{"id":2787,"date":"2014-11-26T11:30:21","date_gmt":"2014-11-26T16:30:21","guid":{"rendered":"http:\/\/sqlity.net\/en\/?p=2787"},"modified":"2014-11-26T12:08:25","modified_gmt":"2014-11-26T17:08:25","slug":"fragile-tests","status":"publish","type":"post","link":"https:\/\/sqlity.net\/en\/2787\/fragile-tests\/","title":{"rendered":"Fragile Tests – A Ticking Time Bomb"},"content":{"rendered":"
\n

Fragile Tests<\/h3>\n

A Fragile Test is a test that works - most of the time. <\/blockquote >\n

\nA test that fails suddenly, even though the tested code is still correct, is called a Fragile Test. Fragile Tests can undermine everything you worked for so hard when writing your test suite.\n<\/p>\n

Why We Test<\/h3>\n

\nOne goal in unit testing or Test Driven Design is to shorten the feedback loop. The shorter the time between making a mistake and discovering that mistake, the simpler, faster, and most importantly cheaper it is to fix.\n<\/p>\n

\nThere are other advantages in testing, particularly in Test Driven Design, but the shortened feedback loop is one of the biggest if not the biggest advantage of unit testing. A short feedback loop gives us trust in the functionality of our software but even more so, it allows us to introduce change confidently.\n<\/p>\n

\nTo get the benefit of the shortened feedback loop, two things have to happen.\n<\/p>\n

    \n
  1. We need to run our test case often. <\/li>\n
  2. We need to trust the test results.<\/li>\n<\/ol>\n

    \nThe longer the time between test case executions, the longer - naturally - is the time it takes for us to receive that feedback. Therefore, only if we regularly and frequently execute all our test cases we will be able to keep that feedback loop short.\n<\/p>\n

    \nThat we also need to trust the results might seem obvious, but let me dive a little deeper into that and show you how fragile tests can undermine that trust.\n<\/p>\n

    \n\"Fragile<\/a><\/div>\n

    The Disabled Test<\/h3>\n

    \nI once was working with a client that had made an odd observation. Their development staff members were avid unit testers and they had implemented a Continuous Integration (CI) loop that ran all tests automatically every time someone checked in a code change to source control.\n<\/p>\n

    \nWith that process in place their code base was clean and mostly defect free. However, occasionally a defect would slip through the cracks and the frequency of that happening had increased lately. They were aware that even the best unit test suite would not guarantee freedom from defects. But what struck them oddly was that in most cases of these defects there actually was a unit test that should have caught it, but that test was disabled.\n<\/p>\n

    \nTo get to the base of this, I first looked at the tests that had been disabled. I had seen this behavior before and had a hunch. I quickly determined that quite a few of the disabled tests where fragile. What happened was that the developers kept getting emails about CI failures. When they went to investigate, they could not replicate the problem, so after a while they just started to disable these tests. That led to a mindset where the first reaction to an unexpected test case failure was the assumption that the test was broken.\n<\/p>\n

    \nIf the developers were not able to immediately see what the problem was, they just assumed it to be \"one of those flukes\" and they disabled the test so it would not stop them from doing their work. The intention certainly was to go back later to investigate what really was the problem and some of those disabled tests even made it on the backlog, but there was always more important work to be done.\n<\/p>\n

    Avoid Fragile Tests<\/h3>\n

    \nBecause of fragile tests, the developers had developed a habit of disabling failing tests. That habit led to more and more holes in their first line of defense. These holes then allowed more and more defects to slip through.\n<\/p>\n

    \nWhat we all can learn from this experience is, that fragile tests tend to undermine the efforts put into unit testing, so they need to be avoided at all cost.\n<\/p>\n

    The Series<\/h3>\n

    \nSeveral root causes can lead to fragile tests. This post is the first in a series about fragile tests that will discuss ways to identify and prevent fragile tests. Below is a list of the ones already published.\n<\/p>\n

    \n
    \n

    Fragile Tests – A Ticking Time Bomb<\/a><\/h2>\n
    <\/div>\n<\/div>\n
    \n

    Testing Time – Three Ways to Avoid Fragile Tests<\/a><\/h2>\n
    <\/div>\n<\/div>\n
    \n

    Test Data – Why Sharing is Not Always Caring<\/a><\/h2>\n
    <\/div>\n<\/div>\n<\/div>\n\n<\/div>\n","protected":false},"excerpt":{"rendered":"

    Fragile tests can rob you of the benefits of unit testing by undermining the developer’s trust in the tests. Read on and discover what makes fragile tests so destructive.<\/p>\n

    [more…]<\/a><\/p>\n","protected":false},"author":3,"featured_media":2788,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[246,5,27,25],"tags":[247,272,67],"class_list":["post-2787","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-fragile-tests","category-general","category-series","category-tdd","tag-fragile-test","tag-tdd","tag-unit-test"],"acf":[],"yoast_head":"\nFragile Tests - A Ticking Time Bomb - sqlity.net<\/title>\n<meta name=\"description\" content=\"Fragile tests can rob you of the benefits of unit testing by undermining the developer's trust. Read on to discover what makes fragile tests so destructive.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/sqlity.net\/en\/2787\/fragile-tests\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Fragile Tests - A Ticking Time Bomb - sqlity.net\" \/>\n<meta property=\"og:description\" content=\"Fragile tests can rob you of the benefits of unit testing by undermining the developer's trust. Read on to discover what makes fragile tests so destructive.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/sqlity.net\/en\/2787\/fragile-tests\/\" \/>\n<meta property=\"og:site_name\" content=\"sqlity.net\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/sqlity.net\" \/>\n<meta property=\"article:published_time\" content=\"2014-11-26T16:30:21+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2014-11-26T17:08:25+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/sqlity.net\/wp-content\/uploads\/2014\/11\/Fragile_Tests.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"450\" \/>\n\t<meta property=\"og:image:height\" content=\"300\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Sebastian Meine\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@sqlity\" \/>\n<meta name=\"twitter:site\" content=\"@sqlity\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Sebastian Meine\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/sqlity.net\\\/en\\\/2787\\\/fragile-tests\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/sqlity.net\\\/en\\\/2787\\\/fragile-tests\\\/\"},\"author\":{\"name\":\"Sebastian Meine\",\"@id\":\"https:\\\/\\\/sqlity.net\\\/en\\\/#\\\/schema\\\/person\\\/bcffd8c572bc2f1bd10fdba80135e53c\"},\"headline\":\"Fragile Tests – A Ticking Time Bomb\",\"datePublished\":\"2014-11-26T16:30:21+00:00\",\"dateModified\":\"2014-11-26T17:08:25+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/sqlity.net\\\/en\\\/2787\\\/fragile-tests\\\/\"},\"wordCount\":695,\"commentCount\":2,\"image\":{\"@id\":\"https:\\\/\\\/sqlity.net\\\/en\\\/2787\\\/fragile-tests\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/sqlity.net\\\/wp-content\\\/uploads\\\/2014\\\/11\\\/Fragile_Tests.jpg\",\"keywords\":[\"fragile test\",\"TDD\",\"unit test\"],\"articleSection\":[\"Fragile Tests\",\"General\",\"Series\",\"TDD\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/sqlity.net\\\/en\\\/2787\\\/fragile-tests\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/sqlity.net\\\/en\\\/2787\\\/fragile-tests\\\/\",\"url\":\"https:\\\/\\\/sqlity.net\\\/en\\\/2787\\\/fragile-tests\\\/\",\"name\":\"Fragile Tests - A Ticking Time Bomb - sqlity.net\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/sqlity.net\\\/en\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/sqlity.net\\\/en\\\/2787\\\/fragile-tests\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/sqlity.net\\\/en\\\/2787\\\/fragile-tests\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/sqlity.net\\\/wp-content\\\/uploads\\\/2014\\\/11\\\/Fragile_Tests.jpg\",\"datePublished\":\"2014-11-26T16:30:21+00:00\",\"dateModified\":\"2014-11-26T17:08:25+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/sqlity.net\\\/en\\\/#\\\/schema\\\/person\\\/bcffd8c572bc2f1bd10fdba80135e53c\"},\"description\":\"Fragile tests can rob you of the benefits of unit testing by undermining the developer's trust. Read on to discover what makes fragile tests so destructive.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/sqlity.net\\\/en\\\/2787\\\/fragile-tests\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/sqlity.net\\\/en\\\/2787\\\/fragile-tests\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/sqlity.net\\\/en\\\/2787\\\/fragile-tests\\\/#primaryimage\",\"url\":\"https:\\\/\\\/sqlity.net\\\/wp-content\\\/uploads\\\/2014\\\/11\\\/Fragile_Tests.jpg\",\"contentUrl\":\"https:\\\/\\\/sqlity.net\\\/wp-content\\\/uploads\\\/2014\\\/11\\\/Fragile_Tests.jpg\",\"width\":450,\"height\":300,\"caption\":\"Fragile Tests - A Ticking Time Bomb\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/sqlity.net\\\/en\\\/2787\\\/fragile-tests\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/sqlity.net\\\/en\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Fragile Tests – A Ticking Time Bomb\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/sqlity.net\\\/en\\\/#website\",\"url\":\"https:\\\/\\\/sqlity.net\\\/en\\\/\",\"name\":\"sqlity.net\",\"description\":\"Quality for SQL\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/sqlity.net\\\/en\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/sqlity.net\\\/en\\\/#\\\/schema\\\/person\\\/bcffd8c572bc2f1bd10fdba80135e53c\",\"name\":\"Sebastian Meine\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/4ab0a6d02dd494849a584a2c3c8bc3bdcef1d0aa5f87e98bf905dbdb9ad2ce3a?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/4ab0a6d02dd494849a584a2c3c8bc3bdcef1d0aa5f87e98bf905dbdb9ad2ce3a?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/4ab0a6d02dd494849a584a2c3c8bc3bdcef1d0aa5f87e98bf905dbdb9ad2ce3a?s=96&d=mm&r=g\",\"caption\":\"Sebastian Meine\"},\"sameAs\":[\"http:\\\/\\\/sqlity.net\",\"https:\\\/\\\/x.com\\\/sqlity\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Fragile Tests - A Ticking Time Bomb - sqlity.net","description":"Fragile tests can rob you of the benefits of unit testing by undermining the developer's trust. Read on to discover what makes fragile tests so destructive.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/sqlity.net\/en\/2787\/fragile-tests\/","og_locale":"en_US","og_type":"article","og_title":"Fragile Tests - A Ticking Time Bomb - sqlity.net","og_description":"Fragile tests can rob you of the benefits of unit testing by undermining the developer's trust. Read on to discover what makes fragile tests so destructive.","og_url":"https:\/\/sqlity.net\/en\/2787\/fragile-tests\/","og_site_name":"sqlity.net","article_publisher":"https:\/\/www.facebook.com\/sqlity.net","article_published_time":"2014-11-26T16:30:21+00:00","article_modified_time":"2014-11-26T17:08:25+00:00","og_image":[{"width":450,"height":300,"url":"https:\/\/sqlity.net\/wp-content\/uploads\/2014\/11\/Fragile_Tests.jpg","type":"image\/jpeg"}],"author":"Sebastian Meine","twitter_card":"summary_large_image","twitter_creator":"@sqlity","twitter_site":"@sqlity","twitter_misc":{"Written by":"Sebastian Meine","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/sqlity.net\/en\/2787\/fragile-tests\/#article","isPartOf":{"@id":"https:\/\/sqlity.net\/en\/2787\/fragile-tests\/"},"author":{"name":"Sebastian Meine","@id":"https:\/\/sqlity.net\/en\/#\/schema\/person\/bcffd8c572bc2f1bd10fdba80135e53c"},"headline":"Fragile Tests – A Ticking Time Bomb","datePublished":"2014-11-26T16:30:21+00:00","dateModified":"2014-11-26T17:08:25+00:00","mainEntityOfPage":{"@id":"https:\/\/sqlity.net\/en\/2787\/fragile-tests\/"},"wordCount":695,"commentCount":2,"image":{"@id":"https:\/\/sqlity.net\/en\/2787\/fragile-tests\/#primaryimage"},"thumbnailUrl":"https:\/\/sqlity.net\/wp-content\/uploads\/2014\/11\/Fragile_Tests.jpg","keywords":["fragile test","TDD","unit test"],"articleSection":["Fragile Tests","General","Series","TDD"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/sqlity.net\/en\/2787\/fragile-tests\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/sqlity.net\/en\/2787\/fragile-tests\/","url":"https:\/\/sqlity.net\/en\/2787\/fragile-tests\/","name":"Fragile Tests - A Ticking Time Bomb - sqlity.net","isPartOf":{"@id":"https:\/\/sqlity.net\/en\/#website"},"primaryImageOfPage":{"@id":"https:\/\/sqlity.net\/en\/2787\/fragile-tests\/#primaryimage"},"image":{"@id":"https:\/\/sqlity.net\/en\/2787\/fragile-tests\/#primaryimage"},"thumbnailUrl":"https:\/\/sqlity.net\/wp-content\/uploads\/2014\/11\/Fragile_Tests.jpg","datePublished":"2014-11-26T16:30:21+00:00","dateModified":"2014-11-26T17:08:25+00:00","author":{"@id":"https:\/\/sqlity.net\/en\/#\/schema\/person\/bcffd8c572bc2f1bd10fdba80135e53c"},"description":"Fragile tests can rob you of the benefits of unit testing by undermining the developer's trust. Read on to discover what makes fragile tests so destructive.","breadcrumb":{"@id":"https:\/\/sqlity.net\/en\/2787\/fragile-tests\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/sqlity.net\/en\/2787\/fragile-tests\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/sqlity.net\/en\/2787\/fragile-tests\/#primaryimage","url":"https:\/\/sqlity.net\/wp-content\/uploads\/2014\/11\/Fragile_Tests.jpg","contentUrl":"https:\/\/sqlity.net\/wp-content\/uploads\/2014\/11\/Fragile_Tests.jpg","width":450,"height":300,"caption":"Fragile Tests - A Ticking Time Bomb"},{"@type":"BreadcrumbList","@id":"https:\/\/sqlity.net\/en\/2787\/fragile-tests\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/sqlity.net\/en\/"},{"@type":"ListItem","position":2,"name":"Fragile Tests – A Ticking Time Bomb"}]},{"@type":"WebSite","@id":"https:\/\/sqlity.net\/en\/#website","url":"https:\/\/sqlity.net\/en\/","name":"sqlity.net","description":"Quality for SQL","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/sqlity.net\/en\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/sqlity.net\/en\/#\/schema\/person\/bcffd8c572bc2f1bd10fdba80135e53c","name":"Sebastian Meine","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/4ab0a6d02dd494849a584a2c3c8bc3bdcef1d0aa5f87e98bf905dbdb9ad2ce3a?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/4ab0a6d02dd494849a584a2c3c8bc3bdcef1d0aa5f87e98bf905dbdb9ad2ce3a?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/4ab0a6d02dd494849a584a2c3c8bc3bdcef1d0aa5f87e98bf905dbdb9ad2ce3a?s=96&d=mm&r=g","caption":"Sebastian Meine"},"sameAs":["http:\/\/sqlity.net","https:\/\/x.com\/sqlity"]}]}},"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/sqlity.net\/wp-content\/uploads\/2014\/11\/Fragile_Tests.jpg","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p2wXuw-IX","jetpack-related-posts":[],"_links":{"self":[{"href":"https:\/\/sqlity.net\/en\/wp-json\/wp\/v2\/posts\/2787","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sqlity.net\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sqlity.net\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sqlity.net\/en\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/sqlity.net\/en\/wp-json\/wp\/v2\/comments?post=2787"}],"version-history":[{"count":0,"href":"https:\/\/sqlity.net\/en\/wp-json\/wp\/v2\/posts\/2787\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/sqlity.net\/en\/wp-json\/wp\/v2\/media\/2788"}],"wp:attachment":[{"href":"https:\/\/sqlity.net\/en\/wp-json\/wp\/v2\/media?parent=2787"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sqlity.net\/en\/wp-json\/wp\/v2\/categories?post=2787"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sqlity.net\/en\/wp-json\/wp\/v2\/tags?post=2787"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}