When TDD Doesn't Work

IMHO TDD, like a lot of the agile stuff, is a good idea with solid foundations that people get wrong all the time and end up making things worse with.<p>Agile was supposed to ease up on process and make teams adapt to changing requirements. It wasn&#x27;t supposed to use up &gt;30% of your working time just to service the methodology, but that&#x27;s what it ends up doing when you get in the Agile Evangelists.<p>TDD was supposed to ensure more correct software at the cost of some overhead (perhaps 30%?) by making sure every unit had its tests written ahead of the code. In practice I&#x27;ve seen it kill productivity entirely as people write test harnesses, dummy systems and frameworks galore, and never produce anything.<p>A combination of these two approaches recently cost an entire team (30+) people their jobs as they produced almost nothing for almost a year, despite being busy and ostensibly working hard all year. We kept one guy to deal with some of the stuff they left behind and do some new development. When asked for an estimate to do a trivial change he gave a massive timescale and then explained that &#x27;in the gateway team we like to write extensive tests before the code&#x27;.<p>The only response we had for him was &#x27;and do you see the rest of the gateway team here now?&#x27;

Except for this statement, beware of absolutism in statements of How To Do Software Development.<p>The specifics of this debate are kind of uninteresting because of the (general) lack of nuance from various sides, albeit all informed by their own lived experience.<p>OTOH, the recurrent reality of &lt;insert topic&gt; debate in our industry <i>is</i> very interesting.<p>I think it&#x27;s some combination of:<p>* a bunch of problems are still unsolved<p>* software is so powerful that sub-optimal solutions are usuallly Good Enough<p>* industry amnesia, driven by developer&#x2F;engineer turnover<p>* the relative infancy of the industry, especially as a function of the rate of change (I&#x27;m not sure how you would normalize for rate-of-change, social structure and communication speed, but it would be interesting to compare these debates to medieval guilds in Europe).<p>* ???<p>To take up the first two above:<p>Things are better than they used to be -- as late as the 90s, code reuse was still an unsolved problem. Of course, code quality is still hard--we are reusing broken code, but at least we &quot;only&quot; have to fix it once.<p>I think it&#x27;s hard to overestimate the importance of Good Enough as a factor in these recurring debates. Everyone can be right from the business&#x27;s point of view--tons of money is still being saved. Once you get past the initial ramp of a company, how to structure for continuing velocity of a team and make headway in your chosen market(s) seems like a different optimization problem than what got you there (again, not a new topic!)<p>Just some partially formed thoughts...

While DHH&#x27;s rant has spawned an interesting discussion, it feels to me like he&#x27;s arguing in reverse in defense of his framework.<p>Many Rails apps are tightly coupled, and many unit tests written by developers using rails test 10% program logic and 90% framework features.<p>Of course this is going to be slow. We can argue about hacks to make it faster but at a certain point it&#x27;s a problem whose solutions start to distract us from solving the important problems.<p>If you have a web app and get to write a single test to determine whether it&#x27;s safe to deploy, that test would be an integration test.<p>The decision to write tests more granular than integration tests is a decision to be made based on assumptions about the rate of change of components of the system.<p>TDD is tangential to the above observation.<p>There are many cases where an implementation is easy to figure out (though possibly time consuming) while the optimal interface design is less obvious. TDD can be really useful to quickly iterate interfaces and verify that all the moving parts work as expected together, before worrying about the implementation details... This makes it possible to work on a larger system with more focus on problem solving, fewer mistakes, and less overall cognitive load.

I have a really crazy idea on how you could test the GUI in a way that would both save time and provide something more valuable than just testing true == true.<p>My idea is to create tests that take screenshots and do a diff of them over time. You then could set a change % threshold that would signal a test &quot;failure&quot; signal. Your QA team could then run through that process and see that something significant changed. Maybe that is fine, but maybe that is hugely unintended.<p>Having a Time Machine of screenshots of different processes, you could compare changes easily and see if they are worth further investigation. For example, this would be useful if you change some CSS or JS for just one page, and it ends up breaking another page.<p>The key point of this system is not that it would tell when your system is broken, but rather that there was a significant change that occurred that might have broken something. It&#x27;s not a substitute for human analysis or thought.<p>Is anyone doing something like this and would it be useful to anyone else?

This isn&#x27;t really true. At smartphone OEMs we certainly do have boxes to put the devices in that perform physical tests like on the touch screen, microphones, speakers, antennas, etc.. And in mobile development we have UI automation tests that confirm buttons are certain color, have certain text or state, the right screens popup when pressed, etc. - heck we have a program called the monkey that presses everything that can be in addition to the UI automation scripts. I know the web side of things has Selenium and similar robots. I think he just hasn&#x27;t ever worked somewhere where everything is tested which is reasonable. In many cases you have nothing to do with the OS your software is running on, for example, so there isn&#x27;t as much point in testing beyond what your app outputs to it.

&quot;Over the years many people have complained about the so-called &quot;religiosity&quot; of some of the proponents of Test Driven Development. &quot;<p>And Bob is guilty of that religiosity himself. See here <a href="https://www.youtube.com/watch?v=WpkDN78P884&amp;feature=youtu.be&amp;t=58m" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=WpkDN78P884&amp;feature=youtu.be...</a><p>Jump to the 58 minute mark if it doesn&#x27;t automatically.

&gt;&gt; So near the physical boundary of the system there is a layer that requires fiddling. It is useless to try to write tests first (or tests at all) for this layer.<p>Maybe I can see what he&#x27;s trying to say, but I don&#x27;t think statement alone is accurate.<p>For the GUI i.e. at the human boundary of the system, the most value (especially to stop regression and catch side affects) is often added with tests e.g. automated tests which perform some user function in the GUI and assert the results.<p>Another physical boundary of the system is a database. Writing tests which cross this boundary add a lot of value too.<p>I&#x27;d favour these tests which hit the boundaries and go over them, over a codebase with only unit tests and endless mocking any day of the week.<p>Also these type of tests can be written first. We do it.

This article is a breath of fresh air. I can&#x27;t how many times I&#x27;ve encountered the dogmatic TDD adherent. I write more tests than anyone I know and what I have learned is TDD is great except when the cost of TDD outweighs its benefits. I&#x27;ve seen a dev spend 8 hours fiddling around with Mocha&#x2F;Chai trying to test if a button changes color in response to a successful callback. Sometimes its good enough to click the button and see if it changes color.

We&#x27;ve seen most of this argument before, but the most interesting (new) part of the article is the implication that CSS is the final layer between software and the physical world and thus hard to test. I&#x27;m sure the people on the Mozilla, Chrome, Opera, and IE projects would disagree that CSS is untestable.<p>It seems Uncle Bob implies that it&#x27;s ok to skip TDD if you think it is hard to test something. There are much better reasons for most apps to avoid testing their CSS. Likewise there are many reasons for some projects to have extensive automated testing around CSS even if it might be hard.

&quot;I have often compared TDD to double-entry bookkeeping.&quot;<p>It always seemed to me, if you were to make perfect, automated tests that 100% cover your application, you would basically have reimplemented it. (Or in other words - if you want to check if your calculations are correct, you have to do the calculations again.) Ideal, fully automated tests are basically taking two implementations, run them side-by-side, and compare the results.<p>That&#x27;s why I am not a big fan of tests, in the sense that there is too much focus on them in the SW industry, and they seem like a hammer (useful but overused).<p>I think there should be more focus on writing the _one_ implementation correctly. This can be done with better abstractions (e.g. actor model for concurrency, functional programming, ..) and asserts (programming by contract), and maybe even automated SW proving. I don&#x27;t think these techniques are as popular as testing, but I wish they were more popular, because they let you write programs only once and correctly.<p>[Update: To specifically expand on point about asserts, if you can trivially convert test to assert, why not do it? Unfortunately tooling doesn&#x27;t support asserts as much as tests.]

A good tool for testing visual interfaces is an image diff combined with a manual testing process.<p>Initially the tester will view each screenshot of an application state that is being tested and set that as &quot;passing&quot;. Next an automated test runs and the latest screenshots are compared to the passing screenshots.<p>If they are different then the test fails. A manual tester then needs to take a look at the tests that failed and decide if the test actually failed or if the changes were supposed to be there.<p>If the changes were supposed to be there the tester can make this image the new passing screenshot. Passing screenshots should probably be reset BEFORE the tests are run. I see no reason why not to just check these images in to the repo along with all of the other test conditions.<p>I&#x27;ve been scheming on ways to do video diffs for testing transitions and animations although I&#x27;m not sure if this provides much value. It would be mostly an academic pursuit.

There seems to be two aspects to this discussion.<p>One involves questions of process, enforcing TDD, and whether or not TDD can save a bad team from producing bad stuff. The other is the question of what TDD can offer for a skilled team with an intelligent approach to development.<p>Mixing these aspects leads people to dismiss TDD because they&#x27;ve seen teams fail by doing TDD in a bad way.<p>Another question: is there a way to structure software so that questions of boundaries and collaborators become less troublesome for testing? I think a promising road is in value-oriented programming without side effects.<p>Another way to see that: if you need a lot of tedious mocking to test your unit, maybe the unit should be redesigned to have fewer collaborators, or maybe you should move the complex logic to a pure function, and so on. Maybe TDD difficulties are showing us that there is something wrong with how we write code. After all that&#x27;s what it&#x27;s supposed to do.

So you&#x27;re telling me that being pragmatic will result in sensible solutions? I could do with more blog posts like this!

<i>&quot;So near the physical boundary of the system there is a layer that requires fiddling. It is useless to try to write tests first (or tests at all) for this layer. The only way to get it right is to use human interaction; and once it&#x27;s right there&#x27;s no point in writing a test.&quot;</i><p>This seems dead wrong. There is probably no way to write tests first in this environment, but with so many different browsers interpreting your CSS (to run with preceding example) you need to be aware of when changes in your code cause changes in rendering that might need to be revalidated and further fiddled! I <i>do</i> agree that it doesn&#x27;t fit well with T<i>D</i>D, but it absolutely can work with automated testing.

&lt;standard TDD comment&gt;<p><i>&quot;...software controls machines that physically interact with the world...&quot;</i><p>See, that&#x27;s not always true. I would love it if all software interacted with the outside world. But a lot of software doesn&#x27;t interact -- just take a look at some of that code sitting in your repository sometime. Some of that isn&#x27;t deployed, isn&#x27;t being used. You could test that until the cows come home and have a whole bucket full of nothing.<p>Because the Bobster and the other TDD guys are correct: you gotta test to know that the code is doing what it&#x27;s supposed to. Testing has to come first. In a way, the test is actually more important than the code. If you get the tests right, and the code passes them, the code itself really doesn&#x27;t matter.<p>Where we fall down is when we confuse the situation of a commercial software development team working on WhipSnapper 2.0 with a startup team working on SnapWhipper 0.1. The commercial guys? They are working on a piece of code with established value, with a funding agent in place, with a future of many years (hopefully) in production. Everything they create will be touched and used over a long period of time. The startup guys? They&#x27;ve got a 1-in-10 shot that they&#x27;re alive next year. Any energy they put into solving a problem that hasn&#x27;t been economically validated is 90% likely to be wasted.<p>Tests are important, but only when you&#x27;re testing the right thing. The test for the startup guys is a <i>business</i> test, not a code test. Is this business doing something useful? If so, then just about any kind of way of accomplishing that -- perhaps without any programming at all -- provides business value.<p>That&#x27;s a powerful lesson for startup junkies to assimilate. In the startup world, you don&#x27;t get rewarded based on the correctness or craftsmanship of your code. You&#x27;re looking at one or two weeds instead of realizing the entire yard needs work.<p>Put a different way, we have Markham&#x27;s Law: The cost of Technical Debt can never exceed the economic value of the software to begin with.<p>&lt;&#x2F;standard TDD comment&gt;

I have never found it to work in visual&#x2F;interactive development. A lot of the time you are working on something you evolve as you develop, try, iterate again.<p>I can see it&#x27;s benefits if you have a simpler I&#x2F;O for your code.

TDD for CSS is indeed an odd concept, but it&#x27;s possible to do automated CSS regression testing:<p><a href="http://tldr.huddle.com/blog/css-testing/" rel="nofollow">http:&#x2F;&#x2F;tldr.huddle.com&#x2F;blog&#x2F;css-testing&#x2F;</a>

As a non religious person, there&#x27;s one thing I don&#x27;t get in the whole to TDD or not to TDD debate that&#x27;s ongoing now.<p>Does it matter if &quot;TDD says this or says that&quot;? Aren&#x27;t these methodologies more of &#x27;suggestions&#x27; for us to adopt as it fits our needs, while trimming the stuff that doesn&#x27;t? Once you adhere to a methodology religiously you lose the flexibility and pragmatism that methodology intended to give you. It just becomes systematic Dogma following of a rule book, like any religion.

<i>&quot;How can I test that the right stuff is drawn on the screen? Either I set up a camera and write code that can interpret what the camera sees, or I look at the screen while running manual tests.&quot;</i><p>Or screenshots, of course, which is still relying on code obviously but probably not <i>your</i> code. Of course, defining what you&#x27;re looking for in a screen shot is going to be nontrivial unless you&#x27;re doing simple check that it hasn&#x27;t changed from the last manually-approved version or something.

I am reminded of this somewhat recent discussion when TDD doesn&#x27;t work (which I believe uncle Bob responded to as well)<p><a href="https://news.ycombinator.com/item?id=7130765" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=7130765</a>

Why is it that we need articles like this to tell us that there is no silver bullet and we had better use our brains instead of the Methodology Du Jour? Sometimes, HN just makes me want to cry...

<i>&quot;However, software is different from accounting in one critical way: software controls machines that physically interact with the world.&quot;</i><p>Doesn&#x27;t accounting?

So the acceptance that there are lots of scenarios where one cannot write tests first.<p>A good post for the TDD evangelists I have met so far.

&quot;But if I want to be sure that the bell rings when the proper signals are sent to the driver, I either have to set up that microphone or just listen to the bell.<p>How can I test that the right stuff is drawn on the screen? Either I set up a camera and write code that can interpret what the camera sees, or I look at the screen while running manual tests.&quot;<p>I have nothing else to add