Unit testing sucks

Testing is, obviously, crucial to the software development process. I am pretty sure that if I wrote a hundred lines of code without running the code even once, it probably wouldn’t do what it’s supposed to do, if it even worked at all without crashing.

So testing, in general doesn’t suck. It’s unit testing that sucks. As you probably know, otherwise you wouldn’t be reading this article, unit testing is the concept that you can write additional code to test your code. Yes, this is something only a hardcore geeky programmer could love: writing code that has no purpose except to run other code. The theory is that your unit tests can be so thorough, testing every single unit of code, that the result is a rock-solid bug-free application.

If only it actually worked as well as the theory! In practice, unit testing is a monumental waste of time. Even worse than a waste of time, the result is often lower quality software that’s harder to maintain. The problems with unit testing are as follows:

(1) It’s impossible to test all possible permutations. You might have to write an infinite number of unit tests. Unit testing advocates may point out “well, it’s better to unit test what you can.” I don’t think so. Time wasted on unit testing could instead be spent by the skilled programmer to improve the quality of his riskier code. The skilled programmer has an intuitive sense of what parts of his code are likely to cause problems.

(2) Whatever you figure out needs to be tested is stuff you would code to work correctly anyway. The bugs that have always showed up in the applications I’ve worked with are unanticipated combinations of user actions or circumstances.

(3) The most important part of the application is the user interface. That’s the part of the program that the user interacts with—programs, after all, are meant to be used by people and not by unit tests. The interface tends to be the most prone to bugs, and it’s impossible to test via unit testing because using the interface the way a real person would use it is just outside the realm of what you can code.

(4) Writing unit tests is incredibly boring. Unit testing advocates might respond, “well that’s too bad, it’s necessary.” But it’s not necessary, and forcing your programming staff to do really boring tasks results in sapping their motivation thus lowering their productivity.

(5) The biggest problem I have with unit testing is its reliance on design patterns concepts. The standard test driven software project requires you to create various application “layers,” with each layer being an interface that receives other interfaces. This allows you to create mock objects that you can substitute for real objects, allowing you to test a small piece of code in isolation from the rest of the project. In order to make your application unit testable, you must make the application a lot more complicated, and complexity introduces errors. Ironically, the unit tests, if they discover errors, are probably only discovering errors that wouldn’t have happened in the first place if you weren’t doing test driven development with overly complicated design patterns.

I know that a lot of people will remain unconvinced. They will think, “I have worked on projects with unit testing, and it has found so many bugs!” But as I said in point five above, the bugs found are often bugs that wouldn’t have been there in the first place were it not for the unit tests. And unit testing generally only finds the big bugs which would have been obvious soon enough when you ran the program.

There is also the theory that unit testing ensures that, when you change some code, it doesn’t break other code. Well that’s only true if it breaks something in a way anticipated by the unit test. But you have to ask yourself, why is your project architected in such a way that it’s so susceptible to a small code change in one place breaking code elsewhere? No, it’s not because of “tight coupling” or failure to follow object oriented design patterns. It’s because of bad planning, and bad developers who confuse unit testing with quality coding. It’s because too many developers are working on the same parts of the project. Too many programmers spoil the code just as surely as too many cooks spoil the broth. The only way to create a quality big project is to divide it up into independent small projects.

Comments

[Mark Hoffman]

Well, you took me up and did write a reasoned argument against unit testing. You're wrong. :'), but at least you explained yourself.

Here's where you are mistaken:

"The skilled programmer has an intuitive sense of what parts of his code are likely to cause problems."

Really? Then how do you explain software defects? If the programmer knows what parts of his code are going to cause problems then we shouldn't have any bugs. Oh..but we do. I wonder why? Lack of skilled programmers? What about when developers have to take responsibility for legacy code? When a developer has to make significant changes to complex code that they didn't write? How do they know that their changes haven't broken something else in the system? (You seem to be a fan of tight-coupling, so surely you are familiar with those types of defects.) Without unit tests, how do you tests those changes? Running the app and checking the results manually? Wouldn't it be a better use of resources to automate those tests?

"Whatever you figure out needs to be tested is stuff you would code to work correctly anyway"

Yes, you would. And then your test would stand as a testament to say "This is how this code should *always* work." With the unit test in place, you will now know when someone comes along and mucks up your code in such a way that the test no longer runs. It's about protecting your code for future changes.

"The interface tends to be the most prone to bugs, and it’s impossible to test via unit testing because using the interface the way a real person would use it is just outside the realm of what you can code."

Wrong. You dismiss design patterns, but a handy one that you should read up on is MVC. Rather than have all of your business logic crammed into the UI, your UI is a very thin layer that is manipulated by a controller. The controller is testable. So yes, it is entirely possible to unit test a great portion of the UI by testing the controller. (Not to mention that there are a slew of testing products that do in fact test the UI through unit tests.)

"But it’s not necessary, and forcing your programming staff to do really boring tasks results in sapping their motivation thus lowering their productivity."

Professionals value quality code that is easy to maintain. Spending time in the debugger tracking down elusive bugs is demoralizing. And without unit tests, that is exactly what you are condemning them to do.

"The standard test driven software project requires you to create various application “layers,”

No. It doesn't. Good design might dicate that, but TDD does not.

"you must make the application a lot more complicated, and complexity introduces errors."

Complex how? Can you provide a code example and show how making something testable makes the code more complex? I think our definitions of complex differ. But is complexity the only factor? What about maintainability? That is of huge importance to me. I'll take additional complexity if it means the code will be easier to maintain down the road.

"And unit testing generally only finds the big bugs which would have been obvious soon enough when you ran the program."

So are you advocating running the entire program, in all it's entirety, every time you commit a checkin? Wow. That sounds pretty inefficient. Why not have unit tests that are automatically run each time a checkin is committed?

In our shop, we use CruiseControl.net to build our app upon each checkin. It then runs all of the unit tests in that project. Not surprisingly, it will identify tests that no longer pass when someone checks in a change. That is a defect that would have gone undetected without a unit test in place. Sure, I could hire a QA team and have them manually test every square inch of the app every day, but I'd rather spend that money on hiring additional professional developers that know how to build testable code.

[Gareth Williams]

I could not disagree more. Unit testing is a huge asset to any complex software project. I have used it succesfully for 8 years. My current company has a large java framework and the unit tests give all my developers the confidence to add new features becuase the tests tell them if they do something stupid.

To address your points individually:

(1)

"It’s impossible to test all possible permutations."

Unit testing means testing as small a portion of the code as possible within each test. Ideally only a single class. If you cant test all permutations of a single class, you need to work harder at your design skills.

"You might have to write an infinite number of unit tests."

Nope, you write tests using experience and skill (plus code coverage tools such as cobertura). I have never come across a project that was infinite in its functional requirements.

Time invested in unit testing fixes problems before they get too embedded in the code - this is vital when you have several developers working on the same project

(2)

"Whatever you figure out needs to be tested is stuff you would code to work correctly anyway."

How do you know it works correctly? Surely writing a well-defined test is better than the alternative - some vague intermittent bug that QA might notice and no one can figure out.

"The bugs that have always showed up in the applications I’ve worked with are unanticipated combinations of user actions or circumstances."

Maybe you're a GUI developer, fair enough - unit testing is not a replacement for acceptance testing. Maybe your apps are CRUD webapps and not so heavy on business logic. Again fair enough, if all you do is shuffle data from a db to HTML and back you're not going to get a lot out of unit testing unless you have "business logic" in the code. But if those are the type of apps you work on, that hardly means unit testing sucks. its a tool to be used where appropriate.

(3)

"The most important part of the application is the user interface. "

OK you must be a GUI/ web app developer. see my previous comments for (2)

(4)

"Writing unit tests is incredibly boring"

Maybe you're in the wrong line of work?

"But it’s not necessary"

It is! Really, it is.

"forcing your programming staff to do really boring tasks results in sapping their motivation thus lowering their productivity"

Really? All the good developers I have worked with love it.

(5)

"The biggest problem I have with unit testing is its reliance on design patterns concepts."

No - you dont need to learn GoF design patterns to unit test. But you will arrive at many of the same conclusions, and learn much more along the way than just reading the book.

"The standard test driven software project requires you to create various application “layers,” with each layer being an interface that receives other interfaces. "

This is just good proven software architecture and has been around a lot longer than unit testing. You're mixing ideas up here.

"Ironically, the unit tests, if they discover errors, are probably only discovering errors that wouldn’t have happened in the first place if you weren’t doing test driven development with overly complicated design patterns."

In my experience, this could only happen if developers lack the skill neccessary. My conclusions from working with a lot of developers is the exact opposite.

"I know that a lot of people will remain unconvinced."

Well, yep, i think you're in a rather small minority

"unit testing generally only finds the big bugs which would have been obvious soon enough when you ran the program."

Nope, it finds others that as I mentioned might only occur under certain combinations of input data. These might only occur occasionally yet completely screw up your customer. Best to find them before you go live, cos QA often miss these types of bugs, and customers hate getting bugs delivered, its bad for business.

"There is also the theory that unit testing ensures that, when you change some code, it doesn’t break other code. Well that’s only true if it breaks something in a way anticipated by the unit test. "

Well yeah, a good developer writes good tests and a mediocre developer writes crappy tests. There is no silver bullet, there is no known antidote to stupidity or laziness.

"But you have to ask yourself, why is your project architected in such a way that it’s so susceptible to a small code change in one place breaking code elsewhere? "

Well imagine you had to store a string in a database eg a product code and someone broke the utility class that created the project code eg misplaced characters. Now, QA should pick it up but might miss it whereas a test would spot it straight away. And if the "product code" object was used as part of a class library say, it could be used across 10 actual apps...so there you go 1 tiny change 1 enormous headache...

For me, its Java + JUnit + Maven + Continuum...
I have been there, tried that for the other approach (no unit tests, no build process, etc) and will never work that way again.

I think there is a lot of BS and snake oil in software, it affects everything MS Linux Agile SOA whatever, but in this case, regardless of how much you choose not to evangelise about unit testing, your arguments don't stand up against the real world experiences of many developers most of whom do not go all "happy clappy" about this stuff but do want to get their job done efficiently, and dislike fixing bugs so much they realise the work required to write unit tests is worth it to save stress and hassle later!

[Andreas T]

amen!

[toolbeing]

@Mark Hoffman:

You asked how making code unit-testable makes it more complex - well, it's because you have to break up your classes into smallest smithereens, which results in more classes and thus more object interactions. That's not so bad in one direction - more modular is good - but now there is more (and more inessential) stuff for someone new to find their way in.

[Bitter]

Couldn't agree more - I'm in the middle of a project right now with a team of ~10 developers that is in an absolute train wreck because of the unit testing "dogma". To be fair, I've done TDD work in the past quite frequently (the biotech and pharma markets love the quantitative evidence it produces), and while I do agree it is *very* beneficial to test, I've arrived at the same conclusion: unit testing (by definition) is coupled too tightly with certain design patterns (ironic, isn't it?). And yes, for *some* projects, I may arrive at the same conclusions for reducing coupling, etc., but not every project requires it (yes, even some big ones where the problem set dictates that decoupling is not required).

In my opinion, it is a weakness in the unit testing tools themselves that we are living with currently. Show me a tool that lets a team write code as they deem correct and test it. To my knowledge, Typemock is one of the only ones that is close - but nobody seems to mention it (perhaps because it isn't free??).

I don't think anyone argues about the merits of unit testing and/or TDD. However, it is total BS that it _also_ dictates how you do your daily job. The last time I checked, there still was some artisan skill required to write code. Time and time again I'm witnessing teams turning to fads like this with hopes they'll solve everyone's maintenance problems, when, in fact, the real issue has nothing to do with technology. Things like organizational habits, individual dicpline, and "drive" cannot be substituted in - these items need to be present in the team dynamic first.

Although, I suppose, just like Agile, perhaps the benefit of unit testing philosophies is merely to showcase the weaknesses in team development efforts - I could grok that. Trouble is, the "religion" isn't taught that way.

Cheers.

[Megacan]

Unit testing is not meant to rid the code off all bugs. Unit testing is one of the testing tools used to test code.

Unit testing tests the code at a very low level, you need additional testing tools and procedures do effectively test all your code.

I think its kind of missing the point to dismiss Unit testing based on the assumption that its objective is to rid your code of all bugs.

[Kelvin]

There are times when test-driven development and unit testing are useful but there are a lot of times when it's a complete waste of time and a distraction.

I work with a team where there are a few coders who are constantly getting sucked into the latest programming fads and rewrite everything they do about ten times trying to get the first step to be perfect. They have certain idols in the coding community that they worship and take everything that they say as gospel that cannot be questioned.

That would be fine if those same guys were the most productive and produced the highest quality final products but unfortunately they tend to be very slow and out of touch with the reality of the project.

[Andy Brice]

Slightly OT, but do you realise http://contrariansoftware.com/ (which is what I see when I follow back a link from http://www.successfulsoftware.net) goes to a parking page? I think you need to tweak your web-server settings.

[retry]

Unit testing is great for avoiding regressions (Regression testing). But that's it. It's UTTERLY STUPID to test before you solve the problem. Solve the problem. Write test cases that deal with the solved problem, and then regressions will get detected everytime. All the TDD development I've seen leads to inefficient, unoptimized design where the coder is worrying more about how it performs in a test than in the real world. It's clearly DOGMA, and all dogma should be distrusted, including this sentence. I'm so sick of religion taking over coding practices telling us how to get the job done. What if people approached painting and film-making this way? You wouldn't have Van Gogh or Jim Jarmusch at all. Just a bunch of lousy Michael Bey's. UGH! Software design is a creative craft. I'm sick of these religious fanatics poisoning the job market with their stupid fad of the week mentality.

[Slevdi]

@retry 'stupid fad of the week mentality'

I started coding in 1970. I have always written separate tests for my code, whether it was assembler, Fortran, BASIC, Perl, VB, VBScript, Javascript, Java or C#. When I realised that I could write the tests first, my world changed. My code got simpler. It took less time to create working code. I could change my code and know it still worked.

@retry 'UTTERLY STUPID to test before you solve the problem'
This is stating the obvious. You need to know what you want to do before writing tests to prove you have done it.