Conseil, audit, formation, mentorat et plus avec Jean-Pierre Lambert
Why test automation fails
… and how YOU will succeed!
Today, automating tests is an obvious choice for most teams and project. Products are always more complex, leading to always bigger non-regression test plans. Running such a non-regression test plan manually requires a lot of people, dedicated to this task. But most projects can’t afford, or do not want, to pay that much people.
So many teams decidedly went for automating their tests — and that’s a good thing! But too often, attempts are failing. Why is that? I’ll try to give you the biggest reasons why such test automation attempts fail. The good news is that, now that you know, you can avoid these pitfalls!
Pitfall: The team did not try hard enough
Teams that just started automating their tests quickly find out that it is much harder than expected. Things don’t come cheap, and they are not willing to pay that much. As a result, many teams just give up before they can take advantage of automated tests.
This is particulary prevalent when working on legacy code, as testing code that has not been designed to be tested is the hardest. Basically, you need to refactor legacy code to enable proper testing.
Implications of this pitfall
As a result of this pitfall, there may not be any automated test at all. The team may end up not writing even the simplest unit tests, even on new code.
Or maybe an even worse situation, is when the team has a set of old, out-of-date tests that they struggle to maintain and steals their time on a daily basis. The worst part is that they probably don’t care when the tests fail!
Bad smells revealing this pitfall
“This code cannot be tested.”
Did you really try? Maybe you wanted to say: “I don’t know how to make this code testable.”
“We spent so much time writing a single test, that we stopped.”
Listen, I know it’s not easy. Maybe you started too big, too fast?
“You know, we are working in <insert whatever industry/technology/team specifics here> so automating tests is not worth it for us.”
Usually heard from people that do not believe in Agile or Scrum either, and use the same words on those topics. Bad faith is common here.
How to avoid this pitfall
So, having automated tests is not always easy. Indeed you have to pay a lot at the beginning before you can reap the benefits. But it is clearly worth it. Actually, it’s not like you have any other option at your disposal… Unless your project is very simple, you can’t guarantee quality without testing it, and you can’t be competitive by testing manually. Even if most of your testing is done by people, you’ll save a lot by having a minimum set of automated tests, checking that the basics work. OK, let’s stop the fairy tale; there is another option: skipping on quality. This is actually what happens on these projects where automated tests are inexistent. Obviously this is not sustainable.
Back to the subject, writing automated tests is hard, especially on legacy code. This is true mainly because it requires skills that your team may not have yet. So your team should train. I can only recommend books like “Working effectively with Legacy Code” which directly adress this issue. Your team can also get help from outside Software Craftsmen who will gladly help you or deliver trainings about how to write tests and how to refactor. As your team will write tests, they’ll get better at it, and not only they’ll reap the benefits of having automated tests but it’ll also be easier and easier to write more tests. Hang on!
Benefits of having automated tests on your project
Peace of mind:Pushing to production with no fear of the regressions. Not being paralyzed by the fact that you don’t know everything about every parts of the system: the test harness will warn you of any wrong assumption you had. Less bugs, meaning less time spent debugging and an improved reputation of quality. Release more often and reduce significantly the time needed to answer to an urgent request. All in all, the team becomes more predictable.
Better functional design:When performing a functional change, the test harness will highlight any unforeseen impacts. As a result, you will be able to either know the limitations of the change, or to provide a better functional change request.
Better code design:Dependencies in a piece of code is what makes it hard to test and maintain, and slow and costly to run. So basically the main skill you’ll learn when writing tests is how to remove and minimize dependencies between pieces of code, so that writing your test is easy and that the resulting tests are efficient and maintainable. It just happens that minimizing dependencies is one of the best ways to devise good code designs. In other words, writing tests will naturally get your code in a good shape.
Better productivity:That may not sound obvious at first as you will focus on writing tests in addition to production code. But as your automated test coverage increases, your productivity will actually increase until it becomes stellar. This is obviously due to the other benefits but also because it shortens significantly the feedback loop: you know when you broke something almost immediately, before you switch to another task. It means that what should be a 5-minutes fix, can indeed be fixed in 5 minutes.
What I’d recommend is to start easy, for instance with automated acceptance tests on new code. Avoid legacy code at first if possible and address it later. Even though your pain come mainly from legacy code, the team needs to learn the basics first. Also, depending on the kind of project you are working on, it may make little sense to cover code that is not meant to change.
On the other hand, if that’s really not possible to avoid legacy code — actually a pretty common case on some projects that build new changes and code on top of existing, legacy code — you may try to train first on non-production code, The goal would be to produce a complete project sample where good coding practices are enforced and which everybody in the company can use afterwards as a reference of how things should be done. The simple fact of seeing that it is possible will have a tremendous effect by itself on convincing people to keep trying until they succeed.
A good recommendation is to try to do things diffently. Peer-programming can be real life-saver when writing tests for the first time. Switch people from teams for a few days. Why not gather several teams together for a single day or afternoon to try to address the issues you encounter? All these initiatives will be incredibly helpful to work differently, to keep going forward, to simply think differently and to find the way out.
In short: do not give up. There is light at the end of the tunnel, and it is definitely worth it. The first steps may be incredibly hard, but every step after that will be ever easier.
Pitfall: Keeping the same organization
A lot of teams do not realize at first, or ever, that this is not only about automating tests. You need to work differently in order to achieve a succesful automated testing strategy, which is sustainable and that actually gets you the maximum benefits.
Implications of this pitfall
Here is the usual, traditional organization and goal of automating the test process: