For when you want to go ATDD-, BDD- or SpecByExample-style
Cet article est également disponible en français.
I just love executable specifications! So here is a match between the main tools that can be used to write them.
Let’s pit against each other:
- Cucumber (and its kin)
- Robot Framework
A word of caution: business-people involvement
Beware: I will evaluate these tools with the mindset that they will be used by the business people. Business people will write the tests, not the developers.
If you’re unable to involve the business people into writing tests/executable specifications, then you might as well go with these more lightweight tools.
Don’t bother using a big, cumbersome tool if the only users of the tool are developers…
OK, that has to be said. Now let’s go!
All these tools are able to handle all kinds of scenarios and will allow you to express your specific business concepts and tests. However, the result will be more or less enjoyable depending on your needs, the resulting tests will be more or less readable, and you’ll need more or less code to make them work. Still, you’ll never face the impossible.
On the other hand, Cucumber will be the easiest to learn for the non-technical ones, and is available whatever your stack. These two decisive arguments are making Cucumber an excellent default option.
Keep reading if you wish to make a more informed choice, tailored to your context!
FitNesse is the old-timer of the group. It is loosely based on Fit, which pioneered specification by example and driving development by acceptance tests.
The most noticeable trait of Fitnesse is that it takes the form of a Wiki, a website whose content can be directly edited and formatted from the interface of the website.
Be-monster not thy feature, wer’t my fitnesse — Shakespeare, King Lear
What’s great about the tool:
- A well-known and widely used tool: go check the FitNesse mailling-list and you’ll find answers! The GitHub accounts hosting the project are well maintained and are still moving in spite of the venerable age of the tool. FitNesse is used in many companies. You’ll find books about FitNesse!
- Community-driven only: there is no company behind FitNesse; it is not a product created by some company, sharing them as an open-sourceproject. What happens next to the tool is not tied to any business model of some specific company.
- Easy-as-pie to install: just type java -jar fitnesse.jar and you’re good to go…
- Integrated Wiki, the resulting documentation is directly accessible: you don’t need any tool to edit tests, no link to do between files and the rendered doc…
- The test Runner is available for many platforms: they are plugged into FitNesse by using specific protocols (FIT or SLIM).
- The prettiest decision tables: truly, the result is nice! It pretty much looks like what the business people would have written into an Excel sheet.
- All the space you need to write some beautiful text: the business poeple can expand on the context and explain it in great detail — as much as they want! They can insert images, format text…
What’s less good about the tool:
- An outdated tool: the tool is venerable, and it feels. The protocol to interface the Wiki with the code server (FIT or SLIM) is clearly questionable. It is rather complicated to use the same test base (i.e. the same Wiki instance) to test several products using different technology stacks. And when something is going wrong, it’s not always easy to understand what exactly went wrong and to debug it.
- Irregular language: several types of tables are available for different types of tests, and each table actually has his own syntax and his own subtleties. These differences are rarely justified, they just happen to be that way and you have to live with it.
- The workflow scenarios are barely readable: among the different types of possible tests, the workflow tests (chaining various steps, actions and checks) are notoriously hard to read and maintain.
- Counter-intuitive concepts: you write tests in a Wiki editor, and the Wiki will make HTML out of it. Yet the test is not the text you’ve written but the generated HTML. Let’s say that you write an URL, then by default the Wiki will make an hyperlink out of the URL; and now when the test is run the character strings passed to the test code won’t be the URL you typed but an <a> HTML tag to your URL.
- Using the Wiki is mandatory, test reports and test writing are not text-friendly: using FitNesse outside of the Wiki is neither natural or practical. This is a downside for the developers, unless they use some FitNesse extension for their IDE — at the condition that such a FitNesse extension is available for their language and their IDE. Likewise versioning of the Wiki content does not come naturally since it is the files that are versioned; the versioning options directly integrated into the Wiki are sub-optimal.
- No advanced editor for the non-developers: the developers may have access to some FitNesse integration into their IDE if they are lucky. The non-developers, on the other hand, have no other option than using the Wiki which is still a rather rudimentary editor. What I really mean is that there is no completion available.