This risk mitigation strategy is made of several parts :
- The changelog
- Risk analysis
- (Internal to the team) Regression testing plan
- (External to the team) Validation by partners
- Push to production strategy
What have we changed?
The changelog is the exhaustive set of changes made by this release.
It was at first written by hand by the developers but later they just made sure they write proper GIT commit messages and now this changelog is simply a dump of the set of commit messages.
What can be possibly broken or impacted by this set of known changes?
Performing a risk analysis is a very special train of thought, one that is typically alien to developers.
If you ask this question to developers, they will immediately think of how to validate the changes which is completely missing the point: these new changes are probably the most thoroughly tested parts of the software right now as everybody tested them over and over before getting to the point where the changes will be released.
Instead you want to check everything that is around the change but is not directly part of the change itself. To manage impacts.
This is one of the most valuable skills of the testers: performing risk analysis is at the core of the tester’s skill set. Developers can learn this skill too (which will make them better developers) but having a tester around to teach the whole team is a big plus. Pairing is your best friend.
Regression testing plan
What tests will we perform to mitigate the risks that we identified?
Again, the goal is not to validate the changes, but to make sure that the changes have not broken anything.
The regression testing plan is devised from the risk analysis:
- The set of test scenarios that must be tested to cover the area at risk
- The set of devices/OS/browsers that must be tested, from a device compatibility point of view
This could also include exploratory testing charters.
Validation by partners
What checks do we recommend partners to do?
We are not working alone. We have several kinds of partners:
- We are producing a library that will be used by internal customers
- Some features require a formal validation by some third-party company to certify that our product follow the market specifications
- Some features are critical to the business path of partners, so they want to make sure we don’t break anything on the critical business features
How do we handle these partners? How do we involve them just enough? While we can’t bypass them, we don’t want to involve them too much either as it would otherwise impede our speed. Indeed, validation by partners is usually a lengthy, blocking process.
In this section we propose a compromise between full involvement and total avoidance of the partners. In any case, this is only a recommendation: it is up to the partners to decide if they want to follow it or not.
It is essential to provide the partners with the full risk mitigation strategy. Otherwise they won’t be able to choose between following and not following our recommendation. It is only with all the information in hand that they are able to compromise and not to go the full regression testing route.
Push to production strategy
Will we push to production incrementally or all-at-once? If we go the incremental way, what are the steps?
Finally, we propose a push to production strategy.
At the moment our tooling is not the most appropriate for real incremental deployment, so we might want to avoid the hassle for trivial changes and speed-up the release pace instead.
There are also cases where incremental push to production does not really help detecting problems early.
It also happened that our partners asked explicitly for an incremental push to production instead of running any regression testing on their side.