The State of Testing survey is back. I liked the idea of the first survey when it hit my Twitter feed a year ago. It offered a chance to provice some kind of state report of the current state of the testing business. This year, I look forward to the results even more since there is something to compare with (i.e. last years´ survey). It is always hard to judge the statistical significance of these kinds of surveys since the samples are not chosen at random. However, the more years this will keep on going, hopefully the participation will increase more and more over time, and we will be able to draw more certain conclusions from it. Also, it will be interesting to monitor the trends over time as our business changes. So, why not take part and make the survey a little bit better than it would have been without your participation?
Disclaimer: This is not a suggestion of a best practice. It´s a heuristic that may be useful in the right context.
Striking the balance between different levels of checks can be tricky when working with automated regression checks. On the lowest level we have the unit check which checks small portions of the code, and on the higher levels the system checks and system integration checks that performs checks on big pieces of the whole system at once. It might be tempting to always go for the higher level checks since they “check that the whole system works”. However, these kind of checks are often brittle and requires a lot of maintenance. Unit checks on the other hand are small and lighweight and fairly easy to maintain. They usually don´t require that much of a complex test environment either. So how can I strike a good balance between different levels of checks? By letting those small and cheap unit checks excersize most of the variations and create a few high level tests that excersizes the main flows of the system. In this way you won´t get overwhelmed by maintaining all your precious checks. There is an existing model called the Test automation pyramid that suggests exactly this. It comes in different flavours with different names on the layers, but the general idea is the same. Read more about it here.
This model tells us something about the relation of the amount of checks between the different levels but not how often they are run. Since high level checks often take longer time to run and often require a more complex test environment, it is probably not a good idea to try to run them as often as the unit checks. If trying to do so, it will probably either make you run your unit checks too seldom or your high level checks too often. How can a check be run too often then? Well, if the checks require a complex test environment you might have to invest in parallel environments which could be very expensive and not worth it. If you have some kind of criteria connected to the checks so that nothing can be committed, you might have created a monster of a slow development process which will frustrate everybody. Also, it takes time to investigate the results of the checks so this might leave you with a big pile of results to investigate and it may be hard to keep up.
So, it might be a good idea to run the unit checks more often and the high level checks a bit more seldom, for instance during a nightly build or over the weekend even. In short we have a heuristic that tells us that “the higher level of check, the more seldom it will be run”. If we want to illustrate this heuristic we can imagine stretching out the test automation pyramid along a time axis, creating a prism. Then imagine inserting holes in the prism illustrating a time period where a check is not run. Now, given the aforementioned heuristic, we will have the biggest holes on the top and no holes in the bottom. So what do we end up with? Something looking like the (delicious) chocolate bar Toblerone.
I find the Toblerone model useful when thinking about balanced automated regression checking, I hope someone else does too. The biggest drawback is the craving for chocolate it always brings.