This morning, I saw the following question posted on LinkedIn:
I suspect that Bright posted this as a thought experiment rather than an assertion. Why? Because the statement is a vast over-simplification. When it comes down to it, we’re just talking about minimizing the person-hours dedicated to the following tasks:
- Test Creation
- Test Execution
- Analysis of Test Results
When we say that a test is a “Regression Test”, we can do a bit of hand-waving and pretend that its “Test Execution” phase will occur some huge number of times. Obviously, if the “Test Execution” phase is being handled manually, then this will consume tons of person-hours over time. We should look into automating this test right away!
What about the tests that will only be executed a handful of times? Can we save person-hours by writing an automated test that is only executed once? Our instinct tells us that the answer is “No”. In this case, the cost of “Test Creation” will typically outweigh the cost of “Test Execution”. Or, more to the point: the cost of writing the test automation will exceed the cost of executing the test manually. Therefore, it is not worth the effort to write the test automation…
… or is it? In some cases, the cost of executing a test manually is so large that it dwarfs the cost of writing the automation. This is perfectly demonstrated by the Four Color Theorem in Mathematics. In this case, mathematicians knew exactly what needed to be tested to prove the correctness of the theorem; however, there were so many combinations that it was simply infeasible to do this verification manually. This demonstrates that our instinct was wrong: in some cases, it is indeed cost-effective to write automation even if a test will only be executed once.
We can still take this even a step further. Until now, we have been implicitly assuming that manual tests can always be created more quickly than automated tests. In reality, even this assumption is incorrect. Tools such as “mutant” allow for new test cases to be generated by mutating existing tests. Similarly, I have written tools that generated thousands of test cases by simply interchanging equivalent sequences of actions (Ex: Don’t just open the dialog. Open it, close it, then open it again.). In the process, we uncovered numerous state-related bugs that we had never even imagined testing for.
In short, it is a vast over-simplification to claim that a functional test is a “100% Waste when it will not be useful for regression”. We certainly should prioritize automating the regression tests, but there are still many cases in which it is very cost-effective to automate one-off tests.