Is it Wasteful to Automate non-Regression Tests?

Posted on Posted in Programming

This morning, I saw the following question posted on LinkedIn:

Functional Automation is 100% waste when it will not be useful for regression. Any thoughts ....

I suspect that Bright posted this as a thought experiment rather than an assertion. Why? Because the statement is a vast over-simplification. When it comes down to it, we’re just talking about minimizing the person-hours dedicated to the following tasks:

  • Test Creation
  • Test Execution
  • Analysis of Test Results

When we say that a test is a “Regression Test”, we can do a bit of hand-waving and pretend that its “Test Execution” phase will occur some huge number of times. Obviously, if the “Test Execution” phase is being handled manually, then this will consume tons of person-hours over time. We should look into automating this test right away!

What about the tests that will only be executed a handful of times? Can we save person-hours by writing an automated test that is only executed once? Our instinct tells us that the answer is “No”. In this case, the cost of “Test Creation” will typically outweigh the cost of “Test Execution”. Or, more to the point: the cost of writing the test automation will exceed the cost of executing the test manually. Therefore, it is not worth the effort to write the test automation…

… or is it? In some cases, the cost of executing a test manually is so large that it dwarfs the cost of writing the automation. This is perfectly demonstrated by the Four Color Theorem in Mathematics. In this case, mathematicians knew exactly what needed to be tested to prove the correctness of the theorem; however, there were so many combinations that it was simply infeasible to do this verification manually. This demonstrates that our instinct was wrong: in some cases, it is indeed cost-effective to write automation even if a test will only be executed once.

We can still take this even a step further. Until now, we have been implicitly assuming that manual tests can always be created more quickly than automated tests. In reality, even this assumption is incorrect. Tools such as “mutant” allow for new test cases to be generated by mutating existing tests. Similarly, I have written tools that generated thousands of test cases by simply interchanging equivalent sequences of actions (Ex: Don’t just open the dialog. Open it, close it, then open it again.). In the process, we uncovered numerous state-related bugs that we had never even imagined testing for.

In short, it is a vast over-simplification to claim that a functional test is a “100% Waste when it will not be useful for regression”. We certainly should prioritize automating the regression tests, but there are still many cases in which it is very cost-effective to automate one-off tests.

2 thoughts on “Is it Wasteful to Automate non-Regression Tests?

  1. Hey Brian, Nice article. It actually depends on budget. But again I think it’s a controversial statement. As a matter of fact it doesn’t even make sense. I believe that manual tester will be helpful only to perform Exploratory, Adhoc testing & Preparation of positive, negative scenarios. The Testing process has been improved a lot. We see Many Techniques, Many Tools, Many Frameworks day by day.
    Lets talk about Model Based Testing http://graphwalker.org
    This will reduce manual QC effort. I can’t elucidate it here. Please go through it.
    Similarly Visual Based Testing & ATDD. Aren’t they useful and reduces the manual effort?
    Recently I came to know about the tool Tellurium. That’s amazing tool. Anyone can automate if they speak English. https://telluriumbycharan.wordpress.com/2016/01/13/how-tellurium-made-my-job-easy/
    Even this can used right from first sprint. All we need is test scenarios (Positive & Negative) given by QC.
    Using Selenium we can test GUI and Backend as well (API/Database). It depends on your framework though.

  2. You bring up some good points: nowadays, the line between “Manual” and “Automated” testing is becoming fairly blurred. People are leveraging a lot of “power tools” to assist with the “Manual” testing. It’s like saying “I cut down a tree” when you actually used a chainsaw. Sure, you held the chainsaw, but the chainsaw did most (all) of the cutting.

Leave a Reply

Your email address will not be published. Required fields are marked *