The Test Automation Effort Ratio

Many organizations are engaged in some kind of process transformation, introducing Test Automation into their Software Engineering practice. Automation is indeed a key element of Agility, proper delivery pipeline, and any modern way of producing software.

In such a transformation, interest is high in metrics that help to follow progression of work in Test Automation adoption. It might be tempting to define KPI on coverage or amount of tests. Something common is to list “tests that should be automated”, from an existing list of manual tests, or created from scratch. Such a list usually suffers of a problem of relevance. Indeed, because the team is at the beginning of its Test Automation Journey, it is highly probable that the list would include “manual” oriented tests, mainly end-to-end, or based on bad test automation paradigm.

Discovering which tests are the right test to automate is a quest every team have to achieve by its own, because the answer depends on the product being tested, the organization of software delivery, the team appetite concerning risk, and many other factors.

In addition, it is not unusual for teams that are progressing for being more mature, not to be able to collect metrics properly. Lack of reliable datasources, manual data collection or estimations, are a common source of bias that would make precise metrics more a tale than a fact.

But something is common to all team : at a certain level of maturity, teams that are on a Test Automation Journey expect to spend an increasing part of their effort in test automation activities, and a decreasing part in manual testing activities.

Timesheets are more reliable than test coverage

Bogrot, Gringotts gobelin

What I suggest you to introduce a metric to evaluate your team progression in the Test Automation Journey, is the Test Automation Effort Ratio.

The Test Automation Effort Ratio is the ratio between the amount of time spent by your team members on any test automation related activities (scripting, debugging, environment management, reporting…), divided by the total amount of time spent by your team members on any kind of test activities, should it be manual or automated. You should calculate this ratio on every cycle you consider relevant (sprint, release, month…)

\tau = \frac{\sum test\;automation\;related\;logged\;timesheets}{\sum any\;test\;related\;logged\;timesheets}

At the beginning of your Test Automation Journey, the value should be near zero. In your most secret dreams, you expect it to reach one. In the real life, you will see reach something like 0.5 quite easily, and a good maturity in the test automation practice will allow you to obtain 0.7 to 0.8. But instead of setting a specific value to obtain, the right goal to set is to have this ratio increasing over time: this is how you will be sure that you are globally doing the right thing.

This metric has several advantages including the following:

  • Almost every organization are good at tracking time, doing timesheets is unfortunately the most applied practice in IT. It is easy to set up.
  • This metric is completely independent from your context, and its specificities. You are free to decide what to automate, and when you will figure out that everything you have started are not appropriate – because, it is part of Test Automation Journey – it won’t affect the metric, since your effort will still continue to increase.
  • The shape of the curve over time will give you a good idea of “when” you will reach a good level of maturity. It is expected to see your numbers to reach an asymptote, which you will discover with the time.