Category: Metrics

The Test Automation Effort Ratio

Many organizations are engaged in some kind of process transformation, introducing Test Automation into their Software Engineering practice. Automation is indeed a key element of Agility, proper delivery pipeline, and any modern way of producing software.

In such a transformation, interest is high in metrics that help to follow progression of work in Test Automation adoption. It might be tempting to define KPI on coverage or amount of tests. Something common is to list “tests that should be automated”, from an existing list of manual tests, or created from scratch. Such a list usually suffers of a problem of relevance. Indeed, because the team is at the beginning of its Test Automation Journey, it is highly probable that the list would include “manual” oriented tests, mainly end-to-end, or based on bad test automation paradigm.

Discovering which tests are the right test to automate is a quest every team have to achieve by its own, because the answer depends on the product being tested, the organization of software delivery, the team appetite concerning risk, and many other factors.

In addition, it is not unusual for teams that are progressing for being more mature, not to be able to collect metrics properly. Lack of reliable datasources, manual data collection or estimations, are a common source of bias that would make precise metrics more a tale than a fact.

But something is common to all team : at a certain level of maturity, teams that are on a Test Automation Journey expect to spend an increasing part of their effort in test automation activities, and a decreasing part in manual testing activities.

Timesheets are more reliable than test coverage

Bogrot, Gringotts gobelin

What I suggest you to introduce a metric to evaluate your team progression in the Test Automation Journey, is the Test Automation Effort Ratio.

The Test Automation Effort Ratio is the ratio between the amount of time spent by your team members on any test automation related activities (scripting, debugging, environment management, reporting…), divided by the total amount of time spent by your team members on any kind of test activities, should it be manual or automated. You should calculate this ratio on every cycle you consider relevant (sprint, release, month…)

\tau = \frac{\sum test\;automation\;related\;logged\;timesheets}{\sum any\;test\;related\;logged\;timesheets}

At the beginning of your Test Automation Journey, the value should be near zero. In your most secret dreams, you expect it to reach one. In the real life, you will see reach something like 0.5 quite easily, and a good maturity in the test automation practice will allow you to obtain 0.7 to 0.8. But instead of setting a specific value to obtain, the right goal to set is to have this ratio increasing over time: this is how you will be sure that you are globally doing the right thing.

This metric has several advantages including the following:

  • Almost every organization are good at tracking time, doing timesheets is unfortunately the most applied practice in IT. It is easy to set up.
  • This metric is completely independent from your context, and its specificities. You are free to decide what to automate, and when you will figure out that everything you have started are not appropriate – because, it is part of Test Automation Journey – it won’t affect the metric, since your effort will still continue to increase.
  • The shape of the curve over time will give you a good idea of “when” you will reach a good level of maturity. It is expected to see your numbers to reach an asymptote, which you will discover with the time.

Team confidence is the best readiness indicator

If you have to keep only one metric for evaluating the product readiness, I argue that collecting development team’s confidence should be that one.

In development team, I include what are usually called devs, testers, QAs, DBAs, devops, sysadmin, and whatever name you use to identify those who are involved in the product construction.

I don’t trust test coverage measurement, since coverage means so different things depending on how you mesure the base (what is supposed to be covered), and how you consider a coverage is done (how many or how deep you’ve tested the base).

I don’t trust test completion either, because if you haven’t listed appropriate tests then completion means nothing.

I don’t trust “definition of done” fulfilment, for same reasons as before.

I trust teams’ feelings more, and team’s confidence, because humans involved in such intellectual process as developing a product, always give part of their heart and soul in what they do. Their confidence is a way better indicator in many situations.

How to measure it

Good news: that’s outrageously simple. Set a board next to the exit door of the floor. Every evening before leaving, any involved individual may get a token – sticky note, card, pin, whatever you want – on the board. Left side means: “I don’t think the product is ready enough for release”, right means the opposite. You can decide a token should be assigned to a person, by using any mark or avatar, or you can decide it should be anonymous, it’s up to you. You can use electronic vote, special survey plugin on your ticket tracking system, multi-value scales or analogic measurement, that’s equal. But measure your team confidence.

By end of the sprint, you should see tokens migrating one after another to the right side of the board. Not all of these will always be on the right side, but the vast majority of it, for sure. The ones staying on the left will probably be here because of some imperfection that the team should accept as a debt for next sprint, or unseen problems by other teammates and corrected ASAP.

Team confidence is the most valuable measurement

I think we struggle too much to get objective measures for our decisions. In many cases, because the element we are trying to measure relies on human thinking, these measure remain highly subjectives. A common example is test completion: the listed tests are the ones that a human thought useful, not the one which were. Then, measuring the completion of those tests, which can be thought as being objective, is really not.

Measuring team confidence is probably the most valuable subjective measurement we can do to evaluate product readiness.