Why We Measured AI Instead of Trusting It

Generating test cases with AI was easy. Working out whether they added value was much harder.

This blog series grew out of a conference presentation exploring how AI-generated UAT test cases could be evaluated through evidence rather than assumptions.

Where we started

AI is everywhere. We’re told AI will revolutionise everything bringing unparalleled efficiency and savings! Happy face!  

Or possibly that it will bring about mass professional unemployment, neo-feudalism and a sea of AI Slop. Sad face!

Either way, my organisation, like may others, is exploring how we can use AI in pretty much every context, including in software testing of enterprise IT systems which is my area.

And we are far from alone in this; looking at the testing conference circuit this year AI was clearly the biggest topic. I should know I was one of the people giving a talk on it.

In our current paradigm shift there is a lot of pressure on people to just “use AI”… Use AI to do what exactly, to solve what problem? Who cares, just use it!

And AI is deceptively easy to start using, be it with specific testing tools sold by specialist vendors or just generic LLMs.  Churning out tests is easy with AI but how useful is it really?

I could easily have just set up an AI pipeline to see how it operates and how I feel about it then told my managers “Yes, we’re doing AI now!” But that’s not exactly a measured or well thought out approach.

I’m a tester, I’m interested in getting solid metrics on how well systems work, applying judgement. That is fundamentally my job and if you’re a test professional it should be yours as well.

Testers shouldn’t run on vibes, they should run on evidence.

I decided to do what I do best and gather evidence to see how effective AI test creation was for me.

What began as a comparison of human and AI-created tests would eventually raise wider questions about quality, human judgement and whether a project is ready to benefit from AI in the first place.

Setting up a study

I run Business Acceptance Testing and User Acceptance Testing on Enterprise IT systems at the BBC. My job involves checking that the various outsourced systems we procure work as intended and do what the end user wants. This frames where I came from in setting up my study; the type of systems I test, the developmental stage they’re at, my resources and so on. 

What works for me might not be what works for you because you’re testing different systems, using different tooling, with different stakeholders and different priorities. My results in terms of value delivered will be different to your results.

Your inputs are different from mine, your results may be different from mine, but the method for evaluating whether AI assisted test creation will work in your environment is transferable.

Setting up a study

The Proof of Concept already existed: AI could generate test cases. This has been demonstrated many times by many people.

But that wasn’t the question I wanted to answer. I wanted to know what value those tests added in the environment where my team worked. What I needed was a Proof of Value study.

The difficult part was not persuading AI to write test cases. It was working out how to recognise whether those test cases had value.

What followed challenged some of my assumptions, not only about what AI could produce, but about requirements, source material, human judgement and what “good” testing means.

Designing the study

With that in mind, how do you design a study that works in your own circumstances? I recommend the following process:

1. Understand your starting point
2. Decide what value means for you and your ogranisation
3. Select key metrics based on those values
4. Set up a fair system of comparison
5. Stabilise your AI approach before you begin measurement
6. Collect and evaluate the evidence from human and AI generated tests

Understanding your starting point

To start with I did some basic analysis of where my team and I were. I ran a SWOT analysis, where I analysed what our Strengths, Weaknesses, Opportunities and Threats were when it came to AI.

I then ran a resource audit where I looked at what was available to work with and what wasn’t available

• team composition and skills
• capacity and workload
• available tools and technologies
• training needs
• constraints and opportunities

Deciding what we valued

Different testing teams are trying to solve different problems. A team under intense delivery pressure might care most about reducing test design effort. A highly regulated environment might prioritise traceability and auditability. Another team might be struggling with test coverage and want to know whether AI helps identify missed scenarios.

Looking at what we valued for us it came down to three broad questions:

• Does AI help us create tests more efficiently?
• Does it create tests of sufficient quality?
• Can it realistically fit into the way we actually work?

Those questions became the foundation for the metrics we selected

Defining key metrics

Once we had defined our values, we could consider the measures that might represent them.

Working with stakeholders and the wider BBC testing community, we generated more than 20 potential metrics. Collecting data for every one of them would have created considerable overhead, so we needed to identify the measures that best represented what we valued.

Setting up a fair system for comparison

Having established what we valued and wanted to measure, we needed a fair way to compare human-created and AI-created tests. That was harder than it sounds.

Once a tester knows that a defect exists, they cannot simply unknow it and independently rediscover it. A perfectly controlled comparison would therefore require different testers working in parallel.

I run on a shoestring, so spare people available to repeat the same work was not going to be easy to find. Our study design had to be credible, but it also had to reflect the resources available in the real world.

Stabilising the AI approach

There was another fairness issue. An experienced tester brought years of context, judgement and learned technique. Our newly established AI-assisted process did not.

We therefore needed to give the AI approach an opportunity to stabilise before formal measurement began. Otherwise, we would have been measuring the immaturity of our first attempt rather than the potential value of the approach.

Collecting and analysing the evidence

Using several measures allowed room for nuance. The outcome would not be reduced to a simple claim that either humans or AI were “better”.

This was not simply a comparison between two sets of test cases. It also raised questions about what experienced testers contribute that is difficult to encode in a prompt or provide through a source document.

What happened next

We had established the question, assessed our starting position and identified the measures that mattered. We had also found a real project that could give us sufficient scale to compare human-created and AI-generated tests.

The eventual results would be specific to our environment. The method for reaching them is something other teams can reuse.

The next step was turning an abstract claim such as “AI adds value” into something we could observe and measure.

---

Next in the series

In the next post, I’ll explain how we defined value, reduced more than 20 possible measures to a practical set, and discovered why the most obvious measures were not always the most useful.