Moving from “why measure AI tests?” to “How do we measure them?”

In my previous blog post, I laid out why I wanted to evaluate how useful AI could be in my test creation process. I wanted evidence I could use to make informed and defensible decisions about what worked well and conversely what didn’t. As testers we should not run vibes, we should work on evidence.

I also gave a high-level account of the study; the initial analysis, the search for useful metrics, the challenge of creating a fair comparison and the project that gave us the opportunity to put the idea into practice.

In this post I want slow down and examine the design properly. How did we establish what mattered to us? How did we avoid creating a study that just confirmed what we wanted to hear? Most importantly, how did we turn the broad claim that “AI adds value” into evidence we could actually collect?

This isn’t a universal recipe. Your systems, stakeholders, constraints and priorities will be different from mine. My aim here is to share a way of thinking you can adapt to your own circumstances.

Understanding your starting point

In my previous post I briefly mentioned that I began with a SWOT analysis and a resource audit. They deserve more attention because they didn’t just serve to document where we stood, they also influenced what we choose the measure and placed practical limits on what the study could support.

SWOT Analysis

A SWOT analysis asks you to consider strengths, weaknesses, opportunities and threats. It is a simple technique, and a general-purpose LLM can help organize the first pass, provided you review the result and remove or generalize anything confidential before sharing it, which is something I had to do here.

One thing that flew out of the SWOT table was access to SMEs. For my work their knowledge is a key strength, but this dependency is also a weakness. This meant that using AI to help reduced the time I would need them for would be a worthwhile exercise. It also meant that SME time could not be treated as an invisible or unlimited resource when we designed the study.

Resource Audit

Running the resource audit brought the practical constraints I had to work with into focus. I looked at the people and their expertise levels and experience. Our skill base in terms of methodology, tooling and domain specific knowledge. The capacity we had and our pipeline of work; peak workloads, known bottlenecks. Our tools and technologies; what we had to work with. Our training needs and budget to gain the new skills we need. Other risks and constraints; capacity risks, compliance risks (think GDPR), broader risks (the BBC funding model). Finally, opportunities; where could we adopt AI, how could we optimize our processes.

This mattered because every extra AI tool required another controlled pipeline and, ideally, fresh eyes for each comparison run. The audit therefore helped shape the number of tools we could examine, the comparison we could realistically run and the evidence we could capture reliably. It also once again highlighted the role of SMEs.

Starting point summary

The SWOT analysis and resource audit gave us two things; a clearer view of what “value” might mean in our context and a realistic understanding of what the study could sustain. It’s worth noting that there are other techniques that can be used at this point. If you work in a field with more external influences, such as a need for regulatory compliance then running a PESTLE analysis will be useful. A good source for looking at what tools will help is your business analysis team. This is their area of expertise.

For now we had enough to establish the principles that would govern our survey.

Principles for defensible evidence

Principle 1: make the comparison fair

Making a like for like comparison involves more than just giving the same source documents to a human and as an AI input. We have to consider differences in experience, who will be running the tests, what those testers already know and how closely the comparison would reflect the way UAT is performed.

  • A human test professional with years of experience has a higher maturity level than a newly created AI pipeline
  • Human created and AI created tests needed to be run independently where possible. Once you know where a defect is you cannot unknow it; don’t think about purple elephants!
  • My approach to UAT is scenario driven to give those running the tests room to explore the system under test. A fair comparison could not change that just because simple scripts are easier to count
  • Both approaches needed equivalent access to source information, test environments, test data and assessment criteria

Principle 2: measure objectively but measure what matters

The point of the exercise was to move beyond anecdote and “vibes”. We wanted numeric evidence in key areas, but numbers on their own were not enough. The metrics we collected had to have utility, had to provide information about whether the approach was useful.

Principle 3: make the study repeatable

The evaluation needed to be repeatable to have lasting value. That meant documenting the process, inputs, prompts, assessment criteria and the way results were captured. If AI-assisted testing was to earn a place in the process, it had to face the same scrutiny as any other testing technique or tool.

Principle 4: failure was always a valid outcome

There was an underlying assumption at the onset that AI could deliver value but that assumption had to be able to be challenged because it was just an assumption. Fairness, meaningful measurement and repeatability were safeguards against confirmation bias. A finding that AI did not help, or only helped in narrow circumstances, needed to be as acceptable as a positive result.

Deciding what value meant

Metrics lie at the heart of all this. It might seem easy enough at first glance; how long does it take to create the tests? How many bugs does each approach find? Add them up, bish bash bosh, job’s a good ’un, let’s knock off early and go down the pub.

Not so fast!

Before selecting any metrics, we needed to ask what our organization really valued and what problem we were trying to solve. The SWOT had already highlighted SME dependency. Some of the systems I test also operate within significant regulatory constraints, so accuracy, traceability and compliance had to be considered alongside speed.

Your priorities almost certainly will be different. You may have limited test capacity, poor requirements coverage, expensive tooling or a need to shorten delivery lead times. The important step is to identify those priorities before deciding what success looks like.

Beware attractive but misleading measures

AI can produce a large number of test cases very quickly, we know that. But quantity does not equal quality. Fast generation can simply shift effort elsewhere by creating review, correction and rework overheads as I think people are beginning to (re)discover.

There is a faint echo here of Fred Brooks’s The Mythical Man-Month. Brooks challenged simplistic assumptions about productivity in software delivery being increased by simply throwing more resources at a problem (if a woman can have a baby in nine months and we need a baby in one month just recruit nine women…). AI generated volume creates a different problem, but the underlying warning still applies: more output does not automatically mean more productivity.

Creation time on its own was therefore weak evidence. Review time mattered. Rework mattered. SME involvement mattered. Even together, those measures still did not tell us whether the finished tests were accurate, usable or capable of finding meaningful defects.

Build a broad candidate list

I organized brainstorming sessions with colleagues from the wider testing community. We began with a broad question: what would we need to measure to decide whether AI-assisted test creation was genuinely valuable?

The candidate metrics could be organized around four connected questions: 

  • Coverage: did the tests address the relevant requirements, risks and end-to-end scenarios?
  • Quality: were the tests accurate, clear, executable and capable of supporting meaningful testing? 
  • Efficiency: after creation, review, rework, test-data preparation and execution were considered, did AI reduce total effort? 
  • Operational fit: could the approach work within our tools, controls, stakeholder relationships and delivery constraints?

This reconciles the shorthand categories used in the first article with the more detailed questions behind them. Coverage and quality describe the output. Efficiency and operational fit describe whether the process adds value in the real world.

Prioritize meaningful evidence

Our brainstorming produced a long list, which is its purpose. The next step is refinement, to examine and assess each idea generated.

We assessed each candidate metric against two criteria: 

  • Ease of capture: could we collect the data consistently and reliably? 
  • Value: would the result materially help us judge the usefulness of AI-assisted test creation? 

We scored both criteria from 1 to 5 then multiplied the values to give a total score from 1 to 25. With each metric now having a numeric value we mapped them to a MoSCoW rating based on that score. Judgement was still involved, but it was visible, documented and defensible rather than hidden.

Designing a fair comparison

With the study principles in place and foundational work completed it was time to design the study itself.

AI tool selection

We considered the available AI tools and their suitability for the work. When doing so we had to consider that each additional tool old need its own controlled pipeline and increase the demand for test resources with fresh eyes. We ultimately selected two general-purpose LLMs and one specialist capability supplied alongside our test management tooling.

Prepare the pipelines before measuring the outputs

A mature human test professional had an obvious advantage over a newly assembled AI pipeline. To reduce that imbalance, the methodology specified three preliminary iterations of AI test creation and test execution on a subset of the work before formal metric collection began.

Those iterations were intended to stabilize the inputs, prompts, review process and result capture. Only after that preparation would we run the concurrent comparisons and begin collecting the formal evidence.

Ready to put the method into practice!

We now had the foundations of a credible study: an understanding of our context, principles intended to keep the comparison honest, a prioritized set of measures tied to what we valued and a method for preparing the AI pipelines before formal measurement.

We also had the Radio Playout project introduced in the previous article, whose scale and repeated testing needs gave us an opportunity to compare human-created and AI-created tests in a real setting.

On paper, the pieces were in place. The next step was to turn the methodology into working test-creation pipelines. As I hinted in the first article, that proved more difficult than expected.

Next in the series

In the next post, I will look at what happened when the design met reality: inconsistent source material, outputs that did not always transfer cleanly between areas, the effort involved in review and rework, and the very human risk of reviewer fatigue

Why We Measured AI Instead of Trusting It

Generating test cases with AI was easy. Working out whether they added value was much harder.

This blog series grew out of a conference presentation exploring how AI-generated UAT test cases could be evaluated through evidence rather than assumptions.

Where we started

AI is everywhere. We’re told AI will revolutionise everything bringing unparalleled efficiency and savings! Happy face!  

Or possibly that it will bring about mass professional unemployment, neo-feudalism and a sea of AI Slop. Sad face!

Either way, my organisation, like may others, is exploring how we can use AI in pretty much every context, including in software testing of enterprise IT systems which is my area.

And we are far from alone in this; looking at the testing conference circuit this year AI was clearly the biggest topic. I should know I was one of the people giving a talk on it.

In our current paradigm shift there is a lot of pressure on people to just “use AI”… Use AI to do what exactly, to solve what problem? Who cares, just use it!

And AI is deceptively easy to start using, be it with specific testing tools sold by specialist vendors or just generic LLMs.  Churning out tests is easy with AI but how useful is it really?

I could easily have just set up an AI pipeline to see how it operates and how I feel about it then told my managers “Yes, we’re doing AI now!” But that’s not exactly a measured or well thought out approach.

I’m a tester, I’m interested in getting solid metrics on how well systems work, applying judgement. That is fundamentally my job and if you’re a test professional it should be yours as well.

Testers shouldn’t run on vibes, they should run on evidence.

I decided to do what I do best and gather evidence to see how effective AI test creation was for me.

What began as a comparison of human and AI-created tests would eventually raise wider questions about quality, human judgement and whether a project is ready to benefit from AI in the first place.

Setting up a study

I run Business Acceptance Testing and User Acceptance Testing on Enterprise IT systems at the BBC. My job involves checking that the various outsourced systems we procure work as intended and do what the end user wants. This frames where I came from in setting up my study; the type of systems I test, the developmental stage they’re at, my resources and so on. 

What works for me might not be what works for you because you’re testing different systems, using different tooling, with different stakeholders and different priorities. My results in terms of value delivered will be different to your results.

Your inputs are different from mine, your results may be different from mine, but the method for evaluating whether AI assisted test creation will work in your environment is transferable.

Setting up a study

The Proof of Concept already existed: AI could generate test cases. This has been demonstrated many times by many people.

But that wasn’t the question I wanted to answer. I wanted to know what value those tests added in the environment where my team worked. What I needed was a Proof of Value study.

The difficult part was not persuading AI to write test cases. It was working out how to recognise whether those test cases had value.

What followed challenged some of my assumptions, not only about what AI could produce, but about requirements, source material, human judgement and what “good” testing means.

Designing the study

With that in mind, how do you design a study that works in your own circumstances? I recommend the following process:

  1. Understand your starting point
  2. Decide what value means for you and your ogranisation
  3. Select key metrics based on those values
  4. Set up a fair system of comparison
  5. Stabilise your AI approach before you begin measurement
  6. Collect and evaluate the evidence from human and AI generated tests

Understanding your starting point

To start with I did some basic analysis of where my team and I were. I ran a SWOT analysis, where I analysed what our Strengths, Weaknesses, Opportunities and Threats were when it came to AI.

I then ran a resource audit where I looked at what was available to work with and what wasn’t available

  • team composition and skills
  • capacity and workload
  • available tools and technologies
  • training needs
  • constraints and opportunities

Deciding what we valued

Different testing teams are trying to solve different problems. A team under intense delivery pressure might care most about reducing test design effort. A highly regulated environment might prioritise traceability and auditability. Another team might be struggling with test coverage and want to know whether AI helps identify missed scenarios.

Looking at what we valued for us it came down to three broad questions:

  • Does AI help us create tests more efficiently?
  • Does it create tests of sufficient quality?
  • Can it realistically fit into the way we actually work?

Those questions became the foundation for the metrics we selected

Defining key metrics

Once we had defined our values, we could consider the measures that might represent them.

Working with stakeholders and the wider BBC testing community, we generated more than 20 potential metrics. Collecting data for every one of them would have created considerable overhead, so we needed to identify the measures that best represented what we valued.

Setting up a fair system for comparison

Having established what we valued and wanted to measure, we needed a fair way to compare human-created and AI-created tests. That was harder than it sounds.

Once a tester knows that a defect exists, they cannot simply unknow it and independently rediscover it. A perfectly controlled comparison would therefore require different testers working in parallel.

I run on a shoestring, so spare people available to repeat the same work was not going to be easy to find. Our study design had to be credible, but it also had to reflect the resources available in the real world.

Stabilising the AI approach

There was another fairness issue. An experienced tester brought years of context, judgement and learned technique. Our newly established AI-assisted process did not.

We therefore needed to give the AI approach an opportunity to stabilise before formal measurement began. Otherwise, we would have been measuring the immaturity of our first attempt rather than the potential value of the approach.

Collecting and analysing the evidence

Using several measures allowed room for nuance. The outcome would not be reduced to a simple claim that either humans or AI were “better”.

This was not simply a comparison between two sets of test cases. It also raised questions about what experienced testers contribute that is difficult to encode in a prompt or provide through a source document.

What happened next

We had established the question, assessed our starting position and identified the measures that mattered. We had also found a real project that could give us sufficient scale to compare human-created and AI-generated tests.

The eventual results would be specific to our environment. The method for reaching them is something other teams can reuse.

The next step was turning an abstract claim such as “AI adds value” into something we could observe and measure.

Next in the series

In the next post, I’ll explain how we defined value, reduced more than 20 possible measures to a practical set, and discovered why the most obvious measures were not always the most useful.

SNAFU, FUBAR and Clusterfudge

Software projects don’t always deliver on time or deliver what’s wanted; according to the Standish Group only 31% of software projects are successful, and of these successful projects only 46% return high value…

Fundamentally testing as an activity is about finding and preventing problems that will affect the quality of what you’re delivering. This isn’t always about plain bug hunting or automated test suites, it’s about smoothly managing a project because when projects slip, as they do, quality suffers.

Generally we use RAG statuses, Red, Amber, Green, to measure how a project is running. But what do these mean? Red, it’s late, Amber it might be late, Green it’s going OK? Or something like that; to be honest I’m not entirely sure because everyone’s take is different. Of course, RAG statuses should be clearly defined at project inception however this isn’t always the case. As I frequently say, if everything that should happen did happen I wouldn’t have had a twenty five year career in testing.

So completely tongue in cheek I’d like to suggest a different taxonomy on how we look at project status based on a hierarchy of military terms. I’ll also suggest what test management should be doing for SNAFU and FUBAR projects. When it comes to a clusterfudge you’re on your own!

SNAFU (Situation Normal All Fouled Up)

This gentleman is Private SNAFU. He featured in a series of cartoons made from 1943 to 1945 by the US Army Airforce First Motion Picture Unit. The cartoons showed what happened when service personnel didn’t follow regulations .The majority of the cartoons were written by one Theodor Geisel who you might of heard of by his more well known writing name, Dr Suess.

Organisations are made of people, and because of this they can be surprisingly functionally messy. When I was young I used to love blooper programmes on TV as they sowed that behind the slick and polished exterior of the TV screen things went wrong on a regular basis.

This the base state of projects. Things don’t run smoothly and there are a series of low level issues to overcome in order to deliver on time.

What should test management be doing?

This is where test management should be making sure that potential project issues have been anticipated and mitigation measures are ready. For me a key element in tackling SNAFU is planning, particularly in keeping up to date RAID logs.

However these need to revisited regularly; as the old adage goes “No plan survives contact with the enemy”. Any plan, including your RAID logs, that is created in a changing environment must routinely adjusted and assessed.

FUBAR (Fouled Up Beyond All Recognition)

It appears this phrase came from the USMC in World War 2 and then was popularised, if that is the correct word, during the US Vietnam conflict. Which when all is said done was pretty FUBARed from the American perspective.

In my taxonomy FUBAR is for projects that are going off the rails. In conventional RAG status these would either be in the Amber or Red.

What should test management be doing?

When trying to turn around a project that is FUBAR realism is key. Testing is unlikely to be the reason it’s shifting right testing but it will be a key part of getting it back on track. As a tester or test manager you need to provide accurate estimates, not just what people what to hear. It’s a regular occurrence on FUBAR projects to be asked to trim testing estimates for activity occurring at the end, such as UAT or SIT. Fine, you could do that but if you do you’ll just be revisiting timelines later.

Be honest with your stakeholders and don’t be afraid to ask for help. Ultimately our job should be to ensure our organisations survive and thrive so don’t be afraid of talking to project sponsors, keeping the sunk cost fallacy in mind.

Of course, it may be that your project sponsors are so deeply committed to the project that they don’t want to hear what you have to say. That brings us to the next status.

The Clusterfudge; Red and beyond

The “clusterfudge” (please note I’ve sanitised the wording) is where things are now seriously wrong. The photo above is by Hubert van Es and shows the last US helicopter to leave Vietnam in 1975, truly a clusterfudge of a situation from the US perspective.

The etymology of the word is somewhat obscure, but it came about during the 1960s and was popularised during the US Vietnam conflict. One attributed is to beat poet Ed Sanders, however my favoured origin and the one that fits with this talks is that is stems from the Oak Leaf Clusters worn by Majors and Lieutenant Colonels in the US forces. A cluster “fudge” came about when these officers got involved in a FUBAR situation and made it worse.

Stanford Prof Bob Sutton describes a clusterfudge as being made up of a mix of illusion, impatience and incompetence. Illusion, when a decision maker believes a goal is easier than it really is. A current high profile example might be Putin believing that Russian troops could take Kyiv in three days. Impatience, trying to release in a hurry or within false time frames. Incompetence, where leaders don’t have technical competence and aren’t willing to listen to experts.

Hopefully your projects will be more SNAFU than FUBAR; chaos and churn are part of life, it’s our job as test professionals to look for problems before they happen and have contingencies in place to deal with them.

Shifting Quality Left with Fagan Inspections

At recent test conferences it seems one of the phrases of the moment is “Shifting Quality Left”. If you test as early as possible bugs are found when they are easier and cheaper to fix. Ideally catch them those you can before coding even starts!

Possibly because the industry has pivoted towards Agile one method that seems to have fallen somewhat out of favour in recent years is Fagan Inspections. If you’ve not come across the term before I’m going to describe it here, it may be something you want to try. A Fagan Inspection is a document review process to make sure that the objects you’re working with are defect free. I say document review, it’s also applicable to code although personally I’ve not used it in that context. This makes it a type of static analysis, testing without actually running the software being created. It’s called a Fagan Inspection as it was devised by Michel Fagan at IBM in 1976. Personally I came across this while working at Sony Ericsson where known as a Granska Bra (“well review”). Sony Ericsson had taken the concept from SAAB and applied it to mobile phone development.

A Fagan Inspection is a formal review process where objects (documents or code) are examined team of people to check to see if they’re ready for use. When I was using this process at Sony Ericsson we estimated that we had a ten fold return on the time we invested in it. So for every hour each person spent reviewing and preparing for reviews, we would save ten hours of time down the line.

How does it work? Well, when the object owner thinks what they’re working on is ready they take it to a moderator and the two of them have a pre-inspection run through just to make sure things really are ready to review. The moderator then sets up a panel of relevant experts to examine the object. For example, when using this at Sony Ericsson we would generally have a developer, a UI designer, a tester and an architect. These reviewers should run through the object from their domain perspective before the meeting.

During the meeting a presenter presents the object. This could be the object owner or it could be the moderator. The reviewers then step through the object and comment where they feel the need to.

There are two dangers at this point and it’s the moderators job to tackle them. Firstly, the moderator has to ensure that the review is a fair review and that the owner of the thing being inspected doesn’t feel like they’re being picked on. Secondly, it’s easy for people to drift onto finding solutions at this point “Ah, we could try doing this instead!” .That’s not the point of the meeting and if you let it start happening the meeting will soon be derailed. A two minute rule works well for this, rigorously enforced by the moderator. People can arrange follow up meetings to work on solutions if they need to.

At the end of the meeting the an exit decision is taken. Does the object need reworking and a subsequent re-review or can it be passed (perhaps with minor modifications with that won’t need re-review).

Adding this process can be seen as a drag, an extra step for people who just want to get on and code. However, ensuring our inputs are actually fit for purpose and clearing up misunderstandings before work is created based on those misunderstandings pays massive dividends.

Why we blog

The aim of this blog is to talk about software testing. It’s a broad area so there’s a lot to talk about from tools and techniques, to methodology and into the philosophy of test,

The inspiration for this blog came from a talk at the UK National Software Testing Conference 2021. As someone with a fancy job title at a well known tech leader I occasionally get invitations to do things like talk at a conference. As it happens the invite for the NSTC talk dropped into my inbox soon after I’d read an interesting article about levels of project failure so I said “Yes, why not?”

After that a blog seemed to be the next thing…