In my previous blog post, I laid out why I wanted to evaluate how useful AI could be in my test creation process. I wanted evidence I could use to make informed and defensible decisions about what worked well and conversely what didn’t. As testers we should not run vibes, we should work on evidence.
I also gave a high-level account of the study; the initial analysis, the search for useful metrics, the challenge of creating a fair comparison and the project that gave us the opportunity to put the idea into practice.
In this post I want slow down and examine the design properly. How did we establish what mattered to us? How did we avoid creating a study that just confirmed what we wanted to hear? Most importantly, how did we turn the broad claim that “AI adds value” into evidence we could actually collect?
This isn’t a universal recipe. Your systems, stakeholders, constraints and priorities will be different from mine. My aim here is to share a way of thinking you can adapt to your own circumstances.
Understanding your starting point
In my previous post I briefly mentioned that I began with a SWOT analysis and a resource audit. They deserve more attention because they didn’t just serve to document where we stood, they also influenced what we choose the measure and placed practical limits on what the study could support.
SWOT Analysis
A SWOT analysis asks you to consider strengths, weaknesses, opportunities and threats. It is a simple technique, and a general-purpose LLM can help organize the first pass, provided you review the result and remove or generalize anything confidential before sharing it, which is something I had to do here.
One thing that flew out of the SWOT table was access to SMEs. For my work their knowledge is a key strength, but this dependency is also a weakness. This meant that using AI to help reduced the time I would need them for would be a worthwhile exercise. It also meant that SME time could not be treated as an invisible or unlimited resource when we designed the study.
Resource Audit
Running the resource audit brought the practical constraints I had to work with into focus. I looked at the people and their expertise levels and experience. Our skill base in terms of methodology, tooling and domain specific knowledge. The capacity we had and our pipeline of work; peak workloads, known bottlenecks. Our tools and technologies; what we had to work with. Our training needs and budget to gain the new skills we need. Other risks and constraints; capacity risks, compliance risks (think GDPR), broader risks (the BBC funding model). Finally, opportunities; where could we adopt AI, how could we optimize our processes.
This mattered because every extra AI tool required another controlled pipeline and, ideally, fresh eyes for each comparison run. The audit therefore helped shape the number of tools we could examine, the comparison we could realistically run and the evidence we could capture reliably. It also once again highlighted the role of SMEs.
Starting point summary
The SWOT analysis and resource audit gave us two things; a clearer view of what “value” might mean in our context and a realistic understanding of what the study could sustain. It’s worth noting that there are other techniques that can be used at this point. If you work in a field with more external influences, such as a need for regulatory compliance then running a PESTLE analysis will be useful. A good source for looking at what tools will help is your business analysis team. This is their area of expertise.
For now we had enough to establish the principles that would govern our survey.
Principles for defensible evidence
Principle 1: make the comparison fair
Making a like for like comparison involves more than just giving the same source documents to a human and as an AI input. We have to consider differences in experience, who will be running the tests, what those testers already know and how closely the comparison would reflect the way UAT is performed.
- A human test professional with years of experience has a higher maturity level than a newly created AI pipeline
- Human created and AI created tests needed to be run independently where possible. Once you know where a defect is you cannot unknow it; don’t think about purple elephants!
- My approach to UAT is scenario driven to give those running the tests room to explore the system under test. A fair comparison could not change that just because simple scripts are easier to count
- Both approaches needed equivalent access to source information, test environments, test data and assessment criteria
Principle 2: measure objectively but measure what matters
The point of the exercise was to move beyond anecdote and “vibes”. We wanted numeric evidence in key areas, but numbers on their own were not enough. The metrics we collected had to have utility, had to provide information about whether the approach was useful.
Principle 3: make the study repeatable
The evaluation needed to be repeatable to have lasting value. That meant documenting the process, inputs, prompts, assessment criteria and the way results were captured. If AI-assisted testing was to earn a place in the process, it had to face the same scrutiny as any other testing technique or tool.
Principle 4: failure was always a valid outcome
There was an underlying assumption at the onset that AI could deliver value but that assumption had to be able to be challenged because it was just an assumption. Fairness, meaningful measurement and repeatability were safeguards against confirmation bias. A finding that AI did not help, or only helped in narrow circumstances, needed to be as acceptable as a positive result.
Deciding what value meant
Metrics lie at the heart of all this. It might seem easy enough at first glance; how long does it take to create the tests? How many bugs does each approach find? Add them up, bish bash bosh, job’s a good ’un, let’s knock off early and go down the pub.
Not so fast!
Before selecting any metrics, we needed to ask what our organization really valued and what problem we were trying to solve. The SWOT had already highlighted SME dependency. Some of the systems I test also operate within significant regulatory constraints, so accuracy, traceability and compliance had to be considered alongside speed.
Your priorities almost certainly will be different. You may have limited test capacity, poor requirements coverage, expensive tooling or a need to shorten delivery lead times. The important step is to identify those priorities before deciding what success looks like.
Beware attractive but misleading measures
AI can produce a large number of test cases very quickly, we know that. But quantity does not equal quality. Fast generation can simply shift effort elsewhere by creating review, correction and rework overheads as I think people are beginning to (re)discover.
There is a faint echo here of Fred Brooks’s The Mythical Man-Month. Brooks challenged simplistic assumptions about productivity in software delivery being increased by simply throwing more resources at a problem (if a woman can have a baby in nine months and we need a baby in one month just recruit nine women…). AI generated volume creates a different problem, but the underlying warning still applies: more output does not automatically mean more productivity.
Creation time on its own was therefore weak evidence. Review time mattered. Rework mattered. SME involvement mattered. Even together, those measures still did not tell us whether the finished tests were accurate, usable or capable of finding meaningful defects.
Build a broad candidate list
I organized brainstorming sessions with colleagues from the wider testing community. We began with a broad question: what would we need to measure to decide whether AI-assisted test creation was genuinely valuable?
The candidate metrics could be organized around four connected questions:
- Coverage: did the tests address the relevant requirements, risks and end-to-end scenarios?
- Quality: were the tests accurate, clear, executable and capable of supporting meaningful testing?
- Efficiency: after creation, review, rework, test-data preparation and execution were considered, did AI reduce total effort?
- Operational fit: could the approach work within our tools, controls, stakeholder relationships and delivery constraints?
This reconciles the shorthand categories used in the first article with the more detailed questions behind them. Coverage and quality describe the output. Efficiency and operational fit describe whether the process adds value in the real world.
Prioritize meaningful evidence
Our brainstorming produced a long list, which is its purpose. The next step is refinement, to examine and assess each idea generated.
We assessed each candidate metric against two criteria:
- Ease of capture: could we collect the data consistently and reliably?
- Value: would the result materially help us judge the usefulness of AI-assisted test creation?
We scored both criteria from 1 to 5 then multiplied the values to give a total score from 1 to 25. With each metric now having a numeric value we mapped them to a MoSCoW rating based on that score. Judgement was still involved, but it was visible, documented and defensible rather than hidden.
Designing a fair comparison
With the study principles in place and foundational work completed it was time to design the study itself.
AI tool selection
We considered the available AI tools and their suitability for the work. When doing so we had to consider that each additional tool old need its own controlled pipeline and increase the demand for test resources with fresh eyes. We ultimately selected two general-purpose LLMs and one specialist capability supplied alongside our test management tooling.
Prepare the pipelines before measuring the outputs
A mature human test professional had an obvious advantage over a newly assembled AI pipeline. To reduce that imbalance, the methodology specified three preliminary iterations of AI test creation and test execution on a subset of the work before formal metric collection began.
Those iterations were intended to stabilize the inputs, prompts, review process and result capture. Only after that preparation would we run the concurrent comparisons and begin collecting the formal evidence.
Ready to put the method into practice!
We now had the foundations of a credible study: an understanding of our context, principles intended to keep the comparison honest, a prioritized set of measures tied to what we valued and a method for preparing the AI pipelines before formal measurement.
We also had the Radio Playout project introduced in the previous article, whose scale and repeated testing needs gave us an opportunity to compare human-created and AI-created tests in a real setting.
On paper, the pieces were in place. The next step was to turn the methodology into working test-creation pipelines. As I hinted in the first article, that proved more difficult than expected.
Next in the series
In the next post, I will look at what happened when the design met reality: inconsistent source material, outputs that did not always transfer cleanly between areas, the effort involved in review and rework, and the very human risk of reviewer fatigue
Bill's test thoughts 