I’d set out to understand how useful AI could be for creating UAT tests. Instead, my study exposed a question I hadn’t thought to ask: was the project suitable for AI in the first place?
Why I thought I had found the ideal project
My study methodology was designed around a few basic principles: compare like with like, reduce avoidable noise and make the study repeatable. That made the project I had in mind look ideal.
It was large, with around 600 new test cases required, and similar testing would be repeated across different environments and by different people. Rather than drawing conclusions from a single set of AI generated tests, we had an opportunity to repeat the comparison at scale.
On paper, it was exactly the sort of project I had been looking for!
When the design met reality
As I detailed in my earlier posts, we had always expected that establishing a robust AI workflow would take several iterations. Part of the study involved experimenting with prompts, refining them and learning how best to provide the AI with the information it needed. Some variation in the early results was therefore expected.
What we hadn’t anticipated was how much that variation would persist even after we had established what appeared to be a reasonably mature workflow.
Carefully constructed prompts could produce a mixture of genuinely useful tests and completely unusable ones within the same batch. This made evaluating the effectiveness of the approach much harder than simply looking at how many tests the AI could generate. Every output still needed human review, and poor tests needed to be identified, rejected or reworked.
Eventually we reached what I started calling our “Shining moment”. In The Shining, Jack Torrance is supposedly writing a novel, only for his wife to discover that page after page contains variations of the same sentence. Our version was less sinister but equally revealing: a section expected to produce more than a hundred tests had instead produced near-identical tests over and over again.
It was funny at first, but it was also the point where we knew something was fundamentally wrong. We had refined the workflow and carefully constructed the prompts, yet the AI still wasn’t producing enough meaningful variation between the tests.
At that point, it became difficult to explain what we were seeing simply as a problem with prompt engineering or an immature workflow. We had iterated on the process, refined the prompts and demonstrated elsewhere that the same approach could generate useful tests
Something more fundamental was going on.
And that, I think, is where the study began to change direction
The hidden human cost
The volume created another problem. Every generated test still needed to be reviewed, including near-duplicates and outputs that initially looked plausible but were not supported by the source material. As the volume increased, so did our reviewer fatigue. The AI could generate tests far faster than we could assess them properly, but speed of generation is of little value if it simply transfers effort into review, rejection and rework.
One last look
By this point, I was close to abandoning the study. We’d spent time refining the workflow and improving the prompts, but I still couldn’t explain why the AI could produce useful tests in one area and poor, repetitive or hallucinated tests in another. I was frustrated that I couldn’t make the approach work consistently, but even more frustrated that I couldn’t work out why.
Before giving up, I went back through the source material and the generated outputs one more time. That was when something struck me.
We were creating tests covering around six years of product updates, but those updates weren’t all the same. Some represented new functionality; others were fixes for defects discovered over the years.
The original requirements for the product had been written around twenty years earlier and we could no longer surface them. Access to the supplier’s defect database wasn’t available to us because it contained information relating to their other clients. In practice, therefore, the AI was working from the information we did have: release notes, training guides and user manuals.
And there was an important difference hidden in those sources.
The release notes mixed both new features and defect fixes. The new features described in the release notes were also covered much more comprehensively by the user manuals and training material. The AI therefore had considerably more context from which to construct meaningful tests.
The defect fixes were different. They appeared in the release notes, but there was no equivalent supplementary material explaining the original defect, the intended behaviour or the context surrounding the change.
Suddenly, those wildly different outputs began to make sense.
We had been concentrating on the prompts, the workflow and the AI. But the quality of the information we were giving it varied dramatically depending on what we were asking it to test.
A clear pattern had emerged: the more useful outputs tended to come from areas with richer supporting information, while the poorest and most repetitive outputs came from areas where little context existed beyond the release notes. The outputs were pointing towards a more fundamental problem with the inputs.
And that changed the direction of the study.
The right project for the study, but the wrong project for AI
There was an uncomfortable irony in all of this. From the perspective of the study, the project had been an ideal candidate. It was large, it involved around 600 new tests, and its repetitive nature gave us opportunities to compare like with like across different environments and testers. It ticked the boxes I had set out for a robust study when designing the experiment.
From a real-world AI perspective, however, it was far from ideal.
The problem wasn’t its size or complexity. It simply did not have enough high-quality source material for what we were asking the AI to do.
That didn’t mean the study design was wrong. The approach was robust and is worth repeating. What had changed was my understanding of where it could reasonably be applied. Continuing to push this particular project through the same process wasn’t going to answer the original question fairly.
And this was not just an academic problem. I have many projects on my plate. Some are shiny and new, with requirements, designs and other source material readily available. Others are legacy systems like this one, where important decisions were made years ago, documentation is incomplete and some of the original thinking has simply been lost in the mists of time.
If AI-assisted testing was going to become genuinely useful rather than an interesting experiment, I needed some way of distinguishing between them.
The question changes…
I started this study with a fairly simple question: how useful could AI be in creating UAT tests?
I still think that is a question worth answering, and the approach we developed remains a sound way of investigating it. But this project revealed that another question needs to come first.
Is this project actually suitable for AI-assisted test creation?
The inconsistent outputs were not, on their own, evidence that AI was either good or bad at creating UAT tests. They showed that the context in which we asked AI to work mattered enormously. Before judging the output, we needed to understand the quality, completeness and relevance of the input.
The study therefore hasn’t given me the neat conclusion I originally expected. I can’t honestly finish by saying that AI is, or isn’t, effective at creating UAT tests. The experiment needs repeating on a project with richer source material before I would be comfortable making that judgement.
But it has left me with what may turn out to be a more useful question:
Before investing time in AI assisted test creation, how can we tell whether a project has the right information to make it worthwhile?
That’s where this experiment has taken me. It wasn’t the question I set out to answer, but following the evidence rather than forcing the original experiment towards a convenient conclusion feels like the right place to stop.
