By Caleb Billingsley, AI Testing and Performance Expert, Foulk Consulting
On one of our projects, AI built three months’ worth of test automation in 80 hours. When we checked that suite against three years of real defects from the same system, it would have caught 6% of them.
The tests ran fast and they passed. The trouble was that they only checked what the AI could see on the screens. Most of the real defects in that system came from business rules that nobody had given the AI.
In June I wrote about six levels of self-healing in operations, an idea I borrowed from the levels used for self-driving cars. The main point was that AI maturity comes down to how much authority you’re willing to hand over to the AI. Test automation climbs the same kind of ladder. Most of the teams we work with are on a lower rung than the vendor demos they’ve seen, which is fine. Problems show up when a team gives the AI more authority than its guardrails can handle.
This is the first of eight posts on AI in test automation. Each one takes a level, or a problem that shows up at a level, and walks through what we’ve learned about making it work.
The six levels for test automation
Level 0: Before AI. Engineers write and fix Playwright or Selenium scripts by hand. The tests do exactly what they’re told. When one fails and nobody changed the code, the cause is usually the environment, the test data or the timing.
Level 1: Machine learning that points at problems. Machine learning flags flaky tests, groups similar failures and helps pick which tests to run for a change. A person still decides what to do about what it finds.
Level 2: AI as a copilot. An AI assistant drafts Playwright or Selenium code from a story, a recording or a description. A person reviews the code and merges it. Most of the teams we meet that use AI for testing are at this level today.
Level 3: Human-approved automation. Agents plan tests, write them and fix them when the application changes. Every change comes in as a proposal that a person approves before it goes into the regression suite.
Level 4: Guardrailed agents. An agent runs tests written in plain English the way a user would. It clicks through the application and decides whether what it sees is right. The team sets the limits it works inside, including which environments and data it can use, which actions it can take and when it has to stop.
Level 5: Full autonomy. You tell the agent to go test your application. It decides what to test and whether the results are good. It makes a great demo, but for enterprise systems with real business rules and real auditors we aren’t there yet.
The step from Level 3 to Level 4
Teams often miss the biggest change on the ladder, which happens between Level 3 and Level 4.
Through Level 3, whatever the AI decides ends up in code. A person reviews that code once and every run after that does exactly the same thing. At Level 4 and above, the agent reads the test and decides what it means every time it runs. If a step can be read two ways, the agent might read it one way on Tuesday and the other way on Wednesday. The test passes one day and fails the next when nothing in the application has changed.
I’ll cover this new kind of flaky test in the next post.
What it takes to move up a level
Moving up the ladder means handing more of the testing work to AI. How much you can safely hand over depends on three things we call GEM.
1. Ground the agent in your truth. Requirements, business rules and defect history are kept in shared context files, so the AI used by every developer, tester and partner works from the same information.
2. Enforce hard limits on what it can do. The team sets approval gates, keeps test data isolated and keeps each change small. The agent never merges its own work. These rules are built into the tools and the pipeline instead of written in a policy document.
3. Measure whether the tests catch faults. We keep a test when it catches a planted fault or a real defect from the past. We report the faults caught rather than test counts or coverage.
We use GEM at every step of the work rather than as three phases you finish once. The suite in the opening story was fast because the AI is good at writing tests. It would have missed 94% of the real defects because nobody grounded it in the business rules and nobody measured it against past defects until we went looking.
Google’s 2024 DORA research found the same pattern at a much larger scale. For every 25% increase in AI adoption, it estimated a 1.5% drop in delivery throughput and a 7.2% drop in delivery stability. AI speeds up the process you already have, including the gaps that were already letting defects through.
We’ve been working on how to bring AI into testing since 2022. GEM came out of that work. Over the next seven posts I’ll work up the ladder, covering flaky tests caused by wording, what your AI needs to know before it tests anything, Playwright and Selenium, self-healing tests, measuring AI-written tests against real defects, testing SAP and what all of this means for the people on a test team.
What you can do this week
Here are three checks you can run on your test suite before your next sprint starts.
1. Find your level. Look at the test suites in your deployment pipeline and decide which level matches what your team actually runs today.
2. Look at your last five production defects. Take five bugs your team fixed recently and ask whether any test in your current automation would catch them if they came back tomorrow.
3. Check where your AI limits live. Find out whether the limits on your AI agents are enforced in your pipeline or only written down in documentation.
Which level is your test automation at today, and what would it take to move up one?
Contact the Foulk Consulting team to learn more about our 90-minute executive briefing for QA leaders and our two-day AI Quality Engineering course.
