By Caleb Billingsley, AI Testing and Performance Expert, Foulk Consulting
Walk through any tech expo or scroll through a DevOps feed today and you’ll hear a familiar promise that hands-off performance testing has arrived. Plug an AI agent into your CI/CD pipeline, point it at your staging environment, and it will simulate millions of users, catch every regression, and tune your architecture while you sleep. It has real promise, but for enterprise systems we aren’t there yet.
The version I see most often is a demo where someone connects an AI agent to a load testing tool through MCP (the Model Context Protocol) and types “go load test our application.” The agent writes the scripts, picks a load, runs the test, and comes back with a report and a list of fixes. The agent has no way of knowing what the test is supposed to prove.
A load test is only as good as the decisions made before it runs. Someone has to decide which business steps matter, how much load is real, what a correct response looks like, and which systems the test is not allowed to touch. Those decisions come from knowing the business and the architecture, which no tool knows out of the box.
That demo can work in an enterprise when a performance engineer and an architect set the guardrails before the agent runs anything.
Where AI is already saving us time
I’m not arguing against AI here. We’ve been working on how to bring AI into performance testing since 2022. Some of it is paying off now. We’re seeing excellent results from SRE agents that combine load test results, APM traces, infrastructure metrics and logs into one performance report. A report that used to take an engineer days to put together now takes less than an hour. The agents also find tricky correlations across those sources that are easy to miss when an engineer is working through one dashboard at a time.
AI helps on the scripting side too. It drafts scripts from recordings and API specs and points out the values that need correlation. An engineer still proves that every script replays correctly before it goes into a test, because a script that replays badly can still produce a clean-looking report.
The engineer gets those days back for the part of the job that needs a person, which is deciding what the test should prove and explaining what the results mean to the business.
Where the demo falls apart
Each of these comes from our own performance work. We’ve left out the names and changed the details to keep our customer details confidential, but the examples are real and meaningful.
Testing the wrong thing. One of our current engagements tests SAP billing for a company with long-term contracts. The business question is whether up to 5,000 invoices a week clear an overnight batch window, so we measure invoices per hour through each stage of the batch. An agent pointed at the billing screens would build virtual users, measure page response times, and report a pass on a test but the issue was in contracts for fixed price billing but cost based where the test was focused.
Counting users the wrong way. On SAP S/4HANA with a Fiori front end, the classic user monitor (SM04) shows close to zero users even at peak, because Fiori’s OData calls don’t keep an ABAP session open between requests. An agent that reads concurrency from that screen would model almost nobody. The logged-on user count lives in a different monitor (SM05). The real capacity limit is how many work processes are busy at once, which is a much smaller number again. A requirement that says “1,000 concurrent users” can’t be tested until someone decides which of those numbers it means.
A fast answer that did no work. A fast 200 OK can be the worst number in the report. When a service runs out of database connections, plenty of application code catches the timeout and returns an empty list with a 200. The response time looks excellent because nothing happened. A content check in the script catches it. Somebody still has to tell the agent what a correct response contains for each business step, because an agent only checks what it has been told to check.
Repeating a fix that already failed. On an operations system we test, the servers were already moved to much larger cloud sizes as insurance. CPU spikes at times and sits low most of the day. The bigger size hasn’t always fixed the slowdowns when they come back. An agent that sees a CPU spike and recommends more capacity would repeat a fix the team has already paid for. When that system is down, people on the floor fall back to paper and key the data in later. The impact runs to hundreds of thousands of dollars a day in lost productivity and rework. The team that tests it has kept more than 20 high-severity performance issues out of production so far.
The load generator becomes the bottleneck. On a public-facing web application with an SAP back end, we moved the load generator off a burstable cloud instance onto a compute-optimized one. Burstable instances can run out of CPU credits partway through a long test. From that point the load generator adds response time that the application never had but the CPU of the cloud machine looked fine. We calibrate each load generator before we commit to a user count. The site’s firewall also has to allow the generators’ addresses, or the test ends up measuring the firewall.
The guardrails we set first
These are the guardrails a performance engineer and an architect set before we let an agent run a test.
- The workload model. The engineer builds it from production data, covering which business steps run, in what mix, at what rate, and how the peak arrives. Virtual users are sized from that throughput rather than from the number of licensed users.
- What a correct response looks like. Each business step gets a content check the business would recognize, so a fast empty answer fails the test.
- What is in scope and what is stubbed. The architect decides which systems the test can touch. Payment, tax, shipping and other outside services are stubbed or virtualized so the test doesn’t run up usage fees, trip someone else’s rate limits or get your addresses blocked.
- What normal looks like for this system. The architect knows which batch jobs are throttled on purpose to protect a downstream system. Test environments are often a path-to-production system and sometimes a scaled-back one. The results have to be read with that difference in mind.
- Test data. Each virtual user gets unique data. The database goes back to a known restore point before each run so one test can be compared with the next.
- Load generators. They are sized and calibrated before anyone quotes a user count. The firewall is set to let them through.
- Limits and stop conditions. The engineer sets the maximum load, the error rate or response time at which the agent stops, and who approves going further.
- Who makes the call. The agent can recommend a decision. A person makes the go or no-go call and explains it to the business.
Inside those guardrails an agent can run the test, combine the results with monitoring data, and draft the report. That is where we’re getting days back. Before we trust a single number, we check that the transaction mix, the busiest services and the peak shape match what production sees. We don’t treat a virtual user count as evidence on its own.
Where to start
In my last article I wrote about Human-Governed AI, where AI speeds up the work and people stay responsible for the outcome. Performance testing is one of the clearest places to see it work. The agent saves days on analysis and reporting while the engineer and the architect decide what the test proves, what it can touch, and when it stops.
Foulk started as a performance testing firm in 2002 and has run more than 1,000 performance engagements since then. We’ve been working on how to bring AI into performance testing since 2022. The guardrails above came out of that work.
If you have a go-live, an upgrade or a peak season coming and you’re thinking about putting an agent in the loop, can we spend a week with your team reviewing your next performance test plan at no cost?
