(And What The Book of Why Explains That Most AI Hype Ignores)
By Caleb Billingsley, AI Testing and Performance Expert, Foulk Consulting
Modern AI is impressive. It writes code, summarizes massive logs, generates test scripts, and spots anomalies faster than any human team could. And yet, when placed in charge of real production environments — complex, versioned, risk-sensitive systems — it consistently falls short.
This isn’t a temporary tooling gap that the next model release will fix. It is a causality problem.
In The Book of Why: The New Science of Cause and Effect, Turing Award winner Judea Pearl and co-author Dana Mackenzie explain the exact limitation that most AI hype glosses over.
Pearl’s Ladder of Causation
Pearl defines three levels of reasoning:
- Association (What is happening?) — correlation, patterns, and statistics.
- Intervention (What happens if I do X?) — cause and effect under direct action.
- Counterfactuals (What would have happened if I had done something different?) — alternate realities, responsibility, blame, and risk assessment.
Large Language Models are exceptional at Level 1. They recognize patterns and predict likely outcomes based on historical similarity across enormous datasets.
To be fair, the research picture is evolving. Recent frontier models post impressive scores on some counterfactual benchmarks, while other studies show them routinely answering interventional questions with associational evidence — a failure researchers call “rung collapse.” But the deeper point survives every benchmark result: reciting causal knowledge absorbed from training data is not the same as holding a causal model of your system. An LLM can retrieve generic cause-and-effect knowledge from millions of postmortems and runbooks. What it cannot do is build and maintain a verified causal model of your specific environment — or tell you when its borrowed knowledge doesn’t apply to your version, your integrations, your business rules.
Enterprise production systems live at Levels 2 and 3.
A Bug That Hid at Midnight
A recent example from our own work. We built a Salesforce-to-Shopify fulfillment integration using AI-assisted development. Orders synced in batches on a five-minute interval, and the AI sensibly suggested a safeguard: only sync orders from the current day.
The human developer read that as a 24-hour rolling window. The AI implemented it as a hard call to today(). The result was a latent bug: an order placed at 11:48 p.m. could age out of the window before the next batch picked it up — and would never sync at all.
No monitoring alarm fired. No test caught it. It was discovered by a developer working late on an adjacent feature who happened to notice an order fail to sync and asked why. Here’s the interesting part: when he questioned the AI, it found the bug instantly. All the evidence was sitting in the system the whole time.
The AI had the data, the code, and the intelligence to diagnose the problem. What it never did — what it had no reason to do — was ask, “What happens to an order placed at 11:58 p.m.?” That is a Level 2 question. The human asked it; the AI answered it. That division of labor is the entire story of AI in production, in miniature.
Why Pattern Recognition Breaks in Complex Systems
In software delivery and performance operations, the critical questions are never “What usually happens?” They are:
- What happens if we deploy this configuration change here?
- What breaks if we fix performance in one layer but neglect another?
- What would have happened if we had not applied that workaround?
- Is this anomaly safe, or the early signal of a cascading failure?
These are interventional and counterfactual questions. An AI trained primarily on association can sound confident answering them — as our integration bug shows, it may even hold the right answer — but it doesn’t know which question to ask about the unique causality of your specific system.
Causal Knowledge Doesn’t Transfer Across Versions
This is why AI struggles with software versioning specifics. Consider the differences between LoadRunner 2021, 2023, and 25.x; SAP ECC and S/4HANA; Kubernetes minor releases; legacy and modernized infrastructure.
From a correlation perspective, these look mostly the same — and for niche enterprise configurations, the training data is sparse and often stale. But a single subtle difference can completely change how a system behaves under load, failure, or scale. The causal knowledge an AI absorbed about one version simply doesn’t transfer to the next, and the AI has no reliable way to know when the transfer fails.
Human engineers learn these differences through the scars of past incidents, failed releases, and postmortems. Where AI sees broad similarity, operations demand exact causal specificity.
Testing Is Fundamentally Counterfactual
Testing is often misconstrued as mere validation. It is actually structured counterfactual reasoning. Every effective test asks a “what if” question: What if users behave differently? What if this integration slows under load? What if this failure happens at the worst possible time — or at 11:58 p.m.?
Performance testing pushes further into Level 3: What happens if concurrency doubles? Which bottleneck causes which downstream symptom?
AI can generate test cases, observe failures, and cluster symptoms. But designing the right test requires knowing which causes matter, which effects are acceptable, and which failures are catastrophic versus tolerable. That requires causal judgment.
Why “Self-Healing” Stops Where It Does
Today’s “self-healing” automation is bounded to three safe actions: restart a service, scale resources, or roll back to a known-good deployment. Going further requires answering: Is this the root cause or a surface symptom? What breaks if I intervene? What happens if I do nothing?
Because today’s AI systems do not maintain explicit causal models of your environment — across shifting versions, integrations, and business rules — their autonomy remains strictly bounded. And there is a second boundary the hype ignores: accountability. When an autonomous change goes wrong, someone must own the risk. That someone is never the model.
The Real Opportunity: Human-Governed AI
When we say human expertise is still required, it isn’t a sentimental argument for “human intuition.” It is a technical one. Humans bring dynamic mental causal models of the system, historical memory of why things were built a certain way, engineering judgment shaped by prior failures and near misses — and accountability for the outcome. When an engineer asks, “What would have happened if we hadn’t caught this?”, that question sits on the top rung of Pearl’s ladder.
For AI to move beyond assistance into true autonomy, it will need explicit causal models, version-aware reasoning, and risk-aware decision frameworks. These are active research areas, not solved problems. Until then, the opportunity isn’t replacing engineers — it’s building Human-Governed AI: AI accelerates insight at Level 1 scale, while humans own causality, formalize counterfactuals through testing, and enforce causal discipline through performance engineering.
AI struggles in complex enterprise environments not because it lacks raw intelligence, but because intelligence without causality is insufficient. If you are testing AI systems or charting the future of your operations, read The Book of Why. Production systems will keep proving its point.
Until an AI can reliably ask — not just answer — “What would have happened if we didn’t do this?”, you will need partners, engineers, and performance operators who can. Not instead of AI, but right alongside it.
Let’s Build Your Causal Strategy Together
At Foulk Consulting, we help organizations navigate the gap between AI hype and enterprise production reality. We don’t just deploy tools; we provide the engineering judgment, version-aware testing, and human-governed AI frameworks that keep your systems stable, resilient, and performant.
Whether you are de-risking AI initiatives, optimizing performance testing strategies, or building guardrails for production environments, our team bridges the gap between association and true causal execution.Contact Foulk Consulting Today to speak with our performance engineering experts.
