By Caleb Billingsley, AI Testing and Performance Expert, Foulk Consulting
I was on a call with a CIO this week discussing a complex SAP EWM implementation, and she said something that immediately caught my attention:
“The developers turned over SAP EWM code for testing, and it had over 50 defects.”
My first question was simple: “Were they using AI?”
The answer was yes. They were using leading AI coding technology.
That response did not surprise me, but not because I believe AI writes bad code. In fact, I believe the opposite. AI is an incredible accelerator for software delivery when used correctly. It can help developers generate code, refactor logic, create test ideas, and move faster through repetitive engineering work.
The problem? AI does not magically understand the business context. It only works from the instructions, examples, and constraints we provide. If the input is incomplete, the output, no matter how polished it looks, will be incomplete too.
The Risky Allure of “Vibe Coding”
In an enterprise environment, we are seeing the rise of “vibe coding,” the practice of building by simply chatting with an AI tool until the output “feels” right. For a prototype or a low-risk internal utility, that might be fine.
But in complex enterprise systems like SAP EWM, Oracle CPQ, or Salesforce, “vibe coding” is a liability. In these environments, the hard part isn’t writing the syntax; it’s understanding the business rules, data dependencies, and integration impacts that determine whether the code actually works in the real world.
From Requirements to “AI Specs”
A traditional requirement might say: “Support warehouse movement confirmation.” A human developer with “tribal knowledge” might be able to fill in the blanks. An AI cannot. To succeed with AI-assisted delivery, we have to shift from vague requirements to structured AI Specs.
An AI Spec acts as a bridge between business intent and machine execution. It must define:
- Business Context: What process does this support?
- Data Constraints: What conditions are valid (and which are “garbage”)?
- Exception Paths: What should the system do when things go wrong?
- Security Roles: Who is allowed to execute this logic?
- Negative Paths: What should the system never do?
If the AI Spec is not shaped by subject matter experts (SMEs) and reviewed against real-world scenarios, quality issues aren’t just possible, they are inevitable.
The Trap of Self-Passing Tests
One of the biggest risks in the AI era is the “mirrored defect.” AI can generate code based on its own assumptions, and then, if asked, it will generate tests that validate those same assumptions. The tests pass, the team feels a false sense of confidence, but the real business process fails the moment it hits production-level data or a domain-specific edge case.
This is why domain expertise is more critical now than ever. Business analysts, QA engineers, and architects shouldn’t be stepping back; they should be leaning in to define the scenarios that matter. AI can generate the volume of coverage, but humans must validate the reality of that coverage.
The Checklist: Is Your AI Spec Ready?
Before handing a task to an AI coding tool, ask if your specification answers these questions:
- What business process does this support?
- What are the valid and invalid data conditions?
- What are the expected positive and negative paths?
- What integrations are touched?
- What security or role-based rules apply?
- What are the specific acceptance tests?
- Does this conflict with existing architecture or business rules?
The Bottom Line
AI is not a replacement for engineering discipline; it is a multiplier of the discipline already present in your process.
- If your process is strong, AI will accelerate it.
- If your process is weak, AI will simply amplify the gaps.
In complex systems, those gaps quickly turn into 50 defects for a single business screen. The organizations that win with AI won’t be the ones who just “code faster.” They will be the ones who learn to communicate business logic with precision.
At Foulk Consulting, we help organizations optimize their software delivery lifecycle for performance and stability. Want to ensure your AI strategy doesn’t compromise your quality? Contact us today.
