By Nate Rich, Principal Engineer, and Caleb Billingsley, Performance Expert, Foulk Consulting
In the world of software performance and QA, there is a recurring nightmare that keeps engineers up at night: The “Clean” Test Run. You know the one. Your automated scripts pass with flying colors in the staging environment. Your load tests show perfect response times. You give the green light for deployment, only for the production environment to catch fire two hours later because users are interacting with the application in ways you never anticipated.
For years, QA teams have relied on “gut feel,” outdated requirements documents, or “happy path” scenarios to build their test suites. But in today’s complex, distributed architectures, guessing is a luxury we can no longer afford.
If you want your testing to be effective, you need to stop imagining how users behave and start looking at how they actually behave. The key lies in your production telemetry—specifically, Real User Monitoring (RUM).
The Gap Between Simulation and Reality
The primary reason test scripts fail to catch production issues isn’t a lack of effort; it’s a lack of context.
We often build scripts based on how the application was designed to be used. However, users are unpredictable. They take shortcuts, they use obscure browser-extension combinations, and they hit “Submit” three times when a page lags.
By leveraging New Relic RUM data, we can close this gap. Instead of guessing which transactions are the most critical, we can use real-time data to drive our testing priorities.
3 Ways to Use New Relic Telemetry to Inform Your QA
At Foulk Consulting, when we help organizations optimize their testing lifecycle, we focus on turning production “noise” into actionable testing “signals.” Here is how you can use New Relic to build a data-driven test strategy:
1. Identify Your “True” High-Traffic Paths
Most teams test the login page and the checkout flow. But what happens in between? By using New Relic’s Service Maps and NRQL queries, you can identify the exact sequence of clicks (User Journeys) that represent 80% of your traffic.
- The Shift: Stop testing every single feature with equal weight. Use telemetry to identify the “Heavy Hitters” and ensure your automated scripts prioritize those high-traffic paths.
2. Mirror Real-World Environments
I’ve seen countless performance tests run on a “standard” high-speed fiber connection from a desktop browser, only for the app to fail for 40% of users who are actually on 4G Android devices in low-bandwidth areas.
- The Shift: Look at your New Relic Browser dashboard. What is the actual distribution of device types, OS versions, and geographic locations? Use this telemetry to configure your test agents (like those in JMeter or Playwright) to simulate the latency and hardware constraints your users actually face.
3. Capturing the “Long Tail” of Errors
Standard testing catches 404s and 500 errors. But telemetry catches the subtle “silent killers”—the JavaScript errors that only trigger under specific conditions or the API calls that take 8 seconds only when a specific database query is overloaded.
- The Shift: Analyze your production error logs in New Relic. If a specific edge case is popping up for 2% of your users, that’s not an “edge case”—it’s a bug waiting to scale. Incorporate those specific failure conditions back into your regression suite.
From Reactive to Proactive: The Foulk Approach
Using telemetry isn’t just about fixing what’s broken; it’s about ROI. Every hour your team spends writing a test script for a feature that no one uses is an hour wasted. Conversely, every minute you spend hardening a script against a known production bottleneck is an investment in your brand’s reputation.
At Foulk Consulting, we specialize in bridging the gap between Observability and Quality. We don’t just help you see what’s happening in your environment; we help you use that data to build a resilient, automated pipeline that reflects reality.
The Bottom Line
Your users are telling you exactly how to test your software—you just have to listen to the data. By integrating New Relic telemetry into your QA process, you move from “hoping it works” to “knowing it works.”Is your testing strategy based on data or assumptions? If you’re ready to stop guessing and start building smarter test scripts, let’s talk. Contact us to see how we can help you turn your production insights into a competitive advantage.
