What are the Tools Used for Performance Testing?

DevOps teams want to be sure they are using the best performance testing tools available. But understanding which tool is best for performance testing has to go beyond a simple consideration of which performance testing tool is easiest. In truth, there are a wide variety of tools that can serve different functions—finding the right set for specific scenarios takes experience.

So what are the tools used for performance testing at Foulk Consulting? Deciding between different test environments and test automation platforms requires strategic thinking about the performance test requirements and how the results align with business objectives. Let’s take a close look at the tools we use for performance testing to give a sense of the scope of tools that are available and how every solution might serve different needs.


What are the Tools Used for Testing?

From pre-release application testing to continuous monitoring and testing releases queued for deployment, these tools are powerful applications we trust to serve DevOps teams in performance testing and beyond.

  • Dynatrace: Dynatrace is an application performance management provider. Built from the experience gained with industry-recognized legacy solution, AppMon, Dynatrace is equipped to carry out five categories of tests including load testing, time-based metrics like response time, front-end performance testing, API testing, and more. The platform fully integrates with performance testing tools like JMeter and NeoLoad.
  • Instana: Instana is an application performance monitoring platform that provides DevOps teams with real-time insight into the performance metrics and changes of their product. Instana empowers teams to pivot testing away from looking for problems that will come up in the future, and instead focus on finding answers that are needed today.
  • NeoLoad: NeoLoad allows for the code-less design and automation of performance tests to be carried out throughout the development pipeline. This allows teams to plan for resource-intensive testing like load testing to be carried out when resources are guaranteed to be available, identifying issues like performance bottlenecks even when advanced user behavior is being modeled. Automatic test updates mean once a test is created, it can be re-used on many different projects when the same requirements need to be tested.
  • Flood.io: Flood.io is a load testing platform tool that is both in the cloud and on-premise. Flood.io’s products include real browser load testing, an on-premise load generator, historical performance analysis, and multi-cloud load generator orchestration. A load test at Flood is anything that uses JMeter, Gatling, Selenium, or Element (Puppeteer) to define user behavior at either the protocol level.
  • Zenoss: Zenoss allows teams to optimize application performance no matter how complex the infrastructure. Real-time data streaming alerts teams when an event occurs, while machine learning analytics help Zenoss understand what notifications need to be escalated for attention. This allows the entire IT ecosystem to be monitored at scale, with integrated capacity planning that lets teams continually improve to ensure responsive systems.


Outsource Performance Testing Tools Comparison and Management to Foulk Consulting

Whether you’re looking for the front-end performance testing tools that will help ensure a satisfactory user experience, or wondering which is the best performance testing tool, Foulk Consulting invites you to take a step back from worrying about the details by reaching out to us for a conversation. Understanding your test requirements, objectives, and the business goals supported by your product is enough work without also having to worry about how to get the testing done. As a high-caliber, experienced solutions IT partner, you can trust Foulk Consulting with your investment of personnel, expertise, and infrastructure.  We empower our clients to better understand, deploy and take full advantage of their software investments while delivering high-quality solutions that exceed expectations.

The Foulk Consulting team is passionate about continually exploring the performance testing tools available to ensure our clients are employing and delivering ahead-of-the-curve solutions. Whether it’s helping you structure and carry out testing internally or serving as third-party software performance testers to add the benefits of deep performance engineering experiences to your efforts, our ultimate goal is helping you serve end-users better. We’d love to start winning your trust—reach out and get to know us.

Think Time Vs. Concurrency

Performance testing is the art and science of generating load or stress against a system or application environment. By replicating real-world situations, as well as the extremes to which the platform might be used, developers can gauge the infrastructure’s ability to support and sustain that load.  

Did you ever do an experiment where you had to build a container that let you throw an egg off a building and have it reach the ground safely? This type of experiment is actually similar to the idea of performance testing. You come to the table with a prototype and then throw it off the building to find out if it works. Once you’ve found what doesn’t work, you redesign and reinforce those weak spots to make sure that egg stays safe and sound when it’s time to submit for a grade—in this case, customer and end user experience.  

This testing, to make sure that systems and applications function properly, comes with a price tag. When companies have already gone over budget on developing a digital product, the idea of more expenses to find out what’s wrong may not be appealing. Some companies, when their budgets start to tighten, will try to save money on performance testing with a bad idea: spend less money by reducing the number of concurrent virtual users being tested. Instead of testing what happens when a hundred thousand people try to access their portal at once, they decide that only ten thousand are in the budget. 

Then, to try to get away with having fewer concurrent virtual users, test designers reduce the think time to ‘trick’ the system into thinking there are more users than there actually are by increasing velocity with fewer virtual users. 

The less that virtual user behavior mimics real user behavior, the more abstract the comparison becomes.  This marginalizes the value of the test by decreasing the accuracy of the performance analysis. How do real people use systems? They need time to think—from reading the text on the screen before clicking “log in” to typing in their personal information, people don’t just click through a system in a few short seconds.

So, when a business wants to test their new app with a lesser number of virtual users, they compensate by also cutting corners on the think time. They believe that making the actions happen faster still adequately tests their system. 


What happens when you reduce the number of virtual users? 

In a nutshell, using this strategy leads to skewed data and an unrealistic testing environment. Let’s take a look. 

Remember the business from earlier? Imagine they need to test the impact of 93,000 transactions per hour total, to recreate the conditions that were seen during a recent peak load event. 

Calculations have indicated to their quality assurance team that with the load test scenarios and scripts, they need 1500 concurrent virtual users to hit the target number of transactions. This works out to effectively 62 transactions per hour per user. This means that each virtual user must iterate—go through a single loop of the load test script or action—62 times per hour, or slightly faster than one iteration per minute. 

However, when the company received the licensing cost for our scenario, it exceeded the budget by a significant margin. They realized they can only spend enough to license 500 virtual users. So, their quality assurance team asked to find a way to configure the test to generate the same load (93,000 total transactions per hour) with 500 virtual users instead of 1500. 

This means that each virtual user would have to do three times the work, or put another way, complete each iteration in 1/3 the time of the 1500 virtual user model. Remember, the original number was 62 transactions per hour to hit our target numbers. So in this decreased virtual user model, that number will have to triple to 186 transactions per hour per user. Or, said another way, each iteration would need to complete in less than 20 seconds. Decreasing think time in this way causes a larger issue. 


Why does it matter if you reduce think time to compensate? 

Reducing think time may seem like a brilliant solution at first. With lots of virtual users, they may be “sitting around doing nothing” for a lot of the time—the total iteration, that is the time from beginning to end of the interaction, can be fairly lengthy. The idea is that having fewer virtual users who do more things quicker and have a shorter iteration could simulate the same effect as having lots of virtual users, and essentially make your server think there are more users than there actually are.

However, server response times are the key variable to hitting these numbers. With a reduced number of virtual users, if an iteration takes longer than 20 seconds (with or without think time), we will be low on our total transaction count. We are also still constrained by server capacity and optimal response times. That user who is “sitting around doing nothing” is still taking up space on the server in a way that matters for real-world performance. Failing to account for them in the test means the test results aren’t fully demonstrating what the team wants to learn. 


What if you tried it anyway?

We did! So, for the sake of the experiment, we assumed that it is possible to run an iteration in as little as 20 seconds. It should be noted, however, that your system may not be able to do this even under ideal/baseline conditions, which would render this entire experiment invalid. 

We tested this hypothesis—that we can reduce concurrent virtual users to 1/3 of the ideal number by reducing think times—and found that we needed to ask what effect this change of think time will have when server response times degrade over time. Remember, we chose think time as our throttling mechanism to try to control throughput (the amount of transactions produced over time during the test), but our actual throughput is ultimately throttled by server response times that we cannot control. 

When we changed the think times and target concurrent virtual users from 1500 to 500, we began to see an unexpected impact to our total transaction throughput. When the servers ran faster than expected, we overshot our transaction volumes by large numbers, and inversely, when server response times slowed down, we greatly undershot our transaction volume target. 

What this showed us is that think time is not a reliable variable to compensate for lowered concurrency of distinct virtual users. Transaction response times have a greater impact on system throughput than think time does in this model.


So, what have we learned? 

We’ve discovered exactly why you shouldn’t use reduced think time to cut corners in performance testing. There are some cases where think time can be used to effectively throttle load, such as stress testing — the goal being to empirically observe what the servers can do under heavy load — but that is very different from trying to recreate an observed load condition or hit a specific set of KPI requirements. 

Think time provides more than just a simulation of human behavior, it is a constructive buffer for your tests that effectively creates a cushion effect when server stress fluctuates. If you remove or dramatically reduce your think times to chase throughput, you will eventually reach a point where you have nowhere else to go but to add concurrent virtual users, so why not just do that right from the start?


How can I find performance testing that I can rely on?

Here at Foulk Consulting, we are committed to helping with the testing methods and metrics that give you a clear look at how your systems and applications will respond to real-world use. We don’t cut corners here, and our industry-proven performance engineering experts can help you with anything from ad-hoc consulting to mature performance engineering processes to regular performance validation. We’re here to give you the power to identify and solve your performance problems using the best technology and processes available. Contact us today to share your story and learn about the solutions we can provide.

What are the Types of Performance Testing?

Performance testing is an essential part of the software development process. It is used to evaluate how well a program or web application performs. It encompasses many different types of non-functional testing that analyze the non-functional requirements of an application or environment to evaluate its readiness to perform under various conditions. Performance testing exposes key performance areas such as speed, capacity, consistency, and recoverability.


There are many different kinds of performance testing, and each type of testing will provide mechanisms to assess different aspects of an application or environment. Read this article to learn about the different types of testing, including the most common types and examples.


What are the different types of performance testing?

The types of performance testing in software testing include: 

  • Load Testing
  • Endurance Testing
  • Stress Testing
  • Spike Testing
  • Scalability Testing
  • Volume Testing
  • Peak Testing
  • Throttle Testing
  • Security Testing


What are the more common performance tests?

Not all of the above performance tests are common or used in most situations. In fact, some teams are beginning to consider security testing as its own unique entity separate from performance testing altogether. Below are the most common types of performance testing with examples.

Load Testing

Load testing measures an application environment’s performance and response time under expected or specific amounts of use. It typically takes place over at least an hour, and should not push beyond expected amounts of use.

Load testing is designed to test an application environment’s speed and capacity. It determines how many users a program can handle, response time under varied amounts of use, and bottleneck issues related to memory leaks and network delays.

A team might perform a load test by sending a large amount of transactions to a server within a short timeframe to see how well it performs. Historically, load testing was performed manually, but today it is automated with the use of performance testing tools.


Endurance Testing

Endurance testing, also called soak testing, is a type of load testing that evaluates an application’s ability to handle average or expected amounts of load over a lengthy period of time, from 4 to 24 hours. 

Endurance testing is designed to test an application’s capacity after the initial deployment. It is especially important as it can identify potential memory leaks, performance degradation, or connection issues that could hinder performance or cause a system failure over time. 

An endurance test may be completed by running an application for hours to see how well it performs.


Stress Testing

Stress testing is similar to load testing, but it is performed at significantly increased load levels. It verifies the stability of an application under extremely heavy use. There are several types of stress testing, including application stress testing, transactional, systemic, and exploratory.

The goal of stress testing is to measure an application’s recoverability, or the way it recovers after a failure. Stress testing, then, should push an application to extreme conditions to illustrate when a system failure occurs and how it performs post failure.

As stress testing is designed to push an application to its limits — and beyond — to determine where they are, a sample test could be any business operation that is the focus of the testing. Options could include creating sessions, creating orders, performing searches, or some other combination of business processes that are used to find the limits of the servers. 


Spike Testing

Like stress testing, spike testing is designed to measure an application environment’s performance under extreme use. However, spike testing specifically assesses performance under unexpected or extreme increases or decreases in load.

Spike testing ultimately seeks to measure an application environment’s consistency and recoverability. It is used to analyze recovery time after an extreme spike to determine how stable it is and how well it recovers.

A spike testing scenario might involve your normal load test, but adds an additional configuration, such as having many users hit a website within a very small window of time, like during the minutes following a flash sale or the stroke of midnight on Black Friday.


Scalability Testing

Scalability testing measures an application’s ability to perform under gradually increasing or decreasing loads. It is designed to test whether the application can handle the growing need and at what point it stops scaling.

The goal of scalability testing is to measure capacity to ensure that an application can handle regular site growth over time, targeted marketing campaigns, or special seasonal loads. 

To run a scalability test, you might gradually increase the amount of network usage on an application to see how it responds.


Volume Testing

Volume testing, also referred to as flood testing, measures a server’s ability to handle a large or extreme volume of data. Volume testing can help identify errors that may occur due to a large amount of data.

Volume testing seeks to measure the capacity of your servers. It can enable your company to see your servers operating in ways that require analysis and work to improve their performance.

An example of volume testing would be to load a large amount of new data into your server to see how well it supports the new data set.


Less Common Performance Tests

Here are brief definitions of types of performance tests that are considered less common. While less common, these tests are still worthwhile to perform for many different programs and applications.

  • Peak testing is similar to endurance testing, but with a much higher load and a much shorter length of time.
  • Throttle testing is done to check how well a system will run for users with low network connectivity. 
  • Security testing, sometimes considered to be its own measure separate from performance testing, is designed to check for vulnerabilities.


Simplify Performance Testing with Foulk Consulting

For nearly 20 years, Foulk Consulting has provided high-caliber, industry-leading services across the IT landscape to both small and large businesses. Foulk Consulting provides companies with highly-experienced quality assurance professionals who can help your team determine what types of performance testing would benefit your business and what types of performance testing are needed to define and measure performance to help guarantee success for your business. Whether you are trying to identify performance issues prior to launch or you have experienced system issues that need correcting, we can help. Contact us today to get started.

What are the Requirements for Performance Testing?

Performance testing is a crucial process in the course of an application’s development. Performance testing assists a development team with information needed to see how software or web applications will perform under a specific workload. It is important that performance testers are looking at the right metrics to really understand how software will perform in the real world. When carrying out a performance test, there are specific requirements that are crucial for the developers to measure so that they know the product will perform under pressure. So what are the requirements for performance testing


Let’s review the common requirements for performance testing, examples of these requirements, and review best testing practices. 


What are the requirements for performance testing?

Performance testing is a progressive, repetitive process. Any given test will uncover some bottleneck or design issue with an application or environment, but there may still be many issues remaining that are masked or blocked by one issue that was just uncovered. Until this first discovered bottleneck is removed, the hidden system performance problems will remain hidden. 

Since performance testing is an iterative process of testing, uncovering bottlenecks or other issues, resolving those issues, and then retesting, each of these cycles of test and remediation is a round of testing that requires a follow-up effort to ensure that the uncovered problems have been resolved.

There are many ways to measure speed, scalability, and stability. The required metrics often measured in performance testing are:

  • Response time: This is the total time to send a request and get a response.
  • Time to first buffer or wait time: This tells developers how long it takes to receive the first byte after a request is sent. It is the measure of the time it took the server to process the request.
  • Peak response time: A peak response time that is significantly higher than average will require some investigation to determine whether it is a fluke or an indicator of a larger underlying problem.
  • Error rate: This calculation is a ratio of requests resulting in errors compared to all requests. Careful analysis of errors is needed to fully understand why they happened, which part of the application architecture generated them, and the conditions that led to them. Many errors are caused by bad test data and faulty assumptions used during the test script definition process.
  • Maximum concurrent users: Also known as concurrency, this is the most common measure of how many active users a system will handle at any point.
  • Requests Per Second: Requests per second is the measure of how many requests are sent to the server in a second. It is the rate at which work is sent to the servers in the given interval of time. This can also be expressed in many other intervals (milliseconds, hours, days) depending on the customer’s business needs.
  • Total requests: The final sum of all requests submitted to the server during a test. This number is helpful when calibrating your tests to meet specific load models.
  • Passed or failed transactions/requests: A measurement of the total numbers of successful or unsuccessful transactions and requests. Generally, a transaction is a logical unit of business work. A business transaction may have one or more requests associated with it. A request is a unit of work sent to a server that we expect a response for. 
  • Throughput: This is the amount of data sent to and from the application servers over time. Throughput shows the amount of bandwidth used during the test and can be measured by kilobytes, megabits, or another measurement.
  • CPU utilization: The amount of CPU required to support the workload of the test. It is one indicator of server performance under load.
  • Memory utilization: The amount of memory required to support the workload of the test. It is another indicator of server performance under load.


Performance test example

For a solid understanding of performance testing, here are a few performance test examples:

  • An e-commerce website creates thousands of new accounts within a specified period of time to test the maximum workload it can handle before bogging down.
  • An e-retailer has a new ad campaign that will go live soon. They expect the link used in the advertisements to generate exceptionally high server demands. How high can the servers go before there is a problem? Is that high enough? What would it take to double the capacity of the current system?
  • A game developer tests a server for a new multiplayer game by having thousands of test accounts log on at the same time to test how many requests it can handle simultaneously. 


Performance testing life cycle

The performance testing life cycle has seven steps: 

  1. Identify the testing environment: Also known as the test bed, a testing environment is where software, hardware, and networks are set up to execute performance tests. Identifying the hardware, software, network configurations, and tools available allows the testing team to design a test that measures the required metrics and may even be repeatable in future use cases.
  2. Identify performance goals: In addition to identifying metrics such as response time, throughput, and constraints, identify the requirements and success and failure criteria for performance testing.
  3. Plan and design performance tests: Identify performance test scenarios that take into account user variability, test data, and target metrics.
  4. Identify Data required for testing: Data will make or break your testing effort. Tasks required to manage your test data include: 
    1. Identifying the types of data needed for each business scenario.
    2. Identifying and meeting the conditions of business requirements for the required data
    3. Identifying how to gather the data for testing
    4. Identifying how to load the test (or production) systems with the necessary data
    5. Identifying how to clean-up data and restore it after testing
  5. Configure the test environment: Prepare the elements of the test environment and instruments needed to monitor resources.
  6. Implement your test design: Taking the test design that the test team has developed, create the tests.
  7. Execute tests: In addition to running the performance tests, monitor and capture the data generated.
  8. Analyze, report, retest: Analyze the data and share the findings. Run the performance tests multiple times using the same parameters and different parameters.


Performance testing best practices

An important principle in performance testing is testing early and often. A single test will not tell developers all they need to know. Performance testing requires a progression of tests that target specific business requirements. In that process, you will find issues that require resolution, and then you must also validate that those issues have been resolved. Do not wait to the end of the development cycle and rush performance testing as the project winds down or is completed — there is value in load testing the various key functional components of a system before they are fully integrated.


In addition to repeated testing, here are 9 performance testing best practices:

  1. Involve developers, IT, and manual testers in creating a performance engineering practice.
  2. Remember real people will be using the software that is undergoing performance testing. Determine how the results will affect users, not just test environment servers.
  3. Go beyond performance test parameters. Develop a model by planning a test environment that takes into account as much user activity as possible.
  4. Always capture baseline measurements, because they provide the starting point for determining success or failure as you push system limits with testing.
  5. Performance tests are best conducted in test environments that are as close to the production systems as possible or ideally in production or a disaster recovery environment.
  6. Isolate the performance test environment from the environment used for quality assurance testing. Don’t allow functional or user acceptance testing to corrupt or negatively impact your performance testing process. If those activities must take place in parallel, ensure that you have an isolated performance testing environment to minimize result contamination and corruption caused by uncontrolled activities that are outside of performance testing.
  7. No performance testing tool can test every application architecture, and limited resources may restrict choice even further. Research performance testing tools to find the appropriate tool for your application architecture and needs.
  8. You can never gather too much data. Use tools that will give you a wide range of statistics like minimums, maximums, averages, standard deviations. The totality of what is absorbed from all the data available will contribute to what you need in order to perform analysis and make recommendations. 
  9. Know the audience when creating reports that share the findings and results of the performance testing efforts, and craft your reporting accordingly. Including system and software change logs in the reports is highly recommended. 


Count On Foulk Consulting For Reliable Software Performance Testing 

Since 2002, Foulk Consulting has been committed to giving businesses the power to solve problems through technology. Our industry-proven performance engineering experts help you determine the testing methods and metrics that will give you a clear look at how your software will respond to real-world use. Our services are available along a spectrum from ad-hoc consulting all the way to mature performance engineering processes and ongoing consulting to support your product from initial release through every update that follows. Allow Foulk to help you assess your needs and capabilities, to find the performance testing tool that best fits what your business needs today.

Whether you are trying to get ahead of performance issues or have already experienced defects that you need to follow up on, we can help. Contact us today to share your story and learn the solutions we can provide.

Crunch Time Clarity

The final Push to Prod is always exciting. I will also say that Go Live week is always Fun because I only remember the good times, which is amazing because the best deployments were typically followed by a pub crawl up Moody Street.

As I work through two “Go Live” weeks, this week, Team A is coalescing into a tightly-knit unit while Team B is bogged down in the swamp of minutiae, too far away from their critical path.

Planning and preparation have been better in Team A, which is performing in-tune and on-time. The lack of planning and preparation in Team B has them learning new skills and techniques during crunch time.

Neither team has any show-stoppers in their application; both applications have good quality. The deployments may go well for both groups. However, if there are any challenges during the deployment action, Team A is relaxed enough to handle adversity with clarity. Team B is not.

The state of mind held by Team A begins with their CEO. He brought the leadership team together this week and said, in his Scottish brogue, “Isolated incidents are OK, we want to be on the watch for systemic problems. Remember gents, it’s just got to be good enough.”

This insight from leadership, in a critical moment of the project, was just what the team needed to release any undue pressure they had and put them on solid footing for the final push to production.

Hopefully, everything will go perfectly for Team B and they won’t face any moments which require dynamic clarity.

Written by Steve McGinley

How New Jersey Prepared for a Flood of COVID-19 Loan and Grant Applications

Check Out Tricentis Flood’s New Blog Post:


#Tricentis #FoulkConsulting

Filtering out Synthetic Monitoring Traffic from Google Analytics

We have been using automated scripts to monitor web sites performance and availability. These scripts are usually launching a real browser and navigating through pre-defined steps to capture the timing and report back at least one checkpoint showing a pass/fail. The issue that recently came up that the Google Analytics reports show a few spots on the map with high traffic and those were a direct result of these scripting running every 15 minutes from 3 locations in the USA. The request came in to come up with a way to filter this monitoring traffic so that the reports would reflect real users only and exclude the monitors.

We explored a few ideas based on what Google Analytics (GA) offered. The first idea was to modify the Accept-Language header to include a test value like zzzz. We tried both adding the value to the header and replacing the entire string with the new value and neither approach worked – GA just reported the same language of en-us.

We also tried adding in a custom header name/value pair. Custom headers have an X as their first letter followed by a name we wanted so we tried X-synthetic-monitor along with value like the name of the script. For some reason, these custom headers could not be viewed in the GA tool.

We then tried a custom cookie name/value pair hoping those would be passed along to GA and alas they were not.

We tried resizing the browser to known value like 1234×987 however the JavaScript running on the page was resizing the viewport based on its own algorithms. The viewport size was also changing during the life the script as the code stepped through a workflow and the JS responded to those inputs. This is how most sites are set up today so we ended up with a range of values for width and height for the browser instead of fixed values we could pick out of the noise. Therefore resize was not a good candidate for filtering traffic.

We also took a look at filtering by the IP addresses of the machines that run the synthetic monitoring scripts but we realized those IPs were dynamic and we had no way of knowing what they were at any given time. If we looked at the current IP of a monitoring machine that could change quickly due to a VM migration, a reboot, or even load balancing to another VM. IPv4 addresses are an unreliable way to identify a particular user even a synthetic one.

Ultimately we settled on blocking the domains for Google Analytics completely in the script’s runtime settings. GA depends on JavaScript code running in the browser, gathering up data, and then sending that data to Google. If we just prevent those requests from being made by blacklisting the domains then the data never makes it to the GA system. This achieved the goal for the team as they no longer had to tweak the filters and could trust that the scripts we deployed were not included in the analytics data.

These are the domain suffixes we chose to prevent or blacklist:
– google-analytics.com
– googletagmanager.com

The other approaches that we tried would probably work for web servers that the customer controls, especially the one at the origin. We could probably investigate the logs and see the custom headers and cookies that are being sent along. We chose to leave those in place just in case the wanted to analyze logs or traffic at other points in the system outside of what Google Analytics shows.

Technical Implementation Details

Tool: LoadRunner 12.60
Protocol: TruClient Web
Domain Blacklisting: Runtime Settings > Download Filters > Exlude only addresses in list
HostSfx: google-analytics.com
HostSfx: googletagmanager.com


Modifying an existing header and merging in the new value:

truclient_step(“1”, “Execute Utils.addAutoHeader ( ‘Accept-Language’ , ‘Zzzz’ , true )”, “snapshot=Action_1.inf”);

Adding in a completely new custom header name and value:

truclient_step(“2”, “Execute Utils.addAutoHeader ( ‘X-synthetic-monitor’ , ‘mymonitoringscript’ , false )”, “snapshot=Action_2.inf”);

Resizing the browser to a specific width and height:

truclient_step(“6”, “Resize browser window to 1234 px / 987 px”, “snapshot=Action_6.inf”)

Written by JJ Welch

How Much Influence Should Developers Have Over the Tools They Use?


Excellent software has been produced by development teams that had total control over what tools they used and also by teams that had their tools mandated by management.  On the other hand, poor software has also been delivered by both approaches.

Here are some thoughts to consider as you decide how your tooling choices are made.


Development teams are more autonomous and have more tool choice

  • The team becomes more invested in the process of building code.
  • The more invested your teams are, the better their product should be.
  • This reduces the need for a systems administration team and the number of employees in it.

The teams are less autonomous and have less tool choice

  • Fewer tools to rely on for producing output means there is less variance to consider in effort projections
  • Fewer pockets of tooling knowledge reduces knowledge fracturing
  • Less license sprawl reduces cost
  • Teams that use the same tools can cross-pollinate the lessons they learn


Development teams are more autonomous and have more tool choice

  • Tool sprawl, which leads to consolidated pockets of tool-centric knowledge and devalues remediation groups such as SRE’s.
  • Lack of standardization in the development process, which leads to difficult deconstruction when remediation is done by groups not involved with the initial development process.
  • An individual developer becomes a chokepoint, should they become unavailable.

The teams are less autonomous and have less tools choice

  • There may be better tools to use.
  • Your team may feel de-valued and become disinterested

Additional Influencers and Considerations

Who is responsible for the maintenance of the code, the Site Reliability Engineer’s or development team?

The team responsible for the viability of the tools should have influence.

Who pays for the tool and its administrative upkeep?

The group fiscally responsible for the tool should have influence.

How well does a particular tool integrate with your existing tooling ecosystem?

Tools that integrate easier and better should carry more weight than those that don’t.

Does a tool have a killer feature that your team MUST have?

This should be considered and weighted appropriately.

How comfortable are you with staff turnover?

If you have a well-defined requisition that is easily filled, then you’re in a position to accommodate a higher turnover rate than if you are trying to replace a specialist.  In other words, let specialists have more influence than less specialized roles.

The easier it is to collect the data required for your quality and delivery KPI’s, the less time is spent on normalization; which reduces your data’s time to market.

Normalization efforts are typically a highly manual effort.  Reducing the amount of manual intervention increases the accuracy of your governance metrics.

By applying control at the point you perform evaluations, the maximum opportunity is provided for individual tool choice.

A happy developer makes useful software and happy users.  We don’t want your engineers feeling like Bill Parcells.

There will be a proper balance of accuracy, time to market, and staffing for your situation.

Written by Steve McGinley

Shared Application Infrastructure Can Be A Critical Performance Constraint and Blindspot To Your Business

The Scenario

Your business has constructed a variety of application environments designed to pursue and expand your mission-critical business plans and gain market share. Over time, security concerns, maintenance and user convenience steered you to building a unified, shared authentication infrastructure such as Identity and Access Management (IAM). 

In this situation your financials system uses IAM, as does your time recording, expense, sales tracking, ticketing and development systems. They all tie-in to a single system for authentication and datacenter asset access control. This has saved your business countless hours in maintenance, unified your account management efforts, and put a singular, controllable and effective system in place to manage this part of your business. So what’s wrong, doesn’t this solve your problems? 

Foulk Consulting can help you identify and avoid the type of problems that the following case study highlights. Here’s how we did it with a high profile Internet E-Retailer. 

Case Study: E-Retailer Preparing for Peak Usage Season

An E-Retailer built multiple, external, customerfacing applications that they also invested a lot of time and effort to performance test them all. The testing goals were ultimately clear and refined, they were based on each of those application’s usage models during the preceding peak season, combined with other business requirements and usage projections for the upcoming peak season that had to be met. In the course of these performance testing efforts, each test was focussed on a specific customer-facing applicationtesting and validating that application’s ability to sustain the expected load and perform within its defined success criteria. 

Through their hard work they built a case internally that they were very well prepared for the upcoming peak season. However, when the peak season hit, the reality of the situation became clearthat something was terribly wrong and the systems could not handle the actual peak customer loads. 

The IAM infrastructure was tested with each application independently and it performed as expected.  

During their annual peak eventuser sessions were dropped at an unexpected and unacceptable level. Most of the new users could noeven register new accounts, nor could most returning customers log in to make their purchases. The failures to convert shopping activities to sales resulted in them missing their projected revenue numbers by a large margin, in fact they faced crippling losses for the year. A story that has been played out too many times in recent years. 

So how did this happen? Didn’t we say that they validated this load before the peak season? It sure seems that they did their due diligence and stressed the machines beyond the traffic levels that they expected, which were even higher than the year before. How is that possible? Did they test the wrong things? No. Did they fail to test dependent subsystems to the right levels? Not exactly. How can you create good tests, find defects and configuration problems, resolve them, validate your efforts, demonstrate that your testing exceeded the loads you were expecting, and still watch your systems fail under real user load? 

As a result of this missed business opportunity and their public hit to customer confidence, Foulk Consulting was contacted to assist in analyzing the situation. We came in and worked with all the involved teams. We reviewed their various testing practices, their planning and assumptionswe dove into the peak load logs and analytics and the event data that was collected, in an attempt to try to find the keys to how this situation manifested itself, and more importantly help this valued customer devise an approach to prevent this sort of thing from happening ever again. 

Critical Analysis & Planning

Coming out of the peak season with this blackeye, the business hired Foulk Consulting to assist them in analyzing the planning and testing leading up to the event as well as the forensic data from the event itself. Our goal was to not only illuminate where the problem originated and manifested itself, but how they could prevent this in the future – to help them solve the actual reasons why they were losing customers and market share, following up on all the hard work they had done before. 

After thorough investigation and analysis, it was found that despite each of their applications being tested thoroughly and each of them proving to meet performance objectives, maintaining stability and performance under load, what appeared out of the piles of data was surprising to everyone in the organization. 

The root cause appeared to be a complete lack of preparation on the IAM tier as well as the dependent login integrations across the various applications when the real customers loaded multiple applications at the same time during peak usage it created a classic bottleneck; the request demand overloaded the system’s ability to respond in a performant manner. 


When each application was tested in isolation, the IAM tier was capable of handling that application load without any problems. When the actual peak production load hit all the applications at the same time, however, the IAM infrastructure became a complete bottleneck and customers could not interact with the systems. This was found to be the root of the peak usage collapse. 

Finding the root cause was only part of the puzzle. Foulk then took that information and applied a performance engineering approach to testing that would ensure that the business was able to identify, isolate and test the systems that demanded attention. The plan was designed to focus the customer’s efforts and identify exactly who was responsible for resolving the issues during their next preparation and deploymencycle. 

Ultimately we were able to work with the various application teams and construct a performance test that stressed the IAM tier realistically, reproducing the production peak load failure model 

Armed with this repeatable test structure, the customer was then able to work with the application support, operations and vendor teams to work through the issues and build the IAM infrastructure to pass the load test criteria and remove their significant production performance bottleneck in the IAM tier. 

Written by Brian Brumfield