Building a suite of automated tests is arguably better than maintaining an army of manual testers, but it can still feel like a leap of faith—especially if this is your first attempt. According to LogiGear’s State of Software Survey Testing, only 19% of respondents were successful with Test Automation implementations the first time. Convincing key stakeholders in your organization who aren’t as knowledgeable about Software Testing and Test Automation often need the oomph of data and numbers.
The goal of this post is to look at the specific upfront and long-term costs of building an Automation framework and comparing it to the benefits you’ll see in a few key areas.
In this article we’ll cover some of the following topics:
- Reducing time-to-market
- Reducing long-term costs
- Increasing product quality
Let’s work through a simple example: We’ll assume the time to design and implement a test framework for your product is fixed at 1 month for 2 engineers, for 100 tests. Let’s say the average salary for an Automation Engineer is $100K per year, which puts the cost of development at $17K. Recruiting and hiring these engineers takes a recruiter (earning about $65K) three weeks to source the right candidates ($4K), and engineering requires 3 similarly-paid engineers 2 hours each to vet the candidate ($2K). The total for your Automation framework is $23K, or $230 per test.
In the same amount of time, you instead hire two manual QA testers at $75K per year. They can run all 100 tests in half the time for $6,250. It takes recruiting 1 week to source the right candidates ($1.5K) and engineering takes 2 engineers 1 hour each to vet the candidate ($200). Running these 100 tests once will only cost you $8K, or $80 per test.
Now, if you’re staring at a major deadline, and the product, marketing, and sales teams are breathing down your neck, manual testing seems irresistible. But in the words of Admiral Ackbar, it’s a trap. The power of Automation doesn’t reveal itself until the second iteration.
Automation Pays For Itself (and Sooner Than You Think)
If your manual development team makes even a relatively minor change, you will need to re-run your suite of tests, costing you about $6K. With Automation, the costs are almost too small to calculate. After 3-4 regression cycles, you’ve already covered your initial investment—but the savings don’t stop there.
As your test suite grows with your product, let’s say each additional 10 tests take 2 Automation Engineers 1 week to automate at a cost of $4K. The same number of tests will take 1 of your manual testers 2 days to write and run. That’s about $300. But remember, that’s for each test run, and we need to add them to our existing suite of 100 tests. So if each release adds 10 tests, that’s only $4K in Automation costs, but $6,000 plus $300 times the number of subsequent code changes for manual testing.
Automation Ensures Timeliness and Quality
Running tests right before a release only lead to last-minute bugs and schedule slips. With Automation, the costs of regression testing are so low, it no longer becomes a chore and potential release blocker; instead, it’s your best tool to deliver a great product. The small increase in overhead to develop automated tests will seem trivial when you’re on your second or third manual regression test cycle.
What you release will be more stable, as Automation brings thoroughness and completeness to testing than manual testing sometimes misses, or doesn’t have time for.
What you release will be more stable, as Automation brings thoroughness and completeness to testing than manual testing sometimes misses, or doesn’t have time for.
Automation frees up your team to now do the following:
- Properly investigate and fix bugs
- Maintain release schedules
- Improve the software through creative exploratory testing
- Improve CI/CD pipelines
Long term, this translates into not just a better, more stable product, but fewer customer service calls, less time spent fixing bugs, and higher team morale.
Automation Supports Growth
Lastly, Automation means you can pull others into your test process. This increases the effective size of your test team, which means that your core QA team can be kept small. Let’s say that as your product grew to require 500 tests, you added 3 more automation engineers to the team, for a total of 5 people at $500K per year. And, for every 50 manual tests that are required, you need 1 additional person to be able to keep up. By the end of the year, your manual testing team is 12 people at $900K. You save almost 50% in payroll alone per year.
While outsourcing part or all of the Automation work to an expert firm like LogiGear isn’t necessary, it can be a beneficial option. The outsourcing firm will supply both scalability and expertise, getting you up and running quickly while keeping the ownership in your core QA team. You’re able to parallelize the effort to build Automation and still maintain the focus of your team. For example, LogiGear previously worked with a small software company that needed assistance in creating an automated test suite. After performing end-to-end testing on their application, LogiGear provided the client with an extensive library created solely for them; this library was highly reusable and allowed them to continue their Automation efforts in-house.
Summary
Designing a maintainable and scalable automation framework can take significant planning and development, and will also need to be maintained over time. But building the right system from day one will mean those ongoing costs don’t escalate or get tied to your product development road map. It’s tempting to take a short term view to satisfy a tight schedule, but patience will be rewarded here.
