Organizations no longer have the luxury to trade-off ‘quality’ over ‘speed,’ or vice versa, when delivering software. The organizations today have started to realize that in order to stay competitive, they need to deliver high-quality software at an accelerated rate by keeping the cost to a minimum. Organizations aspire to reach there by leveraging Agile and DevOps methodologies.
“How to achieve quality faster?” Continuous Testing is the answer to the core question.
Shift-Left + Shift-Right = Continuous Testing
We hear the Agile and DevOps teams speaking about ‘Shift Left’ and ‘Shift Right’ in context to software testing.
Shift-Left means moving to the left in the development lifecycle and test as soon as possible. Though this buzzword is trending, the concept was already well known from the principle of Boehm’s Curve: “The later the defect found, the more expensive it is to fix”
Shift-Right means moving to the right in the development lifecycle and testing in production. This includes alerts, monitoring, feature toggles, staged rollouts, fallbacks, etc.
What is Continuous Testing?
Continuous Testing means testing throughout the development lifecycle. Testing is an ongoing activity and not a phase at the end of development. Continuous Testing is a cultural shift from testing at the end, to testing early, testing often, and testing at all stages of development, by taking advantage of automated tests to ensure the business value is being achieved as expected.
Continuous Testing utilizes automated tests to assess business risks and get instant feedback for the release candidate aiming to increase release velocity and ensure high quality. Continuous Testing is a mindset where you embed testing at every stage of the software development lifecycle.
Continuous Testing is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate (Wikipedia).
The biggest benefit of embracing Continuous Testing is reduced risk and accelerated release velocity and software delivery. Continuous Testing helps you to answer the question: “Does the release candidate have the accepted level of risk?”
Continuous Testing and CI/CD
Traditional testing practices don’t work in a DevOps or Agile setting. The DevOps teams use modern development practices, like:
- Continuous Integration
- Continuous Deployment
- Continuous Delivery
These practices help the team to get early feedback aiming to assess, learn, and improve continuously. The differences between these buzzwords can be often confusing. Let us learn what they mean and how they are related to Continuous Testing.
| Continuous Integration | A practice where developers integrate code to their main branch/common repository frequently— several times a day. Each integration is verified by creating a build and running automated tests against it. This avoids “Integration hell.” |
| Continuous Delivery | Continuous Delivery is an extension of Continuous Integration. A practice where software is built in such a way that it can be released to production at any time.  To achieve Continuous Delivery, we need Continuous Integration in place. A Continuous Delivery pipeline verifies the application via automated tests but keeps the deployment decision as a manual step. |
| Continuous Delivery | A practice where every code change goes through the CI pipeline and is put into production automatically, resulting in daily production deployments. |
Continuous Testing is often implemented as a set of automated tests that are executed in the CI/CD pipeline. Continuous Testing is inevitable for Continuous Integration, Continuous Delivery, and Continuous Deployment.
Continuous Testing-Test Automation
Continuous Testing cannot happen without Test Automation. However, Test Automation is not synonymous with Continuous Testing-Continuous Testing is much more than Test Automation.
Test Automation aims to prevent repetitive tasks, improve test coverage, as well as accelerate the speed and accuracy of testing. But the automated tests may not always be devised with business risks in mind. Low-level tests are also a part of Test Automation. Continuous Testing demands extensive tests that are capable of giving quick feedback and preventing high-risk release candidates from being rolled out to end customers. Continuous Testing is not just about Test Automation. The objective is to run many different types of tests, including exploratory, usability, performance, and security, throughout the delivery process. The Testing Quadrants (Gojko Adzic’s version) outlines the testing types required to achieve continual quality.
Introduced by Brian Marick and further worked upon by Lisa Crispin
Transitioning to Continuous Testing
Even though Continuous Testing is key for Continuous Delivery, most of the organizations are far from reaching it. Continuous Testing requires a significant shift for development teams to succeed. Below are pointers:
- Devise an Intelligent Test Automation Strategy.
Test Automation is the biggest enabler of Continuous Testing. But succeeding in Test Automation is lengthy and multifaceted. Defining your goals and adopting the right approach is the first step in ensuring success; after this, Test Automation success is reliant on choosing the right people and tools, as well as following a structured methodology. Learn more about devising an intelligent Test Automation strategy here.
- Embrace CI/CD.
CI/CD is the backbone of the modern DevOps environment for delivering code changes more frequently and reliably. With Continuous Testing, Automation tests executed in the CI/CD pipeline provides continuous feedback, enabling the product to be always production-ready and can be deployed at the push of a button.
- Continuous Testing Needs Cultural Shift—Everyone Contributes to Quality.
From a cultural point of view, collaboration is the key to Continuous Testing. Dev, QA, and Ops all have to work together. Building a quality mindset in the team becomes essential to find success in Continuous Testing-“Quality is everyone’s responsibility.” Developers contribute by automating unit tests and integrate with CI/CD pipeline. QA contributes by integrating automated integration, functional, and non-functional tests to the pipeline. Ops contribute in terms of production infrastructure automation, monitoring, etc. In this way, everyone is involved in testing and Automation activities.
- Choose the Right Automation Solution for CT.
Continuous Testing is highly tool dependent. You need to choose the right CI/CD tools, Automation tools, service virtualization tools, and the list goes on. The selection of the Automation tool plays a major role in the success of Continuous Testing. Keep in mind the budget, maintenance, support, documentation, and the learning curve are involved when selecting the Automation tool.
Keep in mind that tools need to adapt to us-not the other way around. It’s wise to find a toolset that can work with your available skills, where the learning curve is minimal. There are plenty of codeless Test Automation tools, like TestArchitect, that have freemium options for teams to get up to speed on automating.
- Create Strong Requirements and Define Tests Early.
According to “Continuous Testing Report 2019” by Capgemini, 64% of the defects occur in requirements. Lack of clear requirements will result in rework, as well as wasted time and effort. It is important to ensure that our user stories are testable, and we clearly define the ‘Acceptance Criteria’ at the earliest for a common understanding of the ‘Definition of Done’ between all the team members.
Practices like static testing, behavior-driven development (BDD), acceptance test-driven development (ATDD), keyword-driven testing, Action Based Testing (ABT), and usage of model-based testing tools can greatly aid to overcome requirements ambiguity and define tests early.
- Create Virtualized Test Environments on Demand with Production-like Data.
QA and Dev teams should be provided with the ability to create virtualized test environments on-demand with CI/CD tools to reduce waiting times and Ops dependencies. These environments should be similar to that of production. These are short-lived environments and destroyed after use. These test environment should also include test data on demand. The ‘test data quality’ matters! The test data needs to be as close as possible to that in production, or else there are high chances that we do not find potential issues that might happen on production. Tester’s need to embrace Virtualization and Containerization with likes of Docker and Kubernetes. Virtualization and Containerization also help testers streamline the dependencies of their test tools, as well as parallelize their test execution of tests with different datasets that are highly reducing the execution time.
Summary
Continuous Testing acts as the key driver for DevOps. It’s not enough to release great software code once, organizations need to do it continually. Continuous Testing gives the business the confidence to ship products faster by minimizing risks and unblocking the testing bottlenecks in the Agile/DevOps environments.
Success in Continuous Testing requires expertise and experience. LogiGear is one of the few firms that specialize in Test Automation with the depth and breadth of services to ensure your success. LogiGear can make Continuous Testing to become your business’s competitive edge by continuously releasing higher quality applications faster.
Prashant is an Agile enthusiast and enjoys sharing his experiences by blogging and participating in agile communities around the world. Prashant is a Certified Scrum Master and a frequent speaker at industry conferences. Prashant has helped thousands of ISTQB aspirants to clear their certifications with his free app ISTQB test mentor.