Using DevOps Best Practices to Strategize Continuous Testing

As I have had the opportunity to work in environments that have emphasized the role of Continuous Integration and Continuous Delivery (and with it, the goal of achieving Continuous Testing as well), I have also had the chance to see our Operations Team focus more attention on the goals of DevOps and the methodology that lets us steer towards that goal. In many ways, we are actively working towards getting to that effective DevOps goal and each day gets us closer.

Still, you will notice that I say our goal of reaching an effective DevOps strategy. That is because we are still on that journey and I anticipate that we will be for some time to come. Still, there are a variety of steps we have taken that I think will ultimately help us get to our destination, especially if we continue to execute the following ideas. Thus, I wanted to share these steps in the chance that your organization is working towards the same goal so that you can benefit from them.

Paying Down Technical Debt

Unless I get the opportunity to work with a team starting on a brand new project (rare, but I have had that experience a few times), many of the projects I have worked on have been long in service applications and those applications have accumulated technical debt in a variety of ways. The most common is the number of areas where effective automated tests were either missing or behind where the Development Team was in their feature development. For various reasons, most of which come down to shipping on time and “We will get to that later,” eventually later comes and those uncovered and unspoken areas need to be addressed.

This is where the idea of “Continuous Testing” becomes a central tenet of any DevOps strategy. As more frequent releases and deployments become commonplace, there is a need to be able to run all tests quickly and effectively. Having these tests as a core component of the CI/CD process is critical. Of course, when there is technical debt to be paid down, the goal of Continuous Testing may seem out of reach. I would argue that this is the wrong way to look at the situation. While yes, it would be appropriate to pay down all of the technical debt and get coverage and automated tests so that all areas could fit into the Continuous Testing strategy; this is a case where “perfect is the enemy of the good.” Continuous Testing does not mean that everything is tested, but that critical tests are covered and worked into the CI/CD pipeline so that a high feeling of confidence for the code to be deployed is present.

Too often, the idea of Continuous Testing gets pushed in with Automated Testing as though the 2 terms are synonymous. While Continuous Testing needs a certain amount of Test Automation to be effective, what is more important is to consider that testing happens at all stages of the development, building, and delivery process. To make it truly part of a DevOps approach is to also realize that testing begins at the earliest part of the development cycle, continues the entire way through build and release, and then even after the code is deployed with continued monitoring and feedback. In short, my team and I need to not just commit to a strategy and a skill set, we need to commit to an entire ecosystem of tools and techniques that are spread out through the entire organization. Testing is no longer just the domain of the Software Tester, and testing is not the sole or necessarily the key focus.

Understanding Levels of Testing to Achieve Continuous Testing With DevOps

DevOps is not just a team or a set of scripts that run in a hands-off manner. It’s an approach that requires a team of talented individuals working together to ensure that the software delivery process (initial ideas, development, building, deployment, and monitoring) can work as seamlessly as possible. As I have previously mentioned, the goal of testing is not to have it happen at some specific point in this process but at every point in it. This is reminiscent of the Lean principle of “Quality at Every Step.” If we were to consider the model of an assembly line for a vehicle, we would want to be able to apply testing and examination at any step of the process and be able to plan information. That information shouldn’t just be a snapshot of a moment, but also a set of plans as to what to do at that moment to be able to address any problem that arises.

This is no less important in the assembly line of software deployment. Understanding the domains of testing and what methods make sense at each level is critical. For example:

  • Unit testing is important with the initial development of new code, as well as refactoring of legacy systems. 
  • By isolating sub-systems, we aim to make sure individual components, functions, and methods work as they are designed to. 
  • At the integration level, we start to see how these components play together.

We can continue this process as we add more systems and deploy to a more robust environment, one that mimics the production environment or (gasp!) even be so bold as to push to a production environment itself.

One of the key aspects of DevOps is the understanding that testing doesn’t stop once the code is pushed to production. Continued examination through monitoring, analytics assessment, and experiments, such as A/B tests, allow a team to continuously evaluate the system, even after “the car has driven off the assembly line.”

Transitioning From Traditional SDLCs to Continuous Integration and Deployment

Another key aspect to implementing a DevOps strategy long-term is getting used to the idea of developing smaller increments of code and shipping them more frequently. An organization that is used to shipping products once per year will very likely struggle to accept the idea of shipping code once per day, or multiple times per day. For this change to be successful, there is not only a tool investment needed, but also a mental shift/investment from the team. Both infrastructure and people need to be invested and traditional roles need to be re-examined.

Part of my experience as a Tester was to take on the role of being a “build master” and getting to understand the CI/CD pipeline and what was possible. This is a role where a Technical Tester or one with some experience with coding and Automation can learn a great deal and become an active member of a DevOps process. It’s also important to realize that many areas of the development process––both before and after deployment of code––can benefit from Software Testing skills, as well as software development skills. There is a significant opportunity for learning and getting involved in a variety of activities along the way.

Also, don’t think that Test Automation is the only thing needed; there is a lot of infrastructure building, code deployment, and after-the-fact examination that would be greatly enhanced by having a person well-versed in Software Testing approaches to participate and be active in a DevOps organization. Exploratory Testing skills are just as important as Automation skills and something that should be encouraged throughout the entire team. It’s not enough for just the dedicated Testers to consider these approaches but everyone who is involved in a DevOps Team should be well-versed in testing approaches and able to test anywhere along the way.

Having a strategy is important but just as important is making sure that the team as a whole can deliver on the promise. To encourage this, it is important to ensure that there is communication among the team and specifically to help develop a broad set of skills among all of the participants. While it is not required that everyone on the team be a Full Stack Developer with deep knowledge of every sub-system, it is important that the ability to adapt and be dynamic with the members of the team is possible. When expertise is left to a few or one individual, knowledge silos develop and breaking them down can become its own challenge.

Encouraging that the team as a whole cross-train and learn from each other regularly is vital. Having a central repository of knowledge that is easily accessible, easily understood, and readily shared and curated will help make the whole DevOps and Continuous Testing process run more smoothly.

Creating and Following the Road Map

Implementing a Continuous Testing strategy will require a significant upfront investment or a possible large-scale disruption of current practices to be implemented. However, the key detail to a robust and active DevOps process is to consistently be learning and adapting. This time and attention is not a one-and-done deal. It is something that is––pardon the pun––continuously tested and examined. As new information is learned, the process is tweaked and made more efficient, or in some cases retooled to address new realities.

It is important to realize that many people will have to be willing to get a little uncomfortable. The traditional approach to software development, with key roles defined by certain individuals, is less important than having a well-working team of individuals with many skills that can be leveraged along the way. There is a lot of testing being done but make no mistake, there is no requirement that a dedicated Tester has to be the one to be doing that testing. In a well-run DevOps Team with Continuous Testing implemented, the testing is always happening and everyone is doing it.


The goal of transitioning a team to Continuous Testing in a DevOps-driven process can feel overwhelming but the important part is that our organizations don’t go from A to Z overnight. There are many steps and it takes a total team commitment with lots of reinforcement along the way. Too often, we look at the dream state and think “We can’t get there from here.” Sure, maybe not today, but we certainly can get there if we plot a course and realistically look at what we can do at each milestone. The team I am on now is currently on this journey and to say that we are “done” would be foolish. We are making progress on a regular basis and while it may take many months or even years to get to where we hope to be, that journey and our continued efforts get us closer each day. Once we get to our “destination,” we will likely realize that there is a new and better place we need to be, and the process of continual learning and improvement will continue.

Michael Larsen
Michael Larsen is a Senior Automation Engineer with LTG/PeopleFluent. Over the past three decades, he has been involved in software testing for a range of products and industries, including network routers & switches, virtual machines, capacitance touch devices, video games, and client/server, distributed database & web applications.

Michael is a Black Belt in the Miagi-Do School of Software Testing, helped start and facilitate the Americas chapter of Weekend Testing, is a former Chair of the Education Special Interest Group with the Association for Software Testing (AST), a lead instructor of the Black Box Software Testing courses through AST, and former Board Member and President of AST. Michael writes the TESTHEAD blog and can be found on Twitter at @mkltesthead. A list of books, articles, papers, and presentations can be seen at