DevOps is a big scary thing, but Continuous Delivery is not as scary as the big issues that surround DevOps. More importantly, it is what most teams want to have to get a smooth running software development pipeline.
In fact, whether you are using containers, virtualization, cloud, on-premises cloud, or EaaS– there is a flood of tools for automating IT tasks and moving to the cloud to omit those issues. The Infrastructure as Code movement and the tools to make it happen are increasing in use and ease-of-use. This is a big benefit of the new flood of tools and services, as they are key to Continuous Delivery.
DevOps has brought many cultural and political changes, which are often ignored by many companies during Agile implementation.
Using Continuous Delivery as a manufacturing production line with modern practices, such as Lean Software Development from Lean Manufacturing, gives us easy-to-understand ideas and practices in effort to improve our efficiency. Given the current state of Development integration with Ops, the toolset is not important. Therefore, it is not bizarre to state that Continuous Delivery is the Lean practice of “quality at every step.”
On top of all of this, testing needs to be automated. So, the most important way to look at the impact of Continuous Delivery on test teams is that all tests should be automated—and precisely—because every time there is a change somewhere, some subset of automated tests needs to run in order to validate consistency.
All the information aforementioned is included in the article “Mega Trends in Testing: Continuous Delivery, Production Line, and the Deployment Pipeline.”
If you pay attention to trends in software development—from the perspective of what some sophisticated teams are doing, what articles and books are being written, to conference topics, you may have noticed the tools being developed—there have been transformations in the past decade:
- Releases get smaller and smaller.
- Tools advance building and moving code along toward deployment.
- Infrastructure-as-code with new tools and the use of cloud gets value to customers faster and more often – helps eliminate or reduce environment, installation, provisioning and deployment challenges.
All this has required process and culture change. A number of mindsets and cultures have had to shift for test teams as well.
- Shared ownership of quality–quality tasks are more spread out over the whole team.
- The product along the way must always be deploy-ready. Always release quality. Eliminated technical debt.
- Much smaller releases lower risk, increase speed, and produce more immediate feedback.
To fully understand the shift in development thinking, consider reading Jez Humble and David Farley’s book on Continuous Delivery reviewed in this issue, Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation. It is the definitive book on Continuous Delivery.
Business Need
While most individual software teams do not talk about predictability and scheduling—these are business needs for product sponsors. Software development has become fully mainstreamed. Every modern business organization has a software development team just as it also has an accounting and HR team. Those teams are predictable and deliver a known product. There has been pressure on development teams to work the same way. The idea is, with software development being mainstreamed, it needs to be more predictable, more manageable, and more responsive to the business. It is expected to deliver business value and can no be longer be a mysterious black box.
Manufacturing Production Line
Continuous Delivery combines some XP (eXtreme Programming) and Lean Manufacturing Principles:
- Small releases
- Continuous Integration
- Collective Ownership
- Deliver as fast as Possible
- Quality at every step
- See the whole
Along this line, many people have started to look at software development in terms of a manufacturing production line. While there is great debate about this topic for test teams in many organizations, like it or not, it has arrived.
The lean, XP, and manufacturing production line ideas coupled with tools being used today are pushing us toward product development as a factory production line. Tool chains can automatically flow chunks of code from environment to environment, getting more production-like as it flows, while leading to small releases tested at each step— frequently and quickly delivered for deployment.
The Infrastructure-as-code movement—demystifying infrastructure and turning it into a reproducible and consistent practice by task automation—has already arrived, and the use of cloud services is making this even faster. An important note here is that for many teams, re-engineering the product architecture to fit these paradigms is very often a large and difficult problem in itself.
The tools, infrastructures, practices, and new skills are pushing product development to be a more stable, repeatable, and consistent production line, while producing immediate feedback. The production line flows from build to test, then promoted to new environments with testing at every step. Finally, the new chunks on the production line are deployed to production. All this happens with tool-enabled infrastructure-as-code.
Many people believe software development to be an art form, one that needs unique and creative solutions for critical customers with dynamic needs on quickly changing platforms and devices. This is a place that needs DevOps/CD culture shift in how people view their product development. CI focuses on Dev and Test teams; CD focuses on Ops teams. Where CI ends, CD picks up— promoting builds/versions/deployments up closer to, and finally to the live/production environments.
Deployment Pipeline
Regardless of my personal beliefs about software development being a production line, the idea that the “product pipeline” is the image and description of product flow through the development process and tool chain, has now firmly taken hold. The deployment pipeline flow is a variation of a production line.
It became very clear in Agile that quality is everyone’s job. Likewise everyone owns a piece of quality. In CD, everyone owns deployment. Everyone shares a piece of deployment. If helping in deployment is not your job, then why are you in product development?
Many things must change about the product to get these ideas implemented: architectural changes, tool changes, learning new skills to facilitate, culture changes, rethinking the idea of release all to smooth out and ease deployment. As with all production lines, it will cause bottlenecks.
The goal on pipelines is a constant flow of high quality, finished/deployment-ready work. Keep the flow going. Keep the pipeline moving.
Get lots of, and immediate feedback to the team: Does it work? Are customers using it? What are the actual use cases?
As Jez Humble from Continuous Development says, the solution is to adopt “a more holistic, end-to-end approach to delivering software.”
Find Inefficiencies. Manage Constraints. Eliminate Bottlenecks.
Part of the production line and pipeline mentality is looking for bottlenecks and inefficiencies. This is managing by constraints. Whatever is slowing down the deployment has to be addressed and fixed immediately to keep the flow moving. As a warning for test teams: if the flow is constrained by testing and/or due a lack of test automation—the team has to fix it.
Bottlenecks in the pipeline hold back the entire team. The build, promote, virtualized, or cloud infrastructure task automation and rapid speed of much smaller pushes can quickly show that lack of test automation is the bottleneck. Do all you can to prevent the test effort from being the bottleneck! If not enough testing is automated, it will cause bottlenecks.
This does not mean that you should release poor quality work. It means that you should identify poor quality with automated suites and get feedback of poor quality to the team as soon as possible.
What It Means:
What I want to focus on for test teams, in the topic of product pipeline, is two ideas from lean manufacturing:
- Quality at every step.
- Identify the bottlenecks and fix/remove them.
With my work on testing and test teams, I want to focus on quality at every step and at what insertion points for testing in the product pipeline.
The reason why test teams, and specifically test automation, cannot become the bottleneck is that the bottleneck will be “fixed.” And you may not have any input on how it is fixed.
The reality today is that in Silicon Valley and other places, many companies have done away with QA departments in total and the testing is done by programmers at the unit level, and with the use of Microservices and containers. Testing is done with APIs at the API/services level and there is little to no testing done at the UI level.
This will impact product quality, but to a much lesser degree than it has in the past where unit testing was mysterious, if done at all. In these organizations, unit testing is a formal, repeatable, meaningful practice and a gate for any step in the deployment pipeline. For test teams today to stay meaningful and relevant, test automation levels needs to increase across the board.
Automation and Regression
For the product pipeline to keep moving with immediate feedback does not mean taking one automated regression suite and having it run again and again and again. That does not ensure quality or confidence. There are certain times, particularly with newly committed code, that unit tests are most efficient for immediate feedback. There are other times where API or service level testing is the best strategy and still other places where the tests best inserted into the pipeline are UI driven tests.
On the topic of more and repeatable formalized unit testing, see the triangle image. Most will recognize this inverting or flipping automation triangle. It isn’t new to most, but teams need to recognize that this shift has already happened, and is only going to continue to grow in adoption.
What this deployment pipeline movement means is to automate everything you can. We know that during a sprint there is an absolute need for manual testing. The need for manual testing will never go away. But, it is getting smaller.
We make sure our test automation is as effective and efficient as can be. This does not necessarily mean that we automate more, and more, and more tests. Instead, we automate the most meaningful and rewarding tests.
We automate as much as we need to give the most immediate feedback to the team: smarter automation.
That number could be in the hundreds or thousands or tens of thousands—it depends on the system. But more, and more, and more test automation does not always equal higher and higher quality. We all know this, but it bears repeating.
What we are looking at now for testing in CD is an automation strategy; what tests to run and where to insert them—at what readiness? At what interface? At what environment? Against what data? When? This is an automation strategy.
Remember as you build suites for the CD pipeline, test automation does not show a lack of bugs. Test automation shows consistency with the last iteration. It shows stability and predictability. Not finding bugs does not show that there are no bugs—it shows that the bugs that haven’t been found yet are still hidden.
Other types of tests, manual or automated, still need to happen. We still need to focus on bug finding before the code goes into the main flow.
Feedback
Getting immediate feedback is often overlooked in the rush to automate. Automated tests need to run to show code or that product is ready to move to the next step in the pipeline. Pay attention to the information you are feeding back to the team.
What does the information mean? Is it just a number of passed tests vs. failed tests? A percentage of passed tests? Does it measure coverage? How do you tell the team the product is ready to go to the next place downstream?
Also, be aware of how long it takes you to tell them this. If your current regression automation takes 5 days to fully run— this is not going to work in CD. This will be the bottleneck. Remember the goal of automation is lean and mean, not slow and bloated.
Summary
Product development has become a production line. Deployment pipeline is a flow supported by tool chains to support Ops/IT environments, provisioning, configuration management, deployment, etc. Look for inefficiencies and prevent bottleneck slowdowns.
Make sure testing is not the bottleneck. Automation must be smarter. Regression means lean and mean, not just more and more. When automation is smarter and more efficient, teams get both meaningful and immediate feedback.
Request More Information
