Leading an Organization into Continuous Delivery can be a big task. Nor can businesses just “turn a switch” and start doing Continuous Delivery (CD) overnight. This guide attempts to cover the main points that organizations need to consider when starting to implement and adopt Continuous Delivery:
- Continuous Delivery Pipeline
- Continuous Integration and Continuous Delivery
- Continuous Delivery Tools
- Continuous Delivery Best Practices
- Continuous Delivery Testing
- Continuous Delivery Test Strategy
- How to Get Started with Continuous Delivery
The fundamental underpinnings of CD are dependent on two parts: culture change and commitments to the tool chain. A larger part involves commitment to the tool chain. This is because Continuous Delivery requires an increase in Automation-both Task Automation and Test Automation. Each section contains everything businesses need to consider when it comes to building and implementing a successful continuous testing program; from tool chain, to best practices, to developing a strategy, and discussing options on how to get started.
Continuous Delivery Pipeline
Software development has become fully mainstreamed in the last few years, largely due in part to DevOps. Every modern business organization has a software development team, just as it has an accounting team or an HR team. These teams are all predictable and deliver a known product. With DevOps, there has been an increased pressure on development teams to work this way. The idea is that with software development being mainstreamed, it needs to be more predictable, more manageable and more responsive to the business. Software Development is now expected to deliver business value and can no longer be a “mysterious black box.”
What is the Continuous Delivery Pipeline?
First, let’s establish the fact that Continuous Delivery combines some XP (eXtreme Programming) and Lean Manufacturing Principles. They include:
- Small Releases
- Continuous Integration
- Collective Ownership
- Deliver as fast as possible
- Quality at every step
- See the whole
Many people have started to look at software development in terms of a manufacturing production line. While there is great debate about this topic for test teams, in many organizations—like it or not—it has arrived. The lean, XP, and manufacturing production line ideas (coupled with tools being used today) are pushing us toward product development as a factory production line. Tool chains can automatically flow chunks of code from environment to environment, getting more production-like as it flows. This leads to small releases that are tested at each step that are frequently and quickly delivered for deployment.
The Infrastructure as code movement (the idea of demystifying infrastructure and turning it into a reproducible and consistent practice by task Automation) has already arrived and the use of cloud services is making its movement faster. An important note here is that for many teams, re-engineering the product architecture to fit these paradigms is very often a large and difficult problem in itself. The tools, infrastructures, practices, and new skills associated with Continuous Delivery are pushing product development to be more stable, and repeatable—a consistent production line that produces immediate feedback. The pipeline flows from build to test, then it gets promoted to new environments with testing at every step. Ultimately, the new chunks on the production line are deployed to production. All this happens with tool-enabled infrastructure as code.
Continuous Delivery has been related to a production pipeline. The main idea is that the “Continuous Delivery pipeline” is the image and description of product flow through the development process and tool chain. This pipeline flow is a variation of a manufacturing production line. It became very clear in Agile that quality is everyone’s job. Likewise, everyone owns a piece of quality. In CD, everyone owns deployment and everyone shares a piece of deployment. Many things must change about the product to get these ideas implemented: architectural changes, tool changes, learning new skills to facilitate, culture changes, rethinking the idea release to smooth out, and ease deployment. As with all production lines, there will be bottlenecks. The goal with pipelines is a constant flow of high quality and finished/deployment-ready work. Keep the flow going, Keep the pipeline moving. Get lots of and immediate feedback to the team. Does it work? Are customers using it? What are the actual use cases? As Jez Humble from Continuous Development says, the solution is to adopt “a more holistic, end-to-end approach to delivering software.”
Find Inefficiencies. Manage Constraints. Eliminate Bottlenecks.
Part of the production line and pipeline mentality is looking for bottlenecks and inefficiencies. This is managing by constraints. Whatever is slowing down the deployment has to be addressed and fixed immediately to keep the flow moving. As a warning for test teams, if the flow is constrained by testing and/or due to a lack of Test Automation, the team has to fix it. We will discuss this later in the Continuous Delivery Testing and Continuous Test Strategy sections.
Continuous Integration and Continuous Delivery
First, let’s establish the define Continuous Delivery and Continuous Integration and establish the differences. According to Wikipedia, Continuous Integration (CI) is defined as “A software engineering practice in which the changes made by developers to working copies of code are added to the mainline code base on a frequent basis, and [is] immediately tested.” The goal is to provide rapid feedback so that if a defect is introduced into the mainline, it can be identified quickly and corrected as soon as possible. Continuous Integration software tools are often used to automate testing and build a document trail. Because CI detects deficiencies early on in development, defects are typically smaller, less complex, and easier to resolve. In the end, well-implemented CI reduces the cost of software development and reduces the time to market.
Contrasted, Continuous Delivery is a “Software Engineering approach in which teams produce software in short cycles, ensuring that the software can be reliably released at any time. It aims at building, testing, and releasing software faster and more frequently. The approach helps reduce the cost, time, and risk of delivering changes by allowing for more incremental updates to applications in production. A straightforward and repeatable deployment process is important for Continuous Delivery” (Wikipedia).
The main difference is that CI focuses on Dev and Test teams. CD focuses on Ops teams. Where CI ends, CD picks up; promoting builds, versions, and deployments up closer to and finally, the live/production environments.
Continuous Delivery Tools
The financial commitments to the tool chain often require a large upfront investment for organizations, this can be hard due to the necessity of increasing task Automation. Tools will be needed for the following areas:
- Cloud/EaaS, Iaas, Paas
- Configuration Management (e.g. Chef and Puppet)
- Containers (e.g. Docker and Packer)
- CI/Version Control (e.g. Jenkins, Bamboo, TeamCity, and Buildbot)
- Deployment (e.g. XebiaLabs Deploy, Go Thoughtworks, IBM UrbanCode Deploy, and Code Stream)
- Monitoring (e.g. New Relic andApp Dynamics)
- Soure Control (e.g. Git SVN)
- Test Automation (e.g. Selenium, TestArchitect, Cucumber, and jBehave)
Continuous Delivery Best Practices
The three best practices of Continuous Delivery are shift-left, Lean Software Development’s principles of Quality at Every Step, and Eliminating Waste and Automate –Automate- Automate.
Shift-left
Shift-left is the most important best practice to discuss. For those unfamiliar, Shift-left is an approach to Software Testing and system testing in which testing is performed earlier in the lifecycle—this is a departure from Agile and is part of the maxim: “Test early and often”
It is important to note that the idea of shift-left has been around for a long time. It is grabbing hold much more today. In CD, organizations need to think about shifting all testing, (security, performance, functional, and etc.) as early as possible in the Dev Cycle, no longer to be done as an afterthought at the end.
The biggest impact of shift-left in CD is getting developers to initiate testing earlier with more TDD, unit testing, code review, and other developer tests. Doing this shifts some functional testing as early as possible. It is just as beneficial to shift UI tests earlier to the API level wherever possible. The idea of shift-left leads us to our next best practice…
Lean –Quality at Every Step
According to Wikipedia, Lean software development (LSD) is “a translation of lean manufacturing principles and practices to the software development domain. Adapted from the Toyota production model, it is emerging with the support of a pro-lean subculture within the Agile community. The Seven lean principles are defined as the following:
- Eliminate waste
- Amplify learning
- Decide as Late as possible
- Deliver as fast as possible
- Empower the team
- Build integrity in
- See the whole.
Steps 1, 4, 6, and 7 are the most widely adopted in CD. The principle of “Eliminate Waste” is probably the biggest factor for success in a Continuous Testing implementation. The idea of Testing in Continuous Delivery is that Automation regression suites are supposed to be “lean and mean” not bloated. The goal of Continuous Delivery and DevOps is to provide immediate feedback-aka principle 4—deliver as fast as possible. Teams will need to redesign, optimize, and even cut the size of their automated suites for Testing in Continuous Delivery. While there is no set standard on “ideal times” there is one guiding principle—if your regression suite takes days, maybe weeks to run, then it is too slow for CD. Building integrity-in is sometimes referred to as “Quality at Every Step.” This means exactly what it sounds like. You have to build quality-in at every Step. This means quality user stories, quality code, quality unit tests, quality test cases, quality bugs, quality automated scripts, quality performance tests, quality environments, quality data, and all other deliverables at every step.
Automate Everything vs. Automate Smartly
In Continuous Delivery, everything needs to be Automated or better. Everything that can be automated needs to be. This falls into two categories; both task automation and Test Automation. Task Automation largely deals with deployment and pipeline Automation, whereas Test Automation has several focuses in Continuous Delivery, most notably: performance testing, functional test automation, and security test automation. With this, comes pressure on test teams. According to the Testing in Continuous Delivery Survey when asked “Do you feel that there is more pressure to automate recently?” 60% of respondents stated that “yes, a lot more.” Contrasted, 4% of the group wrote “All of our tests are automated. There is no pressure.”
It is prudent that teams don’t fall into the “automate everything” mindset when it comes to Test Automation. Focus instead on smarter Test Automation, as opposed to more and more, and communicate this idea.
Continuous Delivery Testing / Testing in Continuous Delivery
Without a successful Automation practice in place, any team that attempts DevOps is headed for disaster. DevOps will highlight the shortcomings of a team on a larger scale, faster than Agile ever could.
The reality today is that in Silicon Valley, and other places, many companies have done away with QA departments in total. The Testing is done by programmers at the unit level and with the use of Microservices and containers. Testing is done with APIs at the API/services level and there is little to no Testing done at the UI level. This will impact product quality, but to a much lesser degree than it has in the past where unit testing was mysterious, if done at all. In these organizations, unit testing is a formal repeatable, meaningful practice, and a gate for any step in the deployment pipeline.
The figure below illustrates all the places where Test Automation appears in the Continuous Delivery Pipeline.
In Continuous Delivery, Testing appears at every stage of the pipeline, it is no longer isolated. It is important to note that test teams, including Test Automation, cannot become the bottleneck in the Continuous Delivery pipeline. The main reason is that if it does, it will be “fixed” and you may not have any input on how it is fixed.
For the product pipeline to keep moving with immediate feedback does not mean taking one Automated regression suite and having it run again, and again, and again. That doesn’t ensure quality or confidence. There are certain times, particularly with newly committed code, that unit tests are most efficient for immediate feedback. There are other times where API or service level testing are the best strategies and still other places where the tests best inserted into the pipeline are UI driven tests.
Continuous Delivery Test Strategy
What is commonly referred to as a “Continuous Delivery Test Strategy” is in actuality Continuous Testing. Continuous Testing (CT) is the process of executing Automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release. (Wikipedia) and has its own Automation strategy. It is important to remember that as you build test suites for the CD pipeline, Test Automation does not show a lack of bugs. Test Automation shows consistency with the last iteration; it shows stability and predictability. Not finding bugs does not indicate that there are no bugs—it indicates that the bugs that haven’t been found yet are still hidden.
What test to run and where to insert them—at what readiness? At what interface? In what environment? Against what data? When? These are all questions that will become the foundation of your Continuous Delivery Testing strategy. There are 3 elements that form your strategy:
- Environments
- Regression Automation Suites
- Testing in Production
The test strategy for Continuous Testing is a full reexamination of what gets tested where. This with the Lean Software development (LSD) idea of “quality at every step” is the foundation of a CD test strategy. It involves reexamining, what you have been calling, “regression” and developing multiple lean and mean suites for immediate feedback.
It also means rethinking what and how you automate. Old style Automation—even small isolated automated tests need to be reexamined and replaced by efficient, low level tests as well as longer customer-focused transaction workflows. Rethinking regression and Automation is a big task for even the most effective teams.
Environments
In Agile or Scrum, Dev or Test Teams “progressed” through environments, in Continuous Testing it’s important to consider what and why we test on different environments:
In DevOps/CD, you might be running your Automation on many or all these environments concurrently. For example, what tests need to get run on the build environment? Why? What tests get run in the test environment? Why? What Automated tests get run against, for example, VMs with various configurations, languages, browsers, mobile devices, etc.? What Automated tests get run on the staging server? What is their purpose? What gets run on the staging server? What gets run in production? These are all important considerations to make when developing your testing strategy.
Regression Automation Suites
The term regression is probably the most misunderstood in all software development. The fact is, regression means many things to different people. Rethinking regression and Automation is a big task for even the most effective teams. The goal of Continuous Delivery and DevOps is to provide immediate feedback (aka principle 4—deliver as fast as possible).
To set the record straight, a regression test is any test you run again. Some people in development think the regression suite or full regression is running every test run against a previous build or product. That is rarely the case. Their regression suite is more likely rerunning every Automated test.
For some teams, their automated “full regression” suite can take days to run. Many “full regression” suites have gotten bloated and slow, like we mentioned earlier in the best practices section, this is not ok. There is a speed to Continuous Delivery and Continuous Deployment that leaves no room for “back up plans”. The new description of regression has to be “lean and mean”, fast and furious, bug finding! Not bloated or the same old “whatever” we automated at the time. So for teams that are looking to implement CD, cleaning out their test suites to be lean and mean is a priority.
Testing in Production
It has always been a strict rule: “No testing in production”. The reason being that the last thing you wanted was to have the test team mess around on a live system or environment with customers only to find out that the Testing did something to muck up the system.
This has entirely changed in Continuous Testing. In CD, you may be asked to run a suite of tests to validate the migration to live or monitor the live system. Whatever the case, you need to make sure you are testing what needs to be tested and no more than you need to on live. Make sure the whole team knows the risks of testing in the live production environment, the risk of not testing in the live environment, as well as that tests cannot interfere with the normal user’s workflow performance or operation of the system-be transparent.
Getting started with Continuous Delivery
Continuous Delivery requires both high test coverage levels and an assurance that their practices are in good shape. It is important that businesses do not underestimate the difficulties of Automation. According to the testing essentials survey , released by LogiGear Magazine, almost half of respondents answered that the biggest phase or aspect of testing that is not understood by management is that “Test Automation is not easy”. Automation, and to a larger extent DevOps, can’t be successful if there isn’t serious time and investment from the start.
According to the Test Automation survey released by LogiGear Magazine, only 19% of respondents who did a Test Automation implementation were able to do it the successfully the first time. The Test Automation survey also revealed another shocking insight. When asked what would prevent them from Automating more, the top five results were:
- “Not enough technical skill to build more successful Automation.” (34% of respondents)
- “Investment in Automation program (training, staff, tool maintenance) is too high.” (32% of respondents)
- “Management does not understand what it takes to successfully automate.” ( 31% of respondents)
- “Tool use cost is high.” (31% of respondents)
- “Code or UI is too dynamic to automate more.” (31% of respondents)
For argument’s sake, when looking at these problems, they largely deal with either staffing or tool use. Commitment to the tool chain, as mentioned earlier is often expensive. While there are many tools on the market, there are open source solutions like Selenium, or our own TestArchitect that can be used for Continuous Testing. TestArchitect integrates with most leading tools including: Microsoft Team Foundation Server/ Visual Studio/ Test Manager, Oracle EBS, HP Quality center, Jira, Jenkins, Zephyr, Neoload and more!
TestArchitect is a closed-source tool, but is available for free. TestArchitect can also handle rapid changes to the UI—it uses the Action-Based Testing method (ABT), which focuses on automating keywords, not test cases. Since there are fewer keywords than there are test cases, keyword implementations tend to be shorter than test case implementations; the Automation effort is also more manageable. In a well-designed keyword-based test suite, only a limited number of keywords has to be maintained, after which then all tests run again against the new version of the application.
For staffing there are several options. If you have the requisite staff in-house but are lacking the technical skill for building a successful Automation program, it may make sense to consider engaging with a testing vendor.
The biggest benefits of outsourcing testing are:
- More Automation
- More effective testing
- Faster product release
- And more management oversight
Testing vendors like LogiGear can provide the necessary expertise in testing skills, as well as technical skills to help firms successfully implement Test Automation from the start. LogiGear is one of the few firms that specialize in Test Automation with the depth and breadth of services to ensure your success. LogiGear’s Test Automation solution covers the following areas:
- Test Automation strategy consulting
- Test Automation tool evaluation and selection
- Test Automation scripting and execution using leading tools such as Selenium/Appium, Quick Test UFT, TestComplete and TestArchitect
- Automated Omni-channel testing: Web, Mobile, Wearbles, Voice-App, etc.
- Test Automation for All Major technolgoies & platforms: cross-browser, Web APIs/Services, Mobile, Desktop, (.Net, Java, etc.) ERP (SAP, Oracle EBS, Microsoft Dynamics, etc.)
To learn more about how our Test Automation experts can help you transform your process for CD, visit logigear.com today!
Request More Information
