Jenkins is a popular build and scheduling tool used to create software for many organizations. As such, it is common to see it used in a variety of situations and configurations. While it is common to think that organizations all use the same build server and/or Jenkins environment, many organizations use a variety of Jenkins systems, some of which are configured to run in different environments. As time goes on, there is often a desire to simplify or merge systems. If that’s not an option, then the ability to get Jenkins to communicate with or transition pipelines to different environments becomes something to consider.
I’ve been involved with this process in a few ways, from setting up a monolithic Jenkins environment on a single server and managing everything in one place to spreading out resources into numerous systems. This article will discuss some of the pros and cons of making that transition, what goes into setting up a Jenkins Pipeline, and how some of the changes as that system grows or transitions will help you determine which approach makes the most sense for you.
Starting Small: Your Own Little World
Most individuals or small organizations start out with a simple Jenkins implementation. Often it is a single machine and the conducting Jenkins server is where everything is run. At its heart, Jenkins is more than a fancy scheduling system that will run scripts or programs at designated times. For those first experimenting with Jenkins, this process may feel familiar. We first set up a Jenkins server on a platform. Often, we start with a virtual machine or a dedicated server in our environment. The benefit is that it gives us complete control. We can create the processes we need to build software, run tests, and––pending those test results––create a final installer, repo, or other means to deploy an application. We can then run a deploy script to push the software to our destination machine and be up and running.
Laying a Pipeline
A pipeline is a set of commands or scripts that we can put together to run multiple sequential or parallel processes. If you think about the steps necessary to build a software application, there is first compiling code, which runs through unit tests and other steps necessary to determine if the code changes are appropriate. Then we want to take those compiled codes and build a system with the application up and running. In the smallest of Jenkins environments, that could be run on the same server. Once the application is installed and ready to run, subsequent test jobs can be run. This can help determine if the system as configured and installed is fit for use. Finally, the green light shows that the software is running and ready for everybody to use. Again, it is possible to create this pipeline on a single server, and when starting out, that may be desirable. But, over time and as the application grows in complexity, this model will be quickly outgrown.
Need a Little More Elbow Room
As we build out our application, we realize that the serial nature of a single machine or a single thread for our pipeline will impact us. It will take longer and longer to run our tests and to deploy our systems. Several years back, we reached a threshold where it was taking up to 24 hours to run through all tests and complete our process pipeline. This was all the more frustrating when we had errors or failed tests. Fixing the failing tests or resolving the issues of course takes time; but, when we consider that each cycle of build and test takes close to a day to complete, that’s not practical for growing organizations.
At some point, the need to run tests in parallel comes into play. With that, the need to have a larger platform footprint also comes into play. We faced this issue and determined the best approach was to use cloud resources and set up a variety of systems that we could deploy Docker containers to. We set up 5 machines: One of them was the dedicated Jenkins server (our conductor machine), and the other 4 we would dynamically bring up so that we could create numerous follower systems. Each follower system allows for the creation of multiple Docker containers. They are set up to run as dedicated systems to deploy the code to, run tests, and get a verification/acceptance of the software. Using 4 systems, we created a benchmark that allowed us to divide our total number of tests by 4, and then the number of tests per system would then be further divided up into 10 tests. Those 10 tests would run on an individual Docker container set up to run those specific tests. In the event that the tests need more systems, we could allocate another server in the cloud, that server could spin up the necessary number of Docker containers, and then allocate the tests accordingly.
By setting up this more distributed environment, we were able to piece together our pipeline. The first process in this pipeline consisted of running what we refer to as a “swarm.” This is a dynamic number of Docker containers that correlate to the number of tests we want to run. The process allows us to bring up and allocate the desired number of Docker containers running as servers, based on the number of tests we have committed to run, divide by the number of servers (usually 4), build those individual Docker containers as part of the swarm, spin up the environments, install the latest code on them, run all necessary tests, and if all tests passed, then mark the build as successful, kick off a full build of the software, upload the repos to our repo server, and then finally designate our staging server for the installation and deployment of our final build. Lastly, we deactivate and close down the swarm so it will be ready the next time we need it.
The net result of this change was a time reduction in testing, from 24 hours to 40 minutes (provided everything completed successfully).
Moving Your Pipeline
For various reasons, it may be desirable or even necessary to have Jenkins running across many systems. There are cost and operations trade-offs when it comes to hosting systems in the cloud, hosting them on-premises, or some combination of the 2. When your pipeline is hosted in the cloud, there are benefits of scale and sizing that can be used and leveraged. Additionally, proximity can be determined and systems can be located close to the team working with those systems. We had a need for this because we had a distributed team. Thus, it made sense to deploy 2 different sets of machines, one on the East Coast of the USA and one on the West Coast. In this case, the setup was similar, so the environments could likewise be set up in a similar fashion. Since they were different projects and project teams, the servers themselves were running independently of each other and each of our previously described “swarms” had its own dedicated Jenkins conductor. We benefited from being able to use the Docker swarm and Docker build projects that we had created. Those processes could be set up in our cloud environment and take advantage of similar machine layout and provisioning.
The greater challenge comes when the environments don’t match as well. In our situation, we determined that it made sense to have one set of environments in the cloud for our distributed teams (set up on EC2 instances within AWS) while having a dedicated system in-house for our other co-located teams. As our organization has acquired a variety of companies over several years, it has been a process of transition to get each of the environments to work together. At this time, we have not transitioned to a unified solution where one Jenkins server rules them all. But, it’s possible we may get there at some point in the future. That will require that our pipelines can leverage systems in multiple environments in a hybrid cloud structure.
LogiGear has experience rolling out Continuous Testing pipelines using Jenkins for its clients. Recently, we had to transition a pipeline from various cloud environments: From Amazon Web Services back to VMware and then back to Amazon Web Services to move it to a VMware Horizon.
There were hiccups in the client’s Microsoft Team Foundation Server’s test suites. The client wanted to transition to using Microsoft Azure DevOps (ADO). That way, they could quickly spin up and provision VMs and assign different tests to different VM templates.
Some segments of the pipeline will work the same regardless. That’s one benefit of Docker containers: They will run in a similar way regardless of the base systems they are configured to run on. It’s the setup of the swarm servers that proves to be the bigger challenge. With different rules and processes for setting up machines in different cloud environments, there is no exact plug-and-play option we can use at this time to substitute that first step. However, once we do make that first setup, if the underlying machines are the same (code revisions, operating systems, similar allocation of system resources) then we can run the other parts of the pipeline the same way, regardless of the system they are configured for.
Conclusion
Jenkins can help take a turbulent build, test, and deploy process and give you the tools to contain it. Cloud infrastructure can give you the flexibility to build out as many systems as you need when you need them. Local hardware can give you local control and potential cost savings over cloud infrastructure. In certain cases, it may make sense to leverage one side for smaller projects and expand out only when you need to do something at a greater scale. Regardless, setting up various pipelines and leveraging the ability to connect processes can indeed help make both building and testing your code more timely and easier to manage.
Need help implementing CT pipelines using Jenkins? Let us know and we’ll be happy to help!
Michael is a Black Belt in the Miagi-Do School of Software Testing, helped start and facilitate the Americas chapter of Weekend Testing, is a former Chair of the Education Special Interest Group with the Association for Software Testing (AST), a lead instructor of the Black Box Software Testing courses through AST, and former Board Member and President of AST. Michael writes the TESTHEAD blog and can be found on Twitter at @mkltesthead. A list of books, articles, papers, and presentations can be seen at http://www.linkedin.com/in/mkltesthead.