CI and CD at Scale: Scaling Jenkins with Docker and Apache Mesos

As Continuous Integration and Continuous Delivery usage grows, the underlying infrastructure needs to grow too, scaling horizontally across multiple hosts and dynamically provisioned as needed.

Docker is revolutionizing the way people think about applications and deployments. Apache Mesos enables fault-tolerant distributed systems and allows managing a cluster of Linux servers as a single system, running Docker containers across multiple hosts, offering elasticity and machine abstractions.

By using Apache Mesos and Marathon, an Apache Mesos framework for long-running applications, the Jenkins Continuous Integration environment can be run at scale, dynamically.

Jenkins masters can be run on Docker containers distributed on Marathon, while the Jenkins Mesos and Docker plugins allow slaves and jobs to run in containers distributed across the multiple hosts, and isolating job execution.