Accelerate development at scale with low-touch storage and Jenkins Pipeline2.0 for Continuous Integration

Imagine an integration that dramatically accelerates development at scale in the CI workflow where faster code/build cycle times allow for increased build frequency. Beyond the obvious speed and productivity gains, accelerated iterative testing and identifying bugs at a very early stage of the development cycle leads to improved code quality.  And I haven’t even gotten to the efficiency part yet.

We call this the NetApp-Jenkins Framework2.0.  It is an open source tool that integrates a Source Control Management (SCM) tool, in this case GitLab Community Edition (CE); a CI server, Jenkins Pipeline 2.0; and a binary artifact manager, we used JFrog Artifactory.  All this is done on ONTAP, as the data management platform.

NetApp Service Level Manager (NSLM) makes RESTful APIs calls to ONTAP throughout the CI process. The framework uses native ONTAP features like:

  • FlexVol for the source code, CI nodes, artifacts and binaries
  • Snapshots for quick recovery from build failures and bad source code
  • FlexClones for instant and thin provisioning of developer workspaces allowing solutions to scale to development teams of dozens, or hundreds effortlessly

All the persistent data volumes for GitLab CE, CI nodes and Artifactory are created by this integrated solution using NSLM APIs in ONTAP to provide the scalability and load-balancing of workloads.  The NetApp Docker Volume Plugin mounts these volumes to Docker containers, otherwise referred to as nodes. The data volumes are created with the project_name_ as the prefix to identify Jenkins jobs from different projects. This helps to provide zero storage touch to admins and developers during the CI process.

Typically, developers will fetch their source code from GitLab and artifacts from JFrog Artifactory in order to prepare their environment for writing or modifying their code. This must happen in an ongoing fashion and may require high infrastructure impacts depending on the number of developers working in the environment. This impacts the time it takes to build the code and development environment before they can start coding. This negatively impacts developer productivity and is compounded in large scale development environments.

The NetApp-Jenkins Framework provides developers a more efficient way to write code without any lead time. The framework creates a CI node (that mounts a persistent data volume from ONTAP) that copies the source code from GitLab, gets the corresponding artifacts and performs a full build for the first time. After the CI build completes successfully, a snapshot is created automatically on ONTAP. Every ONTAP snapshot is tagged to a corresponding build number before completing successfully.

When developers request a workspace, a FlexClone is created on ONTAP by NSLM APIs that have the exact replica of the CI volume contents. The developers now have access to source code, artifacts, and binaries in their workspace that enable them to start coding right away. These clone copies can now instantly scale for 100s or 1000s of developers with very little performance impact on the GitLab node. The clones are thin provisioned with very little overhead. The following code from the framework highlights one such clone operation using NSLM to create a developer workspace.

The following code shows a clone created from the parent CI node volume mounted as a service on a Docker container using netappdvp that is resilient to failure. Once developers complete their work, they can blow away the workspaces with no risk to the changes committed to the main code base in GitLab.

The NetApp-Jenkins Framework 2.0 provides a zero touch storage for developers with the ability to instantly scale on demand. DevOps admins who are responsible for setting up a CI environment need to have basic information about the data platform which can be configured in Jenkins “environment” settings.  As every business environment is different, the open source code for the framework is written in python and can be customized based on the development requirements.  For more information, refer to our technical report (TR-4547).

Bikash Roy Choudhury
Bikash Roy Choudhury is a principal architect at NetApp. He is responsible for designing and architecting solutions for DevOps workflows relevant across industry verticals including high tech, financial services, gaming, social media, and web-based development organizations that address customer business requirements in these markets. He also works on validating solutions with Red Hat (RHOSP-IaaS), Apprenda (PaaS), Docker Containers, CloudBees Jenkins, and Perforce Helix using RESTful APIs and integrating them with NetApp ONTAP software in private, hybrid, and public clouds. In his current role, Bikash drives integrations with strategic DevOps partners, including Red Hat, CloudBees, Perforce, Apprenda, JFrog Artifactory, Chef, and Iron.io.

Bikash has over 16 years of data management and storage experience. Before joining NetApp, he worked for five years at key accounts as a professional services engineer. For three years he was a systems administrator working on various UNIX platforms. Bikash received an MSCIS from the University of Phoenix, San Jose, and a BSc in computer science engineering from a distinguished engineering college in India.

Leave a Reply