Announcing the NetApp and CloudBees Jenkins Plug-in

CloudBees Jenkins Enterprise is one of the most popular CI tools that are commonly used by developers. CloudBees provides a supported enterprise version of Jenkins that is lightweight enough to be used in containers, is optimized for performance, has high availability, and is scalable. The enterprise version of Jenkins from CloudBees monitors usage and shared resources across the Jenkins implementation and many more.

During the code development and deployment process, data is generated, stored, processed, and managed on NetApp storage solutions. NetApp offerings provide persistent storage for Docker containers with the NetApp Docker Volume Plug-In (nDVP). NetApp also jointly worked with CloudBees to develop a plug-in that automates and reduces the continuous build and test cycle time, instantaneously create developer workspace and at the same time improves storage space efficiency and reduces storage costs. Integrating Docker containers with storage-persistent Docker volume plug-in and Swarm provides scalability, agility, and resiliency in the CI environment.

The primary reasons for the NetApp and CloudBees joint activity are to abstract and integrate NetApp technologies and empower the CI admins and the developers in seamlessly integrating the CI workflow using RESTful APIs. The CI team and developers no longer have to depend on storage admins to configure and expose the functionalities that accelerate the development process. This integration also leads to additional benefits to the business and the application owners in the development environments.
For more information on these benefits, see my blog in the NetApp Community Technology Section here.

img1

The Jenkins plug-in primarily consists of five different workflow components in the framework:

  1. Version control. Create local SCM (NetApp) and create checkpoint (NetApp).
  2. CI environment. Create CI environment, run builds and create checkpoint (NetApp).
  3. Developer environment. Create user workspace (NetApp).
  4. Code management. Delete checkpoint (NetApp), delete user workspace (NetApp), list SVMs and aggregates, list checkpoint and workspace (NetApp), and list volumes (NetApp).
  5. Building artifacts. Create build artifact (NetApp), zip and copy build artifact (NetApp), and delete build artifact container (NetApp).

Jenkins runs all the runnable tasks as “jobs” and controls and monitors each of these jobs. Each of these phases provides a direct advantage to the CI and the developer workflow, discussed later in this section. NetApp storage is functioning as a shared infrastructure where the volumes are exported and mounted on the VMs or physical hosts or to the Docker container over Network File System v3 (NFSv3). All the NetApp technologies and integrations are transparent to the developers and CI admins.

Version Control and Code Repository

This Jenkins job allows you to create a local source code repository on NetApp storage. It creates a volume or a partition on the NetApp storage and mounts it on a Docker container. A migration or “git clone” of the source code from the GitHub location populates the local repository volume on NetApp storage. The Jenkins plug-in is designed to connect with any software control management (SCM) tool. The plug-in by default uses Gitlab as the SCM tool of choice. There are multiple reasons for creating a local repository on the NetApp storage.

Continuous Integration Environment

It is also recommended to have different development or CI code branches created from the main code base after the developers start to create new code and start working on new features. These development or CI code branches can be organized on different NetApp volumes. If the application that is developed does not have a large codebase, then having a single development branch may suffice. This allows for more control to introduce new features, identify and fix bugs quickly, and run tests independently and iteratively on different development branches in parallel.

Developer Environment

When developers want to check out code, they do not have to access the main codebase in the repository. They check the latest checkpoint or the Snapshot copy from the development branch volume on NetApp and create a workspace for themselves. These workspaces are prepackaged clones that are created instantaneously and mounted on a Docker container. The developer can now perform unit tests, CI tests, or precheckin analysis/Gerrit on the code changes in their respective workspaces before committing the final code changes to the main code repository.

Code Management

This is essentially a clean-up phase in which only the code that needs to be retained is kept, and Snapshot copies or checkpoints and unused user workspaces are purged (deleted). Not all the Snapshot copies or checkpoints created from all the changes submitted to the main code repository and the developer branch baseline volumes have to be retained. After the developers are done using the workspaces or any of those workspaces are marked offline, those can be eliminated for better manageability and control. There are templates that can be used to list SVMs, aggregates, and volume information from NetApp storage.

Build Artifacts

This phase follows right after a successful build in the CI environment. In this phase the build artifact volume is mounted or created (if it does not exist) to a Docker container. All the artifacts in the last successful build in the development or CI code branch volume in the CI environment are moved in this newly created build artifact volume.

For information on various different builder templates that are available in CloudBees Enterprise Jenkins using ONTAP APIs, and on how you can scale the CI environment with Docker containers, refer to the NetApp technical report TR-4547.

The python scripts for the NetApp Jenkins plugin are available for download from our GitHub site. Full details on how to deploy and configure the plugin are available in the documentation. If you have questions or issues, please open an issue or send us an email (opensource@netapp.com).

Try out the plugin as a proof of concept in a development environment and provide us feedback. Suggest ideas and help improve the code with recommendation to add any additional functionality.

Bikash Roy Choudhury
Bikash Roy Choudhury is a principal architect at NetApp. He is responsible for designing and architecting solutions for DevOps workflows relevant across industry verticals including high tech, financial services, gaming, social media, and web-based development organizations that address customer business requirements in these markets. He also works on validating solutions with Red Hat (RHOSP-IaaS), Apprenda (PaaS), Docker Containers, CloudBees Jenkins, and Perforce Helix using RESTful APIs and integrating them with NetApp ONTAP software in private, hybrid, and public clouds. In his current role, Bikash drives integrations with strategic DevOps partners, including Red Hat, CloudBees, Perforce, Apprenda, JFrog Artifactory, Chef, and Iron.io.

Bikash has over 16 years of data management and storage experience. Before joining NetApp, he worked for five years at key accounts as a professional services engineer. For three years he was a systems administrator working on various UNIX platforms. Bikash received an MSCIS from the University of Phoenix, San Jose, and a BSc in computer science engineering from a distinguished engineering college in India.

Leave a Reply