In this blog post, I'll talk about my recent experiments on building a continuous integration service with Jenkins that is, as much as possible, managed through Salt. We've been relying on a Jenkins platform for quite some time at Logilab (Tolosa team). The service was mostly managed by me with sporadic help from other team-mates but I've never been entirely satisfied about the way it was managed because it involved a lot of boilerplate configuration through Jenkins user interface and this does not scale very well nor does it make long term maintenance easy.

So recently, I've taken a stance and decided to go through a Salt-based configuration and management of our Jenkins CI platform. There are actually two aspects here. The first concerns the setup of Jenkins itself (this includes installation, security configuration, plugins management amongst other things). The second concerns the management of client projects (or jobs in Jenkins jargon). For this second aspect, one of the design goals was to enable easy configuration of jobs by users not necessarily familiar with Jenkins setup and to make collaborative maintenance easy. To tackle these two aspects I've essentially been using (or developing) two distinct Salt formulas which I'll detail hereafter.

Core setup: the jenkins formula

The core setup of Jenkins is based on an existing Salt formula, the jenkins-formula which I extended a bit to support map.jinja and which was further improved to support installation of plugins by Yann and Laura (see 3b524d4).

With that, deploying a Jenkins server is as simple as adding the following to your states and pillars top.sls files:

base:
  "jenkins":
    - jenkins
    - jenkins.plugins

Base pillar configuration is used to declare anything that differs from the default Jenkins settings in a jenkins section, e.g.:

jenkins:
  lookup:
    - home: /opt/jenkins

Plugins configuration is declared in plugins subsection as follows:

jenkins:
  lookup:
    plugins:
      scm-api:
        url: 'http://updates.jenkins-ci.org/download/plugins/scm-api/0.2/scm-api.hpi'
        hash: 'md5=9574c07bf6bfd02a57b451145c870f0e'
      mercurial:
        url: 'http://updates.jenkins-ci.org/download/plugins/mercurial/1.54/mercurial.hpi'
        hash: 'md5=1b46e2732be31b078001bcc548149fe5'

(Note that plugins dependency is not handled by Jenkins when installing from the command line, neither by this formula. So in the preceding example, just having an entry for the Mercurial plugin would have not been enough because this plugin depends on scm-api.)

Other aspects (such as security setup) are not handled yet (neither by the original formula, nor by our extension), but I tend to believe that this is acceptable to manage this "by hand" for now.

Jobs management : the jenkins_jobs formula

For this task, I leveraged the excellent jenkins-job-builder tool which makes it possible to configure jobs using a declarative YAML syntax. The tool takes care of installing the job and also handles any housekeeping tasks such as checking configuration validity or deleting old configurations. With this tool, my goal was to let end-users of the Jenkins service add their own project by providing a minima a YAML job description file. So for instance, a simple Job description for a CubicWeb job could be:

- scm:
    name: cubicweb
    scm:
      - hg:
         url: http://hg.logilab.org/review/cubicweb
         clean: true

- job:
    name: cubicweb
    display-name: CubicWeb
    scm:
      - cubicweb
    builders:
      - shell: "find . -name 'tmpdb*' -delete"
      - shell: "tox --hashseed noset"
    publishers:
      - email:
          recipients: cubicweb@lists.cubicweb.org

It consists of two parts:

• the scm section declares, well, SCM information, here the location of the review Mercurial repository, and,

• a job section which consists of some metadata (project name), a reference of the SCM section declared above, some builders (here simple shell builders) and a publisher part to send results by email.

Pretty simple. (Note that most test running configuration is here declared within the source repository, via tox (another story), so that the CI bot holds minimum knowledge and fetches information from the sources repository directly.)

To automate the deployment of this kind of configurations, I made a jenkins_jobs-formula which takes care of:

1. installing jenkins-job-builder,
2. deploying YAML configurations,
3. running jenkins-jobs update to push jobs into the Jenkins instance.

In addition to installing the YAML file and triggering a jenkins-jobs update run upon changes of job files, the formula allows for job to list distribution packages that it would require for building.

Wrapping things up, a pillar declaration of a Jenkins job looks like:

jenkins_jobs:
  lookup:
    jobs:
      cubicweb:
        file: <path to local cubicweb.yaml>
        pkgs:
          - mercurial
          - python-dev
          - libgecode-dev

where the file section indicates the source of the YAML file to install and pkgs lists build dependencies that are not managed by the job itself (typically non Python package in our case).

So, as an end user, all is needed to provide is the YAML file and a pillar snippet similar to the above.

Outlook

This initial setup appears to be enough to greatly reduce the burden of managing a Jenkins server and to allow individual users to contribute jobs for their project based on simple contribution to a Salt configuration.

Later on, there is a few things I'd like to extend on jenkins_jobs-formula side. Most notably the handling of distant sources for YAML configuration file (as well as maybe the packages list file). I'd also like to experiment on configuring slaves for the Jenkins server, possibly relying on Docker (taking advantage of another of my experiment...).