Tollef Fog Heen's blog

tfheen Tue, 04 Sep 2012 - Driving Jenkins using YAML and a bit of python

We recently switched from Buildbot to Jenkins at work, for building Varnish on various platforms. Buildbot worked-ish, but was a bit fiddly to get going on some platforms such as Mac OS and Solaris. Where buildbot has a daemon on each node that is responsible for contacting the central host, Jenkins uses SSH as the transport and centrally manages retries if a host goes down or is rebooted.

All in all, we are pretty happy with Jenkins, except for one thing: The job configurations are a bunch of XML files and the way you are supposed to configure this is through a web interface. That doesn't scale particularly well when you want to build many very similar jobs. We want to build multiple branches, some which are not public and we want to build on many slaves. The latter we could partially solve with matrix builds, except that will fail the entire build if a single slave fails with an error that works on retry. As the number of slaves increases, such failures become more common.

To solve this, I hacked together a crude tool that takes a yaml file and writes the XML files. It's not anywhere near as well structured and pretty as liw's jenkinstool, but it is quite good at translating the YAML into a bunch of XML files. I don't know if it's useful for anybody else, there is no documentation and so on, but if you want to take a look, it's on github.

Feedback is most welcome, as usual. Patches even more so.

[13:03] | varnish | Driving Jenkins using YAML and a bit of python

Tollef Fog Heen <tfheen@err.no>