Why I think you should publish your infrastructure
2010-03-27
3 minutes read

GNOME’s current sysadmin team is entirely volunteer-based, but as they are having problems finding enough (trusted) volunteers they are looking at hiring a part-time sysadmin. From looking at the GNOME wiki, it looks like they have had a meeting about the shortage of sysadmins. Citing from the minutes

The biggest problem that we’ve always had with the maintaining an active sysadmin team is the need for trust. If somebody shows up and wants to help out with a GNOME coding project, then it’s easy to build up trust over time. Suggest a project, have the person send patches, review the patches, if the patches are good, eventually give them direct commit access. However, for sysadmin work, we get a lot of people who want to help out, but it’s very hard for someone to contribute without being given a “dangerous” level of access to the GNOME systems.

Without having looked very hard, I would guess at the GNOME infrastructure being about as open as most proprietary software projects. There’s no way for me, as a third party to take a look at their infrastructure, take a look at their ticket backlog and submit patches for problems. Similarly, their nagios setup is behind a password prompt, so there’s no way for me to look at what services often have performance problems, suggest new monitors or point out any servers or services that are not monitored.

I’m not saying this to pick on GNOME, and as I’ll touch on below, they do seem to mostly do the right thing, and as one of the Freedesktop.org sysadmins, I know we’re not any better at least not yet.

One way to make it at least somewhat easier to contribute and get involved is to use a tool like Chef or Puppet and publishing the recipes. This won’t magically make everything transparent, but it’ll be a big step up. Ideally, the recipes should be complete enough that you can bootstrap a working system from them and so easier reproduce the infrastructure and any problems. It seems like GNOME is using puppet, but I couldn’t find the recipes.

Moving a complete infrastructure from something managed by hand to something managed using automation tools is a fairly big and involved process. However, if you’re serious about getting more people involved in your sysadmin team, I think it’s one of the more reasonable ways to opening up. It also means that when one of your servers is stolen, catches fire or suffers other catastrophic failure you can rebuild the service much quicker.

My last point is to open up your ticket tracker. Most tickets aren’t security sensitive, so provide a way for people to mark those tickets that are sensitive as such and make the rest public. The GNOME wiki makes this a bit confusing as it talks a bit about RT, but it seems like they actually use bugzilla for sysadmin tickets and just hide security-sensitive ones.

Back to posts