Upgrading freedesktop.org hosts
2 minutes read

I recently upgraded kemper.freedesktop.org to lenny. Collabora are nice enough to sponsor some of my sysadmin work for freedesktop and so making sure we are actually running a supported distribution was a good start. The actual dist-upgrade went fine, but when I rebooted with a 2.6.26 kernel, it just hung in the early boot phase. Luckily, a newer kernel worked fine. However, a newer kernel also breaks the NFS kernel server in Lenny. A short backport later, NFS was working fine, except annarchy (which NFS mounts from kemper) didn’t have nfs-common installed at all, meaning it lacked mount.nfs. Ooops.

Now, bugs was broken. It used an SSH tunnel from annarchy to kemper, but the startup script was nowhere to be found. I replaced it with a trivial stunnel setup which has the added advantage of reconnecting if the tunnel goes down.

The ssh config had to be fixed slightly. We used to use an old and patched sshd that stored all the keys in a single file. I added a tiny script to split that again. We also had MkHomeDir in sshd’s config, now replaced with pam_mkhomedir.

Another interesting thing I learnt is that the iLO ssh daemon chucks you out if you try to send enviromental options to it. Like, LANG which is sent by default. Slightly confusing, but easy enough to fix once I knew what the problem was.

In addition to kemper, I upgraded, but did not reboot fruit (the admin and LDAP host), due to not having the iLO password. I did not want to risk sitting there with a non-booting machine I could not fix. It’s going to be rebooted at some later stage. I also did not have the iLO password for gabe, which runs mail and some other faff, so I’ll have to schedule some more downtime in the near future.

Back to posts