Tollef Fog Heen

Why I think you should publish your infrastructure

Sat, 27 Mar 2010 15:55 +0100

GNOME's current sysadmin team is entirely voluneer-based, but as they are having problems finding enough (trusted) volunteers they are looking at hiring a part-time sysadmin. From looking at the GNOME wiki, it looks like they have had a meeting about the shortage of sysadmins. Citing from the minutes

The biggest problem that we've always had with the maintaining an active sysadmin team is the need for trust. If somebody shows up and wants to help out with a GNOME coding project, then it's easy to build up trust over time. Suggest a project, have the person send patches, review the patches, if the patches are good, eventually give them direct commit access. However, for sysadmin work, we get a lot of people who want to help out, but it's very hard for someone to contribute without being given a "dangerous" level of access to the GNOME systems.

Without having looked very hard, I would guess at the GNOME infrastructure being about as open as most proprietary software projects. There's no way for me, as a third party to take a look at their infrastructure, take a look at their ticket backlog and submit patches for problems. Similarly, their nagios setup is behind a password prompt, so there's no way for me to look at what services often have performance problems, suggest new monitors or point out any servers or services that are not monitored.

I'm not saying this to pick on GNOME, and as I'll touch on below, they do seem to mostly do the right thing, and as one of the Freedesktop.org sysadmins, I know we're not any better at least not yet.

One way to make it at least somewhat easier to contribute and get involved is to use a tool like Chef or Puppet and publishing the recipes. This won't magically make everything transparent, but it'll be a big step up. Ideally, the recipes should be complete enough that you can bootstrap a working system from them and so easier reproduce the infrastructure and any problems. It seems like GNOME is using puppet, but I couldn't find the recipes.

Moving a complete infrastructure from something managed by hand to something managed using automation tools is a fairly big and involved process. However, if you're serious about getting more people involved in your sysadmin team, I think it's one of the more reasonable ways to opening up. It also means that when one of your servers is stolen, catches fire or suffers other catastrophic failure you can rebuild the service much quicker.

My last point is to open up your ticket tracker. Most tickets aren't security sensitive, so provide a way for people to mark those tickets that are sensitive as such and make the rest public. The GNOME wiki makes this a bit confusing as it talks a bit about RT, but it seems like they actually use bugzilla for sysadmin tickets and just hide security-sensitive ones.

A small explanation about the yubikey

Tue, 16 Mar 2010 08:41 +0100

Russell Coker recently reviewed the Yubikey. The article mentions me, so I figured I'd correct a minor thing and respond to one of the comments.

First, the yubikey-server-c is my reimplementation of the Yubikey authentication protocol. Yubico provides two implementations, one in PHP and one in Java, neither which I'm particuarly interesting on building my system security on. Any bugs, misfeatures, etc in the C implementation are mine and mine alone.

Barak A. Pearlmutter, one of the commenters on Russell's blog writes:

i don’t understand. isn’t this thing vulnerable to eavesdropping and replaying? even if it has a counter which changes etc, the things it is talking to (web sites) can’t know that some generated string is being reused. and it doesn’t even have a clock, so these things can be old.

The way the Yubikey works is you have a central authentication server. This has a secret shared with the key. Setting this secret is the primary function of the personalisation tool. When you press the button, the key takes its internal state (various counters, uid field, etc) and encrypts this using AES-128. This is then sent to the application you are trying to access, be it Wordpress, SSH or something else. Said application then contacts the authentication server which decrypts the ticket, checks the values of the counters to make sure it's not a replay and responds with OK, bad ticket, replay and various other status codes. Based on this, the application grants or denies access.

There are really two places you could attack this: in the communication between the web browser and application or between application and authentication server. Both of those can be secured using SSL.

There is no way to use a single yubikey in multiple authentication realms without extra software. To do this, you would have a OpenID provider that uses the Yubikey for authentication, or you could have a Kerberos server with cross-realm trust.

As for the PAM modules and other tools so far not being packaged, yes, I know, I might fix it, but the current setup has the bits I use, as I use RADIUS authentication to get services to support both Yubikey and passwords.

Upgrading freedesktop.org hosts

Tue, 16 Feb 2010 08:43 +0100

I recently upgraded kemper.freedesktop.org to lenny. Collabora are nice enough to sponsor some of my sysadmin work for freedesktop and so making sure we are actually running a supported distribution was a good start. The actual dist-upgrade went fine, but when I rebooted with a 2.6.26 kernel, it just hung in the early boot phase. Luckily, a newer kernel worked fine. However, a newer kernel also breaks the NFS kernel server in Lenny. A short backport later, NFS was working fine, except annarchy (which NFS mounts from kemper) didn't have nfs-common installed at all, meaning it lacked mount.nfs. Ooops.

Now, bugs was broken. It used an SSH tunnel from annarchy to kemper, but the startup script was nowhere to be found. I replaced it with a trivial stunnel setup which has the added advantage of reconnecting if the tunnel goes down.

The ssh config had to be fixed slightly. We used to use an old and patched sshd that stored all the keys in a single file. I added a tiny script to split that again. We also had MkHomeDir in sshd's config, now replaced with pam_mkhomedir.

Another interesting thing I learnt is that the iLO ssh daemon chucks you out if you try to send enviromental options to it. Like, LANG which is sent by default. Slightly confusing, but easy enough to fix once I knew what the problem was.

In addition to kemper, I upgraded, but did not reboot fruit (the admin and LDAP host), due to not having the iLO password. I did not want to risk sitting there with a non-booting machine I could not fix. It's going to be rebooted at some later stage. I also did not have the iLO password for gabe, which runs mail and some other faff, so I'll have to schedule some more downtime in the near future.

How free is the N900?

Mon, 25 Jan 2010 18:03 +0100

Lucas asks about how free the N900 is, whether he can download and recompile and reflash. I'll try to answer some of those questions.

No, you can't download all the source. Part of it is just not open. I am not privy to Nokia's decisions on why or why not to open up, but it seems like the user interface bits are only partially open. Hildon itself is open so you can poke at widgets and see how those work. The address book is not open. The telepathy component that talks to the cellular modem is not open.

As for having to accept EULAs, I honestly don't remember accepting one of those, but I'm not going to say there are none. There's at least one which is every time you install a package where you have to check a box saying "Yes, I know this package is third party and will not sue Nokia if it causes my house to burn down, my wife to divorce me or causes somebody to steal the car". It's annoying, but I'm willing to live with it.

The contents of apt's sources.list is:

deb https://downloads.maemo.nokia.com/fremantle/ssu/apps/ ./ 
deb https://downloads.maemo.nokia.com/fremantle/ssu/mr0 ./ 
deb https://downloads.maemo.nokia.com/fremantle/ovi/ ./ 
deb http://repository.maemo.org/extras/ fremantle free non-free
deb http://repository.maemo.org/extras-devel/ fremantle free non-free

(technically, it comes from /etc/apt/sources.list.d/hildon-application-manager.list, not sources.list.)

I believe the built-in applications are generally not free, so rebuilding everything that is free will for instance leave you without any address book UI, the built-in map application or camera. Sadly, the X driver is also proprietary, so you won't be able to see anything either.

I don't think you can usefully install another free distro on the N900. You might be able to, at some point, assuming somebody goes to the effort.

The last question is "- Besides the non-free telephony stack, are there any other “antifeatures” I should be aware of?". The telephony stack is implemented around Telepathy, which is LGPL-ed free software. While it's correct that telepathy-ring (which talks to the cellular modem), the call UI and most of the address book are proprietary, the rest of Telepathy is free. There are SIP and XMPP connection managers that are free, and you can install more connection managers for MSN, IRC and so on.

Also, I think it's important to emphasise that the telephony stack does not contain any antifeatures. The closest thing you would be able to find is probably the restriction to one active and one held call at the same time, but as one of the developers said: "That's to prevent the UI from going mad".

While I like to tout the N900 as a free phone, it is in no way completely free. Large parts of it are free, and almost as importantly: most of the programming interfaces are free and at least somewhat documented, so if somebody wants to replace the built-in camera application with a free one, they can replace the DBus interface that the camera app provides. Ditto for maps applications, the address book and so on.

Moving SMS-es and contacts from iphone to N900

Sun, 17 Jan 2010 09:55 +0100

I've been using an iphone since late 2007 as my primary phone and so I've gotten quite a few contacts and SMS conversations stored on it. Now that Collabora has given me a nice and shiny N900, I wanted to move my contacts and conversations over, but this proved to be a bit more work than expected. Please note that the following procedure worked for me, I have tried to take reasonable steps to prevent anything breaking, but if something breaks, you get to keep both pieces. I am not responsible and this comes with absolutely no warranty. Take backups.

What you need

the addressbook and SMS SQLite databases. On my phone, they live in /var/mobile/Library/AddressBook and /var/mobile/Library/SMS.
A copy of my iphone-contacts-convert script. It's written in Perl and should be reasonably easy to understand. Put it in the same directory as AddressBook.sqlitedb.
A copy of my iphone-export-sms script. It's also written in Perl and should also be reasonably easy to understand. Put it in the same directory as sms.db.
The smstools program you can get from this thread on talk.maemo.org.

The address book conversion script takes the SQLite database structure and converts that into a VCF file. It should be completely safe to run multiple times (it only does SELECT from the different tables in the contacts database, and you have made backups, haven't you?).

If it dies with an "Unknown property", "Unknown label" or other error, you can poke it and see if you can work out what's wrong or drop me an email and I'll see if I can help you. Assuming it doesn't fall over, it will spit out a series of VCards, which you should store in a file, which you then to the N900 and open in the address book. Assuming you have less than 1000 contacts, they should now all be in your address book. If you have more, you need to split the file.

A couple of known limitations:

It doesn't handle some of the attributes, like job title, notes, department, display names, prefix and suffix. None of my contacts used those, so I just didn't care. Patches to change this accepted. Also, it doesn't handle custom attributes and birthdays. I intended to handle birthdays, but forgot and I have few enough contacts with birthdays that I just did it by hand.
When it hits something it doesn't know how to handle, it stops and you need to add the relevant handle to the code. I think it is mostly clear, how to, but again, feel free to contact me with any problems.
Only tested on firmware version 2.2. Yes, ancient, but it's what my iphone is running.
If you have contacts that are organisations, they will come up with a blank full name. Just edit them on the N900 (pressing edit and then save immediately works fine) and they'll be automatically fixed.
No picture support. This looked a bit involved, so I didn't do this bit. Should be possible with a bit of effort.

The procedure for exporting and importing SMS-es is a bit more involved. First, export the sms-es by running the perl script. It spits out a tab-separated file which you should copy to the N900 along with the smsimporter program from the smstools thread. Run ./smsimporter foo.csv and you should get all your SMS-es put into the conversation app. I ended up compiling my own smsimporter based on the 0.2.1 from the thread with the UUID patch too. Read the whole thread and it should be fairly clear.

N900 – first impressions

Tue, 15 Dec 2009 18:53 +0100

Collabora was kind enough to buy N900s for all its employees. Yay! I got mine on Friday and has been playing around with it quite a bit. It's very shiny and the user experience is a lot better than the N810. There are a few graphical glitches, it seems it's XDamage damaging a bit of a window and it's just not quick enough to repaint. Not a problem, and it has far fewer instances of just hanging for half a second which my iPhone has. That is, it hasn't had any of those yet.

The screen is good, but resistive. Takes a short while to get used to when you're used to capacative, but it's not a problem at all. The keyboard is good, but I need to map something as the compose key. Having US/UK key caps and using the Norwegian layout is a bit confusing. Not really the fault of the device though.

The web browser is generally quite good. The gestures take a bit of time to get used to, but they're not hard as such. Some of the default "applications" are implemented as just links to the web pages of services like Twitter, which is a bit silly as you don't even get a version that's optimised for the N900. They're not useless, but they are absolutely nowhere near a real application. Also, the "Store" (Ovi Store) application/web page says "coming soon", which is quite odd.

I'm not sure if I can change the selection of applications on the default application list, but modifying the desktop is easy. There seems to be few themes and background images available so far, at least in anything resembling official repositories. Hopefully this will improve over time.

So far, I haven't actually written any code for the N900. I have some applications I want to write, mostly widget-style apps like "when does the next bus home leave from a bus stop close to me and where is the bus stop", but also some other ones.

Battery life is not great. It almost did 48 hours today with a bit of use underway, and I did charge it before it ran completely out, but when I'm used to closer to a week, it's not that good. Camera seems good and is quite fast, I think it took less than five seconds from opening the camera shutter until I had taken a picture. Shutter delay is quite bad at about a third or half a second, but this is a mobile phone (or mobile computer, as Nokia likes to call it) and not a DSLR, so I'm quite happy with it.

As a phone, it seems fine so far. I can make calls and accept calls and there's no noticeable problems with it. It also functions as a modem/DUN over bluetooth, which is quite useful.

Build quality seems good, there's a good feeling when sliding the keyboard in and out, but only time will tell how good it actually is.

So far, I'm happy with it, it's a big step up from my previous UK phone (which is a Nokia E70; my iPhone is a 2G phone so I can't use it here with the provider I'm using). Hopefully I'll post more happy stories about it in the days to come.

ekey happiness

Thu, 03 Dec 2009 13:38 +0100

In my last post about the ekey, I complained about two things: memory leak in the server and missing reconnects if the client was disconnected for any reason. I've meaning to blog about the follow up for while, but haven't had the time before now.

Quite quickly after my blog post, Simtec engineers got in touch on IRC and we worked together to find out what the memory leak problem was. They also put in the reconnect support I asked for. All this in less than a week, for a device which only cost £36.

To make things even better, they picked up some other small bug fixes/requests from me, such as making ekeyd-egd-linux just Suggest ekeyd and the latest release (1.1.1) seems to have fixed some more problems.

All in all, I'm very happy about it. To make things even better, Ian Molton (of Collabora) has been busy fixing up virtio_rng in the kernel and adding EGD support (including reconnection support) to qemu and thereby KVM. Hopefully all this hits the next stable releases and I can retire my egd-over-stunnel hack.

Package workflow

Thu, 05 Nov 2009 08:31 +0100

As 3.0 format packages are now allowed into the archive, I am thinking about what I would like the workflow to look like and hoping one of them fits me.

For new upstream releases, I am imaginging something like:

New upstream version is released.
git fetch + merge into upstream branch.
Import tarballs, preferably in their original format (bz2/gzip), using pristine-tar.
Merge upstream to debian branch. Do necessary fixups and adjustments. At this point, the upstream..debian branch delta is what I want to apply to the upstream release. The reason I need to apply this delta is so I get all generated files into the package that's built and uploaded.
The source package has two functions at this point: Be a starting point for further hacking; and be the source that buildds use to build the binary Debian packages.

For the former, I need the git repository itself. It is increasingly my preferred form of modification and so I consider it part of the source.

For the latter, it might be easiest just to ship the orig.tar.{gz,bz2} and the upstream..debian delta. This does require the upstream..debian delta not to change any generated files, which I think is a fair requirement.

I'm not actually sure which source format can give me this. I think maybe the 3.0 (git) format can, but I haven't played around with it enough to see. I also don't know if any tools actually support this workflow.

Distributing entropy

Mon, 02 Nov 2009 21:23 +0100

Back at the Debian barbeque party at the end of August, I got myself an EntropyKey from the kind folks at Simtec. It has been working so well that I haven't really had a big need to blog about it. Plug it in and watch /proc/sys/kernel/random/entropy_avail never empty.

However, Collabora, where I am a sysadmin also got one. We are using a few virtual machines rather than physical machines as we want the security domains, but don't have any extreme performance needs. Like most VMs they have been starved from entropy. One problem presents itself: how do we get the entropy from the host system where the key is plugged in to the virtual machines?

Kindly enough the ekeyd package also includes ekeyd-egd-linux which speaks EGD, the TCP protocol the Entropy Gathering Daemon defined a long time ago. ekeyd itself can also output in the same protocol, so this should be easy enough, or so you would think.

Our VMs are all bridged together on the same network that is also exposed to the internet and the EGD protocol doesn't support any kind of encryption, so in order to be safe rather than sorry, I decided to encrypt the entropy. Some people think I'm mad for encrypting what is essentially random bits, but that's me for you.

So, I ended up setting up stunnel, telling ekeyd on the host to listen to localhost on a given port, and stunnel to forward connections to that port. On each VM, I set up stunnel to forward connections from a given port on localhost to the port physical machine where stunnel is listening. ekeyd-linux-egd is then told to connect to the port on localhost where stunnel is listening. After a bit of certificate fiddling and such, I can do:

# pv -rb < /dev/random > /dev/null  
17.5kB [4.39kB/s]

which is way, way better than what you will get without a hardware RNG. The hardware itself seems to be delivering about 32kbit/s of entropy.

My only gripes at this point is that the EGD implementation could use a little bit more work. It seems to leak memory in the EGD server implementation. Also, it would be very useful if the client would reconnect if it was disconnected for any reason. Even with those missing bits, I'm happy about the key so far.

Airport WLAN woes

Thu, 02 Jul 2009 21:52 +0100

Dear whoever runs the Telefonica APs in both Rio de Janeiro and Sao Paulo airports: Your DNS servers are returning SERVFAIL and has been doing so for quite a while. This is not helpful, perhaps you should set up some monitoring of them?