Scalable software distribution systems

Tollef Fog Heen (tfheen@idi.ntnu.no)

\underline{2} Scalable software distribution systems

Chapter 1

Preamble

I’d like to thank my counselor Anders Christensen for a lot of help and guidance with the project. Without him, I would have gotten lost. I also wish to thank my girlfriend, Karianne, without whose support I wouldn’t have finished on time.

Trondheim, November 2004 Tollef Fog Heen

Contents

5

Dissertation text

The text should describe scalability problems in the current software distribution systems for unix -like operation system. The goal is to suggest ways they can be improved.

Introduction

Computers are becoming more and more complex, more and more interconnected and have an ever-increasing amount of software installed and configured. In this paper, I will research one way to handle this problem.

This text requires a good knowledge of unix -based operating systems and concepts. The text has been kept short while still being informative.

2.1 Delimitations

Making a generalized solution for all systems under the sun is, if possible, extremely hard. Also, rather than making a “one-size-fits-none” kind of solution, I will concentrate on a solution for UNIX and UNIX-like operating systems (like Linux). Those are similar enough that that they will be treated as variants rather than different operating systems. Other paradigms like Microsoft Windows and VMS will need other kinds of solutions, which will not be researched here.

Problem definition

Most organizations, be it your family or your workplace, start small. They have few people who then don’t require much in the way of explicit communication and procedures. Today, most organizations use computers extensively, and those need maintainence. As long as you have a small number of hosts, maintaining each as a single entity, troubleshooting, debugging, installing and removing software on each machine by itself, is not too much work.

As the organization grows, so will the number of computers, and at some point, maintaining all the diverging hosts will become a nightmare, the users will be unhappy (since the hosts are broken in various ways: printing works on one, the word processor on another, and so on). Or, huge amounts of resources are spent on maintaining the hosts.

3.1 Packaging systems

What is a package system? It is a tool which helps the user^{1} to:

1. Keep system consistent across system boundaries such as hosts or architectures.

2. Install and upgrade software.

3. Distribute software and configurations in a consistent, predictable manner.

4. Keep track of installed software.

5. Make sure the installed software actually works.

1The user might not always be the end-user, it can also be the local system administrator or a software distributor.

All of this is about handling complexity and scalability, and this is the hard problem at hand. Distributing a piece of software (as a tarball), and unpacking it somewhere is easy. However, making it interact correctly with other software and being able to remove or upgrade it later, without fear of breaking other software makes the problem a lot harder.

Some of the dimensions in which we want scalability are:

1. Autonomous sites

2. Release levels

3. Regional differences (time zones, default language)

4. Package choices (conflicting packages, dependencies)

Philosophy

When making something, one always faces choices on how to do it, weighing the implications for and against the different design choices.

4.1 Good versus bad design

One can argue whether good and bad is relevant when it comes to design. To a certain degree and in a certain context, it is. A quick hacked-together solution is not good design if you want something that lasts and is easy to build on for others. As a software distribution system will be used by and worked on by many different people. Those will of course wary in both roles (installing and maintaining software, building packages, changing the distribution system itself) as well as skill. This is seems obvious, but when one looks at the different systems out there, it is clear that many people coded first and thought about the problem later, rather than the other way around.

It is also important that the different design choices support each other. If not, the user will be confused, the software will be slow and full of errors.

4.2 Who is the user

Since users are different, they will have different goals. A system administrator is usually not interested in working on the packaging system itself. He is more interested in making or installing pre-made packages. To a packager, installing pre-made packages is of little interest, he will be interested in making good packages with as little hassle as possible.

4.2.1 Operations

The number of operations a software distribution tool must support is surprisingly small. Those operations aren’t atomic, but they are operations