First part of major backend overhaul.

First part of major backend overhaul.

*** Please do not use -trunk in production until I say so again ***

I have not entirely decided in the precise terminology, so the following
may sound a lot more complicated than it really is:

In VCL we can now have "backends" and "directors" both of which we
treat as a "backend".

When we define backends and directors in VCL, they refer to "backend
hosts" which is just another way to say "hostname+portname" but later
these will grow other parameters (max connections etc).

A director is a piece of code that selects a "backend host" somehow,
"random" and "round-robin" are the first algorithms.  A backend
can still be specified directly of course, that's the "simple director"
that always return the same "backend host".

This is probably where an example is in order:

	/* A backend as we know it */
	backend b1 {
		.host = "fs";
		.port = "80";
	}

	/* A director */
	director b2 random {
		{
			/* We can refer to named backends */
			.backend        = b1;
			.weight         = 7;
		}
		{
			/* Or define them inline */
			.backend        = {
				.host = "fs2";
			}
			.weight         = 3;
		}
	}

	sub vcl_recv {
		if (req.url ~ "\[[a-z]]") {
			set req.backend = b2;
		} else {
			set req.backend = b1;
		}
	}

This results in quite a lot of changes in the C code, VRT API and
VCL compiler, the major thrust being:

Directors like "simple" and "random" will not have to think about
the actual connections to the backends, but just concentrate on
selecting which backend should be used.

When a new VCL is loaded, it will instantiate all directors, but
try to reuse any preexisting "backend hosts" (which we still
call "backend" in the C code).

This is simple for a backend like "b1" in the example above, but
sligthly more complex for the backend inlined in b2.  The VCL
compiler solves this, by qualifying the ident string for the inlined
backend host with the prefix "b2 random :: 2 :: ", so that a reload
of the same director with the same (unchanged) inline backend host
will match, but none other will.

One implication of instantiating all directors for every VCL load,
is that private statistics cannot be reused, but stats on the
backend host can.  This is likely a very fine point of no consequence.

Once the backend is selected by the director, the generic code in
cache_backend.c will cope with reusing the connection pool,
establishing connections and all that, moving most of the nastyness
out of directors, leaving cache_dir_simple.c with only 96 lines of
code, of which the license is a large fraction.

Until now, we have done automatic DNS re-lookups, but they seem to
cause more grief than advantage (I suspect some of the DNS lookups
to be resposible for long timeouts), so that will be dropped, and
instead we might add an explicit CLI command for this later.

The code as here committed can handle a couple of simple requests,
but there are a large number of INCOMPL()'s that need to be resolved
before this is ready for prime time again.

git-svn-id: svn+ssh://projects.linpro.no/svn/varnish/trunk@2437 d4fa192b-c00b-0410-8231-f00ffab90ce4