From: des Date: Wed, 22 Feb 2006 16:18:23 +0000 (+0000) Subject: Draft specification in DocBook format. X-Git-Url: https://err.no/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=20c02fbb70af0fb322552d2b82738e808cb0de41;p=varnish Draft specification in DocBook format. git-svn-id: svn+ssh://projects.linpro.no/svn/varnish/trunk@12 d4fa192b-c00b-0410-8231-f00ffab90ce4 --- diff --git a/varnish-doc/en/varnish-specification/article.xml b/varnish-doc/en/varnish-specification/article.xml new file mode 100644 index 00000000..6e39f74e --- /dev/null +++ b/varnish-doc/en/varnish-specification/article.xml @@ -0,0 +1,636 @@ + + +
+ + $Id$ + Varnish HTTP Accelerator Draft Specification + + +
+ Introduction + +
+ Overview + + Varnish is a high-performance HTTP accelerator. + + XXX +
+ +
+ Terminology + + The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, + “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and + “OPTIONAL” in this document are to be interpreted as described + in IETF RFC 2119. + + XXX at this time, the above is incorrect because we + started out using MoSCoW prioritisation before realising it was + inadequate for the task. +
+ +
+ To do + + Use consistent terminology throughout the document (see + previous section) + + Clarify the following terms: configuration facility; + logging and statistics facility (split in two?); monitoring and + tuning facility (split in two? does this overlap with + configuration facility?) +
+
+ +
+ General requirements + +
+ License + + XXX two-clause BSD license +
+ +
+ Version control + + All source code and documentation must be kept under + version control in a publicly accessible repository. +
+ +
+ Software platform + + Varnish must be fully functional on the platforms listed + in this section. + + Varnish should also be fully functional on other + POSIX-derived platforms, insofar as this does not require + unreasonable effort. + +
+ FreeBSD + + Varnish must be fully functional on FreeBSD 6.0 or later + officially supported releases, including security and errata + branches. + + The reference platform for FreeBSD compatibility is + FreeBSD 6.1-RELEASE. +
+ +
+ GNU/Linux + + Varnish must be fully functional on GNU/Linux systems + with Linux 2.6.12 or later and GNU libc 2.3.2 or later. + + The reference platform for GNU/Linux compatibility is + Ubuntu 5.10 “Breezy Badger”. +
+
+ +
+ Hardware platform + + Varnish must be fully functional on both 32-bit and 64-bit + Intel-compatible hardware, but may place different limits on + operating parameters (such as cache size) depending on the + platform and the amount of physical and / or virtual memory + available. Varnish must support and take advantage of multiple + CPUs or CPU cores if present. + + The reference hardware platform is a dual-CPU AMD Opteron + system with 4 GB of RAM divided evenly between the CPUs. +
+ +
+ Language + + Varnish must be implemented in C with as few compiler- and + platform-specific extensions as possible. +
+ +
+ Compiler and toolchain + + Varnish must be compilable with the GNU C compiler (GCC) + 3.3.5 or later and toolchain (binutils) 2.15 or later. + + Alternative compilers and toolchains should be supported + insofar as this does not require unreasonable effort. +
+ +
+ Compile-time configuration + + Varnish must use the GNU Autotools for compile-time + configuration. The Makefile templates must be written to work + with any POSIX-compliant make(1) utility. +
+ +
+ Third-party dependencies + + Varnish must not depend on any third-party packages other + than the compiler, toolchain and configuration tools. +
+ +
+ Incidental tools + + Varnish may be accompanied by incidental tools for + purposes such as creating or editing configuration files, + warming up the cache, manipulating or generating source code, + etc. Insofar as these tools are not required at compilation or + installation time nor for the daily operation of Varnish, they + may be written in Perl 5.8 or later, and depend on third-party + Perl modules available from CPAN. +
+ +
+ Coding standards + + All C source code must conform to the FreeBSD style(9) + coding standard. +
+ +
+ Documentation + + Varnish must be accompanied by complete internal and + external documentation. + + All documentation must be in English. + + All documentation must be made available online in HTML + form, and may be made available online in additional formats + such as PDF. + +
+ Internal documentation + + The internal documentation consists of: + + + Code comments. + + + Manual pages describing Varnish internals. + + + Version control history. + + + Requirements and specification in DocBook XML + format. + + + System architecture in DocBook XML format. + + + Developer guidelines and other incidental + documentation either in the project Wiki or in DocBook XML + format. + + +
+ +
+ External documentation + + The external documentation consists of: + + + Manual pages for all daemons and command-line + tools. + + + Installation instructions in DocBook XML + format. + + + Administrator's handbook in DocBook XML + format. + + + Sample configuration files. + + +
+
+
+ +
+ Functional requirements + +
+ Functional description + + Varnish accepts HTTP requests from clients and satisfy + them with documents retrieved from its cache (disk- and / or + memory-based). Documents which are not present in the cache + must be retrieved from a set of preconfigured content servers. + Requests for documents from other servers than the preconfigured + content servers are ignored. +
+ +
+ Protocol support + +
+ HTTP + + Varnish must be able to accept HTTP/1.0 and HTTP/1.1 + requests from both IPv4 and IPv6 clients. + + Varnish must, in general terms, handle these requests in + conformance with IETF RFC 2616. + + Varnish may handle HTTP/0.9 requests in any way it sees + fit, including but not limited to returning a 400 Bad Request + response or simply closing the connection. + + Varnish must use HTTP/1.1 in its communications with the + content servers. + + Varnish may deviate from IETF RFC 2616 when this is + necessary for interoperability with non-conforming clients or + content servers. + + Varnish may deviate from IETF RFC 2616 in cases where + doing so provides a considerable performance advantage without + causing significant harm to interoperability. Any such + deviation must be documented. + + In its communications with clients, Varnish must + interpret IETF RFC 2616 as if it were an origin server. In + its communications with content servers, Varnish must + interpret IETF RFC 2616 as if it were a cache. +
+ +
+ ICP + + Varnish may support ICP for inter-cache coordination. + ICP support may be a compile-time option. +
+ +
+ HTCP + + Varnish may support HTCP for inter-cache coordination. + HTCP support may be a compile-time option. +
+
+ +
+ Content manipulation + + Varnish won't implement content manipulation at this + time. + + Varnish should be designed in such a manner as to make it + possible to implement various kinds of content manipulation + (such as ESI) at a future date. + + XXX ICAP may be worth looking into (but is probably a + performance killer) +
+ +
+ Caching + + Varnish must maintain a local cache of the documets + present on the content server. + +
+ Cached set + + If the amount of memory and / or disk available to + Varnish is not sufficient to cache the entire document set, + Varnish must attempt to identify a subset to cache which + minimizes load on the content servers. + + Varnish should offer multiple alternative cache control + algorithms. At the very least, the LRU (least-recently-used) + and WLRU (LRU weighted by document size) algorithms should be + implemented. +
+ +
+ Expiry + + If a cached document has an expiry time associated with + it, and that time has not yet been reached, Varnish may serve + the document from cache without contacting the content + server. +
+ +
+ Non-cacheable content + + XXX +
+
+ +
+ Management + +
+ Configuration + + XXX +
+ +
+ Logging and statistics + + A separate application is responsible for collecting, + collating and analyzing log data which Varnish makes available + in circular shared memory buffers. + + Varnish must provide the data necessary to compute + lifetime totals and sliding averages for the following: + + + Total size of documents served to clients + + + Total size of data transmitted to clients, including + headers, error messages, etc. + + + Total size of data received from clients, including + request headers etc. + + + Number of client connections received + + + Number of client requests served + + + Client requests broken down by result code + + + Total size of documents retrieved from content + servers + + + Total size of data received from content servers, + including headers, error messages, etc. + + + Total size of data sent to content servers, + including request headers etc. + + + Number of content server connections + initiated + + + Number of content server requests sent + + + Content server requests broken down by result + code + + + Cache effectiveness as the ratio of bytes served to + clients to bytes requested from content servers + + + Cache effectiveness as the ratio of client requests + to content server requests + + + Number of active server processes / threads, broken + down by process / thread type + + + XXX length of request queues + + + + In addition, Varnish must provide the data necessary to + compute the average, median and distribution for the + following: + + + Size of documents served, per unique document + + + Size of documents served, per request + + + Client connection duration + + + Requests per client connection + + + Client request completion time + + + Content server connection duration + + + Requests per content server connection + + + Content server request completion time + + + XXX time spent in request queues + + +
+
+ +
+ Run-time monitoring and tuning + + Varnish must provide low-level monitoring and tuning + facilities. A separate application is responsible for providing + a user-friendly interface to these facilities. + + The following monitoring operations must be + supported: + + + + Cache status of individual documents + + + Cache status of documents matching a glob or regular + expression + + + Access statistics of individual documents + + + Access statistics of documents matching a glob or + regular expression + + + XXX + + + + The following tuning operations must be supported: + + + + Forced invalidation of individual documents + + + Forced invalidation of documents matching a glob or regular expression + + + XXX + + +
+
+ Clustering + + Clustering is defined in this context as a situation where + multiple servers are set up to run Varnish with the same + configuration, serving data from the same content servers to the + same set of clients. + +
+ Configuration + + When multiple Varnish servers act together as a cluster, + the configuration facility is responsible for ensuring that + all nodes share the same configuration and that configuration + changes are applied to all nodes in a timely fashion. +
+ +
+ Logging and statistics + + When multiple Varnish servers act together as a cluster, + the logging and statistics facilities must base its reports on + aggregated data as if the cluster were a single Varnish + server. + + Per-node data may optionally be made available in + addition to aggregated data. +
+ +
+ Run-time monitoring and tuning + + When multiple Varnish servers act together as a cluster, + the run-time monitoring and tuning facilities must propagate + invalidation requests and other administrative commands to all + servers in the cluster. +
+
+
+ +
+ Application structure + +
+ Components + + This section lists the major components in Varnish. + +
+ Listener + + The Listener monitors the listening socket and accepts + incoming client connections. Once the connection is + established, it is passed to the Accepter. + + The Listener should take advantage of accept filters or + similar technologies on systems where they are + available. +
+ +
+ Accepter + + The Accepter reads an HTTP request from a client + connection. It parses the request line and header only to the + extent necessary to establish well-formedness and determine + the requested URL. + + The Accepter then queries the Keeper about the status of + the requested document (identified by its full URL). If the + document is present and valid in the cache, the request is + passed directly to a Sender. Otherwise, it is passed to a + Retriever queue. +
+ +
+ Keeper + + The Keeper manages the document cache. XXX +
+ +
+ Sender + + The Sender transfers the contents of the requested + document to the client. It examines the HTTP request header + to determine the correct way in which to do this – Range, + If-Modified-Since, Content-Encoding and other options may + affect the type and amount of data transferred. + + There may be multiple concurrent Sender threads. +
+ +
+ Retriever + + The Retriever is responsible for retrieving documents + from the content servers. It is triggered either by an + Accepter trying to satisfy a request for a document which is + not in the cache, or by the Janitor when a “hot” document is + nearing expiry. Either way, there may be a queue of requests + waiting for the document to arrive; when it does, the + Retriever passes those requests to a Sender. + + There may be multiple concurrent Retriever + threads. +
+ +
+ Janitor + + The Janitor keeps track of the expiry time of cached + documents and attempts to retrieve fresh copies of documents + which are soon to expire. +
+ +
+ Logger + + The Logger keeps logs of various types of events in + circular shared-memory buffers. These buffers must be managed + using either POSIX shared memory primitives or file-backed + mmap(2). + + It is the responsibility of each module to feed relevant + log data to the Logger. +
+
+
+ + + References + + + + + + +