From: des Date: Mon, 27 Feb 2006 07:28:50 +0000 (+0000) Subject: Separate specification from architecture. X-Git-Url: https://err.no/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=851842080565108b3dd6ccf05300233156943c45;p=varnish Separate specification from architecture. git-svn-id: svn+ssh://projects.linpro.no/svn/varnish/trunk@30 d4fa192b-c00b-0410-8231-f00ffab90ce4 --- diff --git a/varnish-doc/en/varnish-architecture/article.xml b/varnish-doc/en/varnish-architecture/article.xml index fb5bb0fb..e9862da7 100644 --- a/varnish-doc/en/varnish-architecture/article.xml +++ b/varnish-doc/en/varnish-architecture/article.xml @@ -6,696 +6,9 @@
$Id$ - Varnish HTTP Accelerator Draft Specification + Varnish HTTP Accelerator Architecture -
- Introduction - -
- Overview - - Varnish is a high-performance HTTP accelerator. - - XXX -
- -
- Terminology - - The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, - “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and - “OPTIONAL” in this document are to be interpreted as described - in . - - XXX at this time, the above is incorrect because we - started out using MoSCoW prioritisation before realising it was - inadequate for the task. -
- -
- To do - - Use consistent terminology throughout the document (see - previous section) - - Clarify the following terms: configuration facility; - logging and statistics facility (split in two?); monitoring and - tuning facility (split in two? does this overlap with - configuration facility?) -
-
- -
- General requirements - -
- License - - XXX two-clause BSD license -
- -
- Version control - - All source code and documentation must be kept under - version control in a publicly accessible repository. -
- -
- Software platform - - Varnish must be fully functional on the platforms listed - in this section. - - Varnish should also be fully functional on other - POSIX-derived platforms, insofar as this does not require - unreasonable effort. - -
- FreeBSD - - Varnish must be fully functional on FreeBSD 6.0 or later - officially supported releases, including security and errata - branches. - - The reference platform for FreeBSD compatibility is - FreeBSD 6.1-RELEASE. -
- -
- GNU/Linux - - Varnish must be fully functional on GNU/Linux systems - with Linux 2.6.12 or later and GNU libc 2.3.2 or later. - - The reference platform for GNU/Linux compatibility is - Ubuntu 5.10 “Breezy Badger”. -
-
- -
- Hardware platform - - Varnish must be fully functional on both 32-bit and 64-bit - Intel-compatible hardware, but may place different limits on - operating parameters (such as cache size) depending on the - platform and the amount of physical and / or virtual memory - available. Varnish must support and take advantage of multiple - CPUs or CPU cores if present. - - The reference hardware platform is a dual-CPU AMD Opteron - system with 4 GB of RAM divided evenly between the CPUs. -
- -
- Language - - Varnish must be implemented in C with as few compiler- and - platform-specific extensions as possible. -
- -
- Compiler and toolchain - - Varnish must be compilable with the GNU C compiler (GCC) - 3.3.5 or later and toolchain (binutils) 2.15 or later. - - Alternative compilers and toolchains should be supported - insofar as this does not require unreasonable effort. -
- -
- Compile-time configuration - - Varnish must use the GNU Autotools for compile-time - configuration. The Makefile templates must be written to work - with any POSIX-compliant make(1) utility. -
- -
- Third-party dependencies - - Varnish must not depend on any third-party packages other - than the compiler, toolchain and configuration tools. -
- -
- Incidental tools - - Varnish may be accompanied by incidental tools for - purposes such as creating or editing configuration files, - warming up the cache, manipulating or generating source code, - etc. Insofar as these tools are not required at compilation or - installation time nor for the daily operation of Varnish, they - may be written in Perl 5.8 or later, and depend on third-party - Perl modules available from CPAN. -
- -
- Coding standards - - All C source code must conform to the FreeBSD style(9) - coding standard. -
- -
- Documentation - - Varnish must be accompanied by complete internal and - external documentation. - - All documentation must be in English. - - All documentation must be made available online in HTML - form, and may be made available online in additional formats - such as PDF. - -
- Internal documentation - - The internal documentation consists of: - - - Code comments. - - - Manual pages describing Varnish internals. - - - Version control history. - - - Requirements and specification in DocBook XML - format. - - - System architecture in DocBook XML format. - - - Developer guidelines and other incidental - documentation either in the project Wiki or in DocBook XML - format. - - -
- -
- External documentation - - The external documentation consists of: - - - Manual pages for all daemons and command-line - tools. - - - Installation instructions in DocBook XML - format. - - - Administrator's handbook in DocBook XML - format. - - - Sample configuration files. - - -
-
-
- -
- Functional requirements - -
- Functional description - - Varnish accepts HTTP requests from clients and satisfy - them with documents retrieved from its cache (disk- and / or - memory-based). Documents which are not present in the cache - must be retrieved from a set of preconfigured content servers. - Requests for documents from other servers than the preconfigured - content servers are ignored. -
- -
- Protocol support - -
- HTTP - - Varnish must be able to accept HTTP/1.0 and HTTP/1.1 - requests from both IPv4 and IPv6 clients. - - Varnish must, in general terms, handle these requests in - conformance with . - - Varnish may handle HTTP/0.9 requests in any way it sees - fit, including but not limited to returning a 400 Bad Request - response or simply closing the connection. - - Varnish must use HTTP/1.1 in its communications with the - content servers. - - Varnish may deviate from when - this is necessary for interoperability with non-conforming - clients or content servers. - - Varnish may deviate from in - cases where doing so provides a considerable performance - advantage without causing significant harm to - interoperability. Any such deviation must be - documented. - - In its communications with clients, Varnish must - interpret as if it were an origin - server. In its communications with content servers, Varnish - must interpret as if it were a - cache. -
- -
- ICP - - Varnish may support ICP for inter-cache coordination. - ICP support may be a compile-time option. -
- -
- HTCP - - Varnish may support HTCP for inter-cache coordination. - HTCP support may be a compile-time option. -
-
- -
- Content manipulation - - Varnish won't implement content manipulation at this - time. - - Varnish should be designed in such a manner as to make it - possible to implement various kinds of content manipulation - (such as ESI) at a future date. - - XXX ICAP may be worth looking into (but is probably a - performance killer) -
- -
- Caching - - Varnish must maintain a local cache of the documets - present on the content server. - -
- Cached set - - If the amount of memory and / or disk available to - Varnish is not sufficient to cache the entire document set, - Varnish must attempt to identify a subset to cache which - minimizes load on the content servers. - - Varnish should offer multiple alternative cache control - algorithms. At the very least, the LRU (least-recently-used) - and WLRU (LRU weighted by document size) algorithms should be - implemented. -
- -
- Cacheability - - A request which includes authentication headers must not - be served from cache. - - Varnish must interpret Cache-Control directives received - from content servers as follows: - - - - public: the document will be cached even if - authentication headers are present. - - - private: the document will not be cached, since - Varnish is a shared cache. - - - no-cache: the document will not be cached. - - - no-store: XXX - - - s-maxage: overrides max-age, since Varnish is a - shared cache. - - - max-age: overrides the Expires header. - - - min-fresh: ignored. - - - max-stale: ignored. - - - only-if-cached: ignored. - - - must-revalidate: as specified in §14.9.4. - - - proxy-revalidate: as must-revalidate. - - - no-transform: ignored. - - - - Varnish must ignore Cache-Control directives received - from clients. -
- -
- Expiry - - If a content server returns a document with a s-maxage - directive, Varnish will set the expiry time for that document - to the time of the request plus the number of seconds - specified by the directive. - - If a content server returns a document with no s-maxage - directive but a max-age directive, Varnish will set the expiry - time for that document to the time of the request plus the - number of seconds specified by the max-age directive. - - If a content server returns a document with no s-maxage - or max-age directive but an Expires header, Varnish must set - the expiry time for that document to the value specified by - the Expires header. - - When sending a document to a client, Varnish must set - the Expires header to the document's expiry time. -
- -
- Refreshing - - Varnish must attempt to refresh documents before they - expire, in order to avoid stalling a client request while - retrieving an expired document. - - XXX -
-
- -
- Management - -
- Management interface - - Varnish must provide an interface for external - management utilities. This interface must accept connections - on a Unix socket and / or a TCP socket, depending on - configuration. - - Varnish may assume that the management interface is - adequately protected by socket permissions or firewall rules, - as applicable, and that any data it receives through this - interface is valid management data from an authorized - administrator. - - Varnish may further assume that all data received - through the management interface is well-formed and - meaningful. -
- -
- Management library - - Varnish must be accompanied by a C library, hereafter - referred to as the management library, which provides a - reasonably high-level API to the management interface - - Varnish should be accompanied by a Perl library which - provides Perl bindings to the management library. - - Varnish may be accompanied by libraries which provide - appropriate bindings to the management library for other - programming or scripting languages. -
- -
- CLI management tool - - Varnish must be accompanied by a CLI management tool, - written in C, which serves as a front-end to the management - library. - - The CLI management tool must allow management commands - to be passed on the command line. - - In addition, the CLI management tool should offer an - interactive mode using libreadline, libedit or a similar - line-editing library. -
- -
- Web-based management tool - - Varnish should be accompanied by a web-based management - tool. This tool should have the ability to generate graphs - and histograms based on the data described in . -
- -
- Plugins for third-party tools - - Varnish may be accompanied by plugins for third-party - management tools such as Munin, Nagios and NAV. -
- -
- Configuration - - XXX -
- -
- Logging and statistics - - A separate application is responsible for collecting, - collating and analyzing log data which Varnish makes available - in circular shared memory buffers. - - Varnish must provide the data necessary to compute - lifetime totals and sliding averages for the following: - - - - Total size of documents served to clients - - - Total size of data transmitted to clients, including - headers, error messages, etc. - - - Total size of data received from clients, including - request headers etc. - - - Number of client connections received - - - Number of client requests served - - - Client requests broken down by result code - - - Total size of documents retrieved from content - servers - - - Total size of data received from content servers, - including headers, error messages, etc. - - - Total size of data sent to content servers, - including request headers etc. - - - Number of content server connections - initiated - - - Number of content server requests sent - - - Content server requests broken down by result - code - - - Cache effectiveness as the ratio of bytes served to - clients to bytes requested from content servers - - - Cache effectiveness as the ratio of client requests - to content server requests - - - Number of active server processes / threads, broken - down by process / thread type - - - XXX length of request queues - - - - In addition, Varnish must provide the data necessary to - compute the average, median and distribution for the - following: - - - - Size of documents served, per unique document - - - Size of documents served, per request - - - Client connection duration - - - Requests per client connection - - - Client request completion time, broken down by - request type (HEAD / GET), cache status (HIT / MISS) and - outcome (200, 404...) - - - Content server connection duration - - - Requests per content server connection - - - Content server request completion time, broken down - by request type (HEAD / GET) and outcome (200, - 404...) - - - XXX time spent in request queues - - -
-
- -
- Run-time monitoring and tuning - - Varnish must provide low-level monitoring and tuning - facilities. A separate application is responsible for providing - a user-friendly interface to these facilities. - - The following monitoring operations must be - supported: - - - - Cache status of individual documents - - - Cache status of documents matching a glob or regular - expression - - - Access statistics of individual documents - - - Access statistics of documents matching a glob or - regular expression - - - XXX - - - - The following tuning operations must be supported: - - - - Forced invalidation of individual documents - - - Forced invalidation of documents matching a glob or regular expression - - - XXX - - -
-
- Clustering - - Clustering is defined in this context as a situation where - multiple servers are set up to run Varnish with the same - configuration, serving data from the same content servers to the - same set of clients. - -
- Management - - Varnish must be accompanied by a multiplexer for the - management interface which provide a single point of access to - the entire cluster for management tools such as the CLI - management tool or a web-based management interface. - - The management protocol must be designed to allow - management commands to be targeted at individual nodes. The - default behaviour must be to propagate management commands to - all nodes in the cluster. -
- -
- Configuration - - When multiple Varnish servers act together as a cluster, - the configuration facility is responsible for ensuring that - all nodes share the same configuration and that configuration - changes are applied to all nodes in a timely fashion. -
- -
- Logging and statistics - - When multiple Varnish servers act together as a cluster, - the logging and statistics facilities must base its reports on - aggregated data as if the cluster were a single Varnish - server. - - Per-node data may optionally be made available in - addition to aggregated data. -
- -
- Run-time monitoring and tuning - - When multiple Varnish servers act together as a cluster, - the run-time monitoring and tuning facilities must propagate - invalidation requests and other administrative commands to all - servers in the cluster. -
-
-
-
Application structure @@ -789,10 +102,6 @@ References - - - -
diff --git a/varnish-doc/en/varnish-specification/article.xml b/varnish-doc/en/varnish-specification/article.xml index fb5bb0fb..18002379 100644 --- a/varnish-doc/en/varnish-specification/article.xml +++ b/varnish-doc/en/varnish-specification/article.xml @@ -696,96 +696,6 @@ -
- Application structure - -
- Components - - This section lists the major components in Varnish. - -
- Listener - - The Listener monitors the listening socket and accepts - incoming client connections. Once the connection is - established, it is passed to the Accepter. - - The Listener should take advantage of accept filters or - similar technologies on systems where they are - available. -
- -
- Accepter - - The Accepter reads an HTTP request from a client - connection. It parses the request line and header only to the - extent necessary to establish well-formedness and determine - the requested URL. - - The Accepter then queries the Keeper about the status of - the requested document (identified by its full URL). If the - document is present and valid in the cache, the request is - passed directly to a Sender. Otherwise, it is passed to a - Retriever queue. -
- -
- Keeper - - The Keeper manages the document cache. XXX -
- -
- Sender - - The Sender transfers the contents of the requested - document to the client. It examines the HTTP request header - to determine the correct way in which to do this – Range, - If-Modified-Since, Content-Encoding and other options may - affect the type and amount of data transferred. - - There may be multiple concurrent Sender threads. -
- -
- Retriever - - The Retriever is responsible for retrieving documents - from the content servers. It is triggered either by an - Accepter trying to satisfy a request for a document which is - not in the cache, or by the Janitor when a “hot” document is - nearing expiry. Either way, there may be a queue of requests - waiting for the document to arrive; when it does, the - Retriever passes those requests to a Sender. - - There may be multiple concurrent Retriever - threads. -
- -
- Janitor - - The Janitor keeps track of the expiry time of cached - documents and attempts to retrieve fresh copies of documents - which are soon to expire. -
- -
- Logger - - The Logger keeps logs of various types of events in - circular shared-memory buffers. These buffers must be managed - using either POSIX shared memory primitives or file-backed - mmap(2). - - It is the responsibility of each module to feed relevant - log data to the Logger. -
-
-
- References