"regsub" is short for regular expression substitution and it is probably
easiest to explain with some examples:
sub vcl_recv {
set req.url = regsub(req.url, "#.*", "");
}
This will replace the requests URL with the output of the regsub() function
regsub() takes three arguments: the string to be examined, a regular
expression and a replacement string.
In this case, everything after the first '#' is removed (replaced
with nothing).
The replacement string recognizes the following magic sequences:
& - insert everything matched by the regexp
$0 - ditto.
$1 - replace with the first submatch of the regexp
$2 - replace with the second submatch of the regexp
...
$9 - replace with the ninth submatch of the regexp
(The $0..$9 syntax was chosen over the \0...\9 syntax in order to avoid
a nightmare of escape characters in the VCL source code. Arguments and
suggestions are welcome).
A more advanced example:
set bereq.http.ClientIP = regsub(client.ip, "(.*):(.*)", "$2 $1");
The client.ip variable expands to IP:port number, for instance
127.0.0.1:54662
The regular expression "(.*):(.*)" results in the the following matches:
& + $0 "127.0.0.1:54662"
$1 "127.0.0.1"
$2 "54662"
So the replacement string "$2 $1" results in "54662 127.0.0.1"
And the completed header which is sent to the backend will look like:
"ClientIP: 54662 127.0.0.1"
An even more advanced example would be:
set bereq.http.magic = "Client IP = " regsub(client.ip, ":", " port = ");
Where we also exploint the string concatenation ability of the "set" statement.
The result string is built in the request workspace, so you may need
to increase the workspace size if you do a lot of regsub()'s.
Currently there is no decent error handling for running out of workspace.
des [Tue, 3 Jul 2007 14:19:40 +0000 (14:19 +0000)]
Implement "now" and "obj.lastuse", with a note to the effect that the use
of timestamps and clock_gettime() throughout Varnish needs reviewing (as
per IRC discussion with phk)
des [Tue, 3 Jul 2007 14:17:28 +0000 (14:17 +0000)]
Add two VCL variables, "now" and "obj.lastuse". The former returns the
current time, the latter returns the number of seconds since an object
was last requested. The exact semantics of both are slightly fluid at
the moment, and will be revisited at a later date.
des [Tue, 3 Jul 2007 09:09:55 +0000 (09:09 +0000)]
Reorganize the code a little, and add code to wait for all threads to finish
processing pending messages before we exit.
Note that VSL_Dispatch() will read in log data as fast as it can, so when
working from a log file, varnishreplay will usually read in the entire file
into memory within the first few seconds.
Added nagios plugin for varnish. It is very basic, and still needs some more work.
All parameters shown with the use of varnishstat -1 can be monitored.
des [Fri, 29 Jun 2007 12:57:55 +0000 (12:57 +0000)]
Add a regression test for the LRU. Currently, it tests that the child does
not crash when the cache fills up, and that a request for a document that
ought to have been evicted does not result in a cache hit.
des [Thu, 28 Jun 2007 15:10:24 +0000 (15:10 +0000)]
Almost complete rewrite to address a number of interface issues. Most
importantly, the display will now dynamically scale when the terminal is
resized, and will be updated regularly regardless of the rate at which log
data arrive.
des [Thu, 28 Jun 2007 10:29:27 +0000 (10:29 +0000)]
Numerous improvements: display the instance name in the upper right corner;
don't show more lines than the terminal can fit; correctly react to terminal
size changes; react to Ctrl-L (redraw) and Ctrl-C / Ctrl-Q (quit).
The layout code could use a cleanup, and we still fail to DTRT if the width
of the terminal is less than that of our data, but this should not be a
problem in daily use.
des [Thu, 28 Jun 2007 09:35:56 +0000 (09:35 +0000)]
Some of the statistics we gather are accumulated totals, while others are
instantaneous measurements. For instance, we report the total number of
allocator requests made over the child's lifetime, but we also report the
amount of storage in use at any particular moment.
The difference is important, because accumulated totals can be averaged
over the program's lifetime (or over the last N seconds), but instantaneous
measurements can't.
Recycle the format field in MAC_STAT() (it was never used anyway) into a
single-character flag indicating whether each item is an accumulated total
('a') or an instantaneous measure ('i'). Use this in varnishstat to skip
averaging non-averageable numbers.
Also rework varnishstat's "once" mode to show 1) each statistic's symbolic
name, 2) its current value, 3) if appropriate, its value averaged over the
process lifetime, and 4) its description.
The reason for displaying the symbolic name is to simplify scripting, and
to serve as a reference for looking up symbolic names to pass to e.g. the
upcoming Nagios plugin.
des [Wed, 27 Jun 2007 12:56:04 +0000 (12:56 +0000)]
Mostly finish the LRU code and its integration:
- Wrap the storage code so we don't need to duplicate the "toss out some old
crap and try again" logic everywhere. This will also help when / if we
decide to add support for multiple concurrent storage arenas.
- While I'm at it, implement sma_trim().
- Rework the interaction between the LRU and expiry code. Instead of placing
objects retired by the LRU on death row, immediately terminate them.
- Give the LRU code its own fake session and worker so we don't have to pass
it a session pointer.
- Rework the LRU API, and add LRU_DiscardOne() which discards a single
object. This is what the stevedore code uses.
Known or suspected issues:
- The LRU and expiry code should use the same mutex, and / or the possiblity
for races between them should be examined closely.
- LRU_Init() needs to be looked at and possibly moved.
- LRU_DiscardSpace() and LRU_DiscardTime() are unused and quite possibly useless.
- Logging and statistics related to the LRU need more attention.
phk [Wed, 27 Jun 2007 12:37:34 +0000 (12:37 +0000)]
Make sure to reset the workspaces of the backend connection between
requests, otherwise we end up filling it up with Content-Length: headers
if we manage to keep the backend busy.
Also make the snapshot of the preferred workspace-size volatile to ensure
that we are consistent.
des [Tue, 26 Jun 2007 11:52:25 +0000 (11:52 +0000)]
Further refine -n handling. The argument to -n is now either an absolute
path, or a path relative to $localstatedir/varnish. By default, it is set
to the host name, which results in $localstatedir/varnish/$hostname.
This logic is centralized in instance.c, which is compiled into both
varnishd and libvarnishapi, with prototypes in varnishapi.h and heritage.h.
phk [Mon, 25 Jun 2007 21:12:17 +0000 (21:12 +0000)]
Redo the -n argument code, it had too many problems:
We need to process -P and -f arguments before we change directory.
(ticket 120)
(XXX: what about storage and hash arguments ??)
The daemon(3) call should not change our directory subsequently.
(ticket 121)
There is no need to enforce a hostname style format on the argument,
a directory nam makes much more sense, since that is what we need.
Defaulting to /tmp instead of our hostname makes more sense (ticket 119).
This also allows the admin to use a different directory if /tmp is
mounted noexec (ticket 111)
Put the directoryname used in the proctitle (via heritage)
XXX: for docs: vcl.load CLI commands will work relative to the -n directory.
des [Mon, 25 Jun 2007 17:04:09 +0000 (17:04 +0000)]
First step in implementing early retirement of objects when the cache fills
up: implement a "sloppy" LRU list. An object is placed on the list (or moved
to the head of the list if it's already on it and hasn't moved recently) by
calling LRU_Enter(), and removed by calling LRU_Remove(). LRU_DiscardSpace()
will iterate through the LRU list, starting at the back, and retire objects
(by adding them to the deathrow list) until the sum of the length of the
retired objects reaches a certain number. Similarly, LRU_DiscardTime() will
retire objects which haven't moved since a specified cutoff date. In both
cases, vcl_discard() will be given a chance to inspect the object and veto
its retirement.
Currently, LRU_Enter() and LRU_Remove() are called from HSH_Lookup() and
HSH_Deref() respectively. There may be better alternatives.
Neither LRU_DiscardSpace() nor LRU_DiscardTime() is currently called from
anywhere. There are a number of issues to consider: for instance, even if
LRU_DiscardSpace() is called when a high-water mark is reached, there is
still a possibility that the cache might fill up before it has had a chance
to finish and the hangman has had a chance to process the deathrow list.
projects / varnish / log