B File structures and protocols
Each section of a web-site being exported or imported has a
directory which contains accounting information for that web-site section. In
this directory resides a.o. the
report.log. This log contains
information which should be collected at the origin server to make decisions,
statistics and a merged log of requests (which can be converted into a
The report.log format is completely different than apache-style log files,
because this report.log contains much more information than just requests and
has fields which are more suitable for a distributed environment than the
traditional access log formats such as common and combined log formats
To aid future development, the report.log is not a strict format, but instead
is a free-format file with limited rules on how to separate records of
requests and other relevant data and the fields of data inside a record. It
does not describe which fields in which order should be present.
The report.log is a series of unstructured records of events. Each record is
contained on a single line. Lines which start with a hash sign (
should be ignored and can be used for comments. Each line contains one or
multiple fields with data. Fields are in principle separated with one or
multiple spaced or tabs.
A field is either a single letter, used in the report.log to identify
different type of events or is a key--value pair. Key and value are seperated
with either a equal sign (
=), a colon (
:) or semi-colon
;). The different separators serve different purposes:
Used to separate a key from a value, where the value can only be a number.
These numbers should bare some relation to each other. For instance,
identifiers in principle bear no relation to each other as two persons with
ID 3 and one with ID 5 have no logical personal bonds with each other, not
can you induce that there should also be a person with ID 4.
However a timestamp would be suitable to use with this, as there is a
logical enumeration of time.
The semi-colon is a general key--value pair seperator, where the value field
should not be interpreted as a number, but as some identifier. Normally,
there is a limited amount of possible values for a certain key in the
report.log. In other words; you should not expect to see generic text, but
only keywords or identifiers as values in a semi-colon field.
The colon field serves the same purpose as the semi-colon separator, but the
colon can only be used as the last key--value pair and the value in
that comes after the colon may contain spaces and/or tabs.
The following event types can be in the report.log:
- R a document has been requested by some browsing user;
- U a document update has been detected;
- I the document has been invalidated;
- A to indicate that the policy of a document has
- E to indicate that a document has been evicted from
The following fields can be expected:
- t= The timestamp when the event occured.
- path: The path component of a URL, starting without the initial
location from which it was exported (or imported).
- old; The previous (replication) policy that has been used.
- new; The new (replication) policy to be used on a document.
- lastmod= A timestamp with the last modification time of the
- docsize= The (new) document size.
- client; The IP number of the peer (e.g. the browsing user doing
- elapsed= The amount of time needed to do something (serve a
request for instance).
- sndsize= The number of bytes reported to be shipped.
- browser; The User-Agent reported from the browsing user (very
- referer The Referer field in the request reported by the
browsing user (very optional information).
Timestamps and durations are in
apr_time_t precision, normally
microseconds. Sizes are in bytes.
Normally, but not guaranteed, the following fields are present for each event
||t, client, elapsed, sndsize, browser, referer, path
||t, lastmod, docsize, path
||t, old, new, path
February 27, 2006