VENTI-FMT(8) VENTI-FMT(8)
NAME
buildindex, checkarenas, checkindex, conf, fmtarenas,
fmtbloom, fmtindex, fmtisect - prepare and maintain a venti
server
SYNOPSIS
venti/fmtarenas [ -Z ] [ -a arenasize ] [ -b blocksize ]
name file
venti/fmtisect [ -1Z ] [ -b blocksize ] name file
venti/fmtbloom [ -n nblocks | -N nhash ] [ -s size ] file
venti/fmtindex [ -a ] venti.conf
venti/conf [ -w ] partition [ configfile ]
venti/buildindex [ -bd ] [ -i isect ] ... [ -M imemsize ]
venti.conf
venti/checkindex [ -f ] [ -B blockcachesize ] venti.conf tmp
venti/checkarenas [ -afv ] file
DESCRIPTION
These commands aid in the setup, maintenance, and debugging
of venti servers. See venti(6) for an overview of the venti
system and venti(8) for an overview of the data structures
used by the venti server.
Note that the units for the various sizes in the following
commands can be specified by appending `k', `m', or `g' to
indicate kilobytes, megabytes, or gigabytes respectively.
Formatting
To prepare a server for its initial use, the arena parti-
tions and the index sections must be formatted individually,
with fmtarenas and fmtisect. Then the collection of index
sections must be combined into a venti index with fmtindex.
Fmtarenas formats the given file, typically a disk parti-
tion, into an arena partition. The arenas in the partition
are given names of the form name%d, where %d is replaced
with a sequential number starting at 0.
Options to fmtarenas are:
-a arenasize
The arenas are of arenasize bytes. The default is
512M, which was selected to provide a balance between
Page 1 Plan 9 (printed 12/30/24)
VENTI-FMT(8) VENTI-FMT(8)
the number of arenas and the ability to copy an arena
to external media such as recordable CDs and tapes.
-b blocksize
The size, in bytes, for read and write operations to
the file. The size is recorded in the file, and is
used by applications that access the arenas. The
default is 8k.
-4 Create a `version 4' arena partition for backwards com-
patibility with old servers. The default is version 5,
used by the current venti server.
-Z Do not zero the data sections of the arenas. Using
this option reduces the formatting time but should only
be used when it is known that the file was already
zeroed. (Version 4 only; version 5 sections are not
and do not need to be zeroed.)
Fmtisect formats the given file, typically a disk partition,
as a venti index section with the specified name. Each of
the index sections in a venti configuration must have a
unique name.
Options to fmtisect are:
-b bucketsize
The size of an index bucket, in bytes. All the index
sections within a index must have the same bucket size.
The default is 8k.
-1 Create a `version 1' index section for backwards com-
patibility with old servers. The default is version 2,
used by the current venti server.
-Z Do not zero the index. Using this option reduces the
formatting time but should only be used when it is
known that the file was already zeroed. (Version 1
only; version 2 sections are not and do not need to be
zeroed.)
Fmtbloom formats the given file as a Bloom filter (see
venti(6)). The options are:
-n nblock | -N nhash
The number of blocks expected to be indexed by the
filter or the number of hash functions to use. If
the -n option is given, it is used, along with the
total size of the filter, to compute an appropriate
nhash.
Page 2 Plan 9 (printed 12/30/24)
VENTI-FMT(8) VENTI-FMT(8)
-s size The size of the Bloom filter. The default is the
total size of the file. In either case, size is
rounded down to a power of two.
The file argument in the commands above can be of the form
file:lo-hi to specify a range of the file. Lo and hi are
specified in bytes but can have the usual k, m, or g suf-
fixes. Either lo or hi may be omitted. This notation elim-
inates the need to partition raw disks on non-Plan 9 sys-
tems.
Fmtindex reads the configuration file venti.conf and ini-
tializes the index sections to form a usable index struc-
ture. The arena files and index sections must have previ-
ously been formatted using fmtarenas and fmtisect respec-
tively.
The function of a venti index is to map a SHA1 fingerprint
to a location in the data section of one of the arenas. The
index is composed of blocks, each of which contains the map-
ping for a fixed range of possible fingerprint values.
Fmtindex determines the mapping between SHA1 values and the
blocks of the collection of index sections. Once this map-
ping has been determined, it cannot be changed without
rebuilding the index. The basic assumption in the current
implementation is that the index structure is sufficiently
empty that individual blocks of the index will rarely over-
flow. The total size of the index should be about 2% to 10%
of the total size of the arenas, but the exact percentage
depends both on the index block size and the compressed size
of blocks stored. See the discussion in venti(8) for more.
Fmtindex also computes a mapping between a linear address
space and the data section of the collection of arenas. The
-a option can be used to add additional arenas to an index.
To use this feature, add the new arenas to venti.conf after
the existing arenas and then run fmtindex -a.
A copy of the above mappings is stored in the header for
each of the index sections. These copies enable buildindex
to restore a single index section without rebuilding the
entire index.
To make it easier to bootstrap servers, the configuration
file can be stored in otherwise empty space at the beginning
of any venti partitions using conf. A partition so branded
with a configuration file can be used in place of a configu-
ration file when invoking any of the venti commands. By
default, conf prints the configuration stored in partition.
When invoked with the -w flag, conf reads a configuration
file from configfile (or else standard input) and stores it
in partition.
Page 3 Plan 9 (printed 12/30/24)
VENTI-FMT(8) VENTI-FMT(8)
Checking and Rebuilding
Buildindex populates the index for the Venti system
described in venti.conf. The index must have previously been
formatted using fmtindex. This command is typically used to
build a new index for a Venti system when the old index
becomes too small, or to rebuild an index after media fail-
ure. Small errors in an index can usually be fixed with
checkindex, but checkindex requires a large temporary
workspace and buildindex does not.
Options to buildindex are:
-b Reinitialise the Bloom filter, if any.
-d `Dumb' mode; run all three passes.
-i isect Only rebuild index section isect; may be
repeated to rebuild multiple sections. The
name `none' is special and just reads the are-
nas.
-M imemsize The amount of memory, in bytes, to use for
caching raw disk accesses while running
buildindex. (This is not a property of the cre-
ated index.) The usual suffices apply. The
default is 256M.
Checkindex examines the Venti index described in venti.conf.
The program detects various error conditions including:
blocks that are not indexed, index entries for blocks that
do not exist, and duplicate index entries. If requested, an
attempt can be made to fix errors that are found.
The tmp file, usually a disk partition, must be large enough
to store a copy of the index. This temporary space is used
to perform a merge sort of index entries generated by read-
ing the arenas.
Options to checkindex are:
-B blockcachesize
The amount of memory, in bytes, to use for caching raw
disk accesses while running checkindex. The default is
8k.
-f Attempt to fix any errors that are found.
Checkarenas examines the Venti arenas contained in the given
file. The program detects various error conditions, and
optionally attempts to fix any errors that are found.
Page 4 Plan 9 (printed 12/30/24)
VENTI-FMT(8) VENTI-FMT(8)
Options to checkarenas are:
-a For each arena, scan the entire data section. If this
option is omitted, only the end section of the arena is
examined.
-f Attempt to fix any errors that are found.
-v Increase the verbosity of output.
SOURCE
/sys/src/cmd/venti/srv
SEE ALSO
venti(6), venti(8)
BUGS
Buildindex should allow an individual index section to be
rebuilt.
Page 5 Plan 9 (printed 12/30/24)