.Dd December 5, 2013
.Dt EZJAIL 7 USD
.Os
.Sh NAME
.Cm ezjail
.Nd Jail administration framework.
.Sh SYNOPSIS
.Nm ezjail-admin Ar command arguments...
.Sh OVERVIEW
The ezjail commands provide a simple way to create multiple jails
using FreeBSD's jail system. It simplifies jail administration effort
and minimizes jail system resource usage.
.Pp
If you are not familiar with the FreeBSD jail concept, please refer to
.Xr jail 8
before continuing. For additional design information, see the ezjail
site at
.Li http://erdgeist.org/arts/software/ezjail .
.Sh DESCRIPTION
The ezjail system enables the system administrator to create multiple
OS-level virtualization containers called jails. Services like web
servers, mail servers, FTP servers, are typically under frequent attack
from the public Internet and are exposed to possible compromise. The
typical usage of jails is to run a single service in each jail and if
that service becomes compromised the rest of the jails and the host
system are protected from also being compromised.
.Pp
The major shortcoming of jails is that each jail has its own copy of
the world. This eats disk space, inodes, and more importantly,
prevents the sharing of binaries images between jails, thus increasing
the memory pressure on the host system. In addition, this causes a
major administration headache when comes the time to update the host
system, as each jail needs to be updated independently.
.Pp
Ezjail addresses these problems by creating a single basejail (a read-only
.Xr nullfs 4
mounted directory) populated with the same binaries as the host
system which is then shared across all the other service jails created
by ezjail. Is is possible to update the base jail (and thus all the jails) in
a single ezjail command.
.Pp
Typical usage of jails include separation of services, creating test
environments, consolidation of different services on a single physical
host, and more.
.Sh EZJAIL SYSTEM
The administrative interface to the ezjail system is the
.Xr ezjail-admin 8
command. It is used to install the ezjail environment, create new
jails, archive, restore, delete and update jails, open a jail console,
and list the status of all the jails. See below for example usage, and
refer to its man page for complete usage details.
.Pp
Ezjail reads its configuration from its
.Xr ezjail.conf 5 .
Normally it will not be necessary to edit this file, as some sane defaults
are provided. A sample configuration is installed as
.Pa EZJAIL_PREFIX/etc/ezjail.conf.sample .
.Pp
A script is also installed as
.Pa ezjail
in the rc.d system to allow jails under ezjails control to be started at boot
time, given ezjail is enabled by setting the
.Xr rc.conf 5
variable
.Dq Li $ezjail_enable
to
.Dq Li YES .
.Sh WHAT'S IN A JAIL
.Ss The life of an ezjail installation
The base jail is first created by running
.Nm Cm update
or
.Nm Cm install .
Example usage of this command is section
.Sx EXAMPLES .
This will create the base jail, setup a template jail used to
setting up new jails, install an example flavour (see below) and
configure miscellaneous things.
.Pp
This step is necessary before using the ezjail system. In particular,
it is not possible to create new jails without initializing the base
jail in advance.
.Pp
Once the base jail has been created, new jails may be created with
.Nm Cm create .
A new jail is defined by its name and can have one or multiple IP
addresses. Creating a new jail involves copying the template jail to the
new location, configuring
.Xr nullfs 4
mounts for giving access to the base jail, and little more. A jail
that has just be created occupies about 2MB of disk space ; when
running, only a handful of daemons (cron, syslog, sendmail mainly) use
memory.
.Pp
After their creation, jails may be archived to a
.Xr pax 1
archive, restored, and eventually deleted.
.Pp
When a new version of FreeBSD is released, or when an errata is
published, only the base jail need to be updated. Both source upgrades
and binary upgrades (using
.Xr freebsd-update 8 )
are supported. The
.Xr ports 7
collection may also be updated by ezjail, but individual ports need to
be upgraded individually by the administrator.
.Ss Anatomy of a Jail
In the ezjail system, a jail is defined by a root directory and a
couple of configuration values, mainly a name and IP addresses. The
root directory of the jail contains only the jail-specific files:
configuration files, data files, and ports installed by the
administrator. The base system is shared amongst all jails, using a
.Xr nullfs 4
mount. This saves space and inodes (especially when the ports
collection in made available to the jails), and also memory, as the
kernel is now able to share copies of running programs between the
jails.
.Pp
Unless the variable
.Dq Li $ezjail_jaildir
has been set by the administrator, the root directory of the jail is
kept in
.Pa /usr/jails ,
which therefore needs to reside on a partition big enough.
.Pp
There are also file-based jails, in which the storage space for the
jail is kept in a file mounted with
.Xr mdconfig 8 .
There are two advantages to image jails. The amount of disk space
allocated to the jail is limited, while normal jails have no bound on
the amount of disk space they use. On the other hand, the space
dedicated to the jail is no longer available to the host, even if the
jail doesn't use all its allocated space.
.Pp
Image jails may also be encrypted using
.Xr bde 4
or
.Xr geli 8 ,
depending on the options given at creation time.
.Ss Using ZFS
To give more precise control over the resources consumed by a jail,
ezjail allows putting each jail in its own
.Xr zfs 8
filesystem. See
.Sx Jail Creation Examples
for details.
.Pp
Also, ezjail can be configured to install its basejail and the accompanying
template for all new jails into its own filesystem. Set the
.Dq Li $ezjail_use_zfs
variable in your
.Pa ezjail.conf
to
.Dq YES
before running
.Nm Cm update
or
.Nm Cm install .
.Pp
To use any zfs feature in ezjail, you first need to configure the destination
ZFS filesystem using the
.Dq Li $ezjail_jailzfs
variable.
.Pp
You can use ZFS jails without installing the basejail into its own ZFS
filesystem and vice versa. In order to create ZFS jails by default, set the
.Dq Li $ezjail_use_zfs_for_jails
variable to
.Dq YES .
.Ss Per-Jail options
As we saw earlier, a jail is described by a file in
.Pa EZJAIL_PREFIX/etc/ezjail/ .
This file has the same name as the jail it configures. It is a set of
variables interpreted by
.Xr sh 1 ,
much like
.Xr rc.conf 5
is. This file is created at the same time as the jail, and usually
doesn't require tweaking from the administrator.
.Pp
In addition to the variables described below, any variable used by the
init script
.Pa /etc/rc.d/jail
may be added manually by the administrator. The following variables
are handled by ezjail, replacing JAILNAME with the actual name of the jail:
.Bl -tag -width indent
.It jail_JAILNAME_hostname
The hostname of the jail. Defaults to the name of the jail, unless
special characters needed to be stripped.
.It jail_JAILNAME_ip
The IP addresses the jail is allowed to use.
.Pp
Since FreeBSD 7.2,
several IP addresses may be given, separated by commas.
.Pp
Since FreeBSD 9.0
each IP address can be prefixed by an interface name followed by the pipe
symbol. It will then automatically be configured on that interface when the
jail is started and removed from the interface when the jail stops. (You
will probably have to escape the pipe symbol, though.)
.It jail_JAILNAME_rootdir
The directory holding the jail files (the directory used as a mount
point for file-based jails). Defaults to the jail name inside
.Dq Li $ezjail_jaildir .
.It jail_JAILNAME_exec_start
The command to run inside the jail when starting it. Defaults to
.Dq Li $ezjail_exec_start
or
.Dq Li /bin/sh /etc/rc .
.It jail_JAILNAME_exec_stop
The command to run inside the jail when stopping it. Defaults to the
empty string, which means
.Dq Li /bin/sh /etc/rc.shutdown .
.It jail_JAILNAME_mount_enable
A boolean
.Dq ( YES
or
.Dq NO ) ,
that specifies whether the filesystems in
.Pa /etc/fstab. Ar JAILNAME
are carried out. Set by ezjail to
.Dq Li YES ,
set to
.Dq Li NO
at your own risk.
.It jail_JAILNAME_devfs_enable
A boolean specifying whether to mount a
.Pa /dev
filesystem inside the jail. Defaults to
.Dq Li $ezjail_devfs_enable ,
or
.Dq Li YES .
.It jail_JAILNAME_devfs_ruleset
The ruleset to apply when mounting a
.Pa /dev
filesystem inside a jail. Defaults to
.Dq Li $ezjail_devfs_ruleset ,
or
.Dq Li devfsrules_jail .
.It ezjail_JAILNAME_procfs
A boolean specifying whether to mount a
.Pa /proc
filesystem inside the jail. Defaults to
.Dq Li $ezjail_procfs_enable ,
or
.Dq Li YES .
.It ezjail_JAILNAME_fdescfs
A boolean specifying whether to mount a
.Pa /dev/fs
filesystem inside the jail. Defaults to
.Dq Li $ezjail_fdescfs_enable ,
or
.Dq Li YES .
.It ezjail_JAILNAME_image
The path to the image file backing the jail, if the jail is
file-based; or the empty string.
.It ezjail_JAILNAME_imagetype
The type of the image, if the jail is file-based; the empty string
otherwise.
.It ezjail_JAILNAME_attachparams
The parameters to pass to the tool used to decrypt file-based,
encrypted jails. Initialized from the
.Fl C
option when creating such a jail, or the empty string.
.It ezjail_JAILNAME_attachblocking
.Dq Li YES
if the jail requires interaction with the administrator when starting
(typically, encrypted jails that needs a password to be decrypted).
.It ezjail_JAILNAME_forceblocking
If
.Dq Li YES ,
start the jail even when it is marked as blocking.
.It ezjail_JAILNAME_zfs_datasets
For ZFS jails, additional ZFS datasets to attach to the jail when
starting it. Taken from the
.Fl z
option when configuring a jail; the empty string otherwise.
.It ezjail_JAILNAME_cpuset
The processor set to place the jail in when starting it (see
.Xr cpuset 1 ) .
Taken from the
.Fl c
option when configuring a jail; the empty string otherwise.
.It ezjail_JAILNAME_fib
The network view to give to the jail (see
.Xr setfib 1 )
when starting it. Taken from the
.Fl f
option when configuring the jail; the empty string otherwise.
.It ezjail_JAILNAME_parameters
The parameter set to be configured to the jail (see
.Xr jail 8 )
when starting it. You need to configure this by hand.
.It ezjail_JAILNAME_post_start_script
The path to a script that will be executed after the jail
successfully was created. The script receives two parameters,
the jid and the jail name.  You need to configure this by hand.
.El
.Pp
In addition to these
.Xr sh 1 Ns No -style
variables, the administrator may add comment lines starting with
.Dq PROVIDE: ,
.Dq REQUIRE:
and
.Dq BEFORE: .
These comments are used by
.Xr rcorder 8
to determine the order in which the jails are started. The default is
to keep
.Dq REQUIRE
and
.Dq BEFORE
empty, meaning the jails are started in no particular order.
.Ss Flavours
When a jail is created, it is not configured; in particular you likely
want to edit files such as
.Pa /etc/resolv.conf , /etc/localtime
and others. You may also want to create some system users, maybe
enable
.Xr sshd 8 .
Ezjail solves this problem by using the concept of
.Dq flavours .
When a flavour is selected at jail creation time, the flavour
directory tree is merged into the new jail's directory tree. In
addition, the jail is configured so that on its first boot, the file
.Pa ezjail.flavour
is executed.
.Pp
As part of the install sub-command, the flavour base directory
was created as
.Pa /usr/jails/flavours
and populated with an single flavour named
.Cm example .
This flavour contains 3 files customized for running in a
jail
.Pa ( etc/make.conf , etc/periodic.conf , etc/rc.conf ) .
The example
.Pa ezjail.flavour
also show how to create users, and introduce the convention of placing
packages in
.Pa /pkg
that are installed when the jail is first brought up. You are
encouraged to copy the example flavour to create your own flavour.
Typical flavour usages include setting up jails with site-specific
configuration, creating classes of jails for development or testing
(such as a webdev flavour that would install Apache with your
favourite web development framework), pre-creating local users, and so
on.
.Ss Updating the Base Jail
We already mentioned how easy it is to update jails, since only one
copy needs to be updated. Ezjail only handles updating the base
system; updating the ports is left to the administrator (but see
.Dq Li ports-mgmt/jailaudit
for a way to get notified of ports in need of an update). Updates are
handled with the
.Nm Cm update
command. It is possible to update the base jail from source or from
binary packages. If a base jail already exists, the
.Cm update
command installs the world in a temporary directory before moving it
to the basejail, thus leaving intact all installed libraries. After
making sure all software running in the jails is linked with the new
libraries, you may want to remove the old library versions. It is
often a good idea to update the jails when a new kernel is installed
in the host, using the same sources.
.Ss Starting Jails
Like all
.Xr rc 8
scripts, the ezjail script
.Pa EZJAIL_PREFIX/etc/rc.d/ezjail
accepts parameters
.Cm start , restart No and Cm stop, No running, restarting and stopping
all (non-blocking) jails under ezjail's control by default. When passed an
additional list of jails, only these jails are acted upon.
.Pp
The order in which jails are started is determined by the
.Xr rcorder 8
tool, using cues from the jail configurations in ezjails
.Pa EZJAIL_PREFIX/etc/ezjail
control directory.
.Pp
The script examines its config, attaches and mounts images, and sets
variables for each jail in the list before passing its command on
to the
.Pa /etc/rc.d/jail
script.
.Pp
To interactively start all crypto image jails (or those depending on
them), that were not automatically started during booting, use the
.Cm startcrypto
parameter.
.Pp
Note that jails configured to be in the
.Em norun
state (using
.Nm Cm config Fl r Ar norun Ar jailname )
are never started by the ezjail rc script.
.Pp
As a convenient shortcut, the
.Nm
command invokes the rc.d script and passes the corresponding parameters,
if they look like valid parameters.
.Pp
Even if ezjail is not enabled in the
.Xr rc.conf 5 ,
rc.d/ezjail can be used to start and stop jails by prepending
.Cm force No or Cm one No to the Cm start, restart No or Cm stop No parameter.
Refer to
.Xr rc 8
for details.
.Ss Snapshots and retention policies
Jails residing in their own zfs and their corresponding zfs data sets can be
automatically snapshot by the
.Cm ezjail-admin snapshot
subcommand. Taking snapshots of all jails before a major update is considered
best practise. However, when taking snapshots regularly, the amount of disc
space used can be considerable.
.Pp
Therefore ezjail allows you to set retention policies that describe how many
of your snapshots you want to keep for one or all jails or a particular zfs. See
the description of the snapshot command in
.Xr ezjail-admin 5
for details.
.Pp
A retention policy consists of one or multiple windows for which ezjail guarantees
to keep at least one and at most two snapshots. A simple example:
.D1 $ezjail_default_retention_policy="1d 2w 1y"
will ensure ONE snapshot for the last day, for the last two weeks before that day and
then for one snapshot in the year before the two-week window. Valid multipliers are
(m)inutes, (h)ours, (d)ays, (w)eeks and (y)ears.
.Pp
Windows can be repeated by prepending them with a number and the letter x:
.D1 $ezjail_test_com_retention_policy="24x1h 6x1d 3x1w 11x4w KEEP"
will set the retention policy for jail test.com to keep hourly snapshots for one
day, then daily snapshots for the rest of the week, weekly snapshots for the rest of
the month, monthly snapshots for the rest of the year.
.Pp
The magic keyword KEEP at the end of the list will make ezjail not delete snapshots
older than the oldest window. It is your responsibility to keep the list in an order
that makes keeping snapshots possible, i.e. not placing one-hour-windows after
one-year-windows.
.Ss Remarks & Tips
Jails can be either accessed from the network, for instance by using
.Xr ssh 1 ,
or from the host system by using the
.Cm console
command, which gives you an interactive shell inside the jail. It is
also possible to edit the files of a running jail, and the
modifications will appear immediately inside the jail environment.
When dealing image-based, the
.Cm config -i attach
command allows one to access the disk of a file-based jail without starting it.
.Pp
Raw sockets are disallowed by default for all jails. This is not a ezjail
restriction, but a design default of the jail command. This means the
.Xr ping 8
command will get
.Dq Operation not permitted.
error when used from inside of a jail. There are
.Xr sysctl 3
knobs for allowing a jail to access raw sockets, see the
.Xr jail 8
man page for details.
.Pp
Once your jail has network access, then all your normal application
install functions are available, right from the jails console. In
particular, if the ports collection was installed, it can be used as
if from the host system. A modified
.Pa make.conf
file is installed by the example flavour, that enable the ports
collection to work even with a read-only
.Pa /usr/ports .
.Pp
It is possible to change the IP address of a jail by editing its
configuration file in
.Pa EZJAIL_PREFIX/etc/ezjail
and restarting the jail.
.Pp
The jails use the same network stack as the host system. In
particular, that means that if a firewall is needed, it must be
configured in the host system.
.Pp
The ezjail system (and the jails it controls) depends on the
.Dq Li $ezjail_enable
variable being set to
.Dq Li YES
in
.Pa rc.conf .
It is possible to set this variable to
.Dq Li NO
if the administrator wants to temporarily disable ezjail, or if she doesn't
want the jails to be automatically started on boot.
.Pp
The ezjail system may be reset to a pristine state by removing all its
files, that is:
.Bl -item -compact
.It
.Pa /usr/jails/
.It
.Pa EZJAIL_PREFIX/etc/ezjail/
.It
.Pa EZJAIL_PREFIX/etc/ezjail.conf
.It
.Pa /etc/fstab.* No (but check the list of files this matches)
.El
.Sh EXAMPLES
The examples below are only that, examples. The reader is encouraged
to read the
.Xr ezjail-admin 8
man page for definitive documentation of all the options.
.Ss Initial Binary Installation
The ezjail system may be bootstrapped either from binary packages, or
by building from source. The
.Cm install
command allow to bootstrap from binary packages, while the
.Cm update
deals with installations (and updates) from source.
.Bl -tag -width indent
.It Nm Cm install No (without any options)
Fetch and install binaries for populating the base jail from the
FreeBSD FTP server. If the host is not running a -RELEASE version, you
will be asked for the release to install. Neither the man pages nor
the source nor the ports tree are installed. Note that the FreeBSD FTP
server is sometimes so busy the download times out. Use the
.Fl h Ar host
option to specify a less loaded server, or the
.Dq Li $ezjail_ftphost
option in
.Xr ezjail.conf 8 .
.It Nm Cm install Fl ms
Same behavior as above, except that man pages and sources are installed in the
base jail.
.It Nm Cm install Fl p
Same as the first example, but use
.Xr portsnap 8
to fetch and extract a full FreeBSD ports tree from
.Li portsnap.FreeBSD.org
into the base jail. This is necessary if you plan to install ports at later
time into service jails.
.It Nm Cm install Fl P No (note uppercase P)
Only fetch the current version of the ports tree, adding it to the base jail.
This allow to either add the ports tree after the initial installation or update the ports tree in the base jail.
.It Install from a disk image
Mount and use a downloaded
.Pa disc1.iso
CDRom image file.
.Bd -literal -offset indent
mdconfig -a -f /usr/8.0-RELEASE-i386-disc1.iso md0
mount -v -t cd9660 /dev/md0 /mnt
cd /mnt/8.0-RELEASE
ezjail-admin install -h file:// -sm
.Ed
.Pp
When the installation finishes, use the following to release the
.Pa disc1.iso
.Pa md0
file.
.Bd -literal -offset indent
cd /usr
umount /mnt
mdconfig -d -u md0
.Ed
.It Install from a local directory
To fetch the RELEASE base files manually, create a
.Pa .netrc
file in your home directory and populate it with this.
.Bd -literal -offset indent
machine ftp2.jp.FreeBSD.org
login anonymous
password FBSD@home.com
macdef init
prompt off
cd /pub/FreeBSD/releases/i386/8.0-RELEASE
epsv4 off
$ getdir base kernels manpages src
quit
macdef getdir
! mkdir $i
mreget $i/*
.Ed
.Pp
Then issue this command on the command line. If the FTP download
times out re-issue the FTP command again to resume where it left off.
.Bd -literal -offset indent
mkdir /usr/8.0-RELEASE
cd /usr/8.0-RELEASE
ftp -v ftp2.jp.FreeBSD.org
ezjail-admin install -h file:// -sm
.Ed
.Pp
Use this option to target the 8.0-RELEASE files you FTP'ed as the source of
the running binaries used to populate the base jail. In addition the man
pages and sources will be installed into the base jail.
.El
.Ss From Source Installation and Update
The
.Cm update
is used to both install or update from source the base jail, and for
updating the base jail from binary packages.
.Bl -tag -width indent
.It Nm Cm update Fl b
Build and install a world from source. The sources are taken from
.Pa /usr/src
(but see the
.Fl s
flag). This can be used both for creating the initial base jail, and
for updating it after the host has been upgraded.
.It Nm Cm update Fl u
Update the base jail to the next release using
.Xr freebsd-update 8
(i.e. using binary packages). This may be used only to update an
existing installation.
.It Nm Cm update Fl U s Ar 8.0-RELEASE
Upgrade the base jail to the host system's release using
.Xr freebsd-update 8 . This may be used only to upgrade an
existing installation. Tell freebsd-update which OS version to expect
in the basejail via the
.Fl s No option.
.Pp
Note: Check
.Xr uname 1
and especially the
.Pa UNAME_r
environment variable to upgrade to different versions.
.El
.Ss Jail Creation Examples
.Bl -tag -width indent
.It Nm Cm create Ar www.example.com 10.0.10.1
Create a new jail. The jail files will reside in directory
.Pa www_example_com
in
.Pa /usr/jails ,
unless the variable
.Dq Li $ezjail_jaildir
has been set to some other value. The jail will only be allowed to use
the given IP address. A warning will be displayed if this IP address
is not already configured in the host, or if some network daemon is
already listening on this address. The name of the jail which will
appear in the
.Cm list
command or which will need to be given to the
.Cm console
command is
.Ar www.example.com .
.It Nm Cm create Fl f Ar example Fl r Ar webserver www.example.com 10.0.10.2,2001:db8:1:9243::80
Create a new jail, placing it in directory
.Pa webserver
instead of deriving the directory name of the jail from its host name.
The jail will be created with the flavour
.Ar example .
This jail will be given two IP addresses; this is possible only since
FreeBSD 7.2.
.It Nm Cm create Fl i Fl s Ar 600M sandbox2 10.0.10.4
This creates a new file-based jail having a file size of 600 megabytes
in
.Pa /usr/jails/sandbox2.img .
An empty directory,
.Pa /usr/jails/sandbox2 ,
will be created, and used as a mount point when starting the jail.
.It Nm Cm create Fl c Cm bde Fl s Ar 600M sandbox3 10.0.10.5
This creates a new file based image jail, with
.Xr gbde 4
encryption. During the gbde creation process you are asked to enter a
passphrase that is used as the prime seed value of the encryption process.
Remember this passphrase, you will be asked for the passphrase every time
you want to start this jail. As they require administrator interaction,
jails backed by an encrypted file are not automatically started when the
system boots.
.It Nm Cm create Fl c Ar zfs Fl s Ar 1G sandbox4 em1\[rs]|10.0.10.6
This creates a new zfs filesystem based jail with a default quota of 1
gigabyte using lzjb compression. It uses the parent ZFS filesystem configured
in the
.Dq Li $ezjail_jailzfs
variable to create the filesystem in. The jail command will add the ip
address 10.0.10.6 as an alias on the device em1 before starting the jail.
.El
.Sh FILES
.Pa EZJAIL_PREFIX/bin/ezjail-admin
.br
.Pa EZJAIL_PREFIX/etc/rc.d/ezjail
.br
.Pa EZJAIL_PREFIX/etc/ezjail.conf
.br
.Pa EZJAIL_PREFIX/share/examples/ezjail/
.br
.Pa EZJAIL_PREFIX/etc/ezjail/*
.br
.Pa /usr/etc/fstab.*
.Sh SEE ALSO
.Xr ezjail-admin 8 ,
.Xr ezjail.conf 5 ,
.Xr jail 8 ,
.Xr nullfs 4 ,
.Xr zfs 8 .
.Pp
Interesting additional tools include:
.Dq Li ports-mgmt/jailaudit .
.Sh AUTHOR
.An Dirk Engling
.Aq erdgeist@erdgeist.org .
.Pp
The man page is based on a draft by
.An JoeB
.Aq joeb1@a1poweruser.com
and was rewritten by
.An Frederic Perrin
.Aq frederic.perrin@resel.fr .