ezjail – Jail administration framework

News

Overview

A Jail in FreeBSD-speak is one or more tasks with the same kernel Jail-ID, bound on zero or more IP addresses, having the same chroot-environment. One usecase of the FreeBSD Jail Subsystem is to provide virtual FreeBSD-systems within a Host-system. ezjail is about making this as easy as possible, aiming for minimum system resource usage. All further references to the term Jail are to a virtual FreeBSD-system consisting of a host name, an IP-address and a Jail root.

The jail(8) man page outlines the way to create Jails, however, when you need several Jails, complete Jail Directory Trees quickly use much of your valuable hard disc space. ezjail avoids this by using FreeBSDs nullfs feature. Most of the base system (/bin, /boot, /sbin, /lib, /libexec, /rescue, /usr/{bin, include, lib, libexec, ports, sbin, share, src}) only exists in one copy in the Host-system and is being mounted read only into all Jails via nullfs. Those Jails are quite slim (around 2mb each) and only consist of some soft links into the basejail mount point and non-shared directories like /etc, /usr/local, etc.

The ezjail approach offers lots of advantages:

• You save disc space, inodes and even memory since the system only needs to hold one copy of base system binaries for all Jails

• You can update all Jails on a single base directory, since it is so eazy, you might actually end up doing it

• Intruders compromising Jails are unable to install standard rootkits (as the base system is mounted read only)

• Since ezjail is written entirely in sh, there is no need to install other script languages into the hostsystem

• As the base system is provided via soft links, the enjailed users can choose not to use the mounted world

• ezjail offers full zfs integration and can help you automatize your file system configuration

• An often underestimated fact: less complexity means more security.

Quick start

To set up your first very simple ezjail, just install ezjail from sysutils/ezjail port or via pkg_add -r ezjail and enable it by setting ezjail_enable=YES in your in your /etc/rc.conf. Assuming your network interface is em0, just type (as root):

ezjail-admin install
ezjail-admin create example.com 'em0|10.0.0.2'
ezjail-admin start example.com

and you're done. Get a shell in your new jail with the:

ezjail-admin console example.com

command. As with any vanilla FreeBSD installation, you might probably need to touch /etc/ and maybe copy your host's /etc/resolv.conf.

Slow start

ezjail comes with some sane defaults, but can be configured globally and per jail using the config file /usr/local/etc/ezjail.conf (copy the sample from /usr/local/etc/ezjail.conf.sample) and the per-jail config files under /usr/local/etc/ezjail/ (those are created automatically with the jails and managed by the ezjail-admin config command).

ezjail-admin create -f example example.com 'em0|10.0.0.2'

ezjail-admin console -f example.com

ezjail-admin install -r 2.2.8-RELEASE

ezjail-admin setup -i

ezjail-admin setup -b

ezjail-admin update -u

ezjail-admin create -i -s 2G example.com 10.0.0.2

ezjail-admin config -i attach example.com
cd /usr/jails/example.com
# … do your thing …
ezjail-admin config -i detach example.com

ezjail-admin config -i fsck example.com

ezjail-admin create -c eli -i 16G example.com 10.0.0.3

Version History

Since ezjail is useful under FreeBSD only, you might want to install it from ports sysutils/ezjail. An ezjail cvsweb (deprecated) and ezjail gitweb are available, plus there is a fresh checkout. The most current version is available via cvs. Use git clone git://erdgeist.org/ezjail or the legacy view cvs -d :pserver:anoncvs@cvs.erdgeist.org:/home/cvsroot co ezjail with an empty passwort to check it out. Just type sudo make install to install it. Older versions may be found here (checksums in tooltips):

• CURRENT - not packaged yet

• V3.4.2 ezjail-3.4.2.tar.bz2 (2015-11-23)

  • Fixed a bug where ezjail-admin install would fail if the lib32 package is not there.

  • Add an experimental freeze feature, where you can automatically create flavours by letting ezjail inspect the changes in a jail's zfs.

  • Fix a bug in date-string creation (reported by kaltheat).

  • If a zfs jail is forcefully destroyed, also destroy its underlying file system, if there are errors.

• V3.4.1 ezjail-3.4.1.tar.bz2 (2014-06-10)

  • Fixed a fatal bug in unfinished parts of the new troubleshoot subcommand.

• V3.4 ezjail-3.4.tar.bz2 (2014-06-10)

  • Began a troubleshoot subcommand that will in future check for unintuitive reasons for the jails to fail.

  • Fixed a bug where Jails configured .norun were reported failing to start with the ezjail-admin start subcommand. Thanks to John Grasty.

  • Create /var/ports/packages and /var/ports/distfiles in new jails to reflect custom /etc/make.conf.

  • The Makefile now supports staging.

  • Implement a test parameter for the ezjail-admin config -r subcommand that reports the runnable state. This can also be used to test for an ezjail's existence.

  • Fixed a bug where ezjail-admin would not find jails with jid > 99999. Thanks to Mathieu Arnold.

  • Fixed a bug where ezjail-admin delete and ezjail-admin config would not find the zpool the jail was created in.

  • Implemented a jail snapshot mechanism for zfs jails with configurable retention policy.

  • Fixed a bug where ezjail-admin restore with a path to an archive would not use the correct jail name and thus fail. Thanks to Ollivier Robert.

• V3.3 ezjail-3.3.tar.bz2 (2013-04-20)

  • Due to recent circumstances, I added a license file.

  • Properly handle ip addresses with prefixed interface names, so they appear readable in the jail list and they will not be checked for on ezjail-admin create. Since FreeBSD 9.0 you can configure ip addresses with interface prefix (e.g. em1|10.0.0.1). This feature of jail(8) is now also documented in the ezjail's man pages.

  • Using CVS to update ports was deprecated for a while. FreeBSD does not offer the anoncvs server, anymore. So ezjail_portscvsroot is gone.

  • Added a -z option to ezjail-admin create to allow creating zfs jails in a different zpool or under a different zfs parent file system than the one ezjail uses for house keeping. Thanks to Cryx.

  • Add the a man page authors section to the man pages.

  • Also read and dump jail_JAIL_parameters when changing jail options. Thanks to Виталий Владимирович.

  • ezjail-admin update and ezjail-admin upgrade now call freebsd-update install until no more update steps are necessary. Previously upgrades were not installed completely, as freebsd-update uses several passes.

  • Now properly support the new distributions layout.

  • Do not run chflags -R noschg when deleting image jails. The whole file system is being thrown away, anyway. Thanks to tomster.

• V3.2.3 ezjail-3.2.3.tar.bz2 (2012-09-23)

  • Fixed a bug that made ezjail-admin update not find the source tree. Thanks to Paul Macdonald.

  • Fixed ezjail-admin console to not start all jails when the -f option is requested.

• V3.2.2 ezjail-3.2.2.tar.bz2 (2012-09-10)

  • Jails can now default to being created in zfs, set ezjail_use_zfs_for_jails="YES" in your ezjail.conf to do that.

  • Cleaned up the zfs code to ensure the ezjail zfs in a central place.

• V3.2.1 ezjail-3.2.1.tar.bz2 (2012-02-02)

  • Fixed ezjail-admin list to work properly without jails.

  • A helper script ezjail-clone.sh has been added to the repository that aids in importing whole server setups into an ezjail-system.

  • ezjail-admin update now ensures that all files with schg set are also updated. Thanks to Geoffroy Desvernay.

  • Fixed the way ezjail-admin create tests for local IP addresses. The old test using ping with a given TTL did not work reliable. Now we use ping and try to use the IP address as source address.

  • Fixed a bug where pids not belonging to a Jail were reported as jailed. Thanks to Thomas Steen Rasmussen.

  • Fixed a bug in ezjail-admin delete -f where all Jails were stopped, not only the one to be deleted. Tanks to Thomas Steen Rasmussen.

  • Fixed a bug in ezjail-admin create that overwrote all flavours scripts with the successor by not counting the flavours correctly. Thanks to Geoffroy Desvernay.

• V3.2 ezjail-3.2.tar.bz2 (2012-01-07)

  • The man pages have been completely rewritten and are much easier to understand now.

  • Default archive location now is ezjail_archives under ezjail_jaildir. Before it defaulted to the current directory.

  • ezjail-admin install now understands FreeBSD's new naming scheme on the release ftp servers.

  • /usr/games/ has gone.

  • ezjail-admin delete now takes an -f parameter that forces stopping the Jail before deleting it

  • The rc.d/ezjail.sh script has been renamed to rc.d/ezjail in accordance to FreeBSD guide lines.

  • When using ezjail-admin to start Jails, checks now are properly extended to all versions of the (one|force|...) prefixes.

  • When installing the basejail from a server, we now also look in the ftp server's Archive directory.

  • The check for already listening services has been augmented to support multiple IPs and v6 addresses.

  • Before removing Jails the system immutable flag is now cleared.

  • ezjail-admin now collects its Jail list more to the standard.

  • Unknow Jail image types now default to directory based.

  • Experimental freebsd-upgrade support.

  • Fix one misspelling that wrongly reported success for attaching a console to a Jail not running.

  • The ezjail rc script now correctly also re-attaches images on restart.

  • There now is a Jail name auto completion script for zsh.

• V3.1 ezjail-3.1.tar.bz2 (2010-02-26)

  • Introduced support for creating ezjails and the basejail in ZFS file systems.

  • Attaching ZFS-datasets to ezjails is now supported.

  • Support for cpusets and altered network view for ezjails.

  • The ezjail-admin archive command now uses the cpio format for its archives.

  • Since FreeBSD 8 does not support rc.d-scripts ending in .sh, the flavourizing script has been renamed accordingly.

  • The ezjail-admin list command now supports multiple IP addresses for a jail.

  • The ezjail-admin command now returns 0 if no error occurs.

  • You can now specify a default flavour in your ezjail.conf that is applied to any new Jail.

  • Renamed the default flavour to example to avoid confusion. default flavour was never applied by default.

  • ezjail-admin create and ezjail-admin config no longer fail if they can't create a soft link in ezjail_jaildir. They now use mktemp to get a unique one.

  • ezjail-admin create now allow multiple flavours to be applied to new jails. They are executed in the order provided with the -f option.

• V3.0 ezjail-3.0.tar.bz2 (2008-09-26)

  • Introduced an ezjail-admin restore command that can be used to easily deploy a backed up version of an ezjail's archive.

  • Introduced an ezjail-admin archive command that – accompanied by the newly introduced create -a option – can be used to take (automated) backups of your jails and reinstall them.

  • Introduced the ezjail-admin console sub command. This will jexec into a Jail identified by its Jail name. It can also bring up non-running jails if run with the -f switch.

  • ezjail-admin update does not (re)build the whole world by default, anymore. You have to provide -b (buildworld, installworld), -i (installworld only) or (as a new option) -u to use freebsd-update to update your base jail.

  • ezjail-admin now silently proxies everthing that looks like it was meant for ezjail.sh over to ezjail.sh.

  • ezjail-admin now reports its version number along with synopsis information.

  • ezjail.sh now stops ezjails in reverse startup order.

  • ezjail.sh now reports ezjails configured norun as "skipping [jailname]" and blocking as "skipping blocking [jailname]".

  • ezjail-admin now find .s its stuff. Using '*' never found dot-files and could have led to a "too many parameters" situation.

  • Fixed a bug where image jails would not be run with ezjail.sh restart when not running or would not have their images attached when already running. Thanks to hukl for bug report.

  • Fixed the missing lib32 packag on amd64 platforms when doing ezjail-admin install.

  • sudo now checks permissions on its sudoers file and stops, if it is not 0440. Fixed ezjails Makefile to install sudoers to default flavour. (However, if you already installed ezjail, you might want to manually chmod 0440 ${ezjail_flavours}/default/usr/local/etc/sudoers and all corresponding sudoers files in all jails.)

  • Fixed a bug, where an image Jail was incorrectly reported not attached.

• V2.1. ezjail-2.1.tar.bz2 (2007-07-31)

  • Introduced a ezjail-admin config -n newname sub command to enable renaming jails.

  • Introduced a ezjail-admin config -i fsck sub command to fsck images.

  • Use the sendmail_submit_enable, sendmail_outbound_enable and sendmail_msp_queue_enable in the default-flavour rc.conf to allow easier fine-grained control for sendmail.

  • ezjail-admin create now ignores .dot-files in directories meant to be Jail roots. That way directories coming fresh from newfs won't be mistaken as used.

  • Fixed a bug where portsnap would be called with fetch rather than cron from my ezjail-admin update -P.

  • Enable soft updates for image jails.

  • Now taking copies of soft linked packages inside jails. They would not be visible.

  • Fail, when trying to install or update ezjails world, if in a secure level > 0. Too many bad things happen in secure levels.

  • ezjail.sh now also checks for root directory to prevent strange effects in /etc/rc.d/jail.

  • Default flavours /etc/periodic.conf now makes all periodic scripts log to files instead of sending mails. Most often this is what you want in your jails.

  • Started learning groff and went over man pages. Not many visible changes here.

• V2.0.1 ezjail-2.0.1.tar.bz2 (2006-07-24)

  • DESTDIR was ignored for sub targets in an ezjail-admin update -p under certain circumstances. See this report for more details. Thanks go to Simon L. Nielsen.

  • Fixed some qoutation/expansion issues with tr parameters in the form [:alnum:] which were expanded to the files : a l n u and m, if existing. Also an apparent bug in tr was circumvented which incorrectly replaced A-Z to a-z in certain LANG settings. Thanks agains to Simon L. Nielsen.

  • ezjail_ftphost was ignored when specified in PREFIX/etc/ezjail.conf. Thanks to Edwin Groothuis.

• V2.0 ezjail-2.0.tar.bz2 (2006-05-31)

  • The default way to create the basejail will become the newly introduced ezjail-admin install subcommand. In its default configuration the base system is fetched from an ftp-Server. Fetching ports, source, man pages is possible with -p, -s and -m switch. Implementing this feature finally brings the promised ease of use and allows: cd /usr/ports/sysutils/ezjail; make install; ezjail-admin install; ezjail-admin create test.com 10.1.1.1 on a vanilla FreeBSD server, i.e. without installing /usr/src.

  • A new concept of how to manage your Jails has been introduced: Image Jails. They come in plain, gbde and geli encrypted. ezjail-admin create, ezjail-admin delete and ezjail.sh have been updated to provide the functionality needed.

  • ezjail.sh now provides more information to rcorder which took control over the execution of new style localpkg scripts. These changes in FreeBSD rc scripts led to ezjail.sh not being started under certain circumstances. It will REQUIRE: sshd, making administration on a server running many jails a lot more peaceful. Thanks to Oli.

  • When called without specific Jail names (e.g. at system start), ezjail.sh now runs rcorder on its config file list located in PREFIX/etc/ezjail/ before processing them. That way you can manually construct a dependency tree and for example start your name server Jail first and your data base-Jail before your web server Jail. However, some basic knowledge in bourne shell and the rcorder command is required. Beware of circles.

  • ezjail-admin create now installs flavours with an unconditional flag. Previously files were not copied, if an "older" version existed. Thanks to Matthias Lederhofer.

  • ezjail-admin list has been heavily extended. It now lists Jails in the same order they are being run on system startup. This is to help debugging the dependency tree. It further now also lists jails status (Directory based, Image, BDE crypto image or ELI crypto image) and running state (Running, Attached, Stopped or Norun). Thanks to Bas Dakkenhorst for useful diffs.

  • ezjail-admin create now prepends all Jail configs with empty rcorder control statements to make altering the config easier.

  • ezjail.sh now excludes Jail configs whose file name contain '.' (dots). This way it is possible to configure a Jail config test_org.norun to temporary exclude it from auto run process.

  • ezjail introduces an option ezjail_uglyperlhack (defaults to YES) which creates a soft link from /usr/bin/perl to /usr/local/bin/perl in the base jail. This is useful since perl was expelled from base system in FreeBSD 5 but still many scripts rely on #!/usr/bin/perl.

  • /usr/games and /usr/libdata have now become part of the basejail. Someone please kick me for missing that until now. Saves another 100k in newjail. Hopefully I got them all by now ;) Thanks to Vivek Khera.

• V1.3.1 ezjail-1.3.1.tar.bz2 (2006-03-13)

  • ezjail.sh did not ignore the jail_list from rc.conf as it should due to a legacy name space clash. This led to no ezjail being started at system startup when some non-ezjail-Jails exist in the system. Thanks to Cryx.

  • ezjail-admin now prevents a Jail called fulljail to be created and refines the error message in case the user wants to create basejail, newjail or flavours.

  • ezjail-admin now again allows recreating Jails with the -x option. This was broken when introducing the Jail root check in V 1.3. Thanks to Mike Harding.

  • An experimental script ezjail-release.sh has been added to CVS which allows creating the basejail from FreeBSD ftp-Server. However, this script has known issues, like updating existing basejail's ports does not remove obsolete files, also there is no documentation.

• V1.3 ezjail-1.3.tar.bz2 (2006-02-16)

  • Removed an ugly shell variable quoting bug that led to ezjail.sh fail for multiple jails. Thanks to Alex Moura. PLEASE UPDATE FROM 1.2 as fast as possible.

  • ezjail-admin create now checks, whether a given Jail root already exists and refuses to install. Thanks to Mark Bucciarelli for finding this one.

  • /boot and /rescue have now become part of the basejail. Someone please kick me for missing that until now. New jails now are < 2 MB in size.

• V1.2 ezjail-1.2.tar.bz2 (2006-02-10)

  • A subsystem providing auto configuration on Jail has been created. You may specify flavours at Jail creation. See section Flavours.

  • ezjail-admin update now allows cvs co or cvs up of ports in the basejail by providing the -p or -P option.

  • /lib and /libexec have now become part of the basejail. Someone please kick me for missing that until now.

  • ezjail-admin now allows setup as an alias command to update to reduce confusion.

  • ezjail-admin delete now ensures, that the Jail is not running. Deleting running Jails led to panics in several occasions.

  • ezjail-admin update now performs several checks to make sure the source tree is really there and complete.

  • ezjail-admin create now tests for services listening on the new Jails IP or on all IPs and warns if it finds any.

  • ezjail-admin create now replaces any character in Jail name not suitable for shell variables by underscore. This leads to the situation that two jails must not have identical names after them being piped through tr -c [:alnum:] _, thanks go to Gunjin.

  • A default installation PREFIX is provided now. Installing to / seems not too useful and should not be default.

  • Documentation has been rewritten and simplified a lot.

• V1.1 ezjail-1.1.tar.bz2 (2005-10-26)

  • minor typo in ezjail.sh, changed #/bin/sh to #!/bin/sh, thanks go to Alex Samorukov.

  • Makefile ensures that all directories are there when installing. Helps with odd $PREFIXes

  • ezjail-admin now creates $basejail/usr itself, since cpio creates missing intermediate directories in 0700 which is bad

  • ezjail-admin now verifies the existence of its basejail before creating Jails

• V1.0 ezjail-1.0.tar.bz2 (2005-10-14)

  • first release

  • made FreeBSD port for ezjail

  • actually implemented ezjail-admin delete feature

  • documentation completed

  • lots of bug fixes I can't remember anymore, however V0.1 has never been used by anyone but me

• V0.1 ezjail-0.1.tar.bz2 (2005-09-26)

  • highly experimental version

  • first real world testing on a server system with >20 jails

• V0.0 - Not packaged (2005-09-04)

  • first alpha version

FAQ

Q: I have created a Jail but it won't come up when I run /usr/local/etc/rc.d/ezjail start JAILNAME

A: Please check, that the Jail is really "not comming up" by examining the output of the command jls. Unlike the command jail which leaves you with an interactive shell in your Jail, rc.d/ezjail is an rc-script. It is most commonly meant to run at system startup and shutdown where entering an interactive shell would interrupt the boot process. If you want an interactive shell in your Jail after starting it, use the command ezjail-admin console JAILNAME.

A: Please check, that you have enabled ezjail by adding the line ezjail_enable="YES" to rc.conf.

A: Please check, that you have enabled ezjail BEFORE you ran the rc.d/ezjail script the first time. See the rc.d introduction for further details.

A: If this is the case, check the console output in /var/log/jail_JAILNAME_console.log. Most commonly you'll find reasons for your jails startup failure there.

Q: Why does ezjail want an extra copy of ports tree for its base jail? Isn't that just waste of disc space? Can't we just use the base systems ports tree?

A: Counter question: why do we need an extra copy of the base system? Isn't that just waste of disc space? Can't we just mount / as basejail to all jails?

ezjail philosophy very much is influenced by its history: it was created to simplify administration of a shell server with many actual users. Even if you would trust all your users, you may not trust all software they do install and hence not anything done in any Jail. So I decided to strictly seperate what you can access within Jails from everything the host system needs to operate. This boundary is ezjail_root.

For the ports case I would not like everyone to know which port is installed in host system, as might be guessed by work/-directories, this might reveal attack vectors to the host system. A problem like ignoring ro flags when mounting from fstab may give you an idea why I consider that boundary essential to system security.

If you know what you're doing and are sure, your host system has an WRKDIRPREFIX none of the Jails can access, you might follow these instructions to mount ports via nullfs. Another way of doing this is to mv /usr/ports/ /usr/jails/basejail/usr/ports/; ln -s /usr/jails/basejail/usr/ports /usr/ports; in host system. ezjail will not offer this as an option.

Q: What exactly is the difference between the template Jail and a Flavour?

A: In simple words: THE template is all directories that FreeBSD thinks your system needs besides the sharable files in your basejail, like /etc/ and /var/. It is being recreated everytime you ezjail-admin update. So it is not meant to be modified by the user to distribute settings to all Jails to be created. A Flavour on the other hand is set up by the user, contains all non-sharable files not in the base system that you think belongs to a newly created Jail (usually stuff in /usr/local/ and your settings in /etc/). You may have several Flavours each customized for a certain job: an httpd-Flavour, a shell-server only Flavour, a full featured bamp-server and so on. Flavours are never modified by ezjail, the template should not be modified by the user.

Q: I am running version x of ezjail and want to upgrade to version y. Is there a safe upgrade path without losing all of my Jails?

A: ezjail is basically just two scripts. The config files have been designed to be future compatible. So installing any version from ports, cvs or tar ball by ezjails Makefile will do the job of upgrading. Your Jails are safe.

Q: I have created some jails prior to using ezjail. How can I import them as ezjails?

A: There now is an import helper script in the ezjail tar bundle that assist you with moving complete systems into ezjails by turning them into ezjail-admin restore-able ezjail archives.

A: Assume your Jail is under /legacyjail Basically you only need to create an empty directory /legacyjail/basejail/ and call ezjail-admin create -x -r /legacyjail JAILNAME JAILIP and your Jail enjoys all of ezjails starting/stopping capabilities..

To make the Jail a complete ezjail, e.g. use the basejail/, use the following code snippet after stopping the Jail. (Replace /legacyjail with the path to your Jail).

cd /legacyjail
mkdir OLDBASE
for dir in bin boot lib libexec rescue sbin usr/bin usr/include usr/lib usr/libexec usr/ports usr/sbin usr/src usr/share; do
  mv ${dir} OLDBASE/; ln -s /basejail/${dir} ${dir}
done

There is a chance that your binaries require some of the libraries in its old base system. If they don't, you can remove OLDBASE, if they do, use the following snippet to create the old libs.

Q: The clock inside my Jails always shows the wrong time. Why is that?

A: Create a soft link from /etc/localtime to the appropriate time zone file, in my case: /usr/share/zoneinfo/Europe/Berlin.

Q: I messed up the whole show and ezjails files are around EVERYWHERE. Some of them wont even go away, if I rm -rf them as root. How do I get a clean start?

A: On a standard FreeBSD installation several files are being installed with the "system immutable" flag set. Type man chflags for more details. However, to reset everything to a state with no jails and no files lurking around use the following lines, on kern secure level 0, as root, on a standard ezjail.conf

chflags -R noschg /usr/jails/
rm -rf /usr/jails/
rm -rf /usr/local/etc/ezjail/
rm /etc/fstab.*

If you configured ezjail to use zfs, you can just throw away ezjail's parent zfs, as configured in ezjail_jailzfs.

Q: Your HTML sucks.

A: This is not a question. But yeah, I know.

Author/Contact

ezjail was written by Dirk Engling with lot of support from the ezjail community. We like to hear from happy customers. My personal address is erdgeist@erdgeist.org. Requests should go to the project mailing list ezjail@erdgeist.org. You can subscribe to that list at ezjail-subscribe@erdgeist.org. There is an IRC channel #ezjail on ircnet. Please send bug reports, comments and feature requests. Donations are welcome… in form of beer. See…

License

ezjail is considered beer ware.

Thanks

Here I need to re-add a modernized list of thanks.