FreeBSD jails for fun and profit

16th of November, 2020

Docker has stormed into software development in recent years. While its concepts are powerful and valuable, similar tools have been used in systems for decades. FreeBSD's jails in one of those tools which build upon even older chroot(2). To put it shortly, with these tools, you can make a safe environment separated from the rest of the system.

Jails in FreeBSD are by no means a new tool (introduced in 4.x), but for a reason or another, I haven't used them that often, which is a shame since they are so powerful. So I wanted to explore this concept in a concise and summarized manner.

Templates

ZFS datasets are a great way of creating templates for jails since, after the template creation, you can easily create new jails with zfs clone or zfs send/receive. Typically people tend to divide jails to complete and service jails, where the former usually resembles a real FreeBSD system, and the latter is often dedicated to applications/services. I'll cover complete jails for now.

Creating templates starts with creating a dataset for your jail and template. Here I'll make a new dataset for the base installation of FreeBSD 12.2.

$ sudo zfs create -o mountpoint=/vm zroot/vm
$ sudo zfs create zroot/vm/tmpl
$ sudo zfs create zroot/vm/tmpl/12.2

After that, fetch the base installation itself:

$ fetch ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.2-RELEASE/base.txz
# Fetch all the necessary stuff for your template, e.g. lib32 if needed
$ sudo tar -xJvpf base.txz -C /vm/tmpl/12.2

After that, you should write a minimum viable /etc/rc.conf for the template:

$ sudo emacs /vm/tmpl/12.2/etc/rc.conf

# Start or stop services
sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"
syslogd_flags="-ss"
cron_flags="-J 60"

You can also disable some unnecessary jobs for jails:

$ sudo emacs /vm/tmpl/12.2/etc/periodic.conf

# No output for successful script runs.
daily_show_success="NO"
weekly_show_success="NO"
monthly_show_success="NO"
security_show_success="NO"
   
# Output to log files which are rotated by default.
daily_output="/var/log/daily.log"
daily_status_security_output="/var/log/daily.log"
weekly_output="/var/log/weekly.log"
weekly_status_security_output="/var/log/weekly.log"
monthly_output="/var/log/monthly.log"
monthly_status_security_output="/var/log/monthly.log"
   
# No need for those without sendmail
daily_clean_hoststat_enable="NO"
daily_status_mail_rejects_enable="NO"
daily_status_mailq_enable="NO"
daily_queuerun_enable="NO"
   
# Host does those
daily_status_disks_enable="NO"
daily_status_zfs_zpool_list_enable="NO"
daily_status_network_enable="NO"
daily_status_uptime_enable="NO"
daily_ntpd_leapfile_enable="NO"
weekly_locate_enable="NO"
weekly_whatis_enable="NO"
security_status_chksetuid_enable="NO"
security_status_neggrpperm_enable="NO"
security_status_chkuid0_enable="NO"
security_status_ipfwdenied_enable="NO"
security_status_ipfdenied_enable="NO"
security_status_ipfwlimit_enable="NO"
security_status_ipf6denied_enable="NO"
security_status_tcpwrap_enable="NO"

You also might want to enable ports in your jail:

$ sudo mkdir /vm/tmpl/12.2/usr/ports
$ sudo mkdir -p /vm/tmpl/12.2/var/ports/{distfiles,packages}
$ sudo emacs /vm/tmpl/12.2/etc/make.conf

WRKDIRPREFIX = /var/ports
DISTDIR = /var/ports/distfiles
PACKAGES = /var/ports/packages

Apply system updates to the template:

$ sudo freebsd-update -b /vm/tmpl/12.2 fetch install

Lastly, take a snapshot:

Note: Strictly speaking, a template is a snapshot, not a dataset. The snapshot can be cloned or sent/received to generate new datasets for production jails.

$ sudo zfs snapshot zroot/vm/tmpl/12.2@complete

This creates a snapshot of zroot/vm/tmpl/12.2 named complete. You can then check your current snapshots with:

$ sudo zfs list -t snapshot

Creating jails from the template

Now you should create a new jail based on that snapshot. You can do it either with zfs clone or zfs send/receive:

Difference between the two

"A clone is a writable volume or file system whose initial contents are the same as the dataset from which it was created. As with snapshots, creating a clone is nearly instantaneous and initially consumes no additional disk space. In addition, you can snapshot a clone." [1]

"The zfs send command creates a stream representation of a snapshot that is written to standard output. By default, a full stream is generated. You can redirect the output to a file or to a different system. The zfs receive command creates a snapshot whose contents are specified in the stream that is provided on standard input. If a full stream is received, a new file system is created as well. You can send ZFS snapshot data and receive ZFS snapshot data and file systems with these commands. See the examples in the next section." [2]

$ sudo zfs clone zroot/vm/tmpl/12.2@complete zroot/vm/jail1

# OR

$ sudo sh -c "zfs send zroot/vm/tmpl/12.2@complete | zfs receive zroot/vm/jail1"

Jail configurations

# /etc/rc.conf

cloned_interfaces="lo0"

# PF is used for NAT and port forwarding.
pf_enable="YES"
pflog_enable="YES"

jail_enable="YES"
jail_list="jail1"

# /etc/jail.conf

exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;

host.hostname = $name;
path = "/vm/$name";
exec.consolelog = "/var/log/jail_${name}_console.log";
exec.prestart = "cp /etc/resolv.conf $path/etc";
exec.poststop = "rm $path/etc/resolv.conf";

jail1 {
        ip4.addr = "lo0|127.1.1.1/32";
        ip6.addr = "lo0|fd00:1:1:1::1/64";
        allow.chflags;
        allow.raw_sockets;
}

# /etc/hosts

...

127.1.1.1 jail1
fd00:1:1:1::1 jail1

Jail management

FreeBSD provides nifty built-in tools for jail management:

Start all jails.

$ sudo service jail start

Start a specific jail(s).

$ sudo service jail start jail1

Log in to jail.

$ sudo jexec jail1

Run a command on a jail.

$ sudo jexec jail1 ifconfig

List running jails.

$ jls
$ jls -v
$ jls -s

So that's how you can spin up a simple restricted environment on your FreeBSD system. Of course, there are still many things to cover in this topic, e.g., in-depth networking and configurations.

Notes

References

Tags: computers, containers, freebsd, zfs