This guide describes the process of installing Fedora with Root on ZFS while booting using ZFSBootMenu and rEFInd. It has been adapted from the Fedora Root on ZFS guide and the Ubuntu Server ZFSBootMenu install script.

Prerequisites:

• A Fedora 35 live image on a flash drive or disc to boot from (don't use a respin)

• Internet access for the Live Environment

• EFI with Secure Boot disabled

• A single drive to install onto; advanced setups (mirrors, raidz1) are possible, but not covered here

Part 1 - Setting up the Live Environment

Boot into the Live Environment, selecting "Try Fedora" instead of using the visual installer, and open a terminal. Ensure we're running as root:

sudo su -

To allow this guide to be more flexible and future-proof, referencing of direct version numbers is avoided where possible. Therefore, we load the current release info variables:

source /etc/os-release

Remove ZFS Fuse

ZFS on FUSE is deprecated and unsupported. Instead we'll install the full ZFS kernel module after removing ZFS on FUSE:

dnf remove -y zfs-fuse

Note: Depending on the live image, zfs-fuse may not be installed in the first place. This is fine, since we wanted to remove it anyway.

Temporarily Disable SE Linux

Disable SE Linux in the Live Environment (this does not apply to the Installed Environment).

setenforce 0

Install kernel-devel

In order to build and install the ZFS module in the Live Environment, we first need to install the kernel-devel package, containing headers and makefiles needed to build modules for our kernel version:

dnf install -y https://dl.fedoraproject.org/pub/fedora/linux/releases/${VERSION_ID}/Everything/x86_64/os/Packages/k/kernel-devel-$(uname -r).rpm

Add ZFS repo and install ZFS

Now we add the repository provided by ZFS on Linux for our version of Fedora and install the ZFS package. This can take a bit of time, since the module must be compiled first:

dnf install -y https://zfsonlinux.org/fedora/zfs-release.fc${VERSION_ID}.noarch.rpm
dnf install -y zfs

Load the ZFS kernel module

Once the module is compiled and installed, we simply need to load it into the kernel:

modprobe zfs

Install helper tools

These tools and scripts will be useful for us later on:

dnf install -y arch-install-scripts gdisk dosfstools

Part 2 - Partition and ZFS pool layout

Identify the install drive from /dev/disk/by-id. Make certain that this is correct, since everything on the drive will be wiped. Also specify the desired ZFS pool name:

# Replace this with the Disk ID of the Installed Environment.
DISK=/dev/disk/by-id/ata-Crucial_CT500MX200SSD1
# Replace this with the pool name of the Installed Environment.
POOL=super_cool_pool

Wipe the drive

To begin from a clean slate, first ensure that all partition tables have been wiped:

sgdisk --zap-all "$DISK"

If the drive is an SSD, we should issue TRIM instructions the entire drive:

blkdiscard -f "$DISK"

Partition the drive

We start off with the EFI partition, sized at 512MiB. A larger partition doesn't hurt, but this is more than enough for rEFInd and ZFSBootMenu.

sgdisk -n1:1M:+512M -t1:EF00 "$DISK"

Create the swap partition. There's a lot of factors that go into how much swap to allocate, but we won't go into that here. For this guide, 8GiB is used. More may be desired to allow for hibernation, but this can be difficult to setup with ZFS on root. Additionally, this swap partition is optional - Fedora enables zram by default, which is in-memory compressed swap.

sgdisk -n2:0:+8G -t2:8200 "$DISK"

Lastly we create the ZFS partition, which spans the rest of the drive:

sgdisk -n3:0:0 -t3:BF00 "$DISK"

We'll save the partition paths for later use. We use PARTUUID for the EFI and ZFS partition. This can't be used for encrypted swap, since the encryption clobbers the partition header. Instead, we have to use the disk path.

EFI_PART=/dev/disk/by-partuuid/$(blkid -s PARTUUID -o value "$DISK"-part1)
SWAP_PART="$DISK"-part2
ZFS_PART=/dev/disk/by-partuuid/$(blkid -s PARTUUID -o value "$DISK"-part3)

Create the ZFS pool

When creating the pool, we set -R /mnt - until exported, /mnt will be treated as the root for this pool. Enter the pool passphrase when prompted:

zpool create \
    -o ashift=12 \
    -o autotrim=on \
    -O acltype=posixacl \
    -O canmount=off \
    -O compression=lz4 \
    -O dnodesize=auto \
    -O normalization=formD \
    -O utf8only=on \
    -O relatime=on \
    -O xattr=sa \
    -O encryption=aes-256-gcm \
    -O keyformat=passphrase \
    -O keylocation=prompt \
    -R /mnt \
    $POOL \
    $ZFS_PART

The options specfied have the following effects.

• ashift=12 - recommended for modern drives, since they typically have 4KiB physical sectors
• autotrim=on - Lets ZFS issue TRIM commands to the drive automatically
• acltype=posixacl - Lets us use ACLs
• canmount=off - Don't allow the pool itself to ever be mounted
• compression=lz4 - LZ4 is very fast and compresses well
• dnodesize=auto - Allows ZFS to manage dnodesize itself
• normalization=formD and utf8only=on - Requires the filenames to be UTF-8, but will cause compatibility issues with non-UTF compatible filenames
• relatime=on - Decreases number of writes updating file access time (this is the default for most filesystems)
• xattr=sa - Improves performance of extended attributes, but reduces compatibility with older ZFS versions
• encryption=aes-256-gcm - Encrypts the pool with 256-bit AES
• keyformat=passphrase - The key must be a passphrase to decrypt at boot
• keylocation=prompt - Prompt for the key - this will change later on

Create ZFS Datasets

We're creating two sets of datasets - the first set contains the OS and other non-portable data. The second contains user data, such as /home. This lets you rollback your OS while keeping your files and other data. This can also let you share data between different OS installations on the same computer, say Ubuntu and Fedora.

System Datasets

Create the parent dataset for Fedora - this can be called anything, but it'll just be fedora here. This dataset will act simply as a container for its children datasets, so we don't want it to be mountable. The ZFSBootMenu options are inherited, allowing you to have the same command line for all children datasets; these specify the kernel command line and how the root path is passed to the kernel. Fedora can use "root=zfs:".

zfs create \
    -o canmount=off \
    -o mountpoint=none \
    -o org.zfsbootmenu:rootprefix="root=zfs:" \
    -o org.zfsbootmenu:commandline="ro quiet" \
    $POOL/fedora

Then we create the dataset for the system itself. This one mounts at /, but not automatically, letting ZFSBootMenu to choose which system to mount at boot. We'll call it initial for simplicity, and mount it manually.

zfs create \
    -o canmount=noauto \
    -o mountpoint=/ \
    $POOL/fedora/initial

zfs mount $POOL/fedora/initial

User Datasets

Separating out user datasets allows us to snapshot, rollback, and backup user data independently from system data. Which directories to treat as user data is somewhat subjective. For the purposes of this guide, we'll setup /home/ and /var/log. These datasets will live under $POOL/data.

zfs create \
    -o canmount=off \
    -o mountpoint=none \
    $POOL/data

The home directory has a mountpoint of /home, and may automount. Datasets can also be created for individual home directories, but we'll look at that later in the install. Home directories of users will automatically inherit the mountpoint, so a dataset named pool/data/home/ociaw will mount at /home/ociaw.

zfs create \
    -o canmount=on \
    -o mountpoint=/home \
    $POOL/data/home

/var/log is similar, except that the dataset we create for /var isn't mountable, since system data is also stored in various folders. The mountpoint is still inherited, however.

zfs create \
    -o canmount=off \
    -o mountpoint=/var \
    $POOL/data/var

zfs create \
    -o canmount=on \
    $POOL/data/var/log

Other user datasets may be optionally created - an incomplete list of candidates is below.

EFI System Partition (ESP)

Now that we've created our ZFS pool, the last partition to setup is the ESP. We'll format it here.

mkfs.vfat -n EFI $EFI_PART
mkdir -p /mnt/boot/efi/
mount -t vfat $EFI_PART /mnt/boot/efi

Part 3 - Boot Setup

With all our partitions set up, we'll install rEFInd and ZFSBootMenu onto the ESP.

Install rEFInd

Download and verify the refind rpm package:

curl -o /root/refind.rpm -L https://sourceforge.net/projects/refind/files/0.13.2/refind-0.13.2-1.x86_64.rpm/download
sha256sum /root/refind.rpm

(NOTE: As of 2021-11-15, this points the latest version of rEFInd, but newer versions can be used as well.)

The produced hash should match the following:

56e6befe1fb2a302c0355831ea9bc46dd8771d8f5f78d848ecd37ce3fcf62432

Install refind:

dnf install -y /root/refind.rpm

You can verify that the EFI entry has been added correctly via efibootmgr:

efibootmgr -v

Note: The UEFI may not recognize rEFInd as a bootable EFI executable. If this happens you may need to rename and move it, for example from /EFI/refind/refind_x64.efi to /EFI/BOOT/boot_x64.efi.

Install ZFSBootMenu

ZFSBootMenu provides prebuilt initramfs and kernel images, but also can be built from source. For simplicity, we'll use the prebuilt images, downloaded from GitHub:

curl -o /root/zfsbootmenu.tar.gz -L https://github.com/zbm-dev/zfsbootmenu/releases/download/v1.12.0/zfsbootmenu-release-x86_64-v1.12.0.tar.gz

sha256sum /root/zfsbootmenu.tar.gz

(NOTE: As of 2022-03-29, this points the latest version of ZBM, but newer versions can be used as well.)

The produced hash should match the following:

67cd4d4febb965d277665442f893653fdb2cb9a3a144627786abc0b713548b16

Make a directory on the ESP for the ZFSBootMenu kernel and initramfs to live:

mkdir -p /boot/efi/EFI/zbm

Extract the kernel and initramfs into the directory:

tar -xf /root/zfsbootmenu.tar.gz -C /boot/efi/EFI/zbm --strip=1 --no-same-owner

Create the rEFInd configuration file:

echo "\"Boot default\"  \"zfsbootmenu:POOL=$POOL zbm.import_policy=hostid zbm.set_hostid zbm.timeout=5 ro quiet loglevel=0\"" >> /boot/efi/EFI/zbm/refind_linux.conf
echo "\"Boot to menu\"  \"zfsbootmenu:POOL=$POOL zbm.import_policy=hostid zbm.set_hostid zbm.show ro quiet loglevel=0\"" >> /boot/efi/EFI/zbm/refind_linux.conf

The two options allow you to choose to either go straight into booting, or to show the menu instead.

Part 4 - Installing and Configuring Fedora

We can boot, but now we need an operating system to boot into. We'll install the base packages and ZFS first:

dnf --installroot=/mnt --releasever=${VERSION_ID} -y install \
    https://zfsonlinux.org/fedora/zfs-release.fc${VERSION_ID}.noarch.rpm \
    @core kernel kernel-devel python3-dnf-plugin-post-transaction-actions \
    zfs zfs-dracut

fstab

Now we want to setup the mountpoints for the installation.

Begin by adding the entry to mount the EFI partition:

echo "$EFI_PART /boot/efi vfat defaults 0 0" >> /mnt/etc/fstab

Encrypted Swap

Encrypted swap is a bit more complicated to setup, since we'll use crypttab as well. We create an encrypted swap device named "swap", with a key randomly generated on each boot. This is then referenced in fstab as /dev/mapper/swap.

echo "swap $SWAP_PART /dev/urandom plain,swap,cipher=aes-xts-plain64:sha256,size=256,discard" >> /mnt/etc/crypttab

echo "/dev/mapper/swap none swap x-systemd.requires=cryptsetup.target,defaults 0 0" >> /mnt/etc/fstab

More information on encrypted swap is available on the Arch Linux wiki.

Configuring Dracut

We configure dracut to include the pool key into the generated initramfs. This removes the need to input the pool password twice upon boot, but remains secure since the initramfs is encrypted as well.

echo "install_items+=\" /etc/zfs/$POOL.key \"" >> /mnt/etc/dracut.conf.d/zfs.conf
echo 'add_dracutmodules+=" zfs "' >> /mnt/etc/dracut.conf.d/zfs.conf

Now create /mnt/etc/zfs/$POOL.key and set the contents to your ZFS pool passphrase. Ensure the permissions are 600:

nano /mnt/etc/zfs/$POOL.key
chmod 600 /mnt/etc/zfs/$POOL.key

Misc. Setup

Enable timezone synchronization:

hwclock --systohc
systemctl enable systemd-timesyncd --root=/mnt

Interactively set locale, keymap, timezone, and hostname. The root password is a placeholder and is set later on.

rm -f /mnt/etc/localtime
systemd-firstboot --root=/mnt --force --prompt --root-password=PASSWORD

Generate host id:

zgenhostid -f -o /mnt/etc/hostid

Install the locale packages:

dnf --installroot=/mnt install -y glibc-minimal-langpack glibc-langpack-en

ZFS Services

We enable all the ZFS services, including zfs-mount:

systemctl enable zfs-import-scan.service zfs-import.target zfs-zed zfs.target zfs-mount --root=/mnt

SSH and Firewall

Optionally the ssh server can be disabled and the firewall enabled:

systemctl disable sshd --root=/mnt
systemctl enable firewalld --root=/mnt

Part 5 - chroot

We need to chroot into the new installation for the rest of the process. We'll chroot into /mnt and refresh variables since we're in a new shell:

arch-chroot /mnt bash --login

# Replace this with the pool name of the Installed Environment.
POOL=super_cool_pool

More OS configuration and installation

Fix SELinux security contexts on the next boot:

fixfiles -F onboot

Set the root password:

passwd

Build the ZFS kernel modules:

ls -1 /lib/modules \
| while read kernel_version; do
    dkms autoinstall -k $kernel_version
    done

Rebuild initramfs with Dracut

Fedora installs an initramfs by default, but it doesn't include the ZFS module or the pool passphrase. We need to use dracut to rebuild the initramfs with these included:

ls -1 /lib/modules \
| while read kernel_version; do
    dracut /boot/initramfs-$kernel_version.img $kernel_version --force
    done

Update pool key location

Since the key is now located in the initramfs, we can tell ZFS where to find it:

zfs change-key -o keylocation=file:///etc/zfs/$POOL.key -o keyformat=passphrase $POOL

Set pool cachefile

This step isn't immediately necessary, but helps to avoid headaches and problems down the line. Set the cachefile property for the pool so that the file is created and contains the pool:

zpool set cachefile=/etc/zfs/zpool.cache $POOL

If a new pool is imported on the system in the future, ZFS may automatically create the zpool.cache file. This doesn't cause a problem on its own, however if the cachefile exists, the zpool-import service that runs on boot will use that cache instead of scanning for pools. If the main pool is not in the cachefile, it won't be found and you will fail to boot.

Clean up and reboot

Exit the chroot:

exit

Unmount the ESP:

umount /mnt/boot/efi

Recursively snapshot the system and user datasets, then export the pool:

zfs snapshot -r $POOL/fedora@before-first-boot
zfs snapshot -r $POOL/data@before-first-boot
zpool export $POOL

Finally, reboot:

reboot

Important Note: If the UEFI isn't recognizing refind as a bootable EFI executable, you may need to rename it.

Part 6 - Finishing up

Upon reboot, rEFInd should find and boot ZFSBootMenu on ESP. ZFSBootMenu then loads the pool, and prompts for the passphrase to unlock it. Once unlocked, ZFSBootMenu loads and boots the kernel found. Assuming everything is setup correctly, you'll get a login prompt - log in as root with the password you specified earlier.

Create a user

You'll likely want to add a non-root user. You'll want to create a ZFS dataset for the user's home directory, create the user, and give the user the ability to mount, take snapshots of, and destroy snapshots of their home directory. Afterwards, permissions and SE linux contexts need to be fixed, and the user must be given a password.

myUser=UserName
homeDataset=$(df --output=source /home | tail -n +2)
zfs create $homeDataset/${myUser}
useradd -MUd /home/${myUser} -c 'My Name' ${myUser}
zfs allow -u ${myUser} mount,snapshot,destroy $homeDataset/${myUser}
chown -R ${myUser}:${myUser} /home/${myUser}
chmod 700 /home/${myUser}
restorecon /home/${myUser}
passwd ${myUser}

Install package groups

Lastly, you can install package groups to easily install things like desktop environments. Common package groups include

• "Fedora Workstation" - for the standard GNOME environment
• "KDE Plasma Workspaces" - for the KDE environment
• "Web Server" - for a web server

All package groups can be listed with dnf group list. Once you've decided on a group, it can be installed with dnf group install, e.g.

dnf group install 'Fedora Workstation'

Automatic Snapshots

By now you should have a working system. You may want to take advantage of ZFS's snapshots, and setup automatic snapshot taking. This can be accomplished through software such as sanoid, pyznap, or even ordinary cronjobs.

Appendix - Common Non-System Directories

# Home directories
/home

# Directory for root
/root

/srv

# Locally installed or compiled software, separate from the system
/usr/local

# Games
/var/games

# Server files
/var/www

# Snap packages
/var/snap

# User crontabs, mail, other things
/var/spool

# Logs - useful for diagnosing issues while being able to
# rollback a broken system
/var/log

# Gnome user data - may cause issues
/var/lib/AccountsService

# Docker containers
/var/lib/docker

# NFS configuration
/var/lib/nfs

# LXC containers
/var/lib/lxc

# libvirt VM images
/var/lib/libvirt