Fedora Root on ZFS

ZFSBootMenu

This tutorial is based on the GRUB bootloader. Due to its independent implementation of a read-only ZFS driver, GRUB only supports a subset of ZFS features on the boot pool. [In general, bootloader treat disks as read-only to minimize the risk of damaging on-disk data.]

ZFSBootMenu is an alternative bootloader free of such limitations and has support for boot environments. Do not follow instructions on this page if you plan to use ZBM, as the layouts are not compatible. Refer to their site for installation details.

Customization

Unless stated otherwise, it is not recommended to customize system configuration before reboot.

Only use well-tested pool features

You should only use well-tested pool features. Avoid using new features if data integrity is paramount. See, for example, this comment.

Preparation

  1. Disable Secure Boot. ZFS modules can not be loaded if Secure Boot is enabled.

  2. Because the kernel of latest Live CD might be incompatible with ZFS, we will use Alpine Linux Extended, which ships with ZFS by default.

    Download latest extended variant of Alpine Linux live image, verify checksum and boot from it.

    gpg --auto-key-retrieve --keyserver hkps://keyserver.ubuntu.com --verify alpine-extended-*.asc

dd if=input-file of=output-file bs=1M

  3. Login as root user. There is no password.

  4. Configure Internet

    setup-interfaces -r
# You must use "-r" option to start networking services properly
# example:
network interface: wlan0
WiFi name:         <ssid>
ip address:        dhcp
<enter done to finish network config>
manual netconfig:  n

  5. If you are using wireless network and it is not shown, see Alpine Linux wiki for further details. wpa_supplicant can be installed with apk add wpa_supplicant without internet connection.

  6. Configure SSH server

    setup-sshd
# example:
ssh server:        openssh
allow root:        "prohibit-password" or "yes"
ssh key:           "none" or "<public key>"

  7. Set root password or /root/.ssh/authorized_keys.

  8. Connect from another computer

    ssh root@192.168.1.91

  9. Configure NTP client for time synchronization

    setup-ntp busybox

  10. Set up apk-repo. A list of available mirrors is shown. Press space bar to continue

    setup-apkrepos

  11. Throughout this guide, we use predictable disk names generated by udev

    apk update
apk add eudev
setup-devd udev

  12. Target disk

    List available disks with

    find /dev/disk/by-id/

    If virtio is used as disk bus, power off the VM and set serial numbers for disk. For QEMU, use -drive format=raw,file=disk2.img,serial=AaBb. For libvirt, edit domain XML. See this page for examples.

    Declare disk array

    DISK='/dev/disk/by-id/ata-FOO /dev/disk/by-id/nvme-BAR'

    For single disk installation, use

    DISK='/dev/disk/by-id/disk1'

  13. Set a mount point

    MNT=$(mktemp -d)

  14. Set partition size:

    Set swap size in GB, set to 1 if you don’t want swap to take up too much space

    SWAPSIZE=4

    Set how much space should be left at the end of the disk, minimum 1GB

    RESERVE=1

  15. Install ZFS support from live media:

    apk add zfs

  16. Install partition tool

    apk add parted e2fsprogs cryptsetup util-linux

System Installation

  1. Partition the disks.

    Note: you must clear all existing partition tables and data structures from target disks.

    For flash-based storage, this can be done by the blkdiscard command below:

    partition_disk () {
 local disk="${1}"
 blkdiscard -f "${disk}" || true

 parted --script --align=optimal  "${disk}" -- \
 mklabel gpt \
 mkpart EFI 2MiB 1GiB \
 mkpart bpool 1GiB 5GiB \
 mkpart rpool 5GiB -$((SWAPSIZE + RESERVE))GiB \
 mkpart swap  -$((SWAPSIZE + RESERVE))GiB -"${RESERVE}"GiB \
 mkpart BIOS 1MiB 2MiB \
 set 1 esp on \
 set 5 bios_grub on \
 set 5 legacy_boot on

 partprobe "${disk}"
}

for i in ${DISK}; do
   partition_disk "${i}"
done

  2. Setup encrypted swap. This is useful if the available memory is small:

    for i in ${DISK}; do
   cryptsetup open --type plain --key-file /dev/random "${i}"-part4 "${i##*/}"-part4
   mkswap /dev/mapper/"${i##*/}"-part4
   swapon /dev/mapper/"${i##*/}"-part4
done

  3. Load ZFS kernel module

    modprobe zfs

  4. Create boot pool

    # shellcheck disable=SC2046
zpool create -o compatibility=legacy  \
    -o ashift=12 \
    -o autotrim=on \
    -O acltype=posixacl \
    -O canmount=off \
    -O devices=off \
    -O normalization=formD \
    -O relatime=on \
    -O xattr=sa \
    -O mountpoint=/boot \
    -R "${MNT}" \
    bpool \
           mirror \
    $(for i in ${DISK}; do
       printf '%s ' "${i}-part2";
      done)

    If not using a multi-disk setup, remove mirror.

    You should not need to customize any of the options for the boot pool.

    GRUB does not support all of the zpool features. See spa_feature_names in grub-core/fs/zfs/zfs.c. This step creates a separate boot pool for /boot with the features limited to only those that GRUB supports, allowing the root pool to use any/all features.

  5. Create root pool

    # shellcheck disable=SC2046
zpool create \
    -o ashift=12 \
    -o autotrim=on \
    -R "${MNT}" \
    -O acltype=posixacl \
    -O canmount=off \
    -O compression=zstd \
    -O dnodesize=auto \
    -O normalization=formD \
    -O relatime=on \
    -O xattr=sa \
    -O mountpoint=/ \
    rpool \
    mirror \
   $(for i in ${DISK}; do
      printf '%s ' "${i}-part3";
     done)

    If not using a multi-disk setup, remove mirror.

  6. Create root system container:

    • Unencrypted

      zfs create \
 -o canmount=off \
 -o mountpoint=none \
rpool/fedora

    • Encrypted:

      Avoid ZFS send/recv when using native encryption, see `a ZFS developer's comment on this issue`__ and `this spreadsheet of bugs`__. A LUKS-based guide has yet to be written. Once compromised, changing password will not keep your data safe. See zfs-change-key(8) for more info

      zfs create \
  -o canmount=off \
         -o mountpoint=none \
         -o encryption=on \
         -o keylocation=prompt \
         -o keyformat=passphrase \
rpool/fedora

    You can automate this step (insecure) with: echo POOLPASS | zfs create ....

    Create system datasets, manage mountpoints with mountpoint=legacy

    zfs create -o canmount=noauto -o mountpoint=/  rpool/fedora/root
zfs mount rpool/fedora/root
zfs create -o mountpoint=legacy rpool/fedora/home
mkdir "${MNT}"/home
mount -t zfs rpool/fedora/home "${MNT}"/home
zfs create -o mountpoint=legacy  rpool/fedora/var
zfs create -o mountpoint=legacy rpool/fedora/var/lib
zfs create -o mountpoint=legacy rpool/fedora/var/log
zfs create -o mountpoint=none bpool/fedora
zfs create -o mountpoint=legacy bpool/fedora/root
mkdir "${MNT}"/boot
mount -t zfs bpool/fedora/root "${MNT}"/boot
mkdir -p "${MNT}"/var/log
mkdir -p "${MNT}"/var/lib
mount -t zfs rpool/fedora/var/lib "${MNT}"/var/lib
mount -t zfs rpool/fedora/var/log "${MNT}"/var/log

  7. Format and mount ESP

    for i in ${DISK}; do
 mkfs.vfat -n EFI "${i}"-part1
 mkdir -p "${MNT}"/boot/efis/"${i##*/}"-part1
 mount -t vfat -o iocharset=iso8859-1 "${i}"-part1 "${MNT}"/boot/efis/"${i##*/}"-part1
done

mkdir -p "${MNT}"/boot/efi
mount -t vfat -o iocharset=iso8859-1 "$(echo "${DISK}" | sed "s|^ *||"  | cut -f1 -d' '|| true)"-part1 "${MNT}"/boot/efi

System Configuration

  1. Download and extract minimal Fedora root filesystem:

    apk add curl
curl --fail-early --fail -L \
https://dl.fedoraproject.org/pub/fedora/linux/releases/38/Container/x86_64/images/Fedora-Container-Base-38-1.6.x86_64.tar.xz \
-o rootfs.tar.gz
curl --fail-early --fail -L \
https://dl.fedoraproject.org/pub/fedora/linux/releases/38/Container/x86_64/images/Fedora-Container-38-1.6-x86_64-CHECKSUM \
-o checksum

# BusyBox sha256sum treats all lines in the checksum file
# as checksums and requires two spaces "  "
# between filename and checksum

grep 'Container-Base' checksum \
| grep '^SHA256' \
| sed -E 's|.*= ([a-z0-9]*)$|\1  rootfs.tar.gz|' > ./sha256checksum

sha256sum -c ./sha256checksum

rootfs_tar=$(tar t -af rootfs.tar.gz | grep layer.tar)
rootfs_tar_dir=$(dirname "${rootfs_tar}")
tar x -af rootfs.tar.gz "${rootfs_tar}"
ln -s "${MNT}" "${MNT}"/"${rootfs_tar_dir}"
tar x  -C "${MNT}" -af "${rootfs_tar}"
unlink "${MNT}"/"${rootfs_tar_dir}"

  2. Enable community repo

    sed -i '/edge/d' /etc/apk/repositories
sed -i -E 's/#(.*)community/\1community/' /etc/apk/repositories

  3. Generate fstab:

    apk add arch-install-scripts
genfstab -t PARTUUID "${MNT}" \
| grep -v swap \
| sed "s|vfat.*rw|vfat rw,x-systemd.idle-timeout=1min,x-systemd.automount,noauto,nofail|" \
> "${MNT}"/etc/fstab

  4. Chroot

    cp /etc/resolv.conf "${MNT}"/etc/resolv.conf
for i in /dev /proc /sys; do mkdir -p "${MNT}"/"${i}"; mount --rbind "${i}" "${MNT}"/"${i}"; done
chroot "${MNT}" /usr/bin/env DISK="${DISK}" bash

  5. Unset all shell aliases, which can interfere with installation:

    unalias -a

  6. Install base packages

    dnf -y install @core grub2-efi-x64 \
grub2-pc grub2-pc-modules grub2-efi-x64-modules shim-x64  \
efibootmgr kernel kernel-devel

  7. Install ZFS packages

    dnf -y install \
https://zfsonlinux.org/fedora/zfs-release-2-3"$(rpm --eval "%{dist}"||true)".noarch.rpm

dnf -y install zfs zfs-dracut

  8. Check whether ZFS modules are successfully built

    tail -n10 /var/lib/dkms/zfs/**/build/make.log

# ERROR: modpost: GPL-incompatible module zfs.ko uses GPL-only symbol 'bio_start_io_acct'
# ERROR: modpost: GPL-incompatible module zfs.ko uses GPL-only symbol 'bio_end_io_acct_remapped'
# make[4]:  [scripts/Makefile.modpost:138: /var/lib/dkms/zfs/2.1.9/build/module/Module.symvers] Error 1
# make[3]:  [Makefile:1977: modpost] Error 2
# make[3]: Leaving directory '/usr/src/kernels/6.2.9-100.fc36.x86_64'
# make[2]:  [Makefile:55: modules-Linux] Error 2
# make[2]: Leaving directory '/var/lib/dkms/zfs/2.1.9/build/module'
# make[1]:  [Makefile:933: all-recursive] Error 1
# make[1]: Leaving directory '/var/lib/dkms/zfs/2.1.9/build'
# make:  [Makefile:794: all] Error 2

    If the build failed, you need to install an Long Term Support kernel and its headers, then rebuild ZFS module

    # this is a third-party repo!
# you have been warned.
#
# select a kernel from
# https://copr.fedorainfracloud.org/coprs/kwizart/

dnf copr enable -y kwizart/kernel-longterm-VERSION
dnf install -y kernel-longterm kernel-longterm-devel
dnf remove -y kernel-core

    ZFS modules will be built as part of the kernel installation. Check build log again with tail command.

  9. Add zfs modules to dracut

    echo 'add_dracutmodules+=" zfs "' >> /etc/dracut.conf.d/zfs.conf
echo 'force_drivers+=" zfs "' >> /etc/dracut.conf.d/zfs.conf

  10. Add other drivers to dracut:

    if grep mpt3sas /proc/modules; then
  echo 'force_drivers+=" mpt3sas "'  >> /etc/dracut.conf.d/zfs.conf
fi
if grep virtio_blk /proc/modules; then
  echo 'filesystems+=" virtio_blk "' >> /etc/dracut.conf.d/fs.conf
fi

  11. Build initrd

    find -D exec /lib/modules -maxdepth 1 \
-mindepth 1 -type d \
-exec sh -vxc \
'if test -e "$1"/modules.dep;
   then kernel=$(basename "$1");
   dracut --verbose --force --kver "${kernel}";
 fi' sh {} \;

  12. For SELinux, relabel filesystem on reboot:

    fixfiles -F onboot

  13. Enable internet time synchronisation:

    systemctl enable systemd-timesyncd

  14. Generate host id

    zgenhostid -f -o /etc/hostid

  15. Install locale package, example for English locale:

    dnf install -y glibc-minimal-langpack glibc-langpack-en

  16. Set locale, keymap, timezone, hostname

    rm -f /etc/localtime
rm -f /etc/hostname
systemd-firstboot \
--force \
--locale=en_US.UTF-8 \
--timezone=Etc/UTC \
--hostname=testhost \
--keymap=us || true

  17. Set root passwd

    printf 'root:yourpassword' | chpasswd

Bootloader

  1. Apply GRUB workaround

    echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile.d/zpool_vdev_name_path.sh
# shellcheck disable=SC1091
. /etc/profile.d/zpool_vdev_name_path.sh

# GRUB fails to detect rpool name, hard code as "rpool"
sed -i "s|rpool=.*|rpool=rpool|"  /etc/grub.d/10_linux

    This workaround needs to be applied for every GRUB update, as the update will overwrite the changes.

  2. Fedora and RHEL uses Boot Loader Specification module for GRUB, which does not support ZFS. Disable it:

    echo 'GRUB_ENABLE_BLSCFG=false' >> /etc/default/grub

    This means that you need to regenerate GRUB menu and mirror them after every kernel update, otherwise computer will still boot old kernel on reboot.

  3. Install GRUB:

    mkdir -p /boot/efi/fedora/grub-bootdir/i386-pc/
for i in ${DISK}; do
 grub2-install --target=i386-pc --boot-directory \
     /boot/efi/fedora/grub-bootdir/i386-pc/  "${i}"
done
dnf reinstall -y grub2-efi-x64 shim-x64
cp -r /usr/lib/grub/x86_64-efi/ /boot/efi/EFI/fedora/

  4. Generate GRUB menu

    mkdir -p /boot/grub2
grub2-mkconfig -o /boot/grub2/grub.cfg
cp /boot/grub2/grub.cfg \
 /boot/efi/efi/fedora/grub.cfg
cp /boot/grub2/grub.cfg \
 /boot/efi/fedora/grub-bootdir/i386-pc/grub2/grub.cfg

  5. For both legacy and EFI booting: mirror ESP content:

    espdir=$(mktemp -d)
find /boot/efi/ -maxdepth 1 -mindepth 1 -type d -print0 \
| xargs -t -0I '{}' cp -r '{}' "${espdir}"
find "${espdir}" -maxdepth 1 -mindepth 1 -type d -print0 \
| xargs -t -0I '{}' sh -vxc "find /boot/efis/ -maxdepth 1 -mindepth 1 -type d -print0 | xargs -t -0I '[]' cp -r '{}' '[]'"

  6. Exit chroot

    exit

  7. Unmount filesystems and create initial system snapshot You can later create a boot environment from this snapshot. See Root on ZFS maintenance page.

    umount -Rl "${MNT}"
zfs snapshot -r rpool@initial-installation
zfs snapshot -r bpool@initial-installation

  8. Export all pools

    zpool export -a

  9. Reboot

    reboot

  10. For BIOS-legacy boot users only: the GRUB bootloader installed might be unusable. In this case, see Bootloader Recovery section in Root on ZFS maintenance page.

    This issue is not related to Alpine Linux chroot, as Arch Linux installed with this method does not have this issue.

    UEFI bootloader is not affected by this issue.

  11. On first reboot, SELinux policies will be applied, albeit incompletely. The computer will then reboot with incomplete policies and fail to mount /run, resulting in a failure.

    Workaround is to append enforcing=0 to kernel command line in the GRUB menu, as many times as necessary, until the system complete one successful boot. The author of this guide has not found out a way to solve this issue during installation. Help is appreciated.

Post installaion

  1. Install package groups

    dnf group list --hidden -v       # query package groups
dnf group install gnome-desktop

  2. Add new user, configure swap.