Rocky Linux Root on ZFS
ZFSBootMenu
This tutorial is based on the GRUB bootloader. Due to its independent implementation of a read-only ZFS driver, GRUB only supports a subset of ZFS features on the boot pool. [In general, bootloader treat disks as read-only to minimize the risk of damaging on-disk data.]
ZFSBootMenu is an alternative bootloader free of such limitations and has support for boot environments. Do not follow instructions on this page if you plan to use ZBM, as the layouts are not compatible. Refer to their site for installation details.
Customization
Unless stated otherwise, it is not recommended to customize system configuration before reboot.
Only use well-tested pool features
You should only use well-tested pool features. Avoid using new features if data integrity is paramount. See, for example, this comment.
Preparation
Disable Secure Boot. ZFS modules can not be loaded if Secure Boot is enabled.
Because the kernel of latest Live CD might be incompatible with ZFS, we will use Alpine Linux Extended, which ships with ZFS by default.
Download latest extended variant of Alpine Linux live image, verify checksum and boot from it.
gpg --auto-key-retrieve --keyserver hkps://keyserver.ubuntu.com --verify alpine-extended-*.asc dd if=input-file of=output-file bs=1M
Login as root user. There is no password.
Configure Internet
setup-interfaces -r # You must use "-r" option to start networking services properly # example: network interface: wlan0 WiFi name: <ssid> ip address: dhcp <enter done to finish network config> manual netconfig: n
If you are using wireless network and it is not shown, see Alpine Linux wiki for further details.
wpa_supplicant
can be installed withapk add wpa_supplicant
without internet connection.Configure SSH server
setup-sshd # example: ssh server: openssh allow root: "prohibit-password" or "yes" ssh key: "none" or "<public key>"
Set root password or
/root/.ssh/authorized_keys
.Connect from another computer
ssh root@192.168.1.91
Configure NTP client for time synchronization
setup-ntp busybox
Set up apk-repo. A list of available mirrors is shown. Press space bar to continue
setup-apkrepos
Throughout this guide, we use predictable disk names generated by udev
apk update apk add eudev setup-devd udev
Target disk
List available disks with
find /dev/disk/by-id/
If virtio is used as disk bus, power off the VM and set serial numbers for disk. For QEMU, use
-drive format=raw,file=disk2.img,serial=AaBb
. For libvirt, edit domain XML. See this page for examples.Declare disk array
DISK='/dev/disk/by-id/ata-FOO /dev/disk/by-id/nvme-BAR'
For single disk installation, use
DISK='/dev/disk/by-id/disk1'
Set a mount point
MNT=$(mktemp -d)
Set partition size:
Set swap size in GB, set to 1 if you don’t want swap to take up too much space
SWAPSIZE=4
Set how much space should be left at the end of the disk, minimum 1GB
RESERVE=1
Install ZFS support from live media:
apk add zfs
Install partition tool
apk add parted e2fsprogs cryptsetup util-linux
System Installation
Partition the disks.
Note: you must clear all existing partition tables and data structures from target disks.
For flash-based storage, this can be done by the blkdiscard command below:
partition_disk () { local disk="${1}" blkdiscard -f "${disk}" || true parted --script --align=optimal "${disk}" -- \ mklabel gpt \ mkpart EFI 2MiB 1GiB \ mkpart bpool 1GiB 5GiB \ mkpart rpool 5GiB -$((SWAPSIZE + RESERVE))GiB \ mkpart swap -$((SWAPSIZE + RESERVE))GiB -"${RESERVE}"GiB \ mkpart BIOS 1MiB 2MiB \ set 1 esp on \ set 5 bios_grub on \ set 5 legacy_boot on partprobe "${disk}" } for i in ${DISK}; do partition_disk "${i}" done
Setup encrypted swap. This is useful if the available memory is small:
for i in ${DISK}; do cryptsetup open --type plain --key-file /dev/random "${i}"-part4 "${i##*/}"-part4 mkswap /dev/mapper/"${i##*/}"-part4 swapon /dev/mapper/"${i##*/}"-part4 done
Load ZFS kernel module
modprobe zfs
Create boot pool
# shellcheck disable=SC2046 zpool create -o compatibility=legacy \ -o ashift=12 \ -o autotrim=on \ -O acltype=posixacl \ -O canmount=off \ -O devices=off \ -O normalization=formD \ -O relatime=on \ -O xattr=sa \ -O mountpoint=/boot \ -R "${MNT}" \ bpool \ mirror \ $(for i in ${DISK}; do printf '%s ' "${i}-part2"; done)
If not using a multi-disk setup, remove
mirror
.You should not need to customize any of the options for the boot pool.
GRUB does not support all of the zpool features. See
spa_feature_names
in grub-core/fs/zfs/zfs.c. This step creates a separate boot pool for/boot
with the features limited to only those that GRUB supports, allowing the root pool to use any/all features.Create root pool
# shellcheck disable=SC2046 zpool create \ -o ashift=12 \ -o autotrim=on \ -R "${MNT}" \ -O acltype=posixacl \ -O canmount=off \ -O compression=zstd \ -O dnodesize=auto \ -O normalization=formD \ -O relatime=on \ -O xattr=sa \ -O mountpoint=/ \ rpool \ mirror \ $(for i in ${DISK}; do printf '%s ' "${i}-part3"; done)
If not using a multi-disk setup, remove
mirror
.Create root system container:
Unencrypted
zfs create \ -o canmount=off \ -o mountpoint=none \ rpool/rhel
Encrypted:
Avoid ZFS send/recv when using native encryption, see `a ZFS developer's comment on this issue`__ and `this spreadsheet of bugs`__. A LUKS-based guide has yet to be written. Once compromised, changing password will not keep your data safe. See
zfs-change-key(8)
for more infozfs create \ -o canmount=off \ -o mountpoint=none \ -o encryption=on \ -o keylocation=prompt \ -o keyformat=passphrase \ rpool/rhel
You can automate this step (insecure) with:
echo POOLPASS | zfs create ...
.Create system datasets, manage mountpoints with
mountpoint=legacy
zfs create -o canmount=noauto -o mountpoint=/ rpool/rhel/root zfs mount rpool/rhel/root zfs create -o mountpoint=legacy rpool/rhel/home mkdir "${MNT}"/home mount -t zfs rpool/rhel/home "${MNT}"/home zfs create -o mountpoint=legacy rpool/rhel/var zfs create -o mountpoint=legacy rpool/rhel/var/lib zfs create -o mountpoint=legacy rpool/rhel/var/log zfs create -o mountpoint=none bpool/rhel zfs create -o mountpoint=legacy bpool/rhel/root mkdir "${MNT}"/boot mount -t zfs bpool/rhel/root "${MNT}"/boot mkdir -p "${MNT}"/var/log mkdir -p "${MNT}"/var/lib mount -t zfs rpool/rhel/var/lib "${MNT}"/var/lib mount -t zfs rpool/rhel/var/log "${MNT}"/var/log
Format and mount ESP
for i in ${DISK}; do mkfs.vfat -n EFI "${i}"-part1 mkdir -p "${MNT}"/boot/efis/"${i##*/}"-part1 mount -t vfat -o iocharset=iso8859-1 "${i}"-part1 "${MNT}"/boot/efis/"${i##*/}"-part1 done mkdir -p "${MNT}"/boot/efi mount -t vfat -o iocharset=iso8859-1 "$(echo "${DISK}" | sed "s|^ *||" | cut -f1 -d' '|| true)"-part1 "${MNT}"/boot/efi
System Configuration
Download and extract minimal Rhel root filesystem:
apk add curl curl --fail-early --fail -L \ https://dl.rockylinux.org/pub/rocky/9.2/images/x86_64/Rocky-9-Container-Base-9.2-20230513.0.x86_64.tar.xz \ -o rootfs.tar.gz curl --fail-early --fail -L \ https://dl.rockylinux.org/pub/rocky/9.2/images/x86_64/Rocky-9-Container-Base-9.2-20230513.0.x86_64.tar.xz.CHECKSUM \ -o checksum # BusyBox sha256sum treats all lines in the checksum file # as checksums and requires two spaces " " # between filename and checksum grep 'Container-Base' checksum \ | grep '^SHA256' \ | sed -E 's|.*= ([a-z0-9]*)$|\1 rootfs.tar.gz|' > ./sha256checksum sha256sum -c ./sha256checksum tar x -C "${MNT}" -af rootfs.tar.gz
Enable community repo
sed -i '/edge/d' /etc/apk/repositories sed -i -E 's/#(.*)community/\1community/' /etc/apk/repositories
Generate fstab:
apk add arch-install-scripts genfstab -t PARTUUID "${MNT}" \ | grep -v swap \ | sed "s|vfat.*rw|vfat rw,x-systemd.idle-timeout=1min,x-systemd.automount,noauto,nofail|" \ > "${MNT}"/etc/fstab
Chroot
cp /etc/resolv.conf "${MNT}"/etc/resolv.conf for i in /dev /proc /sys; do mkdir -p "${MNT}"/"${i}"; mount --rbind "${i}" "${MNT}"/"${i}"; done chroot "${MNT}" /usr/bin/env DISK="${DISK}" bash
Unset all shell aliases, which can interfere with installation:
unalias -a
Install base packages
dnf -y install --allowerasing @core grub2-efi-x64 \ grub2-pc grub2-pc-modules grub2-efi-x64-modules shim-x64 \ efibootmgr kernel-core
Install ZFS packages:
dnf install -y https://zfsonlinux.org/epel/zfs-release-2-3"$(rpm --eval "%{dist}"|| true)".noarch.rpm dnf config-manager --disable zfs dnf config-manager --enable zfs-kmod dnf install -y zfs zfs-dracut
Add zfs modules to dracut:
echo 'add_dracutmodules+=" zfs "' >> /etc/dracut.conf.d/zfs.conf echo 'force_drivers+=" zfs "' >> /etc/dracut.conf.d/zfs.conf
Add other drivers to dracut:
if grep mpt3sas /proc/modules; then echo 'force_drivers+=" mpt3sas "' >> /etc/dracut.conf.d/zfs.conf fi if grep virtio_blk /proc/modules; then echo 'filesystems+=" virtio_blk "' >> /etc/dracut.conf.d/fs.conf fi
Build initrd:
find -D exec /lib/modules -maxdepth 1 \ -mindepth 1 -type d \ -exec sh -vxc \ 'if test -e "$1"/modules.dep; then kernel=$(basename "$1"); dracut --verbose --force --kver "${kernel}"; fi' sh {} \;
For SELinux, relabel filesystem on reboot:
fixfiles -F onboot
Generate host id:
zgenhostid -f -o /etc/hostid
Install locale package, example for English locale:
dnf install -y glibc-minimal-langpack glibc-langpack-en
Set locale, keymap, timezone, hostname
rm -f /etc/localtime systemd-firstboot \ --force \ --locale=en_US.UTF-8 \ --timezone=Etc/UTC \ --hostname=testhost \ --keymap=us
Set root passwd
printf 'root:yourpassword' | chpasswd
Bootloader
Apply GRUB workaround
echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile.d/zpool_vdev_name_path.sh # shellcheck disable=SC1091 . /etc/profile.d/zpool_vdev_name_path.sh # GRUB fails to detect rpool name, hard code as "rpool" sed -i "s|rpool=.*|rpool=rpool|" /etc/grub.d/10_linux
This workaround needs to be applied for every GRUB update, as the update will overwrite the changes.
RHEL uses Boot Loader Specification module for GRUB, which does not support ZFS. Disable it:
echo 'GRUB_ENABLE_BLSCFG=false' >> /etc/default/grub
This means that you need to regenerate GRUB menu and mirror them after every kernel update, otherwise computer will still boot old kernel on reboot.
Install GRUB:
mkdir -p /boot/efi/rocky/grub-bootdir/i386-pc/ for i in ${DISK}; do grub2-install --target=i386-pc --boot-directory \ /boot/efi/rocky/grub-bootdir/i386-pc/ "${i}" done dnf reinstall -y grub2-efi-x64 shim-x64 cp -r /usr/lib/grub/x86_64-efi/ /boot/efi/EFI/rocky/
Generate GRUB menu:
mkdir -p /boot/grub2 grub2-mkconfig -o /boot/grub2/grub.cfg cp /boot/grub2/grub.cfg \ /boot/efi/efi/rocky/grub.cfg cp /boot/grub2/grub.cfg \ /boot/efi/rocky/grub-bootdir/i386-pc/grub2/grub.cfg
For both legacy and EFI booting: mirror ESP content:
espdir=$(mktemp -d) find /boot/efi/ -maxdepth 1 -mindepth 1 -type d -print0 \ | xargs -t -0I '{}' cp -r '{}' "${espdir}" find "${espdir}" -maxdepth 1 -mindepth 1 -type d -print0 \ | xargs -t -0I '{}' sh -vxc "find /boot/efis/ -maxdepth 1 -mindepth 1 -type d -print0 | xargs -t -0I '[]' cp -r '{}' '[]'"
Exit chroot
exit
Unmount filesystems and create initial system snapshot You can later create a boot environment from this snapshot. See Root on ZFS maintenance page.
umount -Rl "${MNT}" zfs snapshot -r rpool@initial-installation zfs snapshot -r bpool@initial-installation
Export all pools
zpool export -a
Reboot
reboot
For BIOS-legacy boot users only: the GRUB bootloader installed might be unusable. In this case, see Bootloader Recovery section in Root on ZFS maintenance page.
This issue is not related to Alpine Linux chroot, as Arch Linux installed with this method does not have this issue.
UEFI bootloader is not affected by this issue.
Post installaion
Install package groups
dnf group list --hidden -v # query package groups dnf group install gnome-desktop
Add new user, configure swap.