Arch Linux Root on ZFS
ZFSBootMenu
This tutorial is based on the GRUB bootloader. Due to its independent implementation of a read-only ZFS driver, GRUB only supports a subset of ZFS features on the boot pool. [In general, bootloader treat disks as read-only to minimize the risk of damaging on-disk data.]
ZFSBootMenu is an alternative bootloader free of such limitations and has support for boot environments. Do not follow instructions on this page if you plan to use ZBM, as the layouts are not compatible. Refer to their site for installation details.
Customization
Unless stated otherwise, it is not recommended to customize system configuration before reboot.
Only use well-tested pool features
You should only use well-tested pool features. Avoid using new features if data integrity is paramount. See, for example, this comment.
Preparation
Disable Secure Boot. ZFS modules can not be loaded if Secure Boot is enabled.
Because the kernel of latest Live CD might be incompatible with ZFS, we will use Alpine Linux Extended, which ships with ZFS by default.
Download latest extended variant of Alpine Linux live image, verify checksum and boot from it.
gpg --auto-key-retrieve --keyserver hkps://keyserver.ubuntu.com --verify alpine-extended-*.asc dd if=input-file of=output-file bs=1M
Login as root user. There is no password.
Configure Internet
setup-interfaces -r # You must use "-r" option to start networking services properly # example: network interface: wlan0 WiFi name: <ssid> ip address: dhcp <enter done to finish network config> manual netconfig: n
If you are using wireless network and it is not shown, see Alpine Linux wiki for further details.
wpa_supplicant
can be installed withapk add wpa_supplicant
without internet connection.Configure SSH server
setup-sshd # example: ssh server: openssh allow root: "prohibit-password" or "yes" ssh key: "none" or "<public key>"
Set root password or
/root/.ssh/authorized_keys
.Connect from another computer
ssh root@192.168.1.91
Configure NTP client for time synchronization
setup-ntp busybox
Set up apk-repo. A list of available mirrors is shown. Press space bar to continue
setup-apkrepos
Throughout this guide, we use predictable disk names generated by udev
apk update apk add eudev setup-devd udev
Target disk
List available disks with
find /dev/disk/by-id/
If virtio is used as disk bus, power off the VM and set serial numbers for disk. For QEMU, use
-drive format=raw,file=disk2.img,serial=AaBb
. For libvirt, edit domain XML. See this page for examples.Declare disk array
DISK='/dev/disk/by-id/ata-FOO /dev/disk/by-id/nvme-BAR'
For single disk installation, use
DISK='/dev/disk/by-id/disk1'
Set a mount point
MNT=$(mktemp -d)
Set partition size:
Set swap size in GB, set to 1 if you don’t want swap to take up too much space
SWAPSIZE=4
Set how much space should be left at the end of the disk, minimum 1GB
RESERVE=1
Install ZFS support from live media:
apk add zfs
Install partition tool
apk add parted e2fsprogs cryptsetup util-linux
System Installation
Partition the disks.
Note: you must clear all existing partition tables and data structures from target disks.
For flash-based storage, this can be done by the blkdiscard command below:
partition_disk () { local disk="${1}" blkdiscard -f "${disk}" || true parted --script --align=optimal "${disk}" -- \ mklabel gpt \ mkpart EFI 2MiB 1GiB \ mkpart bpool 1GiB 5GiB \ mkpart rpool 5GiB -$((SWAPSIZE + RESERVE))GiB \ mkpart swap -$((SWAPSIZE + RESERVE))GiB -"${RESERVE}"GiB \ mkpart BIOS 1MiB 2MiB \ set 1 esp on \ set 5 bios_grub on \ set 5 legacy_boot on partprobe "${disk}" } for i in ${DISK}; do partition_disk "${i}" done
Setup encrypted swap. This is useful if the available memory is small:
for i in ${DISK}; do cryptsetup open --type plain --key-file /dev/random "${i}"-part4 "${i##*/}"-part4 mkswap /dev/mapper/"${i##*/}"-part4 swapon /dev/mapper/"${i##*/}"-part4 done
Load ZFS kernel module
modprobe zfs
Create boot pool
# shellcheck disable=SC2046 zpool create -o compatibility=legacy \ -o ashift=12 \ -o autotrim=on \ -O acltype=posixacl \ -O canmount=off \ -O devices=off \ -O normalization=formD \ -O relatime=on \ -O xattr=sa \ -O mountpoint=/boot \ -R "${MNT}" \ bpool \ mirror \ $(for i in ${DISK}; do printf '%s ' "${i}-part2"; done)
If not using a multi-disk setup, remove
mirror
.You should not need to customize any of the options for the boot pool.
GRUB does not support all of the zpool features. See
spa_feature_names
in grub-core/fs/zfs/zfs.c. This step creates a separate boot pool for/boot
with the features limited to only those that GRUB supports, allowing the root pool to use any/all features.Create root pool
# shellcheck disable=SC2046 zpool create \ -o ashift=12 \ -o autotrim=on \ -R "${MNT}" \ -O acltype=posixacl \ -O canmount=off \ -O compression=zstd \ -O dnodesize=auto \ -O normalization=formD \ -O relatime=on \ -O xattr=sa \ -O mountpoint=/ \ rpool \ mirror \ $(for i in ${DISK}; do printf '%s ' "${i}-part3"; done)
If not using a multi-disk setup, remove
mirror
.Create root system container:
Unencrypted
zfs create \ -o canmount=off \ -o mountpoint=none \ rpool/archlinux
Encrypted:
Avoid ZFS send/recv when using native encryption, see `a ZFS developer's comment on this issue`__ and `this spreadsheet of bugs`__. A LUKS-based guide has yet to be written. Once compromised, changing password will not keep your data safe. See
zfs-change-key(8)
for more infozfs create \ -o canmount=off \ -o mountpoint=none \ -o encryption=on \ -o keylocation=prompt \ -o keyformat=passphrase \ rpool/archlinux
You can automate this step (insecure) with:
echo POOLPASS | zfs create ...
.Create system datasets, manage mountpoints with
mountpoint=legacy
zfs create -o canmount=noauto -o mountpoint=/ rpool/archlinux/root zfs mount rpool/archlinux/root zfs create -o mountpoint=legacy rpool/archlinux/home mkdir "${MNT}"/home mount -t zfs rpool/archlinux/home "${MNT}"/home zfs create -o mountpoint=legacy rpool/archlinux/var zfs create -o mountpoint=legacy rpool/archlinux/var/lib zfs create -o mountpoint=legacy rpool/archlinux/var/log zfs create -o mountpoint=none bpool/archlinux zfs create -o mountpoint=legacy bpool/archlinux/root mkdir "${MNT}"/boot mount -t zfs bpool/archlinux/root "${MNT}"/boot mkdir -p "${MNT}"/var/log mkdir -p "${MNT}"/var/lib mount -t zfs rpool/archlinux/var/lib "${MNT}"/var/lib mount -t zfs rpool/archlinux/var/log "${MNT}"/var/log
Format and mount ESP
for i in ${DISK}; do mkfs.vfat -n EFI "${i}"-part1 mkdir -p "${MNT}"/boot/efis/"${i##*/}"-part1 mount -t vfat -o iocharset=iso8859-1 "${i}"-part1 "${MNT}"/boot/efis/"${i##*/}"-part1 done mkdir -p "${MNT}"/boot/efi mount -t vfat -o iocharset=iso8859-1 "$(echo "${DISK}" | sed "s|^ *||" | cut -f1 -d' '|| true)"-part1 "${MNT}"/boot/efi
System Configuration
Download and extract minimal Arch Linux root filesystem:
apk add curl curl --fail-early --fail -L \ https://america.archive.pkgbuild.com/iso/2023.09.01/archlinux-bootstrap-x86_64.tar.gz \ -o rootfs.tar.gz curl --fail-early --fail -L \ https://america.archive.pkgbuild.com/iso/2023.09.01/archlinux-bootstrap-x86_64.tar.gz.sig \ -o rootfs.tar.gz.sig apk add gnupg gpg --auto-key-retrieve --keyserver hkps://keyserver.ubuntu.com --verify rootfs.tar.gz.sig ln -s "${MNT}" "${MNT}"/root.x86_64 tar x -C "${MNT}" -af rootfs.tar.gz root.x86_64
Enable community repo
sed -i '/edge/d' /etc/apk/repositories sed -i -E 's/#(.*)community/\1community/' /etc/apk/repositories
Generate fstab:
apk add arch-install-scripts genfstab -t PARTUUID "${MNT}" \ | grep -v swap \ | sed "s|vfat.*rw|vfat rw,x-systemd.idle-timeout=1min,x-systemd.automount,noauto,nofail|" \ > "${MNT}"/etc/fstab
Chroot
cp /etc/resolv.conf "${MNT}"/etc/resolv.conf for i in /dev /proc /sys; do mkdir -p "${MNT}"/"${i}"; mount --rbind "${i}" "${MNT}"/"${i}"; done chroot "${MNT}" /usr/bin/env DISK="${DISK}" bash
Add archzfs repo to pacman config
pacman-key --init pacman-key --refresh-keys pacman-key --populate curl --fail-early --fail -L https://archzfs.com/archzfs.gpg \ | pacman-key -a - --gpgdir /etc/pacman.d/gnupg pacman-key \ --lsign-key \ --gpgdir /etc/pacman.d/gnupg \ DDF7DB817396A49B2A2723F7403BD972F75D9D76 tee -a /etc/pacman.d/mirrorlist-archzfs <<- 'EOF' ## See https://github.com/archzfs/archzfs/wiki ## France #,Server = https://archzfs.com/$repo/$arch ## Germany #,Server = https://mirror.sum7.eu/archlinux/archzfs/$repo/$arch #,Server = https://mirror.biocrafting.net/archlinux/archzfs/$repo/$arch ## India #,Server = https://mirror.in.themindsmaze.com/archzfs/$repo/$arch ## United States #,Server = https://zxcvfdsa.com/archzfs/$repo/$arch EOF tee -a /etc/pacman.conf <<- 'EOF' #[archzfs-testing] #Include = /etc/pacman.d/mirrorlist-archzfs #,[archzfs] #,Include = /etc/pacman.d/mirrorlist-archzfs EOF # this #, prefix is a workaround for ci/cd tests # remove them sed -i 's|#,||' /etc/pacman.d/mirrorlist-archzfs sed -i 's|#,||' /etc/pacman.conf sed -i 's|^#||' /etc/pacman.d/mirrorlist
Install base packages:
pacman -Sy pacman -S --noconfirm mg mandoc grub efibootmgr mkinitcpio kernel_compatible_with_zfs="$(pacman -Si zfs-linux \ | grep 'Depends On' \ | sed "s|.*linux=||" \ | awk '{ print $1 }')" pacman -U --noconfirm https://america.archive.pkgbuild.com/packages/l/linux/linux-"${kernel_compatible_with_zfs}"-x86_64.pkg.tar.zst
Install zfs packages:
pacman -S --noconfirm zfs-linux zfs-utils
Configure mkinitcpio:
sed -i 's|filesystems|zfs filesystems|' /etc/mkinitcpio.conf mkinitcpio -P
For physical machine, install firmware
pacman -S linux-firmware intel-ucode amd-ucode
Enable internet time synchronisation:
systemctl enable systemd-timesyncd
Generate host id:
zgenhostid -f -o /etc/hostid
Generate locales:
echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen locale-gen
Set locale, keymap, timezone, hostname
rm -f /etc/localtime systemd-firstboot \ --force \ --locale=en_US.UTF-8 \ --timezone=Etc/UTC \ --hostname=testhost \ --keymap=us
Set root passwd
printf 'root:yourpassword' | chpasswd
Bootloader
Apply GRUB workaround
echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile.d/zpool_vdev_name_path.sh # shellcheck disable=SC1091 . /etc/profile.d/zpool_vdev_name_path.sh # GRUB fails to detect rpool name, hard code as "rpool" sed -i "s|rpool=.*|rpool=rpool|" /etc/grub.d/10_linux
This workaround needs to be applied for every GRUB update, as the update will overwrite the changes.
Install GRUB:
mkdir -p /boot/efi/archlinux/grub-bootdir/i386-pc/ mkdir -p /boot/efi/archlinux/grub-bootdir/x86_64-efi/ for i in ${DISK}; do grub-install --target=i386-pc --boot-directory \ /boot/efi/archlinux/grub-bootdir/i386-pc/ "${i}" done grub-install --target x86_64-efi --boot-directory \ /boot/efi/archlinux/grub-bootdir/x86_64-efi/ --efi-directory \ /boot/efi --bootloader-id archlinux --removable if test -d /sys/firmware/efi/efivars/; then grub-install --target x86_64-efi --boot-directory \ /boot/efi/archlinux/grub-bootdir/x86_64-efi/ --efi-directory \ /boot/efi --bootloader-id archlinux fi
Import both bpool and rpool at boot:
echo 'GRUB_CMDLINE_LINUX="zfs_import_dir=/dev/"' >> /etc/default/grub
Generate GRUB menu:
mkdir -p /boot/grub grub-mkconfig -o /boot/grub/grub.cfg cp /boot/grub/grub.cfg \ /boot/efi/archlinux/grub-bootdir/x86_64-efi/grub/grub.cfg cp /boot/grub/grub.cfg \ /boot/efi/archlinux/grub-bootdir/i386-pc/grub/grub.cfg
For both legacy and EFI booting: mirror ESP content:
espdir=$(mktemp -d) find /boot/efi/ -maxdepth 1 -mindepth 1 -type d -print0 \ | xargs -t -0I '{}' cp -r '{}' "${espdir}" find "${espdir}" -maxdepth 1 -mindepth 1 -type d -print0 \ | xargs -t -0I '{}' sh -vxc "find /boot/efis/ -maxdepth 1 -mindepth 1 -type d -print0 | xargs -t -0I '[]' cp -r '{}' '[]'"
Exit chroot
exit
Unmount filesystems and create initial system snapshot You can later create a boot environment from this snapshot. See Root on ZFS maintenance page.
umount -Rl "${MNT}" zfs snapshot -r rpool@initial-installation zfs snapshot -r bpool@initial-installation
Export all pools
zpool export -a
Reboot
reboot