Arch Linux Root on ZFS

ZFSBootMenu

This tutorial is based on the GRUB bootloader. Due to its independent implementation of a read-only ZFS driver, GRUB only supports a subset of ZFS features on the boot pool. [In general, bootloader treat disks as read-only to minimize the risk of damaging on-disk data.]

ZFSBootMenu is an alternative bootloader free of such limitations and has support for boot environments. Do not follow instructions on this page if you plan to use ZBM, as the layouts are not compatible. Refer to their site for installation details.

Customization

Unless stated otherwise, it is not recommended to customize system configuration before reboot.

Only use well-tested pool features

You should only use well-tested pool features. Avoid using new features if data integrity is paramount. See, for example, this comment.

Preparation

1. Disable Secure Boot. ZFS modules can not be loaded if Secure Boot is enabled.

2. Because the kernel of latest Live CD might be incompatible with ZFS, we will use Alpine Linux Extended, which ships with ZFS by default.

   Download latest extended variant of Alpine Linux live image, verify checksum and boot from it.

   gpg --auto-key-retrieve --keyserver hkps://keyserver.ubuntu.com --verify alpine-extended-*.asc
   
   dd if=input-file of=output-file bs=1M
   

3. Login as root user. There is no password.

4. Configure Internet

   setup-interfaces -r
   # You must use "-r" option to start networking services properly
   # example:
   network interface: wlan0
   WiFi name:         <ssid>
   ip address:        dhcp
   <enter done to finish network config>
   manual netconfig:  n
   

5. If you are using wireless network and it is not shown, see Alpine Linux wiki for further details. wpa_supplicant can be installed with apk add wpa_supplicant without internet connection.

6. Configure SSH server

   setup-sshd
   # example:
   ssh server:        openssh
   allow root:        "prohibit-password" or "yes"
   ssh key:           "none" or "<public key>"
   

7. Set root password or /root/.ssh/authorized_keys.

8. Connect from another computer

   ssh root@192.168.1.91
   

9. Configure NTP client for time synchronization

   setup-ntp busybox
   

10. Set up apk-repo. A list of available mirrors is shown. Press space bar to continue

    setup-apkrepos
    

11. Throughout this guide, we use predictable disk names generated by udev

    apk update
    apk add eudev
    setup-devd udev
    

12. Target disk

    List available disks with

    find /dev/disk/by-id/
    

    If virtio is used as disk bus, power off the VM and set serial numbers for disk. For QEMU, use -drive format=raw,file=disk2.img,serial=AaBb. For libvirt, edit domain XML. See this page for examples.

    Declare disk array

    DISK='/dev/disk/by-id/ata-FOO /dev/disk/by-id/nvme-BAR'
    

    For single disk installation, use

    DISK='/dev/disk/by-id/disk1'
    

13. Set a mount point

    MNT=$(mktemp -d)
    

14. Set partition size:

    Set swap size in GB, set to 1 if you don’t want swap to take up too much space

    SWAPSIZE=4
    

    Set how much space should be left at the end of the disk, minimum 1GB

    RESERVE=1
    

15. Install ZFS support from live media:

    apk add zfs
    

16. Install partition tool

    apk add parted e2fsprogs cryptsetup util-linux
    

System Installation

1. Partition the disks.

   Note: you must clear all existing partition tables and data structures from target disks.

   For flash-based storage, this can be done by the blkdiscard command below:

   partition_disk () {
    local disk="${1}"
    blkdiscard -f "${disk}" || true
   
    parted --script --align=optimal  "${disk}" -- \
    mklabel gpt \
    mkpart EFI 2MiB 1GiB \
    mkpart bpool 1GiB 5GiB \
    mkpart rpool 5GiB -$((SWAPSIZE + RESERVE))GiB \
    mkpart swap  -$((SWAPSIZE + RESERVE))GiB -"${RESERVE}"GiB \
    mkpart BIOS 1MiB 2MiB \
    set 1 esp on \
    set 5 bios_grub on \
    set 5 legacy_boot on
   
    partprobe "${disk}"
   }
   
   for i in ${DISK}; do
      partition_disk "${i}"
   done
   

2. Setup encrypted swap. This is useful if the available memory is small:

   for i in ${DISK}; do
      cryptsetup open --type plain --key-file /dev/random "${i}"-part4 "${i##*/}"-part4
      mkswap /dev/mapper/"${i##*/}"-part4
      swapon /dev/mapper/"${i##*/}"-part4
   done
   

3. Load ZFS kernel module

   modprobe zfs
   

4. Create boot pool

   # shellcheck disable=SC2046
   zpool create -o compatibility=legacy  \
       -o ashift=12 \
       -o autotrim=on \
       -O acltype=posixacl \
       -O canmount=off \
       -O devices=off \
       -O normalization=formD \
       -O relatime=on \
       -O xattr=sa \
       -O mountpoint=/boot \
       -R "${MNT}" \
       bpool \
              mirror \
       $(for i in ${DISK}; do
          printf '%s ' "${i}-part2";
         done)
   

   If not using a multi-disk setup, remove mirror.

   You should not need to customize any of the options for the boot pool.

   GRUB does not support all of the zpool features. See spa_feature_names in grub-core/fs/zfs/zfs.c. This step creates a separate boot pool for /boot with the features limited to only those that GRUB supports, allowing the root pool to use any/all features.

5. Create root pool

   # shellcheck disable=SC2046
   zpool create \
       -o ashift=12 \
       -o autotrim=on \
       -R "${MNT}" \
       -O acltype=posixacl \
       -O canmount=off \
       -O compression=zstd \
       -O dnodesize=auto \
       -O normalization=formD \
       -O relatime=on \
       -O xattr=sa \
       -O mountpoint=/ \
       rpool \
       mirror \
      $(for i in ${DISK}; do
         printf '%s ' "${i}-part3";
        done)
   

   If not using a multi-disk setup, remove mirror.

6. Create root system container:

   • Unencrypted

     zfs create \
      -o canmount=off \
      -o mountpoint=none \
     rpool/archlinux
     

   • Encrypted:

     Avoid ZFS send/recv when using native encryption, see `a ZFS developer's comment on this issue`__ and `this spreadsheet of bugs`__. A LUKS-based guide has yet to be written. Once compromised, changing password will not keep your data safe. See zfs-change-key(8) for more info

     zfs create \
       -o canmount=off \
              -o mountpoint=none \
              -o encryption=on \
              -o keylocation=prompt \
              -o keyformat=passphrase \
     rpool/archlinux
     

   You can automate this step (insecure) with: echo POOLPASS | zfs create ....

   Create system datasets, manage mountpoints with mountpoint=legacy

   zfs create -o canmount=noauto -o mountpoint=/  rpool/archlinux/root
   zfs mount rpool/archlinux/root
   zfs create -o mountpoint=legacy rpool/archlinux/home
   mkdir "${MNT}"/home
   mount -t zfs rpool/archlinux/home "${MNT}"/home
   zfs create -o mountpoint=legacy  rpool/archlinux/var
   zfs create -o mountpoint=legacy rpool/archlinux/var/lib
   zfs create -o mountpoint=legacy rpool/archlinux/var/log
   zfs create -o mountpoint=none bpool/archlinux
   zfs create -o mountpoint=legacy bpool/archlinux/root
   mkdir "${MNT}"/boot
   mount -t zfs bpool/archlinux/root "${MNT}"/boot
   mkdir -p "${MNT}"/var/log
   mkdir -p "${MNT}"/var/lib
   mount -t zfs rpool/archlinux/var/lib "${MNT}"/var/lib
   mount -t zfs rpool/archlinux/var/log "${MNT}"/var/log
   

7. Format and mount ESP

   for i in ${DISK}; do
    mkfs.vfat -n EFI "${i}"-part1
    mkdir -p "${MNT}"/boot/efis/"${i##*/}"-part1
    mount -t vfat -o iocharset=iso8859-1 "${i}"-part1 "${MNT}"/boot/efis/"${i##*/}"-part1
   done
   
   mkdir -p "${MNT}"/boot/efi
   mount -t vfat -o iocharset=iso8859-1 "$(echo "${DISK}" | sed "s|^ *||"  | cut -f1 -d' '|| true)"-part1 "${MNT}"/boot/efi
   

System Configuration

1. Download and extract minimal Arch Linux root filesystem:

   apk add curl
   
   curl --fail-early --fail -L \
   https://america.archive.pkgbuild.com/iso/2023.09.01/archlinux-bootstrap-x86_64.tar.gz \
   -o rootfs.tar.gz
   curl --fail-early --fail -L \
   https://america.archive.pkgbuild.com/iso/2023.09.01/archlinux-bootstrap-x86_64.tar.gz.sig \
   -o rootfs.tar.gz.sig
   
   apk add gnupg
   gpg --auto-key-retrieve --keyserver hkps://keyserver.ubuntu.com --verify rootfs.tar.gz.sig
   
   ln -s "${MNT}" "${MNT}"/root.x86_64
   tar x  -C "${MNT}" -af rootfs.tar.gz root.x86_64
   

2. Enable community repo

   sed -i '/edge/d' /etc/apk/repositories
   sed -i -E 's/#(.*)community/\1community/' /etc/apk/repositories
   

3. Generate fstab:

   apk add arch-install-scripts
   genfstab -t PARTUUID "${MNT}" \
   | grep -v swap \
   | sed "s|vfat.*rw|vfat rw,x-systemd.idle-timeout=1min,x-systemd.automount,noauto,nofail|" \
   > "${MNT}"/etc/fstab
   

4. Chroot

   cp /etc/resolv.conf "${MNT}"/etc/resolv.conf
   for i in /dev /proc /sys; do mkdir -p "${MNT}"/"${i}"; mount --rbind "${i}" "${MNT}"/"${i}"; done
   chroot "${MNT}" /usr/bin/env DISK="${DISK}" bash
   

5. Add archzfs repo to pacman config

   pacman-key --init
   pacman-key --refresh-keys
   pacman-key --populate
   
   curl --fail-early --fail -L https://archzfs.com/archzfs.gpg \
   |  pacman-key -a - --gpgdir /etc/pacman.d/gnupg
   
   pacman-key \
   --lsign-key \
   --gpgdir /etc/pacman.d/gnupg \
   DDF7DB817396A49B2A2723F7403BD972F75D9D76
   
   tee -a /etc/pacman.d/mirrorlist-archzfs <<- 'EOF'
   ## See https://github.com/archzfs/archzfs/wiki
   ## France
   #,Server = https://archzfs.com/$repo/$arch
   
   ## Germany
   #,Server = https://mirror.sum7.eu/archlinux/archzfs/$repo/$arch
   #,Server = https://mirror.biocrafting.net/archlinux/archzfs/$repo/$arch
   
   ## India
   #,Server = https://mirror.in.themindsmaze.com/archzfs/$repo/$arch
   
   ## United States
   #,Server = https://zxcvfdsa.com/archzfs/$repo/$arch
   EOF
   
   tee -a /etc/pacman.conf <<- 'EOF'
   
   #[archzfs-testing]
   #Include = /etc/pacman.d/mirrorlist-archzfs
   
   #,[archzfs]
   #,Include = /etc/pacman.d/mirrorlist-archzfs
   EOF
   
   # this #, prefix is a workaround for ci/cd tests
   # remove them
   sed -i 's|#,||' /etc/pacman.d/mirrorlist-archzfs
   sed -i 's|#,||' /etc/pacman.conf
   sed -i 's|^#||' /etc/pacman.d/mirrorlist
   

6. Install base packages:

   pacman -Sy
   pacman -S --noconfirm mg mandoc grub efibootmgr mkinitcpio
   
   kernel_compatible_with_zfs="$(pacman -Si zfs-linux \
   | grep 'Depends On' \
   | sed "s|.*linux=||" \
   | awk '{ print $1 }')"
   pacman -U --noconfirm https://america.archive.pkgbuild.com/packages/l/linux/linux-"${kernel_compatible_with_zfs}"-x86_64.pkg.tar.zst
   

7. Install zfs packages:

   pacman -S --noconfirm zfs-linux zfs-utils
   

8. Configure mkinitcpio:

   sed -i 's|filesystems|zfs filesystems|' /etc/mkinitcpio.conf
   mkinitcpio -P
   

9. For physical machine, install firmware

   pacman -S linux-firmware intel-ucode amd-ucode
   

10. Enable internet time synchronisation:

    systemctl enable systemd-timesyncd
    

11. Generate host id:

    zgenhostid -f -o /etc/hostid
    

12. Generate locales:

    echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen
    locale-gen
    

13. Set locale, keymap, timezone, hostname

    rm -f /etc/localtime
    systemd-firstboot \
    --force \
    --locale=en_US.UTF-8 \
    --timezone=Etc/UTC \
    --hostname=testhost \
    --keymap=us
    

14. Set root passwd

    printf 'root:yourpassword' | chpasswd
    

Bootloader

1. Apply GRUB workaround

   echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile.d/zpool_vdev_name_path.sh
   # shellcheck disable=SC1091
   . /etc/profile.d/zpool_vdev_name_path.sh
   
   # GRUB fails to detect rpool name, hard code as "rpool"
   sed -i "s|rpool=.*|rpool=rpool|"  /etc/grub.d/10_linux
   

   This workaround needs to be applied for every GRUB update, as the update will overwrite the changes.

2. Install GRUB:

   mkdir -p /boot/efi/archlinux/grub-bootdir/i386-pc/
   mkdir -p /boot/efi/archlinux/grub-bootdir/x86_64-efi/
   for i in ${DISK}; do
    grub-install --target=i386-pc --boot-directory \
        /boot/efi/archlinux/grub-bootdir/i386-pc/  "${i}"
   done
   grub-install --target x86_64-efi --boot-directory \
    /boot/efi/archlinux/grub-bootdir/x86_64-efi/ --efi-directory \
    /boot/efi --bootloader-id archlinux --removable
   if test -d /sys/firmware/efi/efivars/; then
      grub-install --target x86_64-efi --boot-directory \
       /boot/efi/archlinux/grub-bootdir/x86_64-efi/ --efi-directory \
       /boot/efi --bootloader-id archlinux
   fi
   

3. Import both bpool and rpool at boot:

   echo 'GRUB_CMDLINE_LINUX="zfs_import_dir=/dev/"' >> /etc/default/grub
   

4. Generate GRUB menu:

   mkdir -p /boot/grub
   grub-mkconfig -o /boot/grub/grub.cfg
   cp /boot/grub/grub.cfg \
    /boot/efi/archlinux/grub-bootdir/x86_64-efi/grub/grub.cfg
   cp /boot/grub/grub.cfg \
    /boot/efi/archlinux/grub-bootdir/i386-pc/grub/grub.cfg
   

5. For both legacy and EFI booting: mirror ESP content:

   espdir=$(mktemp -d)
   find /boot/efi/ -maxdepth 1 -mindepth 1 -type d -print0 \
   | xargs -t -0I '{}' cp -r '{}' "${espdir}"
   find "${espdir}" -maxdepth 1 -mindepth 1 -type d -print0 \
   | xargs -t -0I '{}' sh -vxc "find /boot/efis/ -maxdepth 1 -mindepth 1 -type d -print0 | xargs -t -0I '[]' cp -r '{}' '[]'"
   

6. Exit chroot

   exit
   

7. Unmount filesystems and create initial system snapshot You can later create a boot environment from this snapshot. See Root on ZFS maintenance page.

   umount -Rl "${MNT}"
   zfs snapshot -r rpool@initial-installation
   zfs snapshot -r bpool@initial-installation
   

8. Export all pools

   zpool export -a
   

9. Reboot

   reboot