Arch Linux Root on ZFS

Customization

Unless stated otherwise, it is not recommended to customize system configuration before reboot.

Preparation

Disable Secure Boot. ZFS modules can not be loaded if Secure Boot is enabled.
Because the kernel of latest Live CD might be incompatible with ZFS, we will use Alpine Linux Extended, which ships with ZFS by default.

Download latest extended variant of Alpine Linux live image, verify checksum and boot from it.
```
gpg --auto-key-retrieve --keyserver hkps://keyserver.ubuntu.com --verify alpine-extended-*.asc

dd if=input-file of=output-file bs=1M
```
Login as root user. There is no password.

Configure Internet

setup-interfaces -r
# You must use "-r" option to start networking services properly
# example:
network interface: wlan0
WiFi name:         <ssid>
ip address:        dhcp
<enter done to finish network config>
manual netconfig:  n

If you are using wireless network and it is not shown, see Alpine Linux wiki for further details. wpa_supplicant can be installed with apk add wpa_supplicant without internet connection.

Configure SSH server

setup-sshd
# example:
ssh server:        openssh
allow root:        "prohibit-password" or "yes"
ssh key:           "none" or "<public key>"

Set root password or /root/.ssh/authorized_keys.
Connect from another computer
```
ssh root@192.168.1.91
```
Configure NTP client for time synchronization
```
setup-ntp busybox
```
Set up apk-repo. A list of available mirrors is shown. Press space bar to continue
```
setup-apkrepos
```
Throughout this guide, we use predictable disk names generated by udev
```
apk update
apk add eudev
setup-devd udev
```
Target disk

List available disks with
```
find /dev/disk/by-id/
```
If virtio is used as disk bus, power off the VM and set serial numbers for disk. For QEMU, use -drive format=raw,file=disk2.img,serial=AaBb. For libvirt, edit domain XML. See this page for examples.

Declare disk array
```
DISK='/dev/disk/by-id/ata-FOO /dev/disk/by-id/nvme-BAR'
```
For single disk installation, use
```
DISK='/dev/disk/by-id/disk1'
```
Set a mount point
```
MNT=$(mktemp -d)
```
Set partition size:

Set swap size in GB, set to 1 if you don’t want swap to take up too much space
```
SWAPSIZE=4
```
Set how much space should be left at the end of the disk, minimum 1GB
```
RESERVE=1
```
Install ZFS support from live media:
```
apk add zfs
```

Install partition tool

apk add parted e2fsprogs cryptsetup util-linux

System Installation

Partition the disks.

Note: you must clear all existing partition tables and data structures from the disks, especially those with existing ZFS pools or mdraid and those that have been used as live media. Those data structures may interfere with boot process.

For flash-based storage, this can be done by the blkdiscard command below:

partition_disk () {
 local disk="${1}"
 blkdiscard -f "${disk}" || true

 parted --script --align=optimal  "${disk}" -- \
 mklabel gpt \
 mkpart EFI 2MiB 1GiB \
 mkpart bpool 1GiB 5GiB \
 mkpart rpool 5GiB -$((SWAPSIZE + RESERVE))GiB \
 mkpart swap  -$((SWAPSIZE + RESERVE))GiB -"${RESERVE}"GiB \
 mkpart BIOS 1MiB 2MiB \
 set 1 esp on \
 set 5 bios_grub on \
 set 5 legacy_boot on

 partprobe "${disk}"
}

for i in ${DISK}; do
   partition_disk "${i}"
done

Setup encrypted swap. This is useful if the available memory is small:

for i in ${DISK}; do
   cryptsetup open --type plain --key-file /dev/random "${i}"-part4 "${i##*/}"-part4
   mkswap /dev/mapper/"${i##*/}"-part4
   swapon /dev/mapper/"${i##*/}"-part4
done

Load ZFS kernel module
```
modprobe zfs
```

Create boot pool

# shellcheck disable=SC2046
zpool create -d \
    -o feature@async_destroy=enabled \
    -o feature@bookmarks=enabled \
    -o feature@embedded_data=enabled \
    -o feature@empty_bpobj=enabled \
    -o feature@enabled_txg=enabled \
    -o feature@extensible_dataset=enabled \
    -o feature@filesystem_limits=enabled \
    -o feature@hole_birth=enabled \
    -o feature@large_blocks=enabled \
    -o feature@lz4_compress=enabled \
    -o feature@spacemap_histogram=enabled \
    -o ashift=12 \
    -o autotrim=on \
    -O acltype=posixacl \
    -O canmount=off \
    -O compression=lz4 \
    -O devices=off \
    -O normalization=formD \
    -O relatime=on \
    -O xattr=sa \
    -O mountpoint=/boot \
    -R "${MNT}" \
    bpool \
           mirror \
    $(for i in ${DISK}; do
       printf '%s ' "${i}-part2";
      done)

If not using a multi-disk setup, remove mirror.

You should not need to customize any of the options for the boot pool.

GRUB does not support all of the zpool features. See spa_feature_names in grub-core/fs/zfs/zfs.c. This step creates a separate boot pool for /boot with the features limited to only those that GRUB supports, allowing the root pool to use any/all features.

Create root pool

# shellcheck disable=SC2046
zpool create \
    -o ashift=12 \
    -o autotrim=on \
    -R "${MNT}" \
    -O acltype=posixacl \
    -O canmount=off \
    -O compression=zstd \
    -O dnodesize=auto \
    -O normalization=formD \
    -O relatime=on \
    -O xattr=sa \
    -O mountpoint=/ \
    rpool \
    mirror \
   $(for i in ${DISK}; do
      printf '%s ' "${i}-part3";
     done)

If not using a multi-disk setup, remove mirror.

Create root system container:

Unencrypted

zfs create \
 -o canmount=off \
 -o mountpoint=none \
rpool/archlinux

Encrypted:

Pick a strong password. Once compromised, changing password will not keep your data safe. See zfs-change-key(8) for more info

zfs create \
  -o canmount=off \
         -o mountpoint=none \
         -o encryption=on \
         -o keylocation=prompt \
         -o keyformat=passphrase \
rpool/archlinux

You can automate this step (insecure) with: echo POOLPASS | zfs create ....

Create system datasets, manage mountpoints with mountpoint=legacy

zfs create -o canmount=noauto -o mountpoint=/  rpool/archlinux/root
zfs mount rpool/archlinux/root
zfs create -o mountpoint=legacy rpool/archlinux/home
mkdir "${MNT}"/home
mount -t zfs rpool/archlinux/home "${MNT}"/home
zfs create -o mountpoint=legacy  rpool/archlinux/var
zfs create -o mountpoint=legacy rpool/archlinux/var/lib
zfs create -o mountpoint=legacy rpool/archlinux/var/log
zfs create -o mountpoint=none bpool/archlinux
zfs create -o mountpoint=legacy bpool/archlinux/root
mkdir "${MNT}"/boot
mount -t zfs bpool/archlinux/root "${MNT}"/boot
mkdir -p "${MNT}"/var/log
mkdir -p "${MNT}"/var/lib
mount -t zfs rpool/archlinux/var/lib "${MNT}"/var/lib
mount -t zfs rpool/archlinux/var/log "${MNT}"/var/log

Format and mount ESP

for i in ${DISK}; do
 mkfs.vfat -n EFI "${i}"-part1
 mkdir -p "${MNT}"/boot/efis/"${i##*/}"-part1
 mount -t vfat -o iocharset=iso8859-1 "${i}"-part1 "${MNT}"/boot/efis/"${i##*/}"-part1
done

mkdir -p "${MNT}"/boot/efi
mount -t vfat -o iocharset=iso8859-1 "$(echo "${DISK}" | sed "s|^ *||"  | cut -f1 -d' '|| true)"-part1 "${MNT}"/boot/efi

System Configuration

Download and extract minimal Arch Linux root filesystem:

apk add curl

curl --fail-early --fail -L \
https://america.archive.pkgbuild.com/iso/2023.05.03/archlinux-bootstrap-x86_64.tar.gz \
-o rootfs.tar.gz
curl --fail-early --fail -L \
https://america.archive.pkgbuild.com/iso/2023.05.03/archlinux-bootstrap-x86_64.tar.gz.sig \
-o rootfs.tar.gz.sig

apk add gnupg
gpg --auto-key-retrieve --keyserver hkps://keyserver.ubuntu.com --verify rootfs.tar.gz.sig

ln -s "${MNT}" "${MNT}"/root.x86_64
tar x  -C "${MNT}" -af rootfs.tar.gz root.x86_64

Enable community repo

sed -i '/edge/d' /etc/apk/repositories
sed -i -E 's/#(.*)community/\1community/' /etc/apk/repositories

Generate fstab:

apk add arch-install-scripts
genfstab -t PARTUUID "${MNT}" \
| grep -v swap \
| sed "s|vfat.*rw|vfat rw,x-systemd.idle-timeout=1min,x-systemd.automount,noauto,nofail|" \
> "${MNT}"/etc/fstab

Chroot

cp /etc/resolv.conf "${MNT}"/etc/resolv.conf
for i in /dev /proc /sys; do mkdir -p "${MNT}"/"${i}"; mount --rbind "${i}" "${MNT}"/"${i}"; done
chroot "${MNT}" /usr/bin/env DISK="${DISK}" bash

Add archzfs repo to pacman config

pacman-key --init
pacman-key --refresh-keys
pacman-key --populate

curl --fail-early --fail -L https://archzfs.com/archzfs.gpg \
|  pacman-key -a - --gpgdir /etc/pacman.d/gnupg

pacman-key \
--lsign-key \
--gpgdir /etc/pacman.d/gnupg \
DDF7DB817396A49B2A2723F7403BD972F75D9D76

tee -a /etc/pacman.d/mirrorlist-archzfs <<- 'EOF'
## See https://github.com/archzfs/archzfs/wiki
## France
#,Server = https://archzfs.com/$repo/$arch

## Germany
#,Server = https://mirror.sum7.eu/archlinux/archzfs/$repo/$arch
#,Server = https://mirror.biocrafting.net/archlinux/archzfs/$repo/$arch

## India
#,Server = https://mirror.in.themindsmaze.com/archzfs/$repo/$arch

## United States
#,Server = https://zxcvfdsa.com/archzfs/$repo/$arch
EOF

tee -a /etc/pacman.conf <<- 'EOF'

#[archzfs-testing]
#Include = /etc/pacman.d/mirrorlist-archzfs

#,[archzfs]
#,Include = /etc/pacman.d/mirrorlist-archzfs
EOF

# this #, prefix is a workaround for ci/cd tests
# remove them
sed -i 's|#,||' /etc/pacman.d/mirrorlist-archzfs
sed -i 's|#,||' /etc/pacman.conf
sed -i 's|^#||' /etc/pacman.d/mirrorlist

Install base packages:

pacman -Sy
pacman -S --noconfirm mg mandoc grub efibootmgr mkinitcpio

kernel_compatible_with_zfs="$(pacman -Si zfs-linux \
| grep 'Depends On' \
| sed "s|.*linux=||" \
| awk '{ print $1 }')"
pacman -U --noconfirm https://america.archive.pkgbuild.com/packages/l/linux/linux-"${kernel_compatible_with_zfs}"-x86_64.pkg.tar.zst

Install zfs packages:

pacman -S --noconfirm zfs-linux zfs-utils

Configure mkinitcpio:

sed -i 's|filesystems|zfs filesystems|' /etc/mkinitcpio.conf
mkinitcpio -P

For physical machine, install firmware

pacman -S linux-firmware intel-ucode amd-ucode

Enable internet time synchronisation:
```
systemctl enable systemd-timesyncd
```
Generate host id:
```
zgenhostid -f -o /etc/hostid
```

Generate locales:

echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen
locale-gen

Set locale, keymap, timezone, hostname

rm -f /etc/localtime
systemd-firstboot \
--force \
--locale=en_US.UTF-8 \
--timezone=Etc/UTC \
--hostname=testhost \
--keymap=us

Set root passwd
```
printf 'root:yourpassword' | chpasswd
```

Bootloader

Apply GRUB workaround

echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile.d/zpool_vdev_name_path.sh
# shellcheck disable=SC1091
. /etc/profile.d/zpool_vdev_name_path.sh

# GRUB fails to detect rpool name, hard code as "rpool"
sed -i "s|rpool=.*|rpool=rpool|"  /etc/grub.d/10_linux

This workaround needs to be applied for every GRUB update, as the update will overwrite the changes.

Install GRUB:

mkdir -p /boot/efi/archlinux/grub-bootdir/i386-pc/
mkdir -p /boot/efi/archlinux/grub-bootdir/x86_64-efi/
for i in ${DISK}; do
 grub-install --target=i386-pc --boot-directory \
     /boot/efi/archlinux/grub-bootdir/i386-pc/  "${i}"
done
grub-install --target x86_64-efi --boot-directory \
 /boot/efi/archlinux/grub-bootdir/x86_64-efi/ --efi-directory \
 /boot/efi --bootloader-id archlinux --removable
if test -d /sys/firmware/efi/efivars/; then
   grub-install --target x86_64-efi --boot-directory \
    /boot/efi/archlinux/grub-bootdir/x86_64-efi/ --efi-directory \
    /boot/efi --bootloader-id archlinux
fi

Import both bpool and rpool at boot:

echo 'GRUB_CMDLINE_LINUX="zfs_import_dir=/dev/"' >> /etc/default/grub

Generate GRUB menu:

mkdir -p /boot/grub
grub-mkconfig -o /boot/grub/grub.cfg
cp /boot/grub/grub.cfg \
 /boot/efi/archlinux/grub-bootdir/x86_64-efi/grub/grub.cfg
cp /boot/grub/grub.cfg \
 /boot/efi/archlinux/grub-bootdir/i386-pc/grub/grub.cfg

For both legacy and EFI booting: mirror ESP content:

espdir=$(mktemp -d)
find /boot/efi/ -maxdepth 1 -mindepth 1 -type d -print0 \
| xargs -t -0I '{}' cp -r '{}' "${espdir}"
find "${espdir}" -maxdepth 1 -mindepth 1 -type d -print0 \
| xargs -t -0I '{}' sh -vxc "find /boot/efis/ -maxdepth 1 -mindepth 1 -type d -print0 | xargs -t -0I '[]' cp -r '{}' '[]'"

Exit chroot
```
exit
```
Unmount filesystems and create initial system snapshot You can later create a boot environment from this snapshot. See Root on ZFS maintenance page.
```
umount -Rl "${MNT}"
zfs snapshot -r rpool@initial-installation
zfs snapshot -r bpool@initial-installation
```
Export all pools
```
zpool export -a
```
Reboot
```
reboot
```