NixOS Root on ZFS

Note for arm64:

Currently there is a bug with the grub installation script. See here for details.

Note for Immutable Root:

Immutable root can be enabled or disabled by setting zfs-root.boot.immutable option inside per-host configuration.

Customization

Unless stated otherwise, it is not recommended to customize system configuration before reboot.

Only use well-tested pool features

You should only use well-tested pool features. Avoid using new features if data integrity is paramount. See, for example, this comment.

Preparation

Disable Secure Boot. ZFS modules can not be loaded if Secure Boot is enabled.

Download NixOS Live Image and boot from it.

sha256sum -c ./nixos-*.sha256

dd if=input-file of=output-file bs=1M

Connect to the Internet.
Set root password or /root/.ssh/authorized_keys.
Start SSH server
```
systemctl restart sshd
```
Connect from another computer
```
ssh root@192.168.1.91
```
Target disk

List available disks with
```
find /dev/disk/by-id/
```
If virtio is used as disk bus, power off the VM and set serial numbers for disk. For QEMU, use -drive format=raw,file=disk2.img,serial=AaBb. For libvirt, edit domain XML. See this page for examples.

Declare disk array
```
DISK='/dev/disk/by-id/ata-FOO /dev/disk/by-id/nvme-BAR'
```
For single disk installation, use
```
DISK='/dev/disk/by-id/disk1'
```
Set a mount point
```
MNT=$(mktemp -d)
```
Set partition size:

Set swap size in GB, set to 1 if you don’t want swap to take up too much space
```
SWAPSIZE=4
```
Set how much space should be left at the end of the disk, minimum 1GB
```
RESERVE=1
```

Enable Nix Flakes functionality

mkdir -p ~/.config/nix
echo "experimental-features = nix-command flakes" >> ~/.config/nix/nix.conf

Install programs needed for system installation

if ! command -v git; then nix-env -f '<nixpkgs>' -iA git; fi
if ! command -v partprobe;  then nix-env -f '<nixpkgs>' -iA parted; fi

System Installation

Partition the disks.

Note: you must clear all existing partition tables and data structures from target disks.

For flash-based storage, this can be done by the blkdiscard command below:

partition_disk () {
 local disk="${1}"
 blkdiscard -f "${disk}" || true

 parted --script --align=optimal  "${disk}" -- \
 mklabel gpt \
 mkpart EFI 2MiB 1GiB \
 mkpart bpool 1GiB 5GiB \
 mkpart rpool 5GiB -$((SWAPSIZE + RESERVE))GiB \
 mkpart swap  -$((SWAPSIZE + RESERVE))GiB -"${RESERVE}"GiB \
 mkpart BIOS 1MiB 2MiB \
 set 1 esp on \
 set 5 bios_grub on \
 set 5 legacy_boot on

 partprobe "${disk}"
 udevadm settle
}

for i in ${DISK}; do
   partition_disk "${i}"
done

Setup encrypted swap. This is useful if the available memory is small:

for i in ${DISK}; do
   cryptsetup open --type plain --key-file /dev/random "${i}"-part4 "${i##*/}"-part4
   mkswap /dev/mapper/"${i##*/}"-part4
   swapon /dev/mapper/"${i##*/}"-part4
done

LUKS only: Setup encrypted LUKS container for root pool:

for i in ${DISK}; do
   # see PASSPHRASE PROCESSING section in cryptsetup(8)
   printf "YOUR_PASSWD" | cryptsetup luksFormat --type luks2 "${i}"-part3 -
   printf "YOUR_PASSWD" | cryptsetup luksOpen "${i}"-part3 luks-rpool-"${i##*/}"-part3 -
done

Create boot pool

# shellcheck disable=SC2046
zpool create -o compatibility=legacy  \
    -o ashift=12 \
    -o autotrim=on \
    -O acltype=posixacl \
    -O canmount=off \
    -O devices=off \
    -O normalization=formD \
    -O relatime=on \
    -O xattr=sa \
    -O mountpoint=/boot \
    -R "${MNT}" \
    bpool \
  mirror \
    $(for i in ${DISK}; do
       printf '%s ' "${i}-part2";
      done)

If not using a multi-disk setup, remove mirror.

You should not need to customize any of the options for the boot pool.

GRUB does not support all of the zpool features. See spa_feature_names in grub-core/fs/zfs/zfs.c. This step creates a separate boot pool for /boot with the features limited to only those that GRUB supports, allowing the root pool to use any/all features.

Features enabled with -o compatibility=grub2 can be seen here.

Create root pool

Unencrypted

# shellcheck disable=SC2046
zpool create \
    -o ashift=12 \
    -o autotrim=on \
    -R "${MNT}" \
    -O acltype=posixacl \
    -O canmount=off \
    -O compression=zstd \
    -O dnodesize=auto \
    -O normalization=formD \
    -O relatime=on \
    -O xattr=sa \
    -O mountpoint=/ \
    rpool \
    mirror \
   $(for i in ${DISK}; do
      printf '%s ' "${i}-part3";
     done)

LUKS encrypted

# shellcheck disable=SC2046
zpool create \
    -o ashift=12 \
    -o autotrim=on \
    -R "${MNT}" \
    -O acltype=posixacl \
    -O canmount=off \
    -O compression=zstd \
    -O dnodesize=auto \
    -O normalization=formD \
    -O relatime=on \
    -O xattr=sa \
    -O mountpoint=/ \
    rpool \
    mirror \
   $(for i in ${DISK}; do
      printf '/dev/mapper/luks-rpool-%s ' "${i##*/}-part3";
     done)

If not using a multi-disk setup, remove mirror.

Create root system container:

Unencrypted

zfs create \
 -o canmount=off \
 -o mountpoint=none \
rpool/nixos

Encrypted:

Avoid ZFS send/recv when using native encryption, see `a ZFS developer's comment on this issue`__ and `this spreadsheet of bugs`__. In short, if you care about your data, don’t use native encryption. This section has been removed, use LUKS encryption instead.

Create system datasets, manage mountpoints with mountpoint=legacy

zfs create -o mountpoint=legacy     rpool/nixos/root
mount -t zfs rpool/nixos/root "${MNT}"/
zfs create -o mountpoint=legacy rpool/nixos/home
mkdir "${MNT}"/home
mount -t zfs rpool/nixos/home "${MNT}"/home
zfs create -o mountpoint=none   rpool/nixos/var
zfs create -o mountpoint=legacy rpool/nixos/var/lib
zfs create -o mountpoint=legacy rpool/nixos/var/log
zfs create -o mountpoint=none bpool/nixos
zfs create -o mountpoint=legacy bpool/nixos/root
mkdir "${MNT}"/boot
mount -t zfs bpool/nixos/root "${MNT}"/boot
mkdir -p "${MNT}"/var/log
mkdir -p "${MNT}"/var/lib
mount -t zfs rpool/nixos/var/lib "${MNT}"/var/lib
mount -t zfs rpool/nixos/var/log "${MNT}"/var/log
zfs create -o mountpoint=legacy rpool/nixos/empty
zfs snapshot rpool/nixos/empty@start

Format and mount ESP

for i in ${DISK}; do
 mkfs.vfat -n EFI "${i}"-part1
 mkdir -p "${MNT}"/boot/efis/"${i##*/}"-part1
 mount -t vfat -o iocharset=iso8859-1 "${i}"-part1 "${MNT}"/boot/efis/"${i##*/}"-part1
done

System Configuration

Clone template flake configuration

mkdir -p "${MNT}"/etc
git clone --depth 1 --branch openzfs-guide \
  https://github.com/ne9z/dotfiles-flake.git "${MNT}"/etc/nixos

From now on, the complete configuration of the system will be tracked by git, set a user name and email address to continue

rm -rf "${MNT}"/etc/nixos/.git
git -C "${MNT}"/etc/nixos/ init -b main
git -C "${MNT}"/etc/nixos/ add "${MNT}"/etc/nixos/
git -C "${MNT}"/etc/nixos config user.email "you@example.com"
git -C "${MNT}"/etc/nixos config user.name "Alice Q. Nixer"
git -C "${MNT}"/etc/nixos commit -asm 'initial commit'

Customize configuration to your hardware

for i in ${DISK}; do
  sed -i \
  "s|/dev/disk/by-id/|${i%/*}/|" \
  "${MNT}"/etc/nixos/hosts/exampleHost/default.nix
  break
done

diskNames=""
for i in ${DISK}; do
  diskNames="${diskNames} \"${i##*/}\""
done

sed -i "s|\"bootDevices_placeholder\"|${diskNames}|g" \
  "${MNT}"/etc/nixos/hosts/exampleHost/default.nix

sed -i "s|\"abcd1234\"|\"$(head -c4 /dev/urandom | od -A none -t x4| sed 's| ||g' || true)\"|g" \
  "${MNT}"/etc/nixos/hosts/exampleHost/default.nix

sed -i "s|\"x86_64-linux\"|\"$(uname -m || true)-linux\"|g" \
  "${MNT}"/etc/nixos/flake.nix

LUKS only: Enable LUKS support:

sed -i 's|luks.enable = false|luks.enable = true|' "${MNT}"/etc/nixos/hosts/exampleHost/default.nix

Detect kernel modules needed for boot

cp "$(command -v nixos-generate-config || true)" ./nixos-generate-config

chmod a+rw ./nixos-generate-config

# shellcheck disable=SC2016
echo 'print STDOUT $initrdAvailableKernelModules' >> ./nixos-generate-config

kernelModules="$(./nixos-generate-config --show-hardware-config --no-filesystems | tail -n1 || true)"

sed -i "s|\"kernelModules_placeholder\"|${kernelModules}|g" \
  "${MNT}"/etc/nixos/hosts/exampleHost/default.nix

Set root password

rootPwd=$(mkpasswd -m SHA-512)

Declare password in configuration

sed -i \
"s|rootHash_placeholder|${rootPwd}|" \
"${MNT}"/etc/nixos/configuration.nix

You can enable NetworkManager for wireless networks and GNOME desktop environment in configuration.nix.

Commit changes to local repo

git -C "${MNT}"/etc/nixos commit -asm 'initial installation'

Update flake lock file to track latest system version

nix flake update --commit-lock-file \
  "git+file://${MNT}/etc/nixos"

Install system and apply configuration

nixos-install \
--root "${MNT}" \
--no-root-passwd \
--flake "git+file://${MNT}/etc/nixos#exampleHost"

Unmount filesystems
```
umount -Rl "${MNT}"
zpool export -a
```
Reboot
```
reboot
```
For instructions on maintenance tasks, see Root on ZFS maintenance page.