NixOS Root on ZFS

Note for arm64:

Currently there is a bug with the grub installation script. See here for details.

Note for Immutable Root:

Immutable root can be enabled or disabled by setting zfs-root.boot.immutable option inside per-host configuration.

Customization

Unless stated otherwise, it is not recommended to customize system configuration before reboot.

Only use well-tested pool features

You should only use well-tested pool features. Avoid using new features if data integrity is paramount. See, for example, this comment.

Preparation

  1. Disable Secure Boot. ZFS modules can not be loaded if Secure Boot is enabled.

  2. Download NixOS Live Image and boot from it.

    sha256sum -c ./nixos-*.sha256
    
    dd if=input-file of=output-file bs=1M
    
  3. Connect to the Internet.

  4. Set root password or /root/.ssh/authorized_keys.

  5. Start SSH server

    systemctl restart sshd
    
  6. Connect from another computer

    ssh root@192.168.1.91
    
  7. Target disk

    List available disks with

    find /dev/disk/by-id/
    

    If virtio is used as disk bus, power off the VM and set serial numbers for disk. For QEMU, use -drive format=raw,file=disk2.img,serial=AaBb. For libvirt, edit domain XML. See this page for examples.

    Declare disk array

    DISK='/dev/disk/by-id/ata-FOO /dev/disk/by-id/nvme-BAR'
    

    For single disk installation, use

    DISK='/dev/disk/by-id/disk1'
    
  8. Set a mount point

    MNT=$(mktemp -d)
    
  9. Set partition size:

    Set swap size in GB, set to 1 if you don’t want swap to take up too much space

    SWAPSIZE=4
    

    Set how much space should be left at the end of the disk, minimum 1GB

    RESERVE=1
    
  10. Enable Nix Flakes functionality

    mkdir -p ~/.config/nix
    echo "experimental-features = nix-command flakes" >> ~/.config/nix/nix.conf
    
  11. Install programs needed for system installation

    if ! command -v git; then nix-env -f '<nixpkgs>' -iA git; fi
    if ! command -v partprobe;  then nix-env -f '<nixpkgs>' -iA parted; fi
    

System Installation

  1. Partition the disks.

    Note: you must clear all existing partition tables and data structures from target disks.

    For flash-based storage, this can be done by the blkdiscard command below:

    partition_disk () {
     local disk="${1}"
     blkdiscard -f "${disk}" || true
    
     parted --script --align=optimal  "${disk}" -- \
     mklabel gpt \
     mkpart EFI 2MiB 1GiB \
     mkpart bpool 1GiB 5GiB \
     mkpart rpool 5GiB -$((SWAPSIZE + RESERVE))GiB \
     mkpart swap  -$((SWAPSIZE + RESERVE))GiB -"${RESERVE}"GiB \
     mkpart BIOS 1MiB 2MiB \
     set 1 esp on \
     set 5 bios_grub on \
     set 5 legacy_boot on
    
     partprobe "${disk}"
     udevadm settle
    }
    
    for i in ${DISK}; do
       partition_disk "${i}"
    done
    
  2. Setup encrypted swap. This is useful if the available memory is small:

    for i in ${DISK}; do
       cryptsetup open --type plain --key-file /dev/random "${i}"-part4 "${i##*/}"-part4
       mkswap /dev/mapper/"${i##*/}"-part4
       swapon /dev/mapper/"${i##*/}"-part4
    done
    
  3. LUKS only: Setup encrypted LUKS container for root pool:

    for i in ${DISK}; do
       # see PASSPHRASE PROCESSING section in cryptsetup(8)
       printf "YOUR_PASSWD" | cryptsetup luksFormat --type luks2 "${i}"-part3 -
       printf "YOUR_PASSWD" | cryptsetup luksOpen "${i}"-part3 luks-rpool-"${i##*/}"-part3 -
    done
    
  4. Create boot pool

    # shellcheck disable=SC2046
    zpool create -o compatibility=legacy  \
        -o ashift=12 \
        -o autotrim=on \
        -O acltype=posixacl \
        -O canmount=off \
        -O devices=off \
        -O normalization=formD \
        -O relatime=on \
        -O xattr=sa \
        -O mountpoint=/boot \
        -R "${MNT}" \
        bpool \
      mirror \
        $(for i in ${DISK}; do
           printf '%s ' "${i}-part2";
          done)
    

    If not using a multi-disk setup, remove mirror.

    You should not need to customize any of the options for the boot pool.

    GRUB does not support all of the zpool features. See spa_feature_names in grub-core/fs/zfs/zfs.c. This step creates a separate boot pool for /boot with the features limited to only those that GRUB supports, allowing the root pool to use any/all features.

    Features enabled with -o compatibility=grub2 can be seen here.

  5. Create root pool

    • Unencrypted

      # shellcheck disable=SC2046
      zpool create \
          -o ashift=12 \
          -o autotrim=on \
          -R "${MNT}" \
          -O acltype=posixacl \
          -O canmount=off \
          -O compression=zstd \
          -O dnodesize=auto \
          -O normalization=formD \
          -O relatime=on \
          -O xattr=sa \
          -O mountpoint=/ \
          rpool \
          mirror \
         $(for i in ${DISK}; do
            printf '%s ' "${i}-part3";
           done)
      
    • LUKS encrypted

      # shellcheck disable=SC2046
      zpool create \
          -o ashift=12 \
          -o autotrim=on \
          -R "${MNT}" \
          -O acltype=posixacl \
          -O canmount=off \
          -O compression=zstd \
          -O dnodesize=auto \
          -O normalization=formD \
          -O relatime=on \
          -O xattr=sa \
          -O mountpoint=/ \
          rpool \
          mirror \
         $(for i in ${DISK}; do
            printf '/dev/mapper/luks-rpool-%s ' "${i##*/}-part3";
           done)
      

    If not using a multi-disk setup, remove mirror.

  6. Create root system container:

    Create system datasets, manage mountpoints with mountpoint=legacy

    zfs create -o mountpoint=legacy     rpool/nixos/root
    mount -t zfs rpool/nixos/root "${MNT}"/
    zfs create -o mountpoint=legacy rpool/nixos/home
    mkdir "${MNT}"/home
    mount -t zfs rpool/nixos/home "${MNT}"/home
    zfs create -o mountpoint=none   rpool/nixos/var
    zfs create -o mountpoint=legacy rpool/nixos/var/lib
    zfs create -o mountpoint=legacy rpool/nixos/var/log
    zfs create -o mountpoint=none bpool/nixos
    zfs create -o mountpoint=legacy bpool/nixos/root
    mkdir "${MNT}"/boot
    mount -t zfs bpool/nixos/root "${MNT}"/boot
    mkdir -p "${MNT}"/var/log
    mkdir -p "${MNT}"/var/lib
    mount -t zfs rpool/nixos/var/lib "${MNT}"/var/lib
    mount -t zfs rpool/nixos/var/log "${MNT}"/var/log
    zfs create -o mountpoint=legacy rpool/nixos/empty
    zfs snapshot rpool/nixos/empty@start
    
  7. Format and mount ESP

    for i in ${DISK}; do
     mkfs.vfat -n EFI "${i}"-part1
     mkdir -p "${MNT}"/boot/efis/"${i##*/}"-part1
     mount -t vfat -o iocharset=iso8859-1 "${i}"-part1 "${MNT}"/boot/efis/"${i##*/}"-part1
    done
    

System Configuration

  1. Clone template flake configuration

    mkdir -p "${MNT}"/etc
    git clone --depth 1 --branch openzfs-guide \
      https://github.com/ne9z/dotfiles-flake.git "${MNT}"/etc/nixos
    
  2. From now on, the complete configuration of the system will be tracked by git, set a user name and email address to continue

    rm -rf "${MNT}"/etc/nixos/.git
    git -C "${MNT}"/etc/nixos/ init -b main
    git -C "${MNT}"/etc/nixos/ add "${MNT}"/etc/nixos/
    git -C "${MNT}"/etc/nixos config user.email "you@example.com"
    git -C "${MNT}"/etc/nixos config user.name "Alice Q. Nixer"
    git -C "${MNT}"/etc/nixos commit -asm 'initial commit'
    
  3. Customize configuration to your hardware

    for i in ${DISK}; do
      sed -i \
      "s|/dev/disk/by-id/|${i%/*}/|" \
      "${MNT}"/etc/nixos/hosts/exampleHost/default.nix
      break
    done
    
    diskNames=""
    for i in ${DISK}; do
      diskNames="${diskNames} \"${i##*/}\""
    done
    
    sed -i "s|\"bootDevices_placeholder\"|${diskNames}|g" \
      "${MNT}"/etc/nixos/hosts/exampleHost/default.nix
    
    sed -i "s|\"abcd1234\"|\"$(head -c4 /dev/urandom | od -A none -t x4| sed 's| ||g' || true)\"|g" \
      "${MNT}"/etc/nixos/hosts/exampleHost/default.nix
    
    sed -i "s|\"x86_64-linux\"|\"$(uname -m || true)-linux\"|g" \
      "${MNT}"/etc/nixos/flake.nix
    
  4. LUKS only: Enable LUKS support:

    sed -i 's|luks.enable = false|luks.enable = true|' "${MNT}"/etc/nixos/hosts/exampleHost/default.nix
    
  5. Detect kernel modules needed for boot

    cp "$(command -v nixos-generate-config || true)" ./nixos-generate-config
    
    chmod a+rw ./nixos-generate-config
    
    # shellcheck disable=SC2016
    echo 'print STDOUT $initrdAvailableKernelModules' >> ./nixos-generate-config
    
    kernelModules="$(./nixos-generate-config --show-hardware-config --no-filesystems | tail -n1 || true)"
    
    sed -i "s|\"kernelModules_placeholder\"|${kernelModules}|g" \
      "${MNT}"/etc/nixos/hosts/exampleHost/default.nix
    
  6. Set root password

    rootPwd=$(mkpasswd -m SHA-512)
    

    Declare password in configuration

    sed -i \
    "s|rootHash_placeholder|${rootPwd}|" \
    "${MNT}"/etc/nixos/configuration.nix
    
  7. You can enable NetworkManager for wireless networks and GNOME desktop environment in configuration.nix.

  8. Commit changes to local repo

    git -C "${MNT}"/etc/nixos commit -asm 'initial installation'
    
  9. Update flake lock file to track latest system version

    nix flake update --commit-lock-file \
      "git+file://${MNT}/etc/nixos"
    
  10. Install system and apply configuration

    nixos-install \
    --root "${MNT}" \
    --no-root-passwd \
    --flake "git+file://${MNT}/etc/nixos#exampleHost"
    
  11. Unmount filesystems

    umount -Rl "${MNT}"
    zpool export -a
    
  12. Reboot

    reboot
    
  13. For instructions on maintenance tasks, see Root on ZFS maintenance page.