NixOS Root on ZFS
Note for arm64:
Currently there is a bug with the grub installation script. See here for details.
Note for Immutable Root:
Immutable root can be enabled or disabled by setting
zfs-root.boot.immutable
option inside per-host configuration.
Customization
Unless stated otherwise, it is not recommended to customize system configuration before reboot.
Only use well-tested pool features
You should only use well-tested pool features. Avoid using new features if data integrity is paramount. See, for example, this comment.
Preparation
Disable Secure Boot. ZFS modules can not be loaded if Secure Boot is enabled.
Download NixOS Live Image and boot from it.
sha256sum -c ./nixos-*.sha256 dd if=input-file of=output-file bs=1M
Connect to the Internet.
Set root password or
/root/.ssh/authorized_keys
.Start SSH server
systemctl restart sshd
Connect from another computer
ssh root@192.168.1.91
Target disk
List available disks with
find /dev/disk/by-id/
If virtio is used as disk bus, power off the VM and set serial numbers for disk. For QEMU, use
-drive format=raw,file=disk2.img,serial=AaBb
. For libvirt, edit domain XML. See this page for examples.Declare disk array
DISK='/dev/disk/by-id/ata-FOO /dev/disk/by-id/nvme-BAR'
For single disk installation, use
DISK='/dev/disk/by-id/disk1'
Set a mount point
MNT=$(mktemp -d)
Set partition size:
Set swap size in GB, set to 1 if you don’t want swap to take up too much space
SWAPSIZE=4
Set how much space should be left at the end of the disk, minimum 1GB
RESERVE=1
Enable Nix Flakes functionality
mkdir -p ~/.config/nix echo "experimental-features = nix-command flakes" >> ~/.config/nix/nix.conf
Install programs needed for system installation
if ! command -v git; then nix-env -f '<nixpkgs>' -iA git; fi if ! command -v partprobe; then nix-env -f '<nixpkgs>' -iA parted; fi
System Installation
Partition the disks.
Note: you must clear all existing partition tables and data structures from target disks.
For flash-based storage, this can be done by the blkdiscard command below:
partition_disk () { local disk="${1}" blkdiscard -f "${disk}" || true parted --script --align=optimal "${disk}" -- \ mklabel gpt \ mkpart EFI 2MiB 1GiB \ mkpart bpool 1GiB 5GiB \ mkpart rpool 5GiB -$((SWAPSIZE + RESERVE))GiB \ mkpart swap -$((SWAPSIZE + RESERVE))GiB -"${RESERVE}"GiB \ mkpart BIOS 1MiB 2MiB \ set 1 esp on \ set 5 bios_grub on \ set 5 legacy_boot on partprobe "${disk}" udevadm settle } for i in ${DISK}; do partition_disk "${i}" done
Setup encrypted swap. This is useful if the available memory is small:
for i in ${DISK}; do cryptsetup open --type plain --key-file /dev/random "${i}"-part4 "${i##*/}"-part4 mkswap /dev/mapper/"${i##*/}"-part4 swapon /dev/mapper/"${i##*/}"-part4 done
LUKS only: Setup encrypted LUKS container for root pool:
for i in ${DISK}; do # see PASSPHRASE PROCESSING section in cryptsetup(8) printf "YOUR_PASSWD" | cryptsetup luksFormat --type luks2 "${i}"-part3 - printf "YOUR_PASSWD" | cryptsetup luksOpen "${i}"-part3 luks-rpool-"${i##*/}"-part3 - done
Create boot pool
# shellcheck disable=SC2046 zpool create -o compatibility=legacy \ -o ashift=12 \ -o autotrim=on \ -O acltype=posixacl \ -O canmount=off \ -O devices=off \ -O normalization=formD \ -O relatime=on \ -O xattr=sa \ -O mountpoint=/boot \ -R "${MNT}" \ bpool \ mirror \ $(for i in ${DISK}; do printf '%s ' "${i}-part2"; done)
If not using a multi-disk setup, remove
mirror
.You should not need to customize any of the options for the boot pool.
GRUB does not support all of the zpool features. See
spa_feature_names
in grub-core/fs/zfs/zfs.c. This step creates a separate boot pool for/boot
with the features limited to only those that GRUB supports, allowing the root pool to use any/all features.Features enabled with
-o compatibility=grub2
can be seen here.Create root pool
Unencrypted
# shellcheck disable=SC2046 zpool create \ -o ashift=12 \ -o autotrim=on \ -R "${MNT}" \ -O acltype=posixacl \ -O canmount=off \ -O compression=zstd \ -O dnodesize=auto \ -O normalization=formD \ -O relatime=on \ -O xattr=sa \ -O mountpoint=/ \ rpool \ mirror \ $(for i in ${DISK}; do printf '%s ' "${i}-part3"; done)
LUKS encrypted
# shellcheck disable=SC2046 zpool create \ -o ashift=12 \ -o autotrim=on \ -R "${MNT}" \ -O acltype=posixacl \ -O canmount=off \ -O compression=zstd \ -O dnodesize=auto \ -O normalization=formD \ -O relatime=on \ -O xattr=sa \ -O mountpoint=/ \ rpool \ mirror \ $(for i in ${DISK}; do printf '/dev/mapper/luks-rpool-%s ' "${i##*/}-part3"; done)
If not using a multi-disk setup, remove
mirror
.Create root system container:
Unencrypted
zfs create \ -o canmount=off \ -o mountpoint=none \ rpool/nixos
Encrypted:
Avoid ZFS send/recv when using native encryption, see `a ZFS developer's comment on this issue`__ and `this spreadsheet of bugs`__. In short, if you care about your data, don’t use native encryption. This section has been removed, use LUKS encryption instead.
Create system datasets, manage mountpoints with
mountpoint=legacy
zfs create -o mountpoint=legacy rpool/nixos/root mount -t zfs rpool/nixos/root "${MNT}"/ zfs create -o mountpoint=legacy rpool/nixos/home mkdir "${MNT}"/home mount -t zfs rpool/nixos/home "${MNT}"/home zfs create -o mountpoint=none rpool/nixos/var zfs create -o mountpoint=legacy rpool/nixos/var/lib zfs create -o mountpoint=legacy rpool/nixos/var/log zfs create -o mountpoint=none bpool/nixos zfs create -o mountpoint=legacy bpool/nixos/root mkdir "${MNT}"/boot mount -t zfs bpool/nixos/root "${MNT}"/boot mkdir -p "${MNT}"/var/log mkdir -p "${MNT}"/var/lib mount -t zfs rpool/nixos/var/lib "${MNT}"/var/lib mount -t zfs rpool/nixos/var/log "${MNT}"/var/log zfs create -o mountpoint=legacy rpool/nixos/empty zfs snapshot rpool/nixos/empty@start
Format and mount ESP
for i in ${DISK}; do mkfs.vfat -n EFI "${i}"-part1 mkdir -p "${MNT}"/boot/efis/"${i##*/}"-part1 mount -t vfat -o iocharset=iso8859-1 "${i}"-part1 "${MNT}"/boot/efis/"${i##*/}"-part1 done
System Configuration
Clone template flake configuration
mkdir -p "${MNT}"/etc git clone --depth 1 --branch openzfs-guide \ https://github.com/ne9z/dotfiles-flake.git "${MNT}"/etc/nixos
From now on, the complete configuration of the system will be tracked by git, set a user name and email address to continue
rm -rf "${MNT}"/etc/nixos/.git git -C "${MNT}"/etc/nixos/ init -b main git -C "${MNT}"/etc/nixos/ add "${MNT}"/etc/nixos/ git -C "${MNT}"/etc/nixos config user.email "you@example.com" git -C "${MNT}"/etc/nixos config user.name "Alice Q. Nixer" git -C "${MNT}"/etc/nixos commit -asm 'initial commit'
Customize configuration to your hardware
for i in ${DISK}; do sed -i \ "s|/dev/disk/by-id/|${i%/*}/|" \ "${MNT}"/etc/nixos/hosts/exampleHost/default.nix break done diskNames="" for i in ${DISK}; do diskNames="${diskNames} \"${i##*/}\"" done sed -i "s|\"bootDevices_placeholder\"|${diskNames}|g" \ "${MNT}"/etc/nixos/hosts/exampleHost/default.nix sed -i "s|\"abcd1234\"|\"$(head -c4 /dev/urandom | od -A none -t x4| sed 's| ||g' || true)\"|g" \ "${MNT}"/etc/nixos/hosts/exampleHost/default.nix sed -i "s|\"x86_64-linux\"|\"$(uname -m || true)-linux\"|g" \ "${MNT}"/etc/nixos/flake.nix
LUKS only: Enable LUKS support:
sed -i 's|luks.enable = false|luks.enable = true|' "${MNT}"/etc/nixos/hosts/exampleHost/default.nix
Detect kernel modules needed for boot
cp "$(command -v nixos-generate-config || true)" ./nixos-generate-config chmod a+rw ./nixos-generate-config # shellcheck disable=SC2016 echo 'print STDOUT $initrdAvailableKernelModules' >> ./nixos-generate-config kernelModules="$(./nixos-generate-config --show-hardware-config --no-filesystems | tail -n1 || true)" sed -i "s|\"kernelModules_placeholder\"|${kernelModules}|g" \ "${MNT}"/etc/nixos/hosts/exampleHost/default.nix
Set root password
rootPwd=$(mkpasswd -m SHA-512)
Declare password in configuration
sed -i \ "s|rootHash_placeholder|${rootPwd}|" \ "${MNT}"/etc/nixos/configuration.nix
You can enable NetworkManager for wireless networks and GNOME desktop environment in
configuration.nix
.Commit changes to local repo
git -C "${MNT}"/etc/nixos commit -asm 'initial installation'
Update flake lock file to track latest system version
nix flake update --commit-lock-file \ "git+file://${MNT}/etc/nixos"
Install system and apply configuration
nixos-install \ --root "${MNT}" \ --no-root-passwd \ --flake "git+file://${MNT}/etc/nixos#exampleHost"
Unmount filesystems
umount -Rl "${MNT}" zpool export -a
Reboot
reboot
For instructions on maintenance tasks, see Root on ZFS maintenance page.