Overview

This document describes how to install NixOS with ZFS as root file system.

Caution

Partition layout

GUID partition table (GPT) is used. EFI system partition will be referred to as ESP in this document.

Name

legacy boot

ESP

Boot pool

swap

root pool

remaining space

File system

vfat

ZFS

swap

ZFS

Size

1M

2G

4G

depends on RAM size

Optional encryption

Secure Boot

plain dm-crypt

ZFS native encryption

Partition no.

5

1

2

4

3

Mount point

/boot/efis/disk-part1

/boot

/

Dataset layout

The dataset layout used in this guide follows stardard mutable file positions (/var, /etc, …), but can still be modified to a immutable root after installation.

Dataset

canmount

mountpoint

container

notes

bpool

off

/boot

contains sys

rpool

off

/

contains sys

bpool/sys

off

none

contains BOOT

rpool/sys

off

none

contains ROOT

sys is encryptionroot

bpool/sys/BOOT

off

none

contains boot environments

rpool/sys/ROOT

off

none

contains boot environments

rpool/sys/DATA

off

none

contains placeholder “default”

rpool/sys/DATA/default

off

/

contains user datasets

child datsets inherits mountpoint

rpool/sys/DATA/local

off

/

contains /nix datasets

child datsets inherits mountpoint

rpool/sys/DATA/default/ home

on

/home (inherited)

no

user datasets, also called “shared datasets”, “persistent datasets”; also include /var/lib, /srv, …

bpool/sys/BOOT/default

noauto

/boot

no

noauto is used to switch BE. because of noauto, must use fstab to mount

rpool/sys/ROOT/default

noauto

/

no

mounted by initrd zfs hook

Encryption

  • Swap

    Swap is always encrypted. By default, swap is encrypted with plain dm-crypt with key generated from /dev/urandom at every boot. Swap content does not persist between reboots.

  • Root pool

    ZFS native encryption can be optionally enabled for rpool/sys and child datasets.

    User should be aware that, ZFS native encryption does not encrypt some metadata of the datasets. ZFS native encryption also does not change master key when zfs change-key is invoked. Therefore, you should wipe the disk when password is compromised to protect confidentiality. See zfs-load-key.8 and zfs-change-key.8 for more information regarding ZFS native encryption.

    Encryption is enabled at dataset creation and can not be disabled later.

  • Boot pool

    Boot pool can not be encrypted.

  • Bootloader

    Bootloader can not be encrypted.

    However, with Secure Boot, bootloader can be verified by motherboard firmware to be untempered, which should be sufficient for most purposes.

    Secure Boot is not supported out-of-the-box due to ZFS module.

Booting with disk failure

This guide is written with disk failure in mind.

If disks used in Root on ZFS pool failed, but sufficient redundancy for both root pool and boot pool still exists, the system will still boot normally.

Swap partition on the failed disk will fail to mount, after an 1m30s timeout.

This feature is useful for use cases such as an unattended remote server.

Example:

  • System has disks n>1

  • Installed with mirrored setup

  • Mirrored setup can tolerate up to n-1 disk failures

  • Disconnect one or more disks, keep at least one disk connected

  • System still boots, but fails to mount swap and EFI partition