OpenZFS Documentation ===================== Welcome to the OpenZFS Documentation. This resource provides documentation for users and developers working with (or contributing to) the OpenZFS project. New users or system administrators should refer to the documentation for their favorite platform to get started. +----------------------+----------------------+----------------------+ | :doc:`Getting Started| :doc:`Project and | :doc:`Developer | | <./Getting | Community <./Project | Resources ` | and Community/index>`| Resources/index>` | +======================+======================+======================+ | How to get started | About the project | Technical | | with OpenZFS on your | and how to | documentation | | favorite platform | contribute | discussing the | | | | OpenZFS | | | | implementation | +----------------------+----------------------+----------------------+ Table of Contents: ------------------ .. toctree:: :maxdepth: 2 :glob: Getting Started/index Basic Concepts/index Project and Community/index Developer Resources/index Performance and Tuning/index man/index msg/index License Getting Started =============== To get started with OpenZFS refer to the provided documentation for your distribution. It will cover the recommended installation method and any distribution specific information. First time OpenZFS users should then read :doc:`Basic Concepts `, starting with :doc:`Copy-on-Write ` and :doc:`VDEVs ` — pool layout is the decision that is hardest to undo. .. toctree:: :maxdepth: 3 :glob: Alpine Linux/index Arch Linux/index Debian/index Fedora/index FreeBSD Gentoo NixOS/index openSUSE/index RHEL-based distro/index Slackware/index Ubuntu/index Alpine Linux ============ Contents -------- .. toctree:: :maxdepth: 1 :glob: * Installation ------------ Note: this is for installing ZFS on an existing Alpine installation. To use ZFS as root file system, see below. #. Install ZFS package:: apk add zfs zfs-lts #. Load kernel module:: modprobe zfs Automatic zpool importing and mount ----------------------------------- To avoid needing to manually import and mount zpools after the system boots, be sure to enable the related services. #. Import pools on boot:: rc-update add zfs-import default #. Mount pools on boot:: rc-update add zfs-mount default Root on ZFS ----------- .. toctree:: :maxdepth: 1 :glob: * .. highlight:: sh Alpine Linux Root on ZFS ======================== .. ifconfig:: zfs_root_test :: # For the CI/CD test run of this guide, # Enable verbose logging of bash shell and fail immediately when # a command fails. set -vxeuf distro=${1} cp /etc/resolv.conf ./"rootfs-${distro}"/etc/resolv.conf arch-chroot ./"rootfs-${distro}" sh <<-'ZFS_ROOT_GUIDE_TEST' set -vxeuf # install alpine setup scripts apk update apk add alpine-conf curl **ZFSBootMenu** `ZFSBootMenu `__ is an alternative bootloader free of such limitations and has support for boot environments. Do not follow instructions on this page if you plan to use ZBM, as the layouts are not compatible. Refer to their site for installation details. **Customization** Unless stated otherwise, it is not recommended to customize system configuration before reboot. **Only use well-tested pool features** You should only use well-tested pool features. Avoid using new features if data integrity is paramount. See, for example, `this comment `__. **UEFI support only** Only UEFI is supported by this guide. Preparation --------------------------- #. Disable Secure Boot. ZFS modules can not be loaded if Secure Boot is enabled. #. Download latest extended variant of `Alpine Linux live image `__, verify `checksum `__ and boot from it. .. code-block:: sh gpg --auto-key-retrieve --keyserver hkps://keyserver.ubuntu.com --verify alpine-extended-*.asc dd if=input-file of=output-file bs=1M .. ifconfig:: zfs_root_test # check whether the download page exists # alpine version must be in sync with ci/cd test chroot tarball curl --head --fail https://dl-cdn.alpinelinux.org/alpine/v3.19/releases/x86_64/alpine-extended-3.19.0-x86_64.iso curl --head --fail https://dl-cdn.alpinelinux.org/alpine/v3.19/releases/x86_64/alpine-extended-3.19.0-x86_64.iso.asc #. Login as root user. There is no password. #. Configure Internet .. code-block:: sh setup-interfaces -r # You must use "-r" option to start networking services properly # example: network interface: wlan0 WiFi name: ip address: dhcp manual netconfig: n #. If you are using wireless network and it is not shown, see `Alpine Linux wiki `__ for further details. ``wpa_supplicant`` can be installed with ``apk add wpa_supplicant`` without internet connection. #. Configure SSH server .. code-block:: sh setup-sshd # example: ssh server: openssh allow root: "prohibit-password" or "yes" ssh key: "none" or "" Configurations set here will be copied verbatim to the installed system. #. Set root password or ``/root/.ssh/authorized_keys``. Choose a strong root password, as it will be copied to the installed system. However, ``authorized_keys`` is not copied. #. Connect from another computer .. code-block:: sh ssh root@192.168.1.91 #. Configure NTP client for time synchronization .. code-block:: sh setup-ntp busybox .. ifconfig:: zfs_root_test # this step is unnecessary for chroot and returns 1 when executed #. Set up apk-repo. A list of available mirrors is shown. Press space bar to continue .. code-block:: sh setup-apkrepos #. Throughout this guide, we use predictable disk names generated by udev .. code-block:: sh apk update apk add eudev setup-devd udev It can be removed after reboot with ``setup-devd mdev && apk del eudev``. .. ifconfig:: zfs_root_test # for some reason, udev is extremely slow in chroot # it is not needed for chroot anyway. so, skip this step #. Target disk List available disks with .. code-block:: sh find /dev/disk/by-id/ If virtio is used as disk bus, power off the VM and set serial numbers for disk. For QEMU, use ``-drive format=raw,file=disk2.img,serial=AaBb``. For libvirt, edit domain XML. See `this page `__ for examples. Declare disk array .. code-block:: sh DISK='/dev/disk/by-id/ata-FOO /dev/disk/by-id/nvme-BAR' For single disk installation, use .. code-block:: sh DISK='/dev/disk/by-id/disk1' .. ifconfig:: zfs_root_test # for github test run, use chroot and loop devices DISK="$(losetup -a| grep alpine | cut -f1 -d: | xargs -t -I '{}' printf '{} ')" # for maintenance guide test DISK="$(losetup -a| grep maintenance | cut -f1 -d: | xargs -t -I '{}' printf '{} ') ${DISK}" #. Set a mount point :: MNT=$(mktemp -d) #. Set partition size: Set swap size in GB, set to 1 if you don't want swap to take up too much space .. code-block:: sh SWAPSIZE=4 .. ifconfig:: zfs_root_test # For the test run, use 1GB swap space to avoid hitting CI/CD # quota SWAPSIZE=1 Set how much space should be left at the end of the disk, minimum 1GB :: RESERVE=1 #. Install ZFS support from live media:: apk add zfs #. Install bootloader programs and partition tool :: apk add parted e2fsprogs cryptsetup util-linux System Installation --------------------------- #. Partition the disks. Note: you must clear all existing partition tables and data structures from target disks. For flash-based storage, this can be done by the blkdiscard command below: :: partition_disk () { local disk="${1}" blkdiscard -f "${disk}" || true parted --script --align=optimal "${disk}" -- \ mklabel gpt \ mkpart EFI 1MiB 4GiB \ mkpart rpool 4GiB -$((SWAPSIZE + RESERVE))GiB \ mkpart swap -$((SWAPSIZE + RESERVE))GiB -"${RESERVE}"GiB \ set 1 esp on \ partprobe "${disk}" } for i in ${DISK}; do partition_disk "${i}" done .. ifconfig:: zfs_root_test :: # When working with GitHub chroot runners, we are using loop # devices as installation target. However, the alias support for # loop device was just introduced in March 2023. See # https://github.com/systemd/systemd/pull/26693 # For now, we will create the aliases manually as a workaround looppart="1 2 3 4 5" for i in ${DISK}; do for j in ${looppart}; do if test -e "${i}p${j}"; then ln -s "${i}p${j}" "${i}-part${j}" fi done done #. Setup temporary encrypted swap for this installation only. This is useful if the available memory is small:: for i in ${DISK}; do cryptsetup open --type plain --key-file /dev/random "${i}"-part3 "${i##*/}"-part3 mkswap /dev/mapper/"${i##*/}"-part3 swapon /dev/mapper/"${i##*/}"-part3 done #. Load ZFS kernel module .. code-block:: sh modprobe zfs #. Create root pool - Unencrypted:: # shellcheck disable=SC2046 zpool create \ -o ashift=12 \ -o autotrim=on \ -R "${MNT}" \ -O acltype=posixacl \ -O canmount=off \ -O dnodesize=auto \ -O normalization=formD \ -O relatime=on \ -O xattr=sa \ -O mountpoint=none \ rpool \ mirror \ $(for i in ${DISK}; do printf '%s ' "${i}-part2"; done) #. Create root system container: :: zfs create -o canmount=noauto -o mountpoint=legacy rpool/root Create system datasets, manage mountpoints with ``mountpoint=legacy`` :: zfs create -o mountpoint=legacy rpool/home mount -o X-mount.mkdir -t zfs rpool/root "${MNT}" mount -o X-mount.mkdir -t zfs rpool/home "${MNT}"/home #. Format and mount ESP. Only one of them is used as /boot, you need to set up mirroring afterwards :: for i in ${DISK}; do mkfs.vfat -n EFI "${i}"-part1 done for i in ${DISK}; do mount -t vfat -o fmask=0077,dmask=0077,iocharset=iso8859-1,X-mount.mkdir "${i}"-part1 "${MNT}"/boot break done System Configuration --------------------------- #. Install system to disk .. code-block:: sh BOOTLOADER=none setup-disk -k lts -v "${MNT}" The error message about ZFS kernel module can be ignored. .. ifconfig:: zfs_root_test # lts kernel will pull in tons of firmware BOOTLOADER=none setup-disk -k virt -v "${MNT}" #. Install rEFInd boot loader:: # from http://www.rodsbooks.com/refind/getting.html # use Binary Zip File option apk add curl curl -L http://sourceforge.net/projects/refind/files/0.14.0.2/refind-bin-0.14.0.2.zip/download --output refind.zip unzip refind mkdir -p "${MNT}"/boot/EFI/BOOT find ./refind-bin-0.14.0.2/ -name 'refind_x64.efi' -print0 \ | xargs -0I{} mv {} "${MNT}"/boot/EFI/BOOT/BOOTX64.EFI rm -rf refind.zip refind-bin-0.14.0.2 #. Add boot entry:: tee -a "${MNT}"/boot/refind-linux.conf <`__ on `Libera Chat `__. If you have a bug report or feature request related to this HOWTO, please `file a new issue and mention @ne9z `__. Overview -------- Due to license incompatibility, ZFS is not available in Arch Linux official repo. ZFS support is provided by third-party `archzfs repo `__. Installation ------------ See `Archlinux Wiki `__. Root on ZFS ----------- ZFS can be used as root file system for Arch Linux. An installation guide is available. .. toctree:: :maxdepth: 1 :glob: * Contribute ---------- #. Fork and clone `this repo `__. #. Install the tools:: sudo pacman -S --needed python-pip make pip3 install -r docs/requirements.txt # Add ~/.local/bin to your "${PATH}", e.g. by adding this to ~/.bashrc: [ -d "${HOME}"/.local/bin ] && export PATH="${HOME}"/.local/bin:"${PATH}" #. Make your changes. #. Test:: cd docs make html sensible-browser _build/html/index.html #. ``git commit --signoff`` to a branch, ``git push``, and create a pull request. Mention @ne9z. .. highlight:: sh .. ifconfig:: zfs_root_test :: # For the CI/CD test run of this guide, # Enable verbose logging of bash shell and fail immediately when # a command fails. set -vxeuf distro=${1} cp /etc/resolv.conf ./"rootfs-${distro}"/etc/resolv.conf arch-chroot ./"rootfs-${distro}" sh <<-'ZFS_ROOT_GUIDE_TEST' set -vxeuf # install alpine setup scripts apk update apk add alpine-conf curl .. In this document, there are three types of code-block markups: ``::`` are commands intended for both the vm test and the users ``.. ifconfig:: zfs_root_test`` are commands intended only for vm test ``.. code-block:: sh`` are commands intended only for users Arch Linux Root on ZFS ======================================= **ZFSBootMenu** `ZFSBootMenu `__ is an alternative bootloader free of such limitations and has support for boot environments. Do not follow instructions on this page if you plan to use ZBM, as the layouts are not compatible. Refer to their site for installation details. **Customization** Unless stated otherwise, it is not recommended to customize system configuration before reboot. **Only use well-tested pool features** You should only use well-tested pool features. Avoid using new features if data integrity is paramount. See, for example, `this comment `__. **UEFI support only** Only UEFI is supported by this guide. Preparation --------------------------- #. Disable Secure Boot. ZFS modules can not be loaded if Secure Boot is enabled. #. Because the kernel of latest Live CD might be incompatible with ZFS, we will use Alpine Linux Extended, which ships with ZFS by default. Download latest extended variant of `Alpine Linux live image `__, verify `checksum `__ and boot from it. .. code-block:: sh gpg --auto-key-retrieve --keyserver hkps://keyserver.ubuntu.com --verify alpine-extended-*.asc dd if=input-file of=output-file bs=1M .. ifconfig:: zfs_root_test # check whether the download page exists # alpine version must be in sync with ci/cd test chroot tarball #. Login as root user. There is no password. #. Configure Internet .. code-block:: sh setup-interfaces -r # You must use "-r" option to start networking services properly # example: network interface: wlan0 WiFi name: ip address: dhcp manual netconfig: n #. If you are using wireless network and it is not shown, see `Alpine Linux wiki `__ for further details. ``wpa_supplicant`` can be installed with ``apk add wpa_supplicant`` without internet connection. #. Configure SSH server .. code-block:: sh setup-sshd # example: ssh server: openssh allow root: "prohibit-password" or "yes" ssh key: "none" or "" #. Set root password or ``/root/.ssh/authorized_keys``. #. Connect from another computer .. code-block:: sh ssh root@192.168.1.91 #. Configure NTP client for time synchronization .. code-block:: sh setup-ntp busybox .. ifconfig:: zfs_root_test # this step is unnecessary for chroot and returns 1 when executed #. Set up apk-repo. A list of available mirrors is shown. Press space bar to continue .. code-block:: sh setup-apkrepos #. Throughout this guide, we use predictable disk names generated by udev .. code-block:: sh apk update apk add eudev setup-devd udev .. ifconfig:: zfs_root_test # for some reason, udev is extremely slow in chroot # it is not needed for chroot anyway. so, skip this step #. Target disk List available disks with .. code-block:: sh find /dev/disk/by-id/ If virtio is used as disk bus, power off the VM and set serial numbers for disk. For QEMU, use ``-drive format=raw,file=disk2.img,serial=AaBb``. For libvirt, edit domain XML. See `this page `__ for examples. Declare disk array .. code-block:: sh DISK='/dev/disk/by-id/ata-FOO /dev/disk/by-id/nvme-BAR' For single disk installation, use .. code-block:: sh DISK='/dev/disk/by-id/disk1' .. ifconfig:: zfs_root_test # for github test run, use chroot and loop devices DISK="$(losetup -a| grep archlinux | cut -f1 -d: | xargs -t -I '{}' printf '{} ')" #. Set a mount point :: MNT=$(mktemp -d) #. Set partition size: Set swap size in GB, set to 1 if you don't want swap to take up too much space .. code-block:: sh SWAPSIZE=4 .. ifconfig:: zfs_root_test # For the test run, use 1GB swap space to avoid hitting CI/CD # quota SWAPSIZE=1 Set how much space should be left at the end of the disk, minimum 1GB :: RESERVE=1 #. Install ZFS support from live media:: apk add zfs #. Install partition tool :: apk add parted e2fsprogs cryptsetup util-linux System Installation --------------------------- #. Partition the disks. Note: you must clear all existing partition tables and data structures from target disks. For flash-based storage, this can be done by the blkdiscard command below: :: partition_disk () { local disk="${1}" blkdiscard -f "${disk}" || true parted --script --align=optimal "${disk}" -- \ mklabel gpt \ mkpart EFI 1MiB 4GiB \ mkpart rpool 4GiB -$((SWAPSIZE + RESERVE))GiB \ mkpart swap -$((SWAPSIZE + RESERVE))GiB -"${RESERVE}"GiB \ set 1 esp on \ partprobe "${disk}" } for i in ${DISK}; do partition_disk "${i}" done .. ifconfig:: zfs_root_test :: # When working with GitHub chroot runners, we are using loop # devices as installation target. However, the alias support for # loop device was just introduced in March 2023. See # https://github.com/systemd/systemd/pull/26693 # For now, we will create the aliases manually as a workaround looppart="1 2 3 4 5" for i in ${DISK}; do for j in ${looppart}; do if test -e "${i}p${j}"; then ln -s "${i}p${j}" "${i}-part${j}" fi done done #. Setup temporary encrypted swap for this installation only. This is useful if the available memory is small:: for i in ${DISK}; do cryptsetup open --type plain --key-file /dev/random "${i}"-part3 "${i##*/}"-part3 mkswap /dev/mapper/"${i##*/}"-part3 swapon /dev/mapper/"${i##*/}"-part3 done #. Load ZFS kernel module .. code-block:: sh modprobe zfs #. Create root pool - Unencrypted:: # shellcheck disable=SC2046 zpool create \ -o ashift=12 \ -o autotrim=on \ -R "${MNT}" \ -O acltype=posixacl \ -O canmount=off \ -O dnodesize=auto \ -O normalization=formD \ -O relatime=on \ -O xattr=sa \ -O mountpoint=none \ rpool \ mirror \ $(for i in ${DISK}; do printf '%s ' "${i}-part2"; done) #. Create root system container: :: zfs create -o canmount=noauto -o mountpoint=legacy rpool/root Create system datasets, manage mountpoints with ``mountpoint=legacy`` :: zfs create -o mountpoint=legacy rpool/home mount -o X-mount.mkdir -t zfs rpool/root "${MNT}" mount -o X-mount.mkdir -t zfs rpool/home "${MNT}"/home #. Format and mount ESP. Only one of them is used as /boot, you need to set up mirroring afterwards :: for i in ${DISK}; do mkfs.vfat -n EFI "${i}"-part1 done for i in ${DISK}; do mount -t vfat -o fmask=0077,dmask=0077,iocharset=iso8859-1,X-mount.mkdir "${i}"-part1 "${MNT}"/boot break done System Configuration --------------------------- #. Download and extract minimal Arch Linux root filesystem:: apk add curl curl --fail-early --fail -L \ https://archive.archlinux.org/iso/2026.07.01/archlinux-bootstrap-x86_64.tar.zst \ -o rootfs.tar.zst curl --fail-early --fail -L \ https://archive.archlinux.org/iso/2026.07.01/archlinux-bootstrap-x86_64.tar.zst.sig \ -o rootfs.tar.zst.sig apk add gnupg gpg --auto-key-retrieve --keyserver hkps://keyserver.ubuntu.com --verify rootfs.tar.zst.sig apk add zstd ln -s "${MNT}" "${MNT}"/root.x86_64 zstd -d -c rootfs.tar.zst | tar x -C "${MNT}" root.x86_64 #. Enable community repo .. code-block:: sh sed -i '/edge/d' /etc/apk/repositories sed -i -E 's/#(.*)community/\1community/' /etc/apk/repositories #. Generate fstab:: apk add arch-install-scripts genfstab -t PARTUUID "${MNT}" \ | grep -v swap \ | sed "s|vfat.*rw|vfat rw,x-systemd.idle-timeout=1min,x-systemd.automount,noauto,nofail|" \ > "${MNT}"/etc/fstab #. Chroot .. code-block:: sh cp /etc/resolv.conf "${MNT}"/etc/resolv.conf for i in /dev /proc /sys; do mkdir -p "${MNT}"/"${i}"; mount --rbind "${i}" "${MNT}"/"${i}"; done chroot "${MNT}" /usr/bin/env DISK="${DISK}" bash .. ifconfig:: zfs_root_test :: cp /etc/resolv.conf "${MNT}"/etc/resolv.conf for i in /dev /proc /sys; do mkdir -p "${MNT}"/"${i}"; mount --rbind "${i}" "${MNT}"/"${i}"; done chroot "${MNT}" /usr/bin/env DISK="${DISK}" bash <<-'ZFS_ROOT_NESTED_CHROOT' set -vxeuf #. Add archzfs repo to pacman config :: pacman-key --init pacman-key --refresh-keys pacman-key --populate # See https://github.com/archzfs/archzfs/wiki: since 2025-06-19 the # repository is published as a GitHub release and the archzfs.com # one is legacy pacman-key --recv-keys 3A9917BF0DED5C13F69AC68FABEC0A1208037BE9 pacman-key \ --lsign-key \ --gpgdir /etc/pacman.d/gnupg \ 3A9917BF0DED5C13F69AC68FABEC0A1208037BE9 tee -a /etc/pacman.conf <<- 'EOF' #,[archzfs] #,SigLevel = Required #,Server = https://github.com/archzfs/archzfs/releases/download/experimental EOF # this #, prefix is a workaround for ci/cd tests # remove them sed -i 's|#,||' /etc/pacman.conf sed -i 's|^#||' /etc/pacman.d/mirrorlist #. Install base packages:: pacman -Sy # the keyring of the bootstrap image is as old as the image, and the # packages being installed are current, so refresh it before anything # else - otherwise their signatures are "unknown trust" pacman -S --noconfirm archlinux-keyring pacman -S --noconfirm mg mandoc efibootmgr mkinitcpio kernel_compatible_with_zfs="$(pacman -Si zfs-linux \ | grep 'Depends On' \ | sed "s|.*linux=||" \ | awk '{ print $1 }')" pacman -U --noconfirm https://archive.archlinux.org/packages/l/linux/linux-"${kernel_compatible_with_zfs}"-x86_64.pkg.tar.zst #. Install zfs packages:: pacman -S --noconfirm zfs-linux zfs-utils #. Configure mkinitcpio:: sed -i 's|filesystems|zfs filesystems|' /etc/mkinitcpio.conf mkinitcpio -P #. For physical machine, install firmware .. code-block:: sh pacman -S linux-firmware intel-ucode amd-ucode #. Enable internet time synchronisation:: systemctl enable systemd-timesyncd #. Generate host id:: zgenhostid -f -o /etc/hostid #. Generate locales:: echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen locale-gen #. Set locale, keymap, timezone, hostname :: rm -f /etc/localtime systemd-firstboot \ --force \ --locale=en_US.UTF-8 \ --timezone=Etc/UTC \ --hostname=testhost \ --keymap=us #. Set root passwd :: printf 'root:yourpassword' | chpasswd Bootloader --------------------------- #. Install rEFInd boot loader:: # from http://www.rodsbooks.com/refind/getting.html # use Binary Zip File option pacman -S --noconfirm unzip curl -L http://sourceforge.net/projects/refind/files/0.14.0.2/refind-bin-0.14.0.2.zip/download --output refind.zip unzip refind.zip mkdir -p /boot/EFI/BOOT find ./refind-bin-0.14.0.2/ -name 'refind_x64.efi' -print0 \ | xargs -0I{} mv {} /boot/EFI/BOOT/BOOTX64.EFI rm -rf refind.zip refind-bin-0.14.0.2 #. Add boot entry:: tee -a /boot/refind-linux.conf <`__. The `backports repository `__ often provides newer releases of ZFS. You can use it as follows. Add the backports repository:: vi /etc/apt/sources.list.d/trixie-backports.list .. code-block:: sourceslist deb http://deb.debian.org/debian trixie-backports main contrib non-free-firmware deb-src http://deb.debian.org/debian trixie-backports main contrib non-free-firmware :: vi /etc/apt/preferences.d/90_zfs .. code-block:: control Package: src:zfs-linux Pin: release n=trixie-backports Pin-Priority: 990 Install the packages:: apt update apt install linux-headers-$(dpkg --print-architecture) apt install zfs-dkms zfsutils-linux **Caution**: If you are in a poorly configured environment (e.g. certain VM or container consoles), when apt attempts to pop up a message on first install, it may fail to notice a real console is unavailable, and instead appear to hang indefinitely. To circumvent this, you can prefix the `apt install` commands with ``DEBIAN_FRONTEND=noninteractive``, like this:: DEBIAN_FRONTEND=noninteractive apt install zfs-dkms zfsutils-linux Root on ZFS ----------- .. toctree:: :maxdepth: 1 :glob: *Root on ZFS Related topics -------------- .. toctree:: :maxdepth: 1 Debian GNU Linux initrd documentation .. highlight:: sh Debian Bookworm Root on ZFS =========================== .. contents:: Table of Contents :local: Overview -------- Newer release available ~~~~~~~~~~~~~~~~~~~~~~~ - See :doc:`Debian Trixie Root on ZFS <./Debian Trixie Root on ZFS>` for new installs. This guide continues to exist for reference for existing installs that followed it. Caution ~~~~~~~ - This HOWTO uses a whole physical disk. - Do not use these instructions for dual-booting. - Backup your data. Any existing data will be lost. System Requirements ~~~~~~~~~~~~~~~~~~~ - `64-bit Debian GNU/Linux Bookworm Live CD w/ GUI (e.g. gnome iso) `__ - :ref:`A 64-bit kernel is strongly encouraged. <32-bit-vs-64-bit-systems>` - Installing on a drive which presents 4 KiB logical sectors (a “4Kn” drive) only works with UEFI booting. This is not unique to ZFS. `GRUB does not and will not work on 4Kn with legacy (BIOS) booting. `__ Computers that have less than 2 GiB of memory run ZFS slowly. 4 GiB of memory is recommended for normal performance in basic workloads. If you wish to use deduplication, you will need `massive amounts of RAM `__. Enabling deduplication is a permanent change that cannot be easily reverted. Support ~~~~~~~ If you need help, reach out to the community using the :ref:`mailing_lists` or IRC at `#zfsonlinux `__ on `Libera Chat `__. If you have a bug report or feature request related to this HOWTO, please `file a new issue and mention @rlaager `__. Contributing ~~~~~~~~~~~~ #. Fork and clone: https://github.com/openzfs/openzfs-docs #. Install the tools:: sudo apt install python3-pip pip3 install -r docs/requirements.txt # Add ~/.local/bin to your $PATH, e.g. by adding this to ~/.bashrc: PATH=$HOME/.local/bin:$PATH #. Make your changes. #. Test:: cd docs make html sensible-browser _build/html/index.html #. ``git commit --signoff`` to a branch, ``git push``, and create a pull request. Mention @rlaager. Encryption ~~~~~~~~~~ This guide supports three different encryption options: unencrypted, ZFS native encryption, and LUKS. With any option, all ZFS features are fully available. Unencrypted does not encrypt anything, of course. With no encryption happening, this option naturally has the best performance. ZFS native encryption encrypts the data and most metadata in the root pool. It does not encrypt dataset or snapshot names or properties. The boot pool is not encrypted at all, but it only contains the bootloader, kernel, and initrd. (Unless you put a password in ``/etc/fstab``, the initrd is unlikely to contain sensitive data.) The system cannot boot without the passphrase being entered at the console. Performance is good. As the encryption happens in ZFS, even if multiple disks (mirror or raidz topologies) are used, the data only has to be encrypted once. LUKS encrypts almost everything. The only unencrypted data is the bootloader, kernel, and initrd. The system cannot boot without the passphrase being entered at the console. Performance is good, but LUKS sits underneath ZFS, so if multiple disks (mirror or raidz topologies) are used, the data has to be encrypted once per disk. Step 1: Prepare The Install Environment --------------------------------------- #. Boot the Debian GNU/Linux Live CD. If prompted, login with the username ``user`` and password ``live``. Connect your system to the Internet as appropriate (e.g. join your WiFi network). Open a terminal. #. Setup and update the repositories:: sudo vi /etc/apt/sources.list .. code-block:: sourceslist deb http://deb.debian.org/debian bookworm main contrib non-free-firmware :: sudo apt update #. Optional: Install and start the OpenSSH server in the Live CD environment: If you have a second system, using SSH to access the target system can be convenient:: sudo apt install --yes openssh-server sudo systemctl restart ssh **Hint:** You can find your IP address with ``ip addr show scope global | grep inet``. Then, from your main machine, connect with ``ssh user@IP``. #. Disable automounting: If the disk has been used before (with partitions at the same offsets), previous filesystems (e.g. the ESP) will automount if not disabled:: gsettings set org.gnome.desktop.media-handling automount false #. Become root:: sudo -i #. Install ZFS in the Live CD environment:: apt install --yes debootstrap gdisk zfsutils-linux Step 2: Disk Formatting ----------------------- #. Set a variable with the disk name:: DISK=/dev/disk/by-id/scsi-SATA_disk1 Always use the long ``/dev/disk/by-id/*`` aliases with ZFS. Using the ``/dev/sd*`` device nodes directly can cause sporadic import failures, especially on systems that have more than one storage pool. **Hints:** - ``ls -la /dev/disk/by-id`` will list the aliases. - Are you doing this in a virtual machine? If your virtual disk is missing from ``/dev/disk/by-id``, use ``/dev/vda`` if you are using KVM with virtio. Also when using /dev/vda, the partitions used later will be named differently. Otherwise, read the `troubleshooting <#troubleshooting>`__ section. - For a mirror or raidz topology, use ``DISK1``, ``DISK2``, etc. - When choosing a boot pool size, consider how you will use the space. A kernel and initrd may consume around 100M. If you have multiple kernels and take snapshots, you may find yourself low on boot pool space, especially if you need to regenerate your initramfs images, which may be around 85M each. Size your boot pool appropriately for your needs. #. If you are re-using a disk, clear it as necessary: Ensure swap partitions are not in use:: swapoff --all If the disk was previously used in an MD array:: apt install --yes mdadm # See if one or more MD arrays are active: cat /proc/mdstat # If so, stop them (replace ``md0`` as required): mdadm --stop /dev/md0 # For an array using the whole disk: mdadm --zero-superblock --force $DISK # For an array using a partition: mdadm --zero-superblock --force ${DISK}-part2 If the disk was previously used with zfs:: wipefs -a $DISK For flash-based storage, if the disk was previously used, you may wish to do a full-disk discard (TRIM/UNMAP), which can improve performance:: blkdiscard -f $DISK Clear the partition table:: sgdisk --zap-all $DISK If you get a message about the kernel still using the old partition table, you can request the kernel reload the partition information using:: partprobe $DISK If the new partitions still don't show up, you can reboot and start over (except that you can skip this step). #. Partition your disk(s): Run this if you need legacy (BIOS) booting:: sgdisk -a1 -n1:24K:+1000K -t1:EF02 $DISK Run this for UEFI booting (for use now or in the future):: sgdisk -n2:1M:+512M -t2:EF00 $DISK Run this for the boot pool:: sgdisk -n3:0:+1G -t3:BF01 $DISK Choose one of the following options: - Unencrypted or ZFS native encryption:: sgdisk -n4:0:0 -t4:BF00 $DISK - LUKS:: sgdisk -n4:0:0 -t4:8309 $DISK If you are creating a mirror or raidz topology, repeat the partitioning commands for all the disks which will be part of the pool. #. Create the boot pool:: zpool create \ -o ashift=12 \ -o autotrim=on \ -o compatibility=grub2 \ -o cachefile=/etc/zfs/zpool.cache \ -O devices=off \ -O acltype=posixacl -O xattr=sa \ -O compression=lz4 \ -O normalization=formD \ -O relatime=on \ -O canmount=off -O mountpoint=/boot -R /mnt \ bpool ${DISK}-part3 *Note:* GRUB does not support all zpool features (see ``spa_feature_names`` in `grub-core/fs/zfs/zfs.c `_). We create a separate zpool for ``/boot`` here, specifying the ``-o compatibility=grub2`` property which restricts the pool to only those features that GRUB supports, allowing the root pool to use any/all features. See the section on ``Compatibility feature sets`` in the ``zpool-features`` man page for more information. **Hints:** - If you are creating a mirror topology, create the pool using:: zpool create \ ... \ bpool mirror \ /dev/disk/by-id/scsi-SATA_disk1-part3 \ /dev/disk/by-id/scsi-SATA_disk2-part3 - For raidz topologies, replace ``mirror`` in the above command with ``raidz``, ``raidz2``, or ``raidz3`` and list the partitions from the additional disks. - The pool name is arbitrary. If changed, the new name must be used consistently. The ``bpool`` convention originated in this HOWTO. #. Create the root pool: Choose one of the following options: - Unencrypted:: zpool create \ -o ashift=12 \ -o autotrim=on \ -O acltype=posixacl -O xattr=sa -O dnodesize=auto \ -O compression=lz4 \ -O normalization=formD \ -O relatime=on \ -O canmount=off -O mountpoint=/ -R /mnt \ rpool ${DISK}-part4 - ZFS native encryption:: zpool create \ -o ashift=12 \ -o autotrim=on \ -O encryption=on -O keylocation=prompt -O keyformat=passphrase \ -O acltype=posixacl -O xattr=sa -O dnodesize=auto \ -O compression=lz4 \ -O normalization=formD \ -O relatime=on \ -O canmount=off -O mountpoint=/ -R /mnt \ rpool ${DISK}-part4 - LUKS:: apt install --yes cryptsetup cryptsetup luksFormat -c aes-xts-plain64 -s 512 -h sha256 ${DISK}-part4 cryptsetup luksOpen ${DISK}-part4 luks1 zpool create \ -o ashift=12 \ -o autotrim=on \ -O acltype=posixacl -O xattr=sa -O dnodesize=auto \ -O compression=lz4 \ -O normalization=formD \ -O relatime=on \ -O canmount=off -O mountpoint=/ -R /mnt \ rpool /dev/mapper/luks1 **Notes:** - The use of ``ashift=12`` is recommended here because many drives today have 4 KiB (or larger) physical sectors, even though they present 512 B logical sectors. Also, a future replacement drive may have 4 KiB physical sectors (in which case ``ashift=12`` is desirable) or 4 KiB logical sectors (in which case ``ashift=12`` is required). - Setting ``-O acltype=posixacl`` enables POSIX ACLs globally. If you do not want this, remove that option, but later add ``-o acltype=posixacl`` (note: lowercase “o”) to the ``zfs create`` for ``/var/log``, as `journald requires ACLs `__ - Setting ``xattr=sa`` `vastly improves the performance of extended attributes `__. Inside ZFS, extended attributes are used to implement POSIX ACLs. Extended attributes can also be used by user-space applications. `They are used by some desktop GUI applications. `__ `They can be used by Samba to store Windows ACLs and DOS attributes; they are required for a Samba Active Directory domain controller. `__ Note that ``xattr=sa`` is `Linux-specific `__. If you move your ``xattr=sa`` pool to another OpenZFS implementation besides ZFS-on-Linux, extended attributes will not be readable (though your data will be). If portability of extended attributes is important to you, omit the ``-O xattr=sa`` above. Even if you do not want ``xattr=sa`` for the whole pool, it is probably fine to use it for ``/var/log``. - Setting ``normalization=formD`` eliminates some corner cases relating to UTF-8 filename normalization. It also implies ``utf8only=on``, which means that only UTF-8 filenames are allowed. If you care to support non-UTF-8 filenames, do not use this option. For a discussion of why requiring UTF-8 filenames may be a bad idea, see `The problems with enforced UTF-8 only filenames `__. - ``recordsize`` is unset (leaving it at the default of 128 KiB). If you want to tune it (e.g. ``-O recordsize=1M``), see `these `__ `various `__ `blog `__ `posts `__. - Setting ``relatime=on`` is a middle ground between classic POSIX ``atime`` behavior (with its significant performance impact) and ``atime=off`` (which provides the best performance by completely disabling atime updates). Since Linux 2.6.30, ``relatime`` has been the default for other filesystems. See `RedHat’s documentation `__ for further information. - Make sure to include the ``-part4`` portion of the drive path. If you forget that, you are specifying the whole disk, which ZFS will then re-partition, and you will lose the bootloader partition(s). - ZFS native encryption `now `__ defaults to ``aes-256-gcm``. - For LUKS, the key size chosen is 512 bits. However, XTS mode requires two keys, so the LUKS key is split in half. Thus, ``-s 512`` means AES-256. - Your passphrase will likely be the weakest link. Choose wisely. See `section 5 of the cryptsetup FAQ `__ for guidance. **Hints:** - If you are creating a mirror topology, create the pool using:: zpool create \ ... \ rpool mirror \ /dev/disk/by-id/scsi-SATA_disk1-part4 \ /dev/disk/by-id/scsi-SATA_disk2-part4 - For raidz topologies, replace ``mirror`` in the above command with ``raidz``, ``raidz2``, or ``raidz3`` and list the partitions from the additional disks. - When using LUKS with mirror or raidz topologies, use ``/dev/mapper/luks1``, ``/dev/mapper/luks2``, etc., which you will have to create using ``cryptsetup``. - The pool name is arbitrary. If changed, the new name must be used consistently. On systems that can automatically install to ZFS, the root pool is named ``rpool`` by default. Step 3: System Installation --------------------------- #. Create filesystem datasets to act as containers:: zfs create -o canmount=off -o mountpoint=none rpool/ROOT zfs create -o canmount=off -o mountpoint=none bpool/BOOT On Solaris systems, the root filesystem is cloned and the suffix is incremented for major system changes through ``pkg image-update`` or ``beadm``. Similar functionality was implemented in Ubuntu with the ``zsys`` tool, though its dataset layout is more complicated, and ``zsys`` `is on life support `__. Even without such a tool, the `rpool/ROOT` and `bpool/BOOT` containers can still be used for manually created clones. That said, this HOWTO assumes a single filesystem for ``/boot`` for simplicity. #. Create filesystem datasets for the root and boot filesystems:: zfs create -o canmount=noauto -o mountpoint=/ rpool/ROOT/debian zfs mount rpool/ROOT/debian zfs create -o mountpoint=/boot bpool/BOOT/debian With ZFS, it is not normally necessary to use a mount command (either ``mount`` or ``zfs mount``). This situation is an exception because of ``canmount=noauto``. #. Create datasets:: zfs create rpool/home zfs create -o mountpoint=/root rpool/home/root chmod 700 /mnt/root zfs create -o canmount=off rpool/var zfs create -o canmount=off rpool/var/lib zfs create rpool/var/log zfs create rpool/var/spool The datasets below are optional, depending on your preferences and/or software choices. If you wish to separate these to exclude them from snapshots:: zfs create -o com.sun:auto-snapshot=false rpool/var/cache zfs create -o com.sun:auto-snapshot=false rpool/var/lib/nfs zfs create -o com.sun:auto-snapshot=false rpool/var/tmp chmod 1777 /mnt/var/tmp If you use /srv on this system:: zfs create rpool/srv If you use /usr/local on this system:: zfs create -o canmount=off rpool/usr zfs create rpool/usr/local If this system will have games installed:: zfs create rpool/var/games If this system will have a GUI:: zfs create rpool/var/lib/AccountsService zfs create rpool/var/lib/NetworkManager If this system will use Docker (which manages its own datasets & snapshots):: zfs create -o com.sun:auto-snapshot=false rpool/var/lib/docker If this system will store local email in /var/mail:: zfs create rpool/var/mail If this system will use Snap packages:: zfs create rpool/var/snap If you use /var/www on this system:: zfs create rpool/var/www A tmpfs is recommended later, but if you want a separate dataset for ``/tmp``:: zfs create -o com.sun:auto-snapshot=false rpool/tmp chmod 1777 /mnt/tmp The primary goal of this dataset layout is to separate the OS from user data. This allows the root filesystem to be rolled back without rolling back user data. If you do nothing extra, ``/tmp`` will be stored as part of the root filesystem. Alternatively, you can create a separate dataset for ``/tmp``, as shown above. This keeps the ``/tmp`` data out of snapshots of your root filesystem. It also allows you to set a quota on ``rpool/tmp``, if you want to limit the maximum space used. Otherwise, you can use a tmpfs (RAM filesystem) later. **Note:** If you separate a directory required for booting (e.g. ``/etc``) into its own dataset, you must add it to ``ZFS_INITRD_ADDITIONAL_DATASETS`` in ``/etc/default/zfs``. Datasets with ``canmount=off`` (like ``rpool/usr`` above) do not matter for this. #. Mount a tmpfs at /run:: mkdir /mnt/run mount -t tmpfs tmpfs /mnt/run mkdir /mnt/run/lock #. Install the minimal system:: debootstrap bookworm /mnt The ``debootstrap`` command leaves the new system in an unconfigured state. An alternative to using ``debootstrap`` is to copy the entirety of a working system into the new ZFS root. #. Copy in zpool.cache:: mkdir /mnt/etc/zfs cp /etc/zfs/zpool.cache /mnt/etc/zfs/ Step 4: System Configuration ---------------------------- #. Configure the hostname: Replace ``HOSTNAME`` with the desired hostname:: hostname HOSTNAME hostname > /mnt/etc/hostname vi /mnt/etc/hosts .. code-block:: text Add a line: 127.0.1.1 HOSTNAME or if the system has a real name in DNS: 127.0.1.1 FQDN HOSTNAME **Hint:** Use ``nano`` if you find ``vi`` confusing. #. Configure the network interface: Find the interface name:: ip addr show Adjust ``NAME`` below to match your interface name:: vi /mnt/etc/network/interfaces.d/NAME .. code-block:: text auto NAME iface NAME inet dhcp Customize this file if the system is not a DHCP client. #. Configure the package sources:: vi /mnt/etc/apt/sources.list .. code-block:: sourceslist deb http://deb.debian.org/debian bookworm main contrib non-free-firmware deb-src http://deb.debian.org/debian bookworm main contrib non-free-firmware deb http://deb.debian.org/debian-security bookworm-security main contrib non-free-firmware deb-src http://deb.debian.org/debian-security bookworm-security main contrib non-free-firmware deb http://deb.debian.org/debian bookworm-updates main contrib non-free-firmware deb-src http://deb.debian.org/debian bookworm-updates main contrib non-free-firmware #. Bind the virtual filesystems from the LiveCD environment to the new system and ``chroot`` into it:: mount --make-private --rbind /dev /mnt/dev mount --make-private --rbind /proc /mnt/proc mount --make-private --rbind /sys /mnt/sys chroot /mnt /usr/bin/env DISK=$DISK bash --login **Note:** This is using ``--rbind``, not ``--bind``. #. Configure a basic system environment:: apt update apt install --yes console-setup locales Even if you prefer a non-English system language, always ensure that ``en_US.UTF-8`` is available:: dpkg-reconfigure locales tzdata keyboard-configuration console-setup #. Optional: Install driver firmware and WiFi support If you're installing on a laptop or a device where wireless is the primary network option, the network configuration above may not be sufficient as you could lack the appropriate firmware for the device and tools to configure the radio. Install some additional packages to cover that need:: apt install --yes firmware-linux wireless-tools #. Install ZFS in the chroot environment for the new system:: apt install --yes dpkg-dev linux-headers-generic linux-image-generic apt install --yes zfs-initramfs **Note:** Ignore any error messages saying ``ERROR: Couldn't resolve device`` and ``WARNING: Couldn't determine root device``. `cryptsetup does not support ZFS `__. #. For LUKS installs only, setup ``/etc/crypttab``:: apt install --yes cryptsetup cryptsetup-initramfs echo luks1 /dev/disk/by-uuid/$(blkid -s UUID -o value ${DISK}-part4) \ none luks,discard,initramfs > /etc/crypttab The use of ``initramfs`` is a work-around for `cryptsetup does not support ZFS `__. **Hint:** If you are creating a mirror or raidz topology, repeat the ``/etc/crypttab`` entries for ``luks2``, etc. adjusting for each disk. #. Install an NTP service to synchronize time. This step is specific to Bookworm which does not install the package during bootstrap. Although this step is not necessary for ZFS, it is useful for internet browsing where local clock drift can cause login failures:: apt install systemd-timesyncd #. Install GRUB Choose one of the following options: - Install GRUB for legacy (BIOS) booting:: apt install --yes grub-pc - Install GRUB for UEFI booting:: apt install dosfstools mkdosfs -F 32 -s 1 -n EFI ${DISK}-part2 mkdir /boot/efi echo /dev/disk/by-uuid/$(blkid -s UUID -o value ${DISK}-part2) \ /boot/efi vfat defaults 0 0 >> /etc/fstab mount /boot/efi apt install --yes grub-efi-amd64 shim-signed **Notes:** - The ``-s 1`` for ``mkdosfs`` is only necessary for drives which present 4 KiB logical sectors (“4Kn” drives) to meet the minimum cluster size (given the partition size of 512 MiB) for FAT32. It also works fine on drives which present 512 B sectors. - For a mirror or raidz topology, this step only installs GRUB on the first disk. The other disk(s) will be handled later. #. Optional: Remove os-prober:: apt purge --yes os-prober This avoids error messages from `update-grub`. `os-prober` is only necessary in dual-boot configurations. #. Set a root password:: passwd #. Enable importing bpool This ensures that ``bpool`` is always imported, regardless of whether ``/etc/zfs/zpool.cache`` exists, whether it is in the cachefile or not, or whether ``zfs-import-scan.service`` is enabled. :: vi /etc/systemd/system/zfs-import-bpool.service .. code-block:: ini [Unit] DefaultDependencies=no Before=zfs-import-scan.service Before=zfs-import-cache.service [Service] Type=oneshot RemainAfterExit=yes ExecStart=/sbin/zpool import -N -o cachefile=none bpool # Work-around to preserve zpool cache: ExecStartPre=-/bin/mv /etc/zfs/zpool.cache /etc/zfs/preboot_zpool.cache ExecStartPost=-/bin/mv /etc/zfs/preboot_zpool.cache /etc/zfs/zpool.cache [Install] WantedBy=zfs-import.target :: systemctl enable zfs-import-bpool.service **Note:** For some disk configurations (NVMe?), this service `may fail `__ with an error indicating that the ``bpool`` cannot be found. If this happens, add ``-d DISK-part3`` (replace ``DISK`` with the correct device path) to the ``zpool import`` command. #. Optional (but recommended): Mount a tmpfs to ``/tmp`` If you chose to create a ``/tmp`` dataset above, skip this step, as they are mutually exclusive choices. Otherwise, you can put ``/tmp`` on a tmpfs (RAM filesystem) by enabling the ``tmp.mount`` unit. :: cp /usr/share/systemd/tmp.mount /etc/systemd/system/ systemctl enable tmp.mount #. Optional: Install SSH:: apt install --yes openssh-server vi /etc/ssh/sshd_config # Set: PermitRootLogin yes #. Optional: For ZFS native encryption or LUKS, configure Dropbear for remote unlocking:: apt install --yes --no-install-recommends dropbear-initramfs mkdir -p /etc/dropbear/initramfs # Optional: Convert OpenSSH server keys for Dropbear for type in ecdsa ed25519 rsa ; do cp /etc/ssh/ssh_host_${type}_key /tmp/openssh.key ssh-keygen -p -N "" -m PEM -f /tmp/openssh.key dropbearconvert openssh dropbear \ /tmp/openssh.key \ /etc/dropbear/initramfs/dropbear_${type}_host_key done rm /tmp/openssh.key # Add user keys in the same format as ~/.ssh/authorized_keys vi /etc/dropbear/initramfs/authorized_keys # If using a static IP, set it for the initramfs environment: vi /etc/initramfs-tools/initramfs.conf # The syntax is: IP=ADDRESS::GATEWAY:MASK:HOSTNAME:NIC # For example: # IP=192.168.1.100::192.168.1.1:255.255.255.0:myhostname:ens3 # HOSTNAME and NIC are optional. # Rebuild the initramfs (required when changing any of the above): update-initramfs -u -k all **Notes:** - Converting the server keys makes Dropbear use the same keys as OpenSSH, avoiding host key mismatch warnings. Currently, `dropbearconvert doesn't understand the new OpenSSH private key format `__, so the keys need to be converted to the old PEM format first using ``ssh-keygen``. The downside of using the same keys for both OpenSSH and Dropbear is that the OpenSSH keys are then available on-disk, unencrypted in the initramfs. - Later, to use this functionality, SSH to the system (as root) while it is prompting for the passphrase during the boot process. For ZFS native encryption, run ``zfsunlock``. For LUKS, run ``cryptroot-unlock``. - You can optionally add ``command="/usr/bin/zfsunlock"`` or ``command="/bin/cryptroot-unlock"`` in front of the ``authorized_keys`` line to force the unlock command. This way, the unlock command runs automatically and is all that can be run. #. Optional (but kindly requested): Install popcon The ``popularity-contest`` package reports the list of packages installed on your system. Showing that ZFS is popular may be helpful in terms of long-term attention from the distro. :: apt install --yes popularity-contest Choose Yes at the prompt. Step 5: GRUB Installation ------------------------- #. Verify that the ZFS boot filesystem is recognized:: grub-probe /boot #. Refresh the initrd files:: update-initramfs -c -k all **Note:** Ignore any error messages saying ``ERROR: Couldn't resolve device`` and ``WARNING: Couldn't determine root device``. `cryptsetup does not support ZFS `__. #. Workaround GRUB's missing zpool-features support:: vi /etc/default/grub # Set: GRUB_CMDLINE_LINUX="root=ZFS=rpool/ROOT/debian" #. Optional (but highly recommended): Make debugging GRUB easier:: vi /etc/default/grub # Remove quiet from: GRUB_CMDLINE_LINUX_DEFAULT # Uncomment: GRUB_TERMINAL=console # Save and quit. Later, once the system has rebooted twice and you are sure everything is working, you can undo these changes, if desired. #. Update the boot configuration:: update-grub **Note:** Ignore errors from ``osprober``, if present. #. Install the boot loader: Choose one of the following options: - For legacy (BIOS) booting, install GRUB to the MBR:: grub-install $DISK Note that you are installing GRUB to the whole disk, not a partition. If you are creating a mirror or raidz topology, repeat the ``grub-install`` command for each disk in the pool. - For UEFI booting, install GRUB to the ESP:: grub-install --target=x86_64-efi --efi-directory=/boot/efi \ --bootloader-id=debian --recheck It is not necessary to specify the disk here. If you are creating a mirror or raidz topology, the additional disks will be handled later. #. Fix filesystem mount ordering: We need to activate ``zfs-mount-generator``. This makes systemd aware of the separate mountpoints, which is important for things like ``/var/log`` and ``/var/tmp``. In turn, ``rsyslog.service`` depends on ``var-log.mount`` by way of ``local-fs.target`` and services using the ``PrivateTmp`` feature of systemd automatically use ``After=var-tmp.mount``. :: mkdir /etc/zfs/zfs-list.cache touch /etc/zfs/zfs-list.cache/bpool touch /etc/zfs/zfs-list.cache/rpool zed -F & Verify that ``zed`` updated the cache by making sure these are not empty:: cat /etc/zfs/zfs-list.cache/bpool cat /etc/zfs/zfs-list.cache/rpool If either is empty, force a cache update and check again:: zfs set canmount=on bpool/BOOT/debian zfs set canmount=noauto rpool/ROOT/debian If they are still empty, stop zed (as below), start zed (as above) and try again. Once the files have data, stop ``zed``:: fg Press Ctrl-C. Fix the paths to eliminate ``/mnt``:: sed -Ei "s|/mnt/?|/|" /etc/zfs/zfs-list.cache/* Step 6: First Boot ------------------ #. Optional: Snapshot the initial installation:: zfs snapshot bpool/BOOT/debian@install zfs snapshot rpool/ROOT/debian@install In the future, you will likely want to take snapshots before each upgrade, and remove old snapshots (including this one) at some point to save space. #. Exit from the ``chroot`` environment back to the LiveCD environment:: exit #. Run these commands in the LiveCD environment to unmount all filesystems:: mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | \ xargs -i{} umount -lf {} zpool export -a #. If export failed due to `busy` error, try to kill everything that might be using it:: grep [p]ool /proc/*/mounts | cut -d/ -f3 | uniq | xargs kill zpool export -a #. If even after that your pool is busy, mounting it on boot will fail and you will need to ``zpool import -f rpool``, then ``exit`` in the initramfs prompt. #. Reboot:: reboot Wait for the newly installed system to boot normally. Login as root. #. Create a user account: Replace ``YOUR_USERNAME`` with your desired username:: username=YOUR_USERNAME zfs create rpool/home/$username adduser $username cp -a /etc/skel/. /home/$username chown -R $username:$username /home/$username usermod -a -G audio,cdrom,dip,floppy,netdev,plugdev,sudo,video $username #. Mirror GRUB If you installed to multiple disks, install GRUB on the additional disks. - For legacy (BIOS) booting:: dpkg-reconfigure grub-pc Hit enter until you get to the device selection screen. Select (using the space bar) all of the disks (not partitions) in your pool. - For UEFI booting:: umount /boot/efi For the second and subsequent disks (increment debian-2 to -3, etc.):: dd if=/dev/disk/by-id/scsi-SATA_disk1-part2 \ of=/dev/disk/by-id/scsi-SATA_disk2-part2 efibootmgr -c -g -d /dev/disk/by-id/scsi-SATA_disk2 \ -p 2 -L "debian-2" -l '\EFI\debian\grubx64.efi' mount /boot/efi Step 7: Optional: Configure Swap --------------------------------- **Caution**: On systems with extremely high memory pressure, using a zvol for swap can result in lockup, regardless of how much swap is still available. There is `a bug report upstream `__. #. Create a volume dataset (zvol) for use as a swap device:: zfs create -V 4G -b $(getconf PAGESIZE) -o compression=zle \ -o logbias=throughput -o sync=always \ -o primarycache=metadata -o secondarycache=none \ -o com.sun:auto-snapshot=false rpool/swap You can adjust the size (the ``4G`` part) to your needs. The compression algorithm is set to ``zle`` because it is the cheapest available algorithm. As this guide recommends ``ashift=12`` (4 kiB blocks on disk), the common case of a 4 kiB page size means that no compression algorithm can reduce I/O. The exception is all-zero pages, which are dropped by ZFS; but some form of compression has to be enabled to get this behavior. #. Configure the swap device: **Caution**: Always use long ``/dev/zvol`` aliases in configuration files. Never use a short ``/dev/zdX`` device name. :: mkswap -f /dev/zvol/rpool/swap echo /dev/zvol/rpool/swap none swap discard 0 0 >> /etc/fstab echo RESUME=none > /etc/initramfs-tools/conf.d/resume The ``RESUME=none`` is necessary to disable resuming from hibernation. This does not work, as the zvol is not present (because the pool has not yet been imported) at the time the resume script runs. If it is not disabled, the boot process hangs for 30 seconds waiting for the swap zvol to appear. #. Enable the swap device:: swapon -av Step 8: Full Software Installation ---------------------------------- #. Upgrade the minimal system:: apt dist-upgrade --yes #. Install a regular set of software:: tasksel --new-install **Note:** This will check "Debian desktop environment" and "print server" by default. If you want a server installation, unselect those. #. Optional: Disable log compression: As ``/var/log`` is already compressed by ZFS, logrotate’s compression is going to burn CPU and disk I/O for (in most cases) very little gain. Also, if you are making snapshots of ``/var/log``, logrotate’s compression will actually waste space, as the uncompressed data will live on in the snapshot. You can edit the files in ``/etc/logrotate.d`` by hand to comment out ``compress``, or use this loop (copy-and-paste highly recommended):: for file in /etc/logrotate.d/* ; do if grep -Eq "(^|[^#y])compress" "$file" ; then sed -i -r "s/(^|[^#y])(compress)/\1#\2/" "$file" fi done #. Reboot:: reboot Step 9: Final Cleanup --------------------- #. Wait for the system to boot normally. Login using the account you created. Ensure the system (including networking) works normally. #. Optional: Delete the snapshots of the initial installation:: sudo zfs destroy bpool/BOOT/debian@install sudo zfs destroy rpool/ROOT/debian@install #. Optional: Disable the root password:: sudo usermod -p '*' root #. Optional (but highly recommended): Disable root SSH logins: If you installed SSH earlier, revert the temporary change:: sudo vi /etc/ssh/sshd_config # Remove: PermitRootLogin yes sudo systemctl restart ssh #. Optional: Re-enable the graphical boot process: If you prefer the graphical boot process, you can re-enable it now. If you are using LUKS, it makes the prompt look nicer. :: sudo vi /etc/default/grub # Add quiet to GRUB_CMDLINE_LINUX_DEFAULT # Comment out GRUB_TERMINAL=console # Save and quit. sudo update-grub **Note:** Ignore errors from ``osprober``, if present. #. Optional: For LUKS installs only, backup the LUKS header:: sudo cryptsetup luksHeaderBackup /dev/disk/by-id/scsi-SATA_disk1-part4 \ --header-backup-file luks1-header.dat Store that backup somewhere safe (e.g. cloud storage). It is protected by your LUKS passphrase, but you may wish to use additional encryption. **Hint:** If you created a mirror or raidz topology, repeat this for each LUKS volume (``luks2``, etc.). Troubleshooting --------------- Rescuing using a Live CD ~~~~~~~~~~~~~~~~~~~~~~~~ Go through `Step 1: Prepare The Install Environment <#step-1-prepare-the-install-environment>`__. For LUKS, first unlock the disk(s):: apt install --yes cryptsetup cryptsetup luksOpen /dev/disk/by-id/scsi-SATA_disk1-part4 luks1 # Repeat for additional disks, if this is a mirror or raidz topology. Mount everything correctly:: zpool export -a zpool import -N -R /mnt rpool zpool import -N -R /mnt bpool zfs load-key -a zfs mount rpool/ROOT/debian zfs mount -a If needed, you can chroot into your installed environment:: mount --make-private --rbind /dev /mnt/dev mount --make-private --rbind /proc /mnt/proc mount --make-private --rbind /sys /mnt/sys mount -t tmpfs tmpfs /mnt/run mkdir /mnt/run/lock chroot /mnt /bin/bash --login mount /boot/efi mount -a Do whatever you need to do to fix your system. When done, cleanup:: exit mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | \ xargs -i{} umount -lf {} zpool export -a reboot Areca ~~~~~ Systems that require the ``arcsas`` blob driver should add it to the ``/etc/initramfs-tools/modules`` file and run ``update-initramfs -c -k all``. Upgrade or downgrade the Areca driver if something like ``RIP: 0010:[] [] native_read_tsc+0x6/0x20`` appears anywhere in kernel log. ZoL is unstable on systems that emit this error message. MPT2SAS ~~~~~~~ Most problem reports for this tutorial involve ``mpt2sas`` hardware that does slow asynchronous drive initialization, like some IBM M1015 or OEM-branded cards that have been flashed to the reference LSI firmware. The basic problem is that disks on these controllers are not visible to the Linux kernel until after the regular system is started, and ZoL does not hotplug pool members. See `https://github.com/openzfs/zfs/issues/330 `__. Most LSI cards are perfectly compatible with ZoL. If your card has this glitch, try setting ``ZFS_INITRD_PRE_MOUNTROOT_SLEEP=X`` in ``/etc/default/zfs``. The system will wait ``X`` seconds for all drives to appear before importing the pool. QEMU/KVM/XEN ~~~~~~~~~~~~ Set a unique serial number on each virtual disk using libvirt or qemu (e.g. ``-drive if=none,id=disk1,file=disk1.qcow2,serial=1234567890``). To be able to use UEFI in guests (instead of only BIOS booting), run this on the host:: sudo apt install ovmf sudo vi /etc/libvirt/qemu.conf Uncomment these lines: .. code-block:: text nvram = [ "/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd", "/usr/share/OVMF/OVMF_CODE.secboot.fd:/usr/share/OVMF/OVMF_VARS.fd", "/usr/share/AAVMF/AAVMF_CODE.fd:/usr/share/AAVMF/AAVMF_VARS.fd", "/usr/share/AAVMF/AAVMF32_CODE.fd:/usr/share/AAVMF/AAVMF32_VARS.fd" ] :: sudo systemctl restart libvirtd.service VMware ~~~~~~ - Set ``disk.EnableUUID = "TRUE"`` in the vmx file or vsphere configuration. Doing this ensures that ``/dev/disk`` aliases are created in the guest. .. highlight:: sh Debian Bullseye Root on ZFS =========================== .. contents:: Table of Contents :local: Overview -------- Newer release available ~~~~~~~~~~~~~~~~~~~~~~~ - See :doc:`Debian Bookworm Root on ZFS <./Debian Bookworm Root on ZFS>` for new installs. This guide is no longer receiving most updates. It continues to exist for reference for existing installs that followed it. Caution ~~~~~~~ - This HOWTO uses a whole physical disk. - Do not use these instructions for dual-booting. - Backup your data. Any existing data will be lost. System Requirements ~~~~~~~~~~~~~~~~~~~ - `64-bit Debian GNU/Linux Bullseye Live CD w/ GUI (e.g. gnome iso) `__ - :ref:`A 64-bit kernel is strongly encouraged. <32-bit-vs-64-bit-systems>` - Installing on a drive which presents 4 KiB logical sectors (a “4Kn” drive) only works with UEFI booting. This not unique to ZFS. `GRUB does not and will not work on 4Kn with legacy (BIOS) booting. `__ Computers that have less than 2 GiB of memory run ZFS slowly. 4 GiB of memory is recommended for normal performance in basic workloads. If you wish to use deduplication, you will need `massive amounts of RAM `__. Enabling deduplication is a permanent change that cannot be easily reverted. Support ~~~~~~~ If you need help, reach out to the community using the :ref:`mailing_lists` or IRC at `#zfsonlinux `__ on `Libera Chat `__. If you have a bug report or feature request related to this HOWTO, please `file a new issue and mention @rlaager `__. Contributing ~~~~~~~~~~~~ #. Fork and clone: https://github.com/openzfs/openzfs-docs #. Install the tools:: sudo apt install python3-pip pip3 install -r docs/requirements.txt # Add ~/.local/bin to your $PATH, e.g. by adding this to ~/.bashrc: PATH=$HOME/.local/bin:$PATH #. Make your changes. #. Test:: cd docs make html sensible-browser _build/html/index.html #. ``git commit --signoff`` to a branch, ``git push``, and create a pull request. Mention @rlaager. Encryption ~~~~~~~~~~ This guide supports three different encryption options: unencrypted, ZFS native encryption, and LUKS. With any option, all ZFS features are fully available. Unencrypted does not encrypt anything, of course. With no encryption happening, this option naturally has the best performance. ZFS native encryption encrypts the data and most metadata in the root pool. It does not encrypt dataset or snapshot names or properties. The boot pool is not encrypted at all, but it only contains the bootloader, kernel, and initrd. (Unless you put a password in ``/etc/fstab``, the initrd is unlikely to contain sensitive data.) The system cannot boot without the passphrase being entered at the console. Performance is good. As the encryption happens in ZFS, even if multiple disks (mirror or raidz topologies) are used, the data only has to be encrypted once. LUKS encrypts almost everything. The only unencrypted data is the bootloader, kernel, and initrd. The system cannot boot without the passphrase being entered at the console. Performance is good, but LUKS sits underneath ZFS, so if multiple disks (mirror or raidz topologies) are used, the data has to be encrypted once per disk. Step 1: Prepare The Install Environment --------------------------------------- #. Boot the Debian GNU/Linux Live CD. If prompted, login with the username ``user`` and password ``live``. Connect your system to the Internet as appropriate (e.g. join your WiFi network). Open a terminal. #. Setup and update the repositories:: sudo vi /etc/apt/sources.list .. code-block:: sourceslist deb http://deb.debian.org/debian bullseye main contrib :: sudo apt update #. Optional: Install and start the OpenSSH server in the Live CD environment: If you have a second system, using SSH to access the target system can be convenient:: sudo apt install --yes openssh-server sudo systemctl restart ssh **Hint:** You can find your IP address with ``ip addr show scope global | grep inet``. Then, from your main machine, connect with ``ssh user@IP``. #. Disable automounting: If the disk has been used before (with partitions at the same offsets), previous filesystems (e.g. the ESP) will automount if not disabled:: gsettings set org.gnome.desktop.media-handling automount false #. Become root:: sudo -i #. Install ZFS in the Live CD environment:: apt install --yes debootstrap gdisk zfsutils-linux Step 2: Disk Formatting ----------------------- #. Set a variable with the disk name:: DISK=/dev/disk/by-id/scsi-SATA_disk1 Always use the long ``/dev/disk/by-id/*`` aliases with ZFS. Using the ``/dev/sd*`` device nodes directly can cause sporadic import failures, especially on systems that have more than one storage pool. **Hints:** - ``ls -la /dev/disk/by-id`` will list the aliases. - Are you doing this in a virtual machine? If your virtual disk is missing from ``/dev/disk/by-id``, use ``/dev/vda`` if you are using KVM with virtio; otherwise, read the `troubleshooting <#troubleshooting>`__ section. - For a mirror or raidz topology, use ``DISK1``, ``DISK2``, etc. - When choosing a boot pool size, consider how you will use the space. A kernel and initrd may consume around 100M. If you have multiple kernels and take snapshots, you may find yourself low on boot pool space, especially if you need to regenerate your initramfs images, which may be around 85M each. Size your boot pool appropriately for your needs. #. If you are re-using a disk, clear it as necessary: Ensure swap partitions are not in use:: swapoff --all If the disk was previously used in an MD array:: apt install --yes mdadm # See if one or more MD arrays are active: cat /proc/mdstat # If so, stop them (replace ``md0`` as required): mdadm --stop /dev/md0 # For an array using the whole disk: mdadm --zero-superblock --force $DISK # For an array using a partition: mdadm --zero-superblock --force ${DISK}-part2 If the disk was previously used with zfs:: wipefs -a $DISK For flash-based storage, if the disk was previously used, you may wish to do a full-disk discard (TRIM/UNMAP), which can improve performance:: blkdiscard -f $DISK Clear the partition table:: sgdisk --zap-all $DISK If you get a message about the kernel still using the old partition table, you can request the kernel reload the partition information using:: partprobe $DISK If the new partitions still don't show up, you can reboot and start over (except that you can skip this step). #. Partition your disk(s): Run this if you need legacy (BIOS) booting:: sgdisk -a1 -n1:24K:+1000K -t1:EF02 $DISK Run this for UEFI booting (for use now or in the future):: sgdisk -n2:1M:+512M -t2:EF00 $DISK Run this for the boot pool:: sgdisk -n3:0:+1G -t3:BF01 $DISK Choose one of the following options: - Unencrypted or ZFS native encryption:: sgdisk -n4:0:0 -t4:BF00 $DISK - LUKS:: sgdisk -n4:0:0 -t4:8309 $DISK If you are creating a mirror or raidz topology, repeat the partitioning commands for all the disks which will be part of the pool. #. Create the boot pool:: zpool create \ -o ashift=12 \ -o autotrim=on -d \ -o cachefile=/etc/zfs/zpool.cache \ -o feature@async_destroy=enabled \ -o feature@bookmarks=enabled \ -o feature@embedded_data=enabled \ -o feature@empty_bpobj=enabled \ -o feature@enabled_txg=enabled \ -o feature@extensible_dataset=enabled \ -o feature@filesystem_limits=enabled \ -o feature@hole_birth=enabled \ -o feature@large_blocks=enabled \ -o feature@livelist=enabled \ -o feature@lz4_compress=enabled \ -o feature@spacemap_histogram=enabled \ -o feature@zpool_checkpoint=enabled \ -O devices=off \ -O acltype=posixacl -O xattr=sa \ -O compression=lz4 \ -O normalization=formD \ -O relatime=on \ -O canmount=off -O mountpoint=/boot -R /mnt \ bpool ${DISK}-part3 You should not need to customize any of the options for the boot pool. GRUB does not support all of the zpool features. See ``spa_feature_names`` in `grub-core/fs/zfs/zfs.c `__. This step creates a separate boot pool for ``/boot`` with the features limited to only those that GRUB supports, allowing the root pool to use any/all features. Note that GRUB opens the pool read-only, so all read-only compatible features are “supported” by GRUB. **Hints:** - If you are creating a mirror topology, create the pool using:: zpool create \ ... \ bpool mirror \ /dev/disk/by-id/scsi-SATA_disk1-part3 \ /dev/disk/by-id/scsi-SATA_disk2-part3 - For raidz topologies, replace ``mirror`` in the above command with ``raidz``, ``raidz2``, or ``raidz3`` and list the partitions from the additional disks. - The pool name is arbitrary. If changed, the new name must be used consistently. The ``bpool`` convention originated in this HOWTO. **Feature Notes:** - The ``allocation_classes`` feature should be safe to use. However, unless one is using it (i.e. a ``special`` vdev), there is no point to enabling it. It is extremely unlikely that someone would use this feature for a boot pool. If one cares about speeding up the boot pool, it would make more sense to put the whole pool on the faster disk rather than using it as a ``special`` vdev. - The ``device_rebuild`` feature should be safe to use (except on raidz, which it is incompatible with), but the boot pool is small, so this does not matter in practice. - The ``log_spacemap`` and ``spacemap_v2`` features have been tested and are safe to use. The boot pool is small, so these do not matter in practice. - The ``project_quota`` feature has been tested and is safe to use. This feature is extremely unlikely to matter for the boot pool. - The ``resilver_defer`` should be safe but the boot pool is small enough that it is unlikely to be necessary. - As a read-only compatible feature, the ``userobj_accounting`` feature should be compatible in theory, but in practice, GRUB can fail with an “invalid dnode type” error. This feature does not matter for ``/boot`` anyway. #. Create the root pool: Choose one of the following options: - Unencrypted:: zpool create \ -o ashift=12 \ -o autotrim=on \ -O acltype=posixacl -O xattr=sa -O dnodesize=auto \ -O compression=lz4 \ -O normalization=formD \ -O relatime=on \ -O canmount=off -O mountpoint=/ -R /mnt \ rpool ${DISK}-part4 - ZFS native encryption:: zpool create \ -o ashift=12 \ -o autotrim=on \ -O encryption=on -O keylocation=prompt -O keyformat=passphrase \ -O acltype=posixacl -O xattr=sa -O dnodesize=auto \ -O compression=lz4 \ -O normalization=formD \ -O relatime=on \ -O canmount=off -O mountpoint=/ -R /mnt \ rpool ${DISK}-part4 - LUKS:: apt install --yes cryptsetup cryptsetup luksFormat -c aes-xts-plain64 -s 512 -h sha256 ${DISK}-part4 cryptsetup luksOpen ${DISK}-part4 luks1 zpool create \ -o ashift=12 \ -o autotrim=on \ -O acltype=posixacl -O xattr=sa -O dnodesize=auto \ -O compression=lz4 \ -O normalization=formD \ -O relatime=on \ -O canmount=off -O mountpoint=/ -R /mnt \ rpool /dev/mapper/luks1 **Notes:** - The use of ``ashift=12`` is recommended here because many drives today have 4 KiB (or larger) physical sectors, even though they present 512 B logical sectors. Also, a future replacement drive may have 4 KiB physical sectors (in which case ``ashift=12`` is desirable) or 4 KiB logical sectors (in which case ``ashift=12`` is required). - Setting ``-O acltype=posixacl`` enables POSIX ACLs globally. If you do not want this, remove that option, but later add ``-o acltype=posixacl`` (note: lowercase “o”) to the ``zfs create`` for ``/var/log``, as `journald requires ACLs `__ - Setting ``xattr=sa`` `vastly improves the performance of extended attributes `__. Inside ZFS, extended attributes are used to implement POSIX ACLs. Extended attributes can also be used by user-space applications. `They are used by some desktop GUI applications. `__ `They can be used by Samba to store Windows ACLs and DOS attributes; they are required for a Samba Active Directory domain controller. `__ Note that ``xattr=sa`` is `Linux-specific `__. If you move your ``xattr=sa`` pool to another OpenZFS implementation besides ZFS-on-Linux, extended attributes will not be readable (though your data will be). If portability of extended attributes is important to you, omit the ``-O xattr=sa`` above. Even if you do not want ``xattr=sa`` for the whole pool, it is probably fine to use it for ``/var/log``. - Setting ``normalization=formD`` eliminates some corner cases relating to UTF-8 filename normalization. It also implies ``utf8only=on``, which means that only UTF-8 filenames are allowed. If you care to support non-UTF-8 filenames, do not use this option. For a discussion of why requiring UTF-8 filenames may be a bad idea, see `The problems with enforced UTF-8 only filenames `__. - ``recordsize`` is unset (leaving it at the default of 128 KiB). If you want to tune it (e.g. ``-O recordsize=1M``), see `these `__ `various `__ `blog `__ `posts `__. - Setting ``relatime=on`` is a middle ground between classic POSIX ``atime`` behavior (with its significant performance impact) and ``atime=off`` (which provides the best performance by completely disabling atime updates). Since Linux 2.6.30, ``relatime`` has been the default for other filesystems. See `RedHat’s documentation `__ for further information. - Make sure to include the ``-part4`` portion of the drive path. If you forget that, you are specifying the whole disk, which ZFS will then re-partition, and you will lose the bootloader partition(s). - ZFS native encryption `now `__ defaults to ``aes-256-gcm``. - For LUKS, the key size chosen is 512 bits. However, XTS mode requires two keys, so the LUKS key is split in half. Thus, ``-s 512`` means AES-256. - Your passphrase will likely be the weakest link. Choose wisely. See `section 5 of the cryptsetup FAQ `__ for guidance. **Hints:** - If you are creating a mirror topology, create the pool using:: zpool create \ ... \ rpool mirror \ /dev/disk/by-id/scsi-SATA_disk1-part4 \ /dev/disk/by-id/scsi-SATA_disk2-part4 - For raidz topologies, replace ``mirror`` in the above command with ``raidz``, ``raidz2``, or ``raidz3`` and list the partitions from the additional disks. - When using LUKS with mirror or raidz topologies, use ``/dev/mapper/luks1``, ``/dev/mapper/luks2``, etc., which you will have to create using ``cryptsetup``. - The pool name is arbitrary. If changed, the new name must be used consistently. On systems that can automatically install to ZFS, the root pool is named ``rpool`` by default. Step 3: System Installation --------------------------- #. Create filesystem datasets to act as containers:: zfs create -o canmount=off -o mountpoint=none rpool/ROOT zfs create -o canmount=off -o mountpoint=none bpool/BOOT On Solaris systems, the root filesystem is cloned and the suffix is incremented for major system changes through ``pkg image-update`` or ``beadm``. Similar functionality was implemented in Ubuntu with the ``zsys`` tool, though its dataset layout is more complicated, and ``zsys`` `is on life support `__. Even without such a tool, the `rpool/ROOT` and `bpool/BOOT` containers can still be used for manually created clones. That said, this HOWTO assumes a single filesystem for ``/boot`` for simplicity. #. Create filesystem datasets for the root and boot filesystems:: zfs create -o canmount=noauto -o mountpoint=/ rpool/ROOT/debian zfs mount rpool/ROOT/debian zfs create -o mountpoint=/boot bpool/BOOT/debian With ZFS, it is not normally necessary to use a mount command (either ``mount`` or ``zfs mount``). This situation is an exception because of ``canmount=noauto``. #. Create datasets:: zfs create rpool/home zfs create -o mountpoint=/root rpool/home/root chmod 700 /mnt/root zfs create -o canmount=off rpool/var zfs create -o canmount=off rpool/var/lib zfs create rpool/var/log zfs create rpool/var/spool The datasets below are optional, depending on your preferences and/or software choices. If you wish to separate these to exclude them from snapshots:: zfs create -o com.sun:auto-snapshot=false rpool/var/cache zfs create -o com.sun:auto-snapshot=false rpool/var/lib/nfs zfs create -o com.sun:auto-snapshot=false rpool/var/tmp chmod 1777 /mnt/var/tmp If you use /srv on this system:: zfs create rpool/srv If you use /usr/local on this system:: zfs create -o canmount=off rpool/usr zfs create rpool/usr/local If this system will have games installed:: zfs create rpool/var/games If this system will have a GUI:: zfs create rpool/var/lib/AccountsService zfs create rpool/var/lib/NetworkManager If this system will use Docker (which manages its own datasets & snapshots):: zfs create -o com.sun:auto-snapshot=false rpool/var/lib/docker If this system will store local email in /var/mail:: zfs create rpool/var/mail If this system will use Snap packages:: zfs create rpool/var/snap If you use /var/www on this system:: zfs create rpool/var/www A tmpfs is recommended later, but if you want a separate dataset for ``/tmp``:: zfs create -o com.sun:auto-snapshot=false rpool/tmp chmod 1777 /mnt/tmp The primary goal of this dataset layout is to separate the OS from user data. This allows the root filesystem to be rolled back without rolling back user data. If you do nothing extra, ``/tmp`` will be stored as part of the root filesystem. Alternatively, you can create a separate dataset for ``/tmp``, as shown above. This keeps the ``/tmp`` data out of snapshots of your root filesystem. It also allows you to set a quota on ``rpool/tmp``, if you want to limit the maximum space used. Otherwise, you can use a tmpfs (RAM filesystem) later. **Note:** If you separate a directory required for booting (e.g. ``/etc``) into its own dataset, you must add it to ``ZFS_INITRD_ADDITIONAL_DATASETS`` in ``/etc/default/zfs``. Datasets with ``canmount=off`` (like ``rpool/usr`` above) do not matter for this. #. Mount a tmpfs at /run:: mkdir /mnt/run mount -t tmpfs tmpfs /mnt/run mkdir /mnt/run/lock #. Install the minimal system:: debootstrap bullseye /mnt The ``debootstrap`` command leaves the new system in an unconfigured state. An alternative to using ``debootstrap`` is to copy the entirety of a working system into the new ZFS root. #. Copy in zpool.cache:: mkdir /mnt/etc/zfs cp /etc/zfs/zpool.cache /mnt/etc/zfs/ Step 4: System Configuration ---------------------------- #. Configure the hostname: Replace ``HOSTNAME`` with the desired hostname:: hostname HOSTNAME hostname > /mnt/etc/hostname vi /mnt/etc/hosts .. code-block:: text Add a line: 127.0.1.1 HOSTNAME or if the system has a real name in DNS: 127.0.1.1 FQDN HOSTNAME **Hint:** Use ``nano`` if you find ``vi`` confusing. #. Configure the network interface: Find the interface name:: ip addr show Adjust ``NAME`` below to match your interface name:: vi /mnt/etc/network/interfaces.d/NAME .. code-block:: text auto NAME iface NAME inet dhcp Customize this file if the system is not a DHCP client. #. Configure the package sources:: vi /mnt/etc/apt/sources.list .. code-block:: sourceslist deb http://deb.debian.org/debian bullseye main contrib deb-src http://deb.debian.org/debian bullseye main contrib deb http://deb.debian.org/debian-security bullseye-security main contrib deb-src http://deb.debian.org/debian-security bullseye-security main contrib deb http://deb.debian.org/debian bullseye-updates main contrib deb-src http://deb.debian.org/debian bullseye-updates main contrib #. Bind the virtual filesystems from the LiveCD environment to the new system and ``chroot`` into it:: mount --make-private --rbind /dev /mnt/dev mount --make-private --rbind /proc /mnt/proc mount --make-private --rbind /sys /mnt/sys chroot /mnt /usr/bin/env DISK=$DISK bash --login **Note:** This is using ``--rbind``, not ``--bind``. #. Configure a basic system environment:: ln -s /proc/self/mounts /etc/mtab apt update apt install --yes console-setup locales Even if you prefer a non-English system language, always ensure that ``en_US.UTF-8`` is available:: dpkg-reconfigure locales tzdata keyboard-configuration console-setup #. Install ZFS in the chroot environment for the new system:: apt install --yes dpkg-dev linux-headers-generic linux-image-generic apt install --yes zfs-initramfs echo REMAKE_INITRD=yes > /etc/dkms/zfs.conf **Note:** Ignore any error messages saying ``ERROR: Couldn't resolve device`` and ``WARNING: Couldn't determine root device``. `cryptsetup does not support ZFS `__. #. For LUKS installs only, setup ``/etc/crypttab``:: apt install --yes cryptsetup cryptsetup-initramfs echo luks1 /dev/disk/by-uuid/$(blkid -s UUID -o value ${DISK}-part4) \ none luks,discard,initramfs > /etc/crypttab The use of ``initramfs`` is a work-around for `cryptsetup does not support ZFS `__. **Hint:** If you are creating a mirror or raidz topology, repeat the ``/etc/crypttab`` entries for ``luks2``, etc. adjusting for each disk. #. Install an NTP service to synchronize time. This step is specific to Bullseye which does not install the package during bootstrap. Although this step is not necessary for ZFS, it is useful for internet browsing where local clock drift can cause login failures:: apt install systemd-timesyncd timedatectl You should now see "NTP service: active" in the above ``timedatectl`` output. #. Install GRUB Choose one of the following options: - Install GRUB for legacy (BIOS) booting:: apt install --yes grub-pc Select (using the space bar) all of the disks (not partitions) in your pool. - Install GRUB for UEFI booting:: apt install dosfstools mkdosfs -F 32 -s 1 -n EFI ${DISK}-part2 mkdir /boot/efi echo /dev/disk/by-uuid/$(blkid -s UUID -o value ${DISK}-part2) \ /boot/efi vfat defaults 0 0 >> /etc/fstab mount /boot/efi apt install --yes grub-efi-amd64 shim-signed **Notes:** - The ``-s 1`` for ``mkdosfs`` is only necessary for drives which present 4 KiB logical sectors (“4Kn” drives) to meet the minimum cluster size (given the partition size of 512 MiB) for FAT32. It also works fine on drives which present 512 B sectors. - For a mirror or raidz topology, this step only installs GRUB on the first disk. The other disk(s) will be handled later. #. Optional: Remove os-prober:: apt purge --yes os-prober This avoids error messages from `update-grub`. `os-prober` is only necessary in dual-boot configurations. #. Set a root password:: passwd #. Enable importing bpool This ensures that ``bpool`` is always imported, regardless of whether ``/etc/zfs/zpool.cache`` exists, whether it is in the cachefile or not, or whether ``zfs-import-scan.service`` is enabled. :: vi /etc/systemd/system/zfs-import-bpool.service .. code-block:: ini [Unit] DefaultDependencies=no Before=zfs-import-scan.service Before=zfs-import-cache.service [Service] Type=oneshot RemainAfterExit=yes ExecStart=/sbin/zpool import -N -o cachefile=none bpool # Work-around to preserve zpool cache: ExecStartPre=-/bin/mv /etc/zfs/zpool.cache /etc/zfs/preboot_zpool.cache ExecStartPost=-/bin/mv /etc/zfs/preboot_zpool.cache /etc/zfs/zpool.cache [Install] WantedBy=zfs-import.target :: systemctl enable zfs-import-bpool.service **Note:** For some disk configurations (NVMe?), this service `may fail `__ with an error indicating that the ``bpool`` cannot be found. If this happens, add ``-d DISK-part3`` (replace ``DISK`` with the correct device path) to the ``zpool import`` command. #. Optional (but recommended): Mount a tmpfs to ``/tmp`` If you chose to create a ``/tmp`` dataset above, skip this step, as they are mutually exclusive choices. Otherwise, you can put ``/tmp`` on a tmpfs (RAM filesystem) by enabling the ``tmp.mount`` unit. :: cp /usr/share/systemd/tmp.mount /etc/systemd/system/ systemctl enable tmp.mount #. Optional: Install SSH:: apt install --yes openssh-server vi /etc/ssh/sshd_config # Set: PermitRootLogin yes #. Optional: For ZFS native encryption or LUKS, configure Dropbear for remote unlocking:: apt install --yes --no-install-recommends dropbear-initramfs mkdir -p /etc/dropbear-initramfs # Optional: Convert OpenSSH server keys for Dropbear for type in ecdsa ed25519 rsa ; do cp /etc/ssh/ssh_host_${type}_key /tmp/openssh.key ssh-keygen -p -N "" -m PEM -f /tmp/openssh.key dropbearconvert openssh dropbear \ /tmp/openssh.key \ /etc/dropbear-initramfs/dropbear_${type}_host_key done rm /tmp/openssh.key # Add user keys in the same format as ~/.ssh/authorized_keys vi /etc/dropbear-initramfs/authorized_keys # If using a static IP, set it for the initramfs environment: vi /etc/initramfs-tools/initramfs.conf # The syntax is: IP=ADDRESS::GATEWAY:MASK:HOSTNAME:NIC # For example: # IP=192.168.1.100::192.168.1.1:255.255.255.0:myhostname:ens3 # HOSTNAME and NIC are optional. # Rebuild the initramfs (required when changing any of the above): update-initramfs -u -k all **Notes:** - Converting the server keys makes Dropbear use the same keys as OpenSSH, avoiding host key mismatch warnings. Currently, `dropbearconvert doesn't understand the new OpenSSH private key format `__, so the keys need to be converted to the old PEM format first using ``ssh-keygen``. The downside of using the same keys for both OpenSSH and Dropbear is that the OpenSSH keys are then available on-disk, unencrypted in the initramfs. - Later, to use this functionality, SSH to the system (as root) while it is prompting for the passphrase during the boot process. For ZFS native encryption, run ``zfsunlock``. For LUKS, run ``cryptroot-unlock``. - You can optionally add ``command="/usr/bin/zfsunlock"`` or ``command="/bin/cryptroot-unlock"`` in front of the ``authorized_keys`` line to force the unlock command. This way, the unlock command runs automatically and is all that can be run. #. Optional (but kindly requested): Install popcon The ``popularity-contest`` package reports the list of packages installed on your system. Showing that ZFS is popular may be helpful in terms of long-term attention from the distro. :: apt install --yes popularity-contest Choose Yes at the prompt. Step 5: GRUB Installation ------------------------- #. Verify that the ZFS boot filesystem is recognized:: grub-probe /boot #. Refresh the initrd files:: update-initramfs -c -k all **Note:** Ignore any error messages saying ``ERROR: Couldn't resolve device`` and ``WARNING: Couldn't determine root device``. `cryptsetup does not support ZFS `__. #. Workaround GRUB's missing zpool-features support:: vi /etc/default/grub # Set: GRUB_CMDLINE_LINUX="root=ZFS=rpool/ROOT/debian" #. Optional (but highly recommended): Make debugging GRUB easier:: vi /etc/default/grub # Remove quiet from: GRUB_CMDLINE_LINUX_DEFAULT # Uncomment: GRUB_TERMINAL=console # Save and quit. Later, once the system has rebooted twice and you are sure everything is working, you can undo these changes, if desired. #. Update the boot configuration:: update-grub **Note:** Ignore errors from ``osprober``, if present. #. Install the boot loader: #. For legacy (BIOS) booting, install GRUB to the MBR:: grub-install $DISK Note that you are installing GRUB to the whole disk, not a partition. If you are creating a mirror or raidz topology, repeat the ``grub-install`` command for each disk in the pool. #. For UEFI booting, install GRUB to the ESP:: grub-install --target=x86_64-efi --efi-directory=/boot/efi \ --bootloader-id=debian --recheck It is not necessary to specify the disk here. If you are creating a mirror or raidz topology, the additional disks will be handled later. #. Fix filesystem mount ordering: We need to activate ``zfs-mount-generator``. This makes systemd aware of the separate mountpoints, which is important for things like ``/var/log`` and ``/var/tmp``. In turn, ``rsyslog.service`` depends on ``var-log.mount`` by way of ``local-fs.target`` and services using the ``PrivateTmp`` feature of systemd automatically use ``After=var-tmp.mount``. :: mkdir /etc/zfs/zfs-list.cache touch /etc/zfs/zfs-list.cache/bpool touch /etc/zfs/zfs-list.cache/rpool zed -F & Verify that ``zed`` updated the cache by making sure these are not empty:: cat /etc/zfs/zfs-list.cache/bpool cat /etc/zfs/zfs-list.cache/rpool If either is empty, force a cache update and check again:: zfs set canmount=on bpool/BOOT/debian zfs set canmount=noauto rpool/ROOT/debian If they are still empty, stop zed (as below), start zed (as above) and try again. Once the files have data, stop ``zed``:: fg Press Ctrl-C. Fix the paths to eliminate ``/mnt``:: sed -Ei "s|/mnt/?|/|" /etc/zfs/zfs-list.cache/* Step 6: First Boot ------------------ #. Optional: Snapshot the initial installation:: zfs snapshot bpool/BOOT/debian@install zfs snapshot rpool/ROOT/debian@install In the future, you will likely want to take snapshots before each upgrade, and remove old snapshots (including this one) at some point to save space. #. Exit from the ``chroot`` environment back to the LiveCD environment:: exit #. Run these commands in the LiveCD environment to unmount all filesystems:: mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | \ xargs -i{} umount -lf {} zpool export -a #. If this fails for rpool, mounting it on boot will fail and you will need to ``zpool import -f rpool``, then ``exit`` in the initramfs prompt. #. Reboot:: reboot Wait for the newly installed system to boot normally. Login as root. #. Create a user account: Replace ``YOUR_USERNAME`` with your desired username:: username=YOUR_USERNAME zfs create rpool/home/$username adduser $username cp -a /etc/skel/. /home/$username chown -R $username:$username /home/$username usermod -a -G audio,cdrom,dip,floppy,netdev,plugdev,sudo,video $username #. Mirror GRUB If you installed to multiple disks, install GRUB on the additional disks. - For legacy (BIOS) booting:: dpkg-reconfigure grub-pc Hit enter until you get to the device selection screen. Select (using the space bar) all of the disks (not partitions) in your pool. - For UEFI booting:: umount /boot/efi For the second and subsequent disks (increment debian-2 to -3, etc.):: dd if=/dev/disk/by-id/scsi-SATA_disk1-part2 \ of=/dev/disk/by-id/scsi-SATA_disk2-part2 efibootmgr -c -g -d /dev/disk/by-id/scsi-SATA_disk2 \ -p 2 -L "debian-2" -l '\EFI\debian\grubx64.efi' mount /boot/efi Step 7: Optional: Configure Swap --------------------------------- **Caution**: On systems with extremely high memory pressure, using a zvol for swap can result in lockup, regardless of how much swap is still available. There is `a bug report upstream `__. #. Create a volume dataset (zvol) for use as a swap device:: zfs create -V 4G -b $(getconf PAGESIZE) -o compression=zle \ -o logbias=throughput -o sync=always \ -o primarycache=metadata -o secondarycache=none \ -o com.sun:auto-snapshot=false rpool/swap You can adjust the size (the ``4G`` part) to your needs. The compression algorithm is set to ``zle`` because it is the cheapest available algorithm. As this guide recommends ``ashift=12`` (4 kiB blocks on disk), the common case of a 4 kiB page size means that no compression algorithm can reduce I/O. The exception is all-zero pages, which are dropped by ZFS; but some form of compression has to be enabled to get this behavior. #. Configure the swap device: **Caution**: Always use long ``/dev/zvol`` aliases in configuration files. Never use a short ``/dev/zdX`` device name. :: mkswap -f /dev/zvol/rpool/swap echo /dev/zvol/rpool/swap none swap discard 0 0 >> /etc/fstab echo RESUME=none > /etc/initramfs-tools/conf.d/resume The ``RESUME=none`` is necessary to disable resuming from hibernation. This does not work, as the zvol is not present (because the pool has not yet been imported) at the time the resume script runs. If it is not disabled, the boot process hangs for 30 seconds waiting for the swap zvol to appear. #. Enable the swap device:: swapon -av Step 8: Full Software Installation ---------------------------------- #. Upgrade the minimal system:: apt dist-upgrade --yes #. Install a regular set of software:: tasksel --new-install **Note:** This will check "Debian desktop environment" and "print server" by default. If you want a server installation, unselect those. #. Optional: Disable log compression: As ``/var/log`` is already compressed by ZFS, logrotate’s compression is going to burn CPU and disk I/O for (in most cases) very little gain. Also, if you are making snapshots of ``/var/log``, logrotate’s compression will actually waste space, as the uncompressed data will live on in the snapshot. You can edit the files in ``/etc/logrotate.d`` by hand to comment out ``compress``, or use this loop (copy-and-paste highly recommended):: for file in /etc/logrotate.d/* ; do if grep -Eq "(^|[^#y])compress" "$file" ; then sed -i -r "s/(^|[^#y])(compress)/\1#\2/" "$file" fi done #. Reboot:: reboot Step 9: Final Cleanup --------------------- #. Wait for the system to boot normally. Login using the account you created. Ensure the system (including networking) works normally. #. Optional: Delete the snapshots of the initial installation:: sudo zfs destroy bpool/BOOT/debian@install sudo zfs destroy rpool/ROOT/debian@install #. Optional: Disable the root password:: sudo usermod -p '*' root #. Optional (but highly recommended): Disable root SSH logins: If you installed SSH earlier, revert the temporary change:: sudo vi /etc/ssh/sshd_config # Remove: PermitRootLogin yes sudo systemctl restart ssh #. Optional: Re-enable the graphical boot process: If you prefer the graphical boot process, you can re-enable it now. If you are using LUKS, it makes the prompt look nicer. :: sudo vi /etc/default/grub # Add quiet to GRUB_CMDLINE_LINUX_DEFAULT # Comment out GRUB_TERMINAL=console # Save and quit. sudo update-grub **Note:** Ignore errors from ``osprober``, if present. #. Optional: For LUKS installs only, backup the LUKS header:: sudo cryptsetup luksHeaderBackup /dev/disk/by-id/scsi-SATA_disk1-part4 \ --header-backup-file luks1-header.dat Store that backup somewhere safe (e.g. cloud storage). It is protected by your LUKS passphrase, but you may wish to use additional encryption. **Hint:** If you created a mirror or raidz topology, repeat this for each LUKS volume (``luks2``, etc.). Troubleshooting --------------- Rescuing using a Live CD ~~~~~~~~~~~~~~~~~~~~~~~~ Go through `Step 1: Prepare The Install Environment <#step-1-prepare-the-install-environment>`__. For LUKS, first unlock the disk(s):: apt install --yes cryptsetup cryptsetup luksOpen /dev/disk/by-id/scsi-SATA_disk1-part4 luks1 # Repeat for additional disks, if this is a mirror or raidz topology. Mount everything correctly:: zpool export -a zpool import -N -R /mnt rpool zpool import -N -R /mnt bpool zfs load-key -a zfs mount rpool/ROOT/debian zfs mount -a If needed, you can chroot into your installed environment:: mount --make-private --rbind /dev /mnt/dev mount --make-private --rbind /proc /mnt/proc mount --make-private --rbind /sys /mnt/sys mount -t tmpfs tmpfs /mnt/run mkdir /mnt/run/lock chroot /mnt /bin/bash --login mount /boot/efi mount -a Do whatever you need to do to fix your system. When done, cleanup:: exit mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | \ xargs -i{} umount -lf {} zpool export -a reboot Areca ~~~~~ Systems that require the ``arcsas`` blob driver should add it to the ``/etc/initramfs-tools/modules`` file and run ``update-initramfs -c -k all``. Upgrade or downgrade the Areca driver if something like ``RIP: 0010:[] [] native_read_tsc+0x6/0x20`` appears anywhere in kernel log. ZoL is unstable on systems that emit this error message. MPT2SAS ~~~~~~~ Most problem reports for this tutorial involve ``mpt2sas`` hardware that does slow asynchronous drive initialization, like some IBM M1015 or OEM-branded cards that have been flashed to the reference LSI firmware. The basic problem is that disks on these controllers are not visible to the Linux kernel until after the regular system is started, and ZoL does not hotplug pool members. See `https://github.com/openzfs/zfs/issues/330 `__. Most LSI cards are perfectly compatible with ZoL. If your card has this glitch, try setting ``ZFS_INITRD_PRE_MOUNTROOT_SLEEP=X`` in ``/etc/default/zfs``. The system will wait ``X`` seconds for all drives to appear before importing the pool. QEMU/KVM/XEN ~~~~~~~~~~~~ Set a unique serial number on each virtual disk using libvirt or qemu (e.g. ``-drive if=none,id=disk1,file=disk1.qcow2,serial=1234567890``). To be able to use UEFI in guests (instead of only BIOS booting), run this on the host:: sudo apt install ovmf sudo vi /etc/libvirt/qemu.conf Uncomment these lines: .. code-block:: text nvram = [ "/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd", "/usr/share/OVMF/OVMF_CODE.secboot.fd:/usr/share/OVMF/OVMF_VARS.fd", "/usr/share/AAVMF/AAVMF_CODE.fd:/usr/share/AAVMF/AAVMF_VARS.fd", "/usr/share/AAVMF/AAVMF32_CODE.fd:/usr/share/AAVMF/AAVMF32_VARS.fd" ] :: sudo systemctl restart libvirtd.service VMware ~~~~~~ - Set ``disk.EnableUUID = "TRUE"`` in the vmx file or vsphere configuration. Doing this ensures that ``/dev/disk`` aliases are created in the guest. .. highlight:: sh Debian Buster Root on ZFS ========================= .. contents:: Table of Contents :local: Overview -------- Newer release available ~~~~~~~~~~~~~~~~~~~~~~~ - See :doc:`Debian Bullseye Root on ZFS <./Debian Bullseye Root on ZFS>` for new installs. This guide is no longer receiving most updates. It continues to exist for reference for existing installs that followed it. Caution ~~~~~~~ - This HOWTO uses a whole physical disk. - Do not use these instructions for dual-booting. - Backup your data. Any existing data will be lost. System Requirements ~~~~~~~~~~~~~~~~~~~ - `64-bit Debian GNU/Linux Buster Live CD w/ GUI (e.g. gnome iso) `__ - :ref:`A 64-bit kernel is strongly encouraged. <32-bit-vs-64-bit-systems>` - Installing on a drive which presents 4 KiB logical sectors (a “4Kn” drive) only works with UEFI booting. This not unique to ZFS. `GRUB does not and will not work on 4Kn with legacy (BIOS) booting. `__ Computers that have less than 2 GiB of memory run ZFS slowly. 4 GiB of memory is recommended for normal performance in basic workloads. If you wish to use deduplication, you will need `massive amounts of RAM `__. Enabling deduplication is a permanent change that cannot be easily reverted. Support ~~~~~~~ If you need help, reach out to the community using the :ref:`mailing_lists` or IRC at `#zfsonlinux `__ on `Libera Chat `__. If you have a bug report or feature request related to this HOWTO, please `file a new issue and mention @rlaager `__. Contributing ~~~~~~~~~~~~ #. Fork and clone: https://github.com/openzfs/openzfs-docs #. Install the tools:: sudo apt install python3-pip pip3 install -r docs/requirements.txt # Add ~/.local/bin to your $PATH, e.g. by adding this to ~/.bashrc: PATH=$HOME/.local/bin:$PATH #. Make your changes. #. Test:: cd docs make html sensible-browser _build/html/index.html #. ``git commit --signoff`` to a branch, ``git push``, and create a pull request. Mention @rlaager. Encryption ~~~~~~~~~~ This guide supports three different encryption options: unencrypted, ZFS native encryption, and LUKS. With any option, all ZFS features are fully available. Unencrypted does not encrypt anything, of course. With no encryption happening, this option naturally has the best performance. ZFS native encryption encrypts the data and most metadata in the root pool. It does not encrypt dataset or snapshot names or properties. The boot pool is not encrypted at all, but it only contains the bootloader, kernel, and initrd. (Unless you put a password in ``/etc/fstab``, the initrd is unlikely to contain sensitive data.) The system cannot boot without the passphrase being entered at the console. Performance is good. As the encryption happens in ZFS, even if multiple disks (mirror or raidz topologies) are used, the data only has to be encrypted once. LUKS encrypts almost everything. The only unencrypted data is the bootloader, kernel, and initrd. The system cannot boot without the passphrase being entered at the console. Performance is good, but LUKS sits underneath ZFS, so if multiple disks (mirror or raidz topologies) are used, the data has to be encrypted once per disk. Step 1: Prepare The Install Environment --------------------------------------- #. Boot the Debian GNU/Linux Live CD. If prompted, login with the username ``user`` and password ``live``. Connect your system to the Internet as appropriate (e.g. join your WiFi network). Open a terminal. #. Setup and update the repositories:: sudo vi /etc/apt/sources.list .. code-block:: sourceslist deb http://deb.debian.org/debian buster main contrib deb http://deb.debian.org/debian buster-backports main contrib :: sudo apt update #. Optional: Install and start the OpenSSH server in the Live CD environment: If you have a second system, using SSH to access the target system can be convenient:: sudo apt install --yes openssh-server sudo systemctl restart ssh **Hint:** You can find your IP address with ``ip addr show scope global | grep inet``. Then, from your main machine, connect with ``ssh user@IP``. #. Disable automounting: If the disk has been used before (with partitions at the same offsets), previous filesystems (e.g. the ESP) will automount if not disabled:: gsettings set org.gnome.desktop.media-handling automount false #. Become root:: sudo -i #. Install ZFS in the Live CD environment:: apt install --yes debootstrap gdisk dkms dpkg-dev linux-headers-amd64 apt install --yes -t buster-backports --no-install-recommends zfs-dkms modprobe zfs apt install --yes -t buster-backports zfsutils-linux - The dkms dependency is installed manually just so it comes from buster and not buster-backports. This is not critical. - We need to get the module built and loaded before installing zfsutils-linux or `zfs-mount.service will fail to start `__. Step 2: Disk Formatting ----------------------- #. Set a variable with the disk name:: DISK=/dev/disk/by-id/scsi-SATA_disk1 Always use the long ``/dev/disk/by-id/*`` aliases with ZFS. Using the ``/dev/sd*`` device nodes directly can cause sporadic import failures, especially on systems that have more than one storage pool. **Hints:** - ``ls -la /dev/disk/by-id`` will list the aliases. - Are you doing this in a virtual machine? If your virtual disk is missing from ``/dev/disk/by-id``, use ``/dev/vda`` if you are using KVM with virtio; otherwise, read the `troubleshooting <#troubleshooting>`__ section. - For a mirror or raidz topology, use ``DISK1``, ``DISK2``, etc. - When choosing a boot pool size, consider how you will use the space. A kernel and initrd may consume around 100M. If you have multiple kernels and take snapshots, you may find yourself low on boot pool space, especially if you need to regenerate your initramfs images, which may be around 85M each. Size your boot pool appropriately for your needs. #. If you are re-using a disk, clear it as necessary: Ensure swap partitions are not in use:: swapoff --all If the disk was previously used in an MD array:: apt install --yes mdadm # See if one or more MD arrays are active: cat /proc/mdstat # If so, stop them (replace ``md0`` as required): mdadm --stop /dev/md0 # For an array using the whole disk: mdadm --zero-superblock --force $DISK # For an array using a partition: mdadm --zero-superblock --force ${DISK}-part2 Clear the partition table:: sgdisk --zap-all $DISK If you get a message about the kernel still using the old partition table, you can request the kernel reload the partition information using:: partprobe $DISK If the new partitions still don't show up, you can reboot and start over (except that you can skip this step). #. Partition your disk(s): Run this if you need legacy (BIOS) booting:: sgdisk -a1 -n1:24K:+1000K -t1:EF02 $DISK Run this for UEFI booting (for use now or in the future):: sgdisk -n2:1M:+512M -t2:EF00 $DISK Run this for the boot pool:: sgdisk -n3:0:+1G -t3:BF01 $DISK Choose one of the following options: - Unencrypted or ZFS native encryption:: sgdisk -n4:0:0 -t4:BF00 $DISK - LUKS:: sgdisk -n4:0:0 -t4:8309 $DISK If you are creating a mirror or raidz topology, repeat the partitioning commands for all the disks which will be part of the pool. #. Create the boot pool:: zpool create \ -o cachefile=/etc/zfs/zpool.cache \ -o ashift=12 -d \ -o feature@async_destroy=enabled \ -o feature@bookmarks=enabled \ -o feature@embedded_data=enabled \ -o feature@empty_bpobj=enabled \ -o feature@enabled_txg=enabled \ -o feature@extensible_dataset=enabled \ -o feature@filesystem_limits=enabled \ -o feature@hole_birth=enabled \ -o feature@large_blocks=enabled \ -o feature@lz4_compress=enabled \ -o feature@spacemap_histogram=enabled \ -o feature@zpool_checkpoint=enabled \ -O acltype=posixacl -O canmount=off -O compression=lz4 \ -O devices=off -O normalization=formD -O relatime=on -O xattr=sa \ -O mountpoint=/boot -R /mnt \ bpool ${DISK}-part3 You should not need to customize any of the options for the boot pool. GRUB does not support all of the zpool features. See ``spa_feature_names`` in `grub-core/fs/zfs/zfs.c `__. This step creates a separate boot pool for ``/boot`` with the features limited to only those that GRUB supports, allowing the root pool to use any/all features. Note that GRUB opens the pool read-only, so all read-only compatible features are “supported” by GRUB. **Hints:** - If you are creating a mirror topology, create the pool using:: zpool create \ ... \ bpool mirror \ /dev/disk/by-id/scsi-SATA_disk1-part3 \ /dev/disk/by-id/scsi-SATA_disk2-part3 - For raidz topologies, replace ``mirror`` in the above command with ``raidz``, ``raidz2``, or ``raidz3`` and list the partitions from the additional disks. - The pool name is arbitrary. If changed, the new name must be used consistently. The ``bpool`` convention originated in this HOWTO. **Feature Notes:** - The ``allocation_classes`` feature should be safe to use. However, unless one is using it (i.e. a ``special`` vdev), there is no point to enabling it. It is extremely unlikely that someone would use this feature for a boot pool. If one cares about speeding up the boot pool, it would make more sense to put the whole pool on the faster disk rather than using it as a ``special`` vdev. - The ``project_quota`` feature has been tested and is safe to use. This feature is extremely unlikely to matter for the boot pool. - The ``resilver_defer`` should be safe but the boot pool is small enough that it is unlikely to be necessary. - The ``spacemap_v2`` feature has been tested and is safe to use. The boot pool is small, so this does not matter in practice. - As a read-only compatible feature, the ``userobj_accounting`` feature should be compatible in theory, but in practice, GRUB can fail with an “invalid dnode type” error. This feature does not matter for ``/boot`` anyway. #. Create the root pool: Choose one of the following options: - Unencrypted:: zpool create \ -o ashift=12 \ -O acltype=posixacl -O canmount=off -O compression=lz4 \ -O dnodesize=auto -O normalization=formD -O relatime=on \ -O xattr=sa -O mountpoint=/ -R /mnt \ rpool ${DISK}-part4 - ZFS native encryption:: zpool create \ -o ashift=12 \ -O encryption=on \ -O keylocation=prompt -O keyformat=passphrase \ -O acltype=posixacl -O canmount=off -O compression=lz4 \ -O dnodesize=auto -O normalization=formD -O relatime=on \ -O xattr=sa -O mountpoint=/ -R /mnt \ rpool ${DISK}-part4 - LUKS:: apt install --yes cryptsetup cryptsetup luksFormat -c aes-xts-plain64 -s 512 -h sha256 ${DISK}-part4 cryptsetup luksOpen ${DISK}-part4 luks1 zpool create \ -o ashift=12 \ -O acltype=posixacl -O canmount=off -O compression=lz4 \ -O dnodesize=auto -O normalization=formD -O relatime=on \ -O xattr=sa -O mountpoint=/ -R /mnt \ rpool /dev/mapper/luks1 **Notes:** - The use of ``ashift=12`` is recommended here because many drives today have 4 KiB (or larger) physical sectors, even though they present 512 B logical sectors. Also, a future replacement drive may have 4 KiB physical sectors (in which case ``ashift=12`` is desirable) or 4 KiB logical sectors (in which case ``ashift=12`` is required). - Setting ``-O acltype=posixacl`` enables POSIX ACLs globally. If you do not want this, remove that option, but later add ``-o acltype=posixacl`` (note: lowercase “o”) to the ``zfs create`` for ``/var/log``, as `journald requires ACLs `__ - Setting ``normalization=formD`` eliminates some corner cases relating to UTF-8 filename normalization. It also implies ``utf8only=on``, which means that only UTF-8 filenames are allowed. If you care to support non-UTF-8 filenames, do not use this option. For a discussion of why requiring UTF-8 filenames may be a bad idea, see `The problems with enforced UTF-8 only filenames `__. - ``recordsize`` is unset (leaving it at the default of 128 KiB). If you want to tune it (e.g. ``-O recordsize=1M``), see `these `__ `various `__ `blog `__ `posts `__. - Setting ``relatime=on`` is a middle ground between classic POSIX ``atime`` behavior (with its significant performance impact) and ``atime=off`` (which provides the best performance by completely disabling atime updates). Since Linux 2.6.30, ``relatime`` has been the default for other filesystems. See `RedHat’s documentation `__ for further information. - Setting ``xattr=sa`` `vastly improves the performance of extended attributes `__. Inside ZFS, extended attributes are used to implement POSIX ACLs. Extended attributes can also be used by user-space applications. `They are used by some desktop GUI applications. `__ `They can be used by Samba to store Windows ACLs and DOS attributes; they are required for a Samba Active Directory domain controller. `__ Note that ``xattr=sa`` is `Linux-specific `__. If you move your ``xattr=sa`` pool to another OpenZFS implementation besides ZFS-on-Linux, extended attributes will not be readable (though your data will be). If portability of extended attributes is important to you, omit the ``-O xattr=sa`` above. Even if you do not want ``xattr=sa`` for the whole pool, it is probably fine to use it for ``/var/log``. - Make sure to include the ``-part4`` portion of the drive path. If you forget that, you are specifying the whole disk, which ZFS will then re-partition, and you will lose the bootloader partition(s). - ZFS native encryption `now `__ defaults to ``aes-256-gcm``. - For LUKS, the key size chosen is 512 bits. However, XTS mode requires two keys, so the LUKS key is split in half. Thus, ``-s 512`` means AES-256. - Your passphrase will likely be the weakest link. Choose wisely. See `section 5 of the cryptsetup FAQ `__ for guidance. **Hints:** - If you are creating a mirror topology, create the pool using:: zpool create \ ... \ rpool mirror \ /dev/disk/by-id/scsi-SATA_disk1-part4 \ /dev/disk/by-id/scsi-SATA_disk2-part4 - For raidz topologies, replace ``mirror`` in the above command with ``raidz``, ``raidz2``, or ``raidz3`` and list the partitions from the additional disks. - When using LUKS with mirror or raidz topologies, use ``/dev/mapper/luks1``, ``/dev/mapper/luks2``, etc., which you will have to create using ``cryptsetup``. - The pool name is arbitrary. If changed, the new name must be used consistently. On systems that can automatically install to ZFS, the root pool is named ``rpool`` by default. Step 3: System Installation --------------------------- #. Create filesystem datasets to act as containers:: zfs create -o canmount=off -o mountpoint=none rpool/ROOT zfs create -o canmount=off -o mountpoint=none bpool/BOOT On Solaris systems, the root filesystem is cloned and the suffix is incremented for major system changes through ``pkg image-update`` or ``beadm``. Similar functionality was implemented in Ubuntu with the ``zsys`` tool, though its dataset layout is more complicated, and ``zsys`` `is on life support `__. Even without such a tool, the `rpool/ROOT` and `bpool/BOOT` containers can still be used for manually created clones. That said, this HOWTO assumes a single filesystem for ``/boot`` for simplicity. #. Create filesystem datasets for the root and boot filesystems:: zfs create -o canmount=noauto -o mountpoint=/ rpool/ROOT/debian zfs mount rpool/ROOT/debian zfs create -o mountpoint=/boot bpool/BOOT/debian With ZFS, it is not normally necessary to use a mount command (either ``mount`` or ``zfs mount``). This situation is an exception because of ``canmount=noauto``. #. Create datasets:: zfs create rpool/home zfs create -o mountpoint=/root rpool/home/root chmod 700 /mnt/root zfs create -o canmount=off rpool/var zfs create -o canmount=off rpool/var/lib zfs create rpool/var/log zfs create rpool/var/spool The datasets below are optional, depending on your preferences and/or software choices. If you wish to exclude these from snapshots:: zfs create -o com.sun:auto-snapshot=false rpool/var/cache zfs create -o com.sun:auto-snapshot=false rpool/var/tmp chmod 1777 /mnt/var/tmp If you use /opt on this system:: zfs create rpool/opt If you use /srv on this system:: zfs create rpool/srv If you use /usr/local on this system:: zfs create -o canmount=off rpool/usr zfs create rpool/usr/local If this system will have games installed:: zfs create rpool/var/games If this system will store local email in /var/mail:: zfs create rpool/var/mail If this system will use Snap packages:: zfs create rpool/var/snap If you use /var/www on this system:: zfs create rpool/var/www If this system will use GNOME:: zfs create rpool/var/lib/AccountsService If this system will use Docker (which manages its own datasets & snapshots):: zfs create -o com.sun:auto-snapshot=false rpool/var/lib/docker If this system will use NFS (locking):: zfs create -o com.sun:auto-snapshot=false rpool/var/lib/nfs Mount a tmpfs at /run:: mkdir /mnt/run mount -t tmpfs tmpfs /mnt/run mkdir /mnt/run/lock A tmpfs is recommended later, but if you want a separate dataset for ``/tmp``:: zfs create -o com.sun:auto-snapshot=false rpool/tmp chmod 1777 /mnt/tmp The primary goal of this dataset layout is to separate the OS from user data. This allows the root filesystem to be rolled back without rolling back user data. If you do nothing extra, ``/tmp`` will be stored as part of the root filesystem. Alternatively, you can create a separate dataset for ``/tmp``, as shown above. This keeps the ``/tmp`` data out of snapshots of your root filesystem. It also allows you to set a quota on ``rpool/tmp``, if you want to limit the maximum space used. Otherwise, you can use a tmpfs (RAM filesystem) later. #. Install the minimal system:: debootstrap buster /mnt The ``debootstrap`` command leaves the new system in an unconfigured state. An alternative to using ``debootstrap`` is to copy the entirety of a working system into the new ZFS root. #. Copy in zpool.cache:: mkdir /mnt/etc/zfs cp /etc/zfs/zpool.cache /mnt/etc/zfs/ Step 4: System Configuration ---------------------------- #. Configure the hostname: Replace ``HOSTNAME`` with the desired hostname:: hostname HOSTNAME hostname > /mnt/etc/hostname vi /mnt/etc/hosts .. code-block:: text Add a line: 127.0.1.1 HOSTNAME or if the system has a real name in DNS: 127.0.1.1 FQDN HOSTNAME **Hint:** Use ``nano`` if you find ``vi`` confusing. #. Configure the network interface: Find the interface name:: ip addr show Adjust ``NAME`` below to match your interface name:: vi /mnt/etc/network/interfaces.d/NAME .. code-block:: text auto NAME iface NAME inet dhcp Customize this file if the system is not a DHCP client. #. Configure the package sources:: vi /mnt/etc/apt/sources.list .. code-block:: sourceslist deb http://deb.debian.org/debian buster main contrib deb-src http://deb.debian.org/debian buster main contrib deb http://security.debian.org/debian-security buster/updates main contrib deb-src http://security.debian.org/debian-security buster/updates main contrib deb http://deb.debian.org/debian buster-updates main contrib deb-src http://deb.debian.org/debian buster-updates main contrib :: vi /mnt/etc/apt/sources.list.d/buster-backports.list .. code-block:: sourceslist deb http://deb.debian.org/debian buster-backports main contrib deb-src http://deb.debian.org/debian buster-backports main contrib :: vi /mnt/etc/apt/preferences.d/90_zfs .. code-block:: control Package: src:zfs-linux Pin: release n=buster-backports Pin-Priority: 990 #. Bind the virtual filesystems from the LiveCD environment to the new system and ``chroot`` into it:: mount --rbind /dev /mnt/dev mount --rbind /proc /mnt/proc mount --rbind /sys /mnt/sys chroot /mnt /usr/bin/env DISK=$DISK bash --login **Note:** This is using ``--rbind``, not ``--bind``. #. Configure a basic system environment:: ln -s /proc/self/mounts /etc/mtab apt update apt install --yes console-setup locales Even if you prefer a non-English system language, always ensure that ``en_US.UTF-8`` is available:: dpkg-reconfigure locales tzdata keyboard-configuration console-setup #. Install ZFS in the chroot environment for the new system:: apt install --yes dpkg-dev linux-headers-amd64 linux-image-amd64 apt install --yes zfs-initramfs echo REMAKE_INITRD=yes > /etc/dkms/zfs.conf **Note:** Ignore any error messages saying ``ERROR: Couldn't resolve device`` and ``WARNING: Couldn't determine root device``. `cryptsetup does not support ZFS `__. #. For LUKS installs only, setup ``/etc/crypttab``:: apt install --yes cryptsetup echo luks1 /dev/disk/by-uuid/$(blkid -s UUID -o value ${DISK}-part4) \ none luks,discard,initramfs > /etc/crypttab The use of ``initramfs`` is a work-around for `cryptsetup does not support ZFS `__. **Hint:** If you are creating a mirror or raidz topology, repeat the ``/etc/crypttab`` entries for ``luks2``, etc. adjusting for each disk. #. Install GRUB Choose one of the following options: - Install GRUB for legacy (BIOS) booting:: apt install --yes grub-pc Select (using the space bar) all of the disks (not partitions) in your pool. - Install GRUB for UEFI booting:: apt install dosfstools mkdosfs -F 32 -s 1 -n EFI ${DISK}-part2 mkdir /boot/efi echo /dev/disk/by-uuid/$(blkid -s UUID -o value ${DISK}-part2) \ /boot/efi vfat defaults 0 0 >> /etc/fstab mount /boot/efi apt install --yes grub-efi-amd64 shim-signed **Notes:** - The ``-s 1`` for ``mkdosfs`` is only necessary for drives which present 4 KiB logical sectors (“4Kn” drives) to meet the minimum cluster size (given the partition size of 512 MiB) for FAT32. It also works fine on drives which present 512 B sectors. - For a mirror or raidz topology, this step only installs GRUB on the first disk. The other disk(s) will be handled later. #. Optional: Remove os-prober:: apt purge --yes os-prober This avoids error messages from `update-grub`. `os-prober` is only necessary in dual-boot configurations. #. Set a root password:: passwd #. Enable importing bpool This ensures that ``bpool`` is always imported, regardless of whether ``/etc/zfs/zpool.cache`` exists, whether it is in the cachefile or not, or whether ``zfs-import-scan.service`` is enabled. :: vi /etc/systemd/system/zfs-import-bpool.service .. code-block:: ini [Unit] DefaultDependencies=no Before=zfs-import-scan.service Before=zfs-import-cache.service [Service] Type=oneshot RemainAfterExit=yes ExecStart=/sbin/zpool import -N -o cachefile=none bpool # Work-around to preserve zpool cache: ExecStartPre=-/bin/mv /etc/zfs/zpool.cache /etc/zfs/preboot_zpool.cache ExecStartPost=-/bin/mv /etc/zfs/preboot_zpool.cache /etc/zfs/zpool.cache [Install] WantedBy=zfs-import.target :: systemctl enable zfs-import-bpool.service #. Optional (but recommended): Mount a tmpfs to ``/tmp`` If you chose to create a ``/tmp`` dataset above, skip this step, as they are mutually exclusive choices. Otherwise, you can put ``/tmp`` on a tmpfs (RAM filesystem) by enabling the ``tmp.mount`` unit. :: cp /usr/share/systemd/tmp.mount /etc/systemd/system/ systemctl enable tmp.mount #. Optional: Install SSH:: apt install --yes openssh-server vi /etc/ssh/sshd_config # Set: PermitRootLogin yes #. Optional (but kindly requested): Install popcon The ``popularity-contest`` package reports the list of packages installed on your system. Showing that ZFS is popular may be helpful in terms of long-term attention from the distro. :: apt install --yes popularity-contest Choose Yes at the prompt. Step 5: GRUB Installation ------------------------- #. Verify that the ZFS boot filesystem is recognized:: grub-probe /boot #. Refresh the initrd files:: update-initramfs -c -k all **Note:** Ignore any error messages saying ``ERROR: Couldn't resolve device`` and ``WARNING: Couldn't determine root device``. `cryptsetup does not support ZFS `__. #. Workaround GRUB's missing zpool-features support:: vi /etc/default/grub # Set: GRUB_CMDLINE_LINUX="root=ZFS=rpool/ROOT/debian" #. Optional (but highly recommended): Make debugging GRUB easier:: vi /etc/default/grub # Remove quiet from: GRUB_CMDLINE_LINUX_DEFAULT # Uncomment: GRUB_TERMINAL=console # Save and quit. Later, once the system has rebooted twice and you are sure everything is working, you can undo these changes, if desired. #. Update the boot configuration:: update-grub **Note:** Ignore errors from ``osprober``, if present. #. Install the boot loader: #. For legacy (BIOS) booting, install GRUB to the MBR:: grub-install $DISK Note that you are installing GRUB to the whole disk, not a partition. If you are creating a mirror or raidz topology, repeat the ``grub-install`` command for each disk in the pool. #. For UEFI booting, install GRUB to the ESP:: grub-install --target=x86_64-efi --efi-directory=/boot/efi \ --bootloader-id=debian --recheck It is not necessary to specify the disk here. If you are creating a mirror or raidz topology, the additional disks will be handled later. #. Fix filesystem mount ordering: We need to activate ``zfs-mount-generator``. This makes systemd aware of the separate mountpoints, which is important for things like ``/var/log`` and ``/var/tmp``. In turn, ``rsyslog.service`` depends on ``var-log.mount`` by way of ``local-fs.target`` and services using the ``PrivateTmp`` feature of systemd automatically use ``After=var-tmp.mount``. :: mkdir /etc/zfs/zfs-list.cache touch /etc/zfs/zfs-list.cache/bpool touch /etc/zfs/zfs-list.cache/rpool zed -F & Verify that ``zed`` updated the cache by making sure these are not empty:: cat /etc/zfs/zfs-list.cache/bpool cat /etc/zfs/zfs-list.cache/rpool If either is empty, force a cache update and check again:: zfs set canmount=on bpool/BOOT/debian zfs set canmount=noauto rpool/ROOT/debian If they are still empty, stop zed (as below), start zed (as above) and try again. Once the files have data, stop ``zed``:: fg Press Ctrl-C. Fix the paths to eliminate ``/mnt``:: sed -Ei "s|/mnt/?|/|" /etc/zfs/zfs-list.cache/* Step 6: First Boot ------------------ #. Optional: Snapshot the initial installation:: zfs snapshot bpool/BOOT/debian@install zfs snapshot rpool/ROOT/debian@install In the future, you will likely want to take snapshots before each upgrade, and remove old snapshots (including this one) at some point to save space. #. Exit from the ``chroot`` environment back to the LiveCD environment:: exit #. Run these commands in the LiveCD environment to unmount all filesystems:: mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | \ xargs -i{} umount -lf {} zpool export -a #. Reboot:: reboot Wait for the newly installed system to boot normally. Login as root. #. Create a user account: Replace ``YOUR_USERNAME`` with your desired username:: username=YOUR_USERNAME zfs create rpool/home/$username adduser $username cp -a /etc/skel/. /home/$username chown -R $username:$username /home/$username usermod -a -G audio,cdrom,dip,floppy,netdev,plugdev,sudo,video $username #. Mirror GRUB If you installed to multiple disks, install GRUB on the additional disks. - For legacy (BIOS) booting:: dpkg-reconfigure grub-pc Hit enter until you get to the device selection screen. Select (using the space bar) all of the disks (not partitions) in your pool. - For UEFI booting:: umount /boot/efi For the second and subsequent disks (increment debian-2 to -3, etc.):: dd if=/dev/disk/by-id/scsi-SATA_disk1-part2 \ of=/dev/disk/by-id/scsi-SATA_disk2-part2 efibootmgr -c -g -d /dev/disk/by-id/scsi-SATA_disk2 \ -p 2 -L "debian-2" -l '\EFI\debian\grubx64.efi' mount /boot/efi Step 7: Optional: Configure Swap --------------------------------- **Caution**: On systems with extremely high memory pressure, using a zvol for swap can result in lockup, regardless of how much swap is still available. There is `a bug report upstream `__. #. Create a volume dataset (zvol) for use as a swap device:: zfs create -V 4G -b $(getconf PAGESIZE) -o compression=zle \ -o logbias=throughput -o sync=always \ -o primarycache=metadata -o secondarycache=none \ -o com.sun:auto-snapshot=false rpool/swap You can adjust the size (the ``4G`` part) to your needs. The compression algorithm is set to ``zle`` because it is the cheapest available algorithm. As this guide recommends ``ashift=12`` (4 kiB blocks on disk), the common case of a 4 kiB page size means that no compression algorithm can reduce I/O. The exception is all-zero pages, which are dropped by ZFS; but some form of compression has to be enabled to get this behavior. #. Configure the swap device: **Caution**: Always use long ``/dev/zvol`` aliases in configuration files. Never use a short ``/dev/zdX`` device name. :: mkswap -f /dev/zvol/rpool/swap echo /dev/zvol/rpool/swap none swap discard 0 0 >> /etc/fstab echo RESUME=none > /etc/initramfs-tools/conf.d/resume The ``RESUME=none`` is necessary to disable resuming from hibernation. This does not work, as the zvol is not present (because the pool has not yet been imported) at the time the resume script runs. If it is not disabled, the boot process hangs for 30 seconds waiting for the swap zvol to appear. #. Enable the swap device:: swapon -av Step 8: Full Software Installation ---------------------------------- #. Upgrade the minimal system:: apt dist-upgrade --yes #. Install a regular set of software:: tasksel --new-install **Note:** This will check "Debian desktop environment" and "print server" by default. If you want a server installation, unselect those. #. Optional: Disable log compression: As ``/var/log`` is already compressed by ZFS, logrotate’s compression is going to burn CPU and disk I/O for (in most cases) very little gain. Also, if you are making snapshots of ``/var/log``, logrotate’s compression will actually waste space, as the uncompressed data will live on in the snapshot. You can edit the files in ``/etc/logrotate.d`` by hand to comment out ``compress``, or use this loop (copy-and-paste highly recommended):: for file in /etc/logrotate.d/* ; do if grep -Eq "(^|[^#y])compress" "$file" ; then sed -i -r "s/(^|[^#y])(compress)/\1#\2/" "$file" fi done #. Reboot:: reboot Step 9: Final Cleanup --------------------- #. Wait for the system to boot normally. Login using the account you created. Ensure the system (including networking) works normally. #. Optional: Delete the snapshots of the initial installation:: sudo zfs destroy bpool/BOOT/debian@install sudo zfs destroy rpool/ROOT/debian@install #. Optional: Disable the root password:: sudo usermod -p '*' root #. Optional (but highly recommended): Disable root SSH logins: If you installed SSH earlier, revert the temporary change:: sudo vi /etc/ssh/sshd_config # Remove: PermitRootLogin yes sudo systemctl restart ssh #. Optional: Re-enable the graphical boot process: If you prefer the graphical boot process, you can re-enable it now. If you are using LUKS, it makes the prompt look nicer. :: sudo vi /etc/default/grub # Add quiet to GRUB_CMDLINE_LINUX_DEFAULT # Comment out GRUB_TERMINAL=console # Save and quit. sudo update-grub **Note:** Ignore errors from ``osprober``, if present. #. Optional: For LUKS installs only, backup the LUKS header:: sudo cryptsetup luksHeaderBackup /dev/disk/by-id/scsi-SATA_disk1-part4 \ --header-backup-file luks1-header.dat Store that backup somewhere safe (e.g. cloud storage). It is protected by your LUKS passphrase, but you may wish to use additional encryption. **Hint:** If you created a mirror or raidz topology, repeat this for each LUKS volume (``luks2``, etc.). Troubleshooting --------------- Rescuing using a Live CD ~~~~~~~~~~~~~~~~~~~~~~~~ Go through `Step 1: Prepare The Install Environment <#step-1-prepare-the-install-environment>`__. For LUKS, first unlock the disk(s):: apt install --yes cryptsetup cryptsetup luksOpen /dev/disk/by-id/scsi-SATA_disk1-part4 luks1 # Repeat for additional disks, if this is a mirror or raidz topology. Mount everything correctly:: zpool export -a zpool import -N -R /mnt rpool zpool import -N -R /mnt bpool zfs load-key -a zfs mount rpool/ROOT/debian zfs mount -a If needed, you can chroot into your installed environment:: mount --rbind /dev /mnt/dev mount --rbind /proc /mnt/proc mount --rbind /sys /mnt/sys mount -t tmpfs tmpfs /mnt/run mkdir /mnt/run/lock chroot /mnt /bin/bash --login mount /boot/efi mount -a Do whatever you need to do to fix your system. When done, cleanup:: exit mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | \ xargs -i{} umount -lf {} zpool export -a reboot Areca ~~~~~ Systems that require the ``arcsas`` blob driver should add it to the ``/etc/initramfs-tools/modules`` file and run ``update-initramfs -c -k all``. Upgrade or downgrade the Areca driver if something like ``RIP: 0010:[] [] native_read_tsc+0x6/0x20`` appears anywhere in kernel log. ZoL is unstable on systems that emit this error message. MPT2SAS ~~~~~~~ Most problem reports for this tutorial involve ``mpt2sas`` hardware that does slow asynchronous drive initialization, like some IBM M1015 or OEM-branded cards that have been flashed to the reference LSI firmware. The basic problem is that disks on these controllers are not visible to the Linux kernel until after the regular system is started, and ZoL does not hotplug pool members. See `https://github.com/openzfs/zfs/issues/330 `__. Most LSI cards are perfectly compatible with ZoL. If your card has this glitch, try setting ``ZFS_INITRD_PRE_MOUNTROOT_SLEEP=X`` in ``/etc/default/zfs``. The system will wait ``X`` seconds for all drives to appear before importing the pool. QEMU/KVM/XEN ~~~~~~~~~~~~ Set a unique serial number on each virtual disk using libvirt or qemu (e.g. ``-drive if=none,id=disk1,file=disk1.qcow2,serial=1234567890``). To be able to use UEFI in guests (instead of only BIOS booting), run this on the host:: sudo apt install ovmf sudo vi /etc/libvirt/qemu.conf Uncomment these lines: .. code-block:: text nvram = [ "/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd", "/usr/share/OVMF/OVMF_CODE.secboot.fd:/usr/share/OVMF/OVMF_VARS.fd", "/usr/share/AAVMF/AAVMF_CODE.fd:/usr/share/AAVMF/AAVMF_VARS.fd", "/usr/share/AAVMF/AAVMF32_CODE.fd:/usr/share/AAVMF/AAVMF32_VARS.fd" ] :: sudo systemctl restart libvirtd.service VMware ~~~~~~ - Set ``disk.EnableUUID = "TRUE"`` in the vmx file or vsphere configuration. Doing this ensures that ``/dev/disk`` aliases are created in the guest. Debian Stretch Root on ZFS ========================== .. contents:: Table of Contents :local: Overview -------- Newer release available ~~~~~~~~~~~~~~~~~~~~~~~ - See :doc:`Debian Buster Root on ZFS <./Debian Buster Root on ZFS>` for new installs. Caution ~~~~~~~ - This HOWTO uses a whole physical disk. - Do not use these instructions for dual-booting. - Backup your data. Any existing data will be lost. System Requirements ~~~~~~~~~~~~~~~~~~~ - `64-bit Debian GNU/Linux Stretch Live CD `__ - :ref:`A 64-bit kernel is strongly encouraged. <32-bit-vs-64-bit-systems>` - Installing on a drive which presents 4KiB logical sectors (a “4Kn” drive) only works with UEFI booting. This not unique to ZFS. `GRUB does not and will not work on 4Kn with legacy (BIOS) booting. `__ Computers that have less than 2 GiB of memory run ZFS slowly. 4 GiB of memory is recommended for normal performance in basic workloads. If you wish to use deduplication, you will need `massive amounts of RAM `__. Enabling deduplication is a permanent change that cannot be easily reverted. Support ~~~~~~~ If you need help, reach out to the community using the :ref:`mailing_lists` or IRC at `#zfsonlinux `__ on `Libera Chat `__. If you have a bug report or feature request related to this HOWTO, please `file a new issue and mention @rlaager `__. Contributing ~~~~~~~~~~~~ #. Fork and clone: https://github.com/openzfs/openzfs-docs #. Install the tools:: sudo apt install python3-pip pip3 install -r docs/requirements.txt # Add ~/.local/bin to your $PATH, e.g. by adding this to ~/.bashrc: PATH=$HOME/.local/bin:$PATH #. Make your changes. #. Test:: cd docs make html sensible-browser _build/html/index.html #. ``git commit --signoff`` to a branch, ``git push``, and create a pull request. Mention @rlaager. Encryption ~~~~~~~~~~ This guide supports two different encryption options: unencrypted and LUKS (full-disk encryption). ZFS native encryption has not yet been released. With either option, all ZFS features are fully available. Unencrypted does not encrypt anything, of course. With no encryption happening, this option naturally has the best performance. LUKS encrypts almost everything: the OS, swap, home directories, and anything else. The only unencrypted data is the bootloader, kernel, and initrd. The system cannot boot without the passphrase being entered at the console. Performance is good, but LUKS sits underneath ZFS, so if multiple disks (mirror or raidz topologies) are used, the data has to be encrypted once per disk. Step 1: Prepare The Install Environment --------------------------------------- 1.1 Boot the Debian GNU/Linux Live CD. If prompted, login with the username ``user`` and password ``live``. Connect your system to the Internet as appropriate (e.g. join your WiFi network). 1.2 Optional: Install and start the OpenSSH server in the Live CD environment: If you have a second system, using SSH to access the target system can be convenient. :: $ sudo apt update $ sudo apt install --yes openssh-server $ sudo systemctl restart ssh **Hint:** You can find your IP address with ``ip addr show scope global | grep inet``. Then, from your main machine, connect with ``ssh user@IP``. 1.3 Become root: :: $ sudo -i 1.4 Setup and update the repositories: :: # echo deb http://deb.debian.org/debian stretch contrib >> /etc/apt/sources.list # echo deb http://deb.debian.org/debian stretch-backports main contrib >> /etc/apt/sources.list # apt update 1.5 Install ZFS in the Live CD environment: :: # apt install --yes debootstrap gdisk dkms dpkg-dev linux-headers-amd64 # apt install --yes -t stretch-backports zfs-dkms # modprobe zfs - The dkms dependency is installed manually just so it comes from stretch and not stretch-backports. This is not critical. Step 2: Disk Formatting ----------------------- 2.1 If you are re-using a disk, clear it as necessary: :: If the disk was previously used in an MD array, zero the superblock: # apt install --yes mdadm # mdadm --zero-superblock --force /dev/disk/by-id/scsi-SATA_disk1 Clear the partition table: # sgdisk --zap-all /dev/disk/by-id/scsi-SATA_disk1 2.2 Partition your disk(s): :: Run this if you need legacy (BIOS) booting: # sgdisk -a1 -n1:24K:+1000K -t1:EF02 /dev/disk/by-id/scsi-SATA_disk1 Run this for UEFI booting (for use now or in the future): # sgdisk -n2:1M:+512M -t2:EF00 /dev/disk/by-id/scsi-SATA_disk1 Run this for the boot pool: # sgdisk -n3:0:+1G -t3:BF01 /dev/disk/by-id/scsi-SATA_disk1 Choose one of the following options: 2.2a Unencrypted: :: # sgdisk -n4:0:0 -t4:BF01 /dev/disk/by-id/scsi-SATA_disk1 2.2b LUKS: :: # sgdisk -n4:0:0 -t4:8300 /dev/disk/by-id/scsi-SATA_disk1 Always use the long ``/dev/disk/by-id/*`` aliases with ZFS. Using the ``/dev/sd*`` device nodes directly can cause sporadic import failures, especially on systems that have more than one storage pool. **Hints:** - ``ls -la /dev/disk/by-id`` will list the aliases. - Are you doing this in a virtual machine? If your virtual disk is missing from ``/dev/disk/by-id``, use ``/dev/vda`` if you are using KVM with virtio; otherwise, read the `troubleshooting <#troubleshooting>`__ section. - If you are creating a mirror or raidz topology, repeat the partitioning commands for all the disks which will be part of the pool. 2.3 Create the boot pool: :: # zpool create -o ashift=12 -d \ -o feature@async_destroy=enabled \ -o feature@bookmarks=enabled \ -o feature@embedded_data=enabled \ -o feature@empty_bpobj=enabled \ -o feature@enabled_txg=enabled \ -o feature@extensible_dataset=enabled \ -o feature@filesystem_limits=enabled \ -o feature@hole_birth=enabled \ -o feature@large_blocks=enabled \ -o feature@lz4_compress=enabled \ -o feature@spacemap_histogram=enabled \ -o feature@userobj_accounting=enabled \ -O acltype=posixacl -O canmount=off -O compression=lz4 -O devices=off \ -O normalization=formD -O relatime=on -O xattr=sa \ -O mountpoint=/ -R /mnt \ bpool /dev/disk/by-id/scsi-SATA_disk1-part3 You should not need to customize any of the options for the boot pool. GRUB does not support all of the zpool features. See ``spa_feature_names`` in `grub-core/fs/zfs/zfs.c `__. This step creates a separate boot pool for ``/boot`` with the features limited to only those that GRUB supports, allowing the root pool to use any/all features. Note that GRUB opens the pool read-only, so all read-only compatible features are "supported" by GRUB. **Hints:** - If you are creating a mirror or raidz topology, create the pool using ``zpool create ... bpool mirror /dev/disk/by-id/scsi-SATA_disk1-part3 /dev/disk/by-id/scsi-SATA_disk2-part3`` (or replace ``mirror`` with ``raidz``, ``raidz2``, or ``raidz3`` and list the partitions from additional disks). - The pool name is arbitrary. If changed, the new name must be used consistently. The ``bpool`` convention originated in this HOWTO. 2.4 Create the root pool: Choose one of the following options: 2.4a Unencrypted: :: # zpool create -o ashift=12 \ -O acltype=posixacl -O canmount=off -O compression=lz4 \ -O dnodesize=auto -O normalization=formD -O relatime=on -O xattr=sa \ -O mountpoint=/ -R /mnt \ rpool /dev/disk/by-id/scsi-SATA_disk1-part4 2.4b LUKS: :: # apt install --yes cryptsetup # cryptsetup luksFormat -c aes-xts-plain64 -s 512 -h sha256 \ /dev/disk/by-id/scsi-SATA_disk1-part4 # cryptsetup luksOpen /dev/disk/by-id/scsi-SATA_disk1-part4 luks1 # zpool create -o ashift=12 \ -O acltype=posixacl -O canmount=off -O compression=lz4 \ -O dnodesize=auto -O normalization=formD -O relatime=on -O xattr=sa \ -O mountpoint=/ -R /mnt \ rpool /dev/mapper/luks1 - The use of ``ashift=12`` is recommended here because many drives today have 4KiB (or larger) physical sectors, even though they present 512B logical sectors. Also, a future replacement drive may have 4KiB physical sectors (in which case ``ashift=12`` is desirable) or 4KiB logical sectors (in which case ``ashift=12`` is required). - Setting ``-O acltype=posixacl`` enables POSIX ACLs globally. If you do not want this, remove that option, but later add ``-o acltype=posixacl`` (note: lowercase "o") to the ``zfs create`` for ``/var/log``, as `journald requires ACLs `__ - Setting ``normalization=formD`` eliminates some corner cases relating to UTF-8 filename normalization. It also implies ``utf8only=on``, which means that only UTF-8 filenames are allowed. If you care to support non-UTF-8 filenames, do not use this option. For a discussion of why requiring UTF-8 filenames may be a bad idea, see `The problems with enforced UTF-8 only filenames `__. - Setting ``relatime=on`` is a middle ground between classic POSIX ``atime`` behavior (with its significant performance impact) and ``atime=off`` (which provides the best performance by completely disabling atime updates). Since Linux 2.6.30, ``relatime`` has been the default for other filesystems. See `RedHat's documentation `__ for further information. - Setting ``xattr=sa`` `vastly improves the performance of extended attributes `__. Inside ZFS, extended attributes are used to implement POSIX ACLs. Extended attributes can also be used by user-space applications. `They are used by some desktop GUI applications. `__ `They can be used by Samba to store Windows ACLs and DOS attributes; they are required for a Samba Active Directory domain controller. `__ Note that ```xattr=sa`` is Linux-specific. `__ If you move your ``xattr=sa`` pool to another OpenZFS implementation besides ZFS-on-Linux, extended attributes will not be readable (though your data will be). If portability of extended attributes is important to you, omit the ``-O xattr=sa`` above. Even if you do not want ``xattr=sa`` for the whole pool, it is probably fine to use it for ``/var/log``. - Make sure to include the ``-part4`` portion of the drive path. If you forget that, you are specifying the whole disk, which ZFS will then re-partition, and you will lose the bootloader partition(s). - For LUKS, the key size chosen is 512 bits. However, XTS mode requires two keys, so the LUKS key is split in half. Thus, ``-s 512`` means AES-256. - Your passphrase will likely be the weakest link. Choose wisely. See `section 5 of the cryptsetup FAQ `__ for guidance. **Hints:** - If you are creating a mirror or raidz topology, create the pool using ``zpool create ... rpool mirror /dev/disk/by-id/scsi-SATA_disk1-part4 /dev/disk/by-id/scsi-SATA_disk2-part4`` (or replace ``mirror`` with ``raidz``, ``raidz2``, or ``raidz3`` and list the partitions from additional disks). For LUKS, use ``/dev/mapper/luks1``, ``/dev/mapper/luks2``, etc., which you will have to create using ``cryptsetup``. - The pool name is arbitrary. If changed, the new name must be used consistently. On systems that can automatically install to ZFS, the root pool is named ``rpool`` by default. Step 3: System Installation --------------------------- 3.1 Create filesystem datasets to act as containers: :: # zfs create -o canmount=off -o mountpoint=none rpool/ROOT # zfs create -o canmount=off -o mountpoint=none bpool/BOOT On Solaris systems, the root filesystem is cloned and the suffix is incremented for major system changes through ``pkg image-update`` or ``beadm``. Similar functionality for APT is possible but currently unimplemented. Even without such a tool, it can still be used for manually created clones. 3.2 Create filesystem datasets for the root and boot filesystems: :: # zfs create -o canmount=noauto -o mountpoint=/ rpool/ROOT/debian # zfs mount rpool/ROOT/debian # zfs create -o canmount=noauto -o mountpoint=/boot bpool/BOOT/debian # zfs mount bpool/BOOT/debian With ZFS, it is not normally necessary to use a mount command (either ``mount`` or ``zfs mount``). This situation is an exception because of ``canmount=noauto``. 3.3 Create datasets: :: # zfs create rpool/home # zfs create -o mountpoint=/root rpool/home/root # zfs create -o canmount=off rpool/var # zfs create -o canmount=off rpool/var/lib # zfs create rpool/var/log # zfs create rpool/var/spool The datasets below are optional, depending on your preferences and/or software choices: If you wish to exclude these from snapshots: # zfs create -o com.sun:auto-snapshot=false rpool/var/cache # zfs create -o com.sun:auto-snapshot=false rpool/var/tmp # chmod 1777 /mnt/var/tmp If you use /opt on this system: # zfs create rpool/opt If you use /srv on this system: # zfs create rpool/srv If you use /usr/local on this system: # zfs create -o canmount=off rpool/usr # zfs create rpool/usr/local If this system will have games installed: # zfs create rpool/var/games If this system will store local email in /var/mail: # zfs create rpool/var/mail If this system will use Snap packages: # zfs create rpool/var/snap If you use /var/www on this system: # zfs create rpool/var/www If this system will use GNOME: # zfs create rpool/var/lib/AccountsService If this system will use Docker (which manages its own datasets & snapshots): # zfs create -o com.sun:auto-snapshot=false rpool/var/lib/docker If this system will use NFS (locking): # zfs create -o com.sun:auto-snapshot=false rpool/var/lib/nfs A tmpfs is recommended later, but if you want a separate dataset for /tmp: # zfs create -o com.sun:auto-snapshot=false rpool/tmp # chmod 1777 /mnt/tmp The primary goal of this dataset layout is to separate the OS from user data. This allows the root filesystem to be rolled back without rolling back user data such as logs (in ``/var/log``). This will be especially important if/when a ``beadm`` or similar utility is integrated. The ``com.sun:auto-snapshot`` setting is used by some ZFS snapshot utilities to exclude transient data. If you do nothing extra, ``/tmp`` will be stored as part of the root filesystem. Alternatively, you can create a separate dataset for ``/tmp``, as shown above. This keeps the ``/tmp`` data out of snapshots of your root filesystem. It also allows you to set a quota on ``rpool/tmp``, if you want to limit the maximum space used. Otherwise, you can use a tmpfs (RAM filesystem) later. 3.4 Install the minimal system: :: # debootstrap stretch /mnt # zfs set devices=off rpool The ``debootstrap`` command leaves the new system in an unconfigured state. An alternative to using ``debootstrap`` is to copy the entirety of a working system into the new ZFS root. Step 4: System Configuration ---------------------------- 4.1 Configure the hostname (change ``HOSTNAME`` to the desired hostname). :: # echo HOSTNAME > /mnt/etc/hostname # vi /mnt/etc/hosts Add a line: 127.0.1.1 HOSTNAME or if the system has a real name in DNS: 127.0.1.1 FQDN HOSTNAME **Hint:** Use ``nano`` if you find ``vi`` confusing. 4.2 Configure the network interface: :: Find the interface name: # ip addr show # vi /mnt/etc/network/interfaces.d/NAME auto NAME iface NAME inet dhcp Customize this file if the system is not a DHCP client. 4.3 Configure the package sources: :: # vi /mnt/etc/apt/sources.list deb http://deb.debian.org/debian stretch main contrib deb-src http://deb.debian.org/debian stretch main contrib deb http://security.debian.org/debian-security stretch/updates main contrib deb-src http://security.debian.org/debian-security stretch/updates main contrib deb http://deb.debian.org/debian stretch-updates main contrib deb-src http://deb.debian.org/debian stretch-updates main contrib # vi /mnt/etc/apt/sources.list.d/stretch-backports.list deb http://deb.debian.org/debian stretch-backports main contrib deb-src http://deb.debian.org/debian stretch-backports main contrib # vi /mnt/etc/apt/preferences.d/90_zfs Package: src:zfs-linux Pin: release n=stretch-backports Pin-Priority: 990 4.4 Bind the virtual filesystems from the LiveCD environment to the new system and ``chroot`` into it: :: # mount --rbind /dev /mnt/dev # mount --rbind /proc /mnt/proc # mount --rbind /sys /mnt/sys # chroot /mnt /bin/bash --login **Note:** This is using ``--rbind``, not ``--bind``. 4.5 Configure a basic system environment: :: # ln -s /proc/self/mounts /etc/mtab # apt update # apt install --yes locales # dpkg-reconfigure locales Even if you prefer a non-English system language, always ensure that ``en_US.UTF-8`` is available. :: # dpkg-reconfigure tzdata 4.6 Install ZFS in the chroot environment for the new system: :: # apt install --yes dpkg-dev linux-headers-amd64 linux-image-amd64 # apt install --yes zfs-initramfs 4.7 For LUKS installs only, setup crypttab: :: # apt install --yes cryptsetup # echo luks1 UUID=$(blkid -s UUID -o value \ /dev/disk/by-id/scsi-SATA_disk1-part4) none \ luks,discard,initramfs > /etc/crypttab - The use of ``initramfs`` is a work-around for `cryptsetup does not support ZFS `__. **Hint:** If you are creating a mirror or raidz topology, repeat the ``/etc/crypttab`` entries for ``luks2``, etc. adjusting for each disk. 4.8 Install GRUB Choose one of the following options: 4.8a Install GRUB for legacy (BIOS) booting :: # apt install --yes grub-pc Install GRUB to the disk(s), not the partition(s). 4.8b Install GRUB for UEFI booting :: # apt install dosfstools # mkdosfs -F 32 -s 1 -n EFI /dev/disk/by-id/scsi-SATA_disk1-part2 # mkdir /boot/efi # echo PARTUUID=$(blkid -s PARTUUID -o value \ /dev/disk/by-id/scsi-SATA_disk1-part2) \ /boot/efi vfat nofail,x-systemd.device-timeout=1 0 1 >> /etc/fstab # mount /boot/efi # apt install --yes grub-efi-amd64 shim - The ``-s 1`` for ``mkdosfs`` is only necessary for drives which present 4 KiB logical sectors (“4Kn” drives) to meet the minimum cluster size (given the partition size of 512 MiB) for FAT32. It also works fine on drives which present 512 B sectors. **Note:** If you are creating a mirror or raidz topology, this step only installs GRUB on the first disk. The other disk(s) will be handled later. 4.9 Set a root password :: # passwd 4.10 Enable importing bpool This ensures that ``bpool`` is always imported, regardless of whether ``/etc/zfs/zpool.cache`` exists, whether it is in the cachefile or not, or whether ``zfs-import-scan.service`` is enabled. :: # vi /etc/systemd/system/zfs-import-bpool.service [Unit] DefaultDependencies=no Before=zfs-import-scan.service Before=zfs-import-cache.service [Service] Type=oneshot RemainAfterExit=yes ExecStart=/sbin/zpool import -N -o cachefile=none bpool [Install] WantedBy=zfs-import.target # systemctl enable zfs-import-bpool.service 4.11 Optional (but recommended): Mount a tmpfs to /tmp If you chose to create a ``/tmp`` dataset above, skip this step, as they are mutually exclusive choices. Otherwise, you can put ``/tmp`` on a tmpfs (RAM filesystem) by enabling the ``tmp.mount`` unit. :: # cp /usr/share/systemd/tmp.mount /etc/systemd/system/ # systemctl enable tmp.mount 4.12 Optional (but kindly requested): Install popcon The ``popularity-contest`` package reports the list of packages installed on your system. Showing that ZFS is popular may be helpful in terms of long-term attention from the distro. :: # apt install --yes popularity-contest Choose Yes at the prompt. Step 5: GRUB Installation ------------------------- 5.1 Verify that the ZFS boot filesystem is recognized: :: # grub-probe /boot zfs 5.2 Refresh the initrd files: :: # update-initramfs -u -k all update-initramfs: Generating /boot/initrd.img-4.9.0-8-amd64 **Note:** When using LUKS, this will print "WARNING could not determine root device from /etc/fstab". This is because `cryptsetup does not support ZFS `__. 5.3 Workaround GRUB's missing zpool-features support: :: # vi /etc/default/grub Set: GRUB_CMDLINE_LINUX="root=ZFS=rpool/ROOT/debian" 5.4 Optional (but highly recommended): Make debugging GRUB easier: :: # vi /etc/default/grub Remove quiet from: GRUB_CMDLINE_LINUX_DEFAULT Uncomment: GRUB_TERMINAL=console Save and quit. Later, once the system has rebooted twice and you are sure everything is working, you can undo these changes, if desired. 5.5 Update the boot configuration: :: # update-grub Generating grub configuration file ... Found linux image: /boot/vmlinuz-4.9.0-8-amd64 Found initrd image: /boot/initrd.img-4.9.0-8-amd64 done **Note:** Ignore errors from ``osprober``, if present. 5.6 Install the boot loader 5.6a For legacy (BIOS) booting, install GRUB to the MBR: :: # grub-install /dev/disk/by-id/scsi-SATA_disk1 Installing for i386-pc platform. Installation finished. No error reported. Do not reboot the computer until you get exactly that result message. Note that you are installing GRUB to the whole disk, not a partition. If you are creating a mirror or raidz topology, repeat the ``grub-install`` command for each disk in the pool. 5.6b For UEFI booting, install GRUB: :: # grub-install --target=x86_64-efi --efi-directory=/boot/efi \ --bootloader-id=debian --recheck 5.7 Verify that the ZFS module is installed: :: # ls /boot/grub/*/zfs.mod 5.8 Fix filesystem mount ordering `Until ZFS gains a systemd mount generator `__, there are races between mounting filesystems and starting certain daemons. In practice, the issues (e.g. `#5754 `__) seem to be with certain filesystems in ``/var``, specifically ``/var/log`` and ``/var/tmp``. Setting these to use ``legacy`` mounting, and listing them in ``/etc/fstab`` makes systemd aware that these are separate mountpoints. In turn, ``rsyslog.service`` depends on ``var-log.mount`` by way of ``local-fs.target`` and services using the ``PrivateTmp`` feature of systemd automatically use ``After=var-tmp.mount``. Until there is support for mounting ``/boot`` in the initramfs, we also need to mount that, because it was marked ``canmount=noauto``. Also, with UEFI, we need to ensure it is mounted before its child filesystem ``/boot/efi``. ``rpool`` is guaranteed to be imported by the initramfs, so there is no point in adding ``x-systemd.requires=zfs-import.target`` to those filesystems. :: For UEFI booting, unmount /boot/efi first: # umount /boot/efi Everything else applies to both BIOS and UEFI booting: # zfs set mountpoint=legacy bpool/BOOT/debian # echo bpool/BOOT/debian /boot zfs \ nodev,relatime,x-systemd.requires=zfs-import-bpool.service 0 0 >> /etc/fstab # zfs set mountpoint=legacy rpool/var/log # echo rpool/var/log /var/log zfs nodev,relatime 0 0 >> /etc/fstab # zfs set mountpoint=legacy rpool/var/spool # echo rpool/var/spool /var/spool zfs nodev,relatime 0 0 >> /etc/fstab If you created a /var/tmp dataset: # zfs set mountpoint=legacy rpool/var/tmp # echo rpool/var/tmp /var/tmp zfs nodev,relatime 0 0 >> /etc/fstab If you created a /tmp dataset: # zfs set mountpoint=legacy rpool/tmp # echo rpool/tmp /tmp zfs nodev,relatime 0 0 >> /etc/fstab Step 6: First Boot ------------------ 6.1 Snapshot the initial installation: :: # zfs snapshot bpool/BOOT/debian@install # zfs snapshot rpool/ROOT/debian@install In the future, you will likely want to take snapshots before each upgrade, and remove old snapshots (including this one) at some point to save space. 6.2 Exit from the ``chroot`` environment back to the LiveCD environment: :: # exit 6.3 Run these commands in the LiveCD environment to unmount all filesystems: :: # mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {} # zpool export -a 6.4 Reboot: :: # reboot 6.5 Wait for the newly installed system to boot normally. Login as root. 6.6 Create a user account: :: # zfs create rpool/home/YOURUSERNAME # adduser YOURUSERNAME # cp -a /etc/skel/.[!.]* /home/YOURUSERNAME # chown -R YOURUSERNAME:YOURUSERNAME /home/YOURUSERNAME 6.7 Add your user account to the default set of groups for an administrator: :: # usermod -a -G audio,cdrom,dip,floppy,netdev,plugdev,sudo,video YOURUSERNAME 6.8 Mirror GRUB If you installed to multiple disks, install GRUB on the additional disks: 6.8a For legacy (BIOS) booting: :: # dpkg-reconfigure grub-pc Hit enter until you get to the device selection screen. Select (using the space bar) all of the disks (not partitions) in your pool. 6.8b UEFI :: # umount /boot/efi For the second and subsequent disks (increment debian-2 to -3, etc.): # dd if=/dev/disk/by-id/scsi-SATA_disk1-part2 \ of=/dev/disk/by-id/scsi-SATA_disk2-part2 # efibootmgr -c -g -d /dev/disk/by-id/scsi-SATA_disk2 \ -p 2 -L "debian-2" -l '\EFI\debian\grubx64.efi' # mount /boot/efi Step 7: (Optional) Configure Swap --------------------------------- **Caution**: On systems with extremely high memory pressure, using a zvol for swap can result in lockup, regardless of how much swap is still available. This issue is currently being investigated in: `https://github.com/openzfs/zfs/issues/7734 `__ 7.1 Create a volume dataset (zvol) for use as a swap device: :: # zfs create -V 4G -b $(getconf PAGESIZE) -o compression=zle \ -o logbias=throughput -o sync=always \ -o primarycache=metadata -o secondarycache=none \ -o com.sun:auto-snapshot=false rpool/swap You can adjust the size (the ``4G`` part) to your needs. The compression algorithm is set to ``zle`` because it is the cheapest available algorithm. As this guide recommends ``ashift=12`` (4 kiB blocks on disk), the common case of a 4 kiB page size means that no compression algorithm can reduce I/O. The exception is all-zero pages, which are dropped by ZFS; but some form of compression has to be enabled to get this behavior. 7.2 Configure the swap device: **Caution**: Always use long ``/dev/zvol`` aliases in configuration files. Never use a short ``/dev/zdX`` device name. :: # mkswap -f /dev/zvol/rpool/swap # echo /dev/zvol/rpool/swap none swap discard 0 0 >> /etc/fstab # echo RESUME=none > /etc/initramfs-tools/conf.d/resume The ``RESUME=none`` is necessary to disable resuming from hibernation. This does not work, as the zvol is not present (because the pool has not yet been imported) at the time the resume script runs. If it is not disabled, the boot process hangs for 30 seconds waiting for the swap zvol to appear. 7.3 Enable the swap device: :: # swapon -av Step 8: Full Software Installation ---------------------------------- 8.1 Upgrade the minimal system: :: # apt dist-upgrade --yes 8.2 Install a regular set of software: :: # tasksel **Note:** This will check "Debian desktop environment" and "print server" by default. If you want a server installation, unselect those. 8.3 Optional: Disable log compression: As ``/var/log`` is already compressed by ZFS, logrotate’s compression is going to burn CPU and disk I/O for (in most cases) very little gain. Also, if you are making snapshots of ``/var/log``, logrotate’s compression will actually waste space, as the uncompressed data will live on in the snapshot. You can edit the files in ``/etc/logrotate.d`` by hand to comment out ``compress``, or use this loop (copy-and-paste highly recommended): :: # for file in /etc/logrotate.d/* ; do if grep -Eq "(^|[^#y])compress" "$file" ; then sed -i -r "s/(^|[^#y])(compress)/\1#\2/" "$file" fi done 8.4 Reboot: :: # reboot Step 9: Final Cleanup ~~~~~~~~~~~~~~~~~~~~~ 9.1 Wait for the system to boot normally. Login using the account you created. Ensure the system (including networking) works normally. 9.2 Optional: Delete the snapshots of the initial installation: :: $ sudo zfs destroy bpool/BOOT/debian@install $ sudo zfs destroy rpool/ROOT/debian@install 9.3 Optional: Disable the root password :: $ sudo usermod -p '*' root 9.4 Optional: Re-enable the graphical boot process: If you prefer the graphical boot process, you can re-enable it now. If you are using LUKS, it makes the prompt look nicer. :: $ sudo vi /etc/default/grub Add quiet to GRUB_CMDLINE_LINUX_DEFAULT Comment out GRUB_TERMINAL=console Save and quit. $ sudo update-grub **Note:** Ignore errors from ``osprober``, if present. 9.5 Optional: For LUKS installs only, backup the LUKS header: :: $ sudo cryptsetup luksHeaderBackup /dev/disk/by-id/scsi-SATA_disk1-part4 \ --header-backup-file luks1-header.dat Store that backup somewhere safe (e.g. cloud storage). It is protected by your LUKS passphrase, but you may wish to use additional encryption. **Hint:** If you created a mirror or raidz topology, repeat this for each LUKS volume (``luks2``, etc.). Troubleshooting --------------- Rescuing using a Live CD ~~~~~~~~~~~~~~~~~~~~~~~~ Go through `Step 1: Prepare The Install Environment <#step-1-prepare-the-install-environment>`__. This will automatically import your pool. Export it and re-import it to get the mounts right: :: For LUKS, first unlock the disk(s): # apt install --yes cryptsetup # cryptsetup luksOpen /dev/disk/by-id/scsi-SATA_disk1-part4 luks1 Repeat for additional disks, if this is a mirror or raidz topology. # zpool export -a # zpool import -N -R /mnt rpool # zpool import -N -R /mnt bpool # zfs mount rpool/ROOT/debian # zfs mount -a If needed, you can chroot into your installed environment: :: # mount --rbind /dev /mnt/dev # mount --rbind /proc /mnt/proc # mount --rbind /sys /mnt/sys # chroot /mnt /bin/bash --login # mount /boot/efi # mount -a Do whatever you need to do to fix your system. When done, cleanup: :: # exit # mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {} # zpool export -a # reboot MPT2SAS ~~~~~~~ Most problem reports for this tutorial involve ``mpt2sas`` hardware that does slow asynchronous drive initialization, like some IBM M1015 or OEM-branded cards that have been flashed to the reference LSI firmware. The basic problem is that disks on these controllers are not visible to the Linux kernel until after the regular system is started, and ZoL does not hotplug pool members. See `https://github.com/openzfs/zfs/issues/330 `__. Most LSI cards are perfectly compatible with ZoL. If your card has this glitch, try setting ZFS_INITRD_PRE_MOUNTROOT_SLEEP=X in /etc/default/zfs. The system will wait X seconds for all drives to appear before importing the pool. Areca ~~~~~ Systems that require the ``arcsas`` blob driver should add it to the ``/etc/initramfs-tools/modules`` file and run ``update-initramfs -u -k all``. Upgrade or downgrade the Areca driver if something like ``RIP: 0010:[] [] native_read_tsc+0x6/0x20`` appears anywhere in kernel log. ZoL is unstable on systems that emit this error message. VMware ~~~~~~ - Set ``disk.EnableUUID = "TRUE"`` in the vmx file or vsphere configuration. Doing this ensures that ``/dev/disk`` aliases are created in the guest. QEMU/KVM/XEN ~~~~~~~~~~~~ Set a unique serial number on each virtual disk using libvirt or qemu (e.g. ``-drive if=none,id=disk1,file=disk1.qcow2,serial=1234567890``). To be able to use UEFI in guests (instead of only BIOS booting), run this on the host: :: $ sudo apt install ovmf $ sudo vi /etc/libvirt/qemu.conf Uncomment these lines: nvram = [ "/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd", "/usr/share/AAVMF/AAVMF_CODE.fd:/usr/share/AAVMF/AAVMF_VARS.fd" ] $ sudo service libvirt-bin restart .. highlight:: sh Debian Trixie Root on ZFS ========================= .. contents:: Table of Contents :local: Overview -------- Caution ~~~~~~~ - This HOWTO uses a whole physical disk. - Do not use these instructions for dual-booting. - Backup your data. Any existing data will be lost. System Requirements ~~~~~~~~~~~~~~~~~~~ - `64-bit Debian GNU/Linux Trixie Live CD w/ GUI (e.g. gnome iso) `__ - `A 64-bit kernel is strongly encouraged. <../../Project%20and%20Community/FAQ.html#bit-vs-64-bit-systems>`__ - Installing on a drive which presents 4 KiB logical sectors (a “4Kn” drive) only works with UEFI booting. This is not unique to ZFS. `GRUB does not and will not work on 4Kn with legacy (BIOS) booting. `__ Computers that have less than 2 GiB of memory run ZFS slowly. 4 GiB of memory is recommended for normal performance in basic workloads. If you wish to use deduplication, you will need `massive amounts of RAM `__. Enabling deduplication is a permanent change that cannot be easily reverted. Support ~~~~~~~ If you need help, reach out to the community using the :ref:`mailing_lists` or IRC at `#zfsonlinux `__ on `Libera Chat `__. If you have a bug report or feature request related to this HOWTO, please `file a new issue and mention @rlaager `__. Contributing ~~~~~~~~~~~~ #. Fork and clone: https://github.com/openzfs/openzfs-docs #. Install the tools:: sudo apt install python3-pip pip3 install -r docs/requirements.txt # Add ~/.local/bin to your $PATH, e.g. by adding this to ~/.bashrc: PATH=$HOME/.local/bin:$PATH #. Make your changes. #. Test:: cd docs make html sensible-browser _build/html/index.html #. ``git commit --signoff`` to a branch, ``git push``, and create a pull request. Mention @rlaager. Encryption ~~~~~~~~~~ This guide supports three different encryption options: unencrypted, ZFS native encryption, and LUKS. With any option, all ZFS features are fully available. Unencrypted does not encrypt anything, of course. With no encryption happening, this option naturally has the best performance. ZFS native encryption encrypts the data and most metadata in the root pool. It does not encrypt dataset or snapshot names or properties. The boot pool is not encrypted at all, but it only contains the bootloader, kernel, and initrd. (Unless you put a password in ``/etc/fstab``, the initrd is unlikely to contain sensitive data.) The system cannot boot without the passphrase being entered at the console. Performance is good. As the encryption happens in ZFS, even if multiple disks (mirror or raidz topologies) are used, the data only has to be encrypted once. LUKS encrypts almost everything. The only unencrypted data is the bootloader, kernel, and initrd. The system cannot boot without the passphrase being entered at the console. Performance is good, but LUKS sits underneath ZFS, so if multiple disks (mirror or raidz topologies) are used, the data has to be encrypted once per disk. Step 1: Prepare The Install Environment --------------------------------------- #. Boot the Debian GNU/Linux Live CD. If prompted, login with the username ``user`` and password ``live``. Connect your system to the Internet as appropriate (e.g. join your WiFi network). Open a terminal. #. Setup and update the repositories:: sudo vi /etc/apt/sources.list .. code-block:: sourceslist deb http://deb.debian.org/debian trixie main contrib non-free-firmware :: sudo apt update #. Optional: Install and start the OpenSSH server in the Live CD environment: If you have a second system, using SSH to access the target system can be convenient:: sudo apt install --yes openssh-server sudo systemctl restart ssh **Hint:** You can find your IP address with ``ip addr show scope global | grep inet``. Then, from your main machine, connect with ``ssh user@IP``. #. Disable automounting: If the disk has been used before (with partitions at the same offsets), previous filesystems (e.g. the ESP) will automount if not disabled:: gsettings set org.gnome.desktop.media-handling automount false #. Become root:: sudo -i #. Install missing prerequisites in zfsutils-linux package (`bug 1091428 `_):: apt install --yes linux-headers-generic #. Install ZFS in the Live CD environment:: apt install --yes debootstrap gdisk zfsutils-linux Step 2: Disk Formatting ----------------------- #. Set a variable with the disk name:: DISK=/dev/disk/by-id/scsi-SATA_disk1 Always use the long ``/dev/disk/by-id/*`` aliases with ZFS. Using the ``/dev/sd*`` device nodes directly can cause sporadic import failures, especially on systems that have more than one storage pool. **Hints:** - ``ls -la /dev/disk/by-id`` will list the aliases. - Are you doing this in a virtual machine? If your virtual disk is missing from ``/dev/disk/by-id``, use ``/dev/vda`` if you are using KVM with virtio. Also when using /dev/vda, the partitions used later will be named differently. Otherwise, read the `troubleshooting <#troubleshooting>`__ section. - For a mirror or raidz topology, use ``DISK1``, ``DISK2``, etc. - When choosing a boot pool size, consider how you will use the space. A kernel and initrd may consume around 100M. If you have multiple kernels and take snapshots, you may find yourself low on boot pool space, especially if you need to regenerate your initramfs images, which may be around 85M each. Size your boot pool appropriately for your needs. #. If you are re-using a disk, clear it as necessary: Ensure swap partitions are not in use:: swapoff --all If the disk was previously used in an MD array:: apt install --yes mdadm # See if one or more MD arrays are active: cat /proc/mdstat # If so, stop them (replace ``md0`` as required): mdadm --stop /dev/md0 # For an array using the whole disk: mdadm --zero-superblock --force $DISK # For an array using a partition: mdadm --zero-superblock --force ${DISK}-part2 If the disk was previously used with zfs:: wipefs -a $DISK For flash-based storage, if the disk was previously used, you may wish to do a full-disk discard (TRIM/UNMAP), which can improve performance:: blkdiscard -f $DISK Clear the partition table:: sgdisk --zap-all $DISK If you get a message about the kernel still using the old partition table, you can request the kernel reload the partition information using:: partprobe $DISK If the new partitions still don't show up, you can reboot and start over (except that you can skip this step). #. Partition your disk(s): Run this if you need legacy (BIOS) booting:: sgdisk -a1 -n1:24K:+1000K -t1:EF02 $DISK Run this for UEFI booting (for use now or in the future):: sgdisk -n2:1M:+512M -t2:EF00 $DISK Run this for the boot pool:: sgdisk -n3:0:+1G -t3:BF01 $DISK Choose one of the following options: - Unencrypted or ZFS native encryption:: sgdisk -n4:0:0 -t4:BF00 $DISK - LUKS:: sgdisk -n4:0:0 -t4:8309 $DISK If you are creating a mirror or raidz topology, repeat the partitioning commands for all the disks which will be part of the pool. #. Create the boot pool:: zpool create \ -o ashift=12 \ -o autotrim=on \ -o compatibility=grub2 \ -o cachefile=/etc/zfs/zpool.cache \ -O devices=off \ -O acltype=posixacl -O xattr=sa \ -O compression=lz4 \ -O normalization=formD \ -O relatime=on \ -O canmount=off -O mountpoint=/boot -R /mnt \ bpool ${DISK}-part3 *Note:* GRUB does not support all zpool features (see ``spa_feature_names`` in `grub-core/fs/zfs/zfs.c `_). We create a separate zpool for ``/boot`` here, specifying the ``-o compatibility=grub2`` property which restricts the pool to only those features that GRUB supports, allowing the root pool to use any/all features. See the section on ``Compatibility feature sets`` in the ``zpool-features`` man page for more information. **Hints:** - If you are creating a mirror topology, create the pool using:: zpool create \ ... \ bpool mirror \ /dev/disk/by-id/scsi-SATA_disk1-part3 \ /dev/disk/by-id/scsi-SATA_disk2-part3 - For raidz topologies, replace ``mirror`` in the above command with ``raidz``, ``raidz2``, or ``raidz3`` and list the partitions from the additional disks. - The pool name is arbitrary. If changed, the new name must be used consistently. The ``bpool`` convention originated in this HOWTO. #. Create the root pool: Choose one of the following options: - Unencrypted:: zpool create \ -o ashift=12 \ -o autotrim=on \ -O acltype=posixacl -O xattr=sa -O dnodesize=auto \ -O compression=lz4 \ -O normalization=formD \ -O relatime=on \ -O canmount=off -O mountpoint=/ -R /mnt \ rpool ${DISK}-part4 - ZFS native encryption:: zpool create \ -o ashift=12 \ -o autotrim=on \ -O encryption=on -O keylocation=prompt -O keyformat=passphrase \ -O acltype=posixacl -O xattr=sa -O dnodesize=auto \ -O compression=lz4 \ -O normalization=formD \ -O relatime=on \ -O canmount=off -O mountpoint=/ -R /mnt \ rpool ${DISK}-part4 - LUKS:: apt install --yes cryptsetup cryptsetup luksFormat -c aes-xts-plain64 -s 512 -h sha256 ${DISK}-part4 cryptsetup luksOpen ${DISK}-part4 luks1 zpool create \ -o ashift=12 \ -o autotrim=on \ -O acltype=posixacl -O xattr=sa -O dnodesize=auto \ -O compression=lz4 \ -O normalization=formD \ -O relatime=on \ -O canmount=off -O mountpoint=/ -R /mnt \ rpool /dev/mapper/luks1 **Notes:** - The use of ``ashift=12`` is recommended here because many drives today have 4 KiB (or larger) physical sectors, even though they present 512 B logical sectors. Also, a future replacement drive may have 4 KiB physical sectors (in which case ``ashift=12`` is desirable) or 4 KiB logical sectors (in which case ``ashift=12`` is required). - Setting ``-O acltype=posixacl`` enables POSIX ACLs globally. If you do not want this, remove that option, but later add ``-o acltype=posixacl`` (note: lowercase “o”) to the ``zfs create`` for ``/var/log``, as `journald requires ACLs `__ - Setting ``xattr=sa`` `vastly improves the performance of extended attributes `__. Inside ZFS, extended attributes are used to implement POSIX ACLs. Extended attributes can also be used by user-space applications. `They are used by some desktop GUI applications. `__ `They can be used by Samba to store Windows ACLs and DOS attributes; they are required for a Samba Active Directory domain controller. `__ Note that ``xattr=sa`` is `Linux-specific `__. If you move your ``xattr=sa`` pool to another OpenZFS implementation besides ZFS-on-Linux, extended attributes will not be readable (though your data will be). If portability of extended attributes is important to you, omit the ``-O xattr=sa`` above. Even if you do not want ``xattr=sa`` for the whole pool, it is probably fine to use it for ``/var/log``. - Setting ``normalization=formD`` eliminates some corner cases relating to UTF-8 filename normalization. It also implies ``utf8only=on``, which means that only UTF-8 filenames are allowed. If you care to support non-UTF-8 filenames, do not use this option. For a discussion of why requiring UTF-8 filenames may be a bad idea, see `The problems with enforced UTF-8 only filenames `__. - ``recordsize`` is unset (leaving it at the default of 128 KiB). If you want to tune it (e.g. ``-O recordsize=1M``), see `these `__ `various `__ `blog `__ `posts `__. - Setting ``relatime=on`` is a middle ground between classic POSIX ``atime`` behavior (with its significant performance impact) and ``atime=off`` (which provides the best performance by completely disabling atime updates). Since Linux 2.6.30, ``relatime`` has been the default for other filesystems. See `RedHat’s documentation `__ for further information. - Make sure to include the ``-part4`` portion of the drive path. If you forget that, you are specifying the whole disk, which ZFS will then re-partition, and you will lose the bootloader partition(s). - ZFS native encryption `now `__ defaults to ``aes-256-gcm``. - For LUKS, the key size chosen is 512 bits. However, XTS mode requires two keys, so the LUKS key is split in half. Thus, ``-s 512`` means AES-256. - Your passphrase will likely be the weakest link. Choose wisely. See `section 5 of the cryptsetup FAQ `__ for guidance. **Hints:** - If you are creating a mirror topology, create the pool using:: zpool create \ ... \ rpool mirror \ /dev/disk/by-id/scsi-SATA_disk1-part4 \ /dev/disk/by-id/scsi-SATA_disk2-part4 - For raidz topologies, replace ``mirror`` in the above command with ``raidz``, ``raidz2``, or ``raidz3`` and list the partitions from the additional disks. - When using LUKS with mirror or raidz topologies, use ``/dev/mapper/luks1``, ``/dev/mapper/luks2``, etc., which you will have to create using ``cryptsetup``. - The pool name is arbitrary. If changed, the new name must be used consistently. On systems that can automatically install to ZFS, the root pool is named ``rpool`` by default. Step 3: System Installation --------------------------- #. Create filesystem datasets to act as containers:: zfs create -o canmount=off -o mountpoint=none rpool/ROOT zfs create -o canmount=off -o mountpoint=none bpool/BOOT On Solaris systems, the root filesystem is cloned and the suffix is incremented for major system changes through ``pkg image-update`` or ``beadm``. Similar functionality was implemented in Ubuntu with the ``zsys`` tool, though its dataset layout is more complicated, and ``zsys`` `is on life support `__. Even without such a tool, the `rpool/ROOT` and `bpool/BOOT` containers can still be used for manually created clones. That said, this HOWTO assumes a single filesystem for ``/boot`` for simplicity. #. Create filesystem datasets for the root and boot filesystems:: zfs create -o canmount=noauto -o mountpoint=/ rpool/ROOT/debian zfs mount rpool/ROOT/debian zfs create -o mountpoint=/boot bpool/BOOT/debian With ZFS, it is not normally necessary to use a mount command (either ``mount`` or ``zfs mount``). This situation is an exception because of ``canmount=noauto``. #. Create datasets:: zfs create rpool/home zfs create -o mountpoint=/root rpool/home/root chmod 700 /mnt/root zfs create -o canmount=off rpool/var zfs create -o canmount=off rpool/var/lib zfs create rpool/var/log zfs create rpool/var/spool The datasets below are optional, depending on your preferences and/or software choices. If you wish to separate these to exclude them from snapshots:: zfs create -o com.sun:auto-snapshot=false rpool/var/cache zfs create -o com.sun:auto-snapshot=false rpool/var/lib/nfs If you use /srv on this system:: zfs create rpool/srv If you use /usr/local on this system:: zfs create -o canmount=off rpool/usr zfs create rpool/usr/local If this system will have games installed:: zfs create rpool/var/games If this system will have a GUI:: zfs create rpool/var/lib/AccountsService zfs create rpool/var/lib/NetworkManager If this system will use Docker (which manages its own datasets & snapshots):: zfs create -o com.sun:auto-snapshot=false rpool/var/lib/docker If this system will store local email in /var/mail:: zfs create rpool/var/mail If this system will use Snap packages:: zfs create rpool/var/snap If you use /var/www on this system:: zfs create rpool/var/www **Note:** As tmpfs is used by default by Debian Trixie, all ``/tmp`` related operations in previous versions are now obsolete. **Note:** If you separate a directory required for booting (e.g. ``/etc``) into its own dataset, you must add it to ``ZFS_INITRD_ADDITIONAL_DATASETS`` in ``/etc/default/zfs``. Datasets with ``canmount=off`` (like ``rpool/usr`` above) do not matter for this. #. Mount a tmpfs at /run:: mkdir /mnt/run mount -t tmpfs tmpfs /mnt/run mkdir /mnt/run/lock #. Install the minimal system:: debootstrap trixie /mnt The ``debootstrap`` command leaves the new system in an unconfigured state. An alternative to using ``debootstrap`` is to copy the entirety of a working system into the new ZFS root. #. Copy in zpool.cache:: mkdir /mnt/etc/zfs cp /etc/zfs/zpool.cache /mnt/etc/zfs/ Step 4: System Configuration ---------------------------- #. Configure the hostname: Replace ``HOSTNAME`` with the desired hostname:: hostname HOSTNAME hostname > /mnt/etc/hostname vi /mnt/etc/hosts .. code-block:: text Add a line: 127.0.1.1 HOSTNAME or if the system has a real name in DNS: 127.0.1.1 FQDN HOSTNAME **Hint:** Use ``nano`` if you find ``vi`` confusing. #. Configure the network interface: Find the interface name:: ip addr show Adjust ``NAME`` below to match your interface name:: vi /mnt/etc/network/interfaces.d/NAME .. code-block:: text auto NAME iface NAME inet dhcp Customize this file if the system is not a DHCP client. #. Configure the package sources:: vi /mnt/etc/apt/sources.list .. code-block:: sourceslist deb http://deb.debian.org/debian trixie main contrib non-free-firmware deb-src http://deb.debian.org/debian trixie main contrib non-free-firmware deb http://deb.debian.org/debian-security trixie-security main contrib non-free-firmware deb-src http://deb.debian.org/debian-security trixie-security main contrib non-free-firmware deb http://deb.debian.org/debian trixie-updates main contrib non-free-firmware deb-src http://deb.debian.org/debian trixie-updates main contrib non-free-firmware #. Bind the virtual filesystems from the LiveCD environment to the new system and ``chroot`` into it:: mount --make-private --rbind /dev /mnt/dev mount --make-private --rbind /proc /mnt/proc mount --make-private --rbind /sys /mnt/sys chroot /mnt /usr/bin/env DISK=$DISK bash --login **Note:** This is using ``--rbind``, not ``--bind``. #. Configure a basic system environment:: apt update apt install --yes console-setup locales Even if you prefer a non-English system language, always ensure that ``en_US.UTF-8`` is available:: dpkg-reconfigure locales tzdata keyboard-configuration console-setup #. Optional: Install driver firmware and WiFi support If you're installing on a laptop or a device where wireless is the primary network option, the network configuration above may not be sufficient as you could lack the appropriate firmware for the device and tools to configure the radio. Install some additional packages to cover that need:: apt install --yes firmware-linux iw wpasupplicant ``wpasupplicant`` is required to join WPA-protected networks. If you install a desktop environment later, NetworkManager will take over the configuration of the radio. #. Install ZFS in the chroot environment for the new system:: apt install --yes dpkg-dev linux-headers-generic linux-image-generic apt install --yes zfs-initramfs **Note:** Ignore any error messages saying ``ERROR: Couldn't resolve device`` and ``WARNING: Couldn't determine root device``. `cryptsetup does not support ZFS `__. #. For LUKS installs only, setup ``/etc/crypttab``:: apt install --yes cryptsetup cryptsetup-initramfs echo luks1 /dev/disk/by-uuid/$(blkid -s UUID -o value ${DISK}-part4) \ none luks,discard,initramfs > /etc/crypttab The use of ``initramfs`` is a work-around for `cryptsetup does not support ZFS `__. **Hint:** If you are creating a mirror or raidz topology, repeat the ``/etc/crypttab`` entries for ``luks2``, etc. adjusting for each disk. #. Install an NTP service to synchronize time. This step is specific to Trixie which does not install the package during bootstrap. Although this step is not necessary for ZFS, it is useful for internet browsing where local clock drift can cause login failures:: apt install --yes systemd-timesyncd #. Install GRUB Choose one of the following options: - Install GRUB for legacy (BIOS) booting:: apt install --yes grub-pc - Install GRUB for UEFI booting:: apt install --yes dosfstools mkdosfs -F 32 -s 1 -n EFI ${DISK}-part2 mkdir /boot/efi echo /dev/disk/by-uuid/$(blkid -s UUID -o value ${DISK}-part2) \ /boot/efi vfat defaults 0 0 >> /etc/fstab mount /boot/efi apt install --yes grub-efi-amd64 shim-signed **Notes:** - The ``-s 1`` for ``mkdosfs`` is only necessary for drives which present 4 KiB logical sectors (“4Kn” drives) to meet the minimum cluster size (given the partition size of 512 MiB) for FAT32. It also works fine on drives which present 512 B sectors. - For a mirror or raidz topology, this step only installs GRUB on the first disk. The other disk(s) will be handled later. #. Optional: Remove os-prober:: apt purge --yes os-prober This avoids error messages from `update-grub`. `os-prober` is only necessary in dual-boot configurations. #. Set a root password:: passwd #. Enable importing bpool This ensures that ``bpool`` is always imported, regardless of whether ``/etc/zfs/zpool.cache`` exists, whether it is in the cachefile or not, or whether ``zfs-import-scan.service`` is enabled. :: vi /etc/systemd/system/zfs-import-bpool.service .. code-block:: ini [Unit] DefaultDependencies=no Before=zfs-import-scan.service Before=zfs-import-cache.service [Service] Type=oneshot RemainAfterExit=yes ExecStart=/sbin/zpool import -N -o cachefile=none bpool # Work-around to preserve zpool cache: ExecStartPre=-/bin/mv /etc/zfs/zpool.cache /etc/zfs/preboot_zpool.cache ExecStartPost=-/bin/mv /etc/zfs/preboot_zpool.cache /etc/zfs/zpool.cache [Install] WantedBy=zfs-import.target :: systemctl enable zfs-import-bpool.service **Note:** For some disk configurations (NVMe?), this service `may fail `__ with an error indicating that the ``bpool`` cannot be found. If this happens, add ``-d DISK-part3`` (replace ``DISK`` with the correct device path) to the ``zpool import`` command. #. Optional: Install SSH:: apt install --yes openssh-server vi /etc/ssh/sshd_config # Set: PermitRootLogin yes #. Optional: For ZFS native encryption or LUKS, configure Dropbear for remote unlocking:: apt install --yes --no-install-recommends dropbear-initramfs mkdir -p /etc/dropbear/initramfs # Optional: Convert OpenSSH server keys for Dropbear for type in ecdsa ed25519 rsa ; do cp /etc/ssh/ssh_host_${type}_key /tmp/openssh.key ssh-keygen -p -N "" -m PEM -f /tmp/openssh.key dropbearconvert openssh dropbear \ /tmp/openssh.key \ /etc/dropbear/initramfs/dropbear_${type}_host_key done rm /tmp/openssh.key # Add user keys in the same format as ~/.ssh/authorized_keys vi /etc/dropbear/initramfs/authorized_keys # If using a static IP, set it for the initramfs environment: vi /etc/initramfs-tools/initramfs.conf # The syntax is: IP=ADDRESS::GATEWAY:MASK:HOSTNAME:NIC # For example: # IP=192.168.1.100::192.168.1.1:255.255.255.0:myhostname:ens3 # HOSTNAME and NIC are optional. # Rebuild the initramfs (required when changing any of the above): update-initramfs -u -k all **Notes:** - Converting the server keys makes Dropbear use the same keys as OpenSSH, avoiding host key mismatch warnings. Currently, `dropbearconvert doesn't understand the new OpenSSH private key format `__, so the keys need to be converted to the old PEM format first using ``ssh-keygen``. The downside of using the same keys for both OpenSSH and Dropbear is that the OpenSSH keys are then available on-disk, unencrypted in the initramfs. - Later, to use this functionality, SSH to the system (as root) while it is prompting for the passphrase during the boot process. For ZFS native encryption, run ``zfsunlock``. For LUKS, run ``cryptroot-unlock``. - You can optionally add ``command="/usr/bin/zfsunlock"`` or ``command="/bin/cryptroot-unlock"`` in front of the ``authorized_keys`` line to force the unlock command. This way, the unlock command runs automatically and is all that can be run. #. Optional (but kindly requested): Install popcon The ``popularity-contest`` package reports the list of packages installed on your system. Showing that ZFS is popular may be helpful in terms of long-term attention from the distro. :: apt install --yes popularity-contest Choose Yes at the prompt. Step 5: GRUB Installation ------------------------- #. Verify that the ZFS boot filesystem is recognized:: grub-probe /boot #. Refresh the initrd files:: update-initramfs -c -k all **Note:** Ignore any error messages saying ``ERROR: Couldn't resolve device`` and ``WARNING: Couldn't determine root device``. `cryptsetup does not support ZFS `__. #. Workaround GRUB's missing zpool-features support:: vi /etc/default/grub # Set: GRUB_CMDLINE_LINUX="root=ZFS=rpool/ROOT/debian" #. Optional (but highly recommended): Make debugging GRUB easier:: vi /etc/default/grub # Remove quiet from: GRUB_CMDLINE_LINUX_DEFAULT # Uncomment: GRUB_TERMINAL=console # Save and quit. Later, once the system has rebooted twice and you are sure everything is working, you can undo these changes, if desired. #. Update the boot configuration:: update-grub **Note:** Ignore errors from ``osprober``, if present. #. Install the boot loader: Choose one of the following options: - For legacy (BIOS) booting, install GRUB to the MBR:: grub-install $DISK Note that you are installing GRUB to the whole disk, not a partition. If you are creating a mirror or raidz topology, repeat the ``grub-install`` command for each disk in the pool. - For UEFI booting, install GRUB to the ESP:: grub-install --target=x86_64-efi --efi-directory=/boot/efi \ --bootloader-id=debian --recheck It is not necessary to specify the disk here. If you are creating a mirror or raidz topology, the additional disks will be handled later. #. Fix filesystem mount ordering: We need to activate ``zfs-mount-generator``. This makes systemd aware of the separate mountpoints, which is important for things like ``/var/log`` and ``/var/tmp``. In turn, ``rsyslog.service`` depends on ``var-log.mount`` by way of ``local-fs.target`` and services using the ``PrivateTmp`` feature of systemd automatically use ``After=var-tmp.mount``. :: mkdir /etc/zfs/zfs-list.cache touch /etc/zfs/zfs-list.cache/bpool touch /etc/zfs/zfs-list.cache/rpool zed -F & Verify that ``zed`` updated the cache by making sure these are not empty:: cat /etc/zfs/zfs-list.cache/bpool cat /etc/zfs/zfs-list.cache/rpool If either is empty, force a cache update and check again:: zfs set canmount=on bpool/BOOT/debian zfs set canmount=noauto rpool/ROOT/debian If they are still empty, stop zed (as below), start zed (as above) and try again. Once the files have data, stop ``zed``:: fg Press Ctrl-C. Fix the paths to eliminate ``/mnt``:: sed -Ei "s|/mnt/?|/|" /etc/zfs/zfs-list.cache/* Step 6: First Boot ------------------ #. Optional: Snapshot the initial installation:: zfs snapshot bpool/BOOT/debian@install zfs snapshot rpool/ROOT/debian@install In the future, you will likely want to take snapshots before each upgrade, and remove old snapshots (including this one) at some point to save space. #. Exit from the ``chroot`` environment back to the LiveCD environment:: exit #. Run these commands in the LiveCD environment to unmount all filesystems:: mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | \ xargs -i{} umount -lf {} zpool export -a #. If export failed due to `busy` error, try to kill everything that might be using it:: grep [p]ool /proc/*/mounts | cut -d/ -f3 | uniq | xargs kill zpool export -a #. If even after that your pool is busy, mounting it on boot will fail and you will need to ``zpool import -f rpool``, then ``exit`` in the initramfs prompt. #. Reboot:: reboot Wait for the newly installed system to boot normally. Login as root. #. Create a user account: Replace ``YOUR_USERNAME`` with your desired username:: username=YOUR_USERNAME zfs create rpool/home/$username adduser $username cp -a /etc/skel/. /home/$username chown -R $username:$username /home/$username usermod -a -G audio,cdrom,dip,floppy,netdev,plugdev,sudo,video $username #. Mirror GRUB If you installed to multiple disks, install GRUB on the additional disks. - For legacy (BIOS) booting:: dpkg-reconfigure grub-pc Hit enter until you get to the device selection screen. Select (using the space bar) all of the disks (not partitions) in your pool. - For UEFI booting:: umount /boot/efi For the second and subsequent disks (increment debian-2 to -3, etc.):: dd if=/dev/disk/by-id/scsi-SATA_disk1-part2 \ of=/dev/disk/by-id/scsi-SATA_disk2-part2 efibootmgr -c -g -d /dev/disk/by-id/scsi-SATA_disk2 \ -p 2 -L "debian-2" -l '\EFI\debian\grubx64.efi' mount /boot/efi Step 7: Optional: Configure Swap --------------------------------- **Caution**: On systems with extremely high memory pressure, using a zvol for swap can result in lockup, regardless of how much swap is still available. There is `a bug report upstream `__. #. Create a volume dataset (zvol) for use as a swap device:: zfs create -V 4G -b $(getconf PAGESIZE) -o compression=zle \ -o logbias=throughput -o sync=always \ -o primarycache=metadata -o secondarycache=none \ -o com.sun:auto-snapshot=false rpool/swap You can adjust the size (the ``4G`` part) to your needs. The compression algorithm is set to ``zle`` because it is the cheapest available algorithm. As this guide recommends ``ashift=12`` (4 kiB blocks on disk), the common case of a 4 kiB page size means that no compression algorithm can reduce I/O. The exception is all-zero pages, which are dropped by ZFS; but some form of compression has to be enabled to get this behavior. #. Configure the swap device: **Caution**: Always use long ``/dev/zvol`` aliases in configuration files. Never use a short ``/dev/zdX`` device name. :: mkswap -f /dev/zvol/rpool/swap echo /dev/zvol/rpool/swap none swap discard 0 0 >> /etc/fstab echo RESUME=none > /etc/initramfs-tools/conf.d/resume The ``RESUME=none`` is necessary to disable resuming from hibernation. This does not work, as the zvol is not present (because the pool has not yet been imported) at the time the resume script runs. If it is not disabled, the boot process hangs for 30 seconds waiting for the swap zvol to appear. #. Enable the swap device:: swapon -av Step 8: Full Software Installation ---------------------------------- #. Upgrade the minimal system:: apt dist-upgrade --yes #. Install a regular set of software:: apt install --yes tasksel tasksel --new-install **Note:** This will check "Debian desktop environment" and "print server" by default. If you want a server installation, unselect those. #. Optional: Disable log compression: As ``/var/log`` is already compressed by ZFS, logrotate’s compression is going to burn CPU and disk I/O for (in most cases) very little gain. Also, if you are making snapshots of ``/var/log``, logrotate’s compression will actually waste space, as the uncompressed data will live on in the snapshot. You can edit the files in ``/etc/logrotate.d`` by hand to comment out ``compress``, or use this loop (copy-and-paste highly recommended):: for file in /etc/logrotate.d/* ; do if grep -Eq "(^|[^#y])compress" "$file" ; then sed -i -r "s/(^|[^#y])(compress)/\1#\2/" "$file" fi done #. Reboot:: reboot Step 9: Final Cleanup --------------------- #. Wait for the system to boot normally. Login using the account you created. Ensure the system (including networking) works normally. #. Optional: Delete the snapshots of the initial installation:: sudo zfs destroy bpool/BOOT/debian@install sudo zfs destroy rpool/ROOT/debian@install #. Optional: Disable the root password:: sudo usermod -p '*' root #. Optional (but highly recommended): Disable root SSH logins: If you installed SSH earlier, revert the temporary change:: sudo vi /etc/ssh/sshd_config # Remove: PermitRootLogin yes sudo systemctl restart ssh #. Optional: Re-enable the graphical boot process: If you prefer the graphical boot process, you can re-enable it now. If you are using LUKS, it makes the prompt look nicer. :: sudo vi /etc/default/grub # Add quiet to GRUB_CMDLINE_LINUX_DEFAULT # Comment out GRUB_TERMINAL=console # Save and quit. sudo update-grub **Note:** Ignore errors from ``osprober``, if present. #. Optional: For LUKS installs only, backup the LUKS header:: sudo cryptsetup luksHeaderBackup /dev/disk/by-id/scsi-SATA_disk1-part4 \ --header-backup-file luks1-header.dat Store that backup somewhere safe (e.g. cloud storage). It is protected by your LUKS passphrase, but you may wish to use additional encryption. **Hint:** If you created a mirror or raidz topology, repeat this for each LUKS volume (``luks2``, etc.). Troubleshooting --------------- Rescuing using a Live CD ~~~~~~~~~~~~~~~~~~~~~~~~ Go through `Step 1: Prepare The Install Environment <#step-1-prepare-the-install-environment>`__. For LUKS, first unlock the disk(s):: apt install --yes cryptsetup cryptsetup luksOpen /dev/disk/by-id/scsi-SATA_disk1-part4 luks1 # Repeat for additional disks, if this is a mirror or raidz topology. Mount everything correctly:: zpool export -a zpool import -N -R /mnt rpool zpool import -N -R /mnt bpool zfs load-key -a zfs mount rpool/ROOT/debian zfs mount -a If needed, you can chroot into your installed environment:: mount --make-private --rbind /dev /mnt/dev mount --make-private --rbind /proc /mnt/proc mount --make-private --rbind /sys /mnt/sys mount -t tmpfs tmpfs /mnt/run mkdir /mnt/run/lock chroot /mnt /bin/bash --login mount /boot/efi mount -a Do whatever you need to do to fix your system. When done, cleanup:: exit mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | \ xargs -i{} umount -lf {} zpool export -a reboot Areca ~~~~~ Systems that require the ``arcsas`` blob driver should add it to the ``/etc/initramfs-tools/modules`` file and run ``update-initramfs -c -k all``. Upgrade or downgrade the Areca driver if something like ``RIP: 0010:[] [] native_read_tsc+0x6/0x20`` appears anywhere in kernel log. ZoL is unstable on systems that emit this error message. MPT2SAS ~~~~~~~ Most problem reports for this tutorial involve ``mpt2sas`` hardware that does slow asynchronous drive initialization, like some IBM M1015 or OEM-branded cards that have been flashed to the reference LSI firmware. The basic problem is that disks on these controllers are not visible to the Linux kernel until after the regular system is started, and ZoL does not hotplug pool members. See `https://github.com/openzfs/zfs/issues/330 `__. Most LSI cards are perfectly compatible with ZoL. If your card has this glitch, try setting ``ZFS_INITRD_PRE_MOUNTROOT_SLEEP=X`` in ``/etc/default/zfs``. The system will wait ``X`` seconds for all drives to appear before importing the pool. QEMU/KVM/XEN ~~~~~~~~~~~~ Set a unique serial number on each virtual disk using libvirt or qemu (e.g. ``-drive if=none,id=disk1,file=disk1.qcow2,serial=1234567890``). To be able to use UEFI in guests (instead of only BIOS booting), run this on the host:: sudo apt install ovmf sudo vi /etc/libvirt/qemu.conf Uncomment these lines: .. code-block:: text nvram = [ "/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd", "/usr/share/OVMF/OVMF_CODE.secboot.fd:/usr/share/OVMF/OVMF_VARS.fd", "/usr/share/AAVMF/AAVMF_CODE.fd:/usr/share/AAVMF/AAVMF_VARS.fd", "/usr/share/AAVMF/AAVMF32_CODE.fd:/usr/share/AAVMF/AAVMF32_VARS.fd" ] :: sudo systemctl restart libvirtd.service VMware ~~~~~~ - Set ``disk.EnableUUID = "TRUE"`` in the vmx file or vsphere configuration. Doing this ensures that ``/dev/disk`` aliases are created in the guest. Debian GNU Linux initrd documentation ===================================== Supported boot parameters ************************* - rollback= Do a rollback of specified snapshot. - zfs_debug= Debug the initrd script - zfs_force= Force importing the pool. Should not be necessary. - zfs= Don't try to import ANY pool, mount ANY filesystem or even load the module. - rpool= Use this pool for root pool. - bootfs=/ Use this dataset for root filesystem. - root=/ Use this dataset for root filesystem. - root=ZFS=/ Use this dataset for root filesystem. - root=zfs:/ Use this dataset for root filesystem. - root=zfs:AUTO Try to detect both pool and rootfs In all these cases, could also be @. The reason there are so many supported boot options to get the root filesystem, is that there are a lot of different ways too boot ZFS out there, and I wanted to make sure I supported them all. Pool imports ************ Import using /dev/disk/by-\* ---------------------------- The initrd will, if the variable USE_DISK_BY_ID is set in the file /etc/default/zfs, to import using the /dev/disk/by-\* links. It will try to import in this order: 1. /dev/disk/by-vdev 2. /dev/disk/by-\* 3. /dev Import using cache file ----------------------- If all of these imports fail (or if USE_DISK_BY_ID is unset), it will then try to import using the cache file. Last ditch attempt at importing ------------------------------- If that ALSO fails, it will try one more time, without any -d or -c options. Booting ******* Booting from snapshot: ---------------------- Enter the snapshot for the root= parameter like in this example: :: linux /BOOT/debian@/boot/vmlinuz-5.10.0-9-amd64 root=ZFS=rpool/ROOT/debian@some_snapshot ro This will clone the snapshot rpool/ROOT/debian@some_snapshot into the filesystem rpool/ROOT/debian_some_snapshot and use that as root filesystem. The original filesystem and snapshot is left alone in this case. **BEWARE** that it will first destroy, blindingly, the rpool/ROOT/debian_some_snapshot filesystem before trying to clone the snapshot into it again. So if you've booted from the same snapshot previously and done some changes in that root filesystem, they will be undone by the destruction of the filesystem. Snapshot rollback ----------------- From version 0.6.4-1-3 it is now also possible to specify rollback=1 to do a rollback of the snapshot instead of cloning it. **BEWARE** that this will destroy *all* snapshots done after the specified snapshot! Select snapshot dynamically --------------------------- From version 0.6.4-1-3 it is now also possible to specify a NULL snapshot name (such as root=rpool/ROOT/debian@) and if so, the initrd script will discover all snapshots below that filesystem (sans the at), and output a list of snapshot for the user to choose from. Booting from native encrypted filesystem ---------------------------------------- Although there is currently no support for native encryption in ZFS On Linux, there is a patch floating around 'out there' and the initrd supports loading key and unlock such encrypted filesystem. Separated filesystems --------------------- Descended filesystems ~~~~~~~~~~~~~~~~~~~~~ If there are separate filesystems (for example a separate dataset for /usr), the snapshot boot code will try to find the snapshot under each filesystems and clone (or rollback) them. Example: :: rpool/ROOT/debian@some_snapshot rpool/ROOT/debian/usr@some_snapshot These will create the following filesystems respectively (if not doing a rollback): :: rpool/ROOT/debian_some_snapshot rpool/ROOT/debian/usr_some_snapshot The initrd code will use the mountpoint option (if any) in the original (without the snapshot part) dataset to find *where* it should mount the dataset. Or it will use the name of the dataset below the root filesystem (rpool/ROOT/debian in this example) for the mount point. Fedora ====== Contents -------- .. toctree:: :maxdepth: 1 :glob: * Installation ------------ Note: this is for installing ZFS on an existing Fedora installation. To use ZFS as root file system, see below. #. If ``zfs-fuse`` from official Fedora repo is installed, remove it first. It is not maintained and should not be used under any circumstance:: rpm -e --nodeps zfs-fuse #. Add ZFS repo:: dnf install -y https://zfsonlinux.org/fedora/zfs-release-3-1$(rpm --eval "%{dist}").noarch.rpm List of old zfs-release RPMs are available `here `__. #. Install kernel headers:: dnf install -y kernel-devel-$(uname -r | awk -F'-' '{print $1}') ``kernel-devel`` package must be installed before ``zfs`` package. #. Install ZFS packages:: dnf install -y zfs #. Load kernel module:: modprobe zfs If kernel module can not be loaded, your kernel version might be not yet supported by OpenZFS. An option is to an LTS kernel from COPR, provided by a third-party. Use it at your own risk:: # this is a third-party repo! # you have been warned. # # select a kernel from # https://copr.fedorainfracloud.org/coprs/kwizart/ dnf copr enable -y kwizart/kernel-longterm-VERSION dnf install -y kernel-longterm kernel-longterm-devel Reboot to new LTS kernel, then load kernel module:: modprobe zfs #. By default ZFS kernel modules are loaded upon detecting a pool. To always load the modules at boot:: echo zfs > /etc/modules-load.d/zfs.conf #. By default ZFS may be removed by kernel package updates. To lock the kernel version to only ones supported by ZFS to prevent this:: echo 'zfs' > /etc/dnf/protected.d/zfs.conf Pending non-kernel updates can still be applied:: dnf update --exclude=kernel* Latest Repositories (Fedora 41+) -------------------------------- *zfs-latest* repository contains the latest released version of OpenZFS which is under active development. It will contain the newest features, and is considered stable, but will have less real-world testing use than *zfs-legacy*. This repository is equivalent to the default *zfs* repository for Fedora. Packages from the latest repository can be installed as follows. For Fedora 41 and newer run:: sudo dnf config-manager setopt zfs*.enabled=0 sudo dnf config-manager setopt zfs-latest.enabled=1 sudo dnf install zfs Legacy Repositories (Fedora 41+) -------------------------------- *zfs-legacy* repository contains the previous released version of OpenZFS which is still being actively updated. Typically, this repository provides same packages as primary *zfs* repository for RHEL- and CentOS-based distribution. Packages from the legacy repository can be installed as follows. For Fedora 41 and newer run:: sudo dnf config-manager setopt zfs*.enabled=0 sudo dnf config-manager setopt zfs-legacy.enabled=1 sudo dnf install zfs Version Specific Repositories (Fedora 41+) ------------------------------------------ Version specific repositories are provided for users who wants to run a specific branch (e.g. `2.3.x`) of ZFS. Packages from the version specific repository can be installed as follows. For Fedora 41 and newer, to enable version specific repository for ZFS branch x.y, run:: sudo dnf config-manager setopt zfs*.enabled=0 sudo dnf config-manager setopt zfs-x.y.enabled=1 sudo dnf install zfs Testing Repository (DEPRECATED) ------------------------------- *zfs-testing* repository is DEPRECATED in favor of 'zfs-latest'. Root on ZFS ----------- .. toctree:: :maxdepth: 1 :glob: * .. highlight:: sh .. ifconfig:: zfs_root_test :: # For the CI/CD test run of this guide, # Enable verbose logging of bash shell and fail immediately when # a command fails. set -vxeuf distro=${1} cp /etc/resolv.conf ./"rootfs-${distro}"/etc/resolv.conf arch-chroot ./"rootfs-${distro}" sh <<-'ZFS_ROOT_GUIDE_TEST' set -vxeuf # install alpine setup scripts apk update apk add alpine-conf curl .. In this document, there are three types of code-block markups: ``::`` are commands intended for both the vm test and the users ``.. ifconfig:: zfs_root_test`` are commands intended only for vm test ``.. code-block:: sh`` are commands intended only for users Fedora Root on ZFS ======================================= Notes ~~~~~ - As an alternative to the below method of installing Fedora Linux on a ZFS root filesystem, you can use the unofficial script `fedora-on-zfs `__, which is more automated and can generate a Fedora Linux installation that is closer to an official Fedora Linux configuration. The fedora-on-zfs script is different from the below method in that it uses one of Fedora's official kickstarts (`fedora-disk-minimal.ks`, `fedora-disk-workstation.ks`, `fedora-disk-kde.ks`, etc.) to guide the installation, but with a few overrides to add the ZFS functionality. Bug reports should be submitted to Greg's fedora-on-zfs GitHub repo. **ZFSBootMenu** `ZFSBootMenu `__ is an alternative bootloader free of such limitations and has support for boot environments. Do not follow instructions on this page if you plan to use ZBM, as the layouts are not compatible. Refer to their site for installation details. **Customization** Unless stated otherwise, it is not recommended to customize system configuration before reboot. **Only use well-tested pool features** You should only use well-tested pool features. Avoid using new features if data integrity is paramount. See, for example, `this comment `__. **UEFI support only** Only UEFI is supported by this guide. Preparation --------------------------- #. Disable Secure Boot. ZFS modules can not be loaded if Secure Boot is enabled. #. Because the kernel of latest Live CD might be incompatible with ZFS, we will use Alpine Linux Extended, which ships with ZFS by default. Download latest extended variant of `Alpine Linux live image `__, verify `checksum `__ and boot from it. .. code-block:: sh gpg --auto-key-retrieve --keyserver hkps://keyserver.ubuntu.com --verify alpine-extended-*.asc dd if=input-file of=output-file bs=1M .. ifconfig:: zfs_root_test # check whether the download page exists # alpine version must be in sync with ci/cd test chroot tarball #. Login as root user. There is no password. #. Configure Internet .. code-block:: sh setup-interfaces -r # You must use "-r" option to start networking services properly # example: network interface: wlan0 WiFi name: ip address: dhcp manual netconfig: n #. If you are using wireless network and it is not shown, see `Alpine Linux wiki `__ for further details. ``wpa_supplicant`` can be installed with ``apk add wpa_supplicant`` without internet connection. #. Configure SSH server .. code-block:: sh setup-sshd # example: ssh server: openssh allow root: "prohibit-password" or "yes" ssh key: "none" or "" #. Set root password or ``/root/.ssh/authorized_keys``. #. Connect from another computer .. code-block:: sh ssh root@192.168.1.91 #. Configure NTP client for time synchronization .. code-block:: sh setup-ntp busybox .. ifconfig:: zfs_root_test # this step is unnecessary for chroot and returns 1 when executed #. Set up apk-repo. A list of available mirrors is shown. Press space bar to continue .. code-block:: sh setup-apkrepos #. Throughout this guide, we use predictable disk names generated by udev .. code-block:: sh apk update apk add eudev setup-devd udev .. ifconfig:: zfs_root_test # for some reason, udev is extremely slow in chroot # it is not needed for chroot anyway. so, skip this step #. Target disk List available disks with .. code-block:: sh find /dev/disk/by-id/ If virtio is used as disk bus, power off the VM and set serial numbers for disk. For QEMU, use ``-drive format=raw,file=disk2.img,serial=AaBb``. For libvirt, edit domain XML. See `this page `__ for examples. Declare disk array .. code-block:: sh DISK='/dev/disk/by-id/ata-FOO /dev/disk/by-id/nvme-BAR' For single disk installation, use .. code-block:: sh DISK='/dev/disk/by-id/disk1' .. ifconfig:: zfs_root_test # for github test run, use chroot and loop devices DISK="$(losetup -a| grep fedora | cut -f1 -d: | xargs -t -I '{}' printf '{} ')" #. Set a mount point :: MNT=$(mktemp -d) #. Set partition size: Set swap size in GB, set to 1 if you don't want swap to take up too much space .. code-block:: sh SWAPSIZE=4 .. ifconfig:: zfs_root_test # For the test run, use 1GB swap space to avoid hitting CI/CD # quota SWAPSIZE=1 Set how much space should be left at the end of the disk, minimum 1GB :: RESERVE=1 #. Install ZFS support from live media:: apk add zfs #. Install partition tool :: apk add parted e2fsprogs cryptsetup util-linux System Installation --------------------------- #. Partition the disks. Note: you must clear all existing partition tables and data structures from target disks. For flash-based storage, this can be done by the blkdiscard command below: :: partition_disk () { local disk="${1}" blkdiscard -f "${disk}" || true parted --script --align=optimal "${disk}" -- \ mklabel gpt \ mkpart EFI 1MiB 4GiB \ mkpart rpool 4GiB -$((SWAPSIZE + RESERVE))GiB \ mkpart swap -$((SWAPSIZE + RESERVE))GiB -"${RESERVE}"GiB \ set 1 esp on \ partprobe "${disk}" } for i in ${DISK}; do partition_disk "${i}" done .. ifconfig:: zfs_root_test :: # When working with GitHub chroot runners, we are using loop # devices as installation target. However, the alias support for # loop device was just introduced in March 2023. See # https://github.com/systemd/systemd/pull/26693 # For now, we will create the aliases manually as a workaround looppart="1 2 3 4 5" for i in ${DISK}; do for j in ${looppart}; do if test -e "${i}p${j}"; then ln -s "${i}p${j}" "${i}-part${j}" fi done done #. Setup temporary encrypted swap for this installation only. This is useful if the available memory is small:: for i in ${DISK}; do cryptsetup open --type plain --key-file /dev/random "${i}"-part3 "${i##*/}"-part3 mkswap /dev/mapper/"${i##*/}"-part3 swapon /dev/mapper/"${i##*/}"-part3 done #. Load ZFS kernel module .. code-block:: sh modprobe zfs #. Create root pool - Unencrypted:: # shellcheck disable=SC2046 zpool create \ -o ashift=12 \ -o autotrim=on \ -R "${MNT}" \ -O acltype=posixacl \ -O canmount=off \ -O dnodesize=auto \ -O normalization=formD \ -O relatime=on \ -O xattr=sa \ -O mountpoint=none \ rpool \ mirror \ $(for i in ${DISK}; do printf '%s ' "${i}-part2"; done) #. Create root system container: :: # dracut demands system root dataset to have non-legacy mountpoint zfs create -o canmount=noauto -o mountpoint=/ rpool/root Create system datasets, manage mountpoints with ``mountpoint=legacy`` :: zfs create -o mountpoint=legacy rpool/home zfs mount rpool/root mount -o X-mount.mkdir -t zfs rpool/home "${MNT}"/home #. Format and mount ESP. Only one of them is used as /boot, you need to set up mirroring afterwards :: for i in ${DISK}; do mkfs.vfat -n EFI "${i}"-part1 done for i in ${DISK}; do mount -t vfat -o fmask=0077,dmask=0077,iocharset=iso8859-1,X-mount.mkdir "${i}"-part1 "${MNT}"/boot break done System Configuration --------------------------- #. Download and extract minimal Fedora root filesystem:: apk add curl jq curl --fail-early --fail -L \ https://dl.fedoraproject.org/pub/fedora/linux/releases/44/Container/x86_64/images/Fedora-Container-Base-Generic-44-1.7.x86_64.oci.tar.xz \ -o rootfs.oci.tar.xz curl --fail-early --fail -L \ https://dl.fedoraproject.org/pub/fedora/linux/releases/44/Container/x86_64/images/Fedora-Container-44-1.7-x86_64-CHECKSUM \ -o checksum # BusyBox sha256sum treats all lines in the checksum file # as checksums and requires two spaces " " # between filename and checksum grep 'Container-Base-Generic-44' checksum \ | grep '^SHA256' \ | sed -E 's|.*= ([a-z0-9]*)$|\1 rootfs.oci.tar.xz|' > ./sha256checksum sha256sum -c ./sha256checksum # the image is an OCI archive: its index names a manifest, which names # the layer that holds the root filesystem mkdir oci tar x -af rootfs.oci.tar.xz -C oci manifest="oci/blobs/sha256/$(jq -r '.manifests[0].digest' oci/index.json | cut -d: -f2)" layer="oci/blobs/sha256/$(jq -r '.layers[0].digest' "${manifest}" | cut -d: -f2)" tar x -C "${MNT}" -af "${layer}" #. Enable community repo .. code-block:: sh sed -i '/edge/d' /etc/apk/repositories sed -i -E 's/#(.*)community/\1community/' /etc/apk/repositories #. Generate fstab:: apk add arch-install-scripts genfstab -t PARTUUID "${MNT}" \ | grep -v swap \ | sed "s|vfat.*rw|vfat rw,x-systemd.idle-timeout=1min,x-systemd.automount,noauto,nofail|" \ > "${MNT}"/etc/fstab #. Chroot .. code-block:: sh cp /etc/resolv.conf "${MNT}"/etc/resolv.conf for i in /dev /proc /sys; do mkdir -p "${MNT}"/"${i}"; mount --rbind "${i}" "${MNT}"/"${i}"; done chroot "${MNT}" /usr/bin/env DISK="${DISK}" bash .. ifconfig:: zfs_root_test cp /etc/resolv.conf "${MNT}"/etc/resolv.conf for i in /dev /proc /sys; do mkdir -p "${MNT}"/"${i}"; mount --rbind "${i}" "${MNT}"/"${i}"; done chroot "${MNT}" /usr/bin/env DISK="${DISK}" bash <<-'ZFS_ROOT_NESTED_CHROOT' set -vxeuf #. Unset all shell aliases, which can interfere with installation:: unalias -a #. Install base packages .. code-block:: sh dnf -y install @core kernel kernel-devel .. ifconfig:: zfs_root_test # no firmware for test dnf -y install --setopt=install_weak_deps=False @core # kernel-core #. Install ZFS packages .. code-block:: sh dnf -y install \ https://zfsonlinux.org/fedora/zfs-release-3-1"$(rpm --eval "%{dist}"||true)".noarch.rpm dnf -y install zfs zfs-dracut .. ifconfig:: zfs_root_test # this step will build zfs modules and fail # no need to test building in chroot dnf -y install \ https://zfsonlinux.org/fedora/zfs-release-3-1"$(rpm --eval "%{dist}"||true)".noarch.rpm #. Check whether ZFS modules are successfully built .. code-block:: sh tail -n10 /var/lib/dkms/zfs/**/build/make.log # ERROR: modpost: GPL-incompatible module zfs.ko uses GPL-only symbol 'bio_start_io_acct' # ERROR: modpost: GPL-incompatible module zfs.ko uses GPL-only symbol 'bio_end_io_acct_remapped' # make[4]: [scripts/Makefile.modpost:138: /var/lib/dkms/zfs/2.1.9/build/module/Module.symvers] Error 1 # make[3]: [Makefile:1977: modpost] Error 2 # make[3]: Leaving directory '/usr/src/kernels/6.2.9-100.fc36.x86_64' # make[2]: [Makefile:55: modules-Linux] Error 2 # make[2]: Leaving directory '/var/lib/dkms/zfs/2.1.9/build/module' # make[1]: [Makefile:933: all-recursive] Error 1 # make[1]: Leaving directory '/var/lib/dkms/zfs/2.1.9/build' # make: [Makefile:794: all] Error 2 If the build failed, you need to install an Long Term Support kernel and its headers, then rebuild ZFS module .. code-block:: sh # this is a third-party repo! # you have been warned. # # select a kernel from # https://copr.fedorainfracloud.org/coprs/kwizart/ dnf copr enable -y kwizart/kernel-longterm-VERSION dnf install -y kernel-longterm kernel-longterm-devel dnf remove -y kernel-core ZFS modules will be built as part of the kernel installation. Check build log again with ``tail`` command. #. Add zfs modules to dracut .. code-block:: sh echo 'add_dracutmodules+=" zfs "' >> /etc/dracut.conf.d/zfs.conf echo 'force_drivers+=" zfs "' >> /etc/dracut.conf.d/zfs.conf .. ifconfig:: zfs_root_test # skip this in chroot, because we did not build zfs module #. Add other drivers to dracut:: if grep mpt3sas /proc/modules; then echo 'force_drivers+=" mpt3sas "' >> /etc/dracut.conf.d/zfs.conf fi if grep virtio_blk /proc/modules; then echo 'filesystems+=" virtio_blk "' >> /etc/dracut.conf.d/fs.conf fi #. Build initrd :: find -D exec /lib/modules -maxdepth 1 \ -mindepth 1 -type d \ -exec sh -vxc \ 'if test -e "$1"/modules.dep; then kernel=$(basename "$1"); dracut --verbose --force --kver "${kernel}"; fi' sh {} \; #. For SELinux, relabel filesystem on reboot:: fixfiles -F onboot #. Enable internet time synchronisation:: systemctl enable systemd-timesyncd #. Generate host id .. code-block:: sh zgenhostid -f -o /etc/hostid .. ifconfig:: zfs_root_test # because zfs is not installed, skip this step #. Install locale package, example for English locale:: dnf install -y glibc-minimal-langpack glibc-langpack-en #. Set locale, keymap, timezone, hostname :: rm -f /etc/localtime rm -f /etc/hostname systemd-firstboot \ --force \ --locale=en_US.UTF-8 \ --timezone=Etc/UTC \ --hostname=testhost \ --keymap=us || true #. Set root passwd :: printf 'root:yourpassword' | chpasswd Bootloader --------------------------- #. Install rEFInd boot loader:: # from http://www.rodsbooks.com/refind/getting.html # use Binary Zip File option curl -L http://sourceforge.net/projects/refind/files/0.14.0.2/refind-bin-0.14.0.2.zip/download --output refind.zip dnf install -y unzip unzip refind.zip mkdir -p /boot/EFI/BOOT find ./refind-bin-0.14.0.2/ -name 'refind_x64.efi' -print0 \ | xargs -0I{} mv {} /boot/EFI/BOOT/BOOTX64.EFI rm -rf refind.zip refind-bin-0.14.0.2 #. Add boot entry:: tee -a /boot/refind-linux.conf <`__ on `Libera Chat `__. If you have a bug report or feature request related to this HOWTO, please `file a new issue `__. Installation ------------ Note: this is for installing ZFS on an existing NixOS installation. To use ZFS as root file system, see below. NixOS live image ships with ZFS support by default. Note that you need to apply these settings even if you don't need to boot from ZFS. The kernel module 'zfs.ko' will not be available to modprobe until you make these changes and reboot. #. Edit ``/etc/nixos/configuration.nix`` and add the following options:: boot.supportedFilesystems = [ "zfs" ]; boot.zfs.forceImportRoot = false; networking.hostId = "yourHostId"; Where hostID can be generated with:: head -c4 /dev/urandom | od -A none -t x4 #. Apply configuration changes:: nixos-rebuild boot #. Reboot:: reboot Root on ZFS ----------- .. toctree:: :maxdepth: 1 :glob: * Contribute ---------- You can contribute to this documentation. Fork this repo, edit the documentation, then opening a pull request. #. To test your changes locally, clone the repo and build the docs:: git clone https://github.com/openzfs/openzfs-docs cd openzfs-docs make html #. Look for errors and warnings in the make output. If there are no errors:: xdg-open _build/html/index.html #. ``git commit --signoff`` to a branch, ``git push``, and create a pull request. .. highlight:: sh .. ifconfig:: zfs_root_test # For the CI/CD test run of this guide, # Enable verbose logging of bash shell and fail immediately when # a command fails. set -vxeuf .. In this document, there are three types of code-block markups: ``::`` are commands intended for both the vm test and the users ``.. ifconfig:: zfs_root_test`` are commands intended only for vm test ``.. code-block:: sh`` are commands intended only for users NixOS Root on ZFS ======================================= **Customization** Unless stated otherwise, it is not recommended to customize system configuration before reboot. **UEFI support only** Only UEFI is supported by this guide. Make sure your computer is booted in UEFI mode. Preparation --------------------------- #. Download `NixOS Live Image `__ and boot from it. .. code-block:: sh sha256sum -c ./nixos-*.sha256 dd if=input-file of=output-file bs=1M #. Connect to the Internet. #. Set root password or ``/root/.ssh/authorized_keys``. #. Start SSH server .. code-block:: sh systemctl restart sshd #. Connect from another computer .. code-block:: sh ssh root@192.168.1.91 #. Target disk List available disks with .. code-block:: sh find /dev/disk/by-id/ If virtio is used as disk bus, power off the VM and set serial numbers for disk. For QEMU, use ``-drive format=raw,file=disk2.img,serial=AaBb``. For libvirt, edit domain XML. See `this page `__ for examples. Declare disk array .. code-block:: sh DISK='/dev/disk/by-id/ata-FOO /dev/disk/by-id/nvme-BAR' For single disk installation, use .. code-block:: sh DISK='/dev/disk/by-id/disk1' .. ifconfig:: zfs_root_test :: # install installation tools nix-env -f '' -iA nixos-install-tools # for github test run, use chroot and loop devices DISK="$(losetup --all| grep nixos | cut -f1 -d: | xargs -t -I '{}' printf '{} ')" # if there is no loopdev, then we are using qemu virtualized test # run, use sata disks instead if test -z "${DISK}"; then DISK=$(find /dev/disk/by-id -type l | grep -v DVD-ROM | grep -v -- -part | xargs -t -I '{}' printf '{} ') fi #. Set a mount point :: MNT=$(mktemp -d) #. Set partition size: Set swap size in GB, set to 1 if you don't want swap to take up too much space .. code-block:: sh SWAPSIZE=4 .. ifconfig:: zfs_root_test # For the test run, use 1GB swap space to avoid hitting CI/CD # quota SWAPSIZE=1 Set how much space should be left at the end of the disk, minimum 1GB :: RESERVE=1 System Installation --------------------------- #. Partition the disks. Note: you must clear all existing partition tables and data structures from target disks. For flash-based storage, this can be done by the blkdiscard command below: :: partition_disk () { local disk="${1}" blkdiscard -f "${disk}" || true parted --script --align=optimal "${disk}" -- \ mklabel gpt \ mkpart EFI 1MiB 4GiB \ mkpart rpool 4GiB -$((SWAPSIZE + RESERVE))GiB \ mkpart swap -$((SWAPSIZE + RESERVE))GiB -"${RESERVE}"GiB \ set 1 esp on \ partprobe "${disk}" } for i in ${DISK}; do partition_disk "${i}" done .. ifconfig:: zfs_root_test :: # When working with GitHub chroot runners, we are using loop # devices as installation target. However, the alias support for # loop device was just introduced in March 2023. See # https://github.com/systemd/systemd/pull/26693 # For now, we will create the aliases manually as a workaround looppart="1 2 3 4 5" for i in ${DISK}; do for j in ${looppart}; do if test -e "${i}p${j}"; then ln -s "${i}p${j}" "${i}-part${j}" fi done done #. Setup temporary encrypted swap for this installation only. This is useful if the available memory is small:: for i in ${DISK}; do cryptsetup open --type plain --key-file /dev/random "${i}"-part3 "${i##*/}"-part3 mkswap /dev/mapper/"${i##*/}"-part3 swapon /dev/mapper/"${i##*/}"-part3 done #. **LUKS only**: Setup encrypted LUKS container for root pool:: for i in ${DISK}; do # see PASSPHRASE PROCESSING section in cryptsetup(8) printf "YOUR_PASSWD" | cryptsetup luksFormat --type luks2 "${i}"-part2 - printf "YOUR_PASSWD" | cryptsetup luksOpen "${i}"-part2 luks-rpool-"${i##*/}"-part2 - done #. Create root pool - Unencrypted .. code-block:: sh # shellcheck disable=SC2046 zpool create \ -o ashift=12 \ -o autotrim=on \ -R "${MNT}" \ -O acltype=posixacl \ -O canmount=off \ -O dnodesize=auto \ -O normalization=formD \ -O relatime=on \ -O xattr=sa \ -O mountpoint=none \ rpool \ mirror \ $(for i in ${DISK}; do printf '%s ' "${i}-part2"; done) - LUKS encrypted :: # shellcheck disable=SC2046 zpool create \ -o ashift=12 \ -o autotrim=on \ -R "${MNT}" \ -O acltype=posixacl \ -O canmount=off \ -O dnodesize=auto \ -O normalization=formD \ -O relatime=on \ -O xattr=sa \ -O mountpoint=none \ rpool \ mirror \ $(for i in ${DISK}; do printf '/dev/mapper/luks-rpool-%s ' "${i##*/}-part2"; done) If not using a multi-disk setup, remove ``mirror``. #. Create root system container: :: zfs create -o canmount=noauto -o mountpoint=legacy rpool/root Create system datasets, manage mountpoints with ``mountpoint=legacy`` :: zfs create -o mountpoint=legacy rpool/home mount -o X-mount.mkdir -t zfs rpool/root "${MNT}" mount -o X-mount.mkdir -t zfs rpool/home "${MNT}"/home #. Format and mount ESP. Only one of them is used as /boot, you need to set up mirroring afterwards :: for i in ${DISK}; do mkfs.vfat -n EFI "${i}"-part1 done for i in ${DISK}; do mount -t vfat -o fmask=0077,dmask=0077,iocharset=iso8859-1,X-mount.mkdir "${i}"-part1 "${MNT}"/boot break done System Configuration --------------------------- #. Generate system configuration:: nixos-generate-config --root "${MNT}" #. Edit system configuration: .. code-block:: sh nano "${MNT}"/etc/nixos/hardware-configuration.nix #. Set networking.hostId: .. code-block:: sh networking.hostId = "abcd1234"; #. If using LUKS, add the output from following command to system configuration .. code-block:: sh tee <`__ includes such packages of filesystems including OpenZFS. openSUSE progresses through 3 main distribution branches, these are called Tumbleweed, Leap and SLE. There are ZFS packages available for all three. External Links -------------- * `openSUSE OpenZFS page `__ Root on ZFS ----------- .. toctree:: :maxdepth: 1 :glob: *Root on ZFS .. highlight:: sh openSUSE Leap Root on ZFS ========================= .. contents:: Table of Contents :local: Overview -------- Caution ~~~~~~~ - This HOWTO uses a whole physical disk. - Do not use these instructions for dual-booting. - Backup your data. Any existing data will be lost. - This is not an openSUSE official HOWTO page. This document will be updated if Root on ZFS support of openSUSE is added in the future. Also, `openSUSE's default system installer Yast2 does not support zfs `__. The method of setting up system with zypper without Yast2 used in this page is based on openSUSE installation methods written by the experience of the people in the community. For more information about this, please look at the external links. System Requirements ~~~~~~~~~~~~~~~~~~~ - `64-bit openSUSE Leap Live CD w/ GUI (e.g. gnome iso) `__ - :ref:`A 64-bit kernel is strongly encouraged. <32-bit-vs-64-bit-systems>` - Installing on a drive which presents 4 KiB logical sectors (a “4Kn” drive) only works with UEFI booting. This not unique to ZFS. `GRUB does not and will not work on 4Kn with legacy (BIOS) booting. `__ Computers that have less than 2 GiB of memory run ZFS slowly. 4 GiB of memory is recommended for normal performance in basic workloads. If you wish to use deduplication, you will need `massive amounts of RAM `__. Enabling deduplication is a permanent change that cannot be easily reverted. Support ~~~~~~~ If you need help, reach out to the community using the :ref:`mailing_lists` or IRC at `#zfsonlinux `__ on `Libera Chat `__. If you have a bug report or feature request related to this HOWTO, please file a new issue and mention `@Zaryob `__. Contributing ~~~~~~~~~~~~ #. Fork and clone: https://github.com/openzfs/openzfs-docs #. Install the tools:: sudo zypper install python3-pip pip3 install -r docs/requirements.txt # Add ~/.local/bin to your $PATH, e.g. by adding this to ~/.bashrc: PATH=$HOME/.local/bin:$PATH #. Make your changes. #. Test:: cd docs make html sensible-browser _build/html/index.html #. ``git commit --signoff`` to a branch, ``git push``, and create a pull request. Encryption ~~~~~~~~~~ This guide supports three different encryption options: unencrypted, ZFS native encryption, and LUKS. With any option, all ZFS features are fully available. Unencrypted does not encrypt anything, of course. With no encryption happening, this option naturally has the best performance. ZFS native encryption encrypts the data and most metadata in the root pool. It does not encrypt dataset or snapshot names or properties. The boot pool is not encrypted at all, but it only contains the bootloader, kernel, and initrd. (Unless you put a password in ``/etc/fstab``, the initrd is unlikely to contain sensitive data.) The system cannot boot without the passphrase being entered at the console. Performance is good. As the encryption happens in ZFS, even if multiple disks (mirror or raidz topologies) are used, the data only has to be encrypted once. LUKS encrypts almost everything. The only unencrypted data is the bootloader, kernel, and initrd. The system cannot boot without the passphrase being entered at the console. Performance is good, but LUKS sits underneath ZFS, so if multiple disks (mirror or raidz topologies) are used, the data has to be encrypted once per disk. Notes ~~~~~~~ - You can use unofficial script `LroZ `__ (Linux Root On Zfs), which is based on this manual and automates most steps. Step 1: Prepare The Install Environment --------------------------------------- #. Boot the openSUSE Live CD. If prompted, login with the username ``linux`` without password. Connect your system to the Internet as appropriate (e.g. join your WiFi network). Open a terminal. #. Check your openSUSE Leap release:: lsb_release -d Description: openSUSE Leap {$release} #. Setup and update the repositories:: sudo zypper addrepo https://download.opensuse.org/repositories/filesystems/$(lsb_release -rs)/filesystems.repo sudo zypper refresh # Refresh all repositories #. Optional: Install and start the OpenSSH server in the Live CD environment: If you have a second system, using SSH to access the target system can be convenient:: sudo zypper install openssh-server sudo systemctl restart sshd.service **Hint:** You can find your IP address with ``ip addr show scope global | grep inet``. Then, from your main machine, connect with ``ssh user@IP``. Do not forget to set the password for user by ``passwd``. #. Disable automounting: If the disk has been used before (with partitions at the same offsets), previous filesystems (e.g. the ESP) will automount if not disabled:: gsettings set org.gnome.desktop.media-handling automount false #. Become root:: sudo -i #. Install ZFS in the Live CD environment:: zypper install zfs zfs-kmp-default zypper install gdisk dkms modprobe zfs Step 2: Disk Formatting ----------------------- #. Set a variable with the disk name:: DISK=/dev/disk/by-id/scsi-SATA_disk1 Always use the long ``/dev/disk/by-id/*`` aliases with ZFS. Using the ``/dev/sd*`` device nodes directly can cause sporadic import failures, especially on systems that have more than one storage pool. **Hints:** - ``ls -la /dev/disk/by-id`` will list the aliases. - Are you doing this in a virtual machine? If your virtual disk is missing from ``/dev/disk/by-id``, use ``/dev/vda`` if you are using KVM with virtio; otherwise, read the `troubleshooting <#troubleshooting>`__ section. #. If you are re-using a disk, clear it as necessary: If the disk was previously used in an MD array:: zypper install mdadm # See if one or more MD arrays are active: cat /proc/mdstat # If so, stop them (replace ``md0`` as required): mdadm --stop /dev/md0 # For an array using the whole disk: mdadm --zero-superblock --force $DISK # For an array using a partition: mdadm --zero-superblock --force ${DISK}-part2 Clear the partition table:: sgdisk --zap-all $DISK If you get a message about the kernel still using the old partition table, you can request the kernel reload the partition information using:: partprobe $DISK If the new partitions still don't show up, you can reboot and start over (except that you can skip this step). #. Partition your disk(s): Run this if you need legacy (BIOS) booting:: sgdisk -a1 -n1:24K:+1000K -t1:EF02 $DISK Run this for UEFI booting (for use now or in the future):: sgdisk -n2:1M:+512M -t2:EF00 $DISK Run this for the boot pool:: sgdisk -n3:0:+1G -t3:BF01 $DISK Choose one of the following options: - Unencrypted or ZFS native encryption:: sgdisk -n4:0:0 -t4:BF00 $DISK - LUKS:: sgdisk -n4:0:0 -t4:8309 $DISK **Hints:** - If you are creating a mirror or raidz topology, repeat the partitioning commands for all the disks which will be part of the pool. #. Create the boot pool:: zpool create \ -o cachefile=/etc/zfs/zpool.cache \ -o ashift=12 -d \ -o feature@async_destroy=enabled \ -o feature@bookmarks=enabled \ -o feature@embedded_data=enabled \ -o feature@empty_bpobj=enabled \ -o feature@enabled_txg=enabled \ -o feature@extensible_dataset=enabled \ -o feature@filesystem_limits=enabled \ -o feature@hole_birth=enabled \ -o feature@large_blocks=enabled \ -o feature@lz4_compress=enabled \ -o feature@spacemap_histogram=enabled \ -o feature@zpool_checkpoint=enabled \ -O acltype=posixacl -O canmount=off -O compression=lz4 \ -O devices=off -O normalization=formD -O relatime=on -O xattr=sa \ -O mountpoint=/boot -R /mnt \ bpool ${DISK}-part3 You should not need to customize any of the options for the boot pool. GRUB does not support all of the zpool features. See ``spa_feature_names`` in `grub-core/fs/zfs/zfs.c `__. This step creates a separate boot pool for ``/boot`` with the features limited to only those that GRUB supports, allowing the root pool to use any/all features. Note that GRUB opens the pool read-only, so all read-only compatible features are “supported” by GRUB. **Hints:** - If you are creating a mirror topology, create the pool using:: zpool create \ ... \ bpool mirror \ /dev/disk/by-id/scsi-SATA_disk1-part3 \ /dev/disk/by-id/scsi-SATA_disk2-part3 - For raidz topologies, replace ``mirror`` in the above command with ``raidz``, ``raidz2``, or ``raidz3`` and list the partitions from the additional disks. - The pool name is arbitrary. If changed, the new name must be used consistently. The ``bpool`` convention originated in this HOWTO. **Feature Notes:** - The ``allocation_classes`` feature should be safe to use. However, unless one is using it (i.e. a ``special`` vdev), there is no point to enabling it. It is extremely unlikely that someone would use this feature for a boot pool. If one cares about speeding up the boot pool, it would make more sense to put the whole pool on the faster disk rather than using it as a ``special`` vdev. - The ``project_quota`` feature has been tested and is safe to use. This feature is extremely unlikely to matter for the boot pool. - The ``resilver_defer`` should be safe but the boot pool is small enough that it is unlikely to be necessary. - The ``spacemap_v2`` feature has been tested and is safe to use. The boot pool is small, so this does not matter in practice. - As a read-only compatible feature, the ``userobj_accounting`` feature should be compatible in theory, but in practice, GRUB can fail with an “invalid dnode type” error. This feature does not matter for ``/boot`` anyway. #. Create the root pool: Choose one of the following options: - Unencrypted:: zpool create \ -o cachefile=/etc/zfs/zpool.cache \ -o ashift=12 \ -O acltype=posixacl -O canmount=off -O compression=lz4 \ -O dnodesize=auto -O normalization=formD -O relatime=on \ -O xattr=sa -O mountpoint=/ -R /mnt \ rpool ${DISK}-part4 - ZFS native encryption:: zpool create \ -o cachefile=/etc/zfs/zpool.cache \ -o ashift=12 \ -O encryption=on \ -O keylocation=prompt -O keyformat=passphrase \ -O acltype=posixacl -O canmount=off -O compression=lz4 \ -O dnodesize=auto -O normalization=formD -O relatime=on \ -O xattr=sa -O mountpoint=/ -R /mnt \ rpool ${DISK}-part4 - LUKS:: zypper install cryptsetup cryptsetup luksFormat -c aes-xts-plain64 -s 512 -h sha256 ${DISK}-part4 cryptsetup luksOpen ${DISK}-part4 luks1 zpool create \ -o cachefile=/etc/zfs/zpool.cache \ -o ashift=12 \ -O acltype=posixacl -O canmount=off -O compression=lz4 \ -O dnodesize=auto -O normalization=formD -O relatime=on \ -O xattr=sa -O mountpoint=/ -R /mnt \ rpool /dev/mapper/luks1 **Notes:** - The use of ``ashift=12`` is recommended here because many drives today have 4 KiB (or larger) physical sectors, even though they present 512 B logical sectors. Also, a future replacement drive may have 4 KiB physical sectors (in which case ``ashift=12`` is desirable) or 4 KiB logical sectors (in which case ``ashift=12`` is required). - Setting ``-O acltype=posixacl`` enables POSIX ACLs globally. If you do not want this, remove that option, but later add ``-o acltype=posixacl`` (note: lowercase “o”) to the ``zfs create`` for ``/var/log``, as `journald requires ACLs `__ - Setting ``normalization=formD`` eliminates some corner cases relating to UTF-8 filename normalization. It also implies ``utf8only=on``, which means that only UTF-8 filenames are allowed. If you care to support non-UTF-8 filenames, do not use this option. For a discussion of why requiring UTF-8 filenames may be a bad idea, see `The problems with enforced UTF-8 only filenames `__. - ``recordsize`` is unset (leaving it at the default of 128 KiB). If you want to tune it (e.g. ``-O recordsize=1M``), see `these `__ `various `__ `blog `__ `posts `__. - Setting ``relatime=on`` is a middle ground between classic POSIX ``atime`` behavior (with its significant performance impact) and ``atime=off`` (which provides the best performance by completely disabling atime updates). Since Linux 2.6.30, ``relatime`` has been the default for other filesystems. See `RedHat’s documentation `__ for further information. - Setting ``xattr=sa`` `vastly improves the performance of extended attributes `__. Inside ZFS, extended attributes are used to implement POSIX ACLs. Extended attributes can also be used by user-space applications. `They are used by some desktop GUI applications. `__ `They can be used by Samba to store Windows ACLs and DOS attributes; they are required for a Samba Active Directory domain controller. `__ Note that ``xattr=sa`` is `Linux-specific `__. If you move your ``xattr=sa`` pool to another OpenZFS implementation besides ZFS-on-Linux, extended attributes will not be readable (though your data will be). If portability of extended attributes is important to you, omit the ``-O xattr=sa`` above. Even if you do not want ``xattr=sa`` for the whole pool, it is probably fine to use it for ``/var/log``. - Make sure to include the ``-part4`` portion of the drive path. If you forget that, you are specifying the whole disk, which ZFS will then re-partition, and you will lose the bootloader partition(s). - ZFS native encryption `now `__ defaults to ``aes-256-gcm``. - For LUKS, the key size chosen is 512 bits. However, XTS mode requires two keys, so the LUKS key is split in half. Thus, ``-s 512`` means AES-256. - Your passphrase will likely be the weakest link. Choose wisely. See `section 5 of the cryptsetup FAQ `__ for guidance. **Hints:** - If you are creating a mirror topology, create the pool using:: zpool create \ ... \ rpool mirror \ /dev/disk/by-id/scsi-SATA_disk1-part4 \ /dev/disk/by-id/scsi-SATA_disk2-part4 - For raidz topologies, replace ``mirror`` in the above command with ``raidz``, ``raidz2``, or ``raidz3`` and list the partitions from the additional disks. - When using LUKS with mirror or raidz topologies, use ``/dev/mapper/luks1``, ``/dev/mapper/luks2``, etc., which you will have to create using ``cryptsetup``. - The pool name is arbitrary. If changed, the new name must be used consistently. On systems that can automatically install to ZFS, the root pool is named ``rpool`` by default. - If you want to use grub bootloader, you must set:: -o feature@async_destroy=enabled \ -o feature@bookmarks=enabled \ -o feature@embedded_data=enabled \ -o feature@empty_bpobj=enabled \ -o feature@enabled_txg=enabled \ -o feature@extensible_dataset=enabled \ -o feature@filesystem_limits=enabled \ -o feature@hole_birth=enabled \ -o feature@large_blocks=enabled \ -o feature@lz4_compress=enabled \ -o feature@spacemap_histogram=enabled \ -o feature@zpool_checkpoint=enabled \ for your root pool. Relevant for grub 2.04 and Leap 15.3. Don't use zpool upgrade for this pool or you will lost the possibility to use grub2-install command. Step 3: System Installation --------------------------- #. Create filesystem datasets to act as containers:: zfs create -o canmount=off -o mountpoint=none rpool/ROOT zfs create -o canmount=off -o mountpoint=none bpool/BOOT On Solaris systems, the root filesystem is cloned and the suffix is incremented for major system changes through ``pkg image-update`` or ``beadm``. Similar functionality has been implemented in Ubuntu 20.04 with the ``zsys`` tool, though its dataset layout is more complicated. Even without such a tool, the `rpool/ROOT` and `bpool/BOOT` containers can still be used for manually created clones. That said, this HOWTO assumes a single filesystem for ``/boot`` for simplicity. #. Create filesystem datasets for the root and boot filesystems:: zfs create -o canmount=noauto -o mountpoint=/ rpool/ROOT/suse zfs mount rpool/ROOT/suse zfs create -o mountpoint=/boot bpool/BOOT/suse With ZFS, it is not normally necessary to use a mount command (either ``mount`` or ``zfs mount``). This situation is an exception because of ``canmount=noauto``. #. Create datasets:: zfs create rpool/home zfs create -o mountpoint=/root rpool/home/root chmod 700 /mnt/root zfs create -o canmount=off rpool/var zfs create -o canmount=off rpool/var/lib zfs create rpool/var/log zfs create rpool/var/spool The datasets below are optional, depending on your preferences and/or software choices. If you wish to exclude these from snapshots:: zfs create -o com.sun:auto-snapshot=false rpool/var/cache zfs create -o com.sun:auto-snapshot=false rpool/var/tmp chmod 1777 /mnt/var/tmp If you use /opt on this system:: zfs create rpool/opt If you use /srv on this system:: zfs create rpool/srv If you use /usr/local on this system:: zfs create -o canmount=off rpool/usr zfs create rpool/usr/local If this system will have games installed:: zfs create rpool/var/games If this system will store local email in /var/mail:: zfs create rpool/var/mail If this system will use Snap packages:: zfs create rpool/var/snap If this system will use Flatpak packages:: zfs create rpool/var/lib/flatpak If you use /var/www on this system:: zfs create rpool/var/www If this system will use GNOME:: zfs create rpool/var/lib/AccountsService If this system will use Docker (which manages its own datasets & snapshots):: zfs create -o com.sun:auto-snapshot=false rpool/var/lib/docker If this system will use NFS (locking):: zfs create -o com.sun:auto-snapshot=false rpool/var/lib/nfs Mount a tmpfs at /run:: mkdir /mnt/run mount -t tmpfs tmpfs /mnt/run mkdir /mnt/run/lock A tmpfs is recommended later, but if you want a separate dataset for ``/tmp``:: zfs create -o com.sun:auto-snapshot=false rpool/tmp chmod 1777 /mnt/tmp The primary goal of this dataset layout is to separate the OS from user data. This allows the root filesystem to be rolled back without rolling back user data. If you do nothing extra, ``/tmp`` will be stored as part of the root filesystem. Alternatively, you can create a separate dataset for ``/tmp``, as shown above. This keeps the ``/tmp`` data out of snapshots of your root filesystem. It also allows you to set a quota on ``rpool/tmp``, if you want to limit the maximum space used. Otherwise, you can use a tmpfs (RAM filesystem) later. #. Copy in zpool.cache:: mkdir /mnt/etc/zfs -p cp /etc/zfs/zpool.cache /mnt/etc/zfs/ Step 4. Install System ---------------------- #. Add repositories into chrooting directory:: zypper --root /mnt ar http://download.opensuse.org/distribution/leap/$(lsb_release -rs)/repo/non-oss non-oss zypper --root /mnt ar http://download.opensuse.org/distribution/leap/$(lsb_release -rs)/repo/oss oss zypper --root /mnt ar http://download.opensuse.org/update/leap/$(lsb_release -rs)/oss update-oss zypper --root /mnt ar http://download.opensuse.org/update/leap/$(lsb_release -rs)/non-oss update-nonoss #. Generate repository indexes:: zypper --root /mnt refresh You will get fingerprint exception, click a to say always trust and continue.:: New repository or package signing key received: Repository: oss Key Name: openSUSE Project Signing Key Key Fingerprint: 22C07BA5 34178CD0 2EFE22AA B88B2FD4 3DBDC284 Key Created: Mon May 5 11:37:40 2014 Key Expires: Thu May 2 11:37:40 2024 Rpm Name: gpg-pubkey-3dbdc284-53674dd4 Do you want to reject the key, trust temporarily, or trust always? [r/t/a/?] (r): #. Install openSUSE Leap with zypper: If you install `base` pattern, zypper will install `busybox-grep` which masks default kernel package. Thats why I recommend you to install `enhanced_base` pattern, if you're new in openSUSE. But in `enhanced_base`, bloats can annoy you, while you want to use it openSUSE on server. So, you need to select a. Install base packages of openSUSE Leap with zypper (Recommended for server):: zypper --root /mnt install -t pattern base b. Install enhanced base of openSUSE Leap with zypper (Recommended for desktop):: zypper --root /mnt install -t pattern enhanced_base #. Install openSUSE zypper package system into chroot:: zypper --root /mnt install zypper #. Recommended: Install openSUSE yast2 system into chroot:: zypper --root /mnt install yast2 zypper --root /mnt install -t pattern yast2_basis It will make easier to configure network and other configurations for beginners. To install a desktop environment, see the `openSUSE wiki `__ Step 5: System Configuration ---------------------------- #. Configure the hostname: Replace ``HOSTNAME`` with the desired hostname:: echo HOSTNAME > /mnt/etc/hostname vi /mnt/etc/hosts Add a line: .. code-block:: text 127.0.1.1 HOSTNAME or if the system has a real name in DNS: .. code-block:: text 127.0.1.1 FQDN HOSTNAME **Hint:** Use ``nano`` if you find ``vi`` confusing. #. Copy network information:: rm /mnt/etc/resolv.conf cp /etc/resolv.conf /mnt/etc/ You will reconfigure network with yast2 later. #. Bind the virtual filesystems from the LiveCD environment to the new system and ``chroot`` into it:: mount --make-private --rbind /dev /mnt/dev mount --make-private --rbind /proc /mnt/proc mount --make-private --rbind /sys /mnt/sys mount -t tmpfs tmpfs /mnt/run mkdir /mnt/run/lock chroot /mnt /usr/bin/env DISK=$DISK bash --login **Note:** This is using ``--rbind``, not ``--bind``. #. Configure a basic system environment:: ln -s /proc/self/mounts /etc/mtab zypper refresh Even if you prefer a non-English system language, always ensure that ``en_US.UTF-8`` is available:: locale -a Output must include that languages: * C * C.utf8 * en_US.utf8 * POSIX Find your locale from `locale -a` commands output then set it with following command. .. code-block:: text localectl set-locale LANG=en_US.UTF-8 #. Optional: Reinstallation for stability: After installation it may need. Some packages may have minor errors. For that, do this if you wish. Since there is no command like dpkg-reconfigure in openSUSE, `zypper install -f stated as a alternative for it `__ but it will reinstall packages. .. code-block:: text zypper install -f permissions-config iputils ca-certificates ca-certificates-mozilla pam shadow dbus libutempter0 suse-module-tools util-linux #. Install kernel:: zypper install kernel-default kernel-firmware **Note:** If you installed `base` pattern, you need to deinstall busybox-grep to install `kernel-default` package. #. Install ZFS in the chroot environment for the new system .. code-block:: text zypper install lsb-release zypper addrepo https://download.opensuse.org/repositories/filesystems/`lsb_release -rs`/filesystems.repo zypper refresh # Refresh all repositories zypper install zfs zfs-kmp-default Note that if your system uses UEFI with Secure Boot, since openSUSE Leap 15.2 the kernel requires all kernel modules to be signed. The ZFS kernel module built in the ``filesystems`` project *is* signed, but not with the official openSUSE key that was automatically registered with your system when you first booted into openSUSE. In order to make sure that your system trusts the ``filesystems`` signing key, make sure to install the ``zfs-ueficert`` package as well:: zypper install zfs-ueficert On the next boot, you will be prompted by the MOK to enroll the new key. #. For LUKS installs only, setup ``/etc/crypttab``:: zypper install cryptsetup echo luks1 /dev/disk/by-uuid/$(blkid -s UUID -o value ${DISK}-part4) none \ luks,discard,initramfs > /etc/crypttab The use of ``initramfs`` is a work-around for `cryptsetup does not support ZFS `__. **Hint:** If you are creating a mirror or raidz topology, repeat the ``/etc/crypttab`` entries for ``luks2``, etc. adjusting for each disk. #. For LUKS installs only, fix cryptsetup naming for ZFS:: echo 'ENV{DM_NAME}!="", SYMLINK+="$env{DM_NAME}" ENV{DM_NAME}!="", SYMLINK+="dm-name-$env{DM_NAME}"' >> /etc/udev/rules.d/99-local-crypt.rules #. Recommended: Generate and setup hostid:: cd /root zypper install wget wget https://github.com/openzfs/zfs/files/4537537/genhostid.sh.gz gzip -d genhostid.sh.gz chmod +x genhostid.sh zgenhostid `/root/genhostid.sh` Check, that generated and system hostid matches:: /root/genhostid.sh hostid #. Install GRUB Choose one of the following options: - Install GRUB for legacy (BIOS) booting:: zypper install grub2-x86_64-pc If your processor is 32bit use `grub2-i386-pc` instead of x86_64 one. - Install GRUB for UEFI booting:: zypper install grub2-x86_64-efi dosfstools os-prober mkdosfs -F 32 -s 1 -n EFI ${DISK}-part2 mkdir /boot/efi echo /dev/disk/by-uuid/$(blkid -s PARTUUID -o value ${DISK}-part2) \ /boot/efi vfat defaults 0 0 >> /etc/fstab mount /boot/efi **Notes:** - The ``-s 1`` for ``mkdosfs`` is only necessary for drives which present 4 KiB logical sectors (“4Kn” drives) to meet the minimum cluster size (given the partition size of 512 MiB) for FAT32. It also works fine on drives which present 512 B sectors. - For a mirror or raidz topology, this step only installs GRUB on the first disk. The other disk(s) will be handled later. #. Optional: Remove os-prober:: zypper remove os-prober This avoids error messages from `update-bootloader`. `os-prober` is only necessary in dual-boot configurations. #. Set a root password:: passwd #. Enable importing bpool This ensures that ``bpool`` is always imported, regardless of whether ``/etc/zfs/zpool.cache`` exists, whether it is in the cachefile or not, or whether ``zfs-import-scan.service`` is enabled. :: vi /etc/systemd/system/zfs-import-bpool.service .. code-block:: ini [Unit] DefaultDependencies=no Before=zfs-import-scan.service Before=zfs-import-cache.service [Service] Type=oneshot RemainAfterExit=yes ExecStart=/usr/sbin/zpool import -N -o cachefile=none bpool # Work-around to preserve zpool cache: ExecStartPre=-/bin/mv /etc/zfs/zpool.cache /etc/zfs/preboot_zpool.cache ExecStartPost=-/bin/mv /etc/zfs/preboot_zpool.cache /etc/zfs/zpool.cache [Install] WantedBy=zfs-import.target :: systemctl enable zfs-import-bpool.service #. Optional (but recommended): Mount a tmpfs to ``/tmp`` If you chose to create a ``/tmp`` dataset above, skip this step, as they are mutually exclusive choices. Otherwise, you can put ``/tmp`` on a tmpfs (RAM filesystem) by enabling the ``tmp.mount`` unit. :: cp /usr/share/systemd/tmp.mount /etc/systemd/system/ systemctl enable tmp.mount Step 6: Kernel Installation --------------------------- #. Add zfs module into dracut:: echo 'zfs'>> /etc/modules-load.d/zfs.conf #. Kernel version of livecd can differ from currently installed version. Get kernel version of your new OS:: kernel_version=$(find /boot/vmlinuz-* | grep -Eo '[[:digit:]]*\.[[:digit:]]*\.[[:digit:]]*\-.*-default') #. Refresh kernel files:: kernel-install add "$kernel_version" /boot/vmlinuz-"$kernel_version" #. Refresh the initrd files:: mkinitrd **Note:** After some installations, LUKS partition cannot seen by dracut, this will print “Failure occurred during following action: configuring encrypted DM device X VOLUME_CRYPTSETUP_FAILED“. For fix this issue you need to check cryptsetup installation. `See for more information `__ **Note:** Although we add the zfs config to the system module into `/etc/modules.d`, if it is not seen by dracut, we have to add it to dracut by force. `dracut --kver $(uname -r) --force --add-drivers "zfs"` Step 7: Grub2 Installation -------------------------- #. Verify that the ZFS boot filesystem is recognized:: grub2-probe /boot Output must be `zfs` #. If you having trouble with `grub2-probe` command make this:: echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile export ZPOOL_VDEV_NAME_PATH=YES then go back to `grub2-probe` step. #. Optional (but highly recommended): Make debugging GRUB easier:: vi /etc/default/grub # Remove quiet from: GRUB_CMDLINE_LINUX_DEFAULT # Uncomment: GRUB_TERMINAL=console # Save and quit. Later, once the system has rebooted twice and you are sure everything is working, you can undo these changes, if desired. #. Update the boot configuration:: update-bootloader **Note:** Ignore errors from ``osprober``, if present. **Note:** If you have had trouble with the grub2 installation, I suggest you use systemd-boot. **Note:** If this command don't gives any output, use classic grub.cfg generation with following command: ``grub2-mkconfig -o /boot/grub2/grub.cfg`` #. Check that ``/boot/grub2/grub.cfg`` have the menuentry ``root=ZFS=rpool/ROOT/suse``, like this:: linux /boot@/vmlinuz-5.3.18-150300.59.60-default root=ZFS=rpool/ROOT/suse If not, change ``/etc/default/grub``:: GRUB_CMDLINE_LINUX="root=ZFS=rpool/ROOT/suse" and repeat previous step. #. Install the boot loader: #. For legacy (BIOS) booting, install GRUB to the MBR:: grub2-install $DISK Note that you are installing GRUB to the whole disk, not a partition. If you are creating a mirror or raidz topology, repeat the ``grub-install`` command for each disk in the pool. #. For UEFI booting, install GRUB to the ESP:: grub2-install --target=x86_64-efi --efi-directory=/boot/efi \ --bootloader-id=opensuse --recheck It is not necessary to specify the disk here. If you are creating a mirror or raidz topology, the additional disks will be handled later. Step 8: Systemd-Boot Installation --------------------------------- **Warning:** This will break your Yast2 Bootloader Configuration. Make sure that you are not able to fix the problem you are having with grub2. I decided to write this part because sometimes grub2 doesn't see the rpool pool in some cases. #. Install systemd-boot:: bootctl install Note: Only if previous cmd replied "Failed to get machine id: No medium found", you need: systemd-machine-id-setup and repeat installation systemd-boot. #. Configure bootloader configuration:: tee -a /boot/efi/loader/loader.conf << EOF default openSUSE_Leap.conf timeout 5 console-mode auto EOF #. Write Entries:: tee -a /boot/efi/loader/entries/openSUSE_Leap.conf << EOF title openSUSE Leap linux /EFI/openSUSE/vmlinuz initrd /EFI/openSUSE/initrd options root=zfs:rpool/ROOT/suse boot=zfs EOF #. Copy files into EFI:: mkdir /boot/efi/EFI/openSUSE cp /boot/{vmlinuz,initrd} /boot/efi/EFI/openSUSE #. Update systemd-boot variables:: bootctl update Step 9: Filesystem Configuration -------------------------------- #. Fix filesystem mount ordering: We need to activate ``zfs-mount-generator``. This makes systemd aware of the separate mountpoints, which is important for things like ``/var/log`` and ``/var/tmp``. In turn, ``rsyslog.service`` depends on ``var-log.mount`` by way of ``local-fs.target`` and services using the ``PrivateTmp`` feature of systemd automatically use ``After=var-tmp.mount``. :: mkdir /etc/zfs/zfs-list.cache touch /etc/zfs/zfs-list.cache/bpool touch /etc/zfs/zfs-list.cache/rpool ln -s /usr/lib/zfs/zed.d/history_event-zfs-list-cacher.sh /etc/zfs/zed.d zed -F & Verify that ``zed`` updated the cache by making sure these are not empty:: cat /etc/zfs/zfs-list.cache/bpool cat /etc/zfs/zfs-list.cache/rpool If either is empty, force a cache update and check again:: zfs set canmount=on bpool/BOOT/suse zfs set canmount=noauto rpool/ROOT/suse If they are still empty, stop zed (as below), start zed (as above) and try again. Stop ``zed``:: fg Press Ctrl-C. Fix the paths to eliminate ``/mnt``:: sed -Ei "s|/mnt/?|/|" /etc/zfs/zfs-list.cache/* Step 10: First Boot ------------------- #. Optional: Install SSH:: zypper install -y openssh-server vi /etc/ssh/sshd_config # Set: PermitRootLogin yes #. Optional: Snapshot the initial installation:: zfs snapshot -r bpool/BOOT/suse@install zfs snapshot -r rpool/ROOT/suse@install In the future, you will likely want to take snapshots before each upgrade, and remove old snapshots (including this one) at some point to save space. #. Exit from the ``chroot`` environment back to the LiveCD environment:: exit #. Run these commands in the LiveCD environment to unmount all filesystems:: mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | \ xargs -i{} umount -lf {} zpool export -a #. Reboot:: reboot Wait for the newly installed system to boot normally. Login as root. #. Create a user account: Replace ``username`` with your desired username:: zfs create rpool/home/username adduser username cp -a /etc/skel/. /home/username chown -R username:username /home/username usermod -a -G audio,cdrom,dip,floppy,netdev,plugdev,sudo,video username #. Mirror GRUB If you installed to multiple disks, install GRUB on the additional disks. - For legacy (BIOS) booting:: Check to be sure we using efi mode: .. code-block:: text efibootmgr -v This must return a message contains `legacy_boot` Then reconfigure grub: .. code-block:: text grub-install $DISK Hit enter until you get to the device selection screen. Select (using the space bar) all of the disks (not partitions) in your pool. - For UEFI booting:: umount /boot/efi For the second and subsequent disks (increment debian-2 to -3, etc.):: dd if=/dev/disk/by-id/scsi-SATA_disk1-part2 \ of=/dev/disk/by-id/scsi-SATA_disk2-part2 efibootmgr -c -g -d /dev/disk/by-id/scsi-SATA_disk2 \ -p 2 -L "opensuse-2" -l '\EFI\opensuse\grubx64.efi' mount /boot/efi Step 11: Optional: Configure Swap --------------------------------- **Caution**: On systems with extremely high memory pressure, using a zvol for swap can result in lockup, regardless of how much swap is still available. There is `a bug report upstream `__. #. Create a volume dataset (zvol) for use as a swap device:: zfs create -V 4G -b $(getconf PAGESIZE) -o compression=zle \ -o logbias=throughput -o sync=always \ -o primarycache=metadata -o secondarycache=none \ -o com.sun:auto-snapshot=false rpool/swap You can adjust the size (the ``4G`` part) to your needs. The compression algorithm is set to ``zle`` because it is the cheapest available algorithm. As this guide recommends ``ashift=12`` (4 kiB blocks on disk), the common case of a 4 kiB page size means that no compression algorithm can reduce I/O. The exception is all-zero pages, which are dropped by ZFS; but some form of compression has to be enabled to get this behavior. #. Configure the swap device: **Caution**: Always use long ``/dev/zvol`` aliases in configuration files. Never use a short ``/dev/zdX`` device name. :: mkswap -f /dev/zvol/rpool/swap echo /dev/zvol/rpool/swap none swap discard 0 0 >> /etc/fstab echo RESUME=none > /etc/initramfs-tools/conf.d/resume The ``RESUME=none`` is necessary to disable resuming from hibernation. This does not work, as the zvol is not present (because the pool has not yet been imported) at the time the resume script runs. If it is not disabled, the boot process hangs for 30 seconds waiting for the swap zvol to appear. #. Enable the swap device:: swapon -av Step 12: Final Cleanup ---------------------- #. Wait for the system to boot normally. Login using the account you created. Ensure the system (including networking) works normally. #. Optional: Delete the snapshots of the initial installation:: sudo zfs destroy bpool/BOOT/suse@install sudo zfs destroy rpool/ROOT/suse@install #. Optional: Disable the root password:: sudo usermod -p '*' root #. Optional (but highly recommended): Disable root SSH logins: If you installed SSH earlier, revert the temporary change:: vi /etc/ssh/sshd_config # Remove: PermitRootLogin yes systemctl restart sshd #. Optional: Re-enable the graphical boot process: If you prefer the graphical boot process, you can re-enable it now. If you are using LUKS, it makes the prompt look nicer. :: sudo vi /etc/default/grub # Add quiet to GRUB_CMDLINE_LINUX_DEFAULT # Comment out GRUB_TERMINAL=console # Save and quit. sudo update-bootloader **Note:** Ignore errors from ``osprober``, if present. #. Optional: For LUKS installs only, backup the LUKS header:: sudo cryptsetup luksHeaderBackup /dev/disk/by-id/scsi-SATA_disk1-part4 \ --header-backup-file luks1-header.dat Store that backup somewhere safe (e.g. cloud storage). It is protected by your LUKS passphrase, but you may wish to use additional encryption. **Hint:** If you created a mirror or raidz topology, repeat this for each LUKS volume (``luks2``, etc.). Troubleshooting --------------- Rescuing using a Live CD ~~~~~~~~~~~~~~~~~~~~~~~~ Go through `Step 1: Prepare The Install Environment <#step-1-prepare-the-install-environment>`__. For LUKS, first unlock the disk(s):: zypper install cryptsetup cryptsetup luksOpen /dev/disk/by-id/scsi-SATA_disk1-part4 luks1 # Repeat for additional disks, if this is a mirror or raidz topology. Mount everything correctly:: zpool export -a zpool import -N -R /mnt rpool zpool import -N -R /mnt bpool zfs load-key -a zfs mount rpool/ROOT/suse zfs mount -a If needed, you can chroot into your installed environment:: mount --make-private --rbind /dev /mnt/dev mount --make-private --rbind /proc /mnt/proc mount --make-private --rbind /sys /mnt/sys chroot /mnt /bin/bash --login mount /boot/efi mount -a Do whatever you need to do to fix your system. When done, cleanup:: exit mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | \ xargs -i{} umount -lf {} zpool export -a reboot Areca ~~~~~ Systems that require the ``arcsas`` blob driver should add it to the ``/etc/initramfs-tools/modules`` file and run ``update-initramfs -c -k all``. Upgrade or downgrade the Areca driver if something like ``RIP: 0010:[] [] native_read_tsc+0x6/0x20`` appears anywhere in kernel log. ZoL is unstable on systems that emit this error message. MPT2SAS ~~~~~~~ Most problem reports for this tutorial involve ``mpt2sas`` hardware that does slow asynchronous drive initialization, like some IBM M1015 or OEM-branded cards that have been flashed to the reference LSI firmware. The basic problem is that disks on these controllers are not visible to the Linux kernel until after the regular system is started, and ZoL does not hotplug pool members. See `https://github.com/openzfs/zfs/issues/330 `__. Most LSI cards are perfectly compatible with ZoL. If your card has this glitch, try setting ``ZFS_INITRD_PRE_MOUNTROOT_SLEEP=X`` in ``/etc/default/zfs``. The system will wait ``X`` seconds for all drives to appear before importing the pool. QEMU/KVM/XEN ~~~~~~~~~~~~ Set a unique serial number on each virtual disk using libvirt or qemu (e.g. ``-drive if=none,id=disk1,file=disk1.qcow2,serial=1234567890``). To be able to use UEFI in guests (instead of only BIOS booting), run this on the host:: sudo zypper install ovmf sudo vi /etc/libvirt/qemu.conf Uncomment these lines: .. code-block:: text nvram = [ "/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd", "/usr/share/OVMF/OVMF_CODE.secboot.fd:/usr/share/OVMF/OVMF_VARS.fd", "/usr/share/AAVMF/AAVMF_CODE.fd:/usr/share/AAVMF/AAVMF_VARS.fd", "/usr/share/AAVMF/AAVMF32_CODE.fd:/usr/share/AAVMF/AAVMF32_VARS.fd" ] :: sudo systemctl restart libvirtd.service VMware ~~~~~~ - Set ``disk.EnableUUID = "TRUE"`` in the vmx file or vsphere configuration. Doing this ensures that ``/dev/disk`` aliases are created in the guest. External Links ~~~~~~~~~~~~~~ * `OpenZFS on openSUSE `__ * `ZenLinux Blog - How to Setup an openSUSE chroot `__ .. highlight:: sh openSUSE Tumbleweed Root on ZFS =============================== .. contents:: Table of Contents :local: Overview -------- Caution ~~~~~~~ - This HOWTO uses a whole physical disk. - Do not use these instructions for dual-booting. - Backup your data. Any existing data will be lost. - This is not an openSUSE official HOWTO page. This document will be updated if Root on ZFS support of openSUSE is added in the future. Also, `openSUSE's default system installer Yast2 does not support zfs `__. The method of setting up system with zypper without Yast2 used in this page is based on openSUSE installation methods written by the experience of the people in the community. For more information about this, please look at the external links. System Requirements ~~~~~~~~~~~~~~~~~~~ - `64-bit openSUSE Tumbleweed Live CD w/ GUI (e.g. gnome iso) `__ - :ref:`A 64-bit kernel is strongly encouraged. <32-bit-vs-64-bit-systems>` - Installing on a drive which presents 4 KiB logical sectors (a “4Kn” drive) only works with UEFI booting. This not unique to ZFS. `GRUB does not and will not work on 4Kn with legacy (BIOS) booting. `__ Computers that have less than 2 GiB of memory run ZFS slowly. 4 GiB of memory is recommended for normal performance in basic workloads. If you wish to use deduplication, you will need `massive amounts of RAM `__. Enabling deduplication is a permanent change that cannot be easily reverted. Support ~~~~~~~ If you need help, reach out to the community using the :ref:`mailing_lists` or IRC at `#zfsonlinux `__ on `Libera Chat `__. If you have a bug report or feature request related to this HOWTO, please file a new issue and mention `@Zaryob `__. Contributing ~~~~~~~~~~~~ #. Fork and clone: https://github.com/openzfs/openzfs-docs #. Install the tools:: sudo zypper install python3-pip pip3 install -r docs/requirements.txt # Add ~/.local/bin to your $PATH, e.g. by adding this to ~/.bashrc: PATH=$HOME/.local/bin:$PATH #. Make your changes. #. Test:: cd docs make html sensible-browser _build/html/index.html #. ``git commit --signoff`` to a branch, ``git push``, and create a pull request. Encryption ~~~~~~~~~~ This guide supports three different encryption options: unencrypted, ZFS native encryption, and LUKS. With any option, all ZFS features are fully available. Unencrypted does not encrypt anything, of course. With no encryption happening, this option naturally has the best performance. ZFS native encryption encrypts the data and most metadata in the root pool. It does not encrypt dataset or snapshot names or properties. The boot pool is not encrypted at all, but it only contains the bootloader, kernel, and initrd. (Unless you put a password in ``/etc/fstab``, the initrd is unlikely to contain sensitive data.) The system cannot boot without the passphrase being entered at the console. Performance is good. As the encryption happens in ZFS, even if multiple disks (mirror or raidz topologies) are used, the data only has to be encrypted once. LUKS encrypts almost everything. The only unencrypted data is the bootloader, kernel, and initrd. The system cannot boot without the passphrase being entered at the console. Performance is good, but LUKS sits underneath ZFS, so if multiple disks (mirror or raidz topologies) are used, the data has to be encrypted once per disk. Step 1: Prepare The Install Environment --------------------------------------- #. Boot the openSUSE Live CD. If prompted, login with the username ``live`` and password ``live``. Connect your system to the Internet as appropriate (e.g. join your WiFi network). Open a terminal. #. Setup and update the repositories:: sudo zypper addrepo https://download.opensuse.org/repositories/filesystems/openSUSE_Tumbleweed/filesystems.repo sudo zypper refresh # Refresh all repositories #. Optional: Install and start the OpenSSH server in the Live CD environment: If you have a second system, using SSH to access the target system can be convenient:: sudo zypper install openssh-server sudo systemctl restart sshd.service **Hint:** You can find your IP address with ``ip addr show scope global | grep inet``. Then, from your main machine, connect with ``ssh user@IP``. #. Disable automounting: If the disk has been used before (with partitions at the same offsets), previous filesystems (e.g. the ESP) will automount if not disabled:: gsettings set org.gnome.desktop.media-handling automount false #. Become root:: sudo -i #. Install ZFS in the Live CD environment:: zypper install zfs zfs-kmp-default zypper install gdisk modprobe zfs Step 2: Disk Formatting ----------------------- #. Set a variable with the disk name:: DISK=/dev/disk/by-id/scsi-SATA_disk1 Always use the long ``/dev/disk/by-id/*`` aliases with ZFS. Using the ``/dev/sd*`` device nodes directly can cause sporadic import failures, especially on systems that have more than one storage pool. **Hints:** - ``ls -la /dev/disk/by-id`` will list the aliases. - Are you doing this in a virtual machine? If your virtual disk is missing from ``/dev/disk/by-id``, use ``/dev/vda`` if you are using KVM with virtio; otherwise, read the `troubleshooting <#troubleshooting>`__ section. #. If you are re-using a disk, clear it as necessary: If the disk was previously used in an MD array:: zypper install mdadm # See if one or more MD arrays are active: cat /proc/mdstat # If so, stop them (replace ``md0`` as required): mdadm --stop /dev/md0 # For an array using the whole disk: mdadm --zero-superblock --force $DISK # For an array using a partition: mdadm --zero-superblock --force ${DISK}-part2 Clear the partition table:: sgdisk --zap-all $DISK If you get a message about the kernel still using the old partition table, you can request the kernel reload the partition information using:: partprobe $DISK If the new partitions still don't show up, you can reboot and start over (except that you can skip this step). #. Partition your disk(s): Run this if you need legacy (BIOS) booting:: sgdisk -a1 -n1:24K:+1000K -t1:EF02 $DISK Run this for UEFI booting (for use now or in the future):: sgdisk -n2:1M:+512M -t2:EF00 $DISK Run this for the boot pool:: sgdisk -n3:0:+1G -t3:BF01 $DISK Choose one of the following options: - Unencrypted or ZFS native encryption:: sgdisk -n4:0:0 -t4:BF00 $DISK - LUKS:: sgdisk -n4:0:0 -t4:8309 $DISK If you are creating a mirror or raidz topology, repeat the partitioning commands for all the disks which will be part of the pool. #. Create the boot pool:: zpool create \ -o cachefile=/etc/zfs/zpool.cache \ -o ashift=12 -d \ -o feature@async_destroy=enabled \ -o feature@bookmarks=enabled \ -o feature@embedded_data=enabled \ -o feature@empty_bpobj=enabled \ -o feature@enabled_txg=enabled \ -o feature@extensible_dataset=enabled \ -o feature@filesystem_limits=enabled \ -o feature@hole_birth=enabled \ -o feature@large_blocks=enabled \ -o feature@lz4_compress=enabled \ -o feature@spacemap_histogram=enabled \ -o feature@zpool_checkpoint=enabled \ -O acltype=posixacl -O canmount=off -O compression=lz4 \ -O devices=off -O normalization=formD -O relatime=on -O xattr=sa \ -O mountpoint=/boot -R /mnt \ bpool ${DISK}-part3 You should not need to customize any of the options for the boot pool. GRUB does not support all of the zpool features. See ``spa_feature_names`` in `grub-core/fs/zfs/zfs.c `__. This step creates a separate boot pool for ``/boot`` with the features limited to only those that GRUB supports, allowing the root pool to use any/all features. Note that GRUB opens the pool read-only, so all read-only compatible features are “supported” by GRUB. **Hints:** - If you are creating a mirror topology, create the pool using:: zpool create \ ... \ bpool mirror \ /dev/disk/by-id/scsi-SATA_disk1-part3 \ /dev/disk/by-id/scsi-SATA_disk2-part3 - For raidz topologies, replace ``mirror`` in the above command with ``raidz``, ``raidz2``, or ``raidz3`` and list the partitions from the additional disks. - The pool name is arbitrary. If changed, the new name must be used consistently. The ``bpool`` convention originated in this HOWTO. **Feature Notes:** - The ``allocation_classes`` feature should be safe to use. However, unless one is using it (i.e. a ``special`` vdev), there is no point to enabling it. It is extremely unlikely that someone would use this feature for a boot pool. If one cares about speeding up the boot pool, it would make more sense to put the whole pool on the faster disk rather than using it as a ``special`` vdev. - The ``project_quota`` feature has been tested and is safe to use. This feature is extremely unlikely to matter for the boot pool. - The ``resilver_defer`` should be safe but the boot pool is small enough that it is unlikely to be necessary. - The ``spacemap_v2`` feature has been tested and is safe to use. The boot pool is small, so this does not matter in practice. - As a read-only compatible feature, the ``userobj_accounting`` feature should be compatible in theory, but in practice, GRUB can fail with an “invalid dnode type” error. This feature does not matter for ``/boot`` anyway. #. Create the root pool: Choose one of the following options: - Unencrypted:: zpool create \ -o cachefile=/etc/zfs/zpool.cache \ -o ashift=12 \ -O acltype=posixacl -O canmount=off -O compression=lz4 \ -O dnodesize=auto -O normalization=formD -O relatime=on \ -O xattr=sa -O mountpoint=/ -R /mnt \ rpool ${DISK}-part4 - ZFS native encryption:: zpool create \ -o cachefile=/etc/zfs/zpool.cache \ -o ashift=12 \ -O encryption=on \ -O keylocation=prompt -O keyformat=passphrase \ -O acltype=posixacl -O canmount=off -O compression=lz4 \ -O dnodesize=auto -O normalization=formD -O relatime=on \ -O xattr=sa -O mountpoint=/ -R /mnt \ rpool ${DISK}-part4 - LUKS:: zypper install cryptsetup cryptsetup luksFormat -c aes-xts-plain64 -s 512 -h sha256 ${DISK}-part4 cryptsetup luksOpen ${DISK}-part4 luks1 zpool create \ -o cachefile=/etc/zfs/zpool.cache \ -o ashift=12 \ -O acltype=posixacl -O canmount=off -O compression=lz4 \ -O dnodesize=auto -O normalization=formD -O relatime=on \ -O xattr=sa -O mountpoint=/ -R /mnt \ rpool /dev/mapper/luks1 **Notes:** - The use of ``ashift=12`` is recommended here because many drives today have 4 KiB (or larger) physical sectors, even though they present 512 B logical sectors. Also, a future replacement drive may have 4 KiB physical sectors (in which case ``ashift=12`` is desirable) or 4 KiB logical sectors (in which case ``ashift=12`` is required). - Setting ``-O acltype=posixacl`` enables POSIX ACLs globally. If you do not want this, remove that option, but later add ``-o acltype=posixacl`` (note: lowercase “o”) to the ``zfs create`` for ``/var/log``, as `journald requires ACLs `__ - Setting ``normalization=formD`` eliminates some corner cases relating to UTF-8 filename normalization. It also implies ``utf8only=on``, which means that only UTF-8 filenames are allowed. If you care to support non-UTF-8 filenames, do not use this option. For a discussion of why requiring UTF-8 filenames may be a bad idea, see `The problems with enforced UTF-8 only filenames `__. - ``recordsize`` is unset (leaving it at the default of 128 KiB). If you want to tune it (e.g. ``-O recordsize=1M``), see `these `__ `various `__ `blog `__ `posts `__. - Setting ``relatime=on`` is a middle ground between classic POSIX ``atime`` behavior (with its significant performance impact) and ``atime=off`` (which provides the best performance by completely disabling atime updates). Since Linux 2.6.30, ``relatime`` has been the default for other filesystems. See `RedHat’s documentation `__ for further information. - Setting ``xattr=sa`` `vastly improves the performance of extended attributes `__. Inside ZFS, extended attributes are used to implement POSIX ACLs. Extended attributes can also be used by user-space applications. `They are used by some desktop GUI applications. `__ `They can be used by Samba to store Windows ACLs and DOS attributes; they are required for a Samba Active Directory domain controller. `__ Note that ``xattr=sa`` is `Linux-specific `__. If you move your ``xattr=sa`` pool to another OpenZFS implementation besides ZFS-on-Linux, extended attributes will not be readable (though your data will be). If portability of extended attributes is important to you, omit the ``-O xattr=sa`` above. Even if you do not want ``xattr=sa`` for the whole pool, it is probably fine to use it for ``/var/log``. - Make sure to include the ``-part4`` portion of the drive path. If you forget that, you are specifying the whole disk, which ZFS will then re-partition, and you will lose the bootloader partition(s). - ZFS native encryption `now `__ defaults to ``aes-256-gcm``. - For LUKS, the key size chosen is 512 bits. However, XTS mode requires two keys, so the LUKS key is split in half. Thus, ``-s 512`` means AES-256. - Your passphrase will likely be the weakest link. Choose wisely. See `section 5 of the cryptsetup FAQ `__ for guidance. **Hints:** - If you are creating a mirror topology, create the pool using:: zpool create \ ... \ rpool mirror \ /dev/disk/by-id/scsi-SATA_disk1-part4 \ /dev/disk/by-id/scsi-SATA_disk2-part4 - For raidz topologies, replace ``mirror`` in the above command with ``raidz``, ``raidz2``, or ``raidz3`` and list the partitions from the additional disks. - When using LUKS with mirror or raidz topologies, use ``/dev/mapper/luks1``, ``/dev/mapper/luks2``, etc., which you will have to create using ``cryptsetup``. - The pool name is arbitrary. If changed, the new name must be used consistently. On systems that can automatically install to ZFS, the root pool is named ``rpool`` by default. Step 3: System Installation --------------------------- #. Create filesystem datasets to act as containers:: zfs create -o canmount=off -o mountpoint=none rpool/ROOT zfs create -o canmount=off -o mountpoint=none bpool/BOOT On Solaris systems, the root filesystem is cloned and the suffix is incremented for major system changes through ``pkg image-update`` or ``beadm``. Similar functionality has been implemented in Ubuntu 20.04 with the ``zsys`` tool, though its dataset layout is more complicated. Even without such a tool, the `rpool/ROOT` and `bpool/BOOT` containers can still be used for manually created clones. That said, this HOWTO assumes a single filesystem for ``/boot`` for simplicity. #. Create filesystem datasets for the root and boot filesystems:: zfs create -o canmount=noauto -o mountpoint=/ rpool/ROOT/suse zfs mount rpool/ROOT/suse zfs create -o mountpoint=/boot bpool/BOOT/suse With ZFS, it is not normally necessary to use a mount command (either ``mount`` or ``zfs mount``). This situation is an exception because of ``canmount=noauto``. #. Create datasets:: zfs create rpool/home zfs create -o mountpoint=/root rpool/home/root chmod 700 /mnt/root zfs create -o canmount=off rpool/var zfs create -o canmount=off rpool/var/lib zfs create rpool/var/log zfs create rpool/var/spool The datasets below are optional, depending on your preferences and/or software choices. If you wish to exclude these from snapshots:: zfs create -o com.sun:auto-snapshot=false rpool/var/cache zfs create -o com.sun:auto-snapshot=false rpool/var/tmp chmod 1777 /mnt/var/tmp If you use /opt on this system:: zfs create rpool/opt If you use /srv on this system:: zfs create rpool/srv If you use /usr/local on this system:: zfs create -o canmount=off rpool/usr zfs create rpool/usr/local If this system will have games installed:: zfs create rpool/var/games If this system will store local email in /var/mail:: zfs create rpool/var/spool/mail If this system will use Snap packages:: zfs create rpool/var/snap If this system will use Flatpak packages:: zfs create rpool/var/lib/flatpak If you use /var/www on this system:: zfs create rpool/var/www If this system will use GNOME:: zfs create rpool/var/lib/AccountsService If this system will use Docker (which manages its own datasets & snapshots):: zfs create -o com.sun:auto-snapshot=false rpool/var/lib/docker If this system will use NFS (locking):: zfs create -o com.sun:auto-snapshot=false rpool/var/lib/nfs Mount a tmpfs at /run:: mkdir /mnt/run mount -t tmpfs tmpfs /mnt/run mkdir /mnt/run/lock A tmpfs is recommended later, but if you want a separate dataset for ``/tmp``:: zfs create -o com.sun:auto-snapshot=false rpool/tmp chmod 1777 /mnt/tmp The primary goal of this dataset layout is to separate the OS from user data. This allows the root filesystem to be rolled back without rolling back user data. If you do nothing extra, ``/tmp`` will be stored as part of the root filesystem. Alternatively, you can create a separate dataset for ``/tmp``, as shown above. This keeps the ``/tmp`` data out of snapshots of your root filesystem. It also allows you to set a quota on ``rpool/tmp``, if you want to limit the maximum space used. Otherwise, you can use a tmpfs (RAM filesystem) later. #. Copy in zpool.cache:: mkdir /mnt/etc/zfs -p cp /etc/zfs/zpool.cache /mnt/etc/zfs/ Step 4. Install System ---------------------- #. Add repositories into chrooting directory:: zypper --root /mnt ar http://download.opensuse.org/tumbleweed/repo/non-oss/ non-oss zypper --root /mnt ar http://download.opensuse.org/tumbleweed/repo/oss/ oss #. Generate repository indexes:: zypper --root /mnt refresh You will get fingerprint exception, click a to say always trust and continue.:: New repository or package signing key received: Repository: oss Key Name: openSUSE Project Signing Key Key Fingerprint: 22C07BA5 34178CD0 2EFE22AA B88B2FD4 3DBDC284 Key Created: Mon May 5 11:37:40 2014 Key Expires: Thu May 2 11:37:40 2024 Rpm Name: gpg-pubkey-3dbdc284-53674dd4 Do you want to reject the key, trust temporarily, or trust always? [r/t/a/?] (r): #. Install openSUSE Tumbleweed with zypper: If you install `base` pattern, zypper will install `busybox-grep` which masks default kernel package. Thats why I recommend you to install `enhanced_base` pattern, if you're new in openSUSE. But in `enhanced_base`, bloats can annoy you, while you want to use it openSUSE on server. So, you need to select a. Install base packages of openSUSE Tumbleweed with zypper (Recommended for server):: zypper --root /mnt install -t pattern base b. Install enhanced base of openSUSE Tumbleweed with zypper (Recommended for desktop):: zypper --root /mnt install -t pattern enhanced_base #. Install openSUSE zypper package system into chroot:: zypper --root /mnt install zypper #. Recommended: Install openSUSE yast2 system into chroot:: zypper --root /mnt install yast2 .. note:: If your `/etc/resolv.conf` file is empty, proceed this command. echo "nameserver 8.8.4.4" | tee -a /mnt/etc/resolv.conf It will make easier to configure network and other configurations for beginners. To install a desktop environment, see the `openSUSE wiki `__ Step 5: System Configuration ---------------------------- #. Configure the hostname: Replace ``HOSTNAME`` with the desired hostname:: echo HOSTNAME > /mnt/etc/hostname vi /mnt/etc/hosts Add a line: .. code-block:: text 127.0.1.1 HOSTNAME or if the system has a real name in DNS: .. code-block:: text 127.0.1.1 FQDN HOSTNAME **Hint:** Use ``nano`` if you find ``vi`` confusing. #. Copy network information:: cp /etc/resolv.conf /mnt/etc You will reconfigure network with yast2. .. note:: If your `/etc/resolv.conf` file is empty, proceed this command. echo "nameserver 8.8.4.4" | tee -a /mnt/etc/resolv.conf #. Bind the virtual filesystems from the LiveCD environment to the new system and ``chroot`` into it:: mount --make-private --rbind /dev /mnt/dev mount --make-private --rbind /proc /mnt/proc mount --make-private --rbind /sys /mnt/sys mount -t tmpfs tmpfs /mnt/run mkdir /mnt/run/lock chroot /mnt /usr/bin/env DISK=$DISK bash --login **Note:** This is using ``--rbind``, not ``--bind``. #. Configure a basic system environment:: ln -s /proc/self/mounts /etc/mtab zypper refresh Even if you prefer a non-English system language, always ensure that ``en_US.UTF-8`` is available:: locale -a Output must include that languages: * C * C.UTF-8 * en_US.utf8 * POSIX Find your locale from `locale -a` commands output then set it with following command. .. code-block:: text localectl set-locale LANG=en_US.UTF-8 #. Optional: Reinstallation for stability: After installation it may need. Some packages may have minor errors. For that, do this if you wish. Since there is no command like dpkg-reconfigure in openSUSE, `zypper install -f stated as a alternative for it `__ but it will reinstall packages. .. code-block:: text zypper install -f permissions-config iputils ca-certificates ca-certificates-mozilla pam shadow dbus-1 libutempter0 suse-module-tools util-linux #. Install kernel:: zypper install kernel-default kernel-firmware .. note:: If you installed `base` pattern, you need to deinstall busybox-grep to install `kernel-default` package. #. Install ZFS in the chroot environment for the new system .. code-block:: text zypper addrepo https://download.opensuse.org/repositories/filesystems/openSUSE_Tumbleweed/filesystems.repo zypper refresh # Refresh all repositories zypper install zfs Note that if your system uses UEFI with Secure Boot, since Linux 5.4 the kernel requires all kernel modules to be signed. The ZFS kernel module built in the ``filesystems`` project *is* signed, but not with the official openSUSE key that was automatically registered with your system when you first booted into openSUSE. In order to make sure that your system trusts the ``filesystems`` signing key, make sure to install the ``zfs-ueficert`` package as well:: zypper install zfs-ueficert On the next boot, you will be prompted by the MOK to enroll the new key. #. For LUKS installs only, setup ``/etc/crypttab``:: zypper install cryptsetup echo luks1 /dev/disk/by-uuid/$(blkid -s UUID -o value ${DISK}-part4) none \ luks,discard,initramfs > /etc/crypttab The use of ``initramfs`` is a work-around for `cryptsetup does not support ZFS `__. **Hint:** If you are creating a mirror or raidz topology, repeat the ``/etc/crypttab`` entries for ``luks2``, etc. adjusting for each disk. #. For LUKS installs only, fix cryptsetup naming for ZFS:: echo 'ENV{DM_NAME}!="", SYMLINK+="$env{DM_NAME}" ENV{DM_NAME}!="", SYMLINK+="dm-name-$env{DM_NAME}"' >> /etc/udev/rules.d/99-local-crypt.rules #. Install GRUB Choose one of the following options: - Install GRUB for legacy (BIOS) booting:: zypper install grub2-i386-pc - Install GRUB for UEFI booting:: zypper install grub2-x86_64-efi dosfstools os-prober mkdosfs -F 32 -s 1 -n EFI ${DISK}-part2 mkdir /boot/efi echo /dev/disk/by-uuid/$(blkid -s PARTUUID -o value ${DISK}-part2) \ /boot/efi vfat defaults 0 0 >> /etc/fstab mount /boot/efi **Notes:** - The ``-s 1`` for ``mkdosfs`` is only necessary for drives which present 4 KiB logical sectors (“4Kn” drives) to meet the minimum cluster size (given the partition size of 512 MiB) for FAT32. It also works fine on drives which present 512 B sectors. - For a mirror or raidz topology, this step only installs GRUB on the first disk. The other disk(s) will be handled later. #. Optional: Remove os-prober:: zypper remove os-prober This avoids error messages from `update-bootloader`. `os-prober` is only necessary in dual-boot configurations. #. Set a root password:: passwd #. Enable importing bpool This ensures that ``bpool`` is always imported, regardless of whether ``/etc/zfs/zpool.cache`` exists, whether it is in the cachefile or not, or whether ``zfs-import-scan.service`` is enabled. :: vi /etc/systemd/system/zfs-import-bpool.service .. code-block:: ini [Unit] DefaultDependencies=no Before=zfs-import-scan.service Before=zfs-import-cache.service [Service] Type=oneshot RemainAfterExit=yes ExecStart=/sbin/zpool import -N -o cachefile=none bpool # Work-around to preserve zpool cache: ExecStartPre=-/bin/mv /etc/zfs/zpool.cache /etc/zfs/preboot_zpool.cache ExecStartPost=-/bin/mv /etc/zfs/preboot_zpool.cache /etc/zfs/zpool.cache [Install] WantedBy=zfs-import.target :: systemctl enable zfs-import-bpool.service #. Optional (but recommended): Mount a tmpfs to ``/tmp`` If you chose to create a ``/tmp`` dataset above, skip this step, as they are mutually exclusive choices. Otherwise, you can put ``/tmp`` on a tmpfs (RAM filesystem) by enabling the ``tmp.mount`` unit. :: cp /usr/share/systemd/tmp.mount /etc/systemd/system/ systemctl enable tmp.mount Step 6: Kernel Installation --------------------------- #. Add zfs module into dracut:: echo 'zfs'>> /etc/modules-load.d/zfs.conf #. Refresh kernel files:: kernel-install add $(uname -r) /boot/vmlinuz-$(uname -r) #. Refresh the initrd files:: mkinitrd **Note:** After some installations, LUKS partition cannot seen by dracut, this will print “Failure occurred during following action: configuring encrypted DM device X VOLUME_CRYPTSETUP_FAILED“. For fix this issue you need to check cryptsetup installation. `See for more information `__ **Note:** Although we add the zfs config to the system module into `/etc/modules.d`, if it is not seen by dracut, we have to add it to dracut by force. `dracut --kver $(uname -r) --force --add-drivers "zfs"` Step 7: Grub2 Installation -------------------------- #. Verify that the ZFS boot filesystem is recognized:: grub2-probe /boot Output must be `zfs` #. If you having trouble with `grub2-probe` command make this:: echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile export ZPOOL_VDEV_NAME_PATH=YES then go back to `grub2-probe` step. #. Workaround GRUB's missing zpool-features support:: vi /etc/default/grub # Set: GRUB_CMDLINE_LINUX="root=ZFS=rpool/ROOT/suse" #. Optional (but highly recommended): Make debugging GRUB easier:: vi /etc/default/grub # Remove quiet from: GRUB_CMDLINE_LINUX_DEFAULT # Uncomment: GRUB_TERMINAL=console # Save and quit. Later, once the system has rebooted twice and you are sure everything is working, you can undo these changes, if desired. #. Update the boot configuration:: update-bootloader **Note:** Ignore errors from ``osprober``, if present. **Note:** If you have had trouble with the grub2 installation, I suggest you use systemd-boot. **Note:** If this command don't gives any output, use classic grub.cfg generation with following command: ``grub2-mkconfig -o /boot/grub2/grub.cfg`` #. Install the boot loader: #. For legacy (BIOS) booting, install GRUB to the MBR:: grub2-install $DISK Note that you are installing GRUB to the whole disk, not a partition. If you are creating a mirror or raidz topology, repeat the ``grub-install`` command for each disk in the pool. #. For UEFI booting, install GRUB to the ESP:: grub2-install --target=x86_64-efi --efi-directory=/boot/efi \ --bootloader-id=opensuse --recheck It is not necessary to specify the disk here. If you are creating a mirror or raidz topology, the additional disks will be handled later. Step 8: Systemd-Boot Installation --------------------------------- **Warning:** This will break your Yast2 Bootloader Configuration. Make sure that you are not able to fix the problem you are having with grub2. I decided to write this part because sometimes grub2 doesn't see the rpool pool in some cases. #. Install systemd-boot:: bootctl install #. Configure bootloader configuration:: tee -a /boot/efi/loader/loader.conf << EOF default openSUSE_Tumbleweed.conf timeout 5 console-mode auto EOF #. Write Entries:: tee -a /boot/efi/loader/entries/openSUSE_Tumbleweed.conf << EOF title openSUSE Tumbleweed linux /EFI/openSUSE/vmlinuz initrd /EFI/openSUSE/initrd options root=zfs=rpool/ROOT/suse boot=zfs EOF #. Copy files into EFI:: mkdir /boot/efi/EFI/openSUSE cp /boot/{vmlinuz,initrd} /boot/efi/EFI/openSUSE #. Update systemd-boot variables:: bootctl update Step 9: Filesystem Configuration -------------------------------- #. Fix filesystem mount ordering: We need to activate ``zfs-mount-generator``. This makes systemd aware of the separate mountpoints, which is important for things like ``/var/log`` and ``/var/tmp``. In turn, ``rsyslog.service`` depends on ``var-log.mount`` by way of ``local-fs.target`` and services using the ``PrivateTmp`` feature of systemd automatically use ``After=var-tmp.mount``. :: mkdir /etc/zfs/zfs-list.cache touch /etc/zfs/zfs-list.cache/bpool touch /etc/zfs/zfs-list.cache/rpool ln -s /usr/lib/zfs/zed.d/history_event-zfs-list-cacher.sh /etc/zfs/zed.d zed -F & Verify that ``zed`` updated the cache by making sure these are not empty:: cat /etc/zfs/zfs-list.cache/bpool cat /etc/zfs/zfs-list.cache/rpool If either is empty, force a cache update and check again:: zfs set canmount=on bpool/BOOT/suse zfs set canmount=noauto rpool/ROOT/suse If they are still empty, stop zed (as below), start zed (as above) and try again. Stop ``zed``:: fg Press Ctrl-C. Fix the paths to eliminate ``/mnt``:: sed -Ei "s|/mnt/?|/|" /etc/zfs/zfs-list.cache/* Step 10: First Boot ------------------- #. Optional: Install SSH:: zypper install --yes openssh-server vi /etc/ssh/sshd_config # Set: PermitRootLogin yes #. Optional: Snapshot the initial installation:: zfs snapshot bpool/BOOT/suse@install zfs snapshot rpool/ROOT/suse@install In the future, you will likely want to take snapshots before each upgrade, and remove old snapshots (including this one) at some point to save space. #. Exit from the ``chroot`` environment back to the LiveCD environment:: exit #. Run these commands in the LiveCD environment to unmount all filesystems:: mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | \ xargs -i{} umount -lf {} zpool export -a #. Reboot:: reboot Wait for the newly installed system to boot normally. Login as root. #. Create a user account: Replace ``username`` with your desired username:: zfs create rpool/home/username adduser username cp -a /etc/skel/. /home/username chown -R username:username /home/username usermod -a -G audio,cdrom,dip,floppy,netdev,plugdev,sudo,video username #. Mirror GRUB If you installed to multiple disks, install GRUB on the additional disks. - For legacy (BIOS) booting:: Check to be sure we using efi mode: .. code-block:: text efibootmgr -v This must return a message contains `legacy_boot` Then reconfigure grub: .. code-block:: text grub-install $DISK Hit enter until you get to the device selection screen. Select (using the space bar) all of the disks (not partitions) in your pool. - For UEFI booting:: umount /boot/efi For the second and subsequent disks (increment debian-2 to -3, etc.):: dd if=/dev/disk/by-id/scsi-SATA_disk1-part2 \ of=/dev/disk/by-id/scsi-SATA_disk2-part2 efibootmgr -c -g -d /dev/disk/by-id/scsi-SATA_disk2 \ -p 2 -L "opensuse-2" -l '\EFI\opensuse\grubx64.efi' mount /boot/efi Step 11: Optional: Configure Swap --------------------------------- **Caution**: On systems with extremely high memory pressure, using a zvol for swap can result in lockup, regardless of how much swap is still available. There is `a bug report upstream `__. #. Create a volume dataset (zvol) for use as a swap device:: zfs create -V 4G -b $(getconf PAGESIZE) -o compression=zle \ -o logbias=throughput -o sync=always \ -o primarycache=metadata -o secondarycache=none \ -o com.sun:auto-snapshot=false rpool/swap You can adjust the size (the ``4G`` part) to your needs. The compression algorithm is set to ``zle`` because it is the cheapest available algorithm. As this guide recommends ``ashift=12`` (4 kiB blocks on disk), the common case of a 4 kiB page size means that no compression algorithm can reduce I/O. The exception is all-zero pages, which are dropped by ZFS; but some form of compression has to be enabled to get this behavior. #. Configure the swap device: **Caution**: Always use long ``/dev/zvol`` aliases in configuration files. Never use a short ``/dev/zdX`` device name. :: mkswap -f /dev/zvol/rpool/swap echo /dev/zvol/rpool/swap none swap discard 0 0 >> /etc/fstab echo RESUME=none > /etc/initramfs-tools/conf.d/resume The ``RESUME=none`` is necessary to disable resuming from hibernation. This does not work, as the zvol is not present (because the pool has not yet been imported) at the time the resume script runs. If it is not disabled, the boot process hangs for 30 seconds waiting for the swap zvol to appear. #. Enable the swap device:: swapon -av Step 12: Final Cleanup ---------------------- #. Wait for the system to boot normally. Login using the account you created. Ensure the system (including networking) works normally. #. Optional: Delete the snapshots of the initial installation:: sudo zfs destroy bpool/BOOT/suse@install sudo zfs destroy rpool/ROOT/suse@install #. Optional: Disable the root password:: sudo usermod -p '*' root #. Optional (but highly recommended): Disable root SSH logins: If you installed SSH earlier, revert the temporary change:: vi /etc/ssh/sshd_config # Remove: PermitRootLogin yes systemctl restart sshd #. Optional: Re-enable the graphical boot process: If you prefer the graphical boot process, you can re-enable it now. If you are using LUKS, it makes the prompt look nicer. :: sudo vi /etc/default/grub # Add quiet to GRUB_CMDLINE_LINUX_DEFAULT # Comment out GRUB_TERMINAL=console # Save and quit. sudo update-bootloader **Note:** Ignore errors from ``osprober``, if present. #. Optional: For LUKS installs only, backup the LUKS header:: sudo cryptsetup luksHeaderBackup /dev/disk/by-id/scsi-SATA_disk1-part4 \ --header-backup-file luks1-header.dat Store that backup somewhere safe (e.g. cloud storage). It is protected by your LUKS passphrase, but you may wish to use additional encryption. **Hint:** If you created a mirror or raidz topology, repeat this for each LUKS volume (``luks2``, etc.). Troubleshooting --------------- Rescuing using a Live CD ~~~~~~~~~~~~~~~~~~~~~~~~ Go through `Step 1: Prepare The Install Environment <#step-1-prepare-the-install-environment>`__. For LUKS, first unlock the disk(s):: zypper install cryptsetup cryptsetup luksOpen /dev/disk/by-id/scsi-SATA_disk1-part4 luks1 # Repeat for additional disks, if this is a mirror or raidz topology. Mount everything correctly:: zpool export -a zpool import -N -R /mnt rpool zpool import -N -R /mnt bpool zfs load-key -a zfs mount rpool/ROOT/suse zfs mount -a If needed, you can chroot into your installed environment:: mount --make-private --rbind /dev /mnt/dev mount --make-private --rbind /proc /mnt/proc mount --make-private --rbind /sys /mnt/sys chroot /mnt /bin/bash --login mount /boot/efi mount -a Do whatever you need to do to fix your system. When done, cleanup:: exit mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | \ xargs -i{} umount -lf {} zpool export -a reboot Areca ~~~~~ Systems that require the ``arcsas`` blob driver should add it to the ``/etc/initramfs-tools/modules`` file and run ``update-initramfs -c -k all``. Upgrade or downgrade the Areca driver if something like ``RIP: 0010:[] [] native_read_tsc+0x6/0x20`` appears anywhere in kernel log. ZoL is unstable on systems that emit this error message. MPT2SAS ~~~~~~~ Most problem reports for this tutorial involve ``mpt2sas`` hardware that does slow asynchronous drive initialization, like some IBM M1015 or OEM-branded cards that have been flashed to the reference LSI firmware. The basic problem is that disks on these controllers are not visible to the Linux kernel until after the regular system is started, and ZoL does not hotplug pool members. See `https://github.com/openzfs/zfs/issues/330 `__. Most LSI cards are perfectly compatible with ZoL. If your card has this glitch, try setting ``ZFS_INITRD_PRE_MOUNTROOT_SLEEP=X`` in ``/etc/default/zfs``. The system will wait ``X`` seconds for all drives to appear before importing the pool. QEMU/KVM/XEN ~~~~~~~~~~~~ Set a unique serial number on each virtual disk using libvirt or qemu (e.g. ``-drive if=none,id=disk1,file=disk1.qcow2,serial=1234567890``). To be able to use UEFI in guests (instead of only BIOS booting), run this on the host:: sudo zypper install ovmf sudo vi /etc/libvirt/qemu.conf Uncomment these lines: .. code-block:: text nvram = [ "/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd", "/usr/share/OVMF/OVMF_CODE.secboot.fd:/usr/share/OVMF/OVMF_VARS.fd", "/usr/share/AAVMF/AAVMF_CODE.fd:/usr/share/AAVMF/AAVMF_VARS.fd", "/usr/share/AAVMF/AAVMF32_CODE.fd:/usr/share/AAVMF/AAVMF32_VARS.fd" ] :: sudo systemctl restart libvirtd.service VMware ~~~~~~ - Set ``disk.EnableUUID = "TRUE"`` in the vmx file or vsphere configuration. Doing this ensures that ``/dev/disk`` aliases are created in the guest. External Links ~~~~~~~~~~~~~~ * `OpenZFS on openSUSE `__ * `ZenLinux Blog - How to Setup an openSUSE chroot `__ RHEL-based distro ======================= Contents -------- .. toctree:: :maxdepth: 1 :glob: * `DKMS`_ and `kABI-tracking kmod`_ style packages are provided for x86_64 RHEL- and CentOS-based distributions from the OpenZFS repository. These packages are updated as new versions are released. Only the repository for the current minor version of each current major release is updated with new packages. To simplify installation, a *zfs-release* package is provided which includes a zfs.repo configuration file and public signing key. All official OpenZFS packages are signed using this key, and by default yum or dnf will verify a package's signature before allowing it be to installed. Users are strongly encouraged to verify the authenticity of the OpenZFS public key using the fingerprint listed here. | **Key location:** /etc/pki/rpm-gpg/RPM-GPG-KEY-openzfs (previously -zfsonlinux) | **Current release packages:** `EL7`_, `EL8`_, `EL9`_, `EL10`_ | **Archived release packages:** `see repo page `__ | **Signing key1 (EL8 and older, Fedora 36 and older)** `pgp.mit.edu `__ / `direct link `__ | **Fingerprint:** C93A FFFD 9F3F 7B03 C310 CEB6 A9D5 A1C0 F14A B620 | **Signing key2 (EL9+, Fedora 37+)** `pgp.mit.edu `__ / `direct link `__ | **Fingerprint:** 7DC7 299D CF7C 7FD9 CD87 701B A599 FD5E 9DB8 4141 For EL 7 run:: yum install https://zfsonlinux.org/epel/zfs-release-2-3$(rpm --eval "%{dist}").noarch.rpm and for EL 8-10:: dnf install https://zfsonlinux.org/epel/zfs-release-3-0$(rpm --eval "%{dist}").noarch.rpm After installing the *zfs-release* package and verifying the public key users can opt to install either the DKMS or kABI-tracking kmod style packages. DKMS packages are recommended for users running a non-distribution kernel or for users who wish to apply local customizations to OpenZFS. For most users the kABI-tracking kmod packages are recommended in order to avoid needing to rebuild OpenZFS for every kernel update. DKMS ---- To install DKMS style packages issue the following commands. First add the `EPEL repository`_ which provides DKMS by installing the *epel-release* package, then the *kernel-devel* and *zfs* packages. Note that it is important to make sure that the matching *kernel-devel* package is installed for the running kernel since DKMS requires it to build OpenZFS. For EL7, separately run:: yum install -y epel-release yum install -y kernel-devel yum install -y zfs And for EL8 and newer, separately run:: dnf install -y epel-release dnf install -y kernel-devel dnf install -y zfs .. note:: When switching from DKMS to kABI-tracking kmods first uninstall the existing DKMS packages. This should remove the kernel modules for all installed kernels, then the kABI-tracking kmods can be installed as described in the section below. kABI-tracking kmod ------------------ By default the *zfs-release* package is configured to install DKMS style packages so they will work with a wide range of kernels. In order to install the kABI-tracking kmods the default repository must be switched from *zfs* to *zfs-kmod*. Keep in mind that the kABI-tracking kmods are only verified to work with the distribution-provided, non-Stream kernel. For EL7 run:: yum-config-manager --disable zfs yum-config-manager --enable zfs-kmod yum install zfs And for EL8 and newer:: dnf config-manager --disable zfs dnf config-manager --enable zfs-kmod dnf install zfs By default the OpenZFS kernel modules are automatically loaded when a ZFS pool is detected. If you would prefer to always load the modules at boot time you can create such configuration in ``/etc/modules-load.d``:: echo zfs >/etc/modules-load.d/zfs.conf .. note:: When updating to a new EL minor release the existing kmod packages may not work due to upstream kABI changes in the kernel. The configuration of the current release package may have already made an updated package available, but the package manager may not know to install that package if the version number isn't newer. When upgrading, users should verify that the *kmod-zfs* package is providing suitable kernel modules, reinstalling the *kmod-zfs* package if necessary. Previous minor EL releases -------------------------- The current release package uses ``${releasever}`` rather than specify a particular minor release as previous release packages did. Typically ``${releasever}`` will resolve to just the major version (e.g. `8`), and the resulting repository URL will be aliased to the current minor version (e.g. `8.7`), but you can specify ``--releasever`` to use previous repositories. :: [vagrant@localhost ~]$ dnf list available --showduplicates kmod-zfs Last metadata expiration check: 0:00:08 ago on tor 31 jan 2023 17:50:05 UTC. Available Packages kmod-zfs.x86_64 2.1.6-1.el8 zfs-kmod kmod-zfs.x86_64 2.1.7-1.el8 zfs-kmod kmod-zfs.x86_64 2.1.8-1.el8 zfs-kmod kmod-zfs.x86_64 2.1.9-1.el8 zfs-kmod [vagrant@localhost ~]$ dnf list available --showduplicates --releasever=8.6 kmod-zfs Last metadata expiration check: 0:16:13 ago on tor 31 jan 2023 17:34:10 UTC. Available Packages kmod-zfs.x86_64 2.1.4-1.el8 zfs-kmod kmod-zfs.x86_64 2.1.5-1.el8 zfs-kmod kmod-zfs.x86_64 2.1.5-2.el8 zfs-kmod kmod-zfs.x86_64 2.1.6-1.el8 zfs-kmod [vagrant@localhost ~]$ In the above example, the former packages were built for EL8.7, and the latter for EL8.6. Latest Repositories (EL8+) -------------------------- *zfs-latest* repository contains the latest released version of OpenZFS which is under active development. It will contain the newest features, and is considered stable, but will have less real-world testing use than *zfs-legacy*. *zfs-latest* is equivalent to the default *zfs* repo on Fedora. Packages from the latest repository can be installed as follows. For EL8 and newer run:: sudo dnf config-manager --disable "zfs*" sudo dnf config-manager --enable zfs-latest sudo dnf install zfs .. note:: Use *zfs-latest* for DKMS packages and *zfs-latest-kmod* for kABI-tracking kmod packages. Legacy Repositories (EL8+) -------------------------- *zfs-legacy* repository contains the previous released version of OpenZFS which is still being actively updated. Typically, this repository provides same packages as primary *zfs* repository for RHEL- and CentOS-based distribution. Packages from the legacy repository can be installed as follows. For EL8 and newer run:: sudo dnf config-manager --disable "zfs*" sudo dnf config-manager --enable zfs-legacy sudo dnf install zfs .. note:: Use *zfs-legacy* for DKMS packages and *zfs-legacy-kmod* for kABI-tracking kmod packages. Version Specific Repositories (EL8+) ------------------------------------ Version specific repositories are provided for users who wants to run a specific branch (e.g. `2.3.x`) of ZFS. Packages from the version specific repository can be installed as follows. For EL8 and newer, to enable version specific repository for ZFS branch x.y, run:: sudo dnf config-manager --disable "zfs*" sudo dnf config-manager --enable zfs-x.y sudo dnf install zfs .. note:: Use *zfs-x.y* for DKMS packages and *zfs-x.y-kmod* for kABI-tracking kmod packages. Testing Repositories (DEPRECATED) --------------------------------- *zfs-testing* and *zfs-testing-kmod* repositories are DEPRECATED in favor of 'zfs-latest'. Root on ZFS ----------- .. toctree:: :maxdepth: 1 :glob: * .. _kABI-tracking kmod: https://elrepoproject.blogspot.com/2016/02/kabi-tracking-kmod-packages.html .. _DKMS: https://en.wikipedia.org/wiki/Dynamic_Kernel_Module_Support .. _EL7: https://zfsonlinux.org/epel/zfs-release-2-3.el7.noarch.rpm .. _EL8: https://zfsonlinux.org/epel/zfs-release-3-0.el8.noarch.rpm .. _EL9: https://zfsonlinux.org/epel/zfs-release-3-0.el9.noarch.rpm .. _EL10: https://zfsonlinux.org/epel/zfs-release-3-0.el10.noarch.rpm .. _EPEL repository: https://fedoraproject.org/wiki/EPEL .. highlight:: sh .. ifconfig:: zfs_root_test # For the CI/CD test run of this guide, # Enable verbose logging of bash shell and fail immediately when # a command fails. set -vxeuf distro=${1} cp /etc/resolv.conf ./"rootfs-${distro}"/etc/resolv.conf arch-chroot ./"rootfs-${distro}" sh <<-'ZFS_ROOT_GUIDE_TEST' set -vxeuf # install alpine setup scripts apk update apk add alpine-conf curl .. In this document, there are three types of code-block markups: ``::`` are commands intended for both the vm test and the users ``.. ifconfig:: zfs_root_test`` are commands intended only for vm test ``.. code-block:: sh`` are commands intended only for users Rocky Linux Root on ZFS ======================================= **ZFSBootMenu** `ZFSBootMenu `__ is an alternative bootloader free of such limitations and has support for boot environments. Do not follow instructions on this page if you plan to use ZBM, as the layouts are not compatible. Refer to their site for installation details. **Customization** Unless stated otherwise, it is not recommended to customize system configuration before reboot. **Only use well-tested pool features** You should only use well-tested pool features. Avoid using new features if data integrity is paramount. See, for example, `this comment `__. **UEFI support only** Only UEFI is supported by this guide. Preparation --------------------------- #. Disable Secure Boot. ZFS modules can not be loaded if Secure Boot is enabled. #. Because the kernel of latest Live CD might be incompatible with ZFS, we will use Alpine Linux Extended, which ships with ZFS by default. Download latest extended variant of `Alpine Linux live image `__, verify `checksum `__ and boot from it. .. code-block:: sh gpg --auto-key-retrieve --keyserver hkps://keyserver.ubuntu.com --verify alpine-extended-*.asc dd if=input-file of=output-file bs=1M .. ifconfig:: zfs_root_test # check whether the download page exists # alpine version must be in sync with ci/cd test chroot tarball #. Login as root user. There is no password. #. Configure Internet .. code-block:: sh setup-interfaces -r # You must use "-r" option to start networking services properly # example: network interface: wlan0 WiFi name: ip address: dhcp manual netconfig: n #. If you are using wireless network and it is not shown, see `Alpine Linux wiki `__ for further details. ``wpa_supplicant`` can be installed with ``apk add wpa_supplicant`` without internet connection. #. Configure SSH server .. code-block:: sh setup-sshd # example: ssh server: openssh allow root: "prohibit-password" or "yes" ssh key: "none" or "" #. Set root password or ``/root/.ssh/authorized_keys``. #. Connect from another computer .. code-block:: sh ssh root@192.168.1.91 #. Configure NTP client for time synchronization .. code-block:: sh setup-ntp busybox .. ifconfig:: zfs_root_test # this step is unnecessary for chroot and returns 1 when executed #. Set up apk-repo. A list of available mirrors is shown. Press space bar to continue .. code-block:: sh setup-apkrepos #. Throughout this guide, we use predictable disk names generated by udev .. code-block:: sh apk update apk add eudev setup-devd udev .. ifconfig:: zfs_root_test # for some reason, udev is extremely slow in chroot # it is not needed for chroot anyway. so, skip this step #. Target disk List available disks with .. code-block:: sh find /dev/disk/by-id/ If virtio is used as disk bus, power off the VM and set serial numbers for disk. For QEMU, use ``-drive format=raw,file=disk2.img,serial=AaBb``. For libvirt, edit domain XML. See `this page `__ for examples. Declare disk array .. code-block:: sh DISK='/dev/disk/by-id/ata-FOO /dev/disk/by-id/nvme-BAR' For single disk installation, use .. code-block:: sh DISK='/dev/disk/by-id/disk1' .. ifconfig:: zfs_root_test # for github test run, use chroot and loop devices DISK="$(losetup -a| grep rhel | cut -f1 -d: | xargs -t -I '{}' printf '{} ')" #. Set a mount point :: MNT=$(mktemp -d) #. Set partition size: Set swap size in GB, set to 1 if you don't want swap to take up too much space .. code-block:: sh SWAPSIZE=4 .. ifconfig:: zfs_root_test # For the test run, use 1GB swap space to avoid hitting CI/CD # quota SWAPSIZE=1 Set how much space should be left at the end of the disk, minimum 1GB :: RESERVE=1 #. Install ZFS support from live media:: apk add zfs #. Install partition tool :: apk add parted e2fsprogs cryptsetup util-linux System Installation --------------------------- #. Partition the disks. Note: you must clear all existing partition tables and data structures from target disks. For flash-based storage, this can be done by the blkdiscard command below: :: partition_disk () { local disk="${1}" blkdiscard -f "${disk}" || true parted --script --align=optimal "${disk}" -- \ mklabel gpt \ mkpart EFI 1MiB 4GiB \ mkpart rpool 4GiB -$((SWAPSIZE + RESERVE))GiB \ mkpart swap -$((SWAPSIZE + RESERVE))GiB -"${RESERVE}"GiB \ set 1 esp on \ partprobe "${disk}" } for i in ${DISK}; do partition_disk "${i}" done .. ifconfig:: zfs_root_test :: # When working with GitHub chroot runners, we are using loop # devices as installation target. However, the alias support for # loop device was just introduced in March 2023. See # https://github.com/systemd/systemd/pull/26693 # For now, we will create the aliases manually as a workaround looppart="1 2 3 4 5" for i in ${DISK}; do for j in ${looppart}; do if test -e "${i}p${j}"; then ln -s "${i}p${j}" "${i}-part${j}" fi done done #. Setup temporary encrypted swap for this installation only. This is useful if the available memory is small:: for i in ${DISK}; do cryptsetup open --type plain --key-file /dev/random "${i}"-part3 "${i##*/}"-part3 mkswap /dev/mapper/"${i##*/}"-part3 swapon /dev/mapper/"${i##*/}"-part3 done #. Load ZFS kernel module .. code-block:: sh modprobe zfs #. Create root pool - Unencrypted:: # shellcheck disable=SC2046 zpool create \ -o ashift=12 \ -o autotrim=on \ -R "${MNT}" \ -O acltype=posixacl \ -O canmount=off \ -O dnodesize=auto \ -O normalization=formD \ -O relatime=on \ -O xattr=sa \ -O mountpoint=none \ rpool \ mirror \ $(for i in ${DISK}; do printf '%s ' "${i}-part2"; done) #. Create root system container: :: # dracut demands system root dataset to have non-legacy mountpoint zfs create -o canmount=noauto -o mountpoint=/ rpool/root Create system datasets, manage mountpoints with ``mountpoint=legacy`` :: zfs create -o mountpoint=legacy rpool/home zfs mount rpool/root mount -o X-mount.mkdir -t zfs rpool/home "${MNT}"/home #. Format and mount ESP. Only one of them is used as /boot, you need to set up mirroring afterwards :: for i in ${DISK}; do mkfs.vfat -n EFI "${i}"-part1 done for i in ${DISK}; do mount -t vfat -o fmask=0077,dmask=0077,iocharset=iso8859-1,X-mount.mkdir "${i}"-part1 "${MNT}"/boot break done System Configuration --------------------------- #. Download and extract minimal Rhel root filesystem:: apk add curl curl --fail-early --fail -L \ https://dl.rockylinux.org/vault/rocky/9.2/images/x86_64/Rocky-9-Container-Base-9.2-20230513.0.x86_64.tar.xz \ -o rootfs.tar.gz curl --fail-early --fail -L \ https://dl.rockylinux.org/vault/rocky/9.2/images/x86_64/Rocky-9-Container-Base-9.2-20230513.0.x86_64.tar.xz.CHECKSUM \ -o checksum # BusyBox sha256sum treats all lines in the checksum file # as checksums and requires two spaces " " # between filename and checksum grep 'Container-Base' checksum \ | grep '^SHA256' \ | sed -E 's|.*= ([a-z0-9]*)$|\1 rootfs.tar.gz|' > ./sha256checksum sha256sum -c ./sha256checksum tar x -C "${MNT}" -af rootfs.tar.gz #. Enable community repo .. code-block:: sh sed -i '/edge/d' /etc/apk/repositories sed -i -E 's/#(.*)community/\1community/' /etc/apk/repositories #. Generate fstab:: apk add arch-install-scripts genfstab -t PARTUUID "${MNT}" \ | grep -v swap \ | sed "s|vfat.*rw|vfat rw,x-systemd.idle-timeout=1min,x-systemd.automount,noauto,nofail|" \ > "${MNT}"/etc/fstab #. Chroot .. code-block:: sh cp /etc/resolv.conf "${MNT}"/etc/resolv.conf for i in /dev /proc /sys; do mkdir -p "${MNT}"/"${i}"; mount --rbind "${i}" "${MNT}"/"${i}"; done chroot "${MNT}" /usr/bin/env DISK="${DISK}" bash .. ifconfig:: zfs_root_test cp /etc/resolv.conf "${MNT}"/etc/resolv.conf for i in /dev /proc /sys; do mkdir -p "${MNT}"/"${i}"; mount --rbind "${i}" "${MNT}"/"${i}"; done chroot "${MNT}" /usr/bin/env DISK="${DISK}" bash <<-'ZFS_ROOT_NESTED_CHROOT' set -vxeuf #. Unset all shell aliases, which can interfere with installation:: unalias -a #. Install base packages .. code-block:: sh dnf -y install --allowerasing @core kernel-core kernel-modules .. ifconfig:: zfs_root_test # skip installing firmware in test dnf -y install --allowerasing --setopt=install_weak_deps=False \ @core kernel-core #. Install ZFS packages:: dnf install -y https://zfsonlinux.org/epel/zfs-release-2-3"$(rpm --eval "%{dist}"|| true)".noarch.rpm dnf config-manager --disable zfs dnf config-manager --enable zfs-kmod dnf install -y zfs zfs-dracut #. Add zfs modules to dracut:: echo 'add_dracutmodules+=" zfs "' >> /etc/dracut.conf.d/zfs.conf echo 'force_drivers+=" zfs "' >> /etc/dracut.conf.d/zfs.conf #. Add other drivers to dracut:: if grep mpt3sas /proc/modules; then echo 'force_drivers+=" mpt3sas "' >> /etc/dracut.conf.d/zfs.conf fi if grep virtio_blk /proc/modules; then echo 'filesystems+=" virtio_blk "' >> /etc/dracut.conf.d/zfs.conf fi #. Generate host id:: zgenhostid -f -o /etc/hostid #. Build initrd:: find -D exec /lib/modules -maxdepth 1 \ -mindepth 1 -type d \ -exec sh -vxc \ 'if test -e "$1"/modules.dep; then kernel=$(basename "$1"); dracut --verbose --force --kver "${kernel}"; fi' sh {} \; #. For SELinux, relabel filesystem on reboot:: fixfiles -F onboot #. Install locale package, example for English locale:: dnf install -y glibc-minimal-langpack glibc-langpack-en #. Set locale, keymap, timezone, hostname :: rm -f /etc/localtime systemd-firstboot \ --force \ --locale=en_US.UTF-8 \ --timezone=Etc/UTC \ --hostname=testhost \ --keymap=us #. Set root passwd :: printf 'root:yourpassword' | chpasswd Bootloader --------------------------- #. Install rEFInd boot loader:: # from http://www.rodsbooks.com/refind/getting.html # use Binary Zip File option curl -L https://sourceforge.net/projects/refind/files/0.14.0.2/refind-bin-0.14.0.2.zip/download --output refind.zip dnf install -y unzip unzip refind.zip mkdir -p /boot/EFI/BOOT find ./refind-bin-0.14.0.2/ -name 'refind_x64.efi' -print0 \ | xargs -0I{} mv {} /boot/EFI/BOOT/BOOTX64.EFI rm -rf refind.zip refind-bin-0.14.0.2 #. Add boot entry:: tee -a /boot/refind-linux.conf <`__ on `Libera Chat `__. If you have a bug report or feature request related to this HOWTO, please `file a new issue and mention @a-biardi `__. Ubuntu ====== .. contents:: Table of Contents :local: Installation ------------ .. note:: If you want to use ZFS as your root filesystem, see the `Root on ZFS`_ links below instead. On Ubuntu, ZFS is included in the default Linux kernel packages. To install the ZFS utilities, first make sure ``universe`` is enabled in ``/etc/apt/sources.list``:: deb http://archive.ubuntu.com/ubuntu main universe Then install ``zfsutils-linux``:: apt update apt install zfsutils-linux Root on ZFS ----------- .. toctree:: :maxdepth: 1 :glob: * .. highlight:: sh Ubuntu 18.04 Root on ZFS ======================== .. contents:: Table of Contents :local: Overview -------- Newer release available ~~~~~~~~~~~~~~~~~~~~~~~ - See :doc:`Ubuntu 20.04 Root on ZFS <./Ubuntu 20.04 Root on ZFS>` for new installs. This guide is no longer receiving most updates. It continues to exist for reference for existing installs that followed it. Caution ~~~~~~~ - This HOWTO uses a whole physical disk. - Do not use these instructions for dual-booting. - Backup your data. Any existing data will be lost. System Requirements ~~~~~~~~~~~~~~~~~~~ - `Ubuntu 18.04.3 ("Bionic") Desktop CD `__ (*not* any server images) - Installing on a drive which presents 4 KiB logical sectors (a “4Kn” drive) only works with UEFI booting. This not unique to ZFS. `GRUB does not and will not work on 4Kn with legacy (BIOS) booting. `__ Computers that have less than 2 GiB of memory run ZFS slowly. 4 GiB of memory is recommended for normal performance in basic workloads. If you wish to use deduplication, you will need `massive amounts of RAM `__. Enabling deduplication is a permanent change that cannot be easily reverted. Support ~~~~~~~ If you need help, reach out to the community using the :ref:`mailing_lists` or IRC at `#zfsonlinux `__ on `Libera Chat `__. If you have a bug report or feature request related to this HOWTO, please `file a new issue and mention @rlaager `__. Contributing ~~~~~~~~~~~~ #. Fork and clone: https://github.com/openzfs/openzfs-docs #. Install the tools:: sudo apt install python3-pip pip3 install -r docs/requirements.txt # Add ~/.local/bin to your $PATH, e.g. by adding this to ~/.bashrc: PATH=$HOME/.local/bin:$PATH #. Make your changes. #. Test:: cd docs make html sensible-browser _build/html/index.html #. ``git commit --signoff`` to a branch, ``git push``, and create a pull request. Mention @rlaager. Encryption ~~~~~~~~~~ This guide supports two different encryption options: unencrypted and LUKS (full-disk encryption). With either option, all ZFS features are fully available. ZFS native encryption is not available in Ubuntu 18.04. Unencrypted does not encrypt anything, of course. With no encryption happening, this option naturally has the best performance. LUKS encrypts almost everything. The only unencrypted data is the bootloader, kernel, and initrd. The system cannot boot without the passphrase being entered at the console. Performance is good, but LUKS sits underneath ZFS, so if multiple disks (mirror or raidz topologies) are used, the data has to be encrypted once per disk. Step 1: Prepare The Install Environment --------------------------------------- 1.1 Boot the Ubuntu Live CD. Select Try Ubuntu. Connect your system to the Internet as appropriate (e.g. join your WiFi network). Open a terminal (press Ctrl-Alt-T). 1.2 Setup and update the repositories:: sudo apt-add-repository universe sudo apt update 1.3 Optional: Install and start the OpenSSH server in the Live CD environment: If you have a second system, using SSH to access the target system can be convenient:: passwd # There is no current password; hit enter at that prompt. sudo apt install --yes openssh-server **Hint:** You can find your IP address with ``ip addr show scope global | grep inet``. Then, from your main machine, connect with ``ssh ubuntu@IP``. 1.4 Become root:: sudo -i 1.5 Install ZFS in the Live CD environment:: apt install --yes debootstrap gdisk zfs-initramfs Step 2: Disk Formatting ----------------------- 2.1 Set a variable with the disk name:: DISK=/dev/disk/by-id/scsi-SATA_disk1 Always use the long ``/dev/disk/by-id/*`` aliases with ZFS. Using the ``/dev/sd*`` device nodes directly can cause sporadic import failures, especially on systems that have more than one storage pool. **Hints:** - ``ls -la /dev/disk/by-id`` will list the aliases. - Are you doing this in a virtual machine? If your virtual disk is missing from ``/dev/disk/by-id``, use ``/dev/vda`` if you are using KVM with virtio; otherwise, read the `troubleshooting <#troubleshooting>`__ section. - For a mirror or raidz topology, use ``DISK1``, ``DISK2``, etc. - When choosing a boot pool size, consider how you will use the space. A kernel and initrd may consume around 100M. If you have multiple kernels and take snapshots, you may find yourself low on boot pool space, especially if you need to regenerate your initramfs images, which may be around 85M each. Size your boot pool appropriately for your needs. 2.2 If you are re-using a disk, clear it as necessary: If the disk was previously used in an MD array, zero the superblock:: apt install --yes mdadm mdadm --zero-superblock --force $DISK Clear the partition table:: sgdisk --zap-all $DISK 2.3 Partition your disk(s): Run this if you need legacy (BIOS) booting:: sgdisk -a1 -n1:24K:+1000K -t1:EF02 $DISK Run this for UEFI booting (for use now or in the future):: sgdisk -n2:1M:+512M -t2:EF00 $DISK Run this for the boot pool:: sgdisk -n3:0:+1G -t3:BF01 $DISK Choose one of the following options: 2.3a Unencrypted:: sgdisk -n4:0:0 -t4:BF01 $DISK 2.3b LUKS:: sgdisk -n4:0:0 -t4:8300 $DISK If you are creating a mirror or raidz topology, repeat the partitioning commands for all the disks which will be part of the pool. 2.4 Create the boot pool:: zpool create -o ashift=12 -d \ -o feature@async_destroy=enabled \ -o feature@bookmarks=enabled \ -o feature@embedded_data=enabled \ -o feature@empty_bpobj=enabled \ -o feature@enabled_txg=enabled \ -o feature@extensible_dataset=enabled \ -o feature@filesystem_limits=enabled \ -o feature@hole_birth=enabled \ -o feature@large_blocks=enabled \ -o feature@lz4_compress=enabled \ -o feature@spacemap_histogram=enabled \ -O acltype=posixacl -O canmount=off -O compression=lz4 -O devices=off \ -O normalization=formD -O relatime=on -O xattr=sa \ -O mountpoint=/ -R /mnt bpool ${DISK}-part3 You should not need to customize any of the options for the boot pool. GRUB does not support all of the zpool features. See ``spa_feature_names`` in `grub-core/fs/zfs/zfs.c `__. This step creates a separate boot pool for ``/boot`` with the features limited to only those that GRUB supports, allowing the root pool to use any/all features. Note that GRUB opens the pool read-only, so all read-only compatible features are “supported” by GRUB. **Hints:** - If you are creating a mirror or raidz topology, create the pool using ``zpool create ... bpool mirror /dev/disk/by-id/scsi-SATA_disk1-part3 /dev/disk/by-id/scsi-SATA_disk2-part3`` (or replace ``mirror`` with ``raidz``, ``raidz2``, or ``raidz3`` and list the partitions from additional disks). - The pool name is arbitrary. If changed, the new name must be used consistently. The ``bpool`` convention originated in this HOWTO. **Feature Notes:** - As a read-only compatible feature, the ``userobj_accounting`` feature should be compatible in theory, but in practice, GRUB can fail with an “invalid dnode type” error. This feature does not matter for ``/boot`` anyway. 2.5 Create the root pool: Choose one of the following options: 2.5a Unencrypted:: zpool create -o ashift=12 \ -O acltype=posixacl -O canmount=off -O compression=lz4 \ -O dnodesize=auto -O normalization=formD -O relatime=on -O xattr=sa \ -O mountpoint=/ -R /mnt rpool ${DISK}-part4 2.5b LUKS:: cryptsetup luksFormat -c aes-xts-plain64 -s 512 -h sha256 ${DISK}-part4 cryptsetup luksOpen ${DISK}-part4 luks1 zpool create -o ashift=12 \ -O acltype=posixacl -O canmount=off -O compression=lz4 \ -O dnodesize=auto -O normalization=formD -O relatime=on -O xattr=sa \ -O mountpoint=/ -R /mnt rpool /dev/mapper/luks1 **Notes:** - The use of ``ashift=12`` is recommended here because many drives today have 4 KiB (or larger) physical sectors, even though they present 512 B logical sectors. Also, a future replacement drive may have 4 KiB physical sectors (in which case ``ashift=12`` is desirable) or 4 KiB logical sectors (in which case ``ashift=12`` is required). - Setting ``-O acltype=posixacl`` enables POSIX ACLs globally. If you do not want this, remove that option, but later add ``-o acltype=posixacl`` (note: lowercase “o”) to the ``zfs create`` for ``/var/log``, as `journald requires ACLs `__ - Setting ``normalization=formD`` eliminates some corner cases relating to UTF-8 filename normalization. It also implies ``utf8only=on``, which means that only UTF-8 filenames are allowed. If you care to support non-UTF-8 filenames, do not use this option. For a discussion of why requiring UTF-8 filenames may be a bad idea, see `The problems with enforced UTF-8 only filenames `__. - ``recordsize`` is unset (leaving it at the default of 128 KiB). If you want to tune it (e.g. ``-O recordsize=1M``), see `these `__ `various `__ `blog `__ `posts `__. - Setting ``relatime=on`` is a middle ground between classic POSIX ``atime`` behavior (with its significant performance impact) and ``atime=off`` (which provides the best performance by completely disabling atime updates). Since Linux 2.6.30, ``relatime`` has been the default for other filesystems. See `RedHat’s documentation `__ for further information. - Setting ``xattr=sa`` `vastly improves the performance of extended attributes `__. Inside ZFS, extended attributes are used to implement POSIX ACLs. Extended attributes can also be used by user-space applications. `They are used by some desktop GUI applications. `__ `They can be used by Samba to store Windows ACLs and DOS attributes; they are required for a Samba Active Directory domain controller. `__ Note that ``xattr=sa`` is `Linux-specific `__. If you move your ``xattr=sa`` pool to another OpenZFS implementation besides ZFS-on-Linux, extended attributes will not be readable (though your data will be). If portability of extended attributes is important to you, omit the ``-O xattr=sa`` above. Even if you do not want ``xattr=sa`` for the whole pool, it is probably fine to use it for ``/var/log``. - Make sure to include the ``-part4`` portion of the drive path. If you forget that, you are specifying the whole disk, which ZFS will then re-partition, and you will lose the bootloader partition(s). - For LUKS, the key size chosen is 512 bits. However, XTS mode requires two keys, so the LUKS key is split in half. Thus, ``-s 512`` means AES-256. - Your passphrase will likely be the weakest link. Choose wisely. See `section 5 of the cryptsetup FAQ `__ for guidance. **Hints:** - If you are creating a mirror or raidz topology, create the pool using ``zpool create ... rpool mirror /dev/disk/by-id/scsi-SATA_disk1-part4 /dev/disk/by-id/scsi-SATA_disk2-part4`` (or replace ``mirror`` with ``raidz``, ``raidz2``, or ``raidz3`` and list the partitions from additional disks). For LUKS, use ``/dev/mapper/luks1``, ``/dev/mapper/luks2``, etc., which you will have to create using ``cryptsetup``. - The pool name is arbitrary. If changed, the new name must be used consistently. On systems that can automatically install to ZFS, the root pool is named ``rpool`` by default. Step 3: System Installation --------------------------- 3.1 Create filesystem datasets to act as containers:: zfs create -o canmount=off -o mountpoint=none rpool/ROOT zfs create -o canmount=off -o mountpoint=none bpool/BOOT On Solaris systems, the root filesystem is cloned and the suffix is incremented for major system changes through ``pkg image-update`` or ``beadm``. Similar functionality has been implemented in Ubuntu 20.04 with the ``zsys`` tool, though its dataset layout is more complicated. Even without such a tool, the `rpool/ROOT` and `bpool/BOOT` containers can still be used for manually created clones. 3.2 Create filesystem datasets for the root and boot filesystems:: zfs create -o canmount=noauto -o mountpoint=/ rpool/ROOT/ubuntu zfs mount rpool/ROOT/ubuntu zfs create -o canmount=noauto -o mountpoint=/boot bpool/BOOT/ubuntu zfs mount bpool/BOOT/ubuntu With ZFS, it is not normally necessary to use a mount command (either ``mount`` or ``zfs mount``). This situation is an exception because of ``canmount=noauto``. 3.3 Create datasets:: zfs create rpool/home zfs create -o mountpoint=/root rpool/home/root zfs create -o canmount=off rpool/var zfs create -o canmount=off rpool/var/lib zfs create rpool/var/log zfs create rpool/var/spool The datasets below are optional, depending on your preferences and/or software choices. If you wish to exclude these from snapshots:: zfs create -o com.sun:auto-snapshot=false rpool/var/cache zfs create -o com.sun:auto-snapshot=false rpool/var/tmp chmod 1777 /mnt/var/tmp If you use /opt on this system:: zfs create rpool/opt If you use /srv on this system:: zfs create rpool/srv If you use /usr/local on this system:: zfs create -o canmount=off rpool/usr zfs create rpool/usr/local If this system will have games installed:: zfs create rpool/var/games If this system will store local email in /var/mail:: zfs create rpool/var/mail If this system will use Snap packages:: zfs create rpool/var/snap If you use /var/www on this system:: zfs create rpool/var/www If this system will use GNOME:: zfs create rpool/var/lib/AccountsService If this system will use Docker (which manages its own datasets & snapshots):: zfs create -o com.sun:auto-snapshot=false rpool/var/lib/docker If this system will use NFS (locking):: zfs create -o com.sun:auto-snapshot=false rpool/var/lib/nfs A tmpfs is recommended later, but if you want a separate dataset for ``/tmp``:: zfs create -o com.sun:auto-snapshot=false rpool/tmp chmod 1777 /mnt/tmp The primary goal of this dataset layout is to separate the OS from user data. This allows the root filesystem to be rolled back without rolling back user data. The ``com.sun.auto-snapshot`` setting is used by some ZFS snapshot utilities to exclude transient data. If you do nothing extra, ``/tmp`` will be stored as part of the root filesystem. Alternatively, you can create a separate dataset for ``/tmp``, as shown above. This keeps the ``/tmp`` data out of snapshots of your root filesystem. It also allows you to set a quota on ``rpool/tmp``, if you want to limit the maximum space used. Otherwise, you can use a tmpfs (RAM filesystem) later. 3.4 Install the minimal system:: debootstrap bionic /mnt zfs set devices=off rpool The ``debootstrap`` command leaves the new system in an unconfigured state. An alternative to using ``debootstrap`` is to copy the entirety of a working system into the new ZFS root. Step 4: System Configuration ---------------------------- 4.1 Configure the hostname: Replace ``HOSTNAME`` with the desired hostname:: echo HOSTNAME > /mnt/etc/hostname vi /mnt/etc/hosts .. code-block:: text Add a line: 127.0.1.1 HOSTNAME or if the system has a real name in DNS: 127.0.1.1 FQDN HOSTNAME **Hint:** Use ``nano`` if you find ``vi`` confusing. 4.2 Configure the network interface: Find the interface name:: ip addr show Adjust NAME below to match your interface name:: vi /mnt/etc/netplan/01-netcfg.yaml .. code-block:: yaml network: version: 2 ethernets: NAME: dhcp4: true Customize this file if the system is not a DHCP client. 4.3 Configure the package sources:: vi /mnt/etc/apt/sources.list .. code-block:: sourceslist deb http://archive.ubuntu.com/ubuntu bionic main restricted universe multiverse deb http://archive.ubuntu.com/ubuntu bionic-updates main restricted universe multiverse deb http://archive.ubuntu.com/ubuntu bionic-backports main restricted universe multiverse deb http://security.ubuntu.com/ubuntu bionic-security main restricted universe multiverse 4.4 Bind the virtual filesystems from the LiveCD environment to the new system and ``chroot`` into it:: mount --rbind /dev /mnt/dev mount --rbind /proc /mnt/proc mount --rbind /sys /mnt/sys chroot /mnt /usr/bin/env DISK=$DISK bash --login **Note:** This is using ``--rbind``, not ``--bind``. 4.5 Configure a basic system environment:: ln -s /proc/self/mounts /etc/mtab apt update Even if you prefer a non-English system language, always ensure that ``en_US.UTF-8`` is available:: dpkg-reconfigure locales dpkg-reconfigure tzdata If you prefer ``nano`` over ``vi``, install it:: apt install --yes nano 4.6 Install ZFS in the chroot environment for the new system:: apt install --yes --no-install-recommends linux-image-generic apt install --yes zfs-initramfs **Hint:** For the HWE kernel, install ``linux-image-generic-hwe-18.04`` instead of ``linux-image-generic``. 4.7 For LUKS installs only, setup ``/etc/crypttab``:: apt install --yes cryptsetup echo luks1 UUID=$(blkid -s UUID -o value ${DISK}-part4) none \ luks,discard,initramfs > /etc/crypttab The use of ``initramfs`` is a work-around for `cryptsetup does not support ZFS `__. **Hint:** If you are creating a mirror or raidz topology, repeat the ``/etc/crypttab`` entries for ``luks2``, etc. adjusting for each disk. 4.8 Install GRUB Choose one of the following options: 4.8a Install GRUB for legacy (BIOS) booting:: apt install --yes grub-pc Select (using the space bar) all of the disks (not partitions) in your pool. 4.8b Install GRUB for UEFI booting:: apt install dosfstools mkdosfs -F 32 -s 1 -n EFI ${DISK}-part2 mkdir /boot/efi echo PARTUUID=$(blkid -s PARTUUID -o value ${DISK}-part2) \ /boot/efi vfat nofail,x-systemd.device-timeout=1 0 1 >> /etc/fstab mount /boot/efi apt install --yes grub-efi-amd64-signed shim-signed **Notes:** - The ``-s 1`` for ``mkdosfs`` is only necessary for drives which present 4 KiB logical sectors (“4Kn” drives) to meet the minimum cluster size (given the partition size of 512 MiB) for FAT32. It also works fine on drives which present 512 B sectors. - For a mirror or raidz topology, this step only installs GRUB on the first disk. The other disk(s) will be handled later. 4.9 (Optional): Remove os-prober:: apt purge --yes os-prober This avoids error messages from `update-grub`. `os-prober` is only necessary in dual-boot configurations. 4.10 Set a root password:: passwd 4.11 Enable importing bpool This ensures that ``bpool`` is always imported, regardless of whether ``/etc/zfs/zpool.cache`` exists, whether it is in the cachefile or not, or whether ``zfs-import-scan.service`` is enabled. :: vi /etc/systemd/system/zfs-import-bpool.service .. code-block:: ini [Unit] DefaultDependencies=no Before=zfs-import-scan.service Before=zfs-import-cache.service [Service] Type=oneshot RemainAfterExit=yes ExecStart=/sbin/zpool import -N -o cachefile=none bpool [Install] WantedBy=zfs-import.target :: systemctl enable zfs-import-bpool.service 4.12 Optional (but recommended): Mount a tmpfs to ``/tmp`` If you chose to create a ``/tmp`` dataset above, skip this step, as they are mutually exclusive choices. Otherwise, you can put ``/tmp`` on a tmpfs (RAM filesystem) by enabling the ``tmp.mount`` unit. :: cp /usr/share/systemd/tmp.mount /etc/systemd/system/ systemctl enable tmp.mount 4.13 Setup system groups:: addgroup --system lpadmin addgroup --system sambashare Step 5: GRUB Installation ------------------------- 5.1 Verify that the ZFS boot filesystem is recognized:: grub-probe /boot 5.2 Refresh the initrd files:: update-initramfs -c -k all **Note:** When using LUKS, this will print “WARNING could not determine root device from /etc/fstab”. This is because `cryptsetup does not support ZFS `__. 5.3 Workaround GRUB's missing zpool-features support:: vi /etc/default/grub # Set: GRUB_CMDLINE_LINUX="root=ZFS=rpool/ROOT/ubuntu" 5.4 Optional (but highly recommended): Make debugging GRUB easier:: vi /etc/default/grub # Comment out: GRUB_TIMEOUT_STYLE=hidden # Set: GRUB_TIMEOUT=5 # Below GRUB_TIMEOUT, add: GRUB_RECORDFAIL_TIMEOUT=5 # Remove quiet and splash from: GRUB_CMDLINE_LINUX_DEFAULT # Uncomment: GRUB_TERMINAL=console # Save and quit. Later, once the system has rebooted twice and you are sure everything is working, you can undo these changes, if desired. 5.5 Update the boot configuration:: update-grub **Note:** Ignore errors from ``osprober``, if present. 5.6 Install the boot loader: 5.6a For legacy (BIOS) booting, install GRUB to the MBR:: grub-install $DISK Note that you are installing GRUB to the whole disk, not a partition. If you are creating a mirror or raidz topology, repeat the ``grub-install`` command for each disk in the pool. 5.6b For UEFI booting, install GRUB:: grub-install --target=x86_64-efi --efi-directory=/boot/efi \ --bootloader-id=ubuntu --recheck It is not necessary to specify the disk here. If you are creating a mirror or raidz topology, the additional disks will be handled later. 5.7 Fix filesystem mount ordering: `Until ZFS gains a systemd mount generator `__, there are races between mounting filesystems and starting certain daemons. In practice, the issues (e.g. `#5754 `__) seem to be with certain filesystems in ``/var``, specifically ``/var/log`` and ``/var/tmp``. Setting these to use ``legacy`` mounting, and listing them in ``/etc/fstab`` makes systemd aware that these are separate mountpoints. In turn, ``rsyslog.service`` depends on ``var-log.mount`` by way of ``local-fs.target`` and services using the ``PrivateTmp`` feature of systemd automatically use ``After=var-tmp.mount``. Until there is support for mounting ``/boot`` in the initramfs, we also need to mount that, because it was marked ``canmount=noauto``. Also, with UEFI, we need to ensure it is mounted before its child filesystem ``/boot/efi``. ``rpool`` is guaranteed to be imported by the initramfs, so there is no point in adding ``x-systemd.requires=zfs-import.target`` to those filesystems. For UEFI booting, unmount /boot/efi first:: umount /boot/efi Everything else applies to both BIOS and UEFI booting:: zfs set mountpoint=legacy bpool/BOOT/ubuntu echo bpool/BOOT/ubuntu /boot zfs \ nodev,relatime,x-systemd.requires=zfs-import-bpool.service 0 0 >> /etc/fstab zfs set mountpoint=legacy rpool/var/log echo rpool/var/log /var/log zfs nodev,relatime 0 0 >> /etc/fstab zfs set mountpoint=legacy rpool/var/spool echo rpool/var/spool /var/spool zfs nodev,relatime 0 0 >> /etc/fstab If you created a /var/tmp dataset:: zfs set mountpoint=legacy rpool/var/tmp echo rpool/var/tmp /var/tmp zfs nodev,relatime 0 0 >> /etc/fstab If you created a /tmp dataset:: zfs set mountpoint=legacy rpool/tmp echo rpool/tmp /tmp zfs nodev,relatime 0 0 >> /etc/fstab Step 6: First Boot ------------------ 6.1 Snapshot the initial installation:: zfs snapshot bpool/BOOT/ubuntu@install zfs snapshot rpool/ROOT/ubuntu@install In the future, you will likely want to take snapshots before each upgrade, and remove old snapshots (including this one) at some point to save space. 6.2 Exit from the ``chroot`` environment back to the LiveCD environment:: exit 6.3 Run these commands in the LiveCD environment to unmount all filesystems:: mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {} zpool export -a 6.4 Reboot:: reboot Wait for the newly installed system to boot normally. Login as root. 6.5 Create a user account: Replace ``username`` with your desired username:: zfs create rpool/home/username adduser username cp -a /etc/skel/. /home/username chown -R username:username /home/username usermod -a -G audio,cdrom,dip,floppy,netdev,plugdev,sudo,video username 6.6 Mirror GRUB If you installed to multiple disks, install GRUB on the additional disks: 6.6a For legacy (BIOS) booting:: dpkg-reconfigure grub-pc Hit enter until you get to the device selection screen. Select (using the space bar) all of the disks (not partitions) in your pool. 6.6b For UEFI booting:: umount /boot/efi For the second and subsequent disks (increment ubuntu-2 to -3, etc.):: dd if=/dev/disk/by-id/scsi-SATA_disk1-part2 \ of=/dev/disk/by-id/scsi-SATA_disk2-part2 efibootmgr -c -g -d /dev/disk/by-id/scsi-SATA_disk2 \ -p 2 -L "ubuntu-2" -l '\EFI\ubuntu\shimx64.efi' mount /boot/efi Step 7: (Optional) Configure Swap --------------------------------- **Caution**: On systems with extremely high memory pressure, using a zvol for swap can result in lockup, regardless of how much swap is still available. This issue is currently being investigated in: `https://github.com/openzfs/zfs/issues/7734 `__ 7.1 Create a volume dataset (zvol) for use as a swap device:: zfs create -V 4G -b $(getconf PAGESIZE) -o compression=zle \ -o logbias=throughput -o sync=always \ -o primarycache=metadata -o secondarycache=none \ -o com.sun:auto-snapshot=false rpool/swap You can adjust the size (the ``4G`` part) to your needs. The compression algorithm is set to ``zle`` because it is the cheapest available algorithm. As this guide recommends ``ashift=12`` (4 kiB blocks on disk), the common case of a 4 kiB page size means that no compression algorithm can reduce I/O. The exception is all-zero pages, which are dropped by ZFS; but some form of compression has to be enabled to get this behavior. 7.2 Configure the swap device: **Caution**: Always use long ``/dev/zvol`` aliases in configuration files. Never use a short ``/dev/zdX`` device name. :: mkswap -f /dev/zvol/rpool/swap echo /dev/zvol/rpool/swap none swap discard 0 0 >> /etc/fstab echo RESUME=none > /etc/initramfs-tools/conf.d/resume The ``RESUME=none`` is necessary to disable resuming from hibernation. This does not work, as the zvol is not present (because the pool has not yet been imported) at the time the resume script runs. If it is not disabled, the boot process hangs for 30 seconds waiting for the swap zvol to appear. 7.3 Enable the swap device:: swapon -av Step 8: Full Software Installation ---------------------------------- 8.1 Upgrade the minimal system:: apt dist-upgrade --yes 8.2 Install a regular set of software: Choose one of the following options: 8.2a Install a command-line environment only:: apt install --yes ubuntu-standard 8.2b Install a full GUI environment:: apt install --yes ubuntu-desktop vi /etc/gdm3/custom.conf # In the [daemon] section, add: InitialSetupEnable=false **Hint**: If you are installing a full GUI environment, you will likely want to manage your network with NetworkManager:: rm /mnt/etc/netplan/01-netcfg.yaml vi /etc/netplan/01-network-manager-all.yaml .. code-block:: yaml network: version: 2 renderer: NetworkManager 8.3 Optional: Disable log compression: As ``/var/log`` is already compressed by ZFS, logrotate’s compression is going to burn CPU and disk I/O for (in most cases) very little gain. Also, if you are making snapshots of ``/var/log``, logrotate’s compression will actually waste space, as the uncompressed data will live on in the snapshot. You can edit the files in ``/etc/logrotate.d`` by hand to comment out ``compress``, or use this loop (copy-and-paste highly recommended):: for file in /etc/logrotate.d/* ; do if grep -Eq "(^|[^#y])compress" "$file" ; then sed -i -r "s/(^|[^#y])(compress)/\1#\2/" "$file" fi done 8.4 Reboot:: reboot Step 9: Final Cleanup --------------------- 9.1 Wait for the system to boot normally. Login using the account you created. Ensure the system (including networking) works normally. 9.2 Optional: Delete the snapshots of the initial installation:: sudo zfs destroy bpool/BOOT/ubuntu@install sudo zfs destroy rpool/ROOT/ubuntu@install 9.3 Optional: Disable the root password:: sudo usermod -p '*' root 9.4 Optional: Re-enable the graphical boot process: If you prefer the graphical boot process, you can re-enable it now. If you are using LUKS, it makes the prompt look nicer. :: sudo vi /etc/default/grub # Uncomment: GRUB_TIMEOUT_STYLE=hidden # Add quiet and splash to: GRUB_CMDLINE_LINUX_DEFAULT # Comment out: GRUB_TERMINAL=console # Save and quit. sudo update-grub **Note:** Ignore errors from ``osprober``, if present. 9.5 Optional: For LUKS installs only, backup the LUKS header:: sudo cryptsetup luksHeaderBackup /dev/disk/by-id/scsi-SATA_disk1-part4 \ --header-backup-file luks1-header.dat Store that backup somewhere safe (e.g. cloud storage). It is protected by your LUKS passphrase, but you may wish to use additional encryption. **Hint:** If you created a mirror or raidz topology, repeat this for each LUKS volume (``luks2``, etc.). Troubleshooting --------------- Rescuing using a Live CD ~~~~~~~~~~~~~~~~~~~~~~~~ Go through `Step 1: Prepare The Install Environment <#step-1-prepare-the-install-environment>`__. For LUKS, first unlock the disk(s):: cryptsetup luksOpen /dev/disk/by-id/scsi-SATA_disk1-part4 luks1 # Repeat for additional disks, if this is a mirror or raidz topology. Mount everything correctly:: zpool export -a zpool import -N -R /mnt rpool zpool import -N -R /mnt bpool zfs mount rpool/ROOT/ubuntu zfs mount -a If needed, you can chroot into your installed environment:: mount --rbind /dev /mnt/dev mount --rbind /proc /mnt/proc mount --rbind /sys /mnt/sys chroot /mnt /bin/bash --login mount /boot/efi mount -a Do whatever you need to do to fix your system. When done, cleanup:: exit mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {} zpool export -a reboot MPT2SAS ~~~~~~~ Most problem reports for this tutorial involve ``mpt2sas`` hardware that does slow asynchronous drive initialization, like some IBM M1015 or OEM-branded cards that have been flashed to the reference LSI firmware. The basic problem is that disks on these controllers are not visible to the Linux kernel until after the regular system is started, and ZoL does not hotplug pool members. See `https://github.com/openzfs/zfs/issues/330 `__. Most LSI cards are perfectly compatible with ZoL. If your card has this glitch, try setting ``ZFS_INITRD_PRE_MOUNTROOT_SLEEP=X`` in ``/etc/default/zfs``. The system will wait ``X`` seconds for all drives to appear before importing the pool. Areca ~~~~~ Systems that require the ``arcsas`` blob driver should add it to the ``/etc/initramfs-tools/modules`` file and run ``update-initramfs -c -k all``. Upgrade or downgrade the Areca driver if something like ``RIP: 0010:[] [] native_read_tsc+0x6/0x20`` appears anywhere in kernel log. ZoL is unstable on systems that emit this error message. VMware ~~~~~~ - Set ``disk.EnableUUID = "TRUE"`` in the vmx file or vsphere configuration. Doing this ensures that ``/dev/disk`` aliases are created in the guest. QEMU/KVM/XEN ~~~~~~~~~~~~ Set a unique serial number on each virtual disk using libvirt or qemu (e.g. ``-drive if=none,id=disk1,file=disk1.qcow2,serial=1234567890``). To be able to use UEFI in guests (instead of only BIOS booting), run this on the host:: sudo apt install ovmf sudo vi /etc/libvirt/qemu.conf Uncomment these lines: .. code-block:: text nvram = [ "/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd", "/usr/share/AAVMF/AAVMF_CODE.fd:/usr/share/AAVMF/AAVMF_VARS.fd" ] :: sudo systemctl restart libvirtd.service .. highlight:: sh Ubuntu 20.04 Root on ZFS ======================== .. contents:: Table of Contents :local: Newer release available ----------------------- - See :doc:`Ubuntu 22.04 Root on ZFS <./Ubuntu 22.04 Root on ZFS>` for new installs. This guide is no longer receiving most updates. It continues to exist for reference for existing installs that followed it. Errata ------ If you previously installed using this guide, please apply these fixes if applicable: /boot/grub Not Mounted ~~~~~~~~~~~~~~~~~~~~~~ | **Severity:** Normal (previously Grave) | **Fixed:** 2020-12-05 (previously 2020-05-30) For a mirror or raidz topology, ``/boot/grub`` is on a separate dataset. This was originally ``bpool/grub``, then changed on 2020-05-30 to ``bpool/BOOT/ubuntu_UUID/grub`` to work-around zsys setting ``canmount=off`` which would result in ``/boot/grub`` not mounting. This work-around lead to `issues with snapshot restores `__. The underlying `zsys issue `__ was fixed and backported to 20.04, so it is now back to being ``bpool/grub``. * If you never applied the 2020-05-30 errata fix, then ``/boot/grub`` is probably not mounting. Check that:: mount | grep /boot/grub If it is mounted, everything is fine. Stop. Otherwise:: zfs set canmount=on bpool/grub update-initramfs -c -k all update-grub grub-install --target=x86_64-efi --efi-directory=/boot/efi \ --bootloader-id=ubuntu --recheck Run this for the additional disk(s), incrementing the “2” to “3” and so on for both ``/boot/efi2`` and ``ubuntu-2``:: cp -a /boot/efi/EFI /boot/efi2 grub-install --target=x86_64-efi --efi-directory=/boot/efi2 \ --bootloader-id=ubuntu-2 --recheck Check that these have ``set prefix=($root)'/grub@'``:: grep prefix= \ /boot/efi/EFI/ubuntu/grub.cfg \ /boot/efi2/EFI/ubuntu-2/grub.cfg * If you applied the 2020-05-30 errata fix, then you should revert the dataset rename:: umount /boot/grub zfs rename bpool/BOOT/ubuntu_UUID/grub bpool/grub zfs set com.ubuntu.zsys:bootfs=no bpool/grub zfs mount bpool/grub AccountsService Not Mounted ~~~~~~~~~~~~~~~~~~~~~~~~~~~ | **Severity:** Normal | **Fixed:** 2020-05-28 The HOWTO previously had a typo in AccountsService (where Accounts is plural) as AccountServices (where Services is plural). This means that AccountsService data will be written to the root filesystem. This is only harmful in the event of a rollback of the root filesystem that does not include a rollback of the user data. Check it:: zfs list | grep Account If the “s” is on “Accounts”, you are good. If it is on “Services”, fix it:: mv /var/lib/AccountsService /var/lib/AccountsService-old zfs list -r rpool # Replace the UUID twice below: zfs rename rpool/ROOT/ubuntu_UUID/var/lib/AccountServices \ rpool/ROOT/ubuntu_UUID/var/lib/AccountsService mv /var/lib/AccountsService-old/* /var/lib/AccountsService rmdir /var/lib/AccountsService-old Overview -------- Ubuntu Installer ~~~~~~~~~~~~~~~~ The Ubuntu installer has `support for root-on-ZFS `__. This HOWTO produces nearly identical results as the Ubuntu installer because of `bidirectional collaboration `__. If you want a single-disk, unencrypted, desktop install, use the installer. It is far easier and faster than doing everything by hand. If you want a ZFS native encrypted, desktop install, you can `trivially edit the installer `__. The ``-O recordsize=1M`` there is unrelated to encryption; omit that unless you understand it. Make sure to use a password that is at least 8 characters or this hack will crash the installer. Additionally, once the system is installed, you should switch to encrypted swap:: swapon -v # Note the device, including the partition. ls -l /dev/disk/by-id/ # Find the by-id name of the disk. sudo swapoff -a sudo vi /etc/fstab # Remove the swap entry. sudo apt install --yes cryptsetup # Replace DISK-partN as appropriate from above: echo swap /dev/disk/by-id/DISK-partN /dev/urandom \ swap,cipher=aes-xts-plain64:sha256,size=512 | sudo tee -a /etc/crypttab echo /dev/mapper/swap none swap defaults 0 0 | sudo tee -a /etc/fstab `Hopefully the installer will gain encryption support in the future `__. If you want to setup a mirror or raidz topology, use LUKS encryption, and/or install a server (no desktop GUI), use this HOWTO. Raspberry Pi ~~~~~~~~~~~~ If you are looking to install on a Raspberry Pi, see :doc:`Ubuntu 20.04 Root on ZFS for Raspberry Pi`. Caution ~~~~~~~ - This HOWTO uses a whole physical disk. - Do not use these instructions for dual-booting. - Backup your data. Any existing data will be lost. System Requirements ~~~~~~~~~~~~~~~~~~~ - `Ubuntu 20.04.4 (“Focal”) Desktop CD `__ (*not* any server images) - Installing on a drive which presents 4 KiB logical sectors (a “4Kn” drive) only works with UEFI booting. This not unique to ZFS. `GRUB does not and will not work on 4Kn with legacy (BIOS) booting. `__ Computers that have less than 2 GiB of memory run ZFS slowly. 4 GiB of memory is recommended for normal performance in basic workloads. If you wish to use deduplication, you will need `massive amounts of RAM `__. Enabling deduplication is a permanent change that cannot be easily reverted. Support ~~~~~~~ If you need help, reach out to the community using the :ref:`mailing_lists` or IRC at `#zfsonlinux `__ on `Libera Chat `__. If you have a bug report or feature request related to this HOWTO, please `file a new issue and mention @rlaager `__. Contributing ~~~~~~~~~~~~ #. Fork and clone: https://github.com/openzfs/openzfs-docs #. Install the tools:: sudo apt install python3-pip pip3 install -r docs/requirements.txt # Add ~/.local/bin to your $PATH, e.g. by adding this to ~/.bashrc: PATH=$HOME/.local/bin:$PATH #. Make your changes. #. Test:: cd docs make html sensible-browser _build/html/index.html #. ``git commit --signoff`` to a branch, ``git push``, and create a pull request. Mention @rlaager. Encryption ~~~~~~~~~~ This guide supports three different encryption options: unencrypted, ZFS native encryption, and LUKS. With any option, all ZFS features are fully available. Unencrypted does not encrypt anything, of course. With no encryption happening, this option naturally has the best performance. ZFS native encryption encrypts the data and most metadata in the root pool. It does not encrypt dataset or snapshot names or properties. The boot pool is not encrypted at all, but it only contains the bootloader, kernel, and initrd. (Unless you put a password in ``/etc/fstab``, the initrd is unlikely to contain sensitive data.) The system cannot boot without the passphrase being entered at the console. Performance is good. As the encryption happens in ZFS, even if multiple disks (mirror or raidz topologies) are used, the data only has to be encrypted once. LUKS encrypts almost everything. The only unencrypted data is the bootloader, kernel, and initrd. The system cannot boot without the passphrase being entered at the console. Performance is good, but LUKS sits underneath ZFS, so if multiple disks (mirror or raidz topologies) are used, the data has to be encrypted once per disk. Step 1: Prepare The Install Environment --------------------------------------- #. Boot the Ubuntu Live CD. Select Try Ubuntu. Connect your system to the Internet as appropriate (e.g. join your WiFi network). Open a terminal (press Ctrl-Alt-T). #. Setup and update the repositories:: sudo apt update #. Optional: Install and start the OpenSSH server in the Live CD environment: If you have a second system, using SSH to access the target system can be convenient:: passwd # There is no current password. sudo apt install --yes openssh-server vim Installing the full ``vim`` package fixes terminal problems that occur when using the ``vim-tiny`` package (that ships in the Live CD environment) over SSH. **Hint:** You can find your IP address with ``ip addr show scope global | grep inet``. Then, from your main machine, connect with ``ssh ubuntu@IP``. #. Disable automounting: If the disk has been used before (with partitions at the same offsets), previous filesystems (e.g. the ESP) will automount if not disabled:: gsettings set org.gnome.desktop.media-handling automount false #. Become root:: sudo -i #. Install ZFS in the Live CD environment:: apt install --yes debootstrap gdisk zfsutils-linux systemctl stop zed Step 2: Disk Formatting ----------------------- #. Set a variable with the disk name:: DISK=/dev/disk/by-id/scsi-SATA_disk1 Always use the long ``/dev/disk/by-id/*`` aliases with ZFS. Using the ``/dev/sd*`` device nodes directly can cause sporadic import failures, especially on systems that have more than one storage pool. **Hints:** - ``ls -la /dev/disk/by-id`` will list the aliases. - Are you doing this in a virtual machine? If your virtual disk is missing from ``/dev/disk/by-id``, use ``/dev/vda`` if you are using KVM with virtio; otherwise, read the `troubleshooting <#troubleshooting>`__ section. - For a mirror or raidz topology, use ``DISK1``, ``DISK2``, etc. - When choosing a boot pool size, consider how you will use the space. A kernel and initrd may consume around 100M. If you have multiple kernels and take snapshots, you may find yourself low on boot pool space, especially if you need to regenerate your initramfs images, which may be around 85M each. Size your boot pool appropriately for your needs. #. If you are re-using a disk, clear it as necessary: Ensure swap partitions are not in use:: swapoff --all If the disk was previously used in an MD array:: apt install --yes mdadm # See if one or more MD arrays are active: cat /proc/mdstat # If so, stop them (replace ``md0`` as required): mdadm --stop /dev/md0 # For an array using the whole disk: mdadm --zero-superblock --force $DISK # For an array using a partition (e.g. a swap partition per this HOWTO): mdadm --zero-superblock --force ${DISK}-part2 Clear the partition table:: sgdisk --zap-all $DISK If you get a message about the kernel still using the old partition table, you can request the kernel reload the partition information using:: partprobe $DISK If the new partitions still don't show up, you can reboot and start over (except that you can skip this step). #. Create bootloader partition(s):: sgdisk -n1:1M:+512M -t1:EF00 $DISK # For legacy (BIOS) booting: sgdisk -a1 -n5:24K:+1000K -t5:EF02 $DISK **Note:** While the Ubuntu installer uses an MBR label for legacy (BIOS) booting, this HOWTO uses GPT partition labels for both UEFI and legacy (BIOS) booting. This is simpler than having two options. It is also provides forward compatibility (future proofing). In other words, for legacy (BIOS) booting, this will allow you to move the disk(s) to a new system/motherboard in the future without having to rebuild the pool (and restore your data from a backup). The ESP is created in both cases for similar reasons. Additionally, the ESP is used for ``/boot/grub`` in single-disk installs, as :ref:`discussed below `. #. Create a partition for swap: Previous versions of this HOWTO put swap on a zvol. `Ubuntu recommends against this configuration due to deadlocks. `__ There is `a bug report upstream `__. Putting swap on a partition gives up the benefit of ZFS checksums (for your swap). That is probably the right trade-off given the reports of ZFS deadlocks with swap. If you are bothered by this, simply do not enable swap. Choose one of the following options if you want swap: - For a single-disk install:: sgdisk -n2:0:+500M -t2:8200 $DISK - For a mirror or raidz topology:: sgdisk -n2:0:+500M -t2:FD00 $DISK Adjust the swap swize to your needs. If you wish to enable hiberation (which only works for unencrypted installs), the swap partition must be at least as large as the system's RAM. #. Create a boot pool partition:: sgdisk -n3:0:+2G -t3:BE00 $DISK The Ubuntu installer uses 5% of the disk space constrained to a minimum of 500 MiB and a maximum of 2 GiB. `Making this too small (and 500 MiB might be too small) can result in an inability to upgrade the kernel. `__ #. Create a root pool partition: Choose one of the following options: - Unencrypted or ZFS native encryption:: sgdisk -n4:0:0 -t4:BF00 $DISK - LUKS:: sgdisk -n4:0:0 -t4:8309 $DISK If you are creating a mirror or raidz topology, repeat the partitioning commands for all the disks which will be part of the pool. #. Create the boot pool:: zpool create \ -o cachefile=/etc/zfs/zpool.cache \ -o ashift=12 -o autotrim=on -d \ -o feature@async_destroy=enabled \ -o feature@bookmarks=enabled \ -o feature@embedded_data=enabled \ -o feature@empty_bpobj=enabled \ -o feature@enabled_txg=enabled \ -o feature@extensible_dataset=enabled \ -o feature@filesystem_limits=enabled \ -o feature@hole_birth=enabled \ -o feature@large_blocks=enabled \ -o feature@lz4_compress=enabled \ -o feature@spacemap_histogram=enabled \ -O acltype=posixacl -O canmount=off -O compression=lz4 \ -O devices=off -O normalization=formD -O relatime=on -O xattr=sa \ -O mountpoint=/boot -R /mnt \ bpool ${DISK}-part3 You should not need to customize any of the options for the boot pool. GRUB does not support all of the zpool features. See ``spa_feature_names`` in `grub-core/fs/zfs/zfs.c `__. This step creates a separate boot pool for ``/boot`` with the features limited to only those that GRUB supports, allowing the root pool to use any/all features. Note that GRUB opens the pool read-only, so all read-only compatible features are “supported” by GRUB. **Hints:** - If you are creating a mirror topology, create the pool using:: zpool create \ ... \ bpool mirror \ /dev/disk/by-id/scsi-SATA_disk1-part3 \ /dev/disk/by-id/scsi-SATA_disk2-part3 - For raidz topologies, replace ``mirror`` in the above command with ``raidz``, ``raidz2``, or ``raidz3`` and list the partitions from the additional disks. - The boot pool name is no longer arbitrary. It _must_ be ``bpool``. If you really want to rename it, edit ``/etc/grub.d/10_linux_zfs`` later, after GRUB is installed (and run ``update-grub``). **Feature Notes:** - The ``allocation_classes`` feature should be safe to use. However, unless one is using it (i.e. a ``special`` vdev), there is no point to enabling it. It is extremely unlikely that someone would use this feature for a boot pool. If one cares about speeding up the boot pool, it would make more sense to put the whole pool on the faster disk rather than using it as a ``special`` vdev. - The ``project_quota`` feature has been tested and is safe to use. This feature is extremely unlikely to matter for the boot pool. - The ``resilver_defer`` should be safe but the boot pool is small enough that it is unlikely to be necessary. - The ``spacemap_v2`` feature has been tested and is safe to use. The boot pool is small, so this does not matter in practice. - As a read-only compatible feature, the ``userobj_accounting`` feature should be compatible in theory, but in practice, GRUB can fail with an “invalid dnode type” error. This feature does not matter for ``/boot`` anyway. #. Create the root pool: Choose one of the following options: - Unencrypted:: zpool create \ -o ashift=12 -o autotrim=on \ -O acltype=posixacl -O canmount=off -O compression=lz4 \ -O dnodesize=auto -O normalization=formD -O relatime=on \ -O xattr=sa -O mountpoint=/ -R /mnt \ rpool ${DISK}-part4 - ZFS native encryption:: zpool create \ -o ashift=12 -o autotrim=on \ -O encryption=aes-256-gcm \ -O keylocation=prompt -O keyformat=passphrase \ -O acltype=posixacl -O canmount=off -O compression=lz4 \ -O dnodesize=auto -O normalization=formD -O relatime=on \ -O xattr=sa -O mountpoint=/ -R /mnt \ rpool ${DISK}-part4 - LUKS:: cryptsetup luksFormat -c aes-xts-plain64 -s 512 -h sha256 ${DISK}-part4 cryptsetup luksOpen ${DISK}-part4 luks1 zpool create \ -o ashift=12 -o autotrim=on \ -O acltype=posixacl -O canmount=off -O compression=lz4 \ -O dnodesize=auto -O normalization=formD -O relatime=on \ -O xattr=sa -O mountpoint=/ -R /mnt \ rpool /dev/mapper/luks1 **Notes:** - The use of ``ashift=12`` is recommended here because many drives today have 4 KiB (or larger) physical sectors, even though they present 512 B logical sectors. Also, a future replacement drive may have 4 KiB physical sectors (in which case ``ashift=12`` is desirable) or 4 KiB logical sectors (in which case ``ashift=12`` is required). - Setting ``-O acltype=posixacl`` enables POSIX ACLs globally. If you do not want this, remove that option, but later add ``-o acltype=posixacl`` (note: lowercase “o”) to the ``zfs create`` for ``/var/log``, as `journald requires ACLs `__ Also, `disabling ACLs apparently breaks umask handling with NFSv4 `__. - Setting ``normalization=formD`` eliminates some corner cases relating to UTF-8 filename normalization. It also implies ``utf8only=on``, which means that only UTF-8 filenames are allowed. If you care to support non-UTF-8 filenames, do not use this option. For a discussion of why requiring UTF-8 filenames may be a bad idea, see `The problems with enforced UTF-8 only filenames `__. - ``recordsize`` is unset (leaving it at the default of 128 KiB). If you want to tune it (e.g. ``-O recordsize=1M``), see `these `__ `various `__ `blog `__ `posts `__. - Setting ``relatime=on`` is a middle ground between classic POSIX ``atime`` behavior (with its significant performance impact) and ``atime=off`` (which provides the best performance by completely disabling atime updates). Since Linux 2.6.30, ``relatime`` has been the default for other filesystems. See `RedHat’s documentation `__ for further information. - Setting ``xattr=sa`` `vastly improves the performance of extended attributes `__. Inside ZFS, extended attributes are used to implement POSIX ACLs. Extended attributes can also be used by user-space applications. `They are used by some desktop GUI applications. `__ `They can be used by Samba to store Windows ACLs and DOS attributes; they are required for a Samba Active Directory domain controller. `__ Note that ``xattr=sa`` is `Linux-specific `__. If you move your ``xattr=sa`` pool to another OpenZFS implementation besides ZFS-on-Linux, extended attributes will not be readable (though your data will be). If portability of extended attributes is important to you, omit the ``-O xattr=sa`` above. Even if you do not want ``xattr=sa`` for the whole pool, it is probably fine to use it for ``/var/log``. - Make sure to include the ``-part4`` portion of the drive path. If you forget that, you are specifying the whole disk, which ZFS will then re-partition, and you will lose the bootloader partition(s). - ZFS native encryption defaults to ``aes-256-ccm``, but `the default has changed upstream `__ to ``aes-256-gcm``. `AES-GCM seems to be generally preferred over AES-CCM `__, `is faster now `__, and `will be even faster in the future `__. - For LUKS, the key size chosen is 512 bits. However, XTS mode requires two keys, so the LUKS key is split in half. Thus, ``-s 512`` means AES-256. - Your passphrase will likely be the weakest link. Choose wisely. See `section 5 of the cryptsetup FAQ `__ for guidance. **Hints:** - If you are creating a mirror topology, create the pool using:: zpool create \ ... \ rpool mirror \ /dev/disk/by-id/scsi-SATA_disk1-part4 \ /dev/disk/by-id/scsi-SATA_disk2-part4 - For raidz topologies, replace ``mirror`` in the above command with ``raidz``, ``raidz2``, or ``raidz3`` and list the partitions from the additional disks. - When using LUKS with mirror or raidz topologies, use ``/dev/mapper/luks1``, ``/dev/mapper/luks2``, etc., which you will have to create using ``cryptsetup``. - The pool name is arbitrary. If changed, the new name must be used consistently. On systems that can automatically install to ZFS, the root pool is named ``rpool`` by default. Step 3: System Installation --------------------------- #. Create filesystem datasets to act as containers:: zfs create -o canmount=off -o mountpoint=none rpool/ROOT zfs create -o canmount=off -o mountpoint=none bpool/BOOT #. Create filesystem datasets for the root and boot filesystems:: UUID=$(dd if=/dev/urandom bs=1 count=100 2>/dev/null | tr -dc 'a-z0-9' | cut -c-6) zfs create -o mountpoint=/ \ -o com.ubuntu.zsys:bootfs=yes \ -o com.ubuntu.zsys:last-used=$(date +%s) rpool/ROOT/ubuntu_$UUID zfs create -o mountpoint=/boot bpool/BOOT/ubuntu_$UUID #. Create datasets:: zfs create -o com.ubuntu.zsys:bootfs=no \ rpool/ROOT/ubuntu_$UUID/srv zfs create -o com.ubuntu.zsys:bootfs=no -o canmount=off \ rpool/ROOT/ubuntu_$UUID/usr zfs create rpool/ROOT/ubuntu_$UUID/usr/local zfs create -o com.ubuntu.zsys:bootfs=no -o canmount=off \ rpool/ROOT/ubuntu_$UUID/var zfs create rpool/ROOT/ubuntu_$UUID/var/games zfs create rpool/ROOT/ubuntu_$UUID/var/lib zfs create rpool/ROOT/ubuntu_$UUID/var/lib/AccountsService zfs create rpool/ROOT/ubuntu_$UUID/var/lib/apt zfs create rpool/ROOT/ubuntu_$UUID/var/lib/dpkg zfs create rpool/ROOT/ubuntu_$UUID/var/lib/NetworkManager zfs create rpool/ROOT/ubuntu_$UUID/var/log zfs create rpool/ROOT/ubuntu_$UUID/var/mail zfs create rpool/ROOT/ubuntu_$UUID/var/snap zfs create rpool/ROOT/ubuntu_$UUID/var/spool zfs create rpool/ROOT/ubuntu_$UUID/var/www zfs create -o canmount=off -o mountpoint=/ \ rpool/USERDATA zfs create -o com.ubuntu.zsys:bootfs-datasets=rpool/ROOT/ubuntu_$UUID \ -o canmount=on -o mountpoint=/root \ rpool/USERDATA/root_$UUID chmod 700 /mnt/root For a mirror or raidz topology, create a dataset for ``/boot/grub``:: zfs create -o com.ubuntu.zsys:bootfs=no bpool/grub Mount a tmpfs at /run:: mkdir /mnt/run mount -t tmpfs tmpfs /mnt/run mkdir /mnt/run/lock A tmpfs is recommended later, but if you want a separate dataset for ``/tmp``:: zfs create -o com.ubuntu.zsys:bootfs=no \ rpool/ROOT/ubuntu_$UUID/tmp chmod 1777 /mnt/tmp The primary goal of this dataset layout is to separate the OS from user data. This allows the root filesystem to be rolled back without rolling back user data. If you do nothing extra, ``/tmp`` will be stored as part of the root filesystem. Alternatively, you can create a separate dataset for ``/tmp``, as shown above. This keeps the ``/tmp`` data out of snapshots of your root filesystem. It also allows you to set a quota on ``rpool/tmp``, if you want to limit the maximum space used. Otherwise, you can use a tmpfs (RAM filesystem) later. #. Install the minimal system:: debootstrap focal /mnt The ``debootstrap`` command leaves the new system in an unconfigured state. An alternative to using ``debootstrap`` is to copy the entirety of a working system into the new ZFS root. #. Copy in zpool.cache:: mkdir /mnt/etc/zfs cp /etc/zfs/zpool.cache /mnt/etc/zfs/ Step 4: System Configuration ---------------------------- #. Configure the hostname: Replace ``HOSTNAME`` with the desired hostname:: hostname HOSTNAME hostname > /mnt/etc/hostname vi /mnt/etc/hosts .. code-block:: text Add a line: 127.0.1.1 HOSTNAME or if the system has a real name in DNS: 127.0.1.1 FQDN HOSTNAME **Hint:** Use ``nano`` if you find ``vi`` confusing. #. Configure the network interface: Find the interface name:: ip addr show Adjust ``NAME`` below to match your interface name:: vi /mnt/etc/netplan/01-netcfg.yaml .. code-block:: yaml network: version: 2 ethernets: NAME: dhcp4: true Customize this file if the system is not a DHCP client. #. Configure the package sources:: vi /mnt/etc/apt/sources.list .. code-block:: sourceslist deb http://archive.ubuntu.com/ubuntu focal main restricted universe multiverse deb http://archive.ubuntu.com/ubuntu focal-updates main restricted universe multiverse deb http://archive.ubuntu.com/ubuntu focal-backports main restricted universe multiverse deb http://security.ubuntu.com/ubuntu focal-security main restricted universe multiverse #. Bind the virtual filesystems from the LiveCD environment to the new system and ``chroot`` into it:: mount --make-private --rbind /dev /mnt/dev mount --make-private --rbind /proc /mnt/proc mount --make-private --rbind /sys /mnt/sys chroot /mnt /usr/bin/env DISK=$DISK UUID=$UUID bash --login **Note:** This is using ``--rbind``, not ``--bind``. #. Configure a basic system environment:: apt update Even if you prefer a non-English system language, always ensure that ``en_US.UTF-8`` is available:: dpkg-reconfigure locales tzdata keyboard-configuration console-setup Install your preferred text editor:: apt install --yes nano apt install --yes vim Installing the full ``vim`` package fixes terminal problems that occur when using the ``vim-tiny`` package (that is installed by ``debootstrap``) over SSH. #. For LUKS installs only, setup ``/etc/crypttab``:: apt install --yes cryptsetup echo luks1 /dev/disk/by-uuid/$(blkid -s UUID -o value ${DISK}-part4) \ none luks,discard,initramfs > /etc/crypttab The use of ``initramfs`` is a work-around for `cryptsetup does not support ZFS `__. **Hint:** If you are creating a mirror or raidz topology, repeat the ``/etc/crypttab`` entries for ``luks2``, etc. adjusting for each disk. #. Create the EFI filesystem: Perform these steps for both UEFI and legacy (BIOS) booting:: apt install --yes dosfstools mkdosfs -F 32 -s 1 -n EFI ${DISK}-part1 mkdir /boot/efi echo /dev/disk/by-uuid/$(blkid -s UUID -o value ${DISK}-part1) \ /boot/efi vfat defaults 0 0 >> /etc/fstab mount /boot/efi For a mirror or raidz topology, repeat the `mkdosfs` for the additional disks, but do not repeat the other commands. **Note:** The ``-s 1`` for ``mkdosfs`` is only necessary for drives which present 4 KiB logical sectors (“4Kn” drives) to meet the minimum cluster size (given the partition size of 512 MiB) for FAT32. It also works fine on drives which present 512 B sectors. #. Put ``/boot/grub`` on the EFI System Partition: .. _boot-grub-esp: For a single-disk install only:: mkdir /boot/efi/grub /boot/grub echo /boot/efi/grub /boot/grub none defaults,bind 0 0 >> /etc/fstab mount /boot/grub This allows GRUB to write to ``/boot/grub`` (since it is on a FAT-formatted ESP instead of on ZFS), which means that ``/boot/grub/grubenv`` and the ``recordfail`` feature works as expected: if the boot fails, the normally hidden GRUB menu will be shown on the next boot. For a mirror or raidz topology, we do not want GRUB writing to the EFI System Partition. This is because we duplicate it at install without a mechanism to update the copies when the GRUB configuration changes (e.g. as the kernel is upgraded). Thus, we keep ``/boot/grub`` on the boot pool for the mirror or raidz topologies. This preserves correct mirroring/raidz behavior, at the expense of being able to write to ``/boot/grub/grubenv`` and thus the ``recordfail`` behavior. #. Install GRUB/Linux/ZFS in the chroot environment for the new system: Choose one of the following options: - Install GRUB/Linux/ZFS for legacy (BIOS) booting:: apt install --yes grub-pc linux-image-generic zfs-initramfs zsys Select (using the space bar) all of the disks (not partitions) in your pool. - Install GRUB/Linux/ZFS for UEFI booting:: apt install --yes \ grub-efi-amd64 grub-efi-amd64-signed linux-image-generic \ shim-signed zfs-initramfs zsys **Notes:** - Ignore any error messages saying ``ERROR: Couldn't resolve device`` and ``WARNING: Couldn't determine root device``. `cryptsetup does not support ZFS `__. - Ignore any error messages saying ``Module zfs not found`` and ``couldn't connect to zsys daemon``. The first seems to occur due to a version mismatch between the Live CD kernel and the chroot environment, but this is irrelevant since the module is already loaded. The second may be caused by the first but either way is irrelevant since ``zed`` is started manually later. - For a mirror or raidz topology, this step only installs GRUB on the first disk. The other disk(s) will be handled later. For some reason, grub-efi-amd64 does not prompt for ``install_devices`` here, but does after a reboot. #. Optional: Remove os-prober:: apt purge --yes os-prober This avoids error messages from ``update-grub``. ``os-prober`` is only necessary in dual-boot configurations. #. Set a root password:: passwd #. Configure swap: Choose one of the following options if you want swap: - For an unencrypted single-disk install:: mkswap -f ${DISK}-part2 echo /dev/disk/by-uuid/$(blkid -s UUID -o value ${DISK}-part2) \ none swap discard 0 0 >> /etc/fstab swapon -a - For an unencrypted mirror or raidz topology:: apt install --yes mdadm # Adjust the level (ZFS raidz = MD raid5, raidz2 = raid6) and # raid-devices if necessary and specify the actual devices. mdadm --create /dev/md0 --metadata=1.2 --level=mirror \ --raid-devices=2 ${DISK1}-part2 ${DISK2}-part2 mkswap -f /dev/md0 echo /dev/disk/by-uuid/$(blkid -s UUID -o value /dev/md0) \ none swap discard 0 0 >> /etc/fstab - For an encrypted (LUKS or ZFS native encryption) single-disk install:: apt install --yes cryptsetup echo swap ${DISK}-part2 /dev/urandom \ swap,cipher=aes-xts-plain64:sha256,size=512 >> /etc/crypttab echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab - For an encrypted (LUKS or ZFS native encryption) mirror or raidz topology:: apt install --yes cryptsetup mdadm # Adjust the level (ZFS raidz = MD raid5, raidz2 = raid6) and # raid-devices if necessary and specify the actual devices. mdadm --create /dev/md0 --metadata=1.2 --level=mirror \ --raid-devices=2 ${DISK1}-part2 ${DISK2}-part2 echo swap /dev/md0 /dev/urandom \ swap,cipher=aes-xts-plain64:sha256,size=512 >> /etc/crypttab echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab #. Optional (but recommended): Mount a tmpfs to ``/tmp`` If you chose to create a ``/tmp`` dataset above, skip this step, as they are mutually exclusive choices. Otherwise, you can put ``/tmp`` on a tmpfs (RAM filesystem) by enabling the ``tmp.mount`` unit. :: cp /usr/share/systemd/tmp.mount /etc/systemd/system/ systemctl enable tmp.mount #. Setup system groups:: addgroup --system lpadmin addgroup --system lxd addgroup --system sambashare #. Patch a dependency loop: For ZFS native encryption or LUKS:: apt install --yes curl patch curl https://launchpadlibrarian.net/478315221/2150-fix-systemd-dependency-loops.patch | \ sed "s|/etc|/lib|;s|\.in$||" | (cd / ; patch -p1) Ignore the failure in Hunk #2 (say ``n`` twice). This patch is from `Bug #1875577 Encrypted swap won't load on 20.04 with zfs root `__. #. Optional: Install SSH:: apt install --yes openssh-server vi /etc/ssh/sshd_config # Set: PermitRootLogin yes Step 5: GRUB Installation ------------------------- #. Verify that the ZFS boot filesystem is recognized:: grub-probe /boot #. Refresh the initrd files:: update-initramfs -c -k all **Note:** Ignore any error messages saying ``ERROR: Couldn't resolve device`` and ``WARNING: Couldn't determine root device``. `cryptsetup does not support ZFS `__. #. Disable memory zeroing:: vi /etc/default/grub # Add init_on_alloc=0 to: GRUB_CMDLINE_LINUX_DEFAULT # Save and quit (or see the next step). This is to address `performance regressions `__. #. Optional (but highly recommended): Make debugging GRUB easier:: vi /etc/default/grub # Comment out: GRUB_TIMEOUT_STYLE=hidden # Set: GRUB_TIMEOUT=5 # Below GRUB_TIMEOUT, add: GRUB_RECORDFAIL_TIMEOUT=5 # Remove quiet and splash from: GRUB_CMDLINE_LINUX_DEFAULT # Uncomment: GRUB_TERMINAL=console # Save and quit. Later, once the system has rebooted twice and you are sure everything is working, you can undo these changes, if desired. #. Update the boot configuration:: update-grub **Note:** Ignore errors from ``osprober``, if present. #. Install the boot loader: Choose one of the following options: - For legacy (BIOS) booting, install GRUB to the MBR:: grub-install $DISK Note that you are installing GRUB to the whole disk, not a partition. If you are creating a mirror or raidz topology, repeat the ``grub-install`` command for each disk in the pool. - For UEFI booting, install GRUB to the ESP:: grub-install --target=x86_64-efi --efi-directory=/boot/efi \ --bootloader-id=ubuntu --recheck #. Disable grub-initrd-fallback.service For a mirror or raidz topology:: systemctl mask grub-initrd-fallback.service This is the service for ``/boot/grub/grubenv`` which does not work on mirrored or raidz topologies. Disabling this keeps it from blocking subsequent mounts of ``/boot/grub`` if that mount ever fails. Another option would be to set ``RequiresMountsFor=/boot/grub`` via a drop-in unit, but that is more work to do here for no reason. Hopefully `this bug `__ will be fixed upstream. #. Fix filesystem mount ordering: We need to activate ``zfs-mount-generator``. This makes systemd aware of the separate mountpoints, which is important for things like ``/var/log`` and ``/var/tmp``. In turn, ``rsyslog.service`` depends on ``var-log.mount`` by way of ``local-fs.target`` and services using the ``PrivateTmp`` feature of systemd automatically use ``After=var-tmp.mount``. :: mkdir /etc/zfs/zfs-list.cache touch /etc/zfs/zfs-list.cache/bpool touch /etc/zfs/zfs-list.cache/rpool ln -s /usr/lib/zfs-linux/zed.d/history_event-zfs-list-cacher.sh /etc/zfs/zed.d zed -F & Verify that ``zed`` updated the cache by making sure these are not empty:: cat /etc/zfs/zfs-list.cache/bpool cat /etc/zfs/zfs-list.cache/rpool If either is empty, force a cache update and check again:: zfs set canmount=on bpool/BOOT/ubuntu_$UUID zfs set canmount=on rpool/ROOT/ubuntu_$UUID If they are still empty, stop zed (as below), start zed (as above) and try again. Once the files have data, stop ``zed``:: fg Press Ctrl-C. Fix the paths to eliminate ``/mnt``:: sed -Ei "s|/mnt/?|/|" /etc/zfs/zfs-list.cache/* #. Exit from the ``chroot`` environment back to the LiveCD environment:: exit #. Run these commands in the LiveCD environment to unmount all filesystems:: mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | \ xargs -i{} umount -lf {} zpool export -a #. Reboot:: reboot Wait for the newly installed system to boot normally. Login as root. Step 6: First Boot ------------------ #. Install GRUB to additional disks: For a UEFI mirror or raidz topology only:: dpkg-reconfigure grub-efi-amd64 Select (using the space bar) all of the ESP partitions (partition 1 on each of the pool disks). #. Create a user account: Replace ``YOUR_USERNAME`` with your desired username:: username=YOUR_USERNAME UUID=$(dd if=/dev/urandom bs=1 count=100 2>/dev/null | tr -dc 'a-z0-9' | cut -c-6) ROOT_DS=$(zfs list -o name | awk '/ROOT\/ubuntu_/{print $1;exit}') zfs create -o com.ubuntu.zsys:bootfs-datasets=$ROOT_DS \ -o canmount=on -o mountpoint=/home/$username \ rpool/USERDATA/${username}_$UUID adduser $username cp -a /etc/skel/. /home/$username chown -R $username:$username /home/$username usermod -a -G adm,cdrom,dip,lpadmin,lxd,plugdev,sambashare,sudo $username Step 7: Full Software Installation ---------------------------------- #. Upgrade the minimal system:: apt dist-upgrade --yes #. Install a regular set of software: Choose one of the following options: - Install a command-line environment only:: apt install --yes ubuntu-standard - Install a full GUI environment:: apt install --yes ubuntu-desktop **Hint**: If you are installing a full GUI environment, you will likely want to manage your network with NetworkManager:: rm /etc/netplan/01-netcfg.yaml vi /etc/netplan/01-network-manager-all.yaml .. code-block:: yaml network: version: 2 renderer: NetworkManager #. Optional: Disable log compression: As ``/var/log`` is already compressed by ZFS, logrotate’s compression is going to burn CPU and disk I/O for (in most cases) very little gain. Also, if you are making snapshots of ``/var/log``, logrotate’s compression will actually waste space, as the uncompressed data will live on in the snapshot. You can edit the files in ``/etc/logrotate.d`` by hand to comment out ``compress``, or use this loop (copy-and-paste highly recommended):: for file in /etc/logrotate.d/* ; do if grep -Eq "(^|[^#y])compress" "$file" ; then sed -i -r "s/(^|[^#y])(compress)/\1#\2/" "$file" fi done #. Reboot:: reboot Step 8: Final Cleanup --------------------- #. Wait for the system to boot normally. Login using the account you created. Ensure the system (including networking) works normally. #. Optional: Disable the root password:: sudo usermod -p '*' root #. Optional (but highly recommended): Disable root SSH logins: If you installed SSH earlier, revert the temporary change:: sudo vi /etc/ssh/sshd_config # Remove: PermitRootLogin yes sudo systemctl restart ssh #. Optional: Re-enable the graphical boot process: If you prefer the graphical boot process, you can re-enable it now. If you are using LUKS, it makes the prompt look nicer. :: sudo vi /etc/default/grub # Uncomment: GRUB_TIMEOUT_STYLE=hidden # Add quiet and splash to: GRUB_CMDLINE_LINUX_DEFAULT # Comment out: GRUB_TERMINAL=console # Save and quit. sudo update-grub **Note:** Ignore errors from ``osprober``, if present. #. Optional: For LUKS installs only, backup the LUKS header:: sudo cryptsetup luksHeaderBackup /dev/disk/by-id/scsi-SATA_disk1-part4 \ --header-backup-file luks1-header.dat Store that backup somewhere safe (e.g. cloud storage). It is protected by your LUKS passphrase, but you may wish to use additional encryption. **Hint:** If you created a mirror or raidz topology, repeat this for each LUKS volume (``luks2``, etc.). Troubleshooting --------------- Rescuing using a Live CD ~~~~~~~~~~~~~~~~~~~~~~~~ Go through `Step 1: Prepare The Install Environment <#step-1-prepare-the-install-environment>`__. For LUKS, first unlock the disk(s):: cryptsetup luksOpen /dev/disk/by-id/scsi-SATA_disk1-part4 luks1 # Repeat for additional disks, if this is a mirror or raidz topology. Mount everything correctly:: zpool export -a zpool import -N -R /mnt rpool zpool import -N -R /mnt bpool zfs load-key -a # Replace “UUID” as appropriate; use zfs list to find it: zfs mount rpool/ROOT/ubuntu_UUID zfs mount bpool/BOOT/ubuntu_UUID zfs mount -a If needed, you can chroot into your installed environment:: mount --make-private --rbind /dev /mnt/dev mount --make-private --rbind /proc /mnt/proc mount --make-private --rbind /sys /mnt/sys mount -t tmpfs tmpfs /mnt/run mkdir /mnt/run/lock chroot /mnt /bin/bash --login mount -a Do whatever you need to do to fix your system. When done, cleanup:: exit mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | \ xargs -i{} umount -lf {} zpool export -a reboot Areca ~~~~~ Systems that require the ``arcsas`` blob driver should add it to the ``/etc/initramfs-tools/modules`` file and run ``update-initramfs -c -k all``. Upgrade or downgrade the Areca driver if something like ``RIP: 0010:[] [] native_read_tsc+0x6/0x20`` appears anywhere in kernel log. ZoL is unstable on systems that emit this error message. MPT2SAS ~~~~~~~ Most problem reports for this tutorial involve ``mpt2sas`` hardware that does slow asynchronous drive initialization, like some IBM M1015 or OEM-branded cards that have been flashed to the reference LSI firmware. The basic problem is that disks on these controllers are not visible to the Linux kernel until after the regular system is started, and ZoL does not hotplug pool members. See `https://github.com/openzfs/zfs/issues/330 `__. Most LSI cards are perfectly compatible with ZoL. If your card has this glitch, try setting ``ZFS_INITRD_PRE_MOUNTROOT_SLEEP=X`` in ``/etc/default/zfs``. The system will wait ``X`` seconds for all drives to appear before importing the pool. QEMU/KVM/XEN ~~~~~~~~~~~~ Set a unique serial number on each virtual disk using libvirt or qemu (e.g. ``-drive if=none,id=disk1,file=disk1.qcow2,serial=1234567890``). To be able to use UEFI in guests (instead of only BIOS booting), run this on the host:: sudo apt install ovmf sudo vi /etc/libvirt/qemu.conf Uncomment these lines: .. code-block:: text nvram = [ "/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd", "/usr/share/OVMF/OVMF_CODE.secboot.fd:/usr/share/OVMF/OVMF_VARS.fd", "/usr/share/AAVMF/AAVMF_CODE.fd:/usr/share/AAVMF/AAVMF_VARS.fd", "/usr/share/AAVMF/AAVMF32_CODE.fd:/usr/share/AAVMF/AAVMF32_VARS.fd", "/usr/share/OVMF/OVMF_CODE.ms.fd:/usr/share/OVMF/OVMF_VARS.ms.fd" ] :: sudo systemctl restart libvirtd.service VMware ~~~~~~ - Set ``disk.EnableUUID = "TRUE"`` in the vmx file or vsphere configuration. Doing this ensures that ``/dev/disk`` aliases are created in the guest. .. highlight:: sh Ubuntu 20.04 Root on ZFS for Raspberry Pi ========================================= .. contents:: Table of Contents :local: Overview -------- Newer release available ~~~~~~~~~~~~~~~~~~~~~~~ - See :doc:`Ubuntu 22.04 Root on ZFS for Raspberry Pi <./Ubuntu 22.04 Root on ZFS for Raspberry Pi>` for new installs. This guide is no longer receiving most updates. It continues to exist for reference for existing installs that followed it. Caution ~~~~~~~ - This HOWTO uses a whole physical disk. - Backup your data. Any existing data will be lost. System Requirements ~~~~~~~~~~~~~~~~~~~ - A Raspberry Pi 4 B. (If you are looking to install on a regular PC, see :doc:`Ubuntu 20.04 Root on ZFS`.) - `Ubuntu Server 20.04.4 (“Focal”) for Raspberry Pi 4 `__ - A microSD card or USB disk. For microSD card recommendations, see Jeff Geerling's `performance comparison `__. When using a USB enclosure, `ensure it supports UASP `__. - An Ubuntu system (with the ability to write to the microSD card or USB disk) other than the target Raspberry Pi. 4 GiB of memory is recommended. Do not use deduplication, as it needs `massive amounts of RAM `__. Enabling deduplication is a permanent change that cannot be easily reverted. A Raspberry Pi 3 B/B+ would probably work (as the Pi 3 is 64-bit, though it has less RAM), but has not been tested. Please report your results (good or bad) using the issue link below. Support ~~~~~~~ If you need help, reach out to the community using the :ref:`mailing_lists` or IRC at `#zfsonlinux `__ on `Libera Chat `__. If you have a bug report or feature request related to this HOWTO, please `file a new issue and mention @rlaager `__. Contributing ~~~~~~~~~~~~ #. Fork and clone: https://github.com/openzfs/openzfs-docs #. Install the tools:: sudo apt install python3-pip pip3 install -r docs/requirements.txt # Add ~/.local/bin to your $PATH, e.g. by adding this to ~/.bashrc: PATH=$HOME/.local/bin:$PATH #. Make your changes. #. Test:: cd docs make html sensible-browser _build/html/index.html #. ``git commit --signoff`` to a branch, ``git push``, and create a pull request. Mention @rlaager. Encryption ~~~~~~~~~~ **WARNING:** Encryption has not yet been tested on the Raspberry Pi. This guide supports three different encryption options: unencrypted, ZFS native encryption, and LUKS. With any option, all ZFS features are fully available. Unencrypted does not encrypt anything, of course. With no encryption happening, this option naturally has the best performance. ZFS native encryption encrypts the data and most metadata in the root pool. It does not encrypt dataset or snapshot names or properties. The boot pool is not encrypted at all, but it only contains the bootloader, kernel, and initrd. (Unless you put a password in ``/etc/fstab``, the initrd is unlikely to contain sensitive data.) The system cannot boot without the passphrase being entered at the console. Performance is good. As the encryption happens in ZFS, even if multiple disks (mirror or raidz topologies) are used, the data only has to be encrypted once. LUKS encrypts almost everything. The only unencrypted data is the bootloader, kernel, and initrd. The system cannot boot without the passphrase being entered at the console. Performance is good, but LUKS sits underneath ZFS, so if multiple disks (mirror or raidz topologies) are used, the data has to be encrypted once per disk. USB Disks ~~~~~~~~~ The Raspberry Pi 4 runs much faster using a USB Solid State Drive (SSD) than a microSD card. These instructions can also be used to install Ubuntu on a USB-connected SSD or other USB disk. USB disks have three requirements that do not apply to microSD cards: #. The Raspberry Pi's Bootloader EEPROM must be dated 2020-09-03 or later. To check the bootloader version, power up the Raspberry Pi without an SD card inserted or a USB boot device attached; the date will be on the ``bootloader`` line. (If you do not see the ``bootloader`` line, the bootloader is too old.) Alternatively, run ``sudo rpi-eeprom-update`` on an existing OS on the Raspberry Pi (which on Ubuntu requires ``apt install rpi-eeprom``). If needed, the bootloader can be updated from an existing OS on the Raspberry Pi using ``rpi-eeprom-update -a`` and rebooting. For other options, see `Updating the Bootloader `_. #. The Raspberry Pi must configured for USB boot. The bootloader will show a ``boot`` line; if ``order`` includes ``4``, USB boot is enabled. If not already enabled, it can be enabled from an existing OS on the Raspberry Pi using ``rpi-eeprom-config -e``: set ``BOOT_ORDER=0xf41`` and reboot to apply the change. On subsequent reboots, USB boot will be enabled. Otherwise, it can be enabled without an existing OS as follows: - Download the `Raspberry Pi Imager Utility `_. - Flash the ``USB Boot`` image to a microSD card. The ``USB Boot`` image is listed under ``Bootloader`` in the ``Misc utility images`` folder. - Boot the Raspberry Pi from the microSD card. USB Boot should be enabled automatically. #. U-Boot on Ubuntu 20.04 does not seem to support the Raspberry Pi USB. `Ubuntu 20.10 may work `_. As a work-around, the Raspberry Pi bootloader is configured to directly boot Linux. For this to work, the Linux kernel must not be compressed. These instructions decompress the kernel and add a script to ``/etc/kernel/postinst.d`` to handle kernel upgrades. Step 1: Disk Formatting ----------------------- The commands in this step are run on the system other than the Raspberry Pi. This guide has you go to some extra work so that the stock ext4 partition can be deleted. #. Download and unpack the official image:: curl -O https://cdimage.ubuntu.com/releases/20.04.4/release/ubuntu-20.04.4-preinstalled-server-arm64+raspi.img.xz xz -d ubuntu-20.04.4-preinstalled-server-arm64+raspi.img.xz # or combine them to decompress as you download: curl https://cdimage.ubuntu.com/releases/20.04.4/release/ubuntu-20.04.4-preinstalled-server-arm64+raspi.img.xz | \ xz -d > ubuntu-20.04.4-preinstalled-server-arm64+raspi.img #. Dump the partition table for the image:: sfdisk -d ubuntu-20.04.4-preinstalled-server-arm64+raspi.img That will output this:: label: dos label-id: 0xddbefb06 device: ubuntu-20.04.4-preinstalled-server-arm64+raspi.img unit: sectors .img1 : start= 2048, size= 524288, type=c, bootable .img2 : start= 526336, size= 6285628, type=83 The important numbers are 524288 and 6285628. Store those in variables:: BOOT=524288 ROOT=6285628 #. Create a partition script:: cat > partitions << EOF label: dos unit: sectors 1 : start= 2048, size=$BOOT, type=c, bootable 2 : start=$((2048+BOOT)), size=$ROOT, type=83 3 : start=$((2048+BOOT+ROOT)), size=$ROOT, type=83 EOF #. Connect the disk: Connect the disk to a machine other than the target Raspberry Pi. If any filesystems are automatically mounted (e.g. by GNOME) unmount them. Determine the device name. For SD, the device name is almost certainly ``/dev/mmcblk0``. For USB SSDs, the device name is ``/dev/sdX``, where ``X`` is a lowercase letter. ``lsblk`` can help determine the device name. Set the ``DISK`` environment variable to the device name:: DISK=/dev/mmcblk0 # microSD card DISK=/dev/sdX # USB disk Because partitions are named differently for ``/dev/mmcblk0`` and ``/dev/sdX`` devices, set a second variable used when working with partitions:: export DISKP=${DISK}p # microSD card export DISKP=${DISK} # USB disk ($DISKP == $DISK for /dev/sdX devices) **Hint**: microSD cards connected using a USB reader also have ``/dev/sdX`` names. **WARNING**: The following steps destroy the existing data on the disk. Ensure ``DISK`` and ``DISKP`` are correct before proceeding. #. Ensure swap partitions are not in use:: swapon -v # If a partition is in use from the disk, disable it: sudo swapoff THAT_PARTITION #. Clear old ZFS labels:: sudo zpool labelclear -f ${DISK} If a ZFS label still exists from a previous system/attempt, expanding the pool will result in an unbootable system. **Hint:** If you do not already have the ZFS utilities installed, you can install them with: ``sudo apt install zfsutils-linux`` Alternatively, you can zero the entire disk with: ``sudo dd if=/dev/zero of=${DISK} bs=1M status=progress`` #. Delete existing partitions:: echo "label: dos" | sudo sfdisk ${DISK} sudo partprobe ls ${DISKP}* Make sure there are no partitions, just the file for the disk itself. This step is not strictly necessary; it exists to catch problems. #. Create the partitions:: sudo sfdisk $DISK < partitions #. Loopback mount the image:: IMG=$(sudo losetup -fP --show \ ubuntu-20.04.4-preinstalled-server-arm64+raspi.img) #. Copy the bootloader data:: sudo dd if=${IMG}p1 of=${DISKP}1 bs=1M #. Clear old label(s) from partition 2:: sudo wipefs -a ${DISKP}2 If a filesystem with the ``writable`` label from the Ubuntu image is still present in partition 2, the system will not boot initially. #. Copy the root filesystem data:: # NOTE: the destination is p3, not p2. sudo dd if=${IMG}p2 of=${DISKP}3 bs=1M status=progress conv=fsync #. Unmount the image:: sudo losetup -d $IMG #. If setting up a USB disk: Decompress the kernel:: sudo -sE MNT=$(mktemp -d /mnt/XXXXXXXX) mkdir -p $MNT/boot $MNT/root mount ${DISKP}1 $MNT/boot mount ${DISKP}3 $MNT/root zcat -qf $MNT/boot/vmlinuz >$MNT/boot/vmlinux Modify boot config:: cat >> $MNT/boot/usercfg.txt << EOF kernel=vmlinux initramfs initrd.img followkernel boot_delay EOF Create a script to automatically decompress the kernel after an upgrade:: cat >$MNT/root/etc/kernel/postinst.d/zz-decompress-kernel << 'EOF' #!/bin/sh set -eu echo "Updating decompressed kernel..." [ -e /boot/firmware/vmlinux ] && \ cp /boot/firmware/vmlinux /boot/firmware/vmlinux.bak vmlinuxtmp=$(mktemp /boot/firmware/vmlinux.XXXXXXXX) zcat -qf /boot/vmlinuz > "$vmlinuxtmp" mv "$vmlinuxtmp" /boot/firmware/vmlinux EOF chmod +x $MNT/root/etc/kernel/postinst.d/zz-decompress-kernel Cleanup:: umount $MNT/* rm -rf $MNT exit #. Boot the Raspberry Pi. Move the SD/USB disk to the Raspberry Pi. Boot it and login (e.g. via SSH) with ``ubuntu`` as the username and password. If you are using SSH, note that it takes a little bit for cloud-init to enable password logins on the first boot. Set a new password when prompted and login again using that password. If you have your local SSH configured to use ``ControlPersist``, you will have to kill the existing SSH process before logging in the second time. Step 2: Setup ZFS ----------------- #. Become root:: sudo -i #. Set the DISK and DISKP variables again:: DISK=/dev/mmcblk0 # microSD card DISKP=${DISK}p # microSD card DISK=/dev/sdX # USB disk DISKP=${DISK} # USB disk **WARNING:** Device names can change when moving a device to a different computer or switching the microSD card from a USB reader to a built-in slot. Double check the device name before continuing. #. Install ZFS:: apt update apt install pv zfs-initramfs **Note:** Since this is the first boot, you may get ``Waiting for cache lock`` because ``unattended-upgrades`` is running in the background. Wait for it to finish. #. Create the root pool: Choose one of the following options: - Unencrypted:: zpool create \ -o ashift=12 \ -O acltype=posixacl -O canmount=off -O compression=lz4 \ -O dnodesize=auto -O normalization=formD -O relatime=on \ -O xattr=sa -O mountpoint=/ -R /mnt \ rpool ${DISKP}2 **WARNING:** Encryption has not yet been tested on the Raspberry Pi. - ZFS native encryption:: zpool create \ -o ashift=12 \ -O encryption=aes-256-gcm \ -O keylocation=prompt -O keyformat=passphrase \ -O acltype=posixacl -O canmount=off -O compression=lz4 \ -O dnodesize=auto -O normalization=formD -O relatime=on \ -O xattr=sa -O mountpoint=/ -R /mnt \ rpool ${DISKP}2 - LUKS:: cryptsetup luksFormat -c aes-xts-plain64 -s 512 -h sha256 ${DISKP}2 cryptsetup luksOpen ${DISK}-part4 luks1 zpool create \ -o ashift=12 \ -O acltype=posixacl -O canmount=off -O compression=lz4 \ -O dnodesize=auto -O normalization=formD -O relatime=on \ -O xattr=sa -O mountpoint=/ -R /mnt \ rpool /dev/mapper/luks1 **Notes:** - The use of ``ashift=12`` is recommended here because many drives today have 4 KiB (or larger) physical sectors, even though they present 512 B logical sectors. Also, a future replacement drive may have 4 KiB physical sectors (in which case ``ashift=12`` is desirable) or 4 KiB logical sectors (in which case ``ashift=12`` is required). - Setting ``-O acltype=posixacl`` enables POSIX ACLs globally. If you do not want this, remove that option, but later add ``-o acltype=posixacl`` (note: lowercase “o”) to the ``zfs create`` for ``/var/log``, as `journald requires ACLs `__ Also, `disabling ACLs apparently breaks umask handling with NFSv4 `__. - Setting ``normalization=formD`` eliminates some corner cases relating to UTF-8 filename normalization. It also implies ``utf8only=on``, which means that only UTF-8 filenames are allowed. If you care to support non-UTF-8 filenames, do not use this option. For a discussion of why requiring UTF-8 filenames may be a bad idea, see `The problems with enforced UTF-8 only filenames `__. - ``recordsize`` is unset (leaving it at the default of 128 KiB). If you want to tune it (e.g. ``-O recordsize=1M``), see `these `__ `various `__ `blog `__ `posts `__. - Setting ``relatime=on`` is a middle ground between classic POSIX ``atime`` behavior (with its significant performance impact) and ``atime=off`` (which provides the best performance by completely disabling atime updates). Since Linux 2.6.30, ``relatime`` has been the default for other filesystems. See `RedHat’s documentation `__ for further information. - Setting ``xattr=sa`` `vastly improves the performance of extended attributes `__. Inside ZFS, extended attributes are used to implement POSIX ACLs. Extended attributes can also be used by user-space applications. `They are used by some desktop GUI applications. `__ `They can be used by Samba to store Windows ACLs and DOS attributes; they are required for a Samba Active Directory domain controller. `__ Note that ``xattr=sa`` is `Linux-specific `__. If you move your ``xattr=sa`` pool to another OpenZFS implementation besides ZFS-on-Linux, extended attributes will not be readable (though your data will be). If portability of extended attributes is important to you, omit the ``-O xattr=sa`` above. Even if you do not want ``xattr=sa`` for the whole pool, it is probably fine to use it for ``/var/log``. - Make sure to include the ``-part4`` portion of the drive path. If you forget that, you are specifying the whole disk, which ZFS will then re-partition, and you will lose the bootloader partition(s). - ZFS native encryption defaults to ``aes-256-ccm``, but `the default has changed upstream `__ to ``aes-256-gcm``. `AES-GCM seems to be generally preferred over AES-CCM `__, `is faster now `__, and `will be even faster in the future `__. - For LUKS, the key size chosen is 512 bits. However, XTS mode requires two keys, so the LUKS key is split in half. Thus, ``-s 512`` means AES-256. - Your passphrase will likely be the weakest link. Choose wisely. See `section 5 of the cryptsetup FAQ `__ for guidance. Step 3: System Installation --------------------------- #. Create a filesystem dataset to act as a container:: zfs create -o canmount=off -o mountpoint=none rpool/ROOT #. Create a filesystem dataset for the root filesystem:: UUID=$(dd if=/dev/urandom bs=1 count=100 2>/dev/null | tr -dc 'a-z0-9' | cut -c-6) zfs create -o canmount=noauto -o mountpoint=/ \ -o com.ubuntu.zsys:bootfs=yes \ -o com.ubuntu.zsys:last-used=$(date +%s) rpool/ROOT/ubuntu_$UUID zfs mount rpool/ROOT/ubuntu_$UUID With ZFS, it is not normally necessary to use a mount command (either ``mount`` or ``zfs mount``). This situation is an exception because of ``canmount=noauto``. #. Create datasets:: zfs create -o com.ubuntu.zsys:bootfs=no \ rpool/ROOT/ubuntu_$UUID/srv zfs create -o com.ubuntu.zsys:bootfs=no -o canmount=off \ rpool/ROOT/ubuntu_$UUID/usr zfs create rpool/ROOT/ubuntu_$UUID/usr/local zfs create -o com.ubuntu.zsys:bootfs=no -o canmount=off \ rpool/ROOT/ubuntu_$UUID/var zfs create rpool/ROOT/ubuntu_$UUID/var/games zfs create rpool/ROOT/ubuntu_$UUID/var/lib zfs create rpool/ROOT/ubuntu_$UUID/var/lib/AccountsService zfs create rpool/ROOT/ubuntu_$UUID/var/lib/apt zfs create rpool/ROOT/ubuntu_$UUID/var/lib/dpkg zfs create rpool/ROOT/ubuntu_$UUID/var/lib/NetworkManager zfs create rpool/ROOT/ubuntu_$UUID/var/log zfs create rpool/ROOT/ubuntu_$UUID/var/mail zfs create rpool/ROOT/ubuntu_$UUID/var/snap zfs create rpool/ROOT/ubuntu_$UUID/var/spool zfs create rpool/ROOT/ubuntu_$UUID/var/www zfs create -o canmount=off -o mountpoint=/ \ rpool/USERDATA zfs create -o com.ubuntu.zsys:bootfs-datasets=rpool/ROOT/ubuntu_$UUID \ -o canmount=on -o mountpoint=/root \ rpool/USERDATA/root_$UUID If you want a separate dataset for ``/tmp``:: zfs create -o com.ubuntu.zsys:bootfs=no \ rpool/ROOT/ubuntu_$UUID/tmp chmod 1777 /mnt/tmp The primary goal of this dataset layout is to separate the OS from user data. This allows the root filesystem to be rolled back without rolling back user data. If you do nothing extra, ``/tmp`` will be stored as part of the root filesystem. Alternatively, you can create a separate dataset for ``/tmp``, as shown above. This keeps the ``/tmp`` data out of snapshots of your root filesystem. It also allows you to set a quota on ``rpool/tmp``, if you want to limit the maximum space used. Otherwise, you can use a tmpfs (RAM filesystem) later. #. Optional: Ignore synchronous requests: microSD cards are relatively slow. If you want to increase performance (especially when installing packages) at the cost of some safety, you can disable flushing of synchronous requests (e.g. ``fsync()``, ``O_[D]SYNC``): Choose one of the following options: - For the root filesystem, but not user data:: zfs set sync=disabled rpool/ROOT - For everything:: zfs set sync=disabled rpool ZFS is transactional, so it will still be crash consistent. However, you should leave ``sync`` at its default of ``standard`` if this system needs to guarantee persistence (e.g. if it is a database or NFS server). #. Copy the system into the ZFS filesystems:: (cd /; tar -cf - --one-file-system --warning=no-file-ignored .) | \ pv -p -bs $(du -sxm --apparent-size / | cut -f1)m | \ (cd /mnt ; tar -x) Step 4: System Configuration ---------------------------- #. Configure the hostname: Replace ``HOSTNAME`` with the desired hostname:: hostname HOSTNAME hostname > /mnt/etc/hostname vi /mnt/etc/hosts .. code-block:: text Add a line: 127.0.1.1 HOSTNAME or if the system has a real name in DNS: 127.0.1.1 FQDN HOSTNAME **Hint:** Use ``nano`` if you find ``vi`` confusing. #. Stop ``zed``:: systemctl stop zed #. Bind the virtual filesystems from the running environment to the new ZFS environment and ``chroot`` into it:: mount --make-private --rbind /boot/firmware /mnt/boot/firmware mount --make-private --rbind /dev /mnt/dev mount --make-private --rbind /proc /mnt/proc mount --make-private --rbind /run /mnt/run mount --make-private --rbind /sys /mnt/sys chroot /mnt /usr/bin/env DISK=$DISK UUID=$UUID bash --login #. Configure a basic system environment:: apt update Even if you prefer a non-English system language, always ensure that ``en_US.UTF-8`` is available:: dpkg-reconfigure locales dpkg-reconfigure tzdata #. For LUKS installs only, setup ``/etc/crypttab``:: # cryptsetup is already installed, but this marks it as manually # installed so it is not automatically removed. apt install --yes cryptsetup echo luks1 UUID=$(blkid -s UUID -o value ${DISK}-part4) none \ luks,discard,initramfs > /etc/crypttab The use of ``initramfs`` is a work-around for `cryptsetup does not support ZFS `__. #. Optional: Mount a tmpfs to ``/tmp`` If you chose to create a ``/tmp`` dataset above, skip this step, as they are mutually exclusive choices. Otherwise, you can put ``/tmp`` on a tmpfs (RAM filesystem) by enabling the ``tmp.mount`` unit. :: cp /usr/share/systemd/tmp.mount /etc/systemd/system/ systemctl enable tmp.mount #. Setup system groups:: addgroup --system lpadmin addgroup --system sambashare #. Patch a dependency loop: For ZFS native encryption or LUKS:: apt install --yes curl patch curl https://launchpadlibrarian.net/478315221/2150-fix-systemd-dependency-loops.patch | \ sed "s|/etc|/lib|;s|\.in$||" | (cd / ; patch -p1) Ignore the failure in Hunk #2 (say ``n`` twice). This patch is from `Bug #1875577 Encrypted swap won't load on 20.04 with zfs root `__. #. Fix filesystem mount ordering: We need to activate ``zfs-mount-generator``. This makes systemd aware of the separate mountpoints, which is important for things like ``/var/log`` and ``/var/tmp``. In turn, ``rsyslog.service`` depends on ``var-log.mount`` by way of ``local-fs.target`` and services using the ``PrivateTmp`` feature of systemd automatically use ``After=var-tmp.mount``. :: mkdir /etc/zfs/zfs-list.cache touch /etc/zfs/zfs-list.cache/rpool ln -s /usr/lib/zfs-linux/zed.d/history_event-zfs-list-cacher.sh /etc/zfs/zed.d zed -F & Force a cache update:: zfs set canmount=noauto rpool/ROOT/ubuntu_$UUID Verify that ``zed`` updated the cache by making sure this is not empty, which will take a few seconds:: cat /etc/zfs/zfs-list.cache/rpool Stop ``zed``:: fg Press Ctrl-C. Fix the paths to eliminate ``/mnt``:: sed -Ei "s|/mnt/?|/|" /etc/zfs/zfs-list.cache/* #. Remove old filesystem from ``/etc/fstab``:: vi /etc/fstab # Remove the old root filesystem line: # LABEL=writable / ext4 ... #. Configure kernel command line:: cp /boot/firmware/cmdline.txt /boot/firmware/cmdline.txt.bak sed -i "s|root=LABEL=writable rootfstype=ext4|root=ZFS=rpool/ROOT/ubuntu_$UUID|" \ /boot/firmware/cmdline.txt sed -i "s| fixrtc||" /boot/firmware/cmdline.txt sed -i "s|$| init_on_alloc=0|" /boot/firmware/cmdline.txt The ``fixrtc`` script is not compatible with ZFS and will cause the boot to hang for 180 seconds. The ``init_on_alloc=0`` is to address `performance regressions `__. #. Optional (but highly recommended): Make debugging booting easier:: sed -i "s|$| nosplash|" /boot/firmware/cmdline.txt #. Reboot:: exit reboot Wait for the newly installed system to boot normally. Login as ``ubuntu``. Step 5: First Boot ------------------ #. Become root:: sudo -i #. Set the DISK variable again:: DISK=/dev/mmcblk0 # microSD card DISK=/dev/sdX # USB disk #. Delete the ext4 partition and expand the ZFS partition:: sfdisk $DISK --delete 3 echo ", +" | sfdisk --no-reread -N 2 $DISK **Note:** This does not automatically expand the pool. That will be happen on reboot. #. Create a user account: Replace ``YOUR_USERNAME`` with your desired username:: username=YOUR_USERNAME UUID=$(dd if=/dev/urandom bs=1 count=100 2>/dev/null | tr -dc 'a-z0-9' | cut -c-6) ROOT_DS=$(zfs list -o name | awk '/ROOT\/ubuntu_/{print $1;exit}') zfs create -o com.ubuntu.zsys:bootfs-datasets=$ROOT_DS \ -o canmount=on -o mountpoint=/home/$username \ rpool/USERDATA/${username}_$UUID adduser $username cp -a /etc/skel/. /home/$username chown -R $username:$username /home/$username usermod -a -G adm,cdrom,dip,lpadmin,lxd,plugdev,sambashare,sudo $username #. Reboot:: reboot Wait for the system to boot normally. Login using the account you created. #. Become root:: sudo -i #. Expand the ZFS pool: Verify the pool expanded:: zfs list rpool If it did not automatically expand, try to expand it manually:: DISK=/dev/mmcblk0 # microSD card DISKP=${DISK}p # microSD card DISK=/dev/sdX # USB disk DISKP=${DISK} # USB disk zpool online -e rpool ${DISKP}2 #. Delete the ``ubuntu`` user:: deluser --remove-home ubuntu Step 6: Full Software Installation ---------------------------------- #. Optional: Remove cloud-init:: vi /etc/netplan/01-netcfg.yaml .. code-block:: yaml network: version: 2 ethernets: eth0: dhcp4: true :: rm /etc/netplan/50-cloud-init.yaml apt purge --autoremove ^cloud-init rm -rf /etc/cloud #. Optional: Remove other storage packages:: apt purge --autoremove bcache-tools btrfs-progs cloud-guest-utils lvm2 \ mdadm multipath-tools open-iscsi overlayroot xfsprogs #. Upgrade the minimal system:: apt dist-upgrade --yes #. Optional: Install a full GUI environment:: apt install --yes ubuntu-desktop echo dtoverlay=vc4-fkms-v3d >> /boot/firmware/usercfg.txt **Hint**: If you are installing a full GUI environment, you will likely want to remove cloud-init as discussed above but manage your network with NetworkManager:: rm /etc/netplan/*.yaml vi /etc/netplan/01-network-manager-all.yaml .. code-block:: yaml network: version: 2 renderer: NetworkManager #. Optional (but recommended): Disable log compression: As ``/var/log`` is already compressed by ZFS, logrotate’s compression is going to burn CPU and disk I/O for (in most cases) very little gain. Also, if you are making snapshots of ``/var/log``, logrotate’s compression will actually waste space, as the uncompressed data will live on in the snapshot. You can edit the files in ``/etc/logrotate.d`` by hand to comment out ``compress``, or use this loop (copy-and-paste highly recommended):: for file in /etc/logrotate.d/* ; do if grep -Eq "(^|[^#y])compress" "$file" ; then sed -i -r "s/(^|[^#y])(compress)/\1#\2/" "$file" fi done #. Reboot:: reboot Step 7: Final Cleanup --------------------- #. Wait for the system to boot normally. Login using the account you created. Ensure the system (including networking) works normally. #. Optional: For LUKS installs only, backup the LUKS header:: sudo cryptsetup luksHeaderBackup /dev/disk/by-id/scsi-SATA_disk1-part4 \ --header-backup-file luks1-header.dat Store that backup somewhere safe (e.g. cloud storage). It is protected by your LUKS passphrase, but you may wish to use additional encryption. **Hint:** If you created a mirror or raidz topology, repeat this for each LUKS volume (``luks2``, etc.). .. highlight:: sh Ubuntu 22.04 Root on ZFS ======================== .. contents:: Table of Contents :local: Overview -------- Ubuntu Installer ~~~~~~~~~~~~~~~~ The Ubuntu installer still has ZFS support, but `it was almost removed for 22.04 `__ and `it no longer installs zsys `__. At the moment, this HOWTO still uses zsys, but that will be probably be removed in the near future. Raspberry Pi ~~~~~~~~~~~~ If you are looking to install on a Raspberry Pi, see :doc:`Ubuntu 22.04 Root on ZFS for Raspberry Pi`. Caution ~~~~~~~ - This HOWTO uses a whole physical disk. - Do not use these instructions for dual-booting. - Backup your data. Any existing data will be lost. System Requirements ~~~~~~~~~~~~~~~~~~~ - `Ubuntu 22.04.1 (“jammy”) Desktop CD `__ (*not* any server images) - Installing on a drive which presents 4 KiB logical sectors (a “4Kn” drive) only works with UEFI booting. This not unique to ZFS. `GRUB does not and will not work on 4Kn with legacy (BIOS) booting. `__ Computers that have less than 2 GiB of memory run ZFS slowly. 4 GiB of memory is recommended for normal performance in basic workloads. If you wish to use deduplication, you will need `massive amounts of RAM `__. Enabling deduplication is a permanent change that cannot be easily reverted. Support ~~~~~~~ If you need help, reach out to the community using the :ref:`mailing_lists` or IRC at `#zfsonlinux `__ on `Libera Chat `__. If you have a bug report or feature request related to this HOWTO, please `file a new issue and mention @rlaager `__. Contributing ~~~~~~~~~~~~ #. Fork and clone: https://github.com/openzfs/openzfs-docs #. Install the tools:: sudo apt install python3-pip pip3 install -r docs/requirements.txt # Add ~/.local/bin to your $PATH, e.g. by adding this to ~/.bashrc: PATH=$HOME/.local/bin:$PATH #. Make your changes. #. Test:: cd docs make html sensible-browser _build/html/index.html #. ``git commit --signoff`` to a branch, ``git push``, and create a pull request. Mention @rlaager. Encryption ~~~~~~~~~~ This guide supports three different encryption options: unencrypted, ZFS native encryption, and LUKS. With any option, all ZFS features are fully available. Unencrypted does not encrypt anything, of course. With no encryption happening, this option naturally has the best performance. ZFS native encryption encrypts the data and most metadata in the root pool. It does not encrypt dataset or snapshot names or properties. The boot pool is not encrypted at all, but it only contains the bootloader, kernel, and initrd. (Unless you put a password in ``/etc/fstab``, the initrd is unlikely to contain sensitive data.) The system cannot boot without the passphrase being entered at the console. Performance is good. As the encryption happens in ZFS, even if multiple disks (mirror or raidz topologies) are used, the data only has to be encrypted once. LUKS encrypts almost everything. The only unencrypted data is the bootloader, kernel, and initrd. The system cannot boot without the passphrase being entered at the console. Performance is good, but LUKS sits underneath ZFS, so if multiple disks (mirror or raidz topologies) are used, the data has to be encrypted once per disk. Step 1: Prepare The Install Environment --------------------------------------- #. Boot the Ubuntu Live CD. From the GRUB boot menu, select *Try or Install Ubuntu*. On the *Welcome* page, select your preferred language and *Try Ubuntu*. Connect your system to the Internet as appropriate (e.g. join your WiFi network). Open a terminal (press Ctrl-Alt-T). #. Setup and update the repositories:: sudo apt update #. Optional: Install and start the OpenSSH server in the Live CD environment: If you have a second system, using SSH to access the target system can be convenient:: passwd # There is no current password. sudo apt install --yes openssh-server vim Installing the full ``vim`` package fixes terminal problems that occur when using the ``vim-tiny`` package (that ships in the Live CD environment) over SSH. **Hint:** You can find your IP address with ``ip addr show scope global | grep inet``. Then, from your main machine, connect with ``ssh ubuntu@IP``. #. Disable automounting: If the disk has been used before (with partitions at the same offsets), previous filesystems (e.g. the ESP) will automount if not disabled:: gsettings set org.gnome.desktop.media-handling automount false #. Become root:: sudo -i #. Install ZFS in the Live CD environment:: apt install --yes debootstrap gdisk zfsutils-linux systemctl stop zed Step 2: Disk Formatting ----------------------- #. Set a variable with the disk name:: DISK=/dev/disk/by-id/scsi-SATA_disk1 Always use the long ``/dev/disk/by-id/*`` aliases with ZFS. Using the ``/dev/sd*`` device nodes directly can cause sporadic import failures, especially on systems that have more than one storage pool. **Hints:** - ``ls -la /dev/disk/by-id`` will list the aliases. - Are you doing this in a virtual machine? If your virtual disk is missing from ``/dev/disk/by-id``, use ``/dev/vda`` if you are using KVM with virtio; otherwise, read the `troubleshooting <#troubleshooting>`__ section. - For a mirror or raidz topology, use ``DISK1``, ``DISK2``, etc. - When choosing a boot pool size, consider how you will use the space. A kernel and initrd may consume around 100M. If you have multiple kernels and take snapshots, you may find yourself low on boot pool space, especially if you need to regenerate your initramfs images, which may be around 85M each. Size your boot pool appropriately for your needs. #. If you are re-using a disk, clear it as necessary: Ensure swap partitions are not in use:: swapoff --all If the disk was previously used in an MD array:: apt install --yes mdadm # See if one or more MD arrays are active: cat /proc/mdstat # If so, stop them (replace ``md0`` as required): mdadm --stop /dev/md0 # For an array using the whole disk: mdadm --zero-superblock --force $DISK # For an array using a partition (e.g. a swap partition per this HOWTO): mdadm --zero-superblock --force ${DISK}-part2 If the disk was previously used with zfs:: wipefs -a $DISK For flash-based storage, if the disk was previously used, you may wish to do a full-disk discard (TRIM/UNMAP), which can improve performance:: blkdiscard -f $DISK Clear the partition table:: sgdisk --zap-all $DISK If you get a message about the kernel still using the old partition table, you can request the kernel reload the partition information using:: partprobe $DISK If the new partitions still don't show up, you can reboot and start over (except that you can skip this step). #. Create bootloader partition(s):: sgdisk -n1:1M:+512M -t1:EF00 $DISK # For legacy (BIOS) booting: sgdisk -a1 -n5:24K:+1000K -t5:EF02 $DISK **Note:** While the Ubuntu installer uses an MBR label for legacy (BIOS) booting, this HOWTO uses GPT partition labels for both UEFI and legacy (BIOS) booting. This is simpler than having two options. It is also provides forward compatibility (future proofing). In other words, for legacy (BIOS) booting, this will allow you to move the disk(s) to a new system/motherboard in the future without having to rebuild the pool (and restore your data from a backup). The ESP is created in both cases for similar reasons. Additionally, the ESP is used for ``/boot/grub`` in single-disk installs, as :ref:`discussed below `. #. Create a partition for swap: Previous versions of this HOWTO put swap on a zvol. `Ubuntu recommends against this configuration due to deadlocks. `__ There is `a bug report upstream `__. Putting swap on a partition gives up the benefit of ZFS checksums (for your swap). That is probably the right trade-off given the reports of ZFS deadlocks with swap. If you are bothered by this, simply do not enable swap. Choose one of the following options if you want swap: - For a single-disk install:: sgdisk -n2:0:+500M -t2:8200 $DISK - For a mirror or raidz topology:: sgdisk -n2:0:+500M -t2:FD00 $DISK Adjust the swap swize to your needs. If you wish to enable hiberation (which only works for unencrypted installs), the swap partition must be at least as large as the system's RAM. #. Create a boot pool partition:: sgdisk -n3:0:+2G -t3:BE00 $DISK The Ubuntu installer uses 5% of the disk space constrained to a minimum of 500 MiB and a maximum of 2 GiB. `Making this too small (and 500 MiB might be too small) can result in an inability to upgrade the kernel. `__ #. Create a root pool partition: Choose one of the following options: - Unencrypted or ZFS native encryption:: sgdisk -n4:0:0 -t4:BF00 $DISK - LUKS:: sgdisk -n4:0:0 -t4:8309 $DISK If you are creating a mirror or raidz topology, repeat the partitioning commands for all the disks which will be part of the pool. #. Create the boot pool:: zpool create \ -o ashift=12 \ -o autotrim=on \ -o cachefile=/etc/zfs/zpool.cache \ -o compatibility=grub2 \ -o feature@livelist=enabled \ -o feature@zpool_checkpoint=enabled \ -O devices=off \ -O acltype=posixacl -O xattr=sa \ -O compression=lz4 \ -O normalization=formD \ -O relatime=on \ -O canmount=off -O mountpoint=/boot -R /mnt \ bpool ${DISK}-part3 You should not need to customize any of the options for the boot pool. Ignore the warnings about the features “not in specified 'compatibility' feature set.” GRUB does not support all of the zpool features. See ``spa_feature_names`` in `grub-core/fs/zfs/zfs.c `__. This step creates a separate boot pool for ``/boot`` with the features limited to only those that GRUB supports, allowing the root pool to use any/all features. Note that GRUB opens the pool read-only, so all read-only compatible features are “supported” by GRUB. **Hints:** - If you are creating a mirror topology, create the pool using:: zpool create \ ... \ bpool mirror \ /dev/disk/by-id/scsi-SATA_disk1-part3 \ /dev/disk/by-id/scsi-SATA_disk2-part3 - For raidz topologies, replace ``mirror`` in the above command with ``raidz``, ``raidz2``, or ``raidz3`` and list the partitions from the additional disks. - The boot pool name is no longer arbitrary. It _must_ be ``bpool``. If you really want to rename it, edit ``/etc/grub.d/10_linux_zfs`` later, after GRUB is installed (and run ``update-grub``). **Feature Notes:** - The ``allocation_classes`` feature should be safe to use. However, unless one is using it (i.e. a ``special`` vdev), there is no point to enabling it. It is extremely unlikely that someone would use this feature for a boot pool. If one cares about speeding up the boot pool, it would make more sense to put the whole pool on the faster disk rather than using it as a ``special`` vdev. - The ``device_rebuild`` feature should be safe to use (except on raidz, which it is incompatible with), but the boot pool is small, so this does not matter in practice. - The ``log_spacemap`` and ``spacemap_v2`` features have been tested and are safe to use. The boot pool is small, so these do not matter in practice. - The ``project_quota`` feature has been tested and is safe to use. This feature is extremely unlikely to matter for the boot pool. - The ``resilver_defer`` should be safe but the boot pool is small enough that it is unlikely to be necessary. - As a read-only compatible feature, the ``userobj_accounting`` feature should be compatible in theory, but in practice, GRUB can fail with an “invalid dnode type” error. This feature does not matter for ``/boot`` anyway. #. Create the root pool: Choose one of the following options: - Unencrypted:: zpool create \ -o ashift=12 \ -o autotrim=on \ -O acltype=posixacl -O xattr=sa -O dnodesize=auto \ -O compression=lz4 \ -O normalization=formD \ -O relatime=on \ -O canmount=off -O mountpoint=/ -R /mnt \ rpool ${DISK}-part4 - ZFS native encryption:: zpool create \ -o ashift=12 \ -o autotrim=on \ -O encryption=on -O keylocation=prompt -O keyformat=passphrase \ -O acltype=posixacl -O xattr=sa -O dnodesize=auto \ -O compression=lz4 \ -O normalization=formD \ -O relatime=on \ -O canmount=off -O mountpoint=/ -R /mnt \ rpool ${DISK}-part4 - LUKS:: cryptsetup luksFormat -c aes-xts-plain64 -s 512 -h sha256 ${DISK}-part4 cryptsetup luksOpen ${DISK}-part4 luks1 zpool create \ -o ashift=12 \ -o autotrim=on \ -O acltype=posixacl -O xattr=sa -O dnodesize=auto \ -O compression=lz4 \ -O normalization=formD \ -O relatime=on \ -O canmount=off -O mountpoint=/ -R /mnt \ rpool /dev/mapper/luks1 **Notes:** - The use of ``ashift=12`` is recommended here because many drives today have 4 KiB (or larger) physical sectors, even though they present 512 B logical sectors. Also, a future replacement drive may have 4 KiB physical sectors (in which case ``ashift=12`` is desirable) or 4 KiB logical sectors (in which case ``ashift=12`` is required). - Setting ``-O acltype=posixacl`` enables POSIX ACLs globally. If you do not want this, remove that option, but later add ``-o acltype=posixacl`` (note: lowercase “o”) to the ``zfs create`` for ``/var/log``, as `journald requires ACLs `__ Also, `disabling ACLs apparently breaks umask handling with NFSv4 `__. - Setting ``xattr=sa`` `vastly improves the performance of extended attributes `__. Inside ZFS, extended attributes are used to implement POSIX ACLs. Extended attributes can also be used by user-space applications. `They are used by some desktop GUI applications. `__ `They can be used by Samba to store Windows ACLs and DOS attributes; they are required for a Samba Active Directory domain controller. `__ Note that ``xattr=sa`` is `Linux-specific `__. If you move your ``xattr=sa`` pool to another OpenZFS implementation besides ZFS-on-Linux, extended attributes will not be readable (though your data will be). If portability of extended attributes is important to you, omit the ``-O xattr=sa`` above. Even if you do not want ``xattr=sa`` for the whole pool, it is probably fine to use it for ``/var/log``. - Setting ``normalization=formD`` eliminates some corner cases relating to UTF-8 filename normalization. It also implies ``utf8only=on``, which means that only UTF-8 filenames are allowed. If you care to support non-UTF-8 filenames, do not use this option. For a discussion of why requiring UTF-8 filenames may be a bad idea, see `The problems with enforced UTF-8 only filenames `__. - ``recordsize`` is unset (leaving it at the default of 128 KiB). If you want to tune it (e.g. ``-O recordsize=1M``), see `these `__ `various `__ `blog `__ `posts `__. - Setting ``relatime=on`` is a middle ground between classic POSIX ``atime`` behavior (with its significant performance impact) and ``atime=off`` (which provides the best performance by completely disabling atime updates). Since Linux 2.6.30, ``relatime`` has been the default for other filesystems. See `RedHat’s documentation `__ for further information. - Make sure to include the ``-part4`` portion of the drive path. If you forget that, you are specifying the whole disk, which ZFS will then re-partition, and you will lose the bootloader partition(s). - ZFS native encryption `now `__ defaults to ``aes-256-gcm``. - For LUKS, the key size chosen is 512 bits. However, XTS mode requires two keys, so the LUKS key is split in half. Thus, ``-s 512`` means AES-256. - Your passphrase will likely be the weakest link. Choose wisely. See `section 5 of the cryptsetup FAQ `__ for guidance. **Hints:** - If you are creating a mirror topology, create the pool using:: zpool create \ ... \ rpool mirror \ /dev/disk/by-id/scsi-SATA_disk1-part4 \ /dev/disk/by-id/scsi-SATA_disk2-part4 - For raidz topologies, replace ``mirror`` in the above command with ``raidz``, ``raidz2``, or ``raidz3`` and list the partitions from the additional disks. - When using LUKS with mirror or raidz topologies, use ``/dev/mapper/luks1``, ``/dev/mapper/luks2``, etc., which you will have to create using ``cryptsetup``. - The pool name is arbitrary. If changed, the new name must be used consistently. On systems that can automatically install to ZFS, the root pool is named ``rpool`` by default. Step 3: System Installation --------------------------- #. Create filesystem datasets to act as containers:: zfs create -o canmount=off -o mountpoint=none rpool/ROOT zfs create -o canmount=off -o mountpoint=none bpool/BOOT #. Create filesystem datasets for the root and boot filesystems:: UUID=$(dd if=/dev/urandom bs=1 count=100 2>/dev/null | tr -dc 'a-z0-9' | cut -c-6) zfs create -o mountpoint=/ \ -o com.ubuntu.zsys:bootfs=yes \ -o com.ubuntu.zsys:last-used=$(date +%s) rpool/ROOT/ubuntu_$UUID zfs create -o mountpoint=/boot bpool/BOOT/ubuntu_$UUID #. Create datasets:: zfs create -o com.ubuntu.zsys:bootfs=no -o canmount=off \ rpool/ROOT/ubuntu_$UUID/usr zfs create -o com.ubuntu.zsys:bootfs=no -o canmount=off \ rpool/ROOT/ubuntu_$UUID/var zfs create rpool/ROOT/ubuntu_$UUID/var/lib zfs create rpool/ROOT/ubuntu_$UUID/var/log zfs create rpool/ROOT/ubuntu_$UUID/var/spool zfs create -o canmount=off -o mountpoint=/ \ rpool/USERDATA zfs create -o com.ubuntu.zsys:bootfs-datasets=rpool/ROOT/ubuntu_$UUID \ -o canmount=on -o mountpoint=/root \ rpool/USERDATA/root_$UUID chmod 700 /mnt/root The datasets below are optional, depending on your preferences and/or software choices. If you wish to separate these to exclude them from snapshots:: zfs create rpool/ROOT/ubuntu_$UUID/var/cache zfs create rpool/ROOT/ubuntu_$UUID/var/lib/nfs zfs create rpool/ROOT/ubuntu_$UUID/var/tmp chmod 1777 /mnt/var/tmp If desired (the Ubuntu installer creates these):: zfs create rpool/ROOT/ubuntu_$UUID/var/lib/apt zfs create rpool/ROOT/ubuntu_$UUID/var/lib/dpkg If you use /srv on this system:: zfs create -o com.ubuntu.zsys:bootfs=no \ rpool/ROOT/ubuntu_$UUID/srv If you use /usr/local on this system:: zfs create rpool/ROOT/ubuntu_$UUID/usr/local If this system will have games installed:: zfs create rpool/ROOT/ubuntu_$UUID/var/games If this system will have a GUI:: zfs create rpool/ROOT/ubuntu_$UUID/var/lib/AccountsService zfs create rpool/ROOT/ubuntu_$UUID/var/lib/NetworkManager If this system will use Docker (which manages its own datasets & snapshots):: zfs create rpool/ROOT/ubuntu_$UUID/var/lib/docker If this system will store local email in /var/mail:: zfs create rpool/ROOT/ubuntu_$UUID/var/mail If this system will use Snap packages:: zfs create rpool/ROOT/ubuntu_$UUID/var/snap If you use /var/www on this system:: zfs create rpool/ROOT/ubuntu_$UUID/var/www For a mirror or raidz topology, create a dataset for ``/boot/grub``:: zfs create -o com.ubuntu.zsys:bootfs=no bpool/grub A tmpfs is recommended later, but if you want a separate dataset for ``/tmp``:: zfs create -o com.ubuntu.zsys:bootfs=no \ rpool/ROOT/ubuntu_$UUID/tmp chmod 1777 /mnt/tmp The primary goal of this dataset layout is to separate the OS from user data. This allows the root filesystem to be rolled back without rolling back user data. If you do nothing extra, ``/tmp`` will be stored as part of the root filesystem. Alternatively, you can create a separate dataset for ``/tmp``, as shown above. This keeps the ``/tmp`` data out of snapshots of your root filesystem. It also allows you to set a quota on ``rpool/tmp``, if you want to limit the maximum space used. Otherwise, you can use a tmpfs (RAM filesystem) later. **Note:** If you separate a directory required for booting (e.g. ``/etc``) into its own dataset, you must add it to ``ZFS_INITRD_ADDITIONAL_DATASETS`` in ``/etc/default/zfs``. Datasets with ``canmount=off`` (like ``rpool/usr`` above) do not matter for this. #. Mount a tmpfs at /run:: mkdir /mnt/run mount -t tmpfs tmpfs /mnt/run mkdir /mnt/run/lock #. Install the minimal system:: debootstrap jammy /mnt The ``debootstrap`` command leaves the new system in an unconfigured state. An alternative to using ``debootstrap`` is to copy the entirety of a working system into the new ZFS root. #. Copy in zpool.cache:: mkdir /mnt/etc/zfs cp /etc/zfs/zpool.cache /mnt/etc/zfs/ Step 4: System Configuration ---------------------------- #. Configure the hostname: Replace ``HOSTNAME`` with the desired hostname:: hostname HOSTNAME hostname > /mnt/etc/hostname vi /mnt/etc/hosts .. code-block:: text Add a line: 127.0.1.1 HOSTNAME or if the system has a real name in DNS: 127.0.1.1 FQDN HOSTNAME **Hint:** Use ``nano`` if you find ``vi`` confusing. #. Configure the network interface: Find the interface name:: ip addr show Adjust ``NAME`` below to match your interface name:: vi /mnt/etc/netplan/01-netcfg.yaml .. code-block:: yaml network: version: 2 ethernets: NAME: dhcp4: true Customize this file if the system is not a DHCP client. #. Configure the package sources:: vi /mnt/etc/apt/sources.list .. code-block:: sourceslist deb http://archive.ubuntu.com/ubuntu jammy main restricted universe multiverse deb http://archive.ubuntu.com/ubuntu jammy-updates main restricted universe multiverse deb http://archive.ubuntu.com/ubuntu jammy-backports main restricted universe multiverse deb http://security.ubuntu.com/ubuntu jammy-security main restricted universe multiverse #. Bind the virtual filesystems from the LiveCD environment to the new system and ``chroot`` into it:: mount --make-private --rbind /dev /mnt/dev mount --make-private --rbind /proc /mnt/proc mount --make-private --rbind /sys /mnt/sys chroot /mnt /usr/bin/env DISK=$DISK UUID=$UUID bash --login **Note:** This is using ``--rbind``, not ``--bind``. #. Configure a basic system environment:: apt update Even if you prefer a non-English system language, always ensure that ``en_US.UTF-8`` is available:: dpkg-reconfigure locales tzdata keyboard-configuration console-setup Install your preferred text editor:: apt install --yes nano apt install --yes vim Installing the full ``vim`` package fixes terminal problems that occur when using the ``vim-tiny`` package (that is installed by ``debootstrap``) over SSH. #. For LUKS installs only, setup ``/etc/crypttab``:: apt install --yes cryptsetup echo luks1 /dev/disk/by-uuid/$(blkid -s UUID -o value ${DISK}-part4) \ none luks,discard,initramfs > /etc/crypttab The use of ``initramfs`` is a work-around for `cryptsetup does not support ZFS `__. **Hint:** If you are creating a mirror or raidz topology, repeat the ``/etc/crypttab`` entries for ``luks2``, etc. adjusting for each disk. #. Create the EFI filesystem: Perform these steps for both UEFI and legacy (BIOS) booting:: apt install --yes dosfstools mkdosfs -F 32 -s 1 -n EFI ${DISK}-part1 mkdir /boot/efi echo /dev/disk/by-uuid/$(blkid -s UUID -o value ${DISK}-part1) \ /boot/efi vfat defaults 0 0 >> /etc/fstab mount /boot/efi For a mirror or raidz topology, repeat the `mkdosfs` for the additional disks, but do not repeat the other commands. **Note:** The ``-s 1`` for ``mkdosfs`` is only necessary for drives which present 4 KiB logical sectors (“4Kn” drives) to meet the minimum cluster size (given the partition size of 512 MiB) for FAT32. It also works fine on drives which present 512 B sectors. #. Put ``/boot/grub`` on the EFI System Partition: .. _boot-grub-esp: For a single-disk install only:: mkdir /boot/efi/grub /boot/grub echo /boot/efi/grub /boot/grub none defaults,bind 0 0 >> /etc/fstab mount /boot/grub This allows GRUB to write to ``/boot/grub`` (since it is on a FAT-formatted ESP instead of on ZFS), which means that ``/boot/grub/grubenv`` and the ``recordfail`` feature works as expected: if the boot fails, the normally hidden GRUB menu will be shown on the next boot. For a mirror or raidz topology, we do not want GRUB writing to the EFI System Partition. This is because we duplicate it at install without a mechanism to update the copies when the GRUB configuration changes (e.g. as the kernel is upgraded). Thus, we keep ``/boot/grub`` on the boot pool for the mirror or raidz topologies. This preserves correct mirroring/raidz behavior, at the expense of being able to write to ``/boot/grub/grubenv`` and thus the ``recordfail`` behavior. #. Install GRUB/Linux/ZFS in the chroot environment for the new system: Choose one of the following options: - Install GRUB/Linux/ZFS for legacy (BIOS) booting:: apt install --yes grub-pc linux-image-generic zfs-initramfs zsys Select (using the space bar) all of the disks (not partitions) in your pool. - Install GRUB/Linux/ZFS for UEFI booting:: apt install --yes \ grub-efi-amd64 grub-efi-amd64-signed linux-image-generic \ shim-signed zfs-initramfs zsys **Notes:** - Ignore any error messages saying ``ERROR: Couldn't resolve device`` and ``WARNING: Couldn't determine root device``. `cryptsetup does not support ZFS `__. - Ignore any error messages saying ``Module zfs not found`` and ``couldn't connect to zsys daemon``. The first seems to occur due to a version mismatch between the Live CD kernel and the chroot environment, but this is irrelevant since the module is already loaded. The second may be caused by the first but either way is irrelevant since ``zed`` is started manually later. - For a mirror or raidz topology, this step only installs GRUB on the first disk. The other disk(s) will be handled later. For some reason, grub-efi-amd64 does not prompt for ``install_devices`` here, but does after a reboot. #. Optional: Remove os-prober:: apt purge --yes os-prober This avoids error messages from ``update-grub``. ``os-prober`` is only necessary in dual-boot configurations. #. Set a root password:: passwd #. Configure swap: Choose one of the following options if you want swap: - For an unencrypted single-disk install:: mkswap -f ${DISK}-part2 echo /dev/disk/by-uuid/$(blkid -s UUID -o value ${DISK}-part2) \ none swap discard 0 0 >> /etc/fstab swapon -a - For an unencrypted mirror or raidz topology:: apt install --yes mdadm # Adjust the level (ZFS raidz = MD raid5, raidz2 = raid6) and # raid-devices if necessary and specify the actual devices. mdadm --create /dev/md0 --metadata=1.2 --level=mirror \ --raid-devices=2 ${DISK1}-part2 ${DISK2}-part2 mkswap -f /dev/md0 echo /dev/disk/by-uuid/$(blkid -s UUID -o value /dev/md0) \ none swap discard 0 0 >> /etc/fstab - For an encrypted (LUKS or ZFS native encryption) single-disk install:: apt install --yes cryptsetup echo swap ${DISK}-part2 /dev/urandom \ swap,cipher=aes-xts-plain64:sha256,size=512 >> /etc/crypttab echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab - For an encrypted (LUKS or ZFS native encryption) mirror or raidz topology:: apt install --yes cryptsetup mdadm # Adjust the level (ZFS raidz = MD raid5, raidz2 = raid6) and # raid-devices if necessary and specify the actual devices. mdadm --create /dev/md0 --metadata=1.2 --level=mirror \ --raid-devices=2 ${DISK1}-part2 ${DISK2}-part2 echo swap /dev/md0 /dev/urandom \ swap,cipher=aes-xts-plain64:sha256,size=512 >> /etc/crypttab echo /dev/mapper/swap none swap defaults 0 0 >> /etc/fstab #. Optional (but recommended): Mount a tmpfs to ``/tmp`` If you chose to create a ``/tmp`` dataset above, skip this step, as they are mutually exclusive choices. Otherwise, you can put ``/tmp`` on a tmpfs (RAM filesystem) by enabling the ``tmp.mount`` unit. :: cp /usr/share/systemd/tmp.mount /etc/systemd/system/ systemctl enable tmp.mount #. Setup system groups:: addgroup --system lpadmin addgroup --system lxd addgroup --system sambashare #. Optional: Install SSH:: apt install --yes openssh-server vi /etc/ssh/sshd_config # Set: PermitRootLogin yes Step 5: GRUB Installation ------------------------- #. Verify that the ZFS boot filesystem is recognized:: grub-probe /boot #. Refresh the initrd files:: update-initramfs -c -k all **Note:** Ignore any error messages saying ``ERROR: Couldn't resolve device`` and ``WARNING: Couldn't determine root device``. `cryptsetup does not support ZFS `__. #. Disable memory zeroing:: vi /etc/default/grub # Add init_on_alloc=0 to: GRUB_CMDLINE_LINUX_DEFAULT # Save and quit (or see the next step). This is to address `performance regressions `__. #. Optional (but highly recommended): Make debugging GRUB easier:: vi /etc/default/grub # Comment out: GRUB_TIMEOUT_STYLE=hidden # Set: GRUB_TIMEOUT=5 # Below GRUB_TIMEOUT, add: GRUB_RECORDFAIL_TIMEOUT=5 # Remove quiet and splash from: GRUB_CMDLINE_LINUX_DEFAULT # Uncomment: GRUB_TERMINAL=console # Save and quit. Later, once the system has rebooted twice and you are sure everything is working, you can undo these changes, if desired. #. Update the boot configuration:: update-grub **Note:** Ignore errors from ``osprober``, if present. #. Install the boot loader: Choose one of the following options: - For legacy (BIOS) booting, install GRUB to the MBR:: grub-install $DISK Note that you are installing GRUB to the whole disk, not a partition. If you are creating a mirror or raidz topology, repeat the ``grub-install`` command for each disk in the pool. - For UEFI booting, install GRUB to the ESP:: grub-install --target=x86_64-efi --efi-directory=/boot/efi \ --bootloader-id=ubuntu --recheck #. Disable grub-initrd-fallback.service For a mirror or raidz topology:: systemctl mask grub-initrd-fallback.service This is the service for ``/boot/grub/grubenv`` which does not work on mirrored or raidz topologies. Disabling this keeps it from blocking subsequent mounts of ``/boot/grub`` if that mount ever fails. Another option would be to set ``RequiresMountsFor=/boot/grub`` via a drop-in unit, but that is more work to do here for no reason. Hopefully `this bug `__ will be fixed upstream. #. Fix filesystem mount ordering: We need to activate ``zfs-mount-generator``. This makes systemd aware of the separate mountpoints, which is important for things like ``/var/log`` and ``/var/tmp``. In turn, ``rsyslog.service`` depends on ``var-log.mount`` by way of ``local-fs.target`` and services using the ``PrivateTmp`` feature of systemd automatically use ``After=var-tmp.mount``. :: mkdir /etc/zfs/zfs-list.cache touch /etc/zfs/zfs-list.cache/bpool touch /etc/zfs/zfs-list.cache/rpool zed -F & Verify that ``zed`` updated the cache by making sure these are not empty:: cat /etc/zfs/zfs-list.cache/bpool cat /etc/zfs/zfs-list.cache/rpool If either is empty, force a cache update and check again:: zfs set canmount=on bpool/BOOT/ubuntu_$UUID zfs set canmount=on rpool/ROOT/ubuntu_$UUID If they are still empty, stop zed (as below), start zed (as above) and try again. Once the files have data, stop ``zed``:: fg Press Ctrl-C. Fix the paths to eliminate ``/mnt``:: sed -Ei "s|/mnt/?|/|" /etc/zfs/zfs-list.cache/* #. Exit from the ``chroot`` environment back to the LiveCD environment:: exit #. Run these commands in the LiveCD environment to unmount all filesystems:: mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | \ xargs -i{} umount -lf {} zpool export -a #. If export failed due to `busy` error, try to kill everything that might be using it:: grep [p]ool /proc/*/mounts | cut -d/ -f3 | uniq | xargs kill zpool export -a #. If even after that your pool is busy, mounting it on boot will fail and you will need to ``zpool import -f rpool``, then ``exit`` in the initramfs prompt. #. Reboot:: reboot Wait for the newly installed system to boot normally. Login as root. Step 6: First Boot ------------------ #. Install GRUB to additional disks: For a UEFI mirror or raidz topology only:: dpkg-reconfigure grub-efi-amd64 Select (using the space bar) all of the ESP partitions (partition 1 on each of the pool disks). #. Create a user account: Replace ``YOUR_USERNAME`` with your desired username:: username=YOUR_USERNAME UUID=$(dd if=/dev/urandom bs=1 count=100 2>/dev/null | tr -dc 'a-z0-9' | cut -c-6) ROOT_DS=$(zfs list -o name | awk '/ROOT\/ubuntu_/{print $1;exit}') zfs create -o com.ubuntu.zsys:bootfs-datasets=$ROOT_DS \ -o canmount=on -o mountpoint=/home/$username \ rpool/USERDATA/${username}_$UUID adduser $username cp -a /etc/skel/. /home/$username chown -R $username:$username /home/$username usermod -a -G adm,cdrom,dip,lpadmin,lxd,plugdev,sambashare,sudo $username Step 7: Full Software Installation ---------------------------------- #. Upgrade the minimal system:: apt dist-upgrade --yes #. Install a regular set of software: Choose one of the following options: - Install a command-line environment only:: apt install --yes ubuntu-standard - Install a full GUI environment:: apt install --yes ubuntu-desktop **Hint**: If you are installing a full GUI environment, you will likely want to manage your network with NetworkManager:: rm /etc/netplan/01-netcfg.yaml vi /etc/netplan/01-network-manager-all.yaml .. code-block:: yaml network: version: 2 renderer: NetworkManager #. Optional: Disable log compression: As ``/var/log`` is already compressed by ZFS, logrotate’s compression is going to burn CPU and disk I/O for (in most cases) very little gain. Also, if you are making snapshots of ``/var/log``, logrotate’s compression will actually waste space, as the uncompressed data will live on in the snapshot. You can edit the files in ``/etc/logrotate.d`` by hand to comment out ``compress``, or use this loop (copy-and-paste highly recommended):: for file in /etc/logrotate.d/* ; do if grep -Eq "(^|[^#y])compress" "$file" ; then sed -i -r "s/(^|[^#y])(compress)/\1#\2/" "$file" fi done #. Reboot:: reboot Step 8: Final Cleanup --------------------- #. Wait for the system to boot normally. Login using the account you created. Ensure the system (including networking) works normally. #. Optional: Disable the root password:: sudo usermod -p '*' root #. Optional (but highly recommended): Disable root SSH logins: If you installed SSH earlier, revert the temporary change:: sudo vi /etc/ssh/sshd_config # Remove: PermitRootLogin yes sudo systemctl restart ssh #. Optional: Re-enable the graphical boot process: If you prefer the graphical boot process, you can re-enable it now. If you are using LUKS, it makes the prompt look nicer. :: sudo vi /etc/default/grub # Uncomment: GRUB_TIMEOUT_STYLE=hidden # Add quiet and splash to: GRUB_CMDLINE_LINUX_DEFAULT # Comment out: GRUB_TERMINAL=console # Save and quit. sudo update-grub **Note:** Ignore errors from ``osprober``, if present. #. Optional: For LUKS installs only, backup the LUKS header:: sudo cryptsetup luksHeaderBackup /dev/disk/by-id/scsi-SATA_disk1-part4 \ --header-backup-file luks1-header.dat Store that backup somewhere safe (e.g. cloud storage). It is protected by your LUKS passphrase, but you may wish to use additional encryption. **Hint:** If you created a mirror or raidz topology, repeat this for each LUKS volume (``luks2``, etc.). Troubleshooting --------------- Rescuing using a Live CD ~~~~~~~~~~~~~~~~~~~~~~~~ Go through `Step 1: Prepare The Install Environment <#step-1-prepare-the-install-environment>`__. For LUKS, first unlock the disk(s):: cryptsetup luksOpen /dev/disk/by-id/scsi-SATA_disk1-part4 luks1 # Repeat for additional disks, if this is a mirror or raidz topology. Mount everything correctly:: zpool export -a zpool import -N -R /mnt rpool zpool import -N -R /mnt bpool zfs load-key -a # Replace “UUID” as appropriate; use zfs list to find it: zfs mount rpool/ROOT/ubuntu_UUID zfs mount bpool/BOOT/ubuntu_UUID zfs mount -a If needed, you can chroot into your installed environment:: mount --make-private --rbind /dev /mnt/dev mount --make-private --rbind /proc /mnt/proc mount --make-private --rbind /sys /mnt/sys mount -t tmpfs tmpfs /mnt/run mkdir /mnt/run/lock chroot /mnt /bin/bash --login mount -a Do whatever you need to do to fix your system. When done, cleanup:: exit mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | \ xargs -i{} umount -lf {} zpool export -a reboot Areca ~~~~~ Systems that require the ``arcsas`` blob driver should add it to the ``/etc/initramfs-tools/modules`` file and run ``update-initramfs -c -k all``. Upgrade or downgrade the Areca driver if something like ``RIP: 0010:[] [] native_read_tsc+0x6/0x20`` appears anywhere in kernel log. ZoL is unstable on systems that emit this error message. MPT2SAS ~~~~~~~ Most problem reports for this tutorial involve ``mpt2sas`` hardware that does slow asynchronous drive initialization, like some IBM M1015 or OEM-branded cards that have been flashed to the reference LSI firmware. The basic problem is that disks on these controllers are not visible to the Linux kernel until after the regular system is started, and ZoL does not hotplug pool members. See `https://github.com/openzfs/zfs/issues/330 `__. Most LSI cards are perfectly compatible with ZoL. If your card has this glitch, try setting ``ZFS_INITRD_PRE_MOUNTROOT_SLEEP=X`` in ``/etc/default/zfs``. The system will wait ``X`` seconds for all drives to appear before importing the pool. QEMU/KVM/XEN ~~~~~~~~~~~~ Set a unique serial number on each virtual disk using libvirt or qemu (e.g. ``-drive if=none,id=disk1,file=disk1.qcow2,serial=1234567890``). To be able to use UEFI in guests (instead of only BIOS booting), run this on the host:: sudo apt install ovmf sudo vi /etc/libvirt/qemu.conf Uncomment these lines: .. code-block:: text nvram = [ "/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd", "/usr/share/OVMF/OVMF_CODE.secboot.fd:/usr/share/OVMF/OVMF_VARS.fd", "/usr/share/AAVMF/AAVMF_CODE.fd:/usr/share/AAVMF/AAVMF_VARS.fd", "/usr/share/AAVMF/AAVMF32_CODE.fd:/usr/share/AAVMF/AAVMF32_VARS.fd", "/usr/share/OVMF/OVMF_CODE.ms.fd:/usr/share/OVMF/OVMF_VARS.ms.fd" ] :: sudo systemctl restart libvirtd.service VMware ~~~~~~ - Set ``disk.EnableUUID = "TRUE"`` in the vmx file or vsphere configuration. Doing this ensures that ``/dev/disk`` aliases are created in the guest. .. highlight:: sh Ubuntu 22.04 Root on ZFS for Raspberry Pi ========================================= .. contents:: Table of Contents :local: Overview -------- .. note:: These are beta instructions. The author still needs to test them. Additionally, it may be possible to use U-Boot now, which would eliminate some of the customizations. Caution ~~~~~~~ - This HOWTO uses a whole physical disk. - Backup your data. Any existing data will be lost. System Requirements ~~~~~~~~~~~~~~~~~~~ - A Raspberry Pi 4 B. (If you are looking to install on a regular PC, see :doc:`Ubuntu 22.04 Root on ZFS`.) - `Ubuntu Server 22.04 (“Jammy”) for Raspberry Pi 4 `__ - A microSD card or USB disk. For microSD card recommendations, see Jeff Geerling's `performance comparison `__. When using a USB enclosure, `ensure it supports UASP `__. - An Ubuntu system (with the ability to write to the microSD card or USB disk) other than the target Raspberry Pi. 4 GiB of memory is recommended. Do not use deduplication, as it needs `massive amounts of RAM `__. Enabling deduplication is a permanent change that cannot be easily reverted. A Raspberry Pi 3 B/B+ would probably work (as the Pi 3 is 64-bit, though it has less RAM), but has not been tested. Please report your results (good or bad) using the issue link below. Support ~~~~~~~ If you need help, reach out to the community using the :ref:`mailing_lists` or IRC at `#zfsonlinux `__ on `Libera Chat `__. If you have a bug report or feature request related to this HOWTO, please `file a new issue and mention @rlaager `__. Contributing ~~~~~~~~~~~~ #. Fork and clone: https://github.com/openzfs/openzfs-docs #. Install the tools:: sudo apt install python3-pip pip3 install -r docs/requirements.txt # Add ~/.local/bin to your $PATH, e.g. by adding this to ~/.bashrc: PATH=$HOME/.local/bin:$PATH #. Make your changes. #. Test:: cd docs make html sensible-browser _build/html/index.html #. ``git commit --signoff`` to a branch, ``git push``, and create a pull request. Mention @rlaager. Encryption ~~~~~~~~~~ **WARNING:** Encryption has not yet been tested on the Raspberry Pi. This guide supports three different encryption options: unencrypted, ZFS native encryption, and LUKS. With any option, all ZFS features are fully available. Unencrypted does not encrypt anything, of course. With no encryption happening, this option naturally has the best performance. ZFS native encryption encrypts the data and most metadata in the root pool. It does not encrypt dataset or snapshot names or properties. The boot pool is not encrypted at all, but it only contains the bootloader, kernel, and initrd. (Unless you put a password in ``/etc/fstab``, the initrd is unlikely to contain sensitive data.) The system cannot boot without the passphrase being entered at the console. Performance is good. As the encryption happens in ZFS, even if multiple disks (mirror or raidz topologies) are used, the data only has to be encrypted once. LUKS encrypts almost everything. The only unencrypted data is the bootloader, kernel, and initrd. The system cannot boot without the passphrase being entered at the console. Performance is good, but LUKS sits underneath ZFS, so if multiple disks (mirror or raidz topologies) are used, the data has to be encrypted once per disk. USB Disks ~~~~~~~~~ The Raspberry Pi 4 runs much faster using a USB Solid State Drive (SSD) than a microSD card. These instructions can also be used to install Ubuntu on a USB-connected SSD or other USB disk. USB disks have three requirements that do not apply to microSD cards: #. The Raspberry Pi's Bootloader EEPROM must be dated 2020-09-03 or later. To check the bootloader version, power up the Raspberry Pi without an SD card inserted or a USB boot device attached; the date will be on the ``bootloader`` line. (If you do not see the ``bootloader`` line, the bootloader is too old.) Alternatively, run ``sudo rpi-eeprom-update`` on an existing OS on the Raspberry Pi (which on Ubuntu requires ``apt install rpi-eeprom``). If needed, the bootloader can be updated from an existing OS on the Raspberry Pi using ``rpi-eeprom-update -a`` and rebooting. For other options, see `Updating the Bootloader `_. #. The Raspberry Pi must configured for USB boot. The bootloader will show a ``boot`` line; if ``order`` includes ``4``, USB boot is enabled. If not already enabled, it can be enabled from an existing OS on the Raspberry Pi using ``rpi-eeprom-config -e``: set ``BOOT_ORDER=0xf41`` and reboot to apply the change. On subsequent reboots, USB boot will be enabled. Otherwise, it can be enabled without an existing OS as follows: - Download the `Raspberry Pi Imager Utility `_. - Flash the ``USB Boot`` image to a microSD card. The ``USB Boot`` image is listed under ``Bootloader`` in the ``Misc utility images`` folder. - Boot the Raspberry Pi from the microSD card. USB Boot should be enabled automatically. #. U-Boot on Ubuntu 20.04 does not seem to support the Raspberry Pi USB. `Ubuntu 20.10 may work `_. As a work-around, the Raspberry Pi bootloader is configured to directly boot Linux. For this to work, the Linux kernel must not be compressed. These instructions decompress the kernel and add a script to ``/etc/kernel/postinst.d`` to handle kernel upgrades. Step 1: Disk Formatting ----------------------- The commands in this step are run on the system other than the Raspberry Pi. This guide has you go to some extra work so that the stock ext4 partition can be deleted. #. Download and unpack the official image:: curl -O https://cdimage.ubuntu.com/releases/22.04/release/ubuntu-22.04.1-preinstalled-server-arm64+raspi.img.xz xz -d ubuntu-22.04.1-preinstalled-server-arm64+raspi.img.xz # or combine them to decompress as you download: curl https://cdimage.ubuntu.com/releases/22.04/release/ubuntu-22.04.1-preinstalled-server-arm64+raspi.img.xz | \ xz -d > ubuntu-22.04.1-preinstalled-server-arm64+raspi.img #. Dump the partition table for the image:: sfdisk -d ubuntu-22.04.1-preinstalled-server-arm64+raspi.img That will output this:: label: dos label-id: 0x638274e3 device: ubuntu-22.04.1-preinstalled-server-arm64+raspi.img unit: sectors .img1 : start= 2048, size= 524288, type=c, bootable .img2 : start= 526336, size= 7193932, type=83 The important numbers are 524288 and 7193932. Store those in variables:: BOOT=524288 ROOT=7193932 #. Create a partition script:: cat > partitions << EOF label: dos unit: sectors 1 : start= 2048, size=$BOOT, type=c, bootable 2 : start=$((2048+BOOT)), size=$ROOT, type=83 3 : start=$((2048+BOOT+ROOT)), size=$ROOT, type=83 EOF This prepares partition 1 for the boot loader, partition 2 for the initial ZFS pool (sized the same as the original image), and partition 3 for the original image itself. The rest of the disk is left unpartitioned for now. Below, we populate partitions 1 and 3 from the image downloaded above, create the ZFS pool on partition 2, and transfer the files from partition 3 to the pool. As the ZFS partition is sized the same as the original image, everything should fit; while filesystem overhead may be different, the original image is not completely full, and ZFS has compression enabled. Finally, we remove partition 3 and expand partition 2 and the ZFS pool to consume the rest of the disk. #. Connect the disk: Connect the disk to a machine other than the target Raspberry Pi. If any filesystems are automatically mounted (e.g. by GNOME) unmount them. Determine the device name. For SD, the device name is almost certainly ``/dev/mmcblk0``. For USB SSDs, the device name is ``/dev/sdX``, where ``X`` is a lowercase letter. ``lsblk`` can help determine the device name. Set the ``DISK`` environment variable to the device name:: DISK=/dev/mmcblk0 # microSD card DISK=/dev/sdX # USB disk Because partitions are named differently for ``/dev/mmcblk0`` and ``/dev/sdX`` devices, set a second variable used when working with partitions:: export DISKP=${DISK}p # microSD card export DISKP=${DISK} # USB disk ($DISKP == $DISK for /dev/sdX devices) **Hint**: microSD cards connected using a USB reader also have ``/dev/sdX`` names. **WARNING**: The following steps destroy the existing data on the disk. Ensure ``DISK`` and ``DISKP`` are correct before proceeding. #. Ensure swap partitions are not in use:: swapon -v # If a partition is in use from the disk, disable it: sudo swapoff THAT_PARTITION #. Clear old ZFS labels:: sudo zpool labelclear -f ${DISK} If a ZFS label still exists from a previous system/attempt, expanding the pool will result in an unbootable system. **Hint:** If you do not already have the ZFS utilities installed, you can install them with: ``sudo apt install zfsutils-linux`` Alternatively, you can zero the entire disk with: ``sudo dd if=/dev/zero of=${DISK} bs=1M status=progress`` #. Delete existing partitions:: echo "label: dos" | sudo sfdisk ${DISK} sudo partprobe ls ${DISKP}* Make sure there are no partitions, just the file for the disk itself. This step is not strictly necessary; it exists to catch problems. #. Create the partitions:: sudo sfdisk $DISK < partitions #. Loopback mount the image:: IMG=$(sudo losetup -fP --show \ ubuntu-22.04.1-preinstalled-server-arm64+raspi.img) #. Copy the bootloader data:: sudo dd if=${IMG}p1 of=${DISKP}1 bs=1M #. Clear old label(s) from partition 2:: sudo wipefs -a ${DISKP}2 If a filesystem with the ``writable`` label from the Ubuntu image is still present in partition 2, the system will not boot initially. #. Copy the root filesystem data:: # NOTE: the destination is p3, not p2. sudo dd if=${IMG}p2 of=${DISKP}3 bs=1M status=progress conv=fsync #. Unmount the image:: sudo losetup -d $IMG #. If setting up a USB disk: Decompress the kernel:: sudo -sE MNT=$(mktemp -d /mnt/XXXXXXXX) mkdir -p $MNT/boot $MNT/root mount ${DISKP}1 $MNT/boot mount ${DISKP}3 $MNT/root zcat -qf $MNT/boot/vmlinuz >$MNT/boot/vmlinux Modify boot config:: cat >> $MNT/boot/usercfg.txt << EOF kernel=vmlinux initramfs initrd.img followkernel boot_delay EOF Create a script to automatically decompress the kernel after an upgrade:: cat >$MNT/root/etc/kernel/postinst.d/zz-decompress-kernel << 'EOF' #!/bin/sh set -eu echo "Updating decompressed kernel..." [ -e /boot/firmware/vmlinux ] && \ cp /boot/firmware/vmlinux /boot/firmware/vmlinux.bak vmlinuxtmp=$(mktemp /boot/firmware/vmlinux.XXXXXXXX) zcat -qf /boot/vmlinuz > "$vmlinuxtmp" mv "$vmlinuxtmp" /boot/firmware/vmlinux EOF chmod +x $MNT/root/etc/kernel/postinst.d/zz-decompress-kernel Cleanup:: umount $MNT/* rm -rf $MNT exit #. Boot the Raspberry Pi. Move the SD/USB disk to the Raspberry Pi. Boot it and login (e.g. via SSH) with ``ubuntu`` as the username and password. If you are using SSH, note that it takes a little bit for cloud-init to enable password logins on the first boot. Set a new password when prompted and login again using that password. If you have your local SSH configured to use ``ControlPersist``, you will have to kill the existing SSH process before logging in the second time. Step 2: Setup ZFS ----------------- #. Become root:: sudo -i #. Set the DISK and DISKP variables again:: DISK=/dev/mmcblk0 # microSD card DISKP=${DISK}p # microSD card DISK=/dev/sdX # USB disk DISKP=${DISK} # USB disk **WARNING:** Device names can change when moving a device to a different computer or switching the microSD card from a USB reader to a built-in slot. Double check the device name before continuing. #. Install ZFS:: apt update apt install pv zfs-initramfs **Note:** Since this is the first boot, you may get ``Waiting for cache lock`` because ``unattended-upgrades`` is running in the background. Wait for it to finish. #. Create the root pool: Choose one of the following options: - Unencrypted:: zpool create \ -o ashift=12 \ -O acltype=posixacl -O canmount=off -O compression=lz4 \ -O dnodesize=auto -O normalization=formD -O relatime=on \ -O xattr=sa -O mountpoint=/ -R /mnt \ rpool ${DISKP}2 **WARNING:** Encryption has not yet been tested on the Raspberry Pi. - ZFS native encryption:: zpool create \ -o ashift=12 \ -O encryption=on \ -O keylocation=prompt -O keyformat=passphrase \ -O acltype=posixacl -O canmount=off -O compression=lz4 \ -O dnodesize=auto -O normalization=formD -O relatime=on \ -O xattr=sa -O mountpoint=/ -R /mnt \ rpool ${DISKP}2 - LUKS:: cryptsetup luksFormat -c aes-xts-plain64 -s 512 -h sha256 ${DISKP}2 cryptsetup luksOpen ${DISK}-part4 luks1 zpool create \ -o ashift=12 \ -O acltype=posixacl -O canmount=off -O compression=lz4 \ -O dnodesize=auto -O normalization=formD -O relatime=on \ -O xattr=sa -O mountpoint=/ -R /mnt \ rpool /dev/mapper/luks1 **Notes:** - The use of ``ashift=12`` is recommended here because many drives today have 4 KiB (or larger) physical sectors, even though they present 512 B logical sectors. Also, a future replacement drive may have 4 KiB physical sectors (in which case ``ashift=12`` is desirable) or 4 KiB logical sectors (in which case ``ashift=12`` is required). - Setting ``-O acltype=posixacl`` enables POSIX ACLs globally. If you do not want this, remove that option, but later add ``-o acltype=posixacl`` (note: lowercase “o”) to the ``zfs create`` for ``/var/log``, as `journald requires ACLs `__ Also, `disabling ACLs apparently breaks umask handling with NFSv4 `__. - Setting ``normalization=formD`` eliminates some corner cases relating to UTF-8 filename normalization. It also implies ``utf8only=on``, which means that only UTF-8 filenames are allowed. If you care to support non-UTF-8 filenames, do not use this option. For a discussion of why requiring UTF-8 filenames may be a bad idea, see `The problems with enforced UTF-8 only filenames `__. - ``recordsize`` is unset (leaving it at the default of 128 KiB). If you want to tune it (e.g. ``-O recordsize=1M``), see `these `__ `various `__ `blog `__ `posts `__. - Setting ``relatime=on`` is a middle ground between classic POSIX ``atime`` behavior (with its significant performance impact) and ``atime=off`` (which provides the best performance by completely disabling atime updates). Since Linux 2.6.30, ``relatime`` has been the default for other filesystems. See `RedHat’s documentation `__ for further information. - Setting ``xattr=sa`` `vastly improves the performance of extended attributes `__. Inside ZFS, extended attributes are used to implement POSIX ACLs. Extended attributes can also be used by user-space applications. `They are used by some desktop GUI applications. `__ `They can be used by Samba to store Windows ACLs and DOS attributes; they are required for a Samba Active Directory domain controller. `__ Note that ``xattr=sa`` is `Linux-specific `__. If you move your ``xattr=sa`` pool to another OpenZFS implementation besides ZFS-on-Linux, extended attributes will not be readable (though your data will be). If portability of extended attributes is important to you, omit the ``-O xattr=sa`` above. Even if you do not want ``xattr=sa`` for the whole pool, it is probably fine to use it for ``/var/log``. - Make sure to include the ``-part4`` portion of the drive path. If you forget that, you are specifying the whole disk, which ZFS will then re-partition, and you will lose the bootloader partition(s). - ZFS native encryption `now `__ defaults to ``aes-256-gcm``. - For LUKS, the key size chosen is 512 bits. However, XTS mode requires two keys, so the LUKS key is split in half. Thus, ``-s 512`` means AES-256. - Your passphrase will likely be the weakest link. Choose wisely. See `section 5 of the cryptsetup FAQ `__ for guidance. Step 3: System Installation --------------------------- #. Create a filesystem dataset to act as a container:: zfs create -o canmount=off -o mountpoint=none rpool/ROOT #. Create a filesystem dataset for the root filesystem:: UUID=$(dd if=/dev/urandom bs=1 count=100 2>/dev/null | tr -dc 'a-z0-9' | cut -c-6) zfs create -o canmount=noauto -o mountpoint=/ \ -o com.ubuntu.zsys:bootfs=yes \ -o com.ubuntu.zsys:last-used=$(date +%s) rpool/ROOT/ubuntu_$UUID zfs mount rpool/ROOT/ubuntu_$UUID With ZFS, it is not normally necessary to use a mount command (either ``mount`` or ``zfs mount``). This situation is an exception because of ``canmount=noauto``. #. Create datasets:: zfs create -o com.ubuntu.zsys:bootfs=no -o canmount=off \ rpool/ROOT/ubuntu_$UUID/usr zfs create -o com.ubuntu.zsys:bootfs=no -o canmount=off \ rpool/ROOT/ubuntu_$UUID/var zfs create rpool/ROOT/ubuntu_$UUID/var/lib zfs create rpool/ROOT/ubuntu_$UUID/var/log zfs create rpool/ROOT/ubuntu_$UUID/var/spool zfs create -o canmount=off -o mountpoint=/ \ rpool/USERDATA zfs create -o com.ubuntu.zsys:bootfs-datasets=rpool/ROOT/ubuntu_$UUID \ -o canmount=on -o mountpoint=/root \ rpool/USERDATA/root_$UUID chmod 700 /mnt/root The datasets below are optional, depending on your preferences and/or software choices. If you wish to separate these to exclude them from snapshots:: zfs create rpool/ROOT/ubuntu_$UUID/var/cache zfs create rpool/ROOT/ubuntu_$UUID/var/lib/nfs zfs create rpool/ROOT/ubuntu_$UUID/var/tmp chmod 1777 /mnt/var/tmp If desired (the Ubuntu installer creates these):: zfs create rpool/ROOT/ubuntu_$UUID/var/lib/apt zfs create rpool/ROOT/ubuntu_$UUID/var/lib/dpkg If you use /srv on this system:: zfs create -o com.ubuntu.zsys:bootfs=no \ rpool/ROOT/ubuntu_$UUID/srv If you use /usr/local on this system:: zfs create rpool/ROOT/ubuntu_$UUID/usr/local If this system will have games installed:: zfs create rpool/ROOT/ubuntu_$UUID/var/games If this system will have a GUI:: zfs create rpool/ROOT/ubuntu_$UUID/var/lib/AccountsService zfs create rpool/ROOT/ubuntu_$UUID/var/lib/NetworkManager If this system will use Docker (which manages its own datasets & snapshots):: zfs create rpool/ROOT/ubuntu_$UUID/var/lib/docker If this system will store local email in /var/mail:: zfs create rpool/ROOT/ubuntu_$UUID/var/mail If this system will use Snap packages:: zfs create rpool/ROOT/ubuntu_$UUID/var/snap If you use /var/www on this system:: zfs create rpool/ROOT/ubuntu_$UUID/var/www For a mirror or raidz topology, create a dataset for ``/boot/grub``:: zfs create -o com.ubuntu.zsys:bootfs=no bpool/grub A tmpfs is recommended later, but if you want a separate dataset for ``/tmp``:: zfs create -o com.ubuntu.zsys:bootfs=no \ rpool/ROOT/ubuntu_$UUID/tmp chmod 1777 /mnt/tmp The primary goal of this dataset layout is to separate the OS from user data. This allows the root filesystem to be rolled back without rolling back user data. If you do nothing extra, ``/tmp`` will be stored as part of the root filesystem. Alternatively, you can create a separate dataset for ``/tmp``, as shown above. This keeps the ``/tmp`` data out of snapshots of your root filesystem. It also allows you to set a quota on ``rpool/tmp``, if you want to limit the maximum space used. Otherwise, you can use a tmpfs (RAM filesystem) later. **Note:** If you separate a directory required for booting (e.g. ``/etc``) into its own dataset, you must add it to ``ZFS_INITRD_ADDITIONAL_DATASETS`` in ``/etc/default/zfs``. Datasets with ``canmount=off`` (like ``rpool/usr`` above) do not matter for this. #. Optional: Ignore synchronous requests: microSD cards are relatively slow. If you want to increase performance (especially when installing packages) at the cost of some safety, you can disable flushing of synchronous requests (e.g. ``fsync()``, ``O_[D]SYNC``): Choose one of the following options: - For the root filesystem, but not user data:: zfs set sync=disabled rpool/ROOT - For everything:: zfs set sync=disabled rpool ZFS is transactional, so it will still be crash consistent. However, you should leave ``sync`` at its default of ``standard`` if this system needs to guarantee persistence (e.g. if it is a database or NFS server). #. Copy the system into the ZFS filesystems:: (cd /; tar -cf - --one-file-system --warning=no-file-ignored .) | \ pv -p -bs $(du -sxm --apparent-size / | cut -f1)m | \ (cd /mnt ; tar -x) Step 4: System Configuration ---------------------------- #. Configure the hostname: Replace ``HOSTNAME`` with the desired hostname:: hostname HOSTNAME hostname > /mnt/etc/hostname vi /mnt/etc/hosts .. code-block:: text Add a line: 127.0.1.1 HOSTNAME or if the system has a real name in DNS: 127.0.1.1 FQDN HOSTNAME **Hint:** Use ``nano`` if you find ``vi`` confusing. #. Stop ``zed``:: systemctl stop zed #. Bind the virtual filesystems from the running environment to the new ZFS environment and ``chroot`` into it:: mount --make-private --rbind /boot/firmware /mnt/boot/firmware mount --make-private --rbind /dev /mnt/dev mount --make-private --rbind /proc /mnt/proc mount --make-private --rbind /run /mnt/run mount --make-private --rbind /sys /mnt/sys chroot /mnt /usr/bin/env DISK=$DISK UUID=$UUID bash --login #. Configure a basic system environment:: apt update Even if you prefer a non-English system language, always ensure that ``en_US.UTF-8`` is available:: dpkg-reconfigure locales dpkg-reconfigure tzdata #. For LUKS installs only, setup ``/etc/crypttab``:: # cryptsetup is already installed, but this marks it as manually # installed so it is not automatically removed. apt install --yes cryptsetup echo luks1 UUID=$(blkid -s UUID -o value ${DISK}-part4) none \ luks,discard,initramfs > /etc/crypttab The use of ``initramfs`` is a work-around for `cryptsetup does not support ZFS `__. #. Optional: Mount a tmpfs to ``/tmp`` If you chose to create a ``/tmp`` dataset above, skip this step, as they are mutually exclusive choices. Otherwise, you can put ``/tmp`` on a tmpfs (RAM filesystem) by enabling the ``tmp.mount`` unit. :: cp /usr/share/systemd/tmp.mount /etc/systemd/system/ systemctl enable tmp.mount #. Setup system groups:: addgroup --system lpadmin addgroup --system sambashare #. Fix filesystem mount ordering: We need to activate ``zfs-mount-generator``. This makes systemd aware of the separate mountpoints, which is important for things like ``/var/log`` and ``/var/tmp``. In turn, ``rsyslog.service`` depends on ``var-log.mount`` by way of ``local-fs.target`` and services using the ``PrivateTmp`` feature of systemd automatically use ``After=var-tmp.mount``. :: mkdir /etc/zfs/zfs-list.cache touch /etc/zfs/zfs-list.cache/rpool zed -F & Force a cache update:: zfs set canmount=noauto rpool/ROOT/ubuntu_$UUID Verify that ``zed`` updated the cache by making sure this is not empty, which will take a few seconds:: cat /etc/zfs/zfs-list.cache/rpool Stop ``zed``:: fg Press Ctrl-C. Fix the paths to eliminate ``/mnt``:: sed -Ei "s|/mnt/?|/|" /etc/zfs/zfs-list.cache/* #. Remove old filesystem from ``/etc/fstab``:: vi /etc/fstab # Remove the old root filesystem line: # LABEL=writable / ext4 ... #. Configure kernel command line:: cp /boot/firmware/cmdline.txt /boot/firmware/cmdline.txt.bak sed -i "s|root=LABEL=writable rootfstype=ext4|root=ZFS=rpool/ROOT/ubuntu_$UUID|" \ /boot/firmware/cmdline.txt sed -i "s| fixrtc||" /boot/firmware/cmdline.txt sed -i "s|$| init_on_alloc=0|" /boot/firmware/cmdline.txt The ``fixrtc`` script is not compatible with ZFS and will cause the boot to hang for 180 seconds. The ``init_on_alloc=0`` is to address `performance regressions `__. #. Optional (but highly recommended): Make debugging booting easier:: sed -i "s|$| nosplash|" /boot/firmware/cmdline.txt #. Reboot:: exit reboot Wait for the newly installed system to boot normally. Login as ``ubuntu``. Step 5: First Boot ------------------ #. Become root:: sudo -i #. Set the DISK variable again:: DISK=/dev/mmcblk0 # microSD card DISK=/dev/sdX # USB disk #. Delete the ext4 partition and expand the ZFS partition:: sfdisk $DISK --delete 3 echo ", +" | sfdisk --no-reread -N 2 $DISK **Note:** This does not automatically expand the pool. That will be happen on reboot. #. Create a user account: Replace ``YOUR_USERNAME`` with your desired username:: username=YOUR_USERNAME UUID=$(dd if=/dev/urandom bs=1 count=100 2>/dev/null | tr -dc 'a-z0-9' | cut -c-6) ROOT_DS=$(zfs list -o name | awk '/ROOT\/ubuntu_/{print $1;exit}') zfs create -o com.ubuntu.zsys:bootfs-datasets=$ROOT_DS \ -o canmount=on -o mountpoint=/home/$username \ rpool/USERDATA/${username}_$UUID adduser $username cp -a /etc/skel/. /home/$username chown -R $username:$username /home/$username usermod -a -G adm,cdrom,dip,lpadmin,lxd,plugdev,sambashare,sudo $username #. Reboot:: reboot Wait for the system to boot normally. Login using the account you created. #. Become root:: sudo -i #. Expand the ZFS pool: Verify the pool expanded:: zfs list rpool If it did not automatically expand, try to expand it manually:: DISK=/dev/mmcblk0 # microSD card DISKP=${DISK}p # microSD card DISK=/dev/sdX # USB disk DISKP=${DISK} # USB disk zpool online -e rpool ${DISKP}2 #. Delete the ``ubuntu`` user:: deluser --remove-home ubuntu Step 6: Full Software Installation ---------------------------------- #. Optional: Remove cloud-init:: vi /etc/netplan/01-netcfg.yaml .. code-block:: yaml network: version: 2 ethernets: eth0: dhcp4: true :: rm /etc/netplan/50-cloud-init.yaml apt purge --autoremove ^cloud-init rm -rf /etc/cloud #. Optional: Remove other storage packages:: apt purge --autoremove bcache-tools btrfs-progs cloud-guest-utils lvm2 \ mdadm multipath-tools open-iscsi overlayroot xfsprogs #. Upgrade the minimal system:: apt dist-upgrade --yes #. Optional: Install a full GUI environment:: apt install --yes ubuntu-desktop echo dtoverlay=vc4-fkms-v3d >> /boot/firmware/usercfg.txt **Hint**: If you are installing a full GUI environment, you will likely want to remove cloud-init as discussed above but manage your network with NetworkManager:: rm /etc/netplan/*.yaml vi /etc/netplan/01-network-manager-all.yaml .. code-block:: yaml network: version: 2 renderer: NetworkManager #. Optional (but recommended): Disable log compression: As ``/var/log`` is already compressed by ZFS, logrotate’s compression is going to burn CPU and disk I/O for (in most cases) very little gain. Also, if you are making snapshots of ``/var/log``, logrotate’s compression will actually waste space, as the uncompressed data will live on in the snapshot. You can edit the files in ``/etc/logrotate.d`` by hand to comment out ``compress``, or use this loop (copy-and-paste highly recommended):: for file in /etc/logrotate.d/* ; do if grep -Eq "(^|[^#y])compress" "$file" ; then sed -i -r "s/(^|[^#y])(compress)/\1#\2/" "$file" fi done #. Reboot:: reboot Step 7: Final Cleanup --------------------- #. Wait for the system to boot normally. Login using the account you created. Ensure the system (including networking) works normally. #. Optional: For LUKS installs only, backup the LUKS header:: sudo cryptsetup luksHeaderBackup /dev/disk/by-id/scsi-SATA_disk1-part4 \ --header-backup-file luks1-header.dat Store that backup somewhere safe (e.g. cloud storage). It is protected by your LUKS passphrase, but you may wish to use additional encryption. **Hint:** If you created a mirror or raidz topology, repeat this for each LUKS volume (``luks2``, etc.). Basic Concepts ============== How OpenZFS is put together and what its features actually do. If you are new to ZFS, read :doc:`Copy-on-Write ` first — most of what follows is a consequence of it — and then :doc:`VDEVs `, since pool layout is the decision that is hardest to change later. These pages explain concepts and trade-offs; the `man pages <../man/index.html>`__ remain the reference for exact syntax and options. .. toctree:: :maxdepth: 2 Copy-on-write Pool Structure/index Datasets/index Data Storage/index Operations/index Copy-on-Write ============= Almost everything distinctive about ZFS follows from one rule: **a block that is in use is never overwritten.** Changing data means writing it somewhere else and then switching over to the new version. Understanding that rule makes the rest of ZFS — snapshots, ``zfs send``, self-healing, the absence of ``fsck`` — stop looking like separate features and start looking like consequences. The tree ~~~~~~~~ A pool is one large tree. Leaves hold data; interior nodes hold *block pointers*, each of which records where a child lives, how big it is, and — the part that matters — its checksum. A block's checksum is therefore stored in its parent, not next to the block itself, so a damaged block cannot vouch for itself. At the root sits the **uberblock**. Everything in the pool is reachable from it. Writing something ~~~~~~~~~~~~~~~~~ Modifying a block in the middle of the tree cannot happen in place, so ZFS: #. writes the new data to free space; #. writes a new version of the parent, pointing at the new block; #. repeats up the tree, since each parent's checksum has now changed; #. finally writes a new uberblock. .. raw:: html
uberblock root ind 1 ind 2 D1 D2 D3 D4 D4 ind 2 root uberblock D4' ind 2' root' uberblock' untouched — still whole, still consistent superseded — freed, unless a snapshot holds it written by this transaction shared by reference, not copied
In the diagram, only ``D4`` was modified. That forced new copies of ``D4``, its parent ``ind 2``, the ``root`` block and finally the uberblock — the *path from the changed block to the root*, and nothing else. ``D1``, ``D2``, ``D3`` and ``ind 1`` are referenced by the new tree, not duplicated. Until that last step, nothing has changed: the old uberblock still describes a complete, consistent pool. The switch to the new tree is a single atomic write. This is why a ZFS pool is always consistent on disk regardless of when power is lost — there is no window in which the pool is half-updated, and so no equivalent of the RAID-5 write hole and no need for a journal replay of metadata. The cost of a small write is therefore proportional to the *depth* of the tree, not its size — and the old tree is left whole, which is exactly what a snapshot needs. The uberblock is not overwritten either. Each device label carries a **128 KiB uberblock ring** written round-robin, and each device carries **four copies of the label** — two at the start, two at the end — so that losing either end of a disk, or a partition table rewrite, does not lose the root of the tree. On import ZFS picks the valid uberblock with the highest transaction number. Transaction groups ~~~~~~~~~~~~~~~~~~ Writes are not committed one at a time. They are batched into a **transaction group** (txg), and the whole group commits atomically or not at all. Three transaction groups are in flight at once: one **open** and accepting new writes, one **quiescing** (closed to new writes, waiting for stragglers), and one **syncing** to disk. A group closes when ``zfs_txg_timeout`` elapses — 5 seconds by default — or when enough dirty data has accumulated. The practical consequence is that an asynchronous write returns as soon as it is in memory, and reaches disk up to a few seconds later. A crash loses the groups that had not finished syncing, and the pool comes back at the last committed one. It is consistent, just slightly older. That is unacceptable for applications that called ``fsync()``, which is what the :doc:`ZFS Intent Log ` exists for: it records synchronous writes immediately so they can be replayed if the system dies before the txg commits. The ZIL is never read otherwise. What follows from it ~~~~~~~~~~~~~~~~~~~~ **Snapshots are nearly free.** The old tree is already intact and complete — a snapshot is just a reference that stops those blocks being freed. Nothing is copied. See :doc:`Snapshots, Clones and Bookmarks `. **Clones and block cloning are cheap** for the same reason: multiple references to the same block are normal, so sharing costs a reference rather than a copy. **Incremental replication is exact.** Because every block records the txg in which it was born, ZFS can find precisely the blocks that changed between two snapshots without scanning anything. See :doc:`Send and Receive `. **Corruption is detectable and often repairable.** The checksum lives in the parent, so a block that comes back wrong is caught on read, and with redundancy a good copy is used and the bad one rewritten. See :doc:`Checksums ` and :doc:`Scrub and Resilver `. **There is no fsck.** ``fsck.zfs`` exists only to satisfy the interface and does essentially nothing — its man page says so. Nothing needs to be checked for consistency at mount time, because the pool is never left inconsistent. Verifying *content* is a different job, and that is what ``zpool scrub`` does, online. The costs ~~~~~~~~~ Copy-on-write is not free, and its costs explain several tuning topics: **Fragmentation.** Rewriting a file scatters its new blocks wherever there was free space, so data that was written sequentially and then modified randomly does not stay sequential. This is why ZFS is sensitive to how full a pool is: finding contiguous free space gets harder, and the allocator has to work harder. See :doc:`Workload Tuning `. **Free space is needed to write at all.** Even deleting data requires allocating, which is why ZFS reserves slop space — see :doc:`Quotas and Reservations `. **Partial writes are expensive.** Changing part of a record means reading the whole record, modifying it, and writing a new one. Matching ``recordsize`` to the application's I/O size is the usual remedy, and the reason databases and VM images get their own settings. Further reading ~~~~~~~~~~~~~~~ * `zfsconcepts(7) `__, `zpoolconcepts(7) `__ * :doc:`Checksums `, :doc:`Snapshots, Clones and Bookmarks ` * :doc:`ZFS Transaction Delay `, :doc:`ZIO Scheduler ` Pool Structure ============== How a pool is assembled out of devices, what redundancy each layout gives you, and what can still be changed afterwards. .. toctree:: :maxdepth: 1 VDEVs RAIDZ dRAID Howto Caching Special vdev Changing Pool Layout Feature Flags VDEVs ===== What is a VDEV? ~~~~~~~~~~~~~~~ A vdev (virtual device) is a fundamental building block of ZFS storage pools. It represents a logical grouping of physical storage devices, such as hard drives, SSDs, or partitions. What is a leaf vdev? ~~~~~~~~~~~~~~~~~~~~ A leaf vdev is the most basic type of vdev, which directly corresponds to a physical storage device. It is the endpoint of the storage hierarchy in ZFS. What is a top-level vdev? ~~~~~~~~~~~~~~~~~~~~~~~~~ Top-level vdevs are the direct children of the root vdev. They can be single devices or logical groups that aggregate multiple leaf vdevs (like mirrors or RAIDZ groups). ZFS dynamically stripes data across all top-level vdevs in a pool. What is a root vdev? ~~~~~~~~~~~~~~~~~~~~ The root vdev is the top of the pool hierarchy. It aggregates all top-level vdevs into a single logical storage unit (the pool). What are the different types of vdevs? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ OpenZFS supports several types of vdevs. Top-level vdevs carry data and provide redundancy: * **Single device**: A lone disk, file or partition used as a top-level vdev. It provides no redundancy, and its loss loses the whole pool. Note that there is no "stripe" vdev type in ZFS — striping is what the pool does *across* its top-level vdevs, not a vdev in its own right. * **Mirror**: A vdev that stores the same data on two or more drives for redundancy. * :doc:`RAIDZ `: A vdev that uses parity to provide fault tolerance, similar to traditional RAID 5/6. There are three levels of RAIDZ: * **RAIDZ1**: Single parity, similar to RAID 5. Requires at least 2 disks (3+ recommended), can tolerate one drive failure. * **RAIDZ2**: Double parity, similar to RAID 6. Requires at least 3 disks (5+ recommended), can tolerate two drive failures. * **RAIDZ3**: Triple parity. Requires at least 4 disks (7+ recommended), can tolerate three drive failures. * :doc:`dRAID `: Distributed RAID. A vdev that provides distributed parity and hot spares across multiple drives, allowing for much faster rebuild performance after a failure. Auxiliary vdevs provide specific functionality: * **Spare**: A drive that acts as a hot spare, automatically replacing a failed drive in another vdev. * **Cache (L2ARC)**: A Level 2 ARC vdev used for caching frequently accessed data to improve random read performance. It holds only a second copy of data that is already in the pool, so losing it is harmless. * **Log (SLOG)**: A separate log vdev (SLOG) used to store the ZFS Intent Log (ZIL) for improved synchronous write performance. It is not a write cache — only synchronous writes use it. * **Special**: A vdev dedicated to storing metadata, and optionally small file blocks and the Dedup Table (DDT). * **Dedup**: A vdev dedicated strictly to storing the Deduplication Table (DDT). ``special`` and ``dedup`` vdevs are the only data the pool has for those blocks, so they must be as redundant as the pool's normal vdevs — losing one loses the pool. See :doc:`Caching and Auxiliary Devices ` for when each of these is worth adding. How do vdevs relate to storage pools? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Vdevs are the building blocks of ZFS storage pools. A storage pool (zpool) is created by combining one or more top-level vdevs. The overall performance, capacity, and redundancy of the storage pool depend on the configuration and types of vdevs used. Here is an example layout as seen in `zpool-status(8) `_ output for a pool with two RAIDZ1 top-level vdevs and 10 leaf vdevs: :: datapoolname (root vdev) raidz1-0 (top-level vdev) /dev/dsk/disk0 (leaf vdev) /dev/dsk/disk1 (leaf vdev) /dev/dsk/disk2 (leaf vdev) /dev/dsk/disk3 (leaf vdev) /dev/dsk/disk4 (leaf vdev) raidz1-1 (top-level vdev) /dev/dsk/disk5 (leaf vdev) /dev/dsk/disk6 (leaf vdev) /dev/dsk/disk7 (leaf vdev) /dev/dsk/disk8 (leaf vdev) /dev/dsk/disk9 (leaf vdev) How does ZFS handle vdev failures? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Redundancy in ZFS lives **inside** a top-level vdev, not across them. A mirror survives the loss of all but one of its leaf vdevs, and a raidz\ *N* survives the loss of *N*. When a leaf vdev fails, ZFS marks it "faulted", keeps serving reads and writes from the surviving members of that same top-level vdev, and reconstructs the missing data from them. The corollary matters more than the rule: **if a top-level vdev is lost, the pool is lost.** Because data is striped across all top-level vdevs, there is no copy of its contents anywhere else. A single non-redundant disk added to an otherwise mirrored pool therefore puts the entire pool at the mercy of that one disk. Administrators can `zpool-replace(8) `_ a failed leaf vdev with a new one and ZFS will automatically resilver (rebuild) the data onto it to return the pool to a healthy state. See :doc:`Scrub and Resilver `. How do I manage vdevs in ZFS? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Vdevs are managed using the `zpool(8) `_ command-line utility. Common operations include: * **Creating a pool**: `zpool create` allows you to specify the vdev layout. See `zpool-create(8) `_. * **Adding vdevs**: `zpool add` attaches new top-level vdevs to an existing pool, expanding its capacity and performance (by increasing stripe width). See `zpool-add(8) `_. * **Widening a vdev**: `zpool attach` adds a device to a mirror, or to an existing raidz group (RAIDZ expansion). See `zpool-attach(8) `_. * **Removing vdevs**: `zpool remove` can remove certain types of top-level vdevs evacuating their data to other vdevs. It is not possible at all if the pool contains a raidz or draid top-level vdev. See `zpool-remove(8) `_. * **Replacing drives**: `zpool replace` swaps a failed or small drive with a new one. See `zpool-replace(8) `_. * **Monitoring status**: `zpool status` shows the health and layout of all vdevs. See `zpool-status(8) `_. * **Monitoring performance**: `zpool iostat` displays I/O statistics for the pool and individual vdevs. See `zpool-iostat(8) `_. :doc:`Changing Pool Layout ` covers these operations and their constraints in detail. RAIDZ ===== Introduction ~~~~~~~~~~~~ RAIDZ is a variation on RAID-5 that allows for better distribution of parity and eliminates the RAID-5 “write hole” (in which data and parity become inconsistent after a power loss). Data and parity is striped across all disks within a raidz group. A raidz group can have single, double, or triple parity, meaning that the raidz group can sustain one, two, or three failures, respectively, without losing any data. The ``raidz1`` vdev type specifies a single-parity raidz group; the ``raidz2`` vdev type specifies a double-parity raidz group; and the ``raidz3`` vdev type specifies a triple-parity raidz group. The ``raidz`` vdev type is an alias for raidz1. A raidz group of N disks of size X with P parity disks can hold approximately (N-P)*X bytes and can withstand P devices failing without losing data. The minimum number of devices in a raidz group is one more than the number of parity disks. The recommended number is between 3 and 9 to help increase performance. Space efficiency ~~~~~~~~~~~~~~~~ Actual used space for a block in RAIDZ is based on several points: - minimal write size is disk sector size (can be set via `ashift` vdev parameter) - stripe width in RAIDZ is dynamic, and starts with at least one data block part, or up to ``disks count`` minus ``parity number`` parts of data block - one block of data with size of ``recordsize`` is split equally via ``sector size`` parts and written on each stripe on RAIDZ vdev - each stripe of data will have a part of block - in addition to data one, two or three blocks of parity should be written, one per disk; so, for raidz2 of 5 disks there will be 3 blocks of data and 2 blocks of parity Due to these inputs, if ``recordsize`` is less or equal to sector size, then RAIDZ's parity size will be effectively equal to mirror with same redundancy. For example, for raidz1 of 3 disks with ``ashift=12`` and ``recordsize=4K`` we will allocate on disk: - one 4K block of data - one 4K parity block and usable space ratio will be 50%, same as with double mirror. Another example for ``ashift=12`` and ``recordsize=128K`` for raidz1 of 3 disks: - total stripe width is 3 - one stripe can have up to 2 data parts of 4K size because of 1 parity blocks - we will have 128K/8k = 16 stripes with 8K of data and 4K of parity each - 16 stripes each with 12k, means we write 192k to store 128k so usable space ratio in this case will be 66%. The more disks RAIDZ has, the wider the stripe, the greater the space efficiency. You can find actual parity cost per RAIDZ size here: .. raw:: html (`source `__) Performance considerations ~~~~~~~~~~~~~~~~~~~~~~~~~~ Write ^^^^^ A stripe spans across all drives in the array. A one block write will write the stripe part onto each disk. A RAIDZ vdev has a write IOPS of the slowest disk in the array in the worst case because the write operation of all stripe parts must be completed on each disk. Read ^^^^ A read touches only the columns the block actually occupies, not the whole group. A block of ``N`` sectors that does not fill a complete stripe row is laid out across ``N`` data columns; the rest of the group is untouched. Parity is not read at all on a healthy read — it is fetched only when data is missing, or during a scrub or resilver. So the width of the vdev is not what decides read behavior — the ratio between block size and stripe width is: * When a block spans **all** data columns — the usual case, since the default ``recordsize`` of 128 KiB fills any realistic group — every read costs one I/O on every data disk. Random-read IOPS then works out roughly the same as a single disk's, however wide the group. * When blocks are **small** relative to the group — a database with ``recordsize=8K`` or ``16K`` on a wide vdev — each read lands on only a few disks, and concurrent reads to different blocks proceed in parallel. IOPS does scale in that regime, at the cost of the poor space efficiency small records have on RAIDZ (see above). Streaming bandwidth scales with the number of data disks in both cases. The rule of thumb "RAIDZ gives you the IOPS of one disk" is therefore about the default large-record configuration, not a property of RAIDZ itself. It is still the right assumption for general-purpose pools, and mirrors remain the better choice when random-read IOPS is the binding constraint. One slow disk """"""""""""" Because a read completes only when every column it spans has returned, its latency is that of the slowest disk involved. A single sick drive can therefore drag down a whole group (`#9375 `__). Since OpenZFS 2.4 this is detected and worked around: a persistently slow child is put in a *sit out* state, during which reads skip it and reconstruct its data from parity. Writes still go to it, so redundancy is maintained, and scrubs always read it. Up to ``nparity`` disks may sit out at once. The period is set by ``vdev_read_sit_out_secs`` (default 600 s; ``0`` disables detection entirely). Each sit-out bumps the vdev's ``slow_ios`` counter and posts an ``ereport.fs.zfs.delay`` event. Expansion ~~~~~~~~~ A raidz vdev can be widened by attaching another device to it: .. code:: bash zpool attach pool raidz2-0 sdX Fault tolerance is unchanged — a RAID-Z2 stays a RAID-Z2 — and blocks written before the expansion keep their original data-to-parity ratio, just spread over more disks. Only newly written blocks use the wider ratio. See :doc:`Changing Pool Layout `. dRAID ===== .. note:: This page describes functionality which has been added for the OpenZFS 2.1.0 release, it is not in the OpenZFS 2.0.0 release. Introduction ~~~~~~~~~~~~ `dRAID`_ is a variant of raidz that provides integrated distributed hot spares which allows for faster resilvering while retaining the benefits of raidz. A dRAID vdev is constructed from multiple internal raidz groups, each with D data devices and P parity devices. These groups are distributed over all of the children in order to fully utilize the available disk performance. This is known as parity declustering and it has been an active area of research. The image below is simplified, but it helps illustrate this key difference between dRAID and raidz. |draid1| Additionally, a dRAID vdev must shuffle its child vdevs in such a way that regardless of which drive has failed, the rebuild IO (both read and write) will distribute evenly among all surviving drives. This is accomplished by using carefully chosen precomputed permutation maps. This has the advantage of both keeping pool creation fast and making it impossible for the mapping to be damaged or lost. Another way dRAID differs from raidz is that it uses a fixed stripe width (padding as necessary with zeros). This allows a dRAID vdev to be sequentially resilvered, however the fixed stripe width significantly effects both usable capacity and IOPS. For example, with the default D=8 and 4k disk sectors the minimum allocation size is 32k. If using compression, this relatively large allocation size can reduce the effective compression ratio. When using ZFS volumes and dRAID the default volblocksize property is increased to account for the allocation size. If a dRAID pool will hold a significant amount of small blocks, it is recommended to also add a mirrored special vdev to store those blocks. In regards to IO/s, performance is similar to raidz since for any read all D data disks must be accessed. Delivered random IOPS can be reasonably approximated as floor((N-S)/(D+P))*. In summary dRAID can provide the same level of redundancy and performance as raidz, while also providing a fast integrated distributed spare. Create a dRAID vdev ~~~~~~~~~~~~~~~~~~~ A dRAID vdev is created like any other by using the ``zpool create`` command and enumerating the disks which should be used. :: # zpool create draid[1,2,3] Like raidz, the parity level is specified immediately after the ``draid`` vdev type. However, unlike raidz additional colon separated options can be specified. The most important of which is the ``:s`` option which controls the number of distributed hot spares to create. By default, no spares are created. The ``:d`` option can be specified to set the number of data devices to use in each RAID stripe (D+P). When unspecified reasonable defaults are chosen. :: # zpool create draid[][:d][:c][:s] - **parity** - The parity level (1-3). Defaults to one. - **data** - The number of data devices per redundancy group. In general a smaller value of D will increase IOPS, improve the compression ratio, and speed up resilvering at the expense of total usable capacity. Defaults to 8, unless N-P-S is less than 8. - **children** - The expected number of children. Useful as a cross-check when listing a large number of devices. An error is returned when the provided number of children differs. - **spares** - The number of distributed hot spares. Defaults to zero. For example, to create an 11 disk dRAID pool with 4+1 redundancy and a single distributed spare the command would be: :: # zpool create tank draid:4d:1s:11c /dev/sd[a-k] # zpool status tank pool: tank state: ONLINE config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 draid1:4d:11c:1s-0 ONLINE 0 0 0 sda ONLINE 0 0 0 sdb ONLINE 0 0 0 sdc ONLINE 0 0 0 sdd ONLINE 0 0 0 sde ONLINE 0 0 0 sdf ONLINE 0 0 0 sdg ONLINE 0 0 0 sdh ONLINE 0 0 0 sdi ONLINE 0 0 0 sdj ONLINE 0 0 0 sdk ONLINE 0 0 0 spares draid1-0-0 AVAIL Note that the dRAID vdev name, ``draid1:4d:11c:1s``, fully describes the configuration and all of disks which are part of the dRAID are listed. Furthermore, the logical distributed hot spare is shown as an available spare disk. Rebuilding to a Distributed Spare ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ One of the major advantages of dRAID is that it supports both sequential and traditional healing resilvers. When performing a sequential resilver to a distributed hot spare the performance scales with the number of disks divided by the stripe width (D+P). This can greatly reduce resilver times and restore full redundancy in a fraction of the usual time. For example, the following graph shows the observed sequential resilver time in hours for a 90 HDD based dRAID filled to 90% capacity. |draid-resilver| When using dRAID and a distributed spare, the process for handling a failed disk is almost identical to raidz with a traditional hot spare. When a disk failure is detected the ZFS Event Daemon (ZED) will start rebuilding to a spare if one is available. The only difference is that for dRAID a sequential resilver is started, while a healing resilver must be used for raidz. :: # echo offline >/sys/block/sdg/device/state # zpool replace -s tank sdg draid1-0-0 # zpool status pool: tank state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver (draid1:4d:11c:1s-0) in progress since Tue Nov 24 14:34:25 2020 3.51T scanned at 13.4G/s, 1.59T issued 6.07G/s, 6.13T total 326G resilvered, 57.17% done, 00:03:21 to go config: NAME STATE READ WRITE CKSUM tank DEGRADED 0 0 0 draid1:4d:11c:1s-0 DEGRADED 0 0 0 sda ONLINE 0 0 0 (resilvering) sdb ONLINE 0 0 0 (resilvering) sdc ONLINE 0 0 0 (resilvering) sdd ONLINE 0 0 0 (resilvering) sde ONLINE 0 0 0 (resilvering) sdf ONLINE 0 0 0 (resilvering) spare-6 DEGRADED 0 0 0 sdg UNAVAIL 0 0 0 draid1-0-0 ONLINE 0 0 0 (resilvering) sdh ONLINE 0 0 0 (resilvering) sdi ONLINE 0 0 0 (resilvering) sdj ONLINE 0 0 0 (resilvering) sdk ONLINE 0 0 0 (resilvering) spares draid1-0-0 INUSE currently in use While both types of resilvering achieve the same goal it's worth taking a moment to summarize the key differences. - A traditional healing resilver scans the entire block tree. This means the checksum for each block is available while it's being repaired and can be immediately verified. The downside is this creates a random read workload which is not ideal for performance. - A sequential resilver instead scans the space maps in order to determine what space is allocated and what must be repaired. This rebuild process is not limited to block boundaries and can sequentially reads from the disks and make repairs using larger I/Os. The price to pay for this performance improvement is that the block checksums cannot be verified while resilvering. Therefore, a scrub is started to verify the checksums after the sequential resilver completes. For a more in depth explanation of the differences between sequential and healing resilvering check out these `sequential resilver`_ slides which were presented at the OpenZFS Developer Summit. Rebalancing ~~~~~~~~~~~ Distributed spare space can be made available again by simply replacing any failed drive with a new drive. This process is called rebalancing and is essentially a resilver. When performing rebalancing a healing resilver is recommended since the pool is no longer degraded. This ensures all checksums are verified when rebuilding to the new disk and eliminates the need to perform a subsequent scrub of the pool. :: # zpool replace tank sdg sdl # zpool status pool: tank state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Tue Nov 24 14:45:16 2020 6.13T scanned at 7.82G/s, 6.10T issued at 7.78G/s, 6.13T total 565G resilvered, 99.44% done, 00:00:04 to go config: NAME STATE READ WRITE CKSUM tank DEGRADED 0 0 0 draid1:4d:11c:1s-0 DEGRADED 0 0 0 sda ONLINE 0 0 0 (resilvering) sdb ONLINE 0 0 0 (resilvering) sdc ONLINE 0 0 0 (resilvering) sdd ONLINE 0 0 0 (resilvering) sde ONLINE 0 0 0 (resilvering) sdf ONLINE 0 0 0 (resilvering) spare-6 DEGRADED 0 0 0 replacing-0 DEGRADED 0 0 0 sdg UNAVAIL 0 0 0 sdl ONLINE 0 0 0 (resilvering) draid1-0-0 ONLINE 0 0 0 (resilvering) sdh ONLINE 0 0 0 (resilvering) sdi ONLINE 0 0 0 (resilvering) sdj ONLINE 0 0 0 (resilvering) sdk ONLINE 0 0 0 (resilvering) spares draid1-0-0 INUSE currently in use After the resilvering completes the distributed hot spare is once again available for use and the pool has been restored to its normal healthy state. .. |draid1| image:: /_static/img/raidz_draid.png .. |draid-resilver| image:: /_static/img/draid-resilver-hours.png .. _dRAID: https://docs.google.com/presentation/d/1uo0nBfY84HIhEqGWEx-Tbm8fPbJKtIP3ICo4toOPcJo/edit .. _sequential resilver: https://docs.google.com/presentation/d/1vLsgQ1MaHlifw40C9R2sPsSiHiQpxglxMbK2SMthu0Q/edit#slide=id.g995720a6cf_1_39 .. _custom packages: ../../Developer Resources/Custom Packages.html Caching and Auxiliary Devices ============================= ZFS has one read cache in RAM (the ARC), an optional second read cache on disk (L2ARC), and two kinds of write-path device that are often mistaken for caches but are not: the separate intent log (SLOG) and the special allocation class. Adding devices is easy and undoing it is not, so it is worth knowing which problem each one actually solves. ARC ~~~ The Adaptive Replacement Cache is ZFS' in-memory cache. It keeps both recently used and frequently used blocks, and adapts the balance between them, which is why it usually outperforms a simple LRU page cache. Per dataset, ``primarycache`` controls what may enter it: ``all`` (the default), ``metadata``, or ``none``. .. code:: bash zfs set primarycache=metadata pool/bigsequentialdata zarcsummary # arc_summary before OpenZFS 2.4.0 zarcstat 1 # arcstat before OpenZFS 2.4.0 cat /proc/spl/kstat/zfs/arcstats RAM is by far the most effective ZFS "tuning knob". Before adding any cache device, check whether the ARC is simply too small — see :doc:`Module Parameters ` for ``zfs_arc_max`` and friends. L2ARC (``cache`` vdev) ~~~~~~~~~~~~~~~~~~~~~~ A cache device is a second-level read cache between RAM and the pool. It helps when the working set is much larger than RAM and the workload is **random reads of mostly static content**. It does nothing for writes. .. code:: bash zpool add pool cache nvme0n1 zfs set secondarycache=metadata pool/data # all | metadata | none Properties of cache devices: * They cannot be mirrored or part of a raidz group. A read error is simply reissued against the pool, so losing one is harmless. * Their contents survive reboots (persistent L2ARC) and are restored asynchronously on import. This can be disabled with ``l2arc_rebuild_enabled=0``. Devices smaller than 1 GiB do not get the metadata needed for a rebuild. * **L2ARC costs RAM.** Every cached block needs a header in the ARC, so an oversized L2ARC shrinks the cache that actually matters. On a RAM-constrained system an L2ARC can make things slower. SLOG (``log`` vdev) ~~~~~~~~~~~~~~~~~~~ The ZFS Intent Log satisfies POSIX durability requirements: ``fsync()`` and ``O_SYNC`` writes must be on stable storage before the call returns. The ZIL exists on every pool — by default it is allocated from blocks in the main pool. A ``log`` vdev just moves it to a dedicated device. .. code:: bash zpool add pool log mirror nvme0n1 nvme1n1 Consequences worth being precise about: * A SLOG is **not a write cache**. Asynchronous writes never touch the ZIL; they are aggregated in memory and written out at the next transaction group. If your workload has no synchronous writes, a SLOG changes nothing. * It is only ever *read* after a crash, to replay what had not yet been committed. * It should be a low-latency device with power-loss protection. Consumer SSDs without it will happily report data as stable that a power cut then loses. * Multiple log devices load-balance and can be mirrored; raidz is not supported for the intent log. Typical beneficiaries: NFS servers, databases, and VM hosts with sync-heavy guests. The ``sync`` and ``logbias`` dataset properties control the behavior — ``logbias=throughput`` bypasses the log devices for a dataset, and ``sync=disabled`` skips the ZIL entirely, trading recent-write durability for speed. Special vdev (``special`` allocation class) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ A special vdev is not a cache either — it is storage that permanently holds a specific class of blocks: metadata, the indirect blocks of user data, deduplication tables, and optionally small file blocks. .. code:: bash zpool add pool special mirror nvme0n1 nvme1n1 Putting metadata on flash is usually the single biggest win available to a pool of spinning disks: directory traversal, ``zfs list``, scrubs and resilvers all become dramatically faster. But because the blocks live *only* there, the vdev must be as redundant as the rest of the pool, and on a raidz pool it can never be removed again. It has enough sharp edges to deserve its own page — see :doc:`Special vdev `. Choosing between them ~~~~~~~~~~~~~~~~~~~~~ +-------------------------+-------------------------------+------------------+ | Symptom | What helps | Removable later? | +=========================+===============================+==================+ | Slow reads, ARC too | More RAM, then L2ARC | yes | | small for working set | | | +-------------------------+-------------------------------+------------------+ | Slow synchronous writes | SLOG with power-loss | yes | | (NFS, databases, VMs) | protection | | +-------------------------+-------------------------------+------------------+ | Slow metadata operations| special vdev (mirrored) | only if the pool | | on spinning disks | | has no raidz | +-------------------------+-------------------------------+------------------+ | Dedup table thrashing | dedup vdev (mirrored) | same as special | +-------------------------+-------------------------------+------------------+ Further reading ~~~~~~~~~~~~~~~ * `zpoolconcepts(7) `__ — ``Intent Log``, ``Cache Devices``, ``Special Allocation Class`` * `zfsprops(7) `__ — ``primarycache``, ``secondarycache``, ``special_small_blocks``, ``logbias``, ``sync`` * :doc:`VDEVs `, :doc:`Workload Tuning `, :doc:`Module Parameters ` Special vdev ============ A ``special`` vdev is a top-level vdev that holds a specific *allocation class* rather than general pool data: metadata, the indirect blocks of user data, deduplication tables, and — optionally, per dataset — small file blocks. It is not a cache. The blocks routed there exist **only** there. That single fact drives everything else on this page. .. warning:: A ``special`` vdev must be at least as redundant as the pool's normal vdevs. Losing it loses the pool, exactly as losing any other top-level vdev would. Never add one as a single device to a redundant pool. Why it helps ~~~~~~~~~~~~ On a pool of spinning disks, metadata access is the dominant source of random I/O: directory traversal, ``zfs list``, ``find``, backup scans, and — most visibly — scrub and resilver, which must walk the whole block tree. Moving that class onto flash converts those random reads into something the pool can actually serve, without paying for an all-flash pool. This is usually a larger and more reliable win than adding an L2ARC, because it is a permanent placement rather than a cache that must warm up and that costs ARC headers. See :doc:`Caching and Auxiliary Devices `. Adding one ~~~~~~~~~~ .. code:: bash zpool add pool special mirror nvme0n1 nvme1n1 zpool list -v pool # per-vdev usage, special class included A pool must already have at least one normal vdev before ``special`` or ``dedup`` devices can be assigned. Multiple special vdevs load-balance allocations between them. What actually lands there ~~~~~~~~~~~~~~~~~~~~~~~~~ The selection happens per block, in this order: #. **Deduplication tables** go to a ``dedup`` vdev if the pool has one; otherwise to the special vdev. Setting the ``zfs_ddt_data_is_special`` module parameter to 0 keeps them out of the special class. #. **Metadata** — every object type ZFS marks as metadata — goes to the special class. #. **Indirect blocks of user data** (any block above level 0) go there too, controlled by ``zfs_user_indirect_is_special`` (default on). These are the block pointers, so this is much of what makes tree walks fast. #. **Small user data blocks** go there only if the dataset opts in via ``special_small_blocks`` — and only while the class is below its metadata reserve, see below. #. Everything else goes to the normal class. The **ZIL** is a separate case. An intent log block is allocated from, in order: a dedicated ``log`` vdev, then log space reserved on the special vdev, then the special class proper, then log space reserved on normal vdevs, then the normal class. So a special vdev absorbs the ZIL when there is no SLOG, but a SLOG always wins. Small blocks: ``special_small_blocks`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. code:: bash zfs set special_small_blocks=32K pool/data Blocks whose size is less than or equal to this value — measured **after compression and encryption** — are placed in the special class. The default is 0, meaning the feature is off; the maximum is the maximum block size (16 MiB). It is a per-dataset property, so you can enable it only where it pays. Two things surprise people here: **It is compared against the physical size.** A 128 KiB record that compresses to 20 KiB counts as a 20 KiB block. Setting the threshold near your ``recordsize`` will therefore pull far more onto the special vdev than expected — for a well-compressing dataset, potentially all of it. **There is a metadata reserve.** Small blocks are only accepted while the special class is below ``100 - zfs_special_class_metadata_reserve_pct`` percent full — 75% by default. The last quarter is kept for metadata alone. This is a floor for metadata, not a ceiling for the vdev: metadata itself may fill it completely. When it fills up ~~~~~~~~~~~~~~~~ Allocations that no longer fit spill back into the normal class. Nothing fails, and no data is lost — but the metadata written from that point on lands on the slow disks, and the benefit quietly degrades. Existing blocks are never migrated, in either direction. This makes monitoring the fill level worthwhile: .. code:: bash zpool list -v pool Sizing is workload-dependent and hard to predict from first principles. The honest approach is to measure the pool you actually have — ``zdb -bbb pool`` reports space usage broken down by block type, from which the metadata total can be read off — and then leave generous headroom, because the cost of guessing low is a vdev you cannot shrink. Removing one ~~~~~~~~~~~~ .. code:: bash zpool remove pool special-1 This works only under the same constraints as any top-level vdev removal: the pool must contain **no** raidz or draid top-level vdev, all top-level vdevs must share the same ``ashift``, keys for encrypted datasets must be loaded, and the ``device_removal`` feature must be enabled. The practical consequence is blunt: **on a raidz pool, a special vdev can never be removed.** Adding one is a permanent decision. See :doc:`Changing Pool Layout `. Related vdev types ~~~~~~~~~~~~~~~~~~ ``dedup`` The same idea restricted to deduplication tables. Use it when dedup is enabled and you want its table isolated from metadata. Same redundancy requirement, same removal constraints. See :doc:`Deduplication `. ``log`` (SLOG) Not an allocation class in the same sense — it holds the intent log and is read only after a crash. Losing a modern SLOG costs at most the last few seconds of synchronous writes, not the pool. ``cache`` (L2ARC) A genuine cache: a second copy of data that also lives in the pool. Losing it is harmless. Further reading ~~~~~~~~~~~~~~~ * `zpoolconcepts(7) `__ — ``Special Allocation Class`` * `zfsprops(7) `__ — ``special_small_blocks`` * `zpool-add(8) `__, `zpool-remove(8) `__ * :doc:`VDEVs `, :doc:`Caching and Auxiliary Devices ` Changing Pool Layout ==================== Pools are not immutable, but the operations available differ sharply by vdev type — and some of them are one-way. This page collects what can be added, grown, replaced, and taken back out. See :doc:`VDEVs ` for the vocabulary and :doc:`RAIDZ ` for raidz geometry. Adding capacity ~~~~~~~~~~~~~~~ ``zpool add`` appends a new top-level vdev. The pool stripes across all top-level vdevs, so the new one starts taking writes immediately, but existing data is not rebalanced. .. code:: bash zpool add pool mirror sdc sdd zpool add -n pool mirror sdc sdd # dry run — print the resulting layout Match the new vdev's redundancy to the existing ones. ``zpool add`` will warn about a mismatch (adding a bare disk to a mirrored pool, say); overriding that warning with ``-f`` gives the pool a single point of failure that cannot be undone if raidz is involved. Growing existing vdevs ~~~~~~~~~~~~~~~~~~~~~~ **Bigger disks.** Replace each device in a vdev with a larger one, one at a time, waiting for each resilver. Space becomes available only once *every* device in that mirror or raidz group has been replaced. .. code:: bash zpool set autoexpand=on pool # default is off zpool replace pool sda sdX # ...or, after growing a LUN underneath: zpool online -e pool sda ``autoexpand=on`` resizes the pool automatically when a device grows; otherwise ``zpool online -e`` does it explicitly. **RAIDZ expansion.** A device can be attached to an existing raidz vdev to widen it: .. code:: bash zpool attach pool raidz2-0 sdX This reads all allocated data and rewrites it across the wider group; ``zpool status`` shows the progress. Redundancy is maintained throughout, and the expansion pauses if a disk fails until the vdev is healthy again. Points to be clear about: * Fault tolerance does not change — a RAID-Z2 stays a RAID-Z2. * Old blocks keep their original data-to-parity ratio, just spread over more disks; only new blocks use the wider ratio. Reported free space may therefore be slightly pessimistic until old data is rewritten. * A raidz vdev can be expanded more than once. Mirrors ~~~~~~~ ``zpool attach`` on a mirror or a plain device adds another side: a single disk becomes a two-way mirror, a two-way becomes three-way. The new device resilvers immediately and any running scrub is cancelled. ``zpool detach`` removes a side. .. code:: bash zpool attach pool sda sdb # sda becomes a two-way mirror zpool attach -s pool sda sdb # sequential reconstruction, then auto-scrub zpool detach pool sdb ``zpool split`` peels one device off every mirror into a new pool — an instant, consistent copy of the whole pool. Every top-level vdev must be a mirror and no resilver may be in progress. .. code:: bash zpool split pool newpool Replacing a failed device ~~~~~~~~~~~~~~~~~~~~~~~~~ .. code:: bash zpool replace pool sda sdX # new device in a different slot zpool replace pool sda # same slot, device already swapped zpool set autoreplace=on pool ``autoreplace=on`` makes ZFS format and replace any new device appearing in the physical location of one that previously belonged to the pool; the default is ``off``. See :doc:`Scrub and Resilver ` for what happens next. Hot spares ~~~~~~~~~~ A ``spare`` vdev sits idle until an active device fails, then takes its place automatically. .. code:: bash zpool create pool mirror sda sdb spare sdc zpool add pool spare sdd zpool remove pool sdd Once a spare steps in, a new ``spare`` vdev appears in the configuration and stays there until the original device is replaced for real — at which point the spare goes back to being available. So a spare is a stopgap that buys time, not a replacement: the failed disk still has to be swapped. .. code:: bash zpool replace pool sda sdNEW # then the spare frees itself zpool detach pool sdc # cancel an in-progress spare replacement Detaching the *original faulted device* instead makes the spare assume its place permanently, and removes it from the spare list of every pool using it. Spares can be shared between pools, and that carries real risk: a pool currently using a shared spare cannot be exported, and if two pools are imported on different hosts and both lose a device at once, both may claim the same spare — which is not detected and can corrupt data. Share spares only within one host. dRAID has its own integrated distributed spare capacity, which resilvers far faster; see :doc:`dRAID `. Removing devices ~~~~~~~~~~~~~~~~ ``zpool remove`` handles hot spares, cache and log devices trivially. Removing a **top-level data vdev** is the constrained case: it evacuates the vdev by copying its allocated data elsewhere, in the background, and permanently shrinks the pool. .. code:: bash zpool remove pool mirror-1 zpool remove -n -p pool mirror-1 # estimate the mapping table cost first zpool status pool # follow the evacuation It works only if all of these hold: * the pool contains **no** top-level raidz or draid vdev, * all top-level vdevs have the same ``ashift``, * keys for all encrypted datasets are loaded, * the ``device_removal`` feature is enabled. Removal leaves behind an in-memory mapping table for the relocated blocks, which costs RAM for the life of the pool — ``-n -p`` reports how much. An I/O error during evacuation cancels the removal. To take a device out of a mirror, use ``zpool detach``, not ``remove``. .. warning:: **A raidz pool cannot shrink.** If the pool has any raidz or draid top-level vdev, no top-level vdev can ever be removed — including a ``special`` or ``dedup`` vdev added later, and including a plain disk added by mistake. The only fix is to recreate the pool and restore the data. Always ``zpool add -n`` first. Import and export ~~~~~~~~~~~~~~~~~ .. code:: bash zpool export pool zpool import # list importable pools zpool import pool zpool import -d /dev/disk/by-id pool zpool import -o readonly=on -N pool Exporting flushes everything and marks the pool as cleanly detached, which is what makes moving it to another system safe. ``-d`` chooses which device paths to scan — using ``/dev/disk/by-id`` avoids the classic problem of ``sdX`` names shuffling between boots. See the FAQ on :doc:`selecting /dev/ names `. Checkpoints ~~~~~~~~~~~ A pool checkpoint is a whole-pool undo point, intended for risky administrative operations such as an upgrade: .. code:: bash zpool checkpoint pool # ...if it went badly: zpool export pool zpool import --rewind-to-checkpoint pool # ...if it went well: zpool checkpoint -d pool While a checkpoint exists, ``zpool remove``, ``attach``, ``detach``, ``split`` and ``reguid`` are all prohibited, and the space it pins may break reservation boundaries on a pool short of free space. ``zpool status`` shows that a checkpoint exists, and ``zpool list`` shows how much space it holds. Discard it as soon as the operation is confirmed good. Further reading ~~~~~~~~~~~~~~~ * `zpool-add(8) `__, `zpool-attach(8) `__, `zpool-detach(8) `__ * `zpool-replace(8) `__, `zpool-remove(8) `__, `zpool-split(8) `__ * `zpool-import(8) `__, `zpool-export(8) `__, `zpool-checkpoint(8) `__ * `zpoolprops(7) `__ — ``autoexpand``, ``autoreplace`` Feature Flags ============= ZFS on-disk formats were originally versioned with a single number, which increased whenever the format changed. The numbered approach was suitable when development of ZFS was driven by a single organisation. For distributed development of OpenZFS, version numbering was unsuitable. Any change to the number would have required agreement, across all implementations, of each change to the on-disk format. OpenZFS feature flags – an alternative to traditional version numbering – allow **a uniquely named pool property for each change to the on-disk format**. This approach supports: - format changes that are independent - format changes that depend on each other. Feature states -------------- Each feature is exposed as a pool property named ``feature@`` and is in one of three states: ``disabled`` The on-disk format change has not been made and will not be, unless an administrator enables it. ``enabled`` Marked as enabled, but the on-disk format change has **not happened yet**. The pool can still be imported by software that does not know the feature — until something triggers the change, which may happen at any time and moves the feature to ``active``. ``active`` The format change is in effect. Software must support the feature to import the pool read-write. .. code:: bash zpool get all pool | grep feature@ zpool get feature@large_blocks pool **Features cannot be disabled once enabled.** Some can return from ``active`` to ``enabled`` when whatever used them is freed — for example ``block_cloning`` once the last cloned block is gone — but that is feature-specific. Enabling a feature automatically enables anything it depends on. Read-only compatibility ----------------------- Some format changes do not prevent other software from *reading* the pool. If every unsupported feature on a pool is read-only compatible, the pool can still be imported read-only: .. code:: bash zpool import -o readonly=on pool For each unsupported feature on an imported pool, the property ``unsupported@`` explains why the import was allowed: ``inactive`` (the feature is only ``enabled``, so the format is still compatible) or ``readonly`` (it is read-only compatible and the pool was imported read-only). Compatibility ------------- Where all *features* that are used by a pool are supported by multiple implementations of OpenZFS, the on-disk format is portable across those implementations. Features that are exclusive when enabled should be periodically ported to all distributions. Compatibility feature sets -------------------------- The ``compatibility`` pool property pins a pool to a named on-disk format, so it stays importable somewhere else — an older release, another OS, or a bootloader. ``off`` (default) No restriction; any feature may be enabled. ``legacy`` No features may be enabled at all — neither by ``zpool upgrade`` nor by setting ``feature@…=enabled``, and the pool cannot be upgraded to a newer on-disk version. A deliberate safety catch. ``file[,file…]`` One or more feature-set files. Only features present in **all** of the listed files may be enabled. .. code:: bash zpool create -o compatibility=grub2-2.12 bpool ... zpool set compatibility=openzfs-2.1-linux tank zpool get compatibility tank Names are resolved relative to ``/etc/zfs/compatibility.d`` (local, takes precedence) or ``/usr/share/zfs/compatibility.d`` (shipped by the distribution), or given as absolute paths. The shipped sets cover OpenZFS releases per platform (``openzfs-2.1-linux``, ``openzfs-2.2``, ``openzfs-2.3``, ``openzfs-2.4``, ``openzfs-2.0-freebsd``…), older ZoL and FreeBSD versions, GRUB (``grub2-2.06``, ``grub2-2.12``), and yearly baselines (``compat-2018`` … ``compat-2021``). .. code:: bash ls /usr/share/zfs/compatibility.d/ Two practical uses dominate: a **boot pool** restricted to what the bootloader understands, and a pool that must remain importable on an older system. Setting ``compatibility`` also stops ``zpool status`` from nagging about disabled features that are outside the chosen set. Upgrading a pool ---------------- .. code:: bash zpool upgrade # list pools with features not yet enabled zpool upgrade -v # what this build supports zpool upgrade pool # enable them on one pool zpool upgrade -a # ...on every pool ``zpool upgrade`` enables all supported features, subject to ``compatibility``: if a feature set is in force only features in that set are enabled, and with ``legacy`` nothing happens at all. .. warning:: **Upgrading is one-way.** Features cannot be disabled afterwards. Once enabled features become ``active``, the pool is no longer importable by software that does not support them — which includes an older kernel you might need to boot back into, a rescue USB stick, and the bootloader on a root pool. Upgrade deliberately, not reflexively because ``zpool status`` suggested it. A pool that is not upgraded keeps working; it just does not offer the newer features. Note that ``zpool status`` reporting "some supported features are not enabled" is informational. On a root or boot pool in particular, check what your bootloader supports first — see the :doc:`Root on ZFS guides `. ``zfs upgrade`` is a different thing ------------------------------------ Datasets carry their own on-disk version, independent of the pool's features. .. code:: bash zfs upgrade # file systems not at the latest version zfs upgrade -v # versions this build supports zfs upgrade -r pool/data # upgrade a subtree zfs upgrade -a # everything The same one-way warning applies, with an extra consequence: after upgrading, ``zfs send`` streams generated from *new* snapshots of those file systems cannot be received by older ZFS versions either. If you replicate to an older system, that will break the replication, not just the local mount. Reference materials ------------------- `ZFS Feature Flags `_ (Christopher Siden, 2012-01, in the Internet Archive Wayback Machine) in particular: "… Legacy version numbers still exist for pool versions 1-28 …". `zpool-features(7) man page <../../man/master/7/zpool-features.7.html>`_ - OpenZFS `zpool-features `__ (5) – illumos Feature flags implementation per OS ----------------------------------- .. raw:: html
.. raw:: html :file: ../../_build/zfs_feature_matrix.html .. raw:: html
Datasets ======== Datasets are how pool space is carved up and administered: their properties, their point-in-time copies, and the limits placed on them. .. toctree:: :maxdepth: 1 Datasets and Properties ZVOLs Snapshots and Clones Quotas and Reservations ACLs Datasets and Properties ======================= A pool is a pile of space; datasets are how it is carved up. Creating a dataset is cheap and does not preallocate anything, so the usual ZFS practice is to create many of them — one per workload, per user, per application — rather than one big file system with directories. Dataset types ~~~~~~~~~~~~~ **File system** A POSIX file system, mounted somewhere. The default type. **Volume (zvol)** A block device exported under ``/dev/zvol//``. Used for virtual machine disks, iSCSI targets, swap, or file systems ZFS does not implement. **Snapshot** and **bookmark** Read-only, covered in :doc:`Snapshots, Clones and Bookmarks `. .. code:: bash zfs create pool/home zfs create pool/home/alice zfs create -V 40G pool/vm/disk0 # a zvol zfs list # file systems and volumes zfs list -t all -r pool # everything, recursively zfs destroy pool/home/alice Datasets form a hierarchy that mirrors their names, and most properties are inherited down that hierarchy — which is the main reason to nest them deliberately. Properties ~~~~~~~~~~ Everything about a dataset's behavior is a property: where it mounts, how it compresses, whether it is read-only, how much space it may use. .. code:: bash zfs get all pool/home/alice zfs get -r compression,recordsize pool zfs get -o name,property,value,source -s local all pool/home zfs set compression=zstd pool/home zfs inherit compression pool/home/alice # fall back to the parent's value zfs inherit -S compression pool/home/alice # revert to the received value Some properties are read-only (``used``, ``available``, ``creation``, ``compressratio``), some can only be set at creation time (``casesensitivity``, ``normalization``, ``encryption``, ``volblocksize``), and the rest can be changed at any time. Changes to properties that affect how data is written — ``recordsize``, ``compression``, ``checksum``, ``dedup``, ``copies`` — apply only to newly-written blocks. Property sources ^^^^^^^^^^^^^^^^ ``zfs get`` reports where each value came from: ``local`` Set directly on this dataset. ``inherited from `` Set on an ancestor. ``default`` Never set anywhere; the built-in default. ``temporary`` Set for the current mount only, via a mount option. ``received`` Carried in a ``zfs receive`` stream. ``none`` Read-only property with no source. This is the first thing to check when a dataset does not behave as expected — a value inherited from three levels up is easy to miss. Properties worth knowing ~~~~~~~~~~~~~~~~~~~~~~~~ **Layout and mounting** ``mountpoint`` Where the file system mounts. ``legacy`` hands control to ``/etc/fstab`` or ``mount``; ``none`` means it is not mounted. Inherited children extend the parent's path automatically. ``canmount`` ``on``, ``off`` or ``noauto``. ``off`` makes a dataset a pure property-inheritance container that is never mounted, while still having an inheritable ``mountpoint``. ``noauto`` means it mounts only when asked explicitly — the basis of boot environments. ``readonly`` Blocks writes to the dataset. Standard on replication targets. **Data layout** ``recordsize`` The maximum block size for a file system. Default 128 KiB; matching it to the application's I/O size matters for databases and VM images. ``volblocksize`` The fixed block size of a zvol. Set at creation only. ``compression``, ``checksum``, ``dedup``, ``copies`` See :doc:`Compression `, :doc:`Checksums ` and :doc:`Deduplication `. **Behavior** ``atime`` / ``relatime`` Access-time updates. ``atime=off`` removes a write for every read; ``relatime=on`` is the usual compromise. ``sync`` ``standard``, ``always``, or ``disabled``. ``disabled`` ignores fsync — it does not corrupt the pool, but it does lose recently written data on a crash. ``exec``, ``setuid``, ``devices`` The usual mount-option semantics, as properties. ``sharenfs``, ``sharesmb`` Export the dataset; see :doc:`Sharing datasets `. ``quota``, ``refquota``, ``reservation``, ``refreservation`` See :doc:`Quotas and Reservations `. **Space accounting** ``used`` Space charged to this dataset and its descendants. ``referenced`` Space reachable from this dataset alone — what a snapshot of it would reference. ``available`` Space still usable, after quotas and reservations. ``logicalused`` / ``logicalreferenced`` The same before compression. ``usedby*`` ``usedbydataset``, ``usedbysnapshots``, ``usedbychildren`` and ``usedbyrefreservation`` break ``used`` down — the fastest way to find where space went. .. code:: bash zfs list -o name,used,usedbydataset,usedbysnapshots,usedbychildren -r pool User properties ~~~~~~~~~~~~~~~ Arbitrary properties can be attached to datasets for use by tooling. They have no effect on ZFS itself. A user property name must contain a colon, which is what distinguishes it from a native one; the convention is ``module:property``, with the module part typically a reversed domain name. .. code:: bash zfs set com.example:backup=daily pool/home zfs get -r -o name,value com.example:backup pool Because they inherit like native properties, user properties are a convenient way to drive snapshot and replication policy from the dataset tree itself. Further reading ~~~~~~~~~~~~~~~ * `zfsconcepts(7) `__ * `zfsprops(7) `__ — the complete property reference * `zfs-create(8) `__, `zfs-get(8) `__, `zfs-set(8) `__, `zfs-inherit(8) `__ * :doc:`Workload Tuning ` ZVOLs ===== A zvol is a dataset exported as a block device instead of a file system. It gets everything else a dataset gets — checksums, compression, snapshots, clones, replication, encryption — while presenting a raw device to whatever consumes it: a virtual machine disk, an iSCSI LUN, a file system ZFS does not implement, or swap. .. code:: bash zfs create -V 40G pool/vm/disk0 ls -l /dev/zvol/pool/vm/disk0 Sizing: ``volsize`` and ``volblocksize`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ``volsize`` The logical size of the device. It must be a multiple of ``volblocksize`` and cannot be zero. It can be changed later, but shrinking it is as dangerous as shrinking any block device under a live file system. ``volblocksize`` The fixed block size. **Set at creation only** — there is no changing it afterwards short of recreating the volume and copying the data. Valid values are powers of two from 512 B to 16 MiB; the default is 16 KiB. Unlike a file system's ``recordsize``, which is a maximum that ZFS varies per file, ``volblocksize`` is fixed for every block in the volume. That makes it the single most consequential zvol decision. The guiding rule is to match the consumer's I/O size: * too **small**, and each guest I/O costs more metadata, and RAIDZ space efficiency collapses — small blocks on RAIDZ approach mirror-level overhead (see :doc:`RAIDZ `); * too **large**, and every small write becomes a read-modify-write of the whole block, and compression works on data the guest did not change. For a guest file system using 4 KiB blocks, values between 8 KiB and 64 KiB are the usual range; the default 16 KiB is a reasonable starting point. See :doc:`Workload Tuning ` for per-workload advice. Sparse versus thick ~~~~~~~~~~~~~~~~~~~ By default, creating a volume also establishes a ``refreservation`` equal to its size, so the space is guaranteed up front. The reservation is kept equal to the logical size deliberately: without it the volume can run out of space mid-write, which most consumers handle badly. ``zfs create -s`` creates a **sparse** volume with no reservation: .. code:: bash zfs create -s -V 100G pool/vm/thin # sparse zfs create -V 100G pool/vm/thick # reserved zfs set refreservation=auto pool/vm/thin # make it thick afterwards zfs set refreservation=none pool/vm/thick # make it sparse afterwards ``refreservation=auto`` is supported only on volumes and reserves enough for the whole ``volsize`` plus metadata. Sparse volumes let you overcommit, at the price that a write can fail with ``ENOSPC`` when the pool fills — and the guest will see that as a hardware error, not as a full disk. Overcommitting means you must monitor pool space. Note also that snapshots of a thick volume need room for the reservation, so ``volsize`` is not the whole cost. See :doc:`Quotas and Reservations `. How the device appears ~~~~~~~~~~~~~~~~~~~~~~ ``volmode`` controls what the OS sees: ``full`` (alias ``geom``) A complete block device, partitions scanned and exposed. ``dev`` The device without its partitions. Usually what you want for a VM disk or a LUN — the host has no business scanning the guest's partition table. ``none`` Not exposed outside ZFS at all. Still snapshottable, clonable and replicable, which suits a pure backup target. ``default`` Follow the system-wide ``zvol_volmode`` tunable. Snapshot devices appear under ``/dev/zvol/`` only when ``snapdev`` is ``visible``; the default is ``hidden``. There is no ``.zfs/snapshot`` directory for volumes — to reach the contents of a volume snapshot, clone it and mount the clone. Reservations, snapshots and space ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ A zvol's ``used`` can exceed what the guest believes it is using, sometimes dramatically. The reason is that ZFS has no way to know a block has been freed inside the guest's file system unless it is told: file systems tend to allocate into fresh blocks rather than reuse freed ones, so over time ZFS ends up referencing every block the guest ever touched. The fix is discard/TRIM from the consumer: * mount the guest file system with ``discard``, or run ``fstrim`` periodically; * issue the trim **before** taking a snapshot, or the snapshot pins blocks the guest already considers free. See :doc:`TRIM ` and the :doc:`FAQ ` entry on zvol space usage. Swap on a zvol ~~~~~~~~~~~~~~ Possible, but it needs care, and on Linux it can still deadlock under memory pressure — see `issue #7734 `__. If you do it: .. code:: bash zfs create -V 4G -b $(getconf PAGESIZE) \ -o logbias=throughput -o sync=always \ -o primarycache=metadata \ -o com.sun:auto-snapshot=false rpool/swap Match ``volblocksize`` to the page size, keep swap data out of the ARC, and never snapshot a swap device. Further reading ~~~~~~~~~~~~~~~ * `zfsprops(7) `__ — ``volsize``, ``volblocksize``, ``volmode``, ``refreservation``, ``snapdev`` * `zfs-create(8) `__ * :doc:`Datasets and Properties `, :doc:`Sharing Datasets ` * :doc:`Workload Tuning ` — virtual machine and database sections Snapshots, Clones and Bookmarks =============================== Because ZFS never modifies a block in place, keeping an old version of a dataset costs nothing more than a reference to the blocks that were live at that moment. Snapshots, clones and bookmarks are all built on that. Snapshots ~~~~~~~~~ A snapshot is a read-only copy of a file system or volume at a point in time. Creating one is nearly instantaneous and initially consumes no additional space; it starts to account for space as the live dataset diverges and blocks only the snapshot references can no longer be freed. Snapshots are atomic — they include every system call that completed before that point in time. Snapshots created recursively with ``-r`` are all taken at the same instant, which makes a recursive snapshot usable as one consistent image across several datasets. .. code:: bash # snapshot a single dataset zfs snapshot pool/home/user@monday # snapshot a dataset and all of its descendants, atomically zfs snapshot -r pool/home@monday # list snapshots (they are hidden from plain "zfs list" by default) zfs list -t snapshot -r pool/home ``zfs list`` omits snapshots unless ``-t snapshot`` is given; the pool property ``listsnapshots`` (default ``off``) changes that default for a whole pool. Accessing snapshot contents ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File system snapshots are reachable through the ``.zfs/snapshot`` directory in the root of the file system, without being mounted explicitly — they are mounted on demand and unmounted again after a while. This makes restoring a single file a plain ``cp``: .. code:: bash cp /home/user/.zfs/snapshot/monday/notes.txt /home/user/notes.txt The ``snapdir`` property controls whether the ``.zfs`` directory is ``disabled``, ``hidden`` (the default — it exists but does not show up in ``ls``) or ``visible``. For volumes there is no directory to browse. Snapshot device nodes appear under ``/dev/zvol/`` when the volume's ``snapdev`` property is set to ``visible``; the default is ``hidden``. Rolling back ^^^^^^^^^^^^ ``zfs rollback`` discards everything written since the snapshot and returns the dataset to that state. By default it refuses to roll back to anything but the most recent snapshot: .. code:: bash zfs rollback pool/home/user@monday # only if it is the newest snapshot zfs rollback -r pool/home/user@monday # also destroy newer snapshots and bookmarks zfs rollback -R pool/home/user@monday # additionally destroy clones of those snapshots ``-r`` and ``-R`` do not recurse into child datasets: only direct snapshots of the named file system are destroyed. Rolling a recursive snapshot back completely means rolling back each child dataset individually. The file system must be unmounted (or unmountable) for a rollback; ``-f`` used together with ``-R`` forces the unmount of clones that are about to be destroyed. Space accounting ^^^^^^^^^^^^^^^^ Two properties answer most "why is my pool full" questions: * ``used`` on a snapshot is the space that would be freed if *that snapshot alone* were destroyed — blocks referenced by no other snapshot and not by the live dataset. * ``usedbysnapshots`` on a dataset is the space that would be freed if *all* of its snapshots were destroyed. It is not the sum of the snapshots' ``used`` values, because space can be shared between several snapshots. .. code:: bash zfs list -o name,used,referenced,usedbysnapshots pool/home/user zfs list -t snapshot -o name,used,referenced,written -r pool/home/user ``referenced`` is how much data the snapshot (or dataset) can see, and ``written`` is how much data changed since the previous snapshot — useful for sizing incremental replication. A single old snapshot can pin an arbitrary amount of space: if a large file was deleted after the snapshot was taken, the space stays allocated until the snapshot goes away. Holds and deferred destruction ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ A hold is a named reference that prevents a snapshot from being destroyed. Replication tools use holds so that the snapshot a stream is based on cannot disappear mid-transfer. .. code:: bash zfs hold keep pool/home/user@monday zfs holds pool/home/user@monday zfs release keep pool/home/user@monday While a hold exists, ``zfs destroy`` on that snapshot fails with ``EBUSY``. ``zfs destroy -d`` marks such a snapshot for deferred destruction instead of failing: it disappears automatically once the last hold is released and it has no clones. Comparing snapshots ^^^^^^^^^^^^^^^^^^^ ``zfs diff`` reports what changed between a snapshot and a later snapshot (or the live file system): .. code:: bash zfs diff pool/home/user@monday pool/home/user@tuesday zfs diff pool/home/user@monday Limiting snapshot counts ^^^^^^^^^^^^^^^^^^^^^^^^ ``snapshot_limit`` caps the number of snapshots that may exist on a dataset and its descendants, and ``snapshot_count`` reports the current number. A limit set on a descendant does not override an ancestor's limit — it adds to it. The limit is not enforced against a user who is allowed to change it. The ``snapshots_changed`` property records when a snapshot of the dataset was last created or destroyed, which lets monitoring and backup tooling skip datasets that cannot have changed without walking the whole snapshot list. .. _snapshot-scheduling: Taking them on a schedule ^^^^^^^^^^^^^^^^^^^^^^^^^ ZFS creates snapshots but does not schedule or expire them. Something has to decide when to take one and when to destroy it, or a pool quietly fills with years of hourlies. A timer calling ``zfs snapshot -r`` and a second one destroying anything older than *N* is enough for simple cases. Beyond that, third-party tools handle retention policies, and the same ones usually handle replication — ``sanoid``/``syncoid``, ``zrepl`` and ``zfstools`` are among the established ones. None are part of OpenZFS, so treat them as you would any other dependency holding your backups. Some tools key off the ``com.sun:auto-snapshot`` user property to decide which datasets to include, which is why it appears in examples such as the swap zvol in :doc:`ZVOLs `. See :doc:`Send and Receive ` for the replication side. Clones ~~~~~~ A clone is a *writable* dataset whose initial contents are those of a snapshot. Like a snapshot it is created almost instantly and initially consumes no extra space; it then diverges from its origin as it is written to. .. code:: bash zfs snapshot pool/vm/base@golden zfs clone pool/vm/base@golden pool/vm/guest1 zfs clone pool/vm/base@golden pool/vm/guest2 Clones can only be created from a snapshot, and they create a dependency: the origin snapshot cannot be destroyed while a clone of it exists. The clone's ``origin`` property names the snapshot it came from, and ``zfs destroy`` lists such dependencies when it refuses to remove a snapshot. Promoting a clone ^^^^^^^^^^^^^^^^^ ``zfs promote`` reverses the parent/child relationship: the clone becomes the dataset that owns the shared snapshots, and the former origin becomes a clone of it. That is what makes it possible to retire the original dataset: .. code:: bash zfs promote pool/vm/guest1 zfs destroy -r pool/vm/base # now possible Typical uses of clones are virtual machine images from a common golden image, throwaway copies of a production dataset for testing, and recovering from a mistake without discarding the newer data a rollback would delete. Bookmarks ~~~~~~~~~ A bookmark is a read-only marker for the point in time a snapshot was taken. It is even cheaper than a snapshot — it consumes no additional space at all — but it also holds on to no data: it cannot be browsed, mounted, cloned or rolled back to. .. code:: bash zfs bookmark pool/home/user@monday pool/home/user#monday zfs list -t bookmark -r pool/home/user The point of a bookmark is incremental replication. To send an incremental stream, the sending side normally has to keep the previous snapshot around. If it keeps a bookmark instead, the snapshot itself — and the space it pins — can be destroyed, and ``zfs send -i`` can still produce the incremental stream: .. code:: bash zfs send -i pool/home/user#monday pool/home/user@tuesday | ... A bookmark is initially tied to a snapshot, but it survives the destruction of that snapshot. See :doc:`Send and Receive ` for how they fit into a replication workflow. Further reading ~~~~~~~~~~~~~~~ * `zfsconcepts(7) `__ * `zfs-snapshot(8) `__, `zfs-rollback(8) `__, `zfs-destroy(8) `__ * `zfs-clone(8) `__, `zfs-promote(8) `__ * `zfs-bookmark(8) `__ * `zfs-hold(8) `__, `zfs-diff(8) `__ Quotas and Reservations ======================= Datasets share the pool's free space, so by default any one of them can fill it. Quotas cap what a dataset may consume; reservations guarantee what it may always claim. Both are properties, both are inherited-by-position rather than by value, and both come in two flavours: with descendants and snapshots (``quota``, ``reservation``) or without (``refquota``, ``refreservation``). Quotas ~~~~~~ ``quota`` Hard limit on the space used by the dataset **and all its descendants, including snapshots**. ``refquota`` Hard limit on the dataset's own ``referenced`` space — descendants and snapshots do not count. .. code:: bash zfs set quota=100G pool/home/alice zfs set refquota=80G pool/home/alice zfs get quota,refquota,used,referenced,available pool/home/alice A quota set on a descendant does not override an ancestor's — it adds another limit, and the tightest one wins. Quotas cannot be set on volumes: ``volsize`` already acts as an implicit quota. The practical difference matters. Under a plain ``quota``, snapshots eat the user's allowance, so an automatic snapshot schedule can make writes fail even though the user deleted files. Under ``refquota`` the user sees a stable limit and snapshot growth is charged to the parent instead. A common pattern is ``refquota`` for the user-visible limit plus ``quota`` on the parent dataset to bound the total. Reservations ~~~~~~~~~~~~ ``reservation`` Minimum space guaranteed to the dataset and its descendants. ``refreservation`` Minimum space guaranteed to the dataset alone. While usage is below the reservation, the dataset is accounted as if it were using the full reserved amount. That space is charged to the parent's ``used`` and counts against the parent's quotas and reservations — which is exactly what makes it a guarantee: nothing else in the pool can allocate it. .. code:: bash zfs set reservation=50G pool/critical zfs set refreservation=none pool/vm/disk0 With ``refreservation`` set, a snapshot can only be taken if there is enough free pool space outside the reservation for the dataset's current ``referenced`` bytes. ``refreservation=auto`` applies to volumes only: it thick-provisions the zvol, reserving enough space for the whole ``volsize`` plus metadata. Setting it to ``none`` makes the volume sparse — faster to create and thin, but writes can fail with ``ENOSPC`` when the pool fills. User, group and project quotas ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Within a single dataset, space can also be limited per user, per group, or per project: .. code:: bash zfs set userquota@alice=10G pool/home zfs set groupquota@staff=200G pool/home zfs set projectquota@42=50G pool/data zfs userspace pool/home zfs groupspace pool/home ``defaultuserquota``, ``defaultgroupquota`` and ``defaultprojectquota`` set a fallback applied to everyone without a specific quota (``0`` disables them). The ``*objquota@`` variants limit the *number of objects* rather than bytes. Enforcement is asynchronous: a user can exceed their quota by a little before writes start failing with ``EDQUOT``. Limiting counts, not space ~~~~~~~~~~~~~~~~~~~~~~~~~~ ``filesystem_limit`` and ``snapshot_limit`` cap how many datasets or snapshots may exist below a point in the tree. Like quotas, a descendant's limit adds to an ancestor's rather than replacing it, and the limit is not enforced against a user who is permitted to change it. ``filesystem_limit`` requires the ``filesystem_limits`` pool feature. .. code:: bash zfs set snapshot_limit=200 pool/home zfs get filesystem_count,snapshot_count pool/home Slop space ~~~~~~~~~~ You do not need to reserve emergency space by hand — ZFS already does it. The last ``1/2^spa_slop_shift`` of the pool, 3.2% by default, is held back and cannot be consumed by normal writes. Once free space drops below it, ordinary operations such as ``write`` and ``create`` return ``ENOSPC``. The point of the reserve is that recovery still works: * file removal and most administrative actions may use **half** the slop space; * operations that are almost certain to free space, such as ``zfs destroy``, may use up to **three quarters** of it; * a small set of internal operations is always permitted regardless of free space — these are what can, in principle, run a pool genuinely full and leave it permanently read-only. On very small pools the reserve is raised to at least 128 MiB, and on very large ones it is capped at 128 GiB, but it is never more than half the pool. ``spa_slop_shift`` can be raised to claw back some of that space on large pools — 6 gives 1.6%, 7 gives 0.8% — at the cost of a thinner margin for recovery. See :doc:`Module Parameters ` and :doc:`Workload Tuning `, which recommends keeping free space above 10% for allocation performance — a separate concern from the slop reserve, and the one that usually bites first. Further reading ~~~~~~~~~~~~~~~ * `zfsprops(7) `__ — ``quota``, ``refquota``, ``reservation``, ``refreservation``, ``userquota@``, ``filesystem_limit``, ``snapshot_limit`` * `zfs-userspace(8) `__, `zfs-project(8) `__ * :doc:`Datasets and Properties ` ACLs and Extended Attributes ============================ ZFS supports two unrelated ACL models — NFSv4-style ACLs, which are part of ZFS itself, and POSIX ACLs, which are a Linux facility stored as extended attributes. Which one is available depends on the platform, and getting this wrong is a common source of "my permissions do not survive" surprises. Choosing an ACL type ~~~~~~~~~~~~~~~~~~~~ The ``acltype`` property selects the model: ``off`` ACLs disabled. **The default on Linux.** (``noacl`` and ``disabled`` are accepted as aliases.) ``posix`` POSIX ACLs, Linux-only, stored as extended attributes. (``posixacl`` is an alias.) Requires a kernel built with ``CONFIG_FS_POSIX_ACL``; without it, ZFS falls back to ``off``. ``nfsv4`` NFSv4-style ZFS ACLs, managed with ``getfacl``/``setfacl``. **The default on FreeBSD**; not supported on Linux. .. warning:: Setting a type the platform does not support does **not** produce an error. On Linux, ``acltype=nfsv4`` is handled identically to ``off`` — but ``zfs get acltype`` still reports ``nfsv4``. A dataset can therefore look as though it has ACLs enabled while silently having none. .. code:: bash zfs set acltype=posix pool/home # Linux zfs set acltype=nfsv4 pool/data # FreeBSD Because POSIX ACLs live in an extended attribute, they do not overwrite any NFSv4 ACLs that may already be set. On Linux, remember to set ``xattr=sa`` alongside ``acltype=posix`` — see below. NFSv4 ACL behavior ~~~~~~~~~~~~~~~~~~ Two properties govern how ZFS ACLs interact with the traditional Unix mode. Neither applies to POSIX ACLs. ``aclinherit`` — how ACEs are inherited when files and directories are created: ``discard`` Inherit nothing. ``noallow`` Inherit only inheritable ACEs that specify *deny* permissions. ``restricted`` The default. Strips ``write_acl`` and ``write_owner`` on inheritance. (``secure`` is accepted as a legacy alias.) ``passthrough`` Inherit everything unmodified. Files are then created with a mode determined by the inheritable ACEs; if none affect the mode, the application's requested mode is used. ``passthrough-x`` Like ``passthrough``, except ``owner@``, ``group@`` and ``everyone@`` inherit the execute bit only if the creation mode asked for it. ``aclmode`` — what ``chmod(2)`` does to an existing ACL: ``discard`` The default. Deletes every ACE except those representing the requested mode. ``groupmask`` Reduces the permissions in all ALLOW entries so they do not exceed the group permissions given to ``chmod``. ``passthrough`` Changes nothing beyond the entries needed to represent the new mode. ``restricted`` Makes ``chmod`` fail on any file with a non-trivial ACL that cannot be represented as a mode. For a Samba or NFS server where clients manage ACLs, ``aclinherit=passthrough`` with ``aclmode=passthrough`` is the combination that stops a stray ``chmod`` from silently discarding the ACL. The defaults (``restricted`` /``discard``) favour Unix mode semantics instead. Extended attributes ~~~~~~~~~~~~~~~~~~~ The ``xattr`` property picks how extended attributes are stored: ``sa`` System-attribute-based (``on`` is an alias for it). Stored inline in the space reserved for system attributes, which significantly reduces disk I/O. Two limits apply: a single attribute may be at most **32 KiB** (``DXATTR_MAX_ENTRY_SIZE``), and all SA xattrs on one file at most **64 KiB** (``DXATTR_MAX_SA_SIZE``). Exceeding either is not an error — the attribute is transparently written as a directory-based xattr instead. ``dir`` Directory-based: each file with xattrs gets a hidden directory holding them. No practical limit on size or count (though Linux's ``getxattr(2)``/``setxattr(2)`` cap a single attribute at 64K). The most compatible style, supported by every ZFS implementation. ``off`` Disabled. .. note:: **The default changed.** ``xattr`` defaults to ``sa`` from OpenZFS 2.3.0 onward; on 2.2.x and earlier the default was ``dir``. Datasets created under an older release keep ``dir`` — check with ``zfs get xattr`` rather than assuming. This matters for POSIX ACLs in particular, because on Linux they *are* extended attributes: ZFS stores them under the standard names ``system.posix_acl_access`` and ``system.posix_acl_default``, through the same code path as any other xattr. With ``xattr=dir`` every ACL lookup means opening and reading that hidden directory; with ``xattr=sa`` the ACL comes along with the dnode. That is why ``xattr=sa`` is the standard pairing with ``acltype=posix`` on Linux. It also follows that ``xattr=off`` disables POSIX ACLs entirely, whatever ``acltype`` says. .. code:: bash zfs set xattr=sa pool/home zfs set acltype=posix pool/home Note that ``sa``-based xattrs are not readable by ZFS implementations that predate the feature — relevant if the pool must be importable elsewhere. Since both properties only affect newly-written data and newly-created files, set them at dataset creation where possible: .. code:: bash zfs create -o xattr=sa -o acltype=posix pool/home/alice Further reading ~~~~~~~~~~~~~~~ * `zfsprops(7) `__ — ``acltype``, ``aclinherit``, ``aclmode``, ``xattr``, ``nbmand`` * :doc:`Datasets and Properties `, :doc:`Sharing Datasets ` Data Storage ============ What ZFS does to a block on its way to disk — verifying it, shrinking it, sharing it, and encrypting it. .. toctree:: :maxdepth: 1 Checksums Compression Deduplication Block Cloning Encryption Checksums and Their Use in ZFS ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ End-to-end checksums are a key feature of ZFS and an important differentiator for ZFS over other RAID implementations and filesystems. Advantages of end-to-end checksums include: - detects data corruption upon reading from media - blocks that are detected as corrupt are automatically repaired if possible, by using the RAID protection in suitably configured pools, or redundant copies (see the zfs ``copies`` property) - periodic scrubs can check data to detect and repair latent media degradation (bit rot) and corruption from other sources - checksums on ZFS replication streams, ``zfs send`` and ``zfs receive``, ensure the data received is not corrupted by intervening storage or transport mechanisms Checksum Algorithms ^^^^^^^^^^^^^^^^^^^ The checksum algorithms in ZFS can be changed for datasets (filesystems or volumes). The checksum algorithm used for each block is stored in the block pointer (metadata). The block checksum is calculated when the block is written, so changing the algorithm only affects writes occurring after the change. The checksum algorithm for a dataset can be changed by setting the ``checksum`` property: .. code:: bash zfs set checksum=sha256 pool_name/dataset_name +-----------+--------------+------------------------+-------------------------+ | Checksum | Ok for dedup | Compatible with | Notes | | | and nopwrite?| other ZFS | | | | | implementations? | | +===========+==============+========================+=========================+ | on | see notes | yes | ``on`` is a | | | | | short hand for | | | | | ``fletcher4`` | | | | | for non-deduped | | | | | datasets and | | | | | ``sha256`` for | | | | | deduped | | | | | datasets | +-----------+--------------+------------------------+-------------------------+ | off | no | yes | Do not use | | | | | ``off`` | +-----------+--------------+------------------------+-------------------------+ | fletcher2 | no | yes | Deprecated | | | | | implementation | | | | | of Fletcher | | | | | checksum, use | | | | | ``fletcher4`` | | | | | instead | +-----------+--------------+------------------------+-------------------------+ | fletcher4 | no | yes | Fletcher | | | | | algorithm, also | | | | | used for | | | | | ``zfs send`` | | | | | streams | +-----------+--------------+------------------------+-------------------------+ | sha256 | yes | yes | Default for | | | | | deduped | | | | | datasets | +-----------+--------------+------------------------+-------------------------+ | noparity | no | yes | Do not use | | | | | ``noparity`` | +-----------+--------------+------------------------+-------------------------+ | sha512 | yes | requires pool | salted | | | | feature | ``sha512`` | | | | ``org.illumos:sha512`` | currently not | | | | | supported for | | | | | any filesystem | | | | | on the boot | | | | | pools | +-----------+--------------+------------------------+-------------------------+ | skein | yes | requires pool | salted | | | | feature | ``skein`` | | | | ``org.illumos:skein`` | currently not | | | | | supported for | | | | | any filesystem | | | | | on the boot | | | | | pools | +-----------+--------------+------------------------+-------------------------+ | edonr | see notes | requires pool | salted | | | | feature | ``edonr`` | | | | ``org.illumos:edonr`` | currently not | | | | | supported for | | | | | any filesystem | | | | | on the boot | | | | | pools | | | | | | | | | | In an abundance of | | | | | caution, Edon-R requires| | | | | verification when used | | | | | with dedup, so it will | | | | | automatically use | | | | | ``verify``. | | | | | | +-----------+--------------+------------------------+-------------------------+ | blake3 | yes | requires pool | salted | | | | feature | ``blake3`` | | | | ``org.openzfs:blake3`` | currently not | | | | | supported for | | | | | any filesystem | | | | | on the boot | | | | | pools | +-----------+--------------+------------------------+-------------------------+ Checksum Accelerators ^^^^^^^^^^^^^^^^^^^^^ ZFS has the ability to offload checksum operations to the Intel QuickAssist Technology (QAT) adapters. Checksum Microbenchmarks ^^^^^^^^^^^^^^^^^^^^^^^^ Some ZFS features use microbenchmarks when the ``zfs.ko`` kernel module is loaded to determine the optimal algorithm for checksums. The results of the microbenchmarks are observable in the ``/proc/spl/kstat/zfs`` directory. The winning algorithm is reported as the "fastest" and becomes the default. The default can be overridden by setting zfs module parameters. ========= ==================================== ======================== Checksum Results Filename ``zfs`` module parameter ========= ==================================== ======================== Fletcher4 /proc/spl/kstat/zfs/fletcher_4_bench zfs_fletcher_4_impl all-other /proc/spl/kstat/zfs/chksum_bench zfs_blake3_impl, zfs_sha256_impl, zfs_sha512_impl ========= ==================================== ======================== Disabling Checksums ^^^^^^^^^^^^^^^^^^^ While it may be tempting to disable checksums to improve CPU performance, it is widely considered by the ZFS community to be an extraordinarily bad idea. Don't disable checksums. Compression =========== ZFS compresses transparently, per record, per dataset. On most workloads compression is a net win even on fast storage: less data written means fewer I/Os, and modern algorithms compress faster than a disk can absorb the difference. Since OpenZFS 2.2.0 the ``compression`` property defaults to ``on``, so new datasets are compressed unless you say otherwise. Pools and datasets created under older releases defaulted to ``off`` and keep that setting — check before assuming. .. code:: bash zfs get -r compression pool # what is actually set, and where from zfs set compression=zstd pool/data zfs set compression=off pool/incompressible Changing the property affects only newly-written data. Existing blocks keep whatever they were written with; rewriting them (``zfs send | zfs recv``, a copy, or `zfs-rewrite(8) `__) is what re-compresses them. Algorithms ~~~~~~~~~~ ``on`` The default. Unlike every other value it is not a fixed algorithm — it is ``lz4`` when the ``lz4_compress`` feature is enabled, otherwise ``lzjb``, and it may change as new algorithms are added. ``lz4`` Very fast in both directions with a moderate ratio. The safe default for almost everything. Requires the ``lz4_compress`` feature. ``zstd`` / ``zstd-N`` Higher ratios at higher CPU cost. ``N`` runs from 1 (fastest) to 19; plain ``zstd`` is ``zstd-3``. ``zstd-fast-N`` maps to negative zstd levels, with ``N`` in 1–10, 20, 30, …, 100, 500, 1000 — higher ``N`` means faster and weaker. ``zstd-fast`` is ``zstd-fast-1``. ``gzip`` / ``gzip-N`` ``N`` from 1 to 9; plain ``gzip`` is ``gzip-6``. Largely superseded by ``zstd``, which gives similar ratios much faster. ``lzjb`` The original algorithm, kept for compatibility. ``zle`` Compresses runs of zeros only. In practice: ``lz4`` when latency and CPU matter, ``zstd`` (level 3–7) when capacity matters, ``off`` only for data that is already compressed or encrypted. How much actually gets saved ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Blocks are allocated in whole sectors of 2\ :sup:`ashift` bytes, and a compressed record is rounded up to a sector boundary. If compression saves less than one whole sector, the block is stored uncompressed. There is also a 12.5% minimum-saving threshold on top of the rounding. The consequence is that small records compress poorly in accounting terms: with ``recordsize=16K`` and 4K sectors, a record is four sectors, so compression has to save at least 25% before any space is saved on disk. Large records give compression more room to work. Regardless of the algorithm, any setting other than ``off`` detects all-zero blocks and stores them as holes. Checking the result ~~~~~~~~~~~~~~~~~~~ .. code:: bash zfs get compression,compressratio,refcompressratio pool/data zfs list -o name,used,logicalused,compressratio -r pool ``compressratio`` is a multiplier over the dataset's ``used`` space including descendants; ``refcompressratio`` covers only ``referenced`` space. Comparing ``logicalused`` with ``used`` shows the same thing in bytes. Interaction with other features ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ * Compression happens **before** encryption, so ratios are preserved on encrypted datasets. See :doc:`Native Encryption ` for the CRIME-style caveat that follows from this. * ``zfs send -c`` sends already-compressed blocks in their compressed form — smaller streams and less CPU on both sides. See :doc:`Send and Receive `. * Compression interacts with ``recordsize``; see :doc:`Workload Tuning `. Further reading ~~~~~~~~~~~~~~~ * `zfsprops(7) `__ — ``compression``, ``compressratio``, ``logicalused`` * :doc:`Workload Tuning ` Deduplication ============= Deduplication removes redundant data at the block level: identical blocks are stored once and shared between files. It is synchronous — every write is looked up in the dedup table (DDT) before it lands. .. warning:: Unless you have a specific reason, do not enable dedup. It is the most resource-intensive feature in ZFS, and enabling it on an improperly-sized system causes slow I/O, slow administrative operations, and in the worst case a pool that cannot be imported because the DDT does not fit in memory. Try ``compression`` first — see :doc:`Compression `. The cost ~~~~~~~~ Every unique block gets a DDT entry. Lookups happen on every write, and the DDT must be consulted on every free as well. Plan for at least **1.25 GiB of RAM per 1 TiB of stored data**; the real figure depends on block sizes and how much of the data is actually duplicated. Dedup only pays off when the duplication is real and substantial — VM images from a common base, backup targets receiving many similar systems. For general file storage, compression usually gives most of the benefit at a fraction of the cost. Note also that clones and ``zfs send``-based workflows already share blocks without dedup, as does block cloning (``copy_file_range``). Enabling it ~~~~~~~~~~~ ``dedup`` is a dataset property, so it can be limited to the datasets that benefit. It defaults to ``off``, and changing it affects only newly-written data. .. code:: bash zfs set dedup=on pool/vmimages zfs set dedup=blake3,verify pool/vmimages Values are ``off``, ``on``, ``verify``, or an explicit checksum — ``sha256``, ``sha512``, ``skein``, ``edonr`` or ``blake3`` — each optionally with ``,verify``. The default dedup checksum is ``sha256``. When dedup is enabled, this checksum overrides the ``checksum`` property. ``verify`` makes ZFS do a byte-for-byte comparison whenever two blocks have the same signature, at the cost of a read per match. ``verify`` alone means ``sha256,verify``, and it is mandatory with ``edonr``. Watching the table ~~~~~~~~~~~~~~~~~~ .. code:: bash zpool status -D pool zpool get dedupratio,dedup_table_size,dedupused,dedupsaved pool zdb -DD pool # detailed DDT histogram ``dedupratio`` is the ratio of storage that would be needed without dedup to what is actually used; ``dedupsaved`` is the same thing in bytes. Before turning dedup on, ``zdb -S pool`` simulates the DDT for the data already in the pool and prints the ratio you would have gotten. If it is not comfortably above 2x, dedup is not worth it. Keeping the table bounded ~~~~~~~~~~~~~~~~~~~~~~~~~ The failure mode of dedup is an unbounded DDT. Two mechanisms address it: **Quota.** The pool property ``dedup_table_quota`` caps the on-disk size of the DDT. Once reached, no new entries are added — existing entries still have their reference counts updated. The default is ``auto``, which sizes the limit from a dedicated dedup vdev if one exists; ``none`` disables the limit. The enforced size may land slightly above or below the requested value. It applies to both legacy and fast dedup tables. .. code:: bash zpool set dedup_table_quota=100G pool **Pruning.** ``zpool ddtprune`` removes older *unique* (non-duplicate) entries to make room for newer duplicate ones, either by a target percentage of unique entries or by age in days. .. code:: bash zpool ddtprune -d 30 pool # entries older than 30 days zpool ddtprune -p 25 pool # 25% of unique entries **Fast dedup** (the ``fast_dedup`` pool feature) enables the more advanced dedup table format that these mechanisms are built around. It becomes ``active`` when the first deduplicated block is written to a new dedup table, and returns to ``enabled`` once all such blocks are freed. Existing pools keep their legacy tables; a new table is created on pool creation, or when a dataset with dedup enabled starts using a new checksum. A **dedicated dedup vdev** (``zpool add pool dedup ``) moves the DDT onto fast storage, which is what makes dedup practical on larger pools. It should be redundant — losing it loses the pool. Interaction with encryption ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Dedup works on encrypted datasets, but for security a dataset deduplicates only against itself, its snapshots and its clones. It also leaks which blocks within a dataset are identical, and adds CPU cost per written block. See :doc:`Native Encryption `. Turning it off ~~~~~~~~~~~~~~ Setting ``dedup=off`` stops new blocks from being deduplicated, but does not undo anything: existing blocks stay deduplicated and the DDT stays until they are all freed. Getting rid of it entirely means rewriting the data — for example by copying it to a fresh dataset and destroying the old one. Further reading ~~~~~~~~~~~~~~~ * `zfsconcepts(7) `__ — ``Deduplication`` * `zfsprops(7) `__ — ``dedup`` * `zpoolprops(7) `__ — ``dedup_table_quota``, ``dedup_table_size``, ``dedupratio`` * `zpool-ddtprune(8) `__ * :doc:`Workload Tuning ` Block Cloning ============= Block cloning makes a shallow copy of a file, or part of one: the new file references the existing data blocks instead of copying them. A later write to either side takes a private copy of the affected block. This is what implements *reflinks*, or file-level copy-on-write. Copying a 50 GiB file becomes near-instant and consumes no additional space, because no data is read or written — only references are created. Block cloning requires the ``block_cloning`` pool feature, which becomes ``active`` when the first block is cloned and returns to ``enabled`` when the last cloned block is freed. How it is triggered ~~~~~~~~~~~~~~~~~~~ Unlike deduplication, cloning is never automatic — a program has to ask for it, normally via ``copy_file_range(2)``. Many common tools already do: .. code:: bash cp --reflink=auto bigfile copy # GNU coreutils cp bigfile copy # newer versions try this by default Look for "clone", "reflink" or "dedupe" in a tool's documentation to find out whether it will attempt block cloning. Limitations ~~~~~~~~~~~ Blocks cannot be cloned when: * only part of a block would be cloned — cloning works on whole blocks; * the data has not been written to disk yet; * the source and destination ``recordsize`` differ; * the datasets do not share the same master encryption key. ZFS does try to clone across datasets, including encrypted ones, but the OS adds its own restrictions — most versions of Linux will not allow clones across datasets at all. When cloning is not possible the copy silently falls back to reading and writing the data, so a ``cp`` that seems slow is usually a clone that did not happen. Block cloning vs deduplication ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Both end up with multiple references to one block, but they are not alternatives to each other: +------------------+-------------------------+---------------------------+ | | Block cloning | Deduplication | +==================+=========================+===========================+ | Triggered by | an explicit request | every write, synchronously| | | from a program | | +------------------+-------------------------+---------------------------+ | Bookkeeping | Block Reference Table | dedup table (DDT) | | | (BRT), minimal overhead | large, RAM-hungry | +------------------+-------------------------+---------------------------+ | Safe to leave on | yes | only with careful sizing | +------------------+-------------------------+---------------------------+ | Finds duplicates | no — only what it is | yes, anywhere in the pool | | | told to clone | | +------------------+-------------------------+---------------------------+ Because the BRT's overhead is minimal, block cloning can reasonably stay enabled all the time — which is not true of :doc:`deduplication `. Checking the savings ~~~~~~~~~~~~~~~~~~~~ .. code:: bash zpool get bcloneused,bclonesaved,bcloneratio pool zdb -T pool # BRT statistics ``bcloneused`` is the space occupied by cloned blocks, ``bclonesaved`` is what would have been needed without cloning, and ``bcloneratio`` expresses the same as a multiplier. A related feature, ``block_cloning_endian``, corrects an endianness problem in how BRT entries were originally stored; it activates when the first BRT ZAP is created so that existing pools stay compatible. Further reading ~~~~~~~~~~~~~~~ * `zfsconcepts(7) `__ — ``Block cloning`` * `zpool-features(7) `__ — ``block_cloning``, ``block_cloning_endian`` * `zpoolprops(7) `__ — ``bcloneused``, ``bclonesaved``, ``bcloneratio`` * :doc:`Deduplication `, :doc:`Snapshots, Clones and Bookmarks ` Native Encryption ================= ZFS can encrypt datasets itself, without a separate layer such as LUKS or GELI. Encryption is a per-dataset property, so a single pool can hold both encrypted and unencrypted datasets, each encrypted dataset can have its own key, and the pool stays fully manageable — importable, scrubbable, resilverable and replicable — while the keys are not loaded. Native encryption requires the ``encryption`` pool feature. What is and is not encrypted ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Encrypted: * file and volume data * file attributes, ACLs and permission bits * directory listings * FUID mappings and ``userused``/``groupused`` accounting data * the ZFS intent log (ZIL) Not encrypted: * dataset and snapshot names * the dataset hierarchy * dataset properties * file sizes and the location of holes in files * deduplication tables (the deduplicated data itself is encrypted) The dividing line is per object type, and the reason the second list looks the way it does is the **block tree**. Only leaf blocks are encrypted; indirect blocks (level > 0) are authenticated but left in the clear, carrying a MAC over the MACs below them. Block pointers therefore stay readable without keys — which is precisely what lets the pool be scrubbed, resilvered, freed and replicated with the keys unloaded, and equally what leaks how large each file is, where its holes are, and how much data a dataset holds. This is the trade-off that keeps the pool administrable without keys. If the existence and names of your datasets, or their sizes and change patterns, are themselves sensitive, native encryption is not the right layer for that — use full-disk encryption underneath ZFS instead, or in addition. Creating an encrypted dataset ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ``encryption`` and ``keyformat`` must be given at creation time, optionally with ``keylocation`` and ``pbkdf2iters``. The cipher suite cannot be changed afterwards. .. code:: bash # passphrase, entered interactively zfs create -o encryption=on -o keyformat=passphrase pool/secret # raw key from a file dd if=/dev/urandom bs=32 count=1 of=/root/pool.key zfs create -o encryption=on \ -o keyformat=raw \ -o keylocation=file:///root/pool.key \ pool/secret ``encryption=on`` selects the current default suite, which is ``aes-256-gcm``. The suites that can be named explicitly are ``aes-128-ccm``, ``aes-192-ccm``, ``aes-256-ccm``, ``aes-128-gcm``, ``aes-192-gcm`` and ``aes-256-gcm``. The GCM modes are generally faster on hardware with AES acceleration. ``keyformat`` is one of: ``passphrase`` 8 to 512 bytes, run through PBKDF2 before use. The iteration count is set by ``pbkdf2iters`` — currently 350000 by default, with a minimum of 100000. ``raw`` Exactly 32 bytes of random data, regardless of the chosen suite. ``hex`` The same 32 bytes, written as hex. ``keylocation`` is where the key is read from: ``prompt`` (the default — interactive), ``file:///absolute/path``, or an ``http://`` / ``https://`` address, which is how a machine can fetch its key from a key server at boot. Encryption roots and inheritance ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The dataset you supply a key for becomes an *encryption root*. Descendants inherit its key by default, so loading, unloading or changing the key at the root does the same for everything under it. The read-only ``encryptionroot`` property shows which root a dataset currently uses. To break that relationship and give a child its own key, supply a ``keyformat`` when creating it, or run ``zfs change-key`` on it later. Note that: * the child's ``keyformat`` may be the same as the parent's and it still becomes a separate encryption root; * setting the ``encryption`` property alone does *not* create a new encryption root — it just uses a different cipher suite with the same key; * clones always use their origin's key. Because of that last exception, ``keystatus``, ``keyformat``, ``keylocation`` and ``pbkdf2iters`` do not inherit the way other ZFS properties do: their effective value comes from the encryption root. Loading and unloading keys ~~~~~~~~~~~~~~~~~~~~~~~~~~ .. code:: bash zfs load-key pool/secret # prompts, or reads keylocation zfs load-key -a # every encryption root in every imported pool zfs load-key -r pool # recursively, all encryption roots below pool zfs load-key -n pool/secret # dry run: just verify the key is correct zfs mount pool/secret # loading a key does not mount zfs unload-key pool/secret # dataset must not be open or mounted ``zfs mount -l`` asks for the key and mounts in one step. The ``keystatus`` property is ``available`` once the key is loaded and ``unavailable`` while it is not: .. code:: bash zfs get -r keystatus,encryptionroot pool On systems with PAM, ``pam_zfs_key(8)`` can load a user's key and mount their encrypted home directory at login. Changing the key ~~~~~~~~~~~~~~~~ ``zfs change-key`` rewraps the master key with a new user key. It does not re-encrypt the data, so it is instantaneous regardless of dataset size. The existing key must already be loaded (or use ``-l`` to load it first). .. code:: bash zfs change-key pool/secret zfs change-key -o keyformat=passphrase -o pbkdf2iters=500000 pool/secret zfs change-key -i pool/secret/child # give up own key, inherit parent's This is also the only way to change ``keyformat`` and ``pbkdf2iters`` after creation. **Changing the key does not undo a compromise.** Newly written data keeps using the same master key as the existing data, so an attacker who obtained a user key *and* the corresponding wrapped master key can still read what is written later. The old wrapped master key is not overwritten on disk and may remain recoverable by forensic analysis for an indeterminate time. If the master key is compromised, the correct response is to securely erase the drives, create a new pool and copy the data back. This can be approximated in place by creating new datasets, copying the data into them (for example with ``zfs send | zfs recv``), and then clearing the free space with ``zpool trim --secure`` if the hardware supports it, or ``zpool initialize`` otherwise. Raw sends ~~~~~~~~~ ``zfs send -w`` (``--raw``) sends the data exactly as it is on disk, still encrypted. This means: * backups can be taken without loading any keys; * the receiving machine need not be trusted — it cannot read the data, and it cannot alter it without that being detected on the next read; * the received dataset keeps its encryption, and its key, from the source. .. code:: bash zfs send -w pool/secret@daily | ssh backup zfs receive -u tank/secret A non-raw send of an encrypted dataset requires the keys to be loaded and produces a plaintext stream, which is then encrypted (or not) according to the destination's own properties. See :doc:`Send and Receive `. Interaction with other features ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ **Compression.** Encryption is applied after compression, so compression ratios are preserved. The flip side is that a dataset can be vulnerable to a CRIME-like attack if the applications accessing it let an attacker influence what gets compressed alongside secret data. **Checksums.** For encrypted data, the usual 256-bit checksum becomes 128 bits of the chosen checksum plus 128 bits of MAC from the cipher suite. The MAC is what gives protection against *maliciously* altered data, not just accidental corruption. **Deduplication** still works, but for security a dataset deduplicates only against itself, its snapshots and its clones. Dedup with encryption leaks which blocks within a dataset are identical, and costs extra CPU per written block. **Embedded data.** Encrypted data cannot use the ``embedded_data`` feature. **copies=3** is not available on encrypted datasets — the implementation stores encryption metadata where the third copy would go. Further reading ~~~~~~~~~~~~~~~ * `zfs-load-key(8) `__ — including the full ``Encryption`` section * `zfs-change-key(8) `__, `zfs-unload-key(8) `__ * `zfsprops(7) `__ — ``encryption``, ``keyformat``, ``keylocation``, ``pbkdf2iters``, ``keystatus``, ``encryptionroot`` * `pam_zfs_key(8) `__ Operations ========== Running a pool day to day: replicating it, keeping it healthy, exporting it to clients, and handing out administrative rights. .. toctree:: :maxdepth: 1 Send and Receive Scrub and Resilver TRIM Sharing Delegated Administration Boot Process Troubleshooting Send and Receive ================ ``zfs send`` serialises a snapshot into a byte stream, and ``zfs receive`` turns that stream back into a dataset. Together they are ZFS' native replication and backup mechanism. Because the stream is produced from ZFS' own block-level bookkeeping, an incremental send has to read only the blocks that actually changed between two snapshots — it never walks the directory tree looking for modified files. That makes replication cost proportional to the amount of changed data rather than to the size of the dataset, which is the main reason to prefer it over file-based tools such as ``rsync`` on ZFS. Streams are checksummed, so corruption introduced by the network or by intermediate storage is detected on receive. A full stream ~~~~~~~~~~~~~ Everything starts from a snapshot: .. code:: bash zfs snapshot pool/data@2026-07-21 zfs send pool/data@2026-07-21 | ssh backup zfs receive tank/data The receiving side creates ``tank/data`` along with the snapshot ``tank/data@2026-07-21``. The target dataset must not already exist unless you are receiving an incremental stream into it. Streams are just bytes, so they can also be stored in a file or piped through anything: .. code:: bash zfs send pool/data@2026-07-21 | zstd -3 > /mnt/usb/data-full.zfs.zst Storing streams in files is fine for transport, but it is a poor archive format: a single damaged byte can render the whole stream unreceivable, whereas a received dataset is checksummed, scrubable and repairable. Prefer receiving into a real pool for anything you intend to keep. Incremental streams ~~~~~~~~~~~~~~~~~~~ ``-i`` sends the difference between two snapshots. The source snapshot must still exist on the receiving side. .. code:: bash zfs snapshot pool/data@2026-07-22 zfs send -i pool/data@2026-07-21 pool/data@2026-07-22 \ | ssh backup zfs receive tank/data ``-I`` sends every intermediate snapshot between the two, so the receiving side ends up with the same snapshot history rather than just the endpoints: .. code:: bash zfs send -I pool/data@2026-07-21 pool/data@2026-07-28 | ... The incremental source may also be a bookmark, which lets the sending side free the space held by the old snapshot while still being able to compute the difference: .. code:: bash zfs bookmark pool/data@2026-07-21 pool/data#2026-07-21 zfs destroy pool/data@2026-07-21 zfs send -i pool/data#2026-07-21 pool/data@2026-07-22 | ... See :doc:`Snapshots, Clones and Bookmarks ` for what bookmarks are. Replicating a whole hierarchy ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ``-R`` produces a replication stream package: the named dataset and all of its descendants, up to the named snapshot. On receive, properties, snapshots, descendant file systems and clones are preserved. .. code:: bash zfs snapshot -r pool/home@2026-07-21 zfs send -R pool/home@2026-07-21 | ssh backup zfs receive -u tank/home Combined with ``-i``/``-I`` it becomes the usual "make the destination look exactly like the source" tool. On the receiving side, ``-F`` then also destroys snapshots and file systems that no longer exist on the sending side, and rolls the target back to its most recent snapshot before receiving: .. code:: bash zfs send -R -I pool/home@sunday pool/home@monday \ | ssh backup zfs receive -F -u tank/home ``-F`` discards data on the destination. Use ``-n -v`` first if you are not sure what a stream will do. ``-X`` excludes named datasets from a recursive send, and ``-s`` lets a recursive send skip (with a warning) datasets whose snapshots are missing instead of aborting. Useful send options ~~~~~~~~~~~~~~~~~~~ ``-c`` (``--compressed``) Send blocks that are compressed on disk in their compressed form. The stream is smaller and both sides do less work, since nothing has to be decompressed and recompressed. This is the usual default choice for datasets with compression enabled. ``-L`` (``--large-block``) Allow records larger than 128 KiB in the stream. Without it, a dataset with a larger ``recordsize`` is sent as smaller blocks, which is slower and changes the block layout on the receiving side. It has no effect if the ``large_blocks`` pool feature is disabled or ``recordsize`` was never set above 128 KiB. ``-e`` (``--embed``) Send blocks that use embedded data (very small blocks stored inside the block pointer) in that form. Requires the ``embedded_data`` feature on both sides. ``-p`` (``--props``) Include dataset properties in the stream. ``-R`` implies this. ``-w`` (``--raw``) Send the data exactly as it is on disk. For encrypted datasets this means the stream stays encrypted, so a backup can be taken without loading the keys, and the receiving machine never needs them. See :doc:`Encryption `. ``-h`` (``--holds``) Include user holds on the sent snapshots. ``-n -v`` / ``-P`` Dry run with a size estimate; ``-P`` prints it in a machine-parsable form. Worth running before any large or destructive transfer. Useful receive options ~~~~~~~~~~~~~~~~~~~~~~ ``-u`` Do not mount the received file system. On a backup server this is almost always what you want — the destination should not shadow local paths. ``-d`` / ``-e`` Rewrite the target name: ``-d`` drops the first component of the source dataset name, ``-e`` keeps only the last one. Both are relative to the named target file system. ``-o property=value`` / ``-x property`` Override a property carried in the stream, or exclude it so the local setting (or inheritance) wins. ``-o readonly=on`` on the destination prevents accidental writes that would force a rollback on the next incremental receive; ``-x mountpoint`` is a common way to keep a backup copy from claiming the source's mount point. ``-F`` Roll the destination back to its most recent snapshot first, and for incremental replication streams destroy datasets and snapshots that are gone on the sending side. ``-s`` Save the partially received state if the transfer is interrupted, so it can be resumed instead of restarted. See below. Resuming an interrupted transfer ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ A multi-terabyte initial replication over an unreliable link does not have to start over. When the receiving side used ``-s``, the partially received state is kept and exposed as the ``receive_resume_token`` property of the target dataset: .. code:: bash # receiving side zfs receive -s tank/data # ... connection dies ... zfs get -H -o value receive_resume_token tank/data # sending side, with that token zfs send -t | ssh backup zfs receive -s tank/data This requires the ``extensible_dataset`` feature on the receiving pool. An abandoned partial receive can be cleaned up with ``zfs receive -A``, which frees the space it holds. Practical notes ~~~~~~~~~~~~~~~ **Keep a common snapshot.** Retention on both sides must leave at least one snapshot (or bookmark on the sender) in common, otherwise the next incremental fails and a full resend is needed. **Do not modify the destination.** Any write breaks the next incremental receive until the dataset is rolled back; ``readonly=on`` avoids it. **ssh is often the bottleneck.** Both send and receive are bursty, so buffering the pipe on both sides (``mbuffer``, ``dd`` with a large block size) usually helps. **Feature options must match.** Streams made with ``-L``, ``-c``, ``-e`` or ``-w`` need the corresponding features on the receiving pool. Doing the snapshot, retention, hold and resume-token bookkeeping by hand gets tedious. Third-party tools built on ``zfs send``/``receive`` handle it — see :ref:`scheduling snapshots `, since the same tools usually do both. See the :doc:`FAQ ` for stream caveats such as ``hole_birth``. Further reading ~~~~~~~~~~~~~~~ * `zfs-send(8) `__, `zfs-receive(8) `__ * `zfs-bookmark(8) `__ * `zfs-redact(8) `__ — sending a stream with selected data removed * `zstream(8) `__ — inspecting and manipulating stream files * :doc:`FAQ: Sending and Receiving Streams ` Scrub and Resilver ================== ZFS checksums every block, so it knows when data is wrong instead of guessing. Scrub and resilver are the two background operations that act on that knowledge: a scrub verifies everything, a resilver rebuilds what is known to be missing. Self-healing ~~~~~~~~~~~~ Every read verifies the block's checksum. On a mismatch, a pool with redundancy — mirror, raidz, draid, or ``copies>1`` — fetches a good copy, returns it to the application, and rewrites the bad one. The error is counted in ``zpool status`` even though nothing was lost. Without redundancy, ZFS can still detect the corruption and report exactly which file is affected, but it cannot repair it. This is the whole argument for redundant pools: checksums without a second copy only tell you what you have already lost. Scrub ~~~~~ A scrub walks all data in the pool and verifies each block's checksum, repairing damage where redundancy allows. It is how latent corruption — bit rot on media that nothing has read for a year — gets found while it is still repairable. .. code:: bash zpool scrub pool zpool scrub -p pool # pause zpool scrub -s pool # stop zpool scrub -w pool # wait for completion before returning zpool status -v pool A paused scrub stays paused across export and import, and resumes from its last on-disk checkpoint when ``zpool scrub`` is issued again. Useful variants: ``-t`` (thorough) Also decrypt and decompress each block, catching the rare corruption where the checksum matches but decryption or decompression fails. Encrypted datasets need their keys loaded; blocks whose keys are unavailable fall back to a normal scrub. ``-e`` (error scrub) Scrub only files already reported as damaged by ``zpool status -v``. The pool must have been scrubbed once with the ``head_errlog`` feature enabled. ``-C`` Continue from the last saved txg (the ``last_scrubbed_txg`` pool property) instead of starting over. ``-S date`` / ``-E date`` Scrub only blocks created in a date range, in ``"YYYY-MM-DD HH:MM"`` format. Notes: * Scrubs are I/O-intensive; only one scrub or resilver runs at a time, and a scrub cannot start while a resilver is in progress. * A scrub runs in two phases — metadata scanning, which sorts blocks into large sequential ranges, then the actual block reads. * On a live pool, progress can exceed 100% because data keeps changing; no completion estimate is given during that period. * Encryption keys are not needed for ordinary checksum verification, but without them an unrepairable error cannot be attributed to a file name in ``zpool status -v``. How often? Monthly is a common baseline for consumer disks, quarterly for enterprise ones. Most distributions ship a systemd timer or cron job for it — check before adding your own. Resilver ~~~~~~~~ A resilver is the same machinery restricted to data ZFS knows to be out of date: after attaching a device to a mirror, replacing a failed one, or bringing a device back online after an outage. .. code:: bash zpool replace pool /dev/old /dev/new zpool attach pool /dev/existing /dev/new zpool status pool zpool resilver pool # restart a running resilver from the beginning ``zpool resilver`` restarts a resilver from the beginning and picks up any drives scheduled for a deferred resilver; that requires the ``resilver_defer`` pool feature. Sequential reconstruction ^^^^^^^^^^^^^^^^^^^^^^^^^ ``zpool replace -s`` and ``zpool attach -s`` rebuild the new device sequentially rather than by walking the block tree. This restores redundancy much faster, but checksums are not verified during the rebuild — so a scrub is started automatically when it finishes. Sequential reconstruction is not supported for raidz. Reading ``zpool status`` ~~~~~~~~~~~~~~~~~~~~~~~~ The per-device error counters mean different things: ``READ`` / ``WRITE`` I/O errors reported by the device itself. ``CKSUM`` The device returned data that did not match its checksum — silent corruption. Non-zero CKSUM on an otherwise healthy disk usually points at cabling, a controller, or memory rather than the disk. ``zpool status -v`` lists files with unrepairable errors. ``zpool clear`` resets the counters once the underlying cause is dealt with — it does not fix anything by itself. Persistent errors on a file mean the data is gone: restore that file from a backup or a snapshot. See :doc:`Troubleshooting `. Further reading ~~~~~~~~~~~~~~~ * `zpool-scrub(8) `__, `zpool-resilver(8) `__ * `zpool-replace(8) `__, `zpool-attach(8) `__, `zpool-status(8) `__ * :doc:`Checksums `, :doc:`VDEVs `, :doc:`Troubleshooting ` TRIM ==== TRIM (``BLKDISCARD``, ATA TRIM, SCSI UNMAP) tells a device which blocks the pool no longer uses, so an SSD can reclaim them for wear levelling and a thinly provisioned backing store can release the space. Without it, an SSD eventually believes every block it has ever been written to is still in use, and write performance degrades. ZFS offers two modes: an automatic background one and an on-demand one. They are independent — a manual TRIM can be run regardless of the ``autotrim`` setting. Automatic TRIM ~~~~~~~~~~~~~~ .. code:: bash zpool set autotrim=on pool Default is ``off``. When on, recently freed space is trimmed periodically. ZFS deliberately delays this so that small ranges aggregate into fewer, larger requests, which devices handle far better. The caveat is real: continuous trimming can put significant stress on the underlying devices, and how well that is absorbed varies enormously between models. On lower-end hardware, periodic manual TRIM usually captures most of the benefit with much less disruption. TRIM on L2ARC devices is separate — it is enabled by setting the ``l2arc_trim_ahead`` module parameter above 0. On-demand TRIM ~~~~~~~~~~~~~~ .. code:: bash zpool trim pool zpool trim -r 100M pool # rate-limit, bytes/s per leaf vdev zpool trim -w pool # wait for completion zpool trim -s pool # suspend zpool trim pool # ...and resume (no flags) zpool trim -c pool # cancel zpool trim -a # all pools Without ``-r``, TRIM runs as fast as the devices allow, which on a busy pool is worth limiting. Progress is shown per device in ``zpool status -t``. ``zpool trim -d`` (``--secure``) requests a *secure* TRIM: the device guarantees the data on the trimmed blocks is erased. Not all SSDs support it. This is the option referenced when clearing free space after an encryption key compromise — see :doc:`Native Encryption `. Periodic TRIM with systemd ~~~~~~~~~~~~~~~~~~~~~~~~~~ Per-pool weekly and monthly timer units ship with OpenZFS: .. code:: bash systemctl enable zfs-trim-weekly@rpool.timer --now systemctl enable zfs-trim-monthly@tank.timer --now Which devices can be trimmed ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Block devices that support ``BLKDISCARD`` — SSDs, thin-provisioned LUNs — and file vdevs on file systems that support hole punching. Trimming a spinning disk does nothing. If ``zpool trim`` reports that a device does not support TRIM, check the storage path: some HBAs and USB bridges do not pass the command through. See the ATA TRIM discussion in :doc:`Hardware `. Further reading ~~~~~~~~~~~~~~~ * `zpool-trim(8) `__ * `zpoolprops(7) `__ — ``autotrim`` * `zpool-initialize(8) `__ — the write-everything counterpart * :doc:`Hardware ` Sharing Datasets ================ ZFS can drive NFS and SMB exports from dataset properties, so a share follows the dataset instead of living in a separate config file that has to be kept in sync. Block-level export (iSCSI) works differently — it uses zvols and an external target framework. It is only a wrapper ~~~~~~~~~~~~~~~~~~~~ **ZFS does not implement NFS or SMB.** The ``sharenfs`` and ``sharesmb`` properties generate configuration for the platform's normal server and then tell it to reload: +----------+------------------------------+---------------------------------+ | Platform | ZFS writes | ZFS then runs | +==========+==============================+=================================+ | Linux | ``/etc/exports.d/ | ``/usr/sbin/exportfs`` | | (NFS) | zfs.exports`` | | +----------+------------------------------+---------------------------------+ | FreeBSD | ``/etc/zfs/exports`` | signals ``mountd`` | | (NFS) | | | +----------+------------------------------+---------------------------------+ | Linux | a Samba *usershare* | ``net usershare add`` | | (SMB) | | / ``delete`` | +----------+------------------------------+---------------------------------+ Everything else — authentication, Kerberos, ACL semantics, performance, tuning — is the server's job and is configured the server's way. The convenience is real, but so are the consequences: **The server must already be installed and configured.** ZFS checks whether ``exportfs`` exists and quietly does nothing if it does not. Setting ``sharenfs`` on a host without nfs-utils appears to succeed while exporting nothing. Likewise ``sharesmb`` needs Samba listening on ``127.0.0.1`` with usershares enabled (``usershare max shares``, ``usershare owner only``) before it will work at all. **Two config files now describe your exports.** ZFS owns ``/etc/exports.d/zfs.exports``; anything you wrote by hand is still in ``/etc/exports``. Both are live. When a share does not behave, look at both, and at ``exportfs -v`` for what the kernel actually has. **The option set is an allowlist, not passthrough.** Despite appearances, ``sharenfs`` does not accept arbitrary ``exports(5)`` options. On Linux the accepted keys are ``sec``, ``ro``, ``rw`` plus: ``all_squash``, ``anongid``, ``anonuid``, ``async``, ``auth_nlm``, ``crossmnt``, ``fsid``, ``fsuid``, ``hide``, ``insecure``, ``insecure_locks``, ``mountpoint``, ``mp``, ``no_acl``, ``no_all_squash``, ``no_auth_nlm``, ``no_root_squash``, ``no_subtree_check``, ``no_wdelay``, ``nohide``, ``refer``, ``replicas``, ``root_squash``, ``secure``, ``secure_locks``, ``subtree_check``, ``sync``, ``wdelay``. Anything else is rejected as a syntax error. **When you outgrow it, turn it off.** For per-share Samba tuning, Kerberos beyond ``sec=``, or export options outside that list, set the property to ``off`` and manage the export natively. Mixing both for the same dataset is what produces shares that reappear after a reboot or refuse to go away. NFS ~~~ .. code:: bash zfs set sharenfs=on pool/export zfs set sharenfs=rw=@192.168.1.0/24,no_root_squash pool/export zfs set sharenfs=off pool/export ``sharenfs=on`` shares with the default options ``sec=sys,rw,crossmnt,no_subtree_check``. Any other value is an option list, validated against the allowlist above and then translated into an ``/etc/exports``-style entry. Note the syntax: options are **comma-separated**, unlike `exports(5) `__. This avoids quoting and makes the property easy to generate from scripts. On FreeBSD several option sets may be given, separated by semicolons, each applying to different hosts or networks. With ``sharenfs=off``, the dataset is not managed by ZFS at all — export it the traditional way through ``/etc/exports`` and ``exportfs``. Because each dataset is its own file system, a client mounting a parent will not automatically see child datasets unless ``crossmnt`` is in effect (it is, with ``sharenfs=on``). SMB ~~~ .. code:: bash zfs set sharesmb=on pool/export On Linux this uses **Samba usershares**: ZFS invokes ``net(8)`` to create a USERSHARE. Samba must be configured to permit usershares before this works. Points to know: * The share name is derived from the dataset name, with characters that are invalid in a resource name replaced by underscores. * The share is created with the ACL ``Everyone:F`` and no guest access — Samba must be able to authenticate a real user. All finer-grained access control has to be done on the file system itself. * Linux does not support the additional ``sharesmb`` options available on Solaris. See also the Samba section of :doc:`Workload Tuning `. Applying shares ~~~~~~~~~~~~~~~ Datasets are shared and unshared automatically at boot and shutdown, and whenever the properties change: .. code:: bash zfs share -a # share everything (part of the boot process) zfs share pool/export zfs unshare pool/export zfs unshare -a ``zfs share -l`` loads keys for encrypted file systems as they are mounted; with ``keylocation=prompt`` that blocks on the terminal. iSCSI and other block exports ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ There is no ``shareiscsi`` property in OpenZFS. Export a zvol instead, and point an iSCSI target framework (LIO/``targetcli``, SCST, or ``ctld`` on FreeBSD) at the resulting device: .. code:: bash zfs create -V 100G pool/iscsi/lun0 ls -l /dev/zvol/pool/iscsi/lun0 The ``volmode`` property controls how a zvol appears to the OS: ``full`` (alias ``geom``) A fully-fledged block device, partitions included. ``dev`` A block device with its partitions hidden — usually what you want for a LUN or a VM disk, since the host should not be scanning the guest's partition table. ``none`` Not exposed outside ZFS at all. Still snapshottable, clonable and replicable — useful for backup targets. ``default`` Follow the system-wide ``zvol_volmode`` tunable. Set ``volblocksize`` at creation to match the consumer's I/O size; see :doc:`Workload Tuning `. Further reading ~~~~~~~~~~~~~~~ * `zfsprops(7) `__ — ``sharenfs``, ``sharesmb``, ``volmode``, ``volsize`` * `zfs-share(8) `__ — documents both ``zfs share`` and ``zfs unshare`` * :doc:`Datasets and Properties ` Delegated Administration ======================== ``zfs allow`` lets non-privileged users run ZFS operations on specific datasets — take their own snapshots, set their own properties, receive their own replication streams — without handing out root. Delegation is controlled by the pool property ``delegation``, which is ``on`` by default: .. code:: bash zpool get delegation pool Granting and revoking ~~~~~~~~~~~~~~~~~~~~~ .. code:: bash zfs allow alice snapshot,destroy,mount pool/home/alice zfs allow -g devs create,mount,snapshot pool/scratch zfs allow pool/home/alice # show what is delegated zfs unallow alice snapshot pool/home/alice Who: * a bare name is interpreted as ``everyone``, then a user, then a group — use ``-u`` or ``-g`` to be explicit; * ``-e`` (or the literal ``everyone``) grants to all users. Where: * ``-l`` — this dataset only ("locally"); * ``-d`` — descendants only; * neither or both — the dataset and all its descendants (the default). ``zfs unallow`` takes the same selectors, plus ``-r`` to remove the delegation recursively. What can be delegated ~~~~~~~~~~~~~~~~~~~~~ Permission names are just ZFS subcommand and property names: ``snapshot``, ``clone``, ``send``, ``receive``, ``rollback``, ``destroy``, ``hold``, ``load-key``, ``compression``, ``quota``, ``recordsize``, and so on. ``zfs allow`` with no arguments on a dataset prints the current grants. Several permissions imply others, and this is the usual source of confusion: * ``create`` also needs ``mount`` (and ``refreservation`` for non-sparse volumes); * ``destroy``, ``rollback`` and ``snapshot`` also need ``mount``; * ``clone`` needs ``create`` plus ``mount`` in the origin file system; * ``rename`` needs ``mount`` and ``create`` in the new parent; * ``receive`` needs ``mount`` and ``create``, and is what allows ``zfs receive -F`` — use ``receive:append`` for a receive permission that cannot force a rollback; * ``allow`` lets a user delegate onward, but only permissions they hold themselves. Narrower send variants exist for encrypted data: ``send:raw`` permits only raw streams, and ``send:encrypted`` permits raw streams of encrypted datasets only — both prevent an encrypted dataset from being sent in decrypted form. See :doc:`Native Encryption `. .. note:: On Linux the ``mount``, ``unmount``, ``mountpoint``, ``canmount``, ``rename`` and ``share`` permissions cannot be delegated, because ``mount(8)`` restricts changes to the global namespace to root. Since ``create``, ``destroy``, ``snapshot`` and friends require ``mount``, a fully self-service setup on Linux still needs help from root or from a setuid helper. Permission sets ~~~~~~~~~~~~~~~ A named set groups permissions so they can be granted as a unit and changed later in one place. Set names begin with ``@``. .. code:: bash zfs allow -s @backup send,snapshot,hold,destroy pool/data zfs allow backupuser @backup pool/data zfs unallow -s @backup pool/data Create-time permissions ~~~~~~~~~~~~~~~~~~~~~~~ ``zfs allow -c`` sets create-time permissions: they are granted locally to whoever creates a new descendant file system. This is how a user who is allowed to create datasets automatically becomes able to administer the ones they created. .. code:: bash zfs allow -c snapshot,destroy,mount pool/scratch Further reading ~~~~~~~~~~~~~~~ * `zfs-allow(8) `__ — the full permission table * `zfs-unallow(8) `__ * `zpoolprops(7) `__ — ``delegation`` Pools at Boot ============= Nothing about a ZFS pool is described in ``/etc/fstab`` by default. Pools are discovered, imported, mounted and shared by ZFS itself, which is convenient until something does not come up and there is no fstab line to look at. This page covers what happens between power-on and a mounted dataset. The order of events ~~~~~~~~~~~~~~~~~~~ #. **Pools are imported.** Either from the cache file, or by scanning devices. #. **Datasets are mounted**, in mountpoint order, by ``zfs mount -a`` or by generated systemd mount units. #. **Shares are published** by ``zfs share -a`` — see :doc:`Sharing Datasets `. #. **zvols appear** under ``/dev/zvol``; anything waiting on them has to wait for that. #. **zed starts**, so events from this point on are acted upon. A root-on-ZFS system does step 1 earlier and differently: the root pool is imported from the initramfs (Linux) or by the loader (FreeBSD) before the real root can be mounted at all. The :doc:`Root on ZFS guides ` cover that case. The cache file ~~~~~~~~~~~~~~ ``/etc/zfs/zpool.cache`` records which pools were imported, so the system can bring them back without scanning every block device. **Every pool listed there is imported automatically at boot.** The pool property ``cachefile`` controls this: * the default (empty string) uses the standard location; * ``none`` means the pool is never cached — it will *not* come back automatically, which is what you want for removable media, clustering and failover; * an explicit path caches it elsewhere, to be imported later with ``zpool import -c ``. .. code:: bash zpool get cachefile pool zpool set cachefile=none pool # stop importing it at boot zpool set cachefile= pool # back to the default The file is a binary nvlist; ``zdb`` with no arguments prints its contents. If it is stale or missing, a pool can still be found by scanning (``zpool import -d``), which is what the ``zfs-import-scan`` path does. Two common surprises: a pool you imported once keeps coming back after reboot because it is in the cache, and a pool imported with ``-R`` (which sets ``cachefile=none``) does *not* come back, by design. See :doc:`Troubleshooting `. hostid and importing on the wrong machine ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ A pool records the ``hostid`` of the system that imported it. If another system sees a pool that a different host still has imported, it refuses to import it — this is the "pool was last accessed by another system" message, and ``-f`` overrides it. That guard is advisory. It compares a recorded id; it does not check whether the other host is still writing. Forcing an import of a pool that is genuinely active elsewhere will corrupt it. On Linux the id comes from ``/etc/hostid``; generate one with ``zgenhostid`` if it is missing. A missing or duplicated hostid is a classic cause of pools that either refuse to import after a hostname change or fail to protect themselves in a cluster. For shared storage where two hosts really can reach the same devices, the ``multihost`` pool property turns the advisory check into an active one: hosts write periodic heartbeats, and an import is refused — *even with* ``-f`` — while the pool looks alive. Each host must have a unique hostid for it to work. .. code:: bash zgenhostid # if /etc/hostid is missing zpool set multihost=on pool Note what ``multihost`` does not do: it protects the pool at import time, not individual devices from being claimed by two different pools. systemd ~~~~~~~ On Linux the sequence is a set of units, and knowing their names is most of the battle: ``zfs-import-cache.service`` Imports the pools listed in ``zpool.cache``. ``zfs-import-scan.service`` Imports by scanning devices instead. Normally the alternative to the cache service, not a companion to it. ``zfs-import.target`` Reached once importing is done; things needing pools order after it. ``zfs-mount.service`` Runs ``zfs mount -a``. ``zfs-share.service`` Runs ``zfs share -a``. ``zfs-volume-wait.service`` / ``zfs-volumes.target`` Waits until zvol device nodes exist. ``zfs-zed.service`` The event daemon. ``zfs.target`` The aggregate; enable or disable ZFS at boot here. .. code:: bash systemctl list-units 'zfs*' systemctl status zfs-import-cache.service journalctl -u zfs-import-cache.service -b There are also the periodic maintenance timers — ``zfs-scrub-weekly@.timer``, ``zfs-scrub-monthly@.timer`` and the matching ``zfs-trim-*`` ones. See :doc:`Scrub and Resilver ` and :doc:`TRIM `. ``zfs-mount-generator`` ^^^^^^^^^^^^^^^^^^^^^^^ Mounting everything with one ``zfs mount -a`` late in boot means systemd knows nothing about the individual mounts, so units that need a particular dataset cannot order themselves against it. ``zfs-mount-generator`` solves that by generating a native ``.mount`` unit per dataset at generator time, from a cached list of dataset properties. Datasets with ``mountpoint=legacy`` or ``none`` are skipped, as are those with ``canmount=off``. This is what to reach for when a service starts before its data is mounted. It also lets an encrypted dataset's key be requested through the normal systemd credential machinery rather than blocking a shell — see :doc:`Native Encryption `. FreeBSD ~~~~~~~ The cache file lives at ``/boot/zfs/zpool.cache``. Pools are imported by the ``zfs`` rc script, enabled with ``zfs_enable="YES"`` in ``/etc/rc.conf``, and the loader imports the root pool before the kernel starts. The concepts — cache file, hostid, mount then share — are the same. Further reading ~~~~~~~~~~~~~~~ * `zpoolprops(7) `__ — ``cachefile``, ``multihost`` * `zfs-mount-generator(8) `__, `zgenhostid(8) `__ * `zpool-import(8) `__ * :doc:`Troubleshooting `, :doc:`Root on ZFS guides ` Troubleshooting =============== Where to look when a pool misbehaves, how to read what ZFS tells you, and what to collect before filing a bug. First look ~~~~~~~~~~ .. code:: bash zpool status -v # health, and the list of damaged files zpool status -x # only pools with problems zpool status -es # unhealthy vdevs only, with slow-I/O counts zpool events -v # the event history, most detail dmesg -T | grep -i zfs # kernel-level errors zfs version ``zpool status -x`` printing ``all pools are healthy`` rules out most of what follows. Reading ``zpool status`` ~~~~~~~~~~~~~~~~~~~~~~~~ Pool health is one of three states: **ONLINE** (everything normal), **DEGRADED** (a device failed but redundancy is covering it), or **FAULTED** (corrupted metadata, or too few replicas left to function). Individual vdevs report more detail: ``ONLINE`` Working. ``DEGRADED`` Checksum errors, slow I/Os, or I/O errors exceed acceptable levels. ZFS keeps using the device, flagging that something may be wrong. ``FAULTED`` The device opened but its contents did not match expectations, or its error count is high enough that ZFS stopped using it. ``OFFLINE`` Taken offline deliberately with ``zpool offline``. ``REMOVED`` Physically pulled while running. Detection is hardware-dependent. ``UNAVAIL`` Could not be opened at all. If the pool was imported while the device was missing, it shows a GUID rather than a path — the path was never valid. The three counter columns mean different things, and the distinction matters: ``READ`` / ``WRITE`` I/O errors reported by the device itself. ``CKSUM`` The device returned data that failed its checksum — silent corruption. A damaged block that ZFS could reconstruct is charged to the disk that held the bad copy. A block it could **not** reconstruct — three damaged disks in a raidz2, say — cannot be attributed, so the checksum error is reported against *every* disk holding that block. A row of disks all showing CKSUM errors is therefore usually one unrecoverable block, not simultaneous failure of the whole row. Non-zero CKSUM on a disk that reports no I/O errors more often indicates cabling, an HBA, or memory than the disk. ``zpool clear pool`` resets the counters. It does not repair anything — fix the cause first, then clear, then watch whether the counters come back. Permanent errors ~~~~~~~~~~~~~~~~ ``zpool status -v`` lists files with unrepairable errors. That data is gone: restore those files from a backup or a snapshot. If the listing shows a hex object number instead of a path, the file has already been deleted, or the dataset is not mounted. Entries persist until a scrub confirms they are gone, so run a scrub after restoring. See :doc:`Scrub and Resilver `. Errors in metadata rather than a file, or the ```` marker, are more serious — that is the case redundancy exists for, and without it the pool may not be fully recoverable. A pool that will not import ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Work up this ladder, least destructive first. **The options marked below destroy data** — read the next section before using any of them. .. code:: bash zpool import # what is visible at all zpool import -d /dev/disk/by-id # scan a specific path zpool import -o readonly=on -N pool # look without changing anything zpool import -f pool # last used by another system zpool import -m pool # LOSES DATA: discards a missing log zpool import -F -n pool # dry run: is rewind possible? zpool import -F pool # LOSES DATA: discards transactions zpool import -FX pool # LOSES MORE DATA: extended rewind ``zpool import`` with no arguments and ``-d`` are safe — they only scan. ``-f`` overrides the "pool was last used by another system" guard; it does not itself discard anything, but importing a pool that is genuinely still active on another host will corrupt it. ``-o readonly=on -N`` is the right first move on a pool you care about: it imports without mounting and without writing, so you can look before deciding. What each destructive option actually costs ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ``-m`` — **discards the intent log.** This is data loss, and it is easy to miss because the flag reads like a convenience. A missing or dead SLOG holds synchronous writes that were acknowledged to applications but not yet written to the pool. Importing with ``-m`` throws them away: the pool comes up consistent, but recent transactions the application was told were durable are gone. For a database or an NFS server that is silent corruption at the application level, not at the ZFS level — ZFS will report the pool as healthy. Before reaching for ``-m``, try to make the log device readable again. This is also the argument for mirroring a SLOG. ``-F`` — **discards the last few transaction groups.** Rewinds the pool to an earlier consistent state. Everything written after that point is irretrievably lost, and not every damaged pool can be recovered this way. Always run ``-F -n`` first: it reports whether the rewind would succeed without performing it. ``-X`` — **extended rewind.** Searches further back in time, so it can lose correspondingly more. Use only after ``-F`` has failed. If the pool has a checkpoint, ``zpool import --rewind-to-checkpoint`` is a cleaner and better-defined undo than ``-F``, because you chose the point it returns to. Note that it too is irreversible once imported without ``readonly``. See :doc:`Changing Pool Layout ` for checkpoints. A pool that was **destroyed** is not immediately gone. ``zpool import -D`` lists destroyed pools, and importing with ``-D`` brings one back, as long as its devices have not been reused: .. code:: bash zpool import -D # what can still be recovered zpool import -D pool Importing somewhere unusual — a rescue environment, or a pool whose mount points would collide with the running system — is what ``-R`` is for. It sets ``altroot``, so every mount point is prefixed, and ``cachefile=none`` so the import is not remembered: .. code:: bash zpool import -R /mnt -N pool Device names that changed between boots are a common cause of a pool that "disappeared" — import with ``-d /dev/disk/by-id`` and see the :doc:`FAQ ` on selecting ``/dev/`` names. A suspended pool ~~~~~~~~~~~~~~~~ When connectivity to the underlying devices is lost, the pool is suspended and the ``failmode`` property decides what happens: ``wait`` The default. Blocks all I/O until connectivity returns and the errors are cleared with ``zpool clear``. ``continue`` Returns ``EIO`` to new writes but still serves reads from healthy devices. Uncommitted writes stay blocked. ``panic`` Prints a message and crashes the system, generating a dump. A suspended pool with ``failmode=wait`` looks exactly like a hang: commands touching it block forever and are unkillable. Check ``zpool status`` for another pool, and ``dmesg``, before concluding ZFS has deadlocked. Hung processes and slow I/O ~~~~~~~~~~~~~~~~~~~~~~~~~~~ *Symptom:* a ``zfs`` or ``zpool`` command never returns and cannot be killed. *Usual cause:* a stuck kernel thread, a suspended pool, or genuinely slow hardware. ZFS has a *deadman* timer for this. A pool sync taking longer than ``zfs_deadman_synctime_ms``, or a single I/O longer than ``zfs_deadman_ziotime_ms``, is declared hung. With the default ``zfs_deadman_failmode=wait`` this is only *logged* — so the evidence is in the logs, not on your terminal. The deadman is disabled automatically once a pool is suspended. Slow but not hung storage is easier to see directly: .. code:: bash zpool status -s # slow I/O count per leaf vdev zpool iostat -l 1 # latency breakdown zpool iostat -w # latency histograms ``-s`` counts I/Os that took longer than ``zio_slow_io_ms`` (30 s by default). They did not necessarily fail — but a single disk accumulating them is the classic signature of a drive about to go, and on RAIDZ it drags the whole group down (see :doc:`RAIDZ `). Log files ~~~~~~~~~ **Kernel log.** ``dmesg -T``, ``/var/log/syslog``, or wherever ``rsyslogd`` sends kernel messages. Stack traces from stuck threads land here — sometimes only once the deadman timer expires. **ZFS debug messages.** The modules keep an internal log buffer, readable at ``/proc/spl/kstat/zfs/dbgmsg``, when the module parameter `zfs_dbgmsg_enable <../../Performance and Tuning/Module Parameters.html#zfs-dbgmsg-enable>`__ is 1. This is frequently what developers ask for. **Per-pool I/O statistics.** ``/proc/spl/kstat/zfs//`` on Linux. **What was done to this pool.** ``zpool history`` replays the administrative commands the pool has seen — stored in the pool itself, so it survives reboots and moves with the pool to another host. Frequently the fastest way to answer "what changed?". .. code:: bash zpool history pool zpool history -l pool # who, from which host zpool history -i pool # include internally logged events Logging infrastructure — elasticsearch, fluentd, influxdb, splunk — makes correlating these with system events considerably easier. ZFS events ~~~~~~~~~~ ZFS reports notable events through an event channel. The ZFS Event Daemon (``zed``) listens and acts on them; it is extensible, so scripts can subscribe and take action. The script usually installed at ``/etc/zfs/zed.d/all-syslog.sh`` writes a formatted message to syslog. See ``zed(8)``. .. code:: bash zpool events # history since module load, all pools zpool events -v # full detail per event zpool events -f # follow The history lives in RAM and is capped by `zfs_zevent_len_max <../../Performance and Tuning/Module Parameters.html#zfs-zevent-len-max>`__. ``zed`` throttles internally so a failing device cannot exhaust the system. Each event carries a class used for filtering: ``sysevent.fs.zfs.*`` Pool management — import, export, configuration updates, ``zpool history`` entries. ``ereport.*`` Errors. One fault often produces several, as each layer reports it: a failing disk can raise an ``ereport.io`` on the read and an ``ereport.fs.zfs.checksum`` at the pool level. These correspond to the counters in ``zpool status`` — if you see errors there, matching ereports exist here. ``ereport.fs.zfs.delay`` A slow I/O, including a RAIDZ child being put in sit-out. The exact contents of verbose events change between versions; treat them as diagnostic detail, not a stable interface. Reporting a problem ~~~~~~~~~~~~~~~~~~~ Before opening an issue, confirm it is a bug rather than a support question — the :doc:`mailing lists ` are the place for the latter. Then collect: * ``zfs version`` and the distribution and kernel version * ``zpool status -v`` and ``zpool get all `` * the relevant part of ``dmesg`` and of ``/proc/spl/kstat/zfs/dbgmsg`` * ``zpool events -v`` around the failure * non-default module parameters * whether the system uses ECC memory, and whether it is virtualised * a reproducer, if you have one — this matters more than anything else above The :doc:`FAQ ` has the full list and the issue tracker link. Further reading ~~~~~~~~~~~~~~~ * `zpool-status(8) `__, `zpool-events(8) `__, `zpool-import(8) `__ * `zpoolconcepts(7) `__ — ``Device Failure and Recovery`` * `zed(8) `__ * :doc:`ZFS Messages ` — the ``ZFS-8000-*`` codes from ``zpool status`` Project and Community ===================== OpenZFS is storage software which combines the functionality of traditional filesystems, volume manager, and more. OpenZFS includes protection against data corruption, support for high storage capacities, efficient data compression, snapshots and copy-on-write clones, continuous integrity checking and automatic repair, remote replication with ZFS send and receive, and RAID-Z. OpenZFS brings together developers from the illumos, Linux, FreeBSD and OS X platforms, and a wide range of companies -- both online and at the annual OpenZFS Developer Summit. High-level goals of the project include raising awareness of the quality, utility and availability of open-source implementations of ZFS, encouraging open communication about ongoing efforts toward improving open-source variants of ZFS, and ensuring consistent reliability, functionality and performance of all distributions of ZFS. Administration is covered by :doc:`Basic Concepts `, :doc:`Performance and Tuning ` and the `man pages <../man/index.html>`__. .. toctree:: :maxdepth: 2 :caption: Contents: :glob: Donate FAQ Mailing Lists Signing Keys Issue Tracker Releases Roadmap Donate ====== OpenZFS operates under the umbrella of `Software In The Public Interest (SPI) `_. SPI is a 501(c)(3) non-profit organization that provides financial, legal, and organization services for open source software projects. OpenZFS accepts donations via SPI at our `donation page `_. We thank you for your support! FAQ === .. contents:: Table of Contents :local: What is OpenZFS --------------- OpenZFS is an outstanding storage platform that encompasses the functionality of traditional filesystems, volume managers, and more, with consistent reliability, functionality and performance across all distributions. Additional information about OpenZFS can be found in the `OpenZFS wikipedia article `__. Hardware Requirements --------------------- Because ZFS was originally designed for Sun Solaris it was long considered a filesystem for large servers and for companies that could afford the best and most powerful hardware available. But since the porting of ZFS to numerous OpenSource platforms (The BSDs, Illumos and Linux - under the umbrella organization "OpenZFS"), these requirements have been lowered. The suggested hardware requirements are: - ECC memory. This isn't really a requirement, but it's highly recommended. - 8GB+ of memory for the best performance. It's perfectly possible to run with 2GB or less (and people do), but you'll need more if using deduplication. Do I have to use ECC memory for ZFS? ------------------------------------ Using ECC memory for OpenZFS is strongly recommended for enterprise environments where the strongest data integrity guarantees are required. Without ECC memory rare random bit flips caused by cosmic rays or by faulty memory can go undetected. If this were to occur OpenZFS (or any other filesystem) will write the damaged data to disk and be unable to automatically detect the corruption. Unfortunately, ECC memory is not always supported by consumer grade hardware. And even when it is, ECC memory will be more expensive. For home users the additional safety brought by ECC memory might not justify the cost. It's up to you to determine what level of protection your data requires. Installation ------------ OpenZFS is available for FreeBSD and all major Linux distributions. Refer to the :doc:`getting started <../Getting Started/index>` section for links to installation instructions. If your distribution/OS isn't listed you can always build OpenZFS from the latest official `tarball `__. Supported Architectures ----------------------- OpenZFS is regularly compiled for the following architectures: aarch64, arm, ppc, ppc64, x86, x86_64. Supported Linux Kernels ----------------------- The `notes `__ for a given OpenZFS release will include a range of supported kernels. Point releases will be tagged as needed in order to support the *stable* kernel available from `kernel.org `__. The authoritative range for any given branch is the ``Linux-Minimum`` and ``Linux-Maximum`` pair in the `META `__ file of that branch. Older kernels are dropped as enterprise distributions retire them — the minimum has risen well past the 2.6.32 that early ZFS on Linux releases supported. .. _32-bit-vs-64-bit-systems: 32-bit vs 64-bit Systems ------------------------ You are **strongly** encouraged to use a 64-bit kernel. OpenZFS will build for 32-bit systems but you may encounter stability problems. ZFS was originally developed for the Solaris kernel which differs from some OpenZFS platforms in several significant ways. Perhaps most importantly for ZFS it is common practice in the Solaris kernel to make heavy use of the virtual address space. However, use of the virtual address space is strongly discouraged in the Linux kernel. This is particularly true on 32-bit architectures where the virtual address space is limited to 100M by default. Using the virtual address space on 64-bit Linux kernels is also discouraged but the address space is so much larger than physical memory that it is less of an issue. If you are bumping up against the virtual memory limit on a 32-bit system you will see the following message in your system logs. You can increase the virtual address size with the boot option ``vmalloc=512M``. :: vmap allocation for size 4198400 failed: use vmalloc= to increase size. However, even after making this change your system will likely not be entirely stable. Proper support for 32-bit systems would require weaning the OpenZFS code off its dependence on virtual memory, which has not happened. 32-bit systems should be treated as unsupported in practice. Booting from ZFS ---------------- Booting from ZFS on Linux is possible and many people do it. There are excellent walk throughs available for :doc:`Debian <../Getting Started/Debian/index>`, :doc:`Ubuntu <../Getting Started/Ubuntu/index>`, and `Gentoo `__. On FreeBSD 13+ booting from ZFS is supported out of the box. Selecting /dev/ names when creating a pool (Linux) -------------------------------------------------- There are different /dev/ names that can be used when creating a ZFS pool. Each option has advantages and drawbacks, the right choice for your ZFS pool really depends on your requirements. For development and testing using /dev/sdX naming is quick and easy. A typical home server might prefer /dev/disk/by-id/ naming for simplicity and readability. While very large configurations with multiple controllers, enclosures, and switches will likely prefer /dev/disk/by-vdev naming for maximum control. But in the end, how you choose to identify your disks is up to you. - **/dev/sdX, /dev/hdX:** Best for development/test pools - Summary: The top level /dev/ names are the default for consistency with other ZFS implementations. They are available under all Linux distributions and are commonly used. However, because they are not persistent they should only be used with ZFS for development/test pools. - Benefits: This method is easy for a quick test, the names are short, and they will be available on all Linux distributions. - Drawbacks: The names are not persistent and will change depending on what order the disks are detected in. Adding or removing hardware for your system can easily cause the names to change. You would then need to remove the zpool.cache file and re-import the pool using the new names. - Example: ``zpool create tank sda sdb`` - **/dev/disk/by-id/:** Best for small pools (less than 10 disks) - Summary: This directory contains disk identifiers with more human readable names. The disk identifier usually consists of the interface type, vendor name, model number, device serial number, and partition number. This approach is more user friendly because it simplifies identifying a specific disk. - Benefits: Nice for small systems with a single disk controller. Because the names are persistent and guaranteed not to change, it doesn't matter how the disks are attached to the system. You can take them all out, randomly mix them up on the desk, put them back anywhere in the system and your pool will still be automatically imported correctly. - Drawbacks: Configuring redundancy groups based on physical location becomes difficult and error prone. Unreliable on many personal virtual machine setups because the software does not generate persistent unique names by default. - Example: ``zpool create tank scsi-SATA_Hitachi_HTS7220071201DP1D10DGG6HMRP`` - **/dev/disk/by-path/:** Good for large pools (greater than 10 disks) - Summary: This approach is to use device names which include the physical cable layout in the system, which means that a particular disk is tied to a specific location. The name describes the PCI bus number, as well as enclosure names and port numbers. This allows the most control when configuring a large pool. - Benefits: Encoding the storage topology in the name is not only helpful for locating a disk in large installations. But it also allows you to explicitly layout your redundancy groups over multiple adapters or enclosures. - Drawbacks: These names are long, cumbersome, and difficult for a human to manage. - Example: ``zpool create tank pci-0000:00:1f.2-scsi-0:0:0:0 pci-0000:00:1f.2-scsi-1:0:0:0`` - **/dev/disk/by-vdev/:** Best for large pools (greater than 10 disks) - Summary: This approach provides administrative control over device naming using the configuration file /etc/zfs/vdev_id.conf. Names for disks in JBODs can be generated automatically to reflect their physical location by enclosure IDs and slot numbers. The names can also be manually assigned based on existing udev device links, including those in /dev/disk/by-path or /dev/disk/by-id. This allows you to pick your own unique meaningful names for the disks. These names will be displayed by all the zfs utilities so it can be used to clarify the administration of a large complex pool. See the vdev_id and vdev_id.conf man pages for further details. - Benefits: The main benefit of this approach is that it allows you to choose meaningful human-readable names. Beyond that, the benefits depend on the naming method employed. If the names are derived from the physical path the benefits of /dev/disk/by-path are realized. On the other hand, aliasing the names based on drive identifiers or WWNs has the same benefits as using /dev/disk/by-id. - Drawbacks: This method relies on having a /etc/zfs/vdev_id.conf file properly configured for your system. To configure this file please refer to section `Setting up the /etc/zfs/vdev_id.conf file <#setting-up-the-etc-zfs-vdev-id-conf-file>`__. As with benefits, the drawbacks of /dev/disk/by-id or /dev/disk/by-path may apply depending on the naming method employed. - Example: ``zpool create tank mirror A1 B1 mirror A2 B2`` - **/dev/disk/by-uuid/:** Not a great option - Summary: One might think from the use of "UUID" that this would be an ideal option - however, in practice, this ends up listing one device per **pool** ID, which is not very useful for importing pools with multiple disks. - **/dev/disk/by-partuuid/**/**by-partlabel:** Works only for existing partitions - Summary: partition UUID is generated on it's creation, so usage is limited - Drawbacks: you can't refer to a partition unique ID on an unpartitioned disk for ``zpool replace``/``add``/``attach``, and you can't find failed disk easily without a mapping written down ahead of time. Setting up the /etc/zfs/vdev_id.conf file ----------------------------------------- In order to use /dev/disk/by-vdev/ naming the ``/etc/zfs/vdev_id.conf`` must be configured. The format of this file is described in the vdev_id.conf man page. Several examples follow. A non-multipath configuration with direct-attached SAS enclosures and an arbitrary slot re-mapping. :: multipath no topology sas_direct phys_per_port 4 # PCI_SLOT HBA PORT CHANNEL NAME channel 85:00.0 1 A channel 85:00.0 0 B # Linux Mapped # Slot Slot slot 0 2 slot 1 6 slot 2 0 slot 3 3 slot 4 5 slot 5 7 slot 6 4 slot 7 1 A SAS-switch topology. Note that the channel keyword takes only two arguments in this example. :: topology sas_switch # SWITCH PORT CHANNEL NAME channel 1 A channel 2 B channel 3 C channel 4 D A multipath configuration. Note that channel names have multiple definitions - one per physical path. :: multipath yes # PCI_SLOT HBA PORT CHANNEL NAME channel 85:00.0 1 A channel 85:00.0 0 B channel 86:00.0 1 A channel 86:00.0 0 B A configuration using device link aliases. :: # by-vdev # name fully qualified or base name of device link alias d1 /dev/disk/by-id/wwn-0x5000c5002de3b9ca alias d2 wwn-0x5000c5002def789e After defining the new disk names run ``udevadm trigger`` to prompt udev to parse the configuration file. This will result in a new /dev/disk/by-vdev directory which is populated with symlinks to /dev/sdX names. Following the first example above, you could then create the new pool of mirrors with the following command: :: $ zpool create tank \ mirror A0 B0 mirror A1 B1 mirror A2 B2 mirror A3 B3 \ mirror A4 B4 mirror A5 B5 mirror A6 B6 mirror A7 B7 $ zpool status pool: tank state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 A0 ONLINE 0 0 0 B0 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 A1 ONLINE 0 0 0 B1 ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 A2 ONLINE 0 0 0 B2 ONLINE 0 0 0 mirror-3 ONLINE 0 0 0 A3 ONLINE 0 0 0 B3 ONLINE 0 0 0 mirror-4 ONLINE 0 0 0 A4 ONLINE 0 0 0 B4 ONLINE 0 0 0 mirror-5 ONLINE 0 0 0 A5 ONLINE 0 0 0 B5 ONLINE 0 0 0 mirror-6 ONLINE 0 0 0 A6 ONLINE 0 0 0 B6 ONLINE 0 0 0 mirror-7 ONLINE 0 0 0 A7 ONLINE 0 0 0 B7 ONLINE 0 0 0 errors: No known data errors Changing /dev/ names on an existing pool ---------------------------------------- Changing the /dev/ names on an existing pool can be done by simply exporting the pool and re-importing it with the -d option to specify which new names should be used. For example, to use the custom names in /dev/disk/by-vdev: :: $ zpool export tank $ zpool import -d /dev/disk/by-vdev tank .. _the-etczfszpoolcache-file: The /etc/zfs/zpool.cache file ----------------------------- Whenever a pool is imported on the system it will be added to the ``/etc/zfs/zpool.cache`` file. This file stores pool configuration information, such as the device names and pool state. If this file exists when running the ``zpool import`` command then it will be used to determine the list of pools available for import. When a pool is not listed in the cache file it will need to be detected and imported using the ``zpool import -d /dev/disk/by-id`` command. .. _generating-a-new-etczfszpoolcache-file: Generating a new /etc/zfs/zpool.cache file ------------------------------------------ The ``/etc/zfs/zpool.cache`` file will be automatically updated when your pool configuration is changed. However, if for some reason it becomes stale you can force the generation of a new ``/etc/zfs/zpool.cache`` file by setting the cachefile property on the pool. :: $ zpool set cachefile=/etc/zfs/zpool.cache tank Conversely the cache file can be disabled by setting ``cachefile=none``. This is useful for failover configurations where the pool should always be explicitly imported by the failover software. :: $ zpool set cachefile=none tank Sending and Receiving Streams ----------------------------- For how ``zfs send`` and ``zfs receive`` work in general, see :doc:`Send and Receive `. The entries below cover known caveats. hole_birth Bugs ~~~~~~~~~~~~~~~ The hole_birth feature has/had bugs, the result of which is that, if you do a ``zfs send -i`` (or ``-R``, since it uses ``-i``) from an affected dataset, the receiver *will not see any checksum or other errors, but will not match the source*. ZoL versions 0.6.5.8 and 0.7.0-rc1 (and above) default to ignoring the faulty metadata which causes this issue *on the sender side*. For more details, see the :doc:`hole_birth FAQ <./FAQ hole birth>`. Sending Large Blocks ~~~~~~~~~~~~~~~~~~~~ When sending incremental streams which contain large blocks (>128K) the ``--large-block`` flag must be specified. Inconsistent use of the flag between incremental sends can result in files being incorrectly zeroed when they are received. Raw encrypted send/recvs automatically imply the ``--large-block`` flag and are therefore unaffected. For more details, see `issue 6224 `__. CEPH/ZFS -------- There is a lot of tuning that can be done that's dependent on the workload that is being put on CEPH/ZFS, as well as some general guidelines. Some are as follow; ZFS Configuration ~~~~~~~~~~~~~~~~~ The CEPH filestore back-end heavily relies on xattrs, for optimal performance all CEPH workloads will benefit from the following ZFS dataset parameters - ``xattr=sa`` - ``dnodesize=auto`` Beyond that typically rbd/cephfs focused workloads benefit from small recordsize({16K-128K), while objectstore/s3/rados focused workloads benefit from large recordsize (128K-1M). .. _ceph-configuration-cephconf: CEPH Configuration (ceph.conf) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Additionally CEPH sets various values internally for handling xattrs based on the underlying filesystem. As CEPH only officially supports/detects XFS and BTRFS, for all other filesystems it falls back to rather `limited "safe" values `__. On newer releases, the need for larger xattrs will prevent OSD's from even starting. The officially recommended workaround (`see here `__) has some severe downsides, and more specifically is geared toward filesystems with "limited" xattr support such as ext4. ZFS does not have a limit internally to xattrs length, as such we can treat it similarly to how CEPH treats XFS. We can set overrides to set 3 internal values to the same as those used with XFS(`see here `__ and `here `__) and allow it be used without the severe limitations of the "official" workaround. :: [osd] filestore_max_inline_xattrs = 10 filestore_max_inline_xattr_size = 65536 filestore_max_xattr_value_size = 65536 Other General Guidelines ~~~~~~~~~~~~~~~~~~~~~~~~ - Use a separate journal device. Do not collocate CEPH journal on ZFS dataset if at all possible, this will quickly lead to terrible fragmentation, not to mention terrible performance upfront even before fragmentation (CEPH journal does a dsync for every write). - Use a SLOG device, even with a separate CEPH journal device. For some workloads, skipping SLOG and setting ``logbias=throughput`` may be acceptable. - Use a high-quality SLOG/CEPH journal device. A consumer based SSD, or even NVMe WILL NOT DO (Samsung 830, 840, 850, etc) for a variety of reasons. CEPH will kill them quickly, on-top of the performance being quite low in this use. Generally recommended devices are [Intel DC S3610, S3700, S3710, P3600, P3700], or [Samsung SM853, SM863], or better. - If using a high quality SSD or NVMe device (as mentioned above), you CAN share SLOG and CEPH Journal to good results on single device. A ratio of 4 HDDs to 1 SSD (Intel DC S3710 200GB), with each SSD partitioned (remember to align!) to 4x10GB (for ZIL/SLOG) + 4x20GB (for CEPH journal) has been reported to work well. Again - CEPH + ZFS will KILL a consumer based SSD VERY quickly. Even ignoring the lack of power-loss protection, and endurance ratings, you will be very disappointed with performance of consumer based SSD under such a workload. Performance Considerations -------------------------- To achieve good performance with your pool there are some easy best practices you should follow. See also :doc:`Workload Tuning <../Performance and Tuning/Workload Tuning>` and :doc:`Caching and Auxiliary Devices `. - **Evenly balance your disks across controllers:** Often the limiting factor for performance is not the disks but the controller. By balancing your disks evenly across controllers you can often improve throughput. - **Create your pool using whole disks:** When running zpool create use whole disk names. This will allow ZFS to automatically partition the disk to ensure correct alignment. It will also improve interoperability with other OpenZFS implementations which honor the wholedisk property. - **Have enough memory:** A minimum of 2GB of memory is recommended for ZFS. Additional memory is strongly recommended when the compression and deduplication features are enabled. - **Improve performance by setting ashift=12:** You may be able to improve performance for some workloads by setting ``ashift=12``. This tuning can only be set when block devices are first added to a pool, such as when the pool is first created or when a new vdev is added to the pool. This tuning parameter can result in a decrease of capacity for RAIDZ configurations. .. _advanced-format-disks: Advanced Format Disks --------------------- Advanced Format (AF) is a new disk format which natively uses a 4,096 byte, instead of 512 byte, sector size. To maintain compatibility with legacy systems many AF disks emulate a sector size of 512 bytes. By default, ZFS will automatically detect the sector size of the drive. This combination can result in poorly aligned disk accesses which will greatly degrade the pool performance. Therefore, the ability to set the ashift property has been added to the zpool command. This allows users to explicitly assign the sector size when devices are first added to a pool (typically at pool creation time or adding a vdev to the pool). The ashift values range from 9 to 16 with the default value 0 meaning that zfs should auto-detect the sector size. This value is actually a bit shift value, so an ashift value for 512 bytes is 9 (2^9 = 512) while the ashift value for 4,096 bytes is 12 (2^12 = 4,096). To force the pool to use 4,096 byte sectors at pool creation time, you may run: :: $ zpool create -o ashift=12 tank mirror sda sdb To force the pool to use 4,096 byte sectors when adding a vdev to a pool, you may run: :: $ zpool add -o ashift=12 tank mirror sdc sdd ZVOL used space larger than expected ------------------------------------ | Depending on the filesystem used on the zvol (e.g. ext4) and the usage (e.g. deletion and creation of many files) the ``used`` and ``referenced`` properties reported by the zvol may be larger than the "actual" space that is being used as reported by the consumer. | This can happen due to the way some filesystems work, in which they prefer to allocate files in new untouched blocks rather than the fragmented used blocks marked as free. This forces zfs to reference all blocks that the underlying filesystem has ever touched. | This is in itself not much of a problem, as when the ``used`` property reaches the configured ``volsize`` the underlying filesystem will start reusing blocks. But the problem arises if it is desired to snapshot the zvol, as the space referenced by the snapshots will contain the unused blocks. | This issue can be prevented, by issuing the so-called trim (for ex. ``fstrim`` command on Linux) to allow the kernel to specify to zfs which blocks are unused. | Issuing a trim before a snapshot is taken will ensure a minimum snapshot size. | For Linux adding the ``discard`` option for the mounted ZVOL in ``/etc/fstab`` effectively enables the kernel to issue the trim commands continuously, without the need to execute fstrim on-demand. Using a zvol for a swap device on Linux --------------------------------------- You may use a zvol as a swap device but you'll need to configure it appropriately. **CAUTION:** for now swap on zvol may lead to deadlock, in this case please send your logs `here `__. - Set the volume block size to match your systems page size. This tuning prevents ZFS from having to perform read-modify-write options on a larger block while the system is already low on memory. - Set the ``logbias=throughput`` and ``sync=always`` properties. Data written to the volume will be flushed immediately to disk freeing up memory as quickly as possible. - Set ``primarycache=metadata`` to avoid keeping swap data in RAM via the ARC. - Disable automatic snapshots of the swap device. :: $ zfs create -V 4G -b $(getconf PAGESIZE) \ -o logbias=throughput -o sync=always \ -o primarycache=metadata \ -o com.sun:auto-snapshot=false rpool/swap Using ZFS on Xen Hypervisor or Xen Dom0 (Linux) ----------------------------------------------- It is usually recommended to keep virtual machine storage and hypervisor pools, quite separate. Although few people have managed to successfully deploy and run OpenZFS using the same machine configured as Dom0. There are few caveats: - Set a fair amount of memory in grub.conf, dedicated to Dom0. - dom0_mem=16384M,max:16384M - Allocate no more of 30-40% of Dom0's memory to ZFS in ``/etc/modprobe.d/zfs.conf``. - options zfs zfs_arc_max=6442450944 - Disable Xen's auto-ballooning in ``/etc/xen/xl.conf`` - Watch out for any Xen bugs, such as `this one `__ related to ballooning udisks2 creating /dev/mapper/ entries for zvol (Linux) ------------------------------------------------------ To prevent udisks2 from creating /dev/mapper entries that must be manually removed or maintained during zvol remove / rename, create a udev rule such as ``/etc/udev/rules.d/80-udisks2-ignore-zfs.rules`` with the following contents: :: ENV{ID_PART_ENTRY_SCHEME}=="gpt", ENV{ID_FS_TYPE}=="zfs_member", ENV{ID_PART_ENTRY_TYPE}=="6a898cc3-1dd2-11b2-99a6-080020736631", ENV{UDISKS_IGNORE}="1" Licensing --------- License information can be found on the :doc:`License <../License>` page. Reporting a problem ------------------- You can open a new issue and search existing issues using the public `issue tracker `__. The issue tracker is used to organize outstanding bug reports, feature requests, and other development tasks. Anyone may post comments after signing up for a github account. Please make sure that what you're actually seeing is a bug and not a support issue. If in doubt, please ask on the mailing list first, and if you're then asked to file an issue, do so. When opening a new issue include this information at the top of the issue: - What distribution you're using and the version. - What OpenZFS packages you're using and the version, as reported by ``zfs version``. - Describe the problem you're observing. - Describe how to reproduce the problem. - Including any warning/errors/backtraces from the system logs. When a new issue is opened it's not uncommon for a developer to request additional information about the problem. In general, the more detail you share about a problem the quicker a developer can resolve it. For example, providing a simple test case is always exceptionally helpful. Be prepared to work with the developer looking in to your bug in order to get it resolved. They may ask for information like: - Your pool configuration as reported by ``zdb`` or ``zpool status``. - Your hardware configuration, such as - Number of CPUs. - Amount of memory. - Whether your system has ECC memory. - Whether it is running under a VMM/Hypervisor. - Kernel version. - Values of the OpenZFS module parameters. - Stack traces which may be logged to ``dmesg``. Does OpenZFS have a Code of Conduct? ------------------------------------ Yes, the OpenZFS community has a code of conduct. See the `Code of Conduct `__ for details. .. _mailing_lists: Mailing Lists ============= +----------------------+----------------------+----------------------+ |              | Description | List Archive | |             List     | | | |                      | | | +======================+======================+======================+ | `zfs-announce\ | A low-traffic list | `archive | | @list.zfsonlinux.\ | for announcements | `__ | | ups/zfs-announce>`__ | | | +----------------------+----------------------+----------------------+ | `zfs-discuss\ | A user discussion | `archive | | @list.zfsonlinux\ | list for issues | `__ | | oups/zfs-discuss>`__ | usability | | +----------------------+----------------------+----------------------+ | `zfs-\ | A development list | `archive | | devel@list.zfsonlin\ | for developers to | `__ | | groups/zfs-devel>`__ | | | +----------------------+----------------------+----------------------+ | `devel\ | A | `archive `__ | | iki/Mailing_list>`__ | developers to review | | | | ZFS code and | | | | architecture changes | | | | from all platforms | | +----------------------+----------------------+----------------------+ Signing Keys ============ All tagged ZFS on Linux `releases `__ are signed by one of the official maintainers. These signatures are automatically verified by GitHub and can be checked locally by downloading the maintainer's public key. Maintainers ----------- | **Maintainer (inactive):** `Ned Bass `__ | **Download:** `pgp.mit.edu `__ | **Key ID:** C77B9667 | **Fingerprint:** 29D5 610E AE29 41E3 55A2 FE8A B974 67AA C77B 9667 | | **Maintainer:** `Tony Hutter `__ | **Download:** `pgp.mit.edu `__ | **Key ID:** D4598027 | **Fingerprint:** 4F3B A9AB 6D1F 8D68 3DC2 DFB5 6AD8 60EE D459 8027 | | **Maintainer:** `Brian Behlendorf `__ | **Download:** `pgp.mit.edu `__ | **Key ID:** C6AF658B | **Fingerprint:** C33D F142 657E D1F7 C328 A296 0AB9 E991 C6AF 658B Checking the Signature of a Git Tag ----------------------------------- First import the public key listed above in to your key ring. :: $ gpg --keyserver pgp.mit.edu --recv C6AF658B gpg: requesting key C6AF658B from hkp server pgp.mit.edu gpg: key C6AF658B: "Brian Behlendorf " not changed gpg: Total number processed: 1 gpg: unchanged: 1 After the public key is imported the signature of a git tag can be verified as shown. :: $ git tag --verify zfs-0.6.5 object 7a27ad00ae142b38d4aef8cc0af7a72b4c0e44fe type commit tag zfs-0.6.5 tagger Brian Behlendorf 1441996302 -0700 ZFS Version 0.6.5 gpg: Signature made Fri 11 Sep 2015 11:31:42 AM PDT using DSA key ID C6AF658B gpg: Good signature from "Brian Behlendorf " gpg: aka "Brian Behlendorf (LLNL) " Developer Resources =================== .. toctree:: :maxdepth: 2 :caption: Contents: :glob: Custom Packages Building ZFS OpenZFS Documentation Git and GitHub for beginners Custom Packages =============== The following instructions assume you are building from an official `release tarball `__ (version 0.8.0 or newer) or directly from the `git repository `__. Most users should not need to do this and should preferentially use the distribution packages. As a general rule the distribution packages will be more tightly integrated, widely tested, and better supported. However, if your distribution of choice doesn't provide packages, or you're a developer and want to roll your own, here's how to do it. The first thing to be aware of is that the build system is capable of generating several different types of packages. Which type of package you choose depends on what's supported on your platform and exactly what your needs are. - **DKMS** packages contain only the source code and scripts for rebuilding the kernel modules. When the DKMS package is installed kernel modules will be built for all available kernels. Additionally, when the kernel is upgraded new kernel modules will be automatically built for that kernel. This is particularly convenient for desktop systems which receive frequent kernel updates. The downside is that because the DKMS packages build the kernel modules from source a full development environment is required which may not be appropriate for large deployments. - **kmods** packages are binary kernel modules which are compiled against a specific version of the kernel. This means that if you update the kernel you must compile and install a new kmod package. If you don't frequently update your kernel, or if you're managing a large number of systems, then kmod packages are a good choice. - **kABI-tracking kmod** Packages are similar to standard binary kmods and may be used with Enterprise Linux distributions like Red Hat and CentOS. These distributions provide a stable kABI (Kernel Application Binary Interface) which allows the same binary modules to be used with new versions of the distribution provided kernel. Note that these packges do not quite follow Redhat's rules, so there is a slight possibility that a new kernel could fail with one of these packages. The probability is low, but we recommend that for production servers, you disable automatic kernel updates when using one of these builds. By default the build system will generate user packages and both DKMS and kmod style kernel packages if possible. The user packages can be used with either set of kernel packages and do not need to be rebuilt when the kernel is updated. You can also streamline the build process by building only the DKMS or kmod packages as shown below. Be aware that when building directly from a git repository you must first run the *autogen.sh* script to create the *configure* script. This will require installing the GNU autotools packages for your distribution. To perform any of the builds, you must install all the necessary development tools and headers for your distribution. It is important to note that if the development kernel headers for the currently running kernel aren't installed, the modules won't compile properly. - `Red Hat, CentOS and Fedora <#red-hat-centos-and-fedora>`__ - `Debian and Ubuntu <#debian-and-ubuntu>`__ RHEL, CentOS and Fedora ----------------------- Make sure that the required packages are installed to build the latest ZFS 2.1 release: - **RHEL/CentOS 7**: .. code:: sh sudo yum install epel-release gcc make autoconf automake libtool rpm-build libtirpc-devel libblkid-devel libuuid-devel libudev-devel openssl-devel zlib-devel libaio-devel libattr-devel elfutils-libelf-devel kernel-devel-$(uname -r) python python2-devel python-setuptools python-cffi libffi-devel ncompress sudo yum install --enablerepo=epel dkms python-packaging **NOTE:** RHEL/CentOS 7 is end of life. Use dnf instead of yum for install instructions below. - **RHEL/CentOS 8**: .. code:: sh sudo dnf install --skip-broken epel-release gcc make autoconf automake libtool rpm-build kernel-rpm-macros libtirpc-devel libblkid-devel libuuid-devel libudev-devel openssl-devel zlib-devel libaio-devel libattr-devel elfutils-libelf-devel kernel-devel-$(uname -r) kernel-abi-stablelists-$(uname -r | sed 's/\.[^.]\+$//') python3 python3-devel python3-setuptools python3-cffi libffi-devel ncompress sudo dnf install --skip-broken --enablerepo=epel --enablerepo=powertools python3-packaging dkms - **RHEL/CentOS 9**: .. code:: sh sudo dnf config-manager --set-enabled crb sudo dnf install --skip-broken epel-release gcc make autoconf automake libtool rpm-build kernel-rpm-macros libtirpc-devel libblkid-devel libuuid-devel libudev-devel openssl-devel zlib-devel libaio-devel libattr-devel elfutils-libelf-devel kernel-devel-$(uname -r) kernel-abi-stablelists-$(uname -r | sed 's/\.[^.]\+$//') python3 python3-devel python3-setuptools python3-cffi libffi-devel sudo dnf install --skip-broken --enablerepo=epel python3-packaging dkms - **Fedora 41**: .. code:: sh sudo dnf install gcc make autoconf automake libtool rpm-build kernel-rpm-macros libtirpc-devel libblkid-devel libuuid-devel systemd-devel openssl-devel zlib-ng-compat-devel libaio-devel libattr-devel libffi-devel libunwind-devel kernel-devel-$(uname -r) python3 python3-devel openssl ncompress sudo dnf install python3-packaging dkms `Get the source code <#get-the-source-code>`__. DKMS ~~~~ Building rpm-based DKMS and user packages can be done as follows: .. code:: sh $ cd zfs $ ./configure $ make -j1 rpm-utils rpm-dkms $ sudo dnf install *.$(uname -m).rpm *.noarch.rpm kmod ~~~~ The key thing to know when building a kmod package is that a specific Linux kernel must be specified. At configure time the build system will make an educated guess as to which kernel you want to build against. However, if configure is unable to locate your kernel development headers, or you want to build against a different kernel, you must specify the exact path with the ``--with-linux`` and ``--with-linux-obj`` options. .. code:: sh $ cd zfs $ ./configure $ make -j1 rpm-utils rpm-kmod $ sudo dnf install *.$(uname -m).rpm *.noarch.rpm **NOTE:** Fedora 41 Workstation includes the rpm package zfs-fuse which will prevent the installation of your own packages. Remove that single package before dnf install: .. code:: sh $ sudo rpm -e --nodeps zfs-fuse kABI-tracking kmod ~~~~~~~~~~~~~~~~~~ The process for building kABI-tracking kmods is almost identical to for building normal kmods. However, it will only produce binaries which can be used by multiple kernels if the distribution supports a stable kABI. In order to request kABI-tracking package the ``--with-spec=redhat`` option must be passed to configure. Be aware that these packages do not completely follow Redhat's rules, so there is a slight chance that they will not work with a new kernel. We recommend disabling automatic kernel updates when using these builds on a production server. **NOTE:** This type of package is not available for Fedora. .. code:: sh $ cd zfs $ ./configure --with-spec=redhat $ make -j1 rpm-utils rpm-kmod $ sudo dnf install *.$(uname -m).rpm *.noarch.rpm Fedora 41 secure boot with kmod ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The zfs kernel module will fail to load on modern computers that use UEFI and secure boot: .. code:: $ sudo modprobe zfs modprobe: ERROR: could not insert 'zfs': Key was rejected by service Either disable secure boot or create a custom machine owner key (MOK) **once** and manually sign your current and future modules using that key: .. code:: sh $ sudo mkdir /etc/pki/mok $ cd /etc/pki/mok $ sudo openssl req -new -x509 -newkey rsa:2048 -keyout LOCALMOK.priv -outform DER -out LOCALMOK.der -nodes -days 36500 -subj "/CN=LOCALMOK/" $ sudo mokutil --import LOCALMOK.der Mokutil asks for a password that you have to create and remember, then reboot your machine and UEFI will ask to import your key: .. code:: Select "Enroll MOK", "Continue", "Yes", enter mokutil's password, "Reboot" This MOK can then be used to manually sign your zfs kernel modules: .. code:: $ rpm -ql kmod-zfs-$(uname -r) | grep .ko /lib/modules/6.11.8-300.fc41.x86_64/extra/zfs/spl.ko /lib/modules/6.11.8-300.fc41.x86_64/extra/zfs/zfs.ko .. code:: sh $ sudo /usr/src/kernels/$(uname -r)/scripts/sign-file sha256 /etc/pki/mok/LOCALMOK.priv /etc/pki/mok/LOCALMOK.der /lib/modules/$(uname -r)/extra/zfs/spl.ko $ sudo /usr/src/kernels/$(uname -r)/scripts/sign-file sha256 /etc/pki/mok/LOCALMOK.priv /etc/pki/mok/LOCALMOK.der /lib/modules/$(uname -r)/extra/zfs/zfs.ko Load the module and verify it is active: .. code:: $ sudo modprobe zfs $ lsmod | grep zfs zfs 6930432 0 spl 155648 1 zfs Debian and Ubuntu ----------------- Make sure that the required packages are installed: .. code:: sh sudo apt install alien autoconf automake build-essential debhelper-compat dh-dkms dh-python dkms fakeroot gawk libaio-dev libattr1-dev libblkid-dev libcurl4-openssl-dev libelf-dev libffi-dev libpam0g-dev libssl-dev libtirpc-dev libtool libudev-dev linux-headers-generic po-debconf python3 python3-all-dev python3-cffi python3-dev python3-packaging python3-setuptools python3-sphinx uuid-dev zlib1g-dev `Get the source code <#get-the-source-code>`__. .. _kmod-1: kmod ~~~~ The key thing to know when building a kmod package is that a specific Linux kernel must be specified. At configure time the build system will make an educated guess as to which kernel you want to build against. However, if configure is unable to locate your kernel development headers, or you want to build against a different kernel, you must specify the exact path with the ``--with-linux`` and ``--with-linux-obj`` options. To build RPM converted Debian packages: .. code:: sh $ cd zfs $ ./configure --enable-systemd $ make -j1 deb-utils deb-kmod $ sudo apt-get install --fix-missing ./*.deb Starting from openzfs-2.2 release, native Debian packages can be built as follows: .. code:: sh $ cd zfs $ ./configure $ make native-deb-utils native-deb-kmod $ rm ../openzfs-zfs-dkms_*.deb $ rm ../openzfs-zfs-dracut_*.deb # deb-based systems usually use initramfs $ sudo apt-get install --fix-missing ../*.deb Native Debian packages build with pre-configured paths for Debian and Ubuntu. It's best not to override the paths during configure. ``KVERS``, ``KSRC`` and ``KOBJ`` environment variables can be exported to specify the kernel installed in non-default location. .. _dkms-1: DKMS ~~~~ Building RPM converted deb-based DKMS and user packages can be done as follows: .. code:: sh $ cd zfs $ ./configure --enable-systemd $ make -j1 deb-utils deb-dkms $ sudo apt-get install --fix-missing ./*.deb Starting from openzfs-2.2 release, native deb-based DKMS and user packages can be built as follows: .. code:: sh $ sudo apt-get install dh-dkms $ cd zfs $ ./configure $ make native-deb-utils $ rm ../openzfs-zfs-dracut_*.deb # deb-based systems usually use initramfs $ sudo apt-get install --fix-missing ../*.deb Get the Source Code ------------------- Released Tarball ~~~~~~~~~~~~~~~~ The released tarball contains the latest fully tested and released version of ZFS. This is the preferred source code location for use in production systems. If you want to use the official released tarballs, then use the following commands to fetch and prepare the source. .. code:: sh $ wget https://github.com/openzfs/zfs/releases/download/zfs-x.y.z/zfs-x.y.z.tar.gz $ tar -xzf zfs-x.y.z.tar.gz Git Master Branch ~~~~~~~~~~~~~~~~~ The Git *master* branch contains the latest version of the software, and will probably contain fixes that, for some reason, weren't included in the released tarball. This is the preferred source code location for developers who intend to modify ZFS. If you would like to use the git version, you can clone it from Github and prepare the source like this. .. code:: sh $ git clone https://github.com/openzfs/zfs.git $ cd zfs $ ./autogen.sh Once the source has been prepared you'll need to decide what kind of packages you're building and jump the to appropriate section above. Note that not all package types are supported for all platforms. Building ZFS ============ GitHub Repositories ~~~~~~~~~~~~~~~~~~~ The official source for OpenZFS is maintained at GitHub by the `openzfs `__ organization. The primary git repository for the project is the `zfs `__ repository. There are two main components in this repository: - **ZFS**: The ZFS repository contains a copy of the upstream OpenZFS code which has been adapted and extended for Linux and FreeBSD. The vast majority of the core OpenZFS code is self-contained and can be used without modification. - **SPL**: The SPL is a thin shim layer which is responsible for implementing the fundamental interfaces required by OpenZFS. It's this layer which allows OpenZFS to be used across multiple platforms. SPL used to be maintained in a separate repository, but was merged into the `zfs `__ repository in the ``0.8`` major release. Installing Dependencies ~~~~~~~~~~~~~~~~~~~~~~~ The first thing you'll need to do is prepare your environment by installing a full development tool chain. In addition, development headers for both the kernel and the following packages must be available. It is important to note that if the development kernel headers for the currently running kernel aren't installed, the modules won't compile properly. The following dependencies should be installed to build the latest ZFS 2.4 release. - **RHEL/CentOS Stream/AlmaLinux 8**: .. code:: sh sudo dnf install epel-release gcc make autoconf automake libtool rpm-build libtirpc-devel libblkid-devel libuuid-devel libudev-devel openssl-devel zlib-devel libaio-devel libattr-devel elfutils-libelf-devel kernel-devel-$(uname -r) python3 python3-devel python3-setuptools python3-cffi libffi-devel git ncompress libcurl-devel sudo dnf install --enablerepo=epel --enablerepo=powertools python3-packaging dkms - **RHEL/CentOS Stream/AlmaLinux 9, 10, Fedora**: .. code:: sh sudo dnf install epel-release gcc make autoconf automake libtool rpm-build libtirpc-devel libblkid-devel libuuid-devel libudev-devel openssl-devel zlib-devel libaio-devel libattr-devel elfutils-libelf-devel kernel-devel-$(uname -r) python3 python3-devel python3-setuptools python3-cffi libffi-devel git ncompress libcurl-devel sudo dnf install --enablerepo=epel --enablerepo=crb python3-packaging dkms - **Debian, Ubuntu**: .. code:: sh sudo apt install alien autoconf automake build-essential debhelper-compat dh-autoreconf dh-dkms dh-python dkms fakeroot gawk git libaio-dev libattr1-dev libblkid-dev libcurl4-openssl-dev libelf-dev libffi-dev libpam0g-dev libssl-dev libtirpc-dev libtool libudev-dev linux-headers-generic parallel po-debconf python3 python3-all-dev python3-cffi python3-dev python3-packaging python3-setuptools python3-sphinx uuid-dev zlib1g-dev - **FreeBSD**: .. code:: sh pkg install autoconf automake autotools gettext git gmake python devel/py-sysctl sudo Build Options ~~~~~~~~~~~~~ There are two options for building OpenZFS; the correct one largely depends on your requirements. - **Packages**: Often it can be useful to build custom packages from git which can be installed on a system. This is the best way to perform integration testing with systemd, dracut, and udev. The downside to using packages it is greatly increases the time required to build, install, and test a change. - **In-tree**: Development can be done entirely in the SPL/ZFS source tree. This speeds up development by allowing developers to rapidly iterate on a patch. When working in-tree developers can leverage incremental builds, load/unload kernel modules, execute utilities, and verify all their changes with the ZFS Test Suite. The remainder of this page focuses on the **in-tree** option which is the recommended method of development for the majority of changes. See the :doc:`custom packages <./Custom Packages>` page for additional information on building custom packages. Developing In-Tree ~~~~~~~~~~~~~~~~~~ Clone from GitHub ^^^^^^^^^^^^^^^^^ Start by cloning the ZFS repository from GitHub. The repository has a **master** branch for development and a series of **\*-release** branches for tagged releases. After checking out the repository your clone will default to the master branch. Tagged releases may be built by checking out zfs-x.y.z tags with matching version numbers or matching release branches. :: git clone https://github.com/openzfs/zfs Configure and Build ^^^^^^^^^^^^^^^^^^^ For developers working on a change always create a new topic branch based off of master. This will make it easy to open a pull request with your change later. The master branch is kept stable with extensive `regression testing `__ of every pull request before and after it's merged. Every effort is made to catch defects as early as possible and to keep them out of the tree. Developers should be comfortable frequently rebasing their work against the latest master branch. In this example we'll use the master branch and walk through a stock **in-tree** build. Start by checking out the desired branch then build the ZFS and SPL source in the traditional autotools fashion. :: cd ./zfs git checkout master sh autogen.sh ./configure make -s -j$(nproc) | **tip:** ``--with-linux=PATH`` and ``--with-linux-obj=PATH`` can be passed to configure to specify a kernel installed in a non-default location. | **tip:** ``--enable-debug`` can be passed to configure to enable all ASSERTs and additional correctness tests. **Optional** Build packages :: make rpm #Builds RPM packages for CentOS/Fedora make deb #Builds RPM converted DEB packages for Debian/Ubuntu make native-deb #Builds native DEB packages for Debian/Ubuntu | **tip:** Native Debian packages build with pre-configured paths for Debian and Ubuntu. It's best not to override the paths during configure. | **tip:** For native Debian packages, ``KVERS``, ``KSRC`` and ``KOBJ`` environment variables can be exported to specify the kernel installed in non-default location. .. note:: Support for native Debian packaging will be available starting from openzfs-2.2 release. Install ^^^^^^^ You can run ``zfs-tests.sh`` without installing ZFS, see below. If you have reason to install ZFS after building it, pay attention to how your distribution handles kernel modules. On Ubuntu, for example, the modules from this repository install in the ``extra`` kernel module path, which is not in the standard ``depmod`` search path. Therefore, for the duration of your testing, edit ``/etc/depmod.d/ubuntu.conf`` and add ``extra`` to the beginning of the search path. You may then install using ``sudo make install; sudo ldconfig; sudo depmod``. You'd uninstall with ``sudo make uninstall; sudo ldconfig; sudo depmod``. You can install just the kernel modules with ``sudo make -C modules/ install``. .. _running-zloopsh-and-zfs-testssh: Running zloop.sh and zfs-tests.sh ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ If you wish to run the ZFS Test Suite (ZTS), then ``ksh`` and a few additional utilities must be installed. - **RHEL/CentOS Stream/AlmaLinux 8:** .. code:: sh sudo dnf install ksh bc bzip2 fio acl sysstat mdadm lsscsi parted attr nfs-utils samba rng-tools pax perf sudo dnf install --enablerepo=epel dbench - **RHEL/CentOS Stream/AlmaLinux 9, 10, Fedora:** .. code:: sh sudo dnf install ksh bc bzip2 fio acl sysstat mdadm lsscsi parted attr nfs-utils samba rng-tools pax perf sudo dnf install --enablerepo=epel dbench - **Debian:** .. code:: sh sudo apt install ksh bc bzip2 fio acl sysstat mdadm lsscsi parted attr dbench nfs-kernel-server samba rng-tools pax linux-perf selinux-utils quota - **Ubuntu:** .. code:: sh sudo apt install ksh bc bzip2 fio acl sysstat mdadm lsscsi parted attr dbench nfs-kernel-server samba rng-tools pax linux-tools-common selinux-utils quota - **FreeBSD**: .. code:: sh pkg install base64 bash checkbashisms fio hs-ShellCheck ksh93 pamtester devel/py-flake8 sudo There are a few helper scripts provided in the top-level scripts directory designed to aid developers working with in-tree builds. - **zfs-helpers.sh:** Certain functionality (i.e. /dev/zvol/) depends on the ZFS provided udev helper scripts being installed on the system. This script can be used to create symlinks on the system from the installation location to the in-tree helper. These links must be in place to successfully run the ZFS Test Suite. The **-i** and **-r** options can be used to install and remove the symlinks. :: sudo ./scripts/zfs-helpers.sh -i - **zfs.sh:** The freshly built kernel modules can be loaded using ``zfs.sh``. This script can later be used to unload the kernel modules with the **-u** option. :: sudo ./scripts/zfs.sh - **zloop.sh:** A wrapper to run ztest repeatedly with randomized arguments. The ztest command is a user space stress test designed to detect correctness issues by concurrently running a random set of test cases. If a crash is encountered, the ztest logs, any associated vdev files, and core file (if one exists) are collected and moved to the output directory for analysis. :: sudo ./scripts/zloop.sh - **zfs-tests.sh:** A wrapper which can be used to launch the ZFS Test Suite. Three loopback devices are created on top of sparse files located in ``/var/tmp/`` and used for the regression test. Detailed directions for the ZFS Test Suite can be found in the `README `__ located in the top-level tests directory. :: ./scripts/zfs-tests.sh -vx **tip:** The **delegate** tests will be skipped unless group read permission is set on the zfs directory and its parents. Git and GitHub for beginners (ZoL edition) ========================================== This is a very basic rundown of how to use Git and GitHub to make changes. Recommended reading: `ZFS on Linux CONTRIBUTING.md `__ First time setup ---------------- If you've never used Git before, you'll need a little setup to start things off. :: git config --global user.name "My Name" git config --global user.email myemail@noreply.non Cloning the initial repository ------------------------------ The easiest way to get started is to click the fork icon at the top of the main repository page. From there you need to download a copy of the forked repository to your computer: :: git clone https://github.com//zfs.git This sets the "origin" repository to your fork. This will come in handy when creating pull requests. To make pulling from the "upstream" repository as changes are made, it is very useful to establish the upstream repository as another remote (man git-remote): :: cd zfs git remote add upstream https://github.com/openzfs/zfs.git Preparing and making changes ---------------------------- In order to make changes it is recommended to make a branch, this lets you work on several unrelated changes at once. It is also not recommended to make changes to the master branch unless you own the repository. :: git checkout -b my-new-branch From here you can make your changes and move on to the next step. Recommended reading: `C Style and Coding Standards for SunOS `__, :doc:`OpenZFS Developer Resources `, `OpenZFS Developer Resources `__ Testing your patches before pushing ----------------------------------- Before committing and pushing, you may want to test your patches. There are several tests you can run against your branch such as style checking, and functional tests. All pull requests go through these tests before being pushed to the main repository, however testing locally takes the load off the build/test servers. This step is optional but highly recommended, however the test suite should be run on a virtual machine or a host that currently does not use ZFS. You may need to install ``shellcheck`` and ``flake8`` to run the ``checkstyle`` correctly. :: sh autogen.sh ./configure make checkstyle Recommended reading: :doc:`Building ZFS `, `ZFS Test Suite README `__ Committing your changes to be pushed ------------------------------------ When you are done making changes to your branch there are a few more steps before you can make a pull request. :: git commit --all --signoff This command opens an editor and adds all unstaged files from your branch. Here you need to describe your change and add a few things: :: # Please enter the commit message for your changes. Lines starting # with '#' will be ignored, and an empty message aborts the commit. # On branch my-new-branch # Changes to be committed: # (use "git reset HEAD ..." to unstage) # # modified: hello.c # The first thing we need to add is the commit message. This is what is displayed on the git log, and should be a short description of the change. By style guidelines, this has to be less than 72 characters in length. Underneath the commit message you can add a more descriptive text to your commit. The lines in this section have to be less than 72 characters. When you are done, the commit should look like this: :: Add hello command This is a test commit with a descriptive commit message. This message can be more than one line as shown here. Signed-off-by: My Name Closes #9998 Issue #9999 # Please enter the commit message for your changes. Lines starting # with '#' will be ignored, and an empty message aborts the commit. # On branch my-new-branch # Changes to be committed: # (use "git reset HEAD ..." to unstage) # # modified: hello.c # You can also reference issues and pull requests if you are filing a pull request for an existing issue as shown above. Save and exit the editor when you are done. Pushing and creating the pull request ------------------------------------- Home stretch. You've made your change and made the commit. Now it's time to push it. :: git push --set-upstream origin my-new-branch This should ask you for your github credentials and upload your changes to your repository. The last step is to either go to your repository or the upstream repository on GitHub and you should see a button for making a new pull request for your recently committed branch. Correcting issues with your pull request ---------------------------------------- Sometimes things don't always go as planned and you may need to update your pull request with a correction to either your commit message, or your changes. This can be accomplished by re-pushing your branch. If you need to make code changes or ``git add`` a file, you can do those now, along with the following: :: git commit --amend git push --force This will return you to the commit editor screen, and push your changes over top of the old ones. Do note that this will restart the process of any build/test servers currently running and excessively pushing can cause delays in processing of all pull requests. Maintaining your repository --------------------------- When you wish to make changes in the future you will want to have an up-to-date copy of the upstream repository to make your changes on. Here is how you keep updated: :: git checkout master git pull upstream master git push origin master This will make sure you are on the master branch of the repository, grab the changes from upstream, then push them back to your repository. Backporting a commit to a release branch ---------------------------------------- Users may want to backport commits from the ``master`` branch to one of the release branches. To do that, first checkout the "staging" branch for the release version you want to pull the commit into. For example, if you want to backport commit XYZ from ``master`` into the future ``zfs-2.3.6`` release, first checkout the ``zfs-2.3.6-staging`` branch, and then pull the commit in with ``git cherry-pick XYZ`` (and resolve any merge conflicts). You can then open a PR with your backport branch against the ``zfs-2.3.6-staging`` branch (make sure to select this branch in the PR target dropdown box). Please keep the commit message the same when you backport. This means keeping the author the same, and all of the attestation lines the same (Signed-off-by, Reviewed-by, etc). It is assumed those attestation lines refer to the original commit from ``master``, and not necessarily the backport commit (even though it may have needed changes to backport). You may optionally add a ``Backported-by:`` line with your name if desired. Sometimes you may need to add a small, version-specific commit that only goes in the release branch. For those cases, add a tag to the commit line with the release version, in the format of "[x.y.z-only]". For example: :: [2.3.6-only] Disable feature X by default Unlike the master branch, turn feature X off by default for the zfs-2.3.6 release. Signed-off-by: Tony Hutter Final words ----------- This is a very basic introduction to Git and GitHub, but should get you on your way to contributing to many open source projects. Not all projects have style requirements and some may have different processes to getting changes committed so please refer to their documentation to see if you need to do anything different. One topic we have not touched on is the ``git rebase`` command which is a little more advanced for this wiki article. Additional resources: `Github Help `__, `Atlassian Git Tutorials `__ Performance and Tuning ====================== .. toctree:: :maxdepth: 2 :caption: Contents: :glob: * Hardware ******** .. contents:: Table of Contents :local: Introduction ============ Storage before ZFS involved rather expensive hardware that was unable to protect against silent corruption and did not scale very well. The introduction of ZFS has enabled people to use far less expensive hardware than previously used in the industry with superior scaling. This page attempts to provide some basic guidance to people buying hardware for use in ZFS-based servers and workstations. Hardware that adheres to this guidance will enable ZFS to reach its full potential for performance and reliability. Hardware that does not adhere to it will serve as a handicap. Unless otherwise stated, such handicaps apply to all storage stacks and are by no means specific to ZFS. Systems built using competing storage stacks will also benefit from these suggestions. .. _bios_cpu_microcode_updates: BIOS / CPU microcode updates ============================ Running the latest BIOS and CPU microcode is highly recommended. Background ---------- Computer microprocessors are very complex designs that often have bugs, which are called errata. Modern microprocessors are designed to utilize microcode. This puts part of the hardware design into quasi-software that can be patched without replacing the entire chip. Errata are often resolved through CPU microcode updates. These are often bundled in BIOS updates. In some cases, the BIOS interactions with the CPU through machine registers can be modified to fix things with the same microcode. If a newer microcode is not bundled as part of a BIOS update, it can often be loaded by the operating system bootloader or the operating system itself. .. _ecc_memory: ECC Memory ========== Bit flips can have fairly dramatic consequences for all computer filesystems and ZFS is no exception. No technique used in ZFS (or any other filesystem) is capable of protecting against bit flips. Consequently, ECC Memory is highly recommended. .. _background_1: Background ---------- Ordinary background radiation will randomly flip bits in computer memory, which causes undefined behavior. These are known as "bit flips". Each bit flip can have any of four possible consequences depending on which bit is flipped: - Bit flips can have no effect. - Bit flips that have no effect occur in unused memory. - Bit flips can cause runtime failures. - This is the case when a bit flip occurs in something read from disk. - Failures are typically observed when program code is altered. - If the bit flip is in a routine within the system's kernel or /sbin/init, the system will likely crash. Otherwise, reloading the affected data can clear it. This is typically achieved by a reboot. - It can cause data corruption. - This is the case when the bit is in use by data being written to disk. - If the bit flip occurs before ZFS' checksum calculation, ZFS will not realize that the data is corrupt. - If the bit flip occurs after ZFS' checksum calculation, but before write-out, ZFS will detect it, but it might not be able to correct it. - It can cause metadata corruption. - This is the case when a bit flips in an on-disk structure being written to disk. - If the bit flip occurs before ZFS' checksum calculation, ZFS will not realize that the metadata is corrupt. - If the bit flip occurs after ZFS' checksum calculation, but before write-out, ZFS will detect it, but it might not be able to correct it. - Recovery from such an event will depend on what was corrupted. In the worst case, a pool could be rendered unimportable. - All filesystems have poor reliability in their absolute worst case bit-flip failure scenarios. Such scenarios should be considered extraordinarily rare. .. _drive_interfaces: Drive Interfaces ================ .. _sas_versus_sata: SAS versus SATA --------------- ZFS depends on the block device layer for storage. Consequently, ZFS is affected by the same things that affect other filesystems, such as driver support and non-working hardware. Consequently, there are a few things to note: - Never place SATA disks into a SAS expander without a SAS interposer. - If you do this and it does work, it is the exception, rather than the rule. - Do not expect SAS controllers to be compatible with SATA port multipliers. - This configuration is typically not tested. - The disks could be unrecognized. - Support for SATA port multipliers is inconsistent across OpenZFS platforms - Linux drivers generally support them. - Illumos drivers generally do not support them. - FreeBSD drivers are somewhere between Linux and Illumos in terms of support. .. _usb_hard_drives_andor_adapters: USB Hard Drives and/or Adapters ------------------------------- These have problems involving sector size reporting, SMART passthrough, the ability to set ERC, and other areas. ZFS will perform as well on such devices as they are capable of allowing, but try to avoid them. They should not be expected to have the same up-time as SAS and SATA drives and should be considered unreliable. Controllers =========== The ideal storage controller for ZFS has the following attributes: - Driver support on major OpenZFS platforms - Stability is important. - High per-port bandwidth - PCI Express interface bandwidth divided by the number of ports - Low cost - Support for RAID, Battery Backup Units and hardware write caches is unnecessary. Marc Bevand's blog post `From 32 to 2 ports: Ideal SATA/SAS Controllers for ZFS & Linux MD RAID `__ contains an excellent list of storage controllers that meet these criteria. He regularly updates it as newer controllers become available. .. _hardware_raid_controllers: Hardware RAID controllers ------------------------- Hardware RAID controllers should not be used with ZFS. While ZFS will likely be more reliable than other filesystems on Hardware RAID, it will not be as reliable as it would be on its own. - Hardware RAID will limit opportunities for ZFS to perform self healing on checksum failures. When ZFS does RAID-Z or mirroring, a checksum failure on one disk can be corrected by treating the disk containing the sector as bad for the purpose of reconstructing the original information. This cannot be done when a RAID controller handles the redundancy unless a duplicate copy is stored by ZFS. If the copies flag is set or the RAID is part of a mirror/raid-z vdev within ZFS then metadata corruption may be repairable. - Sector size information is not necessarily passed correctly by hardware RAID on RAID 1. Sector size information cannot be passed correctly on RAID 5/6. Hardware RAID 1 is more likely to experience read-modify-write overhead from partial sector writes while Hardware RAID 5/6 will almost certainty suffer from partial stripe writes (i.e. the RAID write hole). ZFS using the disks natively allows it to obtain the sector size information reported by the disks to avoid read-modify-write on sectors, while ZFS avoids partial stripe writes on RAID-Z by design from using copy-on-write. - There can be sector alignment problems on ZFS when a drive misreports its sector size. Such drives are typically NAND-flash based solid state drives and older SATA drives from the advanced format (4K sector size) transition before Windows XP EoL occurred. This can be :ref:`manually corrected ` at vdev creation. - It is possible for the RAID header to cause misalignment of sector writes on RAID 1 by starting the array within a sector on an actual drive, such that manual correction of sector alignment at vdev creation does not solve the problem. - RAID controller failures can require that the controller be replaced with the same model, or in less extreme cases, a model from the same manufacturer. Using ZFS by itself allows any controller to be used. - If a hardware RAID controller's write cache is used, an additional failure point is introduced that can only be partially mitigated by additional complexity from adding flash to save data in power loss events. The data can still be lost if the battery fails when it is required to survive a power loss event or there is no flash and power is not restored in a timely manner. The loss of the data in the write cache can severely damage anything stored on a RAID array when many outstanding writes are cached. In addition, all writes are stored in the cache rather than just synchronous writes that require a write cache, which is inefficient, and the write cache is relatively small. ZFS allows synchronous writes to be written directly to flash, which should provide similar acceleration to hardware RAID and the ability to accelerate many more in-flight operations. - Behavior during RAID reconstruction when silent corruption damages data is undefined. There are reports of RAID 5 and 6 arrays being lost during reconstruction when the controller encounters silent corruption. ZFS' checksums allow it to avoid this situation by determining whether enough information exists to reconstruct data. If not, the file is listed as damaged in zpool status and the system administrator has the opportunity to restore it from a backup. - IO response times will be increased (i.e. reduced performance) whenever the OS blocks on IO operations because the system CPU blocks on a much weaker embedded CPU used in the RAID controller. This lowers IOPS relative to what ZFS could have achieved. - The controller's firmware is an additional layer of complexity that cannot be inspected by arbitrary third parties. The ZFS source code is open source and can be inspected by anyone. - If multiple RAID arrays are formed by the same controller and one fails, the identifiers provided by the arrays exposed to the OS might become inconsistent. Giving the drives directly to the OS allows this to be avoided via naming that maps to a unique port or unique drive identifier. - e.g. If you have arrays A, B, C and D; array B dies, the interaction between the hardware RAID controller and the OS might rename arrays C and D to look like arrays B and C respectively. This can fault pools verbatim imported from the cachefile. - Not all RAID controllers behave this way. This issue has been observed on both Linux and FreeBSD when system administrators used single drive RAID 0 arrays, however. It has also been observed with controllers from different vendors. One might be inclined to try using single-drive RAID 0 arrays to try to use a RAID controller like a HBA, but this is not recommended for many of the reasons listed for other hardware RAID types. It is best to use a HBA instead of a RAID controller, for both performance and reliability. .. _hard_drives: Hard drives =========== .. _sector_size: Sector Size ----------- Historically, all hard drives had 512-byte sectors, with the exception of some SCSI drives that could be modified to support slightly larger sectors. In 2009, the industry migrated from 512-byte sectors to 4096-byte "Advanced Format" sectors. Since Windows XP is not compatible with 4096-byte sectors or drives larger than 2TB, some of the first advanced format drives implemented hacks to maintain Windows XP compatibility. - The first advanced format drives on the market misreported their sector size as 512-bytes for Windows XP compatibility. As of 2013, it is believed that such hard drives are no longer in production. Advanced format hard drives made during or after this time should report their true physical sector size. - Drives storing 2TB and smaller might have a jumper that can be set to map all sectors off by 1. This to provide proper alignment for Windows XP, which started its first partition at sector 63. This jumper setting should be off when using such drives with ZFS. As of 2014, there are still 512-byte and 4096-byte drives on the market, but they are known to properly identify themselves unless behind a USB to SATA controller. Replacing a 512-byte sector drive with a 4096-byte sector drive in a vdev created with 512-byte sector drives will adversely affect performance. Replacing a 4096-byte sector drive with a 512-byte sector drive will have no negative effect on performance. .. _error_recovery_control: Error recovery control ---------------------- ZFS is said to be able to use cheap drives. This was true when it was introduced and hard drives supported error recovery control. Since ZFS' introduction, error recovery control has been removed from low-end drives from certain manufacturers, most notably Western Digital. Consistent performance requires hard drives that support error recovery control. .. _background_2: Background ~~~~~~~~~~ Hard drives store data using small polarized regions on a magnetic surface. Reading from and/or writing to this surface poses a few reliability problems. One is that imperfections in the surface can corrupt bits. Another is that vibrations can cause drive heads to miss their targets. Consequently, hard drive sectors are composed of three regions: - A sector number - The actual data - ECC The sector number and ECC enables hard drives to detect and respond to such events. When either event occurs during a read, hard drives will retry the read many times until they either succeed or conclude that the data cannot be read. The latter case can take a substantial amount of time and consequently, IO to the drive will stall. Enterprise hard drives and some consumer hard drives implement a feature called Time-Limited Error Recovery (TLER) by Western Digital, Error Recovery Control (ERC) by Seagate, and Command Completion Time Limit by Hitachi and Samsung, which permits the time drives are willing to spend on such events to be limited by the system administrator. Drives that lack such functionality can be expected to have arbitrarily high limits. Several minutes is not impossible. Drives with this functionality typically default to 7 seconds. ZFS does not currently adjust this setting on drives. However, it is advisable to write a script to set the error recovery time to a low value, such as 0.1 seconds until ZFS is modified to control it. This must be done on every boot. .. _rpm_speeds: RPM Speeds ---------- High RPM drives have lower seek times, which is historically regarded as being desirable. They increase cost and sacrifice storage density in order to achieve what is typically no more than a factor of 6 improvement over their lower RPM counterparts. To provide some numbers, a 15k RPM drive from a major manufacturer is rated for 3.4 millisecond average read and 3.9 millisecond average write. Presumably, this number assumes that the target sector is at most half the number of drive tracks away from the head and half the disk away. Being even further away is worst-case 2 times slower. Manufacturer numbers for 7200 RPM drives are not available, but they average 13 to 16 milliseconds in empirical measurements. 5400 RPM drives can be expected to be slower. ARC and ZIL are able to mitigate much of the benefit of lower seek times. Far larger increases in IOPS performance can be obtained by adding additional RAM for ARC, L2ARC devices and SLOG devices. Even higher increases in performance can be obtained by replacing hard drives with solid state storage entirely. Such things are typically more cost effective than high RPM drives when considering IOPS. .. _command_queuing: Command Queuing --------------- Drives with command queues are able to reorder IO operations to increase IOPS. This is called Native Command Queuing on SATA and Tagged Command Queuing on PATA/SCSI/SAS. ZFS stores objects in metaslabs and it can use several metaslabs at any given time. Consequently, ZFS is not only designed to take advantage of command queuing, but good ZFS performance requires command queuing. Almost all drives manufactured within the past 10 years can be expected to support command queuing. The exceptions are: - Consumer PATA/IDE drives - First generation SATA drives, which used IDE to SATA translation chips, from 2003 to 2004. - SATA drives operating under IDE emulation that was configured in the system BIOS. Each OpenZFS system has different methods for checking whether command queuing is supported. On Linux, ``hdparm -I /path/to/device \| grep Queue`` is used. On FreeBSD, ``camcontrol identify $DEVICE`` is used. .. _nand_flash_ssds: NAND Flash SSDs =============== As of 2014, Solid state storage is dominated by NAND-flash and most articles on solid state storage focus on it exclusively. As of 2014, the most popular form of flash storage used with ZFS involve drives with SATA interfaces. Enterprise models with SAS interfaces are beginning to become available. As of 2017, Solid state storage using NAND-flash with PCI-E interfaces are widely available on the market. They are predominantly enterprise drives that utilize a NVMe interface that has lower overhead than the ATA used in SATA or SCSI used in SAS. There is also an interface known as M.2 that is primarily used by consumer SSDs, although not necessarily limited to them. It can provide electrical connectivity for multiple buses, such as SATA, PCI-E and USB. M.2 SSDs appear to use either SATA or NVME. .. _nvme_low_level_formatting: NVMe low level formatting ------------------------- Many NVMe SSDs support both 512-byte sectors and 4096-byte sectors. They often ship with 512-byte sectors, which are less performant than 4096-byte sectors. Some also support metadata for T10/DIF CRC to try to improve reliability, although this is unnecessary with ZFS. NVMe drives should be `formatted `__ to use 4096-byte sectors without metadata prior to being given to ZFS for best performance unless they indicate that 512-byte sectors are as performant as 4096-byte sectors, although this is unlikely. Lower numbers in the Rel_Perf of Supported LBA Sizes from ``smartctl -a /dev/$device_namespace`` (for example ``smartctl -a /dev/nvme1n1``) indicate higher performance low level formats, with 0 being the best. The current formatting will be marked by a plus sign under the format Fmt. You may format a drive using ``nvme format /dev/nvme1n1 -l $ID``. The $ID corresponds to the Id field value from the Supported LBA Sizes SMART information. .. _power_failure_protection: Power Failure Protection ------------------------ .. _background_3: Background ~~~~~~~~~~ On-flash data structures are highly complex and traditionally have been highly vulnerable to corruption. In the past, such corruption would result in the loss of \*all\* drive data and an event such as a PSU failure could result in multiple drives simultaneously failing. Since the drive firmware is not available for review, the traditional conclusion was that all drives that lack hardware features to avoid power failure events cannot be trusted, which was found to be the case multiple times in the past [#ssd_analysis]_ [#ssd_analysis2]_ [#ssd_analysis3]_. Discussion of power failures bricking NAND flash SSDs appears to have vanished from literature following the year 2015. SSD manufacturers now claim that firmware power loss protection is robust enough to provide equivalent protection to hardware power loss protection. `Kingston is one example `__. Firmware power loss protection is used to guarantee the protection of flushed data and the drives’ own metadata, which is all that filesystems such as ZFS need. However, those that either need or want strong guarantees that firmware bugs are unlikely to be able to brick drives following power loss events should continue to use drives that provide hardware power loss protection. The basic concept behind how hardware power failure protection works has been `documented by Intel `__ for those who wish to read about the details. As of 2020, use of hardware power loss protection is now a feature solely of enterprise SSDs that attempt to protect unflushed data in addition to drive metadata and flushed data. This additional protection beyond protecting flushed data and the drive metadata provides no additional benefit to ZFS, but it does not hurt it. It should also be noted that drives in data centers and laptops are unlikely to experience power loss events, reducing the usefulness of hardware power loss protection. This is especially the case in datacenters where redundant power, UPS power and the use of IPMI to do forced reboots should prevent most drives from experiencing power loss events. Lists of drives that provide hardware power loss protection are maintained below for those who need/want it. Since ZFS, like other filesystems, only requires power failure protection for flushed data and drive metadata, older drives that only protect these things are included on the lists. .. _nvme_drives_with_power_failure_protection: NVMe drives with power failure protection ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ A non-exhaustive list of NVMe drives with power failure protection is as follows: - Intel 750 - Intel DC P3500/P3600/P3608/P3700 - Kingston DC1000B (M.2 2280 form factor, fits into most laptops) - Micron 7300/7400/7450 PRO/MAX - Samsung PM963 (M.2 form factor) - Samsung PM1725/PM1725a - Samsung XS1715 - Toshiba ZD6300 - Seagate Nytro 5000 M.2 (XP1920LE30002 tested; **read notes below before buying**) - Inexpensive 22110 M.2 enterprise drive using consumer MLC that is optimized for read mostly workloads. It is not a good choice for a SLOG device, which is a write mostly workload. - The `manual `__ for this drive specifies airflow requirements. If the drive does not receive sufficient airflow from case fans, it will overheat at idle. Its thermal throttling will severely degrade performance such that write throughput performance will be limited to 1/10 of the specification and read latencies will reach several hundred milliseconds. Under continuous load, the device will continue to become hotter until it suffers a "degraded reliability" event where all data on at least one NVMe namespace is lost. The NVMe namespace is then unusable until a secure erase is done. Even with sufficient airflow under normal circumstances, data loss is possible under load following the failure of fans in an enterprise environment. Anyone deploying this into production in an enterprise environment should be mindful of this failure mode. - Those who wish to use this drive in a low airflow situation can workaround this failure mode by placing a passive heatsink such as `this `__ on the NAND flash controller. It is the chip under the sticker closest to the capacitors. This was tested by placing the heatsink over the sticker (as removing it was considered undesirable). The heatsink will prevent the drive from overheating to the point of data loss, but it will not fully alleviate the overheating situation under load without active airflow. A scrub will cause it to overheat after a few hundred gigabytes are read. However, the thermal throttling will quickly cool the drive from 76 degrees Celsius to 74 degrees Celsius, restoring performance. - It might be possible to use the heatsink in an enterprise environment to provide protection against data loss following fan failures. However, this was not evaluated. Furthermore, operating temperatures for consumer NAND flash should be at or above 40 degrees Celsius for long term data integrity. Therefore, the use of a heatsink to provide protection against data loss following fan failures in an enterprise environment should be evaluated before deploying drives into production to ensure that the drive is not overcooled. .. _sas_drives_with_power_failure_protection: SAS drives with power failure protection ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ A non-exhaustive list of SAS drives with power failure protection is as follows: - Samsung PM1633/PM1633a - Samsung SM1625 - Samsung PM853T - Toshiba PX05SHB***/PX04SHB***/PX04SHQ**\* - Toshiba PX05SLB***/PX04SLB***/PX04SLQ**\* - Toshiba PX05SMB***/PX04SMB***/PX04SMQ**\* - Toshiba PX05SRB***/PX04SRB***/PX04SRQ**\* - Toshiba PX05SVB***/PX04SVB***/PX04SVQ**\* .. _sata_drives_with_power_failure_protection: SATA drives with power failure protection ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ A non-exhaustive list of SATA drives with power failure protection is as follows: - Crucial MX100/MX200/MX300 - Crucial M500/M550/M600 - Intel 320 - Early reports claimed that the 330 and 335 had power failure protection too, `but they do not `__. - Intel 710 - Intel 730 - Intel DC S3500/S3510/S3610/S3700/S3710 - Kingston DC500R/DC500M - Micron 5210 Ion - First QLC drive on the list. High capacity with a low price per gigabyte. - Samsung PM863/PM863a - Samsung SM843T (do not confuse with SM843) - Samsung SM863/SM863a - Samsung 845DC Evo - Samsung 845DC Pro - `High sustained write IOPS `__ - Toshiba HK4E/HK3E2 - Toshiba HK4R/HK3R2/HK3R .. _criteriaprocess_for_inclusion_into_these_lists: Criteria/process for inclusion into these lists ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ These lists have been compiled on a volunteer basis by OpenZFS contributors (mainly Richard Yao) from trustworthy sources of information. The lists are intended to be vendor neutral and are not intended to benefit any particular manufacturer. Any perceived bias toward any manufacturer is caused by a lack of awareness and a lack of time to research additional options. Confirmation of the presence of adequate power loss protection by a reliable source is the only requirement for inclusion into this list. Adequate power loss protection means that the drive must protect both its own internal metadata and all flushed data. Protection of unflushed data is irrelevant and therefore not a requirement. ZFS only expects storage to protect flushed data. Consequently, solid state drives whose power loss protection only protects flushed data is sufficient for ZFS to ensure that data remains safe. Anyone who believes an unlisted drive to provide adequate power failure protection may contact the :ref:`mailing_lists` with a request for inclusion and substantiation for the claim that power failure protection is provided. Examples of substantiation include pictures of drive internals showing the presence of capacitors, statements by well regarded independent review sites such as Anandtech and manufacturer specification sheets. The latter are accepted on the honor system until a manufacturer is found to misstate reality on the protection of the drives' own internal metadata structures and/or the protection of flushed data. Thus far, all manufacturers have been honest. .. _flash_pages: Flash pages ----------- The smallest unit on a NAND chip that can be written is a flash page. The first NAND-flash SSDs on the market had 4096-byte pages. Further complicating matters is that the the page size has been doubled twice since then. NAND flash SSDs **should** report these pages as being sectors, but so far, all of them incorrectly report 512-byte sectors for Windows XP compatibility. The consequence is that we have a similar situation to what we had with early advanced format hard drives. As of 2014, most NAND-flash SSDs on the market have 8192-byte page sizes. However, models using 128-Gbit NAND from certain manufacturers have a 16384-byte page size. Maximum performance requires that vdevs be created with correct ashift values (13 for 8192-byte and 14 for 16384-byte). However, not all OpenZFS platforms support this. The Linux port supports ashift=13, while others are limited to ashift=12 (4096-byte). As of 2017, NAND-flash SSDs are tuned for 4096-byte IOs. Matching the flash page size is unnecessary and ashift=12 is usually the correct choice. Public documentation on flash page size is also nearly non-existent. .. _ata_trim_scsi_unmap: ATA TRIM / SCSI UNMAP --------------------- It should be noted that this is a separate case from discard on zvols or hole punching on filesystems. Those work regardless of whether ATA TRIM / SCSI UNMAP is sent to the actual block devices. .. _ata_trim_performance_issues: ATA TRIM Performance Issues ~~~~~~~~~~~~~~~~~~~~~~~~~~~ The ATA TRIM command in SATA 3.0 and earlier is a non-queued command. Issuing a TRIM command on a SATA drive conforming to SATA 3.0 or earlier will cause the drive to drain its IO queue and stop servicing requests until it finishes, which hurts performance. SATA 3.1 removed this limitation, but very few SATA drives on the market are conformant to SATA 3.1 and it is difficult to distinguish them from SATA 3.0 drives. At the same time, SCSI UNMAP has no such problems. .. _optane_3d_xpoint_ssds: Optane / 3D XPoint SSDs ======================= These are SSDs with far better latencies and write endurance than NAND flash SSDs. They are byte addressable, such that ashift=9 is fine for use on them. Unlike NAND flash SSDs, they do not require any special power failure protection circuitry for reliability. There is also no need to run TRIM on them. However, they cost more per GB than NAND flash (as of 2020). The enterprise models make excellent SLOG devices. Note that Intel wound down the Optane business in 2022, so these are no longer manufactured. They remain excellent if you already own them or can source them, but for new builds the practical alternative is an enterprise NAND flash SSD with power failure protection — see :ref:`nand_flash_ssds`. Here is a list of models that are known to perform well: - `Intel DC P4800X `__ - `Intel DC P4801X `__ - `Intel DC P1600X `__ Note that SLOG devices rarely have more than 4GB in use at any given time, so the smaller sized devices are generally the best choice in terms of cost, with larger sizes giving no benefit. Larger sizes could be a good choice for other vdev types, depending on performance needs and cost considerations. Power ===== Ensuring that computers are properly grounded is highly recommended. There have been cases in user homes where machines experienced random failures when plugged into power receptacles that had open grounds (i.e. no ground wire at all). This can cause random failures on any computer system, whether it uses ZFS or not. Power should also be relatively stable. Large dips in voltages from brownouts are preferably avoided through the use of UPS units or line conditioners. Systems subject to unstable power that do not outright shutdown can exhibit undefined behavior. PSUs with longer hold-up times should be able to provide partial protection against this, but hold up times are often undocumented and are not a substitute for a UPS or line conditioner. .. _pwr_ok_signal: PWR_OK signal ------------- PSUs are supposed to deassert a PWR_OK signal to indicate that provided voltages are no longer within the rated specification. This should force an immediate shutdown. However, the system clock of a developer workstation was observed to significantly deviate from the expected value during a series of ~1 second brown outs. This machine did not use a UPS at the time. However, the PWR_OK mechanism should have protected against this. The observation of the PWR_OK signal failing to force a shutdown with adverse consequences (to the system clock in this case) suggests that the PWR_OK mechanism is not a strict guarantee. .. _psu_hold_up_times: PSU Hold-up Times ----------------- A PSU hold-up time is the amount of time that a PSU can continue to output power at maximum output within standard voltage tolerances following the loss of input power. This is important for supporting UPS units because `the transfer time `__ taken by a standard UPS to supply power from its battery can leave machines without power for "5-12 ms". `Intel's ATX Power Supply design guide `__ specifies a hold up time of 17 milliseconds at maximum continuous output. The hold-up time is a inverse function of how much power is being output by the PSU, with lower power output increasing holdup times. Capacitor aging in PSUs will lower the hold-up time below what it was when new, which could cause reliability issues as equipment ages. Machines using substandard PSUs with hold-up times below the specification therefore require higher end UPS units for protection to ensure that the transfer time does not exceed the hold-up time. A hold-up time below the transfer time during a transfer to battery power can cause undefined behavior should the PWR_OK signal not become deasserted to force the machine to power off. If in doubt, use a double conversion UPS unit. Double conversion UPS units always run off the battery, such that the transfer time is 0. This is unless they are high efficiency models that are hybrids between standard UPS units and double conversion UPS units, although these are reported to have much lower transfer times than standard PSUs. You could also contact your PSU manufacturer for the hold up time specification, but if reliability for years is a requirement, you should use a higher end UPS with a low transfer time. Note that double conversion units are at most 94% efficient unless they support a high efficiency mode, which adds latency to the time to transition to battery power. .. _ups_batteries: UPS batteries ------------- The lead acid batteries in UPS units generally need to be replaced regularly to ensure that they provide power during power outages. For home systems, this is every 3 to 5 years, although this varies with temperature [#ups_temp]_. For enterprise systems, contact your vendor. .. rubric:: Footnotes .. [#ssd_analysis] .. [#ssd_analysis2] .. [#ssd_analysis3] .. [#ups_temp] Workload Tuning =============== Below are tips for various workloads. .. contents:: Table of Contents :local: .. _basic_concepts: Basic concepts -------------- Descriptions of ZFS internals that have an effect on application performance follow. .. _adaptive_replacement_cache: Adaptive Replacement Cache ~~~~~~~~~~~~~~~~~~~~~~~~~~ See :doc:`Caching and Auxiliary Devices ` for how the ARC relates to L2ARC, SLOG and special vdevs, and when each is worth adding. For decades, operating systems have used RAM as a cache to avoid the necessity of waiting on disk IO, which is extremely slow. This concept is called page replacement. Until ZFS, virtually all filesystems used the Least Recently Used (LRU) page replacement algorithm in which the least recently used pages are the first to be replaced. Unfortunately, the LRU algorithm is vulnerable to cache flushes, where a brief change in workload that occurs occasionally removes all frequently used data from cache. The Adaptive Replacement Cache (ARC) algorithm was implemented in ZFS to replace LRU. It solves this problem by maintaining four lists: #. A list for recently cached entries. #. A list for recently cached entries that have been accessed more than once. #. A list for entries evicted from #1. #. A list of entries evicited from #2. Data is evicted from the first list while an effort is made to keep data in the second list. In this way, ARC is able to outperform LRU by providing a superior hit rate. In addition, a dedicated cache device (typically a SSD) can be added to the pool, with ``zpool add POOLNAME cache DEVICENAME``. The cache device is managed by the L2ARC, which scans entries that are next to be evicted and writes them to the cache device. The data stored in ARC and L2ARC can be controlled via the ``primarycache`` and ``secondarycache`` zfs properties respectively, which can be set on both zvols and datasets. Possible settings are ``all``, ``none`` and ``metadata``. It is possible to improve performance when a zvol or dataset hosts an application that does its own caching by caching only metadata. One example would be a virtual machine using ZFS. Another would be a database system which manages its own cache (Oracle for instance). PostgreSQL, by contrast, depends on the OS-level file cache for the majority of cache. .. _alignment_shift_ashift: Alignment Shift (ashift) ~~~~~~~~~~~~~~~~~~~~~~~~ Top-level vdevs contain an internal property called ashift, which stands for alignment shift. It is set at vdev creation and it is immutable. It can be read using the ``zdb`` command. It is calculated as the maximum base 2 logarithm of the physical sector size of any child vdev and it alters the disk format such that writes are always done according to it. This makes 2^ashift the smallest possible IO on a vdev. Configuring ashift correctly is important because partial sector writes incur a penalty where the sector must be read into a buffer before it can be written. ZFS makes the implicit assumption that the sector size reported by drives is correct and calculates ashift based on that. In an ideal world, physical sector size is always reported correctly and therefore, this requires no attention. Unfortunately, this is not the case. The sector size on all storage devices was 512-bytes prior to the creation of flash-based solid state drives. Some operating systems, such as Windows XP, were written under this assumption and will not function when drives report a different sector size. Flash-based solid state drives came to market around 2007. These devices report 512-byte sectors, but the actual flash pages, which roughly correspond to sectors, are never 512-bytes. The early models used 4096-byte pages while the newer models have moved to an 8192-byte page. In addition, "Advanced Format" hard drives have been created which also use a 4096-byte sector size. Partial page writes suffer from similar performance degradation as partial sector writes. In some cases, the design of NAND-flash makes the performance degradation even worse, but that is beyond the scope of this description. Reporting the correct sector sizes is the responsibility of the block device layer. Since many devices misreport their sector sizes and ZFS relies on the block device layer for this information, each platform has developed different workarounds. The platform-specific methods are as follows: - `sd.conf `__ on illumos - `gnop(8) `__ on FreeBSD; see for example `FreeBSD on 4K sector drives `__ (2011-01-01) - :ref:`ashift= ` on ZFS on Linux - -o ashift= also works with both MacZFS (pool version 8) and ZFS-OSX (pool version 5000). -o ashift= is convenient, but it is flawed in that the creation of pools containing top level vdevs that have multiple optimal sector sizes require the use of multiple commands. `A newer syntax `__ that will rely on the actual sector sizes has been discussed as a cross platform replacement and will likely be implemented in the future. In addition, there is a `database of drives known to misreport sector sizes `__ to the ZFS on Linux project. It is used to automatically adjust ashift without the assistance of the system administrator. This approach is unable to fully compensate for misreported sector sizes whenever drive identifiers are used ambiguously (e.g. virtual machines, iSCSI LUNs, some rare SSDs), but it does a great amount of good. The format is roughly compatible with illumos' sd.conf and it is expected that other implementations will integrate the database in future releases. Strictly speaking, this database does not belong in ZFS, but the difficulty of patching the Linux kernel (especially older ones) necessitated that this be implemented in ZFS itself for Linux. The same is true for MacZFS. However, FreeBSD and illumos are both able to implement this in the correct layer. Compression ~~~~~~~~~~~ Internally, ZFS allocates data using multiples of the device's sector size, typically either 512 bytes or 4KB (see above). When compression is enabled, a smaller number of sectors can be allocated for each block. The uncompressed block size is set by the ``recordsize`` (defaults to 128KB) or ``volblocksize`` (defaults to 16KB since v2.2) property (for filesystems vs volumes). The following compression algorithms are available: - LZ4 - New algorithm added after feature flags were created. It is significantly superior to LZJB in all metrics tested. It is `new default compression algorithm `__ (compression=on) in OpenZFS. It is available on all platforms as of 2020. - LZJB - Original default compression algorithm (compression=on) for ZFS. It was created to satisfy the desire for a compression algorithm suitable for use in filesystems. Specifically, that it provides fair compression, has a high compression speed, has a high decompression speed and detects incompressible data quickly. - GZIP (1 through 9) - Classic Lempel-Ziv implementation. It provides high compression, but it often makes IO CPU-bound. - ZLE (Zero Length Encoding) - A very simple algorithm that only compresses zeroes. - ZSTD (Zstandard) - Zstandard is a modern, high performance, general compression algorithm which provides similar or better compression levels to GZIP, but with much better performance. Zstandard offers a very wide range of performance/compression trade-off, and is backed by an extremely fast decoder. It is available from `OpenZFS 2.0 version `__. If you want to use compression and are uncertain which to use, use LZ4. It averages a 2.1:1 compression ratio while gzip-1 averages 2.7:1, but gzip is much slower. Both figures are obtained from `testing by the LZ4 project `__ on the Silesia corpus. The greater compression ratio of gzip is usually only worthwhile for rarely accessed data. .. _raid_z_stripe_width: RAID-Z stripe width ~~~~~~~~~~~~~~~~~~~ Choose a RAID-Z stripe width based on your IOPS needs and the amount of space you are willing to devote to parity information. If you need more IOPS, use fewer disks per stripe. If you need more usable space, use more disks per stripe. Trying to optimize your RAID-Z stripe width based on exact numbers is irrelevant in nearly all cases. See this `blog post `__ for more details. .. _dataset_recordsize: Dataset recordsize ~~~~~~~~~~~~~~~~~~ ZFS datasets use an internal recordsize of 128KB by default. The dataset recordsize is the basic unit of data used for internal copy-on-write on files. Partial record writes require that data be read from either ARC (cheap) or disk (expensive). recordsize can be set to any power of 2 from 512 bytes to 1 megabyte. Software that writes in fixed record sizes (e.g. databases) will benefit from the use of a matching recordsize. Changing the recordsize on a dataset will only take effect for new files. If you change the recordsize because your application should perform better with a different one, you will need to recreate its files. A cp followed by a mv on each file is sufficient. Alternatively, send/recv should recreate the files with the correct recordsize when a full receive is done. .. _larger_record_sizes: Larger record sizes ^^^^^^^^^^^^^^^^^^^ Record sizes of up to 16M are supported with the large_blocks pool feature, which is enabled by default on new pools on systems that support it. Record sizes larger than 1M were disabled by default before openZFS v2.2, unless the zfs_max_recordsize kernel module parameter was set to allow sizes higher than 1M. \`zfs send\` operations must specify -L to ensure that larger than 128KB blocks are sent and the receiving pools must support the large_blocks feature. .. _zvol_volblocksize: zvol volblocksize ~~~~~~~~~~~~~~~~~ Zvols have a ``volblocksize`` property that is analogous to ``recordsize``. Current default (16KB since v2.2) balances the metadata overhead, compression opportunities and decent space efficiency on majority of pool configurations due to 4KB disk physical block rounding (especially on RAIDZ and DRAID), while incurring some write amplification on guest FSes that run with smaller block sizes [#VOLBLOCKSIZE]_. Users are advised to test their scenarios and see whether the ``volblocksize`` needs to be changed to favor one or the other: - sector alignment of guest FS is crucial - most of guest FSes use default block size of 4-8KB, so: - Larger ``volblocksize`` can help with mostly sequential workloads and will gain a compression efficiency - Smaller ``volblocksize`` can help with random workloads and minimize IO amplification, but will use more metadata (e.g. more small IOs will be generated by ZFS) and may have worse space efficiency (especially on RAIDZ and DRAID) - It's meaningless to set ``volblocksize`` less than guest FS's block size or :ref:`ashift ` - See :ref:`Dataset recordsize ` for additional information Deduplication ~~~~~~~~~~~~~ See :doc:`Deduplication ` for the operational side, including the ``dedup_table_quota`` and ``zpool ddtprune`` mechanisms for keeping the table bounded, and dedicated dedup vdevs. Deduplication uses an on-disk hash table, using `extensible hashing `__ as implemented in the ZAP (ZFS Attribute Processor). Each cached entry uses slightly more than 320 bytes of memory. The DDT code relies on ARC for caching the DDT entries, such that there is no double caching or internal fragmentation from the kernel memory allocator. Each pool has a global deduplication table shared across all datasets and zvols on which deduplication is enabled. Each entry in the hash table is a record of a unique block in the pool. (Where the block size is set by the ``recordsize`` or ``volblocksize`` properties.) The hash table (also known as the DDT or DeDup Table) must be accessed for every dedup-able block that is written or freed (regardless of whether it has multiple references). If there is insufficient memory for the DDT to be cached in memory, each cache miss will require reading a random block from disk, resulting in poor performance. For example, if operating on a single 7200RPM drive that can do 100 io/s, uncached DDT reads would limit overall write throughput to 100 blocks per second, or 400KB/s with 4KB blocks. The consequence is that sufficient memory to store deduplication data is required for good performance. The deduplication data is considered metadata and therefore can be cached if the ``primarycache`` or ``secondarycache`` properties are set to ``metadata``. In addition, the deduplication table will compete with other metadata for metadata storage, which can have a negative effect on performance. Simulation of the number of deduplication table entries needed for a given pool can be done using the ``-S`` option to zdb, which constructs a DDT for the data already in the pool and reports the ratio you would have obtained. (``-D`` displays the DDT that already exists, if dedup is enabled.) Then a simple multiplication by 320-bytes can be done to get the approximate memory requirements. Alternatively, you can estimate an upper bound on the number of unique blocks by dividing the amount of storage you plan to use on each dataset (taking into account that partial records each count as a full recordsize for the purposes of deduplication) by the recordsize and each zvol by the volblocksize, summing and then multiplying by 320-bytes. .. _metaslab_allocator: Metaslab Allocator ~~~~~~~~~~~~~~~~~~ ZFS top level vdevs are divided into metaslabs from which blocks can be independently allocated to allow for concurrent IOs to perform allocations without blocking one another. By default, the selection of a metaslab is biased toward lower LBAs to improve performance of spinning disks, but this does not make sense on solid state media. This behavior can be adjusted globally by setting the ZFS module's global metaslab_lba_weighting_enabled tuanble to 0. This tunable is only advisable on systems that only use solid state media for pools. The metaslab allocator will allocate blocks on a first-fit basis when a metaslab has more than or equal to 4 percent free space and a best-fit basis when a metaslab has less than 4 percent free space. The former is much faster than the latter, but it is not possible to tell when this behavior occurs from the pool's free space. However, the command ``zdb -mmm $POOLNAME`` will provide this information. .. _pool_geometry: Pool Geometry ~~~~~~~~~~~~~ If small random IOPS are of primary importance, mirrored vdevs will outperform raidz vdevs. Read IOPS on mirrors will scale with the number of drives in each mirror while raidz vdevs will each be limited to the IOPS of the slowest drive. If sequential writes are of primary importance, raidz will outperform mirrored vdevs. Sequential write throughput increases linearly with the number of data disks in raidz while writes are limited to the slowest drive in mirrored vdevs. Sequential read performance should be roughly the same on each. Both IOPS and throughput will increase by the respective sums of the IOPS and throughput of each top level vdev, regardless of whether they are raidz or mirrors. .. _whole_disks_versus_partitions: Whole Disks versus Partitions ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ZFS will behave differently on different platforms when given a whole disk. On illumos, ZFS attempts to enable the write cache on a whole disk. The illumos UFS driver cannot ensure integrity with the write cache enabled, so by default Sun/Solaris systems using UFS file system for boot were shipped with drive write cache disabled (long ago, when Sun was still an independent company). For safety on illumos, if ZFS is not given the whole disk, it could be shared with UFS and thus it is not appropriate for ZFS to enable write cache. In this case, the write cache setting is not changed and will remain as-is. Today, most vendors ship drives with write cache enabled by default. On Linux, the Linux IO elevator is largely redundant given that ZFS has its own IO elevator. ZFS will also create a GPT partition table own partitions when given a whole disk under illumos on x86/amd64 and on Linux. This is mainly to make booting through UEFI possible because UEFI requires a small FAT partition to be able to boot the system. The ZFS driver will be able to tell the difference between whether the pool had been given the entire disk or not via the whole_disk field in the label. This is not done on FreeBSD. Pools created by FreeBSD will always have the whole_disk field set to true, such that a pool imported on another platform that was created on FreeBSD will always be treated as the whole disks were given to ZFS. .. _OS_specific: OS/distro-specific recommendations ---------------------------------- .. _linux_specific: Linux ~~~~~ init_on_alloc ^^^^^^^^^^^^^ Some Linux distributions (at least Debian, Ubuntu) enable ``init_on_alloc`` option as security precaution by default. This option can help to [#init_on_alloc]_: prevent possible information leaks and make control-flow bugs that depend on uninitialized values more deterministic. Unfortunately, it can lower ARC throughput considerably (see `bug `__). If you're ready to cope with these security risks [#init_on_alloc]_, you may disable it by setting ``init_on_alloc=0`` in the GRUB kernel boot parameters. .. _general_recommendations: General recommendations ----------------------- .. _alignment_shift: Alignment shift ~~~~~~~~~~~~~~~ Make sure that you create your pools such that the vdevs have the correct alignment shift for your storage device's size. if dealing with flash media, this is going to be either 12 (4K sectors) or 13 (8K sectors). For SSD ephemeral storage on Amazon EC2, the proper setting is 12. .. _atime_updates: Atime Updates ~~~~~~~~~~~~~ Set either relatime=on or atime=off to minimize IOs used to update access time stamps. For backward compatibility with a small percentage of software that supports it, relatime is preferred when available and should be set on your entire pool. atime=off should be used more selectively. .. _free_space: Free Space ~~~~~~~~~~ Keep pool free space above 10% to avoid many metaslabs from reaching the 4% free space threshold to switch from first-fit to best-fit allocation strategies. When the threshold is hit, the :ref:`metaslab_allocator` becomes very CPU intensive in an attempt to protect itself from fragmentation. This reduces IOPS, especially as more metaslabs reach the 4% threshold. The recommendation is 10% rather than 5% because metaslabs selection considers both location and free space unless the global metaslab_lba_weighting_enabled tunable is set to 0. When that tunable is 0, ZFS will consider only free space, so the expense of the best-fit allocator can be avoided by keeping free space above 5%. That setting should only be used on systems with pools that consist of solid state drives because it will reduce sequential IO performance on mechanical disks. .. _lz4_compression: LZ4 compression ~~~~~~~~~~~~~~~ Since OpenZFS 2.2.0 ``compression`` defaults to ``on``, which selects LZ4 on any pool with the ``lz4_compress`` feature enabled, so there is usually nothing to do. On pools created under older releases, or where compression was explicitly disabled, set compression=lz4 (or ``on``) on the pools' root datasets so that all datasets inherit it unless you have a reason not to enable it. Userland tests of LZ4 compression of incompressible data in a single thread has shown that it can process 10GB/sec, so it is unlikely to be a bottleneck even on incompressible data. Furthermore, incompressible data will be stored without compression such that reads of incompressible data with compression enabled will not be subject to decompression. Writes are so fast that in-compressible data is unlikely to see a performance penalty from the use of LZ4 compression. The reduction in IO from LZ4 will typically be a performance win. Note that larger record sizes will increase compression ratios on compressible data by allowing compression algorithms to process more data at a time. .. _nvme_low_level_formatting_link: NVMe low level formatting ~~~~~~~~~~~~~~~~~~~~~~~~~ See :ref:`nvme_low_level_formatting`. .. _pool_geometry_1: Pool Geometry ~~~~~~~~~~~~~ Do not put more than ~16 disks in raidz. The rebuild times on mechanical disks will be excessive when the pool is full. .. _synchronous_io: Synchronous I/O ~~~~~~~~~~~~~~~ If your workload involves fsync or O_SYNC and your pool is backed by mechanical storage, consider adding one or more SLOG devices. Pools that have multiple SLOG devices will distribute ZIL operations across them. Optane / 3D XPoint SSDs remain the best SLOG devices, but they are no longer manufactured — see :ref:`optane_3d_xpoint_ssds`. If an Optane / 3D XPoint SSD is an option, the rest of this section on synchronous I/O need not be read. If Optane / 3D XPoint SSDs is not an option, see :ref:`nand_flash_ssds` for suggestions for NAND flash SSDs and also read the information below. To ensure maximum ZIL performance on NAND flash SSD-based SLOG devices, you should also overprovison spare area to increase IOPS [#ssd_iops]_. Only about 4GB is needed, so the rest can be left as overprovisioned storage. The choice of 4GB is somewhat arbitrary. Most systems do not write anything close to 4GB to ZIL between transaction group commits, so overprovisioning all storage beyond the 4GB partition should be alright. If a workload needs more, then make it no more than the maximum ARC size. Even under extreme workloads, ZFS will not benefit from more SLOG storage than the maximum ARC size. That is half of system memory on Linux and 3/4 of system memory on illumos. .. _overprovisioning_by_secure_erase_and_partition_table_trick: Overprovisioning by secure erase and partition table trick ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ You can do this with a mix of a secure erase and a partition table trick, such as the following: #. Run a secure erase on the NAND-flash SSD. #. Create a partition table on the NAND-flash SSD. #. Create a 4GB partition. #. Give the partition to ZFS to use as a log device. If using the secure erase and partition table trick, do *not* use the unpartitioned space for other things, even temporarily. That will reduce or eliminate the overprovisioning by marking pages as dirty. Alternatively, some devices allow you to change the sizes that they report.This would also work, although a secure erase should be done prior to changing the reported size to ensure that the SSD recognizes the additional spare area. Changing the reported size can be done on drives that support it with \`hdparm -N \` on systems that have laptop-mode-tools. .. _nvme_overprovisioning: NVMe overprovisioning ^^^^^^^^^^^^^^^^^^^^^ On NVMe, you can use namespaces to achieve overprovisioning: #. Do a sanitize command as a precaution to ensure the device is completely clean. #. Delete the default namespace. #. Create a new namespace of size 4GB. #. Give the namespace to ZFS to use as a log device. e.g. zfs add tank log /dev/nvme1n1 .. _whole_disks: Whole disks ~~~~~~~~~~~ Whole disks should be given to ZFS rather than partitions. If you must use a partition, make certain that the partition is properly aligned to avoid read-modify-write overhead. See the section on :ref:`Alignment Shift (ashift) ` for a description of proper alignment. Also, see the section on :ref:`Whole Disks versus Partitions ` for a description of changes in ZFS behavior when operating on a partition. Single disk RAID 0 arrays from RAID controllers are not equivalent to whole disks. The :ref:`hardware_raid_controllers` page explains in detail. .. _bit_torrent: Bit Torrent ----------- Bit torrent performs 16KB random reads/writes. The 16KB writes cause read-modify-write overhead. The read-modify-write overhead can reduce performance by a factor of 16 with 128KB record sizes when the amount of data written exceeds system memory. This can be avoided by using a dedicated dataset for bit torrent downloads with recordsize=16KB. When the files are read sequentially through a HTTP server, the random nature in which the files were generated creates fragmentation that has been observed to reduce sequential read performance by a factor of two on 7200RPM hard disks. If performance is a problem, fragmentation can be eliminated by rewriting the files sequentially in either of two ways: The first method is to configure your client to download the files to a temporary directory and then copy them into their final location when the downloads are finished, provided that your client supports this. The second method is to use send/recv to recreate a dataset sequentially. In practice, defragmenting files obtained through bit torrent should only improve performance when the files are stored on magnetic storage and are subject to significant sequential read workloads after creation. .. _database_workloads: Database workloads ------------------ Setting ``redundant_metadata=most`` can increase IOPS by at least a few percentage points by eliminating redundant metadata at the lowest level of the indirect block tree. This comes with the caveat that data loss will occur if a metadata block pointing to data blocks is corrupted and there are no duplicate copies, but this is generally not a problem in production on mirrored or raidz vdevs. MySQL ~~~~~ InnoDB ^^^^^^ Make separate datasets for InnoDB's data files and log files. Set ``recordsize=16K`` on InnoDB's data files to avoid expensive partial record writes and leave recordsize=128K on the log files. Set ``primarycache=metadata`` on both to prefer InnoDB's caching [#mysql_basic]_. Set ``logbias=throughput`` on the data to stop ZIL from writing twice. Set ``innodb_doublewrite=0`` in my.cnf to prevent innodb from writing twice. The double writes are a data integrity feature meant to protect against corruption from partially-written records, but those are not possible on ZFS. It should be noted that `Percona’s blog had advocated `__ using an ext4 configuration where double writes were turned off for a performance gain, but later recanted it because it caused data corruption. Following a well timed power failure, an in place filesystem such as ext4 can have half of a 8KB record be old while the other half would be new. This would be the corruption that caused Percona to recant its advice. However, ZFS’ copy on write design would cause it to return the old correct data following a power failure (no matter what the timing is). That prevents the corruption that the double write feature is intended to prevent from ever happening. The double write feature is therefore unnecessary on ZFS and can be safely turned off for better performance. On Linux, the driver's AIO implementation is a compatibility shim that just barely passes the POSIX standard. InnoDB performance suffers when using its default AIO codepath. Set ``innodb_use_native_aio=0`` and ``innodb_use_atomic_writes=0`` in my.cnf to disable AIO. Both of these settings must be disabled to disable AIO. PostgreSQL ~~~~~~~~~~ Make separate datasets for PostgreSQL's data and WAL. Set ``compression=lz4`` and ``recordsize=32K`` (64K also work well, as does the 128K default) on both. Previous advice before ``compression=lz4`` was to to set ``recordsize=8K`` to match PostgreSQL's 8K page size to avoid write amplification, but compression often causes even bigger performance improvements [#compression_larger_block_size]_, and compression is only effective when given larger ``recordesize``. Configure ``full_page_writes = off`` for PostgreSQL, as ZFS will never commit a partial write. But do not do this if you set ``recordsize`` < 8K for some reason, or if there is any chance that somebody may copy this database config to a non-ZFS system, as PostgreSQL can end up with corrupt data on power loss if the filesystem does not provide at least Postgres-page-sized atomic writes [#postgres_full_page_writes]_. For a database with large updates, experiment with ``logbias=throughput`` on PostgreSQL's data to avoid writing twice, but be aware that with this setting smaller updates can cause severe fragmentation. SQLite ~~~~~~ Make a separate dataset for the database. Set the recordsize to 64K. Set the SQLite page size to 65536 bytes [#sqlite_ps]_. Note that SQLite databases typically are not exercised enough to merit special tuning, but this will provide it. Note the side effect on cache size mentioned at SQLite.org [#sqlite_ps_change]_. .. _file_servers: File servers ------------ Create a dedicated dataset for files being served. See :ref:`Sequential workloads ` for configuration recommendations. Samba ~~~~~ Windows/DOS clients doesn't support case sensitive file names. If your main workload won't need case sensitivity for other supported clients, create dataset with ``zfs create -o casesensitivity=insensitive`` so Samba may search filenames faster in future [#FS_CASEFOLD_FL]_. See ``case sensitive`` option in `smb.conf(5) `__. .. _sequential_workloads: Sequential workloads -------------------- Set ``recordsize=1M`` on datasets that are subject to sequential workloads. Read :ref:`Larger record sizes ` for documentation on things that should be known before setting 1M record sizes. Set ``compression=lz4`` as per the general recommendation for :ref:`LZ4 compression `. .. _video_games_directories: Video games directories ----------------------- Create a dedicated dataset, use chown to make it user accessible (or create a directory under it and use chown on that) and then configure the game download application to place games there. Specific information on how to configure various ones is below. See :ref:`Sequential workloads ` for configuration recommendations before installing games. Note that the performance gains from this tuning are likely to be small and limited to load times. However, the combination of 1M records and LZ4 will allow more games to be stored, which is why this tuning is documented despite the performance gains being limited. A steam library of 300 games (mostly from humble bundle) that had these tweaks applied to it saw 20% space savings. Both faster load times and significant space savings are possible on compressible games when this tuning has been done. Games whose assets are already compressed will see little to no benefit. Lutris ~~~~~~ Open the context menu by left clicking on the triple bar icon in the upper right. Go to "Preferences" and then the "System options" tab. Change the default installation directory and click save. Steam ~~~~~ Go to "Settings" -> "Downloads" -> "Steam Library Folders" and use "Add Library Folder" to set the directory for steam to use to store games. Make sure to set it to the default by right clicking on it and clicking "Make Default Folder" before closing the dialogue. If you'll use Proton to run non-native games, create dataset with ``zfs create -o casesensitivity=insensitive`` so Wine may search filenames faster in future [#FS_CASEFOLD_FL]_. .. _wine: Wine ---- Windows file systems' standard behavior is to be case-insensitive. Create dataset with ``zfs create -o casesensitivity=insensitive`` so Wine may search filenames faster in future [#FS_CASEFOLD_FL]_. .. _virtual_machines: Virtual machines ---------------- Virtual machine images on ZFS should be stored using either zvols or raw files to avoid unnecessary overhead. The recordsize/volblocksize and guest filesystem may be configured to match to avoid overhead from partial record modification, see :ref:`zvol volblocksize `. If raw files are used, a separate dataset should be used to make it easy to configure recordsize independently of other things stored on ZFS. .. _qemu_kvm_xen: QEMU / KVM / Xen ~~~~~~~~~~~~~~~~ AIO should be used to maximize IOPS when using files for guest storage. .. rubric:: Footnotes .. [#compression_larger_block_size] .. [#postgres_full_page_writes] https://wiki.postgresql.org/wiki/Full_page_writes .. [#ssd_iops] .. [#mysql_basic] .. [#sqlite_ps] .. [#sqlite_ps_change] .. [#FS_CASEFOLD_FL] .. [#init_on_alloc] .. [#VOLBLOCKSIZE] OpenZFS Transaction Delay ========================= OpenZFS write operations are delayed when the backend storage isn't able to accommodate the rate of incoming writes. This delay process is known as the OpenZFS write throttle. As different hardware and workloads have different performance characteristics, tuning the write throttle is hardware and workload specific. Writes are grouped into transactions. Transactions are grouped into transaction groups. When a transaction group is synced to disk, all transactions in that group are considered complete. When a delay is applied to a transaction it delays the transaction's assignment into a transaction group. To check if write throttle is activated on a pool, monitor the dmu_tx_delay and/or dmu_tx_dirty_delay kstat counters. If there is already a write transaction waiting, the delay is relative to when that transaction will finish waiting. Thus the calculated delay time is independent of the number of threads concurrently executing transactions. If there is only one waiter, the delay is relative to when the transaction started, rather than the current time. This credits the transaction for "time already served." For example, if a write transaction requires reading indirect blocks first, then the delay is counted at the start of the transaction, just prior to the indirect block reads. The minimum time for a transaction to take is calculated as: :: min_time = zfs_delay_scale * (dirty - min) / (max - dirty) min_time is then capped at 100 milliseconds The delay has two degrees of freedom that can be adjusted via tunables: 1. The percentage of dirty data at which we start to delay is defined by zfs_delay_min_dirty_percent. This is typically be at or above zfs_vdev_async_write_active_max_dirty_percent so delays occur after writing at full speed has failed to keep up with the incoming write rate. 2. The scale of the curve is defined by zfs_delay_scale. Roughly speaking, this variable determines the amount of delay at the midpoint of the curve. :: delay 10ms +-------------------------------------------------------------*+ | *| 9ms + *+ | *| 8ms + *+ | * | 7ms + * + | * | 6ms + * + | * | 5ms + * + | * | 4ms + * + | * | 3ms + * + | * | 2ms + (midpoint) * + | | ** | 1ms + v *** + | zfs_delay_scale ----------> ******** | 0 +-------------------------------------*********----------------+ 0% <- zfs_dirty_data_max -> 100% Note that since the delay is added to the outstanding time remaining on the most recent transaction, the delay is effectively the inverse of IOPS. Here the midpoint of 500 microseconds translates to 2000 IOPS. The shape of the curve was chosen such that small changes in the amount of accumulated dirty data in the first 3/4 of the curve yield relatively small differences in the amount of delay. The effects can be easier to understand when the amount of delay is represented on a log scale: :: delay 100ms +-------------------------------------------------------------++ + + | | + *+ 10ms + *+ + ** + | (midpoint) ** | + | ** + 1ms + v **** + + zfs_delay_scale ----------> ***** + | **** | + **** + 100us + ** + + * + | * | + * + 10us + * + + + | | + + +--------------------------------------------------------------+ 0% <- zfs_dirty_data_max -> 100% Note here that only as the amount of dirty data approaches its limit does the delay start to increase rapidly. The goal of a properly tuned system should be to keep the amount of dirty data out of that range by first ensuring that the appropriate limits are set for the I/O scheduler to reach optimal throughput on the backend storage, and then by changing the value of zfs_delay_scale to increase the steepness of the curve. Code reference: `dmu_tx.c `_ ZFS I/O (ZIO) Scheduler ======================= ZFS issues I/O operations to leaf :doc:`vdevs ` (usually devices) to satisfy and complete I/Os. The ZIO scheduler determines when and in what order those operations are issued. Operations are divided into nine I/O classes prioritized in the following order: +-----------+--------------+-------------------------------------------+-----------------+ | Priority | I/O Class | Description | I/O Type | +===========+==============+===========================================+=================+ | highest | sync read | most reads | interactive | +-----------+--------------+-------------------------------------------+-----------------+ | | sync write | as defined by application or via 'zfs' | interactive | | | | 'sync' property | | +-----------+--------------+-------------------------------------------+-----------------+ | | async read | prefetch reads | interactive | +-----------+--------------+-------------------------------------------+-----------------+ | see below | async write | most writes | interactive | +-----------+--------------+-------------------------------------------+-----------------+ | | scrub read | scan reads: includes both scrub and | non-interactive | | | | resilver | | +-----------+--------------+-------------------------------------------+-----------------+ | | removal | vdev removal reflow | non-interactive | +-----------+--------------+-------------------------------------------+-----------------+ | | initializing | vdev space initialization | non-interactive | +-----------+--------------+-------------------------------------------+-----------------+ | | trim | TRIM/UNMAP requests | interactive | +-----------+--------------+-------------------------------------------+-----------------+ | lowest | rebuild | sequential reconstruction | non-interactive | +-----------+--------------+-------------------------------------------+-----------------+ For interactive I/Os each queue defines the minimum and maximum number of concurrent operations issued to the device. Each device also respects an aggregate maximum - `zfs_vdev_max_active <../Performance and Tuning/Module Parameters.html#zfs-vdev-max-active>`_. Note that the sum of the per-queue minimums must not exceed the aggregate maximum. If the sum of the per-queue maximums exceeds the aggregate maximum, then the number of active I/Os may reach `zfs_vdev_max_active <../Performance and Tuning/Module Parameters.html#zfs-vdev-max-active>`_, in which case no further I/Os are issued regardless of whether all per-queue minimums have been met. For non-interactive I/O (scrub, resilver, removal, initializing, and rebuild), the number of concurrently-active I/Os is limited to _min_active, unless the vdev is "idle". When there are no interactive I/Os active (sync or async), and `zfs_vdev_nia_delay <../Performance and Tuning/Module Parameters.html#zfs-vdev-nia-delay>`_ I/Os have completed since the last interactive I/O, then the vdev is considered to be "idle", and the number of concurrently-active non-interactive I/Os is increased to _max_active. +-------------+-------------------------------------+-------------------------------------+ | I/O Class | Min Active Parameter | Max Active Parameter | +=============+=====================================+=====================================+ | sync read | ``zfs_vdev_sync_read_min_active`` | ``zfs_vdev_sync_read_max_active`` | +-------------+-------------------------------------+-------------------------------------+ | sync write | ``zfs_vdev_sync_write_min_active`` | ``zfs_vdev_sync_write_max_active`` | +-------------+-------------------------------------+-------------------------------------+ | async read | ``zfs_vdev_async_read_min_active`` | ``zfs_vdev_async_read_max_active`` | +-------------+-------------------------------------+-------------------------------------+ | async write | ``zfs_vdev_async_write_min_active`` | ``zfs_vdev_async_write_max_active`` | +-------------+-------------------------------------+-------------------------------------+ | scrub read | ``zfs_vdev_scrub_min_active`` | ``zfs_vdev_scrub_max_active`` | +-------------+-------------------------------------+-------------------------------------+ | removal | ``zfs_vdev_removal_min_active`` | ``zfs_vdev_removal_max_active`` | +-------------+-------------------------------------+-------------------------------------+ | initializing| ``zfs_vdev_initializing_min_active``| ``zfs_vdev_initializing_max_active``| +-------------+-------------------------------------+-------------------------------------+ | trim | ``zfs_vdev_trim_min_active`` | ``zfs_vdev_trim_max_active`` | +-------------+-------------------------------------+-------------------------------------+ | rebuild | ``zfs_vdev_rebuild_min_active`` | ``zfs_vdev_rebuild_max_active`` | +-------------+-------------------------------------+-------------------------------------+ I/O queue statistics include most of the I/Os classes and can be viewed via the `zpool iostat -q `_ command For many physical devices, throughput increases with the number of concurrent operations, but latency typically suffers. Further, physical devices typically have a limit at which more concurrent operations have no effect on throughput or can cause the disk performance to decrease. The ZIO scheduler selects the next operation to issue by first looking for an I/O class whose minimum has not been satisfied. Once all are satisfied and the aggregate maximum has not been hit, the scheduler looks for classes whose maximum has not been satisfied. Iteration through the I/O classes is done in the order specified above. No further operations are issued if the aggregate maximum number of concurrent operations has been hit or if there are no operations queued for an I/O class that has not hit its maximum. Every time an I/O is queued or an operation completes, the I/O scheduler looks for new operations to issue. In general, smaller max_active's will lead to lower latency of synchronous operations. Larger max_active's may lead to higher overall throughput, depending on underlying storage and the I/O mix. The ratio of the queues' max_actives determines the balance of performance between reads, writes, and scrubs. For example, when there is contention, increasing zfs_vdev_scrub_max_active will cause the scrub or resilver to complete more quickly, but reads and writes to have higher latency and lower throughput. All I/O classes have a fixed maximum number of outstanding operations except for the async write class. Asynchronous writes represent the data that is committed to stable storage during the syncing stage for transaction groups. Transaction groups enter the syncing state periodically so the number of queued async writes quickly bursts up and then reduce down to zero. The zfs_txg_timeout tunable (default=5 seconds) sets the target interval for txg sync. Thus a burst of async writes every 5 seconds is a normal ZFS I/O pattern. Rather than servicing I/Os as quickly as possible, the ZIO scheduler changes the maximum number of active async write I/Os according to the amount of dirty data in the pool. Since both throughput and latency typically increase as the number of concurrent operations issued to physical devices, reducing the burstiness in the number of concurrent operations also stabilizes the response time of operations from other queues. This is particularly important for the sync read and write queues, where the periodic async write bursts of the txg sync can lead to device-level contention. In broad strokes, the ZIO scheduler issues more concurrent operations from the async write queue as there's more dirty data in the pool. Async Write I/O Scheduling ========================== The number of concurrent operations issued for the async write I/O class follows a piece-wise linear function defined by a few adjustable points:: | o---------| <-- zfs_vdev_async_write_max_active ^ | /^ | | | / | | active | / | | I/O | / | | count | / | | | / | | |------------o | | <-- zfs_vdev_async_write_min_active 0|____________^______|_________| 0% | | 100% of zfs_dirty_data_max | | | `-- zfs_vdev_async_write_active_max_dirty_percent `--------- zfs_vdev_async_write_active_min_dirty_percent Until the amount of dirty data exceeds a minimum percentage of the dirty data allowed in the pool, the I/O scheduler will limit the number of concurrent operations to the minimum. As that threshold is crossed, the number of concurrent operations issued increases linearly to the maximum at the specified maximum percentage of the dirty data allowed in the pool. Ideally, the amount of dirty data on a busy pool will stay in the sloped part of the function between zfs_vdev_async_write_active_min_dirty_percent and zfs_vdev_async_write_active_max_dirty_percent. If it exceeds the maximum percentage, this indicates that the rate of incoming data is greater than the rate that the backend storage can handle. In this case, we must further throttle incoming writes. See :doc:`ZFS Transaction Delay ` for details. Code reference: `vdev_queue.c `_ CSTYLE(1) - General Commands Manual # NAME **cstyle** - check for some common stylistic errors in C source files # SYNOPSIS **cstyle** \[**-chpvCP**] \[*file*]… # DESCRIPTION **cstyle** inspects C source files (\*.c and \*.h) for common stylistic errors. It attempts to check for the cstyle documented in [http://www.cis.upenn.edu/~lee/06cse480/data/cstyle.ms.pdf](http://www.cis.upenn.edu/~lee/06cse480/data/cstyle.ms.pdf). Note that there is much in that document that *cannot* be checked for; just because your code is **cstyle**-clean does not mean that you've followed Sun's C style. *Caveat emptor*. # OPTIONS **-c** > Check continuation line indentation inside of functions. > Sun's C style > states that all statements must be indented to an appropriate tab stop, > and any continuation lines after them must be indented > *exactly* > four spaces from the start line. > This option enables a series of checks designed to find > continuation line problems within functions only. > The checks have some limitations; see > **CONTINUATION CHECKING**, > below. **-p** > Performs some of the more picky checks. > Includes ANSI > **#else** > and > **#endif** > rules, and tries to detect spaces after casts. > Used as part of the putback checks. **-v** > Verbose output; includes the text of the line of error, and, for > **-c**, > the first statement in the current continuation block. **-P** > Check for use of non-POSIX types. > Historically, types like > **u\_int** > and > **u\_long** > were used, but they are now deprecated in favor of the POSIX > types > **uint\_t**, > **ulong\_t**, > etc. > This detects any use of the deprecated types. > Used as part of the putback checks. **-g** > Also print GitHub-Actions-style > `::error` > output. # ENVIRONMENT `CI` > If set and nonempty, equivalent to > **-g**. # CONTINUATION CHECKING The continuation checker is a reasonably simple state machine that knows something about how C is laid out, and can match parenthesis, etc. over multiple lines. It does have some limitations: 1. Preprocessor macros which cause unmatched parenthesis will confuse the checker for that line. To fix this, you'll need to make sure that each branch of the **#if** statement has balanced parenthesis. 2. Some cpp(1) macros do not require **;**s after them. Any such macros *must* be ALL\_CAPS; any lower case letters will cause bad output. The bad output will generally be corrected after the next **;**, **{**, or **}**. Some continuation error messages deserve some additional explanation: **multiple statements continued over multiple lines** > A multi-line statement which is not broken at statement boundaries. > For example: > > if (this\_is\_a\_long\_variable == another\_variable) a = > > b + c; > Will trigger this error. > Instead, do: > > if (this\_is\_a\_long\_variable == another\_variable) > > a = b + c; **empty if/for/while body not on its own line** > For visibility, empty bodies for if, for, and while statements should be > on their own line. > For example: > > while (do\_something(&x) == 0); > Will trigger this error. > Instead, do: > > while (do\_something(&x) == 0) > > ; Debian - April 4, 2022 DBUFSTAT(1) - General Commands Manual # NAME **dbufstat** - display statistics for DMU buffer cache # SYNOPSIS **dbufstat** \[**-bdhnrtvx**] \[**-f** *field*\[,*field*…]] \[**-F** *field*=*value*\[,…]] \[**-i** *file*] \[**-o** *file*] \[**-s** *separator*] # DESCRIPTION **dbufstat** displays statistics about the DMU (Data Management Unit) buffer cache, also known as the dbuf cache. This information is read from the **dbufs** kstat (*/proc/spl/kstat/zfs/dbufs* on Linux and **kstat.zfs.misc.dbufs** on FreeBSD) . **dbufstat** can display data in three modes: **-b** > Display a table of information for each individual dbuf > (buffer). > This is the most detailed view. **-d** > Display a table of information for each dnode > (the default mode). > Multiple dbufs belonging to the same dnode are > aggregated. **-t** > Display a table of information for each dnode type. > Multiple dnodes of the same type are aggregated. The following fields are available for display. Not all fields are compatible with all modes; use **-v** to see which fields are incompatible with each mode. **pool** > Pool name. **objset** > Dataset identification number. **object** > Object number. **level** > Indirection level of buffer. **blkid** > Block number of buffer. **offset** > Offset in object of buffer. **dbsize** > Size of buffer. **usize** > Size of attached user data. **meta** > Whether this buffer is metadata. **state** > State of buffer (read, cached, etc). **dbholds** > Number of holds on buffer. **dbc** > Whether this buffer is in the dbuf cache. **list** > Which ARC list contains this buffer. **atype** > ARC header type (data or metadata). **flags** > ARC read flags. **count** > ARC data count. **asize** > Size of this ARC buffer. **access** > Time this ARC buffer was last accessed. **mru** > Hits while on the ARC's MRU list. **gmru** > Hits while on the ARC's MRU ghost list. **mfu** > Hits while on the ARC's MFU list. **gmfu** > Hits while on the ARC's MFU ghost list. **l2** > Hits while on the L2ARC. **l2\_dattr** > L2ARC disk address/offset. **l2\_asize** > L2ARC allocated size (depends on compression). **l2\_comp** > L2ARC compression algorithm for buffer. **aholds** > Number of holds on this ARC buffer. **dtype** > Dnode type. **btype** > Bonus buffer type. **data\_bs** > Data block size. **meta\_bs** > Metadata block size. **bsize** > Bonus buffer size. **lvls** > Number of indirection levels. **dholds** > Number of holds on dnode. **blocks** > Number of allocated blocks. **dsize** > Size of dnode. **cached** > Bytes cached for all blocks. **direct** > Bytes cached for direct blocks. **indirect** > Bytes cached for indirect blocks. **bonus** > Bytes cached for bonus buffer. **spill** > Bytes cached for spill block. # OPTIONS **-b** > Display information for each individual dbuf. **-d** > Display information for each dnode (default). **-f** *field*\[,*field*…] > Display only the specified fields. **-F** *field*=*value*\[,…] > Filter output by field value or regular expression. > Multiple filters may be specified as a comma-separated list. **-h** > Display a help message. **-i** *file* > Redirect input from the specified file instead of the > **dbufs** > kstat. **-n** > Exclude the header from output. **-o** *file* > Redirect output to the specified file instead of standard output. **-r** > Display raw values. > Specify twice to also display numeric type and compression > identifiers instead of names. **-s** *separator* > Override the default field separator (two spaces) with a > custom character or string. **-t** > Display information for each dnode type. **-v** > List all possible field headers, definitions, and > incompatibilities for each mode. **-x** > Display extended statistics. # EXAMPLES Display per-dnode statistics (default mode): # dbufstat -d pool objset object dtype cached zroot 30 34466 DMU_OT_DIRECTORY_CONTENTS 240K zroot 30 0 DMU_OT_DNODE 14M zroot 30 36343 DMU_OT_DIRECTORY_CONTENTS 1024 zroot 30 32921 DMU_OT_PLAIN_FILE_CONTENTS 2.5K Display per-type statistics: # dbufstat -t pool objset dtype cached zroot 30 DMU_OT_DIRECTORY_CONTENTS 1.9M zroot 30 DMU_OT_DNODE 14M zroot 30 DMU_OT_PLAIN_FILE_CONTENTS 8.4M zroot 0 DMU_OT_DSL_PROPS 2.6M Display per-buffer statistics: # dbufstat -b pool objset object level blkid offset dbsize zroot 30 34466 1 0 0 128K zroot 30 0 0 292 4.6M 16K zroot 30 34466 0 8 128K 16K zroot 0 4 0 0 0 128K Display specific fields per dnode: # **dbufstat** **-d** **-f** *pool,object,objset,dsize,cached* Display per-type statistics with comma-separated output: # **dbufstat** **-t** **-s** "" , Display extended per-buffer statistics with filters: # **dbufstat** **-bx** **-F** *dbc=1,pool=zroot* # SEE ALSO zarcstat(1), zfs(8), zpool-iostat(8) # AUTHORS This man page was written by Christos Longros <[chris.longros@gmail.com](mailto:chris.longros@gmail.com)>. Debian - March 19, 2026 RAIDZ\_TEST(1) - General Commands Manual # NAME **raidz\_test** - raidz implementation verification and benchmarking tool # SYNOPSIS **raidz\_test** \[**-StBevTD**] \[**-a** *ashift*] \[**-o** *zio\_off\_shift*] \[**-d** *raidz\_data\_disks*] \[**-s** *zio\_size\_shift*] \[**-r** *reflow\_offset*] # DESCRIPTION The purpose of this tool is to run all supported raidz implementation and verify the results of all methods. It also contains a parameter sweep option where all parameters affecting a RAID-Z block are verified (like ashift size, data offset, data size, etc.). The tool also supports a benchmarking mode using the **-B** option. # OPTION **-h** > Print a help summary. **-a** *ashift* (default: **9**) > Ashift value. **-o** *zio\_off\_shift* (default: **0**) > ZIO offset for each raidz block. > The offset's value is > *2^zio\_off\_shift*. **-d** *raidz\_data\_disks* (default: **8**) > Number of raidz data disks to use. > Additional disks will be used for parity. **-s** *zio\_size\_shift* (default: **19**) > Size of data for raidz block. > The real size is > *2^zio\_size\_shift*. **-r** *reflow\_offset* (default: **uint max**) > Set raidz expansion offset. > The expanded raidz map allocation function will > produce different map configurations depending on this value. **-S**(weep) > Sweep parameter space while verifying the raidz implementations. > This option > will exhaust all most of valid values for the > **-aods** > options. > Runtime using this option will be long. **-t**(imeout) > Wall time for sweep test in seconds. > The actual runtime could be longer. **-B**(enchmark) > All implementations are benchmarked using increasing per disk data size. > Results are given as throughput per disk, measured in MiB/s. **-e**(xpansion) > Use expanded raidz map allocation function. **-v**(erbose) > Increase verbosity. **-T**(est the test) > Debugging option: fail all tests. > This is to check if tests would properly verify bit-exactness. **-D**(ebug) > Debugging option: attach > gdb(1) > when > **SIGSEGV** > or > **SIGABRT** > are received. # SEE ALSO ztest(1) Debian - May 26, 2021 RUN(1) - General Commands Manual # NAME **run** - find, execute, and log the results of tests # SYNOPSIS **run** \[**-dgq**] \[**-o** *outputdir*] \[**-pP** *script*] \[**-t** **-seconds**] \[**-uxX** *username*] *pathname*… **run** **-w** *runfile* \[**-gq**] \[**-o** *outputdir*] \[**-pP** *script*] \[**-t** **-seconds**] \[**-uxX** *username*] *pathname*… **run** **-c** *runfile* \[**-dq**] **run** \[**-h**] # DESCRIPTION **run** command has three basic modes of operation. With neither **-c** nor **-w**, **run** processes the arguments provided on the command line, adding them to the list for this run. If a specified *pathname* is an executable file, it is added as a test. If a specified *pathname* is a directory, the behavior depends upon the presence of **-g**. If **-g** is specified, the directory is treated as a test group. See the section on **Test Groups** below. Without **-g**, **run** simply descends into the directory looking for executable files. The tests are then executed, and the results are logged. With **-w**, **run** finds tests in the manner described above. Rather than executing the tests and logging the results, the test configuration is stored in a *runfile*, which can be used in future invocations, or edited to modify which tests are executed and which options are applied. Options included on the command line with **-w** become defaults in the *runfile*. With **-c**, **run** parses a *runfile*, which can specify a series of tests and test groups to be executed. The tests are then executed, and the results are logged. ## Test Groups A test group is comprised of a set of executable files, all of which exist in one directory. The options specified on the command line or in a *runfile* apply to individual tests in the group. The exception is options pertaining to pre and post scripts, which act on all tests as a group. Rather than running before and after each test, these scripts are run only once each at the start and end of the test group. ## Test Execution The specified tests run serially, and are typically assigned results according to exit values. Tests that exit zero and non-zero are marked **PASS** and **FAIL**, respectively. When a pre script fails for a test group, only the post script is executed, and the remaining tests are marked **SKIPPED**. Any test that exceeds its *timeout* is terminated, and marked **KILLED**. By default, tests are executed with the credentials of the **run** script. Executing tests with other credentials is done via sudo(1m), which must be configured to allow execution without prompting for a password. Environment variables from the calling shell are available to individual tests. During test execution, the working directory is changed to *outputdir*. ## Output Logging By default, **run** will print one line on standard output at the conclusion of each test indicating the test name, result and elapsed time. Additionally, for each invocation of **run**, a directory is created using the ISO 8601 date format. Within this directory is a file named **log** containing all the test output with timestamps, and a directory for each test. Within the test directories, there is one file each for standard output, standard error and merged output. The default location for the *outputdir* is */var/tmp/test\_results*. ## Runfiles The *runfile* is an INI-style configuration file that describes a test run. The file has one section named **DEFAULT**, which contains configuration option names and their values in **name** = *value* format. The values in this section apply to all the subsequent sections, unless they are also specified there, in which case the default is overridden. The remaining section names are the absolute pathnames of files and directories, describing tests and test groups respectively. The legal option names are: **outputdir** = *pathname* > The name of the directory that holds test logs. **pre** = *script* > Run > *script* > prior to the test or test group. **pre\_user** = *username* > Execute the pre script as > *username*. **post** = *script* > Run > *script* > after the test or test group. **post\_user** = *username* > Execute the post script as > *username*. **quiet** = **True**|**False** > If > **True**, > only the results summary is printed to standard out. **tests** = \[*'filename'*, …] > Specify a list of > *filenames* > for this test group. > Only the basename of the absolute path is required. > This option is only valid for test groups, and each > *filename* > must be single quoted. **timeout** = *n* > A timeout value of > *n* > seconds. **user** = *username* > Execute the test or test group as > *username*. # OPTIONS **-c** *runfile* > Specify a > *runfile* > to be consumed by the run command. **-d** > Dry run mode. > Execute no tests, but print a description of each test that would have been run. **-m** > Enable kmemleak reporting (Linux only) **-g** > Create test groups from any directories found while searching for tests. **-o** *outputdir* > Specify the directory in which to write test results. **-p** *script* > Run > *script* > prior to any test or test group. **-P** *script* > Run > *script* > after any test or test group. **-q** > Print only the results summary to the standard output. **-s** *script* > Run > *script* > as a failsafe after any test is killed. **-S** *username* > Execute the failsafe script as > *username*. **-t** *n* > Specify a timeout value of > *n* > seconds per test. **-u** *username* > Execute tests or test groups as > *username*. **-w** *runfile* > Specify the name of the > *runfile* > to create. **-x** *username* > Execute the pre script as > *username*. **-X** *username* > Execute the post script as > *username*. # EXAMPLES **Example 1**: Running ad-hoc tests. > This example demonstrates the simplest invocation of > **run**. > > % **run** *my-tests* > > Test: /home/jkennedy/my-tests/test-01 \[00:02] \[PASS] > > Test: /home/jkennedy/my-tests/test-02 \[00:04] \[PASS] > > Test: /home/jkennedy/my-tests/test-03 \[00:01] \[PASS] > > > > Results Summary > > PASS 3 > > > > Running Time: 00:00:07 > > Percent passed: 100.0% > > Log directory: /var/tmp/test\_results/20120923T180654 **Example 2**: Creating a *runfile* for future use. > This example demonstrates creating a > *runfile* > with non-default options. > > % **run** **-p** *setup* **-x** *root* **-g** **-w** *new-tests.run* *new-tests* > > % **cat** *new-tests.run* > > \[DEFAULT] > > pre = setup > > post\_user = > > quiet = False > > user = > > timeout = 60 > > post = > > pre\_user = root > > outputdir = /var/tmp/test\_results > > > > \[/home/jkennedy/new-tests] > > tests = \['test-01', 'test-02', 'test-03'] # SEE ALSO sudo(1m) Debian - May 26, 2021 ZARCSTAT(1) - General Commands Manual # NAME **zarcstat** - report ZFS ARC and L2ARC statistics # SYNOPSIS **zarcstat** \[**-havxp**] \[**-f** *field*\[,*field*…]] \[**-o** *file*] \[**-s** *string*] \[*interval*] \[*count*] # DESCRIPTION **zarcstat** prints various ZFS ARC and L2ARC statistics in vmstat-like fashion: **c** > ARC target size **dh%** > Demand hit percentage **di%** > Demand I/O hit percentage **dm%** > Demand miss percentage **ddh%** > Demand data hit percentage **ddi%** > Demand data I/O hit percentage **ddm%** > Demand data miss percentage **dmh%** > Demand metadata hit percentage **dmi%** > Demand metadata I/O hit percentage **dmm%** > Demand metadata miss percentage **mfu** > MFU list hits per second **mh%** > Metadata hit percentage **mi%** > Metadata I/O hit percentage **mm%** > Metadata miss percentage **mru** > MRU list hits per second **ph%** > Prefetch hits percentage **pi%** > Prefetch I/O hits percentage **pm%** > Prefetch miss percentage **pdh%** > Prefetch data hits percentage **pdi%** > Prefetch data I/O hits percentage **pdm%** > Prefetch data miss percentage **pmh%** > Prefetch metadata hits percentage **pmi%** > Prefetch metadata I/O hits percentage **pmm%** > Prefetch metadata miss percentage **dhit** > Demand hits per second **dioh** > Demand I/O hits per second **dmis** > Demand misses per second **ddhit** > Demand data hits per second **ddioh** > Demand data I/O hits per second **ddmis** > Demand data misses per second **dmhit** > Demand metadata hits per second **dmioh** > Demand metadata I/O hits per second **dmmis** > Demand metadata misses per second **hit%** > ARC hit percentage **hits** > ARC hits per second **ioh%** > ARC I/O hits percentage **iohs** > ARC I/O hits per second **mfug** > MFU ghost list hits per second **mhit** > Metadata hits per second **mioh** > Metadata I/O hits per second **miss** > ARC misses per second **mmis** > Metadata misses per second **mrug** > MRU ghost list hits per second **phit** > Prefetch hits per second **pioh** > Prefetch I/O hits per second **pmis** > Prefetch misses per second **pdhit** > Prefetch data hits per second **pdioh** > Prefetch data I/O hits per second **pdmis** > Prefetch data misses per second **pmhit** > Prefetch metadata hits per second **pmioh** > Prefetch metadata I/O hits per second **pmmis** > Prefetch metadata misses per second **read** > Total ARC accesses per second **time** > Current time **size** > ARC size **arcsz** > Alias for > **size** **unc** > Uncached list hits per second **dread** > Demand accesses per second **ddread** > Demand data accesses per second **dmread** > Demand metadata accesses per second **eskip** > evict\_skip per second **miss%** > ARC miss percentage **mread** > Metadata accesses per second **pread** > Prefetch accesses per second **pdread** > Prefetch data accesses per second **pmread** > Prefetch metadata accesses per second **l2hit%** > L2ARC access hit percentage **l2hits** > L2ARC hits per second **l2miss** > L2ARC misses per second **l2read** > Total L2ARC accesses per second **l2pref** > L2ARC prefetch allocated size per second **l2pref%** > L2ARC prefetch allocated size percentage **l2mfu** > L2ARC MFU allocated size per second **l2mfu%** > L2ARC MFU allocated size percentage **l2mru** > L2ARC MRU allocated size per second **l2mru%** > L2ARC MRU allocated size percentage **l2data** > L2ARC data (buf content) allocated size per second **l2data%** > L2ARC data (buf content) allocated size percentage **l2meta** > L2ARC metadata (buf content) allocated size per second **l2meta%** > L2ARC metadata (buf content) allocated size percentage **l2size** > Size of the L2ARC **mtxmis** > mutex\_miss per second **l2bytes** > Bytes read per second from the L2ARC **l2wbytes** > Bytes written per second to the L2ARC **l2miss%** > L2ARC access miss percentage **l2asize** > Actual (compressed) size of the L2ARC **cmpsz** > Compressed size **cmpsz%** > Compressed size percentage **ovhsz** > Overhead size **ovhsz%** > Overhead size percentage **bonsz** > Bonus size **bonsz%** > Bonus size percentage **dnosz** > Dnode size **dnosz%** > Dnode size percentage **dbusz** > Dbuf size **dbusz%** > Dbuf size percentage **hdrsz** > Header size **hdrsz%** > Header size percentage **l2hsz** > L2 header size **l2hsz%** > L2 header size percentage **abdsz** > ABD chunk waste size **abdsz%** > ABD chunk waste size percentage **datatg** > ARC data target **datatg%** > ARC data target percentage **datasz** > ARC data size **datasz%** > ARC data size percentage **metatg** > ARC metadata target **metatg%** > ARC metadata target percentage **metasz** > ARC metadata size **metasz%** > ARC metadata size percentage **anosz** > Anonymous size **anosz%** > Anonymous size percentage **anoda** > Anonymous data size **anoda%** > Anonymous data size percentage **anome** > Anonymous metadata size **anome%** > Anonymous metadata size percentage **anoed** > Anonymous evictable data size **anoed%** > Anonymous evictable data size percentage **anoem** > Anonymous evictable metadata size **anoem%** > Anonymous evictable metadata size percentage **mfutg** > MFU target **mfutg%** > MFU target percentage **mfudt** > MFU data target **mfudt%** > MFU data target percentage **mfumt** > MFU metadata target **mfumt%** > MFU metadata target percentage **mfusz** > MFU size **mfusz%** > MFU size percentage **mfuda** > MFU data size **mfuda%** > MFU data size percentage **mfume** > MFU metadata size **mfume%** > MFU metadata size percentage **mfued** > MFU evictable data size **mfued%** > MFU evictable data size percentage **mfuem** > MFU evictable metadata size **mfuem%** > MFU evictable metadata size percentage **mfugsz** > MFU ghost size **mfugd** > MFU ghost data size **mfugm** > MFU ghost metadata size **mrutg** > MRU target **mrutg%** > MRU target percentage **mrudt** > MRU data target **mrudt%** > MRU data target percentage **mrumt** > MRU metadata target **mrumt%** > MRU metadata target percentage **mrusz** > MRU size **mrusz%** > MRU size percentage **mruda** > MRU data size **mruda%** > MRU data size percentage **mrume** > MRU metadata size **mrume%** > MRU metadata size percentage **mrued** > MRU evictable data size **mrued%** > MRU evictable data size percentage **mruem** > MRU evictable metadata size **mruem%** > MRU evictable metadata size percentage **mrugsz** > MRU ghost size **mrugd** > MRU ghost data size **mrugm** > MRU ghost metadata size **uncsz** > Uncached size **uncsz%** > Uncached size percentage **uncda** > Uncached data size **uncda%** > Uncached data size percentage **uncme** > Uncached metadata size **uncme%** > Uncached metadata size percentage **unced** > Uncached evictable data size **unced%** > Uncached evictable data size percentage **uncem** > Uncached evictable metadata size **uncem%** > Uncached evictable metadata size percentage **grow** > ARC grow disabled **need** > ARC reclaim needed **free** > The ARC's idea of how much free memory there is, which includes evictable memory > in the page cache. > Since the ARC tries to keep > **avail** > above zero, > **avail** > is usually more instructive to observe than > **free**. **avail** > The ARC's idea of how much free memory is available to it, which is a bit less > than > **free**. > May temporarily be negative, in which case the ARC will reduce the target size > **c**. # OPTIONS **-a** > Print all possible stats. **-f** > Display only specific fields. > See > *DESCRIPTION* > for supported statistics. **-h** > Display help message. **-o** > Report statistics to a file instead of the standard output. **-p** > Disable auto-scaling of numerical fields (for raw, machine-parsable values). **-s** > Display data with a specified separator (default: 2 spaces). **-x** > Print extended stats > (same as **-f** **time**,**mfu**,**mru**,**mfug**,**mrug**,**eskip**,**mtxmis**,**dread**,**pread**,**read**). **-v** > Show field headers and definitions # OPERANDS The following operands are supported: *interval* > Specify the sampling interval in seconds. *count* > Display only > *count* > reports. Debian - September 19, 2024 ZARCSUMMARY(1) - General Commands Manual # NAME **zarcsummary** - print ZFS ARC and subsystem statistics # SYNOPSIS **zarcsummary** \[**-adghr**] \[**-p** *page*] \[**-s** *section*] # DESCRIPTION **zarcsummary** collects statistics from the running kernel and prints a human-readable report on the state of the ZFS Adaptive Replacement Cache (ARC) and several related subsystems. The output is a snapshot; for periodic sampling, use zarcstat(1) instead. When invoked without arguments, **zarcsummary** prints all sections in sequence. A single section may be selected with the **-s** option. The report is organized into the following sections: **arc** > Current and target ARC sizes, the split between the Most Recently Used > (MRU) > and Most Frequently Used > (MFU) > lists, overall hit and miss ratios, compression savings, and whether the > ARC is experiencing memory pressure. **archits** > Breakdown of ARC hits and misses by demand reads versus prefetches > and data versus metadata. **dmu** > Data Management Unit predictive prefetcher statistics, including stream > hits, misses, and I/O issued ahead of and behind the current stream. **l2arc** > Level-2 ARC statistics for any cache devices attached to imported pools. > This section only appears when an L2ARC device is present and covers > bytes read and written, hit rates, and eviction activity. **spl** > Solaris Porting Layer memory allocator statistics. > This section is only available on Linux. **tunables** > Current values of kernel tunables that govern ARC behavior, including > *zfs\_arc\_max* > and > *zfs\_arc\_min*. > Combining this section with > **-d** > adds a brief explanation of each tunable. **zil** > ZFS Intent Log statistics: committed transaction counts, flush requests > to stable storage, and the byte-level split between normal vdevs and > any separate log > (SLOG) > device. # OPTIONS **-a**, **--alternate** > Use an alternate layout for the tunables and SPL sections. **-d**, **--description** > Print a short description next to each tunable and SPL entry. **-h**, **--help** > Display a help message. **-g**, **--graph** > Print an ASCII graph of current ARC use and exit. **-p** *page*, **--page** *page* > Print a section by its legacy page number. > **Deprecated**; > use > **-s** > instead. **-r**, **--raw** > Dump all available data with minimal formatting. **-s** *section*, **--section** *section* > Print only the named section. > See > *DESCRIPTION* > for the list of valid section names. # ENVIRONMENT **Linux** > Statistics are read from > */proc/spl/kstat/zfs/*. FreeBSD > Statistics are read via > sysctl(3) > from the > *vfs.zfs* > and > *kstat.zfs.misc* > trees. > The > *py-sysctl* > package must be installed for > **zarcsummary** > to function. # EXIT STATUS The **zarcsummary** utility exits 0 on success, and >0 if an error occurs. # EXAMPLES Print the full report: # zarcsummary Display only ARC size and hit rate information: # zarcsummary -s arc Show a visual overview of ARC health: # zarcsummary -g Search for all statistics containing "hit": # zarcsummary -r | grep hit Review all tunables with their descriptions: # zarcsummary -d -s tunables # SEE ALSO zarcstat(1), zpool-status(8) # HISTORY **zarcsummary** originated as a Perl script called **arc\_summary**, written by Ben Rockwood in 2008 for Solaris. Martin Matuska ported it to FreeBSD in 2010, and Jason J. Hellenthal took over maintenance. Scot W. Stevenson contributed a full rewrite in Python 3 in 2017, introducing the **-g**, **-r**, and **-s** options along with the section-based output layout. The tool was renamed from **arc\_summary** to **zarcsummary** in 2025 to align with the OpenZFS command naming convention. # AUTHORS This manual page was written by Christos Longros <[chris.longros@gmail.com](mailto:chris.longros@gmail.com)>. Debian - March 14, 2026 ZHACK(1) - General Commands Manual # NAME **zhack** - libzpool debugging tool # DESCRIPTION This utility pokes configuration changes directly into a ZFS pool, which is dangerous and can cause data corruption. # SYNOPSIS **zhack** **feature stat** *pool* > List feature flags. **zhack** **feature enable** \[**-d** *description*] \[**-r**] *pool* *guid* > Add a new feature to > *pool* > that is uniquely identified by > *guid*, > which is specified in the same form as a > zfs(8) > user property. > The > *description* > is a short human readable explanation of the new feature. > The > **-r** > flag indicates that > *pool* > can be safely opened in read-only mode by a system that does not understand the > *guid* > feature. **zhack** **feature ref** \[**-d**|**-m**] *pool* *guid* > Increment the reference count of the > *guid* > feature in > *pool*. > The > **-d** > flag decrements the reference count of the > *guid* > feature in > *pool* > instead. > The > **-m** > flag indicates that the > *guid* > feature is now required to read the pool MOS. **zhack** **label repair** \[**-cu**] *device* > Repair labels of a specified > *device* > according to options. > Flags may be combined to do their functions simultaneously. > The > **-c** > flag repairs corrupted label checksums > The > **-u** > flag restores the label on a detached device > Example: > **zhack **label repair** **-cu** *device*‌** > Fix checksums and undetach a device **zhack** **mmp reclaim** *pool* > Import a > *pool* > whose uberblock claim cannot reach every mirror leg the configuration still > expects to be present, then mark the unreachable leaves offline so that > ordinary imports succeed, and export it again. > This exists for one situation: a host has failed together with the mirror legs > attached to it, so the surviving labels still describe those legs as healthy, > and every later import demands writes to them that nobody can make. > **This does not make the operation safe on its own**. > A live host whose legs are > *all* > invisible from here cannot be detected by any write and read scheme, and this > command will take the pool from it. > Fence the failed host, or otherwise prove it is powered off, before running > this. > The activity check itself is unchanged: every write the reachable legs allow, > the wait, and the re-read all still happen, so a competing host which shares > any leg with this one is still detected and the import is refused. > Only the number of writes the claim demands is lowered, and only for mirror > legs this host cannot open, each of which is then marked offline. > The pool is left exported, and imports degraded with those leaves offline. > Bring each one back with > zpool-online(8) > once the hardware returns. > A leaf holding the only copy of some data cannot be offlined: it is reported > and left online, and imports from other hosts stay refused. > No leaf is marked offline on a pool which does not have > **multihost** > set, since no claim runs for it and an ordinary import already succeeds. > Leaves of log, cache, and spare vdevs are not touched, as the claim does not > write to them. **zhack** **metaslab leak** \[**-f**] *pool* > Apply a fragmentation profile generated by > **zdb** > to the specified > *pool*. > The > **-f** > flag forces the profile to apply even if the vdevs in the > *pool* > don't have the same number of metaslabs as the fragmentation profile. # GLOBAL OPTIONS The following can be passed to all **zhack** invocations before any subcommand: **-c** *cachefile* > Read > *pool* > configuration from the > *cachefile*, > which is > */etc/zfs/zpool.cache* > by default. **-d** *dir* > Search for > *pool* > members in > *dir*. > Can be specified more than once. **-o** *var*=*value* > Set the given tunable to the provided value. # EXAMPLES # **zhack** **feature stat** *tank* for_read_obj: org.illumos:lz4_compress = 0 for_write_obj: com.delphix:async_destroy = 0 com.delphix:empty_bpobj = 0 descriptions_obj: com.delphix:async_destroy = Destroy filesystems asynchronously. com.delphix:empty_bpobj = Snapshots use less space. org.illumos:lz4_compress = LZ4 compression algorithm support. # **zhack** **feature enable** **-d** 'Predict future disk failures.' *tank com.example:clairvoyance* # **zhack** **feature ref** *tank com.example:clairvoyance* Recover a pool whose peer host died together with its mirror legs, after confirming that host is down: # **zpool** **import** **-f** *tank* cannot import 'tank': the multihost claim could not be written to a device the pool configuration expects to be present. If the device is permanently gone, recover with 'zhack mmp reclaim'. # **zhack** **mmp reclaim** *tank* tank: marked absent leaf 5646371977210937898 offline # **zpool** **import** *tank* # **zpool** **online** *tank* */dev/disk/by-id/...* # SEE ALSO ztest(1), zpool-features(7), zpoolprops(7), zfs(8), zpool-import(8), zpool-online(8) Debian - August 3, 2026 ZILSTAT(1) - General Commands Manual # NAME **zilstat** - report ZFS Intent Log statistics # SYNOPSIS **zilstat** \[**-v**] \[**-a**|**-p** *pool*|**-d** *dataset*\[,*dataset*…]] \[**-f** *field*\[,*field*…]] \[**-s** *separator*] \[**-i** *interval*] # DESCRIPTION **zilstat** prints statistics about the ZFS Intent Log (ZIL). The ZIL is used to log synchronous write operations and is replayed on pool import if the system crashes before those writes are committed to the main pool. By default, global ZIL statistics are displayed. When a pool or dataset is specified, per-dataset statistics are shown instead. If an *interval* is specified, the output is repeated every *interval* seconds showing the change in values since the previous sample. Without an interval, cumulative values are displayed. The following fields are available: **time** > Current time **pool** > Pool name **ds** > Dataset name **obj** > Objset ID **cc** > Commit count **cwc** > Commit writer count **cec** > Commit error count **csc** > Commit stall count **cSc** > Commit suspend count **cCc** > Commit crash count **ic** > In-transaction count **iic** > Indirect in-transaction count **iib** > Indirect in-transaction bytes **icc** > Copied in-transaction count **icb** > Copied in-transaction bytes **inc** > Needcopy in-transaction count **inb** > Needcopy in-transaction bytes **idc** > Direct (copied + needcopy) count **idb** > Direct (copied + needcopy) bytes **iwc** > Total write (indirect + direct) count **iwb** > Total write (indirect + direct) bytes **imnc** > Normal metaslab count **imnb** > Normal metaslab bytes **imnw** > Normal metaslab write bytes **imna** > Normal metaslab alloc bytes **imsc** > SLOG metaslab count **imsb** > SLOG metaslab bytes **imsw** > SLOG metaslab write bytes **imsa** > SLOG metaslab alloc bytes **imc** > Total metaslab (normal + SLOG) count **imb** > Total metaslab (normal + SLOG) bytes **imw** > Total metaslab (normal + SLOG) write bytes **ima** > Total metaslab (normal + SLOG) alloc bytes **se%** > Space efficiency percentage (bytes / alloc) **sen%** > Normal space efficiency percentage **ses%** > SLOG space efficiency percentage **te%** > Total efficiency percentage (bytes / write) **ten%** > Normal total efficiency percentage **tes%** > SLOG total efficiency percentage # OPTIONS **-h** > Display a help message. **-a** > Print statistics for all datasets across all pools. **-d** *dataset* > Print statistics for the specified dataset(s). > Multiple datasets may be given as a comma-separated list. **-f** *field* > Display only the specified fields. > Multiple fields may be given as a comma-separated list. > See > *DESCRIPTION* > for available fields. **-i** *interval* > Print statistics every > *interval* > seconds. > Values are reported as rates per second. **-p** *pool* > Print statistics for all datasets in the specified pool. **-s** *separator* > Override the default field separator > (two spaces) > with a custom string. **-v** > List all available field headers and their definitions. # ENVIRONMENT **Linux** > Statistics are read from > */proc/spl/kstat/zfs/zil* > for global stats and from per-pool objset files under > */proc/spl/kstat/zfs/*‌*pool*‌*/*. **FreeBSD** > Statistics are read via > sysctl(8) > from the > **kstat.zfs** > tree. # SEE ALSO zarcstat(1), zpool-status(8) # AUTHORS This manual page was written by Christos Longros <[chris.longros@gmail.com](mailto:chris.longros@gmail.com)>. Debian - March 9, 2026 ZTEST(1) - General Commands Manual # NAME **ztest** - was written by the ZFS Developers as a ZFS unit test # SYNOPSIS **ztest** \[**-VEG**] \[**-v** *vdevs*] \[**-s** *size\_of\_each\_vdev*] \[**-a** *alignment\_shift*] \[**-m** *mirror\_copies*] \[**-r** *raidz\_disks/draid\_disks*] \[**-R** *raid\_parity*] \[**-K** *raid\_kind*] \[**-D** *draid\_data*] \[**-S** *draid\_spares*] \[**-C** *vdev\_class\_state*] \[**-d** *datasets*] \[**-t** *threads*] \[**-g** *gang\_block\_threshold*] \[**-i** *initialize\_pool\_i\_times*] \[**-k** *kill\_percentage*] \[**-p** *pool\_name*] \[**-T** *time*] \[**-z** *zil\_failure\_rate*] **ztest** **-X** \[**-VG**] \[**-s** *size\_of\_each\_vdev*] \[**-a** *alignment\_shift*] \[**-r** *raidz\_disks*] \[**-R** *raid\_parity*] \[**-d** *datasets*] \[**-t** *threads*] # DESCRIPTION **ztest** was written by the ZFS Developers as a ZFS unit test. The tool was developed in tandem with the ZFS functionality and was executed nightly as one of the many regression tests against the daily build. As features were added to ZFS, unit tests were also added to **ztest**. In addition, a separate test development team wrote and executed more functional and stress tests. By default **ztest** runs for ten minutes and uses block files (stored in */tmp*) to create pools rather than using physical disks. Block files afford **ztest** its flexibility to play around with zpool components without requiring large hardware configurations. However, storing the block files in */tmp* may not work for you if you have a small tmp directory. By default is non-verbose. This is why entering the command above will result in **ztest** quietly executing for 5 minutes. The **-V** option can be used to increase the verbosity of the tool. Adding multiple **-V** options is allowed and the more you add the more chatty **ztest** becomes. After the **ztest** run completes, you should notice many *ztest.\*‌* files lying around. Once the run completes you can safely remove these files. Note that you shouldn't remove these files during a run. You can re-use these files in your next **ztest** run by using the **-E** option. # OPTIONS **-h**, **-?**, **--help** > Print a help summary. **-v**, **--vdevs**= (default: **5**) > Number of vdevs. **-s**, **--vdev-size**= (default: **64M**) > Size of each vdev. **-a**, **--alignment-shift**= (default: **9**) (use **0** for random) > Alignment shift used in test. **-m**, **--mirror-copies**= (default: **2**) > Number of mirror copies. **-r**, **--raid-disks**= (default: **4** for raidz/**16** for draid) > Number of raidz/draid disks. **-R**, **--raid-parity**= (default: **1**) > Raid parity (raidz & draid). **-K**, **--raid-kind**=**raidz**|**eraidz**|**draid**|**random** (default: **random**) > The kind of RAID config to use. > With > **random** > the kind alternates between raidz, eraidz (expandable raidz) and draid. **-D**, **--draid-data**= (default: **4**) > Number of data disks in a dRAID redundancy group. **-S**, **--draid-spares**= (default: **1**) > Number of dRAID distributed spare disks. **-d**, **--datasets**= (default: **7**) > Number of datasets. **-t**, **--threads**= (default: **23**) > Number of threads. **-g**, **--gang-block-threshold**= (default: **32K**) > Gang block threshold. **-i**, **--init-count**= (default: **1**) > Number of pool initializations. **-k**, **--kill-percentage**= (default: **70%**) > Kill percentage. **-p**, **--pool-name**= (default: **ztest**) > Pool name. **-f**, **--vdev-file-directory**= (default: */tmp*) > File directory for vdev files. **-M**, **--multi-host** > Multi-host; simulate pool imported on remote host. **-E**, **--use-existing-pool** > Use existing pool (use existing pool instead of creating new one). **-T**, **--run-time**= (default: **300**s) > Total test run time. **-P**, **--pass-time**= (default: **60**s) > Time per pass. **-F**, **--freeze-loops**= (default: **50**) > Max loops in > **spa\_freeze**(). **-B**, **--alt-ztest**= > Path to alternate ("older") > **ztest** > to drive, which will be used to initialize the pool, and, a stochastic half the > time, to run the tests. > The parallel > *lib* > directory is prepended to > `LD_LIBRARY_PATH`; > i.e. given > **-B** *./chroots/lenny/usr/bin/*‌**ztest**, > *./chroots/lenny/usr/lib* > will be loaded. **-C**, **--vdev-class-state**=**on**|**off**|**random** (default: **random**) > The vdev allocation class state. **-o**, **--option**=*var*=*value*… > Set the given tunable to the provided value. **-G**, **--dump-debug** > Dump zfs\_dbgmsg buffer before exiting due to an error. **-V**, **--verbose** > Verbose (use multiple times for ever more verbosity). **-X**, **--raidz-expansion** > Perform a dedicated raidz expansion test. # EXAMPLES To override */tmp* as your location for block files, you can use the **-f** option: # ztest -f / To get an idea of what **ztest** is actually testing try this: # ztest -f / -VVV Maybe you'd like to run **ztest** for longer? To do so simply use the **-T** option and specify the runlength in seconds like so: # ztest -f / -V -T 120 # ENVIRONMENT VARIABLES `ZFS_HOSTID`=*id* > Use > *id* > instead of the SPL hostid to identify this host. > Intended for use with > **ztest**, but this environment variable will affect any utility which uses > libzpool, including > zpool(8). > Since the kernel is unaware of this setting, > results with utilities other than ztest are undefined. `ZFS_STACK_SIZE`=*stacksize* > Limit the default stack size to > *stacksize* > bytes for the purpose of > detecting and debugging kernel stack overflows. > This value defaults to > *32K* > which is double the default > *16K* > Linux kernel stack size. > In practice, setting the stack size slightly higher is needed because > differences in stack usage between kernel and user space can lead to spurious > stack overflows (especially when debugging is enabled). > The specified value > will be rounded up to a floor of PTHREAD\_STACK\_MIN which is the minimum stack > required for a NULL procedure in user space. > By default the stack size is limited to > *256K*. # SEE ALSO zdb(1), zfs(1), zpool(1), spl(4) Debian - July 12, 2025 ZVOL\_WAIT(1) - General Commands Manual # NAME **zvol\_wait** - wait for ZFS volume links to appear in /dev # SYNOPSIS **zvol\_wait** # DESCRIPTION When a ZFS pool is imported, the volumes within it will appear as block devices. As they're registered, udev(7) asynchronously creates symlinks under */dev/zvol* using the volumes' names. **zvol\_wait** will wait for all those symlinks to be created before exiting. # SEE ALSO udev(7) Debian - May 27, 2021 SPL(4) - Device Drivers Manual # NAME **spl** - parameters of the SPL kernel module # DESCRIPTION **spl\_kmem\_cache\_kmem\_threads**=**4** (uint) > The number of threads created for the spl\_kmem\_cache task queue. > This task queue is responsible for allocating new slabs > for use by the kmem caches. > For the majority of systems and workloads only a small number of threads are > required. **spl\_kmem\_cache\_obj\_per\_slab**=**8** (uint) > The preferred number of objects per slab in the cache. > In general, a larger value will increase the caches memory footprint > while decreasing the time required to perform an allocation. > Conversely, a smaller value will minimize the footprint > and improve cache reclaim time but individual allocations may take longer. **spl\_kmem\_cache\_max\_size**=**32** (64-bit) or **4** (32-bit) (uint) > The maximum size of a kmem cache slab in MiB. > This effectively limits the maximum cache object size to > **spl\_kmem\_cache\_max\_size**/**spl\_kmem\_cache\_obj\_per\_slab**. > Caches may not be created with > object sized larger than this limit. **spl\_kmem\_cache\_slab\_limit**=**16384** (uint) > For small objects the Linux slab allocator should be used to make the most > efficient use of the memory. > However, large objects are not supported by > the Linux slab and therefore the SPL implementation is preferred. > This value is used to determine the cutoff between a small and large object. > Objects of size > **spl\_kmem\_cache\_slab\_limit** > or smaller will be allocated using the Linux slab allocator, > large objects use the SPL allocator. > A cutoff of 16K was determined to be optimal for architectures using 4K pages. **spl\_kmem\_alloc\_warn**=**32768** (uint) > As a general rule > **kmem\_alloc**() > allocations should be small, > preferably just a few pages, since they must by physically contiguous. > Therefore, a rate limited warning will be printed to the console for any > **kmem\_alloc**() > which exceeds a reasonable threshold. > The default warning threshold is set to eight pages but capped at 32K to > accommodate systems using large pages. > This value was selected to be small enough to ensure > the largest allocations are quickly noticed and fixed. > But large enough to avoid logging any warnings when a allocation size is > larger than optimal but not a serious concern. > Since this value is tunable, developers are encouraged to set it lower > when testing so any new largish allocations are quickly caught. > These warnings may be disabled by setting the threshold to zero. **spl\_kmem\_alloc\_max**=**KMALLOC\_MAX\_SIZE**/**4** (uint) > Large > **kmem\_alloc**() > allocations will fail if they exceed > **KMALLOC\_MAX\_SIZE**. > Allocations which are marginally smaller than this limit may succeed but > should still be avoided due to the expense of locating a contiguous range > of free pages. > Therefore, a maximum kmem size with reasonable safely margin of 4x is set. > **kmem\_alloc**() > allocations larger than this maximum will quickly fail. > **vmem\_alloc**() > allocations less than or equal to this value will use > **kmalloc**(), > but shift to > **vmalloc**() > when exceeding this value. **spl\_kmem\_cache\_magazine\_size**=**0** (uint) > Cache magazines are an optimization designed to minimize the cost of > allocating memory. > They do this by keeping a per-CPU cache of recently > freed objects, which can then be reallocated without taking a lock. > This can improve performance on highly contended caches. > However, because objects in magazines will prevent otherwise empty slabs > from being immediately released this may not be ideal for low memory machines. > For this reason, > **spl\_kmem\_cache\_magazine\_size** > can be used to set a maximum magazine size. > When this value is set to 0 the magazine size will > be automatically determined based on the object size. > Otherwise magazines will be limited to 2-256 objects per magazine (i.e. per > CPU). Magazines may never be entirely disabled in this implementation. **spl\_hostid**=**0** (ulong) > The system hostid, when set this can be used to uniquely identify a system. > By default this value is set to zero which indicates the hostid is disabled. > It can be explicitly enabled by placing a unique non-zero value in > */etc/hostid*. **spl\_hostid\_path**=*/etc/hostid* (charp) > The expected path to locate the system hostid when specified. > This value may be overridden for non-standard configurations. **spl\_panic\_halt**=**0** (uint) > Cause a kernel panic on assertion failures. > When not enabled, the thread is halted to facilitate further debugging. > Set to a non-zero value to enable. **spl\_schedule\_hrtimeout\_slack\_us**=**0** (uint) > Slack value in microseconds passed to > **schedule\_hrtimeout\_range**() > when a condition variable times out. > A non-zero value enforces the kernel coalesce the wakeup with other timers > to reduce wakeup count, at the cost of some additional sleep duration. > The maximum is > **1000**, > as defined by > **MAX\_HRTIMEOUT\_SLACK\_US**. > Linux-only. **spl\_taskq\_kick**=**0** (uint) > Kick stuck taskq to spawn threads. > When writing a non-zero value to it, it will scan all the taskqs. > If any of them have a pending task more than 5 seconds old, > it will kick it to spawn more threads. > This can be used if you find a rare > deadlock occurs because one or more taskqs didn't spawn a thread when it should. **spl\_taskq\_thread\_bind**=**0** (int) > Bind taskq threads to specific CPUs. > When enabled all taskq threads will be distributed evenly > across the available CPUs. > By default, this behavior is disabled to allow the Linux scheduler > the maximum flexibility to determine where a thread should run. **spl\_taskq\_thread\_dynamic**=**1** (int) > Allow dynamic taskqs. > When enabled taskqs which set the > **TASKQ\_DYNAMIC** > flag will by default create only a single thread. > New threads will be created on demand up to a maximum allowed number > to facilitate the completion of outstanding tasks. > Threads which are no longer needed will be promptly destroyed. > By default this behavior is enabled but it can be disabled to > aid performance analysis or troubleshooting. **spl\_taskq\_thread\_priority**=**1** (int) > Allow newly created taskq threads to set a non-default scheduler priority. > When enabled, the priority specified when a taskq is created will be applied > to all threads created by that taskq. > When disabled all threads will use the default Linux kernel thread priority. > By default, this behavior is enabled. **spl\_taskq\_thread\_sequential**=**4** (int) > The number of items a taskq worker thread must handle without interruption > before requesting a new worker thread be spawned. > This is used to control > how quickly taskqs ramp up the number of threads processing the queue. > Because Linux thread creation and destruction are relatively inexpensive a > small default value has been selected. > This means that normally threads will be created aggressively which is > desirable. > Increasing this value will > result in a slower thread creation rate which may be preferable for some > configurations. **spl\_taskq\_thread\_timeout\_ms**=**5000** (uint) > Minimum idle threads exit interval for dynamic taskqs. > Smaller values allow idle threads exit more often and potentially be > respawned again on demand, causing more churn. Debian - May 7, 2025 .\" SPDX-License-Identifier: CDDL-1.0 .\" .\" This file and its contents are supplied under the terms of the .\" Common Development and Distribution License ("CDDL"), version 1.0. .\" You may only use this file in accordance with the terms of version .\" 1.0 of the CDDL. .\" .\" A full copy of the text of the CDDL should have accompanied this .\" source. A copy of the CDDL is also available via the Internet at .\" https://opensource.org/license/CDDL-1.0. .\" .\" Copyright (c) 2013 by Turbo Fredriksson . All rights reserved. .\" Copyright (c) 2019, 2021 by Delphix. All rights reserved. .\" Copyright (c) 2019 Datto Inc. .\" Copyright (c) 2023, 2024, 2025, Klara, Inc. .\" Copyright (c) 2026, Mateusz Piotrowski <0mp@FreeBSD.org> .\" Copyright (c) 2026, TrueNAS. .\" .Dd July 12, 2026 .Dt ZFS 4 .Os . .Sh NAME .Nm zfs .Nd tuning of the ZFS kernel module . .Sh DESCRIPTION The ZFS module supports these parameters: .Bl -tag -width Ds .It Sy dbuf_cache_max_bytes Ns = Ns Sy UINT64_MAX Ns B Pq u64 Maximum size in bytes of the dbuf cache. The target size is determined by the MIN versus .No 1/2^ Ns Sy dbuf_cache_shift Pq 1/32nd of the target ARC size. The behavior of the dbuf cache and its associated settings can be observed via the .Pa /proc/spl/kstat/zfs/dbufstats kstat. . .It Sy dbuf_metadata_cache_max_bytes Ns = Ns Sy UINT64_MAX Ns B Pq u64 Maximum size in bytes of the metadata dbuf cache. The target size is determined by the MIN versus .No 1/2^ Ns Sy dbuf_metadata_cache_shift Pq 1/64th of the target ARC size. The behavior of the metadata dbuf cache and its associated settings can be observed via the .Pa /proc/spl/kstat/zfs/dbufstats kstat. . .It Sy dbuf_cache_hiwater_pct Ns = Ns Sy 10 Ns % Pq uint The percentage over .Sy dbuf_cache_max_bytes when dbufs must be evicted directly. . .It Sy dbuf_cache_lowater_pct Ns = Ns Sy 10 Ns % Pq uint The percentage below .Sy dbuf_cache_max_bytes when the evict thread stops evicting dbufs. . .It Sy dbuf_cache_shift Ns = Ns Sy 5 Pq uint Set the size of the dbuf cache .Pq Sy dbuf_cache_max_bytes to a log2 fraction of the target ARC size. . .It Sy dbuf_metadata_cache_shift Ns = Ns Sy 6 Pq uint Set the size of the dbuf metadata cache .Pq Sy dbuf_metadata_cache_max_bytes to a log2 fraction of the target ARC size. . .It Sy dbuf_mutex_cache_shift Ns = Ns Sy 0 Pq uint Set the size of the mutex array for the dbuf cache. When set to .Sy 0 the array is dynamically sized based on total system memory. . .It Sy dmu_object_alloc_chunk_shift Ns = Ns Sy 7 Po 128 Pc Pq uint dnode slots allocated in a single operation as a power of 2. The default value minimizes lock contention for the bulk operation performed. . .It Sy dmu_ddt_copies Ns = Ns Sy 3 Pq uint Controls the number of copies stored for DeDup Table .Pq DDT objects. Reducing the number of copies to 1 from the previous default of 3 can reduce the write inflation caused by deduplication. This assumes redundancy for this data is provided by the vdev layer. If the DDT is damaged, space may be leaked .Pq not freed when the DDT can not report the correct reference count. . .It Sy dmu_prefetch_max Ns = Ns Sy 134217728 Ns B Po 128 MiB Pc Pq uint Limit the amount we can prefetch with one call to this amount in bytes. This helps to limit the amount of memory that can be used by prefetching. . .It Sy l2arc_feed_again Ns = Ns Sy 1 Ns | Ns 0 Pq int Turbo L2ARC warm-up. When the L2ARC is cold the fill interval will be set as fast as possible. . .It Sy l2arc_feed_min_ms Ns = Ns Sy 200 Pq u64 Min feed interval in milliseconds. Requires .Sy l2arc_feed_again Ns = Ns Ar 1 and only applicable in related situations. . .It Sy l2arc_feed_secs Ns = Ns Sy 1 Pq u64 Seconds between L2ARC writing. . .It Sy l2arc_headroom Ns = Ns Sy 8 Pq u64 How far through the ARC lists to search for L2ARC cacheable content per cycle, expressed as a multiplier of the effective write size. Setting to .Sy 0 disables the per-cycle headroom limit. Scan depth is also bounded by .Sy l2arc_ext_headroom_pct when persistent markers are active. . .It Sy l2arc_headroom_boost Ns = Ns Sy 200 Ns % Pq u64 Scales .Sy l2arc_headroom by this percentage when L2ARC contents are being successfully compressed before writing. A value of .Sy 100 disables this feature. . .It Sy l2arc_dwpd_limit Ns = Ns Sy 100 Pq uint Drive Writes Per Day limit for L2ARC devices to protect SSD endurance, specified as a percentage where 100 equals 1.0 DWPD. A value of 100 means each L2ARC device can write its own capacity once per day. Lower values support fractional DWPD (50 = 0.5 DWPD, 30 = 0.3 DWPD for QLC SSDs). Higher values allow more writes (300 = 3.0 DWPD). The effective write rate is always bounded by .Sy l2arc_write_max . A value of 0 disables DWPD rate limiting entirely. DWPD limiting only applies after the initial fill pass completes and when total L2ARC capacity is at least twice arc_c_max. . .It Sy l2arc_exclude_special Ns = Ns Sy 0 Ns | Ns 1 Pq int Controls whether buffers present on special vdevs are eligible for caching into L2ARC. If set to 1, exclude dbufs on special vdevs from being cached to L2ARC. . .It Sy l2arc_mfuonly Ns = Ns Sy 0 Ns | Ns 1 Ns | Ns 2 Pq int Controls whether only MFU metadata and data are cached from ARC into L2ARC. This may be desired to avoid wasting space on L2ARC when reading/writing large amounts of data that are not expected to be accessed more than once. .Pp The default is 0, meaning both MRU and MFU data and metadata are cached. When turning off this feature (setting it to 0), some MRU buffers will still be present in ARC and eventually cached on L2ARC. .No If Sy l2arc_noprefetch Ns = Ns Sy 0 , some prefetched buffers will be cached to L2ARC, and those might later transition to MRU, in which case the .Sy l2arc_mru_asize No arcstat will not be Sy 0 . .Pp Setting it to 1 means to L2 cache only MFU data and metadata. .Pp Setting it to 2 means to L2 cache all metadata (MRU+MFU) but only MFU data (i.e. MRU data are not cached). This can be the right setting to cache as much metadata as possible even when having high data turnover. .Pp Regardless of .Sy l2arc_noprefetch , some MFU buffers might be evicted from ARC, accessed later on as prefetches and transition to MRU as prefetches. If accessed again they are counted as MRU and the .Sy l2arc_mru_asize No arcstat will not be Sy 0 . .Pp The ARC status of L2ARC buffers when they were first cached in L2ARC can be seen in the .Sy l2arc_mru_asize , Sy l2arc_mfu_asize , No and Sy l2arc_prefetch_asize arcstats when importing the pool or onlining a cache device if persistent L2ARC is enabled. .Pp The .Sy evict_l2_eligible_mru arcstat does not take into account if this option is enabled as the information provided by the .Sy evict_l2_eligible_m[rf]u arcstats can be used to decide if toggling this option is appropriate for the current workload. . .It Sy l2arc_meta_percent Ns = Ns Sy 33 Ns % Pq uint Percent of ARC size allowed for L2ARC-only headers. Since L2ARC buffers are not evicted on memory pressure, too many headers on a system with an irrationally large L2ARC can render it slow or unusable. This parameter limits L2ARC writes and rebuilds to achieve the target. . .It Sy l2arc_trim_ahead Ns = Ns Sy 0 Ns % Pq u64 Trims ahead of the current write size on L2ARC devices by this percentage of write size if we have filled the device. If set to .Sy 100 we TRIM twice the space required to accommodate upcoming writes. A minimum of .Sy 64 MiB will be trimmed. It also enables TRIM of the whole L2ARC device upon creation or addition to an existing pool or if the header of the device is invalid upon importing a pool or onlining a cache device. A value of .Sy 0 disables TRIM on L2ARC altogether and is the default as it can put significant stress on the underlying storage devices. This will vary depending of how well the specific device handles these commands. . .It Sy l2arc_noprefetch Ns = Ns Sy 1 Ns | Ns 0 Pq int Do not write buffers to L2ARC if they were prefetched but not used by applications. In case there are prefetched buffers in L2ARC and this option is later set, we do not read the prefetched buffers from L2ARC. Unsetting this option is useful for caching sequential reads from the disks to L2ARC and serve those reads from L2ARC later on. This may be beneficial in case the L2ARC device is significantly faster in sequential reads than the disks of the pool. .Pp Use .Sy 1 to disable and .Sy 0 to enable caching/reading prefetches to/from L2ARC. . .It Sy l2arc_norw Ns = Ns Sy 0 Ns | Ns 1 Pq int No reads during writes. . .It Sy l2arc_ext_headroom_pct Ns = Ns Sy 25 Pq u64 Percentage of each ARC state's size that a pass may scan before resetting its markers to the tail. Lower values keep the marker closer to the tail under active workloads. Set to 0 to disable the depth cap. . .It Sy l2arc_meta_cycles Ns = Ns Sy 2 Pq u64 How many consecutive cycles metadata may monopolize the write budget before being skipped to let data run. The default of 2 gives metadata roughly 67% and data 33% of L2ARC write bandwidth under sustained load. Higher values favor metadata; set to 0 to disable. . .It Sy l2arc_write_max Ns = Ns Sy 33554432 Ns B Po 32 MiB Pc Pq u64 Maximum write rate in bytes per second for each L2ARC device. Used directly during initial fill, when DWPD limiting is disabled, or for non-persistent L2ARC. When DWPD limiting is active, writes are capped by this rate. Total L2ARC throughput scales with the number of cache devices in a pool. . .It Sy l2arc_rebuild_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int Rebuild the L2ARC when importing a pool (persistent L2ARC). This can be disabled if there are problems importing a pool or attaching an L2ARC device (e.g. the L2ARC device is slow in reading stored log metadata, or the metadata has become somehow fragmented/unusable). . .It Sy l2arc_rebuild_blocks_min_l2size Ns = Ns Sy 1073741824 Ns B Po 1 GiB Pc Pq u64 Minimum size of an L2ARC device required in order to write log blocks in it. The log blocks are used upon importing the pool to rebuild the persistent L2ARC. .Pp For L2ARC devices less than 1 GiB, the amount of data .Fn l2arc_evict evicts is significant compared to the amount of restored L2ARC data. In this case, do not write log blocks in L2ARC in order not to waste space. . .It Sy metaslab_aliquot Ns = Ns Sy 2097152 Ns B Po 2 MiB Pc Pq u64 Metaslab group's per child vdev allocation granularity, in bytes. This is roughly similar to what would be referred to as the "stripe size" in traditional RAID arrays. In normal operation, ZFS will try to write this amount of data to each child of a top-level vdev before moving on to the next top-level vdev. . .It Sy metaslab_bias_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int Enable metaslab groups biasing based on their over- or under-utilization relative to the metaslab class average. If disabled, each metaslab group will receive allocations proportional to its capacity. . .It Sy metaslab_perf_bias Ns = Ns Sy 1 Ns | Ns 0 Ns | Ns 2 Pq int Controls metaslab groups biasing based on their write performance. Setting to 0 makes all metaslab groups receive fixed amounts of allocations. Setting to 2 allows faster metaslab groups to allocate more. Setting to 1 equals to 2 if the pool is write-bound or 0 otherwise. That is, if the pool is limited by write throughput, then allocate more from faster metaslab groups, but if not, try to evenly distribute the allocations. . .It Sy metaslab_force_ganging Ns = Ns Sy 16777217 Ns B Po 16 MiB + 1 B Pc Pq u64 Make some blocks above a certain size be gang blocks. This option is used by the test suite to facilitate testing. . .It Sy metaslab_force_ganging_pct Ns = Ns Sy 3 Ns % Pq uint For blocks that could be forced to be a gang block (due to .Sy metaslab_force_ganging ) , force this many of them to be gang blocks. . .It Sy brt_zap_prefetch Ns = Ns Sy 1 Ns | Ns 0 Pq int Controls prefetching BRT records for blocks which are going to be cloned. . .It Sy brt_zap_default_bs Ns = Ns Sy 13 Po 8 KiB Pc Pq int Default BRT ZAP data block size as a power of 2. Note that changing this after creating a BRT on the pool will not affect existing BRTs, only newly created ones. . .It Sy brt_zap_default_ibs Ns = Ns Sy 13 Po 8 KiB Pc Pq int Default BRT ZAP indirect block size as a power of 2. Note that changing this after creating a BRT on the pool will not affect existing BRTs, only newly created ones. . .It Sy ddt_zap_default_bs Ns = Ns Sy 15 Po 32 KiB Pc Pq int Default DDT ZAP data block size as a power of 2. Note that changing this after creating a DDT on the pool will not affect existing DDTs, only newly created ones. . .It Sy ddt_zap_default_ibs Ns = Ns Sy 15 Po 32 KiB Pc Pq int Default DDT ZAP indirect block size as a power of 2. Note that changing this after creating a DDT on the pool will not affect existing DDTs, only newly created ones. . .It Sy zfs_default_bs Ns = Ns Sy 9 Po 512 B Pc Pq int Default dnode block size as a power of 2. . .It Sy zfs_default_ibs Ns = Ns Sy 17 Po 128 KiB Pc Pq int Default dnode indirect block size as a power of 2. . .It Sy zfs_dio_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int Enable Direct I/O. If this setting is 0, then all I/O requests will be directed through the ARC acting as though the dataset property .Sy direct was set to .Sy disabled . . .It Sy zfs_dio_strict Ns = Ns Sy 0 Ns | Ns 1 Pq int Strictly enforce alignment for Direct I/O requests, returning .Sy EINVAL if not page-aligned instead of silently falling back to uncached I/O. . .It Sy zfs_history_output_max Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq u64 When attempting to log an output nvlist of an ioctl in the on-disk history, the output will not be stored if it is larger than this size (in bytes). This must be less than .Sy DMU_MAX_ACCESS Pq 64 MiB . This applies primarily to .Fn zfs_ioc_channel_program Pq cf. Xr zfs-program 8 . . .It Sy zfs_keep_log_spacemaps_at_export Ns = Ns Sy 0 Ns | Ns 1 Pq int Prevent log spacemaps from being destroyed during pool exports and destroys. . .It Sy zfs_metaslab_segment_weight_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int Enable/disable segment-based metaslab selection. . .It Sy zfs_metaslab_switch_threshold Ns = Ns Sy 2 Pq int When using segment-based metaslab selection, continue allocating from the active metaslab until this option's worth of buckets have been exhausted. . .It Sy metaslab_debug_load Ns = Ns Sy 0 Ns | Ns 1 Pq int Load all metaslabs during pool import. . .It Sy metaslab_debug_unload Ns = Ns Sy 0 Ns | Ns 1 Pq int Prevent metaslabs from being unloaded. . .It Sy metaslab_fragmentation_factor_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int Enable use of the fragmentation metric in computing metaslab weights. . .It Sy metaslab_df_max_search Ns = Ns Sy 16777216 Ns B Po 16 MiB Pc Pq uint Maximum distance to search forward from the last offset. Without this limit, fragmented pools can see .Em >100`000 iterations and .Fn metaslab_block_picker becomes the performance limiting factor on high-performance storage. .Pp With the default setting of .Sy 16 MiB , we typically see less than .Em 500 iterations, even with very fragmented .Sy ashift Ns = Ns Sy 9 pools. The maximum number of iterations possible is .Sy metaslab_df_max_search / 2^(ashift+1) . With the default setting of .Sy 16 MiB this is .Em 16*1024 Pq with Sy ashift Ns = Ns Sy 9 or .Em 2*1024 Pq with Sy ashift Ns = Ns Sy 12 . . .It Sy metaslab_df_alloc_threshold Ns = Ns Sy 131072 Ns B Po 128 KiB Pc Pq u64 Minimum size which forces the dynamic allocator to change its allocation strategy. Once the space map cannot satisfy an allocation of this size, it switches to a more aggressive strategy (searching by size rather than offset). . .It Sy metaslab_df_free_pct Ns = Ns Sy 4 Ns % Pq uint The minimum free space, in percent, which must be available in a space map to continue allocations in a first-fit fashion. Once free space drops below this level, allocations switch to a best-fit strategy. . .It Sy metaslab_df_use_largest_segment Ns = Ns Sy 0 Ns | Ns 1 Pq int If not searching forward (due to .Sy metaslab_df_max_search , metaslab_df_free_pct , .No or Sy metaslab_df_alloc_threshold ) , this tunable controls which segment is used. If set, we will use the largest free segment. If unset, we will use a segment of at least the requested size. . .It Sy zfs_metaslab_max_size_cache_sec Ns = Ns Sy 3600 Ns s Po 1 hour Pc Pq u64 When we unload a metaslab, we cache the size of the largest free chunk. We use that cached size to determine whether or not to load a metaslab for a given allocation. As more frees accumulate in that metaslab while it's unloaded, the cached max size becomes less and less accurate. After a number of seconds controlled by this tunable, we stop considering the cached max size and start considering only the histogram instead. . .It Sy zfs_metaslab_mem_limit Ns = Ns Sy 25 Ns % Pq uint When we are loading a new metaslab, we check the amount of memory being used to store metaslab range trees. If it is over a threshold, we attempt to unload the least recently used metaslab to prevent the system from clogging all of its memory with range trees. This tunable sets the percentage of total system memory that is the threshold. . .It Sy zfs_metaslab_try_hard_before_gang Ns = Ns Sy 0 Ns | Ns 1 Pq int .Bl -item -compact .It If unset, we will first try normal allocation. .It If that fails then we will do a gang allocation. .It If that fails then we will do a "try hard" gang allocation. .It If that fails then we will have a multi-layer gang block. .El .Pp .Bl -item -compact .It If set, we will first try normal allocation. .It If that fails then we will do a "try hard" allocation. .It If that fails we will do a gang allocation. .It If that fails we will do a "try hard" gang allocation. .It If that fails then we will have a multi-layer gang block. .El . .It Sy zfs_metaslab_find_max_tries Ns = Ns Sy 100 Pq uint When not trying hard, we only consider this number of the best metaslabs. This improves performance, especially when there are many metaslabs per vdev and the allocation can't actually be satisfied (so we would otherwise iterate all metaslabs). . .It Sy zfs_metaslab_sm_blksz_no_log Ns = Ns Sy 16384 Ns B Po 16 KiB Pc Pq int Block size for the metaslab space maps in pools where the .Sy log_spacemap feature is disabled. Multiple metaslabs are modified per transaction group, so a smaller block size lets more, scattered I/O operations be issued. Must be a power of 2 greater than .Sy 4096 . This parameter can only be set at module load time. . .It Sy zfs_metaslab_sm_blksz_with_log Ns = Ns Sy 131072 Ns B Po 128 KiB Pc Pq int Block size for the metaslab space maps in pools where the .Sy log_spacemap feature is enabled. Changes are batched in the per-pool log spacemap and flushed to each metaslab's space map only occasionally, so a larger block size is more efficient. Must be a power of 2 greater than .Sy 4096 . This parameter can only be set at module load time. . .It Sy zfs_metaslab_condense_pct Ns = Ns Sy 200 Ns % Pq uint Condense an on-disk space map when its size exceeds this percentage of the in-memory representation. The minimum is .Sy 100 . . .It Sy zfs_vdev_default_ms_count Ns = Ns Sy 200 Pq uint When a vdev is added, target this number of metaslabs per top-level vdev. . .It Sy zfs_vdev_default_ms_shift Ns = Ns Sy 29 Po 512 MiB Pc Pq uint Default lower limit for metaslab size. . .It Sy zfs_vdev_max_ms_shift Ns = Ns Sy 34 Po 16 GiB Pc Pq uint Default upper limit for metaslab size. . .It Sy zfs_vdev_max_auto_ashift Ns = Ns Sy 14 Pq uint Maximum ashift used when optimizing for logical \[->] physical sector size on new top-level vdevs. May be increased up to .Sy ASHIFT_MAX Po 16 Pc , but this may negatively impact pool space efficiency. . .It Sy zfs_vdev_direct_write_verify Ns = Ns Sy Linux 1 | FreeBSD 0 Pq uint If non-zero, then a Direct I/O write's checksum will be verified every time the write is issued and before it is committed to the block pointer. In the event the checksum is not valid then the I/O operation will return EIO. This module parameter can be used to detect if the contents of the users buffer have changed in the process of doing a Direct I/O write. It can also help to identify if reported checksum errors are tied to Direct I/O writes. Each verify error causes a .Sy dio_verify_wr zevent. Direct Write I/O checksum verify errors can be seen with .Nm zpool Cm status Fl d . The default value for this is 1 on Linux, but is 0 for .Fx because user pages can be placed under write protection in .Fx before the Direct I/O write is issued. . .It Sy zfs_vdev_min_auto_ashift Ns = Ns Sy ASHIFT_MIN Po 9 Pc Pq uint Minimum ashift used when creating new top-level vdevs. . .It Sy zfs_vdev_min_ms_count Ns = Ns Sy 16 Pq uint Minimum number of metaslabs to create in a top-level vdev. . .It Sy vdev_validate_skip Ns = Ns Sy 0 Ns | Ns 1 Pq int Skip label validation steps during pool import. Changing is not recommended unless you know what you're doing and are recovering a damaged label. . .It Sy zfs_vdev_ms_count_limit Ns = Ns Sy 131072 Po 128k Pc Pq uint Practical upper limit of total metaslabs per top-level vdev. . .It Sy zfs_vdev_dtl_sm_blksz Ns = Ns Sy 4096 Ns B Po 4 KiB Pc Pq int Since the DTL space map of a vdev is not expected to have a lot of entries, we default its block size to 4K. . .It Sy zfs_vdev_standard_sm_blksz Ns = Ns Sy 131072 Ns B Po 128 KiB Pc Pq int vdev-wide space maps that have lots of entries written to them at the end of each transaction can benefit from a higher I/O bandwidth .Pq e.g. Sy vdev_obsolete_sm , thus we default their block size to 128K. . .It Sy metaslab_preload_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int Enable metaslab group preloading. . .It Sy metaslab_preload_limit Ns = Ns Sy 10 Pq uint Maximum number of metaslabs per group to preload . .It Sy metaslab_preload_pct Ns = Ns Sy 50 Pq uint Percentage of CPUs to run a metaslab preload taskq . .It Sy metaslab_lba_weighting_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int Give more weight to metaslabs with lower LBAs, assuming they have greater bandwidth, as is typically the case on a modern constant angular velocity disk drive. . .It Sy metaslab_unload_delay Ns = Ns Sy 32 Pq uint After a metaslab is used, we keep it loaded for this many TXGs, to attempt to reduce unnecessary reloading. Note that both this many TXGs and .Sy metaslab_unload_delay_ms milliseconds must pass before unloading will occur. . .It Sy metaslab_unload_delay_ms Ns = Ns Sy 600000 Ns ms Po 10 min Pc Pq uint After a metaslab is used, we keep it loaded for this many milliseconds, to attempt to reduce unnecessary reloading. Note, that both this many milliseconds and .Sy metaslab_unload_delay TXGs must pass before unloading will occur. . .It Sy reference_history Ns = Ns Sy 3 Pq uint Maximum reference holders being tracked when reference_tracking_enable is active. .It Sy raidz_expand_max_copy_bytes Ns = Ns Sy 160MB Pq ulong Max amount of memory to use for RAID-Z expansion I/O. This limits how much I/O can be outstanding at once. . .It Sy raidz_expand_max_reflow_bytes Ns = Ns Sy 0 Pq ulong For testing, pause RAID-Z expansion when reflow amount reaches this value. . .It Sy raidz_io_aggregate_rows Ns = Ns Sy 4 Pq ulong For expanded RAID-Z, aggregate reads that have more rows than this. . .It Sy reference_history Ns = Ns Sy 3 Pq int Maximum reference holders being tracked when reference_tracking_enable is active. . .It Sy reference_tracking_enable Ns = Ns Sy 0 Ns | Ns 1 Pq int Track reference holders to .Sy refcount_t objects (debug builds only). . .It Sy send_holes_without_birth_time Ns = Ns Sy 1 Ns | Ns 0 Pq int When set, the .Sy hole_birth optimization will not be used, and all holes will always be sent during a .Nm zfs Cm send . This is useful if you suspect your datasets are affected by a bug in .Sy hole_birth . . .It Sy spa_config_path Ns = Ns Pa /etc/zfs/zpool.cache Pq charp SPA config file. . .It Sy spa_asize_inflation Ns = Ns Sy 24 Pq uint Multiplication factor used to estimate actual disk consumption from the size of data being written. The default value is a worst case estimate, but lower values may be valid for a given pool depending on its configuration. Pool administrators who understand the factors involved may wish to specify a more realistic inflation factor, particularly if they operate close to quota or capacity limits. . .It Sy spa_load_print_vdev_tree Ns = Ns Sy 0 Ns | Ns 1 Pq int Whether to print the vdev tree in the debugging message buffer during pool import. . .It Sy spa_load_verify_data Ns = Ns Sy 1 Ns | Ns 0 Pq int Whether to traverse data blocks during an "extreme rewind" .Pq Fl X import. .Pp An extreme rewind import normally performs a full traversal of all blocks in the pool for verification. If this parameter is unset, the traversal skips non-metadata blocks. It can be toggled once the import has started to stop or start the traversal of non-metadata blocks. . .It Sy spa_load_verify_metadata Ns = Ns Sy 1 Ns | Ns 0 Pq int Whether to traverse blocks during an "extreme rewind" .Pq Fl X pool import. .Pp An extreme rewind import normally performs a full traversal of all blocks in the pool for verification. If this parameter is unset, the traversal is not performed. It can be toggled once the import has started to stop or start the traversal. . .It Sy spa_load_verify_shift Ns = Ns Sy 4 Po 1/16th Pc Pq uint Sets the maximum number of bytes to consume during pool import to the log2 fraction of the target ARC size. . .It Sy spa_slop_shift Ns = Ns Sy 5 Po 1/32nd Pc Pq int Normally, we don't allow the last .Sy 3.2% Pq Sy 1/2^spa_slop_shift of space in the pool to be consumed. This ensures that we don't run the pool completely out of space, due to unaccounted changes (e.g. to the MOS). It also limits the worst-case time to allocate space. If we have less than this amount of free space, most ZPL operations (e.g. write, create) will return .Sy ENOSPC . . .It Sy spa_num_allocators Ns = Ns Sy 4 Pq int Determines the number of block allocators to use per spa instance. Capped by the number of actual CPUs in the system via .Sy spa_cpus_per_allocator . .Pp Note that setting this value too high could result in performance degradation and/or excess fragmentation. Set value only applies to pools imported/created after that. . .It Sy spa_cpus_per_allocator Ns = Ns Sy 4 Pq int Determines the minimum number of CPUs in a system for block allocator per spa instance. Set value only applies to pools imported/created after that. . .It Sy spa_upgrade_errlog_limit Ns = Ns Sy 0 Pq uint Limits the number of on-disk error log entries that will be converted to the new format when enabling the .Sy head_errlog feature. The default is to convert all log entries. . .It Sy vdev_read_sit_out_secs Ns = Ns Sy 600 Ns s Po 10 min Pc Pq ulong When a slow disk outlier is detected it is placed in a sit out state. While sitting out the disk will not participate in normal reads, instead its data will be reconstructed as needed from parity. Scrub operations will always read from a disk, even if it's sitting out. A number of disks in a RAID-Z or dRAID vdev may sit out at the same time, up to the number of parity devices. Writes will still be issued to a disk which is sitting out to maintain full redundancy. Defaults to 600 seconds and a value of zero disables disk sit-outs in general, including slow disk outlier detection. . .It Sy vdev_raidz_outlier_check_interval_ms Ns = Ns Sy 1000 Ns ms Po 1 sec Pc Pq ulong How often each RAID-Z and dRAID vdev will check for slow disk outliers. Increasing this interval will reduce the sensitivity of detection (since all I/Os since the last check are included in the statistics), but will slow the response to a disk developing a problem. Defaults to once per second; setting extremely small values may cause negative performance effects. . .It Sy vdev_raidz_outlier_insensitivity Ns = Ns Sy 50 Pq uint When performing slow outlier checks for RAID-Z and dRAID vdevs, this value is used to determine how far out an outlier must be before it counts as an event worth consdering. This is phrased as "insensitivity" because larger values result in fewer detections. Smaller values will result in more aggressive sitting out of disks that may have problems, but may significantly increase the rate of spurious sit-outs. .Pp To provide a more technical definition of this parameter, this is the multiple of the inter-quartile range (IQR) that is being used in a Tukey's Fence detection algorithm. This is much higher than a normal Tukey's Fence k-value, because the distribution under consideration is probably an extreme-value distribution, rather than a more typical Gaussian distribution. . .It Sy vdev_removal_max_span Ns = Ns Sy 32768 Ns B Po 32 KiB Pc Pq uint During top-level vdev removal, chunks of data are copied from the vdev which may include free space in order to trade bandwidth for IOPS. This parameter determines the maximum span of free space, in bytes, which will be included as "unnecessary" data in a chunk of copied data. .Pp The default value here was chosen to align with .Sy zfs_vdev_read_gap_limit , which is a similar concept when doing regular reads (but there's no reason it has to be the same). . .It Sy vdev_file_logical_ashift Ns = Ns Sy 9 Po 512 B Pc Pq u64 Logical ashift for file-based devices. . .It Sy vdev_file_physical_ashift Ns = Ns Sy 9 Po 512 B Pc Pq u64 Physical ashift for file-based devices. . .It Sy zap_iterate_prefetch Ns = Ns Sy 1 Ns | Ns 0 Pq int If set, when we start iterating over a ZAP object, prefetch the entire object (all leaf blocks). However, this is limited by .Sy dmu_prefetch_max . . .It Sy zap_micro_max_size Ns = Ns Sy 131072 Ns B Po 128 KiB Pc Pq int Maximum micro ZAP size. A "micro" ZAP is upgraded to a "fat" ZAP once it grows beyond the specified size. Sizes higher than 128KiB will be clamped to 128KiB unless the .Sy large_microzap feature is enabled. . .It Sy zap_shrink_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int If set, adjacent empty ZAP blocks will be collapsed, reducing disk space. . .It Sy zfetch_min_distance Ns = Ns Sy 4194304 Ns B Po 4 MiB Pc Pq uint Min bytes to prefetch per stream. Prefetch distance starts from the demand access size and quickly grows to this value, doubling on each hit. After that it may grow further by 1/8 per hit, but only if some prefetch since last time haven't completed in time to satisfy demand request, i.e. prefetch depth didn't cover the read latency or the pool got saturated. . .It Sy zfetch_max_distance Ns = Ns Sy 67108864 Ns B Po 64 MiB Pc Pq uint Max bytes to prefetch per stream. . .It Sy zfetch_max_idistance Ns = Ns Sy 67108864 Ns B Po 64 MiB Pc Pq uint Max bytes to prefetch indirects for per stream. . .It Sy zfetch_max_reorder Ns = Ns Sy 16777216 Ns B Po 16 MiB Pc Pq uint Requests within this byte distance from the current prefetch stream position are considered parts of the stream, reordered due to parallel processing. Such requests do not advance the stream position immediately unless .Sy zfetch_hole_shift fill threshold is reached, but saved to fill holes in the stream later. . .It Sy zfetch_max_streams Ns = Ns Sy 8 Pq uint Max number of streams per zfetch (prefetch streams per file). . .It Sy zfetch_min_sec_reap Ns = Ns Sy 1 Pq uint Min time before inactive prefetch stream can be reclaimed . .It Sy zfetch_max_sec_reap Ns = Ns Sy 2 Pq uint Max time before inactive prefetch stream can be deleted . .It Sy zfs_abd_scatter_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int Enables ARC from using scatter/gather lists and forces all allocations to be linear in kernel memory. Disabling can improve performance in some code paths at the expense of fragmented kernel memory. . .It Sy zfs_abd_scatter_max_order Ns = Ns Sy MAX_ORDER\-1 Pq uint Maximum number of consecutive memory pages allocated in a single block for scatter/gather lists. .Pp The value of .Sy MAX_ORDER depends on kernel configuration. . .It Sy zfs_abd_scatter_min_size Ns = Ns Sy 1536 Ns B Po 1.5 KiB Pc Pq uint This is the minimum allocation size that will use scatter (page-based) ABDs. Smaller allocations will use linear ABDs. . .It Sy zfs_active_allocator Ns = Ns Sy dynamic Pq charp Select the SPA metaslab allocator. Valid values are .Sy dynamic and .Sy cursor . . .It Sy zfs_arc_dnode_limit Ns = Ns Sy 0 Ns B Pq u64 When the number of bytes consumed by dnodes in the ARC exceeds this number of bytes, try to unpin some of it in response to demand for non-metadata. This value acts as a ceiling to the amount of dnode metadata, and defaults to .Sy 0 , which indicates that a percent which is based on .Sy zfs_arc_dnode_limit_percent of the ARC meta buffers that may be used for dnodes. .It Sy zfs_arc_dnode_limit_percent Ns = Ns Sy 10 Ns % Pq u64 Percentage that can be consumed by dnodes of ARC meta buffers. .Pp See also .Sy zfs_arc_dnode_limit , which serves a similar purpose but has a higher priority if nonzero. . .It Sy zfs_arc_dnode_reduce_percent Ns = Ns Sy 10 Ns % Pq u64 Percentage used to size dnode prune requests. The request size is the larger of two values: .Sy zfs_arc_dnode_reduce_percent applied to the dnode count above .Sy zfs_arc_dnode_limit , or .Sy zfs_arc_dnode_reduce_percent applied to the total dnode count when non-evictable metadata exceeds 3/4 of the metadata target. . .It Sy zfs_arc_average_blocksize Ns = Ns Sy 8192 Ns B Po 8 KiB Pc Pq uint The ARC's buffer hash table is sized based on the assumption of an average block size of this value. This works out to roughly 1 MiB of hash table per 1 GiB of physical memory with 8-byte pointers. For configurations with a known larger average block size, this value can be increased to reduce the memory footprint. . .It Sy zfs_arc_eviction_pct Ns = Ns Sy 200 Ns % Pq uint When .Fn arc_is_overflowing , .Fn arc_get_data_impl waits for this percent of the requested amount of data to be evicted. For example, by default, for every .Em 2 KiB that's evicted, .Em 1 KiB of it may be "reused" by a new allocation. Since this is above .Sy 100 Ns % , it ensures that progress is made towards getting .Sy arc_size No under Sy arc_c . Since this is finite, it ensures that allocations can still happen, even during the potentially long time that .Sy arc_size No is more than Sy arc_c . . .It Sy zfs_arc_evict_batch_limit Ns = Ns Sy 10 Pq uint Number ARC headers to evict per sub-list before proceeding to another sub-list. This batch-style operation prevents entire sub-lists from being evicted at once but comes at a cost of additional unlocking and locking. . .It Sy zfs_arc_evict_batches_limit Ns = Ns Sy 5 Pq uint Number of .Sy zfs_arc_evict_batch_limit batches to process per parallel eviction task under heavy load to reduce number of context switches. . .It Sy zfs_arc_evict_threads Ns = Ns Sy 0 Pq int Sets the number of ARC eviction threads to be used. .Pp If set greater than 0, ZFS will dedicate up to that many threads to ARC eviction. Each thread will process one sub-list at a time, until the eviction target is reached or all sub-lists have been processed. When set to 0, ZFS will compute a reasonable number of eviction threads based on the number of CPUs. .TS box; lb l l . CPUs Threads _ 1-4 1 5-8 2 9-15 3 16-31 4 32-63 6 64-95 8 96-127 9 128-160 11 160-191 12 192-223 13 224-255 14 256+ 16 .TE .Pp More threads may improve the responsiveness of ZFS to memory pressure. This can be important for performance when eviction from the ARC becomes a bottleneck for reads and writes. .Pp This parameter can only be set at module load time. . .It Sy zfs_arc_grow_retry Ns = Ns Sy 0 Ns s Pq uint If set to a non zero value, it will replace the .Sy arc_grow_retry value with this value. The .Sy arc_grow_retry .No value Pq default Sy 5 Ns s is the number of seconds the ARC will wait before trying to resume growth after a memory pressure event. . .It Sy zfs_arc_lotsfree_percent Ns = Ns Sy 10 Ns % Pq int Throttle I/O when free system memory drops below this percentage of total system memory. Setting this value to .Sy 0 will disable the throttle. . .It Sy zfs_arc_max Ns = Ns Sy 0 Ns B Pq u64 Max size of ARC in bytes. If .Sy 0 , then the max size of ARC is determined by the amount of system memory installed. The larger of .Sy all_system_memory No \- Sy 1 GiB and .Sy 5/8 No \(mu Sy all_system_memory will be used as the limit. This value must be at least .Sy 67108864 Ns B Pq 64 MiB . .Pp This value can be changed dynamically, with some caveats. It cannot be set back to .Sy 0 while running, and reducing it below the current ARC size will not cause the ARC to shrink without memory pressure to induce shrinking. . .It Sy zfs_arc_meta_balance Ns = Ns Sy 500 Pq uint Balance between metadata and data on ghost hits. Values above 100 increase metadata caching by proportionally reducing effect of ghost data hits on target data/metadata rate. . .It Sy zfs_arc_min Ns = Ns Sy 0 Ns B Pq u64 Min size of ARC in bytes. .No If set to Sy 0 , arc_c_min will default to consuming the larger of .Sy 32 MiB and .Sy all_system_memory No / Sy 32 . . .It Sy zfs_arc_min_prefetch_ms Ns = Ns Sy 0 Ns ms Ns Po Ns ≡ Ns 1s Pc Pq uint Minimum time prefetched blocks are locked in the ARC. . .It Sy zfs_arc_min_prescient_prefetch_ms Ns = Ns Sy 0 Ns ms Ns Po Ns ≡ Ns 6s Pc Pq uint Minimum time "prescient prefetched" blocks are locked in the ARC. These blocks are meant to be prefetched fairly aggressively ahead of the code that may use them. . .It Sy zfs_arc_prune_task_threads Ns = Ns Sy 1 Pq int Number of arc_prune threads. .Fx does not need more than one. Linux may theoretically use one per mount point up to number of CPUs, but that was not proven to be useful. . .It Sy zfs_max_missing_tvds Ns = Ns Sy 0 Pq int Number of missing top-level vdevs which will be allowed during pool import (only in read-only mode). . .It Sy zfs_max_missing_tvds_cachefile Ns = Ns Sy 2 Pq u64 Number of missing top-level vdevs tolerated when importing a pool from a cachefile, before the trusted config is read from the MOS. A cachefile can fall out of sync with the on-disk config after a device removal that did not rewrite the cachefile, so the default of 2 still lets the import reach a copy of the MOS. . .It Sy zfs_max_missing_tvds_scan Ns = Ns Sy 0 Pq u64 Number of missing top-level vdevs tolerated when importing a pool by scanning device paths, before the trusted config is read from the MOS. Defaults to 0 because a scan should detect every present device. . .It Sy zfs_max_nvlist_src_size Ns = Sy 0 Pq u64 Maximum size in bytes allowed to be passed as .Sy zc_nvlist_src_size for ioctls on .Pa /dev/zfs . This prevents a user from causing the kernel to allocate an excessive amount of memory. When the limit is exceeded, the ioctl fails with .Sy EINVAL and a description of the error is sent to the .Pa zfs-dbgmsg log. This parameter should not need to be touched under normal circumstances. If .Sy 0 , equivalent to a quarter of the user-wired memory limit under .Fx and to .Sy 134217728 Ns B Pq 128 MiB under Linux. . .It Sy zfs_multilist_num_sublists Ns = Ns Sy 0 Pq uint To allow more fine-grained locking, each ARC state contains a series of lists for both data and metadata objects. Locking is performed at the level of these "sub-lists". This parameters controls the number of sub-lists per ARC state, and also applies to other uses of the multilist data structure. .Pp If .Sy 0 , equivalent to the greater of the number of online CPUs and .Sy 4 . . .It Sy zfs_arc_no_grow_shift Ns = Ns Sy 5 Pq uint If less than .Sy arc_c No >> Sy zfs_arc_no_grow_shift free memory is available, the ARC is not allowed to grow. . .It Sy zfs_arc_overflow_shift Ns = Ns Sy 8 Pq int The ARC size is considered to be overflowing if it exceeds the current ARC target size .Pq Sy arc_c by thresholds determined by this parameter. Exceeding by .Sy ( arc_c No >> Sy zfs_arc_overflow_shift ) No / Sy 2 starts ARC reclamation process. If that appears insufficient, exceeding by .Sy ( arc_c No >> Sy zfs_arc_overflow_shift ) No \(mu Sy 1.5 blocks new buffer allocation until the reclaim thread catches up. Started reclamation process continues till ARC size returns below the target size. .Pp The default value of .Sy 8 causes the ARC to start reclamation if it exceeds the target size by .Em 0.2% of the target size, and block allocations by .Em 0.6% . . .It Sy zfs_arc_shrink_shift Ns = Ns Sy 0 Pq uint If nonzero, this will update .Sy arc_shrink_shift Pq default Sy 7 with the new value. . .It Sy zfs_arc_pc_percent Ns = Ns Sy 0 Ns % Po off Pc Pq uint Percent of pagecache to reclaim ARC to. .Pp This tunable allows the ZFS ARC to play more nicely with the kernel's LRU pagecache. It can guarantee that the ARC size won't collapse under scanning pressure on the pagecache, yet still allows the ARC to be reclaimed down to .Sy zfs_arc_min if necessary. This value is specified as percent of pagecache size (as measured by .Sy NR_ACTIVE_FILE + .Sy NR_INACTIVE_FILE ) , where that percent may exceed .Sy 100 . This only operates during memory pressure/reclaim. . .It Sy zfs_arc_shrinker_limit Ns = Ns Sy 0 Pq int This is a limit on how many pages the ARC shrinker makes available for eviction in response to one page allocation attempt. Note that in practice, the kernel's shrinker can ask us to evict up to about four times this for one allocation attempt. To reduce OOM risk, this limit is applied for kswapd reclaims only. .Pp For example a value of .Sy 10000 Pq in practice, Em 160 MiB No per allocation attempt with 4 KiB pages limits the amount of time spent attempting to reclaim ARC memory to less than 100 ms per allocation attempt, even with a small average compressed block size of ~8 KiB. .Pp The parameter can be set to 0 (zero) to disable the limit, and only applies on Linux. . .It Sy zfs_arc_shrinker_seeks Ns = Ns Sy 2 Pq int Relative cost of ARC eviction on Linux, AKA number of seeks needed to restore evicted page. Bigger values make ARC more precious and evictions smaller, comparing to other kernel subsystems. Value of 4 means parity with page cache. . .It Sy zfs_arc_sys_free Ns = Ns Sy 0 Ns B Pq u64 The target number of bytes the ARC should leave as free memory on the system. If zero, equivalent to the bigger of .Sy 512 KiB No and Sy all_system_memory/64 . .It Sy zfs_arc_free_target Pq uint Desired number of free pages below which the ARC triggers reclaim. Initialized at boot to the kernel's .Va vm.v_free_target value and can be adjusted at runtime. This parameter is .Fx Ns -specific and uses pages, unlike the Linux-specific .Sy zfs_arc_sys_free which is measured in bytes. . .It Sy zfs_ccw_retry_interval Ns = Ns Sy 300 Ns s Pq int Interval, in seconds, at which a failed write of the configuration cache file is retried. . .It Sy zfs_checksum_events_per_second Ns = Ns Sy 20 Ns /s Pq uint Rate limit checksum events to this many per second. Note that this should not be set below the ZED thresholds (currently 10 checksums over 10 seconds) or else the daemon may not trigger any action. . .It Sy zfs_commit_timeout_pct Ns = Ns Sy 10 Ns % Pq uint This controls the amount of time that a ZIL block (lwb) will remain "open" when it isn't "full", and it has a thread waiting for it to be committed to stable storage. The timeout is scaled based on a percentage of the last lwb latency to avoid significantly impacting the latency of each individual transaction record (itx). . .It Sy zfs_compressed_arc_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int Enables storing ARC buffers in their on-disk compressed form, reducing memory pressure. When disabled, buffers are decompressed before being cached. . .It Sy zfs_condense_indirect_commit_entry_delay_ms Ns = Ns Sy 0 Ns ms Pq int Vdev indirection layer (used for device removal) sleeps for this many milliseconds during mapping generation. Intended for use with the test suite to throttle vdev removal speed. . .It Sy zfs_condense_indirect_obsolete_pct Ns = Ns Sy 25 Ns % Pq uint Minimum percent of obsolete bytes in vdev mapping required to attempt to condense .Pq see Sy zfs_condense_indirect_vdevs_enable . Intended for use with the test suite to facilitate triggering condensing as needed. . .It Sy zfs_condense_indirect_vdevs_enable Ns = Ns Sy 1 Ns | Ns 0 Pq int Enable condensing indirect vdev mappings. When set, attempt to condense indirect vdev mappings if the mapping uses more than .Sy zfs_condense_min_mapping_bytes bytes of memory and if the obsolete space map object uses more than .Sy zfs_condense_max_obsolete_bytes bytes on-disk. The condensing process is an attempt to save memory by removing obsolete mappings. . .It Sy zfs_condense_max_obsolete_bytes Ns = Ns Sy 1073741824 Ns B Po 1 GiB Pc Pq u64 Only attempt to condense indirect vdev mappings if the on-disk size of the obsolete space map object is greater than this number of bytes .Pq see Sy zfs_condense_indirect_vdevs_enable . . .It Sy zfs_condense_min_mapping_bytes Ns = Ns Sy 131072 Ns B Po 128 KiB Pc Pq u64 Minimum size vdev mapping to attempt to condense .Pq see Sy zfs_condense_indirect_vdevs_enable . . .It Sy zfs_dbgmsg_enable Ns = Ns Sy 1 Ns | Ns 0 Pq int Internally ZFS keeps a small log to facilitate debugging. The log is enabled by default, and can be disabled by unsetting this option. The contents of the log can be accessed by reading .Pa /proc/spl/kstat/zfs/dbgmsg . Writing .Sy 0 to the file clears the log. .Pp This setting does not influence debug prints due to .Sy zfs_flags . . .It Sy zfs_dbgmsg_maxsize Ns = Ns Sy 4194304 Ns B Po 4 MiB Pc Pq uint Maximum size of the internal ZFS debug log. . .It Sy zfs_dbuf_state_index Ns = Ns Sy 0 Pq int Historically used for controlling what reporting was available under .Pa /proc/spl/kstat/zfs . No effect. . .It Sy zfs_deadman_checktime_ms Ns = Ns Sy 60000 Ns ms Po 1 min Pc Pq u64 Check time in milliseconds. This defines the frequency at which we check for hung I/O requests and potentially invoke the .Sy zfs_deadman_failmode behavior. . .It Sy zfs_deadman_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int When a pool sync operation takes longer than .Sy zfs_deadman_synctime_ms , or when an individual I/O operation takes longer than .Sy zfs_deadman_ziotime_ms , then the operation is considered to be "hung". If .Sy zfs_deadman_enabled is set, then the deadman behavior is invoked as described by .Sy zfs_deadman_failmode . By default, the deadman is enabled and set to .Sy wait which results in "hung" I/O operations only being logged. The deadman is automatically disabled when a pool gets suspended. . .It Sy zfs_deadman_events_per_second Ns = Ns Sy 1 Ns /s Pq int Rate limit deadman zevents (which report hung I/O operations) to this many per second. . .It Sy zfs_deadman_failmode Ns = Ns Sy wait Pq charp Controls the failure behavior when the deadman detects a "hung" I/O operation. Valid values are: .Bl -tag -compact -offset 4n -width "continue" .It Sy wait Wait for a "hung" operation to complete. For each "hung" operation a "deadman" event will be posted describing that operation. .It Sy continue Attempt to recover from a "hung" operation by re-dispatching it to the I/O pipeline if possible. .It Sy panic Panic the system. This can be used to facilitate automatic fail-over to a properly configured fail-over partner. .El . .It Sy zfs_deadman_synctime_ms Ns = Ns Sy 600000 Ns ms Po 10 min Pc Pq u64 Interval in milliseconds after which the deadman is triggered and also the interval after which a pool sync operation is considered to be "hung". Once this limit is exceeded the deadman will be invoked every .Sy zfs_deadman_checktime_ms milliseconds until the pool sync completes. . .It Sy zfs_deadman_ziotime_ms Ns = Ns Sy 300000 Ns ms Po 5 min Pc Pq u64 Interval in milliseconds after which the deadman is triggered and an individual I/O operation is considered to be "hung". As long as the operation remains "hung", the deadman will be invoked every .Sy zfs_deadman_checktime_ms milliseconds until the operation completes. . .It Sy zfs_dedup_prefetch Ns = Ns Sy 0 Ns | Ns 1 Pq int Enable prefetching dedup-ed blocks which are going to be freed. . .It Sy zfs_dedup_log_flush_min_time_ms Ns = Ns Sy 1000 Ns Pq uint Minimum time to spend on dedup log flush each transaction. .Pp At least this long will be spent flushing dedup log entries each transaction, up to .Sy zfs_txg_timeout . This occurs even if doing so would delay the transaction, that is, other IO completes under this time. . .It Sy zfs_dedup_log_flush_entries_min Ns = Ns Sy 100 Ns Pq uint Flush at least this many entries each transaction. .Pp OpenZFS will flush a fraction of the log every TXG, to keep the size proportional to the ingest rate (see .Sy zfs_dedup_log_flush_txgs ) . This sets the minimum for that estimate, which prevents the backlog from completely draining if the ingest rate falls. Raising it can force OpenZFS to flush more aggressively, reducing the backlog to zero more quickly, but can make it less able to back off if log flushing would compete with other IO too much. . .It Sy zfs_dedup_log_flush_entries_max Ns = Ns Sy UINT_MAX Ns Pq uint Flush at most this many entries each transaction. .Pp Mostly used for debugging purposes. .It Sy zfs_dedup_log_flush_txgs Ns = Ns Sy 100 Ns Pq uint Target number of TXGs to process the whole dedup log. .Pp Every TXG, OpenZFS will process the inverse of this number times the size of the DDT backlog. This will keep the backlog at a size roughly equal to the ingest rate times this value. This offers a balance between a more efficient DDT log, with better aggregation, and shorter import times, which increase as the size of the DDT log increases. Increasing this value will result in a more efficient DDT log, but longer import times. .It Sy zfs_dedup_log_cap Ns = Ns Sy UINT_MAX Ns Pq uint Soft cap for the size of the current dedup log. .Pp If the log is larger than this size, we increase the aggressiveness of the flushing to try to bring it back down to the soft cap. Setting it will reduce import times, but will reduce the efficiency of the DDT log, increasing the expected number of IOs required to flush the same amount of data. .It Sy zfs_dedup_log_hard_cap Ns = Ns Sy 0 Ns | Ns 1 Pq uint Whether to treat the log cap as a firm cap or not. .Pp When set to 0 (the default), the .Sy zfs_dedup_log_cap will increase the maximum number of log entries we flush in a given txg. This will bring the backlog size down towards the cap, but not at the expense of making TXG syncs take longer. If this is set to 1, the cap acts more like a hard cap than a soft cap; it will also increase the minimum number of log entries we flush per TXG. Enabling it will reduce worst-case import times, at the cost of increased TXG sync times. .It Sy zfs_dedup_log_flush_flow_rate_txgs Ns = Ns Sy 10 Ns Pq uint Number of transactions to use to compute the flow rate. .Pp OpenZFS will estimate number of entries changed (ingest rate), number of entries flushed (flush rate) and time spent flushing (flush time rate) and combining these into an overall "flow rate". It will use an exponential weighted moving average over some number of recent transactions to compute these rates. This sets the number of transactions to compute these averages over. Setting it higher can help to smooth out the flow rate in the face of spiky workloads, but will take longer for the flow rate to adjust to a sustained change in the ingress rate. . .It Sy zfs_dedup_log_txg_max Ns = Ns Sy 8 Ns Pq uint Max transactions to before starting to flush dedup logs. .Pp OpenZFS maintains two dedup logs, one receiving new changes, one flushing. If there is nothing to flush, it will accumulate changes for no more than this many transactions before switching the logs and starting to flush entries out. . .It Sy zfs_dedup_log_mem_max Ns = Ns Sy 0 Ns Pq u64 Max memory to use for dedup logs. .Pp OpenZFS will spend no more than this much memory on maintaining the in-memory dedup log. Flushing will begin when around half this amount is being spent on logs. The default value of .Sy 0 will cause it to be set by .Sy zfs_dedup_log_mem_max_percent instead. . .It Sy zfs_dedup_log_mem_max_percent Ns = Ns Sy 1 Ns % Pq uint Max memory to use for dedup logs, as a percentage of total memory. .Pp If .Sy zfs_dedup_log_mem_max is not set, it will be initialized as a percentage of the total memory in the system. . .It Sy zfs_delay_min_dirty_percent Ns = Ns Sy 60 Ns % Pq uint Start to delay each transaction once there is this amount of dirty data, expressed as a percentage of .Sy zfs_dirty_data_max . This value should be at least .Sy zfs_vdev_async_write_active_max_dirty_percent . .No See Sx ZFS TRANSACTION DELAY . . .It Sy zfs_delay_scale Ns = Ns Sy 500000 Pq int This controls how quickly the transaction delay approaches infinity. Larger values cause longer delays for a given amount of dirty data. .Pp For the smoothest delay, this value should be about 1 billion divided by the maximum number of operations per second. This will smoothly handle between ten times and a tenth of this number. .No See Sx ZFS TRANSACTION DELAY . .Pp .Sy zfs_delay_scale No \(mu Sy zfs_dirty_data_max Em must No be smaller than Sy 2^64 . . .It Sy zfs_dio_write_verify_events_per_second Ns = Ns Sy 20 Ns /s Pq uint Rate limit Direct I/O write verify events to this many per second. . .It Sy zfs_disable_ivset_guid_check Ns = Ns Sy 0 Ns | Ns 1 Pq int Disables requirement for IVset GUIDs to be present and match when doing a raw receive of encrypted datasets. Intended for users whose pools were created with OpenZFS pre-release versions and now have compatibility issues. . .It Sy zfs_key_max_salt_uses Ns = Ns Sy 400000000 Po 4*10^8 Pc Pq ulong Maximum number of uses of a single salt value before generating a new one for encrypted datasets. The default value is also the maximum. . .It Sy zfs_object_mutex_size Ns = Ns Sy 64 Pq uint Size of the znode hashtable used for holds. .Pp Due to the need to hold locks on objects that may not exist yet, kernel mutexes are not created per-object and instead a hashtable is used where collisions will result in objects waiting when there is not actually contention on the same object. . .It Sy zfs_slow_io_events_per_second Ns = Ns Sy 20 Ns /s Pq int Rate limit delay zevents (which report slow I/O operations) to this many per second. . .It Sy zfs_unflushed_max_mem_amt Ns = Ns Sy 1073741824 Ns B Po 1 GiB Pc Pq u64 Upper-bound limit for unflushed metadata changes to be held by the log spacemap in memory, in bytes. . .It Sy zfs_unflushed_max_mem_ppm Ns = Ns Sy 1000 Ns ppm Po 0.1% Pc Pq u64 Part of overall system memory that ZFS allows to be used for unflushed metadata changes by the log spacemap, in millionths. . .It Sy zfs_unflushed_log_block_max Ns = Ns Sy 131072 Po 128k Pc Pq u64 Describes the maximum number of log spacemap blocks allowed for each pool. The default value means that the space in all the log spacemaps can add up to no more than .Sy 131072 blocks (which means .Em 16 GiB of logical space before compression and ditto blocks, assuming that blocksize is .Em 128 KiB ) . .Pp This tunable is important because it involves a trade-off between import time after an unclean export and the frequency of flushing metaslabs. The higher this number is, the more log blocks we allow when the pool is active which means that we flush metaslabs less often and thus decrease the number of I/O operations for spacemap updates per TXG. At the same time though, that means that in the event of an unclean export, there will be more log spacemap blocks for us to read, inducing overhead in the import time of the pool. The lower the number, the amount of flushing increases, destroying log blocks quicker as they become obsolete faster, which leaves less blocks to be read during import time after a crash. .Pp Each log spacemap block existing during pool import leads to approximately one extra logical I/O issued. This is the reason why this tunable is exposed in terms of blocks rather than space used. . .It Sy zfs_unflushed_log_block_min Ns = Ns Sy 1000 Pq u64 If the number of metaslabs is small and our incoming rate is high, we could get into a situation that we are flushing all our metaslabs every TXG. Thus we always allow at least this many log blocks. . .It Sy zfs_unflushed_log_block_pct Ns = Ns Sy 400 Ns % Pq u64 Tunable used to determine the number of blocks that can be used for the spacemap log, expressed as a percentage of the total number of unflushed metaslabs in the pool. . .It Sy zfs_unflushed_log_txg_max Ns = Ns Sy 1000 Pq u64 Tunable limiting maximum time in TXGs any metaslab may remain unflushed. It effectively limits maximum number of unflushed per-TXG spacemap logs that need to be read after unclean pool export. . .It Sy zfs_unlink_suspend_progress Ns = Ns Sy 0 Ns | Ns 1 Pq uint When enabled, files will not be asynchronously removed from the list of pending unlinks and the space they consume will be leaked. Once this option has been disabled and the dataset is remounted, the pending unlinks will be processed and the freed space returned to the pool. This option is used by the test suite. . .It Sy zfs_delete_blocks Ns = Ns Sy 20480 Pq ulong This is the used to define a large file for the purposes of deletion. Files containing more than .Sy zfs_delete_blocks will be deleted asynchronously, while smaller files are deleted synchronously. Decreasing this value will reduce the time spent in an .Xr unlink 2 system call, at the expense of a longer delay before the freed space is available. This only applies on Linux. . .It Sy zfs_dirty_data_max Ns = Pq int Determines the dirty space limit in bytes. Once this limit is exceeded, new writes are halted until space frees up. This parameter takes precedence over .Sy zfs_dirty_data_max_percent . .No See Sx ZFS TRANSACTION DELAY . .Pp Defaults to .Sy physical_ram/10 , capped at .Sy zfs_dirty_data_max_max . . .It Sy zfs_dirty_data_max_max Ns = Pq int Maximum allowable value of .Sy zfs_dirty_data_max , expressed in bytes. This limit is only enforced at module load time, and will be ignored if .Sy zfs_dirty_data_max is later changed. This parameter takes precedence over .Sy zfs_dirty_data_max_max_percent . .No See Sx ZFS TRANSACTION DELAY . .Pp Defaults to .Sy min(physical_ram/4, 4GiB) , or .Sy min(physical_ram/4, 1GiB) for 32-bit systems. . .It Sy zfs_dirty_data_max_max_percent Ns = Ns Sy 25 Ns % Pq uint Maximum allowable value of .Sy zfs_dirty_data_max , expressed as a percentage of physical RAM. This limit is only enforced at module load time, and will be ignored if .Sy zfs_dirty_data_max is later changed. The parameter .Sy zfs_dirty_data_max_max takes precedence over this one. .No See Sx ZFS TRANSACTION DELAY . . .It Sy zfs_dirty_data_max_percent Ns = Ns Sy 10 Ns % Pq uint Determines the dirty space limit, expressed as a percentage of all memory. Once this limit is exceeded, new writes are halted until space frees up. The parameter .Sy zfs_dirty_data_max takes precedence over this one. .No See Sx ZFS TRANSACTION DELAY . .Pp Subject to .Sy zfs_dirty_data_max_max . . .It Sy zfs_dirty_data_sync_percent Ns = Ns Sy 20 Ns % Pq uint Start syncing out a transaction group if there's at least this much dirty data .Pq as a percentage of Sy zfs_dirty_data_max . This should be less than .Sy zfs_vdev_async_write_active_min_dirty_percent . . .It Sy zfs_wrlog_data_max Ns = Pq int The upper limit of write-transaction ZIL log data size in bytes. Write operations are throttled when approaching the limit until log data is cleared out after transaction group sync. Because of some overhead, it should be set at least 2 times the size of .Sy zfs_dirty_data_max .No to prevent harming normal write throughput . It also should be smaller than the size of the slog device if slog is present. .Pp Defaults to .Sy zfs_dirty_data_max*2 . .It Sy zfs_fallocate_reserve_percent Ns = Ns Sy 110 Ns % Pq uint Since ZFS is a copy-on-write filesystem with snapshots, blocks cannot be preallocated for a file in order to guarantee that later writes will not run out of space. Instead, .Xr fallocate 2 space preallocation only checks that sufficient space is currently available in the pool or the user's project quota allocation, and then creates a sparse file of the requested size. The requested space is multiplied by .Sy zfs_fallocate_reserve_percent to allow additional space for indirect blocks and other internal metadata. Setting this to .Sy 0 disables support for .Xr fallocate 2 and causes it to return .Sy EOPNOTSUPP . . .It Sy zfs_fletcher_4_impl Ns = Ns Sy fastest Pq string Select a fletcher 4 implementation. .Pp Supported selectors are: .Sy fastest , scalar , sse2 , ssse3 , avx2 , avx512f , avx512bw , .No and Sy aarch64_neon . All except .Sy fastest No and Sy scalar require instruction set extensions to be available, and will only appear if ZFS detects that they are present at runtime. If multiple implementations of fletcher 4 are available, the .Sy fastest will be chosen using a micro benchmark. Selecting .Sy scalar results in the original CPU-based calculation being used. Selecting any option other than .Sy fastest No or Sy scalar results in vector instructions from the respective CPU instruction set being used. . .It Sy zfs_bclone_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int Enables access to the block cloning feature. If this setting is 0, then even if feature@block_cloning is enabled, using functions and system calls that attempt to clone blocks will act as though the feature is disabled. . .It Sy zfs_bclone_strict_properties Ns = Ns Sy 1 Ns | Ns 0 Pq int Restricts block cloning between datasets with different properties (checksum, compression, copies, dedup, or special_small_blocks). . .It Sy zfs_bclone_wait_dirty Ns = Ns Sy 1 Ns | Ns 0 Pq int When set to 1 the FICLONE and FICLONERANGE ioctls will wait for any dirty data to be written to disk before proceeding. This ensures that the clone operation reliably succeeds, even if a file is modified and then immediately cloned. Note that for small files this may be slower than simply copying the file. When set to 0 the clone operation will immediately fail if it encounters any dirty blocks. By default waiting is enabled. A FIDEDUPERANGE request always waits, as it has no copy fallback that could absorb the failure. . .It Sy zfs_blake3_impl Ns = Ns Sy fastest Pq string Select a BLAKE3 implementation. .Pp Supported selectors are: .Sy cycle , fastest , generic , sse2 , sse41 , avx2 , avx512 . All except .Sy cycle , fastest No and Sy generic require instruction set extensions to be available, and will only appear if ZFS detects that they are present at runtime. If multiple implementations of BLAKE3 are available, the .Sy fastest will be chosen using a micro benchmark. You can see the benchmark results by reading this kstat file: .Pa /proc/spl/kstat/zfs/chksum_bench . . .It Sy zfs_free_bpobj_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int Enable/disable the processing of the free_bpobj object. . .It Sy zfs_async_block_max_blocks Ns = Ns Sy UINT64_MAX Po unlimited Pc Pq u64 Maximum number of blocks freed in a single TXG. . .It Sy zfs_max_async_dedup_frees Ns = Ns Sy 250000 Pq u64 Maximum number of dedup, clone or gang blocks freed in a single TXG. These frees may require additional I/O, making them more expensive. . .It Sy zfs_async_free_zio_wait_interval Ns = Ns Sy 2000 Pq u64 After freeing this many dedup, clone or gang blocks wait for all pending I/Os to complete before continuing. . .It Sy zfs_vdev_async_read_max_active Ns = Ns Sy 3 Pq uint Maximum asynchronous read I/O operations active to each device. .No See Sx ZFS I/O SCHEDULER . . .It Sy zfs_vdev_async_read_min_active Ns = Ns Sy 1 Pq uint Minimum asynchronous read I/O operation active to each device. .No See Sx ZFS I/O SCHEDULER . . .It Sy zfs_vdev_async_write_active_max_dirty_percent Ns = Ns Sy 60 Ns % Pq uint When the pool has more than this much dirty data, use .Sy zfs_vdev_async_write_max_active to limit active async writes. If the dirty data is between the minimum and maximum, the active I/O limit is linearly interpolated. .No See Sx ZFS I/O SCHEDULER . . .It Sy zfs_vdev_async_write_active_min_dirty_percent Ns = Ns Sy 30 Ns % Pq uint When the pool has less than this much dirty data, use .Sy zfs_vdev_async_write_min_active to limit active async writes. If the dirty data is between the minimum and maximum, the active I/O limit is linearly interpolated. .No See Sx ZFS I/O SCHEDULER . . .It Sy zfs_vdev_async_write_max_active Ns = Ns Sy 10 Pq uint Maximum asynchronous write I/O operations active to each device. .No See Sx ZFS I/O SCHEDULER . . .It Sy zfs_vdev_async_write_min_active Ns = Ns Sy 2 Pq uint Minimum asynchronous write I/O operations active to each device. .No See Sx ZFS I/O SCHEDULER . .Pp Lower values are associated with better latency on rotational media but poorer resilver performance. The default value of .Sy 2 was chosen as a compromise. A value of .Sy 3 has been shown to improve resilver performance further at a cost of further increasing latency. . .It Sy zfs_vdev_initializing_max_active Ns = Ns Sy 1 Pq uint Maximum initializing I/O operations active to each device. .No See Sx ZFS I/O SCHEDULER . . .It Sy zfs_vdev_initializing_min_active Ns = Ns Sy 1 Pq uint Minimum initializing I/O operations active to each device. .No See Sx ZFS I/O SCHEDULER . . .It Sy zfs_vdev_max_active Ns = Ns Sy 1000 Pq uint The maximum number of I/O operations active to each device. Ideally, this will be at least the sum of each queue's .Sy max_active . .No See Sx ZFS I/O SCHEDULER . . .It Sy zfs_vdev_open_timeout_ms Ns = Ns Sy 1000 Pq uint Timeout value to wait before determining a device is missing during import. This is helpful for transient missing paths due to links being briefly removed and recreated in response to udev events. . .It Sy zfs_vdev_rebuild_max_active Ns = Ns Sy 3 Pq uint Maximum sequential resilver I/O operations active to each device. .No See Sx ZFS I/O SCHEDULER . . .It Sy zfs_vdev_rebuild_min_active Ns = Ns Sy 1 Pq uint Minimum sequential resilver I/O operations active to each device. .No See Sx ZFS I/O SCHEDULER . . .It Sy zfs_vdev_removal_max_active Ns = Ns Sy 2 Pq uint Maximum removal I/O operations active to each device. .No See Sx ZFS I/O SCHEDULER . . .It Sy zfs_vdev_removal_min_active Ns = Ns Sy 1 Pq uint Minimum removal I/O operations active to each device. .No See Sx ZFS I/O SCHEDULER . . .It Sy zfs_vdev_scrub_max_active Ns = Ns Sy 2 Pq uint Maximum scrub I/O operations active to each device. .No See Sx ZFS I/O SCHEDULER . . .It Sy zfs_vdev_scrub_min_active Ns = Ns Sy 1 Pq uint Minimum scrub I/O operations active to each device. .No See Sx ZFS I/O SCHEDULER . . .It Sy zfs_vdev_sync_read_max_active Ns = Ns Sy 10 Pq uint Maximum synchronous read I/O operations active to each device. .No See Sx ZFS I/O SCHEDULER . . .It Sy zfs_vdev_sync_read_min_active Ns = Ns Sy 10 Pq uint Minimum synchronous read I/O operations active to each device. .No See Sx ZFS I/O SCHEDULER . . .It Sy zfs_vdev_sync_write_max_active Ns = Ns Sy 10 Pq uint Maximum synchronous write I/O operations active to each device. .No See Sx ZFS I/O SCHEDULER . . .It Sy zfs_vdev_sync_write_min_active Ns = Ns Sy 10 Pq uint Minimum synchronous write I/O operations active to each device. .No See Sx ZFS I/O SCHEDULER . . .It Sy zfs_vdev_trim_max_active Ns = Ns Sy 2 Pq uint Maximum trim/discard I/O operations active to each device. .No See Sx ZFS I/O SCHEDULER . . .It Sy zfs_vdev_trim_min_active Ns = Ns Sy 1 Pq uint Minimum trim/discard I/O operations active to each device. .No See Sx ZFS I/O SCHEDULER . . .It Sy zfs_vdev_nia_delay Ns = Ns Sy 5 Pq uint For non-interactive I/O (scrub, resilver, removal, initialize and rebuild), the number of concurrently-active I/O operations is limited to .Sy zfs_*_min_active , unless the vdev is "idle". When there are no interactive I/O operations active (synchronous or otherwise), and .Sy zfs_vdev_nia_delay operations have completed since the last interactive operation, then the vdev is considered to be "idle", and the number of concurrently-active non-interactive operations is increased to .Sy zfs_*_max_active . .No See Sx ZFS I/O SCHEDULER . . .It Sy zfs_vdev_nia_credit Ns = Ns Sy 5 Pq uint Some HDDs tend to prioritize sequential I/O so strongly, that concurrent random I/O latency reaches several seconds. On some HDDs this happens even if sequential I/O operations are submitted one at a time, and so setting .Sy zfs_*_max_active Ns = Sy 1 does not help. To prevent non-interactive I/O, like scrub, from monopolizing the device, no more than .Sy zfs_vdev_nia_credit operations can be sent while there are outstanding incomplete interactive operations. This enforced wait ensures the HDD services the interactive I/O within a reasonable amount of time. .No See Sx ZFS I/O SCHEDULER . . .It Sy zfs_vdev_failfast_mask Ns = Ns Sy 1 Pq uint Defines if the driver should retire on a given error type. The following options may be bitwise-ored together: .TS box; lbz r l l . Value Name Description _ 1 Device No driver retries on device errors 2 Transport No driver retries on transport errors. 4 Driver No driver retries on driver errors. .TE . .It Sy zfs_vdev_disk_max_segs Ns = Ns Sy 0 Pq uint Maximum number of segments to add to a BIO (min 4). If this is higher than the maximum allowed by the device queue or the kernel itself, it will be clamped. Setting it to zero will cause the kernel's ideal size to be used. This parameter only applies on Linux. . .It Sy zfs_vdev_disk_calling_thread_io Ns = Ns Sy 0 Ns | Ns 1 Pq uint Controls calling thread io, note that we only wait for the zio to complete if it bypassed the vdev queue, all this module parameter does is enable that capability. May lead to performance improvements when enabled if backing vdev devices are fast and low latency. May impact performance with certain workloads when enabled on raidz or draid zpool configurations. This parameter currently only applies on Linux. . .It Sy zfs_expire_snapshot Ns = Ns Sy 300 Ns s Pq int Time before expiring .Pa .zfs/snapshot . . .It Sy zfs_admin_snapshot Ns = Ns Sy 0 Ns | Ns 1 Pq int Allow the creation, removal, or renaming of entries in the .Sy .zfs/snapshot directory to cause the creation, destruction, or renaming of snapshots. When enabled, this functionality works both locally and over NFS exports which have the .Em no_root_squash option set. . .It Sy zfs_snapshot_history_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int Whether snapshot creation and destruction events are recorded in the pool history log, viewable with .Nm zpool Cm history . . .It Sy zfs_snapshot_no_setuid Ns = Ns Sy 0 Ns | Ns 1 Pq int Whether to disable .Em setuid/setgid support for snapshot mounts triggered by access to the .Sy .zfs/snapshot directory by setting the .Em nosuid mount option. . .It Sy zfs_flags Ns = Ns Sy 0 Pq int Set additional debugging flags. The following flags may be bitwise-ored together: .TS box; lbz r l l . Value Name Description _ 1 ZFS_DEBUG_DPRINTF Enable dprintf entries in the debug log. * 2 ZFS_DEBUG_DBUF_VERIFY Enable extra dbuf verifications. * 4 ZFS_DEBUG_DNODE_VERIFY Enable extra dnode verifications. 8 ZFS_DEBUG_SNAPNAMES Enable snapshot name verification. * 16 ZFS_DEBUG_MODIFY Check for illegally modified ARC buffers. 64 ZFS_DEBUG_ZIO_FREE Enable verification of block frees. 128 ZFS_DEBUG_HISTOGRAM_VERIFY Enable extra spacemap histogram verifications. 256 ZFS_DEBUG_METASLAB_VERIFY Verify space accounting on disk matches in-memory \fBrange_trees\fP. 512 ZFS_DEBUG_SET_ERROR Enable \fBSET_ERROR\fP and dprintf entries in the debug log. 1024 ZFS_DEBUG_INDIRECT_REMAP Verify split blocks created by device removal. 2048 ZFS_DEBUG_TRIM Verify TRIM ranges are always within the allocatable range tree. 4096 ZFS_DEBUG_LOG_SPACEMAP Verify that the log summary is consistent with the spacemap log and enable \fBzfs_dbgmsgs\fP for metaslab loading and flushing. 8192 ZFS_DEBUG_METASLAB_ALLOC Enable debugging messages when allocations fail. 16384 ZFS_DEBUG_BRT Enable BRT-related debugging messages. 32768 ZFS_DEBUG_RAIDZ_RECONSTRUCT Enabled debugging messages for raidz reconstruction. 65536 ZFS_DEBUG_DDT Enable DDT-related debugging messages. .TE .Sy \& * No Requires debug build . . .It Sy zfs_btree_verify_intensity Ns = Ns Sy 0 Pq uint Enables btree verification. The following settings are cumulative: .TS box; lbz r l l . Value Description 1 Verify height. 2 Verify pointers from children to parent. 3 Verify element counts. 4 Verify element order. (expensive) * 5 Verify unused memory is poisoned. (expensive) .TE .Sy \& * No Requires debug build . . .It Sy zfs_free_leak_on_eio Ns = Ns Sy 0 Ns | Ns 1 Pq int If destroy encounters an .Sy EIO while reading metadata (e.g. indirect blocks), space referenced by the missing metadata can not be freed. Normally this causes the background destroy to become "stalled", as it is unable to make forward progress. While in this stalled state, all remaining space to free from the error-encountering filesystem is "temporarily leaked". Set this flag to cause it to ignore the .Sy EIO , permanently leak the space from indirect blocks that can not be read, and continue to free everything else that it can. .Pp The default "stalling" behavior is useful if the storage partially fails (i.e. some but not all I/O operations fail), and then later recovers. In this case, we will be able to continue pool operations while it is partially failed, and when it recovers, we can continue to free the space, with no leaks. Note, however, that this case is actually fairly rare. .Pp Typically pools either .Bl -enum -compact -offset 4n -width "1." .It fail completely (but perhaps temporarily, e.g. due to a top-level vdev going offline), or .It have localized, permanent errors (e.g. disk returns the wrong data due to bit flip or firmware bug). .El In the former case, this setting does not matter because the pool will be suspended and the sync thread will not be able to make forward progress regardless. In the latter, because the error is permanent, the best we can do is leak the minimum amount of space, which is what setting this flag will do. It is therefore reasonable for this flag to normally be set, but we chose the more conservative approach of not setting it, so that there is no possibility of leaking space in the "partial temporary" failure case. . .It Sy zfs_free_min_time_ms Ns = Ns Sy 500 Ns ms Po 1s Pc Pq uint During a .Nm zfs Cm destroy operation using the .Sy async_destroy feature, a minimum of this much time will be spent working on freeing blocks per TXG. . .It Sy zfs_obsolete_min_time_ms Ns = Ns Sy 500 Ns ms Pq uint Similar to .Sy zfs_free_min_time_ms , but for cleanup of old indirection records for removed vdevs. . .It Sy zfs_immediate_write_sz Ns = Ns Sy 32768 Ns B Po 32 KiB Pc Pq s64 Largest write size to store the data directly into the ZIL if .Sy logbias Ns = Ns Sy latency . Larger writes may be written indirectly similar to .Sy logbias Ns = Ns Sy throughput . In presence of SLOG this parameter is ignored, as if it was set to infinity, storing all written data into ZIL to not depend on regular vdev latency. . .It Sy zil_special_is_slog Ns = Ns Sy 1 Ns | Ns 0 Pq int When enabled, and written blocks go to normal vdevs, treat present special vdevs as SLOGs. Blocks that go to the special vdevs are still written indirectly, as with .Sy logbias Ns = Ns Sy throughput . This parameter is ignored if an SLOG is present. . .It Sy zfs_import_defer_txgs Ns = Ns Sy 5 Pq uint Number of transaction groups to wait after pool import before starting background work such as asynchronous block freeing .Pq from snapshots, clones, and deduplication and scrub or resilver operations. This allows the pool import and filesystem mounting to complete more quickly without interference from background activities. The default value of 5 transaction groups typically provides sufficient time for import and mount operations to complete on most systems. . .It Sy zfs_initialize_value Ns = Ns Sy 16045690984833335022 Po 0xDEADBEEFDEADBEEE Pc Pq u64 Default pattern written to vdev free space by .Xr zpool-initialize 8 , used unless .Nm zpool Cm initialize Fl z requests zeroes instead. The value in effect when a device starts initializing is recorded per device and reused if the run is suspended and later resumed. . .It Sy zfs_initialize_chunk_size Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq u64 Size of writes used by .Xr zpool-initialize 8 . This option is used by the test suite. . .It Sy zfs_livelist_max_entries Ns = Ns Sy 500000 Po 5*10^5 Pc Pq u64 The threshold size (in block pointers) at which we create a new sub-livelist. Larger sublists are more costly from a memory perspective but the fewer sublists there are, the lower the cost of insertion. . .It Sy zfs_livelist_min_percent_shared Ns = Ns Sy 75 Ns % Pq int If the amount of shared space between a snapshot and its clone drops below this threshold, the clone turns off the livelist and reverts to the old deletion method. This is in place because livelists no long give us a benefit once a clone has been overwritten enough. . .It Sy zfs_livelist_condense_new_alloc Ns = Ns Sy 0 Pq int Incremented each time an extra ALLOC blkptr is added to a livelist entry while it is being condensed. This option is used by the test suite to track race conditions. . .It Sy zfs_livelist_condense_sync_cancel Ns = Ns Sy 0 Pq int Incremented each time livelist condensing is canceled while in .Fn spa_livelist_condense_sync . This option is used by the test suite to track race conditions. . .It Sy zfs_livelist_condense_sync_pause Ns = Ns Sy 0 Ns | Ns 1 Pq int When set, the livelist condense process pauses indefinitely before executing the synctask \(em .Fn spa_livelist_condense_sync . This option is used by the test suite to trigger race conditions. . .It Sy zfs_livelist_condense_zthr_cancel Ns = Ns Sy 0 Pq int Incremented each time livelist condensing is canceled while in .Fn spa_livelist_condense_cb . This option is used by the test suite to track race conditions. . .It Sy zfs_livelist_condense_zthr_pause Ns = Ns Sy 0 Ns | Ns 1 Pq int When set, the livelist condense process pauses indefinitely before executing the open context condensing work in .Fn spa_livelist_condense_cb . This option is used by the test suite to trigger race conditions. . .It Sy zfs_lua_max_instrlimit Ns = Ns Sy 100000000 Po 10^8 Pc Pq u64 The maximum execution time limit that can be set for a ZFS channel program, specified as a number of Lua instructions. . .It Sy zfs_lua_max_memlimit Ns = Ns Sy 104857600 Po 100 MiB Pc Pq u64 The maximum memory limit that can be set for a ZFS channel program, specified in bytes. . .It Sy zfs_max_dataset_nesting Ns = Ns Sy 50 Pq int The maximum depth of nested datasets. This value can be tuned temporarily to fix existing datasets that exceed the predefined limit. . .It Sy zfs_max_log_walking Ns = Ns Sy 5 Pq u64 The number of past TXGs that the flushing algorithm of the log spacemap feature uses to estimate incoming log blocks. . .It Sy zfs_max_logsm_summary_length Ns = Ns Sy 10 Pq u64 Maximum number of rows allowed in the summary of the spacemap log. . .It Sy zfs_max_recordsize Ns = Ns Sy 16777216 Po 16 MiB Pc Pq uint We currently support block sizes from .Em 512 Po 512 B Pc No to Em 16777216 Po 16 MiB Pc . The benefits of larger blocks, and thus larger I/O, need to be weighed against the cost of COWing a giant block to modify one byte. Additionally, very large blocks can have an impact on I/O latency, and also potentially on the memory allocator. Therefore, we formerly forbade creating blocks larger than 1M. Larger blocks could be created by changing it, and pools with larger blocks can always be imported and used, regardless of this setting. .Pp Note that it is still limited by default to .Ar 1 MiB on x86_32, because Linux's 3/1 memory split doesn't leave much room for 16M chunks. . .It Sy zfs_allow_redacted_dataset_mount Ns = Ns Sy 0 Ns | Ns 1 Pq int Allow datasets received with redacted send/receive to be mounted. Normally disabled because these datasets may be missing key data. . .It Sy zfs_min_metaslabs_to_flush Ns = Ns Sy 1 Pq u64 Minimum number of metaslabs to flush per dirty TXG. . .It Sy zfs_min_metaslabs_to_condense_pct Ns = Ns Sy 5 Pq u64 Minimum number of metaslabs to flush every TXG when condensing log spacemaps, as a percentage of the number of unflushed metaslabs at the time the .Nm zpool Cm condense command was run. . .It Sy zfs_metaslab_fragmentation_threshold Ns = Ns Sy 77 Ns % Pq uint Allow metaslabs to keep their active state as long as their fragmentation percentage is no more than this value. An active metaslab that exceeds this threshold will no longer keep its active status allowing better metaslabs to be selected. . .It Sy zfs_mg_fragmentation_threshold Ns = Ns Sy 95 Ns % Pq uint Metaslab groups are considered eligible for allocations if their fragmentation metric (measured as a percentage) is less than or equal to this value. If a metaslab group exceeds this threshold then it will be skipped unless all metaslab groups within the metaslab class have also crossed this threshold. . .It Sy zfs_mg_noalloc_threshold Ns = Ns Sy 0 Ns % Pq uint Defines a threshold at which metaslab groups should be eligible for allocations. The value is expressed as a percentage of free space beyond which a metaslab group is always eligible for allocations. If a metaslab group's free space is less than or equal to the threshold, the allocator will avoid allocating to that group unless all groups in the pool have reached the threshold. Once all groups have reached the threshold, all groups are allowed to accept allocations. The default value of .Sy 0 disables the feature and causes all metaslab groups to be eligible for allocations. .Pp This parameter allows one to deal with pools having heavily imbalanced vdevs such as would be the case when a new vdev has been added. Setting the threshold to a non-zero percentage will stop allocations from being made to vdevs that aren't filled to the specified percentage and allow lesser filled vdevs to acquire more allocations than they otherwise would under the old .Sy zfs_mg_alloc_failures facility. . .It Sy zfs_ddt_data_is_special Ns = Ns Sy 1 Ns | Ns 0 Pq int If enabled, ZFS will place DDT data into the special allocation class. . .It Sy zfs_user_indirect_is_special Ns = Ns Sy 1 Ns | Ns 0 Pq int If enabled, ZFS will place user data indirect blocks into the special allocation class. . .It Sy zfs_multihost_history Ns = Ns Sy 0 Pq uint Historical statistics for this many latest multihost updates will be available in .Pa /proc/spl/kstat/zfs/ Ns Ao Ar pool Ac Ns Pa /multihost . . .It Sy zfs_multihost_interval Ns = Ns Sy 1000 Ns ms Po 1 s Pc Pq u64 Used to control the frequency of multihost writes which are performed when the .Sy multihost pool property is on. This is one of the factors used to determine the length of the activity check during import. .Pp The multihost write period is .Sy zfs_multihost_interval No / Sy leaf-vdevs . On average a multihost write will be issued for each leaf vdev every .Sy zfs_multihost_interval milliseconds. In practice, the observed period can vary with the I/O load and this observed value is the delay which is stored in the uberblock. . .It Sy zfs_multihost_import_intervals Ns = Ns Sy 20 Pq uint Used to control the duration of the activity test on import. Smaller values of .Sy zfs_multihost_import_intervals will reduce the import time but increase the risk of failing to detect an active pool. The total activity check time is never allowed to drop below one second. .Pp On import the activity check waits a minimum amount of time determined by .Sy zfs_multihost_interval No \(mu Sy zfs_multihost_import_intervals , or the same product computed on the host which last had the pool imported, whichever is greater. The activity check time may be further extended if the value of MMP delay found in the best uberblock indicates actual multihost updates happened at longer intervals than .Sy zfs_multihost_interval . A minimum of .Em 100 ms is enforced. .Pp .Sy 0 No is equivalent to Sy 1 . . .It Sy zfs_multihost_fail_intervals Ns = Ns Sy 10 Pq uint Controls the behavior of the pool when multihost write failures or delays are detected. .Pp When .Sy 0 , multihost write failures or delays are ignored. The failures will still be reported to the ZED which depending on its configuration may take action such as suspending the pool or offlining a device. .Pp Otherwise, the pool will be suspended if .Sy zfs_multihost_fail_intervals No \(mu Sy zfs_multihost_interval milliseconds pass without a successful MMP write. This guarantees the activity test will see MMP writes if the pool is imported. .Sy 1 No is equivalent to Sy 2 ; this is necessary to prevent the pool from being suspended due to normal, small I/O latency variations. . .It Sy zfs_no_scrub_io Ns = Ns Sy 0 Ns | Ns 1 Pq int Set to disable scrub I/O. This results in scrubs not actually scrubbing data and simply doing a metadata crawl of the pool instead. . .It Sy zfs_no_scrub_prefetch Ns = Ns Sy 0 Ns | Ns 1 Pq int Set to disable block prefetching for scrubs. . .It Sy zfs_nocacheflush Ns = Ns Sy 0 Ns | Ns 1 Pq int Disable cache flush operations on disks when writing. Setting this will cause pool corruption on power loss if a volatile out-of-order write cache is enabled. . .It Sy zfs_nopwrite_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int Allow no-operation writes. The occurrence of nopwrites will further depend on other pool properties .Pq i.a. the checksumming and compression algorithms . . .It Sy zfs_dmu_offset_next_sync Ns = Ns Sy 1 Ns | Ns 0 Pq int Enable forcing TXG sync to find holes. When enabled forces ZFS to sync data when .Sy SEEK_HOLE No or Sy SEEK_DATA flags are used allowing holes in a file to be accurately reported. When disabled holes will not be reported in recently dirtied files. . .It Sy zfs_pd_bytes_max Ns = Ns Sy 52428800 Ns B Po 50 MiB Pc Pq int The number of bytes which should be prefetched during a pool traversal, like .Nm zfs Cm send or other data crawling operations. . .It Sy zfs_traverse_indirect_prefetch_limit Ns = Ns Sy 32 Pq uint The number of blocks pointed by indirect (non-L0) block which should be prefetched during a pool traversal, like .Nm zfs Cm send or other data crawling operations. . .It Sy zfs_per_txg_dirty_frees_percent Ns = Ns Sy 30 Ns % Pq u64 Control percentage of dirtied indirect blocks from frees allowed into one TXG. After this threshold is crossed, additional frees will wait until the next TXG. .Sy 0 No disables this throttle . . .It Sy zfs_prefetch_disable Ns = Ns Sy 0 Ns | Ns 1 Pq int Disable predictive prefetch. Note that it leaves "prescient" prefetch .Pq for, e.g., Nm zfs Cm send intact. Unlike predictive prefetch, prescient prefetch never issues I/O that ends up not being needed, so it can't hurt performance. . .It Sy zfs_qat_checksum_disable Ns = Ns Sy 0 Ns | Ns 1 Pq int Disable QAT hardware acceleration for SHA256 checksums. May be unset after the ZFS modules have been loaded to initialize the QAT hardware as long as support is compiled in and the QAT driver is present. . .It Sy zfs_qat_compress_disable Ns = Ns Sy 0 Ns | Ns 1 Pq int Disable QAT hardware acceleration for gzip compression. May be unset after the ZFS modules have been loaded to initialize the QAT hardware as long as support is compiled in and the QAT driver is present. . .It Sy zfs_qat_encrypt_disable Ns = Ns Sy 0 Ns | Ns 1 Pq int Disable QAT hardware acceleration for AES-GCM encryption. May be unset after the ZFS modules have been loaded to initialize the QAT hardware as long as support is compiled in and the QAT driver is present. . .It Sy zfs_vnops_read_chunk_size Ns = Ns Sy 33554432 Ns B Po 32 MiB Pc Pq u64 Bytes to read per chunk. . .It Sy zfs_read_history Ns = Ns Sy 0 Pq uint Historical statistics for this many latest reads will be available in .Pa /proc/spl/kstat/zfs/ Ns Ao Ar pool Ac Ns Pa /reads . . .It Sy zfs_read_history_hits Ns = Ns Sy 0 Ns | Ns 1 Pq int Include cache hits in read history . .It Sy zfs_rebuild_max_segment Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq u64 Maximum read segment size to issue when sequentially resilvering a top-level vdev. . .It Sy zfs_rebuild_scrub_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int Automatically start a pool scrub when the last active sequential resilver completes in order to verify the checksums of all blocks which have been resilvered. This is enabled by default and strongly recommended. . .It Sy zfs_rebuild_vdev_limit Ns = Ns Sy 67108864 Ns B Po 64 MiB Pc Pq u64 Maximum amount of I/O that can be concurrently issued for a sequential resilver per leaf device, given in bytes. . .It Sy zfs_reconstruct_indirect_combinations_max Ns = Ns Sy 4096 Pq int If an indirect split block contains more than this many possible unique combinations when being reconstructed, consider it too computationally expensive to check them all. Instead, try at most this many randomly selected combinations each time the block is accessed. This allows all segment copies to participate fairly in the reconstruction when all combinations cannot be checked and prevents repeated use of one bad copy. . .It Sy zfs_recover Ns = Ns Sy 0 Ns | Ns 1 Pq int Set to attempt to recover from fatal errors. This should only be used as a last resort, as it typically results in leaked space, or worse. . .It Sy zfs_removal_ignore_errors Ns = Ns Sy 0 Ns | Ns 1 Pq int Ignore hard I/O errors during device removal. When set, if a device encounters a hard I/O error during the removal process the removal will not be canceled. This can result in a normally recoverable block becoming permanently damaged and is hence not recommended. This should only be used as a last resort when the pool cannot be returned to a healthy state prior to removing the device. . .It Sy zfs_removal_suspend_progress Ns = Ns Sy 0 Ns | Ns 1 Pq uint This is used by the test suite so that it can ensure that certain actions happen while in the middle of a removal. . .It Sy zfs_remove_max_segment Ns = Ns Sy 16777216 Ns B Po 16 MiB Pc Pq uint The largest contiguous segment that we will attempt to allocate when removing a device. If there is a performance problem with attempting to allocate large blocks, consider decreasing this. The default value is also the maximum. . .It Sy zfs_resilver_disable_defer Ns = Ns Sy 0 Ns | Ns 1 Pq int Ignore the .Sy resilver_defer feature, causing an operation that would start a resilver to immediately restart the one in progress. . .It Sy zfs_resilver_defer_percent Ns = Ns Sy 10 Ns % Pq uint If the ongoing resilver progress is below this threshold, a new resilver will restart from scratch instead of being deferred after the current one finishes, even if the .Sy resilver_defer feature is enabled. . .It Sy zfs_resilver_min_time_ms Ns = Ns Sy 1500 Ns ms Pq uint Resilvers are processed by the sync thread. While resilvering, it will spend at least this much time working on a resilver between TXG flushes. . .It Sy zfs_scan_ignore_errors Ns = Ns Sy 0 Ns | Ns 1 Pq int If set, remove the DTL (dirty time list) upon completion of a pool scan (scrub), even if there were unrepairable errors. Intended to be used during pool repair or recovery to stop resilvering when the pool is next imported. . .It Sy zfs_scrub_after_expand Ns = Ns Sy 1 Ns | Ns 0 Pq int Automatically start a pool scrub after a RAIDZ expansion completes in order to verify the checksums of all blocks which have been copied during the expansion. This is enabled by default and strongly recommended. . .It Sy zfs_scrub_min_time_ms Ns = Ns Sy 750 Ns ms Pq uint Scrubs are processed by the sync thread. While scrubbing, it will spend at least this much time working on a scrub between TXG flushes. . .It Sy zfs_scrub_error_blocks_per_txg Ns = Ns Sy 4096 Pq uint Error blocks to be scrubbed in one txg. . .It Sy zfs_scan_blkstats Ns = Ns Sy 0 Ns | Ns 1 Pq int When enabled, ZFS counts blocks by type and indirection level during a scrub. The counts are kept in memory for debugging and are not exposed to userspace. . .It Sy zfs_scan_checkpoint_intval Ns = Ns Sy 7200 Ns s Po 2 hours Pc Pq uint To preserve progress across reboots, the sequential scan algorithm periodically needs to stop metadata scanning and issue all the verification I/O to disk. The frequency of this flushing is determined by this tunable. . .It Sy zfs_scan_fill_weight Ns = Ns Sy 3 Pq uint This tunable affects how scrub and resilver I/O segments are ordered. A higher number indicates that we care more about how filled in a segment is, while a lower number indicates we care more about the size of the extent without considering the gaps within a segment. This value is only tunable upon module insertion. Changing the value afterwards will have no effect on scrub or resilver performance. . .It Sy zfs_scan_issue_strategy Ns = Ns Sy 0 Pq uint Determines the order that data will be verified while scrubbing or resilvering: .Bl -tag -compact -offset 4n -width "a" .It Sy 1 Data will be verified as sequentially as possible, given the amount of memory reserved for scrubbing .Pq see Sy zfs_scan_mem_lim_fact . This may improve scrub performance if the pool's data is very fragmented. .It Sy 2 The largest mostly-contiguous chunk of found data will be verified first. By deferring scrubbing of small segments, we may later find adjacent data to coalesce and increase the segment size. .It Sy 0 .No Use strategy Sy 1 No during normal verification .No and strategy Sy 2 No while taking a checkpoint . .El . .It Sy zfs_scan_legacy Ns = Ns Sy 0 Ns | Ns 1 Pq int If unset, indicates that scrubs and resilvers will gather metadata in memory before issuing sequential I/O. Otherwise indicates that the legacy algorithm will be used, where I/O is initiated as soon as it is discovered. Unsetting will not affect scrubs or resilvers that are already in progress. . .It Sy zfs_scan_max_ext_gap Ns = Ns Sy 2097152 Ns B Po 2 MiB Pc Pq int Sets the largest gap in bytes between scrub/resilver I/O operations that will still be considered sequential for sorting purposes. Changing this value will not affect scrubs or resilvers that are already in progress. . .It Sy zfs_scan_mem_lim_fact Ns = Ns Sy 20 Ns ^-1 Pq uint Maximum fraction of RAM used for I/O sorting by sequential scan algorithm. This tunable determines the hard limit for I/O sorting memory usage. When the hard limit is reached we stop scanning metadata and start issuing data verification I/O. This is done until we get below the soft limit. . .It Sy zfs_scan_mem_lim_soft_fact Ns = Ns Sy 20 Ns ^-1 Pq uint The fraction of the hard limit used to determined the soft limit for I/O sorting by the sequential scan algorithm. When we cross this limit from below no action is taken. When we cross this limit from above it is because we are issuing verification I/O. In this case (unless the metadata scan is done) we stop issuing verification I/O and start scanning metadata again until we get to the hard limit. . .It Sy zfs_scan_report_txgs Ns = Ns Sy 0 Ns | Ns 1 Pq uint When reporting resilver throughput and estimated completion time use the performance observed over roughly the last .Sy zfs_scan_report_txgs TXGs. When set to zero performance is calculated over the time between checkpoints. . .It Sy zfs_scan_strict_mem_lim Ns = Ns Sy 0 Ns | Ns 1 Pq int Enforce tight memory limits on pool scans when a sequential scan is in progress. When disabled, the memory limit may be exceeded by fast disks. . .It Sy zfs_scan_suspend_progress Ns = Ns Sy 0 Ns | Ns 1 Pq int Freezes a scrub/resilver in progress without actually pausing it. Intended for testing/debugging. . .It Sy zfs_scan_vdev_limit Ns = Ns Sy 16777216 Ns B Po 16 MiB Pc Pq int Maximum amount of data that can be concurrently issued at once for scrubs and resilvers per leaf device, given in bytes. . .It Sy zfs_send_corrupt_data Ns = Ns Sy 0 Ns | Ns 1 Pq int Allow sending of corrupt data (ignore read/checksum errors when sending). . .It Sy zfs_send_unmodified_spill_blocks Ns = Ns Sy 1 Ns | Ns 0 Pq int Include unmodified spill blocks in the send stream. Under certain circumstances, previous versions of ZFS could incorrectly remove the spill block from an existing object. Including unmodified copies of the spill blocks creates a backwards-compatible stream which will recreate a spill block if it was incorrectly removed. . .It Sy zfs_send_no_prefetch_queue_ff Ns = Ns Sy 20 Ns ^\-1 Pq uint The fill fraction of the .Nm zfs Cm send internal queues. The fill fraction controls the timing with which internal threads are woken up. . .It Sy zfs_send_no_prefetch_queue_length Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq uint The maximum number of bytes allowed in .Nm zfs Cm send Ns 's internal queues. . .It Sy zfs_send_queue_ff Ns = Ns Sy 20 Ns ^\-1 Pq uint The fill fraction of the .Nm zfs Cm send prefetch queue. The fill fraction controls the timing with which internal threads are woken up. . .It Sy zfs_send_queue_length Ns = Ns Sy 16777216 Ns B Po 16 MiB Pc Pq uint The maximum number of bytes allowed that will be prefetched by .Nm zfs Cm send . This value must be at least twice the maximum block size in use. . .It Sy zfs_recv_queue_ff Ns = Ns Sy 20 Ns ^\-1 Pq uint The fill fraction of the .Nm zfs Cm receive queue. The fill fraction controls the timing with which internal threads are woken up. . .It Sy zfs_recv_queue_length Ns = Ns Sy 16777216 Ns B Po 16 MiB Pc Pq uint The maximum number of bytes allowed in the .Nm zfs Cm receive queue. This value must be at least twice the maximum block size in use. . .It Sy zfs_recv_write_batch_size Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq uint The maximum amount of data, in bytes, that .Nm zfs Cm receive will write in one DMU transaction. This is the uncompressed size, even when receiving a compressed send stream. This setting will not reduce the write size below a single block. Capped at a maximum of .Sy 32 MiB . . .It Sy zfs_recv_defer_batch_size Ns = Ns Sy 33554432 Ns B Po 32 MiB Pc Pq uint When a non-raw incremental stream reallocates an object at a different dnode size, .Nm zfs Cm receive must wait for the old dnode's free to be written out before the object number can be reused. Rather than forcing one transaction group sync per reallocated object, affected records are parked, up to this many bytes at a time, and applied together behind a single sync. Parked records hold their payloads in memory, so a receive may use up to this much memory in addition to .Sy zfs_recv_queue_length . While records are parked, the saved resume state is held back at the first parked record, so interrupting a receive can require resending up to this much of the stream. Setting this to .Sy 0 restores the previous behavior of one sync per reallocated object. . .It Sy zfs_recv_best_effort_corrective Ns = Ns Sy 0 Pq int When this variable is set to non-zero a corrective receive: .Bl -enum -compact -offset 4n -width "1." .It Does not enforce the restriction of source & destination snapshot GUIDs matching. .It If there is an error during healing, the healing receive is not terminated instead it moves on to the next record. .El . .It Sy zfs_override_estimate_recordsize Ns = Ns Sy 0 Ns | Ns 1 Pq uint Setting this variable overrides the default logic for estimating block sizes when doing a .Nm zfs Cm send . The default heuristic is that the average block size will be the current recordsize. Override this value if most data in your dataset is not of that size and you require accurate zfs send size estimates. . .It Sy zfs_sync_pass_deferred_free Ns = Ns Sy 2 Pq uint Flushing of data to disk is done in passes. Defer frees starting in this pass. . .It Sy zfs_spa_discard_memory_limit Ns = Ns Sy 16777216 Ns B Po 16 MiB Pc Pq int Maximum memory used for prefetching a checkpoint's space map on each vdev while discarding the checkpoint. . .It Sy zfs_spa_note_txg_time Ns = Ns Sy 600 Pq uint This parameter defines, in seconds, how often the TXG time database will record a new TXG if it has changed. After the specified time interval has passed, and if the TXG number has changed, the new value is recorded in the database. These timestamps can later be used for more granular operations, such as scrubbing. . .It Sy zfs_spa_flush_txg_time Ns = Ns Sy 600 Pq uint This parameter defines, in seconds, how often the ZFS will flush the TXG time database to disk. It ensures that the data is actually written to persistent storage, which helps preserve the database in case of unexpected shutdown. The database is also automatically flushed during the export sequence. . .It Sy zfs_special_class_metadata_reserve_pct Ns = Ns Sy 25 Ns % Pq uint Only allow small data blocks to be allocated on the special and dedup vdev types when the available free space percentage on these vdevs exceeds this value. This ensures reserved space is available for pool metadata as the special vdevs approach capacity. . .It Sy zfs_sync_pass_dont_compress Ns = Ns Sy 8 Pq uint Starting in this sync pass, disable compression (including of metadata). With the default setting, in practice, we don't have this many sync passes, so this has no effect. .Pp The original intent was that disabling compression would help the sync passes to converge. However, in practice, disabling compression increases the average number of sync passes; because when we turn compression off, many blocks' size will change, and thus we have to re-allocate (not overwrite) them. It also increases the number of .Em 128 KiB allocations (e.g. for indirect blocks and spacemaps) because these will not be compressed. The .Em 128 KiB allocations are especially detrimental to performance on highly fragmented systems, which may have very few free segments of this size, and may need to load new metaslabs to satisfy these allocations. . .It Sy zfs_sync_pass_rewrite Ns = Ns Sy 2 Pq uint Rewrite new block pointers starting in this pass. . .It Sy zfs_scrub_partial_writes Ns = Ns Sy 1 Ns | Ns 0 Pq int If a write to a multi-disk vdev fails, but the data is recoverable, the data is persisted on disk but may not be as redundant as the vdev usually ensures. If this tunable is set, we issue a read after such a write error to detect the full extent of the problem and attempt to recover from it. Note: This currently only works with RAID-Z and dRAID. . .It Sy zfs_trim_extent_bytes_max Ns = Ns Sy 134217728 Ns B Po 128 MiB Pc Pq uint Maximum size of TRIM command. Larger ranges will be split into chunks no larger than this value before issuing. . .It Sy zfs_trim_extent_bytes_min Ns = Ns Sy 32768 Ns B Po 32 KiB Pc Pq uint Minimum size of TRIM commands. TRIM ranges smaller than this will be skipped, unless they're part of a larger range which was chunked. This is done because it's common for these small TRIMs to negatively impact overall performance. . .It Sy zfs_trim_metaslab_skip Ns = Ns Sy 0 Ns | Ns 1 Pq uint Skip uninitialized metaslabs during the TRIM process. This option is useful for pools constructed from large thinly-provisioned devices where TRIM operations are slow. As a pool ages, an increasing fraction of the pool's metaslabs will be initialized, progressively degrading the usefulness of this option. This setting is stored when starting a manual TRIM and will persist for the duration of the requested TRIM. . .It Sy zfs_trim_queue_limit Ns = Ns Sy 10 Pq uint Maximum number of queued TRIMs outstanding per leaf vdev. The number of concurrent TRIM commands issued to the device is controlled by .Sy zfs_vdev_trim_min_active No and Sy zfs_vdev_trim_max_active . . .It Sy zfs_trim_txg_batch Ns = Ns Sy 32 Pq uint The number of transaction groups' worth of frees which should be aggregated before TRIM operations are issued to the device. This setting represents a trade-off between issuing larger, more efficient TRIM operations and the delay before the recently trimmed space is available for use by the device. .Pp Increasing this value will allow frees to be aggregated for a longer time. This will result is larger TRIM operations and potentially increased memory usage. Decreasing this value will have the opposite effect. The default of .Sy 32 was determined to be a reasonable compromise. . .It Sy zfs_txg_history Ns = Ns Sy 100 Pq uint Historical statistics for this many latest TXGs will be available in .Pa /proc/spl/kstat/zfs/ Ns Ao Ar pool Ac Ns Pa /TXGs . . .It Sy zfs_txg_timeout Ns = Ns Sy 5 Ns s Pq uint Flush dirty data to disk at least every this many seconds (maximum TXG duration). . .It Sy zfs_vdev_aggregation_limit Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq uint Max vdev I/O aggregation size. . .It Sy zfs_vdev_aggregation_limit_non_rotating Ns = Ns Sy 131072 Ns B Po 128 KiB Pc Pq uint Max vdev I/O aggregation size for non-rotating media. . .It Sy zfs_vdev_mirror_rotating_inc Ns = Ns Sy 0 Pq int A number by which the balancing algorithm increments the load calculation for the purpose of selecting the least busy mirror member when an I/O operation immediately follows its predecessor on rotational vdevs for the purpose of making decisions based on load. . .It Sy zfs_vdev_mirror_rotating_seek_inc Ns = Ns Sy 5 Pq int A number by which the balancing algorithm increments the load calculation for the purpose of selecting the least busy mirror member when an I/O operation lacks locality as defined by .Sy zfs_vdev_mirror_rotating_seek_offset . Operations within this that are not immediately following the previous operation are incremented by half. . .It Sy zfs_vdev_mirror_rotating_seek_offset Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq int The maximum distance for the last queued I/O operation in which the balancing algorithm considers an operation to have locality. .No See Sx ZFS I/O SCHEDULER . . .It Sy zfs_vdev_mirror_non_rotating_inc Ns = Ns Sy 0 Pq int A number by which the balancing algorithm increments the load calculation for the purpose of selecting the least busy mirror member on non-rotational vdevs when I/O operations do not immediately follow one another. . .It Sy zfs_vdev_mirror_non_rotating_seek_inc Ns = Ns Sy 1 Pq int A number by which the balancing algorithm increments the load calculation for the purpose of selecting the least busy mirror member when an I/O operation lacks locality as defined by the .Sy zfs_vdev_mirror_rotating_seek_offset . Operations within this that are not immediately following the previous operation are incremented by half. . .It Sy zfs_vdev_read_gap_limit Ns = Ns Sy 32768 Ns B Po 32 KiB Pc Pq uint Aggregate read I/O operations if the on-disk gap between them is within this threshold. . .It Sy zfs_vdev_write_gap_limit Ns = Ns Sy 4096 Ns B Po 4 KiB Pc Pq uint Aggregate write I/O operations if the on-disk gap between them is within this threshold. . .It Sy zfs_vdev_raidz_impl Ns = Ns Sy fastest Pq string Select the raidz parity implementation to use. .Pp Variants that don't depend on CPU-specific features may be selected on module load, as they are supported on all systems. The remaining options may only be set after the module is loaded, as they are available only if the implementations are compiled in and supported on the running system. .Pp Once the module is loaded, .Pa /sys/module/zfs/parameters/zfs_vdev_raidz_impl will show the available options, with the currently selected one enclosed in square brackets. .Pp .TS lb l l . fastest selected by built-in benchmark original original implementation scalar scalar implementation sse2 SSE2 instruction set 64-bit x86 ssse3 SSSE3 instruction set 64-bit x86 avx2 AVX2 instruction set 64-bit x86 avx512f AVX512F instruction set 64-bit x86 avx512bw AVX512F & AVX512BW instruction sets 64-bit x86 aarch64_neon NEON Aarch64/64-bit ARMv8 aarch64_neonx2 NEON with more unrolling Aarch64/64-bit ARMv8 powerpc_altivec Altivec PowerPC .TE . .It Sy zfs_zevent_len_max Ns = Ns Sy 512 Pq uint Max event queue length. Events in the queue can be viewed with .Xr zpool-events 8 . . .It Sy zfs_zevent_retain_max Ns = Ns Sy 2000 Pq int Maximum recent zevent records to retain for duplicate checking. Setting this to .Sy 0 disables duplicate detection. . .It Sy zfs_zevent_retain_expire_secs Ns = Ns Sy 900 Ns s Po 15 min Pc Pq int Lifespan for a recent ereport that was retained for duplicate checking. . .It Sy zfs_zil_clean_taskq_maxalloc Ns = Ns Sy 1048576 Pq int The maximum number of taskq entries that are allowed to be cached. When this limit is exceeded transaction records (itxs) will be cleaned synchronously. . .It Sy zfs_zil_clean_taskq_minalloc Ns = Ns Sy 1024 Pq int The number of taskq entries that are pre-populated when the taskq is first created and are immediately available for use. . .It Sy zfs_zil_clean_taskq_nthr_pct Ns = Ns Sy 100 Ns % Pq int This controls the number of threads used by .Sy dp_zil_clean_taskq . The default value of .Sy 100% will create a maximum of one thread per CPU. . .It Sy zil_maxblocksize Ns = Ns Sy 131072 Ns B Po 128 KiB Pc Pq uint This sets the maximum block size used by the ZIL. On very fragmented pools, lowering this .Pq typically to Sy 36 KiB can improve performance. . .It Sy zil_maxcopied Ns = Ns Sy 7680 Ns B Po 7.5 KiB Pc Pq uint This sets the maximum number of write bytes logged via WR_COPIED. It tunes a tradeoff between additional memory copy and possibly worse log space efficiency vs additional range lock/unlock. . .It Sy zil_nocacheflush Ns = Ns Sy 0 Ns | Ns 1 Pq int Disable the cache flush commands that are normally sent to disk by the ZIL after an LWB write has completed. Setting this will cause ZIL corruption on power loss if a volatile out-of-order write cache is enabled. . .It Sy zil_replay_disable Ns = Ns Sy 0 Ns | Ns 1 Pq int Disable intent logging replay. Can be disabled for recovery from corrupted ZIL. . .It Sy zil_slog_bulk Ns = Ns Sy 67108864 Ns B Po 64 MiB Pc Pq u64 Limit SLOG write size per commit executed with synchronous priority. Any writes above that will be executed with lower (asynchronous) priority to limit potential SLOG device abuse by single active ZIL writer. . .It Sy zfs_zil_saxattr Ns = Ns Sy 1 Ns | Ns 0 Pq int Setting this tunable to zero disables ZIL logging of new .Sy xattr Ns = Ns Sy sa records if the .Sy org.openzfs:zilsaxattr feature is enabled on the pool. This would only be necessary to work around bugs in the ZIL logging or replay code for this record type. The tunable has no effect if the feature is disabled. . .It Sy zfs_embedded_slog_min_ms Ns = Ns Sy 64 Pq uint Usually, one metaslab from each normal and special class vdev is dedicated for use by the ZIL to log synchronous writes. However, if there are fewer than .Sy zfs_embedded_slog_min_ms metaslabs in the vdev, this functionality is disabled. This ensures that we don't set aside an unreasonable amount of space for the ZIL. . .It Sy zstd_earlyabort_pass Ns = Ns Sy 1 Pq uint Whether heuristic for detection of incompressible data with zstd levels >= 3 using LZ4 and zstd-1 passes is enabled. . .It Sy zstd_abort_size Ns = Ns Sy 131072 Pq uint Minimal uncompressed size (inclusive) of a record before the early abort heuristic will be attempted. . .It Sy zio_deadman_log_all Ns = Ns Sy 0 Ns | Ns 1 Pq int If non-zero, the zio deadman will produce debugging messages .Pq see Sy zfs_dbgmsg_enable for all zios, rather than only for leaf zios possessing a vdev. This is meant to be used by developers to gain diagnostic information for hang conditions which don't involve a mutex or other locking primitive: typically conditions in which a thread in the zio pipeline is looping indefinitely. . .It Sy zio_slow_io_ms Ns = Ns Sy 30000 Ns ms Po 30 s Pc Pq int When an I/O operation takes more than this much time to complete, it's marked as slow. Each slow operation causes a delay zevent. Slow I/O counters can be seen with .Nm zpool Cm status Fl s . . .It Sy zio_dva_throttle_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int Throttle block allocations in the I/O pipeline. This allows for dynamic allocation distribution based on device performance. . .It Sy zfs_xattr_compat Ns = Ns 0 Ns | Ns 1 Pq int Control the naming scheme used when setting new xattrs in the user namespace. If .Sy 0 .Pq the default on Linux , user namespace xattr names are prefixed with the namespace, to be backwards compatible with previous versions of ZFS on Linux. If .Sy 1 .Pq the default on Fx , user namespace xattr names are not prefixed, to be backwards compatible with previous versions of ZFS on illumos and .Fx . .Pp Either naming scheme can be read on this and future versions of ZFS, regardless of this tunable, but legacy ZFS on illumos or .Fx are unable to read user namespace xattrs written in the Linux format, and legacy versions of ZFS on Linux are unable to read user namespace xattrs written in the legacy ZFS format. .Pp An existing xattr with the alternate naming scheme is removed when overwriting the xattr so as to not accumulate duplicates. . .It Sy zio_requeue_io_start_cut_in_line Ns = Ns Sy 0 Ns | Ns 1 Pq int Prioritize requeued I/O. . .It Sy zfs_delete_inode Ns = Ns Sy 0 Ns | Ns 1 Pq int Sets whether the kernel should free an inode structure when the last reference is released, or cache it in memory. Intended for testing/debugging. .Pp A live inode structure "pins" versious internal OpenZFS structures in memory, which can result in large amounts of "unusable" memory on systems with lots of infrequently-accessed files, until the kernel's memory pressure mechanism asks OpenZFS to release them. .Pp The default value of .Sy 0 always caches inodes that appear to still exist on disk. Setting it to .Sy 1 will immediately release unused inodes and their associated memory back to the dbuf cache or the ARC for reuse, but may reduce performance if inodes are frequently evicted and reloaded. .Pp This parameter is only available on Linux. . .It Sy zfs_delete_dentry Ns = Ns Sy 0 Ns | Ns 1 Pq int Sets whether the kernel should free a dentry structure when it is no longer required, or hold it in the dentry cache. Intended for testing/debugging. . Since a dentry structure holds an inode reference, a cached dentry can "pin" an inode in memory indefinitely, along with associated OpenZFS structures (See .Sy zfs_delete_inode ) . .Pp The default value of .Sy 0 instructs the kernel to cache entries and their associated inodes when they are no longer directly referenced. They will be reclaimed as part of the kernel's normal cache management processes. Setting it to .Sy 1 will instruct the kernel to release directory entries and their inodes as soon as they are no longer referenced by the filesystem. .Pp This parameter is only available on Linux. . .It Sy zio_taskq_batch_pct Ns = Ns Sy 80 Ns % Pq uint Percentage of online CPUs which will run a worker thread for I/O. These workers are responsible for I/O work such as compression, encryption, checksum and parity calculations. Fractional number of CPUs will be rounded down. .Pp The default value of .Sy 80% was chosen to avoid using all CPUs which can result in latency issues and inconsistent application performance, especially when slower compression and/or checksumming is enabled. Set value only applies to pools imported/created after that. . .It Sy zio_taskq_batch_tpq Ns = Ns Sy 0 Pq uint Number of worker threads per taskq. Higher values improve I/O ordering and CPU utilization, while lower reduce lock contention. Set value only applies to pools imported/created after that. .Pp If .Sy 0 , generate a system-dependent value close to 6 threads per taskq. Set value only applies to pools imported/created after that. . .It Sy zio_taskq_write_tpq Ns = Ns Sy 16 Pq uint Determines the minimum number of threads per write issue taskq. Higher values improve CPU utilization on high throughput, while lower reduce taskq locks contention on high IOPS. Set value only applies to pools imported/created after that. . .It Sy zio_taskq_read Ns = Ns Sy fixed,1,8 null scale null Pq charp Set the queue and thread configuration for the IO read queues. This is an advanced debugging parameter. Don't change this unless you understand what it does. Each of the four values corresponds to the issue, issue high-priority, interrupt, and interrupt high-priority queues. Valid values are .Sy fixed,N,M (M queues with N threads each), .Sy scale[,MIN] (scale with CPUs, minimum MIN total threads), .Sy sync , and .Sy null . Set values only apply to pools imported/created after that. . .It Sy zio_taskq_write Ns = Ns Sy sync null scale null Pq charp Set the queue and thread configuration for the IO write queues. This is an advanced debugging parameter. Don't change this unless you understand what it does. Each of the four values corresponds to the issue, issue high-priority, interrupt, and interrupt high-priority queues. Valid values are .Sy fixed,N,M (M queues with N threads each), .Sy scale[,MIN] (scale with CPUs, minimum MIN total threads), .Sy sync , and .Sy null . Set values only apply to pools imported/created after that. . .It Sy zio_taskq_free Ns = Ns Sy scale,32 null null null Pq charp Set the queue and thread configuration for the IO free queues. This is an advanced debugging parameter. Don't change this unless you understand what it does. Each of the four values corresponds to the issue, issue high-priority, interrupt, and interrupt high-priority queues. Valid values are .Sy fixed,N,M (M queues with N threads each), .Sy scale[,MIN] (scale with CPUs, minimum MIN total threads), .Sy sync , and .Sy null . The default uses a minimum of 32 threads to improve parallelism for DDT and BRT metadata operations during frees. Set values only apply to pools imported/created after that. . .It Sy zvol_inhibit_dev Ns = Ns Sy 0 Ns | Ns 1 Pq uint Do not create zvol device nodes. This may slightly improve startup time on systems with a very large number of zvols. . .It Sy zvol_major Ns = Ns Sy 230 Pq uint Major number for zvol block devices. . .It Sy zvol_max_discard_blocks Ns = Ns Sy 16384 Pq long Discard (TRIM) operations done on zvols will be done in batches of this many blocks, where block size is determined by the .Sy volblocksize property of a zvol. . .It Sy zvol_prefetch_bytes Ns = Ns Sy 131072 Ns B Po 128 KiB Pc Pq uint When adding a zvol to the system, prefetch this many bytes from the start and end of the volume. Prefetching these regions of the volume is desirable, because they are likely to be accessed immediately by .Xr blkid 8 or the kernel partitioner. . .It Sy zvol_request_sync Ns = Ns Sy 0 Ns | Ns 1 Pq uint When processing I/O requests for a zvol, submit them synchronously. This effectively limits the queue depth to .Em 1 for each I/O submitter. When unset, requests are handled asynchronously by a thread pool. The number of requests which can be handled concurrently is controlled by .Sy zvol_threads . This parameter applies regardless of the .Sy zvol_use_blk_mq setting. . .It Sy zvol_num_taskqs Ns = Ns Sy 0 Pq uint Number of zvol taskqs. If .Sy 0 (the default) then scaling is done internally to prefer 6 threads per taskq. This only applies on Linux. . .It Sy zvol_threads Ns = Ns Sy 0 Pq uint The number of system wide threads to use for processing zvol block IOs. If .Sy 0 (the default) then internally set .Sy zvol_threads to the number of CPUs present or 32 (whichever is greater). . .It Sy zvol_blk_mq_threads Ns = Ns Sy 0 Pq uint The number of threads per zvol to use for queuing IO requests. This parameter will only appear if your kernel supports .Li blk-mq and is only read and assigned to a zvol at zvol load time. If .Sy 0 (the default) then internally set .Sy zvol_blk_mq_threads to the number of CPUs present. . .It Sy zvol_use_blk_mq Ns = Ns Sy 0 Ns | Ns 1 Pq uint Set to .Sy 1 to use the .Li blk-mq API for zvols. Set to .Sy 0 (the default) to use the legacy zvol APIs. This setting can give better or worse zvol performance depending on the workload. This parameter will only appear if your kernel supports .Li blk-mq and is only read and assigned to a zvol at zvol load time. . .It Sy zvol_blk_mq_blocks_per_thread Ns = Ns Sy 8 Pq uint If .Sy zvol_use_blk_mq is enabled, then process this number of .Sy volblocksize Ns -sized blocks per zvol thread. This tunable can be use to favor better performance for zvol reads (lower values) or writes (higher values). If set to .Sy 0 , then the zvol layer will process the maximum number of blocks per thread that it can. This parameter will only appear if your kernel supports .Li blk-mq and is only applied at each zvol's load time. . .It Sy zvol_blk_mq_queue_depth Ns = Ns Sy 0 Pq uint The queue_depth value for the zvol .Li blk-mq interface. This parameter will only appear if your kernel supports .Li blk-mq and is only applied at each zvol's load time. If .Sy 0 (the default) then use the kernel's default queue depth. Values are clamped to the kernel's .Dv BLKDEV_MIN_RQ and .Dv BLKDEV_MAX_RQ Ns / Ns Dv BLKDEV_DEFAULT_RQ limits. . .It Sy zvol_volmode Ns = Ns Sy 1 Pq uint Defines zvol block devices behavior when .Sy volmode Ns = Ns Sy default : .Bl -tag -compact -offset 4n -width "a" .It Sy 1 .No equivalent to Sy full .It Sy 2 .No equivalent to Sy dev .It Sy 3 .No equivalent to Sy none .El . .It Sy zvol_enforce_quotas Ns = Ns Sy 0 Ns | Ns 1 Pq uint Enable strict ZVOL quota enforcement. The strict quota enforcement may have a performance impact. .El . .Sh ZFS I/O SCHEDULER ZFS issues I/O operations to leaf vdevs to satisfy and complete I/O operations. The scheduler determines when and in what order those operations are issued. The scheduler divides operations into five I/O classes, prioritized in the following order: sync read, sync write, async read, async write, and scrub/resilver. Each queue defines the minimum and maximum number of concurrent operations that may be issued to the device. In addition, the device has an aggregate maximum, .Sy zfs_vdev_max_active . Note that the sum of the per-queue minima must not exceed the aggregate maximum. If the sum of the per-queue maxima exceeds the aggregate maximum, then the number of active operations may reach .Sy zfs_vdev_max_active , in which case no further operations will be issued, regardless of whether all per-queue minima have been met. .Pp For many physical devices, throughput increases with the number of concurrent operations, but latency typically suffers. Furthermore, physical devices typically have a limit at which more concurrent operations have no effect on throughput or can actually cause it to decrease. .Pp The scheduler selects the next operation to issue by first looking for an I/O class whose minimum has not been satisfied. Once all are satisfied and the aggregate maximum has not been hit, the scheduler looks for classes whose maximum has not been satisfied. Iteration through the I/O classes is done in the order specified above. No further operations are issued if the aggregate maximum number of concurrent operations has been hit, or if there are no operations queued for an I/O class that has not hit its maximum. Every time an I/O operation is queued or an operation completes, the scheduler looks for new operations to issue. .Pp In general, smaller .Sy max_active Ns s will lead to lower latency of synchronous operations. Larger .Sy max_active Ns s may lead to higher overall throughput, depending on underlying storage. .Pp The ratio of the queues' .Sy max_active Ns s determines the balance of performance between reads, writes, and scrubs. For example, increasing .Sy zfs_vdev_scrub_max_active will cause the scrub or resilver to complete more quickly, but reads and writes to have higher latency and lower throughput. .Pp All I/O classes have a fixed maximum number of outstanding operations, except for the async write class. Asynchronous writes represent the data that is committed to stable storage during the syncing stage for transaction groups. Transaction groups enter the syncing state periodically, so the number of queued async writes will quickly burst up and then bleed down to zero. Rather than servicing them as quickly as possible, the I/O scheduler changes the maximum number of active async write operations according to the amount of dirty data in the pool. Since both throughput and latency typically increase with the number of concurrent operations issued to physical devices, reducing the burstiness in the number of simultaneous operations also stabilizes the response time of operations from other queues, in particular synchronous ones. In broad strokes, the I/O scheduler will issue more concurrent operations from the async write queue as there is more dirty data in the pool. . .Ss Async Writes The number of concurrent operations issued for the async write I/O class follows a piece-wise linear function defined by a few adjustable points: .Bd -literal | o---------| <-- \fBzfs_vdev_async_write_max_active\fP ^ | /^ | | | / | | active | / | | I/O | / | | count | / | | | / | | |-------o | | <-- \fBzfs_vdev_async_write_min_active\fP 0|_______^______|_________| 0% | | 100% of \fBzfs_dirty_data_max\fP | | | `-- \fBzfs_vdev_async_write_active_max_dirty_percent\fP `--------- \fBzfs_vdev_async_write_active_min_dirty_percent\fP .Ed .Pp Until the amount of dirty data exceeds a minimum percentage of the dirty data allowed in the pool, the I/O scheduler will limit the number of concurrent operations to the minimum. As that threshold is crossed, the number of concurrent operations issued increases linearly to the maximum at the specified maximum percentage of the dirty data allowed in the pool. .Pp Ideally, the amount of dirty data on a busy pool will stay in the sloped part of the function between .Sy zfs_vdev_async_write_active_min_dirty_percent and .Sy zfs_vdev_async_write_active_max_dirty_percent . If it exceeds the maximum percentage, this indicates that the rate of incoming data is greater than the rate that the backend storage can handle. In this case, we must further throttle incoming writes, as described in the next section. . .Sh ZFS TRANSACTION DELAY We delay transactions when we've determined that the backend storage isn't able to accommodate the rate of incoming writes. .Pp If there is already a transaction waiting, we delay relative to when that transaction will finish waiting. This way the calculated delay time is independent of the number of threads concurrently executing transactions. .Pp If we are the only waiter, wait relative to when the transaction started, rather than the current time. This credits the transaction for "time already served", e.g. reading indirect blocks. .Pp The minimum time for a transaction to take is calculated as .D1 min_time = min( Ns Sy zfs_delay_scale No \(mu Po Sy dirty No \- Sy min Pc / Po Sy max No \- Sy dirty Pc , 100ms) .Pp The delay has two degrees of freedom that can be adjusted via tunables. The percentage of dirty data at which we start to delay is defined by .Sy zfs_delay_min_dirty_percent . This should typically be at or above .Sy zfs_vdev_async_write_active_max_dirty_percent , so that we only start to delay after writing at full speed has failed to keep up with the incoming write rate. The scale of the curve is defined by .Sy zfs_delay_scale . Roughly speaking, this variable determines the amount of delay at the midpoint of the curve. .Bd -literal delay 10ms +-------------------------------------------------------------*+ | *| 9ms + *+ | *| 8ms + *+ | * | 7ms + * + | * | 6ms + * + | * | 5ms + * + | * | 4ms + * + | * | 3ms + * + | * | 2ms + (midpoint) * + | | ** | 1ms + v *** + | \fBzfs_delay_scale\fP ----------> ******** | 0 +-------------------------------------*********----------------+ 0% <- \fBzfs_dirty_data_max\fP -> 100% .Ed .Pp Note, that since the delay is added to the outstanding time remaining on the most recent transaction it's effectively the inverse of IOPS. Here, the midpoint of .Em 500 us translates to .Em 2000 IOPS . The shape of the curve was chosen such that small changes in the amount of accumulated dirty data in the first three quarters of the curve yield relatively small differences in the amount of delay. .Pp The effects can be easier to understand when the amount of delay is represented on a logarithmic scale: .Bd -literal delay 100ms +-------------------------------------------------------------++ + + | | + *+ 10ms + *+ + ** + | (midpoint) ** | + | ** + 1ms + v **** + + \fBzfs_delay_scale\fP ----------> ***** + | **** | + **** + 100us + ** + + * + | * | + * + 10us + * + + + | | + + +--------------------------------------------------------------+ 0% <- \fBzfs_dirty_data_max\fP -> 100% .Ed .Pp Note here that only as the amount of dirty data approaches its limit does the delay start to increase rapidly. The goal of a properly tuned system should be to keep the amount of dirty data out of that range by first ensuring that the appropriate limits are set for the I/O scheduler to reach optimal throughput on the back-end storage, and then by changing the value of .Sy zfs_delay_scale to increase the steepness of the curve. VDEV\_ID.CONF(5) - File Formats Manual # NAME **vdev\_id.conf** - configuration file for vdev\_id(8) # DESCRIPTION **vdev\_id.conf** is the configuration file for vdev\_id(8). It controls the default behavior of vdev\_id(8) while it is mapping a disk device name to an alias. The **vdev\_id.conf** file uses a simple format consisting of a keyword followed by one or more values on a single line. Any line not beginning with a recognized keyword is ignored. Comments may optionally begin with a hash character. The following keywords and values are used. **alias** *name* *devlink* > Maps a device link in the > */dev* > directory hierarchy to a new device name. > The udev rule defining the device link must have run prior to > vdev\_id(8). > A defined alias takes precedence over a topology-derived name, but the > two naming methods can otherwise coexist. > For example, one might name drives in a JBOD with the > **sas\_direct** > topology while naming an internal L2ARC device with an alias. > *name* > is the name of the link to the device that will by created under > */dev/disk/by-vdev*. > *devlink* > is the name of the device link that has already been > defined by udev. > This may be an absolute path or the base filename. **channel** \[*pci\_slot*] *port* *name* > Maps a physical path to a channel name (typically representing a single > disk enclosure). **enclosure\_symlinks** **yes**|**no** > Additionally create > */dev/by-enclosure* > symlinks to the disk enclosure > *sg* > devices using the naming scheme from > *vdev\_id.conf*. > **enclosure\_symlinks** > is only allowed for > **sas\_direct** > mode. **enclosure\_symlinks\_prefix** *prefix* > Specify the prefix for the enclosure symlinks in the form > */dev/by-enclosure/*<*prefix*>-<*channel*><*num*> > Defaults to > "*enc*". **slot** *prefix* *new* \[*channel*] > Maps a disk slot number as reported by the operating system to an > alternative slot number. > If the > *channel* > parameter is specified > then the mapping is only applied to slots in the named channel, > otherwise the mapping is applied to all channels. > The first-specified > *slot* > rule that can match a slot takes precedence. > Therefore a channel-specific mapping for a given slot should generally appear > before a generic mapping for the same slot. > In this way a custom mapping may be applied to a particular channel > and a default mapping applied to the others. **zpad\_slot** *digits* > Pad slot numbers with zeros to make them > *digits* > long, which can help to make disk names a consistent length and easier to sort. **multipath** **yes**|**no** > Specifies whether > vdev\_id(8) > will handle only dm-multipath devices. > If set to > **yes** > then > vdev\_id(8) > will examine the first running component disk of a dm-multipath > device as provided by the driver command to determine the physical path. **topology** **sas\_direct**|**sas\_switch**|**scsi** > Identifies a physical topology that governs how physical paths are > mapped to channels: > **sas\_direct** and **scsi** > > channels are uniquely identified by a PCI slot and HBA port number > **sas\_switch** > > channels are uniquely identified by a SAS switch port number **phys\_per\_port** *num* > Specifies the number of PHY devices associated with a SAS HBA port or SAS > switch port. > vdev\_id(8) > internally uses this value to determine which HBA or switch port a > device is connected to. > The default is > **4**. **slot** **bay**|**phy**|**port**|**id**|**lun**|**bay\_lun**|**ses** > Specifies from which element of a SAS identifier the slot number is > taken. > The default is > **bay**: > **bay** > > read the slot number from the bay identifier. > **phy** > > read the slot number from the phy identifier. > **port** > > use the SAS port as the slot number. > **id** > > use the scsi id as the slot number. > **lun** > > use the scsi lun as the slot number. > **bay\_lun** > > read the slot number from the bay identifier and append the lun number. > > Useful for multi-lun multi-actuator hard drives. > **ses** > > use the SCSI Enclosure Services (SES) enclosure device slot number, > > as reported by > > sg\_ses(8). > > Intended for use only on systems where > > **bay** > > is unsupported, > > noting that > > **port** > > and > > **id** > > may be unstable across disk replacement. # FILES */etc/zfs/vdev\_id.conf* > The configuration file for > vdev\_id(8). # EXAMPLES A non-multipath configuration with direct-attached SAS enclosures and an arbitrary slot re-mapping: multipath no topology sas_direct phys_per_port 4 slot bay # PCI_SLOT HBA PORT CHANNEL NAME channel 85:00.0 1 A channel 85:00.0 0 B channel 86:00.0 1 C channel 86:00.0 0 D # Custom mapping for Channel A # Linux Mapped # Slot Slot Channel slot 1 7 A slot 2 10 A slot 3 3 A slot 4 6 A # Default mapping for B, C, and D slot 1 4 slot 2 2 slot 3 1 slot 4 3 A SAS-switch topology. Note, that the *channel* keyword takes only two arguments in this example: topology sas_switch # SWITCH PORT CHANNEL NAME channel 1 A channel 2 B channel 3 C channel 4 D A multipath configuration. Note that channel names have multiple definitions - one per physical path: multipath yes # PCI_SLOT HBA PORT CHANNEL NAME channel 85:00.0 1 A channel 85:00.0 0 B channel 86:00.0 1 A channel 86:00.0 0 B A configuration with enclosure\_symlinks enabled: multipath yes enclosure_symlinks yes # PCI_ID HBA PORT CHANNEL NAME channel 05:00.0 1 U channel 05:00.0 0 L channel 06:00.0 1 U channel 06:00.0 0 L In addition to the disks symlinks, this configuration will create: /dev/by-enclosure/enc-L0 /dev/by-enclosure/enc-L1 /dev/by-enclosure/enc-U0 /dev/by-enclosure/enc-U1 A configuration using device link aliases: # by-vdev # name fully qualified or base name of device link alias d1 /dev/disk/by-id/wwn-0x5000c5002de3b9ca alias d2 wwn-0x5000c5002def789e # SEE ALSO vdev\_id(8) Debian - October 8, 2024 DRACUT.ZFS(7) - Miscellaneous Information Manual # NAME **dracut.zfs** - overview of ZFS dracut hooks # SYNOPSIS parse-zfs.sh -> dracut-cmdline.service | |v | | |v \-----------------> dracut-initqueue.service | zfs-import-opts.sh zfs-load-module.service |v | | | | sysinit.target |v | |v | | zfs-import-scan.service |v zfs-import-scan.service |v |v | zfs-import-cache.service | zfs-import-cache.service basic.target | | \__________________| | |v |v |v | zfs-load-key.sh zfs-env-bootfs.service | | |v |v |v zfs-import.target -> dracut-pre-mount.service | |^ | | dracut-zfs-generator | | _____________________/| |/ |v | sysroot.mount <------- dracut-zfs-generator | | | |v | initrd-root-fs.target <--- zfs-nonroot-necessities.service | | | | |v | |v dracut-mount.service | zfs-snapshot-bootfs.service | | | |v | |v | zfs-rollback-bootfs.service | | | |v | | /sysroot/{usr,etc,lib,&c.} <---------------------------------------/ | | | |v | initrd-fs.target \______________________ | \| |v export-zfs.sh initrd.target | | |v |v dracut-shutdown.service | |v zfs-needshutdown.sh -> initrd-cleanup.service Compare dracut.bootup(7) for the full flowchart. # DESCRIPTION Under dracut, booting with ZFS-on-*/* is facilitated by a number of hooks in the **90zfs** module. Booting into a ZFS dataset requires **mountpoint**=*/* to be set on the dataset containing the root filesystem (henceforth "the boot dataset") and at the very least either the **bootfs** property to be set to that dataset, or the **root=** kernel cmdline (or dracut drop-in) argument to specify it. All children of the boot dataset with **canmount**=**on** with **mountpoint**s matching */etc*, */bin*, */lib*, */lib??*, */libx32*, and */usr* globs are deemed essential and will be mounted as well. zfs-mount-generator(8) is recommended for proper functioning of the system afterward (correct mount properties, remounting, &c.). # CMDLINE ## Standard **root=zfs:**‌*dataset*, **root=ZFS=**‌*dataset* > Use > *dataset* > as the boot dataset. > All pluses > ('+') > are replaced with spaces > (' '). **root=zfs:AUTO**, **root=zfs:**, **root=zfs**, \[**root=**] > After import, search for the first pool with the > **bootfs** > property set, use its value as-if specified as the > *dataset* > above. **rootfstype=zfs root=**‌*dataset* > Equivalent to > **root=zfs:**‌*dataset*. **rootfstype=zfs** \[**root=**] > Equivalent to > **root=zfs:AUTO**. **rootflags=**‌*flags* > Mount the boot dataset with > **-o** *flags*; > cf. > *Temporary Mount Point Properties* > in > zfsprops(7). > These properties will not last, since all filesystems will be re-mounted from > the real root. **debug** > If specified, > **dracut-zfs-generator** > logs to the journal. Be careful about setting neither **rootfstype=zfs** nor **root=zfs:**‌*dataset* — other automatic boot selection methods, like **systemd-gpt-auto-generator** and **systemd-fstab-generator** might take precedent. ## ZFS-specific **bootfs.snapshot**\[**=**‌*snapshot-name*] > Execute > **zfs** **snapshot** *boot-dataset*‌**@**‌*snapshot-name* > before pivoting to the real root. > *snapshot-name* > defaults to the current kernel release. **bootfs.rollback**\[**=**‌*snapshot-name*] > Execute > **zfs** **rollback** **-Rf** *boot-dataset*‌**@**‌*snapshot-name* > before pivoting to the real root. > *snapshot-name* > defaults to the current kernel release. **spl\_hostid=**‌*host-id* > Use > zgenhostid(8) > to set the host ID to > *host-id*; > otherwise, > */etc/hostid* > inherited from the real root is used. **zfs\_force**, **zfs.force**, **zfsforce** > Appends > **-f** > to all > **zpool** **import** > invocations; primarily useful in conjunction with > **spl\_hostid=**, > or if no host ID was inherited. # FILES *parse-zfs.sh* (**cmdline**) > Processes > **spl\_hostid=**. > If > **root=** > matches a known pattern, above, provides > */dev/root* > and delays the initqueue until > zfs(4) > is loaded, *zfs-import-opts.sh* (**systemd** environment generator) > Turns > **zfs\_force**, **zfs.force**, or **zfsforce** > into > `ZPOOL_IMPORT_OPTS`=**-f** > for > *zfs-import-scan.service* > or > *zfs-import-cache.service*. *zfs-load-key.sh* (**pre-mount**) > Loads encryption keys for the boot dataset and its essential descendants. > **keylocation**=**prompt** > > Is prompted for via > > **systemd-ask-password** > > thrice. > **keylocation**=**https://**‌*URL*, **keylocation**=**http://**‌*URL* > > *network-online.target* > > is started before loading. > **keylocation**=**file://**‌*path* > > If > > *path* > > doesn't exist, > > **udevadm** is **settle**d. > > If it still doesn't, it's waited for for up to > > **10**s. *zfs-env-bootfs.service* (**systemd** service) > After pool import, sets > `BOOTFS`= > in the systemd environment to the first non-null > **bootfs** > value in iteration order. *dracut-zfs-generator* (**systemd** generator) > Generates > *sysroot.mount* (using **rootflags=**, if any). > If an explicit boot dataset was specified, also generates essential mountpoints > (*sysroot-etc.mount*, *sysroot-bin.mount*, &c.), > otherwise generates > *zfs-nonroot-necessities.service* > which mounts them explicitly after > */sysroot* > using > `BOOTFS`=. *zfs-snapshot-bootfs.service*, *zfs-rollback-bootfs.service* (**systemd** services) > Consume > **bootfs.snapshot** > and > **bootfs.rollback** > as described in > *CMDLINE*. > Use > `BOOTFS`= > if no explicit boot dataset was specified. *zfs-needshutdown.sh* (**cleanup**) > If any pools were imported, signals that shutdown hooks are required. *export-zfs.sh* (**shutdown**) > Forcibly exports all pools. */etc/hostid*, */etc/zfs/zpool.cache*, */etc/zfs/vdev\_id.conf* (regular files) > Included verbatim, hostonly. *mount-zfs.sh* (**mount**) > Does nothing on > **systemd** > systems > (if *dracut-zfs-generator* succeeded). > Otherwise, loads encryption key for the boot dataset from the console or via > plymouth. > It may not work at all! # SEE ALSO dracut.bootup(7), zfsprops(7), zpoolprops(7), dracut-shutdown.service(8), systemd-fstab-generator(8), systemd-gpt-auto-generator(8), zfs-mount-generator(8), zgenhostid(8) Debian - July 13, 2024 VDEVPROPS(7) - Miscellaneous Information Manual # NAME **vdevprops** - native and user-defined properties of ZFS vdevs # DESCRIPTION Properties are divided into two types, native properties and user-defined (or "user") properties. Native properties either export internal statistics or control ZFS behavior. In addition, native properties are either editable or read-only. User properties have no effect on ZFS behavior, but you can use them to annotate vdevs in a way that is meaningful in your environment. For more information about user properties, see the *User Properties* section, below. ## Native Properties Every vdev has a set of properties that export statistics about the vdev as well as control various behaviors. Properties are not inherited from top-level vdevs, with the exception of checksum\_n, checksum\_t, io\_n, io\_t, slow\_io\_events, slow\_io\_n, and slow\_io\_t. The values of numeric properties can be specified using human-readable suffixes (for example, **k**, **KB**, **M**, **Gb**, and so forth, up to **Z** for zettabyte) . The following are all valid (and equal) specifications: `1536M`, `1.5g`, `1.50GB`. The values of non-numeric properties are case sensitive and must be lowercase. The following native properties consist of read-only statistics about the vdev. These properties can not be changed. **capacity** > Percentage of vdev space used **state** > state of this vdev such as online, faulted, or offline **guid** > globally unique id of this vdev **asize** > The allocatable size of this vdev **psize** > The physical size of this vdev **ashift** > The physical sector size of this vdev expressed as the power of two **size** > The total size of this vdev **free** > The amount of remaining free space on this vdev **allocated** > The amount of allocated space on this vdev **expandsize** > How much this vdev can expand by **fragmentation** > Percent of fragmentation in this vdev **parity** > The level of parity for this vdev **failure\_domain** > Failure domain id of this child vdev in > **dRAID** > vdev with failure domains feature **failure\_group** > Failure group id of this child vdev in > **dRAID** > vdev with failure domains feature **devid** > The device id for this vdev **physpath** > The physical path to the device **encpath** > The enclosure path to the device **fru** > Field Replaceable Unit, usually a model number **parent** > Parent of this vdev **children** > Comma separated list of children of this vdev **numchildren** > The number of children belonging to this vdev **read\_errors**, **write\_errors**, **checksum\_errors**, **initialize\_errors**, **trim\_errors** > The number of errors of each type encountered by this vdev **slow\_ios** > This indicates the number of slow I/O operations encountered by this vdev. > A slow I/O is defined as an operation that did not complete within the > **zio\_slow\_io\_ms** > threshold in milliseconds > (**30000** by default). > For > **RAIDZ** > and > **dRAID** > configurations, this value also represents the number of times the vdev was > identified as an outlier and excluded from participating in read I/O operations. **null\_ops**, **read\_ops**, **write\_ops**, **free\_ops**, **claim\_ops**, **trim\_ops** > The number of I/O operations of each type performed by this vdev **null\_bytes**, **read\_bytes**, **write\_bytes**, **free\_bytes**, **claim\_bytes**, **trim\_bytes** > The cumulative size of all operations of each type performed by this vdev **removing** > If this device is currently being removed from the pool **raidz\_expanding** > Set while a > **zpool** **attach** > expansion is in progress on this RAID-Z vdev; cleared on completion. > See > zpool-attach(8). **trim\_support** > Indicates if a leaf device supports trim operations. **rotational** > Indicates whether the device backing this vdev uses rotating media. The following native properties can be used to change the behavior of a vdev. **checksum\_n**, **checksum\_t**, **io\_n**, **io\_t**, **slow\_io\_n**, **slow\_io\_t** > Tune the fault management daemon by specifying checksum/io thresholds of <N> > errors in <T> seconds, respectively. > These properties can be set on leaf and top-level vdevs. > When the property is set on the leaf and top-level vdev, the value of the leaf > vdev will be used. > If the property is only set on the top-level vdev, this value will be used. > The value of these properties do not persist across vdev replacement. > For this reason, it is advisable to set the property on the top-level vdev - > not on the leaf vdev itself. > The > **slow\_io\_n** > and > **slow\_io\_t** > properties must be manually set to enable slow I/O diagnosis; > there are no built-in defaults. > The default values for > **checksum\_n**, **checksum\_t**, **io\_n**, > and > **io\_t** > on > **OpenZFS on Linux** > are 10 errors in 600 seconds. > For > **OpenZFS on FreeBSD** > defaults see > zfsd(8). > The **slow\_io\_events** > property controls whether slow I/O events are generated. > Even when disabled, slow I/Os will be included in the > **zpool** **status** **-s** > output. **comment** > A text comment up to 8192 characters long **bootsize** > The amount of space to reserve for the EFI system partition **failfast**=**inherit**|**on**|**off** > If this device should propagate BIO errors back to ZFS, used to disable > failfast. > **inherit** > causes the vdev to adopt the behavior of its parent vdev, > recursively up the tree. **sit\_out** > Only valid for > **RAIDZ** > and > **dRAID** > vdevs. > True when a slow disk outlier was detected and the vdev is currently in a sit > out state. > This property can be manually set to cause vdevs to sit out. > It will also be automatically set by the > **autosit** > logic if that is enabled. > While sitting out, the vdev will not participate in normal reads, instead its > data will be reconstructed as needed from parity. **autosit** > Only valid for > **RAIDZ** > and > **dRAID** > vdevs. > If set, this enables the kernel-level slow disk detection logic. > This logic automatically causes any vdevs that are significant negative > performance outliers to sit out, as described in the > **sit\_out** > property. **path** > The path to the device for this vdev **allocating** > If this device should perform new allocations, used to disable a device > when it is scheduled for later removal. > See > zpool-remove(8). **alloc\_bias**=**none**|**log**|**special**|**dedup** > Controls the allocation class for a top-level vdev. > Changes take effect after an export and import of the pool. > Changing to/from log is not implemented, since it may lead to data loss in > case of the log device failure. > Setting to > **special** > and > **dedup** > requires > **feature@allocation\_classes** > to be enabled. > At least one top-level vdev must remain in the normal > (**none**) > class. **scheduler**=**auto**|**on**|**off** > Controls how I/O requests are added to the vdev queue when reading or > writing to this vdev. > This property can be set on leaf vdevs. > The value of these properties do not persist across vdev replacement. > *auto* > > Let ZFS choose which scheduler it thinks will be best. > > Currently, the scheduler will queue I/O if the vdev is backed by a rotational > > block device or file, and not queue otherwise. > *on* > > Always adds I/O requests to the vdev queue. > *off* > > Never adds I/O requests to the vdev queue. > > This is not recommended for vdevs backed by spinning disks as it could > > result in starvation. ## User Properties In addition to the standard native properties, ZFS supports arbitrary user properties. User properties have no effect on ZFS behavior, but applications or administrators can use them to annotate vdevs. User property names must contain a colon ("**:**") character to distinguish them from native properties. They may contain lowercase letters, numbers, and the following punctuation characters: colon ("**:**"), dash ("**-**"), period ("**.**"), and underscore ("**\_**"). The expected convention is that the property name is divided into two portions such as *module*:*property*, but this namespace is not enforced by ZFS. User property names can be at most 256 characters, and cannot begin with a dash ("**-**"). When making programmatic use of user properties, it is strongly suggested to use a reversed DNS domain name for the *module* component of property names to reduce the chance that two independently-developed packages use the same property name for different purposes. The values of user properties are arbitrary strings and are never validated. Use the **zpool** **set** command with a blank value to clear a user property. Property values are limited to 8192 bytes. # SEE ALSO zpoolprops(7), zpool-set(8) Debian - July 23, 2024 ZFSCONCEPTS(7) - Miscellaneous Information Manual # NAME **zfsconcepts** - overview of ZFS concepts # DESCRIPTION ## ZFS File System Hierarchy A ZFS storage pool is a logical collection of devices that provide space for datasets. A storage pool is also the root of the ZFS file system hierarchy. The root of the pool can be accessed as a file system, such as mounting and unmounting, taking snapshots, and setting properties. The physical storage characteristics, however, are managed by the zpool(8) command. See zpool(8) for more information on creating and administering pools. ## Snapshots A snapshot is a read-only copy of a file system or volume. Snapshots can be created extremely quickly, and initially consume no additional space within the pool. As data within the active dataset changes, the snapshot consumes more data than would otherwise be shared with the active dataset. Snapshots can have arbitrary names. Snapshots of volumes can be cloned or rolled back, visibility is determined by the **snapdev** property of the parent volume. File system snapshots can be accessed under the *.zfs/snapshot* directory in the root of the file system. Snapshots are automatically mounted on demand and may be unmounted at regular intervals. The availability and visibility of the *.zfs* directory can be controlled by the **snapdir** property. ## Bookmarks A bookmark is like a snapshot, a read-only copy of a file system or volume. Bookmarks can be created extremely quickly, compared to snapshots, and they consume no additional space within the pool. Bookmarks can also have arbitrary names, much like snapshots. Unlike snapshots, bookmarks can not be accessed through the filesystem in any way. From a storage standpoint a bookmark just provides a way to reference when a snapshot was created as a distinct object. Bookmarks are initially tied to a snapshot, not the filesystem or volume, and they will survive if the snapshot itself is destroyed. Since they are very light weight there's little incentive to destroy them. ## Clones A clone is a writable volume or file system whose initial contents are the same as another dataset. As with snapshots, creating a clone is nearly instantaneous, and initially consumes no additional space. Clones can only be created from a snapshot. When a snapshot is cloned, it creates an implicit dependency between the parent and child. Even though the clone is created somewhere else in the dataset hierarchy, the original snapshot cannot be destroyed as long as a clone exists. The **origin** property exposes this dependency, and the **destroy** command lists any such dependencies, if they exist. The clone parent-child dependency relationship can be reversed by using the **promote** subcommand. This causes the "origin" file system to become a clone of the specified file system, which makes it possible to destroy the file system that the clone was created from. ## Mount Points Creating a ZFS file system is a simple operation, so the number of file systems per system is likely to be numerous. To cope with this, ZFS automatically manages mounting and unmounting file systems without the need to edit the */etc/fstab* file. All automatically managed file systems are mounted by ZFS at boot time. By default, file systems are mounted under */path*, where *path* is the name of the file system in the ZFS namespace. Directories are created and destroyed as needed. A file system can also have a mount point set in the **mountpoint** property. This directory is created as needed, and ZFS automatically mounts the file system when the **zfs** **mount** **-a** command is invoked (without editing */etc/fstab*) . The **mountpoint** property can be inherited, so if *pool/home* has a mount point of */export/stuff*, then *pool/home/user* automatically inherits a mount point of */export/stuff/user*. A file system **mountpoint** property of **none** prevents the file system from being mounted. If needed, ZFS file systems can also be managed with traditional tools (**mount**, **umount**, */etc/fstab*) . If a file system's mount point is set to **legacy**, ZFS makes no attempt to manage the file system, and the administrator is responsible for mounting and unmounting the file system. Because pools must be imported before a legacy mount can succeed, administrators should ensure that legacy mounts are only attempted after the zpool import process finishes at boot time. For example, on machines using systemd, the mount option **x-systemd.requires=zfs-import.target** will ensure that the zfs-import completes before systemd attempts mounting the filesystem. See systemd.mount(5) for details. ## Deduplication Deduplication is the process for removing redundant data at the block level, reducing the total amount of data stored. If a file system has the **dedup** property enabled, duplicate data blocks are removed synchronously. The result is that only unique data is stored and common components are shared among files. Deduplicating data is a very resource-intensive operation. It is generally recommended that you have at least 1.25 GiB of RAM per 1 TiB of storage when you enable deduplication. Calculating the exact requirement depends heavily on the type of data stored in the pool. Enabling deduplication on an improperly-designed system can result in performance issues (slow I/O and administrative operations). It can potentially lead to problems importing a pool due to memory exhaustion. Deduplication can consume significant processing power (CPU) and memory as well as generate additional disk I/O. Before creating a pool with deduplication enabled, ensure that you have planned your hardware requirements appropriately and implemented appropriate recovery practices, such as regular backups. Consider using the **compression** property as a less resource-intensive alternative. ## Block cloning Block cloning is a facility that allows a file (or parts of a file) to be "cloned", that is, a shallow copy made where the existing data blocks are referenced rather than copied. Later modifications to the data will cause a copy of the data block to be taken and that copy modified. This facility is used to implement "reflinks" or "file-level copy-on-write". Cloned blocks are tracked in a special on-disk structure called the Block Reference Table (BRT) . Unlike deduplication, this table has minimal overhead, so can be enabled at all times. Also unlike deduplication, cloning must be requested by a user program. Many common file copying programs, including newer versions of **/bin/cp**, will try to create clones automatically. Look for "clone", "dedupe" or "reflink" in the documentation for more information. There are some limitations to block cloning. Only whole blocks can be cloned, and blocks can not be cloned if they are not yet written to disk, or if they are encrypted, or the source and destination **recordsize** properties differ. The OS may add additional restrictions; for example, most versions of Linux will not allow clones across datasets. Debian - October 2, 2024 ZFSPROPS(7) - Miscellaneous Information Manual # NAME **zfsprops** - native and user-defined properties of ZFS datasets # DESCRIPTION Properties are divided into two types, native properties and user-defined (or "user") properties. Native properties either export internal statistics or control ZFS behavior. In addition, native properties are either editable or read-only. User properties have no effect on ZFS behavior, but you can use them to annotate datasets in a way that is meaningful in your environment. For more information about user properties, see the *User Properties* section, below. ## Native Properties Every dataset has a set of properties that export statistics about the dataset as well as control various behaviors. Properties are inherited from the parent unless overridden by the child. Some properties apply only to certain types of datasets (file systems, volumes, or snapshots). The values of numeric properties can be specified using human-readable suffixes (for example, **k**, **KB**, **M**, **Gb**, and so forth, up to **Z** for zettabyte) . The following are all valid (and equal) specifications: `1536M`, `1.5g`, `1.50GB`. The values of non-numeric properties are case sensitive and must be lowercase, except for **mountpoint**, **sharenfs**, and **sharesmb**. The following native properties consist of read-only statistics about the dataset. These properties can be neither set, nor inherited. Native properties apply to all dataset types unless otherwise noted. **available** > The amount of space available to the dataset and all its children, assuming that > there is no other activity in the pool. > Because space is shared within a pool, availability can be limited by any number > of factors, including physical pool size, quotas, reservations, or other > datasets within the pool. > This property can also be referred to by its shortened column name, > **avail**. **compressratio** > For non-snapshots, the compression ratio achieved for the > **used** > space of this dataset, expressed as a multiplier. > The > **used** > property includes descendant datasets, and, for clones, does not include the > space shared with the origin snapshot. > For snapshots, the > **compressratio** > is the same as the > **refcompressratio** > property. > Compression can be turned on by running: > **zfs** **set** **compression**=**on** *dataset*. > The default value is > **off**. **createtxg** > The transaction group (txg) in which the dataset was created. > Bookmarks have the same > **createtxg** > as the snapshot they are initially tied to. > This property is suitable for ordering a list of snapshots, > e.g. for incremental send and receive. **creation** > The time this dataset was created. **clones** > For snapshots, this property is a comma-separated list of filesystems or volumes > which are clones of this snapshot. > The clones' > **origin** > property is this snapshot. > If the > **clones** > property is not empty, then this snapshot can not be destroyed > (even with the > **-r** > or > **-f** > options) > . > The roles of origin and clone can be swapped by promoting the clone with the > **zfs** **promote** > command. **defer\_destroy** > This property is > **on** > if the snapshot has been marked for deferred destroy by using the > **zfs** **destroy** **-d** > command. > Otherwise, the property is > **off**. **encryptionroot** > For encrypted datasets, indicates where the dataset is currently inheriting its > encryption key from. > Loading or unloading a key for the > **encryptionroot** > will implicitly load / unload the key for any inheriting datasets (see > **zfs** **load-key** > and > **zfs** **unload-key** > for details). > Clones will always share an > encryption key with their origin. > See the > *Encryption* > section of > zfs-load-key(8) > for details. **filesystem\_count** > The total number of filesystems and volumes that exist under this location in > the dataset tree. > This value is only available when a > **filesystem\_limit** > has been set somewhere in the tree under which the dataset resides. **keystatus** > Indicates if an encryption key is currently loaded into ZFS. > The possible values are > **none**, > **available**, > and > **unavailable**. > See > **zfs** **load-key** > and > **zfs** **unload-key**. **guid** > The 64 bit GUID of this dataset or bookmark which does not change over its > entire lifetime. > When a snapshot is sent to another pool, the received snapshot has the same > GUID. > Thus, the > **guid** > is suitable to identify a snapshot across pools. **logicalreferenced** > The amount of space that is > "logically" > accessible by this dataset. > See the > **referenced** > property. > The logical space ignores the effect of the > **compression** > and > **copies** > properties, giving a quantity closer to the amount of data that applications > see. > However, it does include space consumed by metadata. > This property can also be referred to by its shortened column name, > **lrefer**. **logicalused** > The amount of space that is > "logically" > consumed by this dataset and all its descendants. > See the > **used** > property. > The logical space ignores the effect of the > **compression** > and > **copies** > properties, giving a quantity closer to the amount of data that applications > see. > However, it does include space consumed by metadata. > This property can also be referred to by its shortened column name, > **lused**. **mounted** > For file systems, indicates whether the file system is currently mounted. > This property can be either > **yes** > or > **no**. **objsetid** > A unique identifier for this dataset within the pool. > Unlike the dataset's > **guid**, the **objsetid** > of a dataset is not transferred to other pools when the snapshot is copied > with a send/receive operation. > The > **objsetid** > can be reused (for a new dataset) after the dataset is deleted. **origin** > For cloned file systems or volumes, the snapshot from which the clone was > created. > See also the > **clones** > property. **receive\_resume\_token** > For filesystems or volumes which have saved partially-completed state from > **zfs** **receive** **-s**, > this opaque token can be provided to > **zfs** **send** **-t** > to resume and complete the > **zfs** **receive**. **redact\_snaps** > For bookmarks, this is the list of snapshot GUIDs the bookmark contains a > redaction > list for. > For snapshots, this is the list of snapshot GUIDs the snapshot is redacted with > respect to. **referenced** > The amount of data that is accessible by this dataset, which may or may not be > shared with other datasets in the pool. > When a snapshot or clone is created, it initially references the same amount of > space as the file system or snapshot it was created from, since its contents are > identical. > This property can also be referred to by its shortened column name, > **refer**. **refcompressratio** > The compression ratio achieved for the > **referenced** > space of this dataset, expressed as a multiplier. > See also the > **compressratio** > property. **snapshot\_count** > The total number of snapshots that exist under this location in the dataset > tree. > This value is only available when a > **snapshot\_limit** > has been set somewhere in the tree under which the dataset resides. **type** > The type of dataset: > **filesystem**, > **volume**, > **snapshot**, > or > **bookmark**. **used** > The amount of space consumed by this dataset and all its descendants. > This is the value that is checked against this dataset's quota and reservation. > The space used does not include this dataset's reservation, but does take into > account the reservations of any descendent datasets. > The amount of space that a dataset consumes from its parent, as well as the > amount of space that is freed if this dataset is recursively destroyed, is the > greater of its space used and its reservation. > The used space of a snapshot > (see the > *Snapshots* > section of > zfsconcepts(7)) > is space that is referenced exclusively by this snapshot. > If this snapshot is destroyed, the amount of > **used** > space will be freed. > Space that is shared by multiple snapshots isn't accounted for in this metric. > When a snapshot is destroyed, space that was previously shared with this > snapshot can become unique to snapshots adjacent to it, thus changing the used > space of those snapshots. > The used space of the latest snapshot can also be affected by changes in the > file system. > Note that the > **used** > space of a snapshot is a subset of the > **written** > space of the snapshot. > The amount of space used, available, or referenced does not take into account > pending changes. > Pending changes are generally accounted for within a few seconds. > Committing a change to a disk using > fsync(2) > or > **O\_SYNC** > does not necessarily guarantee that the space usage information is updated > immediately. **usedby\*‌** > The > **usedby\*‌** > properties decompose the > **used** > properties into the various reasons that space is used. > Specifically, > **used** = > **usedbychildren** + > **usedbydataset** + > **usedbyrefreservation** + > **usedbysnapshots**. > These properties are only available for datasets created on > **zpool** > "version 13" > pools. **usedbychildren** > The amount of space used by children of this dataset, which would be freed if > all the dataset's children were destroyed. **usedbydataset** > The amount of space used by this dataset itself, which would be freed if the > dataset were destroyed > (after first removing any > **refreservation** > and destroying any necessary snapshots or descendants) > . **usedbyrefreservation** > The amount of space used by a > **refreservation** > set on this dataset, which would be freed if the > **refreservation** > was removed. **usedbysnapshots** > The amount of space consumed by snapshots of this dataset. > In particular, it is the amount of space that would be freed if all of this > dataset's snapshots were destroyed. > Note that this is not simply the sum of the snapshots' > **used** > properties because space can be shared by multiple snapshots. **userused**@*user* > The amount of space consumed by the specified user in this dataset. > Space is charged to the owner of each file, as displayed by > **ls** **-l**. > The amount of space charged is displayed by > **du** and **ls** **-s**. > See the > **zfs** **userspace** > command for more information. > Unprivileged users can access only their own space usage. > The root user, or a user who has been granted the > **userused** > privilege with > **zfs** **allow**, > can access everyone's usage. > The > **userused**@*…* > properties are not displayed by > **zfs** **get** **all**. > The user's name must be appended after the > **@** > symbol, using one of the following forms: > * POSIX name > ("joe") > * POSIX numeric ID > ("789") > * SID name > ("joe.smith@mydomain") > * SID numeric ID > ("S-1-123-456-789") > Files created on Linux always have POSIX owners. **userobjused**@*user* > The > **userobjused** > property is similar to > **userused** > but instead it counts the number of objects consumed by a user. > This property counts all objects allocated on behalf of the user, > it may differ from the results of system tools such as > **df** **-i**. > When the property > **xattr**=**on** > is set on a file system additional objects will be created per-file to store > extended attributes. > These additional objects are reflected in the > **userobjused** > value and are counted against the user's > **userobjquota**. > When a file system is configured to use > **xattr**=**sa** > no additional internal objects are normally required. **userrefs** > This property is set to the number of user holds on this snapshot. > User holds are set by using the > **zfs** **hold** > command. **groupused**@*group* > The amount of space consumed by the specified group in this dataset. > Space is charged to the group of each file, as displayed by > **ls** **-l**. > See the > **userused**@*user* > property for more information. > Unprivileged users can only access their own groups' space usage. > The root user, or a user who has been granted the > **groupused** > privilege with > **zfs** **allow**, > can access all groups' usage. **groupobjused**@*group* > The number of objects consumed by the specified group in this dataset. > Multiple objects may be charged to the group for each file when extended > attributes are in use. > See the > **userobjused**@*user* > property for more information. > Unprivileged users can only access their own groups' space usage. > The root user, or a user who has been granted the > **groupobjused** > privilege with > **zfs** **allow**, > can access all groups' usage. **projectused**@*project* > The amount of space consumed by the specified project in this dataset. > Project is identified via the project identifier (ID) that is object-based > numeral attribute. > An object can inherit the project ID from its parent object (if the > parent has the flag of inherit project ID that can be set and changed via > **chattr** **-/+P** > or > **zfs project** **-s**) > when being created. > The privileged user can set and change object's project > ID via > **chattr** **-p** > or > **zfs project** **-s** > anytime. > Space is charged to the project of each file, as displayed by > **lsattr** **-p** > or > **zfs project**. > See the > **userused**@*user* > property for more information. > The root user, or a user who has been granted the > **projectused** > privilege with > **zfs allow**, > can access all projects' usage. **projectobjused**@*project* > The > **projectobjused** > is similar to > **projectused** > but instead it counts the number of objects consumed by project. > When the property > **xattr**=**on** > is set on a fileset, ZFS will create additional objects per-file to store > extended attributes. > These additional objects are reflected in the > **projectobjused** > value and are counted against the project's > **projectobjquota**. > When a filesystem is configured to use > **xattr**=**sa** > no additional internal objects are required. > See the > **userobjused**@*user* > property for more information. > The root user, or a user who has been granted the > **projectobjused** > privilege with > **zfs allow**, > can access all projects' objects usage. **snapshots\_changed** > Provides a mechanism to quickly determine whether snapshot list has > changed without having to mount a dataset or iterate the snapshot list. > Specifies the time at which a snapshot for a dataset was last > created or deleted. > This allows us to be more efficient how often we query snapshots. > The property is persistent across mount and unmount operations only if the > **extensible\_dataset** > feature is enabled. **snapshots\_changed\_nsecs** > Specifies the UTC time at which a snapshot for a dataset was last created > or deleted, expressed as the number of nanoseconds since the Unix epoch. > This is a high-precision version of > **snapshots\_changed**, > representing the same instant with nanosecond instead of second resolution. > The property is persistent across mount and unmount operations only if the > **extensible\_dataset** > feature is enabled. **volblocksize** > For volumes, specifies the block size of the volume. > The > **blocksize** > cannot be changed once the volume has been written, so it should be set at > volume creation time. > The size specified must be a power of two greater than or equal to > *512 B* > and less than or equal to > *128 KiB*. > If the > **large\_blocks** > feature is enabled on the pool, the size may be up to > *16 MiB*. > The default size is > *16 KiB*. > This property can also be referred to by its shortened column name, > **volblock**. **written** > The amount of space > **referenced** > by this dataset, that was written since the previous snapshot > (i.e. that is not referenced by the previous snapshot). **written**@*snapshot* > The amount of > **referenced** > space written to this dataset since the specified snapshot. > This is the space that is referenced by this dataset but was not referenced by > the specified snapshot. > The > *snapshot* > may be specified as a short snapshot name > (just the part after the **@**), > in which case it will be interpreted as a snapshot in the same filesystem as > this dataset. > The > *snapshot* > may be a full snapshot name > (*filesystem*@*snapshot*), > which for clones may be a snapshot in the origin's filesystem > (or the origin of the origin's filesystem, etc.) The following native properties can be used to change the behavior of a ZFS dataset. **aclinherit**=**discard**|**noallow**|**restricted**|**passthrough**|**passthrough-x** > Controls how ACEs are inherited when files and directories are created. > **discard** > > does not inherit any ACEs. > **noallow** > > only inherits inheritable ACEs that specify > > "deny" > > permissions. > **restricted** > > default, removes the > > **write\_acl** > > and > > **write\_owner** > > permissions when the ACE is inherited. > **passthrough** > > inherits all inheritable ACEs without any modifications. > **passthrough-x** > > same meaning as > > **passthrough**, > > except that the > > **owner@**, **group@**, and **everyone@** > > ACEs inherit the execute permission only if the file creation mode also requests > > the execute bit. > When the property value is set to > **passthrough**, > files are created with a mode determined by the inheritable ACEs. > If no inheritable ACEs exist that affect the mode, then the mode is set in > accordance to the requested mode from the application. > The > **aclinherit** > property does not apply to POSIX ACLs. **aclmode**=**discard**|**groupmask**|**passthrough**|**restricted** > Controls how an ACL is modified during chmod(2) and how inherited ACEs > are modified by the file creation mode: > **discard** > > default, deletes all > > **ACEs** > > except for those representing > > the mode of the file or directory requested by > > chmod(2). > **groupmask** > > reduces permissions granted in all > > **ALLOW** > > entries found in the > > **ACL** > > such that they are no greater than the group permissions specified by > > chmod(2). > **passthrough** > > indicates that no changes are made to the ACL other than creating or updating > > the necessary ACL entries to represent the new mode of the file or directory. > **restricted** > > will cause the > > chmod(2) > > operation to return an error when used on any file or directory which has > > a non-trivial ACL whose entries can not be represented by a mode. > > chmod(2) > > is required to change the set user ID, set group ID, or sticky bits on a file > > or directory, as they do not have equivalent ACL entries. > > In order to use > > chmod(2) > > on a file or directory with a non-trivial ACL when > > **aclmode** > > is set to > > **restricted**, > > you must first remove all ACL entries which do not represent the current mode. **acltype**=**off**|**nfsv4**|**posix** > Controls whether ACLs are enabled and if so what type of ACL to use. > When this property is set to a type of ACL not supported by the current > platform, the behavior is the same as if it were set to > **off**. > **off** > > default on Linux, when a file system has the > > **acltype** > > property set to off then ACLs are disabled. > **noacl** > > an alias for > > **off** > **nfsv4** > > default on > > FreeBSD, > > indicates that NFSv4-style ZFS ACLs should be used. > > These ACLs can be managed with the > > getfacl(1) > > and > > setfacl(1). > > The > > **nfsv4** > > ZFS ACL type is not yet supported on Linux. > **posix** > > indicates POSIX ACLs should be used. > > POSIX ACLs are specific to Linux and are not functional on other platforms. > > POSIX ACLs are stored as an extended > > attribute and therefore will not overwrite any existing NFSv4 ACLs which > > may be set. > **posixacl** > > an alias for > > **posix** > To obtain the best performance when setting > **posix** > users are strongly encouraged to set the > **xattr**=**sa** > property. > This will result in the POSIX ACL being stored more efficiently on disk. > But as a consequence, all new extended attributes will only be > accessible from OpenZFS implementations which support the > **xattr**=**sa** > property. > See the > **xattr** > property for more details. **atime**=**on**|**off** > Controls whether the access time for files is updated when they are read. > Turning this property off avoids producing write traffic when reading files and > can result in significant performance gains, though it might confuse mailers > and other similar utilities. > The values > **on** > and > **off** > are equivalent to the > **atime** > and > **noatime** > mount options. > The default value is > **on**. > See also > **relatime** > below. **canmount**=**on**|**off**|**noauto** > If this property is set to > **off**, > the file system cannot be mounted, and is ignored by > **zfs** **mount** **-a**. > Setting this property to > **off** > is similar to setting the > **mountpoint** > property to > **none**, > except that the dataset still has a normal > **mountpoint** > property, which can be inherited. > Setting this property to > **off** > allows datasets to be used solely as a mechanism to inherit properties. > One example of setting > **canmount**=**off** > is to have two datasets with the same > **mountpoint**, > so that the children of both datasets appear in the same directory, but might > have different inherited characteristics. > When set to > **noauto**, > a dataset can only be mounted and unmounted explicitly. > The dataset is not mounted automatically when the dataset is created or > imported, nor is it mounted by the > **zfs** **mount** **-a** > command or unmounted by the > **zfs** **unmount** **-a** > command. > This property is not inherited. **checksum**=**on**|**off**|**fletcher2**|**fletcher4**|**sha256**|**noparity**|**sha512**|**skein**|**edonr**|**blake3** > Controls the checksum used to verify data integrity. > The default value is > **on**, > which automatically selects an appropriate algorithm > (currently, > **fletcher4**, > but this may change in future releases) > . > The value > **off** > disables integrity checking on user data. > The value > **noparity** > not only disables integrity but also disables maintaining parity for user data. > This setting is used internally by a dump device residing on a RAID-Z pool and > should not be used by any other dataset. > Disabling checksums is > *NOT* > a recommended practice. > The > **sha512**, > **skein**, > **edonr**, > and > **blake3** > checksum algorithms require enabling the appropriate features on the pool. > Please see > zpool-features(7) > for more information on these algorithms. > Changing this property affects only newly-written data. **compression**=**on**|**off**|**gzip**|**gzip-**‌*N*|**lz4**|**lzjb**|**zle**|**zstd**|**zstd-**‌*N*|**zstd-fast**|**zstd-fast-**‌*N* > Controls the compression algorithm used for this dataset. > When set to > **on** > (the default), indicates that the current default compression algorithm should > be used. > The default balances compression and decompression speed, with compression ratio > and is expected to work well on a wide variety of workloads. > Unlike all other settings for this property, > **on** > does not select a fixed compression type. > As new compression algorithms are added to ZFS and enabled on a pool, the > default compression algorithm may change. > The current default compression algorithm is either > **lzjb** > or, if the > **lz4\_compress** > feature is enabled, > **lz4**. > The > **lz4** > compression algorithm is a high-performance replacement for the > **lzjb** > algorithm. > It features significantly faster compression and decompression, as well as a > moderately higher compression ratio than > **lzjb**, > but can only be used on pools with the > **lz4\_compress** > feature set to > **enabled**. > See > zpool-features(7) > for details on ZFS feature flags and the > **lz4\_compress** > feature. > The > **lzjb** > compression algorithm is optimized for performance while providing decent data > compression. > The > **gzip** > compression algorithm uses the same compression as the > gzip(1) > command. > You can specify the > **gzip** > level by using the value > **gzip-**‌*N*, > where > *N* > is an integer from 1 > (fastest) > to 9 > (best compression ratio). > Currently, > **gzip** > is equivalent to > **gzip-6** > (which is also the default for > gzip(1)) > . > The > **zstd** > compression algorithm provides both high compression ratios and good > performance. > You can specify the > **zstd** > level by using the value > **zstd-**‌*N*, > where > *N* > is an integer from 1 > (fastest) > to 19 > (best compression ratio). > **zstd** > is equivalent to > **zstd-3**. > Faster speeds at the cost of the compression ratio can be requested by > setting a negative > **zstd** > level. > This is done using > **zstd-fast-**‌*N*, > where > *N* > is an integer in > \[**1**-**10**, **20**, **30**, …, **100**, **500**, **1000**] > which maps to a negative > **zstd** > level. > The lower the level the faster the compression — > **1000** > provides the fastest compression and lowest compression ratio. > **zstd-fast** > is equivalent to > **zstd-fast-**‌*1*. > The > **zle** > compression algorithm compresses runs of zeros. > This property can also be referred to by its shortened column name > **compress**. > Changing this property affects only newly-written data. > When any setting except > **off** > is selected, compression will explicitly check for blocks consisting of only > zeroes (the NUL byte). > When a zero-filled block is detected, it is stored as > a hole and not compressed using the indicated compression algorithm. > All blocks are allocated as a whole number of sectors > (chunks of 2^**ashift** bytes, e.g. **512B** or **4KB**). > Compression may result in a non-sector-aligned size, which will be rounded up > to a whole number of sectors. > If compression saves less than one whole sector, > the block will be stored uncompressed. > Therefore, blocks whose logical size is a small number of sectors will > experience less compression > (e.g. for > **recordsize**=**16K** > with > **4K** > sectors, which have 4 sectors per block, > compression needs to save at least 25% to actually save space on disk). > There is > **12.5%** > default compression threshold in addition to sector rounding. **context**=**none**|*SELinux-User*:*SELinux-Role*:*SELinux-Type*:*Sensitivity-Level* > This flag sets the SELinux context for all files in the file system under > a mount point for that file system. > See > selinux(8) > for more information. **fscontext**=**none**|*SELinux-User*:*SELinux-Role*:*SELinux-Type*:*Sensitivity-Level* > This flag sets the SELinux context for the file system file system being > mounted. > See > selinux(8) > for more information. **defcontext**=**none**|*SELinux-User*:*SELinux-Role*:*SELinux-Type*:*Sensitivity-Level* > This flag sets the SELinux default context for unlabeled files. > See > selinux(8) > for more information. **rootcontext**=**none**|*SELinux-User*:*SELinux-Role*:*SELinux-Type*:*Sensitivity-Level* > This flag sets the SELinux context for the root inode of the file system. > See > selinux(8) > for more information. **copies**=**1**|**2**|**3** > Controls the number of copies of data stored for this dataset. > These copies are in addition to any redundancy provided by the pool, for > example, mirroring or RAID-Z. > The copies are stored on different disks, if possible. > The space used by multiple copies is charged to the associated file and dataset, > changing the > **used** > property and counting against quotas and reservations. > Changing this property only affects newly-written data. > Therefore, set this property at file system creation time by using the > **-o** **copies**=*N* > option. > Remember that ZFS will not import a pool with a missing top-level vdev. > Do > *NOT* > create, for example a two-disk striped pool and set > **copies**=*2* > on some datasets thinking you have setup redundancy for them. > When a disk fails you will not be able to import the pool > and will have lost all of your data. > Encrypted datasets may not have > **copies**=*3* > since the implementation stores some encryption metadata where the third copy > would normally be. **devices**=**on**|**off** > Controls whether device nodes can be opened on this file system. > The default value is > **on**. > The values > **on** > and > **off** > are equivalent to the > **dev** > and > **nodev** > mount options. **dedup**=**off**|**on**|**verify**|**sha256**\[,**verify**]|**sha512**\[,**verify**]|**skein**\[,**verify**]|**edonr**,**verify**|**blake3**\[,**verify**] > Configures deduplication for a dataset. > The default value is > **off**. > The default deduplication checksum is > **sha256** > (this may change in the future). > When > **dedup** > is enabled, the checksum defined here overrides the > **checksum** > property. > Setting the value to > **verify** > has the same effect as the setting > **sha256**,**verify**. > If set to > **verify**, > ZFS will do a byte-to-byte comparison in case of two blocks having the same > signature to make sure the block contents are identical. > Specifying > **verify** > is mandatory for the > **edonr** > algorithm. > Unless necessary, deduplication should > *not* > be enabled on a system. > See the > *Deduplication* > section of > zfsconcepts(7). **direct**=**disabled**|**standard**|**always** > Controls the behavior of Direct I/O requests > (e.g. `O_DIRECT`). > The > **standard** > behavior for Direct I/O requests is to bypass the ARC when possible. > These requests will not be cached and performance will be limited by the > raw speed of the underlying disks > (`this is the default`). > **always** > causes every properly aligned read or write to be treated as a direct request. > **disabled** > causes the O\_DIRECT flag to be silently ignored and all direct requests will > be handled by the ARC. > This is the default behavior for OpenZFS 2.2 and prior releases. > Bypassing the ARC requires that a direct request be correctly aligned. > For write requests the starting offset and size of the request must be > **recordsize**-aligned, if not then the unaligned portion of the request will be silently > redirected through the ARC. > For read requests there is no > **recordsize** > alignment restriction on either the starting offset or size. > All direct requests must use a page-aligned memory buffer and the request > size must be a multiple of the page size or an error is returned. > Concurrently mixing buffered and direct requests to overlapping regions of > a file can decrease performance. > However, the resulting file will always be coherent. > For example, a direct read after a buffered write will return the data > from the buffered write. > Furthermore, if an application uses > mmap(2) > based file access then in order to maintain coherency all direct requests > are converted to buffered requests while the file is mapped. > Currently Direct I/O is not supported with zvols. > If dedup is enabled on a dataset, Direct I/O writes will not check for > deduplication. > Deduplication and Direct I/O writes are currently incompatible. **dnodesize**=**legacy**|**auto**|**1k**|**2k**|**4k**|**8k**|**16k** > Specifies a compatibility mode or literal value for the size of dnodes in the > file system. > The default value is > **legacy**. > Setting this property to a value other than > **legacy** requires the **large\_dnode** pool feature to be enabled. > Consider setting > **dnodesize** > to > **auto** > if the dataset uses the > **xattr**=**sa** > property setting and the workload makes heavy use of extended attributes. > This > may be applicable to SELinux-enabled systems, Lustre servers, and Samba > servers, for example. > Literal values are supported for cases where the optimal > size is known in advance and for performance testing. > Leave > **dnodesize** > set to > **legacy** > if you need to receive a send stream of this dataset on a pool that doesn't > enable the > **large\_dnode** > feature, or if you need to import this pool on a system that doesn't support the > **large\_dnode** feature. > This property can also be referred to by its shortened column name, > **dnsize**. **encryption**=**off**|**on**|**aes-128-ccm**|**aes-192-ccm**|**aes-256-ccm**|**aes-128-gcm**|**aes-192-gcm**|**aes-256-gcm** > Controls the encryption cipher suite (block cipher, key length, and mode) used > for this dataset. > Requires the > **encryption** > feature to be enabled on the pool. > Requires a > **keyformat** > to be set at dataset creation time. > Selecting > **encryption**=**on** > when creating a dataset indicates that the default encryption suite will be > selected, which is currently > **aes-256-gcm**. > In order to provide consistent data protection, encryption must be specified at > dataset creation time and it cannot be changed afterwards. > For more details and caveats about encryption see the > *Encryption* > section of > zfs-load-key(8). **keyformat**=**raw**|**hex**|**passphrase** > Controls what format the user's encryption key will be provided as. > This property is only set when the dataset is encrypted. > Raw keys and hex keys must be 32 bytes long (regardless of the chosen > encryption suite) and must be randomly generated. > A raw key can be generated with the following command: > > \# **dd** **if=/dev/urandom bs=32 count=1** **of=**‌*/path/to/output/key* > Passphrases must be between 8 and 512 bytes long and will be processed through > PBKDF2 before being used (see the > **pbkdf2iters** > property). > Even though the encryption suite cannot be changed after dataset creation, > the keyformat can be with > **zfs** **change-key**. **keylocation**=**prompt**|**file://**‌*/absolute/file/path*|**https://**‌*address*|**http://**‌*address* > Controls where the user's encryption key will be loaded from by default for > commands such as > **zfs** **load-key** > and > **zfs** **mount** **-l**. > This property is only set for encrypted datasets which are encryption roots. > If unspecified, the default is > **prompt**. > Even though the encryption suite cannot be changed after dataset creation, the > keylocation can be with either > **zfs** **set** > or > **zfs** **change-key**. > If > **prompt** > is selected, ZFS will expect the key to be provided when it is required to > access the encrypted data (see > **zfs** **load-key** > for details). > If stdin is a TTY, then ZFS will ask for the key to be provided. > Otherwise, stdin is expected to be the key to use and will be processed as such. > Users should be careful not to place keys which should be kept secret on the > command line, as most operating systems may expose command line arguments to > other processes. > If the > "raw" > **keyformat** > was used, then the key must be provided via stdin. > If a file URL is selected, the key will be loaded from the > specified absolute file path. > If an HTTPS or HTTP URL is selected, it will be GETted using > fetch(3), > libcurl, or nothing, depending on compile-time configuration and run-time > availability. > The > **SSL\_CA\_CERT\_FILE** > environment variable can be set to set the location > of the concatenated certificate store. > The > **SSL\_CA\_CERT\_PATH** > environment variable can be set to override the location > of the directory containing the certificate authority bundle. > The > **SSL\_CLIENT\_CERT\_FILE** > and > **SSL\_CLIENT\_KEY\_FILE** > environment variables can be set to configure the path > to the client certificate and its key. **pbkdf2iters**=*iterations* > Controls the number of PBKDF2 iterations that a > **passphrase** > encryption key should be run through when processing it into an encryption key. > This property is only defined when encryption is enabled and a keyformat of > **passphrase** > is selected. > The goal of PBKDF2 is to significantly increase the > computational difficulty needed to brute force a user's passphrase. > This is accomplished by forcing the attacker to run each passphrase through a > computationally expensive hashing function many times before they arrive at the > resulting key. > A user who actually knows the passphrase will only have to pay this cost once. > As CPUs become better at processing, this number should be > raised to ensure that a brute force attack is still not possible. > The current default is > **350000** > and the minimum is > **100000**. > This property may be changed with > **zfs** **change-key**. **exec**=**on**|**off** > Controls whether processes can be executed from within this file system. > The default value is > **on**. > The values > **on** > and > **off** > are equivalent to the > **exec** > and > **noexec** > mount options. **volthreading**=**on**|**off** > Controls internal zvol threading. > The value > **off** > disables zvol threading, and zvol relies on application threads. > The default value is > **on**, > which enables threading within a zvol. > Please note that this property will be overridden by > **zvol\_request\_sync** > module parameter. > This property is only applicable to Linux. **filesystem\_limit**=*count*|**none** > Limits the number of filesystems and volumes that can exist under this point in > the dataset tree. > The limit is not enforced if the user is allowed to change the limit. > Setting a > **filesystem\_limit** > to > **on** > a descendant of a filesystem that already has a > **filesystem\_limit** > does not override the ancestor's > **filesystem\_limit**, > but rather imposes an additional limit. > This feature must be enabled to be used > (see > zpool-features(7)) > . **special\_small\_blocks**=*size* > This value represents the threshold block size for including small file > or zvol blocks into the special allocation class. > Blocks smaller than or equal to this value after compression and encryption > will be assigned to the special allocation class, while greater blocks will > be assigned to the regular class. > Valid values are from 0 to maximum block size ( > *16 MiB* > ). > The default size is 0 which means no small file or zvol blocks > will be allocated in the special class. > Before setting this property, a special class vdev must be added to the > pool. > See > zpoolconcepts(7) > for more details on the special allocation class. **mountpoint**=*path*|**none**|**legacy** > Controls the mount point used for this file system. > See the > *Mount Points* > section of > zfsconcepts(7) > for more information on how this property is used. > When the > **mountpoint** > property is changed for a file system, the file system and any children that > inherit the mount point are unmounted. > If the new value is > **legacy**, > then they remain unmounted. > Otherwise, they are automatically remounted in the new location if the property > was previously > **legacy** > or > **none**. > In addition, any shared file systems are unshared and shared in the new > location. > When the > **mountpoint** > property is set with > **zfs** **set** **-u** > , the > **mountpoint** > property is updated but dataset is not mounted or unmounted and remains > as it was before. **nbmand**=**on**|**off** > Controls whether the file system should be mounted with > **nbmand** > (Non-blocking mandatory locks). > Changes to this property only take effect when the file system is unmounted and > remounted. > This was only supported by Linux prior to 5.15, and was buggy there, > and is not supported by > FreeBSD. > On Solaris it's used for SMB clients. **longname**=**on**|**off** > Controls support for filenames longer than 255 bytes, up to 1023 bytes. > The default is > **off**. > Setting this property to > **on** > activates the > **longname** > pool feature, which must be enabled > (see > zpool-features(7)) > . > Once a file with a long name is created, the feature becomes active and > the pool can no longer be imported by an OpenZFS implementation that does > not support it. **overlay**=**on**|**off** > Allow mounting on a busy directory or a directory which already contains > files or directories. > This is the default mount behavior for Linux and > FreeBSD > file systems. > On these platforms the property is > **on** > by default. > Set to > **off** > to disable overlay mounts for consistency with OpenZFS on other platforms. **primarycache**=**all**|**none**|**metadata** > Controls what is cached in the primary cache > (ARC). > If this property is set to > **all**, > then both user data and metadata is cached. > If this property is set to > **none**, > then neither user data nor metadata is cached. > If this property is set to > **metadata**, > then only metadata is cached. > The default value is > **all**. **quota**=*size*|**none** > Limits the amount of space a dataset and its descendants can consume. > This property enforces a hard limit on the amount of space used. > This includes all space consumed by descendants, including file systems and > snapshots. > Setting a quota on a descendant of a dataset that already has a quota does not > override the ancestor's quota, but rather imposes an additional limit. > Quotas cannot be set on volumes, as the > **volsize** > property acts as an implicit quota. **snapshot\_limit**=*count*|**none** > Limits the number of snapshots that can be created on a dataset and its > descendants. > Setting a > **snapshot\_limit** > on a descendant of a dataset that already has a > **snapshot\_limit** > does not override the ancestor's > **snapshot\_limit**, > but rather imposes an additional limit. > The limit is not enforced if the user is allowed to change the limit. > For example, this means that recursive snapshots taken from the global zone are > counted against each delegated dataset within a zone. > This feature must be enabled to be used > (see > zpool-features(7)) > . **userquota@**‌*user*=*size*|**none** > Limits the amount of space consumed by the specified user. > User space consumption is identified by the > **userspace@**‌*user* > property. > Enforcement of user quotas may be delayed by several seconds. > This delay means that a user might exceed their quota before the system notices > that they are over quota and begins to refuse additional writes with the > `EDQUOT` > error message. > See the > **zfs** **userspace** > command for more information. > Unprivileged users can only access their own groups' space usage. > The root user, or a user who has been granted the > **userquota** > privilege with > **zfs** **allow**, > can get and set everyone's quota. > This property is not available on volumes, on file systems before version 4, or > on pools before version 15. > The > **userquota@**‌*…* > properties are not displayed by > **zfs** **get** **all**. > The user's name must be appended after the > **@** > symbol, using one of the following forms: > * POSIX name > ("joe") > * POSIX numeric ID > ("789") > * SID name > ("joe.smith@mydomain") > * SID numeric ID > ("S-1-123-456-789") > Files created on Linux always have POSIX owners. **defaultuserquota**=*size*|**none** > Sets a default user quota to be applied to each user for whom no > user-specific quota is set. > The value > **0** > disables defaultuserquota. **userobjquota@**‌*user*=*size*|**none** > The > **userobjquota** > is similar to > **userquota** > but it limits the number of objects a user can create. > Please refer to > **userobjused** > for more information about how objects are counted. **defaultuserobjquota**=*size*|**none** > Sets a default user object quota to be applied to each user for > whom no userobj-specific quota is set. > The value > **0** > disables defaultuserobjquota. **groupquota@**‌*group*=*size*|**none** > Limits the amount of space consumed by the specified group. > Group space consumption is identified by the > **groupused@**‌*group* > property. > Unprivileged users can access only their own groups' space usage. > The root user, or a user who has been granted the > **groupquota** > privilege with > **zfs** **allow**, > can get and set all groups' quotas. **defaultgroupquota**=*size*|**none** > Sets a default group quota to be applied to each group for whom no > group-specific quota is set. > The value > **0** > disables defaultgroupquota. **groupobjquota@**‌*group*=*size*|**none** > The > **groupobjquota** > is similar to > **groupquota** > but it limits number of objects a group can consume. > Please refer to > **userobjused** > for more information about how objects are counted. **defaultgroupobjquota**=*size*|**none** > Sets a default group object quota to be applied to each group for > whom no groupobj-specific quota is set. > The value > **0** > disables defaultgroupobjquota. **projectquota@**‌*project*=*size*|**none** > Limits the amount of space consumed by the specified project. > Project space consumption is identified by the > **projectused@**‌*project* > property. > Please refer to > **projectused** > for more information about how project is identified and set/changed. > The root user, or a user who has been granted the > **projectquota** > privilege with > **zfs allow**, > can access all projects' quota. **defaultprojectquota**=*size*|**none** > Sets a default project quota to be applied to each project for whom no > project-specific quota is set. > The value > **0** > disables defaultprojectquota. **projectobjquota@**‌*project*=*size*|**none** > The > **projectobjquota** > is similar to > **projectquota** > but it limits number of objects a project can consume. > Please refer to > **userobjused** > for more information about how objects are counted. **defaultprojectobjquota**=*size*|**none** > Sets a default project object quota to be applied to each project for > whom no projectobj-specific quota is set. > The value > **0** > disables defaultprojectobjquota. **readonly**=**on**|**off** > Controls whether this dataset can be modified. > The default value is > **off**. > The values > **on** > and > **off** > are equivalent to the > **ro** > and > **rw** > mount options. > This property can also be referred to by its shortened column name, > **rdonly**. **recordsize**=*size* > Specifies a suggested block size for files in the file system. > This property is designed solely for use with database workloads that access > files in fixed-size records. > ZFS automatically tunes block sizes according to internal algorithms optimized > for typical access patterns. > For databases that create very large files but access them in small random > chunks, these algorithms may be suboptimal. > Specifying a > **recordsize** > greater than or equal to the record size of the database can result in > significant performance gains. > Use of this property for general purpose file systems is strongly discouraged, > and may adversely affect performance. > The size specified must be a power of two greater than or equal to > *512 B* > and less than or equal to > *128 KiB*. > If the > **large\_blocks** > feature is enabled on the pool, the size may be up to > *16 MiB*. > See > zpool-features(7) > for details on ZFS feature flags. > Note that maximum size is still limited by default to > *1 MiB* > on x86\_32, see > **zfs\_max\_recordsize** > module parameter. > Changing the file system's > **recordsize** > affects only files created afterward; existing files are unaffected. > This property can also be referred to by its shortened column name, > **recsize**. **redundant\_metadata**=**all**|**most**|**some**|**none** > Controls what types of metadata are stored redundantly. > ZFS stores an extra copy of metadata, so that if a single block is corrupted, > the amount of user data lost is limited. > This extra copy is in addition to any redundancy provided at the pool level > (e.g. by mirroring or RAID-Z), > and is in addition to an extra copy specified by the > **copies** > property > (up to a total of 3 copies). > For example if the pool is mirrored, > **copies**=2, > and > **redundant\_metadata**=**most**, > then ZFS stores 6 copies of most metadata, and 4 copies of data and some > metadata. > When set to > **all**, > ZFS stores an extra copy of all metadata. > If a single on-disk block is corrupt, at worst a single block of user data > (which is > **recordsize** > bytes long) > can be lost. > When set to > **most**, > ZFS stores an extra copy of most types of metadata. > This can improve performance of random writes, because less metadata must be > written. > In practice, at worst about 1000 blocks > (of > **recordsize** > bytes each) > of user data can be lost if a single on-disk block is corrupt. > The exact behavior of which metadata blocks are stored redundantly may change in > future releases. > When set to > **some**, > ZFS stores an extra copy of only critical metadata. > This can improve file create performance since less metadata > needs to be written. > If a single on-disk block is corrupt, multiple user files or directories > can be lost. > When set to > **none**, > ZFS does not store any copies of metadata redundantly. > If a single on-disk block is corrupt, an entire dataset can be lost. > The default value is > **all**. **refquota**=*size*|**none** > Limits the amount of space a dataset can consume. > This property enforces a hard limit on the amount of space used. > This hard limit does not include space used by descendants, including file > systems and snapshots. **refreservation**=*size*|**none**|**auto** > The minimum amount of space guaranteed to a dataset, not including its > descendants. > When the amount of space used is below this value, the dataset is treated as if > it were taking up the amount of space specified by > **refreservation**. > The > **refreservation** > reservation is accounted for in the parent datasets' space used, and counts > against the parent datasets' quotas and reservations. > If > **refreservation** > is set, a snapshot is only allowed if there is enough free pool space outside of > this reservation to accommodate the current number of > "referenced" > bytes in the dataset. > If > **refreservation** > is set to > **auto**, > a volume is thick provisioned > (or > "not sparse") > . > **refreservation**=**auto** > is only supported on volumes. > See > **volsize** > in the > *Native Properties* > section for more information about sparse volumes. > This property can also be referred to by its shortened column name, > **refreserv**. **relatime**=**on**|**off** > Controls the manner in which the access time is updated when > **atime**=**on** > is set. > Turning this property on causes the access time to be updated relative > to the modify or change time. > Access time is only updated if the previous > access time was earlier than the current modify or change time or if the > existing access time hasn't been updated within the past 24 hours. > The default value is > **on**. > The values > **on** > and > **off** > are equivalent to the > **relatime** > and > **norelatime** > mount options. **reservation**=*size*|**none** > The minimum amount of space guaranteed to a dataset and its descendants. > When the amount of space used is below this value, the dataset is treated as if > it were taking up the amount of space specified by its reservation. > Reservations are accounted for in the parent datasets' space used, and count > against the parent datasets' quotas and reservations. > This property can also be referred to by its shortened column name, > **reserv**. **secondarycache**=**all**|**none**|**metadata** > Controls what is cached in the secondary cache > (L2ARC). > If this property is set to > **all**, > then both user data and metadata is cached. > If this property is set to > **none**, > then neither user data nor metadata is cached. > If this property is set to > **metadata**, > then only metadata is cached. > The default value is > **all**. **prefetch**=**all**|**none**|**metadata** > Controls what speculative prefetch does. > If this property is set to > **all**, > then both user data and metadata are prefetched. > If this property is set to > **none**, > then neither user data nor metadata are prefetched. > If this property is set to > **metadata**, > then only metadata are prefetched. > The default value is > **all**. > Please note that the module parameter zfs\_prefetch\_disable=1 can > be used to totally disable speculative prefetch, bypassing anything > this property does. **setuid**=**on**|**off** > Controls whether the setuid bit is respected for the file system. > The default value is > **on**. > The values > **on** > and > **off** > are equivalent to the > **suid** > and > **nosuid** > mount options. **sharesmb**=**on**|**off**|*opts* > Controls whether the file system is shared by using > **Samba USERSHARES** > and what options are to be used. > Otherwise, the file system is automatically shared and unshared with the > **zfs** **share** > and > **zfs** **unshare** > commands. > If the property is set to on, the > net(8) > command is invoked to create a > **USERSHARE**. > Because SMB shares requires a resource name, a unique resource name is > constructed from the dataset name. > The constructed name is a copy of the > dataset name except that the characters in the dataset name, which would be > invalid in the resource name, are replaced with underscore (\_) characters. > Linux does not currently support additional options which might be available > on Solaris. > If the > **sharesmb** > property is set to > **off**, > the file systems are unshared. > The share is created with the ACL (Access Control List) "Everyone:F" ("F" > stands for "full permissions", i.e. read and write permissions) and no guest > access (which means Samba must be able to authenticate a real user — > passwd(5)/shadow(5)-, > LDAP- or > smbpasswd(5)-based) > by default. > This means that any additional access control > (disallow specific user specific access etc) must be done on the underlying file > system. > When the > **sharesmb** > property is updated with > **zfs** **set** **-u** > , the property is set to desired value, but the operation to share, reshare > or unshare the the dataset is not performed. **sharenfs**=**on**|**off**|*opts* > Controls whether the file system is shared via NFS, and what options are to be > used. > A file system with a > **sharenfs** > property of > **off** > is managed with the > exportfs(8) > command and entries in the > */etc/exports* > file. > Otherwise, the file system is automatically shared and unshared with the > **zfs** **share** > and > **zfs** **unshare** > commands. > If the property is set to > **on**, > the dataset is shared using the default options: > > sec=sys,rw,crossmnt,no\_subtree\_check > Please note that the options are comma-separated, unlike those found in > exports(5). > This is done to negate the need for quoting, as well as to make parsing > with scripts easier. > For > FreeBSD, > there may be multiple sets of options separated by semicolon(s). > Each set of options must apply to different hosts or networks and each > set of options will create a separate line for > exports(5). > Any semicolon separated option set that consists entirely of whitespace > will be ignored. > This use of semicolons is only for > FreeBSD > at this time. > See > exports(5) > for the meaning of the default options. > Otherwise, the > exportfs(8) > command is invoked with options equivalent to the contents of this property. > When the > **sharenfs** > property is changed for a dataset, the dataset and any children inheriting the > property are re-shared with the new options, only if the property was previously > **off**, > or if they were shared before the property was changed. > If the new property is > **off**, > the file systems are unshared. > When the > **sharenfs** > property is updated with > **zfs** **set** **-u** > , the property is set to desired value, but the operation to share, reshare > or unshare the the dataset is not performed. **logbias**=**latency**|**throughput** > Provide a hint to ZFS about handling of synchronous write requests in this > dataset. > If > **logbias** > is set to > **latency** > (the default), > ZFS will use pool log devices > (if configured) > to handle the write requests at low latency. > If > **logbias** > is set to > **throughput**, > ZFS will not use configured pool log devices to store written data. > ZFS will instead optimize synchronous operations for global pool throughput and > efficient use of resources. **snapdev**=**hidden**|**visible** > Controls whether the volume snapshot devices under > */dev/zvol/*<*pool*> > are hidden or visible. > The default value is > **hidden**. **snapdir**=**disabled**|**hidden**|**visible** > Controls whether the > *.zfs* > directory is disabled, hidden or visible in the root of the file system as > discussed in the > *Snapshots* > section of > zfsconcepts(7). > The default value is > **hidden**. **sync**=**standard**|**always**|**disabled** > Controls the behavior of synchronous requests > (e.g. fsync, O\_DSYNC). > **standard** > is the POSIX-specified behavior of ensuring all synchronous requests > are written to stable storage and all devices are flushed to ensure > data is not cached by device controllers > (this is the default). > **always** > causes every file system transaction to be written and flushed before its > system call returns. > This has a large performance penalty. > **disabled** > disables synchronous requests. > File system transactions are only committed to stable storage periodically. > This option will give the highest performance. > However, it is very dangerous as ZFS would be ignoring the synchronous > transaction demands of applications such as databases or NFS. > Administrators should only use this option when the risks are understood. **version**=*N*|**current** > The on-disk version of this file system, which is independent of the pool > version. > This property can only be set to later supported versions. > See the > **zfs** **upgrade** > command. **volsize**=*size* > For volumes, specifies the logical size of the volume. > By default, creating a volume establishes a reservation of equal size. > For storage pools with a version number of 9 or higher, a > **refreservation** > is set instead. > Any changes to > **volsize** > are reflected in an equivalent change to the reservation > (or **refreservation**). > The > **volsize** > can only be set to a multiple of > **volblocksize**, > and cannot be zero. > The reservation is kept equal to the volume's logical size to prevent unexpected > behavior for consumers. > Without the reservation, the volume could run out of space, resulting in > undefined behavior or data corruption, depending on how the volume is used. > These effects can also occur when the volume size is changed while it is in use > (particularly when shrinking the size). > Extreme care should be used when adjusting the volume size. > Though not recommended, a > "sparse volume" > (also known as > "thin provisioned") > can be created by specifying the > **-s** > option to the > **zfs** **create** **-V** > command, or by changing the value of the > **refreservation** > property > (or > **reservation** > property on pool version 8 or earlier) > after the volume has been created. > A > "sparse volume" > is a volume where the value of > **refreservation** > is less than the size of the volume plus the space required to store its > metadata. > Consequently, writes to a sparse volume can fail with > `ENOSPC` > when the pool is low on space. > For a sparse volume, changes to > **volsize** > are not reflected in the > **refreservation**. > A volume that is not sparse is said to be > "thick provisioned". > A sparse volume can become thick provisioned by setting > **refreservation** > to > **auto**. **volmode**=**default**|**full**|**geom**|**dev**|**none** > This property specifies how volumes should be exposed to the OS. > Setting it to > **full** > exposes volumes as fully fledged block devices, providing maximal > functionality. > The value > **geom** > is just an alias for > **full** > and is kept for compatibility. > Setting it to > **dev** > hides its partitions. > Volumes with property set to > **none** > are not exposed outside ZFS, but can be snapshotted, cloned, replicated, etc, > that can be suitable for backup purposes. > Value > **default** > means that volumes exposition is controlled by system-wide tunable > **zvol\_volmode**, > where > **full**, > **dev** > and > **none** > are encoded as 1, 2 and 3 respectively. > The default value is > **full**. **vscan**=**on**|**off** > Controls whether regular files should be scanned for viruses when a file is > opened and closed. > In addition to enabling this property, the virus scan service must also be > enabled for virus scanning to occur. > The default value is > **off**. > This property is not used by OpenZFS. **xattr**=**on**|**off**|**dir**|**sa** > Controls whether extended attributes are enabled for this file system. > Two styles of extended attributes are supported: either directory-based > or system-attribute-based. > Directory-based extended attributes can be enabled by setting the value to > **dir**. > This style of extended attribute imposes no practical limit > on either the size or number of attributes which can be set on a file. > Although under Linux the > getxattr(2) > and > setxattr(2) > system calls limit the maximum size to > **64K**. > This is the most compatible > style of extended attribute and is supported by all ZFS implementations. > System-attribute-based xattrs can be enabled by setting the value to > **sa** > (default and equal to > **on** > ) . > The key advantage of this type of xattr is improved performance. > Storing extended attributes as system attributes > significantly decreases the amount of disk I/O required. > Up to > **64K** > of data may be stored per-file in the space reserved for system attributes. > If there is not enough space available for an extended attribute > then it will be automatically written as a directory-based xattr. > System-attribute-based extended attributes are not accessible > on platforms which do not support the > **xattr**=**sa** > feature. > OpenZFS supports > **xattr**=**sa** > on both > FreeBSD > and Linux. > The use of system-attribute-based xattrs is strongly encouraged for users of > SELinux or POSIX ACLs. > Both of these features heavily rely on extended > attributes and benefit significantly from the reduced access time. > The values > **on** > and > **off** > are equivalent to the > **xattr** > and > **noxattr** > mount options. **jailed**=**off**|**on** > Controls whether the dataset is managed from a jail. > See > zfs-jail(8) > for more information. > Jails are a > FreeBSD > feature and this property is not available on other platforms. **zoned**=**off**|**on** > Controls whether the dataset is managed from a non-global zone or namespace. > See > zfs-zone(8) > for more information. > Zoning is a > Linux > feature and this property is not available on other platforms. **zoned\_uid**=*uid* > Delegates dataset visibility and administration to all user namespaces > owned by the specified UID. > This property enables rootless container support with native ZFS storage. > For example, setting > **zoned\_uid**=1000 > allows user 1000's rootless Podman containers to use ZFS for storage layers. > This is a Linux-only feature. > Authorization uses an additive three-layer model where all layers must pass: > **L0 (authentication)** > > The user namespace owner UID must match the > > **zoned\_uid** > > value. > **L1 (dsl\_deleg)** > > The pool administrator must grant per-operation permissions on the > > delegation root using > > zfs-allow(8). > > When pool delegation is OFF > > (**zpool** **set** **delegation**=**off**), > > all write operations are denied regardless of capabilities. > **L2 (capability tier)** > > Linux capabilities within the user namespace determine the permitted > > operation class: > > **CAP\_FOWNER** > > for non-destructive operations > > (create, snapshot, set property), > > **CAP\_SYS\_ADMIN** > > for destructive operations > > (destroy, rename, clone). > > Both are namespaced capabilities scoped to the user namespace, > > not the init namespace. > Read-only operations > (**zfs** **list**, **zfs** **get**) > require no capabilities and no > **zfs** **allow** > grants; visibility is controlled solely by the > **zoned\_uid** > delegation scoping. > Write operations that can be delegated include > **zfs** **create**, > **zfs** **destroy**, > **zfs** **snapshot**, > **zfs** **clone**, > **zfs** **rename** > (within the delegation subtree), > and > **zfs** **set**. > The delegation root dataset > (where zoned\_uid is locally set) > cannot be destroyed from within the user namespace, protecting the > parent dataset from unauthorized removal. > Renames are also constrained to remain within the delegation subtree. > The namespace user cannot modify the > **zoned\_uid** > or > **zoned** > properties, and cannot override > **filesystem\_limit** > or > **snapshot\_limit** > set by the administrator on the delegation root > (but can impose tighter sub-limits on child datasets). > Set to > **0** > (or inherit) > to disable UID-based delegation. > Unlike > **zfs** **zone** > which requires an existing namespace file, > **zoned\_uid** > applies to any user namespace owned by the specified UID, > making it suitable for container runtimes that create new namespaces > on each invocation. > See > zfs-zone(8) > for namespace-specific delegation. > Example setup for rootless Podman: > > \# zfs create tank/containers > > \# zfs set zoned\_uid=1000 tank/containers > > \# zfs set mountpoint=none tank/containers > > \# zfs allow -u 1000 create,destroy,mount,snapshot,rename,clone tank/containers The following three properties cannot be changed after the file system is created, and therefore, should be set when the file system is created. If the properties are not set with the **zfs** **create** or **zpool** **create** commands, these properties are inherited from the parent dataset. If the parent dataset lacks these properties due to having been created prior to these features being supported, the new file system will have the default values for these properties. **casesensitivity**=**sensitive**|**insensitive**|**mixed** > Indicates whether the file name matching algorithm used by the file system > should be case-sensitive, case-insensitive, or allow a combination of both > styles of matching. > The default value for the > **casesensitivity** > property is > **sensitive**. > Traditionally, > UNIX > and POSIX file systems have case-sensitive file names. > The > **mixed** > value for the > **casesensitivity** > property indicates that the file system can support requests for both > case-sensitive and case-insensitive matching behavior. > Currently, case-insensitive matching behavior on a file system that supports > mixed behavior is limited to the SMB server product. > For more information about the > **mixed** > value behavior, see the "ZFS Administration Guide". **normalization**=**none**|**formC**|**formD**|**formKC**|**formKD** > Indicates whether the file system should perform a > **Unicode** > normalization of file names whenever two file names are compared, and which > normalization algorithm should be used. > File names are always stored unmodified, names are normalized as part of any > comparison process. > If this property is set to a legal value other than > **none**, > and the > **utf8only** > property was left unspecified, the > **utf8only** > property is automatically set to > **on**. > The default value of the > **normalization** > property is > **none**. > This property cannot be changed after the file system is created. **utf8only**=**on**|**off** > Indicates whether the file system should reject file names that include > characters that are not present in the > **UTF-8** > character code set. > If this property is explicitly set to > **off**, > the normalization property must either not be explicitly set or be set to > **none**. > The default value for the > **utf8only** > property is > **off**. > This property cannot be changed after the file system is created. The **casesensitivity**, **normalization**, and **utf8only** properties are also new permissions that can be assigned to non-privileged users by using the ZFS delegated administration feature. ## Temporary Mount Point Properties When a file system is mounted, either through mount(8) for legacy mounts or the **zfs** **mount** command for normal file systems, its mount options are set according to its properties. The correlation between properties and mount options is as follows: **atime** > atime/noatime **canmount** > auto/noauto **devices** > dev/nodev **exec** > exec/noexec **readonly** > ro/rw **relatime** > relatime/norelatime **setuid** > suid/nosuid **xattr** > xattr/noxattr **nbmand** > mand/nomand **context**= > context= **fscontext**= > fscontext= **defcontext**= > defcontext= **rootcontext**= > rootcontext= In addition, these options can be set on a per-mount basis using the **-o** option, without affecting the property that is stored on disk. The values specified on the command line override the values stored in the dataset. The **nosuid** option is an alias for **nodevices**,**nosetuid**. These properties are reported as "temporary" by the **zfs** **get** command. If the properties are changed while the dataset is mounted, the new setting overrides any temporary settings. ## User Properties In addition to the standard native properties, ZFS supports arbitrary user properties. User properties have no effect on ZFS behavior, but applications or administrators can use them to annotate datasets (file systems, volumes, and snapshots). User property names must contain a colon ("**:**") character to distinguish them from native properties. They may contain lowercase letters, numbers, and the following punctuation characters: colon ("**:**"), dash ("**-**"), period ("**.**"), and underscore ("**\_**"). The expected convention is that the property name is divided into two portions such as *module*:*property*, but this namespace is not enforced by ZFS. User property names can be at most 256 characters, and cannot begin with a dash ("**-**"). When making programmatic use of user properties, it is strongly suggested to use a reversed DNS domain name for the *module* component of property names to reduce the chance that two independently-developed packages use the same property name for different purposes. The values of user properties are arbitrary strings, are always inherited, and are never validated. All of the commands that operate on properties (**zfs** **list**, **zfs** **get**, **zfs** **set**, and so forth) can be used to manipulate both native properties and user properties. Use the **zfs** **inherit** command to clear a user property. If the property is not defined in any parent dataset, it is removed entirely. Property values are limited to 8192 bytes. Debian - September 13, 2025 ZPOOL-FEATURES(7) - Miscellaneous Information Manual # NAME **zpool-features** - description of ZFS pool features # DESCRIPTION ZFS pool on-disk format versions are specified via "features" which replace the old on-disk format numbers (the last supported on-disk format number is 28). To enable a feature on a pool use the **zpool** **upgrade**, or set the **feature**@*feature-name* property to **enabled**. Please also see the *Compatibility feature sets* section for information on how sets of features may be enabled together. The pool format does not affect file system version compatibility or the ability to send file systems between pools. Since most features can be enabled independently of each other, the on-disk format of the pool is specified by the set of all features marked as **active** on the pool. If the pool was created by another software version this set may include unsupported features. ## Identifying features Every feature has a GUID of the form *com.example*:*feature-name*. The reversed DNS name ensures that the feature's GUID is unique across all ZFS implementations. When unsupported features are encountered on a pool they will be identified by their GUIDs. Refer to the documentation for the ZFS implementation that created the pool for information about those features. Each supported feature also has a short name. By convention a feature's short name is the portion of its GUID which follows the ':' (i.e. *com.example*:*feature-name* would have the short name *feature-name*) , however a feature's short name may differ across ZFS implementations if following the convention would result in name conflicts. ## Feature states Features can be in one of three states: **active** > This feature's on-disk format changes are in effect on the pool. > Support for this feature is required to import the pool in read-write mode. > If this feature is not read-only compatible, > support is also required to import the pool in read-only mode > (see *Read-only compatibility*). **enabled** > An administrator has marked this feature as enabled on the pool, but the > feature's on-disk format changes have not been made yet. > The pool can still be imported by software that does not support this feature, > but changes may be made to the on-disk format at any time > which will move the feature to the > **active** > state. > Some features may support returning to the > **enabled** > state after becoming > **active**. > See feature-specific documentation for details. **disabled** > This feature's on-disk format changes have not been made and will not be made > unless an administrator moves the feature to the > **enabled** > state. > Features cannot be disabled once they have been enabled. The state of supported features is exposed through pool properties of the form **feature**@*short-name*. ## Read-only compatibility Some features may make on-disk format changes that do not interfere with other software's ability to read from the pool. These features are referred to as "read-only compatible". If all unsupported features on a pool are read-only compatible, the pool can be imported in read-only mode by setting the **readonly** property during import (see zpool-import(8) for details on importing pools) . ## Unsupported features For each unsupported feature enabled on an imported pool, a pool property named **unsupported**@*feature-name* will indicate why the import was allowed despite the unsupported feature. Possible values for this property are: **inactive** > The feature is in the > **enabled** > state and therefore the pool's on-disk > format is still compatible with software that does not support this feature. **readonly** > The feature is read-only compatible and the pool has been imported in > read-only mode. ## Feature dependencies Some features depend on other features being enabled in order to function. Enabling a feature will automatically enable any features it depends on. ## Compatibility feature sets It is sometimes necessary for a pool to maintain compatibility with a specific on-disk format, by enabling and disabling particular features. The **compatibility** feature facilitates this by allowing feature sets to be read from text files. When set to **off** (the default), compatibility feature sets are disabled (i.e. all features are enabled); when set to **legacy**, no features are enabled. When set to a comma-separated list of filenames (each filename may either be an absolute path, or relative to */etc/zfs/compatibility.d* or */usr/share/zfs/compatibility.d*) , the lists of requested features are read from those files, separated by whitespace and/or commas. Only features present in all files are enabled. Simple sanity checks are applied to the files: they must be between 1 B and 16 KiB in size, and must end with a newline character. The requested features are applied when a pool is created using **zpool** **create** **-o** **compatibility**=*…* and controls which features are enabled when using **zpool** **upgrade**. **zpool** **status** will not show a warning about disabled features which are not part of the requested feature set. The special value **legacy** prevents any features from being enabled, either via **zpool** **upgrade** or **zpool** **set** **feature**@*feature-name*=**enabled**. This setting also prevents pools from being upgraded to newer on-disk versions. This is a safety measure to prevent new features from being accidentally enabled, breaking compatibility. By convention, compatibility files in */usr/share/zfs/compatibility.d* are provided by the distribution, and include feature sets supported by important versions of popular distributions, and feature sets commonly supported at the start of each year. Compatibility files in */etc/zfs/compatibility.d*, if present, will take precedence over files with the same name in */usr/share/zfs/compatibility.d*. If an unrecognized feature is found in these files, an error message will be shown. If the unrecognized feature is in a file in */etc/zfs/compatibility.d*, this is treated as an error and processing will stop. If the unrecognized feature is under */usr/share/zfs/compatibility.d*, this is treated as a warning and processing will continue. This difference is to allow distributions to include features which might not be recognized by the currently-installed binaries. Compatibility files may include comments: any text from '#' to the end of the line is ignored. **Example**: example# **cat** */usr/share/zfs/compatibility.d/grub2* # Features which are supported by GRUB2 versions from v2.12 onwards. allocation_classes async_destroy block_cloning bookmarks device_rebuild embedded_data empty_bpobj enabled_txg extensible_dataset filesystem_limits hole_birth large_blocks livelist log_spacemap lz4_compress project_quota resilver_defer spacemap_histogram spacemap_v2 userobj_accounting zilsaxattr zpool_checkpoint example# **cat** */usr/share/zfs/compatibility.d/grub2-2.06* # Features which are supported by GRUB2 versions prior to v2.12. # # GRUB is not able to detect ZFS pool if snapshot of top level boot pool # is created. This issue is observed with GRUB versions before v2.12 if # extensible_dataset feature is enabled on ZFS boot pool. # # This file lists all read-only compatible features except # extensible_dataset and any other feature that depends on it. # allocation_classes async_destroy block_cloning device_rebuild embedded_data empty_bpobj enabled_txg hole_birth log_spacemap lz4_compress resilver_defer spacemap_histogram spacemap_v2 zpool_checkpoint example# **zpool** **create** **-o** **compatibility**=*grub2* *bootpool* *vdev* See zpool-create(8) and zpool-upgrade(8) for more information on how these commands are affected by feature sets. # FEATURES The following features are supported on this system: **allocation\_classes** > GUID > > **org.zfsonlinux:allocation\_classes** > READ-ONLY COMPATIBLE > > yes > This feature enables support for separate allocation classes. > This feature becomes > **active** > when a dedicated allocation class vdev > (dedup or special) > is created with the > **zpool** **create** or **zpool** **add** commands. > With device removal, it can be returned to the > **enabled** > state if all the dedicated allocation class vdevs are removed. **async\_destroy** > GUID > > **com.delphix:async\_destroy** > READ-ONLY COMPATIBLE > > yes > Destroying a file system requires traversing all of its data in order to > return its used space to the pool. > Without > **async\_destroy**, > the file system is not fully removed until all space has been reclaimed. > If the destroy operation is interrupted by a reboot or power outage, > the next attempt to open the pool will need to complete the destroy > operation synchronously. > When > **async\_destroy** > is enabled, the file system's data will be reclaimed by a background process, > allowing the destroy operation to complete > without traversing the entire file system. > The background process is able to resume > interrupted destroys after the pool has been opened, eliminating the need > to finish interrupted destroys as part of the open operation. > The amount of space remaining to be reclaimed by the background process > is available through the > **freeing** > property. > This feature is only > **active** > while > **freeing** > is non-zero. **blake3** > GUID > > **org.openzfs:blake3** > DEPENDENCIES > > **extensible\_dataset** > READ-ONLY COMPATIBLE > > no > This feature enables the use of the BLAKE3 hash algorithm for checksum and > dedup. > BLAKE3 is a secure hash algorithm focused on high performance. > When the > **blake3** > feature is set to > **enabled**, > the administrator can turn on the > **blake3** > checksum on any dataset using > **zfs** **set** **checksum**=**blake3** *dset* > (see zfs-set(8)). > This feature becomes > **active** > once a > **checksum** > property has been set to > **blake3**, > and will return to being > **enabled** > once all filesystems that have ever had their checksum set to > **blake3** > are destroyed. **block\_cloning** > GUID > > **com.fudosecurity:block\_cloning** > READ-ONLY COMPATIBLE > > yes > When this feature is enabled ZFS will use block cloning for operations like > **copy\_file\_range**(*2*). > Block cloning allows to create multiple references to a single block. > It is much faster than copying the data (as the actual data is neither read nor > written) and takes no additional space. > Blocks can be cloned across datasets under some conditions (like equal > **recordsize**, > the same master encryption key, etc.). > ZFS tries its best to clone across datasets including encrypted ones. > This is limited for various (nontrivial) reasons depending on the OS > and/or ZFS internals. > This feature becomes > **active** > when first block is cloned. > When the last cloned block is freed, it goes back to the enabled state. **block\_cloning\_endian** > GUID > > **com.truenas:block\_cloning\_endian** > READ-ONLY COMPATIBLE > > yes > This feature corrects ZAP entry endianness issues in the Block Reference > Table (BRT) used by block cloning. > During the original block cloning implementation, BRT ZAP entries were > mistakenly stored as arrays of 8 single-byte entries instead of single > 8-byte entries, making pools non-endian-safe. > This feature is activated when the first BRT ZAP is created (that way > ensuring compatibility with existing pools). > When active, new BRT entries are stored in the correct endian-safe format. > The feature becomes inactive when all BRT ZAPs are destroyed. **bookmarks** > GUID > > **com.delphix:bookmarks** > DEPENDENCIES > > **extensible\_dataset** > READ-ONLY COMPATIBLE > > yes > This feature enables use of the > **zfs** **bookmark** > command. > This feature is > **active** > while any bookmarks exist in the pool. > All bookmarks in the pool can be listed by running > **zfs** **list** **-t** **bookmark** **-r** *poolname*. **bookmark\_v2** > GUID > > **com.datto:bookmark\_v2** > DEPENDENCIES > > **bookmark** > > , **extensible\_dataset** > READ-ONLY COMPATIBLE > > no > This feature enables the creation and management of larger bookmarks which are > needed for other features in ZFS. > This feature becomes > **active** > when a v2 bookmark is created and will be returned to the > **enabled** > state when all v2 bookmarks are destroyed. **bookmark\_written** > GUID > > **com.delphix:bookmark\_written** > DEPENDENCIES > > **bookmark** > > , **extensible\_dataset** > > , **bookmark\_v2** > READ-ONLY COMPATIBLE > > no > This feature enables additional bookmark accounting fields, enabling the > **written**#*bookmark* > property > (space written since a bookmark) > and estimates of send stream sizes for incrementals from bookmarks. > This feature becomes > **active** > when a bookmark is created and will be > returned to the > **enabled** > state when all bookmarks with these fields are destroyed. **device\_rebuild** > GUID > > **org.openzfs:device\_rebuild** > READ-ONLY COMPATIBLE > > yes > This feature enables the ability for the > **zpool** **attach** > and > **zpool** **replace** > commands to perform sequential reconstruction > (instead of healing reconstruction) > when resilvering. > Sequential reconstruction resilvers a device in LBA order without immediately > verifying the checksums. > Once complete, a scrub is started, which then verifies the checksums. > This approach allows full redundancy to be restored to the pool > in the minimum amount of time. > This two-phase approach will take longer than a healing resilver > when the time to verify the checksums is included. > However, unless there is additional pool damage, > no checksum errors should be reported by the scrub. > This feature is incompatible with raidz configurations. > This feature becomes > **active** > while a sequential resilver is in progress, and returns to > **enabled** > when the resilver completes. **device\_removal** > GUID > > **com.delphix:device\_removal** > READ-ONLY COMPATIBLE > > no > This feature enables the > **zpool** **remove** > command to remove top-level vdevs, > evacuating them to reduce the total size of the pool. > This feature becomes > **active** > when the > **zpool** **remove** > command is used > on a top-level vdev, and will never return to being > **enabled**. **draid** > GUID > > **org.openzfs:draid** > READ-ONLY COMPATIBLE > > no > This feature enables use of the > **draid** > vdev type. > dRAID is a variant of RAID-Z which provides integrated distributed > hot spares that allow faster resilvering while retaining the benefits of RAID-Z. > Data, parity, and spare space are organized in redundancy groups > and distributed evenly over all of the devices. > This feature becomes > **active** > when creating a pool which uses the > **draid** > vdev type, or when adding a new > **draid** > vdev to an existing pool. **draid\_failure\_domains** > GUID > > **com.seagate:draid\_failure\_domains** > DEPENDENCIES > > **draid** > READ-ONLY COMPATIBLE > > no > This feature enables use of failure domains in > **draid** > vdev type. > Failure domains allow for an entire set of devices that belong to a domain > to fail without taking the pool offline. > Devices that are likely to fail together due to sharing a common component, > such as an enclosure, HBA, or SAS expander, are good candidates to form a > failure domain. > For example, on a setup with several enclosures the user could define > the failure domains to be the set of devices in each enclosure, then the > failure groups span the enclosures. > Or in other words, the dRAID is arranged such that the i-th device in > every failure group belongs to i-th enclosure. > This will allow tolerating the failure of the whole enclosure. > The size of the failure group is equal to the number of failure domains, > and it cannot be less than the size of the redundancy group > (parity + data + spares). > This feature becomes > **active** > when creating a pool which uses the > **draid** > vdev type with failure domains configured, or when adding a new > **draid** > vdev with failure domains to an existing pool. **dynamic\_gang\_header** > GUID > > **com.klarasystems:dynamic\_gang\_header** > READ-ONLY COMPATIBLE > > no > This feature enables larger gang headers based on the sector size of the pool. > When enabled, gang headers will use the entire space allocated for them, instead > of always restricting themselves to 512 bytes. > This can reduce the need for nested gang trees in extreme fragmentation > scenarios. > This feature becomes active when a gang header is written that is larger than > 512 bytes. > This feature is not enabled by > zpool-upgrade(8). > Instead, it must be manually enabled, or be part of a compatibility file. **edonr** > GUID > > **org.illumos:edonr** > DEPENDENCIES > > **extensible\_dataset** > READ-ONLY COMPATIBLE > > no > This feature enables the use of the Edon-R hash algorithm for checksum, > including for nopwrite > (if compression is also enabled, an overwrite of > a block whose checksum matches the data being written will be ignored) > . > In an abundance of caution, Edon-R requires verification when used with > dedup: > **zfs** **set** **dedup**=**edonr**,**verify** > (see zfs-set(8)). > Edon-R is a very high-performance hash algorithm that was part > of the NIST SHA-3 competition. > It provides extremely high hash performance > (over 350% faster than SHA-256), > but was not selected because of its unsuitability > as a general purpose secure hash algorithm. > This implementation utilizes the new salted checksumming functionality > in ZFS, which means that the checksum is pre-seeded with a secret > 256-bit random key > (stored on the pool) > before being fed the data block to be checksummed. > Thus the produced checksums are unique to a given pool, > preventing hash collision attacks on systems with dedup. > When the > **edonr** > feature is set to > **enabled**, > the administrator can turn on the > **edonr** > checksum on any dataset using > **zfs** **set** **checksum**=**edonr** *dset* > (see zfs-set(8)). > This feature becomes > **active** > once a > **checksum** > property has been set to > **edonr**, > and will return to being > **enabled** > once all filesystems that have ever had their checksum set to > **edonr** > are destroyed. **embedded\_data** > GUID > > **com.delphix:embedded\_data** > READ-ONLY COMPATIBLE > > no > This feature improves the performance and compression ratio of > highly-compressible blocks. > Blocks whose contents can compress to 112 bytes > or smaller can take advantage of this feature. > When this feature is enabled, the contents of highly-compressible blocks are > stored in the block > "pointer" > itself > (a misnomer in this case, as it contains > the compressed data, rather than a pointer to its location on disk) > . > Thus the space of the block > (one sector, typically 512 B or 4 KiB) > is saved, and no additional I/O is needed to read and write the data block. > This feature becomes **active** as soon as it is enabled and will never return to being **enabled**. **empty\_bpobj** > GUID > > **com.delphix:empty\_bpobj** > READ-ONLY COMPATIBLE > > yes > This feature increases the performance of creating and using a large > number of snapshots of a single filesystem or volume, and also reduces > the disk space required. > When there are many snapshots, each snapshot uses many Block Pointer > Objects > (bpobjs) > to track blocks associated with that snapshot. > However, in common use cases, most of these bpobjs are empty. > This feature allows us to create each bpobj on-demand, > thus eliminating the empty bpobjs. > This feature is > **active** > while there are any filesystems, volumes, > or snapshots which were created after enabling this feature. **enabled\_txg** > GUID > > **com.delphix:enabled\_txg** > READ-ONLY COMPATIBLE > > yes > Once this feature is enabled, ZFS records the transaction group number > in which new features are enabled. > This has no user-visible impact, but other features may depend on this feature. > This feature becomes > **active** > as soon as it is enabled and will never return to being > **enabled**. **encryption** > GUID > > **com.datto:encryption** > DEPENDENCIES > > **bookmark\_v2** > > , **extensible\_dataset** > READ-ONLY COMPATIBLE > > no > This feature enables the creation and management of natively encrypted datasets. > This feature becomes > **active** > when an encrypted dataset is created and will be returned to the > **enabled** > state when all datasets that use this feature are destroyed. **fast\_dedup** > GUID > > **com.klarasystems:fast\_dedup** > READ-ONLY COMPATIBLE > > yes > This feature allows more advanced deduplication features to be enabled on new > dedup tables. > This feature will be > **active** > when the first deduplicated block is written after a new dedup table is created > (i.e. after a new pool creation, or new checksum used on a dataset with > **dedup** > enabled). > It will be returned to the > **enabled** > state when all deduplicated blocks using it are freed. **extensible\_dataset** > GUID > > **com.delphix:extensible\_dataset** > READ-ONLY COMPATIBLE > > no > This feature allows more flexible use of internal ZFS data structures, > and exists for other features to depend on. > This feature will be > **active** > when the first dependent feature uses it, and will be returned to the > **enabled** > state when all datasets that use this feature are destroyed. **filesystem\_limits** > GUID > > **com.joyent:filesystem\_limits** > DEPENDENCIES > > **extensible\_dataset** > READ-ONLY COMPATIBLE > > yes > This feature enables filesystem and snapshot limits. > These limits can be used to control how many filesystems and/or snapshots > can be created at the point in the tree on which the limits are set. > This feature is > **active** > once either of the limit properties has been set on a dataset > and will never return to being > **enabled**. **head\_errlog** > GUID > > **com.delphix:head\_errlog** > READ-ONLY COMPATIBLE > > no > This feature enables the upgraded version of errlog, which required an on-disk > error log format change. > Now the error log of each head dataset is stored separately in the zap object > and keyed by the head id. > With this feature enabled, every dataset affected by an error block is listed > in the output of > **zpool** **status**. > In case of encrypted filesystems with unloaded keys we are unable to check > their snapshots or clones for errors and these will not be reported. > An "access denied" error will be reported. > This feature becomes **active** as soon as it is enabled and will never return to being **enabled**. **hole\_birth** > GUID > > **com.delphix:hole\_birth** > DEPENDENCIES > > **enabled\_txg** > READ-ONLY COMPATIBLE > > no > This feature has/had bugs, the result of which is that, if you do a > **zfs** **send** **-i** > (or **-R**, since it uses **-i**) > from an affected dataset, the receiving party will not see any checksum > or other errors, but the resulting destination snapshot > will not match the source. > Its use by > **zfs** **send** **-i** > has been disabled by default > (see > **send\_holes\_without\_birth\_time** > in > zfs(4)) > . > This feature improves performance of incremental sends > (**zfs** **send** **-i**) > and receives for objects with many holes. > The most common case of hole-filled objects is zvols. > An incremental send stream from snapshot > **A** to snapshot **B** > contains information about every block that changed between > **A** and **B**. > Blocks which did not change between those snapshots can be > identified and omitted from the stream using a piece of metadata called > the > "block birth time", > but birth times are not recorded for holes > (blocks filled only with zeroes). > Since holes created after > **A** cannot be distinguished from holes created before **A**, > information about every hole in the entire filesystem or zvol > is included in the send stream. > For workloads where holes are rare this is not a problem. > However, when incrementally replicating filesystems or zvols with many holes > (for example a zvol formatted with another filesystem) > a lot of time will be spent sending and receiving unnecessary information > about holes that already exist on the receiving side. > Once the > **hole\_birth** > feature has been enabled the block birth times > of all new holes will be recorded. > Incremental sends between snapshots created after this feature is enabled > will use this new metadata to avoid sending information about holes that > already exist on the receiving side. > This feature becomes **active** as soon as it is enabled and will never return to being **enabled**. **large\_blocks** > GUID > > **org.open-zfs:large\_blocks** > DEPENDENCIES > > **extensible\_dataset** > READ-ONLY COMPATIBLE > > no > This feature allows the record size on a dataset to be set larger than 128 KiB. > This feature becomes > **active** > once a dataset contains a file with a block size larger than 128 KiB, > and will return to being > **enabled** > once all filesystems that have ever had their recordsize larger than 128 KiB > are destroyed. **large\_dnode** > GUID > > **org.zfsonlinux:large\_dnode** > DEPENDENCIES > > **extensible\_dataset** > READ-ONLY COMPATIBLE > > no > This feature allows the size of dnodes in a dataset to be set larger than 512 B. > This feature becomes > **active** > once a dataset contains an object with a dnode larger than 512 B, > which occurs as a result of setting the > **dnodesize** > dataset property to a value other than > **legacy**. > The feature will return to being > **enabled** > once all filesystems that have ever contained a dnode larger than 512 B > are destroyed. > Large dnodes allow more data to be stored in the bonus buffer, > thus potentially improving performance by avoiding the use of spill blocks. **large\_microzap** > GUID > > **com.klarasystems:large\_microzap** > DEPENDENCIES > > **extensible\_dataset** > > , **large\_blocks** > READ-ONLY COMPATIBLE > > yes > This feature allows "micro" ZAPs to grow larger than 128 KiB without being > upgraded to "fat" ZAPs. > This feature becomes > **active** > the first time a micro ZAP grows larger than 128KiB. > It will only be returned to the > **enabled** > state when all datasets that ever had a large micro ZAP are destroyed. > Note that even when this feature is enabled, micro ZAPs cannot grow larger > than 128 KiB without also changing the > **zap\_micro\_max\_size** > module parameter. > See > zfs(4). **livelist** > GUID > > **com.delphix:livelist** > DEPENDENCIES > > **extensible\_dataset** > READ-ONLY COMPATIBLE > > yes > This feature allows clones to be deleted faster than the traditional method > when a large number of random/sparse writes have been made to the clone. > All blocks allocated and freed after a clone is created are tracked by the > the clone's livelist which is referenced during the deletion of the clone. > The feature is activated when a clone is created and remains > **active** > until all clones have been destroyed. **log\_spacemap** > GUID > > **com.delphix:log\_spacemap** > DEPENDENCIES > > **com.delphix:spacemap\_v2** > READ-ONLY COMPATIBLE > > yes > This feature improves performance for heavily-fragmented pools, > especially when workloads are heavy in random-writes. > It does so by logging all the metaslab changes on a single spacemap every TXG > instead of scattering multiple writes to all the metaslab spacemaps. > This feature becomes **active** as soon as it is enabled and will never return to being **enabled**. **longname** > GUID > > **org.zfsonlinux:longname** > DEPENDENCIES > > **extensible\_dataset** > READ-ONLY COMPATIBLE > > no > This feature allows creating files and directories with name up to 1023 bytes > in length. > A new dataset property > **longname** > is also introduced to toggle longname support for each dataset individually. > This property can be disabled even if it contains longname files. > In such case, new file cannot be created with longname but existing longname > files can still be looked up. > This feature becomes > **active** > when a file name greater than 255 is created in a dataset, and returns to > being > **enabled** > when all such datasets are destroyed. **lz4\_compress** > GUID > > **org.illumos:lz4\_compress** > READ-ONLY COMPATIBLE > > no > **lz4** > is a high-performance real-time compression algorithm that > features significantly faster compression and decompression as well as a > higher compression ratio than the older > **lzjb** > compression. > Typically, > **lz4** > compression is approximately 50% faster on compressible data and 200% faster > on incompressible data than > **lzjb**. > It is also approximately 80% faster on decompression, > while giving approximately a 10% better compression ratio. > When the > **lz4\_compress** > feature is set to > **enabled**, > the administrator can turn on > **lz4** > compression on any dataset on the pool using the > zfs-set(8) > command. > All newly written metadata will be compressed with the > **lz4** > algorithm. > This feature becomes **active** as soon as it is enabled and will never return to being **enabled**. **multi\_vdev\_crash\_dump** > GUID > > **com.joyent:multi\_vdev\_crash\_dump** > READ-ONLY COMPATIBLE > > no > This feature allows a dump device to be configured with a pool comprised > of multiple vdevs. > Those vdevs may be arranged in any mirrored or raidz configuration. > When the > **multi\_vdev\_crash\_dump** > feature is set to > **enabled**, > the administrator can use > dumpadm(8) > to configure a dump device on a pool comprised of multiple vdevs. > Under > FreeBSD > and Linux this feature is unused, but registered for compatibility. > New pools created on these systems will have the feature > **enabled** > but will never transition to > **active**, > as this functionality is not required for crash dump support. > Existing pools where this feature is > **active** > can be imported. **obsolete\_counts** > GUID > > **com.delphix:obsolete\_counts** > DEPENDENCIES > > **device\_removal** > READ-ONLY COMPATIBLE > > yes > This feature is an enhancement of > **device\_removal**, > which will over time reduce the memory used to track removed devices. > When indirect blocks are freed or remapped, > we note that their part of the indirect mapping is > "obsolete" > – no longer needed. > This feature becomes > **active** > when the > **zpool** **remove** > command is used on a top-level vdev, and will never return to being > **enabled**. **physical\_rewrite** > GUID > > **com.truenas:physical\_rewrite** > DEPENDENCIES > > **extensible\_dataset** > READ-ONLY COMPATIBLE > > yes > This feature enables physical block rewriting that preserves logical birth > times, avoiding unnecessary inclusion of rewritten blocks in incremental > **zfs** **send** > streams. > When enabled, the > **zfs** **rewrite** **-P** > command can be used. > This feature becomes > **active** > the first time > **zfs** **rewrite** **-P** > is used on any dataset, and will return to being > **enabled** > once all datasets that have ever used physical rewrite are destroyed. **project\_quota** > GUID > > **org.zfsonlinux:project\_quota** > DEPENDENCIES > > **extensible\_dataset** > READ-ONLY COMPATIBLE > > yes > This feature allows administrators to account the spaces and objects usage > information against the project identifier > (ID). > The project ID is an object-based attribute. > When upgrading an existing filesystem, > objects without a project ID will be assigned a zero project ID. > When this feature is enabled, newly created objects inherit > their parent directories' project ID if the parent's inherit flag is set > (via **chattr** **\[+-]P** or **zfs** **project** **-s**|**-C**). > Otherwise, the new object's project ID will be zero. > An object's project ID can be changed at any time by the owner > (or privileged user) > via > **chattr** **-p** *prjid* > or > **zfs** **project** **-p** *prjid*. > This feature will become > **active** > as soon as it is enabled and will never return to being > **disabled**. > Each filesystem will be upgraded automatically when remounted, or when a new file is created under that filesystem. The upgrade can also be triggered on filesystems via **zfs** **set** **version**=**current** *fs*. The upgrade process runs in the background and may take a while to complete for filesystems containing large amounts of files. **raidz\_expansion** > GUID > > **org.openzfs:raidz\_expansion** > DEPENDENCIES > > **none** > READ-ONLY COMPATIBLE > > no > This feature enables the > **zpool** **attach** > subcommand to attach a new device to a RAID-Z group, expanding the total > amount usable space in the pool. > See > zpool-attach(8). **redaction\_bookmarks** > GUID > > **com.delphix:redaction\_bookmarks** > DEPENDENCIES > > **bookmarks** > > , **extensible\_dataset** > READ-ONLY COMPATIBLE > > no > This feature enables the use of redacted > **zfs** **send**s, > which create redaction bookmarks storing the list of blocks > redacted by the send that created them. > For more information about redacted sends, see > zfs-send(8). **redacted\_datasets** > GUID > > **com.delphix:redacted\_datasets** > DEPENDENCIES > > **extensible\_dataset** > READ-ONLY COMPATIBLE > > no > This feature enables the receiving of redacted > **zfs** **send** > streams, which create redacted datasets when received. > These datasets are missing some of their blocks, > and so cannot be safely mounted, and their contents cannot be safely read. > For more information about redacted receives, see > zfs-send(8). **redaction\_list\_spill** > GUID > > **com.delphix:redaction\_list\_spill** > DEPENDENCIES > > **redaction\_bookmarks** > READ-ONLY COMPATIBLE > > no > This feature enables the redaction list created by zfs redact to store > many more entries. > It becomes > **active** > when a redaction list is created with more than 36 entries, > and returns to being > **enabled** > when no long redaction lists remain in the pool. > For more information about redacted sends, see > zfs-send(8). **resilver\_defer** > GUID > > **com.datto:resilver\_defer** > READ-ONLY COMPATIBLE > > yes > This feature allows ZFS to postpone new resilvers if an existing one is already > in progress. > Without this feature, any new resilvers will cause the currently > running one to be immediately restarted from the beginning. > This feature becomes > **active** > once a resilver has been deferred, and returns to being > **enabled** > when the deferred resilver begins. **sha512** > GUID > > **org.illumos:sha512** > DEPENDENCIES > > **extensible\_dataset** > READ-ONLY COMPATIBLE > > no > This feature enables the use of the SHA-512/256 truncated hash algorithm > (FIPS 180-4) > for checksum and dedup. > The native 64-bit arithmetic of SHA-512 provides an approximate 50% > performance boost over SHA-256 on 64-bit hardware > and is thus a good minimum-change replacement candidate > for systems where hash performance is important, > but these systems cannot for whatever reason utilize the faster > **skein** and **edonr** > algorithms. > When the > **sha512** > feature is set to > **enabled**, > the administrator can turn on the > **sha512** > checksum on any dataset using > **zfs** **set** **checksum**=**sha512** *dset* > (see zfs-set(8)). > This feature becomes > **active** > once a > **checksum** > property has been set to > **sha512**, > and will return to being > **enabled** > once all filesystems that have ever had their checksum set to > **sha512** > are destroyed. **skein** > GUID > > **org.illumos:skein** > DEPENDENCIES > > **extensible\_dataset** > READ-ONLY COMPATIBLE > > no > This feature enables the use of the Skein hash algorithm for checksum and dedup. > Skein is a high-performance secure hash algorithm that was a > finalist in the NIST SHA-3 competition. > It provides a very high security margin and high performance on 64-bit hardware > (80% faster than SHA-256). > This implementation also utilizes the new salted checksumming > functionality in ZFS, which means that the checksum is pre-seeded with a > secret 256-bit random key > (stored on the pool) > before being fed the data block to be checksummed. > Thus the produced checksums are unique to a given pool, > preventing hash collision attacks on systems with dedup. > When the > **skein** > feature is set to > **enabled**, > the administrator can turn on the > **skein** > checksum on any dataset using > **zfs** **set** **checksum**=**skein** *dset* > (see zfs-set(8)). > This feature becomes > **active** > once a > **checksum** > property has been set to > **skein**, > and will return to being > **enabled** > once all filesystems that have ever had their checksum set to > **skein** > are destroyed. **spacemap\_histogram** > GUID > > **com.delphix:spacemap\_histogram** > READ-ONLY COMPATIBLE > > yes > This features allows ZFS to maintain more information about how free space > is organized within the pool. > If this feature is > **enabled**, > it will be activated when a new space map object is created, or > an existing space map is upgraded to the new format, > and never returns back to being > **enabled**. **spacemap\_v2** > GUID > > **com.delphix:spacemap\_v2** > READ-ONLY COMPATIBLE > > yes > This feature enables the use of the new space map encoding which > consists of two words > (instead of one) > whenever it is advantageous. > The new encoding allows space maps to represent large regions of > space more efficiently on-disk while also increasing their maximum > addressable offset. > This feature becomes > **active** > once it is > **enabled**, > and never returns back to being > **enabled**. **userobj\_accounting** > GUID > > **org.zfsonlinux:userobj\_accounting** > DEPENDENCIES > > **extensible\_dataset** > READ-ONLY COMPATIBLE > > yes > This feature allows administrators to account the object usage information > by user and group. > This feature becomes **active** as soon as it is enabled and will never return to being **enabled**. > Each filesystem will be upgraded automatically when remounted, or when a new file is created under that filesystem. The upgrade can also be triggered on filesystems via **zfs** **set** **version**=**current** *fs*. The upgrade process runs in the background and may take a while to complete for filesystems containing large amounts of files. **vdev\_zaps\_v2** > GUID > > **com.klarasystems:vdev\_zaps\_v2** > READ-ONLY COMPATIBLE > > no > This feature creates a ZAP object for the root vdev. > This feature becomes active after the next > **zpool** **import** > or > **zpool reguid**. > Properties can be retrieved or set on the root vdev using > **zpool** **get** > and > **zpool** **set** > with > **root** > as the vdev name which is an alias for > **root-0**. **zilsaxattr** > GUID > > **org.openzfs:zilsaxattr** > DEPENDENCIES > > **extensible\_dataset** > READ-ONLY COMPATIBLE > > yes > This feature enables > **xattr**=**sa** > extended attribute logging in the ZIL. > If enabled, extended attribute changes > (both **xattrdir**=**dir** and **xattr**=**sa**) > are guaranteed to be durable if either the dataset had > **sync**=**always** > set at the time the changes were made, or > sync(2) > is called on the dataset after the changes were made. > This feature becomes > **active** > when a ZIL is created for at least one dataset and will be returned to the > **enabled** > state when it is destroyed for all datasets that use this feature. **zpool\_checkpoint** > GUID > > **com.delphix:zpool\_checkpoint** > READ-ONLY COMPATIBLE > > yes > This feature enables the > **zpool** **checkpoint** > command that can checkpoint the state of the pool > at the time it was issued and later rewind back to it or discard it. > This feature becomes > **active** > when the > **zpool** **checkpoint** > command is used to checkpoint the pool. > The feature will only return back to being > **enabled** > when the pool is rewound or the checkpoint has been discarded. **zstd\_compress** > GUID > > **org.freebsd:zstd\_compress** > DEPENDENCIES > > **extensible\_dataset** > READ-ONLY COMPATIBLE > > no > **zstd** > is a high-performance compression algorithm that features a > combination of high compression ratios and high speed. > Compared to > **gzip**, > **zstd** > offers slightly better compression at much higher speeds. > Compared to > **lz4**, > **zstd** > offers much better compression while being only modestly slower. > Typically, > **zstd** > compression speed ranges from 250 to 500 MB/s per thread > and decompression speed is over 1 GB/s per thread. > When the > **zstd** > feature is set to > **enabled**, > the administrator can turn on > **zstd** > compression of any dataset using > **zfs** **set** **compress**=**zstd** *dset* > (see zfs-set(8)). > This feature becomes > **active** > once a > **compress** > property has been set to > **zstd**, > and will return to being > **enabled** > once all filesystems that have ever had their > **compress** > property set to > **zstd** > are destroyed. # SEE ALSO zfs(8), zpool(8) Debian - July 23, 2025 ZPOOLCONCEPTS(7) - Miscellaneous Information Manual # NAME **zpoolconcepts** - overview of ZFS storage pools # DESCRIPTION ## Virtual Devices (vdevs) A "virtual device" describes a single device or a collection of devices, organized according to certain performance and fault characteristics. The following virtual devices are supported: **disk** > A block device, typically located under > */dev*. > ZFS can use individual slices or partitions, though the recommended mode of > operation is to use whole disks. > A disk can be specified by a full path, or it can be a shorthand name > (the relative portion of the path under > */dev*) > . > A whole disk can be specified by omitting the slice or partition designation. > For example, > *sda* > is equivalent to > */dev/sda*. > When given a whole disk, ZFS automatically labels the disk, if necessary. **file** > A regular file. > The use of files as a backing store is strongly discouraged. > It is designed primarily for experimental purposes, as the fault tolerance of a > file is only as good as the file system on which it resides. > A file must be specified by a full path. **mirror** > A mirror of two or more devices. > Data is replicated in an identical fashion across all components of a mirror. > A mirror with > *N* disks of size *X* can hold *X* bytes and can withstand *N-1* > devices failing, without losing data. **raidz**, **raidz1**, **raidz2**, **raidz3** > A distributed-parity layout, similar to RAID-5/6, with improved distribution of > parity, and which does not suffer from the RAID-5/6 > "write hole", > (in which data and parity become inconsistent after a power loss). > Data and parity is striped across all disks within a raidz group, though not > necessarily in a consistent stripe width. > A raidz group can have single, double, or triple parity, meaning that the > raidz group can sustain one, two, or three failures, respectively, without > losing any data. > The > **raidz1** > vdev type specifies a single-parity raidz group; the > **raidz2** > vdev type specifies a double-parity raidz group; and the > **raidz3** > vdev type specifies a triple-parity raidz group. > The > **raidz** > vdev type is an alias for > **raidz1**. > A raidz group with > *N* disks of size *X* with *P* parity disks can hold approximately > *(N-P)\*X* bytes and can withstand *P* devices failing without losing data. > The minimum number of devices in a raidz group is one more than the number of > parity disks. > The recommended number is between 3 and 9 to help increase performance. **draid**, **draid1**, **draid2**, **draid3** > A variant of raidz that provides integrated distributed hot spares, allowing > for faster resilvering, while retaining the benefits of raidz. > A dRAID vdev is constructed from multiple internal raidz groups, each with > *D* data devices and *P* parity devices. > These groups are distributed over all of the children in order to fully > utilize the available disk performance. > Unlike raidz, dRAID uses a fixed stripe width (padding as necessary with > zeros) to allow fully sequential resilvering. > This fixed stripe width significantly affects both usable capacity and IOPS. > For example, with the default > *D=8* and *4 KiB* disk sectors the minimum allocation size is *32 KiB*. > If using compression, this relatively large allocation size can reduce the > effective compression ratio. > When using ZFS volumes (zvols) and dRAID, the default of the > **volblocksize** > property is increased to account for the allocation size. > If a dRAID pool will hold a significant amount of small blocks, it is > recommended to also add a mirrored > **special** > vdev to store those blocks. > In regards to I/O, performance is similar to raidz since, for any read, all > *D* data disks must be accessed. > Delivered random IOPS can be reasonably approximated as > **floor((N-S)/(D+P))\*single\_drive\_IOPS**. > Like raidz, a dRAID can have single-, double-, or triple-parity. > The > **draid1**, > **draid2**, > and > **draid3** > types can be used to specify the parity level. > The > **draid** > vdev type is an alias for > **draid1**. > A dRAID with > *N* disks of size *X*, *D* data disks per redundancy group, *P* > parity level, and *S* distributed hot spares can hold approximately > *(N-S)\*(D/(D+P))\*X* bytes and can withstand *P* > devices failing without losing data. **draid**\[*parity*]\[**:**‌*data*‌**d**]\[**:**‌*children*‌**c**]\[**:**‌*width*‌**w**]\[**:**‌*spares*‌**s**] > A non-default dRAID configuration can be specified by appending one or more > of the following optional arguments to the > **draid** > keyword: > *parity* > > The parity level (1-3). > *data* > > The number of data devices per redundancy group. > > In general, a smaller value of > > *D* will increase IOPS, improve the compression ratio, > > and speed up resilvering at the expense of total usable capacity. > > Defaults to > > *8*, unless *N-P-S* is less than *8*. > *children* > > The expected number of children. > > Useful as a cross-check when listing a large number of devices. > > An error is returned when the provided number of children differs. > *width* > > You can configure several groups of children in the same row, in which case > > *width* would be a multiple of *children*. > > Such configurations allow the creation of failure groups with every i-th device > > in each group being from different failure domain (for example an enclosure) > > so that if all devices in one domain fail, the > > *draid* vdev still will be operational with enough redundancy to > > rebuild the data. > > In case of > > *draid2*, two domains can fail at a time, in case of > > *draid3* — three domains (provided there are no other failures > > in any failure group). > > For each group, it will be only one, two or three failures. > *spares* > > The number of distributed hot spares. > > All spares are shared between failure groups. > > Defaults to zero. > > Note: to support domain failure, we cannot have more than > > *parity-1* failures in any failure group, no matter if the failed > > devices are rebuilt to draid hot spares or not — the blocks of those > > spares can be mapped to the devices from the failed domain, and we cannot > > tolerate more than > > *parity* failures in any failure group. **spare** > A pseudo-vdev which keeps track of available hot spares for a pool. > For more information, see the > *Hot Spares* > section. **log** > A separate intent log device. > If more than one log device is specified, then writes are load-balanced between > devices. > Log devices can be mirrored. > However, raidz vdev types are not supported for the intent log. > For more information, see the > *Intent Log* > section. **dedup** > A device solely dedicated for deduplication tables. > The redundancy of this device should match the redundancy of the other normal > devices in the pool. > If more than one dedup device is specified, then > allocations are load-balanced between those devices. **special** > A device dedicated solely for allocating various kinds of internal metadata, > and optionally small file blocks. > The redundancy of this device should match the redundancy of the other normal > devices in the pool. > If more than one special device is specified, then > allocations are load-balanced between those devices. > For more information on special allocations, see the > *Special Allocation Class* > section. **cache** > A device used to cache storage pool data. > A cache device cannot be configured as a mirror or raidz group. > For more information, see the > *Cache Devices* > section. **fdomain** or **failure\_domain** > Denotes the list of failure domain devices for dRAID vdev. **fgroup** or **failure\_group** > Denotes the list of failure group devices for dRAID vdev. Virtual devices cannot be nested arbitrarily. A mirror, raidz or draid virtual device can only be created with files or disks. Mirrors of mirrors or other such combinations are not allowed. A pool can have any number of virtual devices at the top of the configuration (known as "root vdevs") . Data is dynamically distributed across all top-level devices to balance data among devices. As new virtual devices are added, ZFS automatically places data on the newly available devices. Virtual devices are specified one at a time on the command line, separated by whitespace. Keywords like **mirror** and **raidz** are used to distinguish where a group ends and another begins. For example, the following creates a pool with two root vdevs, each a mirror of two disks: # **zpool** **create** *mypool* **mirror** *sda sdb* **mirror** *sdc sdd* ## Device Failure and Recovery ZFS supports a rich set of mechanisms for handling device failure and data corruption. All metadata and data is checksummed, and ZFS automatically repairs bad data from a good copy, when corruption is detected. In order to take advantage of these features, a pool must make use of some form of redundancy, using either mirrored or raidz groups. While ZFS supports running in a non-redundant configuration, where each root vdev is simply a disk or file, this is strongly discouraged. A single case of bit corruption can render some or all of your data unavailable. A pool's health status is described by one of three states: **online**, **degraded**, or **faulted**. An online pool has all devices operating normally. A degraded pool is one in which one or more devices have failed, but the data is still available due to a redundant configuration. A faulted pool has corrupted metadata, or one or more faulted devices, and insufficient replicas to continue functioning. The health of the top-level vdev, such as a mirror or raidz device, is potentially impacted by the state of its associated vdevs or component devices. A top-level vdev or component device is in one of the following states: **DEGRADED** > One or more top-level vdevs is in the degraded state because one or more > component devices are offline. > Sufficient replicas exist to continue functioning. > One or more component devices is in the degraded or faulted state, but > sufficient replicas exist to continue functioning. > The underlying conditions are as follows: > * The number of checksum errors or slow I/Os exceeds acceptable levels and the > device is degraded as an indication that something may be wrong. > ZFS continues to use the device as necessary. > * The number of I/O errors exceeds acceptable levels. > The device could not be marked as faulted because there are insufficient > replicas to continue functioning. **FAULTED** > One or more top-level vdevs is in the faulted state because one or more > component devices are offline. > Insufficient replicas exist to continue functioning. > One or more component devices is in the faulted state, and insufficient > replicas exist to continue functioning. > The underlying conditions are as follows: > * The device could be opened, but the contents did not match expected values. > * The number of I/O errors exceeds acceptable levels and the device is faulted to > prevent further use of the device. **OFFLINE** > The device was explicitly taken offline by the > **zpool** **offline** > command. **ONLINE** > The device is online and functioning. **REMOVED** > The device was physically removed while the system was running. > Device removal detection is hardware-dependent and may not be supported on all > platforms. **UNAVAIL** > The device could not be opened. > If a pool is imported when a device was unavailable, then the device will be > identified by a unique identifier instead of its path since the path was never > correct in the first place. Checksum errors represent events where a disk returned data that was expected to be correct, but was not. In other words, these are instances of silent data corruption. The checksum errors are reported in **zpool** **status** and **zpool** **events**. When a block is stored redundantly, a damaged block may be reconstructed (e.g. from raidz parity or a mirrored copy). In this case, ZFS reports the checksum error against the disks that contained damaged data. If a block is unable to be reconstructed (e.g. due to 3 disks being damaged in a raidz2 group), it is not possible to determine which disks were silently corrupted. In this case, checksum errors are reported for all disks on which the block is stored. If a device is removed and later re-attached to the system, ZFS attempts to bring the device online automatically. Device attachment detection is hardware-dependent and might not be supported on all platforms. ## Hot Spares ZFS allows devices to be associated with pools as "hot spares". These devices are not actively used in the pool. But, when an active device fails, it is automatically replaced by a hot spare. To create a pool with hot spares, specify a **spare** vdev with any number of devices. For example, # **zpool** **create** *pool* **mirror** *sda sdb* **spare** *sdc sdd* Spares can be shared across multiple pools, and can be added with the **zpool** **add** command and removed with the **zpool** **remove** command. Once a spare replacement is initiated, a new **spare** vdev is created within the configuration that will remain there until the original device is replaced. At this point, the hot spare becomes available again, if another device fails. If a pool has a shared spare that is currently being used, the pool cannot be exported, since other pools may use this shared spare, which may lead to potential data corruption. Shared spares add some risk. If the pools are imported on different hosts, and both pools suffer a device failure at the same time, both could attempt to use the spare at the same time. This may not be detected, resulting in data corruption. An in-progress spare replacement can be canceled by detaching the hot spare. If the original faulted device is detached, then the hot spare assumes its place in the configuration, and is removed from the spare list of all active pools. The **draid** vdev type provides distributed hot spares. These are virtual devices whose blocks are reserved and distributed among all real devices, which makes resilvering to them much faster because one device is not a bottleneck anymore. Fast resilvering is crucial for data durability, it decreases the time of having degraded data redundancy in the pool, thus decreasing the chance of losing more devices at a time which cannot be tolerate. dRAID hot spares are named after the draid vdev they're a part of (**draid1**-*2*-*3* specifies spare *3* of vdev *2*, which is a single parity dRAID) and may only be used by that dRAID vdev. Otherwise, they behave the same as normal hot spares. Spares cannot replace log devices. ## Intent Log The ZFS Intent Log (ZIL) satisfies POSIX requirements for synchronous transactions. For instance, databases often require their transactions to be on stable storage devices when returning from a system call. NFS and other applications can also use fsync(2) to ensure data stability. By default, the intent log is allocated from blocks within the main pool. However, it might be possible to get better performance using separate intent log devices such as NVRAM or a dedicated disk. For example: # **zpool** **create** *pool sda sdb* **log** *sdc* Multiple log devices can also be specified, and they can be mirrored. See the *EXAMPLES* section for an example of mirroring multiple log devices. ## Cache Devices Devices can be added to a storage pool as "cache devices". These devices provide an additional layer of caching between main memory and disk. For read-heavy workloads, where the working set size is much larger than what can be cached in main memory, using cache devices allows much more of this working set to be served from low latency media. Using cache devices provides the greatest performance improvement for random read-workloads of mostly static content. To create a pool with cache devices, specify a **cache** vdev with any number of devices. For example: # **zpool** **create** *pool sda sdb* **cache** *sdc sdd* Cache devices cannot be mirrored or part of a raidz configuration. If a read error is encountered on a cache device, that read I/O is reissued to the original storage pool device, which might be part of a mirrored or raidz configuration. The content of the cache devices is persistent across reboots and restored asynchronously when importing the pool in L2ARC (persistent L2ARC). This can be disabled by setting **l2arc\_rebuild\_enabled**=**0**. For cache devices smaller than *1 GiB*, ZFS does not write the metadata structures required for rebuilding the L2ARC, to conserve space. This can be changed with **l2arc\_rebuild\_blocks\_min\_l2size**. The cache device header (*512 B*) is updated even if no metadata structures are written. L2ARC operates in one of two modes depending on total cache capacity. When total L2ARC capacity is less than twice **arc\_c\_max**, L2ARC writes buffers to cache as they are evicted from ARC. When total capacity is at least twice **arc\_c\_max**, L2ARC uses persistent markers that track scan positions across iterations, writing cacheable content as write throughput allows. A depth cap (**l2arc\_ext\_headroom\_pct**) limits how far markers advance from the tail, keeping them focused on buffers soon to be evicted where caching adds the most value. If a cache device is added with **zpool** **add**, its label and header will be overwritten and its contents will not be restored in L2ARC, even if the device was previously part of the pool. If a cache device is onlined with **zpool** **online**, its contents will be restored in L2ARC. This is useful in case of memory pressure, where the contents of the cache device are not fully restored in L2ARC. The user can off- and online the cache device when there is less memory pressure, to fully restore its contents to L2ARC. ## Pool checkpoint Before starting critical procedures that include destructive actions (like **zfs** **destroy**), an administrator can checkpoint the pool's state and, in the case of a mistake or failure, rewind the entire pool back to the checkpoint. Otherwise, the checkpoint can be discarded when the procedure has completed successfully. A pool checkpoint can be thought of as a pool-wide snapshot and should be used with care as it contains every part of the pool's state, from properties to vdev configuration. Thus, certain operations are not allowed while a pool has a checkpoint. Specifically, vdev removal/attach/detach, mirror splitting, and changing the pool's GUID. Adding a new vdev is supported, but in the case of a rewind it will have to be added again. Finally, users of this feature should keep in mind that scrubs in a pool that has a checkpoint do not repair checkpointed data. To create a checkpoint for a pool: # **zpool** **checkpoint** *pool* To later rewind to its checkpointed state, you need to first export it and then rewind it during import: # **zpool** **export** *pool* # **zpool** **import** **--rewind-to-checkpoint** *pool* Note that rewinding to a checkpoint will **permanently discard it.** Once the pool has been successfully imported with the above rewind command, you cannot rewind to the same checkpoint. To discard the checkpoint from a pool: # **zpool** **checkpoint** **-d** *pool* Dataset reservations (controlled by the **reservation** and **refreservation** properties) may be unenforceable while a checkpoint exists, because the checkpoint is allowed to consume the dataset's reservation. Finally, data that is part of the checkpoint but has been freed in the current state of the pool won't be scanned during a scrub. ## Special Allocation Class Allocations in the special class are dedicated to specific block types. By default, this includes all metadata, the indirect blocks of user data, intent log (in absence of separate log device), and deduplication tables. The class can also be provisioned to accept small file blocks or zvol blocks on a per dataset granularity. A pool must always have at least one normal (non-**dedup**/-**special**) vdev before other devices can be assigned to the special class. If the **special** class becomes full, then allocations intended for it will spill back into the normal class. Deduplication tables can be excluded from the special class by unsetting the **zfs\_ddt\_data\_is\_special** ZFS module parameter. Inclusion of small file or zvol blocks in the special class is opt-in. Each dataset can control the size of small file blocks allowed in the special class by setting the **special\_small\_blocks** property to nonzero. See zfsprops(7) for more info on this property. Debian - August 6, 2025 ZPOOLPROPS(7) - Miscellaneous Information Manual # NAME **zpoolprops** - properties of ZFS storage pools # DESCRIPTION Each pool has several properties associated with it. Some properties are read-only statistics while others are configurable and change the behavior of the pool. User properties have no effect on ZFS behavior. Use them to annotate pools in a way that is meaningful in your environment. For more information about user properties, see the *User Properties* section. The following are read-only properties: **allocated** > Amount of storage used within the pool. > See > **fragmentation** > and > **free** > for more information. **available** > The amount of free space available in the pool, adjusted for raidz parity > overhead. > The > **available** > property is calculated as: > **usable - used** > See > **usable**, > **used**, > and > **free** > for more information. **bcloneratio** > The ratio of the total amount of storage that would be required to store all > the cloned blocks without cloning to the actual storage used. > The > **bcloneratio** > property is calculated as: > ((**bclonesaved + bcloneused**) **\* 100**) **/ bcloneused** **bclonesaved** > The amount of additional storage that would be required if block cloning > was not used. **bcloneused** > The amount of storage used by cloned blocks. **capacity** > Percentage of pool space used. > This property can also be referred to by its shortened column name, > **cap**. **dedupcached** > Total size of the deduplication table currently loaded into the ARC. > See > zpool-prefetch(8). **dedup\_table\_size** > Total on-disk size of the deduplication table. **dedupratio** > The ratio of the total amount of storage that would be required to store all > the deduplicated blocks without deduplication to the actual storage used. > The > **dedupratio** > property is calculated as: > ((**dedupsaved + dedupused**) **\* 100**) **/ dedupused** **dedupsaved** > The amount of additional storage that would be required if deduplication > was not used. > This represents the space saved by deduplication. **dedupused** > The amount of storage used by deduplicated blocks. > This is the actual physical space occupied on disk after deduplication. **expandsize** > Amount of uninitialized space within the pool or device that can be used to > increase the total capacity of the pool. > On whole-disk vdevs, this is the space beyond the end of the GPT – > typically occurring when a LUN is dynamically expanded > or a disk replaced with a larger one. > On partition vdevs, this is the space appended to the partition after it was > added to the pool – most likely by resizing it in-place. > The space can be claimed for the pool by bringing it online with > **autoexpand=on** > or using > **zpool** **online** **-e**. **fragmentation** > The amount of fragmentation in the pool. > As the amount of space > **allocated** > increases, it becomes more difficult to locate > **free** > space. > This may result in lower write performance compared to pools with more > unfragmented free space. **free** > The amount of free space available in the pool. > By contrast, the > zfs(8) > **available** > property describes how much new data can be written to ZFS filesystems/volumes. > The zpool > **free** > property is not generally useful for this purpose, and can be substantially more > than the zfs > **available** > space. > This discrepancy is due to several factors, including raidz parity; > zfs reservation, quota, refreservation, and refquota properties; and space set > aside by > **spa\_slop\_shift** > (see > zfs(4) > for more information). **freeing** > After a file system or snapshot is destroyed, the space it was using is > returned to the pool asynchronously. > **freeing** > is the amount of space remaining to be reclaimed. > Over time > **freeing** > will decrease while > **free** > increases. **guid** > A unique identifier for the pool. **health** > The current health of the pool. > Health can be one of > **ONLINE**, **DEGRADED**, **FAULTED**, **OFFLINE, REMOVED**, **UNAVAIL**. **last\_scrubbed\_txg** > Indicates the transaction group (TXG) up to which the most recent scrub > operation has checked and repaired the dataset. > This provides insight into the data integrity status of their pool at > a specific point in time. > zpool-scrub(8) > can utilize this property to scan only data that has changed since the last > scrub completed, when given the > **-C** > flag. > This property is not updated when performing an error scrub with the > **-e** > flag. **leaked** > Space not released while > **freeing** > due to corruption, now permanently leaked into the pool. **load\_guid** > A unique identifier for the pool. > Unlike the > **guid** > property, this identifier is generated every time we load the pool (i.e. does > not persist across imports/exports) and never changes while the pool is loaded > (even if a > **reguid** > operation takes place). **size** > Total size of the storage pool. **unsupported@**‌*guid* > Information about unsupported features that are enabled on the pool. > See > zpool-features(7) > for details. **usable** > Estimate of total storage pool size, adjusted for raidz parity overhead. > The adjustment assumes a fixed 128KiB record size to compute a data-to-parity > ratio. > **usable** > is a heuristic for allocation and should not be interpreted as an exact measure > of usable space. **used** > Estimate of storage used within the storage pool, adjusted for raidz parity > overhead. > The adjustment assumes a fixed 128KiB record size to compute a data-to-parity > ratio. > **used** > is a heuristic for allocation and should not be interpreted as an exact measure > of used space. The space usage properties report actual physical space available to the storage pool. The physical space can be different from the total amount of space that any contained datasets can actually use. The amount of space used in a raidz configuration depends on the characteristics of the data being written. In addition, ZFS reserves some space for internal accounting that the zfs(8) command takes into account, but the **zpoolprops** command does not. For non-full pools of a reasonable size, these effects should be invisible. For small pools, or pools that are close to being completely full, these discrepancies may become more noticeable. The following properties are read-only metrics for allocation classes: **class\_dedup\_allocated**, **class\_dedup\_available**, **class\_dedup\_capacity**, **class\_dedup\_expandsize**, **class\_dedup\_fragmentation**, **class\_dedup\_free**, **class\_dedup\_size**, **class\_dedup\_usable**, **class\_dedup\_used** > Space usage properties of the pool's > **dedup** > metaslab allocator class. **class\_elog\_allocated**, **class\_elog\_available**, **class\_elog\_capacity**, **class\_elog\_expandsize**, **class\_elog\_fragmentation**, **class\_elog\_free**, **class\_elog\_size**, **class\_elog\_usable**, **class\_elog\_used** > Space usage properties of the pool's > **embedded\_log** > metaslab allocator class. **class\_log\_allocated**, **class\_log\_available**, **class\_log\_capacity**, **class\_log\_expandsize**, **class\_log\_fragmentation**, **class\_log\_free**, **class\_log\_size**, **class\_log\_usable**, **class\_log\_used** > Space usage properties of the pool's > **log** > metaslab allocator class. **class\_normal\_allocated**, **class\_normal\_available**, **class\_normal\_capacity**, **class\_normal\_expandsize**, **class\_normal\_fragmentation**, **class\_normal\_free**, **class\_normal\_size**, **class\_normal\_usable**, **class\_normal\_used** > Space usage properties of the pool's > **normal** > metaslab allocator class. **class\_special\_allocated**, **class\_special\_available**, **class\_special\_capacity**, **class\_special\_expandsize**, **class\_special\_fragmentation**, **class\_special\_free**, **class\_special\_size**, **class\_special\_usable**, **class\_special\_used** > Space usage properties of the pool's > **special** > metaslab allocator class. **class\_special\_elog\_allocated**, **class\_special\_elog\_available**, **class\_special\_elog\_capacity**, **class\_special\_elog\_expandsize**, **class\_special\_elog\_fragmentation**, **class\_special\_elog\_free**, **class\_special\_elog\_size**, **class\_special\_elog\_usable**, **class\_special\_elog\_used** > Space usage properties of the pool's > **special\_embedded\_log** > metaslab allocator class. Each allocation class is described by these values: **allocated** > Amount of storage used within the allocation class. **available** > Estimate of free space available in the allocation class, adjusted for raidz > parity overhead. > The adjustment assumes a fixed 128KiB record size to compute a data-to-parity > ratio. > **available** > is a heuristic for allocation and should not be interpreted as an exact measure > of usable space. > Actual usable space depends on a variety of factors such as dataset record size > and compression. **capacity** > Percentage of allocation class space used. **expandsize** > Amount of uninitialized space within the allocation class that can be used to > increase the capacity of the allocator. > See the pool-wide > **expandsize** > property. **fragmentation** > The amount of free space fragmentation in the allocation class. > See the pool-wide > **fragmentation** > property. **free** > The amount of free space available in the allocation class. > This value is not adjusted for factors such as dataset record size, compression, > or raidz parity overhead. > See the pool-wide > **free** > property. **size** > Total size of the allocation class. **usable** > Estimate of total allocation class size, adjusted for raidz parity overhead. > The adjustment assumes a fixed 128KiB record size to compute a data-to-parity > ratio. > **usable** > is a heuristic for allocation and should not be interpreted as an exact measure > of usable space. **used** > Estimate of storage used within the allocation class, adjusted for raidz parity > overhead. > The adjustment assumes a fixed 128KiB record size to compute a data-to-parity > ratio. > **used** > is a heuristic for allocation and should not be interpreted as an exact measure > of used space. The following property can be set at creation time and import time: **altroot** > Alternate root directory. > If set, this directory is prepended to any mount points within the pool. > This can be used when examining an unknown pool where the mount points cannot be > trusted, or in an alternate boot environment, where the typical paths are not > valid. > **altroot** > is not a persistent property. > It is valid only while the system is up. > Setting > **altroot** > defaults to using > **cachefile**=**none**, > though this may be overridden using an explicit setting. The following property can be set only at import time: **readonly**=**on**|**off** > If set to > **on**, > the pool will be imported in read-only mode. > This property can also be referred to by its shortened column name, > **rdonly**. The following properties can be set at creation time and import time, and later changed with the **zpool** **set** command: **ashift**=*ashift* > Pool sector size exponent, to the power of > **2** > (internally referred to as > **ashift**). > Values from 9 to 16, inclusive, are valid; also, the > value 0 (the default) means to auto-detect using the kernel's block > layer and a ZFS internal exception list. > I/O operations will be aligned to the specified size boundaries. > Additionally, the minimum (disk) > write size will be set to the specified size, so this represents a > space/performance trade-off. > For optimal performance, the pool sector size should be greater than > or equal to the sector size of the underlying disks. > The typical case for setting this property is when > performance is important and the underlying disks use 4KiB sectors but > report 512B sectors to the OS (for compatibility reasons); in that > case, set > **ashift**=**12** > (which is > **1<<12** = **4096**). > When set, this property is > used as the default hint value in subsequent vdev operations (add, > attach and replace). > Changing this value will not modify any existing > vdev, not even on disk replacement; however it can be used, for > instance, to replace a dying 512B sectors disk with a newer 4KiB > sectors device: this will probably result in bad performance but at the > same time could prevent loss of data. **autoexpand**=**on**|**off** > Controls automatic pool expansion when the underlying LUN is grown. > If set to > **on**, > the pool will be resized according to the size of the expanded device. > If the device is part of a mirror or raidz then all devices within that > mirror/raidz group must be expanded before the new space is made available to > the pool. > The default behavior is > **off**. > This property can also be referred to by its shortened column name, > **expand**. **autoreplace**=**on**|**off** > Controls automatic device replacement. > If set to > **off**, > device replacement must be initiated by the administrator by using the > **zpool** **replace** > command. > If set to > **on**, > any new device, found in the same physical location as a device that previously > belonged to the pool, is automatically formatted and replaced. > The default behavior is > **off**. > This property can also be referred to by its shortened column name, > **replace**. > Autoreplace can also be used with virtual disks (like device > mapper) provided that you use the /dev/disk/by-vdev paths setup by > vdev\_id.conf. > See the > vdev\_id(8) > manual page for more details. > Autoreplace and autoonline require the ZFS Event Daemon be configured and > running. > See the > zed(8) > manual page for more details. **autotrim**=**on**|**off** > When set to > **on** > space which has been recently freed, and is no longer allocated by the pool, > will be periodically trimmed. > This allows block device vdevs which support > BLKDISCARD, such as SSDs, or file vdevs on which the underlying file system > supports hole-punching, to reclaim unused blocks. > The default value for this property is > **off**. > Automatic TRIM does not immediately reclaim blocks after a free. > Instead, it will optimistically delay allowing smaller ranges to be aggregated > into a few larger ones. > These can then be issued more efficiently to the storage. > TRIM on L2ARC devices is enabled by setting > **l2arc\_trim\_ahead > 0**. > Be aware that automatic trimming of recently freed data blocks can put > significant stress on the underlying storage devices. > This will vary depending of how well the specific device handles these commands. > For lower-end devices it is often possible to achieve most of the benefits > of automatic trimming by running an on-demand (manual) TRIM periodically > using the > **zpool** **trim** > command. **bootfs**=**(unset)**|*pool*\[/*dataset*] > Identifies the default bootable dataset for the root pool. > This property is expected to be set mainly by the installation and upgrade > programs. > Not all Linux distribution boot processes use the bootfs property. **cachefile**=*path*|**none** > Controls the location of where the pool configuration is cached. > Discovering all pools on system startup requires a cached copy of the > configuration data that is stored on the root file system. > All pools in this cache are automatically imported when the system boots. > Some environments, such as install and clustering, need to cache this > information in a different location so that pools are not automatically > imported. > Setting this property caches the pool configuration in a different location that > can later be imported with > **zpool** **import** **-c**. > Setting it to the value > **none** > creates a temporary pool that is never cached, and the > "" > (empty string) > uses the default location. > Multiple pools can share the same cache file. > Because the kernel destroys and recreates this file when pools are added and > removed, care should be taken when attempting to access this file. > When the last pool using a > **cachefile** > is exported or destroyed, the file will be empty. **comment**=*text* > A text string consisting of printable ASCII characters that will be stored > such that it is available even if the pool becomes faulted. > An administrator can provide additional information about a pool using this > property. **compatibility**=**off**|**legacy**|*file*\[,*file*]… > Specifies that the pool maintain compatibility with specific feature sets. > When set to > **off** > (or unset) compatibility is disabled (all features may be enabled); when set to > **legacy** > no features may be enabled. > When set to a comma-separated list of filenames > (each filename may either be an absolute path, or relative to > */etc/zfs/compatibility.d* > or > */usr/share/zfs/compatibility.d*) > the lists of requested features are read from those files, separated by > whitespace and/or commas. > Only features present in all files may be enabled. > See > zpool-features(7), > zpool-create(8) > and > zpool-upgrade(8) > for more information on the operation of compatibility feature sets. **dedup\_table\_quota**=*number*|**none**|**auto** > This property sets a limit on the on-disk size of the pool's dedup table. > Entries will not be added to the dedup table once this size is reached; > if a dedup table already exists, and is larger than this size, they > will not be removed as part of setting this property. > Existing entries will still have their reference counts updated. > The actual size limit of the table may be above or below the quota, > depending on the actual on-disk size of the entries (which may be > approximated for purposes of calculating the quota). > That is, setting a quota size of 1M may result in the maximum size being > slightly below, or slightly above, that value. > Set to > **'none'** > to disable. > In automatic mode, which is the default, the size of a dedicated dedup vdev > is used as the quota limit. > The > **dedup\_table\_quota** > property works for both legacy and fast dedup tables. **dedupditto**=*number* > This property is deprecated and no longer has any effect. **delegation**=**on**|**off** > Controls whether a non-privileged user is granted access based on the dataset > permissions defined on the dataset. > See > zfs(8) > for more information on ZFS delegated administration. **failmode**=**wait**|**continue**|**panic** > Controls the system behavior in the event of catastrophic pool failure. > This condition is typically a result of a loss of connectivity to the underlying > storage device(s) or a failure of all devices within the pool. > The behavior of such an event is determined as follows: > **wait** > > Blocks all I/O access until the device connectivity is recovered and the errors > > are cleared with > > **zpool** **clear**. > > This is the default behavior. > **continue** > > Returns > > `EIO` > > to any new write I/O requests but allows reads to any of the remaining healthy > > devices. > > Any write requests that have yet to be committed to disk would be blocked. > **panic** > > Prints out a message to the console and generates a system crash dump. **feature@**‌*feature\_name*=**enabled** > The value of this property is the current state of > *feature\_name*. > The only valid value when setting this property is > **enabled** > which moves > *feature\_name* > to the enabled state. > See > zpool-features(7) > for details on feature states. **listsnapshots**=**on**|**off** > Controls whether information about snapshots associated with this pool is > output when > **zfs** **list** > is run without the > **-t** > option. > The default value is > **off**. > This property can also be referred to by its shortened name, > **listsnaps**. **multihost**=**on**|**off** > Controls whether a pool activity check should be performed during > **zpool** **import**. > When a pool is determined to be active it cannot be imported, even with the > **-f** > option. > This property is intended to be used in failover configurations > where multiple hosts have access to a pool on shared storage. > Multihost provides protection on import only. > It does not protect against an > individual device being used in multiple pools, regardless of the type of vdev. > See the discussion under > **zpool** **create**. > When this property is on, periodic writes to storage occur to show the pool is > in use. > See > **zfs\_multihost\_interval** > in the > zfs(4) > manual page. > In order to enable this property each host must set a unique hostid. > See > genhostid(1) > zgenhostid(8) > spl(4) > for additional details. > The default value is > **off**. **version**=*version* > The current on-disk version of the pool. > This can be increased, but never decreased. > The preferred method of updating pools is with the > **zpool** **upgrade** > command, though this property can be used when a specific version is needed for > backwards compatibility. > Once feature flags are enabled on a pool this property will no longer have a > value. ## User Properties In addition to the standard native properties, ZFS supports arbitrary user properties. User properties have no effect on ZFS behavior, but applications or administrators can use them to annotate pools. User property names must contain a colon ("**:**") character to distinguish them from native properties. They may contain lowercase letters, numbers, and the following punctuation characters: colon ("**:**"), dash ("**-**"), period ("**.**"), and underscore ("**\_**"). The expected convention is that the property name is divided into two portions such as *module*:*property*, but this namespace is not enforced by ZFS. User property names can be at most 255 characters, and cannot begin with a dash ("**-**"). When making programmatic use of user properties, it is strongly suggested to use a reversed DNS domain name for the *module* component of property names to reduce the chance that two independently-developed packages use the same property name for different purposes. The values of user properties are arbitrary strings and are never validated. All of the commands that operate on properties (**zpool** **list**, **zpool** **get**, **zpool** **set**, and so forth) can be used to manipulate both native properties and user properties. Use **zpool** **set** *name*= to clear a user property. Property values are limited to 8192 bytes. Debian - March 2, 2026 FSCK.ZFS(8) - System Manager's Manual # NAME **fsck.zfs** - dummy ZFS filesystem checker # SYNOPSIS **fsck.zfs** \[*options*] *dataset*… # DESCRIPTION **fsck.zfs** is a thin shell wrapper that at most checks the status of a dataset's container pool. It is installed by OpenZFS because some Linux distributions expect a fsck helper for all filesystems. If more than one *dataset* is specified, each is checked in turn and the results binary-ored. # OPTIONS Ignored. # NOTES ZFS datasets are checked by running **zpool** **scrub** on the containing pool. An individual ZFS dataset is never checked independently of its pool, which is unlike a regular filesystem. However, the fsck(8) interface still allows it to communicate some errors: if the *dataset* is in a degraded pool, then **fsck.zfs** will return exit code **4** to indicate an uncorrected filesystem error. Similarly, if the *dataset* is in a faulted pool and has a legacy */etc/fstab* record, then **fsck.zfs** will return exit code **8** to indicate a fatal operational error. # SEE ALSO fstab(5), fsck(8), zpool-scrub(8) Debian - May 26, 2021 MOUNT.ZFS(8) - System Manager's Manual # NAME **mount.zfs** - mount ZFS filesystem # SYNOPSIS **mount.zfs** \[**-sfnvh**] \[**-o** *options*] *dataset* *mountpoint* # DESCRIPTION The **mount.zfs** helper is used by mount(8) to mount filesystem snapshots and **mountpoint=**‌*legacy* ZFS filesystems, as well as by zfs(8) when the **ZFS\_MOUNT\_HELPER** environment variable is not set. Users should should invoke either mount(8) or zfs(8) in most cases. *options* are handled according to the *Temporary Mount Point Properties* section in zfsprops(7), except for those described below. If */etc/mtab* is a regular file and **-n** was not specified, it will be updated via libmount. # OPTIONS **-s** > Ignore unknown (sloppy) mount options. **-f** > Do everything except actually executing the system call. **-n** > Never update > */etc/mtab*. **-v** > Print resolved mount options and parser state. **-h** > Print the usage message. **-o** *zfsutil* > This private flag indicates that > mount(8) > is being called by the > zfs(8) > command. # SEE ALSO fstab(5), mount(8), zfs-mount(8) Debian - May 24, 2021 PAM\_ZFS\_KEY(8) - System Manager's Manual # NAME **pam\_zfs\_key** - PAM module for ZFS encryption key management # SYNOPSIS **pam\_zfs\_key.so** \[*options*] # DESCRIPTION **pam\_zfs\_key** is a PAM module that automatically manages encryption keys for ZFS datasets during user authentication and session management. When a user logs in, the module uses their password to unlock their encrypted home directory. When the last session closes, the module unmounts the dataset and unloads the key. The module tracks active sessions using reference counting to support multiple simultaneous logins from the same user. ## Multiple Home Prefixes When configured with multiple home prefixes, the module attempts operations on all matching datasets. Operations that succeed are not rolled back if others fail. The module returns success only if all operations succeed. For example, with datasets 1, 2, 3 where dataset 2 fails: * Auth/session: datasets 1 and 3 are unlocked and mounted, dataset 2 is not. * Password change: datasets 1 and 3 have the new password, dataset 2 retains the old. With **required**, login fails even though datasets 1 and 3 succeeded. With **optional**, login proceeds. For password changes, datasets 1 and 3 are updated while dataset 2 retains the old password. With **required**, the user sees an error. With **optional**, the user sees success and may not notice the inconsistency. Either way, passwords are left out of sync. Errors are logged to syslog. Use zfs-change-key(8) to resync passwords after partial failure. # OPTIONS **homes**=*path*\[,*path2*...] > Comma-separated list of dataset prefixes where user home directories > are located. > The module constructs the full dataset path as > *prefix*/*username*. > Default: > **zroot/home** > on > FreeBSD, > **rpool/home** > on Linux. **runstatedir**=*path* > Directory for storing session reference counts. > Default: > */var/run/pam\_zfs\_key*. **uid\_min**=*uid* > Minimum user ID for which the module will operate. > Default: 1000. **uid\_max**=*uid* > Maximum user ID for which the module will operate. > Default: MAXUID. **nounmount** > Do not unmount datasets or unload encryption keys when sessions close. > Datasets remain mounted and keys remain loaded. **forceunmount** > Force unmount datasets even if busy > (`MS_FORCE`). **recursive\_homes** > Recursively search for encrypted datasets under the homes prefix. **mount\_recursively** > Mount and unmount child datasets recursively. **prop\_mountpoint** > Find the user's dataset by matching the dataset's > **mountpoint** > property to the user's home directory from > */etc/passwd*, > instead of constructing the dataset name as > *prefix*/*username*. # FILES */var/run/pam\_zfs\_key/*‌*uid* > Session reference count files tracking active logins per user. # EXAMPLES ## Example 1: Basic Configuration Add to */etc/pam.d/system-auth*: auth optional pam_zfs_key.so password optional pam_zfs_key.so session optional pam_zfs_key.so This configuration uses default settings. User home datasets are expected at **zroot/home/**‌*username* on FreeBSD or **rpool/home/**‌*username* on Linux. ## Example 2: Custom Home Directory Prefix auth optional pam_zfs_key.so homes=tank/users password optional pam_zfs_key.so homes=tank/users session optional pam_zfs_key.so homes=tank/users Looks for user datasets at **tank/users/**‌*username*. ## Example 3: Multiple Dataset Prefixes session optional pam_zfs_key.so homes=rpool/home,tank/users Searches for user datasets in both **rpool/home** and **tank/users**. ## Example 4: Keep Datasets Mounted session optional pam_zfs_key.so nounmount Leaves datasets mounted and keys loaded when sessions close. Useful for systems with background processes accessing the home directory. ## Example 5: Recursive Mounting session optional pam_zfs_key.so mount_recursively Mounts child datasets recursively, useful when user data is organized hierarchically like **rpool/home/alice/documents** and **rpool/home/alice/photos**. ## Example 6: Creating an Encrypted Home Dataset # zfs create -o encryption=on \ -o keyformat=passphrase \ -o keylocation=prompt \ -o canmount=on \ -o mountpoint=/home/alice \ rpool/home/alice The user's login password must match the dataset passphrase for automatic unlocking to work. The dataset must have a ZFS-managed mountpoint (not legacy) and **canmount**=**on** for automatic mounting. ## Example 7: Multiple Homes with Password Sync Check auth optional pam_zfs_key.so homes=rpool/home,tank/home password required pam_zfs_key.so homes=rpool/home,tank/home session optional pam_zfs_key.so homes=rpool/home,tank/home Login proceeds even if some datasets are unavailable. Password changes fail if any dataset cannot be updated, ensuring the user is notified of sync issues. See *Multiple Home Prefixes* for failure behavior. # SEE ALSO pam(8), zfs-change-key(8), zfs-load-key(8), zfs-mount(8) # NOTES * Only works with datasets using **keyformat**=**passphrase**. * Datasets must have **keylocation**=**prompt**. * Datasets with **mountpoint**=**legacy**, **canmount**=**off**, or **canmount**=**noauto** will have keys loaded but not be automatically mounted. Debian - December 24, 2025 VDEV\_ID(8) - System Manager's Manual # NAME **vdev\_id** - generate user-friendly names for JBOD disks # SYNOPSIS **vdev\_id** **-d** *dev* **-c** *config\_file* **-g** **sas\_direct**|**sas\_switch**|**scsi** **-m** **-p** *phys\_per\_port* # DESCRIPTION **vdev\_id** is an udev helper which parses vdev\_id.conf(5) to map a physical path in a storage topology to a channel name. The channel name is combined with a disk enclosure slot number to create an alias that reflects the physical location of the drive. This is particularly helpful when it comes to tasks like replacing failed drives. Slot numbers may also be remapped in case the default numbering is unsatisfactory. The drive aliases will be created as symbolic links in */dev/disk/by-vdev*. The currently supported topologies are **sas\_direct**, **sas\_switch**, and **scsi**. A multipath mode is supported in which dm-mpath devices are handled by examining the first running component disk as reported by the driver. In multipath mode the configuration file should contain a channel definition with the same name for each path to a given enclosure. **vdev\_id** also supports creating aliases based on existing udev links in the /dev hierarchy using the **alias** configuration file keyword. See vdev\_id.conf(5) for details. # OPTIONS **-d** *device* > The device node to classify, like > */dev/sda*. **-c** *config\_file* > Specifies the path to an alternate configuration file. > The default is > */etc/zfs/vdev\_id.conf*. **-g** **sas\_direct**|**sas\_switch**|**scsi** > Identifies a physical topology that governs how physical paths are > mapped to channels: > **sas\_direct** and **scsi** > > channels are uniquely identified by a PCI slot and HBA port number > **sas\_switch** > > channels are uniquely identified by a SAS switch port number **-m** > Only handle dm-multipath devices. > If specified, examine the first running component disk of a dm-multipath > device as provided by the driver to determine the physical path. **-p** *phys\_per\_port* > Specifies the number of PHY devices associated with a SAS HBA port or SAS > switch port. > **vdev\_id** > internally uses this value to determine which HBA or switch port a > device is connected to. > The default is > **4**. **-h** > Print a usage summary. # SEE ALSO vdev\_id.conf(5) Debian - May 26, 2021 ZDB(8) - System Manager's Manual # NAME **zdb** - display ZFS storage pool debugging and consistency information # SYNOPSIS **zdb** \[**-AbcdDFGhikLMNPsTvXYy**] \[**-e** \[**-V**] \[**-p** *path*]…] \[**-I** *inflight-I/O-ops*] \[**-o** *var*=*value*]… \[**-t** *txg*] \[**-U** *cache*] \[**-x** *dumpdir*] \[**-K** *key*] \[*poolname*\[/*dataset*|*objset-ID*]] \[*object*|*range*…] **zdb** \[**-AdiPv**] \[**-e** \[**-V**] \[**-p** *path*]…] \[**-U** *cache*] \[**-K** *key*] *poolname*\[*/*‌*dataset*|*objset-ID*] \[*object*|*range*…] **zdb** **-B** \[**-e** \[**-V**] \[**-p** *path*]…] \[**-U** *cache*] \[**-K** *key*] *poolname*‌*/*‌*objset-ID* \[*backup-flags*] **zdb** **-C** \[**-A**] \[**-U** *cache*] \[*poolname*] **zdb** **-E** \[**-A**] *word0*:*word1*:…:*word15* **zdb** **-l** \[**-Aqu**] *device* **zdb** **-m** \[**-AFLPXY**] \[**-e** \[**-V**] \[**-p** *path*]…] \[**-t** *txg*] \[**-U** *cache*] *poolname* \[*vdev* \[*metaslab*]…] **zdb** **--allocated-map** \[**-mAFLPXY**] \[**-e** \[**-V**] \[**-p** *path*]…] \[**-t** *txg*] \[**-U** *cache*] *poolname* \[*vdev* \[*metaslab*]…] **zdb** **-O** \[**-K** *key*] *dataset path* **zdb** **-r** \[**-K** *key*] *dataset path destination* **zdb** **-r** **-O** \[**-K** *key*] *dataset object-id destination* **zdb** **-R** \[**-A**] \[**-e** \[**-V**] \[**-p** *path*]…] \[**-U** *cache*] *poolname vdev*:*offset*:\[*lsize*/]*psize*\[:*flags*] **zdb** **-S** \[**-AP**] \[**-e** \[**-V**] \[**-p** *path*]…] \[**-U** *cache*] *poolname* # DESCRIPTION The **zdb** utility displays information about a ZFS pool useful for debugging and performs some amount of consistency checking. It is a not a general purpose tool and options (and facilities) may change. It is not a fsck(8) utility. The output of this command in general reflects the on-disk structure of a ZFS pool, and is inherently unstable. The precise output of most invocations is not documented, a knowledge of ZFS internals is assumed. If the *dataset* argument does not contain any "**/**" or "**@**" characters, it is interpreted as a pool name. The root dataset can be specified as "*pool*/". **zdb** is an "offline" tool; it accesses the block devices underneath the pools directly from userspace and does not care if the pool is imported or datasets are mounted (or even if the system understands ZFS at all). When operating on an imported and active pool it is possible, though unlikely, that zdb may interpret inconsistent pool data and behave erratically. # OPTIONS Display options: **-**‌**-allocated-map** > Prints out a list of all the allocated regions in the pool. > Primarily intended for use with the > **zhack metaslab leak** > subcommand. **-b**, **--block-stats** > Display statistics regarding the number, size > (logical, physical and allocated) > and deduplication of blocks. **--bin**=(`lsize`|`psize`|`asize`) > When used with > **-bb**, > sort blocks into all three bins according to the given size (instead of binning > a block for each size separately). > For instance, with > **--bin**=`lsize`, > a block with lsize of 16K and psize of 4K will be added to the 16K bin > in all three columns. **--class**=(`normal`|`special`|`dedup`|`other`)\[,…] > When used with > **-bb**, > only consider blocks from these allocation classes. **-B**, **--backup** > Generate a backup stream, similar to > **zfs** **send**, > but for the numeric objset ID, and without opening the dataset. > This can be useful in recovery scenarios if dataset metadata has become > corrupted but the dataset itself is readable. > The optional > *flags* > argument is a string of one or more of the letters > **e**, > **L**, > **c**, > and > **w**, > which correspond to the same flags in > zfs-send(8). **-c**, **--checksum** > Verify the checksum of all metadata blocks while printing block statistics > (see > **-b**) > . > If specified multiple times, verify the checksums of all blocks. **-C**, **--config** > Display information about the configuration. > If specified with no other options, instead display information about the cache > file > (*/etc/zfs/zpool.cache*). > To specify the cache file to display, see > **-U**. > If specified multiple times, and a pool name is also specified display both the > cached configuration and the on-disk configuration. > If specified multiple times with > **-e** > also display the configuration that would be used were the pool to be imported. **-d**, **--datasets** > Display information about datasets. > Specified once, displays basic dataset information: ID, create transaction, > size, and object count. > See > **-N** > for determining if > *poolname*\[/*dataset*|*objset-ID*] > is to use the specified > *dataset*|*objset-ID* > as a string (dataset name) or a number (objset ID) when > datasets have numeric names. > If specified multiple times provides greater and greater verbosity. > If object IDs or object ID ranges are specified, display information about > those specific objects or ranges only. > An object ID range is specified in terms of a colon-separated tuple of > the form > <start>:<end>\[:<flags>]. > The fields > *start* > and > *end* > are integer object identifiers that denote the upper and lower bounds > of the range. > An > *end* > value of -1 specifies a range with no upper bound. > The > *flags* > field optionally specifies a set of flags, described below, that control > which object types are dumped. > By default, all object types are dumped. > A minus sign > (-) > negates the effect of the flag that follows it and has no effect unless > preceded by the > *A* > flag. > For example, the range 0:-1:A-d will dump all object types except for > directories. > **A** > > Dump all objects (this is the default) > **d** > > Dump ZFS directory objects > **f** > > Dump ZFS plain file objects > **m** > > Dump SPA space map objects > **z** > > Dump ZAP objects > **-** > > Negate the effect of next flag **-D**, **--dedup-stats** > Display deduplication statistics, including the deduplication ratio > (**dedup**), > compression ratio > (**compress**), > inflation due to the zfs copies property > (**copies**), > and an overall effective ratio > (**dedup** × **compress** / **copies**). **-DD** > Display a histogram of deduplication statistics, showing the allocated > (physically present on disk) > and referenced > (logically referenced in the pool) > block counts and sizes by reference count. **-DDD** > Display the statistics independently for each deduplication table. **-DDDD** > Dump the contents of the deduplication tables describing duplicate blocks. **-DDDDD** > Also dump the contents of the deduplication tables describing unique blocks. **-E**, **--embedded-block-pointer**=*word0*:*word1*:…:*word15* > Decode and display block from an embedded block pointer specified by the > *word* > arguments. **-f**, **--file-layout** > Display the file layout of an object for the disks of a raidz vdev. > Numeric values in the disply are hexadecimal. > With > **-H**, > the output is in scripted mode for easy parsing, with all values > being presented as 512 byte blocks in decimal; with > **-v**, > the block type (parity or data) is displayed; with > **-vv**, > the offset into the file for each block is also printed. > Only a single top-level raidz vdev is supported. **-h**, **--history** > Display pool history similar to > **zpool** **history**, > but include internal changes, transaction, and dataset information. **-i**, **--intent-logs** > Display information about intent log > (ZIL) > entries relating to each dataset. > If specified multiple times, display counts of each intent log transaction type. **-k**, **--checkpointed-state** > Examine the checkpointed state of the pool. > Note, the on disk format of the pool is not reverted to the checkpointed state. **-l**, **--label**=*device* > Read the vdev labels and L2ARC header from the specified device. > **zdb** **-l** > will return 0 if valid label was found, 1 if error occurred, and 2 if no valid > labels were found. > The presence of L2ARC header is indicated by a specific > sequence (L2ARC\_DEV\_HDR\_MAGIC). > If there is an accounting error in the size or the number of L2ARC log blocks > **zdb** **-l** > will return 1. > Each unique configuration is displayed only once. **-ll** *device* > In addition display label space usage stats. > If a valid L2ARC header was found > also display the properties of log blocks used for restoring L2ARC contents > (persistent L2ARC). **-lll** *device* > Display every configuration, unique or not. > If a valid L2ARC header was found > also display the properties of log entries in log blocks used for restoring > L2ARC contents (persistent L2ARC). > If the > **-q** > option is also specified, don't print the labels or the L2ARC header. > If the > **-u** > option is also specified, also display the uberblocks on this device. > Specify multiple times to increase verbosity. **-L**, **--disable-leak-tracking** > Disable leak detection and the loading of space maps. > By default, > **zdb** > verifies that all non-free blocks are referenced, which can be very expensive. **-m**, **--metaslabs** > Display the offset, spacemap, free space of each metaslab, all the log > spacemaps and their obsolete entry statistics. **-mm** > Also display information about the on-disk free space histogram associated with > each metaslab. **-mmm** > Display the maximum contiguous free space, the in-core free space histogram, and > the percentage of free space in each space map. **-mmmm** > Display every spacemap record. **-M**, **--metaslab-groups** > Display all "normal" vdev metaslab group information - per-vdev metaslab count, > fragmentation, > and free space histogram, as well as overall pool fragmentation and histogram. **-MM** > "Special" vdevs are added to -M's normal output. > Also display information about the maximum contiguous free space and the > percentage of free space in each space map. **-MMM** > Display every spacemap record. **-N** > Same as > **-d** > but force zdb to interpret the > \[*dataset*|*objset-ID*] > in > \[*poolname*\[/*dataset*|*objset-ID*]] > as a numeric objset ID. **-O**, **--object-lookups**=*dataset path* > Look up the specified > *path* > inside of the > *dataset* > and display its metadata and indirect blocks. > Specified > *path* > must be relative to the root of > *dataset*. > This option can be combined with > **-v** > for increasing verbosity. **-r**, **--copy-object**=*dataset path destination* > Copy the specified > *path* > inside of the > *dataset* > to the specified destination. > Specified > *path* > must be relative to the root of > *dataset*. > When used with > **-O**, > the > *path* > argument is interpreted as an object identifier, > not a path. > This option can be combined with > **-v** > for increasing verbosity. **-R**, **--read-block**=*poolname vdev*:*offset*:\[*lsize*/]*psize*\[:*flags*] > Read and display a block from the specified device. > By default the block is displayed as a hex dump, but see the description of the > **r** > flag, below. > The block is specified in terms of a colon-separated tuple > *vdev* > (an integer vdev identifier) > *offset* > (the offset within the vdev) > *size* > (the physical size, or logical size / physical size) > of the block to read and, optionally, > *flags* > (a set of flags, described below). > **b** *offset* > > Print block pointer at hex offset > **c** > > Calculate and display checksums > **d** > > Decompress the block. > > Set environment variable > > **ZDB\_NO\_ZLE** > > to skip zle when guessing. > **e** > > Byte swap the block > **g** > > Dump gang block header > **i** > > Dump indirect block > **r** > > Dump raw uninterpreted block data > **v** > > Verbose output for guessing compression algorithm **-s**, **--io-stats** > Report statistics on > **zdb** > I/O. > Display operation counts, bandwidth, and error counts of I/O to the pool from > **zdb**. **-S**, **--simulate-dedup** > Simulate the effects of deduplication, constructing a DDT and then display > that DDT as with > **-DD**. **-T**, **--brt-stats** > Display block reference table (BRT) statistics, including the size of uniques > blocks cloned, the space saving as a result of cloning, and the saving ratio. **-TT** > Display the per-vdev BRT statistics, including total references. **-TTT** > Display histograms of per-vdev BRT refcounts. **-TTTT** > Dump the contents of the block reference tables. **-u**, **--uberblock** > Display the current uberblock. Other options: **-A**, **--ignore-assertions** > Do not abort should any assertion fail. **-AA** > Enable panic recovery, certain errors which would otherwise be fatal are > demoted to warnings. **-AAA** > Do not abort if asserts fail and also enable panic recovery. **-e**, **--exported**=\[**-p** *path*]… > Operate on an exported pool, not present in > */etc/zfs/zpool.cache*. > The > **-p** > flag specifies the path under which devices are to be searched. **-x**, **--dump-blocks**=*dumpdir* > All blocks accessed will be copied to files in the specified directory. > The blocks will be placed in sparse files whose name is the same as > that of the file or device read. > **zdb** > can be then run on the generated files. > Note that the > **-bbc** > flags are sufficient to access > (and thus copy) > all metadata on the pool. **-F**, **--automatic-rewind** > Attempt to make an unreadable pool readable by trying progressively older > transactions. **-G**, **--dump-debug-msg** > Dump the contents of the zfs\_dbgmsg buffer before exiting > **zdb**. > zfs\_dbgmsg is a buffer used by ZFS to dump advanced debug information. **-I**, **--inflight**=*inflight-I/O-ops* > Limit the number of outstanding checksum I/O operations to the specified value. > The default value is 200. > This option affects the performance of the > **-c** > option. **-K**, **--key**=*key* > Decryption key needed to access an encrypted dataset. > This will cause > **zdb** > to attempt to unlock the dataset using the encryption root, key format and other > encryption parameters on the given dataset. > **zdb** > can still inspect pool and dataset structures on encrypted datasets without > unlocking them, but will not be able to access file names and attributes and > object contents. **WARNING:** The raw decryption key and any decrypted data > will be in user memory while > **zdb** > is running. > Other user programs may be able to extract it by inspecting > **zdb** > as it runs. > Exercise extreme caution when using this option in shared or uncontrolled > environments. **-o**, **--option**=*var*=*value*… > Set the given tunable to the provided value. **-o**, **--option**=*var*… > Show the value of the given tunable. **-o**, **--option**=show > Show all tunables and their values. **-o**, **--option**=info=*value*… > Show info about a tunable, including their name, type and description. **-o**, **--option**=info > Show info about all tunables. **-P**, **--parseable** > Print numbers in an unscaled form more amenable to parsing, e.g. > **1000000** > rather than > **1M**. **-t**, **--txg**=*transaction* > Specify the highest transaction to use when searching for uberblocks. > See also the > **-u** > and > **-l** > options for a means to see the available uberblocks and their associated > transaction numbers. **-U**, **--cachefile**=*cachefile* > Use a cache file other than > */etc/zfs/zpool.cache*. **-v**, **--verbose** > Enable verbosity. > Specify multiple times for increased verbosity. **-V**, **--verbatim** > Attempt verbatim import. > This mimics the behavior of the kernel when loading a pool from a cachefile. > Only usable with > **-e**. **-X**, **--extreme-rewind** > Attempt > "extreme" > transaction rewind, that is attempt the same recovery as > **-F** > but read transactions otherwise deemed too old. **-Y**, **--all-reconstruction** > Attempt all possible combinations when reconstructing indirect split blocks. > This flag disables the individual I/O deadman timer in order to allow as > much time as required for the attempted reconstruction. **-y**, **--livelist** > Perform validation for livelists that are being deleted. > Scans through the livelist and metaslabs, checking for duplicate entries > and compares the two, checking for potential double frees. > If it encounters issues, warnings will be printed, but the command will not > necessarily fail. Specifying a display option more than once enables verbosity for only that option, with more occurrences enabling more verbosity. If no options are specified, all information about the named pool will be displayed at default verbosity. # EXIT STATUS The **zdb** utility exits **0** on success, **1** if a fatal error occurs, **2** if invalid command line options were specified, or **3** if on-disk corruption was detected, but was not fatal. # EXAMPLES ## Example 1: Display the configuration of imported pool *rpool* # **zdb** **-C** *rpool* MOS Configuration: version: 28 name: 'rpool' ## Example 2: Display basic dataset information about *rpool* # **zdb** **-d** *rpool* Dataset mos [META], ID 0, cr_txg 4, 26.9M, 1051 objects Dataset rpool/swap [ZVOL], ID 59, cr_txg 356, 486M, 2 objects ## Example 3: Display basic information about object 0 in *rpool/export/home* # **zdb** **-d** *rpool/export/home 0* Dataset rpool/export/home [ZPL], ID 137, cr_txg 1546, 32K, 8 objects Object lvl iblk dblk dsize lsize %full type 0 7 16K 16K 15.0K 16K 25.00 DMU dnode ## Example 4: Display the predicted effect of enabling deduplication on *rpool* # **zdb** **-S** *rpool* Simulated DDT histogram: bucket allocated referenced ______ ______________________________ ______________________________ refcnt blocks LSIZE PSIZE DSIZE blocks LSIZE PSIZE DSIZE ------ ------ ----- ----- ----- ------ ----- ----- ----- 1 694K 27.1G 15.0G 15.0G 694K 27.1G 15.0G 15.0G 2 35.0K 1.33G 699M 699M 74.7K 2.79G 1.45G 1.45G dedup = 1.11, compress = 1.80, copies = 1.00, dedup * compress / copies = 2.00 # SEE ALSO zfs(8), zpool(8) Debian - August 12, 2025 ZED(8) - System Manager's Manual # NAME **ZED** - ZFS Event Daemon # SYNOPSIS **ZED** \[**-fFhILMvVZ**] \[**-d** *zedletdir*] \[**-p** *pidfile*] \[**-P** *path*] \[**-s** *statefile*] \[**-j** *jobs*] \[**-b** *buflen*] # DESCRIPTION The **ZED** (ZFS Event Daemon) monitors events generated by the ZFS kernel module. When a zevent (ZFS Event) is posted, the **ZED** will run any ZEDLETs (ZFS Event Daemon Linkage for Executable Tasks) that have been enabled for the corresponding zevent class. # OPTIONS **-h** > Display a summary of the command-line options. **-L** > Display license information. **-V** > Display version information. **-v** > Be verbose. **-f** > Force the daemon to run if at all possible, disabling security checks and > throwing caution to the wind. > Not recommended for use in production. **-F** > Don't daemonize: remain attached to the controlling terminal, > log to the standard I/O streams. **-M** > Lock all current and future pages in the virtual memory address space. > This may help the daemon remain responsive when the system is under heavy > memory pressure. **-I** > Request that the daemon idle rather than exit when the kernel modules are not > loaded. > Processing of events will start, or resume, when the kernel modules are > (re)loaded. > Under Linux the kernel modules cannot be unloaded while the daemon is running. **-Z** > Zero the daemon's state, thereby allowing zevents still within the kernel > to be reprocessed. **-d** *zedletdir* > Read the enabled ZEDLETs from the specified directory. **-p** *pidfile* > Write the daemon's process ID to the specified file. **-P** *path* > Custom > `$PATH` > for zedlets to use. > Normally zedlets run in a locked-down environment, with hardcoded paths to the > ZFS commands > (`$ZFS`, `$ZPOOL`, `$ZED`, ``), > and a hard-coded > `$PATH`. > This is done for security reasons. > However, the ZFS test suite uses a custom PATH for its ZFS commands, and passes > it to > **ZED** > with > **-P**. > In short, > **-P** > is only to be used by the ZFS test suite; never use > it in production! **-s** *statefile* > Write the daemon's state to the specified file. **-j** *jobs* > Allow at most > *jobs* > ZEDLETs to run concurrently, > delaying execution of new ones until they finish. > Defaults to > **16**. **-b** *buflen* > Cap kernel event buffer growth to > *buflen* > entries. > This buffer is grown when the daemon misses an event, but results in > unreclaimable memory use in the kernel. > A value of > **0** > removes the cap. > Defaults to > **1048576**. # ZEVENTS A zevent is comprised of a list of nvpairs (name/value pairs). Each zevent contains an EID (Event IDentifier) that uniquely identifies it throughout the lifetime of the loaded ZFS kernel module; this EID is a monotonically increasing integer that resets to 1 each time the kernel module is loaded. Each zevent also contains a class string that identifies the type of event. For brevity, a subclass string is defined that omits the leading components of the class string. Additional nvpairs exist to provide event details. The kernel maintains a list of recent zevents that can be viewed (along with their associated lists of nvpairs) using the **zpool** **events** **-v** command. # CONFIGURATION ZEDLETs to be invoked in response to zevents are located in the *enabled-zedlets* directory (*zedletdir*). These can be symlinked or copied from the *installed-zedlets* directory; symlinks allow for automatic updates from the installed ZEDLETs, whereas copies preserve local modifications. As a security measure, since ownership change is a privileged operation, ZEDLETs must be owned by root. They must have execute permissions for the user, but they must not have write permissions for group or other. Dotfiles are ignored. ZEDLETs are named after the zevent class for which they should be invoked. In particular, a ZEDLET will be invoked for a given zevent if either its class or subclass string is a prefix of its filename (and is followed by a non-alphabetic character). As a special case, the prefix **all** matches all zevents. Multiple ZEDLETs may be invoked for a given zevent. # ZEDLETS ZEDLETs are executables invoked by the ZED in response to a given zevent. They should be written under the presumption they can be invoked concurrently, and they should use appropriate locking to access any shared resources. The one exception to this are "synchronous zedlets", which are described later in this page. Common variables used by ZEDLETs can be stored in the default rc file which is sourced by scripts; these variables should be prefixed with **ZED\_**. The zevent nvpairs are passed to ZEDLETs as environment variables. Each nvpair name is converted to an environment variable in the following manner: 1. it is prefixed with **ZEVENT\_**, 2. it is converted to uppercase, and 3. each non-alphanumeric character is converted to an underscore. Some additional environment variables have been defined to present certain nvpair values in a more convenient form. An incomplete list of zevent environment variables is as follows: **ZEVENT\_EID** > The Event IDentifier. **ZEVENT\_CLASS** > The zevent class string. **ZEVENT\_SUBCLASS** > The zevent subclass string. **ZEVENT\_TIME** > The time at which the zevent was posted as > "*seconds nanoseconds*" > since the Epoch. **ZEVENT\_TIME\_SECS** > The > *seconds* > component of > **ZEVENT\_TIME**. **ZEVENT\_TIME\_NSECS** > The > *nanoseconds* > component of > **ZEVENT\_TIME**. **ZEVENT\_TIME\_STRING** > An almost-RFC3339-compliant string for > **ZEVENT\_TIME**. Additionally, the following ZED & ZFS variables are defined: **ZED\_PID** > The daemon's process ID. **ZED\_ZEDLET\_DIR** > The daemon's current > *enabled-zedlets* > directory. **ZFS\_ALIAS** > The alias > ("*name*-*version*-*release*") > string of the ZFS distribution the daemon is part of. **ZFS\_VERSION** > The ZFS version the daemon is part of. **ZFS\_RELEASE** > The ZFS release the daemon is part of. ZEDLETs may need to call other ZFS commands. The installation paths of the following executables are defined as environment variables: **ZDB**, **ZED**, **ZFS**, **ZINJECT**, and **ZPOOL**. These variables may be overridden in the rc file. # Synchronous ZEDLETS ZED's normal behavior is to spawn off zedlets in parallel and ignore their completion order. This means that ZED can potentially have zedlets for event ID number 2 starting before zedlets for event ID number 1 have finished. Most of the time this is fine, and it actually helps when the system is getting hammered with hundreds of events. However, there are times when you want your zedlets to be executed in sequence with the event ID. That is where synchronous zedlets come in. ZED will wait for all previously spawned zedlets to finish before running a synchronous zedlet. Synchronous zedlets are guaranteed to be the only zedlet running. No other zedlets may run in parallel with a synchronous zedlet. Users should be careful to only use synchronous zedlets when needed, since they decrease parallelism. To make a zedlet synchronous, simply add a "-sync-" immediately following the event name in the zedlet's file name: **EVENT\_NAME-sync-ZEDLETNAME.sh** For example, if you wanted a synchronous statechange script: **statechange-sync-myzedlet.sh** # FILES *@sysconfdir@/zfs/zed.d* > The default directory for enabled ZEDLETs. *@sysconfdir@/zfs/zed.d/zed.rc* > The default rc file for common variables used by ZEDLETs. *@zfsexecdir@/zed.d* > The default directory for installed ZEDLETs. *@runstatedir@/zed.pid* > The default file containing the daemon's process ID. *@runstatedir@/zed.state* > The default file containing the daemon's state. # SIGNALS **SIGHUP** > Reconfigure the daemon and rescan the directory for enabled ZEDLETs. **SIGTERM**, **SIGINT** > Terminate the daemon. # SEE ALSO zfs(8), zpool(8), zpool-events(8) # NOTES The **ZED** requires root privileges. Do not taunt the **ZED**. # BUGS ZEDLETs are unable to return state/status information to the kernel. Internationalization support via gettext has not been added. Debian - August 22, 2022 .\" SPDX-License-Identifier: CDDL-1.0 .\" .\" This file and its contents are supplied under the terms of the .\" Common Development and Distribution License ("CDDL"), version 1.0. .\" You may only use this file in accordance with the terms of version .\" 1.0 of the CDDL. .\" .\" A full copy of the text of the CDDL should have accompanied this .\" source. A copy of the CDDL is also available via the Internet at .\" https://opensource.org/license/CDDL-1.0. .\" .\" Copyright (c) 2009 Sun Microsystems, Inc. All Rights Reserved. .\" Copyright 2011 Joshua M. Clulow .\" Copyright (c) 2011, 2019 by Delphix. All rights reserved. .\" Copyright (c) 2013 by Saso Kiselkov. All rights reserved. .\" Copyright (c) 2014, Joyent, Inc. All rights reserved. .\" Copyright (c) 2014 by Adam Stevko. All rights reserved. .\" Copyright (c) 2014 Integros [integros.com] .\" Copyright 2019 Richard Laager. All rights reserved. .\" Copyright 2018 Nexenta Systems, Inc. .\" Copyright 2019 Joyent, Inc. .\" Copyright (c) 2026, TrueNAS. .\" .Dd June 14, 2026 .Dt ZFS-ALLOW 8 .Os . .Sh NAME .Nm zfs-allow .Nd delegate ZFS administration permissions to unprivileged users .Sh SYNOPSIS .Nm zfs .Cm allow .Op Fl dglu .Ar user Ns | Ns Ar group Ns Oo , Ns Ar user Ns | Ns Ar group Oc Ns … .Ar perm Ns | Ns @ Ns Ar setname Ns Oo , Ns Ar perm Ns | Ns @ Ns .Ar setname Oc Ns … .Ar filesystem Ns | Ns Ar volume .Nm zfs .Cm allow .Op Fl dl .Fl e Ns | Ns Sy everyone .Ar perm Ns | Ns @ Ns Ar setname Ns Oo , Ns Ar perm Ns | Ns @ Ns .Ar setname Oc Ns … .Ar filesystem Ns | Ns Ar volume .Nm zfs .Cm allow .Fl c .Ar perm Ns | Ns @ Ns Ar setname Ns Oo , Ns Ar perm Ns | Ns @ Ns .Ar setname Oc Ns … .Ar filesystem Ns | Ns Ar volume .Nm zfs .Cm allow .Fl s No @ Ns Ar setname .Ar perm Ns | Ns @ Ns Ar setname Ns Oo , Ns Ar perm Ns | Ns @ Ns .Ar setname Oc Ns … .Ar filesystem Ns | Ns Ar volume .Nm zfs .Cm unallow .Op Fl dglru .Ar user Ns | Ns Ar group Ns Oo , Ns Ar user Ns | Ns Ar group Oc Ns … .Oo Ar perm Ns | Ns @ Ns Ar setname Ns Oo , Ns Ar perm Ns | Ns @ Ns .Ar setname Oc Ns … Oc .Ar filesystem Ns | Ns Ar volume .Nm zfs .Cm unallow .Op Fl dlr .Fl e Ns | Ns Sy everyone .Oo Ar perm Ns | Ns @ Ns Ar setname Ns Oo , Ns Ar perm Ns | Ns @ Ns .Ar setname Oc Ns … Oc .Ar filesystem Ns | Ns Ar volume .Nm zfs .Cm unallow .Op Fl r .Fl c .Oo Ar perm Ns | Ns @ Ns Ar setname Ns Oo , Ns Ar perm Ns | Ns @ Ns .Ar setname Oc Ns … Oc .Ar filesystem Ns | Ns Ar volume .Nm zfs .Cm unallow .Op Fl r .Fl s No @ Ns Ar setname .Oo Ar perm Ns | Ns @ Ns Ar setname Ns Oo , Ns Ar perm Ns | Ns @ Ns .Ar setname Oc Ns … Oc .Ar filesystem Ns | Ns Ar volume . .Sh DESCRIPTION .Bl -tag -width "" .It Xo .Nm zfs .Cm allow .Ar filesystem Ns | Ns Ar volume .Xc Displays permissions that have been delegated on the specified filesystem or volume. See the other forms of .Nm zfs Cm allow for more information. .Pp Delegations are supported under Linux with the exception of .Sy mount , .Sy unmount , .Sy mountpoint , .Sy canmount , .Sy rename , and .Sy share . These permissions cannot be delegated because the Linux .Xr mount 8 command restricts modifications of the global namespace to the root user. .It Xo .Nm zfs .Cm allow .Op Fl dglu .Ar user Ns | Ns Ar group Ns Oo , Ns Ar user Ns | Ns Ar group Oc Ns … .Ar perm Ns | Ns @ Ns Ar setname Ns Oo , Ns Ar perm Ns | Ns @ Ns .Ar setname Oc Ns … .Ar filesystem Ns | Ns Ar volume .Xc .It Xo .Nm zfs .Cm allow .Op Fl dl .Fl e Ns | Ns Sy everyone .Ar perm Ns | Ns @ Ns Ar setname Ns Oo , Ns Ar perm Ns | Ns @ Ns .Ar setname Oc Ns … .Ar filesystem Ns | Ns Ar volume .Xc Delegates ZFS administration permission for the file systems to non-privileged users. .Bl -tag -width "-d" .It Fl d Allow only for the descendent file systems. .It Fl e Ns | Ns Sy everyone Specifies that the permissions be delegated to everyone. .It Fl g Ar group Ns Oo , Ns Ar group Oc Ns … Explicitly specify that permissions are delegated to the group. .It Fl l Allow .Qq locally only for the specified file system. .It Fl u Ar user Ns Oo , Ns Ar user Oc Ns … Explicitly specify that permissions are delegated to the user. .It Ar user Ns | Ns Ar group Ns Oo , Ns Ar user Ns | Ns Ar group Oc Ns … Specifies to whom the permissions are delegated. Multiple entities can be specified as a comma-separated list. If neither of the .Fl gu options are specified, then the argument is interpreted preferentially as the keyword .Sy everyone , then as a user name, and lastly as a group name. To specify a user or group named .Qq everyone , use the .Fl g or .Fl u options. To specify a group with the same name as a user, use the .Fl g options. .It Xo .Ar perm Ns | Ns @ Ns Ar setname Ns Oo , Ns Ar perm Ns | Ns @ Ns .Ar setname Oc Ns … .Xc The permissions to delegate. Multiple permissions may be specified as a comma-separated list. Permission names are the same as ZFS subcommand and property names. See the property list below. Property set names, which begin with .Sy @ , may be specified. See the .Fl s form below for details. .El .Pp If neither of the .Fl dl options are specified, or both are, then the permissions are allowed for the file system or volume, and all of its descendants. .Pp Permissions are generally the ability to use a ZFS subcommand or change a ZFS property. The following permissions are available: .TS l l l . NAME TYPE NOTES _ _ _ allow subcommand Must also have the permission that is being allowed bookmark subcommand clone subcommand Must also have the \fBcreate\fR ability and \fBmount\fR ability in the origin file system create subcommand Must also have the \fBmount\fR ability. Must also have the \fBrefreservation\fR ability to create a non-sparse volume. destroy subcommand Must also have the \fBmount\fR ability diff subcommand Allows lookup of paths within a dataset given an object number, and the ability to create snapshots necessary to \fBzfs diff\fR. hold subcommand Allows adding a user hold to a snapshot load-key subcommand Allows loading and unloading of encryption key (see \fBzfs load-key\fR and \fBzfs unload-key\fR). change-key subcommand Allows changing an encryption key via \fBzfs change-key\fR. mount subcommand Allows mounting/unmounting ZFS datasets promote subcommand Must also have the \fBmount\fR and \fBpromote\fR ability in the origin file system receive subcommand Must also have the \fBmount\fR and \fBcreate\fR ability, required for \fBzfs receive -F\fR (see also \fBreceive:append\fR for limited, non forced receive) release subcommand Allows releasing a user hold which might destroy the snapshot rename subcommand Must also have the \fBmount\fR and \fBcreate\fR ability in the new parent rollback subcommand Must also have the \fBmount\fR ability send subcommand Allows sending a replication stream of a dataset. send:raw subcommand Only allows sending raw replication streams, preventing encrypted datasets being sent in decrypted form. send:encrypted subcommand Only allows sending raw replication streams of encrypted datasets, preventing encrypted datasets being sent in decrypted form and preventing decrypted datasets being sent at all. share subcommand Allows sharing file systems over NFS or SMB protocols snapshot subcommand Must also have the \fBmount\fR ability receive:append other Must also have the \fBmount\fR and \fBcreate\fR ability, limited receive ability (can not do receive -F) groupquota other Allows accessing any \fBgroupquota@\fI…\fR property groupobjquota other Allows accessing any \fBgroupobjquota@\fI…\fR property groupused other Allows reading any \fBgroupused@\fI…\fR property groupobjused other Allows reading any \fBgroupobjused@\fI…\fR property userprop other Allows changing any user property userquota other Allows accessing any \fBuserquota@\fI…\fR property userobjquota other Allows accessing any \fBuserobjquota@\fI…\fR property userused other Allows reading any \fBuserused@\fI…\fR property userobjused other Allows reading any \fBuserobjused@\fI…\fR property projectobjquota other Allows accessing any \fBprojectobjquota@\fI…\fR property projectquota other Allows accessing any \fBprojectquota@\fI…\fR property projectobjused other Allows reading any \fBprojectobjused@\fI…\fR property projectused other Allows reading any \fBprojectused@\fI…\fR property aclinherit property aclmode property acltype property atime property canmount property casesensitivity property checksum property compression property context property copies property dedup property defcontext property devices property dnodesize property encryption property exec property filesystem_limit property fscontext property keyformat property keylocation property logbias property mlslabel property mountpoint property nbmand property normalization property overlay property pbkdf2iters property primarycache property quota property readonly property recordsize property redundant_metadata property refquota property refreservation property relatime property reservation property rootcontext property secondarycache property setuid property sharenfs property sharesmb property snapdev property snapdir property snapshot_limit property special_small_blocks property sync property utf8only property version property volblocksize property volmode property volsize property vscan property xattr property zoned property .TE .It Xo .Nm zfs .Cm allow .Fl c .Ar perm Ns | Ns @ Ns Ar setname Ns Oo , Ns Ar perm Ns | Ns @ Ns .Ar setname Oc Ns … .Ar filesystem Ns | Ns Ar volume .Xc Sets .Qq create time permissions. These permissions are granted .Pq locally to the creator of any newly-created descendent file system. .It Xo .Nm zfs .Cm allow .Fl s No @ Ns Ar setname .Ar perm Ns | Ns @ Ns Ar setname Ns Oo , Ns Ar perm Ns | Ns @ Ns .Ar setname Oc Ns … .Ar filesystem Ns | Ns Ar volume .Xc Defines or adds permissions to a permission set. The set can be used by other .Nm zfs Cm allow commands for the specified file system and its descendants. Sets are evaluated dynamically, so changes to a set are immediately reflected. Permission sets follow the same naming restrictions as ZFS file systems, but the name must begin with .Sy @ , and can be no more than 64 characters long. .It Xo .Nm zfs .Cm unallow .Op Fl dglru .Ar user Ns | Ns Ar group Ns Oo , Ns Ar user Ns | Ns Ar group Oc Ns … .Oo Ar perm Ns | Ns @ Ns Ar setname Ns Oo , Ns Ar perm Ns | Ns @ Ns .Ar setname Oc Ns … Oc .Ar filesystem Ns | Ns Ar volume .Xc .It Xo .Nm zfs .Cm unallow .Op Fl dlr .Fl e Ns | Ns Sy everyone .Oo Ar perm Ns | Ns @ Ns Ar setname Ns Oo , Ns Ar perm Ns | Ns @ Ns .Ar setname Oc Ns … Oc .Ar filesystem Ns | Ns Ar volume .Xc .It Xo .Nm zfs .Cm unallow .Op Fl r .Fl c .Oo Ar perm Ns | Ns @ Ns Ar setname Ns Oo , Ns Ar perm Ns | Ns @ Ns .Ar setname Oc Ns … Oc .Ar filesystem Ns | Ns Ar volume .Xc Removes permissions that were granted with the .Nm zfs Cm allow command. No permissions are explicitly denied, so other permissions granted are still in effect. For example, if the permission is granted by an ancestor. If no permissions are specified, then all permissions for the specified .Ar user , .Ar group , or .Sy everyone are removed. Specifying .Sy everyone .Po or using the .Fl e option .Pc only removes the permissions that were granted to everyone, not all permissions for every user and group. See the .Nm zfs Cm allow command for a description of the .Fl ldugec options. .Bl -tag -width "-r" .It Fl r Recursively remove the permissions from this file system and all descendants. .El .It Xo .Nm zfs .Cm unallow .Op Fl r .Fl s No @ Ns Ar setname .Oo Ar perm Ns | Ns @ Ns Ar setname Ns Oo , Ns Ar perm Ns | Ns @ Ns .Ar setname Oc Ns … Oc .Ar filesystem Ns | Ns Ar volume .Xc Removes permissions from a permission set. If no permissions are specified, then all permissions are removed, thus removing the set entirely. .El . .Sh EXAMPLES .\" These are, respectively, examples 17, 18, 19, 20 from zfs.8 .\" Make sure to update them bidirectionally .Ss Example 1 : No Delegating ZFS Administration Permissions on a ZFS Dataset The following example shows how to set permissions so that user .Ar cindys can create, destroy, mount, and take snapshots on .Ar tank/cindys . The permissions on .Ar tank/cindys are also displayed. .Bd -literal -compact -offset Ds .No # Nm zfs Cm allow Sy cindys create , Ns Sy destroy , Ns Sy mount , Ns Sy snapshot Ar tank/cindys .No # Nm zfs Cm allow Ar tank/cindys ---- Permissions on tank/cindys -------------------------------------- Local+Descendent permissions: user cindys create,destroy,mount,snapshot .Ed .Pp Because the .Ar tank/cindys mount point permission is set to 755 by default, user .Ar cindys will be unable to mount file systems under .Ar tank/cindys . Add an ACE similar to the following syntax to provide mount point access: .Dl # Cm chmod No A+user : Ns Ar cindys Ns :add_subdirectory:allow Ar /tank/cindys . .Ss Example 2 : No Delegating Create Time Permissions on a ZFS Dataset The following example shows how to grant anyone in the group .Ar staff to create file systems in .Ar tank/users . This syntax also allows staff members to destroy their own file systems, but not destroy anyone else's file system. The permissions on .Ar tank/users are also displayed. .Bd -literal -compact -offset Ds .No # Nm zfs Cm allow Ar staff Sy create , Ns Sy mount Ar tank/users .No # Nm zfs Cm allow Fl c Sy destroy Ar tank/users .No # Nm zfs Cm allow Ar tank/users ---- Permissions on tank/users --------------------------------------- Permission sets: destroy Local+Descendent permissions: group staff create,mount .Ed . .Ss Example 3 : No Defining and Granting a Permission Set on a ZFS Dataset The following example shows how to define and grant a permission set on the .Ar tank/users file system. The permissions on .Ar tank/users are also displayed. .Bd -literal -compact -offset Ds .No # Nm zfs Cm allow Fl s No @ Ns Ar pset Sy create , Ns Sy destroy , Ns Sy snapshot , Ns Sy mount Ar tank/users .No # Nm zfs Cm allow staff No @ Ns Ar pset tank/users .No # Nm zfs Cm allow Ar tank/users ---- Permissions on tank/users --------------------------------------- Permission sets: @pset create,destroy,mount,snapshot Local+Descendent permissions: group staff @pset .Ed . .Ss Example 4 : No Delegating Property Permissions on a ZFS Dataset The following example shows to grant the ability to set quotas and reservations on the .Ar users/home file system. The permissions on .Ar users/home are also displayed. .Bd -literal -compact -offset Ds .No # Nm zfs Cm allow Ar cindys Sy quota , Ns Sy reservation Ar users/home .No # Nm zfs Cm allow Ar users/home ---- Permissions on users/home --------------------------------------- Local+Descendent permissions: user cindys quota,reservation cindys% zfs set quota=10G users/home/marks cindys% zfs get quota users/home/marks NAME PROPERTY VALUE SOURCE users/home/marks quota 10G local .Ed . .Ss Example 5 : No Removing ZFS Delegated Permissions on a ZFS Dataset The following example shows how to remove the snapshot permission from the .Ar staff group on the .Sy tank/users file system. The permissions on .Sy tank/users are also displayed. .Bd -literal -compact -offset Ds .No # Nm zfs Cm unallow Ar staff Sy snapshot Ar tank/users .No # Nm zfs Cm allow Ar tank/users ---- Permissions on tank/users --------------------------------------- Permission sets: @pset create,destroy,mount,snapshot Local+Descendent permissions: group staff @pset .Ed ZFS-BOOKMARK(8) - System Manager's Manual # NAME **zfs-bookmark** - create bookmark of ZFS snapshot # SYNOPSIS **zfs** **bookmark** \[**-r**] *snapshot*|*bookmark* *newbookmark* # DESCRIPTION Creates a new bookmark of the given snapshot or bookmark. Bookmarks mark the point in time when the snapshot was created, and can be used as the incremental source for a **zfs** **send**. **-r** > Recursively create bookmarks for the given source snapshot of all descendent > datasets. > The source must be a snapshot, and a bookmark is created only for the > descendants that have a snapshot of that name; descendants without it are > skipped. When creating a bookmark from an existing redaction bookmark, the resulting bookmark is *not* a redaction bookmark. This feature must be enabled to be used. See zpool-features(7) for details on ZFS feature flags and the **bookmarks** feature. # EXAMPLES ## Example 1: Creating a bookmark The following example creates a bookmark to a snapshot. This bookmark can then be used instead of a snapshot in send streams. # **zfs** **bookmark** *rpool*@*snapshot rpool*#*bookmark* # SEE ALSO zfs-destroy(8), zfs-send(8), zfs-snapshot(8) Debian - July 11, 2022 ZFS-LOAD-KEY(8) - System Manager's Manual # NAME **zfs-load-key** - load, unload, or change encryption key of ZFS dataset # SYNOPSIS **zfs** **load-key** \[**-nr**] \[**-L** *keylocation*] **-a**|*filesystem* **zfs** **unload-key** \[**-r**] **-a**|*filesystem* **zfs** **change-key** \[**-l**] \[**-o** *keylocation*=*value*] \[**-o** *keyformat*=*value*] \[**-o** *pbkdf2iters*=*value*] \[**-o** *user:prop*=*value*] *filesystem* **zfs** **change-key** **-i** \[**-l**] *filesystem* # DESCRIPTION **zfs** **load-key** \[**-nr**] \[**-L** *keylocation*] **-a**|*filesystem* > Load the key for > *filesystem*, > allowing it and all children that inherit the > **keylocation** > property to be accessed. > The key will be expected in the format specified by the > **keyformat** > and location specified by the > **keylocation** > property. > Note that if the > **keylocation** > is set to > **prompt** > the terminal will interactively wait for the key to be entered. > Loading a key will not automatically mount the dataset. > If that functionality is desired, > **zfs** **mount** **-l** > will ask for the key and mount the dataset > (see > zfs-mount(8)) > . > For automated key management during user login, > pam\_zfs\_key(8) > can load keys and mount encrypted home directories on systems with PAM support. > Once the key is loaded the > **keystatus** > property will become > **available**. > **-r** > > Recursively loads the keys for the specified filesystem and all descendent > > encryption roots. > **-a** > > Loads the keys for all encryption roots in all imported pools. > **-n** > > Do a dry-run > > ("No-op") > > **load-key**. > > This will cause > > **zfs** > > to simply check that the provided key is correct. > > This command may be run even if the key is already loaded. > **-L** *keylocation* > > Use > > *keylocation* > > instead of the > > **keylocation** > > property. > > This will not change the value of the property on the dataset. > > Note that if used with either > > **-r** > > or > > **-a**, > > *keylocation* > > may only be given as > > **prompt**. **zfs** **unload-key** \[**-r**] **-a**|*filesystem* > Unloads a key from ZFS, removing the ability to access the dataset and all of > its children that inherit the > **keylocation** > property. > This requires that the dataset is not currently open or mounted. > Once the key is unloaded the > **keystatus** > property will become > **unavailable**. > **-r** > > Recursively unloads the keys for the specified filesystem and all descendent > > encryption roots. > **-a** > > Unloads the keys for all encryption roots in all imported pools. **zfs** **change-key** \[**-l**] \[**-o** *keylocation*=*value*] \[**-o** *keyformat*=*value*] \[**-o** *pbkdf2iters*=*value*] \[**-o** *user:prop*=*value*] *filesystem* **zfs** **change-key** **-i** \[**-l**] *filesystem* > Changes the user's key (e.g. a passphrase) used to access a dataset. > This command requires that the existing key for the dataset is already loaded. > This command may also be used to change the > **keylocation**, > **keyformat**, > and > **pbkdf2iters** > properties as needed, as well as set user properties. > If the dataset was not previously an encryption root it will become one. > Alternatively, the > **-i** > flag may be provided to cause an encryption root to inherit the parent's key > instead. > If the user's key is compromised, > **zfs** **change-key** > does not necessarily protect existing or newly-written data from attack. > Newly-written data will continue to be encrypted with the same master key as > the existing data. > The master key is compromised if an attacker obtains a > user key and the corresponding wrapped master key. > Currently, > **zfs** **change-key** > does not overwrite the previous wrapped master key on disk, so it is > accessible via forensic analysis for an indeterminate length of time. > In the event of a master key compromise, ideally the drives should be securely > erased to remove all the old data (which is readable using the compromised > master key), a new pool created, and the data copied back. > This can be approximated in place by creating new datasets, copying the data > (e.g. using **zfs** **send** | **zfs** **recv**), > and then clearing the free space with > **zpool** **trim** **--secure** > if supported by your hardware, otherwise > **zpool** **initialize**. > **-l** > > Ensures the key is loaded before attempting to change the key. > > This is effectively equivalent to running > > **zfs** **load-key** *filesystem*; **zfs** **change-key** *filesystem* > **-o** *property*=*value* > > Allows the user to set encryption key properties > > (**keyformat**, **keylocation**, and **pbkdf2iters**) > > and user properties while changing the key. > > This is the only way to alter > > **keyformat** > > and > > **pbkdf2iters** > > after the dataset has been created. > **-i** > > Indicates that zfs should make > > *filesystem* > > inherit the key of its parent. > > Note that this command can only be run on an encryption root > > that has an encrypted parent. ## Encryption Enabling the **encryption** feature allows for the creation of encrypted filesystems and volumes. ZFS will encrypt file and volume data, file attributes, ACLs, permission bits, directory listings, FUID mappings, and **userused**/**groupused** data. ZFS will not encrypt metadata related to the pool structure, including dataset and snapshot names, dataset hierarchy, properties, file size, file holes, and deduplication tables (though the deduplicated data itself is encrypted). Key rotation is managed by ZFS. Changing the user's key (e.g. a passphrase) does not require re-encrypting the entire dataset. Datasets can be scrubbed, resilvered, renamed, and deleted without the encryption keys being loaded (see the **load-key** subcommand for more info on key loading). Creating an encrypted dataset requires specifying the **encryption** and **keyformat** properties at creation time, along with an optional **keylocation** and **pbkdf2iters**. After entering an encryption key, the created dataset will become an encryption root. Any descendant datasets will inherit their encryption key from the encryption root by default, meaning that loading, unloading, or changing the key for the encryption root will implicitly do the same for all inheriting datasets. If this inheritance is not desired, simply supply a **keyformat** when creating the child dataset or use **zfs** **change-key** to break an existing relationship, creating a new encryption root on the child. Note that the child's **keyformat** may match that of the parent while still creating a new encryption root, and that changing the **encryption** property alone does not create a new encryption root; this would simply use a different cipher suite with the same key as its encryption root. The one exception is that clones will always use their origin's encryption key. As a result of this exception, some encryption-related properties (namely **keystatus**, **keyformat**, **keylocation**, and **pbkdf2iters**) do not inherit like other ZFS properties and instead use the value determined by their encryption root. Encryption root inheritance can be tracked via the read-only **encryptionroot** property. Encryption changes the behavior of a few ZFS operations. Encryption is applied after compression so compression ratios are preserved. Normally checksums in ZFS are 256 bits long, but for encrypted data the checksum is 128 bits of the user-chosen checksum and 128 bits of MAC from the encryption suite, which provides additional protection against maliciously altered data. Deduplication is still possible with encryption enabled but for security, datasets will only deduplicate against themselves, their snapshots, and their clones. There are a few limitations on encrypted datasets. Encrypted data cannot be embedded via the **embedded\_data** feature. Encrypted datasets may not have **copies**=*3* since the implementation stores some encryption metadata where the third copy would normally be. Since compression is applied before encryption, datasets may be vulnerable to a CRIME-like attack if applications accessing the data allow for it. Deduplication with encryption will leak information about which blocks are equivalent in a dataset and will incur an extra CPU cost for each block written. # SEE ALSO zfsprops(7), pam\_zfs\_key(8), zfs-create(8), zfs-set(8) Debian - January 30, 2026 ZFS-CLONE(8) - System Manager's Manual # NAME **zfs-clone** - clone snapshot of ZFS dataset # SYNOPSIS **zfs** **clone** \[**-pu**] \[**-o** *property*=*value*]… *snapshot* *filesystem*|*volume* # DESCRIPTION See the *Clones* section of zfsconcepts(7) for details. The target dataset can be located anywhere in the ZFS hierarchy, and is created as the same type as the original. **-o** *property*=*value* > Sets the specified property; see > **zfs** **create** > for details. **-p** > Creates all the non-existing parent datasets. > Datasets created in this manner are automatically mounted according to the > **mountpoint** > property inherited from their parent. > If the > **-p** > option is specified multiple times, parent datasets are created with > **canmount**=*off*. > If the target filesystem or volume already exists, the operation completes > successfully. **-u** > Do not mount the newly created file system. # EXAMPLES ## Example 1: Creating a ZFS Clone The following command creates a writable file system whose initial contents are the same as *pool/home/bob@yesterday*. # **zfs** **clone** *pool/home/bob@yesterday pool/clone* ## Example 2: Promoting a ZFS Clone The following commands illustrate how to test out changes to a file system, and then replace the original file system with the changed one, using clones, clone promotion, and renaming: # **zfs** **create** *pool/project/production* populate /pool/project/production with data # **zfs** **snapshot** *pool/project/production*@*today* # **zfs** **clone** *pool/project/production@today pool/project/beta* make changes to /pool/project/beta and test them # **zfs** **promote** *pool/project/beta* # **zfs** **rename** *pool/project/production pool/project/legacy* # **zfs** **rename** *pool/project/beta pool/project/production* once the legacy version is no longer needed, it can be destroyed # **zfs** **destroy** *pool/project/legacy* # SEE ALSO zfs-promote(8), zfs-snapshot(8) Debian - July 11, 2022 ZFS-CREATE(8) - System Manager's Manual # NAME **zfs-create** - create ZFS dataset # SYNOPSIS **zfs** **create** \[**-Pnpuv**] \[**-o** *property*=*value*]… *filesystem* **zfs** **create** \[**-ps**] \[**-b** *blocksize*] \[**-o** *property*=*value*]… **-V** *size* *volume* # DESCRIPTION **zfs** **create** \[**-Pnpuv**] \[**-o** *property*=*value*]… *filesystem* > Creates a new ZFS file system. > The file system is automatically mounted according to the > **mountpoint** > property inherited from the parent, unless the > **-u** > option is used. > **-o** *property*=*value* > > Sets the specified property as if the command > > **zfs** **set** *property*=*value* > > was invoked at the same time the dataset was created. > > Any editable ZFS property can also be set at creation time. > > Multiple > > **-o** > > options can be specified. > > An error results if the same property is specified in multiple > > **-o** > > options. > **-p** > > Creates all the non-existing parent datasets. > > Datasets created in this manner are automatically mounted according to the > > **mountpoint** > > property inherited from their parent. > > Any property specified on the command line using the > > **-o** > > option is ignored. > > If the > > **-p** > > option is specified multiple times, parent datasets are created with > > **canmount**=*off*. > > If the target filesystem already exists, the operation completes successfully. > **-n** > > Do a dry-run > > ("No-op") > > creation. > > No datasets will be created. > > This is useful in conjunction with the > > **-v** > > or > > **-P** > > flags to validate properties that are passed via > > **-o** > > options and those implied by other options. > > The actual dataset creation can still fail due to insufficient privileges or > > available capacity. > **-P** > > Print machine-parsable verbose information about the created dataset. > > Each line of output contains a key and one or two values, all separated by tabs. > > The > > **create\_ancestors** > > and > > **create** > > keys have > > *filesystem* > > as their only value. > > The > > **create\_ancestors** > > key only appears if the > > **-p** > > option is used. > > The > > **property** > > key has two values, a property name and that property's value. > > The > > **property** > > key may appear zero or more times, once for each property that will be set local > > to > > *filesystem* > > due to the use of the > > **-o** > > option. > **-u** > > Do not mount the newly created file system. > **-v** > > Print verbose information about the created dataset. **zfs** **create** \[**-ps**] \[**-b** *blocksize*] \[**-o** *property*=*value*]… **-V** *size* *volume* > Creates a volume of the given size. > The volume is exported as a block device in > */dev/zvol/path*, > where > *path* > is the name of the volume in the ZFS namespace. > The size represents the logical size as exported by the device. > By default, a reservation of equal size is created. > *size* > is automatically rounded up to the nearest multiple of the > **blocksize**. > **-b** *blocksize* > > Equivalent to > > **-o** **volblocksize**=*blocksize*. > > If this option is specified in conjunction with > > **-o** **volblocksize**, > > the resulting behavior is undefined. > **-o** *property*=*value* > > Sets the specified property as if the > > **zfs** **set** *property*=*value* > > command was invoked at the same time the dataset was created. > > Any editable ZFS property can also be set at creation time. > > Multiple > > **-o** > > options can be specified. > > An error results if the same property is specified in multiple > > **-o** > > options. > **-p** > > Creates all the non-existing parent datasets. > > Datasets created in this manner are automatically mounted according to the > > **mountpoint** > > property inherited from their parent. > > Any property specified on the command line using the > > **-o** > > option is ignored. > > If the > > **-p** > > option is specified multiple times, parent datasets are created with > > **canmount**=*off*. > > If the target filesystem already exists, the operation completes successfully. > **-s** > > Creates a sparse volume with no reservation. > > See > > **volsize** > > in the > > *Native Properties* > > section of > > zfsprops(7) > > for more information about sparse volumes. > **-n** > > Do a dry-run > > ("No-op") > > creation. > > No datasets will be created. > > This is useful in conjunction with the > > **-v** > > or > > **-P** > > flags to validate properties that are passed via > > **-o** > > options and those implied by other options. > > The actual dataset creation can still fail due to insufficient privileges or > > available capacity. > **-P** > > Print machine-parsable verbose information about the created dataset. > > Each line of output contains a key and one or two values, all separated by tabs. > > The > > **create\_ancestors** > > and > > **create** > > keys have > > *volume* > > as their only value. > > The > > **create\_ancestors** > > key only appears if the > > **-p** > > option is used. > > The > > **property** > > key has two values, a property name that property's value. > > The > > **property** > > key may appear zero or more times, once for each property that will be set local > > to > > *volume* > > due to the use of the > > **-b** > > or > > **-o** > > options, as well as > > **refreservation** > > if the volume is not sparse. > **-v** > > Print verbose information about the created dataset. ## ZFS for Swap Swapping to a ZFS volume is prone to deadlock and not recommended. See OpenZFS FAQ. Swapping to a file on a ZFS filesystem is not supported. # EXAMPLES ## Example 1: Creating a ZFS File System Hierarchy The following commands create a file system named *pool/home* and a file system named *pool/home/bob*. The mount point */export/home* is set for the parent file system, and is automatically inherited by the child file system. # **zfs** **create** *pool/home* # **zfs** **set** **mountpoint**=*/export/home pool/home* # **zfs** **create** *pool/home/bob* ## Example 2: Promoting a ZFS Clone The following commands illustrate how to test out changes to a file system, and then replace the original file system with the changed one, using clones, clone promotion, and renaming: # **zfs** **create** *pool/project/production* populate /pool/project/production with data # **zfs** **snapshot** *pool/project/production*@*today* # **zfs** **clone** *pool/project/production@today pool/project/beta* make changes to /pool/project/beta and test them # **zfs** **promote** *pool/project/beta* # **zfs** **rename** *pool/project/production pool/project/legacy* # **zfs** **rename** *pool/project/beta pool/project/production* once the legacy version is no longer needed, it can be destroyed # **zfs** **destroy** *pool/project/legacy* # SEE ALSO zfs-destroy(8), zfs-list(8), zpool-create(8) Debian - June 2, 2023 ZFS-DESTROY(8) - System Manager's Manual # NAME **zfs-destroy** - destroy ZFS dataset, snapshots, or bookmark # SYNOPSIS **zfs** **destroy** \[**-Rfnprv**] *filesystem*|*volume* **zfs** **destroy** \[**-Rdnprv**] *filesystem*|*volume*@*snap*\[%*snap*\[,*snap*\[%*snap*]]]… **zfs** **destroy** *filesystem*|*volume*#*bookmark* # DESCRIPTION **zfs** **destroy** \[**-Rfnprv**] *filesystem*|*volume* > Destroys the given dataset. > By default, the command unshares any file systems that are currently shared, > unmounts any file systems that are currently mounted, and refuses to destroy a > dataset that has active dependents > (children or clones). > **-R** > > Recursively destroy all dependents, including cloned file systems outside the > > target hierarchy. > **-f** > > Forcibly unmount file systems. > > This option has no effect on non-file systems or unmounted file systems. > **-n** > > Do a dry-run > > ("No-op") > > deletion. > > No data will be deleted. > > This is useful in conjunction with the > > **-v** > > or > > **-p** > > flags to determine what data would be deleted. > **-p** > > Print machine-parsable verbose information about the deleted data. > **-r** > > Recursively destroy all children. > **-v** > > Print verbose information about the deleted data. > Extreme care should be taken when applying either the > **-r** > or the > **-R** > options, as they can destroy large portions of a pool and cause unexpected > behavior for mounted file systems in use. **zfs** **destroy** \[**-Rdnprv**] *filesystem*|*volume*@*snap*\[%*snap*\[,*snap*\[%*snap*]]]… > Attempts to destroy the given snapshot(s). > This will fail if any clones of the snapshot exist or if the snapshot is held. > In this case, by default, > **zfs** **destroy** > will have no effect and exit in error. > If the > **-d** > option is applied, the command will instead mark the given snapshot for > automatic destruction as soon as it becomes eligible. > While marked for destruction, a snapshot remains visible, and the user may > create new clones from it and place new holds on it. > The read-only snapshot properties > **defer\_destroy** > and > **userrefs** > are used by > **zfs** **destroy** > to determine eligibility and marked status. > An inclusive range of snapshots may be specified by separating the first and > last snapshots with a percent sign. > The first and/or last snapshots may be left blank, in which case the > filesystem's oldest or newest snapshot will be implied. > Multiple snapshots > (or ranges of snapshots) > of the same filesystem or volume may be specified in a comma-separated list of > snapshots. > Only the snapshot's short name > (the part after the > **@**) > should be specified when using a range or comma-separated list to identify > multiple snapshots. > **-R** > > Recursively destroy all clones of these snapshots, including the clones, > > snapshots, and children. > > If this flag is specified, the > > **-d** > > flag will have no effect. > **-d** > > Rather than returning error if the given snapshot is ineligible for immediate > > destruction, mark it for deferred, automatic destruction once it becomes > > eligible. > **-n** > > Do a dry-run > > ("No-op") > > deletion. > > No data will be deleted. > > This is useful in conjunction with the > > **-p** > > or > > **-v** > > flags to determine what data would be deleted. > **-p** > > Print machine-parsable verbose information about the deleted data. > **-r** > > Destroy > > (or mark for deferred deletion) > > all snapshots with this name in descendent file systems. > **-v** > > Print verbose information about the deleted data. > Extreme care should be taken when applying either the > **-r** > or the > **-R** > options, as they can destroy large portions of a pool and cause unexpected > behavior for mounted file systems in use. **zfs** **destroy** *filesystem*|*volume*#*bookmark* > The given bookmark is destroyed. # EXAMPLES ## Example 1: Creating and Destroying Multiple Snapshots The following command creates snapshots named *yesterday* of *pool/home* and all of its descendent file systems. Each snapshot is mounted on demand in the *.zfs/snapshot* directory at the root of its file system. The second command destroys the newly created snapshots. # **zfs** **snapshot** **-r** *pool/home*@*yesterday* # **zfs** **destroy** **-r** *pool/home*@*yesterday* ## Example 2: Promoting a ZFS Clone The following commands illustrate how to test out changes to a file system, and then replace the original file system with the changed one, using clones, clone promotion, and renaming: # **zfs** **create** *pool/project/production* populate /pool/project/production with data # **zfs** **snapshot** *pool/project/production*@*today* # **zfs** **clone** *pool/project/production@today pool/project/beta* make changes to /pool/project/beta and test them # **zfs** **promote** *pool/project/beta* # **zfs** **rename** *pool/project/production pool/project/legacy* # **zfs** **rename** *pool/project/beta pool/project/production* once the legacy version is no longer needed, it can be destroyed # **zfs** **destroy** *pool/project/legacy* ## Example 3: Performing a Rolling Snapshot The following example shows how to maintain a history of snapshots with a consistent naming scheme. To keep a week's worth of snapshots, the user destroys the oldest snapshot, renames the remaining snapshots, and then creates a new snapshot, as follows: # **zfs** **destroy** **-r** *pool/users@7daysago* # **zfs** **rename** **-r** *pool/users@6daysago* @*7daysago* # **zfs** **rename** **-r** *pool/users@5daysago* @*6daysago* # **zfs** **rename** **-r** *pool/users@4daysago* @*5daysago* # **zfs** **rename** **-r** *pool/users@3daysago* @*4daysago* # **zfs** **rename** **-r** *pool/users@2daysago* @*3daysago* # **zfs** **rename** **-r** *pool/users@yesterday* @*2daysago* # **zfs** **rename** **-r** *pool/users@today* @*yesterday* # **zfs** **snapshot** **-r** *pool/users*@*today* # SEE ALSO zfs-create(8), zfs-hold(8), zfsprops(8) Debian - February 5, 2025 ZFS-DIFF(8) - System Manager's Manual # NAME **zfs-diff** - show difference between ZFS snapshots # SYNOPSIS **zfs** **diff** \[**-FHth**] *snapshot* *snapshot*|*filesystem* # DESCRIPTION Display the difference between a snapshot of a given filesystem and another snapshot of that filesystem from a later time or the current contents of the filesystem. The first column is a character indicating the type of change, the other columns indicate pathname, new pathname (in case of rename), change in link count, and optionally file type and/or change time. The types of change are: **-** > The path has been removed **+** > The path has been created **M** > The path has been modified **R** > The path has been renamed **-F** > Display an indication of the type of file, in a manner similar to the > **-F** > option of > ls(1). > **B** > > Block device > **C** > > Character device > **/** > > Directory > **>** > > Door > **|** > > Named pipe > **@** > > Symbolic link > **P** > > Event port > **=** > > Socket > **F** > > Regular file **-H** > Give more parsable tab-separated output, without header lines and without > arrows. **-t** > Display the path's inode change time as the first column of output. **-h** > Do not > **\0**‌*ooo*-escape > non-ASCII paths. # EXAMPLES ## Example 1: Showing the differences between a snapshot and a ZFS Dataset The following example shows how to see what has changed between a prior snapshot of a ZFS dataset and its current state. The **-F** option is used to indicate type information for the files affected. # **zfs** **diff** **-F** *tank/test@before tank/test* M / /tank/test/ M F /tank/test/linked (+1) R F /tank/test/oldname -> /tank/test/newname - F /tank/test/deleted + F /tank/test/created M F /tank/test/modified # SEE ALSO zfs-snapshot(8) Debian - July 11, 2022 ZFS-SET(8) - System Manager's Manual # NAME **zfs-set** - set properties on ZFS datasets # SYNOPSIS **zfs** **set** \[**-u**] *property*=*value* \[*property*=*value*]… *filesystem*|*volume*|*snapshot*… **zfs** **get** \[**-r**|**-d** *depth*] \[**-Hp**] \[**-j** \[*--json-int*]] \[**-o** *field*\[,*field*]…] \[**-s** *source*\[,*source*]…] \[**-t** *type*\[,*type*]…] **all**|*property*\[,*property*]… \[*filesystem*|*volume*|*snapshot*|*bookmark*]… **zfs** **inherit** \[**-rS**] *property* *filesystem*|*volume*|*snapshot*… # DESCRIPTION **zfs** **set** \[**-u**] *property*=*value* \[*property*=*value*]… *filesystem*|*volume*|*snapshot*… > Only some properties can be edited. > See > zfsprops(7) > for more information on what properties can be set and acceptable > values. > Numeric values can be specified as exact values, or in a human-readable form > with a suffix of > **B**, **K**, **M**, **G**, **T**, **P**, **E**, **Z** > (for bytes, kilobytes, megabytes, gigabytes, terabytes, petabytes, exabytes, > or zettabytes, respectively) > . > User properties can be set on snapshots. > For more information, see the > *User Properties* > section of > zfsprops(7). > **-u** > > Update mountpoint, sharenfs, sharesmb property but do not mount or share the > > dataset. **zfs** **get** \[**-r**|**-d** *depth*] \[**-Hp**] \[**-j** \[*--json-int*]] \[**-o** *field*\[,*field*]…] \[**-s** *source*\[,*source*]…] \[**-t** *type*\[,*type*]…] **all**|*property*\[,*property*]… \[*filesystem*|*volume*|*snapshot*|*bookmark*]… > Displays properties for the given datasets. > If no datasets are specified, then the command displays properties for all > datasets on the system. > For each property, the following columns are displayed: > **name** > > Dataset name > **property** > > Property name > **value** > > Property value > **source** > > Property source > > **local**, **default**, **inherited**, **temporary**, **received**, or **-** (none). > **received** > > The received value of the property, if any. > > This column is not displayed by default. > The > **name**, **property**, **value**, > and > **source** > columns are displayed by default, though this can be > controlled by using the > **-o** > option. > This command takes a comma-separated list of properties as described in the > *Native Properties* > and > *User Properties* > sections of > zfsprops(7). > The value > **all** > can be used to display all properties that apply to the given dataset's type > (**filesystem**, **volume**, **snapshot**, or **bookmark**). > **-j**, **--json** \[*--json-int*] > > Display the output in JSON format. > > Specify > > **--json-int** > > to display numbers in integer format instead of strings for JSON output. > **-H** > > Display output in a form more easily parsed by scripts. > > Any headers are omitted, and fields are explicitly separated by a single tab > > instead of an arbitrary amount of space. > **-d** *depth* > > Recursively display any children of the dataset, limiting the recursion to > > *depth*. > > A depth of > > **1** > > will display only the dataset and its direct children. > **-o** *field* > > A comma-separated list of columns to display. > > Supported fields are > > **name**, **property**, **value**, **received**, **source**, > > or > > **all** > > to select all five columns. > > The default value is > > **name**,**property**,**value**,**source**. > **-p** > > Display numbers in parsable > > (exact) > > values. > **-r** > > Recursively display properties for any children. > **-s** *source* > > A comma-separated list of sources to display. > > Those properties coming from a source other than those in this list are ignored. > > Each source must be one of the following: > > **local**, **default**, **inherited**, **temporary**, **received**, or **none**. > > The default value is all sources. > **-t** *type* > > A comma-separated list of types to display, where > > *type* > > is one of > > **filesystem**, **snapshot**, **volume**, **bookmark**, or **all**. > > **fs**, > > **snap**, > > or > > **vol** > > can be used as aliases for > > **filesystem**, > > **snapshot**, > > or > > **volume**. **zfs** **inherit** \[**-rS**] *property* *filesystem*|*volume*|*snapshot*… > Clears the specified property, causing it to be inherited from an ancestor, > restored to default if no ancestor has the property set, or with the > **-S** > option reverted to the received value if one exists. > See > zfsprops(7) > for a listing of default values, and details on which properties can be > inherited. > **-r** > > Recursively inherit the given property for all children. > **-S** > > Revert the property to the received value, if one exists; > > otherwise, for non-inheritable properties, to the default; > > otherwise, operate as if the > > **-S** > > option was not specified. # EXAMPLES ## Example 1: Creating a ZFS File System Hierarchy The following commands create a file system named *pool/home* and a file system named *pool/home/bob*. The mount point */export/home* is set for the parent file system, and is automatically inherited by the child file system. # **zfs** **create** *pool/home* # **zfs** **set** **mountpoint**=*/export/home pool/home* # **zfs** **create** *pool/home/bob* ## Example 2: Disabling and Enabling File System Compression The following command disables the **compression** property for all file systems under *pool/home*. The next command explicitly enables **compression** for *pool/home/anne*. # **zfs** **set** **compression**=**off** *pool/home* # **zfs** **set** **compression**=**on** *pool/home/anne* ## Example 3: Setting a Quota on a ZFS File System The following command sets a quota of 50 Gbytes for *pool/home/bob*: # **zfs** **set** **quota**=*50G pool/home/bob* ## Example 4: Listing ZFS Properties The following command lists all properties for *pool/home/bob*: # **zfs** **get** **all** *pool/home/bob* NAME PROPERTY VALUE SOURCE pool/home/bob type filesystem - pool/home/bob creation Tue Jul 21 15:53 2009 - pool/home/bob used 21K - pool/home/bob available 20.0G - pool/home/bob referenced 21K - pool/home/bob compressratio 1.00x - pool/home/bob mounted yes - pool/home/bob quota 20G local pool/home/bob reservation none default pool/home/bob recordsize 128K default pool/home/bob mountpoint /pool/home/bob default pool/home/bob sharenfs off default pool/home/bob checksum on default pool/home/bob compression on local pool/home/bob atime on default pool/home/bob devices on default pool/home/bob exec on default pool/home/bob setuid on default pool/home/bob readonly off default pool/home/bob zoned off default pool/home/bob snapdir hidden default pool/home/bob acltype off default pool/home/bob aclmode discard default pool/home/bob aclinherit restricted default pool/home/bob canmount on default pool/home/bob xattr on default pool/home/bob copies 1 default pool/home/bob version 4 - pool/home/bob utf8only off - pool/home/bob normalization none - pool/home/bob casesensitivity sensitive - pool/home/bob vscan off default pool/home/bob nbmand off default pool/home/bob sharesmb off default pool/home/bob refquota none default pool/home/bob refreservation none default pool/home/bob primarycache all default pool/home/bob secondarycache all default pool/home/bob usedbysnapshots 0 - pool/home/bob usedbydataset 21K - pool/home/bob usedbychildren 0 - pool/home/bob usedbyrefreservation 0 - The following command gets a single property value: # **zfs** **get** **-H** **-o** **value compression** *pool/home/bob* on The following command gets a single property value recursively in JSON format: # **zfs** **get** **-j** **-r** **mountpoint** *pool/home* | **jq** { "output_version": { "command": "zfs get", "vers_major": 0, "vers_minor": 1 }, "datasets": { "pool/home": { "name": "pool/home", "type": "FILESYSTEM", "pool": "pool", "createtxg": "10", "properties": { "mountpoint": { "value": "/pool/home", "source": { "type": "DEFAULT", "data": "-" } } } }, "pool/home/bob": { "name": "pool/home/bob", "type": "FILESYSTEM", "pool": "pool", "createtxg": "1176", "properties": { "mountpoint": { "value": "/pool/home/bob", "source": { "type": "DEFAULT", "data": "-" } } } } } } The following command lists all properties with local settings for *pool/home/bob*: # **zfs** **get** **-r** **-s** **local** **-o** **name**,**property**,**value all** *pool/home/bob* NAME PROPERTY VALUE pool/home/bob quota 20G pool/home/bob compression on ## Example 5: Inheriting ZFS Properties The following command causes *pool/home/bob* and *pool/home/anne* to inherit the **checksum** property from their parent. # **zfs** **inherit** **checksum** *pool/home/bob pool/home/anne* ## Example 6: Setting User Properties The following example sets the user-defined *com.example*:*department* property for a dataset: # **zfs** **set** *com.example*:*department*=*12345 tank/accounting* ## Example 7: Setting sharenfs Property Options on a ZFS File System The following commands show how to set **sharenfs** property options to enable read-write access for a set of IP addresses and to enable root access for system "neo" on the *tank/home* file system: # **zfs** **set** **sharenfs**='*rw*=@123.123.0.0/16:[::1],root=*neo*' tank/home If you are using DNS for host name resolution, specify the fully-qualified hostname. # SEE ALSO zfsprops(7), zfs-list(8) Debian - March 15, 2026 ZFS-USERSPACE(8) - System Manager's Manual # NAME **zfs-userspace** - display space and quotas of ZFS dataset # SYNOPSIS **zfs** **userspace** \[**-Hinp**] \[**-o** *field*\[,*field*]…] \[**-s** *field*]… \[**-S** *field*]… \[**-t** *type*\[,*type*]…] *filesystem*|*snapshot*|*path* **zfs** **groupspace** \[**-Hinp**] \[**-o** *field*\[,*field*]…] \[**-s** *field*]… \[**-S** *field*]… \[**-t** *type*\[,*type*]…] *filesystem*|*snapshot*|*path* **zfs** **projectspace** \[**-Hp**] \[**-o** *field*\[,*field*]…] \[**-s** *field*]… \[**-S** *field*]… *filesystem*|*snapshot*|*path* # DESCRIPTION **zfs** **userspace** \[**-Hinp**] \[**-o** *field*\[,*field*]…] \[**-s** *field*]… \[**-S** *field*]… \[**-t** *type*\[,*type*]…] *filesystem*|*snapshot*|*path* > Displays space consumed by, and quotas on, each user in the specified > filesystem, > snapshot, or path. > If a path is given, the filesystem that contains that path will be used. > This corresponds to the > **userused@**‌*user*, > **userobjused@**‌*user*, > **userquota@**‌*user*, > and > **userobjquota@**‌*user* > properties. > **-H** > > Do not print headers, use tab-delimited output. > **-S** *field* > > Sort by this field in reverse order. > > See > > **-s**. > **-i** > > Translate SID to POSIX ID. > > The POSIX ID may be ephemeral if no mapping exists. > > Normal POSIX interfaces > > (like stat(2), **ls** **-l**) > > perform this translation, so the > > **-i** > > option allows the output from > > **zfs** **userspace** > > to be compared directly with those utilities. > > However, > > **-i** > > may lead to confusion if some files were created by an SMB user before a > > SMB-to-POSIX name mapping was established. > > In such a case, some files will be owned by the SMB entity and some by the POSIX > > entity. > > However, the > > **-i** > > option will report that the POSIX entity has the total usage and quota for both. > **-n** > > Print numeric ID instead of user/group name. > **-o** *field*\[,*field*]… > > Display only the specified fields from the following set: > > **type**, > > **name**, > > **used**, > > **quota**. > > The default is to display all fields. > **-p** > > Use exact > > (parsable) > > numeric output. > **-s** *field* > > Sort output by this field. > > The > > **-s** > > and > > **-S** > > flags may be specified multiple times to sort first by one field, then by > > another. > > The default is > > **-s** **type** **-s** **name**. > **-t** *type*\[,*type*]… > > Print only the specified types from the following set: > > **all**, > > **posixuser**, > > **smbuser**, > > **posixgroup**, > > **smbgroup**. > > The default is > > **-t** **posixuser**,**smbuser**. > > The default can be changed to include group types. **zfs** **groupspace** \[**-Hinp**] \[**-o** *field*\[,*field*]…] \[**-s** *field*]… \[**-S** *field*]… \[**-t** *type*\[,*type*]…] *filesystem*|*snapshot* > Displays space consumed by, and quotas on, each group in the specified > filesystem or snapshot. > This subcommand is identical to > **userspace**, > except that the default types to display are > **-t** **posixgroup**,**smbgroup**. **zfs** **projectspace** \[**-Hp**] \[**-o** *field*\[,*field*]…] \[**-s** *field*]… \[**-S** *field*]… *filesystem*|*snapshot*|*path* > Displays space consumed by, and quotas on, each project in the specified > filesystem or snapshot. > This subcommand is identical to > **userspace**, > except that the project identifier is a numeral, not a name. > So need neither the option > **-i** > for SID to POSIX ID nor > **-n** > for numeric ID, nor > **-t** > for types. # SEE ALSO zfsprops(7), zfs-set(8) Debian - July 11, 2022 ZFS-HOLD(8) - System Manager's Manual # NAME **zfs-hold** - hold ZFS snapshots to prevent their removal # SYNOPSIS **zfs** **hold** \[**-r**] *tag* *snapshot*… **zfs** **holds** \[**-rHp**] *snapshot*… **zfs** **release** \[**-r**] *tag* *snapshot*… # DESCRIPTION **zfs** **hold** \[**-r**] *tag* *snapshot*… > Adds a single reference, named with the > *tag* > argument, to the specified snapshots. > Each snapshot has its own tag namespace, and tags must be unique within that > space. > If a hold exists on a snapshot, attempts to destroy that snapshot by using the > **zfs** **destroy** > command return > **EBUSY**. > **-r** > > Specifies that a hold with the given tag is applied recursively to the snapshots > > of all descendent file systems. **zfs** **holds** \[**-rHp**] *snapshot*… > Lists all existing user references for the given snapshot or snapshots. > **-r** > > Lists the holds that are set on the named descendent snapshots, in addition to > > listing the holds on the named snapshot. > **-H** > > Do not print headers, use tab-delimited output. > **-p** > > Prints holds timestamps as Unix epoch timestamps. **zfs** **release** \[**-r**] *tag* *snapshot*… > Removes a single reference, named with the > *tag* > argument, from the specified snapshot or snapshots. > The tag must already exist for each snapshot. > If a hold exists on a snapshot, attempts to destroy that snapshot by using the > **zfs** **destroy** > command return > **EBUSY**. > **-r** > > Recursively releases a hold with the given tag on the snapshots of all > > descendent file systems. # SEE ALSO zfs-destroy(8) Debian - November 8, 2022 ZFS-SET(8) - System Manager's Manual # NAME **zfs-set** - set properties on ZFS datasets # SYNOPSIS **zfs** **set** \[**-u**] *property*=*value* \[*property*=*value*]… *filesystem*|*volume*|*snapshot*… **zfs** **get** \[**-r**|**-d** *depth*] \[**-Hp**] \[**-j** \[*--json-int*]] \[**-o** *field*\[,*field*]…] \[**-s** *source*\[,*source*]…] \[**-t** *type*\[,*type*]…] **all**|*property*\[,*property*]… \[*filesystem*|*volume*|*snapshot*|*bookmark*]… **zfs** **inherit** \[**-rS**] *property* *filesystem*|*volume*|*snapshot*… # DESCRIPTION **zfs** **set** \[**-u**] *property*=*value* \[*property*=*value*]… *filesystem*|*volume*|*snapshot*… > Only some properties can be edited. > See > zfsprops(7) > for more information on what properties can be set and acceptable > values. > Numeric values can be specified as exact values, or in a human-readable form > with a suffix of > **B**, **K**, **M**, **G**, **T**, **P**, **E**, **Z** > (for bytes, kilobytes, megabytes, gigabytes, terabytes, petabytes, exabytes, > or zettabytes, respectively) > . > User properties can be set on snapshots. > For more information, see the > *User Properties* > section of > zfsprops(7). > **-u** > > Update mountpoint, sharenfs, sharesmb property but do not mount or share the > > dataset. **zfs** **get** \[**-r**|**-d** *depth*] \[**-Hp**] \[**-j** \[*--json-int*]] \[**-o** *field*\[,*field*]…] \[**-s** *source*\[,*source*]…] \[**-t** *type*\[,*type*]…] **all**|*property*\[,*property*]… \[*filesystem*|*volume*|*snapshot*|*bookmark*]… > Displays properties for the given datasets. > If no datasets are specified, then the command displays properties for all > datasets on the system. > For each property, the following columns are displayed: > **name** > > Dataset name > **property** > > Property name > **value** > > Property value > **source** > > Property source > > **local**, **default**, **inherited**, **temporary**, **received**, or **-** (none). > **received** > > The received value of the property, if any. > > This column is not displayed by default. > The > **name**, **property**, **value**, > and > **source** > columns are displayed by default, though this can be > controlled by using the > **-o** > option. > This command takes a comma-separated list of properties as described in the > *Native Properties* > and > *User Properties* > sections of > zfsprops(7). > The value > **all** > can be used to display all properties that apply to the given dataset's type > (**filesystem**, **volume**, **snapshot**, or **bookmark**). > **-j**, **--json** \[*--json-int*] > > Display the output in JSON format. > > Specify > > **--json-int** > > to display numbers in integer format instead of strings for JSON output. > **-H** > > Display output in a form more easily parsed by scripts. > > Any headers are omitted, and fields are explicitly separated by a single tab > > instead of an arbitrary amount of space. > **-d** *depth* > > Recursively display any children of the dataset, limiting the recursion to > > *depth*. > > A depth of > > **1** > > will display only the dataset and its direct children. > **-o** *field* > > A comma-separated list of columns to display. > > Supported fields are > > **name**, **property**, **value**, **received**, **source**, > > or > > **all** > > to select all five columns. > > The default value is > > **name**,**property**,**value**,**source**. > **-p** > > Display numbers in parsable > > (exact) > > values. > **-r** > > Recursively display properties for any children. > **-s** *source* > > A comma-separated list of sources to display. > > Those properties coming from a source other than those in this list are ignored. > > Each source must be one of the following: > > **local**, **default**, **inherited**, **temporary**, **received**, or **none**. > > The default value is all sources. > **-t** *type* > > A comma-separated list of types to display, where > > *type* > > is one of > > **filesystem**, **snapshot**, **volume**, **bookmark**, or **all**. > > **fs**, > > **snap**, > > or > > **vol** > > can be used as aliases for > > **filesystem**, > > **snapshot**, > > or > > **volume**. **zfs** **inherit** \[**-rS**] *property* *filesystem*|*volume*|*snapshot*… > Clears the specified property, causing it to be inherited from an ancestor, > restored to default if no ancestor has the property set, or with the > **-S** > option reverted to the received value if one exists. > See > zfsprops(7) > for a listing of default values, and details on which properties can be > inherited. > **-r** > > Recursively inherit the given property for all children. > **-S** > > Revert the property to the received value, if one exists; > > otherwise, for non-inheritable properties, to the default; > > otherwise, operate as if the > > **-S** > > option was not specified. # EXAMPLES ## Example 1: Creating a ZFS File System Hierarchy The following commands create a file system named *pool/home* and a file system named *pool/home/bob*. The mount point */export/home* is set for the parent file system, and is automatically inherited by the child file system. # **zfs** **create** *pool/home* # **zfs** **set** **mountpoint**=*/export/home pool/home* # **zfs** **create** *pool/home/bob* ## Example 2: Disabling and Enabling File System Compression The following command disables the **compression** property for all file systems under *pool/home*. The next command explicitly enables **compression** for *pool/home/anne*. # **zfs** **set** **compression**=**off** *pool/home* # **zfs** **set** **compression**=**on** *pool/home/anne* ## Example 3: Setting a Quota on a ZFS File System The following command sets a quota of 50 Gbytes for *pool/home/bob*: # **zfs** **set** **quota**=*50G pool/home/bob* ## Example 4: Listing ZFS Properties The following command lists all properties for *pool/home/bob*: # **zfs** **get** **all** *pool/home/bob* NAME PROPERTY VALUE SOURCE pool/home/bob type filesystem - pool/home/bob creation Tue Jul 21 15:53 2009 - pool/home/bob used 21K - pool/home/bob available 20.0G - pool/home/bob referenced 21K - pool/home/bob compressratio 1.00x - pool/home/bob mounted yes - pool/home/bob quota 20G local pool/home/bob reservation none default pool/home/bob recordsize 128K default pool/home/bob mountpoint /pool/home/bob default pool/home/bob sharenfs off default pool/home/bob checksum on default pool/home/bob compression on local pool/home/bob atime on default pool/home/bob devices on default pool/home/bob exec on default pool/home/bob setuid on default pool/home/bob readonly off default pool/home/bob zoned off default pool/home/bob snapdir hidden default pool/home/bob acltype off default pool/home/bob aclmode discard default pool/home/bob aclinherit restricted default pool/home/bob canmount on default pool/home/bob xattr on default pool/home/bob copies 1 default pool/home/bob version 4 - pool/home/bob utf8only off - pool/home/bob normalization none - pool/home/bob casesensitivity sensitive - pool/home/bob vscan off default pool/home/bob nbmand off default pool/home/bob sharesmb off default pool/home/bob refquota none default pool/home/bob refreservation none default pool/home/bob primarycache all default pool/home/bob secondarycache all default pool/home/bob usedbysnapshots 0 - pool/home/bob usedbydataset 21K - pool/home/bob usedbychildren 0 - pool/home/bob usedbyrefreservation 0 - The following command gets a single property value: # **zfs** **get** **-H** **-o** **value compression** *pool/home/bob* on The following command gets a single property value recursively in JSON format: # **zfs** **get** **-j** **-r** **mountpoint** *pool/home* | **jq** { "output_version": { "command": "zfs get", "vers_major": 0, "vers_minor": 1 }, "datasets": { "pool/home": { "name": "pool/home", "type": "FILESYSTEM", "pool": "pool", "createtxg": "10", "properties": { "mountpoint": { "value": "/pool/home", "source": { "type": "DEFAULT", "data": "-" } } } }, "pool/home/bob": { "name": "pool/home/bob", "type": "FILESYSTEM", "pool": "pool", "createtxg": "1176", "properties": { "mountpoint": { "value": "/pool/home/bob", "source": { "type": "DEFAULT", "data": "-" } } } } } } The following command lists all properties with local settings for *pool/home/bob*: # **zfs** **get** **-r** **-s** **local** **-o** **name**,**property**,**value all** *pool/home/bob* NAME PROPERTY VALUE pool/home/bob quota 20G pool/home/bob compression on ## Example 5: Inheriting ZFS Properties The following command causes *pool/home/bob* and *pool/home/anne* to inherit the **checksum** property from their parent. # **zfs** **inherit** **checksum** *pool/home/bob pool/home/anne* ## Example 6: Setting User Properties The following example sets the user-defined *com.example*:*department* property for a dataset: # **zfs** **set** *com.example*:*department*=*12345 tank/accounting* ## Example 7: Setting sharenfs Property Options on a ZFS File System The following commands show how to set **sharenfs** property options to enable read-write access for a set of IP addresses and to enable root access for system "neo" on the *tank/home* file system: # **zfs** **set** **sharenfs**='*rw*=@123.123.0.0/16:[::1],root=*neo*' tank/home If you are using DNS for host name resolution, specify the fully-qualified hostname. # SEE ALSO zfsprops(7), zfs-list(8) Debian - March 15, 2026 ZFS-JAIL(8) - System Manager's Manual # NAME **zfs-jail** - attach or detach ZFS filesystem from FreeBSD jail # SYNOPSIS **zfs **jail**‌** *jailid*|*jailname* *filesystem* **zfs **unjail**‌** *jailid*|*jailname* *filesystem* # DESCRIPTION The **zfs-jail** functionality can be used to assign a dataset onto a running FreeBSD system jail(4), allowing zfs(8) management utilities to be run inside of the jail(4). To allow management of the dataset from within a jail, the **jailed** property should be set and the required devfs.conf(5) entries to expose */dev/zfs* device within the jail must be present. The **quota** property cannot be changed from within a jail. To use this functionality, the jail needs the **allow.mount** and **allow.mount.zfs** parameters set to **1** and the **enforce\_statfs** parameter set to a value lower than **2**. The subcommands are as follows: **jail** *jailid*|*jailname* *filesystem* > Attach the specified > *filesystem* > to the jail identified by JID > *jailid* > or name > *jailname*. > From now on this file system tree can be managed from within a jail if the > **jailed** > property has been set. > You cannot attach a jailed dataset's children to another jail. > You can also not attach the root file system > of the jail or any dataset which needs to be mounted before the zfs rc script > is run inside the jail, as it would be attached unmounted until it is > mounted from the rc script inside the jail. > After a dataset is attached to a jail and the > **jailed** > property is set, a jailed file system cannot be mounted outside the jail, > since the jail administrator might have set the mount point to an unacceptable > value. **unjail** *jailid*|*jailname* *filesystem* > Detaches the specified > *filesystem* > from the jail identified by JID > *jailid* > or name > *jailname*. # SEE ALSO devfs.conf(5), zfsprops(7), jail(8) # CAVEATS The root directory of jail can not be delegated to the jail with this utility because the jail must be running with a valid root directory. Jails are a FreeBSD feature and are not relevant on other platforms. See jail(8) for more information on managing jails, or zfs-zone(8) for the equivalent functionality on Linux. Debian - November 4, 2025 ZFS-LIST(8) - System Manager's Manual # NAME **zfs-list** - list properties of ZFS datasets # SYNOPSIS **zfs** **list** \[**-r**|**-d** *depth*] \[**-Hp**] \[**-j** \[*--json-int*]] \[**-o** *property*\[,*property*]…] \[**-s** *property*]… \[**-S** *property*]… \[**-t** *type*\[,*type*]…] \[*filesystem*|*volume*|*snapshot*]… # DESCRIPTION By default, all file systems and volumes are displayed, with the following fields: **name**, **used**, **available**, **referenced**, **mountpoint**. Snapshots are displayed if the **listsnapshots** pool property is **on** (the default is **off**) or if the **-t** **snapshot** or **-t** **all** options are specified. **-H** > Used for scripting mode. > Do not print headers, and separate fields by a single tab instead of arbitrary > white space. **-j**, **--json** \[*--json-int*] > Print the output in JSON format. > Specify > **--json-int** > to print the numbers in integer format instead of strings in JSON output. **-d** *depth* > Recursively display any children of the dataset, limiting the recursion to > *depth*. > A > *depth* > of > **1** > will display only the dataset and its direct children. **-o** *property* > A comma-separated list of properties to display. > Each property must be: > * One of the properties described in the > *Native Properties* > section of > zfsprops(7) > * A user property > * The value > **name** > to display the dataset name > * The value > **space** > to display space usage properties on file systems and volumes. > This is a shortcut for specifying > **-o** **name**,**avail**,**used**,**usedsnap**,**usedds**,**usedrefreserv**,**usedchild** > **-t** **filesystem**,**volume**. **-p** > Display numbers in parsable > (exact) > values. **-r** > Recursively display any children of the dataset on the command line. **-s** *property* > A property for sorting the output by column in ascending order based on the > value of the property. > The property must be one of the properties described in the > *Properties* > section of > zfsprops(7) > or the value > **name** > to sort by the dataset name. > Multiple properties can be specified to operate together using multiple > **-s** > or > **-S** > options. > Multiple > **-s** > and > **-S** > options are evaluated from left to right to supply sort keys in > decreasing order of priority. > Property types operate as follows: > * Numeric types sort in numeric order. > * String types sort in alphabetical order. > * Types inappropriate for a row sort that row to the literal bottom, > regardless of the specified ordering. > If no sort columns are specified, or if two lines of output would sort > equally across all specified columns, then datasets and bookmarks are > sorted by name, whereas snapshots are sorted first by the name of their > dataset and then by the time of their creation. > When no sort columns are specified but snapshots are listed, this > default behavior causes snapshots to be grouped under their datasets in > chronological order by creation time. **-S** *property* > Same as > **-s**, > but sorts by > *property* > in descending order. **-t** *type* > A comma-separated list of types to display, where > *type* > is one of > **filesystem**, > **snapshot**, > **volume**, > **bookmark**, > or > **all**. > For example, specifying > **-t** **snapshot** > displays only snapshots. > **fs**, > **snap**, > or > **vol** > can be used as aliases for > **filesystem**, > **snapshot**, > or > **volume**. # EXAMPLES ## Example 1: Listing ZFS Datasets The following command lists all active file systems and volumes in the system. Snapshots are displayed if **listsnaps**=**on**. The default is **off**. See zpoolprops(7) for more information on pool properties. # **zfs** **list** NAME USED AVAIL REFER MOUNTPOINT pool 450K 457G 18K /pool pool/home 315K 457G 21K /export/home pool/home/anne 18K 457G 18K /export/home/anne pool/home/bob 276K 457G 276K /export/home/bob ## Example 2: Listing ZFS filesystems and snapshots in JSON format # **zfs** **list** **-j** **-t** *filesystem,snapshot* | **jq** { "output_version": { "command": "zfs list", "vers_major": 0, "vers_minor": 1 }, "datasets": { "pool": { "name": "pool", "type": "FILESYSTEM", "pool": "pool", "properties": { "used": { "value": "290K", "source": { "type": "NONE", "data": "-" } }, "available": { "value": "30.5G", "source": { "type": "NONE", "data": "-" } }, "referenced": { "value": "24K", "source": { "type": "NONE", "data": "-" } }, "mountpoint": { "value": "/pool", "source": { "type": "DEFAULT", "data": "-" } } } }, "pool/home": { "name": "pool/home", "type": "FILESYSTEM", "pool": "pool", "properties": { "used": { "value": "48K", "source": { "type": "NONE", "data": "-" } }, "available": { "value": "30.5G", "source": { "type": "NONE", "data": "-" } }, "referenced": { "value": "24K", "source": { "type": "NONE", "data": "-" } }, "mountpoint": { "value": "/mnt/home", "source": { "type": "LOCAL", "data": "-" } } } }, "pool/home/bob": { "name": "pool/home/bob", "type": "FILESYSTEM", "pool": "pool", "properties": { "used": { "value": "24K", "source": { "type": "NONE", "data": "-" } }, "available": { "value": "30.5G", "source": { "type": "NONE", "data": "-" } }, "referenced": { "value": "24K", "source": { "type": "NONE", "data": "-" } }, "mountpoint": { "value": "/mnt/home/bob", "source": { "type": "INHERITED", "data": "pool/home" } } } }, "pool/home/bob@v1": { "name": "pool/home/bob@v1", "type": "SNAPSHOT", "pool": "pool", "dataset": "pool/home/bob", "snapshot_name": "v1", "properties": { "used": { "value": "0B", "source": { "type": "NONE", "data": "-" } }, "available": { "value": "-", "source": { "type": "NONE", "data": "-" } }, "referenced": { "value": "24K", "source": { "type": "NONE", "data": "-" } }, "mountpoint": { "value": "-", "source": { "type": "NONE", "data": "-" } } } } } } # SEE ALSO zfsprops(7), zfs-get(8) Debian - August 25, 2025 ZFS-LOAD-KEY(8) - System Manager's Manual # NAME **zfs-load-key** - load, unload, or change encryption key of ZFS dataset # SYNOPSIS **zfs** **load-key** \[**-nr**] \[**-L** *keylocation*] **-a**|*filesystem* **zfs** **unload-key** \[**-r**] **-a**|*filesystem* **zfs** **change-key** \[**-l**] \[**-o** *keylocation*=*value*] \[**-o** *keyformat*=*value*] \[**-o** *pbkdf2iters*=*value*] \[**-o** *user:prop*=*value*] *filesystem* **zfs** **change-key** **-i** \[**-l**] *filesystem* # DESCRIPTION **zfs** **load-key** \[**-nr**] \[**-L** *keylocation*] **-a**|*filesystem* > Load the key for > *filesystem*, > allowing it and all children that inherit the > **keylocation** > property to be accessed. > The key will be expected in the format specified by the > **keyformat** > and location specified by the > **keylocation** > property. > Note that if the > **keylocation** > is set to > **prompt** > the terminal will interactively wait for the key to be entered. > Loading a key will not automatically mount the dataset. > If that functionality is desired, > **zfs** **mount** **-l** > will ask for the key and mount the dataset > (see > zfs-mount(8)) > . > For automated key management during user login, > pam\_zfs\_key(8) > can load keys and mount encrypted home directories on systems with PAM support. > Once the key is loaded the > **keystatus** > property will become > **available**. > **-r** > > Recursively loads the keys for the specified filesystem and all descendent > > encryption roots. > **-a** > > Loads the keys for all encryption roots in all imported pools. > **-n** > > Do a dry-run > > ("No-op") > > **load-key**. > > This will cause > > **zfs** > > to simply check that the provided key is correct. > > This command may be run even if the key is already loaded. > **-L** *keylocation* > > Use > > *keylocation* > > instead of the > > **keylocation** > > property. > > This will not change the value of the property on the dataset. > > Note that if used with either > > **-r** > > or > > **-a**, > > *keylocation* > > may only be given as > > **prompt**. **zfs** **unload-key** \[**-r**] **-a**|*filesystem* > Unloads a key from ZFS, removing the ability to access the dataset and all of > its children that inherit the > **keylocation** > property. > This requires that the dataset is not currently open or mounted. > Once the key is unloaded the > **keystatus** > property will become > **unavailable**. > **-r** > > Recursively unloads the keys for the specified filesystem and all descendent > > encryption roots. > **-a** > > Unloads the keys for all encryption roots in all imported pools. **zfs** **change-key** \[**-l**] \[**-o** *keylocation*=*value*] \[**-o** *keyformat*=*value*] \[**-o** *pbkdf2iters*=*value*] \[**-o** *user:prop*=*value*] *filesystem* **zfs** **change-key** **-i** \[**-l**] *filesystem* > Changes the user's key (e.g. a passphrase) used to access a dataset. > This command requires that the existing key for the dataset is already loaded. > This command may also be used to change the > **keylocation**, > **keyformat**, > and > **pbkdf2iters** > properties as needed, as well as set user properties. > If the dataset was not previously an encryption root it will become one. > Alternatively, the > **-i** > flag may be provided to cause an encryption root to inherit the parent's key > instead. > If the user's key is compromised, > **zfs** **change-key** > does not necessarily protect existing or newly-written data from attack. > Newly-written data will continue to be encrypted with the same master key as > the existing data. > The master key is compromised if an attacker obtains a > user key and the corresponding wrapped master key. > Currently, > **zfs** **change-key** > does not overwrite the previous wrapped master key on disk, so it is > accessible via forensic analysis for an indeterminate length of time. > In the event of a master key compromise, ideally the drives should be securely > erased to remove all the old data (which is readable using the compromised > master key), a new pool created, and the data copied back. > This can be approximated in place by creating new datasets, copying the data > (e.g. using **zfs** **send** | **zfs** **recv**), > and then clearing the free space with > **zpool** **trim** **--secure** > if supported by your hardware, otherwise > **zpool** **initialize**. > **-l** > > Ensures the key is loaded before attempting to change the key. > > This is effectively equivalent to running > > **zfs** **load-key** *filesystem*; **zfs** **change-key** *filesystem* > **-o** *property*=*value* > > Allows the user to set encryption key properties > > (**keyformat**, **keylocation**, and **pbkdf2iters**) > > and user properties while changing the key. > > This is the only way to alter > > **keyformat** > > and > > **pbkdf2iters** > > after the dataset has been created. > **-i** > > Indicates that zfs should make > > *filesystem* > > inherit the key of its parent. > > Note that this command can only be run on an encryption root > > that has an encrypted parent. ## Encryption Enabling the **encryption** feature allows for the creation of encrypted filesystems and volumes. ZFS will encrypt file and volume data, file attributes, ACLs, permission bits, directory listings, FUID mappings, and **userused**/**groupused** data. ZFS will not encrypt metadata related to the pool structure, including dataset and snapshot names, dataset hierarchy, properties, file size, file holes, and deduplication tables (though the deduplicated data itself is encrypted). Key rotation is managed by ZFS. Changing the user's key (e.g. a passphrase) does not require re-encrypting the entire dataset. Datasets can be scrubbed, resilvered, renamed, and deleted without the encryption keys being loaded (see the **load-key** subcommand for more info on key loading). Creating an encrypted dataset requires specifying the **encryption** and **keyformat** properties at creation time, along with an optional **keylocation** and **pbkdf2iters**. After entering an encryption key, the created dataset will become an encryption root. Any descendant datasets will inherit their encryption key from the encryption root by default, meaning that loading, unloading, or changing the key for the encryption root will implicitly do the same for all inheriting datasets. If this inheritance is not desired, simply supply a **keyformat** when creating the child dataset or use **zfs** **change-key** to break an existing relationship, creating a new encryption root on the child. Note that the child's **keyformat** may match that of the parent while still creating a new encryption root, and that changing the **encryption** property alone does not create a new encryption root; this would simply use a different cipher suite with the same key as its encryption root. The one exception is that clones will always use their origin's encryption key. As a result of this exception, some encryption-related properties (namely **keystatus**, **keyformat**, **keylocation**, and **pbkdf2iters**) do not inherit like other ZFS properties and instead use the value determined by their encryption root. Encryption root inheritance can be tracked via the read-only **encryptionroot** property. Encryption changes the behavior of a few ZFS operations. Encryption is applied after compression so compression ratios are preserved. Normally checksums in ZFS are 256 bits long, but for encrypted data the checksum is 128 bits of the user-chosen checksum and 128 bits of MAC from the encryption suite, which provides additional protection against maliciously altered data. Deduplication is still possible with encryption enabled but for security, datasets will only deduplicate against themselves, their snapshots, and their clones. There are a few limitations on encrypted datasets. Encrypted data cannot be embedded via the **embedded\_data** feature. Encrypted datasets may not have **copies**=*3* since the implementation stores some encryption metadata where the third copy would normally be. Since compression is applied before encryption, datasets may be vulnerable to a CRIME-like attack if applications accessing the data allow for it. Deduplication with encryption will leak information about which blocks are equivalent in a dataset and will incur an extra CPU cost for each block written. # SEE ALSO zfsprops(7), pam\_zfs\_key(8), zfs-create(8), zfs-set(8) Debian - January 30, 2026 ZFS-MOUNT-GENERATOR(8) - System Manager's Manual # NAME **zfs-mount-generator** - generate systemd mount units for ZFS filesystems # SYNOPSIS *@systemdgeneratordir@/zfs-mount-generator* # DESCRIPTION **zfs-mount-generator** is a systemd.generator(7) that generates native systemd.mount(5) units for configured ZFS datasets. ## Properties **mountpoint**= > Skipped if **legacy** or **none**. **canmount**= > Skipped if **off**. > Skipped if only **noauto** > datasets exist for a given mountpoint and there's more than one. > Datasets with **yes** take precedence over ones with **noauto** for the same mountpoint. > Sets logical *noauto* flag if **noauto**. > Encryption roots always generate > **zfs-load-key@**‌*root*‌**.service**, > even if > **off**. **atime**=, **relatime**=, **devices**=, **exec**=, **readonly**=, **setuid**=, **nbmand**= > Used to generate mount options equivalent to > **zfs** **mount**. **encroot**=, **keylocation**= > If the dataset is an encryption root, its mount unit will bind to > **zfs-load-key@**‌*root*‌**.service**, > with additional dependencies as follows: > **keylocation**=**prompt** > > None, uses > > systemd-ask-password(1) > **keylocation**=**https://**‌*URL* (et al.) > > **Wants**=, **After**=: *network-online.target* > **keylocation**=**file://**<*path*> > > **RequiresMountsFor**=*path* > The service also uses the same > **Wants**=, > **After**=, > **Requires**=, and > **RequiresMountsFor**=, > as the mount unit. **org.openzfs.systemd:requires**=*path*\[ *path*]… > Sets **Requires**= for the mount- and key-loading unit. **org.openzfs.systemd:requires-mounts-for**=*path*\[ *path*]… > Sets **RequiresMountsFor**= for the mount- and key-loading unit. **org.openzfs.systemd:before**=*unit*\[ *unit*]… > Sets **Before**= for the mount unit. **org.openzfs.systemd:after**=*unit*\[ *unit*]… > Sets **After**= for the mount unit. **org.openzfs.systemd:wanted-by**=*unit*\[ *unit*]… > Sets logical *noauto* flag (see below). > If not **none**, sets **WantedBy**= for the mount unit. **org.openzfs.systemd:required-by**=*unit*\[ *unit*]… > Sets logical *noauto* flag (see below). > If not **none**, sets **RequiredBy**= for the mount unit. **org.openzfs.systemd:nofail**=(unset)|**on**|**off** > Waxes or wanes strength of default reverse dependencies of the mount unit, see > below. **org.openzfs.systemd:ignore**=**on**|**off** > Skip if **on**. > Defaults to **off**. ## Unit Ordering And Dependencies Additionally, unless the pool the dataset resides on is imported at generation time, both units gain **Wants**=*zfs-import.target* and **After**=*zfs-import.target*. Additionally, unless the logical *noauto* flag is set, the mount unit gains a reverse-dependency for *local-fs.target* of strength (unset) > **WantedBy**= + **Before**= **on** > **WantedBy**= **off** > **RequiredBy**= + **Before**= ## Cache File Because ZFS pools may not be available very early in the boot process, information on ZFS mountpoints must be stored separately. The output of **zfs** **list** **-Ho** *name*, for datasets that should be mounted by systemd should be kept at *@sysconfdir@/zfs/zfs-list.cache/*‌*poolname*, and, if writeable, will be kept synchronized for the entire pool by the *history\_event-zfs-list-cacher.sh* ZEDLET, if enabled (see zed(8)). # ENVIRONMENT If the **ZFS\_DEBUG** environment variable is nonzero (or unset and */proc/cmdline* contains "**debug**"), print summary accounting information at the end. # EXAMPLES To begin, enable tracking for the pool: # **touch** *@sysconfdir@/zfs/zfs-list.cache/**poolname* Then enable the tracking ZEDLET: # **ln** **-s** *@zfsexecdir@/zed.d/history_event-zfs-list-cacher.sh @sysconfdir@/zfs/zed.d* # **systemctl** **enable** *zfs-zed.service* # **systemctl** **restart** *zfs-zed.service* If no history event is in the queue, inject one to ensure the ZEDLET runs to refresh the cache file by setting a monitored property somewhere on the pool: # **zfs** **set** **relatime**=**off** *poolname/dset* # **zfs** **inherit** **relatime** *poolname/dset* To test the generator output: $ **mkdir** */tmp/zfs-mount-generator* $ **@systemdgeneratordir@/zfs-mount-generator** */tmp/zfs-mount-generator* If the generated units are satisfactory, instruct **systemd** to re-run all generators: # **systemctl daemon-reload** # SEE ALSO systemd.mount(5), systemd.target(5), zfs(5), systemd.generator(7), systemd.special(7), zed(8), zpool-events(8) Debian - November 30, 2021 ZFS-MOUNT(8) - System Manager's Manual # NAME **zfs-mount** - manage mount state of ZFS filesystems # SYNOPSIS **zfs** **mount** \[**-j**] **zfs** **mount** \[**-Oflv**] \[**-o** *options*] **-a**|**-R** *filesystem*|*filesystem* **zfs** **unmount** \[**-fu**] **-a**|*filesystem*|*mountpoint* # DESCRIPTION **zfs** **mount** \[**-j**] > Displays all ZFS file systems currently mounted. > **-j**, **--json** > > Displays all mounted file systems in JSON format. **zfs** **mount** \[**-Oflv**] \[**-o** *options*] **-a**|**-R** *filesystem*|*filesystem* > Mount ZFS filesystem on a path described by its > **mountpoint** > property, if the path exists and is empty. > If > **mountpoint** > is set to > *legacy*, > the filesystem should be instead mounted using > mount(8). > **-O** > > Perform an overlay mount. > > Allows mounting in non-empty > > **mountpoint**. > > See > > mount(8) > > for more information. > **-a** > > Mount all available ZFS file systems. > > Invoked automatically as part of the boot process if configured. > **-R** > > Mount the specified filesystems along with all their children. > *filesystem* > > Mount the specified filesystem. > **-o** *options* > > An optional, comma-separated list of mount options to use temporarily for the > > duration of the mount. > > See the > > *Temporary Mount Point Properties* > > section of > > zfsprops(7) > > for details. > **-l** > > Load keys for encrypted filesystems as they are being mounted. > > This is equivalent to executing > > **zfs** **load-key** > > on each encryption root before mounting it. > > Note that if a filesystem has > > **keylocation**=**prompt**, > > this will cause the terminal to interactively block after asking for the key. > > On systems with PAM support, > > pam\_zfs\_key(8) > > can automate this process during user login. > **-v** > > Report mount progress. > **-f** > > Attempt to force mounting of all filesystems, even those that couldn't normally > > be mounted (e.g. redacted datasets). **zfs** **unmount** \[**-fu**] **-a**|*filesystem*|*mountpoint* > Unmounts currently mounted ZFS file systems. > **-a** > > Unmount all available ZFS file systems. > > Invoked automatically as part of the shutdown process. > **-f** > > Forcefully unmount the file system, even if it is currently in use. > > This option is not supported on Linux. > **-u** > > Unload keys for any encryption roots unmounted by this command. > *filesystem*|*mountpoint* > > Unmount the specified filesystem. > > The command can also be given a path to a ZFS file system mount point on the > > system. # SEE ALSO pam\_zfs\_key(8) Debian - October 12, 2024 .\" SPDX-License-Identifier: CDDL-1.0 .\" .\" This file and its contents are supplied under the terms of the .\" Common Development and Distribution License ("CDDL"), version 1.0. .\" You may only use this file in accordance with the terms of version .\" 1.0 of the CDDL. .\" .\" A full copy of the text of the CDDL should have accompanied this .\" source. A copy of the CDDL is also available via the Internet at .\" https://opensource.org/license/CDDL-1.0. .\" .\" Copyright (c) 2016, 2019 by Delphix. All Rights Reserved. .\" Copyright (c) 2019, 2020 by Christian Schwarz. All Rights Reserved. .\" Copyright 2020 Joyent, Inc. .\" Copyright (c) 2025, Rob Norris .\" .Dd June 5, 2025 .Dt ZFS-PROGRAM 8 .Os . .Sh NAME .Nm zfs-program .Nd execute ZFS channel programs .Sh SYNOPSIS .Nm zfs .Cm program .Op Fl jn .Op Fl t Ar instruction-limit .Op Fl m Ar memory-limit .Ar pool .Ar script .Op Ar script arguments . .Sh DESCRIPTION The ZFS channel program interface allows ZFS administrative operations to be run programmatically as a Lua script. The entire script is executed atomically, with no other administrative operations taking effect concurrently. A library of ZFS calls is made available to channel program scripts. Channel programs may only be run with root privileges. .Pp A modified version of the Lua 5.2 interpreter is used to run channel program scripts. The Lua 5.2 manual can be found at .Lk http://www.lua.org/manual/5.2/ .Pp The channel program given by .Ar script will be run on .Ar pool , and any attempts to access or modify other pools will cause an error. . .Sh OPTIONS .Bl -tag -width "-t" .It Fl j , -json Display channel program output in JSON format. When this flag is specified and standard output is empty - channel program encountered an error. The details of such an error will be printed to standard error in plain text. .It Fl n Executes a read-only channel program, which runs faster. The program cannot change on-disk state by calling functions from the zfs.sync submodule. The program can be used to gather information such as properties and determining if changes would succeed (zfs.check.*). Without this flag, all pending changes must be synced to disk before a channel program can complete. .It Fl t Ar instruction-limit Limit the number of Lua instructions to execute. If a channel program executes more than the specified number of instructions, it will be stopped and an error will be returned. The default limit is 10 million instructions, and it can be set to a maximum of 100 million instructions. .It Fl m Ar memory-limit Memory limit, in bytes. If a channel program attempts to allocate more memory than the given limit, it will be stopped and an error returned. The default memory limit is 10 MiB, and can be set to a maximum of 100 MiB. .El .Pp All remaining argument strings will be passed directly to the Lua script as described in the .Sx LUA INTERFACE section below. . .Sh LUA INTERFACE A channel program can be invoked either from the command line, or via a library call to .Fn lzc_channel_program . . .Ss Arguments Arguments passed to the channel program are converted to a Lua table. If invoked from the command line, extra arguments to the Lua script will be accessible as an array stored in the argument table with the key 'argv': .Bd -literal -compact -offset indent args = ... argv = args["argv"] -- argv == {1="arg1", 2="arg2", ...} .Ed .Pp If invoked from the libzfs interface, an arbitrary argument list can be passed to the channel program, which is accessible via the same .Qq Li ... syntax in Lua: .Bd -literal -compact -offset indent args = ... -- args == {"foo"="bar", "baz"={...}, ...} .Ed .Pp Note that because Lua arrays are 1-indexed, arrays passed to Lua from the libzfs interface will have their indices incremented by 1. That is, the element in .Va arr[0] in a C array passed to a channel program will be stored in .Va arr[1] when accessed from Lua. . .Ss Return Values Lua return statements take the form: .Dl return ret0, ret1, ret2, ... .Pp Return statements returning multiple values are permitted internally in a channel program script, but attempting to return more than one value from the top level of the channel program is not permitted and will throw an error. However, tables containing multiple values can still be returned. If invoked from the command line, a return statement: .Bd -literal -compact -offset indent a = {foo="bar", baz=2} return a .Ed .Pp Will be output formatted as: .Bd -literal -compact -offset indent Channel program fully executed with return value: return: baz: 2 foo: 'bar' .Ed . .Ss Fatal Errors If the channel program encounters a fatal error while running, a non-zero exit status will be returned. If more information about the error is available, a singleton list will be returned detailing the error: .Dl error: \&"error string, including Lua stack trace" .Pp If a fatal error is returned, the channel program may have not executed at all, may have partially executed, or may have fully executed but failed to pass a return value back to userland. .Pp If the channel program exhausts an instruction or memory limit, a fatal error will be generated and the program will be stopped, leaving the program partially executed. No attempt is made to reverse or undo any operations already performed. Note that because both the instruction count and amount of memory used by a channel program are deterministic when run against the same inputs and filesystem state, as long as a channel program has run successfully once, you can guarantee that it will finish successfully against a similar size system. .Pp If a channel program attempts to return too large a value, the program will fully execute but exit with a nonzero status code and no return value. .Pp .Em Note : ZFS API functions do not generate Fatal Errors when correctly invoked, they return an error code and the channel program continues executing. See the .Sx ZFS API section below for function-specific details on error return codes. . .Ss Lua to C Value Conversion When invoking a channel program via the libzfs interface, it is necessary to translate arguments and return values from Lua values to their C equivalents, and vice-versa. .Pp There is a correspondence between nvlist values in C and Lua tables. A Lua table which is returned from the channel program will be recursively converted to an nvlist, with table values converted to their natural equivalents: .TS cw3 l c l . string -> string number -> int64 boolean -> boolean_value nil -> boolean (no value) table -> nvlist .TE .Pp Likewise, table keys are replaced by string equivalents as follows: .TS cw3 l c l . string -> no change number -> signed decimal string ("%lld") boolean -> "true" | "false" .TE .Pp Any collision of table key strings (for example, the string "true" and a true boolean value) will cause a fatal error. .Pp Lua numbers are represented internally as signed 64-bit integers. . .Sh LUA STANDARD LIBRARY The following Lua built-in base library functions are available: .TS cw3 l l l l . assert rawlen collectgarbage rawget error rawset getmetatable select ipairs setmetatable next tonumber pairs tostring rawequal type .TE .Pp All functions in the .Em coroutine , .Em string , and .Em table built-in submodules are also available. A complete list and documentation of these modules is available in the Lua manual. .Pp The following functions base library functions have been disabled and are not available for use in channel programs: .TS cw3 l l l l l l . dofile loadfile load pcall print xpcall .TE . .Sh ZFS API . .Ss Function Arguments Each API function takes a fixed set of required positional arguments and optional keyword arguments. For example, the destroy function takes a single positional string argument (the name of the dataset to destroy) and an optional "defer" keyword boolean argument. When using parentheses to specify the arguments to a Lua function, only positional arguments can be used: .Dl Sy zfs.sync.destroy Ns Pq \&"rpool@snap" .Pp To use keyword arguments, functions must be called with a single argument that is a Lua table containing entries mapping integers to positional arguments and strings to keyword arguments: .Dl Sy zfs.sync.destroy Ns Pq {1="rpool@snap", defer=true} .Pp The Lua language allows curly braces to be used in place of parenthesis as syntactic sugar for this calling convention: .Dl Sy zfs.sync.snapshot Ns {"rpool@snap", defer=true} . .Ss Function Return Values If an API function succeeds, it returns 0. If it fails, it returns an error code and the channel program continues executing. API functions do not generate Fatal Errors except in the case of an unrecoverable internal file system error. .Pp In addition to returning an error code, some functions also return extra details describing what caused the error. This extra description is given as a second return value, and will always be a Lua table, or Nil if no error details were returned. Different keys will exist in the error details table depending on the function and error case. Any such function may be called expecting a single return value: .Dl errno = Sy zfs.sync.promote Ns Pq dataset .Pp Or, the error details can be retrieved: .Bd -literal -compact -offset indent .No errno, details = Sy zfs.sync.promote Ns Pq dataset if (errno == EEXIST) then assert(details ~= Nil) list_of_conflicting_snapshots = details end .Ed .Pp The following global aliases for API function error return codes are defined for use in channel programs: .TS cw3 l l l l l l l . EPERM ECHILD ENODEV ENOSPC ENOENT EAGAIN ENOTDIR ESPIPE ESRCH ENOMEM EISDIR EROFS EINTR EACCES EINVAL EMLINK EIO EFAULT ENFILE EPIPE ENXIO ENOTBLK EMFILE EDOM E2BIG EBUSY ENOTTY ERANGE ENOEXEC EEXIST ETXTBSY EDQUOT EBADF EXDEV EFBIG .TE . .Ss API Functions For detailed descriptions of the exact behavior of any ZFS administrative operations, see the main .Xr zfs 8 manual page. .Bl -tag -width "xx" .It Fn zfs.debug msg Record a debug message in the zfs_dbgmsg log. A log of these messages can be printed via mdb's "::zfs_dbgmsg" command, or can be monitored live by running .Dl dtrace -n 'zfs-dbgmsg{trace(stringof(arg0))}' .Pp .Bl -tag -compact -width "property (string)" .It Ar msg Pq string Debug message to be printed. .El .It Fn zfs.exists dataset Returns true if the given dataset exists, or false if it doesn't. A fatal error will be thrown if the dataset is not in the target pool. That is, in a channel program running on rpool, .Sy zfs.exists Ns Pq \&"rpool/nonexistent_fs" returns false, but .Sy zfs.exists Ns Pq \&"somepool/fs_that_may_exist" will error. .Pp .Bl -tag -compact -width "property (string)" .It Ar dataset Pq string Dataset to check for existence. Must be in the target pool. .El .It Fn zfs.get_prop dataset property Returns two values. First, a string, number or table containing the property value for the given dataset. Second, a string containing the source of the property (i.e. the name of the dataset in which it was set or nil if it is readonly). Throws a Lua error if the dataset is invalid or the property doesn't exist. Note that Lua only supports int64 number types whereas ZFS number properties are uint64. This means very large values (like GUIDs) may wrap around and appear negative. .Pp .Bl -tag -compact -width "property (string)" .It Ar dataset Pq string Filesystem or snapshot path to retrieve properties from. .It Ar property Pq string Name of property to retrieve. All filesystem, snapshot and volume properties are supported except for .Sy mounted and .Sy iscsioptions . Also supports the .Sy written@ Ns Ar snap and .Sy written# Ns Ar bookmark properties and the .Ao Sy user Ns | Ns Sy group Ac Ns Ao Sy quota Ns | Ns Sy used Ac Ns Sy @ Ns Ar id properties, though the id must be in numeric form. .El .El .Bl -tag -width "xx" .It Sy zfs.sync submodule The sync submodule contains functions that modify the on-disk state. They are executed in "syncing context". .Pp The available sync submodule functions are as follows: .Bl -tag -width "xx" .It Fn zfs.sync.clone snapshot newdataset Create a new filesystem from a snapshot. Returns 0 if the filesystem was successfully created, and a nonzero error code otherwise. .Pp Note: Due to general limitations in channel programs, a filesystem created this way will not be mounted, regardless of the value of the .Sy mountpoint and .Sy canmount properties. This limitation may be removed in the future, so it is recommended that you set .Sy mountpoint Ns = Ns Sy none or .Sy canmount Ns = Ns Sy off or .Sy noauto to avoid surprises. .Pp .Bl -tag -compact -width "newbookmark (string)" .It Ar snapshot Pq string Name of the source snapshot to clone. .It Ar newdataset Pq string Name of the target dataset to create. .El .It Sy zfs.sync.destroy Ns Pq Ar dataset , Op Ar defer Ns = Ns Sy true Ns | Ns Sy false Destroy the given dataset. Returns 0 on successful destroy, or a nonzero error code if the dataset could not be destroyed (for example, if the dataset has any active children or clones). .Pp .Bl -tag -compact -width "newbookmark (string)" .It Ar dataset Pq string Filesystem or snapshot to be destroyed. .It Op Ar defer Pq boolean Valid only for destroying snapshots. If set to true, and the snapshot has holds or clones, allows the snapshot to be marked for deferred deletion rather than failing. .El .It Fn zfs.sync.inherit dataset property Clears the specified property in the given dataset, causing it to be inherited from an ancestor, or restored to the default if no ancestor property is set. The .Nm zfs Cm inherit Fl S option has not been implemented. Returns 0 on success, or a nonzero error code if the property could not be cleared. .Pp .Bl -tag -compact -width "newbookmark (string)" .It Ar dataset Pq string Filesystem or snapshot containing the property to clear. .It Ar property Pq string The property to clear. Allowed properties are the same as those for the .Nm zfs Cm inherit command. .El .It Fn zfs.sync.promote dataset Promote the given clone to a filesystem. Returns 0 on successful promotion, or a nonzero error code otherwise. If EEXIST is returned, the second return value will be an array of the clone's snapshots whose names collide with snapshots of the parent filesystem. .Pp .Bl -tag -compact -width "newbookmark (string)" .It Ar dataset Pq string Clone to be promoted. .El .It Fn zfs.sync.rollback filesystem Rollback to the previous snapshot for a dataset. Returns 0 on successful rollback, or a nonzero error code otherwise. Rollbacks can be performed on filesystems or zvols, but not on snapshots or mounted datasets. EBUSY is returned in the case where the filesystem is mounted. .Pp .Bl -tag -compact -width "newbookmark (string)" .It Ar filesystem Pq string Filesystem to rollback. .El .It Fn zfs.sync.set_prop dataset property value Sets the given property on a dataset. Currently only user properties are supported. Returns 0 if the property was set, or a nonzero error code otherwise. .Pp .Bl -tag -compact -width "newbookmark (string)" .It Ar dataset Pq string The dataset where the property will be set. .It Ar property Pq string The property to set. .It Ar value Pq string The value of the property to be set. .El .It Fn zfs.sync.snapshot dataset Create a snapshot of a filesystem. Returns 0 if the snapshot was successfully created, and a nonzero error code otherwise. .Pp Note: Taking a snapshot will fail on any pool older than legacy version 27. To enable taking snapshots from ZCP scripts, the pool must be upgraded. .Pp .Bl -tag -compact -width "newbookmark (string)" .It Ar dataset Pq string Name of snapshot to create. .El .It Fn zfs.sync.rename_snapshot dataset oldsnapname newsnapname Rename a snapshot of a filesystem or a volume. Returns 0 if the snapshot was successfully renamed, and a nonzero error code otherwise. .Pp .Bl -tag -compact -width "newbookmark (string)" .It Ar dataset Pq string Name of the snapshot's parent dataset. .It Ar oldsnapname Pq string Original name of the snapshot. .It Ar newsnapname Pq string New name of the snapshot. .El .It Fn zfs.sync.bookmark source newbookmark Create a bookmark of an existing source snapshot or bookmark. Returns 0 if the new bookmark was successfully created, and a nonzero error code otherwise. .Pp Note: Bookmarking requires the corresponding pool feature to be enabled. .Pp .Bl -tag -compact -width "newbookmark (string)" .It Ar source Pq string Full name of the existing snapshot or bookmark. .It Ar newbookmark Pq string Full name of the new bookmark. .El .El .It Sy zfs.check submodule For each function in the .Sy zfs.sync submodule, there is a corresponding .Sy zfs.check function which performs a "dry run" of the same operation. Each takes the same arguments as its .Sy zfs.sync counterpart and returns 0 if the operation would succeed, or a non-zero error code if it would fail, along with any other error details. That is, each has the same behavior as the corresponding sync function except for actually executing the requested change. For example, .Fn zfs.check.destroy \&"fs" returns 0 if .Fn zfs.sync.destroy \&"fs" would successfully destroy the dataset. .Pp The available .Sy zfs.check functions are: .Bl -tag -compact -width "xx" .It Fn zfs.check.clone snapshot newdataset .It Sy zfs.check.destroy Ns Pq Ar dataset , Op Ar defer Ns = Ns Sy true Ns | Ns Sy false .It Fn zfs.check.promote dataset .It Fn zfs.check.rollback filesystem .It Fn zfs.check.set_property dataset property value .It Fn zfs.check.snapshot dataset .El .It Sy zfs.list submodule The zfs.list submodule provides functions for iterating over datasets and properties. Rather than returning tables, these functions act as Lua iterators, and are generally used as follows: .Bd -literal -compact -offset indent .No for child in Fn zfs.list.children \&"rpool" No do ... end .Ed .Pp The available .Sy zfs.list functions are: .Bl -tag -width "xx" .It Fn zfs.list.clones snapshot Iterate through all clones of the given snapshot. .Pp .Bl -tag -compact -width "snapshot (string)" .It Ar snapshot Pq string Must be a valid snapshot path in the current pool. .El .It Fn zfs.list.snapshots dataset Iterate through all snapshots of the given dataset. Each snapshot is returned as a string containing the full dataset name, e.g. "pool/fs@snap". .Pp .Bl -tag -compact -width "snapshot (string)" .It Ar dataset Pq string Must be a valid filesystem or volume. .El .It Fn zfs.list.children dataset Iterate through all direct children of the given dataset. Each child is returned as a string containing the full dataset name, e.g. "pool/fs/child". .Pp .Bl -tag -compact -width "snapshot (string)" .It Ar dataset Pq string Must be a valid filesystem or volume. .El .It Fn zfs.list.bookmarks dataset Iterate through all bookmarks of the given dataset. Each bookmark is returned as a string containing the full dataset name, e.g. "pool/fs#bookmark". .Pp .Bl -tag -compact -width "snapshot (string)" .It Ar dataset Pq string Must be a valid filesystem or volume. .El .It Fn zfs.list.holds snapshot Iterate through all user holds on the given snapshot. Each hold is returned as a pair of the hold's tag and the timestamp (in seconds since the epoch) at which it was created. .Pp .Bl -tag -compact -width "snapshot (string)" .It Ar snapshot Pq string Must be a valid snapshot. .El .It Fn zfs.list.properties dataset An alias for zfs.list.user_properties (see relevant entry). .Pp .Bl -tag -compact -width "snapshot (string)" .It Ar dataset Pq string Must be a valid filesystem, snapshot, or volume. .El .It Fn zfs.list.user_properties dataset Iterate through all user properties for the given dataset. For each step of the iteration, output the property name, its value, and its source. Throws a Lua error if the dataset is invalid. .Pp .Bl -tag -compact -width "snapshot (string)" .It Ar dataset Pq string Must be a valid filesystem, snapshot, or volume. .El .It Fn zfs.list.system_properties dataset Returns an array of strings, the names of the valid system (non-user defined) properties for the given dataset. Throws a Lua error if the dataset is invalid. .Pp .Bl -tag -compact -width "snapshot (string)" .It Ar dataset Pq string Must be a valid filesystem, snapshot or volume. .El .El .El . .Sh EXAMPLES . .Ss Example 1 The following channel program recursively destroys a filesystem and all its snapshots and children in a naive manner. Note that this does not involve any error handling or reporting. .Bd -literal -offset indent function destroy_recursive(root) for child in zfs.list.children(root) do destroy_recursive(child) end for snap in zfs.list.snapshots(root) do zfs.sync.destroy(snap) end zfs.sync.destroy(root) end destroy_recursive("pool/somefs") .Ed . .Ss Example 2 A more verbose and robust version of the same channel program, which properly detects and reports errors, and also takes the dataset to destroy as a command line argument, would be as follows: .Bd -literal -offset indent succeeded = {} failed = {} function destroy_recursive(root) for child in zfs.list.children(root) do destroy_recursive(child) end for snap in zfs.list.snapshots(root) do err = zfs.sync.destroy(snap) if (err ~= 0) then failed[snap] = err else succeeded[snap] = err end end err = zfs.sync.destroy(root) if (err ~= 0) then failed[root] = err else succeeded[root] = err end end args = ... argv = args["argv"] destroy_recursive(argv[1]) results = {} results["succeeded"] = succeeded results["failed"] = failed return results .Ed . .Ss Example 3 The following function performs a forced promote operation by attempting to promote the given clone and destroying any conflicting snapshots. .Bd -literal -offset indent function force_promote(ds) errno, details = zfs.check.promote(ds) if (errno == EEXIST) then assert(details ~= Nil) for i, snap in ipairs(details) do zfs.sync.destroy(ds .. "@" .. snap) end elseif (errno ~= 0) then return errno end return zfs.sync.promote(ds) end .Ed ZFS-PROJECT(8) - System Manager's Manual # NAME **zfs-project** - manage projects in ZFS filesystem # SYNOPSIS **zfs** **project** \[**-d**|**-r**] *file*|*directory*… **zfs** **project** **-C** \[**-kr**] *file*|*directory*… **zfs** **project** **-c** \[**-0**] \[**-d**|**-r**] \[**-p** *id*] *file*|*directory*… **zfs** **project** \[**-p** *id*] \[**-rs**] *file*|*directory*… # DESCRIPTION **zfs** **project** \[**-d**|**-r**] *file*|*directory*… > List project identifier (ID) and inherit flag of files and directories. > **-d** > > Show the directory project ID and inherit flag, not its children. > **-r** > > List subdirectories recursively. **zfs** **project** **-C** \[**-kr**] *file*|*directory*… > Clear project inherit flag and/or ID on the files and directories. > **-k** > > Keep the project ID unchanged. > > If not specified, the project ID will be reset to zero. > **-r** > > Clear subdirectories' flags recursively. **zfs** **project** **-c** \[**-0**] \[**-d**|**-r**] \[**-p** *id*] *file*|*directory*… > Check project ID and inherit flag on the files and directories: > report entries without the project inherit flag, or with project IDs different > from the > target directory's project ID or the one specified with > **-p**. > **-0** > > Delimit filenames with a NUL byte instead of newline, don't output diagnoses. > **-d** > > Check the directory project ID and inherit flag, not its children. > **-p** *id* > > Compare to > > *id* > > instead of the target files and directories' project IDs. > **-r** > > Check subdirectories recursively. **zfs** **project** **-p** *id* \[**-rs**] *file*|*directory*… > Set project ID and/or inherit flag on the files and directories. > **-p** *id* > > Set the project ID to the given value. > **-r** > > Set on subdirectories recursively. > **-s** > > Set project inherit flag on the given files and directories. > > This is usually used for setting up tree quotas with > > **-r**. > > In that case, the directory's project ID > > will be set for all its descendants, unless specified explicitly with > > **-p**. # SEE ALSO zfs-projectspace(8) Debian - July 11, 2022 ZFS-USERSPACE(8) - System Manager's Manual # NAME **zfs-userspace** - display space and quotas of ZFS dataset # SYNOPSIS **zfs** **userspace** \[**-Hinp**] \[**-o** *field*\[,*field*]…] \[**-s** *field*]… \[**-S** *field*]… \[**-t** *type*\[,*type*]…] *filesystem*|*snapshot*|*path* **zfs** **groupspace** \[**-Hinp**] \[**-o** *field*\[,*field*]…] \[**-s** *field*]… \[**-S** *field*]… \[**-t** *type*\[,*type*]…] *filesystem*|*snapshot*|*path* **zfs** **projectspace** \[**-Hp**] \[**-o** *field*\[,*field*]…] \[**-s** *field*]… \[**-S** *field*]… *filesystem*|*snapshot*|*path* # DESCRIPTION **zfs** **userspace** \[**-Hinp**] \[**-o** *field*\[,*field*]…] \[**-s** *field*]… \[**-S** *field*]… \[**-t** *type*\[,*type*]…] *filesystem*|*snapshot*|*path* > Displays space consumed by, and quotas on, each user in the specified > filesystem, > snapshot, or path. > If a path is given, the filesystem that contains that path will be used. > This corresponds to the > **userused@**‌*user*, > **userobjused@**‌*user*, > **userquota@**‌*user*, > and > **userobjquota@**‌*user* > properties. > **-H** > > Do not print headers, use tab-delimited output. > **-S** *field* > > Sort by this field in reverse order. > > See > > **-s**. > **-i** > > Translate SID to POSIX ID. > > The POSIX ID may be ephemeral if no mapping exists. > > Normal POSIX interfaces > > (like stat(2), **ls** **-l**) > > perform this translation, so the > > **-i** > > option allows the output from > > **zfs** **userspace** > > to be compared directly with those utilities. > > However, > > **-i** > > may lead to confusion if some files were created by an SMB user before a > > SMB-to-POSIX name mapping was established. > > In such a case, some files will be owned by the SMB entity and some by the POSIX > > entity. > > However, the > > **-i** > > option will report that the POSIX entity has the total usage and quota for both. > **-n** > > Print numeric ID instead of user/group name. > **-o** *field*\[,*field*]… > > Display only the specified fields from the following set: > > **type**, > > **name**, > > **used**, > > **quota**. > > The default is to display all fields. > **-p** > > Use exact > > (parsable) > > numeric output. > **-s** *field* > > Sort output by this field. > > The > > **-s** > > and > > **-S** > > flags may be specified multiple times to sort first by one field, then by > > another. > > The default is > > **-s** **type** **-s** **name**. > **-t** *type*\[,*type*]… > > Print only the specified types from the following set: > > **all**, > > **posixuser**, > > **smbuser**, > > **posixgroup**, > > **smbgroup**. > > The default is > > **-t** **posixuser**,**smbuser**. > > The default can be changed to include group types. **zfs** **groupspace** \[**-Hinp**] \[**-o** *field*\[,*field*]…] \[**-s** *field*]… \[**-S** *field*]… \[**-t** *type*\[,*type*]…] *filesystem*|*snapshot* > Displays space consumed by, and quotas on, each group in the specified > filesystem or snapshot. > This subcommand is identical to > **userspace**, > except that the default types to display are > **-t** **posixgroup**,**smbgroup**. **zfs** **projectspace** \[**-Hp**] \[**-o** *field*\[,*field*]…] \[**-s** *field*]… \[**-S** *field*]… *filesystem*|*snapshot*|*path* > Displays space consumed by, and quotas on, each project in the specified > filesystem or snapshot. > This subcommand is identical to > **userspace**, > except that the project identifier is a numeral, not a name. > So need neither the option > **-i** > for SID to POSIX ID nor > **-n** > for numeric ID, nor > **-t** > for types. # SEE ALSO zfsprops(7), zfs-set(8) Debian - July 11, 2022 ZFS-PROMOTE(8) - System Manager's Manual # NAME **zfs-promote** - promote clone dataset to no longer depend on origin snapshot # SYNOPSIS **zfs** **promote** *clone* # DESCRIPTION The **zfs** **promote** command makes it possible to destroy the dataset that the clone was created from. The clone parent-child dependency relationship is reversed, so that the origin dataset becomes a clone of the specified dataset. The snapshot that was cloned, and any snapshots previous to this snapshot, are now owned by the promoted clone. The space they use moves from the origin dataset to the promoted clone, so enough space must be available to accommodate these snapshots. No new space is consumed by this operation, but the space accounting is adjusted. The promoted clone must not have any conflicting snapshot names of its own. The **zfs** **rename** subcommand can be used to rename any conflicting snapshots. # EXAMPLES ## Example 1: Promoting a ZFS Clone The following commands illustrate how to test out changes to a file system, and then replace the original file system with the changed one, using clones, clone promotion, and renaming: # **zfs** **create** *pool/project/production* populate /pool/project/production with data # **zfs** **snapshot** *pool/project/production*@*today* # **zfs** **clone** *pool/project/production@today pool/project/beta* make changes to /pool/project/beta and test them # **zfs** **promote** *pool/project/beta* # **zfs** **rename** *pool/project/production pool/project/legacy* # **zfs** **rename** *pool/project/beta pool/project/production* once the legacy version is no longer needed, it can be destroyed # **zfs** **destroy** *pool/project/legacy* # SEE ALSO zfs-clone(8), zfs-rename(8) Debian - July 11, 2022 ZFS-RECEIVE(8) - System Manager's Manual # NAME **zfs-receive** - create snapshot from backup stream # SYNOPSIS **zfs** **receive** \[**-FhMnsuv**] \[**-o** **origin**=*snapshot*] \[**-o** *property*=*value*] \[**-x** *property*] *filesystem*|*volume*|*snapshot* **zfs** **receive** \[**-FhMnsuv**] \[**-d**|**-e**] \[**-o** **origin**=*snapshot*] \[**-o** *property*=*value*] \[**-x** *property*] *filesystem* **zfs** **receive** **-A** *filesystem*|*volume* **zfs** **receive** **-c** \[**-vn**] *filesystem*|*snapshot* # DESCRIPTION **zfs** **receive** \[**-FhMnsuv**] \[**-o** **origin**=*snapshot*] \[**-o** *property*=*value*] \[**-x** *property*] *filesystem*|*volume*|*snapshot* **zfs** **receive** \[**-FhMnsuv**] \[**-d**|**-e**] \[**-o** **origin**=*snapshot*] \[**-o** *property*=*value*] \[**-x** *property*] *filesystem* > Creates a snapshot whose contents are as specified in the stream provided on > standard input. > If a full stream is received, then a new file system is created as well. > Streams are created using the > **zfs** **send** > subcommand, which by default creates a full stream. > **zfs** **recv** > can be used as an alias for > **zfs** **receive**. > If an incremental stream is received, then the destination file system must > already exist, and its most recent snapshot must match the incremental stream's > source. > For > **zvols**, > the destination device link is destroyed and recreated, which means the > **zvol** > cannot be accessed during the > **receive** > operation. > When a snapshot replication package stream that is generated by using the > **zfs** **send** **-R** > command is received, any snapshots that do not exist on the sending location are > destroyed by using the > **zfs** **destroy** **-d** > command. > The ability to send and receive deduplicated send streams has been removed. > However, a deduplicated send stream created with older software can be converted > to a regular (non-deduplicated) stream by using the > **zstream** **redup** > command. > If > **-o** *property*=*value* > or > **-x** *property* > is specified, it applies to the effective value of the property throughout > the entire subtree of replicated datasets. > Effective property values will be set > (**-o**) > or inherited > (**-x**) > on the topmost in the replicated subtree. > In descendant datasets, if the > property is set by the send stream, it will be overridden by forcing the > property to be inherited from the top‐most file system. > Received properties are retained in spite of being overridden > and may be restored with > **zfs** **inherit** **-S**. > Specifying > **-o** **origin**=*snapshot* > is a special case because, even if > **origin** > is a read-only property and cannot be set, it's allowed to receive the send > stream as a clone of the given snapshot. > Raw encrypted send streams (created with > **zfs** **send** **-w**) > may only be received as is, and cannot be re-encrypted, decrypted, or > recompressed by the receive process. > Unencrypted streams can be received as > encrypted datasets, either through inheritance or by specifying encryption > parameters with the > **-o** > options. > Note that the > **keylocation** > property cannot be overridden to > **prompt** > during a receive. > This is because the receive process itself is already using > the standard input for the send stream. > Instead, the property can be overridden after the receive completes. > The added security provided by raw sends adds some restrictions to the send > and receive process. > ZFS will not allow a mix of raw receives and non-raw receives. > Specifically, any raw incremental receives that are attempted after > a non-raw receive will fail. > Non-raw receives do not have this restriction and, > therefore, are always possible. > Because of this, it is best practice to always > use either raw sends for their security benefits or non-raw sends for their > flexibility when working with encrypted datasets, but not a combination. > The reason for this restriction stems from the inherent restrictions of the > AEAD ciphers that ZFS uses to encrypt data. > When using ZFS native encryption, > each block of data is encrypted against a randomly generated number known as > the "initialization vector" (IV), which is stored in the filesystem metadata. > This number is required by the encryption algorithms whenever the data is to > be decrypted. > Together, all of the IVs provided for all of the blocks in a > given snapshot are collectively called an "IV set". > When ZFS performs a raw send, the IV set is transferred from the source > to the destination in the send stream. > When ZFS performs a non-raw send, the data is decrypted by the source > system and re-encrypted by the destination system, creating a snapshot with > effectively the same data, but a different IV set. > In order for decryption to work after a raw send, ZFS must ensure that > the IV set used on both the source and destination side match. > When an incremental raw receive is performed on > top of an existing snapshot, ZFS will check to confirm that the "from" > snapshot on both the source and destination were using the same IV set, > ensuring the new IV set is consistent. > Note that receiving a non-raw stream on top of a raw received file system is > allowed, but replaces the IV set of the newly received snapshot with a > different one. > Any subsequent raw incremental receive based on such a snapshot will > therefore fail with an "IV set guid mismatch" error. > Raw replication can be resumed by rolling the destination back to the most > recent snapshot that was received raw and taking the raw incremental from > there, or by receiving a new full raw stream. > The name of the snapshot > (and file system, if a full stream is received) > that this subcommand creates depends on the argument type and the use of the > **-d** > or > **-e** > options. > If the argument is a snapshot name, the specified > *snapshot* > is created. > If the argument is a file system or volume name, a snapshot with the same name > as the sent snapshot is created within the specified > *filesystem* > or > *volume*. > If neither of the > **-d** > or > **-e** > options are specified, the provided target snapshot name is used exactly as > provided. > The > **-d** > and > **-e** > options cause the file system name of the target snapshot to be determined by > appending a portion of the sent snapshot's name to the specified target > *filesystem*. > If the > **-d** > option is specified, all but the first element of the sent snapshot's file > system path > (usually the pool name) > is used and any required intermediate file systems within the specified one are > created. > If the > **-e** > option is specified, then only the last element of the sent snapshot's file > system name > (i.e. the name of the source file system itself) > is used as the target file system name. > **-F** > > Force a rollback of the file system to the most recent snapshot before > > performing the receive operation. > > If receiving an incremental replication stream > > (for example, one generated by > > **zfs** **send** **-R** \[**-i**|**-I**]) > > , > > destroy snapshots and file systems that do not exist on the sending side. > **-d** > > Discard the first element of the sent snapshot's file system name, using the > > remaining elements to determine the name of the target file system for the new > > snapshot as described in the paragraph above. > **-e** > > Discard all but the last element of the sent snapshot's file system name, using > > that element to determine the name of the target file system for the new > > snapshot as described in the paragraph above. > **-h** > > Skip the receive of holds. > > There is no effect if holds are not sent. > **-M** > > Force an unmount of the file system while receiving a snapshot. > > This option is not supported on Linux. > **-n** > > Do not actually receive the stream. > > This can be useful in conjunction with the > > **-v** > > option to verify the name the receive operation would use. > **-o** **origin**=*snapshot* > > Forces the stream to be received as a clone of the given snapshot. > > If the stream is a full send stream, this will create the filesystem > > described by the stream as a clone of the specified snapshot. > > Which snapshot was specified will not affect the success or failure of the > > receive, as long as the snapshot does exist. > > If the stream is an incremental send stream, all the normal verification will be > > performed. > **-o** *property*=*value* > > Sets the specified property as if the command > > **zfs** **set** *property*=*value* > > was invoked immediately before the receive. > > When receiving a stream from > > **zfs** **send** **-R**, > > causes the property to be inherited by all descendant datasets, as through > > **zfs** **inherit** *property* > > was run on any descendant datasets that have this property set on the > > sending system. > > If the send stream was sent with > > **-c** > > then overriding the > > **compression** > > property will have no effect on received data but the > > **compression** > > property will be set. > > To have the data recompressed on receive remove the > > **-c** > > flag from the send stream. > > Any editable property can be set at receive time. > > Set-once properties bound > > to the received data, such as > > **normalization** > > and > > **casesensitivity**, > > cannot be set at receive time even when the datasets are newly created by > > **zfs** **receive**. > > Additionally both settable properties > > **version** > > and > > **volsize** > > cannot be set at receive time. > > The > > **-o** > > option may be specified multiple times, for different properties. > > An error results if the same property is specified in multiple > > **-o** > > or > > **-x** > > options. > > The > > **-o** > > option may also be used to override encryption properties upon initial receive. > > This allows unencrypted streams to be received as encrypted datasets. > > To cause the received dataset (or root dataset of a recursive stream) to be > > received as an encryption root, specify encryption properties in the same > > manner as is required for > > **zfs** **create**. > > For instance: > > > \# **zfs** **send** *tank/test@snap1* | **zfs** **recv** **-o** **encryption**=**on** **-o** **keyformat**=**passphrase** **-o** **keylocation**=*file:///path/to/keyfile* > > Note that > > **-o** **keylocation**=**prompt** > > may not be specified here, since the standard input > > is already being utilized for the send stream. > > Once the receive has completed, you can use > > **zfs** **set** > > to change this setting after the fact. > > Similarly, you can receive a dataset as an encrypted child by specifying > > **-x** **encryption** > > to force the property to be inherited. > > Overriding encryption properties (except for > > **keylocation**) > > is not possible with raw send streams. > **-s** > > If the receive is interrupted, save the partially received state, rather > > than deleting it. > > Interruption may be due to premature termination of the stream > > (e.g. due to network failure or failure of the remote system > > if the stream is being read over a network connection) > > , > > a checksum error in the stream, termination of the > > **zfs** **receive** > > process, or unclean shutdown of the system. > > The receive can be resumed with a stream generated by > > **zfs** **send** **-t** *token*, > > where the > > *token* > > is the value of the > > **receive\_resume\_token** > > property of the filesystem or volume which is received into. > > To use this flag, the storage pool must have the > > **extensible\_dataset** > > feature enabled. > > See > > zpool-features(7) > > for details on ZFS feature flags. > **-u** > > File system that is associated with the received stream is not mounted. > **-v** > > Print verbose information about the stream and the time required to perform the > > receive operation. > **-x** *property* > > Ensures that the effective value of the specified property after the > > receive is unaffected by the value of that property in the send stream (if any), > > as if the property had been excluded from the send stream. > > If the specified property is not present in the send stream, this option does > > nothing. > > If a received property needs to be overridden, the effective value will be > > set or inherited, depending on whether the property is inheritable or not. > > In the case of an incremental update, > > **-x** > > leaves any existing local setting or explicit inheritance unchanged. > > All > > **-o** > > restrictions (e.g. set-once) apply equally to > > **-x**. **zfs** **receive** **-A** *filesystem*|*volume* > Abort an interrupted > **zfs** **receive** **-s**, > deleting its saved partially received state. **zfs** **receive** **-c** \[**-vn**] *filesystem*|*snapshot* > Attempt to repair data corruption in the specified dataset, > by using the provided stream as the source of healthy data. > This method of healing can only heal data blocks present in the stream. > Metadata can not be healed by corrective receive. > Running a scrub is recommended post-healing to ensure all data corruption was > repaired. > It's important to consider why corruption has happened in the first place. > If you have slowly failing hardware - periodically repairing the data > is not going to save you from data loss later on when the hardware fails > completely. # EXAMPLES ## Example 1: Remotely Replicating ZFS Data The following commands send a full stream and then an incremental stream to a remote machine, restoring them into *poolB/received/fs@a* and *poolB/received/fs@b*, respectively. *poolB* must contain the file system *poolB/received*, and must not initially contain *poolB/received/fs*. # **zfs** **send** *pool/fs@a* | **ssh** *host* **zfs** **receive** *poolB/received/fs*@*a* # **zfs** **send** **-i** *a pool/fs@b* | **ssh** *host* **zfs** **receive** *poolB/received/fs* ## Example 2: Using the **zfs** **receive** **-d** Option The following command sends a full stream of *poolA/fsA/fsB@snap* to a remote machine, receiving it into *poolB/received/fsA/fsB@snap*. The *fsA/fsB@snap* portion of the received snapshot's name is determined from the name of the sent snapshot. *poolB* must contain the file system *poolB/received*. If *poolB/received/fsA* does not exist, it is created as an empty file system. # **zfs** **send** *poolA/fsA/fsB@snap* | **ssh** *host* **zfs** **receive** **-d** *poolB/received* # SEE ALSO zfs-send(8), zstream(8) Debian - March 12, 2023 ZFS-RECEIVE(8) - System Manager's Manual # NAME **zfs-receive** - create snapshot from backup stream # SYNOPSIS **zfs** **receive** \[**-FhMnsuv**] \[**-o** **origin**=*snapshot*] \[**-o** *property*=*value*] \[**-x** *property*] *filesystem*|*volume*|*snapshot* **zfs** **receive** \[**-FhMnsuv**] \[**-d**|**-e**] \[**-o** **origin**=*snapshot*] \[**-o** *property*=*value*] \[**-x** *property*] *filesystem* **zfs** **receive** **-A** *filesystem*|*volume* **zfs** **receive** **-c** \[**-vn**] *filesystem*|*snapshot* # DESCRIPTION **zfs** **receive** \[**-FhMnsuv**] \[**-o** **origin**=*snapshot*] \[**-o** *property*=*value*] \[**-x** *property*] *filesystem*|*volume*|*snapshot* **zfs** **receive** \[**-FhMnsuv**] \[**-d**|**-e**] \[**-o** **origin**=*snapshot*] \[**-o** *property*=*value*] \[**-x** *property*] *filesystem* > Creates a snapshot whose contents are as specified in the stream provided on > standard input. > If a full stream is received, then a new file system is created as well. > Streams are created using the > **zfs** **send** > subcommand, which by default creates a full stream. > **zfs** **recv** > can be used as an alias for > **zfs** **receive**. > If an incremental stream is received, then the destination file system must > already exist, and its most recent snapshot must match the incremental stream's > source. > For > **zvols**, > the destination device link is destroyed and recreated, which means the > **zvol** > cannot be accessed during the > **receive** > operation. > When a snapshot replication package stream that is generated by using the > **zfs** **send** **-R** > command is received, any snapshots that do not exist on the sending location are > destroyed by using the > **zfs** **destroy** **-d** > command. > The ability to send and receive deduplicated send streams has been removed. > However, a deduplicated send stream created with older software can be converted > to a regular (non-deduplicated) stream by using the > **zstream** **redup** > command. > If > **-o** *property*=*value* > or > **-x** *property* > is specified, it applies to the effective value of the property throughout > the entire subtree of replicated datasets. > Effective property values will be set > (**-o**) > or inherited > (**-x**) > on the topmost in the replicated subtree. > In descendant datasets, if the > property is set by the send stream, it will be overridden by forcing the > property to be inherited from the top‐most file system. > Received properties are retained in spite of being overridden > and may be restored with > **zfs** **inherit** **-S**. > Specifying > **-o** **origin**=*snapshot* > is a special case because, even if > **origin** > is a read-only property and cannot be set, it's allowed to receive the send > stream as a clone of the given snapshot. > Raw encrypted send streams (created with > **zfs** **send** **-w**) > may only be received as is, and cannot be re-encrypted, decrypted, or > recompressed by the receive process. > Unencrypted streams can be received as > encrypted datasets, either through inheritance or by specifying encryption > parameters with the > **-o** > options. > Note that the > **keylocation** > property cannot be overridden to > **prompt** > during a receive. > This is because the receive process itself is already using > the standard input for the send stream. > Instead, the property can be overridden after the receive completes. > The added security provided by raw sends adds some restrictions to the send > and receive process. > ZFS will not allow a mix of raw receives and non-raw receives. > Specifically, any raw incremental receives that are attempted after > a non-raw receive will fail. > Non-raw receives do not have this restriction and, > therefore, are always possible. > Because of this, it is best practice to always > use either raw sends for their security benefits or non-raw sends for their > flexibility when working with encrypted datasets, but not a combination. > The reason for this restriction stems from the inherent restrictions of the > AEAD ciphers that ZFS uses to encrypt data. > When using ZFS native encryption, > each block of data is encrypted against a randomly generated number known as > the "initialization vector" (IV), which is stored in the filesystem metadata. > This number is required by the encryption algorithms whenever the data is to > be decrypted. > Together, all of the IVs provided for all of the blocks in a > given snapshot are collectively called an "IV set". > When ZFS performs a raw send, the IV set is transferred from the source > to the destination in the send stream. > When ZFS performs a non-raw send, the data is decrypted by the source > system and re-encrypted by the destination system, creating a snapshot with > effectively the same data, but a different IV set. > In order for decryption to work after a raw send, ZFS must ensure that > the IV set used on both the source and destination side match. > When an incremental raw receive is performed on > top of an existing snapshot, ZFS will check to confirm that the "from" > snapshot on both the source and destination were using the same IV set, > ensuring the new IV set is consistent. > Note that receiving a non-raw stream on top of a raw received file system is > allowed, but replaces the IV set of the newly received snapshot with a > different one. > Any subsequent raw incremental receive based on such a snapshot will > therefore fail with an "IV set guid mismatch" error. > Raw replication can be resumed by rolling the destination back to the most > recent snapshot that was received raw and taking the raw incremental from > there, or by receiving a new full raw stream. > The name of the snapshot > (and file system, if a full stream is received) > that this subcommand creates depends on the argument type and the use of the > **-d** > or > **-e** > options. > If the argument is a snapshot name, the specified > *snapshot* > is created. > If the argument is a file system or volume name, a snapshot with the same name > as the sent snapshot is created within the specified > *filesystem* > or > *volume*. > If neither of the > **-d** > or > **-e** > options are specified, the provided target snapshot name is used exactly as > provided. > The > **-d** > and > **-e** > options cause the file system name of the target snapshot to be determined by > appending a portion of the sent snapshot's name to the specified target > *filesystem*. > If the > **-d** > option is specified, all but the first element of the sent snapshot's file > system path > (usually the pool name) > is used and any required intermediate file systems within the specified one are > created. > If the > **-e** > option is specified, then only the last element of the sent snapshot's file > system name > (i.e. the name of the source file system itself) > is used as the target file system name. > **-F** > > Force a rollback of the file system to the most recent snapshot before > > performing the receive operation. > > If receiving an incremental replication stream > > (for example, one generated by > > **zfs** **send** **-R** \[**-i**|**-I**]) > > , > > destroy snapshots and file systems that do not exist on the sending side. > **-d** > > Discard the first element of the sent snapshot's file system name, using the > > remaining elements to determine the name of the target file system for the new > > snapshot as described in the paragraph above. > **-e** > > Discard all but the last element of the sent snapshot's file system name, using > > that element to determine the name of the target file system for the new > > snapshot as described in the paragraph above. > **-h** > > Skip the receive of holds. > > There is no effect if holds are not sent. > **-M** > > Force an unmount of the file system while receiving a snapshot. > > This option is not supported on Linux. > **-n** > > Do not actually receive the stream. > > This can be useful in conjunction with the > > **-v** > > option to verify the name the receive operation would use. > **-o** **origin**=*snapshot* > > Forces the stream to be received as a clone of the given snapshot. > > If the stream is a full send stream, this will create the filesystem > > described by the stream as a clone of the specified snapshot. > > Which snapshot was specified will not affect the success or failure of the > > receive, as long as the snapshot does exist. > > If the stream is an incremental send stream, all the normal verification will be > > performed. > **-o** *property*=*value* > > Sets the specified property as if the command > > **zfs** **set** *property*=*value* > > was invoked immediately before the receive. > > When receiving a stream from > > **zfs** **send** **-R**, > > causes the property to be inherited by all descendant datasets, as through > > **zfs** **inherit** *property* > > was run on any descendant datasets that have this property set on the > > sending system. > > If the send stream was sent with > > **-c** > > then overriding the > > **compression** > > property will have no effect on received data but the > > **compression** > > property will be set. > > To have the data recompressed on receive remove the > > **-c** > > flag from the send stream. > > Any editable property can be set at receive time. > > Set-once properties bound > > to the received data, such as > > **normalization** > > and > > **casesensitivity**, > > cannot be set at receive time even when the datasets are newly created by > > **zfs** **receive**. > > Additionally both settable properties > > **version** > > and > > **volsize** > > cannot be set at receive time. > > The > > **-o** > > option may be specified multiple times, for different properties. > > An error results if the same property is specified in multiple > > **-o** > > or > > **-x** > > options. > > The > > **-o** > > option may also be used to override encryption properties upon initial receive. > > This allows unencrypted streams to be received as encrypted datasets. > > To cause the received dataset (or root dataset of a recursive stream) to be > > received as an encryption root, specify encryption properties in the same > > manner as is required for > > **zfs** **create**. > > For instance: > > > \# **zfs** **send** *tank/test@snap1* | **zfs** **recv** **-o** **encryption**=**on** **-o** **keyformat**=**passphrase** **-o** **keylocation**=*file:///path/to/keyfile* > > Note that > > **-o** **keylocation**=**prompt** > > may not be specified here, since the standard input > > is already being utilized for the send stream. > > Once the receive has completed, you can use > > **zfs** **set** > > to change this setting after the fact. > > Similarly, you can receive a dataset as an encrypted child by specifying > > **-x** **encryption** > > to force the property to be inherited. > > Overriding encryption properties (except for > > **keylocation**) > > is not possible with raw send streams. > **-s** > > If the receive is interrupted, save the partially received state, rather > > than deleting it. > > Interruption may be due to premature termination of the stream > > (e.g. due to network failure or failure of the remote system > > if the stream is being read over a network connection) > > , > > a checksum error in the stream, termination of the > > **zfs** **receive** > > process, or unclean shutdown of the system. > > The receive can be resumed with a stream generated by > > **zfs** **send** **-t** *token*, > > where the > > *token* > > is the value of the > > **receive\_resume\_token** > > property of the filesystem or volume which is received into. > > To use this flag, the storage pool must have the > > **extensible\_dataset** > > feature enabled. > > See > > zpool-features(7) > > for details on ZFS feature flags. > **-u** > > File system that is associated with the received stream is not mounted. > **-v** > > Print verbose information about the stream and the time required to perform the > > receive operation. > **-x** *property* > > Ensures that the effective value of the specified property after the > > receive is unaffected by the value of that property in the send stream (if any), > > as if the property had been excluded from the send stream. > > If the specified property is not present in the send stream, this option does > > nothing. > > If a received property needs to be overridden, the effective value will be > > set or inherited, depending on whether the property is inheritable or not. > > In the case of an incremental update, > > **-x** > > leaves any existing local setting or explicit inheritance unchanged. > > All > > **-o** > > restrictions (e.g. set-once) apply equally to > > **-x**. **zfs** **receive** **-A** *filesystem*|*volume* > Abort an interrupted > **zfs** **receive** **-s**, > deleting its saved partially received state. **zfs** **receive** **-c** \[**-vn**] *filesystem*|*snapshot* > Attempt to repair data corruption in the specified dataset, > by using the provided stream as the source of healthy data. > This method of healing can only heal data blocks present in the stream. > Metadata can not be healed by corrective receive. > Running a scrub is recommended post-healing to ensure all data corruption was > repaired. > It's important to consider why corruption has happened in the first place. > If you have slowly failing hardware - periodically repairing the data > is not going to save you from data loss later on when the hardware fails > completely. # EXAMPLES ## Example 1: Remotely Replicating ZFS Data The following commands send a full stream and then an incremental stream to a remote machine, restoring them into *poolB/received/fs@a* and *poolB/received/fs@b*, respectively. *poolB* must contain the file system *poolB/received*, and must not initially contain *poolB/received/fs*. # **zfs** **send** *pool/fs@a* | **ssh** *host* **zfs** **receive** *poolB/received/fs*@*a* # **zfs** **send** **-i** *a pool/fs@b* | **ssh** *host* **zfs** **receive** *poolB/received/fs* ## Example 2: Using the **zfs** **receive** **-d** Option The following command sends a full stream of *poolA/fsA/fsB@snap* to a remote machine, receiving it into *poolB/received/fsA/fsB@snap*. The *fsA/fsB@snap* portion of the received snapshot's name is determined from the name of the sent snapshot. *poolB* must contain the file system *poolB/received*. If *poolB/received/fsA* does not exist, it is created as an empty file system. # **zfs** **send** *poolA/fsA/fsB@snap* | **ssh** *host* **zfs** **receive** **-d** *poolB/received* # SEE ALSO zfs-send(8), zstream(8) Debian - March 12, 2023 ZFS-SEND(8) - System Manager's Manual # NAME **zfs-send** - generate backup stream of ZFS dataset # SYNOPSIS **zfs** **send** \[**-DLPUVbcehnpsvw**] \[**-R** \[**-X** *dataset*\[,*dataset*]…]] \[\[**-I**|**-i**] *snapshot*] *snapshot* **zfs** **send** \[**-DLPUVcensvw**] \[**-i** *snapshot*|*bookmark*] *filesystem*|*volume*|*snapshot* **zfs** **send** **--redact** *redaction\_bookmark* \[**-DLPVcenpv**] \[**-i** *snapshot*|*bookmark*] *snapshot* **zfs** **send** \[**-PVenv**] **-t** *receive\_resume\_token* **zfs** **send** \[**-PVnv**] **-S** *filesystem* **zfs** **redact** *snapshot redaction\_bookmark* *redaction\_snapshot*… # DESCRIPTION **zfs** **send** \[**-DLPUVbcehnpsvw**] \[**-R** \[**-X** *dataset*\[,*dataset*]…]] \[\[**-I**|**-i**] *snapshot*] *snapshot* > Creates a stream representation of the second > *snapshot*, > which is written to standard output. > The output can be redirected to a file or to a different system > (for example, using > ssh(1)) > . > By default, a full stream is generated. > **-D**, **--dedup** > > Deduplicated send is no longer supported. > > This flag is accepted for backwards compatibility, but a regular, > > non-deduplicated stream will be generated. > **-I** *snapshot* > > Generate a stream package that sends all intermediary snapshots from the first > > snapshot to the second snapshot. > > For example, > > **-I** *@a* *fs@d* > > is similar to > > **-i** *@a* *fs@b*; **-i** *@b* *fs@c*; **-i** *@c* *fs@d*. > > The incremental source may be specified as with the > > **-i** > > option. > **-L**, **--large-block** > > Generate a stream which may contain blocks larger than 128 KiB. > > This flag has no effect if the > > **large\_blocks** > > pool feature is disabled, or if the > > **recordsize** > > property of this filesystem has never been set above 128 KiB. > > The receiving system must have the > > **large\_blocks** > > pool feature enabled as well. > > This flag is required if the > > **large\_microzap** > > pool feature is active. > > See > > zpool-features(7) > > for details on ZFS feature flags and the > > **large\_blocks** > > feature. > **-P**, **--parsable** > > Print machine-parsable verbose information about the stream package generated. > **-R**, **--replicate** > > Generate a replication stream package, which will replicate the specified > > file system, and all descendent file systems, up to the named snapshot. > > When received, all properties, snapshots, descendent file systems, and clones > > are preserved. > > If the > > **-i** > > or > > **-I** > > flags are used in conjunction with the > > **-R** > > flag, an incremental replication stream is generated. > > The current values of properties, and current snapshot and file system names are > > set when the stream is received. > > If the > > **-F** > > flag is specified when this stream is received, snapshots and file systems that > > do not exist on the sending side are destroyed. > > If the > > **-R** > > flag is used to send encrypted datasets, then > > **-w** > > or > > **-U** > > must also be specified. > **-V**, **--proctitle** > > Set the process title to a per-second report of how much data has been sent. > **-X**, **--exclude** *dataset*\[,*dataset*]… > > With > > **-R**, > > **-X** > > specifies a set of datasets (and, hence, their descendants), > > to be excluded from the send stream. > > The root dataset may not be excluded. > > **-X** *a* **-X** *b* > > is equivalent to > > **-X** *a*,*b*. > **-e**, **--embed** > > Generate a more compact stream by using > > **WRITE\_EMBEDDED** > > records for blocks which are stored more compactly on disk by the > > **embedded\_data** > > pool feature. > > This flag has no effect if the > > **embedded\_data** > > feature is disabled. > > The receiving system must have the > > **embedded\_data** > > feature enabled. > > If the > > **lz4\_compress** > > or > > **zstd\_compress** > > features are active on the sending system, then the receiving system must have > > the corresponding features enabled as well. > > Datasets that are sent with this flag may not be > > received as an encrypted dataset, since encrypted datasets cannot use the > > **embedded\_data** > > feature. > > See > > zpool-features(7) > > for details on ZFS feature flags and the > > **embedded\_data** > > feature. > **-b**, **--backup** > > Sends only received property values whether or not they are overridden by local > > settings, but only if the dataset has ever been received. > > Use this option when you want > > **zfs** **receive** > > to restore received properties backed up on the sent dataset and to avoid > > sending local settings that may have nothing to do with the source dataset, > > but only with how the data is backed up. > **-c**, **--compressed** > > Generate a more compact stream by using compressed WRITE records for blocks > > which are compressed on disk and in memory > > (see the > > **compression** > > property for details) > > . > > If the > > **lz4\_compress** > > or > > **zstd\_compress** > > features are active on the sending system, then the receiving system must have > > the corresponding features enabled as well. > > If the > > **large\_blocks** > > feature is enabled on the sending system but the > > **-L** > > option is not supplied in conjunction with > > **-c**, > > then the data will be decompressed before sending so it can be split into > > smaller block sizes. > > Streams sent with > > **-c** > > will not have their data recompressed on the receiver side using > > **-o** **compress**= *value*. > > The data will stay compressed as it was from the sender. > > The new compression property will be set for future data. > > Note that uncompressed data from the sender will still attempt to > > compress on the receiver, unless you specify > > **-o** **compress**= *off*. > **-w**, **--raw** > > For encrypted datasets, send data exactly as it exists on disk. > > This allows backups to be taken even if encryption keys are not currently > > loaded. > > The backup may then be received on an untrusted machine since that machine will > > not have the encryption keys to read the protected data or alter it without > > being detected. > > Upon being received, the dataset will have the same encryption > > keys as it did on the send side, although the > > **keylocation** > > property will be defaulted to > > **prompt** > > if not otherwise provided. > > For unencrypted datasets, this flag will be equivalent to > > **-Lec**. > > Note that if you do not use this flag for sending encrypted datasets, data will > > be sent unencrypted and may be re-encrypted with a different encryption key on > > the receiving system, which will disable the ability to do a raw send to that > > system for incrementals. > **-h**, **--holds** > > Generate a stream package that includes any snapshot holds (created with the > > **zfs** **hold** > > command), and indicating to > > **zfs** **receive** > > that the holds be applied to the dataset on the receiving system. > **-i** *snapshot* > > Generate an incremental stream from the first > > *snapshot* > > (the incremental source) > > to the second > > *snapshot* > > (the incremental target). > > The incremental source can be specified as the last component of the snapshot > > name > > (the > > **@** > > character and following) > > and it is assumed to be from the same file system as the incremental target. > > If the destination is a clone, the source may be the origin snapshot, which must > > be fully specified > > (for example, > > *pool/fs@origin*, > > not just > > *@origin*) > > . > **-n**, **--dryrun** > > Do a dry-run > > ("No-op") > > send. > > Do not generate any actual send data. > > This is useful in conjunction with the > > **-v** > > or > > **-P** > > flags to determine what data will be sent. > > In this case, the verbose output will be written to standard output > > (contrast with a non-dry-run, where the stream is written to standard output > > and the verbose output goes to standard error) > > . > **-p**, **--props** > > Include the dataset's properties in the stream. > > This flag is implicit when > > **-R** > > is specified. > > The receiving system must also support this feature. > > Sends of encrypted datasets must use > > **-w** > > or > > **-U** > > when using this flag. > **-s**, **--skip-missing** > > Allows sending a replication stream even when there are snapshots missing in the > > hierarchy. > > When a snapshot is missing, instead of throwing an error and aborting the send, > > a warning is printed to the standard error stream and the dataset to which it > > belongs > > and its descendants are skipped. > > This flag can only be used in conjunction with > > **-R**. > **-U**, **--no-preserve-encryption** > > Allow sending an encrypted dataset with properties, but without keeping > > encryption. > > When this flag is specified, encrypted datasets that would otherwise be blocked > > from sending are sent as unencrypted data. > **-v**, **--verbose** > > Print verbose information about the stream package generated. > > This information includes a per-second report of how much data has been sent. > > The same report can be requested by sending > > `SIGINFO` > > or > > `SIGUSR1`, > > regardless of > > **-v**. > > The format of the stream is committed. > > You will be able to receive your streams on future versions of ZFS. **zfs** **send** \[**-DLPVcenvw**] \[**-i** *snapshot*|*bookmark*] *filesystem*|*volume*|*snapshot* > Generate a send stream, which may be of a filesystem, and may be incremental > from a bookmark. > If the destination is a filesystem or volume, the pool must be read-only, or the > filesystem must not be mounted. > When the stream generated from a filesystem or volume is received, the default > snapshot name will be > "--head--". > **-D**, **--dedup** > > Deduplicated send is no longer supported. > > This flag is accepted for backwards compatibility, but a regular, > > non-deduplicated stream will be generated. > **-L**, **--large-block** > > Generate a stream which may contain blocks larger than 128 KiB. > > This flag has no effect if the > > **large\_blocks** > > pool feature is disabled, or if the > > **recordsize** > > property of this filesystem has never been set above 128 KiB. > > The receiving system must have the > > **large\_blocks** > > pool feature enabled as well. > > See > > zpool-features(7) > > for details on ZFS feature flags and the > > **large\_blocks** > > feature. > **-P**, **--parsable** > > Print machine-parsable verbose information about the stream package generated. > **-c**, **--compressed** > > Generate a more compact stream by using compressed WRITE records for blocks > > which are compressed on disk and in memory > > (see the > > **compression** > > property for details) > > . > > If the > > **lz4\_compress** > > or > > **zstd\_compress** > > features are active on the sending system, then the receiving system must have > > the corresponding features enabled as well. > > If the > > **large\_blocks** > > feature is enabled on the sending system but the > > **-L** > > option is not supplied in conjunction with > > **-c**, > > then the data will be decompressed before sending so it can be split into > > smaller block sizes. > **-w**, **--raw** > > For encrypted datasets, send data exactly as it exists on disk. > > This allows backups to be taken even if encryption keys are not currently > > loaded. > > The backup may then be received on an untrusted machine since that machine will > > not have the encryption keys to read the protected data or alter it without > > being detected. > > Upon being received, the dataset will have the same encryption > > keys as it did on the send side, although the > > **keylocation** > > property will be defaulted to > > **prompt** > > if not otherwise provided. > > For unencrypted datasets, this flag will be equivalent to > > **-Lec**. > > Note that if you do not use this flag for sending encrypted datasets, data will > > be sent unencrypted and may be re-encrypted with a different encryption key on > > the receiving system, which will disable the ability to do a raw send to that > > system for incrementals. > **-e**, **--embed** > > Generate a more compact stream by using > > **WRITE\_EMBEDDED** > > records for blocks which are stored more compactly on disk by the > > **embedded\_data** > > pool feature. > > This flag has no effect if the > > **embedded\_data** > > feature is disabled. > > The receiving system must have the > > **embedded\_data** > > feature enabled. > > If the > > **lz4\_compress** > > or > > **zstd\_compress** > > features are active on the sending system, then the receiving system must have > > the corresponding features enabled as well. > > Datasets that are sent with this flag may not be received as an encrypted > > dataset, > > since encrypted datasets cannot use the > > **embedded\_data** > > feature. > > See > > zpool-features(7) > > for details on ZFS feature flags and the > > **embedded\_data** > > feature. > **-i** *snapshot*|*bookmark* > > Generate an incremental send stream. > > The incremental source must be an earlier snapshot in the destination's history. > > It will commonly be an earlier snapshot in the destination's file system, in > > which case it can be specified as the last component of the name > > (the > > **#** > > or > > **@** > > character and following) > > . > > If the incremental target is a clone, the incremental source can be the origin > > snapshot, or an earlier snapshot in the origin's filesystem, or the origin's > > origin, etc. > **-n**, **--dryrun** > > Do a dry-run > > ("No-op") > > send. > > Do not generate any actual send data. > > This is useful in conjunction with the > > **-v** > > or > > **-P** > > flags to determine what data will be sent. > > In this case, the verbose output will be written to standard output > > (contrast with a non-dry-run, where the stream is written to standard output > > and the verbose output goes to standard error) > > . > **-v**, **--verbose** > > Print verbose information about the stream package generated. > > This information includes a per-second report of how much data has been sent. > > The same report can be requested by sending > > `SIGINFO` > > or > > `SIGUSR1`, > > regardless of > > **-v**. **zfs** **send** **--redact** *redaction\_bookmark* \[**-DLPVcenpv**] \[**-i** *snapshot*|*bookmark*] *snapshot* > Generate a redacted send stream. > This send stream contains all blocks from the snapshot being sent that aren't > included in the redaction list contained in the bookmark specified by the > **--redact** > (or > **-d**) > flag. > The resulting send stream is said to be redacted with respect to the snapshots > the bookmark specified by the > **--redact** flag was created with. > The bookmark must have been created by running > **zfs** **redact** > on the snapshot being sent. > This feature can be used to allow clones of a filesystem to be made available on > a remote system, in the case where their parent need not (or needs to not) be > usable. > For example, if a filesystem contains sensitive data, and it has clones where > that sensitive data has been secured or replaced with dummy data, redacted sends > can be used to replicate the secured data without replicating the original > sensitive data, while still sharing all possible blocks. > A snapshot that has been redacted with respect to a set of snapshots will > contain all blocks referenced by at least one snapshot in the set, but will > contain none of the blocks referenced by none of the snapshots in the set. > In other words, if all snapshots in the set have modified a given block in the > parent, that block will not be sent; but if one or more snapshots have not > modified a block in the parent, they will still reference the parent's block, so > that block will be sent. > Note that only user data will be redacted. > When the redacted send stream is received, we will generate a redacted > snapshot. > Due to the nature of redaction, a redacted dataset can only be used in the > following ways: > 1. To receive, as a clone, an incremental send from the original snapshot to one > of the snapshots it was redacted with respect to. > In this case, the stream will produce a valid dataset when received because all > blocks that were redacted in the parent are guaranteed to be present in the > child's send stream. > This use case will produce a normal snapshot, which can be used just like other > snapshots. > 2. To receive an incremental send from the original snapshot to something > redacted with respect to a subset of the set of snapshots the initial snapshot > was redacted with respect to. > In this case, each block that was redacted in the original is still redacted > (redacting with respect to additional snapshots causes less data to be redacted > (because the snapshots define what is permitted, and everything else is > redacted)). > This use case will produce a new redacted snapshot. > 3. To receive an incremental send from a redaction bookmark of the original > snapshot that was created when redacting with respect to a subset of the set of > snapshots the initial snapshot was created with respect to > anything else. > A send stream from such a redaction bookmark will contain all of the blocks > necessary to fill in any redacted data, should it be needed, because the sending > system is aware of what blocks were originally redacted. > This will either produce a normal snapshot or a redacted one, depending on > whether the new send stream is redacted. > 4. To receive an incremental send from a redacted version of the initial > snapshot that is redacted with respect to a subject of the set of snapshots the > initial snapshot was created with respect to. > A send stream from a compatible redacted dataset will contain all of the blocks > necessary to fill in any redacted data. > This will either produce a normal snapshot or a redacted one, depending on > whether the new send stream is redacted. > 5. To receive a full send as a clone of the redacted snapshot. > Since the stream is a full send, it definitionally contains all the data needed > to create a new dataset. > This use case will either produce a normal snapshot or a redacted one, depending > on whether the full send stream was redacted. > These restrictions are detected and enforced by > **zfs** **receive**; > a redacted send stream will contain the list of snapshots that the stream is > redacted with respect to. > These are stored with the redacted snapshot, and are used to detect and > correctly handle the cases above. > Note that for technical reasons, > raw sends and redacted sends cannot be combined at this time. **zfs** **send** \[**-PVenv**] **-t** *receive\_resume\_token* > Creates a send stream which resumes an interrupted receive. > The > *receive\_resume\_token* > is the value of this property on the filesystem or volume that was being > received into. > See the documentation for > **zfs** **receive** **-s** > for more details. **zfs** **send** \[**-PVnv**] \[**-i** *snapshot*|*bookmark*] **-S** *filesystem* > Generate a send stream from a dataset that has been partially received. > **-S**, **--saved** > > This flag requires that the specified filesystem previously received a resumable > > send that did not finish and was interrupted. > > In such scenarios this flag > > enables the user to send this partially received state. > > Using this flag will always use the last fully received snapshot > > as the incremental source if it exists. **zfs** **redact** *snapshot redaction\_bookmark* *redaction\_snapshot*… > Generate a new redaction bookmark. > In addition to the typical bookmark information, a redaction bookmark contains > the list of redacted blocks and the list of redaction snapshots specified. > The redacted blocks are blocks in the snapshot which are not referenced by any > of the redaction snapshots. > These blocks are found by iterating over the metadata in each redaction snapshot > to determine what has been changed since the target snapshot. > Redaction is designed to support redacted zfs sends; see the entry for > **zfs** **send** > for more information on the purpose of this operation. > If a redact operation fails partway through (due to an error or a system > failure), the redaction can be resumed by rerunning the same command. ## Redaction ZFS has support for a limited version of data subsetting, in the form of redaction. Using the **zfs** **redact** command, a **redaction bookmark** can be created that stores a list of blocks containing sensitive information. When provided to **zfs** **send**, this causes a **redacted send** to occur. Redacted sends omit the blocks containing sensitive information, replacing them with REDACT records. When these send streams are received, a **redacted dataset** is created. A redacted dataset cannot be mounted by default, since it is incomplete. It can be used to receive other send streams. In this way datasets can be used for data backup and replication, with all the benefits that zfs send and receive have to offer, while protecting sensitive information from being stored on less-trusted machines or services. For the purposes of redaction, there are two steps to the process. A redact step, and a send/receive step. First, a redaction bookmark is created. This is done by providing the **zfs** **redact** command with a parent snapshot, a bookmark to be created, and a number of redaction snapshots. These redaction snapshots must be descendants of the parent snapshot, and they should modify data that is considered sensitive in some way. Any blocks of data modified by all of the redaction snapshots will be listed in the redaction bookmark, because it represents the truly sensitive information. When it comes to the send step, the send process will not send the blocks listed in the redaction bookmark, instead replacing them with REDACT records. When received on the target system, this will create a redacted dataset, missing the data that corresponds to the blocks in the redaction bookmark on the sending system. The incremental send streams from the original parent to the redaction snapshots can then also be received on the target system, and this will produce a complete snapshot that can be used normally. Incrementals from one snapshot on the parent filesystem and another can also be done by sending from the redaction bookmark, rather than the snapshots themselves. In order to make the purpose of the feature more clear, an example is provided. Consider a zfs filesystem containing four files. These files represent information for an online shopping service. One file contains a list of usernames and passwords, another contains purchase histories, a third contains click tracking data, and a fourth contains user preferences. The owner of this data wants to make it available for their development teams to test against, and their market research teams to do analysis on. The development teams need information about user preferences and the click tracking data, while the market research teams need information about purchase histories and user preferences. Neither needs access to the usernames and passwords. However, because all of this data is stored in one ZFS filesystem, it must all be sent and received together. In addition, the owner of the data wants to take advantage of features like compression, checksumming, and snapshots, so they do want to continue to use ZFS to store and transmit their data. Redaction can help them do so. First, they would make two clones of a snapshot of the data on the source. In one clone, they create the setup they want their market research team to see; they delete the usernames and passwords file, and overwrite the click tracking data with dummy information. In another, they create the setup they want the development teams to see, by replacing the passwords with fake information and replacing the purchase histories with randomly generated ones. They would then create a redaction bookmark on the parent snapshot, using snapshots on the two clones as redaction snapshots. The parent can then be sent, redacted, to the target server where the research and development teams have access. Finally, incremental sends from the parent snapshot to each of the clones can be sent to and received on the target server; these snapshots are identical to the ones on the source, and are ready to be used, while the parent snapshot on the target contains none of the username and password data present on the source, because it was removed by the redacted send operation. # SIGNALS See **-v**. # EXAMPLES ## Example 1: Remotely Replicating ZFS Data The following commands send a full stream and then an incremental stream to a remote machine, restoring them into *poolB/received/fs@a* and *poolB/received/fs@b*, respectively. *poolB* must contain the file system *poolB/received*, and must not initially contain *poolB/received/fs*. # **zfs** **send** *pool/fs@a* | **ssh** *host* **zfs** **receive** *poolB/received/fs*@*a* # **zfs** **send** **-i** *a pool/fs@b* | **ssh** *host* **zfs** **receive** *poolB/received/fs* ## Example 2: Using the **zfs** **receive** **-d** Option The following command sends a full stream of *poolA/fsA/fsB@snap* to a remote machine, receiving it into *poolB/received/fsA/fsB@snap*. The *fsA/fsB@snap* portion of the received snapshot's name is determined from the name of the sent snapshot. *poolB* must contain the file system *poolB/received*. If *poolB/received/fsA* does not exist, it is created as an empty file system. # **zfs** **send** *poolA/fsA/fsB@snap* | **ssh** *host* **zfs** **receive** **-d** *poolB/received* # SEE ALSO zfs-bookmark(8), zfs-receive(8), zfs-redact(8), zfs-snapshot(8) Debian - February 20, 2026 ZFS-HOLD(8) - System Manager's Manual # NAME **zfs-hold** - hold ZFS snapshots to prevent their removal # SYNOPSIS **zfs** **hold** \[**-r**] *tag* *snapshot*… **zfs** **holds** \[**-rHp**] *snapshot*… **zfs** **release** \[**-r**] *tag* *snapshot*… # DESCRIPTION **zfs** **hold** \[**-r**] *tag* *snapshot*… > Adds a single reference, named with the > *tag* > argument, to the specified snapshots. > Each snapshot has its own tag namespace, and tags must be unique within that > space. > If a hold exists on a snapshot, attempts to destroy that snapshot by using the > **zfs** **destroy** > command return > **EBUSY**. > **-r** > > Specifies that a hold with the given tag is applied recursively to the snapshots > > of all descendent file systems. **zfs** **holds** \[**-rHp**] *snapshot*… > Lists all existing user references for the given snapshot or snapshots. > **-r** > > Lists the holds that are set on the named descendent snapshots, in addition to > > listing the holds on the named snapshot. > **-H** > > Do not print headers, use tab-delimited output. > **-p** > > Prints holds timestamps as Unix epoch timestamps. **zfs** **release** \[**-r**] *tag* *snapshot*… > Removes a single reference, named with the > *tag* > argument, from the specified snapshot or snapshots. > The tag must already exist for each snapshot. > If a hold exists on a snapshot, attempts to destroy that snapshot by using the > **zfs** **destroy** > command return > **EBUSY**. > **-r** > > Recursively releases a hold with the given tag on the snapshots of all > > descendent file systems. # SEE ALSO zfs-destroy(8) Debian - November 8, 2022 ZFS-RENAME(8) - System Manager's Manual # NAME **zfs-rename** - rename ZFS dataset # SYNOPSIS **zfs** **rename** \[**-f**] *filesystem*|*volume*|*snapshot* *filesystem*|*volume*|*snapshot* **zfs** **rename** **-p** \[**-f**] *filesystem*|*volume* *filesystem*|*volume* **zfs** **rename** **-u** \[**-f**] *filesystem* *filesystem* **zfs** **rename** **-r** *snapshot* *snapshot* # DESCRIPTION **zfs** **rename** \[**-f**] *filesystem*|*volume*|*snapshot* *filesystem*|*volume*|*snapshot* **zfs** **rename** **-p** \[**-f**] *filesystem*|*volume* *filesystem*|*volume* **zfs** **rename** **-u** \[**-f**] *filesystem* *filesystem* > Renames the given dataset. > The new target can be located anywhere in the ZFS hierarchy, with the exception > of snapshots. > Snapshots can only be renamed within the parent file system or volume. > When renaming a snapshot, the parent file system of the snapshot does not need > to be specified as part of the second argument. > Renamed file systems can inherit new mount points, in which case they are > unmounted and remounted at the new mount point. > **-f** > > Force unmount any file systems that need to be unmounted in the process. > > This flag has no effect if used together with the > > **-u** > > flag. > **-p** > > Creates all the non-existing parent datasets. > > Datasets created in this manner are automatically mounted according to the > > **mountpoint** > > property inherited from their parent. > > Alernatively, if the > > **-p** > > option is specified multiple times, parent datasets are created with > > **canmount**=*off*. > **-u** > > Do not remount file systems during rename. > > If a file system's > > **mountpoint** > > property is set to > > **legacy** > > or > > **none**, > > the file system is not unmounted even if this option is not given. **zfs** **rename** **-r** *snapshot* *snapshot* > Recursively rename the snapshots of all descendent datasets. > Snapshots are the only dataset that can be renamed recursively. # EXAMPLES ## Example 1: Promoting a ZFS Clone The following commands illustrate how to test out changes to a file system, and then replace the original file system with the changed one, using clones, clone promotion, and renaming: # **zfs** **create** *pool/project/production* populate /pool/project/production with data # **zfs** **snapshot** *pool/project/production*@*today* # **zfs** **clone** *pool/project/production@today pool/project/beta* make changes to /pool/project/beta and test them # **zfs** **promote** *pool/project/beta* # **zfs** **rename** *pool/project/production pool/project/legacy* # **zfs** **rename** *pool/project/beta pool/project/production* once the legacy version is no longer needed, it can be destroyed # **zfs** **destroy** *pool/project/legacy* ## Example 2: Performing a Rolling Snapshot The following example shows how to maintain a history of snapshots with a consistent naming scheme. To keep a week's worth of snapshots, the user destroys the oldest snapshot, renames the remaining snapshots, and then creates a new snapshot, as follows: # **zfs** **destroy** **-r** *pool/users@7daysago* # **zfs** **rename** **-r** *pool/users@6daysago* @*7daysago* # **zfs** **rename** **-r** *pool/users@5daysago* @*6daysago* # **zfs** **rename** **-r** *pool/users@4daysago* @*5daysago* # **zfs** **rename** **-r** *pool/users@3daysago* @*4daysago* # **zfs** **rename** **-r** *pool/users@2daysago* @*3daysago* # **zfs** **rename** **-r** *pool/users@yesterday* @*2daysago* # **zfs** **rename** **-r** *pool/users@today* @*yesterday* # **zfs** **snapshot** **-r** *pool/users*@*today* Debian - July 11, 2022 ZFS-REWRITE(8) - System Manager's Manual # NAME **zfs-rewrite** - rewrite specified files without modification # SYNOPSIS **zfs** **rewrite** \[**-CPSrvx**] \[**-l** *length*] \[**-o** *offset*] *file*|*directory*… # DESCRIPTION Rewrite blocks of specified *file* as is without modification at a new location and possibly with new properties, as if they were atomically read and written back. See *NOTES*. for more information about property changes that may be applied during rewrite. **-C** > Skip blocks that are shared via block cloning (BRT). > Cloned blocks are referenced by multiple files or datasets. > Rewriting these blocks would create separate copies and increase space usage. > This flag prevents such expansion by skipping cloned blocks. **-P** > Perform physical rewrite, preserving logical birth time of blocks. > By default, rewrite updates logical birth times, making blocks appear > as modified in snapshots and incremental send streams. > Physical rewrite preserves logical birth times, avoiding unnecessary > inclusion in incremental streams. > Physical rewrite requires the > **physical\_rewrite** > feature to be enabled on the pool. **-S** > Skip blocks that are shared with snapshots. > Blocks created before the most recent snapshot are shared with that snapshot. > Rewriting these blocks would create new copies, leaving the old copies for > the snapshot and increasing space usage. > This flag prevents such expansion by skipping snapshot-shared blocks. **-l** *length* > Rewrite at most this number of bytes. **-o** *offset* > Start at this offset in bytes. **-r** > Recurse into directories. **-v** > Print names of all successfully rewritten files. **-x** > Don't cross file system mount points when recursing. # NOTES Rewrite works by replacing an existing block with a new block of the same logical size. Changed dataset properties that operate on the data or metadata without changing the logical size will be applied. These include **checksum**, **compression**, **dedup** and **copies**. Changes to properties that affect the size of a logical block, like **recordsize**, will have no effect. Rewrite of cloned blocks and blocks that are part of any snapshots, same as some property changes may increase pool space usage. Use the **-C** and **-S** flags to skip cloned and snapshot-shared blocks respectively to prevent this expansion. Holes that were never written or were previously zero-compressed are not rewritten and will remain holes even if compression is disabled. If a **-l** or **-o** value request a rewrite to regions past the end of the file, then those regions are silently ignored, and no error is reported. By default, rewritten blocks update their logical birth time, meaning they will be included in incremental **zfs** **send** streams as modified data. When the **-P** flag is used, rewritten blocks preserve their logical birth time, since there are no user data changes. # SEE ALSO zfsprops(7), zpool-features(7) Debian - November 5, 2025 ZFS-ROLLBACK(8) - System Manager's Manual # NAME **zfs-rollback** - roll ZFS dataset back to snapshot # SYNOPSIS **zfs** **rollback** \[**-Rfr**] *snapshot* # DESCRIPTION When a dataset is rolled back, all data that has changed since the snapshot is discarded, and the dataset reverts to the state at the time of the snapshot. By default, the command refuses to roll back to a snapshot other than the most recent one. In order to do so, all intermediate snapshots and bookmarks must be destroyed by specifying the **-r** option. The **-rR** options do not recursively destroy the child snapshots of a recursive snapshot. Only direct snapshots of the specified filesystem are destroyed by either of these options. To completely roll back a recursive snapshot, you must roll back the individual child snapshots. **-R** > Destroy any more recent snapshots and bookmarks, as well as any clones of those > snapshots. **-f** > Used with the > **-R** > option to force an unmount of any clone file systems that are to be destroyed. **-r** > Destroy any snapshots and bookmarks more recent than the one specified. # EXAMPLES ## Example 1: Rolling Back a ZFS File System The following command reverts the contents of *pool/home/anne* to the snapshot named *yesterday*, deleting all intermediate snapshots: # **zfs** **rollback** **-r** *pool/home/anne*@*yesterday* # SEE ALSO zfs-snapshot(8) Debian - April 28, 2025 ZFS-SEND(8) - System Manager's Manual # NAME **zfs-send** - generate backup stream of ZFS dataset # SYNOPSIS **zfs** **send** \[**-DLPUVbcehnpsvw**] \[**-R** \[**-X** *dataset*\[,*dataset*]…]] \[\[**-I**|**-i**] *snapshot*] *snapshot* **zfs** **send** \[**-DLPUVcensvw**] \[**-i** *snapshot*|*bookmark*] *filesystem*|*volume*|*snapshot* **zfs** **send** **--redact** *redaction\_bookmark* \[**-DLPVcenpv**] \[**-i** *snapshot*|*bookmark*] *snapshot* **zfs** **send** \[**-PVenv**] **-t** *receive\_resume\_token* **zfs** **send** \[**-PVnv**] **-S** *filesystem* **zfs** **redact** *snapshot redaction\_bookmark* *redaction\_snapshot*… # DESCRIPTION **zfs** **send** \[**-DLPUVbcehnpsvw**] \[**-R** \[**-X** *dataset*\[,*dataset*]…]] \[\[**-I**|**-i**] *snapshot*] *snapshot* > Creates a stream representation of the second > *snapshot*, > which is written to standard output. > The output can be redirected to a file or to a different system > (for example, using > ssh(1)) > . > By default, a full stream is generated. > **-D**, **--dedup** > > Deduplicated send is no longer supported. > > This flag is accepted for backwards compatibility, but a regular, > > non-deduplicated stream will be generated. > **-I** *snapshot* > > Generate a stream package that sends all intermediary snapshots from the first > > snapshot to the second snapshot. > > For example, > > **-I** *@a* *fs@d* > > is similar to > > **-i** *@a* *fs@b*; **-i** *@b* *fs@c*; **-i** *@c* *fs@d*. > > The incremental source may be specified as with the > > **-i** > > option. > **-L**, **--large-block** > > Generate a stream which may contain blocks larger than 128 KiB. > > This flag has no effect if the > > **large\_blocks** > > pool feature is disabled, or if the > > **recordsize** > > property of this filesystem has never been set above 128 KiB. > > The receiving system must have the > > **large\_blocks** > > pool feature enabled as well. > > This flag is required if the > > **large\_microzap** > > pool feature is active. > > See > > zpool-features(7) > > for details on ZFS feature flags and the > > **large\_blocks** > > feature. > **-P**, **--parsable** > > Print machine-parsable verbose information about the stream package generated. > **-R**, **--replicate** > > Generate a replication stream package, which will replicate the specified > > file system, and all descendent file systems, up to the named snapshot. > > When received, all properties, snapshots, descendent file systems, and clones > > are preserved. > > If the > > **-i** > > or > > **-I** > > flags are used in conjunction with the > > **-R** > > flag, an incremental replication stream is generated. > > The current values of properties, and current snapshot and file system names are > > set when the stream is received. > > If the > > **-F** > > flag is specified when this stream is received, snapshots and file systems that > > do not exist on the sending side are destroyed. > > If the > > **-R** > > flag is used to send encrypted datasets, then > > **-w** > > or > > **-U** > > must also be specified. > **-V**, **--proctitle** > > Set the process title to a per-second report of how much data has been sent. > **-X**, **--exclude** *dataset*\[,*dataset*]… > > With > > **-R**, > > **-X** > > specifies a set of datasets (and, hence, their descendants), > > to be excluded from the send stream. > > The root dataset may not be excluded. > > **-X** *a* **-X** *b* > > is equivalent to > > **-X** *a*,*b*. > **-e**, **--embed** > > Generate a more compact stream by using > > **WRITE\_EMBEDDED** > > records for blocks which are stored more compactly on disk by the > > **embedded\_data** > > pool feature. > > This flag has no effect if the > > **embedded\_data** > > feature is disabled. > > The receiving system must have the > > **embedded\_data** > > feature enabled. > > If the > > **lz4\_compress** > > or > > **zstd\_compress** > > features are active on the sending system, then the receiving system must have > > the corresponding features enabled as well. > > Datasets that are sent with this flag may not be > > received as an encrypted dataset, since encrypted datasets cannot use the > > **embedded\_data** > > feature. > > See > > zpool-features(7) > > for details on ZFS feature flags and the > > **embedded\_data** > > feature. > **-b**, **--backup** > > Sends only received property values whether or not they are overridden by local > > settings, but only if the dataset has ever been received. > > Use this option when you want > > **zfs** **receive** > > to restore received properties backed up on the sent dataset and to avoid > > sending local settings that may have nothing to do with the source dataset, > > but only with how the data is backed up. > **-c**, **--compressed** > > Generate a more compact stream by using compressed WRITE records for blocks > > which are compressed on disk and in memory > > (see the > > **compression** > > property for details) > > . > > If the > > **lz4\_compress** > > or > > **zstd\_compress** > > features are active on the sending system, then the receiving system must have > > the corresponding features enabled as well. > > If the > > **large\_blocks** > > feature is enabled on the sending system but the > > **-L** > > option is not supplied in conjunction with > > **-c**, > > then the data will be decompressed before sending so it can be split into > > smaller block sizes. > > Streams sent with > > **-c** > > will not have their data recompressed on the receiver side using > > **-o** **compress**= *value*. > > The data will stay compressed as it was from the sender. > > The new compression property will be set for future data. > > Note that uncompressed data from the sender will still attempt to > > compress on the receiver, unless you specify > > **-o** **compress**= *off*. > **-w**, **--raw** > > For encrypted datasets, send data exactly as it exists on disk. > > This allows backups to be taken even if encryption keys are not currently > > loaded. > > The backup may then be received on an untrusted machine since that machine will > > not have the encryption keys to read the protected data or alter it without > > being detected. > > Upon being received, the dataset will have the same encryption > > keys as it did on the send side, although the > > **keylocation** > > property will be defaulted to > > **prompt** > > if not otherwise provided. > > For unencrypted datasets, this flag will be equivalent to > > **-Lec**. > > Note that if you do not use this flag for sending encrypted datasets, data will > > be sent unencrypted and may be re-encrypted with a different encryption key on > > the receiving system, which will disable the ability to do a raw send to that > > system for incrementals. > **-h**, **--holds** > > Generate a stream package that includes any snapshot holds (created with the > > **zfs** **hold** > > command), and indicating to > > **zfs** **receive** > > that the holds be applied to the dataset on the receiving system. > **-i** *snapshot* > > Generate an incremental stream from the first > > *snapshot* > > (the incremental source) > > to the second > > *snapshot* > > (the incremental target). > > The incremental source can be specified as the last component of the snapshot > > name > > (the > > **@** > > character and following) > > and it is assumed to be from the same file system as the incremental target. > > If the destination is a clone, the source may be the origin snapshot, which must > > be fully specified > > (for example, > > *pool/fs@origin*, > > not just > > *@origin*) > > . > **-n**, **--dryrun** > > Do a dry-run > > ("No-op") > > send. > > Do not generate any actual send data. > > This is useful in conjunction with the > > **-v** > > or > > **-P** > > flags to determine what data will be sent. > > In this case, the verbose output will be written to standard output > > (contrast with a non-dry-run, where the stream is written to standard output > > and the verbose output goes to standard error) > > . > **-p**, **--props** > > Include the dataset's properties in the stream. > > This flag is implicit when > > **-R** > > is specified. > > The receiving system must also support this feature. > > Sends of encrypted datasets must use > > **-w** > > or > > **-U** > > when using this flag. > **-s**, **--skip-missing** > > Allows sending a replication stream even when there are snapshots missing in the > > hierarchy. > > When a snapshot is missing, instead of throwing an error and aborting the send, > > a warning is printed to the standard error stream and the dataset to which it > > belongs > > and its descendants are skipped. > > This flag can only be used in conjunction with > > **-R**. > **-U**, **--no-preserve-encryption** > > Allow sending an encrypted dataset with properties, but without keeping > > encryption. > > When this flag is specified, encrypted datasets that would otherwise be blocked > > from sending are sent as unencrypted data. > **-v**, **--verbose** > > Print verbose information about the stream package generated. > > This information includes a per-second report of how much data has been sent. > > The same report can be requested by sending > > `SIGINFO` > > or > > `SIGUSR1`, > > regardless of > > **-v**. > > The format of the stream is committed. > > You will be able to receive your streams on future versions of ZFS. **zfs** **send** \[**-DLPVcenvw**] \[**-i** *snapshot*|*bookmark*] *filesystem*|*volume*|*snapshot* > Generate a send stream, which may be of a filesystem, and may be incremental > from a bookmark. > If the destination is a filesystem or volume, the pool must be read-only, or the > filesystem must not be mounted. > When the stream generated from a filesystem or volume is received, the default > snapshot name will be > "--head--". > **-D**, **--dedup** > > Deduplicated send is no longer supported. > > This flag is accepted for backwards compatibility, but a regular, > > non-deduplicated stream will be generated. > **-L**, **--large-block** > > Generate a stream which may contain blocks larger than 128 KiB. > > This flag has no effect if the > > **large\_blocks** > > pool feature is disabled, or if the > > **recordsize** > > property of this filesystem has never been set above 128 KiB. > > The receiving system must have the > > **large\_blocks** > > pool feature enabled as well. > > See > > zpool-features(7) > > for details on ZFS feature flags and the > > **large\_blocks** > > feature. > **-P**, **--parsable** > > Print machine-parsable verbose information about the stream package generated. > **-c**, **--compressed** > > Generate a more compact stream by using compressed WRITE records for blocks > > which are compressed on disk and in memory > > (see the > > **compression** > > property for details) > > . > > If the > > **lz4\_compress** > > or > > **zstd\_compress** > > features are active on the sending system, then the receiving system must have > > the corresponding features enabled as well. > > If the > > **large\_blocks** > > feature is enabled on the sending system but the > > **-L** > > option is not supplied in conjunction with > > **-c**, > > then the data will be decompressed before sending so it can be split into > > smaller block sizes. > **-w**, **--raw** > > For encrypted datasets, send data exactly as it exists on disk. > > This allows backups to be taken even if encryption keys are not currently > > loaded. > > The backup may then be received on an untrusted machine since that machine will > > not have the encryption keys to read the protected data or alter it without > > being detected. > > Upon being received, the dataset will have the same encryption > > keys as it did on the send side, although the > > **keylocation** > > property will be defaulted to > > **prompt** > > if not otherwise provided. > > For unencrypted datasets, this flag will be equivalent to > > **-Lec**. > > Note that if you do not use this flag for sending encrypted datasets, data will > > be sent unencrypted and may be re-encrypted with a different encryption key on > > the receiving system, which will disable the ability to do a raw send to that > > system for incrementals. > **-e**, **--embed** > > Generate a more compact stream by using > > **WRITE\_EMBEDDED** > > records for blocks which are stored more compactly on disk by the > > **embedded\_data** > > pool feature. > > This flag has no effect if the > > **embedded\_data** > > feature is disabled. > > The receiving system must have the > > **embedded\_data** > > feature enabled. > > If the > > **lz4\_compress** > > or > > **zstd\_compress** > > features are active on the sending system, then the receiving system must have > > the corresponding features enabled as well. > > Datasets that are sent with this flag may not be received as an encrypted > > dataset, > > since encrypted datasets cannot use the > > **embedded\_data** > > feature. > > See > > zpool-features(7) > > for details on ZFS feature flags and the > > **embedded\_data** > > feature. > **-i** *snapshot*|*bookmark* > > Generate an incremental send stream. > > The incremental source must be an earlier snapshot in the destination's history. > > It will commonly be an earlier snapshot in the destination's file system, in > > which case it can be specified as the last component of the name > > (the > > **#** > > or > > **@** > > character and following) > > . > > If the incremental target is a clone, the incremental source can be the origin > > snapshot, or an earlier snapshot in the origin's filesystem, or the origin's > > origin, etc. > **-n**, **--dryrun** > > Do a dry-run > > ("No-op") > > send. > > Do not generate any actual send data. > > This is useful in conjunction with the > > **-v** > > or > > **-P** > > flags to determine what data will be sent. > > In this case, the verbose output will be written to standard output > > (contrast with a non-dry-run, where the stream is written to standard output > > and the verbose output goes to standard error) > > . > **-v**, **--verbose** > > Print verbose information about the stream package generated. > > This information includes a per-second report of how much data has been sent. > > The same report can be requested by sending > > `SIGINFO` > > or > > `SIGUSR1`, > > regardless of > > **-v**. **zfs** **send** **--redact** *redaction\_bookmark* \[**-DLPVcenpv**] \[**-i** *snapshot*|*bookmark*] *snapshot* > Generate a redacted send stream. > This send stream contains all blocks from the snapshot being sent that aren't > included in the redaction list contained in the bookmark specified by the > **--redact** > (or > **-d**) > flag. > The resulting send stream is said to be redacted with respect to the snapshots > the bookmark specified by the > **--redact** flag was created with. > The bookmark must have been created by running > **zfs** **redact** > on the snapshot being sent. > This feature can be used to allow clones of a filesystem to be made available on > a remote system, in the case where their parent need not (or needs to not) be > usable. > For example, if a filesystem contains sensitive data, and it has clones where > that sensitive data has been secured or replaced with dummy data, redacted sends > can be used to replicate the secured data without replicating the original > sensitive data, while still sharing all possible blocks. > A snapshot that has been redacted with respect to a set of snapshots will > contain all blocks referenced by at least one snapshot in the set, but will > contain none of the blocks referenced by none of the snapshots in the set. > In other words, if all snapshots in the set have modified a given block in the > parent, that block will not be sent; but if one or more snapshots have not > modified a block in the parent, they will still reference the parent's block, so > that block will be sent. > Note that only user data will be redacted. > When the redacted send stream is received, we will generate a redacted > snapshot. > Due to the nature of redaction, a redacted dataset can only be used in the > following ways: > 1. To receive, as a clone, an incremental send from the original snapshot to one > of the snapshots it was redacted with respect to. > In this case, the stream will produce a valid dataset when received because all > blocks that were redacted in the parent are guaranteed to be present in the > child's send stream. > This use case will produce a normal snapshot, which can be used just like other > snapshots. > 2. To receive an incremental send from the original snapshot to something > redacted with respect to a subset of the set of snapshots the initial snapshot > was redacted with respect to. > In this case, each block that was redacted in the original is still redacted > (redacting with respect to additional snapshots causes less data to be redacted > (because the snapshots define what is permitted, and everything else is > redacted)). > This use case will produce a new redacted snapshot. > 3. To receive an incremental send from a redaction bookmark of the original > snapshot that was created when redacting with respect to a subset of the set of > snapshots the initial snapshot was created with respect to > anything else. > A send stream from such a redaction bookmark will contain all of the blocks > necessary to fill in any redacted data, should it be needed, because the sending > system is aware of what blocks were originally redacted. > This will either produce a normal snapshot or a redacted one, depending on > whether the new send stream is redacted. > 4. To receive an incremental send from a redacted version of the initial > snapshot that is redacted with respect to a subject of the set of snapshots the > initial snapshot was created with respect to. > A send stream from a compatible redacted dataset will contain all of the blocks > necessary to fill in any redacted data. > This will either produce a normal snapshot or a redacted one, depending on > whether the new send stream is redacted. > 5. To receive a full send as a clone of the redacted snapshot. > Since the stream is a full send, it definitionally contains all the data needed > to create a new dataset. > This use case will either produce a normal snapshot or a redacted one, depending > on whether the full send stream was redacted. > These restrictions are detected and enforced by > **zfs** **receive**; > a redacted send stream will contain the list of snapshots that the stream is > redacted with respect to. > These are stored with the redacted snapshot, and are used to detect and > correctly handle the cases above. > Note that for technical reasons, > raw sends and redacted sends cannot be combined at this time. **zfs** **send** \[**-PVenv**] **-t** *receive\_resume\_token* > Creates a send stream which resumes an interrupted receive. > The > *receive\_resume\_token* > is the value of this property on the filesystem or volume that was being > received into. > See the documentation for > **zfs** **receive** **-s** > for more details. **zfs** **send** \[**-PVnv**] \[**-i** *snapshot*|*bookmark*] **-S** *filesystem* > Generate a send stream from a dataset that has been partially received. > **-S**, **--saved** > > This flag requires that the specified filesystem previously received a resumable > > send that did not finish and was interrupted. > > In such scenarios this flag > > enables the user to send this partially received state. > > Using this flag will always use the last fully received snapshot > > as the incremental source if it exists. **zfs** **redact** *snapshot redaction\_bookmark* *redaction\_snapshot*… > Generate a new redaction bookmark. > In addition to the typical bookmark information, a redaction bookmark contains > the list of redacted blocks and the list of redaction snapshots specified. > The redacted blocks are blocks in the snapshot which are not referenced by any > of the redaction snapshots. > These blocks are found by iterating over the metadata in each redaction snapshot > to determine what has been changed since the target snapshot. > Redaction is designed to support redacted zfs sends; see the entry for > **zfs** **send** > for more information on the purpose of this operation. > If a redact operation fails partway through (due to an error or a system > failure), the redaction can be resumed by rerunning the same command. ## Redaction ZFS has support for a limited version of data subsetting, in the form of redaction. Using the **zfs** **redact** command, a **redaction bookmark** can be created that stores a list of blocks containing sensitive information. When provided to **zfs** **send**, this causes a **redacted send** to occur. Redacted sends omit the blocks containing sensitive information, replacing them with REDACT records. When these send streams are received, a **redacted dataset** is created. A redacted dataset cannot be mounted by default, since it is incomplete. It can be used to receive other send streams. In this way datasets can be used for data backup and replication, with all the benefits that zfs send and receive have to offer, while protecting sensitive information from being stored on less-trusted machines or services. For the purposes of redaction, there are two steps to the process. A redact step, and a send/receive step. First, a redaction bookmark is created. This is done by providing the **zfs** **redact** command with a parent snapshot, a bookmark to be created, and a number of redaction snapshots. These redaction snapshots must be descendants of the parent snapshot, and they should modify data that is considered sensitive in some way. Any blocks of data modified by all of the redaction snapshots will be listed in the redaction bookmark, because it represents the truly sensitive information. When it comes to the send step, the send process will not send the blocks listed in the redaction bookmark, instead replacing them with REDACT records. When received on the target system, this will create a redacted dataset, missing the data that corresponds to the blocks in the redaction bookmark on the sending system. The incremental send streams from the original parent to the redaction snapshots can then also be received on the target system, and this will produce a complete snapshot that can be used normally. Incrementals from one snapshot on the parent filesystem and another can also be done by sending from the redaction bookmark, rather than the snapshots themselves. In order to make the purpose of the feature more clear, an example is provided. Consider a zfs filesystem containing four files. These files represent information for an online shopping service. One file contains a list of usernames and passwords, another contains purchase histories, a third contains click tracking data, and a fourth contains user preferences. The owner of this data wants to make it available for their development teams to test against, and their market research teams to do analysis on. The development teams need information about user preferences and the click tracking data, while the market research teams need information about purchase histories and user preferences. Neither needs access to the usernames and passwords. However, because all of this data is stored in one ZFS filesystem, it must all be sent and received together. In addition, the owner of the data wants to take advantage of features like compression, checksumming, and snapshots, so they do want to continue to use ZFS to store and transmit their data. Redaction can help them do so. First, they would make two clones of a snapshot of the data on the source. In one clone, they create the setup they want their market research team to see; they delete the usernames and passwords file, and overwrite the click tracking data with dummy information. In another, they create the setup they want the development teams to see, by replacing the passwords with fake information and replacing the purchase histories with randomly generated ones. They would then create a redaction bookmark on the parent snapshot, using snapshots on the two clones as redaction snapshots. The parent can then be sent, redacted, to the target server where the research and development teams have access. Finally, incremental sends from the parent snapshot to each of the clones can be sent to and received on the target server; these snapshots are identical to the ones on the source, and are ready to be used, while the parent snapshot on the target contains none of the username and password data present on the source, because it was removed by the redacted send operation. # SIGNALS See **-v**. # EXAMPLES ## Example 1: Remotely Replicating ZFS Data The following commands send a full stream and then an incremental stream to a remote machine, restoring them into *poolB/received/fs@a* and *poolB/received/fs@b*, respectively. *poolB* must contain the file system *poolB/received*, and must not initially contain *poolB/received/fs*. # **zfs** **send** *pool/fs@a* | **ssh** *host* **zfs** **receive** *poolB/received/fs*@*a* # **zfs** **send** **-i** *a pool/fs@b* | **ssh** *host* **zfs** **receive** *poolB/received/fs* ## Example 2: Using the **zfs** **receive** **-d** Option The following command sends a full stream of *poolA/fsA/fsB@snap* to a remote machine, receiving it into *poolB/received/fsA/fsB@snap*. The *fsA/fsB@snap* portion of the received snapshot's name is determined from the name of the sent snapshot. *poolB* must contain the file system *poolB/received*. If *poolB/received/fsA* does not exist, it is created as an empty file system. # **zfs** **send** *poolA/fsA/fsB@snap* | **ssh** *host* **zfs** **receive** **-d** *poolB/received* # SEE ALSO zfs-bookmark(8), zfs-receive(8), zfs-redact(8), zfs-snapshot(8) Debian - February 20, 2026 ZFS-SET(8) - System Manager's Manual # NAME **zfs-set** - set properties on ZFS datasets # SYNOPSIS **zfs** **set** \[**-u**] *property*=*value* \[*property*=*value*]… *filesystem*|*volume*|*snapshot*… **zfs** **get** \[**-r**|**-d** *depth*] \[**-Hp**] \[**-j** \[*--json-int*]] \[**-o** *field*\[,*field*]…] \[**-s** *source*\[,*source*]…] \[**-t** *type*\[,*type*]…] **all**|*property*\[,*property*]… \[*filesystem*|*volume*|*snapshot*|*bookmark*]… **zfs** **inherit** \[**-rS**] *property* *filesystem*|*volume*|*snapshot*… # DESCRIPTION **zfs** **set** \[**-u**] *property*=*value* \[*property*=*value*]… *filesystem*|*volume*|*snapshot*… > Only some properties can be edited. > See > zfsprops(7) > for more information on what properties can be set and acceptable > values. > Numeric values can be specified as exact values, or in a human-readable form > with a suffix of > **B**, **K**, **M**, **G**, **T**, **P**, **E**, **Z** > (for bytes, kilobytes, megabytes, gigabytes, terabytes, petabytes, exabytes, > or zettabytes, respectively) > . > User properties can be set on snapshots. > For more information, see the > *User Properties* > section of > zfsprops(7). > **-u** > > Update mountpoint, sharenfs, sharesmb property but do not mount or share the > > dataset. **zfs** **get** \[**-r**|**-d** *depth*] \[**-Hp**] \[**-j** \[*--json-int*]] \[**-o** *field*\[,*field*]…] \[**-s** *source*\[,*source*]…] \[**-t** *type*\[,*type*]…] **all**|*property*\[,*property*]… \[*filesystem*|*volume*|*snapshot*|*bookmark*]… > Displays properties for the given datasets. > If no datasets are specified, then the command displays properties for all > datasets on the system. > For each property, the following columns are displayed: > **name** > > Dataset name > **property** > > Property name > **value** > > Property value > **source** > > Property source > > **local**, **default**, **inherited**, **temporary**, **received**, or **-** (none). > **received** > > The received value of the property, if any. > > This column is not displayed by default. > The > **name**, **property**, **value**, > and > **source** > columns are displayed by default, though this can be > controlled by using the > **-o** > option. > This command takes a comma-separated list of properties as described in the > *Native Properties* > and > *User Properties* > sections of > zfsprops(7). > The value > **all** > can be used to display all properties that apply to the given dataset's type > (**filesystem**, **volume**, **snapshot**, or **bookmark**). > **-j**, **--json** \[*--json-int*] > > Display the output in JSON format. > > Specify > > **--json-int** > > to display numbers in integer format instead of strings for JSON output. > **-H** > > Display output in a form more easily parsed by scripts. > > Any headers are omitted, and fields are explicitly separated by a single tab > > instead of an arbitrary amount of space. > **-d** *depth* > > Recursively display any children of the dataset, limiting the recursion to > > *depth*. > > A depth of > > **1** > > will display only the dataset and its direct children. > **-o** *field* > > A comma-separated list of columns to display. > > Supported fields are > > **name**, **property**, **value**, **received**, **source**, > > or > > **all** > > to select all five columns. > > The default value is > > **name**,**property**,**value**,**source**. > **-p** > > Display numbers in parsable > > (exact) > > values. > **-r** > > Recursively display properties for any children. > **-s** *source* > > A comma-separated list of sources to display. > > Those properties coming from a source other than those in this list are ignored. > > Each source must be one of the following: > > **local**, **default**, **inherited**, **temporary**, **received**, or **none**. > > The default value is all sources. > **-t** *type* > > A comma-separated list of types to display, where > > *type* > > is one of > > **filesystem**, **snapshot**, **volume**, **bookmark**, or **all**. > > **fs**, > > **snap**, > > or > > **vol** > > can be used as aliases for > > **filesystem**, > > **snapshot**, > > or > > **volume**. **zfs** **inherit** \[**-rS**] *property* *filesystem*|*volume*|*snapshot*… > Clears the specified property, causing it to be inherited from an ancestor, > restored to default if no ancestor has the property set, or with the > **-S** > option reverted to the received value if one exists. > See > zfsprops(7) > for a listing of default values, and details on which properties can be > inherited. > **-r** > > Recursively inherit the given property for all children. > **-S** > > Revert the property to the received value, if one exists; > > otherwise, for non-inheritable properties, to the default; > > otherwise, operate as if the > > **-S** > > option was not specified. # EXAMPLES ## Example 1: Creating a ZFS File System Hierarchy The following commands create a file system named *pool/home* and a file system named *pool/home/bob*. The mount point */export/home* is set for the parent file system, and is automatically inherited by the child file system. # **zfs** **create** *pool/home* # **zfs** **set** **mountpoint**=*/export/home pool/home* # **zfs** **create** *pool/home/bob* ## Example 2: Disabling and Enabling File System Compression The following command disables the **compression** property for all file systems under *pool/home*. The next command explicitly enables **compression** for *pool/home/anne*. # **zfs** **set** **compression**=**off** *pool/home* # **zfs** **set** **compression**=**on** *pool/home/anne* ## Example 3: Setting a Quota on a ZFS File System The following command sets a quota of 50 Gbytes for *pool/home/bob*: # **zfs** **set** **quota**=*50G pool/home/bob* ## Example 4: Listing ZFS Properties The following command lists all properties for *pool/home/bob*: # **zfs** **get** **all** *pool/home/bob* NAME PROPERTY VALUE SOURCE pool/home/bob type filesystem - pool/home/bob creation Tue Jul 21 15:53 2009 - pool/home/bob used 21K - pool/home/bob available 20.0G - pool/home/bob referenced 21K - pool/home/bob compressratio 1.00x - pool/home/bob mounted yes - pool/home/bob quota 20G local pool/home/bob reservation none default pool/home/bob recordsize 128K default pool/home/bob mountpoint /pool/home/bob default pool/home/bob sharenfs off default pool/home/bob checksum on default pool/home/bob compression on local pool/home/bob atime on default pool/home/bob devices on default pool/home/bob exec on default pool/home/bob setuid on default pool/home/bob readonly off default pool/home/bob zoned off default pool/home/bob snapdir hidden default pool/home/bob acltype off default pool/home/bob aclmode discard default pool/home/bob aclinherit restricted default pool/home/bob canmount on default pool/home/bob xattr on default pool/home/bob copies 1 default pool/home/bob version 4 - pool/home/bob utf8only off - pool/home/bob normalization none - pool/home/bob casesensitivity sensitive - pool/home/bob vscan off default pool/home/bob nbmand off default pool/home/bob sharesmb off default pool/home/bob refquota none default pool/home/bob refreservation none default pool/home/bob primarycache all default pool/home/bob secondarycache all default pool/home/bob usedbysnapshots 0 - pool/home/bob usedbydataset 21K - pool/home/bob usedbychildren 0 - pool/home/bob usedbyrefreservation 0 - The following command gets a single property value: # **zfs** **get** **-H** **-o** **value compression** *pool/home/bob* on The following command gets a single property value recursively in JSON format: # **zfs** **get** **-j** **-r** **mountpoint** *pool/home* | **jq** { "output_version": { "command": "zfs get", "vers_major": 0, "vers_minor": 1 }, "datasets": { "pool/home": { "name": "pool/home", "type": "FILESYSTEM", "pool": "pool", "createtxg": "10", "properties": { "mountpoint": { "value": "/pool/home", "source": { "type": "DEFAULT", "data": "-" } } } }, "pool/home/bob": { "name": "pool/home/bob", "type": "FILESYSTEM", "pool": "pool", "createtxg": "1176", "properties": { "mountpoint": { "value": "/pool/home/bob", "source": { "type": "DEFAULT", "data": "-" } } } } } } The following command lists all properties with local settings for *pool/home/bob*: # **zfs** **get** **-r** **-s** **local** **-o** **name**,**property**,**value all** *pool/home/bob* NAME PROPERTY VALUE pool/home/bob quota 20G pool/home/bob compression on ## Example 5: Inheriting ZFS Properties The following command causes *pool/home/bob* and *pool/home/anne* to inherit the **checksum** property from their parent. # **zfs** **inherit** **checksum** *pool/home/bob pool/home/anne* ## Example 6: Setting User Properties The following example sets the user-defined *com.example*:*department* property for a dataset: # **zfs** **set** *com.example*:*department*=*12345 tank/accounting* ## Example 7: Setting sharenfs Property Options on a ZFS File System The following commands show how to set **sharenfs** property options to enable read-write access for a set of IP addresses and to enable root access for system "neo" on the *tank/home* file system: # **zfs** **set** **sharenfs**='*rw*=@123.123.0.0/16:[::1],root=*neo*' tank/home If you are using DNS for host name resolution, specify the fully-qualified hostname. # SEE ALSO zfsprops(7), zfs-list(8) Debian - March 15, 2026 ZFS-SHARE(8) - System Manager's Manual # NAME **zfs-share** - share and unshare ZFS filesystems # SYNOPSIS **zfs** **share** \[**-l**] **-a**|*filesystem* **zfs** **unshare** **-a**|*filesystem*|*mountpoint* # DESCRIPTION **zfs** **share** \[**-l**] **-a**|*filesystem* > Shares available ZFS file systems. > **-l** > > Load keys for encrypted filesystems as they are being mounted. > > This is equivalent to executing > > **zfs** **load-key** > > on each encryption root before mounting it. > > Note that if a filesystem has > > **keylocation**=**prompt**, > > this will cause the terminal to interactively block after asking for the key. > **-a** > > Share all available ZFS file systems. > > Invoked automatically as part of the boot process. > *filesystem* > > Share the specified filesystem according to the > > **sharenfs** > > and > > **sharesmb** > > properties. > > File systems are shared when the > > **sharenfs** > > or > > **sharesmb** > > property is set. **zfs** **unshare** **-a**|*filesystem*|*mountpoint* > Unshares currently shared ZFS file systems. > **-a** > > Unshare all available ZFS file systems. > > Invoked automatically as part of the shutdown process. > *filesystem*|*mountpoint* > > Unshare the specified filesystem. > > The command can also be given a path to a ZFS file system shared on the system. # SEE ALSO exports(5), smb.conf(5), zfsprops(7) Debian - July 11, 2022 ZFS-SNAPSHOT(8) - System Manager's Manual # NAME **zfs-snapshot** - create snapshots of ZFS datasets # SYNOPSIS **zfs** **snapshot** \[**-r**] \[**-o** *property*=*value*]… *dataset*@*snapname*… # DESCRIPTION Creates a snapshot of a dataset or multiple snapshots of different datasets. Snapshots are created atomically. That is, a snapshot is a consistent image of a dataset at a specific point in time; it includes all modifications to the dataset made by system calls that have successfully completed before that point in time. Recursive snapshots created through the **-r** option are all created at the same time. **zfs** **snap** can be used as an alias for **zfs** **snapshot**. See the *Snapshots* section of zfsconcepts(7) for details. **-o** *property*=*value* > Set the specified property; see > **zfs** **create** > for details. **-r** > Recursively create snapshots of all descendent datasets # EXAMPLES ## Example 1: Creating a ZFS Snapshot The following command creates a snapshot named *yesterday*. This snapshot is mounted on demand in the *.zfs/snapshot* directory at the root of the *pool/home/bob* file system. # **zfs** **snapshot** *pool/home/bob*@*yesterday* ## Example 2: Creating and Destroying Multiple Snapshots The following command creates snapshots named *yesterday* of *pool/home* and all of its descendent file systems. Each snapshot is mounted on demand in the *.zfs/snapshot* directory at the root of its file system. The second command destroys the newly created snapshots. # **zfs** **snapshot** **-r** *pool/home*@*yesterday* # **zfs** **destroy** **-r** *pool/home*@*yesterday* ## Example 3: Promoting a ZFS Clone The following commands illustrate how to test out changes to a file system, and then replace the original file system with the changed one, using clones, clone promotion, and renaming: # **zfs** **create** *pool/project/production* populate /pool/project/production with data # **zfs** **snapshot** *pool/project/production*@*today* # **zfs** **clone** *pool/project/production@today pool/project/beta* make changes to /pool/project/beta and test them # **zfs** **promote** *pool/project/beta* # **zfs** **rename** *pool/project/production pool/project/legacy* # **zfs** **rename** *pool/project/beta pool/project/production* once the legacy version is no longer needed, it can be destroyed # **zfs** **destroy** *pool/project/legacy* ## Example 4: Performing a Rolling Snapshot The following example shows how to maintain a history of snapshots with a consistent naming scheme. To keep a week's worth of snapshots, the user destroys the oldest snapshot, renames the remaining snapshots, and then creates a new snapshot, as follows: # **zfs** **destroy** **-r** *pool/users@7daysago* # **zfs** **rename** **-r** *pool/users@6daysago* @*7daysago* # **zfs** **rename** **-r** *pool/users@5daysago* @*6daysago* # **zfs** **rename** **-r** *pool/users@4daysago* @*5daysago* # **zfs** **rename** **-r** *pool/users@3daysago* @*4daysago* # **zfs** **rename** **-r** *pool/users@2daysago* @*3daysago* # **zfs** **rename** **-r** *pool/users@yesterday* @*2daysago* # **zfs** **rename** **-r** *pool/users@today* @*yesterday* # **zfs** **snapshot** **-r** *pool/users*@*today* # SEE ALSO zfs-bookmark(8), zfs-clone(8), zfs-destroy(8), zfs-diff(8), zfs-hold(8), zfs-rename(8), zfs-rollback(8), zfs-send(8) Debian - July 11, 2022 .\" SPDX-License-Identifier: CDDL-1.0 .\" .\" This file and its contents are supplied under the terms of the .\" Common Development and Distribution License ("CDDL"), version 1.0. .\" You may only use this file in accordance with the terms of version .\" 1.0 of the CDDL. .\" .\" A full copy of the text of the CDDL should have accompanied this .\" source. A copy of the CDDL is also available via the Internet at .\" https://opensource.org/license/CDDL-1.0. .\" .\" Copyright (c) 2009 Sun Microsystems, Inc. All Rights Reserved. .\" Copyright 2011 Joshua M. Clulow .\" Copyright (c) 2011, 2019 by Delphix. All rights reserved. .\" Copyright (c) 2013 by Saso Kiselkov. All rights reserved. .\" Copyright (c) 2014, Joyent, Inc. All rights reserved. .\" Copyright (c) 2014 by Adam Stevko. All rights reserved. .\" Copyright (c) 2014 Integros [integros.com] .\" Copyright 2019 Richard Laager. All rights reserved. .\" Copyright 2018 Nexenta Systems, Inc. .\" Copyright 2019 Joyent, Inc. .\" Copyright (c) 2026, TrueNAS. .\" .Dd June 14, 2026 .Dt ZFS-ALLOW 8 .Os . .Sh NAME .Nm zfs-allow .Nd delegate ZFS administration permissions to unprivileged users .Sh SYNOPSIS .Nm zfs .Cm allow .Op Fl dglu .Ar user Ns | Ns Ar group Ns Oo , Ns Ar user Ns | Ns Ar group Oc Ns … .Ar perm Ns | Ns @ Ns Ar setname Ns Oo , Ns Ar perm Ns | Ns @ Ns .Ar setname Oc Ns … .Ar filesystem Ns | Ns Ar volume .Nm zfs .Cm allow .Op Fl dl .Fl e Ns | Ns Sy everyone .Ar perm Ns | Ns @ Ns Ar setname Ns Oo , Ns Ar perm Ns | Ns @ Ns .Ar setname Oc Ns … .Ar filesystem Ns | Ns Ar volume .Nm zfs .Cm allow .Fl c .Ar perm Ns | Ns @ Ns Ar setname Ns Oo , Ns Ar perm Ns | Ns @ Ns .Ar setname Oc Ns … .Ar filesystem Ns | Ns Ar volume .Nm zfs .Cm allow .Fl s No @ Ns Ar setname .Ar perm Ns | Ns @ Ns Ar setname Ns Oo , Ns Ar perm Ns | Ns @ Ns .Ar setname Oc Ns … .Ar filesystem Ns | Ns Ar volume .Nm zfs .Cm unallow .Op Fl dglru .Ar user Ns | Ns Ar group Ns Oo , Ns Ar user Ns | Ns Ar group Oc Ns … .Oo Ar perm Ns | Ns @ Ns Ar setname Ns Oo , Ns Ar perm Ns | Ns @ Ns .Ar setname Oc Ns … Oc .Ar filesystem Ns | Ns Ar volume .Nm zfs .Cm unallow .Op Fl dlr .Fl e Ns | Ns Sy everyone .Oo Ar perm Ns | Ns @ Ns Ar setname Ns Oo , Ns Ar perm Ns | Ns @ Ns .Ar setname Oc Ns … Oc .Ar filesystem Ns | Ns Ar volume .Nm zfs .Cm unallow .Op Fl r .Fl c .Oo Ar perm Ns | Ns @ Ns Ar setname Ns Oo , Ns Ar perm Ns | Ns @ Ns .Ar setname Oc Ns … Oc .Ar filesystem Ns | Ns Ar volume .Nm zfs .Cm unallow .Op Fl r .Fl s No @ Ns Ar setname .Oo Ar perm Ns | Ns @ Ns Ar setname Ns Oo , Ns Ar perm Ns | Ns @ Ns .Ar setname Oc Ns … Oc .Ar filesystem Ns | Ns Ar volume . .Sh DESCRIPTION .Bl -tag -width "" .It Xo .Nm zfs .Cm allow .Ar filesystem Ns | Ns Ar volume .Xc Displays permissions that have been delegated on the specified filesystem or volume. See the other forms of .Nm zfs Cm allow for more information. .Pp Delegations are supported under Linux with the exception of .Sy mount , .Sy unmount , .Sy mountpoint , .Sy canmount , .Sy rename , and .Sy share . These permissions cannot be delegated because the Linux .Xr mount 8 command restricts modifications of the global namespace to the root user. .It Xo .Nm zfs .Cm allow .Op Fl dglu .Ar user Ns | Ns Ar group Ns Oo , Ns Ar user Ns | Ns Ar group Oc Ns … .Ar perm Ns | Ns @ Ns Ar setname Ns Oo , Ns Ar perm Ns | Ns @ Ns .Ar setname Oc Ns … .Ar filesystem Ns | Ns Ar volume .Xc .It Xo .Nm zfs .Cm allow .Op Fl dl .Fl e Ns | Ns Sy everyone .Ar perm Ns | Ns @ Ns Ar setname Ns Oo , Ns Ar perm Ns | Ns @ Ns .Ar setname Oc Ns … .Ar filesystem Ns | Ns Ar volume .Xc Delegates ZFS administration permission for the file systems to non-privileged users. .Bl -tag -width "-d" .It Fl d Allow only for the descendent file systems. .It Fl e Ns | Ns Sy everyone Specifies that the permissions be delegated to everyone. .It Fl g Ar group Ns Oo , Ns Ar group Oc Ns … Explicitly specify that permissions are delegated to the group. .It Fl l Allow .Qq locally only for the specified file system. .It Fl u Ar user Ns Oo , Ns Ar user Oc Ns … Explicitly specify that permissions are delegated to the user. .It Ar user Ns | Ns Ar group Ns Oo , Ns Ar user Ns | Ns Ar group Oc Ns … Specifies to whom the permissions are delegated. Multiple entities can be specified as a comma-separated list. If neither of the .Fl gu options are specified, then the argument is interpreted preferentially as the keyword .Sy everyone , then as a user name, and lastly as a group name. To specify a user or group named .Qq everyone , use the .Fl g or .Fl u options. To specify a group with the same name as a user, use the .Fl g options. .It Xo .Ar perm Ns | Ns @ Ns Ar setname Ns Oo , Ns Ar perm Ns | Ns @ Ns .Ar setname Oc Ns … .Xc The permissions to delegate. Multiple permissions may be specified as a comma-separated list. Permission names are the same as ZFS subcommand and property names. See the property list below. Property set names, which begin with .Sy @ , may be specified. See the .Fl s form below for details. .El .Pp If neither of the .Fl dl options are specified, or both are, then the permissions are allowed for the file system or volume, and all of its descendants. .Pp Permissions are generally the ability to use a ZFS subcommand or change a ZFS property. The following permissions are available: .TS l l l . NAME TYPE NOTES _ _ _ allow subcommand Must also have the permission that is being allowed bookmark subcommand clone subcommand Must also have the \fBcreate\fR ability and \fBmount\fR ability in the origin file system create subcommand Must also have the \fBmount\fR ability. Must also have the \fBrefreservation\fR ability to create a non-sparse volume. destroy subcommand Must also have the \fBmount\fR ability diff subcommand Allows lookup of paths within a dataset given an object number, and the ability to create snapshots necessary to \fBzfs diff\fR. hold subcommand Allows adding a user hold to a snapshot load-key subcommand Allows loading and unloading of encryption key (see \fBzfs load-key\fR and \fBzfs unload-key\fR). change-key subcommand Allows changing an encryption key via \fBzfs change-key\fR. mount subcommand Allows mounting/unmounting ZFS datasets promote subcommand Must also have the \fBmount\fR and \fBpromote\fR ability in the origin file system receive subcommand Must also have the \fBmount\fR and \fBcreate\fR ability, required for \fBzfs receive -F\fR (see also \fBreceive:append\fR for limited, non forced receive) release subcommand Allows releasing a user hold which might destroy the snapshot rename subcommand Must also have the \fBmount\fR and \fBcreate\fR ability in the new parent rollback subcommand Must also have the \fBmount\fR ability send subcommand Allows sending a replication stream of a dataset. send:raw subcommand Only allows sending raw replication streams, preventing encrypted datasets being sent in decrypted form. send:encrypted subcommand Only allows sending raw replication streams of encrypted datasets, preventing encrypted datasets being sent in decrypted form and preventing decrypted datasets being sent at all. share subcommand Allows sharing file systems over NFS or SMB protocols snapshot subcommand Must also have the \fBmount\fR ability receive:append other Must also have the \fBmount\fR and \fBcreate\fR ability, limited receive ability (can not do receive -F) groupquota other Allows accessing any \fBgroupquota@\fI…\fR property groupobjquota other Allows accessing any \fBgroupobjquota@\fI…\fR property groupused other Allows reading any \fBgroupused@\fI…\fR property groupobjused other Allows reading any \fBgroupobjused@\fI…\fR property userprop other Allows changing any user property userquota other Allows accessing any \fBuserquota@\fI…\fR property userobjquota other Allows accessing any \fBuserobjquota@\fI…\fR property userused other Allows reading any \fBuserused@\fI…\fR property userobjused other Allows reading any \fBuserobjused@\fI…\fR property projectobjquota other Allows accessing any \fBprojectobjquota@\fI…\fR property projectquota other Allows accessing any \fBprojectquota@\fI…\fR property projectobjused other Allows reading any \fBprojectobjused@\fI…\fR property projectused other Allows reading any \fBprojectused@\fI…\fR property aclinherit property aclmode property acltype property atime property canmount property casesensitivity property checksum property compression property context property copies property dedup property defcontext property devices property dnodesize property encryption property exec property filesystem_limit property fscontext property keyformat property keylocation property logbias property mlslabel property mountpoint property nbmand property normalization property overlay property pbkdf2iters property primarycache property quota property readonly property recordsize property redundant_metadata property refquota property refreservation property relatime property reservation property rootcontext property secondarycache property setuid property sharenfs property sharesmb property snapdev property snapdir property snapshot_limit property special_small_blocks property sync property utf8only property version property volblocksize property volmode property volsize property vscan property xattr property zoned property .TE .It Xo .Nm zfs .Cm allow .Fl c .Ar perm Ns | Ns @ Ns Ar setname Ns Oo , Ns Ar perm Ns | Ns @ Ns .Ar setname Oc Ns … .Ar filesystem Ns | Ns Ar volume .Xc Sets .Qq create time permissions. These permissions are granted .Pq locally to the creator of any newly-created descendent file system. .It Xo .Nm zfs .Cm allow .Fl s No @ Ns Ar setname .Ar perm Ns | Ns @ Ns Ar setname Ns Oo , Ns Ar perm Ns | Ns @ Ns .Ar setname Oc Ns … .Ar filesystem Ns | Ns Ar volume .Xc Defines or adds permissions to a permission set. The set can be used by other .Nm zfs Cm allow commands for the specified file system and its descendants. Sets are evaluated dynamically, so changes to a set are immediately reflected. Permission sets follow the same naming restrictions as ZFS file systems, but the name must begin with .Sy @ , and can be no more than 64 characters long. .It Xo .Nm zfs .Cm unallow .Op Fl dglru .Ar user Ns | Ns Ar group Ns Oo , Ns Ar user Ns | Ns Ar group Oc Ns … .Oo Ar perm Ns | Ns @ Ns Ar setname Ns Oo , Ns Ar perm Ns | Ns @ Ns .Ar setname Oc Ns … Oc .Ar filesystem Ns | Ns Ar volume .Xc .It Xo .Nm zfs .Cm unallow .Op Fl dlr .Fl e Ns | Ns Sy everyone .Oo Ar perm Ns | Ns @ Ns Ar setname Ns Oo , Ns Ar perm Ns | Ns @ Ns .Ar setname Oc Ns … Oc .Ar filesystem Ns | Ns Ar volume .Xc .It Xo .Nm zfs .Cm unallow .Op Fl r .Fl c .Oo Ar perm Ns | Ns @ Ns Ar setname Ns Oo , Ns Ar perm Ns | Ns @ Ns .Ar setname Oc Ns … Oc .Ar filesystem Ns | Ns Ar volume .Xc Removes permissions that were granted with the .Nm zfs Cm allow command. No permissions are explicitly denied, so other permissions granted are still in effect. For example, if the permission is granted by an ancestor. If no permissions are specified, then all permissions for the specified .Ar user , .Ar group , or .Sy everyone are removed. Specifying .Sy everyone .Po or using the .Fl e option .Pc only removes the permissions that were granted to everyone, not all permissions for every user and group. See the .Nm zfs Cm allow command for a description of the .Fl ldugec options. .Bl -tag -width "-r" .It Fl r Recursively remove the permissions from this file system and all descendants. .El .It Xo .Nm zfs .Cm unallow .Op Fl r .Fl s No @ Ns Ar setname .Oo Ar perm Ns | Ns @ Ns Ar setname Ns Oo , Ns Ar perm Ns | Ns @ Ns .Ar setname Oc Ns … Oc .Ar filesystem Ns | Ns Ar volume .Xc Removes permissions from a permission set. If no permissions are specified, then all permissions are removed, thus removing the set entirely. .El . .Sh EXAMPLES .\" These are, respectively, examples 17, 18, 19, 20 from zfs.8 .\" Make sure to update them bidirectionally .Ss Example 1 : No Delegating ZFS Administration Permissions on a ZFS Dataset The following example shows how to set permissions so that user .Ar cindys can create, destroy, mount, and take snapshots on .Ar tank/cindys . The permissions on .Ar tank/cindys are also displayed. .Bd -literal -compact -offset Ds .No # Nm zfs Cm allow Sy cindys create , Ns Sy destroy , Ns Sy mount , Ns Sy snapshot Ar tank/cindys .No # Nm zfs Cm allow Ar tank/cindys ---- Permissions on tank/cindys -------------------------------------- Local+Descendent permissions: user cindys create,destroy,mount,snapshot .Ed .Pp Because the .Ar tank/cindys mount point permission is set to 755 by default, user .Ar cindys will be unable to mount file systems under .Ar tank/cindys . Add an ACE similar to the following syntax to provide mount point access: .Dl # Cm chmod No A+user : Ns Ar cindys Ns :add_subdirectory:allow Ar /tank/cindys . .Ss Example 2 : No Delegating Create Time Permissions on a ZFS Dataset The following example shows how to grant anyone in the group .Ar staff to create file systems in .Ar tank/users . This syntax also allows staff members to destroy their own file systems, but not destroy anyone else's file system. The permissions on .Ar tank/users are also displayed. .Bd -literal -compact -offset Ds .No # Nm zfs Cm allow Ar staff Sy create , Ns Sy mount Ar tank/users .No # Nm zfs Cm allow Fl c Sy destroy Ar tank/users .No # Nm zfs Cm allow Ar tank/users ---- Permissions on tank/users --------------------------------------- Permission sets: destroy Local+Descendent permissions: group staff create,mount .Ed . .Ss Example 3 : No Defining and Granting a Permission Set on a ZFS Dataset The following example shows how to define and grant a permission set on the .Ar tank/users file system. The permissions on .Ar tank/users are also displayed. .Bd -literal -compact -offset Ds .No # Nm zfs Cm allow Fl s No @ Ns Ar pset Sy create , Ns Sy destroy , Ns Sy snapshot , Ns Sy mount Ar tank/users .No # Nm zfs Cm allow staff No @ Ns Ar pset tank/users .No # Nm zfs Cm allow Ar tank/users ---- Permissions on tank/users --------------------------------------- Permission sets: @pset create,destroy,mount,snapshot Local+Descendent permissions: group staff @pset .Ed . .Ss Example 4 : No Delegating Property Permissions on a ZFS Dataset The following example shows to grant the ability to set quotas and reservations on the .Ar users/home file system. The permissions on .Ar users/home are also displayed. .Bd -literal -compact -offset Ds .No # Nm zfs Cm allow Ar cindys Sy quota , Ns Sy reservation Ar users/home .No # Nm zfs Cm allow Ar users/home ---- Permissions on users/home --------------------------------------- Local+Descendent permissions: user cindys quota,reservation cindys% zfs set quota=10G users/home/marks cindys% zfs get quota users/home/marks NAME PROPERTY VALUE SOURCE users/home/marks quota 10G local .Ed . .Ss Example 5 : No Removing ZFS Delegated Permissions on a ZFS Dataset The following example shows how to remove the snapshot permission from the .Ar staff group on the .Sy tank/users file system. The permissions on .Sy tank/users are also displayed. .Bd -literal -compact -offset Ds .No # Nm zfs Cm unallow Ar staff Sy snapshot Ar tank/users .No # Nm zfs Cm allow Ar tank/users ---- Permissions on tank/users --------------------------------------- Permission sets: @pset create,destroy,mount,snapshot Local+Descendent permissions: group staff @pset .Ed ZFS-JAIL(8) - System Manager's Manual # NAME **zfs-jail** - attach or detach ZFS filesystem from FreeBSD jail # SYNOPSIS **zfs **jail**‌** *jailid*|*jailname* *filesystem* **zfs **unjail**‌** *jailid*|*jailname* *filesystem* # DESCRIPTION The **zfs-jail** functionality can be used to assign a dataset onto a running FreeBSD system jail(4), allowing zfs(8) management utilities to be run inside of the jail(4). To allow management of the dataset from within a jail, the **jailed** property should be set and the required devfs.conf(5) entries to expose */dev/zfs* device within the jail must be present. The **quota** property cannot be changed from within a jail. To use this functionality, the jail needs the **allow.mount** and **allow.mount.zfs** parameters set to **1** and the **enforce\_statfs** parameter set to a value lower than **2**. The subcommands are as follows: **jail** *jailid*|*jailname* *filesystem* > Attach the specified > *filesystem* > to the jail identified by JID > *jailid* > or name > *jailname*. > From now on this file system tree can be managed from within a jail if the > **jailed** > property has been set. > You cannot attach a jailed dataset's children to another jail. > You can also not attach the root file system > of the jail or any dataset which needs to be mounted before the zfs rc script > is run inside the jail, as it would be attached unmounted until it is > mounted from the rc script inside the jail. > After a dataset is attached to a jail and the > **jailed** > property is set, a jailed file system cannot be mounted outside the jail, > since the jail administrator might have set the mount point to an unacceptable > value. **unjail** *jailid*|*jailname* *filesystem* > Detaches the specified > *filesystem* > from the jail identified by JID > *jailid* > or name > *jailname*. # SEE ALSO devfs.conf(5), zfsprops(7), jail(8) # CAVEATS The root directory of jail can not be delegated to the jail with this utility because the jail must be running with a valid root directory. Jails are a FreeBSD feature and are not relevant on other platforms. See jail(8) for more information on managing jails, or zfs-zone(8) for the equivalent functionality on Linux. Debian - November 4, 2025 ZFS-LOAD-KEY(8) - System Manager's Manual # NAME **zfs-load-key** - load, unload, or change encryption key of ZFS dataset # SYNOPSIS **zfs** **load-key** \[**-nr**] \[**-L** *keylocation*] **-a**|*filesystem* **zfs** **unload-key** \[**-r**] **-a**|*filesystem* **zfs** **change-key** \[**-l**] \[**-o** *keylocation*=*value*] \[**-o** *keyformat*=*value*] \[**-o** *pbkdf2iters*=*value*] \[**-o** *user:prop*=*value*] *filesystem* **zfs** **change-key** **-i** \[**-l**] *filesystem* # DESCRIPTION **zfs** **load-key** \[**-nr**] \[**-L** *keylocation*] **-a**|*filesystem* > Load the key for > *filesystem*, > allowing it and all children that inherit the > **keylocation** > property to be accessed. > The key will be expected in the format specified by the > **keyformat** > and location specified by the > **keylocation** > property. > Note that if the > **keylocation** > is set to > **prompt** > the terminal will interactively wait for the key to be entered. > Loading a key will not automatically mount the dataset. > If that functionality is desired, > **zfs** **mount** **-l** > will ask for the key and mount the dataset > (see > zfs-mount(8)) > . > For automated key management during user login, > pam\_zfs\_key(8) > can load keys and mount encrypted home directories on systems with PAM support. > Once the key is loaded the > **keystatus** > property will become > **available**. > **-r** > > Recursively loads the keys for the specified filesystem and all descendent > > encryption roots. > **-a** > > Loads the keys for all encryption roots in all imported pools. > **-n** > > Do a dry-run > > ("No-op") > > **load-key**. > > This will cause > > **zfs** > > to simply check that the provided key is correct. > > This command may be run even if the key is already loaded. > **-L** *keylocation* > > Use > > *keylocation* > > instead of the > > **keylocation** > > property. > > This will not change the value of the property on the dataset. > > Note that if used with either > > **-r** > > or > > **-a**, > > *keylocation* > > may only be given as > > **prompt**. **zfs** **unload-key** \[**-r**] **-a**|*filesystem* > Unloads a key from ZFS, removing the ability to access the dataset and all of > its children that inherit the > **keylocation** > property. > This requires that the dataset is not currently open or mounted. > Once the key is unloaded the > **keystatus** > property will become > **unavailable**. > **-r** > > Recursively unloads the keys for the specified filesystem and all descendent > > encryption roots. > **-a** > > Unloads the keys for all encryption roots in all imported pools. **zfs** **change-key** \[**-l**] \[**-o** *keylocation*=*value*] \[**-o** *keyformat*=*value*] \[**-o** *pbkdf2iters*=*value*] \[**-o** *user:prop*=*value*] *filesystem* **zfs** **change-key** **-i** \[**-l**] *filesystem* > Changes the user's key (e.g. a passphrase) used to access a dataset. > This command requires that the existing key for the dataset is already loaded. > This command may also be used to change the > **keylocation**, > **keyformat**, > and > **pbkdf2iters** > properties as needed, as well as set user properties. > If the dataset was not previously an encryption root it will become one. > Alternatively, the > **-i** > flag may be provided to cause an encryption root to inherit the parent's key > instead. > If the user's key is compromised, > **zfs** **change-key** > does not necessarily protect existing or newly-written data from attack. > Newly-written data will continue to be encrypted with the same master key as > the existing data. > The master key is compromised if an attacker obtains a > user key and the corresponding wrapped master key. > Currently, > **zfs** **change-key** > does not overwrite the previous wrapped master key on disk, so it is > accessible via forensic analysis for an indeterminate length of time. > In the event of a master key compromise, ideally the drives should be securely > erased to remove all the old data (which is readable using the compromised > master key), a new pool created, and the data copied back. > This can be approximated in place by creating new datasets, copying the data > (e.g. using **zfs** **send** | **zfs** **recv**), > and then clearing the free space with > **zpool** **trim** **--secure** > if supported by your hardware, otherwise > **zpool** **initialize**. > **-l** > > Ensures the key is loaded before attempting to change the key. > > This is effectively equivalent to running > > **zfs** **load-key** *filesystem*; **zfs** **change-key** *filesystem* > **-o** *property*=*value* > > Allows the user to set encryption key properties > > (**keyformat**, **keylocation**, and **pbkdf2iters**) > > and user properties while changing the key. > > This is the only way to alter > > **keyformat** > > and > > **pbkdf2iters** > > after the dataset has been created. > **-i** > > Indicates that zfs should make > > *filesystem* > > inherit the key of its parent. > > Note that this command can only be run on an encryption root > > that has an encrypted parent. ## Encryption Enabling the **encryption** feature allows for the creation of encrypted filesystems and volumes. ZFS will encrypt file and volume data, file attributes, ACLs, permission bits, directory listings, FUID mappings, and **userused**/**groupused** data. ZFS will not encrypt metadata related to the pool structure, including dataset and snapshot names, dataset hierarchy, properties, file size, file holes, and deduplication tables (though the deduplicated data itself is encrypted). Key rotation is managed by ZFS. Changing the user's key (e.g. a passphrase) does not require re-encrypting the entire dataset. Datasets can be scrubbed, resilvered, renamed, and deleted without the encryption keys being loaded (see the **load-key** subcommand for more info on key loading). Creating an encrypted dataset requires specifying the **encryption** and **keyformat** properties at creation time, along with an optional **keylocation** and **pbkdf2iters**. After entering an encryption key, the created dataset will become an encryption root. Any descendant datasets will inherit their encryption key from the encryption root by default, meaning that loading, unloading, or changing the key for the encryption root will implicitly do the same for all inheriting datasets. If this inheritance is not desired, simply supply a **keyformat** when creating the child dataset or use **zfs** **change-key** to break an existing relationship, creating a new encryption root on the child. Note that the child's **keyformat** may match that of the parent while still creating a new encryption root, and that changing the **encryption** property alone does not create a new encryption root; this would simply use a different cipher suite with the same key as its encryption root. The one exception is that clones will always use their origin's encryption key. As a result of this exception, some encryption-related properties (namely **keystatus**, **keyformat**, **keylocation**, and **pbkdf2iters**) do not inherit like other ZFS properties and instead use the value determined by their encryption root. Encryption root inheritance can be tracked via the read-only **encryptionroot** property. Encryption changes the behavior of a few ZFS operations. Encryption is applied after compression so compression ratios are preserved. Normally checksums in ZFS are 256 bits long, but for encrypted data the checksum is 128 bits of the user-chosen checksum and 128 bits of MAC from the encryption suite, which provides additional protection against maliciously altered data. Deduplication is still possible with encryption enabled but for security, datasets will only deduplicate against themselves, their snapshots, and their clones. There are a few limitations on encrypted datasets. Encrypted data cannot be embedded via the **embedded\_data** feature. Encrypted datasets may not have **copies**=*3* since the implementation stores some encryption metadata where the third copy would normally be. Since compression is applied before encryption, datasets may be vulnerable to a CRIME-like attack if applications accessing the data allow for it. Deduplication with encryption will leak information about which blocks are equivalent in a dataset and will incur an extra CPU cost for each block written. # SEE ALSO zfsprops(7), pam\_zfs\_key(8), zfs-create(8), zfs-set(8) Debian - January 30, 2026 ZFS-MOUNT(8) - System Manager's Manual # NAME **zfs-mount** - manage mount state of ZFS filesystems # SYNOPSIS **zfs** **mount** \[**-j**] **zfs** **mount** \[**-Oflv**] \[**-o** *options*] **-a**|**-R** *filesystem*|*filesystem* **zfs** **unmount** \[**-fu**] **-a**|*filesystem*|*mountpoint* # DESCRIPTION **zfs** **mount** \[**-j**] > Displays all ZFS file systems currently mounted. > **-j**, **--json** > > Displays all mounted file systems in JSON format. **zfs** **mount** \[**-Oflv**] \[**-o** *options*] **-a**|**-R** *filesystem*|*filesystem* > Mount ZFS filesystem on a path described by its > **mountpoint** > property, if the path exists and is empty. > If > **mountpoint** > is set to > *legacy*, > the filesystem should be instead mounted using > mount(8). > **-O** > > Perform an overlay mount. > > Allows mounting in non-empty > > **mountpoint**. > > See > > mount(8) > > for more information. > **-a** > > Mount all available ZFS file systems. > > Invoked automatically as part of the boot process if configured. > **-R** > > Mount the specified filesystems along with all their children. > *filesystem* > > Mount the specified filesystem. > **-o** *options* > > An optional, comma-separated list of mount options to use temporarily for the > > duration of the mount. > > See the > > *Temporary Mount Point Properties* > > section of > > zfsprops(7) > > for details. > **-l** > > Load keys for encrypted filesystems as they are being mounted. > > This is equivalent to executing > > **zfs** **load-key** > > on each encryption root before mounting it. > > Note that if a filesystem has > > **keylocation**=**prompt**, > > this will cause the terminal to interactively block after asking for the key. > > On systems with PAM support, > > pam\_zfs\_key(8) > > can automate this process during user login. > **-v** > > Report mount progress. > **-f** > > Attempt to force mounting of all filesystems, even those that couldn't normally > > be mounted (e.g. redacted datasets). **zfs** **unmount** \[**-fu**] **-a**|*filesystem*|*mountpoint* > Unmounts currently mounted ZFS file systems. > **-a** > > Unmount all available ZFS file systems. > > Invoked automatically as part of the shutdown process. > **-f** > > Forcefully unmount the file system, even if it is currently in use. > > This option is not supported on Linux. > **-u** > > Unload keys for any encryption roots unmounted by this command. > *filesystem*|*mountpoint* > > Unmount the specified filesystem. > > The command can also be given a path to a ZFS file system mount point on the > > system. # SEE ALSO pam\_zfs\_key(8) Debian - October 12, 2024 ZFS-ZONE(8) - System Manager's Manual # NAME **zfs-zone**, **zfs-unzone** - attach and detach ZFS filesystems to user namespaces # SYNOPSIS **zfs **zone**‌** *nsfile* *filesystem* **zfs **unzone**‌** *nsfile* *filesystem* # DESCRIPTION **zfs** **zone** *nsfile* *filesystem* > Attach the specified > *filesystem* > to the user namespace identified by > *nsfile*. > From now on this file system tree can be managed from within a user namespace > if the > **zoned** > property has been set. > You cannot attach a zoned dataset's children to another user namespace. > You can also not attach the root file system > of the user namespace or any dataset > which needs to be mounted before the zfs service > is run inside the user namespace, > as it would be attached unmounted until it is > mounted from the service inside the user namespace. > To allow management of the dataset from within a user namespace, the > **zoned** > property has to be set and the user namespaces needs access to the > */dev/zfs* > device. > The > **quota** > property cannot be changed from within a user namespace. > After a dataset is attached to a user namespace and the > **zoned** > property is set, > a zoned file system cannot be mounted outside the user namespace, > since the user namespace administrator might have set the mount point > to an unacceptable value. **zfs** **unzone** *nsfile* *filesystem* > Detach the specified > *filesystem* > from the user namespace identified by > *nsfile*. # EXAMPLES ## Example 1: Delegating a Dataset to a User Namespace The following example delegates the *tank/users* dataset to a user namespace identified by user namespace file */proc/1234/ns/user*. # **zfs** **zone** */proc/1234/ns/user* *tank/users* # SEE ALSO zfsprops(7), zfs-allow(8) For rootless container use cases where the namespace is ephemeral, consider using the **zoned\_uid** property instead, which delegates to all namespaces owned by a UID rather than requiring attachment to a specific namespace file. The **zoned\_uid** property uses a three-layer additive authorization model (UID match, dsl\_deleg grants, capability tiers) described in zfsprops(7). Debian - July 11, 2022 ZFS-UPGRADE(8) - System Manager's Manual # NAME **zfs-upgrade** - manage on-disk version of ZFS filesystems # SYNOPSIS **zfs** **upgrade** **zfs** **upgrade** **-v** **zfs** **upgrade** \[**-r**] \[**-V** *version*] **-a**|*filesystem* # DESCRIPTION **zfs** **upgrade** > Displays a list of file systems that are not the most recent version. **zfs** **upgrade** **-v** > Displays a list of currently supported file system versions. **zfs** **upgrade** \[**-r**] \[**-V** *version*] **-a**|*filesystem* > Upgrades file systems to a new on-disk version. > Once this is done, the file systems will no longer be accessible on systems > running older versions of ZFS. > **zfs** **send** > streams generated from new snapshots of these file systems cannot be accessed on > systems running older versions of ZFS. > In general, the file system version is independent of the pool version. > See > zpool-features(7) > for information on features of ZFS storage pools. > In some cases, the file system version and the pool version are interrelated and > the pool version must be upgraded before the file system version can be > upgraded. > **-V** *version* > > Upgrade to > > *version*. > > If not specified, upgrade to the most recent version. > > This > > option can only be used to increase the version number, and only up to the most > > recent version supported by this version of ZFS. > **-a** > > Upgrade all file systems on all imported pools. > *filesystem* > > Upgrade the specified file system. > **-r** > > Upgrade the specified file system and all descendent file systems. # SEE ALSO zpool-upgrade(8) Debian - July 11, 2022 ZFS-USERSPACE(8) - System Manager's Manual # NAME **zfs-userspace** - display space and quotas of ZFS dataset # SYNOPSIS **zfs** **userspace** \[**-Hinp**] \[**-o** *field*\[,*field*]…] \[**-s** *field*]… \[**-S** *field*]… \[**-t** *type*\[,*type*]…] *filesystem*|*snapshot*|*path* **zfs** **groupspace** \[**-Hinp**] \[**-o** *field*\[,*field*]…] \[**-s** *field*]… \[**-S** *field*]… \[**-t** *type*\[,*type*]…] *filesystem*|*snapshot*|*path* **zfs** **projectspace** \[**-Hp**] \[**-o** *field*\[,*field*]…] \[**-s** *field*]… \[**-S** *field*]… *filesystem*|*snapshot*|*path* # DESCRIPTION **zfs** **userspace** \[**-Hinp**] \[**-o** *field*\[,*field*]…] \[**-s** *field*]… \[**-S** *field*]… \[**-t** *type*\[,*type*]…] *filesystem*|*snapshot*|*path* > Displays space consumed by, and quotas on, each user in the specified > filesystem, > snapshot, or path. > If a path is given, the filesystem that contains that path will be used. > This corresponds to the > **userused@**‌*user*, > **userobjused@**‌*user*, > **userquota@**‌*user*, > and > **userobjquota@**‌*user* > properties. > **-H** > > Do not print headers, use tab-delimited output. > **-S** *field* > > Sort by this field in reverse order. > > See > > **-s**. > **-i** > > Translate SID to POSIX ID. > > The POSIX ID may be ephemeral if no mapping exists. > > Normal POSIX interfaces > > (like stat(2), **ls** **-l**) > > perform this translation, so the > > **-i** > > option allows the output from > > **zfs** **userspace** > > to be compared directly with those utilities. > > However, > > **-i** > > may lead to confusion if some files were created by an SMB user before a > > SMB-to-POSIX name mapping was established. > > In such a case, some files will be owned by the SMB entity and some by the POSIX > > entity. > > However, the > > **-i** > > option will report that the POSIX entity has the total usage and quota for both. > **-n** > > Print numeric ID instead of user/group name. > **-o** *field*\[,*field*]… > > Display only the specified fields from the following set: > > **type**, > > **name**, > > **used**, > > **quota**. > > The default is to display all fields. > **-p** > > Use exact > > (parsable) > > numeric output. > **-s** *field* > > Sort output by this field. > > The > > **-s** > > and > > **-S** > > flags may be specified multiple times to sort first by one field, then by > > another. > > The default is > > **-s** **type** **-s** **name**. > **-t** *type*\[,*type*]… > > Print only the specified types from the following set: > > **all**, > > **posixuser**, > > **smbuser**, > > **posixgroup**, > > **smbgroup**. > > The default is > > **-t** **posixuser**,**smbuser**. > > The default can be changed to include group types. **zfs** **groupspace** \[**-Hinp**] \[**-o** *field*\[,*field*]…] \[**-s** *field*]… \[**-S** *field*]… \[**-t** *type*\[,*type*]…] *filesystem*|*snapshot* > Displays space consumed by, and quotas on, each group in the specified > filesystem or snapshot. > This subcommand is identical to > **userspace**, > except that the default types to display are > **-t** **posixgroup**,**smbgroup**. **zfs** **projectspace** \[**-Hp**] \[**-o** *field*\[,*field*]…] \[**-s** *field*]… \[**-S** *field*]… *filesystem*|*snapshot*|*path* > Displays space consumed by, and quotas on, each project in the specified > filesystem or snapshot. > This subcommand is identical to > **userspace**, > except that the project identifier is a numeral, not a name. > So need neither the option > **-i** > for SID to POSIX ID nor > **-n** > for numeric ID, nor > **-t** > for types. # SEE ALSO zfsprops(7), zfs-set(8) Debian - July 11, 2022 ZFS-WAIT(8) - System Manager's Manual # NAME **zfs-wait** - wait for activity in ZFS filesystem to stop # SYNOPSIS **zfs** **wait** \[**-t** *activity*\[,*activity*]…] *filesystem* # DESCRIPTION Waits until all background activity of the given types has ceased in the given filesystem. The activity could cease because it has completed or because the filesystem has been destroyed or unmounted. If no activities are specified, the command waits until background activity of every type listed below has ceased. If there is no activity of the given types in progress, the command returns immediately. These are the possible values for *activity*, along with what each one waits for: **deleteq** > The filesystem's internal delete queue to empty Note that the internal delete queue does not finish draining until all large files have had time to be fully destroyed and all open file handles to unlinked files are closed. # SEE ALSO lsof(8) Debian - July 11, 2022 ZFS-ZONE(8) - System Manager's Manual # NAME **zfs-zone**, **zfs-unzone** - attach and detach ZFS filesystems to user namespaces # SYNOPSIS **zfs **zone**‌** *nsfile* *filesystem* **zfs **unzone**‌** *nsfile* *filesystem* # DESCRIPTION **zfs** **zone** *nsfile* *filesystem* > Attach the specified > *filesystem* > to the user namespace identified by > *nsfile*. > From now on this file system tree can be managed from within a user namespace > if the > **zoned** > property has been set. > You cannot attach a zoned dataset's children to another user namespace. > You can also not attach the root file system > of the user namespace or any dataset > which needs to be mounted before the zfs service > is run inside the user namespace, > as it would be attached unmounted until it is > mounted from the service inside the user namespace. > To allow management of the dataset from within a user namespace, the > **zoned** > property has to be set and the user namespaces needs access to the > */dev/zfs* > device. > The > **quota** > property cannot be changed from within a user namespace. > After a dataset is attached to a user namespace and the > **zoned** > property is set, > a zoned file system cannot be mounted outside the user namespace, > since the user namespace administrator might have set the mount point > to an unacceptable value. **zfs** **unzone** *nsfile* *filesystem* > Detach the specified > *filesystem* > from the user namespace identified by > *nsfile*. # EXAMPLES ## Example 1: Delegating a Dataset to a User Namespace The following example delegates the *tank/users* dataset to a user namespace identified by user namespace file */proc/1234/ns/user*. # **zfs** **zone** */proc/1234/ns/user* *tank/users* # SEE ALSO zfsprops(7), zfs-allow(8) For rootless container use cases where the namespace is ephemeral, consider using the **zoned\_uid** property instead, which delegates to all namespaces owned by a UID rather than requiring attachment to a specific namespace file. The **zoned\_uid** property uses a three-layer additive authorization model (UID match, dsl\_deleg grants, capability tiers) described in zfsprops(7). Debian - July 11, 2022 ZFS(8) - System Manager's Manual # NAME **zfs** - configure ZFS datasets # SYNOPSIS **zfs** **-?V** **zfs** **version** \[**-j**] **zfs** **subcommand** \[*arguments*] # DESCRIPTION The **zfs** command configures ZFS datasets within a ZFS storage pool, as described in zpool(8). A dataset is identified by a unique path within the ZFS namespace: > *pool*\[**/**‌*component*]**/**‌*component* for example: rpool/var/log The maximum length of a dataset name is **ZFS\_MAX\_DATASET\_NAME\_LEN** - 1 ASCII characters (currently 255) satisfying **\[A-Za-z\_.:/ -]**. Additionally snapshots are allowed to contain a single **@** character, while bookmarks are allowed to contain a single **#** character. **/** is used as separator between components. The maximum amount of nesting allowed in a path is **zfs\_max\_dataset\_nesting** levels deep. ZFS tunables (**zfs\_\*‌**) are explained in zfs(4). A dataset can be one of the following: **file system** > Can be mounted within the standard system namespace and behaves like other file > systems. > While ZFS file systems are designed to be POSIX-compliant, known issues exist > that prevent compliance in some cases. > Applications that depend on standards conformance might fail due to non-standard > behavior when checking file system free space. **volume** > A logical volume exported as a raw or block device. > This type of dataset should only be used when a block device is required. > File systems are typically used in most environments. **snapshot** > A read-only version of a file system or volume at a given point in time. > It is specified as > *filesystem*@*name* > or > *volume*@*name*. **bookmark** > Much like a > **snapshot**, > but without the hold on on-disk data. > It can be used as the source of a send (but not for a receive). > It is specified as > *filesystem*#*name* > or > *volume*#*name*. See zfsconcepts(7) for details. ## Properties Properties are divided into two types: native properties and user-defined (or "user") properties. Native properties either export internal statistics or control ZFS behavior. In addition, native properties are either editable or read-only. User properties have no effect on ZFS behavior, but you can use them to annotate datasets in a way that is meaningful in your environment. For more information about properties, see zfsprops(7). ## Encryption Enabling the **encryption** feature allows for the creation of encrypted filesystems and volumes. ZFS will encrypt file and zvol data, file attributes, ACLs, permission bits, directory listings, FUID mappings, and **userused**/**groupused**/**projectused** data. For an overview of encryption, see zfs-load-key(8). # SUBCOMMANDS All subcommands that modify state are logged persistently to the pool in their original form. **zfs** **-**? > Displays a help message. **zfs** **-V**, **--version** **zfs** **version** \[**-j**] > Displays the software version of the > **zfs** > userland utility and the zfs kernel module. > Use > **-j** > option to output in JSON format. ## Dataset Management zfs-list(8) > Lists the property information for the given datasets in tabular form. zfs-create(8) > Creates a new ZFS file system or volume. zfs-destroy(8) > Destroys the given dataset(s), snapshot(s), or bookmark. zfs-rename(8) > Renames the given dataset (filesystem or snapshot). zfs-upgrade(8) > Manage upgrading the on-disk version of filesystems. ## Snapshots zfs-snapshot(8) > Creates snapshots with the given names. zfs-rollback(8) > Roll back the given dataset to a previous snapshot. zfs-hold(8)/zfs-release(8) > Add or remove a hold reference to the specified snapshot or snapshots. > If a hold exists on a snapshot, attempts to destroy that snapshot by using the > **zfs** **destroy** > command return > **EBUSY**. zfs-diff(8) > Display the difference between a snapshot of a given filesystem and another > snapshot of that filesystem from a later time or the current contents of the > filesystem. ## Clones zfs-clone(8) > Creates a clone of the given snapshot. zfs-promote(8) > Promotes a clone file system to no longer be dependent on its > "origin" > snapshot. ## Send & Receive zfs-send(8) > Generate a send stream, which may be of a filesystem, and may be incremental > from a bookmark. zfs-receive(8) > Creates a snapshot whose contents are as specified in the stream provided on > standard input. > If a full stream is received, then a new file system is created as well. > Streams are created using the > zfs-send(8) > subcommand, which by default creates a full stream. zfs-bookmark(8) > Creates a new bookmark of the given snapshot or bookmark. > Bookmarks mark the point in time when the snapshot was created, and can be used > as the incremental source for a > **zfs** **send** > command. zfs-redact(8) > Generate a new redaction bookmark. > This feature can be used to allow clones of a filesystem to be made available on > a remote system, in the case where their parent need not (or needs to not) be > usable. ## Properties zfs-get(8) > Displays properties for the given datasets. zfs-set(8) > Sets the property or list of properties to the given value(s) for each dataset. zfs-inherit(8) > Clears the specified property, causing it to be inherited from an ancestor, > restored to default if no ancestor has the property set, or with the > **-S** > option reverted to the received value if one exists. ## Quotas zfs-userspace(8)/zfs-groupspace(8)/zfs-projectspace(8) > Displays space consumed by, and quotas on, each user, group, or project > in the specified filesystem or snapshot. zfs-project(8) > List, set, or clear project ID and/or inherit flag on the files or directories. ## Mountpoints zfs-mount(8) > Displays all ZFS file systems currently mounted, or mount ZFS filesystem > on a path described by its > **mountpoint** > property. zfs-unmount(8) > Unmounts currently mounted ZFS file systems. ## Shares zfs-share(8) > Shares available ZFS file systems. zfs-unshare(8) > Unshares currently shared ZFS file systems. ## Delegated Administration zfs-allow(8) > Delegate permissions on the specified filesystem or volume. zfs-unallow(8) > Remove delegated permissions on the specified filesystem or volume. ## Encryption zfs-change-key(8) > Add or change an encryption key on the specified dataset. zfs-load-key(8) > Load the key for the specified encrypted dataset, enabling access. zfs-unload-key(8) > Unload a key for the specified dataset, > removing the ability to access the dataset. ## Channel Programs zfs-program(8) > Execute ZFS administrative operations > programmatically via a Lua script-language channel program. ## Data rewrite zfs-rewrite(8) > Rewrite specified files without modification. ## Jails zfs-jail(8) > Attaches a filesystem to a jail. zfs-unjail(8) > Detaches a filesystem from a jail. ## Waiting zfs-wait(8) > Wait for background activity in a filesystem to complete. # EXIT STATUS The **zfs** utility exits **0** on success, **1** if an error occurs, and **2** if invalid command line options were specified. # EXAMPLES ## Example 1: Creating a ZFS File System Hierarchy The following commands create a file system named *pool/home* and a file system named *pool/home/bob*. The mount point */export/home* is set for the parent file system, and is automatically inherited by the child file system. # **zfs** **create** *pool/home* # **zfs** **set** **mountpoint**=*/export/home pool/home* # **zfs** **create** *pool/home/bob* ## Example 2: Creating a ZFS Snapshot The following command creates a snapshot named *yesterday*. This snapshot is mounted on demand in the *.zfs/snapshot* directory at the root of the *pool/home/bob* file system. # **zfs** **snapshot** *pool/home/bob*@*yesterday* ## Example 3: Creating and Destroying Multiple Snapshots The following command creates snapshots named *yesterday* of *pool/home* and all of its descendent file systems. Each snapshot is mounted on demand in the *.zfs/snapshot* directory at the root of its file system. The second command destroys the newly created snapshots. # **zfs** **snapshot** **-r** *pool/home*@*yesterday* # **zfs** **destroy** **-r** *pool/home*@*yesterday* ## Example 4: Disabling and Enabling File System Compression The following command disables the **compression** property for all file systems under *pool/home*. The next command explicitly enables **compression** for *pool/home/anne*. # **zfs** **set** **compression**=**off** *pool/home* # **zfs** **set** **compression**=**on** *pool/home/anne* ## Example 5: Listing ZFS Datasets The following command lists all active file systems and volumes in the system. Snapshots are displayed if **listsnaps**=**on**. The default is **off**. See zpoolprops(7) for more information on pool properties. # **zfs** **list** NAME USED AVAIL REFER MOUNTPOINT pool 450K 457G 18K /pool pool/home 315K 457G 21K /export/home pool/home/anne 18K 457G 18K /export/home/anne pool/home/bob 276K 457G 276K /export/home/bob ## Example 6: Setting a Quota on a ZFS File System The following command sets a quota of 50 Gbytes for *pool/home/bob*: # **zfs** **set** **quota**=*50G pool/home/bob* ## Example 7: Listing ZFS Properties The following command lists all properties for *pool/home/bob*: # **zfs** **get** **all** *pool/home/bob* NAME PROPERTY VALUE SOURCE pool/home/bob type filesystem - pool/home/bob creation Tue Jul 21 15:53 2009 - pool/home/bob used 21K - pool/home/bob available 20.0G - pool/home/bob referenced 21K - pool/home/bob compressratio 1.00x - pool/home/bob mounted yes - pool/home/bob quota 20G local pool/home/bob reservation none default pool/home/bob recordsize 128K default pool/home/bob mountpoint /pool/home/bob default pool/home/bob sharenfs off default pool/home/bob checksum on default pool/home/bob compression on local pool/home/bob atime on default pool/home/bob devices on default pool/home/bob exec on default pool/home/bob setuid on default pool/home/bob readonly off default pool/home/bob zoned off default pool/home/bob snapdir hidden default pool/home/bob acltype off default pool/home/bob aclmode discard default pool/home/bob aclinherit restricted default pool/home/bob canmount on default pool/home/bob xattr on default pool/home/bob copies 1 default pool/home/bob version 4 - pool/home/bob utf8only off - pool/home/bob normalization none - pool/home/bob casesensitivity sensitive - pool/home/bob vscan off default pool/home/bob nbmand off default pool/home/bob sharesmb off default pool/home/bob refquota none default pool/home/bob refreservation none default pool/home/bob primarycache all default pool/home/bob secondarycache all default pool/home/bob usedbysnapshots 0 - pool/home/bob usedbydataset 21K - pool/home/bob usedbychildren 0 - pool/home/bob usedbyrefreservation 0 - The following command gets a single property value: # **zfs** **get** **-H** **-o** **value compression** *pool/home/bob* on The following command lists all properties with local settings for *pool/home/bob*: # **zfs** **get** **-r** **-s** **local** **-o** **name**,**property**,**value all** *pool/home/bob* NAME PROPERTY VALUE pool/home/bob quota 20G pool/home/bob compression on ## Example 8: Rolling Back a ZFS File System The following command reverts the contents of *pool/home/anne* to the snapshot named *yesterday*, deleting all intermediate snapshots: # **zfs** **rollback** **-r** *pool/home/anne*@*yesterday* ## Example 9: Creating a ZFS Clone The following command creates a writable file system whose initial contents are the same as *pool/home/bob@yesterday*. # **zfs** **clone** *pool/home/bob@yesterday pool/clone* ## Example 10: Promoting a ZFS Clone The following commands illustrate how to test out changes to a file system, and then replace the original file system with the changed one, using clones, clone promotion, and renaming: # **zfs** **create** *pool/project/production* populate /pool/project/production with data # **zfs** **snapshot** *pool/project/production*@*today* # **zfs** **clone** *pool/project/production@today pool/project/beta* make changes to /pool/project/beta and test them # **zfs** **promote** *pool/project/beta* # **zfs** **rename** *pool/project/production pool/project/legacy* # **zfs** **rename** *pool/project/beta pool/project/production* once the legacy version is no longer needed, it can be destroyed # **zfs** **destroy** *pool/project/legacy* ## Example 11: Inheriting ZFS Properties The following command causes *pool/home/bob* and *pool/home/anne* to inherit the **checksum** property from their parent. # **zfs** **inherit** **checksum** *pool/home/bob pool/home/anne* ## Example 12: Remotely Replicating ZFS Data The following commands send a full stream and then an incremental stream to a remote machine, restoring them into *poolB/received/fs@a* and *poolB/received/fs@b*, respectively. *poolB* must contain the file system *poolB/received*, and must not initially contain *poolB/received/fs*. # **zfs** **send** *pool/fs@a* | **ssh** *host* **zfs** **receive** *poolB/received/fs*@*a* # **zfs** **send** **-i** *a pool/fs@b* | **ssh** *host* **zfs** **receive** *poolB/received/fs* ## Example 13: Using the **zfs** **receive** **-d** Option The following command sends a full stream of *poolA/fsA/fsB@snap* to a remote machine, receiving it into *poolB/received/fsA/fsB@snap*. The *fsA/fsB@snap* portion of the received snapshot's name is determined from the name of the sent snapshot. *poolB* must contain the file system *poolB/received*. If *poolB/received/fsA* does not exist, it is created as an empty file system. # **zfs** **send** *poolA/fsA/fsB@snap* | **ssh** *host* **zfs** **receive** **-d** *poolB/received* ## Example 14: Setting User Properties The following example sets the user-defined *com.example*:*department* property for a dataset: # **zfs** **set** *com.example*:*department*=*12345 tank/accounting* ## Example 15: Performing a Rolling Snapshot The following example shows how to maintain a history of snapshots with a consistent naming scheme. To keep a week's worth of snapshots, the user destroys the oldest snapshot, renames the remaining snapshots, and then creates a new snapshot, as follows: # **zfs** **destroy** **-r** *pool/users@7daysago* # **zfs** **rename** **-r** *pool/users@6daysago* @*7daysago* # **zfs** **rename** **-r** *pool/users@5daysago* @*6daysago* # **zfs** **rename** **-r** *pool/users@4daysago* @*5daysago* # **zfs** **rename** **-r** *pool/users@3daysago* @*4daysago* # **zfs** **rename** **-r** *pool/users@2daysago* @*3daysago* # **zfs** **rename** **-r** *pool/users@yesterday* @*2daysago* # **zfs** **rename** **-r** *pool/users@today* @*yesterday* # **zfs** **snapshot** **-r** *pool/users*@*today* ## Example 16: Setting sharenfs Property Options on a ZFS File System The following commands show how to set **sharenfs** property options to enable read-write access for a set of IP addresses and to enable root access for system "neo" on the *tank/home* file system: # **zfs** **set** **sharenfs**='*rw*=@123.123.0.0/16:[::1],root=*neo*' tank/home If you are using DNS for host name resolution, specify the fully-qualified hostname. ## Example 17: Delegating ZFS Administration Permissions on a ZFS Dataset The following example shows how to set permissions so that user *cindys* can create, destroy, mount, and take snapshots on *tank/cindys*. The permissions on *tank/cindys* are also displayed. # **zfs** **allow** **cindys create**,**destroy**,**mount**,**snapshot** *tank/cindys* # **zfs** **allow** *tank/cindys* ---- Permissions on tank/cindys -------------------------------------- Local+Descendent permissions: user cindys create,destroy,mount,snapshot Because the *tank/cindys* mount point permission is set to 755 by default, user *cindys* will be unable to mount file systems under *tank/cindys*. Add an ACE similar to the following syntax to provide mount point access: # **chmod** A+user:*cindys*:add_subdirectory:allow */tank/cindys* ## Example 18: Delegating Create Time Permissions on a ZFS Dataset The following example shows how to grant anyone in the group *staff* to create file systems in *tank/users*. This syntax also allows staff members to destroy their own file systems, but not destroy anyone else's file system. The permissions on *tank/users* are also displayed. # **zfs** **allow** *staff* **create**,**mount** *tank/users* # **zfs** **allow** **-c** **destroy** *tank/users* # **zfs** **allow** *tank/users* ---- Permissions on tank/users --------------------------------------- Permission sets: destroy Local+Descendent permissions: group staff create,mount ## Example 19: Defining and Granting a Permission Set on a ZFS Dataset The following example shows how to define and grant a permission set on the *tank/users* file system. The permissions on *tank/users* are also displayed. # **zfs** **allow** **-s** @*pset* **create**,**destroy**,**snapshot**,**mount** *tank/users* # **zfs** **allow staff** @*pset tank/users* # **zfs** **allow** *tank/users* ---- Permissions on tank/users --------------------------------------- Permission sets: @pset create,destroy,mount,snapshot Local+Descendent permissions: group staff @pset ## Example 20: Delegating Property Permissions on a ZFS Dataset The following example shows to grant the ability to set quotas and reservations on the *users/home* file system. The permissions on *users/home* are also displayed. # **zfs** **allow** *cindys* **quota**,**reservation** *users/home* # **zfs** **allow** *users/home* ---- Permissions on users/home --------------------------------------- Local+Descendent permissions: user cindys quota,reservation cindys% zfs set quota=10G users/home/marks cindys% zfs get quota users/home/marks NAME PROPERTY VALUE SOURCE users/home/marks quota 10G local ## Example 21: Removing ZFS Delegated Permissions on a ZFS Dataset The following example shows how to remove the snapshot permission from the *staff* group on the **tank/users** file system. The permissions on **tank/users** are also displayed. # **zfs** **unallow** *staff* **snapshot** *tank/users* # **zfs** **allow** *tank/users* ---- Permissions on tank/users --------------------------------------- Permission sets: @pset create,destroy,mount,snapshot Local+Descendent permissions: group staff @pset ## Example 22: Showing the differences between a snapshot and a ZFS Dataset The following example shows how to see what has changed between a prior snapshot of a ZFS dataset and its current state. The **-F** option is used to indicate type information for the files affected. # **zfs** **diff** **-F** *tank/test@before tank/test* M / /tank/test/ M F /tank/test/linked (+1) R F /tank/test/oldname -> /tank/test/newname - F /tank/test/deleted + F /tank/test/created M F /tank/test/modified ## Example 23: Creating a bookmark The following example creates a bookmark to a snapshot. This bookmark can then be used instead of a snapshot in send streams. # **zfs** **bookmark** *rpool*@*snapshot rpool*#*bookmark* ## Example 24: Setting **sharesmb** Property Options on a ZFS File System The following example show how to share SMB filesystem through ZFS. Note that a user and their password must be given. # **smbmount** *//127.0.0.1/share_tmp /mnt/tmp* **-o** user=workgroup/turbo,password=obrut,uid=1000 Minimal */etc/samba/smb.conf* configuration is required, as follows. Samba will need to bind to the loopback interface for the ZFS utilities to communicate with Samba. This is the default behavior for most Linux distributions. Samba must be able to authenticate a user. This can be done in a number of ways (passwd(5), LDAP, smbpasswd(5), &c.). How to do this is outside the scope of this document – refer to smb.conf(5) for more information. See the *USERSHARES* section for all configuration options, in case you need to modify any options of the share afterwards. Do note that any changes done with the net(8) command will be undone if the share is ever unshared (like via a reboot). # ENVIRONMENT VARIABLES **ZFS\_COLOR** > Use ANSI color in > **zfs** **diff** > and > **zfs** **list** > output. **ZFS\_MOUNT\_HELPER** > Cause > **zfs** **mount** > to use > mount(8) > to mount ZFS datasets. > This option is provided for backwards compatibility with older ZFS versions. **ZFS\_SET\_PIPE\_MAX** > Tells > **zfs** > to set the maximum pipe size for sends/receives. > Disabled by default on Linux > due to an unfixed deadlock in Linux's pipe size handling code. **ZFS\_MODULE\_TIMEOUT** > Time, in seconds, to wait for > */dev/zfs* > to appear. > Defaults to > **10**, > max > **600** (10 minutes). > If > <**0**, > wait forever; if > **0**, > don't wait. # INTERFACE STABILITY **Committed**. # SEE ALSO attr(1), gzip(1), ssh(1), chmod(2), fsync(2), stat(2), write(2), acl(5), attributes(5), exports(5), zfsconcepts(7), zfsprops(7), exportfs(8), mount(8), net(8), pam\_zfs\_key(8), selinux(8), zfs-allow(8), zfs-bookmark(8), zfs-change-key(8), zfs-clone(8), zfs-create(8), zfs-destroy(8), zfs-diff(8), zfs-get(8), zfs-groupspace(8), zfs-hold(8), zfs-inherit(8), zfs-jail(8), zfs-list(8), zfs-load-key(8), zfs-mount(8), zfs-program(8), zfs-project(8), zfs-projectspace(8), zfs-promote(8), zfs-receive(8), zfs-redact(8), zfs-release(8), zfs-rename(8), zfs-rollback(8), zfs-send(8), zfs-set(8), zfs-share(8), zfs-snapshot(8), zfs-unallow(8), zfs-unjail(8), zfs-unload-key(8), zfs-unmount(8), zfs-unshare(8), zfs-upgrade(8), zfs-userspace(8), zfs-wait(8), zpool(8) Debian - May 12, 2025 ZFS\_IDS\_TO\_PATH(8) - System Manager's Manual # NAME **zfs\_ids\_to\_path** - convert objset and object ids to names and paths # SYNOPSIS **zfs\_ids\_to\_path** \[**-v**] *pool* *objset-id* *object-id* # DESCRIPTION The **zfs\_ids\_to\_path** utility converts a provided objset and object ids into a path to the file they refer to. **-v** > Verbose. > Print the dataset name and the file path within the dataset separately. > This will work correctly even if the dataset is not mounted. # SEE ALSO zdb(8), zfs(8) Debian - July 11, 2022 ZFS\_PREPARE\_DISK(8) - System Manager's Manual # NAME **zfs\_prepare\_disk** - special script that gets run before bringing a disk into a pool # DESCRIPTION **zfs\_prepare\_disk** is an optional script that gets called by libzfs before bringing a disk into a pool. It can be modified by the user to run whatever commands are necessary to prepare a disk for inclusion into the pool. For example, users can add lines to **zfs\_prepare\_disk** to do things like update the drive's firmware or check the drive's health. **zfs\_prepare\_disk** is optional and can be removed if not needed. libzfs will look for the script at @zfsexecdir@/zfs\_prepare\_disk. ## Properties **zfs\_prepare\_disk** will be passed the following environment variables: **POOL\_NAME** > Name of the pool **VDEV\_PATH** > Path to the disk (like /dev/sda) **VDEV\_PREPARE** > Reason why the disk is being prepared for inclusion > ('create', 'add', 'replace', or 'autoreplace'). > This can be useful if you only want the script to be run under certain actions. **VDEV\_UPATH** > Path to one of the underlying devices for the > disk. > For multipath this would return one of the /dev/sd\* paths to the disk. > If the device is not a device mapper device, then > **VDEV\_UPATH** > just returns the same value as > **VDEV\_PATH** **VDEV\_ENC\_SYSFS\_PATH** > Path to the disk's enclosure sysfs path, if available Note that some of these variables may have a blank value. **POOL\_NAME** is blank at pool creation time, for example. # ENVIRONMENT **zfs\_prepare\_disk** runs with a limited $PATH. # EXIT STATUS **zfs\_prepare\_disk** should return 0 on success, non-zero otherwise. If non-zero is returned, the disk will not be included in the pool. Debian - August 30, 2023 ZGENHOSTID(8) - System Manager's Manual # NAME **zgenhostid** - generate host ID into /etc/hostid # SYNOPSIS **zgenhostid** \[**-f**] \[**-o** *filename*] \[*hostid*] # DESCRIPTION Creates */etc/hostid* file and stores the host ID in it. If *hostid* was provided, validate and store that value. Otherwise, randomly generate an ID. # OPTIONS **-h** > Display a summary of the command-line options. **-f** > Allow output overwrite. **-o** *filename* > Write to > *filename* > instead of the default > */etc/hostid*. *hostid* > Specifies the value to be placed in > */etc/hostid*. > It should be a number with a value between 1 and 2^32-1. > If > **0**, > generate a random ID. > This value > *must* > be unique among your systems. > It > *must* > be an 8-digit-long hexadecimal number, optionally prefixed by > "0x". # FILES */etc/hostid* # EXAMPLES Generate a random hostid and store it > > \# **zgenhostid** Record the libc-generated hostid in */etc/hostid* > > \# **zgenhostid** "$(**hostid**)" Record a custom hostid (*0xdeadbeef*) in */etc/hostid* > > \# **zgenhostid** *deadbeef* Record a custom hostid (*0x01234567*) in */tmp/hostid* and overwrite the file if it exists > > \# **zgenhostid** **-f** **-o** */tmp/hostid 0x01234567* # SEE ALSO genhostid(1), hostid(1), sethostid(3), spl(4) # HISTORY **zgenhostid** emulates the genhostid(1) utility and is provided for use on systems which do not include the utility or do not provide the sethostid(3) function. Debian - July 11, 2022 ZINJECT(8) - System Manager's Manual # NAME **zinject** - ZFS Fault Injector # DESCRIPTION **zinject** creates artificial problems in a ZFS pool by simulating data corruption or device failures. This program is dangerous. # SYNOPSIS **zinject** > List injection records. **zinject** **-b** *objset*:*object*:*level*:*start*:*end* \[**-f** *frequency*] **-amu** \[pool] > Force an error into the pool at a bookmark. **zinject** **-c** *id*|**all** > Cancel injection records. **zinject** **-d** *vdev* **-A** **degrade**|**fault** *pool* > Force a vdev into the DEGRADED or FAULTED state. **zinject** **-d** *vdev* **-D** *latency*:*lanes* \[**-T** *read|write*] *pool* > Add an artificial delay to I/O requests on a particular > device, such that the requests take a minimum of > *latency* > milliseconds to complete. > Each delay has an associated number of > *lanes* > which defines the number of concurrent > I/O requests that can be processed. > For example, with a single lane delay of 10 ms > (**-D** *10*:*1*), > the device will only be able to service a single I/O request > at a time with each request taking 10 ms to complete. > So, if only a single request is submitted every 10 ms, the > average latency will be 10 ms; but if more than one request > is submitted every 10 ms, the average latency will be more > than 10 ms. > Similarly, if a delay of 10 ms is specified to have two > lanes > (**-D** *10*:*2*), > then the device will be able to service > two requests at a time, each with a minimum latency of 10 ms. > So, if two requests are submitted every 10 ms, then > the average latency will be 10 ms; but if more than two > requests are submitted every 10 ms, the average latency > will be more than 10 ms. > Also note, these delays are additive. > So two invocations of > **-D** *10*:*1* > are roughly equivalent to a single invocation of > **-D** *10*:*2*. > This also means, that one can specify multiple > lanes with differing target latencies. > For example, an invocation of > **-D** *10*:*1* > followed by > **-D** *25*:*2* > will create 3 lanes on the device: one lane with a latency > of 10 ms and two lanes with a 25 ms latency. **zinject** **-d** *vdev* \[**-e** *device\_error*] \[**-L** *label\_error*] \[**-T** *failure*] \[**-f** *frequency*] \[**-F**] *pool* > Force a vdev error. **zinject** **-i** *seconds* *pool* > Add an artificial delay during the future import of a pool. > This injector is automatically cleared after the import is finished. **zinject** **-E** *delay* \[**-a**] \[**-m**] \[**-f** *freq*] \[**-l** *level*] \[**-r** *range*] \[**-T** *iotype*] \[**-t** *type*|**-b** *bookmark*] > Inject pipeline ready stage delays for the given object or bookmark. > The delay is specified in milliseconds. **zinject** **-I** \[**-s** *seconds*|**-g** *txgs*] *pool* > Simulate a hardware failure that fails to honor a cache flush. **zinject** **-p** *function* *pool* > Panic inside the specified function. **zinject** **-t** **data** **-C** *dvas* \[**-e** *device\_error*] \[**-f** *frequency*] \[**-l** *level*] \[**-r** *range*] \[**-amq**] *path* > Force an error into the contents of a file. **zinject** **-t** **dnode** **-C** *dvas* \[**-e** *device\_error*] \[**-f** *frequency*] \[**-l** *level*] \[**-amq**] *path* > Force an error into the metadnode for a file or directory. **zinject** **-t** *mos\_type* **-C** *dvas* \[**-e** *device\_error*] \[**-f** *frequency*] \[**-l** *level*] \[**-r** *range*] \[**-amqu**] *pool* > Force an error into the MOS of a pool. # OPTIONS **-a** > Flush the ARC before injection. **-b** *objset*:*object*:*level*:*start*:*end* > Force an error into the pool at this bookmark tuple. > Each number is in hexadecimal, and only one block can be specified. **-C** *dvas* > Inject the given error only into specific DVAs. > The mask should be specified as a list of 0-indexed DVAs separated by commas > (e.g. *0,2*). > This option is not applicable to logical data errors such as > **decompress** > and > **decrypt**. **-d** *vdev* > A vdev specified by path or GUID. **-e** *device\_error* > Specify > **checksum** > > for an ECKSUM error, > **decompress** > > for a data decompression error, > **decrypt** > > for a data decryption error, > **corrupt** > > to flip a bit in the data after a read, > **dtl** > > for an ECHILD error, > **io** > > for an EIO error where reopening the device will succeed, > **nxio** > > for an ENXIO error where reopening the device will fail, or > **io-prefail** > > to drop the IO without executing it and return failure, or > **noop** > > to drop the IO without executing it and return success. > For EIO and ENXIO, the "failed" reads or writes still occur. > The probe simply sets the error value reported by the I/O pipeline > so it appears the read or write failed. > Decryption errors only currently work with file data. **-f** *frequency* > Only inject errors a fraction of the time. > Expressed as a real number percentage between > **0.0001** > and > **100**. **-F** > Fail faster. > Do fewer checks. **-f** *txgs* > Run for this many transaction groups before reporting failure. **-h** > Print the usage message. **-l** *level* > Inject an error at a particular block level. > The default is > **0**. **-L** *label\_error* > Set the label error region to one of > **nvlist**, > **pad1**, > **pad2**, > or > **uber**. **-m** > Automatically remount the underlying filesystem. **-q** > Quiet mode. > Only print the handler number added. **-r** *range* > Inject an error over a particular logical range of an object, specified as > *start*\[,*end*]. > Numeric suffixes > (K, M, G, T, P, E) > are accepted. > The range will be translated to the appropriate blkid range according to the > object's properties. **-s** *seconds* > Run for this many seconds before reporting failure. **-T** *type* > Inject the error into I/O of this type. > **read**, **write**, **flush**, **claim**, **free** > > Fundamental I/O types > **all** > > All fundamental I/O types > **probe** > > Device probe I/O **-t** *mos\_type* > Set this to > **mos** > > for any data in the MOS, > **mosdir** > > for an object directory, > **config** > > for the pool configuration, > **bpobj** > > for the block pointer list, > **spacemap** > > for the space map, > **metaslab** > > for the metaslab, or > **errlog** > > for the persistent error log. **-u** > Unload the pool after injection. # ENVIRONMENT VARIABLES `ZFS_HOSTID` > Run > **zinject** > in debug mode. # SEE ALSO zfs(8), zpool(8) Debian - April 1, 2026 ZPOOL-ADD(8) - System Manager's Manual # NAME **zpool-add** - add vdevs to ZFS storage pool # SYNOPSIS **zpool** **add** \[**-fgLnP**] \[**--allow-in-use** **--allow-replication-mismatch** **--allow-ashift-mismatch**] \[**-o** *property*=*value*] *pool vdev*… # DESCRIPTION Adds the specified virtual devices to the given pool. The *vdev* specification is described in the *Virtual Devices* section of zpoolconcepts(7). The behavior of the **-f** option, and the device checks performed are described in the **zpool** **create** subcommand. **-f** > Forces use of > *vdev*s, > even if they appear in use, have conflicting ashift values, or specify > a conflicting replication level. > Not all devices can be overridden in this manner. **-g** > Display > *vdev*, > GUIDs instead of the normal device names. > These GUIDs can be used in place of > device names for the zpool detach/offline/remove/replace commands. **-L** > Display real paths for > *vdev*s > resolving all symbolic links. > This can be used to look up the current block > device name regardless of the > */dev/disk* > path used to open it. **-n** > Displays the configuration that would be used without actually adding the > *vdev*s. > The actual pool creation can still fail due to insufficient privileges or > device sharing. **-P** > Display real paths for > *vdev*s > instead of only the last component of the path. > This can be used in conjunction with the > **-L** > flag. **-o** *property*=*value* > Sets the given pool properties. > See the > zpoolprops(7) > manual page for a list of valid properties that can be set. > The only property supported at the moment is > **ashift**. **--allow-ashift-mismatch** > Disable the ashift validation which allows mismatched ashift values in the > pool. > Adding top-level > *vdev*s > with different sector sizes will prohibit future device removal operations, see > zpool-remove(8). **--allow-in-use** > Allow vdevs to be added even if they might be in use in another pool. **--allow-replication-mismatch** > Allow vdevs with conflicting replication levels to be added to the pool. # EXAMPLES ## Example 1: Adding a Mirror to a ZFS Storage Pool The following command adds two mirrored disks to the pool *tank*, assuming the pool is already made up of two-way mirrors. The additional space is immediately available to any datasets within the pool. # **zpool** **add** *tank* **mirror** *sda sdb* ## Example 2: Adding Cache Devices to a ZFS Pool The following command adds two disks for use as cache devices to a ZFS storage pool: # **zpool** **add** *pool* **cache** *sdc sdd* Once added, the cache devices gradually fill with content from main memory. Depending on the size of your cache devices, it could take over an hour for them to fill. Capacity and reads can be monitored using the **iostat** subcommand as follows: # **zpool** **iostat** **-v** *pool 5* # SEE ALSO zpool-attach(8), zpool-import(8), zpool-initialize(8), zpool-online(8), zpool-remove(8) Debian - March 8, 2024 ZPOOL-ATTACH(8) - System Manager's Manual # NAME **zpool-attach** - attach new device to existing ZFS vdev # SYNOPSIS **zpool** **attach** \[**-fsw**] \[**-o** *property*=*value*] *pool vdev new\_device* # DESCRIPTION Attaches *new\_device* to the existing *vdev*. The behavior differs depending on if the existing *vdev* is a RAID-Z device, or a mirror/plain device. If the existing vdev is a mirror or plain device (e.g. specified as "`sda`" or "`mirror-7`"), the new device will be mirrored with the existing vdev, a resilver will be initiated, and the new device will contribute to additional redundancy once the resilver completes. If *vdev* is not currently part of a mirrored configuration, *device* automatically transforms into a two-way mirror of *device* and *new\_device*. If *vdev* is part of a two-way mirror, attaching *new\_device* creates a three-way mirror, and so on. In either case, *new\_device* begins to resilver immediately and any running scrub is canceled. If the existing vdev is a RAID-Z device (e.g. specified as "*raidz2-0*"), the new device will become part of that RAID-Z group. A "raidz expansion" will be initiated, and once the expansion completes, the new device will contribute additional space to the RAID-Z group. The expansion entails reading all allocated space from existing disks in the RAID-Z group, and rewriting it to the new disks in the RAID-Z group (including the newly added *device*). Its progress can be monitored with **zpool** **status**. Data redundancy is maintained during and after the expansion. If a disk fails while the expansion is in progress, the expansion pauses until the health of the RAID-Z vdev is restored (e.g. by replacing the failed disk and waiting for reconstruction to complete). Expansion does not change the number of failures that can be tolerated without data loss (e.g. a RAID-Z2 is still a RAID-Z2 even after expansion). A RAID-Z vdev can be expanded multiple times. After the expansion completes, old blocks retain their old data-to-parity ratio (e.g. 5-wide RAID-Z2 has 3 data and 2 parity) but distributed among the larger set of disks. New blocks will be written with the new data-to-parity ratio (e.g. a 5-wide RAID-Z2 which has been expanded once to 6-wide, has 4 data and 2 parity). However, the vdev's assumed parity ratio does not change, so slightly less space than is expected may be reported for newly-written blocks, according to **zfs** **list**, **df**, **ls** **-s**, and similar tools. A pool-wide scrub is initiated at the end of the expansion in order to verify the checksums of all blocks which have been copied during the expansion. **-f** > Forces use of > *new\_device*, > even if it appears to be in use. > Not all vdevs can be overridden in this manner. **-o** *property*=*value* > Sets the given pool properties. > See the > zpoolprops(7) > manual page for a list of valid properties that can be set. > The only property supported at the moment is > **ashift**. **-s** > When attaching to a mirror or plain vdev, the > *new\_device* > is reconstructed sequentially to restore redundancy as quickly as possible. > Checksums are not verified during sequential reconstruction so a scrub is > started when the resilver completes. **-w** > Waits until > *new\_device* > has finished resilvering or expanding before returning. # EXAMPLES ## Example 1: Making a non-mirrored ZFS Storage Pool mirrored The following command converts an existing single device *sda* into a mirror by attaching a second device to it, *sdb*. # **zpool** **attach** *tank* *sda sdb* ## Example 2: Expanding a RAID-Z vdev with an additional disk The following command adds *sdg* to the existing *raidz2-0* vdev in *tank*, turning a 6-wide RAID-Z2 into a 7-wide RAID-Z2: # **zpool** **attach** *tank raidz2-0* *sdg* Progress is reported by **zpool** **status**. The operation requires the **raidz\_expansion** pool feature, and *sdg* must be at least as large as the smallest existing disk in the vdev. Old blocks keep their original data-to-parity ratio; only blocks written after the expansion use the new ratio. # SEE ALSO zpool-add(8), zpool-detach(8), zpool-import(8), zpool-initialize(8), zpool-online(8), zpool-replace(8), zpool-resilver(8) Debian - May 9, 2026 ZPOOL-CHECKPOINT(8) - System Manager's Manual # NAME **zpool-checkpoint** - check-point current ZFS storage pool state # SYNOPSIS **zpool** **checkpoint** \[**-d** \[**-w**]] *pool* # DESCRIPTION Checkpoints the current state of *pool* , which can be later restored by **zpool** **import --rewind-to-checkpoint**. The existence of a checkpoint in a pool prohibits the following **zpool** subcommands: **remove**, **attach**, **detach**, **split**, and **reguid**. In addition, it may break reservation boundaries if the pool lacks free space. The **zpool** **status** command indicates the existence of a checkpoint or the progress of discarding a checkpoint from a pool. **zpool** **list** can be used to check how much space the checkpoint takes from the pool. # OPTIONS **-d**, **--discard** > Discards an existing checkpoint from > *pool*. **-w**, **--wait** > Waits until the checkpoint has finished being discarded before returning. # SEE ALSO zfs-snapshot(8), zpool-import(8), zpool-status(8) Debian - July 11, 2022 ZPOOL-CLEAR(8) - System Manager's Manual # NAME **zpool-clear** - clear device errors in ZFS storage pool # SYNOPSIS **zpool** **clear** \[**--power**] *pool* \[*device*]… # DESCRIPTION Clears device errors in a pool. If no arguments are specified, all device errors within the pool are cleared. If one or more devices is specified, only those errors associated with the specified device or devices are cleared. If the pool was suspended it will be brought back online provided the devices can be accessed. Pools with **multihost** enabled which have been suspended cannot be resumed when there is evidence that the pool was imported by another host. The same checks performed during an import will be applied before the clear proceeds. **--power** > Power on the devices's slot in the storage enclosure and wait for the device > to show up before attempting to clear errors. > This is done on all the devices specified. > Alternatively, you can set the > **ZPOOL\_AUTO\_POWER\_ON\_SLOT** > environment variable to always enable this behavior. > Note: This flag currently works on Linux only. # SEE ALSO zdb(8), zpool-reopen(8), zpool-status(8) Debian - April 29, 2024 ZPOOL-CONDENSE(8) - System Manager's Manual # NAME **zpool-condense** - Condense, flush, garbage collect or otherwise clean up pool metadata # SYNOPSIS **zpool** **condense** \[**-t** *type*] \[**-c** | **-w**] **-a**|*pool*… **zpool** **condense** **--types** # DESCRIPTION Many internal pool metadata updates are performed in the background at a rate chosen to limit the performance impact to normal use of the pool. Sometimes it is desirable to accelerate these operations, even if it affects overall performance. **condense** allows an operator to request that a specific background operation be prioritised to complete as soon as possible. **-t**, **--type** > What to condense. > If not specified, all types will be condensed. > These are the possible values for > *type*: > **log\_spacemap** > > flushing log spacemap entries to their underlying metaslabs. **-a**, **--all** > Begin, cancel or wait on condense for all pools. **-c**, **--cancel** > Cancel a previous > **condense**. > This will return background updates to their normal rate. **-w**, **--wait** > Wait until the condense has completed before returning. **--types** > List condense types supported by this build of > **condense**. # EXAMPLES ## Example 1 Status of pool with **log\_spacemap** condense in progress: # **zpool** **condense -t log_spacemap** # **zpool** **status** pool: tank state: ONLINE condense: log_spacemap: condensing, 104/321 metaslabs done in 00:00:15 ... # SEE ALSO zpool-status(8), zpool-wait(8) Debian - May 26, 2026 ZPOOL-CREATE(8) - System Manager's Manual # NAME **zpool-create** - create ZFS storage pool # SYNOPSIS **zpool** **create** \[**-dfn**] \[**-m** *mountpoint*] \[**-o** *property*=*value*]… \[**-o** **feature@**‌*feature*=*value*] \[**-o** *compatibility*=**off**|**legacy**|*file*\[,*file*]…] \[**-O** *file-system-property*=*value*]… \[**-R** *root*] \[**-t** *tname*] *pool* *vdev*… # DESCRIPTION Creates a new storage pool containing the virtual devices specified on the command line. The pool name must begin with a letter, and can only contain alphanumeric characters as well as the underscore ("**\_**"), dash ("**-**"), colon ("**:**"), space ("** **"), and period ("**.**"). The pool names **mirror**, **raidz**, **draid**, **spare** and **log** are reserved, as are names beginning with **mirror**, **raidz**, **draid**, and **spare**. The *vdev* specification is described in the *Virtual Devices* section of zpoolconcepts(7). The command attempts to verify that each device specified is accessible and not currently in use by another subsystem. However this check is not robust enough to detect simultaneous attempts to use a new device in different pools, even if **multihost**= **enabled**. The administrator must ensure that simultaneous invocations of any combination of **zpool** **replace**, **zpool** **create**, **zpool** **add**, or **zpool** **labelclear** do not refer to the same device. Using the same device in two pools will result in pool corruption. There are some uses, such as being currently mounted, or specified as the dedicated dump device, that prevents a device from ever being used by ZFS. Other uses, such as having a preexisting UFS file system, can be overridden with **-f**. The command also checks that the replication strategy for the pool is consistent. An attempt to combine redundant and non-redundant storage in a single pool, or to mix disks and files, results in an error unless **-f** is specified. The use of differently-sized devices within a single raidz or mirror group is also flagged as an error unless **-f** is specified. Unless the **-R** option is specified, the default mount point is */*‌*pool*. The mount point must not exist or must be empty, or else the root dataset will not be able to be be mounted. This can be overridden with the **-m** option. By default all supported features are enabled on the new pool. The **-d** option and the **-o** *compatibility* property (e.g **-o** **compatibility**=*2020*) can be used to restrict the features that are enabled, so that the pool can be imported on other releases of ZFS. **-d** > Do not enable any features on the new pool. > Individual features can be enabled by setting their corresponding properties to > **enabled** > with > **-o**. > See > zpool-features(7) > for details about feature properties. **-f** > Forces use of > *vdev*s, > even if they appear in use or specify a conflicting replication level. > Not all devices can be overridden in this manner. **-m** *mountpoint* > Sets the mount point for the root dataset. > The default mount point is > */pool* > or > *altroot/pool* > if > **altroot** > is specified. > The mount point must be an absolute path, > **legacy**, > or > **none**. > For more information on dataset mount points, see > zfsprops(7). **-n** > Displays the configuration that would be used without actually creating the > pool. > The actual pool creation can still fail due to insufficient privileges or > device sharing. **-o** *property*=*value* > Sets the given pool properties. > See > zpoolprops(7) > for a list of valid properties that can be set. **-o** *compatibility*=**off**|**legacy**|*file*\[,*file*]… > Specifies compatibility feature sets. > See > zpool-features(7) > for more information about compatibility feature sets. **-o** **feature@**‌*feature*=*value* > Sets the given pool feature. > See the > zpool-features(7) > section for a list of valid features that can be set. > Value can be either disabled or enabled. **-O** *file-system-property*=*value* > Sets the given file system properties in the root file system of the pool. > See > zfsprops(7) > for a list of valid properties that can be set. **-R** *root* > Equivalent to > **-o** **cachefile**=**none** **-o** **altroot**=*root* **-t** *tname* > Sets the in-core pool name to > *tname* > while the on-disk name will be the name specified as > *pool*. > This will set the default of the > **cachefile** > property to > **none**. > This is intended > to handle name space collisions when creating pools for other systems, > such as virtual machines or physical machines whose pools live on network > block devices. # EXAMPLES ## Example 1: Creating a RAID-Z Storage Pool The following command creates a pool with a single raidz root vdev that consists of six disks: # **zpool** **create** *tank* **raidz** *sda sdb sdc sdd sde sdf* ## Example 2: Creating a Mirrored Storage Pool The following command creates a pool with two mirrors, where each mirror contains two disks: # **zpool** **create** *tank* **mirror** *sda sdb* **mirror** *sdc sdd* ## Example 3: Creating a ZFS Storage Pool by Using Partitions The following command creates a non-redundant pool using two disk partitions: # **zpool** **create** *tank* *sda1 sdb2* ## Example 4: Creating a ZFS Storage Pool by Using Files The following command creates a non-redundant pool using files. While not recommended, a pool based on files can be useful for experimental purposes. # **zpool** **create** *tank* */path/to/file/a /path/to/file/b* ## Example 5: Managing Hot Spares The following command creates a new pool with an available hot spare: # **zpool** **create** *tank* **mirror** *sda sdb* **spare** *sdc* ## Example 6: Creating a ZFS Pool with Mirrored Separate Intent Logs The following command creates a ZFS storage pool consisting of two, two-way mirrors and mirrored log devices: # **zpool** **create** *pool* **mirror** *sda sdb* **mirror** *sdc sdd* **log mirror** *sde sdf* ## Example 7: Creating a ZFS Pool with dRAID vdev The following command creates a ZFS storage pool with dRAID vdev with one parity, four data and one spare devices, 6 devices in total: # **zpool** **create** *pool* **draid1:4d:6c:1s** *sda sdb sdc sdd sde sdf* ## Example 8: Creating a ZFS Pool with dRAID vdev with failure domains The following commands create a ZFS storage pool with dRAID vdev with five failure groups and six failure domains (for example, enclosures). The commands are equivalent: # **zpool** **create** *pool* **draid1:4d:6c:30w:5s** \ enc0d0 enc1d0 enc2d0 enc3d0 enc4d0 enc5d0 \ enc0d1 enc1d1 enc2d1 enc3d1 enc4d1 enc5d1 \ enc0d2 enc1d2 enc2d2 enc3d2 enc4d2 enc5d2 \ enc0d3 enc1d3 enc2d3 enc3d3 enc4d3 enc5d3 \ enc0d4 enc1d4 enc2d4 enc3d4 enc4d4 enc5d4 # **zpool** **create** *pool* **draid1:5s** \ fgroup enc0d0 enc1d0 enc2d0 enc3d0 enc4d0 enc5d0 \ fgroup enc0d1 enc1d1 enc2d1 enc3d1 enc4d1 enc5d1 \ fgroup enc0d2 enc1d2 enc2d2 enc3d2 enc4d2 enc5d2 \ fgroup enc0d3 enc1d3 enc2d3 enc3d3 enc4d3 enc5d3 \ fgroup enc0d4 enc1d4 enc2d4 enc3d4 enc4d4 enc5d4 # **zpool** **create** *pool* **draid1:5s** \ fdomain enc0d0 enc0d1 enc0d2 enc0d3 enc0d4 \ fdomain enc1d0 enc1d1 enc1d2 enc1d3 enc1d4 \ fdomain enc2d0 enc2d1 enc2d2 enc2d3 enc2d4 \ fdomain enc3d0 enc3d1 enc3d2 enc3d3 enc3d4 \ fdomain enc4d0 enc4d1 enc4d2 enc4d3 enc4d4 \ fdomain enc5d0 enc5d1 enc5d2 enc5d3 enc5d4 # SEE ALSO zpool-destroy(8), zpool-export(8), zpool-import(8) Debian - July 11, 2022 ZPOOL-DDTPRUNE(8) - System Manager's Manual # NAME **zpool-ddtprune** - Prunes the oldest entries from the single reference dedup table(s) # SYNOPSIS **zpool** **ddtprune** **-d** *days* | **-p** *percentage* *pool* # DESCRIPTION This command prunes older unique entries from the dedup table. As a complement to the dedup quota feature, **ddtprune** allows removal of older non-duplicate entries to make room for newer duplicate entries. The amount to prune can be based on a target percentage of the unique entries or based on the age (i.e., every unique entry older than N days). # SEE ALSO zdb(8), zpool-status(8) Debian - June 17, 2024 ZPOOL-DESTROY(8) - System Manager's Manual # NAME **zpool-destroy** - destroy ZFS storage pool # SYNOPSIS **zpool** **destroy** \[**-f**] *pool* # DESCRIPTION Destroys the given pool, freeing up any devices for other use. This command tries to unmount any active datasets before destroying the pool. **-f** > Forcefully unmount all active datasets. # EXAMPLES ## Example 1: Destroying a ZFS Storage Pool The following command destroys the pool *tank* and any datasets contained within: # **zpool** **destroy** **-f** *tank* Debian - July 11, 2022 ZPOOL-DETACH(8) - System Manager's Manual # NAME **zpool-detach** - detach device from ZFS mirror # SYNOPSIS **zpool** **detach** *pool device* # DESCRIPTION Detaches *device* from a mirror. The operation is refused if there are no other valid replicas of the data. If *device* may be re-added to the pool later on then consider the **zpool** **offline** command instead. # SEE ALSO zpool-attach(8), zpool-labelclear(8), zpool-offline(8), zpool-remove(8), zpool-replace(8), zpool-split(8) Debian - July 11, 2022 .\" SPDX-License-Identifier: CDDL-1.0 .\" .\" This file and its contents are supplied under the terms of the .\" Common Development and Distribution License ("CDDL"), version 1.0. .\" You may only use this file in accordance with the terms of version .\" 1.0 of the CDDL. .\" .\" A full copy of the text of the CDDL should have accompanied this .\" source. A copy of the CDDL is also available via the Internet at .\" https://opensource.org/license/CDDL-1.0. .\" .\" Copyright (c) 2007, Sun Microsystems, Inc. All Rights Reserved. .\" Copyright (c) 2012, 2018 by Delphix. All rights reserved. .\" Copyright (c) 2012 Cyril Plisko. All Rights Reserved. .\" Copyright (c) 2017 Datto Inc. .\" Copyright (c) 2018 George Melikov. All Rights Reserved. .\" Copyright 2017 Nexenta Systems, Inc. .\" Copyright (c) 2017 Open-E, Inc. All Rights Reserved. .\" Copyright (c) 2024, 2025, Klara, Inc. .\" .Dd July 3, 2025 .Dt ZPOOL-EVENTS 8 .Os . .Sh NAME .Nm zpool-events .Nd list recent events generated by kernel .Sh SYNOPSIS .Nm zpool .Cm events .Op Fl vHf .Op Ar pool .Nm zpool .Cm events .Fl c . .Sh DESCRIPTION Lists all recent events generated by the ZFS kernel modules. These events are consumed by the .Xr zed 8 and used to automate administrative tasks such as replacing a failed device with a hot spare. For more information about the subclasses and event payloads that can be generated see .Sx EVENTS and the following sections. . .Sh OPTIONS .Bl -tag -compact -width Ds .It Fl c Clear all previous events. .It Fl f Follow mode. .It Fl H Scripted mode. Do not display headers, and separate fields by a single tab instead of arbitrary space. .It Fl v Print the entire payload for each event. .El . .Sh EVENTS These are the different event subclasses. The full event name would be .Sy ereport.fs.zfs.\& Ns Em SUBCLASS , but only the last part is listed here. .Pp .Bl -tag -compact -width "vdev.bad_guid_sum" .It Sy checksum Issued when a checksum error has been detected. .It Sy io Issued when there is an I/O error in a vdev in the pool. .It Sy data Issued when there have been data errors in the pool. .It Sy deadman Issued when an I/O request is determined to be "hung", this can be caused by lost completion events due to flaky hardware or drivers. See .Sy zfs_deadman_failmode in .Xr zfs 4 for additional information regarding "hung" I/O detection and configuration. .It Sy delay Issued when a completed I/O request exceeds the maximum allowed time specified by the .Sy zio_slow_io_ms module parameter. This can be an indicator of problems with the underlying storage device. The number of delay events is ratelimited by the .Sy zfs_slow_io_events_per_second module parameter. .It Sy dio_verify_rd Issued when there was a checksum verify error after a Direct I/O read has been issued. .It Sy dio_verify_wr Issued when there was a checksum verify error after a Direct I/O write has been issued. This event can only take place if the module parameter .Sy zfs_vdev_direct_write_verify is not set to zero. See .Xr zfs 4 for more details on the .Sy zfs_vdev_direct_write_verify module parameter. .It Sy config_sync Issued every time a vdev change have been done to the pool. .It Sy zpool Issued when a pool cannot be imported. .It Sy pool_create Issued when a pool is created. .It Sy pool_destroy Issued when a pool is destroyed. .It Sy pool_export Issued when a pool is exported. .It Sy pool_import Issued when a pool is imported. .It Sy pool_reguid Issued when a REGUID (new unique identifier for the pool have been regenerated) have been detected. .It Sy vdev.unknown Issued when the vdev is unknown. Such as trying to clear device errors on a vdev that have failed/been kicked from the system/pool and is no longer available. .It Sy vdev.open_failed Issued when a vdev could not be opened (because it didn't exist for example). .It Sy vdev.corrupt_data Issued when corrupt data have been detected on a vdev. .It Sy vdev.no_replicas Issued when there are no more replicas to sustain the pool. This would lead to the pool being .Em DEGRADED . .It Sy vdev.bad_guid_sum Issued when a missing device in the pool have been detected. .It Sy vdev.too_small Issued when the system (kernel) have removed a device, and ZFS notices that the device isn't there any more. This is usually followed by a .Sy probe_failure event. .It Sy vdev.bad_label Issued when the label is OK but invalid. .It Sy vdev.bad_ashift Issued when the ashift alignment requirement has increased. .It Sy vdev_remove Issued when a vdev is detached from a mirror (or a spare detached from a vdev where it have been used to replace a failed drive - only works if the original drive have been re-added). .It Sy vdev_remove_aux Issued when an auxiliary vdev is removed. .It Sy vdev_remove_dev Issued when a specific device is removed from a vdev. .It Sy vdev_clear Issued when clearing device errors in a pool. Such as running .Nm zpool Cm clear on a device in the pool. .It Sy vdev_check Issued when a check to see if a given vdev could be opened is started. .It Sy vdev_spare Issued when a spare have kicked in to replace a failed device. .It Sy vdev_autoexpand Issued when a vdev can be automatically expanded. .It Sy vdev_add Issued when a vdev is added to a pool. .It Sy vdev_attach Issued when a vdev is attached to a mirror or raidz vdev type. .It Sy vdev_online Issued when an offline vdev is brought online .It Sy io_failure Issued when there is an I/O failure in a vdev in the pool. .It Sy probe_failure Issued when a probe fails on a vdev. This would occur if a vdev have been kicked from the system outside of ZFS (such as the kernel have removed the device). .It Sy log_replay Issued when the intent log cannot be replayed. The can occur in the case of a missing or damaged log device. .It Sy resilver_start Issued when a resilver is started. .It Sy resilver_finish Issued when the running resilver have finished. .It Sy scrub_start Issued when a scrub is started on a pool. .It Sy scrub_finish Issued when a pool has finished scrubbing. .It Sy scrub_abort Issued when a scrub is aborted on a pool. .It Sy scrub_resume Issued when a scrub is resumed on a pool. .It Sy scrub_paused Issued when a scrub is paused on a pool. .It Sy errorscrub_start Issued when a errorscrub is started on a pool. .It Sy errorscrub_finish Issued when a pool has finished errorscrubbing. .It Sy errorscrub_abort Issued when a errorscrub is aborted on a pool. .It Sy errorscrub_resume Issued when a errorscrub is resumed on a pool. .It Sy errorscrub_paused Issued when a errorscrub is paused on a pool. .It Sy trim_start Issued when a trim is started on a pool. .It Sy trim_finish Issued when a pool has finished trimbing. .It Sy trim_cancel Issued when a trim is canceled on a pool. .It Sy trim_resume Issued when a trim is resumed on a pool. .It Sy trim_suspend Issued when a trim is suspend on a pool. .It Sy authentication Issued when there is a decryption / authentication error. .It Sy config_cache_write Issued when the config cache file cannot be written. .It Sy bootfs_vdev_attach Issued when a vdev is attached to a root pool with the bootfs property set. .It Sy sitout Issued when a .Sy RAIDZ or .Sy DRAID vdev triggers the .Sy autosit logic. This logic detects when a disk in such a vdev is significantly slower than its peers, and sits them out temporarily to preserve the performance of the pool. .El . .Sh PAYLOADS This is the payload (data, information) that accompanies an event. .Pp For .Xr zed 8 , these are set to uppercase and prefixed with .Sy ZEVENT_ . .Pp .Bl -tag -compact -width "vdev_cksum_errors" .It Sy pool Pool name. .It Sy pool_failmode Failmode - .Sy wait , .Sy continue , or .Sy panic . See the .Sy failmode property in .Xr zpoolprops 7 for more information. .It Sy pool_guid The GUID of the pool. .It Sy pool_context The load state for the pool (0=none, 1=open, 2=import, 3=tryimport, 4=recover 5=error). .It Sy vdev_guid The GUID of the vdev in question (the vdev failing or operated upon with .Nm zpool Cm clear , etc.). .It Sy vdev_type Type of vdev - .Sy disk , .Sy file , .Sy mirror , etc. See the .Sy Virtual Devices section of .Xr zpoolconcepts 7 for more information on possible values. .It Sy vdev_path Full path of the vdev, including any .Em -partX . .It Sy vdev_devid ID of vdev (if any). .It Sy vdev_fru Physical FRU location. .It Sy vdev_state State of vdev (0=uninitialized, 1=closed, 2=offline, 3=removed, 4=failed to open, 5=faulted, 6=degraded, 7=healthy). .It Sy vdev_ashift The ashift value of the vdev. .It Sy vdev_complete_ts The time the last I/O request completed for the specified vdev. .It Sy vdev_delta_ts The time since the last I/O request completed for the specified vdev. .It Sy vdev_spare_paths List of spares, including full path and any .Em -partX . .It Sy vdev_spare_guids GUID(s) of spares. .It Sy vdev_read_errors How many read errors that have been detected on the vdev. .It Sy vdev_write_errors How many write errors that have been detected on the vdev. .It Sy vdev_cksum_errors How many checksum errors that have been detected on the vdev. .It Sy parent_guid GUID of the vdev parent. .It Sy parent_type Type of parent. See .Sy vdev_type . .It Sy parent_path Path of the vdev parent (if any). .It Sy parent_devid ID of the vdev parent (if any). .It Sy zio_objset The object set number for a given I/O request. .It Sy zio_object The object number for a given I/O request. .It Sy zio_level The indirect level for the block. Level 0 is the lowest level and includes data blocks. Values > 0 indicate metadata blocks at the appropriate level. .It Sy zio_blkid The block ID for a given I/O request. .It Sy zio_err The error number for a failure when handling a given I/O request, compatible with .Xr errno 3 with the value of .Sy EBADE used to indicate a ZFS checksum error. .It Sy zio_offset The offset in bytes of where to write the I/O request for the specified vdev. .It Sy zio_size The size in bytes of the I/O request. .It Sy zio_flags The current flags describing how the I/O request should be handled. See the .Sy I/O FLAGS section for the full list of I/O flags. .It Sy zio_stage The current stage of the I/O in the pipeline. See the .Sy I/O STAGES section for a full list of all the I/O stages. .It Sy zio_pipeline The valid pipeline stages for the I/O. See the .Sy I/O STAGES section for a full list of all the I/O stages. .It Sy zio_priority The queue priority of the I/O request. See the .Sy I/O PRIORITIES section for a full list of all the I/O priorities. .It Sy zio_tyoe The type of the I/O request. See the .Sy I/O TYPES section for a full list of all the I/O types. .It Sy zio_delay The time elapsed (in nanoseconds) waiting for the block layer to complete the I/O request. Unlike .Sy zio_delta , this does not include any vdev queuing time and is therefore solely a measure of the block layer performance. .It Sy zio_timestamp The time when a given I/O request was submitted. .It Sy zio_delta The time required to service a given I/O request. .It Sy prev_state The previous state of the vdev. .It Sy cksum_algorithm Checksum algorithm used. See .Xr zfsprops 7 for more information on the available checksum algorithms. .It Sy cksum_byteswap Whether or not the data is byteswapped. .It Sy bad_ranges .No [\& Ns Ar start , end ) pairs of corruption offsets. Offsets are always aligned on a 64-bit boundary, and can include some gaps of non-corruption. (See .Sy bad_ranges_min_gap ) .It Sy bad_ranges_min_gap In order to bound the size of the .Sy bad_ranges array, gaps of non-corruption less than or equal to .Sy bad_ranges_min_gap bytes have been merged with adjacent corruption. Always at least 8 bytes, since corruption is detected on a 64-bit word basis. .It Sy bad_range_sets This array has one element per range in .Sy bad_ranges . Each element contains the count of bits in that range which were clear in the good data and set in the bad data. .It Sy bad_range_clears This array has one element per range in .Sy bad_ranges . Each element contains the count of bits for that range which were set in the good data and clear in the bad data. .It Sy bad_set_bits If this field exists, it is an array of .Pq Ar bad data No & ~( Ns Ar good data ) ; that is, the bits set in the bad data which are cleared in the good data. Each element corresponds a byte whose offset is in a range in .Sy bad_ranges , and the array is ordered by offset. Thus, the first element is the first byte in the first .Sy bad_ranges range, and the last element is the last byte in the last .Sy bad_ranges range. .It Sy bad_cleared_bits Like .Sy bad_set_bits , but contains .Pq Ar good data No & ~( Ns Ar bad data ) ; that is, the bits set in the good data which are cleared in the bad data. .El . .Sh I/O STAGES The ZFS I/O pipeline is comprised of various stages which are defined below. The individual stages are used to construct these basic I/O operations: Read, Write, Free, Claim, Flush and Trim. These stages may be set on an event to describe the life cycle of a given I/O request. .Pp .TS tab(:); l l l . Stage:Bit Mask:Operations _:_:_ ZIO_STAGE_OPEN:0x00000001:RWFCXT ZIO_STAGE_READ_BP_INIT:0x00000002:R----- ZIO_STAGE_WRITE_BP_INIT:0x00000004:-W---- ZIO_STAGE_FREE_BP_INIT:0x00000008:--F--- ZIO_STAGE_ISSUE_ASYNC:0x00000010:-WF--T ZIO_STAGE_WRITE_COMPRESS:0x00000020:-W---- ZIO_STAGE_ENCRYPT:0x00000040:-W---- ZIO_STAGE_CHECKSUM_GENERATE:0x00000080:-W---- ZIO_STAGE_NOP_WRITE:0x00000100:-W---- ZIO_STAGE_DDT_READ_START:0x00000200:R----- ZIO_STAGE_DDT_READ_DONE:0x00000400:R----- ZIO_STAGE_DDT_WRITE:0x00000800:-W---- ZIO_STAGE_DDT_FREE:0x00001000:--F--- ZIO_STAGE_BRT_FREE:0x00002000:--F--- ZIO_STAGE_GANG_ASSEMBLE:0x00004000:RWFC-- ZIO_STAGE_GANG_ISSUE:0x00008000:RWFC-- ZIO_STAGE_DVA_THROTTLE:0x00010000:-W---- ZIO_STAGE_DVA_ALLOCATE:0x00020000:-W---- ZIO_STAGE_DVA_FREE:0x00040000:--F--- ZIO_STAGE_DVA_CLAIM:0x00080000:---C-- ZIO_STAGE_READY:0x00100000:RWFCIT ZIO_STAGE_VDEV_IO_START:0x00200000:RW--XT ZIO_STAGE_VDEV_IO_DONE:0x00400000:RW--XT ZIO_STAGE_VDEV_IO_ASSESS:0x00800000:RW--XT ZIO_STAGE_CHECKSUM_VERIFY:0x01000000:R----- ZIO_STAGE_DIO_CHECKSUM_VERIFY:0x02000000:-W---- ZIO_STAGE_DONE:0x04000000:RWFCXT .TE . .Sh I/O FLAGS Every I/O request in the pipeline contains a set of flags which describe its function and are used to govern its behavior. These flags will be set in an event as a .Sy zio_flags payload entry. .Pp .TS tab(:); l l . Flag:Bit Mask _:_ ZIO_FLAG_DONT_AGGREGATE:0x00000001 ZIO_FLAG_IO_REPAIR:0x00000002 ZIO_FLAG_SELF_HEAL:0x00000004 ZIO_FLAG_RESILVER:0x00000008 ZIO_FLAG_SCRUB:0x00000010 ZIO_FLAG_SCAN_THREAD:0x00000020 ZIO_FLAG_PHYSICAL:0x00000040 ZIO_FLAG_CANFAIL:0x00000080 ZIO_FLAG_SPECULATIVE:0x00000100 ZIO_FLAG_CONFIG_WRITER:0x00000200 ZIO_FLAG_DONT_RETRY:0x00000400 ZIO_FLAG_NODATA:0x00001000 ZIO_FLAG_INDUCE_DAMAGE:0x00002000 ZIO_FLAG_ALLOC_THROTTLED:0x00004000 ZIO_FLAG_IO_RETRY:0x00008000 ZIO_FLAG_PROBE:0x00010000 ZIO_FLAG_TRYHARD:0x00020000 ZIO_FLAG_OPTIONAL:0x00040000 ZIO_FLAG_DIO_READ:0x00080000 ZIO_FLAG_BYPASSED_QUEUE:0x00100000 ZIO_FLAG_DONT_QUEUE:0x00200000 ZIO_FLAG_DONT_PROPAGATE:0x00400000 ZIO_FLAG_IO_BYPASS:0x00800000 ZIO_FLAG_IO_REWRITE:0x01000000 ZIO_FLAG_RAW_COMPRESS:0x02000000 ZIO_FLAG_RAW_ENCRYPT:0x04000000 ZIO_FLAG_GANG_CHILD:0x08000000 ZIO_FLAG_DDT_CHILD:0x10000000 ZIO_FLAG_GODFATHER:0x20000000 ZIO_FLAG_NOPWRITE:0x40000000 ZIO_FLAG_REEXECUTED:0x80000000 ZIO_FLAG_DELEGATED:0x100000000 ZIO_FLAG_PREALLOCATED:0x200000000 ZIO_FLAG_POSTREAD:0x400000000 .TE . .Sh I/O TYPES Every I/O request in the pipeline has a single type value. This value describes the kind of low-level work the I/O represents. This value will be set in an event as a .Sy zio_type payload entry. .Pp .TS tab(:); l l l . Type:Value:Description _:_:_ ZIO_TYPE_NULL:0x0:internal I/O sync point ZIO_TYPE_READ:0x1:data read ZIO_TYPE_WRITE:0x2:data write ZIO_TYPE_FREE:0x3:block free ZIO_TYPE_CLAIM:0x4:block claim (ZIL replay) ZIO_TYPE_FLUSH:0x5:disk cache flush request ZIO_TYPE_TRIM:0x6:trim (discard) .TE . .Sh I/O PRIORITIES Every I/O request in the pipeline has a single priority value. This value is used by the queuing code to decide which I/O to issue next. This value will be set in an event as a .Sy zio_priority payload entry. .Pp .TS tab(:); l l l . Type:Value:Description _:_:_ ZIO_PRIORITY_SYNC_READ:0x0: ZIO_PRIORITY_SYNC_WRITE:0x1:ZIL ZIO_PRIORITY_ASYNC_READ:0x2:prefetch ZIO_PRIORITY_ASYNC_WRITE:0x3:spa_sync() ZIO_PRIORITY_SCRUB:0x4:asynchronous scrub/resilver reads ZIO_PRIORITY_REMOVAL:0x5:reads/writes for vdev removal ZIO_PRIORITY_INITIALIZING:0x6:initializing I/O ZIO_PRIORITY_TRIM:0x7:trim I/O (discard) ZIO_PRIORITY_REBUILD:0x8:reads/writes for vdev rebuild ZIO_PRIORITY_NOW:0xa:non-queued i/os (e.g. free) .TE . .Sh SEE ALSO .Xr zfs 4 , .Xr zed 8 , .Xr zpool-wait 8 ZPOOL-EXPORT(8) - System Manager's Manual # NAME **zpool-export** - export ZFS storage pools # SYNOPSIS **zpool** **export** \[**-f**] **-a**|*pool*… # DESCRIPTION Exports the given pools from the system. All devices are marked as exported, but are still considered in use by other subsystems. The devices can be moved between systems (even those of different endianness) and imported as long as a sufficient number of devices are present. Before exporting the pool, all datasets within the pool are unmounted. A pool can not be exported if it has a shared spare that is currently being used. For pools to be portable, you must give the **zpool** command whole disks, not just partitions, so that ZFS can label the disks with portable EFI labels. Otherwise, disk drivers on platforms of different endianness will not recognize the disks. **-a** > Exports all pools imported on the system. **-f** > Forcefully unmount all datasets, and allow export of pools with active shared > spares. > This command will forcefully export the pool even if it has a shared spare that > is currently being used. > This may lead to potential data corruption. # EXAMPLES ## Example 1: Exporting a ZFS Storage Pool The following command exports the devices in pool *tank* so that they can be relocated or later imported: # **zpool** **export** *tank* # SEE ALSO zpool-import(8) Debian - July 11, 2022 ZPOOL-GET(8) - System Manager's Manual # NAME **zpool-get** - retrieve properties of ZFS storage pools # SYNOPSIS **zpool** **get** \[**-Hp**] \[**-j** \[*--json-int, --json-pool-key-guid*]] \[**-o** *field*\[,*field*]…] **all**|*property*\[,*property*]… \[*pool*]… **zpool** **get** \[**-Hp**] \[**-j** \[*--json-int*]] \[**-o** *field*\[,*field*]…] **all**|*property*\[,*property*]… *pool* \[**all-vdevs**|*vdev*]… **zpool** **set** *property*=*value* *pool* **zpool** **set** *property*=*value* *pool* *vdev* # DESCRIPTION **zpool** **get** \[**-Hp**] \[**-j** \[*--json-int, --json-pool-key-guid*]] \[**-o** *field*\[,*field*]…] **all**|*property*\[,*property*]… \[*pool*]… > Retrieves the given list of properties > (or all properties if > **all** > is used) > for the specified storage pool(s). > These properties are displayed with the following fields: > **name** > > Name of storage pool. > **property** > > Property name. > **value** > > Property value. > **source** > > Property source, either > > **default** or **local**. > See the > zpoolprops(7) > manual page for more information on the available pool properties. > **-j**, **--json** \[*--json-int, --json-pool-key-guid*] > > Display the list of properties in JSON format. > > Specify > > **--json-int** > > to display the numbers in integer format instead of strings in JSON output. > > Specify > > **--json-pool-key-guid** > > to set pool GUID as key for pool objects instead of pool name. > **-H** > > Scripted mode. > > Do not display headers, and separate fields by a single tab instead of arbitrary > > space. > **-o** *field* > > A comma-separated list of columns to display, defaults to > > **name**,**property**,**value**,**source**. > **-p** > > Display numbers in parsable (exact) values. **zpool** **get** \[**-j** \[*--json-int*]] \[**-Hp**] \[**-o** *field*\[,*field*]…] **all**|*property*\[,*property*]… *pool* \[**all-vdevs**|*vdev*]… > Retrieves the given list of properties > (or all properties if > **all** > is used) > for the specified vdevs > (or all vdevs if > **all-vdevs** > is used) > in the specified pool. > These properties are displayed with the following fields: > **name** > > Name of vdev. > **property** > > Property name. > **value** > > Property value. > **source** > > Property source, either > > **default** or **local**. > See the > vdevprops(7) > manual page for more information on the available pool properties. > **-j**, **--json** \[*--json-int*] > > Display the list of properties in JSON format. > > Specify > > **--json-int** > > to display the numbers in integer format instead of strings in JSON output. > **-H** > > Scripted mode. > > Do not display headers, and separate fields by a single tab instead of arbitrary > > space. > **-o** *field* > > A comma-separated list of columns to display, defaults to > > **name**,**property**,**value**,**source**. > **-p** > > Display numbers in parsable (exact) values. **zpool** **set** *property*=*value* *pool* > Sets the given property on the specified pool. > See the > zpoolprops(7) > manual page for more information on what properties can be set and acceptable > values. **zpool** **set** *property*=*value* *pool* *vdev* > Sets the given property on the specified vdev in the specified pool. > See the > vdevprops(7) > manual page for more information on what properties can be set and acceptable > values. # SEE ALSO vdevprops(7), zpool-features(7), zpoolprops(7), zpool-list(8) Debian - October 12, 2024 ZPOOL-HISTORY(8) - System Manager's Manual # NAME **zpool-history** - inspect command history of ZFS storage pools # SYNOPSIS **zpool** **history** \[**-il**] \[*pool*]… # DESCRIPTION Displays the command history of the specified pool(s) or all pools if no pool is specified. **-i** > Displays internally logged ZFS events in addition to user initiated events. **-l** > Displays log records in long format, which in addition to standard format > includes, the user name, the hostname, and the zone in which the operation was > performed. # SEE ALSO zpool-checkpoint(8), zpool-events(8), zpool-status(8), zpool-wait(8) Debian - July 11, 2022 ZPOOL-IMPORT(8) - System Manager's Manual # NAME **zpool-import** - import ZFS storage pools or list available pools # SYNOPSIS **zpool** **import** \[**-D**] \[**-d** *dir*|*device*]… **zpool** **import** **-a** \[**-DflmN**] \[**-F** \[**-nTX**]] \[**--rewind-to-checkpoint**] \[**-c** *cachefile*|**-d** *dir*|*device*] \[**-o** *mntopts*] \[**-o** *property*=*value*]… \[**-R** *root*] **zpool** **import** \[**-Dflmt**] \[**-F** \[**-nTX**]] \[**--rewind-to-checkpoint**] \[**-c** *cachefile*|**-d** *dir*|*device*] \[**-o** *mntopts*] \[**-o** *property*=*value*]… \[**-R** *root*] \[**-s**] *pool*|*id* \[*newpool*] # DESCRIPTION **zpool** **import** \[**-D**] \[**-d** *dir*|*device*]… > Lists pools available to import. > If the > **-d** > or > **-c** > options are not specified, this command searches for devices using libblkid > on Linux and geom on > FreeBSD. > The > **-d** > option can be specified multiple times, and all directories are searched. > If the device appears to be part of an exported pool, this command displays a > summary of the pool with the name of the pool, a numeric identifier, as well as > the vdev layout and current health of the device for each device or file. > Destroyed pools, pools that were previously destroyed with the > **zpool** **destroy** > command, are not listed unless the > **-D** > option is specified. > The numeric identifier is unique, and can be used instead of the pool name when > multiple exported pools of the same name are available. > **-c** *cachefile* > > Reads configuration from the given > > *cachefile* > > that was created with the > > **cachefile** > > pool property. > > This > > *cachefile* > > is used instead of searching for devices. > **-d** *dir*|*device* > > Uses > > *device* > > or searches for devices or files in > > *dir*. > > The > > **-d** > > option can be specified multiple times. > **-D** > > Lists destroyed pools only. **zpool** **import** **-a** \[**-DflmN**] \[**-F** \[**-nTX**]] \[**-c** *cachefile*|**-d** *dir*|*device*] \[**-o** *mntopts*] \[**-o** *property*=*value*]… \[**-R** *root*] \[**-s**] > Imports all pools found in the search directories. > Identical to the previous command, except that all pools with a sufficient > number of devices available are imported. > Destroyed pools, pools that were previously destroyed with the > **zpool** **destroy** > command, will not be imported unless the > **-D** > option is specified. > **-a** > > Searches for and imports all pools found. > **-c** *cachefile* > > Reads configuration from the given > > *cachefile* > > that was created with the > > **cachefile** > > pool property. > > This > > *cachefile* > > is used instead of searching for devices. > **-d** *dir*|*device* > > Uses > > *device* > > or searches for devices or files in > > *dir*. > > The > > **-d** > > option can be specified multiple times. > > This option is incompatible with the > > **-c** > > option. > **-D** > > Imports destroyed pools only. > > The > > **-f** > > option is also required. > **-f** > > Forces import, even if the pool appears to be potentially active. > **-F** > > Recovery mode for a non-importable pool. > > Attempt to return the pool to an importable state by discarding the last few > > transactions. > > Not all damaged pools can be recovered by using this option. > > If successful, the data from the discarded transactions is irretrievably lost. > > This option is ignored if the pool is importable or already imported. > **-l** > > Indicates that this command will request encryption keys for all encrypted > > datasets it attempts to mount as it is bringing the pool online. > > Note that if any datasets have a > > **keylocation** > > of > > **prompt** > > this command will block waiting for the keys to be entered. > > Without this flag > > encrypted datasets will be left unavailable until the keys are loaded. > **-m** > > Allows a pool to import when there is a missing log device. > > Recent transactions can be lost because the log device will be discarded. > **-n** > > Used with the > > **-F** > > recovery option. > > Determines whether a non-importable pool can be made importable again, but does > > not actually perform the pool recovery. > > For more details about pool recovery mode, see the > > **-F** > > option, above. > **-N** > > Import the pool without mounting any file systems. > **-o** *mntopts* > > Comma-separated list of mount options to use when mounting datasets within the > > pool. > > See > > zfs(8) > > for a description of dataset properties and mount options. > **-o** *property*=*value* > > Sets the specified property on the imported pool. > > See the > > zpoolprops(7) > > manual page for more information on the available pool properties. > **-R** *root* > > Sets the > > **cachefile** > > property to > > **none** > > and the > > **altroot** > > property to > > *root*. > **--rewind-to-checkpoint** > > Rewinds pool to the checkpointed state. > > Once the pool is imported with this flag there is no way to undo the rewind. > > All changes and data that were written after the checkpoint are lost! > > The only exception is when the > > **readonly** > > mounting option is enabled. > > In this case, the checkpointed state of the pool is opened and an > > administrator can see how the pool would look like if they were > > to fully rewind. > **-s** > > Scan using the default search path, the libblkid cache will not be > > consulted. > > A custom search path may be specified by setting the > > **ZPOOL\_IMPORT\_PATH** > > environment variable. > **-X** > > Used with the > > **-F** > > recovery option. > > Determines whether extreme measures to find a valid txg should take place. > > This allows the pool to > > be rolled back to a txg which is no longer guaranteed to be consistent. > > Pools imported at an inconsistent txg may contain uncorrectable checksum errors. > > For more details about pool recovery mode, see the > > **-F** > > option, above. > > WARNING: This option can be extremely hazardous to the > > health of your pool and should only be used as a last resort. > **-T** > > Specify the txg to use for rollback. > > Implies > > **-FX**. > > For more details > > about pool recovery mode, see the > > **-X** > > option, above. > > WARNING: This option can be extremely hazardous to the > > health of your pool and should only be used as a last resort. **zpool** **import** \[**-Dflmt**] \[**-F** \[**-nTX**]] \[**-c** *cachefile*|**-d** *dir*|*device*] \[**-o** *mntopts*] \[**-o** *property*=*value*]… \[**-R** *root*] \[**-s**] *pool*|*id* \[*newpool*] > Imports a specific pool. > A pool can be identified by its name or the numeric identifier. > If > *newpool* > is specified, the pool is imported using the name > *newpool*. > Otherwise, it is imported with the same name as its exported name. > If a device is removed from a system without running > **zpool** **export** > first, the device appears as potentially active. > It cannot be determined if this was a failed export, or whether the device is > really in use from another host. > To import a pool in this state, the > **-f** > option is required. > **-c** *cachefile* > > Reads configuration from the given > > *cachefile* > > that was created with the > > **cachefile** > > pool property. > > This > > *cachefile* > > is used instead of searching for devices. > **-d** *dir*|*device* > > Uses > > *device* > > or searches for devices or files in > > *dir*. > > The > > **-d** > > option can be specified multiple times. > > This option is incompatible with the > > **-c** > > option. > **-D** > > Imports destroyed pool. > > The > > **-f** > > option is also required. > **-f** > > Forces import, even if the pool appears to be potentially active. > **-F** > > Recovery mode for a non-importable pool. > > Attempt to return the pool to an importable state by discarding the last few > > transactions. > > Not all damaged pools can be recovered by using this option. > > If successful, the data from the discarded transactions is irretrievably lost. > > This option is ignored if the pool is importable or already imported. > **-l** > > Indicates that this command will request encryption keys for all encrypted > > datasets it attempts to mount as it is bringing the pool online. > > Note that if any datasets have a > > **keylocation** > > of > > **prompt** > > this command will block waiting for the keys to be entered. > > Without this flag > > encrypted datasets will be left unavailable until the keys are loaded. > **-m** > > Allows a pool to import when there is a missing log device. > > Recent transactions can be lost because the log device will be discarded. > **-n** > > Used with the > > **-F** > > recovery option. > > Determines whether a non-importable pool can be made importable again, but does > > not actually perform the pool recovery. > > For more details about pool recovery mode, see the > > **-F** > > option, above. > **-o** *mntopts* > > Comma-separated list of mount options to use when mounting datasets within the > > pool. > > See > > zfs(8) > > for a description of dataset properties and mount options. > **-o** *property*=*value* > > Sets the specified property on the imported pool. > > See the > > zpoolprops(7) > > manual page for more information on the available pool properties. > **-R** *root* > > Sets the > > **cachefile** > > property to > > **none** > > and the > > **altroot** > > property to > > *root*. > **-s** > > Scan using the default search path, the libblkid cache will not be > > consulted. > > A custom search path may be specified by setting the > > **ZPOOL\_IMPORT\_PATH** > > environment variable. > **-X** > > Used with the > > **-F** > > recovery option. > > Determines whether extreme measures to find a valid txg should take place. > > This allows the pool to > > be rolled back to a txg which is no longer guaranteed to be consistent. > > Pools imported at an inconsistent txg may contain uncorrectable > > checksum errors. > > For more details about pool recovery mode, see the > > **-F** > > option, above. > > WARNING: This option can be extremely hazardous to the > > health of your pool and should only be used as a last resort. > **-T** > > Specify the txg to use for rollback. > > Implies > > **-FX**. > > For more details > > about pool recovery mode, see the > > **-X** > > option, above. > > *WARNING*: > > This option can be extremely hazardous to the > > health of your pool and should only be used as a last resort. > **-t** > > Used with > > *newpool*. > > Specifies that > > *newpool* > > is temporary. > > Temporary pool names last until export. > > Ensures that the original pool name will be used > > in all label updates and therefore is retained upon export. > > Will also set > > **-o** **cachefile**=**none** > > when not explicitly specified. # EXAMPLES ## Example 9: Importing a ZFS Storage Pool The following command displays available pools, and then imports the pool *tank* for use on the system. The results from this command are similar to the following: # **zpool** **import** pool: tank id: 15451357997522795478 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: tank ONLINE mirror ONLINE sda ONLINE sdb ONLINE # **zpool** **import** *tank* # SEE ALSO zpool-export(8), zpool-list(8), zpool-status(8) Debian - July 11, 2022 ZPOOL-INITIALIZE(8) - System Manager's Manual # NAME **zpool-initialize** - write to unallocated regions of ZFS storage pool # SYNOPSIS **zpool** **initialize** \[**-c**|**-s** |**-u**] \[**-w**] \[**-z**] **-a**|*pool* \[*device*]… # DESCRIPTION Begins initializing by writing to all unallocated regions on the specified devices, or all eligible devices in the pool if no individual devices are specified. Only leaf data or log devices may be initialized. By default a non-zero pattern is written; see **-z** to write zeroes instead. **-a**, **--all** > Begin, cancel, suspend initializing on > all > pools. **-c**, **--cancel** > Cancel initializing on the specified devices, or all eligible devices if none > are specified. > If one or more target devices are invalid or are not currently being > initialized, the command will fail and no cancellation will occur on any device. **-s**, **--suspend** > Suspend initializing on the specified devices, or all eligible devices if none > are specified. > If one or more target devices are invalid or are not currently being > initialized, the command will fail and no suspension will occur on any device. > Initializing can then be resumed by running > **zpool** **initialize** > with no flags on the relevant target devices. **-u**, **--uninit** > Clears the initialization state on the specified devices, or all eligible > devices if none are specified. > If the devices are being actively initialized the command will fail. > After being cleared > **zpool** **initialize** > with no flags can be used to re-initialize all unallocated regions on > the relevant target devices. **-w**, **--wait** > Wait until the devices have finished initializing before returning. **-z**, **--zero** > Write zeroes to the unallocated regions instead of the default non-zero > pattern. > This is useful for reclaiming space on thinly provisioned backing storage which > does not support TRIM. > If the backing storage supports TRIM, > zpool-trim(8) > is typically the more efficient way to return space to it. > The chosen value is recorded per device, so a run that is suspended and later > resumed keeps writing zeroes. # SEE ALSO zpool-add(8), zpool-attach(8), zpool-create(8), zpool-online(8), zpool-replace(8), zpool-trim(8) Debian - July 12, 2026 ZPOOL-IOSTAT(8) - System Manager's Manual # NAME **zpool-iostat** - display logical I/O statistics for ZFS storage pools # SYNOPSIS **zpool** **iostat** \[\[\[**-c** *SCRIPT*] \[**-lq**]]|**-rw**] \[**-T** **u**|**d**] \[**-ghHLnpPvy**] \[*pool*…|\[*pool vdev*…]|*vdev*…] \[*interval* \[*count*]] # DESCRIPTION Displays logical I/O statistics for the given pools/vdevs. Physical I/O statistics may be observed via iostat(1). If writes are located nearby, they may be merged into a single larger operation. Additional I/O may be generated depending on the level of vdev redundancy. To filter output, you may pass in a list of pools, a pool and list of vdevs in that pool, or a list of any vdevs from any pool. If no items are specified, statistics for every pool in the system are shown. When given an *interval*, the statistics are printed every *interval* seconds until killed. If **-n** flag is specified the headers are displayed only once, otherwise they are displayed periodically. If *count* is specified, the command exits after *count* reports are printed. The first report printed is always the average per-second statistics since boot regardless of whether *interval* and *count* are passed. However, this behavior can be suppressed with the **-y** flag. Also note that the units of **K**, **M**, **G**… that are printed in the report are in base 1024. To get the raw values, use the **-p** flag. **-c** \[*SCRIPT1*\[,*SCRIPT2*]…] > Run a script (or scripts) on each vdev and include the output as a new column > in the > **zpool** **iostat** > output. > Users can run any script found in their > *~/.zpool.d* > directory or from the system > */etc/zfs/zpool.d* > directory. > Script names containing the slash > (**/**) > character are not allowed. > The default search path can be overridden by setting the > **ZPOOL\_SCRIPTS\_PATH** > environment variable. > A privileged user can only run > **-c** > if they have the > **ZPOOL\_SCRIPTS\_AS\_ROOT** > environment variable set. > If a script requires the use of a privileged command, like > smartctl(8), > then it's recommended you allow the user access to it in > */etc/sudoers*. > If > **-c** > is passed without a script name, it prints a list of all scripts. > **-c** > also sets verbose mode > (**-v**). > Script output should be in the form of "name=value". > The column name is set to "name" and the value is set to "value". > Multiple lines can be used to output multiple columns. > The first line of output not in the > "name=value" format is displayed without a column title, > and no more output after that is displayed. > This can be useful for printing error messages. > Blank or NULL values are printed as a '-' to make output AWKable. > The following environment variables are set before running each script: > **VDEV\_PATH** > > Full path to the vdev > **VDEV\_UPATH** > > Underlying path to the vdev > > (*/dev/sd\*‌*). > > For use with device mapper, multipath, or partitioned vdevs. > **VDEV\_ENC\_SYSFS\_PATH** > > The sysfs path to the enclosure for the vdev (if any). **-T** **u**|**d** > Display a time stamp. > Specify > **u** > for a printed representation of the internal representation of time. > See > time(1). > Specify > **d** > for standard date format. > See > date(1). **-g** > Display vdev GUIDs instead of the normal device names. > These GUIDs can be used in place of device names for the zpool > detach/offline/remove/replace commands. **-H** > Scripted mode. > Do not display headers, and separate fields by a > single tab instead of arbitrary space. **-L** > Display real paths for vdevs resolving all symbolic links. > This can be used to look up the current block device name regardless of the > */dev/disk/* > path used to open it. **-n** > Print headers only once when passed **-p** > Display numbers in parsable (exact) values. > Time values are in nanoseconds. **-P** > Display full paths for vdevs instead of only the last component of the path. > This can be used in conjunction with the > **-L** > flag. **-r** > Print request size histograms for the leaf vdev's I/O. > This includes histograms of individual I/O (ind) and aggregate I/O (agg). > These stats can be useful for observing how well I/O aggregation is working. > Note that TRIM I/O may exceed 16M, but will be counted as 16M. **-v** > Verbose statistics Reports usage statistics for individual vdevs within the > pool, in addition to the pool-wide statistics. **-y** > Normally the first line of output reports the statistics since boot: > suppress it. **-w** > Display latency histograms: > **total\_wait** > > Total I/O time (queuing + disk I/O time). > **disk\_wait** > > Disk I/O time (time reading/writing the disk). > **syncq\_wait** > > Amount of time I/O spent in synchronous priority queues. > > Does not include disk time. > **asyncq\_wait** > > Amount of time I/O spent in asynchronous priority queues. > > Does not include disk time. > **scrub** > > Amount of time I/O spent in scrub queue. > > Does not include disk time. > **rebuild** > > Amount of time I/O spent in rebuild queue. > > Does not include disk time. **-l** > Include average latency statistics: > **total\_wait** > > Average total I/O time (queuing + disk I/O time). > **disk\_wait** > > Average disk I/O time (time reading/writing the disk). > **syncq\_wait** > > Average amount of time I/O spent in synchronous priority queues. > > Does not include disk time. > **asyncq\_wait** > > Average amount of time I/O spent in asynchronous priority queues. > > Does not include disk time. > **scrub** > > Average queuing time in scrub queue. > > Does not include disk time. > **trim** > > Average queuing time in trim queue. > > Does not include disk time. > **rebuild** > > Average queuing time in rebuild queue. > > Does not include disk time. **-q** > Include active queue statistics. > Each priority queue has both pending > (**pend**) > and active > (**activ**) > I/O requests. > Pending requests are waiting to be issued to the disk, > and active requests have been issued to disk and are waiting for completion. > These stats are broken out by priority queue: > **syncq\_read/write** > > Current number of entries in synchronous priority > > queues. > **asyncq\_read/write** > > Current number of entries in asynchronous priority queues. > **scrubq\_read** > > Current number of entries in scrub queue. > **trimq\_write** > > Current number of entries in trim queue. > **rebuildq\_write** > > Current number of entries in rebuild queue. > All queue statistics are instantaneous measurements of the number of > entries in the queues. > If you specify an interval, > the measurements will be sampled from the end of the interval. # EXAMPLES ## Example 13: Adding Cache Devices to a ZFS Pool The following command adds two disks for use as cache devices to a ZFS storage pool: # **zpool** **add** *pool* **cache** *sdc sdd* Once added, the cache devices gradually fill with content from main memory. Depending on the size of your cache devices, it could take over an hour for them to fill. Capacity and reads can be monitored using the **iostat** subcommand as follows: # **zpool** **iostat** **-v** *pool 5* ## Example 16: Adding output columns Additional columns can be added to the **zpool** **status** and **zpool** **iostat** output with **-c**. # **zpool** **status** **-c** *vendor*,*model*,*size* NAME STATE READ WRITE CKSUM vendor model size tank ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 U1 ONLINE 0 0 0 SEAGATE ST8000NM0075 7.3T U10 ONLINE 0 0 0 SEAGATE ST8000NM0075 7.3T U11 ONLINE 0 0 0 SEAGATE ST8000NM0075 7.3T U12 ONLINE 0 0 0 SEAGATE ST8000NM0075 7.3T U13 ONLINE 0 0 0 SEAGATE ST8000NM0075 7.3T U14 ONLINE 0 0 0 SEAGATE ST8000NM0075 7.3T # **zpool** **iostat** **-vc** *size* capacity operations bandwidth pool alloc free read write read write size ---------- ----- ----- ----- ----- ----- ----- ---- rpool 14.6G 54.9G 4 55 250K 2.69M sda1 14.6G 54.9G 4 55 250K 2.69M 70G ---------- ----- ----- ----- ----- ----- ----- ---- # SEE ALSO iostat(1), smartctl(8), zpool-list(8), zpool-status(8) Debian - January 29, 2024 ZPOOL-LABELCLEAR(8) - System Manager's Manual # NAME **zpool-labelclear** - remove ZFS label information from device # SYNOPSIS **zpool** **labelclear** \[**-f**] *device* # DESCRIPTION Removes ZFS label information from the specified *device*. If the *device* is a cache device, it also removes the L2ARC header (persistent L2ARC). The *device* must not be part of an active pool configuration. This overwrites pool metadata, making all data on the *device* inaccessible without specialized recovery tools. Unlike **zpool** **destroy**, this cannot be undone by ZFS tooling. Only use this when you don't care about the data on the *device*. **-f** > Treat exported or foreign devices as inactive. # SEE ALSO zpool-destroy(8), zpool-detach(8), zpool-remove(8), zpool-replace(8) Debian - July 11, 2022 ZPOOL-LIST(8) - System Manager's Manual # NAME **zpool-list** - list information about ZFS storage pools # SYNOPSIS **zpool** **list** \[**-HgLpPv**] \[**-j** \[*--json-int, --json-pool-key-guid*]] \[**-o** *property*\[,*property*]…] \[**-T** **u**|**d**] \[*pool*]… \[*interval* \[*count*]] # DESCRIPTION Lists the given pools along with a health status and space usage. If no *pool*s are specified, all pools currently imported are listed. For listing pools available for import, see zpool-import(8). When given an *interval*, the information is printed every *interval* seconds until killed. If *count* is specified, the command exits after *count* reports are printed. **-j**, **--json** \[*--json-int, --json-pool-key-guid*] > Display the list of pools in JSON format. > Specify > **--json-int** > to display the numbers in integer format instead of strings. > Specify > **--json-pool-key-guid** > to set pool GUID as key for pool objects instead of pool names. **-g** > Display vdev GUIDs instead of the normal device names. > These GUIDs can be used in place of device names for the zpool > detach/offline/remove/replace commands. **-H** > Scripted mode. > Do not display headers, and separate fields by a single tab instead of arbitrary > space. **-o** *property* > Comma-separated list of properties to display. > See the > zpoolprops(7) > manual page for a list of valid properties. > The default list is > **name**, **size**, **allocated**, **free**, **checkpoint, expandsize**, **fragmentation**, > **capacity**, **dedupratio**, **health**, **altroot**. **-L** > Display real paths for vdevs resolving all symbolic links. > This can be used to look up the current block device name regardless of the > */dev/disk* > path used to open it. **-p** > Display numbers in parsable > (exact) > values. **-P** > Display full paths for vdevs instead of only the last component of > the path. > This can be used in conjunction with the > **-L** > flag. **-T** **u**|**d** > Display a time stamp. > Specify > **u** > for a printed representation of the internal representation of time. > See > time(1). > Specify > **d** > for standard date format. > See > date(1). **-v** > Verbose statistics. > Reports usage statistics for individual vdevs within the pool, in addition to > the pool-wide statistics. # EXAMPLES ## Example 1: Listing Available ZFS Storage Pools The following command lists all available pools on the system. In this case, the pool *zion* is faulted due to a missing device. The results from this command are similar to the following: # **zpool** **list** NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT rpool 19.9G 8.43G 11.4G - 33% 42% 1.00x ONLINE - tank 61.5G 20.0G 41.5G - 48% 32% 1.00x ONLINE - zion - - - - - - - FAULTED - ## Example 2: Displaying expanded space on a device The following command displays the detailed information for the pool *data*. This pool is comprised of a single raidz vdev where one of its devices increased its capacity by 10 GiB. In this example, the pool will not be able to utilize this extra capacity until all the devices under the raidz vdev have been expanded. # **zpool** **list** **-v** *data* NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT data 23.9G 14.6G 9.30G - 48% 61% 1.00x ONLINE - raidz1 23.9G 14.6G 9.30G - 48% sda - - - - - sdb - - - 10G - sdc - - - - - ## Example 3: Displaying expanded space on a device The following command lists all available pools on the system in JSON format. # **zpool** **list** **-j** | **jq** { "output_version": { "command": "zpool list", "vers_major": 0, "vers_minor": 1 }, "pools": { "tank": { "name": "tank", "type": "POOL", "state": "ONLINE", "guid": "15220353080205405147", "txg": "2671", "spa_version": "5000", "zpl_version": "5", "properties": { "size": { "value": "111G", "source": { "type": "NONE", "data": "-" } }, "allocated": { "value": "30.8G", "source": { "type": "NONE", "data": "-" } }, "free": { "value": "80.2G", "source": { "type": "NONE", "data": "-" } }, "checkpoint": { "value": "-", "source": { "type": "NONE", "data": "-" } }, "expandsize": { "value": "-", "source": { "type": "NONE", "data": "-" } }, "fragmentation": { "value": "0%", "source": { "type": "NONE", "data": "-" } }, "capacity": { "value": "27%", "source": { "type": "NONE", "data": "-" } }, "dedupratio": { "value": "1.00x", "source": { "type": "NONE", "data": "-" } }, "health": { "value": "ONLINE", "source": { "type": "NONE", "data": "-" } }, "altroot": { "value": "-", "source": { "type": "DEFAULT", "data": "-" } } } } } } # SEE ALSO zpool-import(8), zpool-status(8) Debian - October 12, 2024 ZPOOL-OFFLINE(8) - System Manager's Manual # NAME **zpool-offline** - take physical devices offline in ZFS storage pool # SYNOPSIS **zpool** **offline** \[**-**‌**-power**|\[**-**‌**ft**]] *pool* *device*… **zpool** **online** \[**-**‌**-power**] \[**-**‌**e**] *pool* *device*… # DESCRIPTION **zpool** **offline** \[**-**‌**-power**|\[**-**‌**ft**]] *pool* *device*… > Takes the specified physical device offline or force-fault it. > While the > *device* > is offline or force-faulted, no attempt is made to read or write to the device. > dRAID spares can not be offlined or force faulted. > Traditional spares can only be offlined or force-faulted when they are active. > **--power** > > Power off the device's slot in the storage enclosure. > > This flag currently works on Linux only > **-f** > > Force fault. > > Instead of offlining the disk, put it into a faulted state. > > The fault will persist across imports unless the > > **-t** > > flag was specified. > **-t** > > Temporary. > > Upon reboot, the specified physical device reverts to its previous state. **zpool** **online** \[**--power**] \[**-e**] *pool* *device*… > Brings the specified physical device online. > This command is not applicable to spares. > **--power** > > Power on the device's slot in the storage enclosure and wait for the device > > to show up before attempting to online it. > > Alternatively, you can set the > > **ZPOOL\_AUTO\_POWER\_ON\_SLOT** > > environment variable to always enable this behavior. > > This flag currently works on Linux only > **-e** > > Expand the device to use all available space. > > If the device is part of a mirror or raidz then all devices must be expanded > > before the new space will become available to the pool. # SEE ALSO zpool-detach(8), zpool-remove(8), zpool-reopen(8), zpool-resilver(8) Debian - December 21, 2023 ZPOOL-OFFLINE(8) - System Manager's Manual # NAME **zpool-offline** - take physical devices offline in ZFS storage pool # SYNOPSIS **zpool** **offline** \[**-**‌**-power**|\[**-**‌**ft**]] *pool* *device*… **zpool** **online** \[**-**‌**-power**] \[**-**‌**e**] *pool* *device*… # DESCRIPTION **zpool** **offline** \[**-**‌**-power**|\[**-**‌**ft**]] *pool* *device*… > Takes the specified physical device offline or force-fault it. > While the > *device* > is offline or force-faulted, no attempt is made to read or write to the device. > dRAID spares can not be offlined or force faulted. > Traditional spares can only be offlined or force-faulted when they are active. > **--power** > > Power off the device's slot in the storage enclosure. > > This flag currently works on Linux only > **-f** > > Force fault. > > Instead of offlining the disk, put it into a faulted state. > > The fault will persist across imports unless the > > **-t** > > flag was specified. > **-t** > > Temporary. > > Upon reboot, the specified physical device reverts to its previous state. **zpool** **online** \[**--power**] \[**-e**] *pool* *device*… > Brings the specified physical device online. > This command is not applicable to spares. > **--power** > > Power on the device's slot in the storage enclosure and wait for the device > > to show up before attempting to online it. > > Alternatively, you can set the > > **ZPOOL\_AUTO\_POWER\_ON\_SLOT** > > environment variable to always enable this behavior. > > This flag currently works on Linux only > **-e** > > Expand the device to use all available space. > > If the device is part of a mirror or raidz then all devices must be expanded > > before the new space will become available to the pool. # SEE ALSO zpool-detach(8), zpool-remove(8), zpool-reopen(8), zpool-resilver(8) Debian - December 21, 2023 ZPOOL-PREFETCH(8) - System Manager's Manual # NAME **zpool-prefetch** - Prefetches pool metadata into ARC # SYNOPSIS **zpool** **prefetch** \[**-t** *type*] *pool* # DESCRIPTION Massively prefetch metadata of a specific type for the given pool into the ARC to reduce latency of some operations later. If no type is specified, all types are prefetched. The following types are supported: **brt** > Prefetch the BRT (block reference table). > This may improve performance for block cloning operations, > and frees for earlier cloned blocks. **ddt** > Prefetch the DDT (deduplication table). > This may improve performance of writes when deduplication is enabled, > and frees for earlier deduplicated blocks. Debian - February 14, 2024 ZPOOL-REGUID(8) - System Manager's Manual # NAME **zpool-reguid** - generate new unique identifier for ZFS storage pool # SYNOPSIS **zpool** **reguid** \[**-g** *guid*] *pool* # DESCRIPTION Generates a new unique identifier for the pool. You must ensure that all devices in this pool are online and healthy before performing this action. **-g** *guid* > Set the pool GUID to the provided value. > The GUID can be any 64-bit value accepted by > strtoull(3) > in base 10. > **zpool-reguid** > will return an error if the provided GUID is already in use. # SEE ALSO zpool-export(8), zpool-import(8) Debian - August 26, 2024 ZPOOL-REMOVE(8) - System Manager's Manual # NAME **zpool-remove** - remove devices from ZFS storage pool # SYNOPSIS **zpool** **remove** \[**-npw**] *pool* *device*… **zpool** **remove** **-s** *pool* # DESCRIPTION **zpool** **remove** \[**-npw**] *pool* *device*… > Removes the specified device from the pool. > This command supports removing hot spare, cache, log, and both mirrored and > non-redundant primary top-level vdevs, including dedup and special vdevs. > Top-level vdevs can only be removed if the primary pool storage does not contain > a top-level raidz or draid vdev, all top-level vdevs have the same ashift size, > and the keys for all encrypted datasets are loaded. > Removing a top-level vdev reduces the total amount of space in the storage pool. > The specified device will be evacuated by copying all allocated space from it to > the other devices in the pool. > In this case, the > **zpool** **remove** > command initiates the removal and returns, while the evacuation continues in > the background. > The removal progress can be monitored with > **zpool** **status**. > If an I/O error is encountered during the removal process it will be canceled. > The > **device\_removal** > feature flag must be enabled to remove a top-level vdev, see > zpool-features(7). > A mirrored top-level device (log or data) can be removed by specifying the top- > level mirror for the > same. > Non-log devices or data devices that are part of a mirrored configuration can be > removed using > the > **zpool** **detach** > command. > **-n** > > Do not actually perform the removal > > ("No-op"). > > Instead, print the estimated amount of memory that will be used by the > > mapping table after the removal completes. > > This is nonzero only for top-level vdevs. > **-p** > > Used in conjunction with the > > **-n** > > flag, displays numbers as parsable (exact) values. > **-w** > > Waits until the removal has completed before returning. **zpool** **remove** **-s** *pool* > Stops and cancels an in-progress removal of a top-level vdev. # EXAMPLES ## Example 1: Removing a Mirrored top-level (Log or Data) Device The following commands remove the mirrored log device **mirror-2** and mirrored top-level data device **mirror-1**. Given this configuration: pool: tank state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 sda ONLINE 0 0 0 sdb ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 sdc ONLINE 0 0 0 sdd ONLINE 0 0 0 logs mirror-2 ONLINE 0 0 0 sde ONLINE 0 0 0 sdf ONLINE 0 0 0 The command to remove the mirrored log *mirror-2* is: # **zpool** **remove** *tank mirror-2* At this point, the log device no longer exists (both sides of the mirror have been removed): pool: tank state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 sda ONLINE 0 0 0 sdb ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 sdc ONLINE 0 0 0 sdd ONLINE 0 0 0 The command to remove the mirrored data *mirror-1* is: # **zpool** **remove** *tank mirror-1* After *mirror-1* has been evacuated, the pool remains redundant, but the total amount of space is reduced: pool: tank state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 sda ONLINE 0 0 0 sdb ONLINE 0 0 0 # SEE ALSO zpool-add(8), zpool-detach(8), zpool-labelclear(8), zpool-offline(8), zpool-replace(8), zpool-split(8) Debian - November 19, 2024 ZPOOL-REOPEN(8) - System Manager's Manual # NAME **zpool-reopen** - reopen vdevs associated with ZFS storage pools # SYNOPSIS **zpool** **reopen** \[**-n**] \[*pool*]… # DESCRIPTION Reopen all vdevs associated with the specified pools, or all pools if none specified. # OPTIONS **-n** > Do not restart an in-progress scrub operation. > This is not recommended and can > result in partially resilvered devices unless a second scrub is performed. Debian - July 11, 2022 ZPOOL-REPLACE(8) - System Manager's Manual # NAME **zpool-replace** - replace one device with another in ZFS storage pool # SYNOPSIS **zpool** **replace** \[**-fsw**] \[**-o** *property*=*value*] *pool* *device* \[*new-device*] # DESCRIPTION Replaces *device* with *new-device*. This is equivalent to attaching *new-device*, waiting for it to resilver, and then detaching *device*. Any in progress scrub will be canceled. The size of *new-device* must be greater than or equal to the minimum size of all the devices in a mirror or raidz configuration. *new-device* is required if the pool is not redundant. If *new-device* is not specified, it defaults to *device*. This form of replacement is useful after an existing disk has failed and has been physically replaced. In this case, the new disk may have the same */dev* path as the old device, even though it is actually a different disk. ZFS recognizes this. **-f** > Forces use of > *new-device*, > even if it appears to be in use. > Not all devices can be overridden in this manner. **-o** *property*=*value* > Sets the given pool properties. > See the > zpoolprops(7) > manual page for a list of valid properties that can be set. > The only property supported at the moment is > **ashift**. **-s** > The > *new-device* > is reconstructed sequentially to restore redundancy as quickly as possible. > Checksums are not verified during sequential reconstruction so a scrub is > started when the resilver completes. > Sequential reconstruction is not supported for raidz configurations. **-w** > Waits until the replacement has completed before returning. # SEE ALSO zpool-detach(8), zpool-initialize(8), zpool-online(8), zpool-resilver(8) Debian - July 11, 2022 ZPOOL-RESILVER(8) - System Manager's Manual # NAME **zpool-resilver** - resilver devices in ZFS storage pools # SYNOPSIS **zpool** **resilver** *pool*… # DESCRIPTION Starts a resilver of the specified pools. If an existing resilver is already running it will be restarted from the beginning. Any drives that were scheduled for a deferred resilver will be added to the new one. This requires the **resilver\_defer** pool feature. Due to concurrent writes on a live system, it is possible for resilver to progress beyond 100% completion. # SEE ALSO zpool-iostat(8), zpool-online(8), zpool-reopen(8), zpool-replace(8), zpool-scrub(8), zpool-status(8) Debian - July 11, 2022 ZPOOL-SCRUB(8) - System Manager's Manual # NAME **zpool-scrub** - begin or resume scrub of ZFS storage pools # SYNOPSIS **zpool** **scrub** \[**-w**] \[**-t**] \[**-C** | \[**-S** *date*] \[**-E** *date*]] **-a**|*pool*… **zpool** **scrub** **-e** \[**-w**] **-a**|*pool*… **zpool** **scrub** **-p** **-a**|*pool*… **zpool** **scrub** **-s** **-a**|*pool*… # DESCRIPTION Begins a scrub or resumes a paused scrub. A normal scrub examines all data in the specified pools and verifies each block's checksum. A thorough scrub additionally decrypts and/or decompresses blocks as they are read. For replicated (mirror, raidz, or draid) devices, ZFS automatically repairs any damage discovered during the scrub. The **zpool** **status** command reports the progress of the scrub and summarizes the results of the scrub upon completion. Scrubbing and resilvering are very similar operations. The difference is that resilvering only examines data that ZFS knows to be out of date (for example, when attaching a new device to a mirror or replacing an existing device) , whereas scrubbing examines all data to discover silent errors due to hardware faults or disk failure. When scrubbing a pool with encrypted filesystems the keys do not need to be loaded for ordinary checksum verification but are needed for thorough scrub (see **-t**) below. If the keys are not loaded and an unrepairable checksum error is detected the file name cannot be included in the **zpool** **status** **-v** verbose error report. Because scrubbing and resilvering are I/O-intensive operations, ZFS only allows one at a time. A scrub is split into two parts: metadata scanning and block scrubbing. The metadata scanning sorts blocks into large sequential ranges which can then be read much more efficiently from disk when issuing the scrub I/O. If a scrub is paused, the **zpool** **scrub** resumes it. If a resilver is in progress, ZFS does not allow a scrub to be started until the resilver completes. Note that, due to changes in pool data on a live system, it is possible for scrubs and resilvers to progress beyond 100% completion. During this period, no completion time estimate will be provided. # OPTIONS **-a**, **--all** > Begin, pause > **-p**, > or stop > **-s**, > scrub on all pools. > Initiating scrubs on multiple pools can put considerable load and memory > pressure on the system, so this operation should be performed with caution. **-s** > Stop scrubbing. > Cannot be combined with other options except > **-a**. **-p** > Pause scrubbing. > Cannot be combined with other options except > **-a**. > Scrub pause state and progress are periodically synced to disk. > If the system is restarted or pool is exported during a paused scrub, > even after import, scrub will remain paused until it is resumed. > Once resumed the scrub will pick up from the place where it was last > checkpointed to disk. > To resume a paused scrub issue > **zpool** **scrub**. > The scrub resumes in the same mode > (normal or thorough) > it had when it started. > To resume paused error scrub, issue > **zpool** **scrub** **-e**. **-w** > Wait until scrub has completed before returning. **-e** > Only scrub files with known data errors as reported by > **zpool** **status** **-v**. > The pool must have been scrubbed at least once with the > **head\_errlog** > feature enabled to use this option. > Error scrubbing cannot be run simultaneously with other scrubbing or > resilvering, nor can it be run when a scrub is paused. > This option cannot be combined with > **-C**, > **-p**, > **-s**, > or > **-t**. **-C** > Continue scrub from last saved txg (see zpool > **last\_scrubbed\_txg** > property). > Cannot be combined with > **-S**, > **-E**, > **-e**, > **-p**, > or > **-s**. > May be combined with thorough scrub > (**-t**). **-t** > Thorough scrub. > Will cause scrub to decrypt and decompress blocks it reads so that it will > catch the rare type of corruption where the checksum matches the data, but > decryption and/or decompression fails. > For encrypted datasets keys have to be loaded in order to perform > thorough scrub. > If keys are not loaded or unloaded during a thorough scrub the scrub will revert > to doing a normal scrub for the encrypted blocks with key(s) unloaded. > Cannot be combined with > **-e**, > **-p**, > or > **-s**. > May be combined with > **-a**, > **-w**, > **-C** > or > start/end date options > (**-S** **-/** **-E**). **-S** *date*, **-E** *date* > Allows specifying the date range for blocks created between these dates. > * **-S** > Defines a start date. > If not specified, scrubbing begins from the start of the pool's > existence. > * **-E** > Defines an end date. > If not specified, scrubbing continues up to the most recent data. > The provided date should be in the format: > "YYYY-MM-DD HH:MM". > Where: > * "YYYY" > is the year. > * "MM" > is the numeric representation of the month. > * "DD" > is the day of the month. > * "HH" > is the hour. > * "MM" > is the minutes. > The hour and minutes parameters can be omitted. > The time should be provided in machine local time zone. > Specifying dates prior to enabling this feature will result in scrubbing > starting from the date the pool was created. > Large or discontinuous changes to the system clock (forward or backward) > can skew how the start and/or end dates map to pool transaction groups. > As a result, the scrub may not cover the intended dates. > Cannot be combined with > **-C**. # EXAMPLES ## Example 1 Status of pool with ongoing scrub: # **zpool** **status** ... scan: scrub in progress since Sun Jul 25 16:07:49 2021 403M / 405M scanned at 100M/s, 68.4M / 405M issued at 10.0M/s 0B repaired, 16.91% done, 00:00:04 to go ... Where metadata which references 403M of file data has been scanned at 100M/s, and 68.4M of that file data has been scrubbed sequentially at 10.0M/s. # PERIODIC SCRUB On machines using systemd, scrub timers can be enabled on per-pool basis. **weekly** and **monthly** timer units are provided. > **systemctl** > **enable** > **zfs-scrub-*weekly*‌**@**‌*rpool*‌**.timer**‌** > **--now** > **systemctl** > **enable** > **zfs-scrub-*monthly*‌**@**‌*otherpool*‌**.timer**‌** > **--now** # SEE ALSO systemd.timer(5), zpool-iostat(8), zpool-resilver(8), zpool-status(8) Debian - May 1, 2026 ZPOOL-GET(8) - System Manager's Manual # NAME **zpool-get** - retrieve properties of ZFS storage pools # SYNOPSIS **zpool** **get** \[**-Hp**] \[**-j** \[*--json-int, --json-pool-key-guid*]] \[**-o** *field*\[,*field*]…] **all**|*property*\[,*property*]… \[*pool*]… **zpool** **get** \[**-Hp**] \[**-j** \[*--json-int*]] \[**-o** *field*\[,*field*]…] **all**|*property*\[,*property*]… *pool* \[**all-vdevs**|*vdev*]… **zpool** **set** *property*=*value* *pool* **zpool** **set** *property*=*value* *pool* *vdev* # DESCRIPTION **zpool** **get** \[**-Hp**] \[**-j** \[*--json-int, --json-pool-key-guid*]] \[**-o** *field*\[,*field*]…] **all**|*property*\[,*property*]… \[*pool*]… > Retrieves the given list of properties > (or all properties if > **all** > is used) > for the specified storage pool(s). > These properties are displayed with the following fields: > **name** > > Name of storage pool. > **property** > > Property name. > **value** > > Property value. > **source** > > Property source, either > > **default** or **local**. > See the > zpoolprops(7) > manual page for more information on the available pool properties. > **-j**, **--json** \[*--json-int, --json-pool-key-guid*] > > Display the list of properties in JSON format. > > Specify > > **--json-int** > > to display the numbers in integer format instead of strings in JSON output. > > Specify > > **--json-pool-key-guid** > > to set pool GUID as key for pool objects instead of pool name. > **-H** > > Scripted mode. > > Do not display headers, and separate fields by a single tab instead of arbitrary > > space. > **-o** *field* > > A comma-separated list of columns to display, defaults to > > **name**,**property**,**value**,**source**. > **-p** > > Display numbers in parsable (exact) values. **zpool** **get** \[**-j** \[*--json-int*]] \[**-Hp**] \[**-o** *field*\[,*field*]…] **all**|*property*\[,*property*]… *pool* \[**all-vdevs**|*vdev*]… > Retrieves the given list of properties > (or all properties if > **all** > is used) > for the specified vdevs > (or all vdevs if > **all-vdevs** > is used) > in the specified pool. > These properties are displayed with the following fields: > **name** > > Name of vdev. > **property** > > Property name. > **value** > > Property value. > **source** > > Property source, either > > **default** or **local**. > See the > vdevprops(7) > manual page for more information on the available pool properties. > **-j**, **--json** \[*--json-int*] > > Display the list of properties in JSON format. > > Specify > > **--json-int** > > to display the numbers in integer format instead of strings in JSON output. > **-H** > > Scripted mode. > > Do not display headers, and separate fields by a single tab instead of arbitrary > > space. > **-o** *field* > > A comma-separated list of columns to display, defaults to > > **name**,**property**,**value**,**source**. > **-p** > > Display numbers in parsable (exact) values. **zpool** **set** *property*=*value* *pool* > Sets the given property on the specified pool. > See the > zpoolprops(7) > manual page for more information on what properties can be set and acceptable > values. **zpool** **set** *property*=*value* *pool* *vdev* > Sets the given property on the specified vdev in the specified pool. > See the > vdevprops(7) > manual page for more information on what properties can be set and acceptable > values. # SEE ALSO vdevprops(7), zpool-features(7), zpoolprops(7), zpool-list(8) Debian - October 12, 2024 ZPOOL-SPLIT(8) - System Manager's Manual # NAME **zpool-split** - split devices off ZFS storage pool, creating new pool # SYNOPSIS **zpool** **split** \[**-gLlnP**] \[**-o** *property*=*value*]… \[**-R** *root*] *pool newpool* \[*device*]… # DESCRIPTION Splits devices off *pool* creating *newpool*. All vdevs in *pool* must be mirrors and the pool must not be in the process of resilvering. At the time of the split, *newpool* will be a replica of *pool*. By default, the last device in each mirror is split from *pool* to create *newpool*. The optional device specification causes the specified device(s) to be included in the new *pool* and, should any devices remain unspecified, the last device in each mirror is used as would be by default. **-g** > Display vdev GUIDs instead of the normal device names. > These GUIDs can be used in place of device names for the zpool > detach/offline/remove/replace commands. **-L** > Display real paths for vdevs resolving all symbolic links. > This can be used to look up the current block device name regardless of the > */dev/disk/* > path used to open it. **-l** > Indicates that this command will request encryption keys for all encrypted > datasets it attempts to mount as it is bringing the new pool online. > Note that if any datasets have > **keylocation**=**prompt**, > this command will block waiting for the keys to be entered. > Without this flag, encrypted datasets will be left unavailable until the keys > are loaded. **-n** > Do a dry-run > ("No-op") > split: do not actually perform it. > Print out the expected configuration of > *newpool*. **-P** > Display full paths for vdevs instead of only the last component of > the path. > This can be used in conjunction with the > **-L** > flag. **-o** *property*=*value* > Sets the specified property for > *newpool*. > See the > zpoolprops(7) > manual page for more information on the available pool properties. **-R** *root* > Set > **altroot** > for > *newpool* > to > *root* > and automatically import it. # SEE ALSO zpool-import(8), zpool-list(8), zpool-remove(8) Debian - July 11, 2022 ZPOOL-STATUS(8) - System Manager's Manual # NAME **zpool-status** - show detailed health status for ZFS storage pools # SYNOPSIS **zpool** **status** \[**-DdegiLPpstvx**] \[**-c** *script1*\[,*script2*,…]] \[**-j|--json** \[**--json-flat-vdevs**] \[**--json-int**] \[**--json-pool-key-guid**]] \[**-T** *d|u*] \[**--power**] \[*pool*] \[*interval* \[*count*]] # DESCRIPTION Displays the detailed health status for the given pools. If no *pool* is specified, then the status of each pool in the system is displayed. For more information on pool and device health, see the *Device Failure and Recovery* section of zpoolconcepts(7). If a scrub or resilver is in progress, this command reports the percentage done and the estimated time to completion. Both of these are only approximate, because the amount of data in the pool and the other workloads on the system can change. Due to concurrent writes on a live system, it is possible for both scrubs and resilvers to progress beyond 100% completion. **-c** *script1*\[,*script2*,…] > Run a script (or scripts) on each vdev and include the output as a new column > in the > **zpool** **status** > output. > See the > **-c** > option of > **zpool** **iostat** > for complete details. **-D** > Display a histogram of deduplication statistics, showing the allocated > (physically present on disk) > and referenced > (logically referenced in the pool) > block counts and sizes by reference count. > If repeated, (-DD), also shows statistics on how much of the DDT is resident > in the ARC. **-d** > Display the number of Direct I/O read/write checksum verify errors that have > occurred on a top-level VDEV. > See > *zfs\_vdev\_direct\_write\_verify* > in > zfs(4) > for details about the conditions that can cause Direct I/O write checksum > verify failures to occur. > Direct I/O reads checksum verify errors can also occur if the contents of the > buffer are being manipulated after the I/O has been issued and is in flight. > In the case of Direct I/O read checksum verify errors, the I/O will be reissued > through the ARC. **-e** > Only show unhealthy vdevs (not-ONLINE or with errors). **-g** > Display vdev GUIDs instead of the normal device names > These GUIDs can be used in place of device names for the zpool > detach/offline/remove/replace commands. **-i** > Display vdev initialization status. **-j**, **--json** \[**--json-flat-vdevs**] \[**--json-int**] \[**--json-pool-key-guid**] > Display the status for ZFS pools in JSON format. > Specify > **--json-flat-vdevs** > to display vdevs in flat hierarchy instead of nested vdev objects. > Specify > **--json-int** > to display numbers in integer format instead of strings. > Specify > **--json-pool-key-guid** > to set pool GUID as key for pool objects instead of pool names. **-L** > Display real paths for vdevs resolving all symbolic links. > This can be used to look up the current block device name regardless of the > */dev/disk/* > path used to open it. **-P** > Display full paths for vdevs instead of only the last component of > the path. > This can be used in conjunction with the > **-L** > flag. **-p** > Display numbers in parsable (exact) values. **--power** > Display vdev enclosure slot power status (on or off). **-s** > Display the number of leaf vdev slow I/O operations. > This is the number of I/O operations that didn't complete in > **zio\_slow\_io\_ms** > milliseconds > (**30000** by default). > This does not necessarily mean the I/O operations failed to complete, just took > an > unreasonably long amount of time. > This may indicate a problem with the underlying storage. **-T** **d**|**u** > Display a time stamp. > Specify > **d** > for standard date format. > See > date(1). > Specify > **u** > for a printed representation of the internal representation of time. > See > time(1). **-t** > Display vdev TRIM status. **-v** > Displays verbose data error information, printing out a complete list of all > data errors since the last complete pool scrub. > If the head\_errlog feature is enabled and files containing errors have been > removed then the respective filenames will not be reported in subsequent runs > of this command. **-x** > Only display status for pools that are exhibiting errors or are otherwise > unavailable. > Warnings about pools not using the latest on-disk format will not be included. # EXAMPLES ## Example 1: Adding output columns Additional columns can be added to the **zpool** **status** and **zpool** **iostat** output with **-c**. # **zpool** **status** **-c** *vendor*,*model*,*size* NAME STATE READ WRITE CKSUM vendor model size tank ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 U1 ONLINE 0 0 0 SEAGATE ST8000NM0075 7.3T U10 ONLINE 0 0 0 SEAGATE ST8000NM0075 7.3T U11 ONLINE 0 0 0 SEAGATE ST8000NM0075 7.3T U12 ONLINE 0 0 0 SEAGATE ST8000NM0075 7.3T U13 ONLINE 0 0 0 SEAGATE ST8000NM0075 7.3T U14 ONLINE 0 0 0 SEAGATE ST8000NM0075 7.3T # **zpool** **iostat** **-vc** *size* capacity operations bandwidth pool alloc free read write read write size ---------- ----- ----- ----- ----- ----- ----- ---- rpool 14.6G 54.9G 4 55 250K 2.69M sda1 14.6G 54.9G 4 55 250K 2.69M 70G ---------- ----- ----- ----- ----- ----- ----- ---- ## Example 2: Display the status output in JSON format **zpool** **status** can output in JSON format if **-j** is specified. **-c** can be used to run a script on each VDEV. # **zpool** **status** **-j** **-c** *vendor*,*model*,*size* | **jq** { "output_version": { "command": "zpool status", "vers_major": 0, "vers_minor": 1 }, "pools": { "tank": { "name": "tank", "state": "ONLINE", "guid": "3920273586464696295", "txg": "16597", "spa_version": "5000", "zpl_version": "5", "status": "OK", "vdevs": { "tank": { "name": "tank", "alloc_space": "62.6G", "total_space": "15.0T", "def_space": "11.3T", "read_errors": "0", "write_errors": "0", "checksum_errors": "0", "vdevs": { "raidz1-0": { "name": "raidz1-0", "vdev_type": "raidz", "guid": "763132626387621737", "state": "HEALTHY", "alloc_space": "62.5G", "total_space": "10.9T", "def_space": "7.26T", "rep_dev_size": "10.9T", "read_errors": "0", "write_errors": "0", "checksum_errors": "0", "vdevs": { "ca1eb824-c371-491d-ac13-37637e35c683": { "name": "ca1eb824-c371-491d-ac13-37637e35c683", "vdev_type": "disk", "guid": "12841765308123764671", "path": "/dev/disk/by-partuuid/ca1eb824-c371-491d-ac13-37637e35c683", "state": "HEALTHY", "rep_dev_size": "3.64T", "phys_space": "3.64T", "read_errors": "0", "write_errors": "0", "checksum_errors": "0", "vendor": "ATA", "model": "WDC WD40EFZX-68AWUN0", "size": "3.6T" }, "97cd98fb-8fb8-4ac4-bc84-bd8950a7ace7": { "name": "97cd98fb-8fb8-4ac4-bc84-bd8950a7ace7", "vdev_type": "disk", "guid": "1527839927278881561", "path": "/dev/disk/by-partuuid/97cd98fb-8fb8-4ac4-bc84-bd8950a7ace7", "state": "HEALTHY", "rep_dev_size": "3.64T", "phys_space": "3.64T", "read_errors": "0", "write_errors": "0", "checksum_errors": "0", "vendor": "ATA", "model": "WDC WD40EFZX-68AWUN0", "size": "3.6T" }, "e9ddba5f-f948-4734-a472-cb8aa5f0ff65": { "name": "e9ddba5f-f948-4734-a472-cb8aa5f0ff65", "vdev_type": "disk", "guid": "6982750226085199860", "path": "/dev/disk/by-partuuid/e9ddba5f-f948-4734-a472-cb8aa5f0ff65", "state": "HEALTHY", "rep_dev_size": "3.64T", "phys_space": "3.64T", "read_errors": "0", "write_errors": "0", "checksum_errors": "0", "vendor": "ATA", "model": "WDC WD40EFZX-68AWUN0", "size": "3.6T" } } } } } }, "dedup": { "mirror-2": { "name": "mirror-2", "vdev_type": "mirror", "guid": "2227766268377771003", "state": "HEALTHY", "alloc_space": "89.1M", "total_space": "3.62T", "def_space": "3.62T", "rep_dev_size": "3.62T", "read_errors": "0", "write_errors": "0", "checksum_errors": "0", "vdevs": { "db017360-d8e9-4163-961b-144ca75293a3": { "name": "db017360-d8e9-4163-961b-144ca75293a3", "vdev_type": "disk", "guid": "17880913061695450307", "path": "/dev/disk/by-partuuid/db017360-d8e9-4163-961b-144ca75293a3", "state": "HEALTHY", "rep_dev_size": "3.63T", "phys_space": "3.64T", "read_errors": "0", "write_errors": "0", "checksum_errors": "0", "vendor": "ATA", "model": "WDC WD40EFZX-68AWUN0", "size": "3.6T" }, "952c3baf-b08a-4a8c-b7fa-33a07af5fe6f": { "name": "952c3baf-b08a-4a8c-b7fa-33a07af5fe6f", "vdev_type": "disk", "guid": "10276374011610020557", "path": "/dev/disk/by-partuuid/952c3baf-b08a-4a8c-b7fa-33a07af5fe6f", "state": "HEALTHY", "rep_dev_size": "3.63T", "phys_space": "3.64T", "read_errors": "0", "write_errors": "0", "checksum_errors": "0", "vendor": "ATA", "model": "WDC WD40EFZX-68AWUN0", "size": "3.6T" } } } }, "special": { "25d418f8-92bd-4327-b59f-7ef5d5f50d81": { "name": "25d418f8-92bd-4327-b59f-7ef5d5f50d81", "vdev_type": "disk", "guid": "3935742873387713123", "path": "/dev/disk/by-partuuid/25d418f8-92bd-4327-b59f-7ef5d5f50d81", "state": "HEALTHY", "alloc_space": "37.4M", "total_space": "444G", "def_space": "444G", "rep_dev_size": "444G", "phys_space": "447G", "read_errors": "0", "write_errors": "0", "checksum_errors": "0", "vendor": "ATA", "model": "Micron_5300_MTFDDAK480TDS", "size": "447.1G" } }, "error_count": "0" } } } # SEE ALSO zpool-condense(8), zpool-events(8), zpool-history(8), zpool-iostat(8), zpool-list(8), zpool-resilver(8), zpool-scrub(8), zpool-wait(8) Debian - May 26, 2026 ZPOOL-SYNC(8) - System Manager's Manual # NAME **zpool-sync** - flush data to primary storage of ZFS storage pools # SYNOPSIS **zpool** **sync** \[*pool*]… # DESCRIPTION This command forces all in-core dirty data to be written to the primary pool storage and not the ZIL. It will also update administrative information including quota reporting. Without arguments, **zpool** **sync** will sync all pools on the system. Otherwise, it will sync only the specified pools. # SEE ALSO zpoolconcepts(7), zpool-export(8), zpool-iostat(8) Debian - July 11, 2022 ZPOOL-TRIM(8) - System Manager's Manual # NAME **zpool-trim** - initiate TRIM of free space in ZFS storage pool # SYNOPSIS **zpool** **trim** \[**-dw**] \[**-r** *rate*] \[**-c**|**-s**] **-a**|*pool* \[*device*]… # DESCRIPTION Initiates an immediate on-demand TRIM operation for all of the free space in a pool. This operation informs the underlying storage devices of all blocks in the pool which are no longer allocated and allows thinly provisioned devices to reclaim the space. A manual on-demand TRIM operation can be initiated irrespective of the **autotrim** pool property setting. See the documentation for the **autotrim** property above for the types of vdev devices which can be trimmed. **-a**, **--all** > Perform TRIM operation on > all > pools. **-d**, **--secure** > Causes a secure TRIM to be initiated. > When performing a secure TRIM, the > device guarantees that data stored on the trimmed blocks has been erased. > This requires support from the device and is not supported by all SSDs. **-r**, **--rate** *rate* > Controls the rate at which the TRIM operation progresses. > Without this > option TRIM is executed as quickly as possible. > The rate, expressed in bytes > per second, is applied on a per-vdev basis and may be set differently for > each leaf vdev. **-c**, **--cancel** > Cancel trimming on the specified devices, or all eligible devices if none > are specified. > If one or more target devices are invalid or are not currently being > trimmed, the command will fail and no cancellation will occur on any device. **-s**, **--suspend** > Suspend trimming on the specified devices, or all eligible devices if none > are specified. > If one or more target devices are invalid or are not currently being > trimmed, the command will fail and no suspension will occur on any device. > Trimming can then be resumed by running > **zpool** **trim** > with no flags on the relevant target devices. **-w**, **--wait** > Wait until the devices are done being trimmed before returning. # PERIODIC TRIM On machines using systemd, trim timers can be enabled on a per-pool basis. **weekly** and **monthly** timer units are provided. > **systemctl** > **enable** > **zfs-trim-*weekly*‌**@**‌*rpool*‌**.timer**‌** > **--now** > **systemctl** > **enable** > **zfs-trim-*monthly*‌**@**‌*otherpool*‌**.timer**‌** > **--now** # SEE ALSO systemd.timer(5), zpoolprops(7), zpool-initialize(8), zpool-wait(8) Debian - July 30, 2025 ZPOOL-UPGRADE(8) - System Manager's Manual # NAME **zpool-upgrade** - manage version and feature flags of ZFS storage pools # SYNOPSIS **zpool** **upgrade** **zpool** **upgrade** **-v** **zpool** **upgrade** \[**-V** *version*] **-a**|*pool*… # DESCRIPTION **zpool** **upgrade** > Displays pools which do not have all supported features enabled and pools > formatted using a legacy ZFS version number. > These pools can continue to be used, but some features may not be available. > Use > **zpool** **upgrade** **-a** > to enable all features on all pools (subject to the > **-o** **compatibility** > property). **zpool** **upgrade** **-v** > Displays legacy ZFS versions supported by this version of ZFS. > See > zpool-features(7) > for a description of features supported by this version of ZFS. **zpool** **upgrade** \[**-V** *version*] **-a**|*pool*… > Enables all supported features on the given pool. > If the pool has specified compatibility feature sets using the > **-o** **compatibility** > property, only the features present in all requested compatibility sets will be > enabled. > If this property is set to > *legacy* > then no upgrade will take place. > Once this is done, the pool will no longer be accessible on systems that do not > support feature flags. > See > zpool-features(7) > for details on compatibility with systems that support feature flags, but do not > support all features enabled on the pool. > **-a** > > Enables all supported features (from specified compatibility sets, if any) on > > all > > pools. > **-V** *version* > > Upgrade to the specified legacy version. > > If specified, no features will be enabled on the pool. > > This option can only be used to increase the version number up to the last > > supported legacy version number. # EXAMPLES ## Example 1: Upgrading All ZFS Storage Pools to the Current Version The following command upgrades all ZFS Storage pools to the current version of the software: # **zpool** **upgrade** **-a** This system is currently running ZFS version 2. # SEE ALSO zpool-features(7), zpoolconcepts(7), zpoolprops(7), zpool-history(8) Debian - July 11, 2022 ZPOOL-WAIT(8) - System Manager's Manual # NAME **zpool-wait** - wait for activity to stop in a ZFS storage pool # SYNOPSIS **zpool** **wait** \[**-Hp**] \[**-T** **u**|**d**] \[**-t** *activity*\[,*activity*]…] *pool* \[*interval*] # DESCRIPTION Waits until all background activity of the given types has ceased in the given pool. The activity could cease because it has completed, or because it has been paused or canceled by a user, or because the pool has been exported or destroyed. If no activities are specified, the command waits until background activity of every type listed below has ceased. If there is no activity of the given types in progress, the command returns immediately. These are the possible values for *activity*, along with what each one waits for: **discard** > Checkpoint to be discarded **free** > **freeing** > property to become > **0** **initialize** > All initializations to cease **replace** > All device replacements to cease **remove** > Device removal to cease **resilver** > Resilver to cease **scrub** > Scrub to cease **trim** > Manual trim to cease **raidz\_expand** > Attaching to a RAID-Z vdev to complete **condense** > Metadata condense operations to complete If an *interval* is provided, the amount of work remaining, in bytes, for each activity is printed every *interval* seconds. **-H** > Scripted mode. > Do not display headers, and separate fields by a single tab instead of arbitrary > space. **-p** > Display numbers in parsable (exact) values. **-T** **u**|**d** > Display a time stamp. > Specify > **u** > for a printed representation of the internal representation of time. > See > time(1). > Specify > **d** > for standard date format. > See > date(1). # SEE ALSO zpool-checkpoint(8), zpool-condense(8), zpool-initialize(8), zpool-remove(8), zpool-replace(8), zpool-resilver(8), zpool-scrub(8), zpool-status(8), zpool-trim(8) Debian - May 26, 2026 ZPOOL(8) - System Manager's Manual # NAME **zpool** - configure ZFS storage pools # SYNOPSIS **zpool** **-?V** **zpool** **version** \[**-j**] **zpool** **subcommand** \[*arguments*] # DESCRIPTION The **zpool** command configures ZFS storage pools. A storage pool is a collection of devices that provides physical storage and data replication for ZFS datasets. All datasets within a storage pool share the same space. See zfs(8) for information on managing datasets. For an overview of creating and managing ZFS storage pools see the zpoolconcepts(7) manual page. # SUBCOMMANDS All subcommands that modify state are logged persistently to the pool in their original form. The **zpool** command provides subcommands to create and destroy storage pools, add capacity to storage pools, and provide information about the storage pools. The following subcommands are supported: **zpool** **-?** > Displays a help message. **zpool** **-V**, **--version** **zpool** **version** \[**-j**] > Displays the software version of the > **zpool** > userland utility and the ZFS kernel module. > Use > **-j** > option to output in JSON format. ## Creation zpool-create(8) > Creates a new storage pool containing the virtual devices specified on the > command line. zpool-initialize(8) > Begins initializing by writing to all unallocated regions on the specified > devices, or all eligible devices in the pool if no individual devices are > specified. ## Destruction zpool-destroy(8) > Destroys the given pool, freeing up any devices for other use. zpool-labelclear(8) > Removes ZFS label information from the specified > *device*, > making all data inaccessible without specialized recovery tools. > Cannot be undone with ZFS tooling. ## Virtual Devices zpool-attach(8)/zpool-detach(8) > Converts a non-redundant disk into a mirror, or increases > the redundancy level of an existing mirror > (**attach**), or performs the inverse operation (**detach**). zpool-add(8)/zpool-remove(8) > Adds the specified virtual devices to the given pool, > or removes the specified device from the pool. zpool-replace(8) > Replaces an existing device (which may be faulted) with a new one. zpool-split(8) > Creates a new pool by splitting all mirrors in an existing pool (which decreases > its redundancy). ## Properties Available pool properties listed in the zpoolprops(7) manual page. zpool-list(8) > Lists the given pools along with a health status and space usage. zpool-get(8)/zpool-set(8) > Retrieves the given list of properties > (or all properties if > **all** > is used) > for the specified storage pool(s). ## Monitoring zpool-status(8) > Displays the detailed health status for the given pools. zpool-iostat(8) > Displays logical I/O statistics for the given pools/vdevs. > Physical I/O operations may be observed via > iostat(1). zpool-events(8) > Lists all recent events generated by the ZFS kernel modules. > These events are consumed by the > zed(8) > and used to automate administrative tasks such as replacing a failed device > with a hot spare. > That manual page also describes the subclasses and event payloads > that can be generated. zpool-history(8) > Displays the command history of the specified pool(s) or all pools if no pool is > specified. ## Maintenance zpool-prefetch(8) > Prefetches specific types of pool data. zpool-scrub(8) > Begins a scrub or resumes a paused scrub. zpool-condense(8) > Condense, flush, garbage collect or otherwise clean up pool metadata. zpool-checkpoint(8) > Checkpoints the current state of > *pool*, > which can be later restored by > **zpool** **import** **--rewind-to-checkpoint**. zpool-trim(8) > Initiates an immediate on-demand TRIM operation for all of the free space in a > pool. > This operation informs the underlying storage devices of all blocks > in the pool which are no longer allocated and allows thinly provisioned > devices to reclaim the space. zpool-sync(8) > This command forces all in-core dirty data to be written to the primary > pool storage and not the ZIL. > It will also update administrative information including quota reporting. > Without arguments, > **zpool** **sync** > will sync all pools on the system. > Otherwise, it will sync only the specified pool(s). zpool-upgrade(8) > Manage the on-disk format version of storage pools. zpool-wait(8) > Waits until all background activity of the given types has ceased in the given > pool. ## Fault Resolution zpool-offline(8)/zpool-online(8) > Takes the specified physical device offline or brings it online. zpool-resilver(8) > Starts a resilver. > If an existing resilver is already running it will be restarted from the > beginning. zpool-reopen(8) > Reopen all the vdevs associated with the pool. zpool-clear(8) > Clears device errors in a pool. ## Import & Export zpool-import(8) > Make disks containing ZFS storage pools available for use on the system. zpool-export(8) > Exports the given pools from the system. zpool-reguid(8) > Generates a new unique identifier for the pool. # EXIT STATUS The following exit values are returned: **0** > Successful completion. **1** > An error occurred. **2** > Invalid command line options were specified. # EXAMPLES ## Example 1: Creating a RAID-Z Storage Pool The following command creates a pool with a single raidz root vdev that consists of six disks: # **zpool** **create** *tank* **raidz** *sda sdb sdc sdd sde sdf* ## Example 2: Creating a Mirrored Storage Pool The following command creates a pool with two mirrors, where each mirror contains two disks: # **zpool** **create** *tank* **mirror** *sda sdb* **mirror** *sdc sdd* ## Example 3: Creating a ZFS Storage Pool by Using Partitions The following command creates a non-redundant pool using two disk partitions: # **zpool** **create** *tank* *sda1 sdb2* ## Example 4: Creating a ZFS Storage Pool by Using Files The following command creates a non-redundant pool using files. While not recommended, a pool based on files can be useful for experimental purposes. # **zpool** **create** *tank* */path/to/file/a /path/to/file/b* ## Example 5: Making a non-mirrored ZFS Storage Pool mirrored The following command converts an existing single device *sda* into a mirror by attaching a second device to it, *sdb*. # **zpool** **attach** *tank* *sda sdb* ## Example 6: Adding a Mirror to a ZFS Storage Pool The following command adds two mirrored disks to the pool *tank*, assuming the pool is already made up of two-way mirrors. The additional space is immediately available to any datasets within the pool. # **zpool** **add** *tank* **mirror** *sda sdb* ## Example 7: Listing Available ZFS Storage Pools The following command lists all available pools on the system. In this case, the pool *zion* is faulted due to a missing device. The results from this command are similar to the following: # **zpool** **list** NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT rpool 19.9G 8.43G 11.4G - 33% 42% 1.00x ONLINE - tank 61.5G 20.0G 41.5G - 48% 32% 1.00x ONLINE - zion - - - - - - - FAULTED - ## Example 8: Destroying a ZFS Storage Pool The following command destroys the pool *tank* and any datasets contained within: # **zpool** **destroy** **-f** *tank* ## Example 9: Exporting a ZFS Storage Pool The following command exports the devices in pool *tank* so that they can be relocated or later imported: # **zpool** **export** *tank* ## Example 10: Importing a ZFS Storage Pool The following command displays available pools, and then imports the pool *tank* for use on the system. The results from this command are similar to the following: # **zpool** **import** pool: tank id: 15451357997522795478 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: tank ONLINE mirror ONLINE sda ONLINE sdb ONLINE # **zpool** **import** *tank* ## Example 11: Upgrading All ZFS Storage Pools to the Current Version The following command upgrades all ZFS Storage pools to the current version of the software: # **zpool** **upgrade** **-a** This system is currently running ZFS version 2. ## Example 12: Managing Hot Spares The following command creates a new pool with an available hot spare: # **zpool** **create** *tank* **mirror** *sda sdb* **spare** *sdc* If one of the disks were to fail, the pool would be reduced to the degraded state. The failed device can be replaced using the following command: # **zpool** **replace** *tank* *sda sdd* Once the data has been resilvered, the spare is automatically removed and is made available for use should another device fail. The hot spare can be permanently removed from the pool using the following command: # **zpool** **remove** *tank* *sdc* ## Example 13: Creating a ZFS Pool with Mirrored Separate Intent Logs The following command creates a ZFS storage pool consisting of two, two-way mirrors and mirrored log devices: # **zpool** **create** *pool* **mirror** *sda sdb* **mirror** *sdc sdd* **log mirror** *sde sdf* ## Example 14: Adding Cache Devices to a ZFS Pool The following command adds two disks for use as cache devices to a ZFS storage pool: # **zpool** **add** *pool* **cache** *sdc sdd* Once added, the cache devices gradually fill with content from main memory. Depending on the size of your cache devices, it could take over an hour for them to fill. Capacity and reads can be monitored using the **iostat** subcommand as follows: # **zpool** **iostat** **-v** *pool 5* ## Example 15: Removing a Mirrored top-level (Log or Data) Device The following commands remove the mirrored log device **mirror-2** and mirrored top-level data device **mirror-1**. Given this configuration: pool: tank state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 sda ONLINE 0 0 0 sdb ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 sdc ONLINE 0 0 0 sdd ONLINE 0 0 0 logs mirror-2 ONLINE 0 0 0 sde ONLINE 0 0 0 sdf ONLINE 0 0 0 The command to remove the mirrored log *mirror-2* is: # **zpool** **remove** *tank mirror-2* At this point, the log device no longer exists (both sides of the mirror have been removed): pool: tank state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 sda ONLINE 0 0 0 sdb ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 sdc ONLINE 0 0 0 sdd ONLINE 0 0 0 The command to remove the mirrored data *mirror-1* is: # **zpool** **remove** *tank mirror-1* After *mirror-1* has been evacuated, the pool remains redundant, but the total amount of space is reduced: pool: tank state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 sda ONLINE 0 0 0 sdb ONLINE 0 0 0 ## Example 16: Displaying expanded space on a device The following command displays the detailed information for the pool *data*. This pool is comprised of a single raidz vdev where one of its devices increased its capacity by 10 GiB. In this example, the pool will not be able to utilize this extra capacity until all the devices under the raidz vdev have been expanded. # **zpool** **list** **-v** *data* NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT data 23.9G 14.6G 9.30G - 48% 61% 1.00x ONLINE - raidz1 23.9G 14.6G 9.30G - 48% sda - - - - - sdb - - - 10G - sdc - - - - - ## Example 17: Adding output columns Additional columns can be added to the **zpool** **status** and **zpool** **iostat** output with **-c**. # **zpool** **status** **-c** *vendor*,*model*,*size* NAME STATE READ WRITE CKSUM vendor model size tank ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 U1 ONLINE 0 0 0 SEAGATE ST8000NM0075 7.3T U10 ONLINE 0 0 0 SEAGATE ST8000NM0075 7.3T U11 ONLINE 0 0 0 SEAGATE ST8000NM0075 7.3T U12 ONLINE 0 0 0 SEAGATE ST8000NM0075 7.3T U13 ONLINE 0 0 0 SEAGATE ST8000NM0075 7.3T U14 ONLINE 0 0 0 SEAGATE ST8000NM0075 7.3T # **zpool** **iostat** **-vc** *size* capacity operations bandwidth pool alloc free read write read write size ---------- ----- ----- ----- ----- ----- ----- ---- rpool 14.6G 54.9G 4 55 250K 2.69M sda1 14.6G 54.9G 4 55 250K 2.69M 70G ---------- ----- ----- ----- ----- ----- ----- ---- # ENVIRONMENT VARIABLES **ZFS\_ABORT** > Cause > **zpool** > to dump core on exit for the purposes of running > **::findleaks**. **ZFS\_COLOR** > Use ANSI color in > **zpool** **status** > and > **zpool** **iostat** > output. **ZPOOL\_AUTO\_POWER\_ON\_SLOT** > Automatically attempt to turn on the drives enclosure slot power to a drive when > running the > **zpool** **online** > or > **zpool** **clear** > commands. > This has the same effect as passing the > **--power** > option to those commands. **ZPOOL\_POWER\_ON\_SLOT\_TIMEOUT\_MS** > The maximum time in milliseconds to wait for a slot power sysfs value > to return the correct value after writing it. > For example, after writing "on" to the sysfs enclosure slot power\_control file, > it can take some time for the enclosure to power down the slot and return > "on" if you read back the 'power\_control' value. > Defaults to 30 seconds (30000ms) if not set. **ZPOOL\_IMPORT\_PATH** > The search path for devices or files to use with the pool. > This is a colon-separated list of directories in which > **zpool** > looks for device nodes and files. > Similar to the > **-d** > option in > **zpool import**. **ZPOOL\_IMPORT\_UDEV\_TIMEOUT\_MS** > The maximum time in milliseconds that > **zpool import** > will wait for an expected device to be available. **ZPOOL\_STATUS\_NON\_NATIVE\_ASHIFT\_IGNORE** > If set, suppress warning about non-native vdev ashift in > **zpool** **status**. > The value is not used, only the presence or absence of the variable matters. **ZPOOL\_VDEV\_NAME\_GUID** > Cause > **zpool** > subcommands to output vdev GUIDs by default. > This behavior is identical to the > **zpool** **status** **-g** > command line option. **ZPOOL\_VDEV\_NAME\_FOLLOW\_LINKS** > Cause > **zpool** > subcommands to follow links for vdev names by default. > This behavior is identical to the > **zpool** **status** **-L** > command line option. **ZPOOL\_VDEV\_NAME\_PATH** > Cause > **zpool** > subcommands to output full vdev path names by default. > This behavior is identical to the > **zpool** **status** **-P** > command line option. **ZFS\_VDEV\_DEVID\_OPT\_OUT** > Older OpenZFS implementations had issues when attempting to display pool > config vdev names if a > **devid** > NVP value is present in the pool's config. > For example, a pool that originated on illumos platform would have a > **devid** > value in the config and > **zpool** **status** > would fail when listing the config. > This would also be true for future Linux-based pools. > A pool can be stripped of any > **devid** > values on import or prevented from adding > them on > **zpool** **create** > or > **zpool** **add** > by setting > **ZFS\_VDEV\_DEVID\_OPT\_OUT**. **ZPOOL\_SCRIPTS\_AS\_ROOT** > Allow a privileged user to run > **zpool** **status**/**iostat** **-c**. > Normally, only unprivileged users are allowed to run > **-c**. **ZPOOL\_SCRIPTS\_PATH** > The search path for scripts when running > **zpool** **status**/**iostat** **-c**. > This is a colon-separated list of directories and overrides the default > *~/.zpool.d* > and > */etc/zfs/zpool.d* > search paths. **ZPOOL\_SCRIPTS\_ENABLED** > Allow a user to run > **zpool** **status**/**iostat** **-c**. > If > **ZPOOL\_SCRIPTS\_ENABLED** > is not set, it is assumed that the user is allowed to run > **zpool** **status**/**iostat** **-c**. **ZFS\_MODULE\_TIMEOUT** > Time, in seconds, to wait for > */dev/zfs* > to appear. > Defaults to > **10**, > max > **600** (10 minutes). > If > <**0**, > wait forever; if > **0**, > don't wait. # INTERFACE STABILITY **Evolving** # SEE ALSO zfs(4), zpool-features(7), zpoolconcepts(7), zpoolprops(7), zed(8), zfs(8), zpool-add(8), zpool-attach(8), zpool-checkpoint(8), zpool-clear(8), zpool-create(8), zpool-ddtprune(8), zpool-destroy(8), zpool-detach(8), zpool-events(8), zpool-export(8), zpool-get(8), zpool-history(8), zpool-import(8), zpool-initialize(8), zpool-iostat(8), zpool-labelclear(8), zpool-list(8), zpool-offline(8), zpool-online(8), zpool-prefetch(8), zpool-reguid(8), zpool-remove(8), zpool-reopen(8), zpool-replace(8), zpool-resilver(8), zpool-scrub(8), zpool-set(8), zpool-split(8), zpool-status(8), zpool-sync(8), zpool-trim(8), zpool-upgrade(8), zpool-wait(8) Debian - May 26, 2026 ZPOOL\_INFLUXDB(8) - System Manager's Manual # NAME **zpool\_influxdb** - collect ZFS pool statistics in InfluxDB line protocol format # SYNOPSIS **zpool\_influxdb** \[**-e**|**--execd**] \[**-n**|**--no-histogram**] \[**-s**|**--sum-histogram-buckets**] \[**-t**|**--tags** *key*=*value*\[,*key*=*value*]…] \[*pool*] # DESCRIPTION **zpool\_influxdb** produces InfluxDB-line-protocol-compatible metrics from zpools. Like the **zpool** command, **zpool\_influxdb** reads the current pool status and statistics. Unlike the **zpool** command which is intended for humans, **zpool\_influxdb** formats the output in the InfluxDB line protocol. The expected use is as a plugin to a metrics collector or aggregator, such as Telegraf. By default, **zpool\_influxdb** prints pool metrics and status in the InfluxDB line protocol format. All pools are printed, similar to the **zpool** **status** command. Providing a pool name restricts the output to the named pool. # OPTIONS **-e**, **--execd** > Run in daemon mode compatible with Telegraf's > **execd** > plugin. > In this mode, the pools are sampled every time a > newline appears on the standard input. **-n**, **--no-histogram** > Do not print latency and I/O size histograms. > This can reduce the total > amount of data, but one should consider the value brought by the insights > that latency and I/O size distributions provide. > The resulting values > are suitable for graphing with Grafana's heatmap plugin. **-s**, **--sum-histogram-buckets** > Accumulates bucket values. > By default, the values are not accumulated and the raw data appears as shown by > **zpool** **iostat**. > This works well for Grafana's heatmap plugin. > Summing the buckets produces output similar to Prometheus histograms. **-t**, **--tags** *key*=*value*\[,*key*=*value*]… > Adds specified tags to the tag set. > No sanity checking is performed. > See the InfluxDB Line Protocol format documentation for details on escaping > special characters used in tags. **-h**, **--help** > Print a usage summary. # SEE ALSO zpool-iostat(8), zpool-status(8), [InfluxDB](https://github.com/influxdata/influxdb), [Telegraf](https://github.com/influxdata/telegraf), [Grafana](https://grafana.com), [Prometheus](https://prometheus.io) Debian - May 26, 2021 ZSTREAM(8) - System Manager's Manual # NAME **zstream** - manipulate ZFS send streams # SYNOPSIS **zstream** **dump** \[**-Cvd**] \[*file*] **zstream** **decompress** \[**-v**] \[*object*‌**,**‌*offset*\[**,**‌*type*...]] **zstream** **drop\_record** \[**-v**] \[*object*‌**,**‌*offset*\[**‌**...]] **zstream** **redup** \[**-v**] *file* **zstream** **token** *resume\_token* **zstream** **raw** \[**-v**] \[**-b** *maxbufs*] \[**-g** *fromguid*] *image|device* **zstream** **recompress** \[**-t** *num\_threads*] \[**-l** *level*] *algorithm* # DESCRIPTION The **zstream** utility manipulates ZFS send streams output by the **zfs send** command. **zstream** **dump** \[**-Cvd**] \[*file*] > Print information about the specified send stream, including headers and > record counts. > The send stream may either be in the specified > *file*, > or provided on standard input. > **-C** > > Suppress the validation of checksums. > **-v** > > Verbose. > > Print metadata for each record. > **-d** > > Dump data contained in each record. > > Implies verbose. > The > **zstreamdump** > alias is provided for compatibility and is equivalent to running > **zstream** > **dump**. **zstream** **token** *resume\_token* > Dumps zfs resume token information **zstream** **decompress** \[**-v**] \[*object*‌**,**‌*offset*\[**,**‌*type*...]] > Decompress selected records in a ZFS send stream provided on standard input, > when the compression type recorded in ZFS metadata may be incorrect. > Specify the object number and byte offset of each record that you wish to > decompress. > Optionally specify the compression type. > Valid compression types include > **off**, > **gzip**, > **lz4**, > **lzjb**, > **zstd**, > and > **zle**. > The default is > **lz4**. > Every record for that object beginning at that offset will be decompressed, if > possible. > It may not be possible, because the record may be corrupted in some but not > all of the stream's snapshots. > Specifying a compression type of > **off** > will change the stream's metadata accordingly, without attempting decompression. > This can be useful if the record is already uncompressed but the metadata > insists otherwise. > The repaired stream will be written to standard output. > **-v** > > Verbose. > > Print summary of decompressed records. **zstream** **drop\_record** \[**-v**] \[*object*‌**,**‌*offset*\[** ...**]] > Drop selected records from a ZFS send stream provided on standard input, > specified by object number and byte offset. > Only WRITE and WRITE\_EMBEDDED are records are supported, currently. > The repaired stream will be written to standard output. > **-v** > > Verbose. > > Print summary of dropped records. **zstream** **redup** \[**-v**] *file* > Deduplicated send streams can be generated by using the > **zfs** **send** **-D** > command. > The ability to send deduplicated send streams is deprecated. > In the future, the ability to receive a deduplicated send stream with > **zfs** **receive** > will be removed. > However, deduplicated send streams can still be received by utilizing > **zstream** **redup**. > The > **zstream** **redup** > command is provided a > *file* > containing a deduplicated send stream, and outputs an equivalent > non-deduplicated send stream on standard output. > Therefore, a deduplicated send stream can be received by running: > > \# **zstream** **redup** *DEDUP\_STREAM\_FILE* | **zfs** **receive** … > **-v** > > Verbose. > > Print summary of converted records. **zstream** **raw** \[**-v**] \[**-b** *maxbufs*] \[**-g** *fromguid*] *image|device* > Apply a zvol send stream to a raw image or block device. > By default, at most 32 > `SPA_MAXBLOCKSIZE` > buffers can be written at a time. > With > *maxbufs* > set to some value between 1 and > `IOV_MAX`, > contiguous write records will be combined into one write operation using up to > this many buffers, at the cost of additional memory use. > When > *fromguid* > is provided, the initial fromguid of the stream will be checked to ensure it > matches the given value. > The final toguid of the stream is printed to stdout on completion. > With > **-v**, > details of the records in the stream are printed in similar fashion to > **zstream** > **dump**. **zstream** **recompress** \[**-t** *num\_threads*] \[**-l** *level*] *algorithm* > Recompresses a send stream from standard input using the specified compression > *algorithm* > and optional > *level*, > and writes the modified stream to standard output. > All WRITE records in the send stream will be compressed unless compression > fails to reduce their size compared to being left uncompressed. > The > *algorithm* > can be any valid value to the > **compress** > property. > Note that encrypted send streams cannot be recompressed. > **-t** *num\_threads* > > Specifies the number of compression worker threads. > > By default, > > **zstream** **recompress** > > creates a thread for every CPU core. > > It's generally unproductive to raise > > *num\_threads* > > above this default. > > On some architectures, contention for resources such as memory caches can > > reduce efficiency when many threads are active. > > On those systems, it may be helpful to run with fewer threads, > > particularly when automating operations on shared servers. > **-l** *level* > > Specifies compression level. > > Only needed for algorithms where the level is not implied as part of the name > > of the algorithm (e.g. gzip-3 does not require it, while zstd does, if a > > non-default level is desired). # EXAMPLES ## Recovering from OpenZFS bug #12762 First, determine which records are corrupt. That cannot be done automatically; it requires information beyond ZFS's metadata. If object **128** is corrupted at offset **0** and is compressed using **lz4**, then run this command: # **zfs** *send* **-c** ** | **zstream decompress** *128,0,lz4* | **zfs recv** ** ## Recovering from OpenZFS bug #18239 The bogus records typically have an absurdly large offset, and can be seen with a command like **zdb** **-ddddd** *dataset* *object* or **zstream** *dump* **-v**. To recover, send the dataset and use **zstream** to drop the bogus record, then receive into a new dataset. # **zfs** *send* *...* | **zstream drop_record** *3545761,18446744073709486080* | **zfs recv** *...* ## Migrating from a zvol to a physical disk Given a zvol tank/vol with snapshots @1 @2 and @3, and an adequately sized physical disk /dev/sdd: # **zfs** **send** **-Lec** *tank* | **zstream** **raw** */dev/sdd* 16731506615198184313 # **zfs** **send** **-Lec** **-I** *@1* *tank/vol@3* | **zstream** **raw** **-g** *16731506615198184313* */dev/sdd* 10690368765373298656 # SEE ALSO zdb(8), zfs(8), zfs-receive(8), zfs-send(8), [https://github.com/openzfs/zfs/issues/12762](https://github.com/openzfs/zfs/issues/12762) [https://github.com/openzfs/zfs/issues/18239](https://github.com/openzfs/zfs/issues/18239) Debian - May 11, 2026 ZSTREAM(8) - System Manager's Manual # NAME **zstream** - manipulate ZFS send streams # SYNOPSIS **zstream** **dump** \[**-Cvd**] \[*file*] **zstream** **decompress** \[**-v**] \[*object*‌**,**‌*offset*\[**,**‌*type*...]] **zstream** **drop\_record** \[**-v**] \[*object*‌**,**‌*offset*\[**‌**...]] **zstream** **redup** \[**-v**] *file* **zstream** **token** *resume\_token* **zstream** **raw** \[**-v**] \[**-b** *maxbufs*] \[**-g** *fromguid*] *image|device* **zstream** **recompress** \[**-t** *num\_threads*] \[**-l** *level*] *algorithm* # DESCRIPTION The **zstream** utility manipulates ZFS send streams output by the **zfs send** command. **zstream** **dump** \[**-Cvd**] \[*file*] > Print information about the specified send stream, including headers and > record counts. > The send stream may either be in the specified > *file*, > or provided on standard input. > **-C** > > Suppress the validation of checksums. > **-v** > > Verbose. > > Print metadata for each record. > **-d** > > Dump data contained in each record. > > Implies verbose. > The > **zstreamdump** > alias is provided for compatibility and is equivalent to running > **zstream** > **dump**. **zstream** **token** *resume\_token* > Dumps zfs resume token information **zstream** **decompress** \[**-v**] \[*object*‌**,**‌*offset*\[**,**‌*type*...]] > Decompress selected records in a ZFS send stream provided on standard input, > when the compression type recorded in ZFS metadata may be incorrect. > Specify the object number and byte offset of each record that you wish to > decompress. > Optionally specify the compression type. > Valid compression types include > **off**, > **gzip**, > **lz4**, > **lzjb**, > **zstd**, > and > **zle**. > The default is > **lz4**. > Every record for that object beginning at that offset will be decompressed, if > possible. > It may not be possible, because the record may be corrupted in some but not > all of the stream's snapshots. > Specifying a compression type of > **off** > will change the stream's metadata accordingly, without attempting decompression. > This can be useful if the record is already uncompressed but the metadata > insists otherwise. > The repaired stream will be written to standard output. > **-v** > > Verbose. > > Print summary of decompressed records. **zstream** **drop\_record** \[**-v**] \[*object*‌**,**‌*offset*\[** ...**]] > Drop selected records from a ZFS send stream provided on standard input, > specified by object number and byte offset. > Only WRITE and WRITE\_EMBEDDED are records are supported, currently. > The repaired stream will be written to standard output. > **-v** > > Verbose. > > Print summary of dropped records. **zstream** **redup** \[**-v**] *file* > Deduplicated send streams can be generated by using the > **zfs** **send** **-D** > command. > The ability to send deduplicated send streams is deprecated. > In the future, the ability to receive a deduplicated send stream with > **zfs** **receive** > will be removed. > However, deduplicated send streams can still be received by utilizing > **zstream** **redup**. > The > **zstream** **redup** > command is provided a > *file* > containing a deduplicated send stream, and outputs an equivalent > non-deduplicated send stream on standard output. > Therefore, a deduplicated send stream can be received by running: > > \# **zstream** **redup** *DEDUP\_STREAM\_FILE* | **zfs** **receive** … > **-v** > > Verbose. > > Print summary of converted records. **zstream** **raw** \[**-v**] \[**-b** *maxbufs*] \[**-g** *fromguid*] *image|device* > Apply a zvol send stream to a raw image or block device. > By default, at most 32 > `SPA_MAXBLOCKSIZE` > buffers can be written at a time. > With > *maxbufs* > set to some value between 1 and > `IOV_MAX`, > contiguous write records will be combined into one write operation using up to > this many buffers, at the cost of additional memory use. > When > *fromguid* > is provided, the initial fromguid of the stream will be checked to ensure it > matches the given value. > The final toguid of the stream is printed to stdout on completion. > With > **-v**, > details of the records in the stream are printed in similar fashion to > **zstream** > **dump**. **zstream** **recompress** \[**-t** *num\_threads*] \[**-l** *level*] *algorithm* > Recompresses a send stream from standard input using the specified compression > *algorithm* > and optional > *level*, > and writes the modified stream to standard output. > All WRITE records in the send stream will be compressed unless compression > fails to reduce their size compared to being left uncompressed. > The > *algorithm* > can be any valid value to the > **compress** > property. > Note that encrypted send streams cannot be recompressed. > **-t** *num\_threads* > > Specifies the number of compression worker threads. > > By default, > > **zstream** **recompress** > > creates a thread for every CPU core. > > It's generally unproductive to raise > > *num\_threads* > > above this default. > > On some architectures, contention for resources such as memory caches can > > reduce efficiency when many threads are active. > > On those systems, it may be helpful to run with fewer threads, > > particularly when automating operations on shared servers. > **-l** *level* > > Specifies compression level. > > Only needed for algorithms where the level is not implied as part of the name > > of the algorithm (e.g. gzip-3 does not require it, while zstd does, if a > > non-default level is desired). # EXAMPLES ## Recovering from OpenZFS bug #12762 First, determine which records are corrupt. That cannot be done automatically; it requires information beyond ZFS's metadata. If object **128** is corrupted at offset **0** and is compressed using **lz4**, then run this command: # **zfs** *send* **-c** ** | **zstream decompress** *128,0,lz4* | **zfs recv** ** ## Recovering from OpenZFS bug #18239 The bogus records typically have an absurdly large offset, and can be seen with a command like **zdb** **-ddddd** *dataset* *object* or **zstream** *dump* **-v**. To recover, send the dataset and use **zstream** to drop the bogus record, then receive into a new dataset. # **zfs** *send* *...* | **zstream drop_record** *3545761,18446744073709486080* | **zfs recv** *...* ## Migrating from a zvol to a physical disk Given a zvol tank/vol with snapshots @1 @2 and @3, and an adequately sized physical disk /dev/sdd: # **zfs** **send** **-Lec** *tank* | **zstream** **raw** */dev/sdd* 16731506615198184313 # **zfs** **send** **-Lec** **-I** *@1* *tank/vol@3* | **zstream** **raw** **-g** *16731506615198184313* */dev/sdd* 10690368765373298656 # SEE ALSO zdb(8), zfs(8), zfs-receive(8), zfs-send(8), [https://github.com/openzfs/zfs/issues/12762](https://github.com/openzfs/zfs/issues/12762) [https://github.com/openzfs/zfs/issues/18239](https://github.com/openzfs/zfs/issues/18239) Debian - May 11, 2026 ZFS Messages ============ .. toctree:: :maxdepth: 2 :caption: Contents: :glob: ZFS-*/index .. CDDL HEADER START The contents of this file are subject to the terms of the Common Development and Distribution License (the "License"). You may not use this file except in compliance with the License. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE or http://www.opensolaris.org/os/licensing. See the License for the specific language governing permissions and limitations under the License. When distributing Covered Code, include this CDDL HEADER in each file and include the License file at usr/src/OPENSOLARIS.LICENSE. If applicable, add the following below this CDDL HEADER, with the fields enclosed by brackets "[]" replaced with your own identifying information: Portions Copyright [yyyy] [name of copyright owner] CDDL HEADER END Portions Copyright 2007 Sun Microsystems, Inc. .. highlight:: none Message ID: ZFS-8000-14 ======================= Corrupt ZFS cache ----------------- +-------------------------+--------------------------------------+ | **Type:** | Error | +-------------------------+--------------------------------------+ | **Severity:** | Critical | +-------------------------+--------------------------------------+ | **Description:** | The ZFS cache file is corrupted. | +-------------------------+--------------------------------------+ | **Automated Response:** | No automated response will be taken. | +-------------------------+--------------------------------------+ | **Impact:** | ZFS filesystems are not available. | +-------------------------+--------------------------------------+ .. rubric:: Suggested Action for System Administrator ZFS keeps a list of active pools on the filesystem to avoid having to scan all devices when the system is booted. If this file is corrupted, then normally active pools will not be automatically opened. The pools can be recovered using the ``zpool import`` command: :: # zpool import pool: test id: 12743384782310107047 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: test ONLINE sda9 ONLINE This will automatically scan ``/dev`` for any devices part of a pool. If devices have been made available in an alternate location, use the ``-d`` option to ``zpool import`` to search for devices in a different directory. Once you have determined which pools are available for import, you can import the pool explicitly by specifying the name or numeric identifier: :: # zpool import test Alternately, you can import all available pools by specifying the ``-a`` option. Once a pool has been imported, the ZFS cache will be repaired so that the pool will appear normally in the future. .. rubric:: Details The Message ID: ``ZFS-8000-14`` indicates a corrupted ZFS cache file. Take the documented action to resolve the problem. .. CDDL HEADER START The contents of this file are subject to the terms of the Common Development and Distribution License (the "License"). You may not use this file except in compliance with the License. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE or http://www.opensolaris.org/os/licensing. See the License for the specific language governing permissions and limitations under the License. When distributing Covered Code, include this CDDL HEADER in each file and include the License file at usr/src/OPENSOLARIS.LICENSE. If applicable, add the following below this CDDL HEADER, with the fields enclosed by brackets "[]" replaced with your own identifying information: Portions Copyright [yyyy] [name of copyright owner] CDDL HEADER END Portions Copyright 2007 Sun Microsystems, Inc. .. highlight:: none Message ID: ZFS-8000-2Q ======================= Missing device in replicated configuration ------------------------------------------ +-------------------------+--------------------------------------------------+ | **Type:** | Error | +-------------------------+--------------------------------------------------+ | **Severity:** | Major | +-------------------------+--------------------------------------------------+ | **Description:** | A device in a replicated configuration could not | | | be opened. | +-------------------------+--------------------------------------------------+ | **Automated Response:** | A hot spare will be activated if available. | +-------------------------+--------------------------------------------------+ | **Impact:** | The pool is no longer providing the configured | | | level of replication. | +-------------------------+--------------------------------------------------+ .. rubric:: Suggested Action for System Administrator .. rubric:: For an active pool: If this error was encountered while running ``zpool import``, please see the section below. Otherwise, run ``zpool status -x`` to determine which pool has experienced a failure: :: # zpool status -x pool: test state: DEGRADED status: One or more devices could not be opened. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Attach the missing device and online it using 'zpool online'. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-2Q scrub: none requested config: NAME STATE READ WRITE CKSUM test DEGRADED 0 0 0 mirror DEGRADED 0 0 0 sda ONLINE 0 0 0 sdb FAULTED 0 0 0 cannot open errors: No known data errors Determine which device failed to open by looking for a FAULTED device with an additional 'cannot open' message. If this device has been inadvertently removed from the system, attach the device and bring it online with ``zpool online``: :: # zpool online test sdb If the device is no longer available, the device can be replaced using the ``zpool replace`` command: :: # zpool replace test sdb sdc If the device has been replaced by another disk in the same physical slot, then the device can be replaced using a single argument to the ``zpool replace`` command: :: # zpool replace test sdb Existing data will be resilvered to the new device. Once the resilvering completes, the device will be removed from the pool. .. rubric:: For an exported pool: If this error is encountered during a ``zpool import``, it means that one of the devices is not attached to the system: :: # zpool import pool: test id: 10121266328238932306 state: DEGRADED status: One or more devices are missing from the system. action: The pool can be imported despite missing or damaged devices. The fault tolerance of the pool may be compromised if imported. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-2Q config: test DEGRADED mirror DEGRADED sda ONLINE sdb FAULTED cannot open Unlike when the pool is active on the system, the device cannot be replaced while the pool is exported. If the device can be attached to the system, attach the device and run ``zpool import`` again. Alternatively, the pool can be imported as-is, though it will be placed in the DEGRADED state due to a missing device. The device will be marked as UNAVAIL. Once the pool has been imported, the missing device can be replaced as described above. .. rubric:: Details The Message ID: ``ZFS-8000-2Q`` indicates a device which was unable to be opened by the ZFS subsystem. .. CDDL HEADER START The contents of this file are subject to the terms of the Common Development and Distribution License (the "License"). You may not use this file except in compliance with the License. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE or http://www.opensolaris.org/os/licensing. See the License for the specific language governing permissions and limitations under the License. When distributing Covered Code, include this CDDL HEADER in each file and include the License file at usr/src/OPENSOLARIS.LICENSE. If applicable, add the following below this CDDL HEADER, with the fields enclosed by brackets "[]" replaced with your own identifying information: Portions Copyright [yyyy] [name of copyright owner] CDDL HEADER END Portions Copyright 2007 Sun Microsystems, Inc. .. highlight:: none Message ID: ZFS-8000-3C ======================= Missing device in non-replicated configuration ---------------------------------------------- +-------------------------+--------------------------------------------------+ | **Type:** | Error | +-------------------------+--------------------------------------------------+ | **Severity:** | Critical | +-------------------------+--------------------------------------------------+ | **Description:** | A device could not be opened and no replicas are | | | available. | +-------------------------+--------------------------------------------------+ | **Automated Response:** | No automated response will be taken. | +-------------------------+--------------------------------------------------+ | **Impact:** | The pool is no longer available. | +-------------------------+--------------------------------------------------+ .. rubric:: Suggested Action for System Administrator .. rubric:: For an active pool: If this error was encountered while running ``zpool import``, please see the section below. Otherwise, run ``zpool status -x`` to determine which pool has experienced a failure: :: # zpool status -x pool: test state: FAULTED status: One or more devices could not be opened. There are insufficient replicas for the pool to continue functioning. action: Attach the missing device and online it using 'zpool online'. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-3C scrub: none requested config: NAME STATE READ WRITE CKSUM test FAULTED 0 0 0 insufficient replicas sda ONLINE 0 0 0 sdb FAULTED 0 0 0 cannot open errors: No known data errors If the device has been temporarily detached from the system, attach the device to the system and run ``zpool status`` again. The pool should automatically detect the newly attached device and resume functioning. You may have to mount the filesystems in the pool explicitly using ``zfs mount -a``. If the device is no longer available and cannot be reattached to the system, then the pool must be destroyed and re-created from a backup source. .. rubric:: For an exported pool: If this error is encountered during a ``zpool import``, it means that one of the devices is not attached to the system: :: # zpool import pool: test id: 10121266328238932306 state: FAULTED status: One or more devices are missing from the system. action: The pool cannot be imported. Attach the missing devices and try again. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-3C config: test FAULTED insufficient replicas sda ONLINE sdb FAULTED cannot open The pool cannot be imported until the missing device is attached to the system. If the device has been made available in an alternate location, use the ``-d`` option to ``zpool import`` to search for devices in a different directory. If the missing device is unavailable, then the pool cannot be imported. .. rubric:: Details The Message ID: ``ZFS-8000-3C`` indicates a device which was unable to be opened by the ZFS subsystem. .. CDDL HEADER START The contents of this file are subject to the terms of the Common Development and Distribution License (the "License"). You may not use this file except in compliance with the License. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE or http://www.opensolaris.org/os/licensing. See the License for the specific language governing permissions and limitations under the License. When distributing Covered Code, include this CDDL HEADER in each file and include the License file at usr/src/OPENSOLARIS.LICENSE. If applicable, add the following below this CDDL HEADER, with the fields enclosed by brackets "[]" replaced with your own identifying information: Portions Copyright [yyyy] [name of copyright owner] CDDL HEADER END Portions Copyright 2007 Sun Microsystems, Inc. .. highlight:: none Message ID: ZFS-8000-4J ======================= Corrupted device label in a replicated configuration ---------------------------------------------------- +-------------------------+--------------------------------------------------+ | **Type:** | Error | +-------------------------+--------------------------------------------------+ | **Severity:** | Major | +-------------------------+--------------------------------------------------+ | **Description:** | A device could not be opened due to a missing or | | | invalid device label. | +-------------------------+--------------------------------------------------+ | **Automated Response:** | A hot spare will be activated if available. | +-------------------------+--------------------------------------------------+ | **Impact:** | The pool is no longer providing the configured | | | level of replication. | +-------------------------+--------------------------------------------------+ .. rubric:: Suggested Action for System Administrator .. rubric:: For an active pool: If this error was encountered while running ``zpool import``, please see the section below. Otherwise, run ``zpool status -x`` to determine which pool has experienced a failure: :: # zpool status -x pool: test state: DEGRADED status: One or more devices could not be used because the label is missing or invalid. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the device using 'zpool replace'. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J scrub: none requested config: NAME STATE READ WRITE CKSUM test DEGRADED 0 0 0 mirror DEGRADED 0 0 0 sda ONLINE 0 0 0 sdb FAULTED 0 0 0 corrupted data errors: No known data errors If the device has been temporarily detached from the system, attach the device to the system and run ``zpool status`` again. The pool should automatically detect the newly attached device and resume functioning. If the device is no longer available, it can be replaced using ``zpool replace``: :: # zpool replace test sdb sdc If the device has been replaced by another disk in the same physical slot, then the device can be replaced using a single argument to the ``zpool replace`` command: :: # zpool replace test sdb ZFS will begin migrating data to the new device as soon as the replace is issued. Once the resilvering completes, the original device (if different from the replacement) will be removed, and the pool will be restored to the ONLINE state. .. rubric:: For an exported pool: If this error is encountered while running ``zpool import``, the pool can be still be imported despite the failure: :: # zpool import pool: test id: 5187963178597328409 state: DEGRADED status: One or more devices contains corrupted data. The fault tolerance of the pool may be compromised if imported. action: The pool can be imported using its name or numeric identifier. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J config: test DEGRADED mirror DEGRADED sda ONLINE sdb FAULTED corrupted data To import the pool, run ``zpool import``: :: # zpool import test Once the pool has been imported, the damaged device can be replaced according to the above procedure. .. rubric:: Details The Message ID: ``ZFS-8000-4J`` indicates a device which was unable to be opened by the ZFS subsystem. .. CDDL HEADER START The contents of this file are subject to the terms of the Common Development and Distribution License (the "License"). You may not use this file except in compliance with the License. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE or http://www.opensolaris.org/os/licensing. See the License for the specific language governing permissions and limitations under the License. When distributing Covered Code, include this CDDL HEADER in each file and include the License file at usr/src/OPENSOLARIS.LICENSE. If applicable, add the following below this CDDL HEADER, with the fields enclosed by brackets "[]" replaced with your own identifying information: Portions Copyright [yyyy] [name of copyright owner] CDDL HEADER END Portions Copyright 2007 Sun Microsystems, Inc. .. highlight:: none Message ID: ZFS-8000-5E ======================= Corrupted device label in non-replicated configuration ------------------------------------------------------ +-------------------------+--------------------------------------------------+ | **Type:** | Error | +-------------------------+--------------------------------------------------+ | **Severity:** | Critical | +-------------------------+--------------------------------------------------+ | **Description:** | A device could not be opened due to a missing or | | | invalid device label and no replicas are | | | available. | +-------------------------+--------------------------------------------------+ | **Automated Response:** | No automated response will be taken. | +-------------------------+--------------------------------------------------+ | **Impact:** | The pool is no longer available. | +-------------------------+--------------------------------------------------+ .. rubric:: Suggested Action for System Administrator .. rubric:: For an active pool: If this error was encountered while running ``zpool import``, please see the section below. Otherwise, run ``zpool status -x`` to determine which pool has experienced a failure: :: # zpool status -x pool: test state: FAULTED status: One or more devices could not be used because the label is missing or invalid. There are insufficient replicas for the pool to continue functioning. action: Destroy and re-create the pool from a backup source. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-5E scrub: none requested config: NAME STATE READ WRITE CKSUM test FAULTED 0 0 0 insufficient replicas sda FAULTED 0 0 0 corrupted data sdb ONLINE 0 0 0 errors: No known data errors The device listed as FAULTED with 'corrupted data' cannot be opened due to a corrupt label. ZFS will be unable to use the pool, and all data within the pool is irrevocably lost. The pool must be destroyed and recreated from an appropriate backup source. Using replicated configurations will prevent this from happening in the future. .. rubric:: For an exported pool: If this error is encountered during ``zpool import``, the action is the same. The pool cannot be imported - all data is lost and must be restored from an appropriate backup source. .. rubric:: Details The Message ID: ``ZFS-8000-5E`` indicates a device which was unable to be opened by the ZFS subsystem. .. CDDL HEADER START The contents of this file are subject to the terms of the Common Development and Distribution License (the "License"). You may not use this file except in compliance with the License. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE or http://www.opensolaris.org/os/licensing. See the License for the specific language governing permissions and limitations under the License. When distributing Covered Code, include this CDDL HEADER in each file and include the License file at usr/src/OPENSOLARIS.LICENSE. If applicable, add the following below this CDDL HEADER, with the fields enclosed by brackets "[]" replaced with your own identifying information: Portions Copyright [yyyy] [name of copyright owner] CDDL HEADER END Portions Copyright 2007 Sun Microsystems, Inc. .. highlight:: none Message ID: ZFS-8000-6X ======================= Missing top level device ------------------------ +-------------------------+--------------------------------------------+ | **Type:** | Error | +-------------------------+--------------------------------------------+ | **Severity:** | Critical | +-------------------------+--------------------------------------------+ | **Description:** | One or more top level devices are missing. | +-------------------------+--------------------------------------------+ | **Automated Response:** | No automated response will be taken. | +-------------------------+--------------------------------------------+ | **Impact:** | The pool cannot be imported. | +-------------------------+--------------------------------------------+ .. rubric:: Suggested Action for System Administrator Run ``zpool import`` to list which pool cannot be imported: :: # zpool import pool: test id: 13783646421373024673 state: FAULTED status: One or more devices are missing from the system. action: The pool cannot be imported. Attach the missing devices and try again. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-6X config: test FAULTED missing device sda ONLINE Additional devices are known to be part of this pool, though their exact configuration cannot be determined. ZFS attempts to store enough configuration data on the devices such that the configuration is recoverable from any subset of devices. In some cases, particularly when an entire toplevel virtual device is not attached to the system, ZFS will be unable to determine the complete configuration. It will always detect that these devices are missing, even if it cannot identify all of the devices. The pool cannot be imported until the unknown missing device is attached to the system. If the device has been made available in an alternate location, use the ``-d`` option to ``zpool import`` to search for devices in a different directory. If the missing device is unavailable, then the pool cannot be imported. .. rubric:: Details The Message ID: ``ZFS-8000-6X`` indicates one or more top level devices are missing from the configuration. .. CDDL HEADER START The contents of this file are subject to the terms of the Common Development and Distribution License (the "License"). You may not use this file except in compliance with the License. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE or http://www.opensolaris.org/os/licensing. See the License for the specific language governing permissions and limitations under the License. When distributing Covered Code, include this CDDL HEADER in each file and include the License file at usr/src/OPENSOLARIS.LICENSE. If applicable, add the following below this CDDL HEADER, with the fields enclosed by brackets "[]" replaced with your own identifying information: Portions Copyright [yyyy] [name of copyright owner] CDDL HEADER END Portions Copyright 2007 Sun Microsystems, Inc. .. highlight:: none Message ID: ZFS-8000-72 ======================= Corrupted pool metadata ----------------------- +-------------------------+-------------------------------------------+ | **Type:** | Error | +-------------------------+-------------------------------------------+ | **Severity:** | Critical | +-------------------------+-------------------------------------------+ | **Description:** | The metadata required to open the pool is | | | corrupt. | +-------------------------+-------------------------------------------+ | **Automated Response:** | No automated response will be taken. | +-------------------------+-------------------------------------------+ | **Impact:** | The pool is no longer available. | +-------------------------+-------------------------------------------+ .. rubric:: Suggested Action for System Administrator Even though all the devices are available, the on-disk data has been corrupted such that the pool cannot be opened. If a recovery action is presented, the pool can be returned to a usable state. Otherwise, all data within the pool is lost, and the pool must be destroyed and restored from an appropriate backup source. ZFS includes built-in metadata replication to prevent this from happening even for unreplicated pools, but running in a replicated configuration will decrease the chances of this happening in the future. If this error is encountered during ``zpool import``, see the section below. Otherwise, run ``zpool status -x`` to determine which pool is faulted and if a recovery option is available: :: # zpool status -x pool: test id: 13783646421373024673 state: FAULTED status: The pool metadata is corrupted and cannot be opened. action: Recovery is possible, but will result in some data loss. Returning the pool to its state as of Mon Sep 28 10:24:39 2009 should correct the problem. Approximately 59 seconds of data will have to be discarded, irreversibly. Recovery can be attempted by executing 'zpool clear -F test'. A scrub of the pool is strongly recommended following a successful recovery. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-72 config: NAME STATE READ WRITE CKSUM test FAULTED 0 0 2 corrupted data sda ONLINE 0 0 2 sdb ONLINE 0 0 2 If recovery is unavailable, the recommended action will be: Destroy the pool and restore from backup. If this error is encountered during ``zpool import``, and if no recovery option is mentioned, the pool is unrecoverable and cannot be imported. The pool must be restored from an appropriate backup source. If a recovery option is available, the output from ``zpool import`` will look something like the following: :: # zpool import share cannot import 'share': I/O error Recovery is possible, but will result in some data loss. Returning the pool to its state as of Sun Sep 27 12:31:07 2009 should correct the problem. Approximately 53 seconds of data will have to be discarded, irreversibly. Recovery can be attempted by executing 'zpool import -F share'. A scrub of the pool is strongly recommended following a successful recovery. Recovery actions are requested with the -F option to either ``zpool clear`` or ``zpool import``. Recovery will result in some data loss, because it reverts the pool to an earlier state. A dry-run recovery check can be performed by adding the ``-n`` option, affirming if recovery is possible without actually reverting the pool to its earlier state. .. rubric:: Details The Message ID: ``ZFS-8000-72`` indicates a pool was unable to be opened due to a detected corruption in the pool metadata. .. CDDL HEADER START The contents of this file are subject to the terms of the Common Development and Distribution License (the "License"). You may not use this file except in compliance with the License. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE or http://www.opensolaris.org/os/licensing. See the License for the specific language governing permissions and limitations under the License. When distributing Covered Code, include this CDDL HEADER in each file and include the License file at usr/src/OPENSOLARIS.LICENSE. If applicable, add the following below this CDDL HEADER, with the fields enclosed by brackets "[]" replaced with your own identifying information: Portions Copyright [yyyy] [name of copyright owner] CDDL HEADER END Portions Copyright 2007 Sun Microsystems, Inc. .. highlight:: none Message ID: ZFS-8000-8A ======================= Corrupted data -------------- +-------------------------+----------------------------------------------+ | **Type:** | Error | +-------------------------+----------------------------------------------+ | **Severity:** | Critical | +-------------------------+----------------------------------------------+ | **Description:** | A file or directory could not be read due to | | | corrupt data. | +-------------------------+----------------------------------------------+ | **Automated Response:** | No automated response will be taken. | +-------------------------+----------------------------------------------+ | **Impact:** | The file or directory is unavailable. | +-------------------------+----------------------------------------------+ .. rubric:: Suggested Action for System Administrator Run ``zpool status -x`` to determine which pool is damaged: :: # zpool status -x pool: test state: ONLINE status: One or more devices has experienced an error and no valid replicas are available. Some filesystem data is corrupt, and applications may have been affected. action: Destroy the pool and restore from backup. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A scrub: none requested config: NAME STATE READ WRITE CKSUM test ONLINE 0 0 2 sda ONLINE 0 0 2 sdb ONLINE 0 0 0 errors: 1 data errors, use '-v' for a list Unfortunately, the data cannot be repaired, and the only choice to repair the data is to restore the pool from backup. Applications attempting to access the corrupted data will get an error (EIO), and data may be permanently lost. The list of affected files can be retrieved by using the ``-v`` option to ``zpool status``: :: # zpool status -xv pool: test state: ONLINE status: One or more devices has experienced an error and no valid replicas are available. Some filesystem data is corrupt, and applications may have been affected. action: Destroy the pool and restore from backup. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A scrub: none requested config: NAME STATE READ WRITE CKSUM test ONLINE 0 0 2 sda ONLINE 0 0 2 sdb ONLINE 0 0 0 errors: Permanent errors have been detected in the following files: /export/example/foo Damaged files may or may not be able to be removed depending on the type of corruption. If the corruption is within the plain data, the file should be removable. If the corruption is in the file metadata, then the file cannot be removed, though it can be moved to an alternate location. In either case, the data should be restored from a backup source. It is also possible for the corruption to be within pool-wide metadata, resulting in entire datasets being unavailable. If this is the case, the only option is to destroy the pool and re-create the datasets from backup. .. rubric:: Details The Message ID: ``ZFS-8000-8A`` indicates corrupted data exists in the current pool. .. CDDL HEADER START The contents of this file are subject to the terms of the Common Development and Distribution License (the "License"). You may not use this file except in compliance with the License. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE or http://www.opensolaris.org/os/licensing. See the License for the specific language governing permissions and limitations under the License. When distributing Covered Code, include this CDDL HEADER in each file and include the License file at usr/src/OPENSOLARIS.LICENSE. If applicable, add the following below this CDDL HEADER, with the fields enclosed by brackets "[]" replaced with your own identifying information: Portions Copyright [yyyy] [name of copyright owner] CDDL HEADER END Portions Copyright 2007 Sun Microsystems, Inc. .. highlight:: none Message ID: ZFS-8000-9P ======================= Failing device in replicated configuration ------------------------------------------ +-------------------------+----------------------------------------------------+ | **Type:** | Error | +-------------------------+----------------------------------------------------+ | **Severity:** | Minor | +-------------------------+----------------------------------------------------+ | **Description:** | A device has experienced uncorrectable errors in a | | | replicated configuration. | +-------------------------+----------------------------------------------------+ | **Automated Response:** | ZFS has attempted to repair the affected data. | +-------------------------+----------------------------------------------------+ | **Impact:** | The system is unaffected, though errors may | | | indicate future failure. Future errors may cause | | | ZFS to automatically fault the device. | +-------------------------+----------------------------------------------------+ .. rubric:: Suggested Action for System Administrator Run ``zpool status -x`` to determine which pool has experienced errors: :: # zpool status -x pool: test state: ONLINE status: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected. action: Determine if the device needs to be replaced, and clear the errors using 'zpool online' or replace the device with 'zpool replace'. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P scrub: none requested config: NAME STATE READ WRITE CKSUM test ONLINE 0 0 0 mirror ONLINE 0 0 0 sda ONLINE 0 0 2 sdb ONLINE 0 0 0 errors: No known data errors Find the device with a non-zero error count for READ, WRITE, or CKSUM. This indicates that the device has experienced a read I/O error, write I/O error, or checksum validation error. Because the device is part of a mirror or RAID-Z device, ZFS was able to recover from the error and subsequently repair the damaged data. If these errors persist over a period of time, ZFS may determine the device is faulty and mark it as such. However, these error counts may or may not indicate that the device is unusable. It depends on how the errors were caused, which the administrator can determine in advance of any ZFS diagnosis. For example, the following cases will all produce errors that do not indicate potential device failure: - A network attached device lost connectivity but has now recovered - A device suffered from a bit flip, an expected event over long periods of time - An administrator accidentally wrote over a portion of the disk using another program In these cases, the presence of errors does not indicate that the device is likely to fail in the future, and therefore does not need to be replaced. If this is the case, then the device errors should be cleared using ``zpool clear``: :: # zpool clear test sda On the other hand, errors may very well indicate that the device has failed or is about to fail. If there are continual I/O errors to a device that is otherwise attached and functioning on the system, it most likely needs to be replaced. The administrator should check the system log for any driver messages that may indicate hardware failure. If it is determined that the device needs to be replaced, then the ``zpool replace`` command should be used: :: # zpool replace test sda sdc This will attach the new device to the pool and begin resilvering data to it. Once the resilvering process is complete, the old device will automatically be removed from the pool, at which point it can safely be removed from the system. If the device needs to be replaced in-place (because there are no available spare devices), the original device can be removed and replaced with a new device, at which point a different form of ``zpool replace`` can be used: :: # zpool replace test sda This assumes that the original device at 'sda' has been replaced with a new device under the same path, and will be replaced appropriately. You can monitor the progress of the resilvering operation by using the ``zpool status -x`` command: :: # zpool status -x pool: test state: DEGRADED status: One or more devices is currently being replaced. The pool may not be providing the necessary level of replication. action: Wait for the resilvering operation to complete scrub: resilver in progress, 0.14% done, 0h0m to go config: NAME STATE READ WRITE CKSUM test ONLINE 0 0 0 mirror ONLINE 0 0 0 replacing ONLINE 0 0 0 sda ONLINE 0 0 3 sdc ONLINE 0 0 0 58.5K resilvered sdb ONLINE 0 0 0 errors: No known data errors .. rubric:: Details The Message ID: ``ZFS-8000-9P`` indicates a device has exceeded the acceptable limit of errors allowed by the system. .. CDDL HEADER START The contents of this file are subject to the terms of the Common Development and Distribution License (the "License"). You may not use this file except in compliance with the License. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE or http://www.opensolaris.org/os/licensing. See the License for the specific language governing permissions and limitations under the License. When distributing Covered Code, include this CDDL HEADER in each file and include the License file at usr/src/OPENSOLARIS.LICENSE. If applicable, add the following below this CDDL HEADER, with the fields enclosed by brackets "[]" replaced with your own identifying information: Portions Copyright [yyyy] [name of copyright owner] CDDL HEADER END Portions Copyright 2007 Sun Microsystems, Inc. .. highlight:: none Message ID: ZFS-8000-A5 ======================= Incompatible version -------------------- +-------------------------+------------------------------------------------+ | **Type:** | Error | +-------------------------+------------------------------------------------+ | **Severity:** | Major | +-------------------------+------------------------------------------------+ | **Description:** | The on-disk version is not compatible with the | | | running system. | +-------------------------+------------------------------------------------+ | **Automated Response:** | No automated response will occur. | +-------------------------+------------------------------------------------+ | **Impact:** | The pool is unavailable. | +-------------------------+------------------------------------------------+ .. rubric:: Suggested Action for System Administrator If this error is seen during ``zpool import``, see the section below. Otherwise, run ``zpool status -x`` to determine which pool is faulted: :: # zpool status -x pool: test state: FAULTED status: The ZFS version for the pool is incompatible with the software running on this system. action: Destroy and re-create the pool. scrub: none requested config: NAME STATE READ WRITE CKSUM test FAULTED 0 0 0 incompatible version mirror ONLINE 0 0 0 sda9 ONLINE 0 0 0 sdb9 ONLINE 0 0 0 errors: No known errors The pool cannot be used on this system. Either move the storage to the system where the pool was originally created, upgrade the current system software to a more recent version, or destroy the pool and re-create it from backup. If this error is seen during import, the pool cannot be imported on the current system. The disks must be attached to the system which originally created the pool, and imported there. The list of currently supported versions can be displayed using ``zpool upgrade -v``. .. rubric:: Details The Message ID: ``ZFS-8000-A5`` indicates a version mismatch exists between the running system and the on-disk data. .. CDDL HEADER START The contents of this file are subject to the terms of the Common Development and Distribution License (the "License"). You may not use this file except in compliance with the License. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE or http://www.opensolaris.org/os/licensing. See the License for the specific language governing permissions and limitations under the License. When distributing Covered Code, include this CDDL HEADER in each file and include the License file at usr/src/OPENSOLARIS.LICENSE. If applicable, add the following below this CDDL HEADER, with the fields enclosed by brackets "[]" replaced with your own identifying information: Portions Copyright [yyyy] [name of copyright owner] CDDL HEADER END Portions Copyright 2007 Sun Microsystems, Inc. .. highlight:: none Message ID: ZFS-8000-ER ======================= ZFS Errata #1 ------------- +-------------------------+--------------------------------------------------+ | **Type:** | Compatibility | +-------------------------+--------------------------------------------------+ | **Severity:** | Moderate | +-------------------------+--------------------------------------------------+ | **Description:** | The ZFS pool contains an on-disk format | | | incompatibility. | +-------------------------+--------------------------------------------------+ | **Automated Response:** | No automated response will be taken. | +-------------------------+--------------------------------------------------+ | **Impact:** | Until the pool is scrubbed using OpenZFS version | | | 0.6.3 or newer the pool may not be imported by | | | older versions of OpenZFS or other ZFS | | | implementations. | +-------------------------+--------------------------------------------------+ .. rubric:: Suggested Action for System Administrator The pool contains an on-disk format incompatibility. Affected pools must be imported and scrubbed using the current version of ZFS. This will return the pool to a state in which it may be imported by other implementations. This errata only impacts compatibility between ZFS versions, no user data is at risk as result of this erratum. :: # zpool status -x pool: test state: ONLINE status: Errata #1 detected. action: To correct the issue run 'zpool scrub'. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-ER scan: none requested config: NAME STATE READ WRITE CKSUM test ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 vdev0 ONLINE 0 0 0 vdev1 ONLINE 0 0 0 vdev2 ONLINE 0 0 0 vdev3 ONLINE 0 0 0 errors: No known data errors # zpool scrub test # zpool status -x all pools are healthy ZFS Errata #2 ------------- +-------------------------+---------------------------------------------------+ | **Type:** | Compatibility | +-------------------------+---------------------------------------------------+ | **Severity:** | Moderate | +-------------------------+---------------------------------------------------+ | **Description:** | The ZFS packages were updated while an | | | asynchronous destroy was in progress and the pool | | | contains an on-disk format incompatibility. | +-------------------------+---------------------------------------------------+ | **Automated Response:** | No automated response will be taken. | +-------------------------+---------------------------------------------------+ | **Impact:** | The pool cannot be imported until the issue is | | | corrected. | +-------------------------+---------------------------------------------------+ .. rubric:: Suggested Action for System Administrator Affected pools must be reverted to the previous ZFS version where they can be correctly imported. Once imported, all asynchronous destroy operations must be allowed to complete. The ZFS packages may then be updated and the pool can be imported cleanly by the newer software. :: # zpool import pool: test id: 1165955789558693437 state: ONLINE status: Errata #2 detected. action: The pool cannot be imported with this version of ZFS due to an active asynchronous destroy. Revert to an earlier version and allow the destroy to complete before updating. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-ER config: test ONLINE raidz1-0 ONLINE vdev0 ONLINE vdev1 ONLINE vdev2 ONLINE vdev3 ONLINE Revert to previous ZFS version, import the pool, then wait for the ``freeing`` property to drop to zero. This indicates that all outstanding asynchronous destroys have completed. :: # zpool get freeing NAME PROPERTY VALUE SOURCE test freeing 0 default The ZFS packages may be now be updated and the pool imported. The on-disk format incompatibility can now be corrected online as described in `Errata #1 <#1>`__. ZFS Errata #3 ------------- +-------------------------+----------------------------------------------------+ | **Type:** | Compatibility | +-------------------------+----------------------------------------------------+ | **Severity:** | Moderate | +-------------------------+----------------------------------------------------+ | **Description:** | An encrypted dataset contains an on-disk format | | | incompatibility. | +-------------------------+----------------------------------------------------+ | **Automated Response:** | No automated response will be taken. | +-------------------------+----------------------------------------------------+ | **Impact:** | Encrypted datasets created before the ZFS packages | | | were updated cannot be mounted or opened for | | | write. The errata impacts the ability of ZFS to | | | correctly perform raw sends, so this functionality | | | has been disabled for these datasets. | +-------------------------+----------------------------------------------------+ .. rubric:: Suggested Action for System Administrator System administrators with affected pools will need to recreate any encrypted datasets created before the new version of ZFS was used. This can be accomplished by using ``zfs send`` and ``zfs receive``. Note, however, that backups can NOT be done with a raw ``zfs send -w``, since this would preserve the on-disk incompatibility. Alternatively, system administrators can use conventional tools to back up data to new encrypted datasets. The new version of ZFS will prevent new data from being written to the impacted datasets, but they can still be mounted read-only. :: # zpool status pool: test id: 1165955789558693437 state: ONLINE status: Errata #3 detected. action: To correct the issue backup existing encrypted datasets to new encrypted datasets and destroy the old ones. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-ER config: test ONLINE raidz1-0 ONLINE vdev0 ONLINE vdev1 ONLINE vdev2 ONLINE vdev3 ONLINE Import the pool and backup any existing encrypted datasets to new datasets. To ensure the new datasets are re-encrypted, be sure to receive them below an encryption root or use ``zfs receive -o encryption=on``, then destroy the source dataset. :: # zfs send test/crypt1@snap1 | zfs receive -o encryption=on -o keyformat=passphrase -o keylocation=file:///path/to/keyfile test/newcrypt1 # zfs send -I test/crypt1@snap1 test/crypt1@snap5 | zfs receive test/newcrypt1 # zfs destroy -R test/crypt1 New datasets can be mounted read-write and used normally. The errata will be cleared upon reimporting the pool and the alert will only be shown again if another dataset is found with the errata. To ensure that all datasets are on the new version reimport the pool, load all keys, mount all encrypted datasets, and check ``zpool status``. :: # zpool export test # zpool import test # zfs load-key -a Enter passphrase for 'test/crypt1': 1 / 1 key(s) successfully loaded # zfs mount -a # zpool status -x all pools are healthy ZFS Errata #4 ------------- +-------------------------+----------------------------------------------------+ | **Type:** | Compatibility | +-------------------------+----------------------------------------------------+ | **Severity:** | Moderate | +-------------------------+----------------------------------------------------+ | **Description:** | An encrypted dataset contains an on-disk format | | | incompatibility. | +-------------------------+----------------------------------------------------+ | **Automated Response:** | No automated response will be taken. | +-------------------------+----------------------------------------------------+ | **Impact:** | Encrypted datasets created before the ZFS packages | | | were updated cannot be backed up via a raw send to | | | an updated system. These datasets also cannot | | | receive additional snapshots. New encrypted | | | datasets cannot be created until the | | | ``bookmark_v2`` feature has been enabled. | +-------------------------+----------------------------------------------------+ .. rubric:: Suggested Action for System Administrator First, system administrators with affected pools will need to enable the ``bookmark_v2`` feature on their pools. Enabling this feature will prevent this pool from being imported by previous versions of the ZFS software after any new bookmarks are created (including read-only imports). If the pool contains no encrypted datasets, this is the only step required. If there are existing encrypted datasets, administrators will then need to back these datasets up. This can be done in several ways. Non-raw ``zfs send`` and ``zfs receive`` can be used as per usual, as can traditional backup tools. Raw receives of existing encrypted datasets and raw receives into existing encrypted datasets are currently disabled because ZFS is not able to guarantee that the stream and the existing dataset came from a consistent source. This check can be disabled which will allow ZFS to receive these streams anyway. Note that this can result in datasets with data that cannot be accessed due to authentication errors if raw and non-raw receives are mixed over the course of several incremental backups. To disable this restriction, set the ``zfs_disable_ivset_guid_check`` module parameter to 1. Streams received this way (as well as any received before the upgrade) will need to be manually checked by reading the data to ensure they are not corrupted. Note that ``zpool scrub`` cannot be used for this purpose because the scrub does not check the cryptographic authentication codes. For more information on this issue, please refer to the zfs man page section on ``zfs receive`` which describes the restrictions on raw sends. :: # zpool status pool: test state: ONLINE status: Errata #4 detected. Existing encrypted datasets contain an on-disk incompatibility which needs to be corrected. action: To correct the issue enable the bookmark_v2 feature and backup any existing encrypted datasets to new encrypted datasets and destroy the old ones. If this pool does not contain any encrypted datasets, simply enable the bookmark_v2 feature. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-ER scan: none requested config: NAME STATE READ WRITE CKSUM test ONLINE 0 0 0 /root/vdev0 ONLINE 0 0 0 errors: No known data errors Import the pool and enable the ``bookmark_v2`` feature. Then backup any existing encrypted datasets to new datasets. This can be done with traditional tools or via ``zfs send``. Raw sends will require that the ``zfs_disable_ivset_guid_check`` is set to 1 on the receive side. Once this is done, the original datasets should be destroyed. :: # zpool set feature@bookmark_v2=enabled test # echo 1 > /sys/module/zfs/parameters/zfs_disable_ivset_guid_check # zfs send -Rw test/crypt1@snap1 | zfs receive test/newcrypt1 # zfs send -I test/crypt1@snap1 test/crypt1@snap5 | zfs receive test/newcrypt1 # zfs destroy -R test/crypt1 # echo 0 > /sys/module/zfs/parameters/zfs_disable_ivset_guid_check The errata will be cleared upon reimporting the pool and the alert will only be shown again if another dataset is found with the errata. To check that all datasets are fixed, perform a ``zfs list -t all``, and check ``zpool status`` once it is completed. :: # zpool export test # zpool import test # zpool scrub # wait for completion # zpool status -x all pools are healthy .. CDDL HEADER START The contents of this file are subject to the terms of the Common Development and Distribution License (the "License"). You may not use this file except in compliance with the License. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE or http://www.opensolaris.org/os/licensing. See the License for the specific language governing permissions and limitations under the License. When distributing Covered Code, include this CDDL HEADER in each file and include the License file at usr/src/OPENSOLARIS.LICENSE. If applicable, add the following below this CDDL HEADER, with the fields enclosed by brackets "[]" replaced with your own identifying information: Portions Copyright [yyyy] [name of copyright owner] CDDL HEADER END Portions Copyright 2007 Sun Microsystems, Inc. .. highlight:: none Message ID: ZFS-8000-EY ======================= ZFS label hostid mismatch ------------------------- +-------------------------+---------------------------------------------------+ | **Type:** | Error | +-------------------------+---------------------------------------------------+ | **Severity:** | Major | +-------------------------+---------------------------------------------------+ | **Description:** | The ZFS pool was last accessed by another system. | +-------------------------+---------------------------------------------------+ | **Automated Response:** | No automated response will be taken. | +-------------------------+---------------------------------------------------+ | **Impact:** | ZFS filesystems are not available. | +-------------------------+---------------------------------------------------+ .. rubric:: Suggested Action for System Administrator The pool has been written to from another host, and was not cleanly exported from the other system. Actively importing a pool on multiple systems will corrupt the pool and leave it in an unrecoverable state. To determine which system last accessed the pool, run the ``zpool import`` command: :: # zpool import pool: test id: 14702934086626715962 state: ONLINE status: The pool was last accessed by another system. action: The pool can be imported using its name or numeric identifier and the '-f' flag. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY config: test ONLINE sda ONLINE # zpool import test cannot import 'test': pool may be in use from other system, it was last accessed by 'tank' (hostid: 0x1435718c) on Fri Mar 9 15:42:47 2007 use '-f' to import anyway If you are certain that the pool is not being actively accessed by another system, then you can use the ``-f`` option to ``zpool import`` to forcibly import the pool. .. rubric:: Details The Message ID: ``ZFS-8000-EY`` indicates that the pool cannot be imported as it was last accessed by another system. Take the documented action to resolve the problem. .. CDDL HEADER START The contents of this file are subject to the terms of the Common Development and Distribution License (the "License"). You may not use this file except in compliance with the License. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE or http://www.opensolaris.org/os/licensing. See the License for the specific language governing permissions and limitations under the License. When distributing Covered Code, include this CDDL HEADER in each file and include the License file at usr/src/OPENSOLARIS.LICENSE. If applicable, add the following below this CDDL HEADER, with the fields enclosed by brackets "[]" replaced with your own identifying information: Portions Copyright [yyyy] [name of copyright owner] CDDL HEADER END Portions Copyright 2007 Sun Microsystems, Inc. .. highlight:: none Message ID: ZFS-8000-HC ======================= ZFS pool I/O failures --------------------- +-------------------------+-----------------------------------------+ | **Type:** | Error | +-------------------------+-----------------------------------------+ | **Severity:** | Major | +-------------------------+-----------------------------------------+ | **Description:** | The ZFS pool has experienced currently | | | unrecoverable I/O failures. | +-------------------------+-----------------------------------------+ | **Automated Response:** | No automated response will be taken. | +-------------------------+-----------------------------------------+ | **Impact:** | Read and write I/Os cannot be serviced. | +-------------------------+-----------------------------------------+ .. rubric:: Suggested Action for System Administrator The pool has experienced I/O failures. Since the ZFS pool property ``failmode`` is set to 'wait', all I/Os (reads and writes) are blocked. See the zpoolprops(8) manpage for more information on the ``failmode`` property. Manual intervention is required for I/Os to be serviced. You can see which devices are affected by running ``zpool status -x``: :: # zpool status -x pool: test state: FAULTED status: There are I/O failures. action: Make sure the affected devices are connected, then run 'zpool clear'. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-HC scrub: none requested config: NAME STATE READ WRITE CKSUM test FAULTED 0 13 0 insufficient replicas sda FAULTED 0 7 0 experienced I/O failures sdb ONLINE 0 0 0 errors: 1 data errors, use '-v' for a list After you have made sure the affected devices are connected, run ``zpool clear`` to allow I/O to the pool again: :: # zpool clear test If I/O failures continue to happen, then applications and commands for the pool may hang. At this point, a reboot may be necessary to allow I/O to the pool again. .. rubric:: Details The Message ID: ``ZFS-8000-HC`` indicates that the pool has experienced I/O failures. Take the documented action to resolve the problem. .. CDDL HEADER START The contents of this file are subject to the terms of the Common Development and Distribution License (the "License"). You may not use this file except in compliance with the License. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE or http://www.opensolaris.org/os/licensing. See the License for the specific language governing permissions and limitations under the License. When distributing Covered Code, include this CDDL HEADER in each file and include the License file at usr/src/OPENSOLARIS.LICENSE. If applicable, add the following below this CDDL HEADER, with the fields enclosed by brackets "[]" replaced with your own identifying information: Portions Copyright [yyyy] [name of copyright owner] CDDL HEADER END Portions Copyright 2007 Sun Microsystems, Inc. .. highlight:: none Message ID: ZFS-8000-JQ ======================= ZFS pool I/O failures --------------------- +-------------------------+----------------------------------------+ | **Type:** | Error | +-------------------------+----------------------------------------+ | **Severity:** | Major | +-------------------------+----------------------------------------+ | **Description:** | The ZFS pool has experienced currently | | | unrecoverable I/O failures. | +-------------------------+----------------------------------------+ | **Automated Response:** | No automated response will be taken. | +-------------------------+----------------------------------------+ | **Impact:** | Write I/Os cannot be serviced. | +-------------------------+----------------------------------------+ .. rubric:: Suggested Action for System Administrator The pool has experienced I/O failures. Since the ZFS pool property ``failmode`` is set to 'continue', read I/Os will continue to be serviced, but write I/Os are blocked. See the zpoolprops(8) manpage for more information on the ``failmode`` property. Manual intervention is required for write I/Os to be serviced. You can see which devices are affected by running ``zpool status -x``: :: # zpool status -x pool: test state: FAULTED status: There are I/O failures. action: Make sure the affected devices are connected, then run 'zpool clear'. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-HC scrub: none requested config: NAME STATE READ WRITE CKSUM test FAULTED 0 13 0 insufficient replicas sda9 FAULTED 0 7 0 experienced I/O failures sdb9 ONLINE 0 0 0 errors: 1 data errors, use '-v' for a list After you have made sure the affected devices are connected, run ``zpool clear`` to allow write I/O to the pool again: :: # zpool clear test If I/O failures continue to happen, then applications and commands for the pool may hang. At this point, a reboot may be necessary to allow I/O to the pool again. .. rubric:: Details The Message ID: ``ZFS-8000-JQ`` indicates that the pool has experienced I/O failures. Take the documented action to resolve the problem. .. CDDL HEADER START The contents of this file are subject to the terms of the Common Development and Distribution License (the "License"). You may not use this file except in compliance with the License. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE or http://www.opensolaris.org/os/licensing. See the License for the specific language governing permissions and limitations under the License. When distributing Covered Code, include this CDDL HEADER in each file and include the License file at usr/src/OPENSOLARIS.LICENSE. If applicable, add the following below this CDDL HEADER, with the fields enclosed by brackets "[]" replaced with your own identifying information: Portions Copyright [yyyy] [name of copyright owner] CDDL HEADER END Portions Copyright 2007 Sun Microsystems, Inc. .. highlight:: none Message ID: ZFS-8000-K4 ======================= ZFS intent log read failure --------------------------- +-------------------------+--------------------------------------------+ | **Type:** | Error | +-------------------------+--------------------------------------------+ | **Severity:** | Major | +-------------------------+--------------------------------------------+ | **Description:** | A ZFS intent log device could not be read. | +-------------------------+--------------------------------------------+ | **Automated Response:** | No automated response will be taken. | +-------------------------+--------------------------------------------+ | **Impact:** | The intent log(s) cannot be replayed. | +-------------------------+--------------------------------------------+ .. rubric:: Suggested Action for System Administrator A ZFS intent log record could not be read due to an error. This may be due to a missing or broken log device, or a device within the pool may be experiencing I/O errors. The pool itself is not corrupt but is missing some pool changes that happened shortly before a power loss or system failure. These are pool changes that applications had requested to be written synchronously but had not been committed in the pool. This transaction group commit currently occurs every five seconds, and so typically at most five seconds worth of synchronous writes have been lost. ZFS itself cannot determine if the pool changes lost are critical to those applications running at the time of the system failure. This is a decision the administrator must make. You may want to consider mirroring log devices. First determine which pool is in error: :: # zpool status -x pool: test state: FAULTED status: One or more of the intent logs could not be read. Waiting for administrator intervention to fix the faulted pool. action: Either restore the affected device(s) and run 'zpool online', or ignore the intent log records by running 'zpool clear'. scrub: none requested config: NAME STATE READ WRITE CKSUM test FAULTED 0 0 0 bad intent log c3t2d0 ONLINE 0 0 0 logs FAULTED 0 0 0 bad intent log c5t3d0 UNAVAIL 0 0 0 cannot open There are two courses of action to resolve this problem. If the validity of the pool from an application perspective requires the pool changes then the log devices must be recovered. Make sure power and cables are connected and that the affected device is online. Then run ``zpool online`` and then ``zpool clear``: :: # zpool online test c5t3d0 # zpool clear test # zpool status test pool: test state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM test ONLINE 0 0 0 c3t2d0 ONLINE 0 0 0 logs ONLINE 0 0 0 c5t3d0 ONLINE 0 0 0 errors: No known data errors The second alternative action is to ignore the most recent pool changes that could not be read. To do this run ``zpool clear``: :: # zpool clear test # zpool status test pool: test state: DEGRADED status: One or more devices could not be opened. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Attach the missing device and online it using 'zpool online'. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-2Q scrub: none requested config: NAME STATE READ WRITE CKSUM test DEGRADED 0 0 0 c3t2d0 ONLINE 0 0 0 logs DEGRADED 0 0 0 c5t3d0 UNAVAIL 0 0 0 cannot open errors: No known data errors Future log records will not use a failed log device but will be written to the main pool. You should fix or replace any failed log devices. .. rubric:: Details The Message ID: ``ZFS-8000-K4`` indicates that a log device is missing or cannot be read. License ======= - The OpenZFS software is licensed under the Common Development and Distribution License (`CDDL `__) unless otherwise noted. - The OpenZFS documentation content is licensed under a Creative Commons Attribution-ShareAlike license (`CC BY-SA 3.0 `__) unless otherwise noted. - OpenZFS is an associated project of SPI (`Software in the Public Interest `__). SPI is a 501(c)(3) nonprofit organization which handles the donations, finances, and legal holdings of the project. .. note:: The Linux Kernel is licensed under the GNU General Public License Version 2 (`GPLv2 `__). While both (OpenZFS and Linux Kernel) are free open source licenses they are restrictive licenses. The combination of them causes problems because it prevents using pieces of code exclusively available under one license with pieces of code exclusively available under the other in the same binary. In the case of the Linux Kernel, this prevents us from distributing OpenZFS as part of the Linux Kernel binary. However, there is nothing in either license that prevents distributing it in the form of a binary module or in the form of source code. Additional reading and opinions: - `Software Freedom Law Center `__ - `Software Freedom Conservancy `__ - `Free Software Foundation `__ - `Encouraging closed source modules `__ CC BY-SA 3.0: |Creative Commons License| .. |Creative Commons License| image:: https://i.creativecommons.org/l/by-sa/3.0/88x31.png :target: http://creativecommons.org/licenses/by-sa/3.0/ :orphan: Fedora ======================= This page has been moved to `here `__. :orphan: RHEL and CentOS ======================= This page has been moved to `RHEL-based distro `__. :orphan: FAQ Hole birth ============== Short explanation ~~~~~~~~~~~~~~~~~ The hole_birth feature has/had bugs, the result of which is that, if you do a ``zfs send -i`` (or ``-R``, since it uses ``-i``) from an affected dataset, the receiver will not see any checksum or other errors, but the resulting destination snapshot will not match the source. ZoL versions 0.6.5.8 and 0.7.0-rc1 (and above) default to ignoring the faulty metadata which causes this issue *on the sender side*. FAQ ~~~ I have a pool with hole_birth enabled, how do I know if I am affected? ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ It is technically possible to calculate whether you have any affected files, but it requires scraping zdb output for each file in each snapshot in each dataset, which is a combinatoric nightmare. (If you really want it, there is a proof of concept `here `__. Is there any less painful way to fix this if we have already received an affected snapshot? ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ No, the data you need was simply not present in the send stream, unfortunately, and cannot feasibly be rewritten in place. Long explanation ~~~~~~~~~~~~~~~~ hole_birth is a feature to speed up ZFS send -i - in particular, ZFS used to not store metadata on when "holes" (sparse regions) in files were created, so every zfs send -i needed to include every hole. hole_birth, as the name implies, added tracking for the txg (transaction group) when a hole was created, so that zfs send -i could only send holes that had a birth_time between (starting snapshot txg) and (ending snapshot txg), and life was wonderful. Unfortunately, hole_birth had a number of edge cases where it could "forget" to set the birth_time of holes in some cases, causing it to record the birth_time as 0 (the value used prior to hole_birth, and essentially equivalent to "since file creation"). This meant that, when you did a zfs send -i, since zfs send does not have any knowledge of the surrounding snapshots when sending a given snapshot, it would see the creation txg as 0, conclude "oh, it is 0, I must have already sent this before", and not include it. This means that, on the receiving side, it does not know those holes should exist, and does not create them. This leads to differences between the source and the destination. ZoL versions 0.6.5.8 and 0.7.0-rc1 (and above) default to ignoring this metadata and always sending holes with birth_time 0, configurable using the tunable known as ``ignore_hole_birth`` or ``send_holes_without_birth_time``. The latter is what OpenZFS standardized on. ZoL version 0.6.5.8 only has the former, but for any ZoL version with ``send_holes_without_birth_time``, they point to the same value, so changing either will work.