Copy-on-Write

Almost everything distinctive about ZFS follows from one rule: a block that is in use is never overwritten. Changing data means writing it somewhere else and then switching over to the new version. Understanding that rule makes the rest of ZFS — snapshots, zfs send, self-healing, the absence of fsck — stop looking like separate features and start looking like consequences.

The tree

A pool is one large tree. Leaves hold data; interior nodes hold block pointers, each of which records where a child lives, how big it is, and — the part that matters — its checksum. A block’s checksum is therefore stored in its parent, not next to the block itself, so a damaged block cannot vouch for itself.

At the root sits the uberblock. Everything in the pool is reachable from it.

Writing something

Modifying a block in the middle of the tree cannot happen in place, so ZFS:

  1. writes the new data to free space;

  2. writes a new version of the parent, pointing at the new block;

  3. repeats up the tree, since each parent’s checksum has now changed;

  4. finally writes a new uberblock.

uberblock root ind 1 ind 2 D1 D2 D3 D4 D4 ind 2 root uberblock D4' ind 2' root' uberblock' untouched — still whole, still consistent superseded — freed, unless a snapshot holds it written by this transaction shared by reference, not copied

In the diagram, only D4 was modified. That forced new copies of D4, its parent ind 2, the root block and finally the uberblock — the path from the changed block to the root, and nothing else. D1, D2, D3 and ind 1 are referenced by the new tree, not duplicated.

Until that last step, nothing has changed: the old uberblock still describes a complete, consistent pool. The switch to the new tree is a single atomic write. This is why a ZFS pool is always consistent on disk regardless of when power is lost — there is no window in which the pool is half-updated, and so no equivalent of the RAID-5 write hole and no need for a journal replay of metadata.

The cost of a small write is therefore proportional to the depth of the tree, not its size — and the old tree is left whole, which is exactly what a snapshot needs.

The uberblock is not overwritten either. Each device label carries a 128 KiB uberblock ring written round-robin, and each device carries four copies of the label — two at the start, two at the end — so that losing either end of a disk, or a partition table rewrite, does not lose the root of the tree. On import ZFS picks the valid uberblock with the highest transaction number.

Transaction groups

Writes are not committed one at a time. They are batched into a transaction group (txg), and the whole group commits atomically or not at all.

Three transaction groups are in flight at once: one open and accepting new writes, one quiescing (closed to new writes, waiting for stragglers), and one syncing to disk. A group closes when zfs_txg_timeout elapses — 5 seconds by default — or when enough dirty data has accumulated.

The practical consequence is that an asynchronous write returns as soon as it is in memory, and reaches disk up to a few seconds later. A crash loses the groups that had not finished syncing, and the pool comes back at the last committed one. It is consistent, just slightly older.

That is unacceptable for applications that called fsync(), which is what the ZFS Intent Log exists for: it records synchronous writes immediately so they can be replayed if the system dies before the txg commits. The ZIL is never read otherwise.

What follows from it

Snapshots are nearly free. The old tree is already intact and complete — a snapshot is just a reference that stops those blocks being freed. Nothing is copied. See Snapshots, Clones and Bookmarks.

Clones and block cloning are cheap for the same reason: multiple references to the same block are normal, so sharing costs a reference rather than a copy.

Incremental replication is exact. Because every block records the txg in which it was born, ZFS can find precisely the blocks that changed between two snapshots without scanning anything. See Send and Receive.

Corruption is detectable and often repairable. The checksum lives in the parent, so a block that comes back wrong is caught on read, and with redundancy a good copy is used and the bad one rewritten. See Checksums and Scrub and Resilver.

There is no fsck. fsck.zfs exists only to satisfy the interface and does essentially nothing — its man page says so. Nothing needs to be checked for consistency at mount time, because the pool is never left inconsistent. Verifying content is a different job, and that is what zpool scrub does, online.

The costs

Copy-on-write is not free, and its costs explain several tuning topics:

Fragmentation. Rewriting a file scatters its new blocks wherever there was free space, so data that was written sequentially and then modified randomly does not stay sequential. This is why ZFS is sensitive to how full a pool is: finding contiguous free space gets harder, and the allocator has to work harder. See Workload Tuning.

Free space is needed to write at all. Even deleting data requires allocating, which is why ZFS reserves slop space — see Quotas and Reservations.

Partial writes are expensive. Changing part of a record means reading the whole record, modifying it, and writing a new one. Matching recordsize to the application’s I/O size is the usual remedy, and the reason databases and VM images get their own settings.

Further reading