Module Parameters 

Turbo L2ARC warmup

When to change: If cache devices exist and it is desired to fill them as fast as possible

Notes: Turbo L2ARC cache warm-up. When the L2ARC is cold the fill interval will be set to aggressively fill as fast as possible.

l2arc_feed_min_ms

Versions:: v0.6 - master
Platforms:: Linux, FreeBSD
Type:: u64
Default:: 200
Range:: 0 to (1000 * l2arc_feed_secs)
Change:: Dynamic
Tags:: ARC, L2ARC

Min feed interval in milliseconds

When to change: If cache devices exist and l2arc_feed_again and the feed is too aggressive, then this tunable can be adjusted to reduce the impact of the fill

Notes: Minimum time period for aggressively feeding the L2ARC. The L2ARC feed thread wakes up once per second (see l2arc_feed_secs) to look for data to feed into the L2ARC. l2arc_feed_min_ms only affects the turbo L2ARC cache warm-up and allows the aggressiveness to be adjusted.

l2arc_feed_secs

Versions:: v0.6 - master
Platforms:: Linux, FreeBSD
Type:: u64
Default:: 1
Range:: 1 to UINT64_MAX
Change:: Dynamic
Tags:: ARC, L2ARC

Seconds between L2ARC writing

When to change: Do not change

Notes: Seconds between waking the L2ARC feed thread. One feed thread works for all cache devices in turn. If the pool that owns a cache device is imported readonly, then the feed thread is delayed 5 * l2arc_feed_secs before moving onto the next cache device. If multiple pools are imported with cache devices and one pool with cache is imported readonly, the L2ARC feed rate to all caches can be slowed.

l2arc_headroom

Versions:

v0.6 - master

Platforms:

Linux, FreeBSD

Type:

u64

Default:

v2.2 - master	`8`
v0.6 - v2.1	`2`

Change:

Dynamic

Tags:

Number of max device writes to precache

When to change: If the rate of change in the ARC is faster than the overall L2ARC feed rate, then increasing l2arc_headroom can increase L2ARC efficiency. Setting the value too large can cause the L2ARC feed thread to consume more CPU time looking for data to feed. Setting to 0 disables the headroom limit, allowing the full length of ARC lists to be searched for cacheable content.

Notes: How far through the ARC lists to search for L2ARC cacheable content, expressed as a multiplier of l2arc_write_max

l2arc_headroom_boost

Versions:: v0.6 - master
Platforms:: Linux, FreeBSD
Type:: u64
Default:: 200
Range:: 100 to UINT64_MAX, when set to 100, the L2ARC headroom boost feature is effectively disabled
Change:: Dynamic
Tags:: ARC, L2ARC

Compressed l2arc_headroom multiplier

When to change: If average compression efficiency is greater than 2:1, then increasing l2arc_headroom_boost can increase the L2ARC feed rate

Notes: Percentage scale for l2arc_headroom when L2ARC contents are being successfully compressed before writing.

l2arc_meta_cycles

Versions:: master
Platforms:: Linux, FreeBSD
Type:: u64
Default:: 2
Change:: Dynamic
Tags:: ARC, L2ARC

Consecutive metadata cycles before skipping to let data run

l2arc_meta_percent

Versions:: v2.0 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 33
Range:: 0 to 100
Change:: Dynamic
Tags:: ARC, L2ARC

Percent of ARC size allowed for L2ARC-only headers

When to change: When workload really require enormous L2ARC.

Notes: Percent of ARC size allowed for L2ARC-only headers. Since L2ARC buffers are not evicted on memory pressure, too large amount of headers on system with irrationally large L2ARC can render it slow or unusable. This parameter limits L2ARC writes and rebuild to achieve it.

l2arc_mfuonly

Versions:

v2.0 - master

Platforms:

Linux, FreeBSD

Type:

int

Default:

0

Range:

`0`	store MRU and MFU blocks in cache device
`1`	store MFU blocks in cache device

Change:

Dynamic

Tags:

Cache only MFU data from ARC into L2ARC

When to change: When accessing a large amount of data only once.

Notes: Controls whether only MFU metadata and data are cached from ARC into L2ARC. This may be desirable to avoid wasting space on L2ARC when reading/writing large amounts of data that are not expected to be accessed more than once. By default both MRU and MFU data and metadata are cached in the L2ARC.

l2arc_nocompress

Versions:

v0.6

Platforms:

Linux, FreeBSD

Type:

int

Range:

`0`	store compressed blocks in cache device
`1`	store uncompressed blocks in cache device

Change:

Dynamic

Tags:

Skip compressing L2ARC buffers

When to change: When testing compressed L2ARC feature

Notes: Removed in v0.7.0 by commit d3c2ae1c0 (“OpenZFS 6950 - ARC should cache compressed data”). The ARC now always caches compressed data, making this toggle unnecessary. No replacement parameter. Disable writing compressed data to cache devices. Disabling allows the legacy behavior of writing decompressed data to cache devices.

l2arc_noprefetch

Versions:

v0.6 - master

Platforms:

Linux, FreeBSD

Type:

int

Default:

1

Range:

`0`	write prefetched but unused buffers to cache devices
`1`	do not write prefetched but unused buffers to cache devices

Change:

Dynamic

Tags:

ARC, L2ARC, prefetch

Skip caching prefetched buffers

When to change: Setting to 0 can increase L2ARC hit rates for workloads where the ARC is too small for a read workload that benefits from prefetching. Also, if the main pool devices are very slow, setting to 0 can improve some workloads such as backups.

l2arc_norw

Versions:

v0.6 - master

Platforms:

Linux, FreeBSD

Type:

int

Default:

0

Range:

`0`	read and write simultaneously
`1`	avoid writes when reading for antique SSDs

Change:

Dynamic

Tags:

No reads during writes

When to change: In the early days of SSDs, some devices did not perform well when reading and writing simultaneously. Modern SSDs do not have these issues.

Notes: Disables writing to cache devices while they are being read.

l2arc_rebuild_blocks_min_l2size

Versions:: v2.0 - master
Platforms:: Linux, FreeBSD
Type:: u64
Default:: 1073741824
Change:: Dynamic
Tags:: ARC, L2ARC

Min size in bytes to write rebuild log blocks in L2ARC

When to change: The cache device is small and the pool is frequently imported.

Notes: The minimum required size (in bytes) of an L2ARC device in order to write log blocks in it. The log blocks are used upon importing the pool to rebuild the persistent L2ARC. For L2ARC devices less than 1GB the overhead involved offsets most of benefit so log blocks are not written for cache devices smaller than this.

l2arc_rebuild_enabled

Versions:

v2.0 - master

Platforms:

Linux, FreeBSD

Type:

int

Default:

1

Range:

`0`	disable persistent L2ARC rebuild
`1`	enable persistent L2ARC rebuild

Change:

Dynamic

Tags:

Rebuild the L2ARC when importing a pool

When to change: If there are problems importing a pool or attaching an L2ARC device.

l2arc_trim_ahead

Versions:: v2.0 - master
Platforms:: Linux, FreeBSD
Type:: u64
Default:: 0
Range:: 0 to 100
Change:: Dynamic
Tags:: ARC, L2ARC, trim

TRIM ahead L2ARC write size multiplier

When to change: Consider setting for cache devices which efficiently handle TRIM commands.

Notes: Once the cache device has been filled TRIM ahead of the current write size l2arc_write_max on L2ARC devices by this percentage. This can speed up future writes depending on the performance characteristics of the cache device. When set to 100% TRIM twice the space required to accommodate upcoming writes. A minimum of 64MB will be trimmed. If set it enables TRIM of the whole L2ARC device when it is added to a pool. By default, this option is disabled since it can put significant stress on the underlying storage devices.

l2arc_write_boost

Versions:

v0.6 - v2.4

Platforms:

Linux, FreeBSD

Type:

u64

Default:

v2.2 - v2.4	`33554432`
v0.6 - v2.1	`8,388,608`

Change:

Dynamic

Tags:

Extra write bytes during device warmup

When to change: To fill the cache devices more aggressively after pool import.

Notes: Until the ARC fills, increases the L2ARC fill rate l2arc_write_max by l2arc_write_boost.

l2arc_write_max

Versions:

v0.6 - master

Platforms:

Linux, FreeBSD

Type:

u64

Default:

v2.2 - master	`33554432`
v0.6 - v2.1	`8,388,608`

Range:

1 to UINT64_MAX

Change:

Dynamic

Tags:

Max write bytes per interval

When to change: If the cache devices can sustain the write workload, increasing the rate of cache device fill when workloads generate new data at a rate higher than l2arc_write_max can increase L2ARC hit rate

Notes: Maximum number of bytes to be written to each cache device for each L2ARC feed thread interval (see l2arc_feed_secs). Until v2.4 the limit could be raised during warmup by l2arc_write_boost, which has since been removed. By default l2arc_feed_secs is 1 second, delivering a maximum write workload to cache devices of 32 MiB/sec.

metaslab_aliquot

Versions:

v0.6 - master

Platforms:

Linux, FreeBSD

Type:

u64

Default:

v2.4 - master	`2097152`
v2.1 - v2.3	`1048576`
v0.6 - v2.0	`524,288`

Change:

Dynamic

Tags:

allocation, metaslab, vdev

Allocation granularity (a.k.a. stripe size)

When to change: If write performance increases as devices more efficiently write larger, contiguous blocks

Notes: Sets the metaslab granularity. Nominally, ZFS will try to allocate this amount of data to a top-level vdev before moving on to the next top-level vdev. This is roughly similar to what would be referred to as the “stripe size” in traditional RAID arrays. When tuning for HDDs, it can be more efficient to have a few larger, sequential writes to a device rather than switching to the next device. Monitoring the size of contiguous writes to the disks relative to the write throughput can be used to determine if increasing metaslab_aliquot can help. For modern devices, it is unlikely that decreasing metaslab_aliquot from the default will help. If there is only one top-level vdev, this tunable is not used.

metaslab_bias_enabled

Versions:

v0.6 - master

Platforms:

Linux, FreeBSD

Type:

int

Default:

1

Range:

`0`	spread evenly across top-level vdevs
`1`	bias spread to favor less full top-level vdevs

Change:

Dynamic

Tags:

allocation, metaslab, vdev

Enable space-based metaslab group biasing

When to change: If a new top-level vdev is added and you do not want to bias new allocations to the new top-level vdev

Notes: Enables metaslab group biasing based on a top-level vdev’s utilization relative to the pool. Nominally, all top-level devs are the same size and the allocation is spread evenly. When the top-level vdevs are not of the same size, for example if a new (empty) top-level is added to the pool, this allows the new top-level vdev to get a larger portion of new allocations.

metaslab_debug_load

Versions:

v0.6 - master

Platforms:

Linux, FreeBSD

Type:

int

Default:

0

Range:

`0`	do not load all metaslab info at pool import
`1`	dynamically load metaslab info as needed

Change:

Dynamic

Tags:

allocation, debug, memory, metaslab

Load all metaslabs when pool is first opened

When to change: When RAM is plentiful and pool import time is not a consideration

Notes: When enabled, all metaslabs are loaded into memory during pool import. Nominally, metaslab space map information is loaded and unloaded as needed (see metaslab_debug_unload) It is difficult to predict how much RAM is required to store a space map. An empty or completely full metaslab has a small space map. However, a highly fragmented space map can consume significantly more memory. Enabling metaslab_debug_load can increase pool import time.

metaslab_debug_unload

Versions:

v0.6 - master

Platforms:

Linux, FreeBSD

Type:

int

Default:

0

Range:

`0`	dynamically unload metaslab info
`1`	unload metaslab info only upon pool export

Change:

Dynamic

Tags:

allocation, debug, memory, metaslab

Prevent metaslabs from being unloaded

When to change: When RAM is plentiful and the penalty for dynamically reloading metaslab info from the pool is high

Notes: When enabled, prevents metaslab information from being dynamically unloaded from RAM. Nominally, metaslab space map information is loaded and unloaded as needed (see metaslab_debug_load) It is difficult to predict how much RAM is required to store a space map. An empty or completely full metaslab has a small space map. However, a highly fragmented space map can consume significantly more memory. Enabling metaslab_debug_unload consumes RAM that would otherwise be freed.

metaslab_df_alloc_threshold

Versions:: master
Platforms:: Linux, FreeBSD
Type:: u64
Default:: 131072
Change:: Dynamic
Tags:: metaslab

Minimum size which forces the dynamic allocator to change its allocation strategy

metaslab_df_free_pct

Versions:: master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 4
Change:: Dynamic
Tags:: metaslab

The minimum free space, in percent, to continue allocations in a first-fit fashion

metaslab_df_max_search

Versions:: v0.8 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 16777216
Change:: Dynamic
Tags:: metaslab

Max distance (bytes) to search forward before using size tree

metaslab_df_use_largest_segment

Versions:: v0.8 - master
Platforms:: Linux, FreeBSD
Type:: int
Default:: 0
Change:: Dynamic
Tags:: metaslab

When looking in size tree, use largest segment instead of exact fit

metaslab_force_ganging

Versions:: v0.8 - master
Platforms:: Linux, FreeBSD
Type:: u64
Default:: 16777217
Range:: SPA_MINBLOCKSIZE to (SPA_MAXBLOCKSIZE + 1)
Change:: Dynamic
Tags:: allocation, metaslab

Blocks larger than this size are sometimes forced to be gang blocks

When to change: for development testing purposes only

Notes: When testing allocation code, metaslab_force_ganging forces blocks above the specified size to be ganged.

metaslab_force_ganging_pct

Versions:: v2.2 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 3
Change:: Dynamic
Tags:: metaslab

Percentage of large blocks that will be forced to be gang blocks

metaslab_fragmentation_factor_enabled

Versions:

v0.6 - master

Platforms:

Linux, FreeBSD

Type:

int

Default:

1

Range:

`0`	do not consider metaslab free space fragmentation
`1`	try to avoid fragmented metaslabs

Change:

Dynamic

Tags:

allocation, metaslab

Use the fragmentation metric to prefer less fragmented metaslabs

When to change: To test metaslab fragmentation

Notes: Enable use of the fragmentation metric in computing metaslab weights. In version v0.7.0, if zfs_metaslab_segment_weight_enabled is enabled, then metaslab_fragmentation_factor_enabled is ignored.

metaslab_lba_weighting_enabled

Versions:

v0.6 - master

Platforms:

Linux, FreeBSD

Type:

int

Default:

1

Range:

`0`	do not use LBA weighting
`1`	use LBA weighting

Change:

Dynamic

Tags:

allocation, HDD, metaslab, SSD

Prefer metaslabs with lower LBAs

When to change: disable if using only SSDs and version v0.6.4 or earlier

Verification: The rotational setting described by a block device in sysfs by observing /sys/ block/DISK_NAME/queue/rotational

Notes: Modern HDDs have uniform bit density and constant angular velocity. Therefore, the outer recording zones are faster (higher bandwidth) than the inner zones by the ratio of outer to inner track diameter. The difference in bandwidth can be 2:1, and is often available in the HDD detailed specifications or drive manual. For HDDs when metaslab_lba_weighting_enabled is true, write allocation preference is given to the metaslabs representing the outer recording zones. Thus the allocation to metaslabs prefers faster bandwidth over free space. If the devices are not rotational, yet misrepresent themselves to the OS as rotational, then disabling metaslab_lba_weighting_enabled can result in more even, free-space-based allocation.

metaslab_perf_bias

Versions:: v2.4 - master
Platforms:: Linux, FreeBSD
Type:: int
Default:: 1
Change:: Dynamic
Tags:: metaslab

Enable performance-based metaslab group biasing

metaslab_preload_enabled

Versions:

v0.6 - master

Platforms:

Linux, FreeBSD

Type:

int

Default:

1

Range:

`0`	do not preload metaslab info
`1`	preload up to 3 metaslabs

Change:

Dynamic

Tags:

allocation, metaslab

Preload potential metaslabs during reassessment

When to change: When testing metaslab allocation

Notes: Enable metaslab group preloading. Each top-level vdev has a metaslab group. By default, up to 3 copies of metadata can exist and are distributed across multiple top-level vdevs. metaslab_preload_enabled allows the corresponding metaslabs to be preloaded, thus improving allocation efficiency.

metaslab_preload_limit

Versions:: v2.2 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 10
Change:: Dynamic
Tags:: metaslab

Max number of metaslabs per group to preload

metaslab_preload_pct

Versions:: v2.2 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 50
Change:: Dynamic
Tags:: metaslab, SPA

Percentage of CPUs to run a metaslab preload taskq

metaslab_trace_enabled

Versions:: master
Platforms:: Linux, FreeBSD
Type:: int
Change:: Dynamic
Tags:: metaslab

Enable metaslab allocation tracing

metaslab_trace_max_entries

Versions:: master
Platforms:: Linux, FreeBSD
Type:: u64
Change:: Dynamic
Tags:: metaslab

Maximum entries for metaslab allocation tracing

metaslab_unload_delay

Versions:: v2.0 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 32
Change:: Dynamic
Tags:: delay, metaslab

Delay in txgs after metaslab was last used before unloading

metaslab_unload_delay_ms

Versions:: v2.0 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 600000
Change:: Dynamic
Tags:: delay, metaslab

Delay in milliseconds after metaslab was last used before unloading

metaslabs_per_vdev

Versions:: v0.6 - v0.7
Platforms:: Linux, FreeBSD
Type:: int
Default:: 200
Range:: 16 to UINT64_MAX
Change:: Dynamic
Tags:: allocation, metaslab, vdev

Divide added vdev into approximately (but no more than) this number of metaslabs

When to change: When testing metaslab allocation

raidz_expand_max_copy_bytes

Versions:: v2.3 - master
Platforms:: Linux, FreeBSD
Type:: ulong
Default:: 160MB
Change:: Dynamic
Tags:: raidz, vdev

Max amount of concurrent i/o for RAIDZ expansion

raidz_expand_max_reflow_bytes

Versions:: v2.3 - master
Platforms:: Linux, FreeBSD
Type:: ulong
Default:: 0
Change:: Dynamic
Tags:: raidz, vdev

For testing, pause RAIDZ expansion after reflowing this many bytes

raidz_io_aggregate_rows

Versions:: v2.3 - master
Platforms:: Linux, FreeBSD
Type:: ulong
Default:: 4
Change:: Dynamic
Tags:: raidz, vdev

For expanded RAIDZ, aggregate reads that have more rows than this

reference_history

Versions:: v2.1 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 3
Change:: Dynamic
Tags:: refcount

Maximum reference holders being tracked

reference_tracking_enable

Versions:: v2.1 - master
Platforms:: Linux, FreeBSD
Type:: int
Default:: 0
Change:: Dynamic
Tags:: refcount

Track reference holders to refcount_t objects

send_holes_without_birth_time

Versions:: v0.7 - master
Platforms:: Linux, FreeBSD
Type:: int
Default:: 1
Change:: Dynamic
Tags:: DMU, dmu_traverse, send

Ignore hole_birth txg for zfs send

Notes: Renamed from ignore_hole_birth in v2.4.0.

spa_asize_inflation

Versions:: v0.6 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 24
Range:: 1 to 24
Change:: Dynamic
Tags:: allocation, SPA

SPA size estimate multiplication factor

When to change: If the allocation requirements for the workload are well known and quotas are used

Notes: Multiplication factor used to estimate actual disk consumption from the size of data being written. The default value is a worst case estimate, but lower values may be valid for a given pool depending on its configuration. Pool administrators who understand the factors involved may wish to specify a more realistic inflation factor, particularly if they operate close to quota or capacity limits. The worst case space requirement for allocation is single-sector max-parity RAIDZ blocks, in which case the space requirement is exactly 4 times the size, accounting for a maximum of 3 parity blocks. This is added to the maximum number of ZFS copies parameter (copies max=3). Additional space is required if the block could impact deduplication tables. Altogether, the worst case is 24. If the estimation is not correct, then quotas or out-of-space conditions can lead to optimistic expectations of the ability to allocate. Applications are typically not prepared to deal with such failures and can misbehave.

spa_config_path

Versions:: v0.6 - master
Platforms:: Linux, FreeBSD
Type:: string
Change:: Prior to module load
Tags:: import, SPA, spa_config

SPA config file (/etc/zfs/zpool.cache)

When to change: If creating a non-standard distribution and the cachefile property is inconvenient

Notes: By default, the zpool import command searches for pool information in the zpool.cache file. If the pool to be imported has an entry in zpool.cache then the devices do not have to be scanned to determine if they are pool members. The path to the cache file is spa_config_path. For more information on zpool import and the -o cachefile and -d options, see the man page for zpool(8) See also zfs_autoimport_disable (removed after v2.3)

spa_cpus_per_allocator

Versions:: v2.3 - master
Platforms:: Linux, FreeBSD
Type:: int
Default:: 4
Change:: Dynamic
Tags:: SPA

Minimum number of CPUs per allocators

spa_flush_txg_time

Versions:: v2.4 - master
Platforms:: Linux, FreeBSD
Type:: uint
Change:: Dynamic
Tags:: SPA

How frequently the TXG timestamps database should be flushed to disk (in seconds)

spa_load_print_vdev_tree

Versions:

v0.8 - master

Platforms:

Linux, FreeBSD

Type:

int

Default:

0

Range:

`0`	do not print pool configuration in logs
`1`	print pool configuration in logs

Change:

Dynamic

Tags:

import, SPA, vdev

Print vdev tree to zfs_dbgmsg during pool import

When to change: troubleshooting pool import failures

Notes: spa_load_print_vdev_tree enables printing of the attempted pool import’s vdev tree to kernel message to the ZFS debug message log /proc/spl/kstat/zfs/dbgmsg Both the provided vdev tree and MOS vdev tree are printed, which can be useful for debugging problems with the zpool cachefile

spa_load_verify_data

Versions:

v0.6 - master

Platforms:

Linux, FreeBSD

Type:

int

Default:

1

Range:

`0`	do not verify data upon pool import
`1`	verify pool data upon import

Change:

Dynamic

Tags:

allocation, SPA

Set to traverse data on pool import

When to change: At the risk of data integrity, to speed extreme import of large pool

Notes: An extreme rewind import (see zpool import -X) normally performs a full traversal of all blocks in the pool for verification. If this parameter is set to 0, the traversal skips non-metadata blocks. It can be toggled once the import has started to stop or start the traversal of non-metadata blocks. See also spa_load_verify_metadata.

spa_load_verify_maxinflight

Versions:: v0.6 - v0.7
Platforms:: Linux, FreeBSD
Type:: int
Default:: 10000
Range:: 1 to MAX_INT
Change:: Dynamic
Tags:: import, SPA

Max concurrent traversal I/Os while verifying pool during import -X

When to change: During an extreme rewind import, to match the concurrent I/O capabilities of the pool devices

Notes: Maximum number of concurrent I/Os during the data verification performed during an extreme rewind import (see zpool import -X)

spa_load_verify_metadata

Versions:

v0.6 - master

Platforms:

Linux, FreeBSD

Type:

int

Default:

1

Range:

`0`	do not verify metadata upon pool import
`1`	verify pool metadata upon import

Change:

Dynamic

Tags:

import, metadata, SPA

Set to traverse metadata on pool import

When to change: At the risk of data integrity, to speed extreme import of large pool

Notes: An extreme rewind import (see zpool import -X) normally performs a full traversal of all blocks in the pool for verification. If this parameter is set to 0, the traversal is not performed. It can be toggled once the import has started to stop or start the traversal. See spa_load_verify_data

spa_load_verify_shift

Versions:: v0.8 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 4
Range:: 1 to MAX_INT
Change:: Dynamic
Tags:: ARC, import, SPA

log2 fraction of arc that can be used by inflight I/Os when verifying pool during import

When to change: troubleshooting pool import on large memory machines

Notes: spa_load_verify_shift sets the fraction of ARC that can be used by inflight I/Os when verifying the pool during import. This value is a “shift” representing the fraction of ARC target size (grep -w c /proc/spl/kstat/zfs/arcstats). The ARC target size is shifted to the right. Thus a value of ‘2’ results in the fraction = 1/4, while a value of ‘4’ results in the fraction = 1/8. For large memory machines, pool import can consume large amounts of ARC: much larger than the value of maxinflight. This can result in spa_load_verify_maxinflight (removed after v0.7) having a value of 0 causing the system to hang. Setting spa_load_verify_shift can reduce this limit and allow importing without hanging.

spa_note_txg_time

Versions:: v2.4 - master
Platforms:: Linux, FreeBSD
Type:: uint
Change:: Dynamic
Tags:: SPA

How frequently TXG timestamps are stored internally (in seconds)

spa_num_allocators

Versions:: v2.3 - master
Platforms:: Linux, FreeBSD
Type:: int
Default:: 4
Change:: Dynamic
Tags:: SPA

Number of allocators per spa

spa_slop_shift

Versions:: v0.6 - master
Platforms:: Linux, FreeBSD
Type:: int
Default:: 5
Range:: 1 to MAX_INT, however the practical upper limit is 15 for a system with 4TB of RAM
Change:: Dynamic
Tags:: allocation, SPA

Reserved free space in pool

When to change: For large pools, when 3.2% may be too conservative and more usable space is desired, consider increasing spa_slop_shift

Notes: Normally, the last 3.2% (1/(2^spa_slop_shift)) of pool space is reserved to ensure the pool doesn’t run completely out of space, due to unaccounted changes (e.g. to the MOS). This also limits the worst-case time to allocate space. When less than this amount of free space exists, most ZPL operations (e.g. write, create) return error:no space (ENOSPC). Changing spa_slop_shift affects the currently loaded ZFS module and all imported pools. spa_slop_shift is not stored on disk. Beware when importing full pools on systems with larger spa_slop_shift can lead to over-full conditions. The minimum SPA slop space is limited to 128 MiB. The maximum SPA slop space is limited to 128 GiB.

spa_upgrade_errlog_limit

Versions:: v2.2 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 0
Change:: Dynamic
Tags:: SPA

Limit the number of errors which will be upgraded to the new on-disk error log when enabling head_errlog

spl_hostid

Versions:: v0.8 - master
Platforms:: Linux
Type:: ulong
Default:: 0
Range:: 0=ignore hostid, 1 to 4,294,967,295 (32-bits or 0xffffffff)
Change:: Dynamic
Tags:: generic, hostid, MMP, SPL

The system hostid.

When to change: to uniquely identify a system when vdevs can be shared across multiple systems

Notes: spl_hostid is a unique system id number. It originated in Sun’s products where most systems had a unique id assigned at the factory. This assignment does not exist in modern hardware. In ZFS, the hostid is stored in the vdev label and can be used to determine if another system had imported the pool. When set spl_hostid can be used to uniquely identify a system. By default this value is set to zero which indicates the hostid is disabled. It can be explicitly enabled by placing a unique non-zero value in the file shown in spl_hostid_path

spl_hostid_path

Versions:: v0.8 - master
Platforms:: Linux
Type:: charp
Change:: Prior to module load
Tags:: generic, hostid, MMP, SPL

The system hostid file (/etc/hostid)

When to change: when creating a new ZFS distribution where the default value is inappropriate

Notes: spl_hostid_path is the path name for a file that can contain a unique hostid. For testing purposes, spl_hostid_path can be overridden by the ZFS_HOSTID environment variable.

spl_kmem_alloc_max

Versions:

v0.8 - master

Platforms:

Linux

Type:

uint

Default:

v2.1 - master	`KMALLOC_MAX_SIZE`
v0.8 - v2.0	`KMALLOC_MAX_SIZE/4`

Change:

Dynamic

Tags:

kmem, memory, SPL

Maximum size in bytes for a kmem_alloc()

Notes: Large kmem_alloc() allocations fail if they exceed KMALLOC_MAX_SIZE, as determined by the kernel source. Allocations which are marginally smaller than this limit may succeed but should still be avoided due to the expense of locating a contiguous range of free pages. Therefore, a maximum kmem size with reasonable safely margin of 4x is set. kmem_alloc() allocations larger than this maximum will quickly fail. vmem_alloc() allocations less than or equal to this value will use kmalloc(), but shift to vmalloc() when exceeding this value.

spl_kmem_alloc_warn

Versions:: v0.8 - master
Platforms:: Linux
Type:: uint
Default:: 32768
Range:: 0=disable the warnings,
Change:: Dynamic
Tags:: kmem, memory, SPL

Warning threshold in bytes for a kmem_alloc()

When to change: developers are encouraged lower when testing so any new, large allocations are quickly caught

Notes: As a general rule kmem_alloc() allocations should be small, preferably just a few pages since they must by physically contiguous. Therefore, a rate limited warning is printed to the console for any kmem_alloc() which exceeds the threshold spl_kmem_alloc_warn The default warning threshold is set to eight pages but capped at 32K to accommodate systems using large pages. This value was selected to be small enough to ensure the largest allocations are quickly noticed and fixed. But large enough to avoid logging any warnings when a allocation size is larger than optimal but not a serious concern. Since this value is tunable, developers are encouraged to set it lower when testing so any new largish allocations are quickly caught. These warnings may be disabled by setting the threshold to zero.

spl_kmem_cache_expire

Versions:: v0.8
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 0x02
Range:: 0x01 - Aging (illumos), 0x02 - Low memory (Linux)
Change:: Dynamic
Tags:: kmem, kmem_cache, memory, SPL

By age (0x1) or low memory (0x2)

Notes: Cache expiration is part of default illumos cache behavior. The idea is that objects in magazines which have not been recently accessed should be returned to the slabs periodically. This is known as cache aging and when enabled objects will be typically returned after 15 seconds. On the other hand Linux slabs are designed to never move objects back to the slabs unless there is memory pressure. This is possible because under Linux the cache will be notified when memory is low and objects can be released. By default only the Linux method is enabled. It has been shown to improve responsiveness on low memory systems and not negatively impact the performance of systems with more memory. This policy may be changed by setting the spl_kmem_cache_expire bit mask as follows, both policies may be enabled concurrently.

spl_kmem_cache_kmem_limit

Versions:: v0.8
Platforms:: Linux, FreeBSD
Type:: uint
Default:: PAGE_SIZE/4
Change:: Dynamic
Tags:: kmem, kmem_cache, memory, SPL

Objects less than N bytes use the kmalloc

Notes: Depending on the size of a memory cache object it may be backed by kmalloc() or vmalloc() memory. This is because the size of the required allocation greatly impacts the best way to allocate the memory. When objects are small and only a small number of memory pages need to be allocated, ideally just one, then kmalloc() is very efficient. However, allocating multiple pages with kmalloc() gets increasingly expensive because the pages must be physically contiguous. For this reason we shift to vmalloc() for slabs of large objects which which removes the need for contiguous pages. vmalloc() cannot be used in all cases because there is significant locking overhead involved. This function takes a single global lock over the entire virtual address range which serializes all allocations. Using slightly different allocation functions for small and large objects allows us to handle a wide range of object sizes. The spl_kmem_cache_kmem_limit value is used to determine this cutoff size. One quarter of the kernel’s compiled PAGE_SIZE is used as the default value because spl_kmem_cache_obj_per_slab defaults to 8. With these default values, at most two contiguous pages are allocated.

spl_kmem_cache_kmem_threads

Versions:: v0.8 - master
Platforms:: Linux
Type:: uint
Default:: 4
Range:: 1 to MAX_INT
Change:: Prior to module load
Tags:: CPU, kmem, kmem_cache, memory, SPL

Number of spl_kmem_cache threads

When to change: read-only

Notes: spl_kmem_cache_kmem_threads shows the current number of spl_kmem_cache threads. This task queue is responsible for allocating new slabs for use by the kmem caches. For the majority of systems and workloads only a small number of threads are required.

spl_kmem_cache_magazine_size

Versions:: v0.8 - master
Platforms:: Linux
Type:: uint
Default:: 0
Range:: 0=automatically scale magazine size, otherwise 2 to 256
Change:: Prior to module load
Tags:: CPU, kmem, kmem_cache, memory, SPL

Default magazine size (2-256), set automatically (0)

Notes: spl_kmem_cache_magazine_size shows the current . Cache magazines are an optimization designed to minimize the cost of allocating memory. They do this by keeping a per-cpu cache of recently freed objects, which can then be reallocated without taking a lock. This can improve performance on highly contended caches. However, because objects in magazines will prevent otherwise empty slabs from being immediately released this may not be ideal for low memory machines. For this reason spl_kmem_cache_magazine_size can be used to set a maximum magazine size. When this value is set to 0 the magazine size will be automatically determined based on the object size. Otherwise magazines will be limited to 2-256 objects per magazine (eg per CPU). Magazines cannot be disabled entirely in this implementation.

spl_kmem_cache_max_size

Versions:: v0.8 - master
Platforms:: Linux
Type:: uint
Default:: 32
Change:: Dynamic
Tags:: kmem, kmem_cache, memory, SPL

Maximum size of slab in MB

Notes: spl_kmem_cache_max_size is the maximum size of a kmem cache slab in MiB. This effectively limits the maximum cache object size to spl_kmem_cache_max_size / spl_kmem_cache_obj_per_slab Kmem caches may not be created with object sized larger than this limit.

spl_kmem_cache_obj_per_slab

Versions:: v0.8 - master
Platforms:: Linux
Type:: uint
Default:: 8
Change:: Dynamic
Tags:: kmem, kmem_cache, memory, SPL

Number of objects per slab

Notes: spl_kmem_cache_obj_per_slab is the preferred number of objects per slab in the kmem cache. In general, a larger value will increase the caches memory footprint while decreasing the time required to perform an allocation. Conversely, a smaller value will minimize the footprint and improve cache reclaim time but individual allocations may take longer.

spl_kmem_cache_obj_per_slab_min

Versions:: v0.8
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 1
Change:: Dynamic
Tags:: kmem, kmem_cache, memory, SPL

Minimal number of objects per slab

When to change: debugging kmem cache operations

Notes: spl_kmem_cache_obj_per_slab_min is the minimum number of objects allowed per slab. Normally slabs will contain spl_kmem_cache_obj_per_slab objects but for caches that contain very large objects it’s desirable to only have a few, or even just one, object per slab.

spl_kmem_cache_reclaim

Versions:

v0.8 - v2.1

Platforms:

Linux

Type:

uint

Default:

0

Range:

`0`	enable rapid memory reclaim from kmem caches
`1`	disable rapid memory reclaim from kmem caches

Change:

Dynamic

Tags:

kmem, kmem_cache, memory, SPL

Single reclaim pass (0x1)

Notes: spl_kmem_cache_reclaim prevents Linux from being able to rapidly reclaim all the memory held by the kmem caches. This may be useful in circumstances where it’s preferable that Linux reclaim memory from some other subsystem first. Setting spl_kmem_cache_reclaim increases the likelihood out of memory events on a memory constrained system.

spl_kmem_cache_slab_limit

Versions:: v0.8 - master
Platforms:: Linux
Type:: uint
Default:: 16384
Change:: Dynamic
Tags:: kmem, kmem_cache, memory, SPL

Objects less than N bytes use the Linux slab

Notes: For small objects the Linux slab allocator should be used to make the most efficient use of the memory. However, large objects are not supported by the Linux slab allocator and therefore the SPL implementation is preferred. spl_kmem_cache_slab_limit is used to determine the cutoff between a small and large object. Objects of spl_kmem_cache_slab_limit or smaller will be allocated using the Linux slab allocator, large objects use the SPL allocator. A cutoff of 16 KiB was determined to be optimal for architectures using 4 KiB pages.

spl_max_show_tasks

Versions:: v0.8 - v2.2
Platforms:: Linux
Type:: uint
Default:: 512
Range:: 0 disables the limit, 1 to MAX_UINT
Change:: Dynamic
Tags:: SPL, taskq

Max number of tasks shown in taskq proc

Notes: spl_max_show_tasks is the limit of tasks per pending list in each taskq shown in /proc/spl/taskq and /proc/spl/taskq-all. Reading the ProcFS files walks the lists with lock held and it could cause a lock up if the list grow too large. If the list is larger than the limit, the string "(truncated)" is printed.

spl_panic_halt

Versions:

v0.8 - master

Platforms:

Linux

Type:

uint

Default:

0

Range:

`0`	halt thread upon assertion
`1`	panic kernel upon assertion

Change:

Dynamic

Tags:

debug, panic, SPL

Cause kernel panic on assertion failures

When to change: when debugging assertions and kernel core dumps are desired

Notes: spl_panic_halt enables kernel panic upon assertion failures. When not enabled, the asserting thread is halted to facilitate further debugging.

spl_schedule_hrtimeout_slack_us

Versions:: v0.8 - master
Platforms:: Linux
Type:: uint
Default:: 0
Change:: Dynamic
Tags:: SPL

schedule_hrtimeout_range() delta/slack value in us, default(0)

spl_taskq_kick

Versions:: v0.8 - master
Platforms:: Linux
Type:: uint
Default:: 0
Change:: Dynamic
Tags:: SPL, taskq

Write nonzero to kick stuck taskqs to spawn more threads

When to change: See description above

Notes: Upon writing a non-zero value to spl_taskq_kick, all taskqs are scanned. If any taskq has a pending task more than 5 seconds old, the taskq spawns more threads. This can be useful in rare deadlock situations caused by one or more taskqs not spawning a thread when it should.

spl_taskq_thread_bind

Versions:

v0.8 - master

Platforms:

Linux

Type:

int

Default:

0

Range:

`0`	taskqs are not bound to specific CPUs
`1`	taskqs are bound to CPUs

Change:

Dynamic

Tags:

CPU, SPL, taskq

Bind taskq thread to CPU by default

When to change: when debugging CPU scheduling options

Notes: spl_taskq_thread_bind enables binding taskq threads to specific CPUs, distributed evenly over the available CPUs. By default, this behavior is disabled to allow the Linux scheduler the maximum flexibility to determine where a thread should run.

spl_taskq_thread_dynamic

Versions:

v0.8 - master

Platforms:

Linux

Type:

int

Default:

1

Range:

`0`	taskq threads are not dynamic
`1`	taskq threads are dynamically created and destroyed

Change:

Prior to module load

Tags:

SPL, taskq

Allow dynamic taskq threads

When to change: disable for performance analysis or troubleshooting

Notes: spl_taskq_thread_dynamic enables taskqs to set the TASKQ_DYNAMIC flag will by default create only a single thread. New threads will be created on demand up to a maximum allowed number to facilitate the completion of outstanding tasks. Threads which are no longer needed are promptly destroyed. By default this behavior is enabled but it can be d. See also zfs_zil_clean_taskq_nthr_pct, zio_taskq_batch_pct

spl_taskq_thread_priority

Versions:

v0.8 - master

Platforms:

Linux

Type:

int

Default:

1

Range:

`0`	taskq threads use the default Linux kernel thread priority
`1`

Change:

Dynamic

Tags:

CPU, SPL, taskq

Allow non-default priority for taskq threads

When to change: when troubleshooting CPU scheduling-related performance issues

Notes: spl_taskq_thread_priority allows newly created taskq threads to set a non-default scheduler priority. When enabled the priority specified when a taskq is created will be applied to all threads created by that taskq. When disabled all threads will use the default Linux kernel thread priority.

spl_taskq_thread_sequential

Versions:: v0.8 - master
Platforms:: Linux
Type:: uint
Default:: 4
Range:: 1 to MAX_INT
Change:: Dynamic
Tags:: CPU, SPL, taskq

Create new taskq threads after N sequential tasks

Notes: spl_taskq_thread_sequential is the number of items a taskq worker thread must handle without interruption before requesting a new worker thread be spawned. spl_taskq_thread_sequential controls how quickly taskqs ramp up the number of threads processing the queue. Because Linux thread creation and destruction are relatively inexpensive a small default value has been selected. Thus threads are created aggressively, which is typically desirable. Increasing this value results in a slower thread creation rate which may be preferable for some configurations.

spl_taskq_thread_timeout_ms

Versions:: v2.2 - master
Platforms:: Linux
Type:: uint
Default:: 5000
Change:: Dynamic
Tags:: SPL, taskq

Minimum idle threads exit interval for dynamic taskqs

vdev_file_logical_ashift

Versions:: v2.0 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 9
Change:: Dynamic
Tags:: vdev, vdev_file

Logical ashift for file-based devices

vdev_file_physical_ashift

Versions:: v2.0 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 9
Change:: Dynamic
Tags:: vdev, vdev_file

Physical ashift for file-based devices

vdev_raidz_outlier_check_interval_ms

Versions:: v2.4 - master
Platforms:: Linux, FreeBSD
Type:: u64
Default:: 1000
Change:: Dynamic
Tags:: raidz, vdev

Interval to check for slow raidz/draid children

vdev_raidz_outlier_insensitivity

Versions:: v2.4 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 50
Change:: Dynamic
Tags:: raidz, vdev

How insensitive the slow raidz/draid child check should be

vdev_read_sit_out_secs

Versions:: v2.4 - master
Platforms:: Linux, FreeBSD
Type:: ulong
Default:: 600
Change:: Dynamic
Tags:: raidz, vdev

Raidz/draid slow disk sit out time period in seconds

vdev_removal_max_span

Versions:: v0.8 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 32768
Range:: 0 to MAX_INT
Change:: Dynamic
Tags:: vdev, vdev_removal

Largest span of free chunks a remap segment can span

Notes: During top-level vdev removal, chunks of data are copied from the vdev which may include free space in order to trade bandwidth for IOPS. vdev_removal_max_span sets the maximum span of free space included as unnecessary data in a chunk of copied data.

vdev_validate_skip

Versions:

v0.8 - master

Platforms:

Linux, FreeBSD

Type:

int

Default:

0

Range:

`0`	validate labels during pool import
`1`	do not validate vdev labels during pool import

Change:

Dynamic

Tags:

vdev

Bypass vdev_validate()

When to change: do not change

Notes: vdev_validate_skip disables label validation steps during pool import. Changing is not recommended unless you know what you are doing and are recovering a damaged label.

zap_iterate_prefetch

Versions:: v0.8 - master
Platforms:: Linux, FreeBSD
Type:: int
Default:: 1
Change:: Dynamic
Tags:: prefetch, ZAP, zap_fat

When iterating ZAP object, prefetch it

zap_micro_max_size

Versions:: v2.2 - master
Platforms:: Linux, FreeBSD
Type:: int
Default:: 131072
Change:: Dynamic
Tags:: ZAP

Maximum micro ZAP size before converting to a fat ZAP, in bytes (max 1M)

zap_shrink_enabled

Versions:: v2.3 - master
Platforms:: Linux, FreeBSD
Type:: int
Default:: 1
Change:: Dynamic
Tags:: ZAP, zap_fat

Enable ZAP shrinking

zfetch_array_rd_sz

Versions:: v0.6 - v2.1
Platforms:: Linux, FreeBSD
Type:: ulong
Default:: 1048576
Range:: 0 to MAX_ULONG
Change:: Dynamic
Tags:: DMU, dmu_zfetch, prefetch

Number of bytes in a array_read

When to change: To allow prefetching when using large block sizes

Notes: If prefetching is enabled, do not prefetch blocks larger than zfetch_array_rd_sz size.

zfetch_block_cap

Versions:: v0.6
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 256
Change:: Dynamic
Tags:: DMU, dmu_zfetch

Max number of blocks to fetch at a time

zfetch_hole_shift

Versions:: v2.2 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 2
Change:: Dynamic
Tags:: DMU, dmu_zfetch, prefetch

Max log2 fraction of holes in a stream

zfetch_max_distance

Versions:

v0.7 - master

Platforms:

Linux, FreeBSD

Type:

uint

Default:

v2.1 - master	`67108864`
v0.7 - v2.0	`8,388,608`

Change:

Dynamic

Tags:

DMU, dmu_zfetch, prefetch

Max bytes to prefetch per stream

When to change: Consider increasing read workloads that use large blocks and exhibit high prefetch hit ratios

zfetch_max_idistance

Versions:: v2.0 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 67108864
Change:: Dynamic
Tags:: DMU, dmu_zfetch, prefetch

Max bytes to prefetch indirects for per stream

zfetch_max_reorder

Versions:: v2.2 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 16777216
Change:: Dynamic
Tags:: DMU, dmu_zfetch, prefetch

Max request reorder distance within a stream

zfetch_max_sec_reap

Versions:: v2.1 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 2
Change:: Dynamic
Tags:: DMU, dmu_zfetch, prefetch

Max time before stream delete

zfetch_max_streams

Versions:: v0.6 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 8
Range:: 1 to MAX_UINT
Change:: Dynamic
Tags:: DMU, dmu_zfetch, prefetch

Max number of streams per zfetch

When to change: If the workload benefits from prefetching and has more than zfetch_max_streams concurrent reader threads

Notes: Maximum number of prefetch streams per file. For version v0.7.0 and later, when prefetching small files the number of prefetch streams is automatically reduced below to prevent the streams from overlapping.

zfetch_min_distance

Versions:: v2.1 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 4194304
Change:: Dynamic
Tags:: DMU, dmu_zfetch, prefetch

Min bytes to prefetch per stream

zfetch_min_sec_reap

Versions:

v0.6 - master

Platforms:

Linux, FreeBSD

Type:

uint

Default:

v2.1 - master	`1`
v0.6 - v2.0	`2`

Range:

0 to MAX_UINT

Change:

Dynamic

Tags:

DMU, dmu_zfetch, prefetch

Min time before stream reclaim

When to change: To test prefetch efficiency

Notes: Prefetch streams that have been accessed in zfetch_min_sec_reap seconds are automatically stopped.

zfs_abd_scatter_enabled

Versions:

v0.7 - master

Platforms:

Linux

Type:

int

Default:

1

Range:

`0`	use linear allocation only
`1`	allow scatter/gather

Change:

Dynamic

Tags:

ABD, memory

Toggle whether ABD allocations must be linear.

When to change: Testing ABD

Verification: ABD statistics are observable in /proc/spl/kstat/zfs/abdstats. Slab allocations are observable in /proc/slabinfo

Notes: zfs_abd_scatter_enabled controls the ARC Buffer Data (ABD) scatter/gather feature. When disabled, the legacy behaviour is selected using linear buffers. For linear buffers, all the data in the ABD is stored in one contiguous buffer in memory (from a zio_[data_]buf_* kmem cache). When enabled (default), the data in the ABD is split into equal-sized chunks (from the abd_chunk_cache kmem_cache), with pointers to the chunks recorded in an array at the end of the ABD structure. This allows more efficient memory allocation for buffers, especially when large recordsizes are used.

zfs_abd_scatter_max_order

Versions:

v0.7 - master

Platforms:

Linux

Type:

uint

Default:

v2.1 - master	`MAX_ORDER-1`
v2.0	`10 at the time of this writing`

Range:

1 to 10 (upper limit is hardware-dependent)

Change:

Dynamic

Tags:

ABD, memory

Maximum order allocation used for a scatter ABD.

When to change: Testing ABD features

Verification: ABD statistics are observable in /proc/spl/kstat/zfs/abdstats

Notes: zfs_abd_scatter_max_order sets the maximum order for physical page allocation when ABD is enabled (see zfs_abd_scatter_enabled) See also Buddy Memory Allocation in the Linux kernel documentation.

zfs_abd_scatter_min_size

Versions:: v0.8 - master
Platforms:: Linux
Type:: int
Default:: 1536
Range:: 0 to MAX_INT
Change:: Dynamic
Tags:: ARC

Minimum size of scatter allocations.

When to change: debugging memory allocation, especially for large pages

Notes: zfs_abd_scatter_min_size changes the ARC buffer data (ABD) allocator’s threshold for using linear or page-based scatter buffers. Allocations smaller than zfs_abd_scatter_min_size use linear ABDs. Scatter ABD’s use at least one page each, so sub-page allocations waste some space when allocated as scatter allocations. For example, 2KB scatter allocation wastes half of each page. Using linear ABD’s for small allocations results in slabs containing many allocations. This can improve memory efficiency, at the expense of more work for ARC evictions attempting to free pages, because all the buffers on one slab need to be freed in order to free the slab and its underlying pages. Typically, 512B and 1KB kmem caches have 16 buffers per slab, so it’s possible for them to actually waste more memory than scatter allocations: - one page per buf = wasting 3/4 or 7/8 - one buf per slab = wasting 15/16 Spill blocks are typically 512B and are heavily used on systems running selinux with the default dnode size and the xattr=sa property set. By default, linear allocations for 512B and 1KB, and scatter allocations for larger (>= 1.5KB) allocation requests.

zfs_active_allocator

Versions:: v2.3 - master
Platforms:: Linux, FreeBSD
Type:: charp
Default:: dynamic
Change:: Dynamic
Tags:: metaslab

SPA active allocator

zfs_admin_snapshot

Versions:

v0.6 - master

Platforms:

Linux

Type:

int

Default:

0

Range:

`0`	do not allow snapshot manipulation via the filesystem
`1`	allow snapshot manipulation via the filesystem

Change:

Dynamic

Tags:

ctldir, filesystem, snapshot

Enable mkdir/rmdir/mv in .zfs/snapshot

Notes: Allow the creation, removal, or renaming of entries in the .zfs/snapshot subdirectory to cause the creation, destruction, or renaming of snapshots. When enabled this functionality works both locally and over NFS exports which have the “no_root_squash” option set.

zfs_allow_redacted_dataset_mount

Versions:: v2.0 - master
Platforms:: Linux, FreeBSD
Type:: int
Default:: 0
Change:: Dynamic
Tags:: DSL, dsl_dataset

Allow mounting of redacted datasets

zfs_arc_average_blocksize

Versions:: v0.6 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 8192
Range:: 512 to 16,777,216
Change:: Prior to module load
Tags:: ARC, memory

Target average block size

When to change: For workloads where the known average blocksize is larger, increasing zfs_arc_average_blocksize can reduce memory usage

Notes: The ARC’s buffer hash table is sized based on the assumption of an average block size of zfs_arc_average_blocksize. The default of 8 KiB uses approximately 1 MiB of hash table per 1 GiB of physical memory with 8-byte pointers.

zfs_arc_dnode_limit

Versions:: v0.7 - master
Platforms:: Linux, FreeBSD
Type:: u64
Default:: 0
Change:: Dynamic
Tags:: ARC

Minimum bytes of dnodes in ARC

When to change: Consider increasing if arc_prune is using excessive system time and /proc/spl/kstat/zfs/arcstats shows arc_dnode_size is near or over arc_dnode_limit

Notes: When the number of bytes consumed by dnodes in the ARC exceeds zfs_arc_dnode_limit bytes, demand for new metadata can take from the space consumed by dnodes. The default value 0, indicates that a percent which is based on zfs_arc_dnode_limit_percent of the ARC meta buffers that may be used for dnodes.

zfs_arc_dnode_limit_percent

Versions:: v0.7 - master
Platforms:: Linux, FreeBSD
Type:: u64
Default:: 10
Range:: 0 to 100
Change:: Dynamic
Tags:: ARC

Percent of ARC meta buffers for dnodes

When to change: Consider increasing if arc_prune is using excessive system time and /proc/spl/kstat/zfs/arcstats shows arc_dnode_size is near or over arc_dnode_limit

Notes: Percentage of ARC metadata space that can be used for dnodes. The value calculated for zfs_arc_dnode_limit_percent can be overridden by zfs_arc_dnode_limit.

zfs_arc_dnode_reduce_percent

Versions:: v0.7 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 10
Range:: 0 to 100
Change:: Dynamic
Tags:: ARC

Percentage of excess dnodes to try to unpin

When to change: Testing dnode cache efficiency

Notes: Percentage of ARC dnodes to try to evict in response to demand for non-metadata when the number of bytes consumed by dnodes exceeds zfs_arc_dnode_limit.

zfs_arc_evict_batch_limit

Versions:: v2.0 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 10
Range:: 1 to INT_MAX
Change:: Dynamic
Tags:: ARC

The number of headers to evict per sublist before moving to the next

When to change: Testing ARC multilist features

Notes: Number ARC headers to evict per sublist before proceeding to another sublist. This batch-style operation prevents entire sublists from being evicted at once but comes at a cost of additional unlocking and locking.

zfs_arc_evict_batches_limit

Versions:: v2.4 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 5
Change:: Dynamic
Tags:: ARC

The number of batches to run per parallel eviction task

zfs_arc_evict_threads

Versions:: v2.3 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 0
Change:: Prior to module load
Tags:: ARC

Number of threads to use for ARC eviction.

zfs_arc_eviction_pct

Versions:: v2.0 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 200
Change:: Dynamic
Tags:: ARC

When full, ARC allocation waits for eviction of this % of alloc size

zfs_arc_free_target

Versions:: v2.2 - master
Platforms:: FreeBSD
Type:: uint
Tags:: ARC, memory

Desired number of free pages below which ARC triggers reclaim

When to change: When the ARC is not releasing memory fast enough to keep up with other system demands

Notes: zfs_arc_free_target is the desired number of free pages below which the ARC triggers reclaim. Initialized at boot to the kernel’s vm.v_free_target value. This parameter is FreeBSD-specific and uses pages as its unit, unlike zfs_arc_sys_free which is measured in bytes.

zfs_arc_grow_retry

Versions:

v0.6 - master

Platforms:

Linux, FreeBSD

Type:

uint

Default:

v0.7 - master	`0`
v0.6	`5`

Range:

0=use the built-in default of 5 seconds, otherwise 1 to INT_MAX

Change:

Dynamic

Tags:

ARC, memory

Seconds before growing ARC size

Notes: When the ARC is shrunk due to memory demand, do not retry growing the ARC for zfs_arc_grow_retry seconds. This operates as a damper to prevent oscillating grow/shrink cycles when there is memory pressure. If zfs_arc_grow_retry = 0, the internal default of 5 seconds is used.

zfs_arc_lotsfree_percent

Versions:: v0.6 - master
Platforms:: Linux, FreeBSD
Type:: int
Default:: 10
Range:: 0 to 100
Change:: Dynamic
Tags:: ARC, memory

System free memory I/O throttle in bytes

Notes: Throttle ARC memory consumption, effectively throttling I/O, when free system memory drops below this percentage of total system memory. Setting zfs_arc_lotsfree_percent to 0 disables the throttle. The arcstat_memory_throttle_count counter in /proc/spl/kstat/arcstats can indicate throttle activity.

zfs_arc_max

Versions:: v0.6 - master
Platforms:: Linux, FreeBSD
Type:: u64
Default:: 0
Range:: 67,108,864 to RAM size in bytes
Change:: Dynamic
Tags:: ARC, memory

Maximum ARC size in bytes

When to change: Reduce if ARC competes too much with other applications, increase if ZFS is the primary application and can use more RAM

Verification: c column of zarcstat (arcstat before 2.4.0) or /proc/spl/kstat/zfs/arcstats entry c_max

Notes:

If set to 0, the limit is derived from the amount of memory installed. Since v2.3.0 every platform uses the larger of all_system_memory - 1 GiB and 5/8 x all_system_memory. Before that the platforms differed:

Linux: half of system memory
FreeBSD: the larger of all_system_memory - 1 GiB and 5/8 x all_system_memory

zfs_arc_max can be changed dynamically, with some caveats. It cannot be set back to 0 while running, and reducing it below the current ARC size will not cause the ARC to shrink without memory pressure to induce shrinking.

zfs_arc_meta_adjust_restarts

Versions:: v0.6 - v2.1
Platforms:: Linux, FreeBSD
Type:: int
Default:: 4096
Range:: 0 to INT_MAX
Change:: Dynamic
Tags:: ARC

Limit number of restarts in arc_evict_meta

When to change: Testing ARC metadata adjustment feature

Notes: Removed in v2.2.0 by commit a8d83e2a (“More adaptive ARC eviction”). Replaced by adaptive eviction with zfs_arc_meta_balance. The number of restart passes to make while scanning the ARC attempting the free buffers in order to stay below the zfs_arc_meta_limit.

zfs_arc_meta_balance

Versions:: v2.2 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 500
Change:: Dynamic
Tags:: ARC

Balance between metadata and data on ghost hits.

When to change: When the ARC is not caching enough metadata (or too much) for your workload

Notes: zfs_arc_meta_balance controls how the ARC balances metadata and data caching. When evicted metadata is re-requested (a “ghost hit”), the ARC shifts its target to cache more metadata. This parameter scales the strength of that shift relative to data ghost hits. Higher values give more preference to metadata. The default of 500 means metadata ghost hits have 5x the effect of data ghost hits. A value of 100 weights them equally. This parameter replaced the manual metadata limit tunables (zfs_arc_meta_limit, zfs_arc_meta_min, etc.) that were removed in v2.2.0.

zfs_arc_meta_limit

Versions:: v0.6 - v2.1
Platforms:: Linux, FreeBSD
Type:: ulong
Default:: 0
Range:: 0 to c_max
Change:: Dynamic
Tags:: ARC

Metadata limit for arc size

When to change: For workloads where the metadata to data ratio in the ARC can be changed to improve ARC hit rates

Verification: /proc/spl/kstat/zfs/arcstats entry arc_meta_limit

Notes: Removed in v2.2.0 by commit a8d83e2a (“More adaptive ARC eviction”). Replaced by adaptive eviction with zfs_arc_meta_balance. Sets the maximum allowed size metadata buffers in the ARC. When zfs_arc_meta_limit is reached metadata buffers are reclaimed, even if the overall c_max has not been reached. In version v0.7.0, with a default value = 0, zfs_arc_meta_limit_percent is used to set arc_meta_limit

zfs_arc_meta_limit_percent

Versions:: v0.7 - v2.1
Platforms:: Linux, FreeBSD
Type:: ulong
Default:: 75
Range:: 0 to 100
Change:: Dynamic
Tags:: ARC

Percent of arc size for arc meta limit

When to change: For workloads where the metadata to data ratio in the ARC can be changed to improve ARC hit rates

Verification: /proc/spl/kstat/zfs/arcstats entry arc_meta_limit

Notes: Removed in v2.2.0 by commit a8d83e2a (“More adaptive ARC eviction”). Replaced by adaptive eviction with zfs_arc_meta_balance. Sets the limit to ARC metadata, arc_meta_limit, as a percentage of the maximum size target of the ARC, c_max Prior to version v0.7.0, the zfs_arc_meta_limit was used to set the limit as a fixed size. zfs_arc_meta_limit_percent provides a more convenient interface for setting the limit.

zfs_arc_meta_min

Versions:: v0.6 - v2.1
Platforms:: Linux, FreeBSD
Type:: ulong
Default:: 0
Range:: 16,777,216 to c_max
Change:: Dynamic
Tags:: ARC

Min arc metadata

Verification: /proc/spl/kstat/zfs/arcstats entry arc_meta_min

Notes: Removed in v2.2.0 by commit a8d83e2a (“More adaptive ARC eviction”). Replaced by adaptive eviction with zfs_arc_meta_balance. The minimum allowed size in bytes that metadata buffers may consume in the ARC. This value defaults to 0 which disables a floor on the amount of the ARC devoted meta data. When evicting data from the ARC, if the metadata_size is less than arc_meta_min then data is evicted instead of metadata.

zfs_arc_meta_prune

Versions:: v0.6 - v2.1
Platforms:: Linux, FreeBSD
Type:: int
Default:: 10000
Range:: 0 to INT_MAX
Change:: Dynamic
Tags:: ARC

Meta objects to scan for prune

Notes: Removed in v2.2.0 by commit a8d83e2a (“More adaptive ARC eviction”). Replaced by adaptive eviction with zfs_arc_meta_balance. zfs_arc_meta_prune sets the number of dentries and znodes to be scanned looking for entries which can be dropped. This provides a mechanism to ensure the ARC can honor the arc_meta_limit and reclaim otherwise pinned ARC buffers. Pruning may be required when the ARC size drops to arc_meta_limit because dentries and znodes can pin buffers in the ARC. Increasing this value will cause to dentry and znode caches to be pruned more aggressively and the arc_prune thread becomes more active. Setting zfs_arc_meta_prune to 0 will disable pruning.

zfs_arc_meta_strategy

Versions:

v0.6 - v2.1

Platforms:

Linux, FreeBSD

Type:

int

Default:

1

Range:

`0`	evict metadata only
`1`	also evict data buffers if they can free metadata buffers for eviction

Change:

Dynamic

Tags:

Meta reclaim strategy

When to change: Testing ARC metadata eviction

Notes: Removed in v2.2.0 by commit a8d83e2a (“More adaptive ARC eviction”). Replaced by adaptive eviction with zfs_arc_meta_balance. Defines the strategy for ARC metadata eviction (meta reclaim strategy). A value of 0 (META_ONLY) will evict only the ARC metadata. A value of 1 (BALANCED) indicates that additional data may be evicted if required in order to evict the requested amount of metadata.

zfs_arc_min

Versions:

v0.6 - master

Platforms:

Linux, FreeBSD

Type:

u64

Default:

v0.7 - master	`0`
v0.6	`100`

Range:

33,554,432 to c_max

Change:

Dynamic

Tags:

Minimum ARC size in bytes

When to change: If the primary focus of the system is ZFS, then increasing can ensure the ARC gets a minimum amount of RAM

Verification: /proc/spl/kstat/zfs/arcstats entry c_min

Notes: Minimum ARC size limit. When the ARC is asked to shrink, it will stop shrinking at c_min as tuned by zfs_arc_min.

zfs_arc_min_prefetch_lifespan

Versions:

v0.6 - v0.7

Platforms:

Linux, FreeBSD

Type:

int

Default:

v0.7	`0`
v0.6	`100`

Range:

0 = use default value

Change:

Dynamic

Tags:

ARC, prefetch

Min life of prefetch block

Notes: Removed in v0.8.0 by commit d4a72f238 (“Sequential scrub and resilvers”). Prefetch lifetime logic was reworked as part of the sequential scan changes. Replaced by zfs_arc_min_prefetch_ms and zfs_arc_min_prescient_prefetch_ms. arc_min_prefetch_lifespan is the minimum time for a prefetched block to remain in ARC before it is eligible for eviction.

zfs_arc_min_prefetch_ms

Versions:: v0.8 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 0
Units:: ms
Range:: 0=use the built-in default of 1 second, otherwise 1 to INT_MAX
Change:: Dynamic
Tags:: ARC, prefetch

Min life of prefetch block in ms

Notes: Minimum time prefetched blocks are locked in the ARC. A value of 0 represents the default of 1 second. However, once changed, dynamically setting to 0 will not return to the default.

zfs_arc_min_prescient_prefetch_ms

Versions:: v0.8 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 0
Units:: ms
Range:: 0=use the built-in default of 6 seconds, otherwise 1 to INT_MAX
Change:: Dynamic
Tags:: ARC, prefetch

Min life of prescient prefetched block in ms

Notes: Minimum time “prescient prefetched” blocks are locked in the ARC. These blocks are meant to be prefetched fairly aggressively ahead of the code that may use them. A value of 0 represents the default of 6 seconds. However, once changed, dynamically setting to 0 will not return to the default.

zfs_arc_no_grow_shift

Versions:: v2.2 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 5
Change:: Dynamic
Tags:: ARC

log2(fraction of ARC which must be free to allow growing)

zfs_arc_num_sublists_per_state

Versions:: v0.6
Platforms:: Linux, FreeBSD
Type:: int
Default:: 1 or the number of on-online CPUs, whichever is greater
Change:: Dynamic
Tags:: ARC

Number of sublists used in each of the ARC state lists

zfs_arc_p_aggressive_disable

Versions:: v0.6
Platforms:: Linux, FreeBSD
Type:: int
Change:: Dynamic
Tags:: ARC

disable aggressive arc_p grow

zfs_arc_p_dampener_disable

Versions:

v0.6 - v2.1

Platforms:

Linux, FreeBSD

Type:

int

Default:

1

Range:

`0`	avoid large adjustments
`1`	permit large adjustments

Change:

Dynamic

Tags:

Disable arc_p adapt dampener

When to change: Testing ARC ghost list behaviour

Notes: Removed in v2.2.0 by commit a8d83e2a (“More adaptive ARC eviction”). Replaced by adaptive eviction with zfs_arc_meta_balance. When data is being added to the ghost lists, the MRU target size is adjusted. The amount of adjustment is based on the ratio of the MRU/MFU sizes. When enabled, the ratio is capped to 10, avoiding large adjustments.

zfs_arc_p_min_shift

Versions:

v0.6 - v2.1

Platforms:

Linux, FreeBSD

Type:

int

Default:

v0.7 - v2.1	`0`
v0.6	`4`

Range:

0=use the built-in default of 4, otherwise 1 to INT_MAX

Change:

Dynamic

Tags:

arc_c shift to calc min/max arc_p

Verification: Observe changes to /proc/spl/kstat/zfs/arcstats entry p

Notes: Removed in v2.2.0 by commit a8d83e2a (“More adaptive ARC eviction”). Replaced by adaptive eviction with zfs_arc_meta_balance. arc_p_min_shift is used to shift of ARC target size (/proc/spl/kstat/zfs/arcstats entry c) for calculating both minimum and maximum most recently used (MRU) target size (/proc/spl/kstat/zfs/arcstats entry p) A value of 0 represents the default setting of arc_p_min_shift = 4. However, once changed, dynamically setting zfs_arc_p_min_shift to 0 will not return to the default.

zfs_arc_pc_percent

Versions:: v0.7 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 0
Range:: 0 to 100
Change:: Dynamic
Tags:: ARC, memory

Percent of pagecache to reclaim ARC to

When to change: When using file systems under memory shortfall, if the page scanner causes the ARC to shrink too fast, then adjusting zfs_arc_pc_percent can reduce the shrink rate

Notes: zfs_arc_pc_percent allows ZFS arc to play more nicely with the kernel’s LRU pagecache. It can guarantee that the arc size won’t collapse under scanning pressure on the pagecache, yet still allows arc to be reclaimed down to zfs_arc_min if necessary. This value is specified as percent of pagecache size (as measured by NR_FILE_PAGES) where that percent may exceed 100. This only operates during memory pressure/reclaim.

zfs_arc_prune_task_threads

Versions:: v2.1 - master
Platforms:: Linux, FreeBSD
Type:: int
Default:: 1
Change:: Dynamic
Tags:: ARC

Number of arc_prune threads

zfs_arc_shrink_shift

Versions:

v0.6 - master

Platforms:

Linux, FreeBSD

Type:

uint

Default:

v0.7 - master	`0`
v0.6	`5`

Range:

0=use the built-in default of 7, otherwise 1 to INT_MAX

Change:

Dynamic

Tags:

ARC, memory

log2(fraction of ARC to reclaim)

When to change: During memory shortfall, reducing zfs_arc_shrink_shift increases the rate of ARC shrinkage

Notes: arc_shrink_shift is used to adjust the ARC target sizes when large reduction is required. The current ARC target size, c, and MRU size p can be reduced by by the current size >> arc_shrink_shift. For the default value of 7, this reduces the target by approximately 0.8%. A value of 0 represents the default setting of arc_shrink_shift = 7. However, once changed, dynamically setting arc_shrink_shift to 0 will not return to the default.

zfs_arc_shrinker_limit

Versions:

v2.0 - master

Platforms:

Linux

Type:

int

Default:

v2.3 - master	`0`
v2.0 - v2.2	`10,000`

Change:

Dynamic

Tags:

Limit on number of pages that ARC shrinker can reclaim at once

zfs_arc_shrinker_seeks

Versions:: v2.2 - master
Platforms:: Linux
Type:: int
Default:: 2
Change:: Prior to module load
Tags:: ARC

Relative cost of ARC eviction vs other kernel subsystems

zfs_arc_sys_free

Versions:: v0.6 - master
Platforms:: Linux, FreeBSD
Type:: u64
Default:: 0
Range:: 0 to ULONG_MAX
Change:: Dynamic
Tags:: ARC, memory

System free memory target size in bytes

When to change: Change if more free memory is desired as a margin against memory demand by applications

Notes: zfs_arc_sys_free is the target number of bytes the ARC should leave as free memory on the system. Defaults to the larger of 1/64 of physical memory or 512K. Setting this option to a non-zero value will override the default. However, once changed, dynamically setting zfs_arc_sys_free to 0 will not return to the default.

zfs_async_block_max_blocks

Versions:

v0.8 - master

Platforms:

Linux, FreeBSD

Type:

u64

Default:

v2.2 - master	`UINT64_MAX`
v2.0 - v2.1	`ULONG_MAX (unlimited)`
v0.8	`100,000`

Range:

1 to MAX_ULONG

Change:

Dynamic

Tags:

delete, DMU, DSL, dsl_scan

Max number of blocks freed in one txg

Notes: zfs_async_block_max_blocks limits the number of blocks freed in a single transaction group commit. During deletes of large objects, such as snapshots, the number of freed blocks can cause the DMU to extend txg sync times well beyond zfs_txg_timeout. zfs_async_block_max_blocks is used to limit these effects.

zfs_async_free_zio_wait_interval

Versions:: v2.4 - master
Platforms:: Linux, FreeBSD
Type:: u64
Default:: 2000
Change:: Dynamic
Tags:: DSL, dsl_scan

Wait for pending free I/Os after issuing this many asynchronously

zfs_autoimport_disable

Versions:

v0.6 - v2.3

Platforms:

Linux, FreeBSD

Type:

int

Default:

1

Range:

`0`	read `zpool.cache` at module load
`1`	do not read `zpool.cache` at module load

Change:

Dynamic

Tags:

import, SPA, spa_config

Disable pool import at module load

When to change: Leave as default so that zfs behaves as other Linux kernel modules

Notes: Disable reading zpool.cache file (see spa_config_path) when loading the zfs module.

zfs_bclone_enabled

Versions:: v2.2 - master
Platforms:: Linux, FreeBSD
Type:: int
Default:: 1
Change:: Dynamic
Tags:: vnops

Enable block cloning

zfs_bclone_strict_properties

Versions:: master
Platforms:: Linux, FreeBSD
Type:: int
Default:: 1
Change:: Dynamic
Tags:: vnops

Restrict cross-dataset cloning with different properties

zfs_bclone_wait_dirty

Versions:

v2.2 - master

Platforms:

Linux, FreeBSD

Type:

int

Default:

v2.3 - master	`1`
v2.2	`0`

Change:

Dynamic

Tags:

vnops

Wait for dirty blocks when cloning

zfs_btree_verify_intensity

Versions:: v2.1 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 0
Change:: Dynamic
Tags:: btree

Enable btree verification. Levels above 4 require ZFS be built with debugging

zfs_ccw_retry_interval

Versions:: master
Platforms:: Linux, FreeBSD
Type:: int
Default:: 300
Change:: Dynamic
Tags:: SPA

Configuration cache file write, retry after failure, interval (seconds)

zfs_checksum_events_per_second

Versions:: v0.8 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 20
Range:: zed threshold to MAX_UINT
Change:: Dynamic
Tags:: vdev

Rate limit checksum events to this many checksum errors per second (do not set below ZED threshold).

Notes: zfs_checksum_events_per_second is a rate limit for checksum events. Note that this should not be set below the zed thresholds (currently 10 checksums over 10 sec) or else zed may not trigger any action.

zfs_checksums_per_second

Versions:: v0.7
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 20
Change:: Dynamic
Tags:: checksum, vdev, zed

Rate limit checksum events to this many checksum errors per second (do not set below zedthreshold).

When to change: If processing checksum error events at a higher rate is desired

Notes: Renamed to zfs_checksum_events_per_second in v0.8.0 (commit ad796b8a3). The ZFS Event Daemon (zed) processes events from ZFS. However, it can be overwhelmed by high rates of error reports which can be generated by failing, high-performance devices. zfs_checksums_per_second limits the rate of checksum events reported to zed. Note: do not set this value lower than the SERD limit for checksum in zed. By default, checksum_N = 10 and checksum_T = 10 minutes, resulting in a practical lower limit of 1.

zfs_commit_timeout_pct

Versions:

v0.8 - master

Platforms:

Linux, FreeBSD

Type:

uint

Default:

v2.2 - master	`10`
v0.8 - v2.1	`5%`

Range:

1 to 100

Change:

Dynamic

Tags:

ZIL

ZIL block open timeout percentage

Notes: zfs_commit_timeout_pct controls the amount of time that a log (ZIL) write block (lwb) remains “open” when it isn’t “full” and it has a thread waiting to commit to stable storage. The timeout is scaled based on a percentage of the last lwb latency to avoid significantly impacting the latency of each individual intent log transaction (itx).

zfs_compressed_arc_enabled

Versions:

v0.7 - master

Platforms:

Linux, FreeBSD

Type:

int

Default:

1

Range:

`0`	compressed ARC disabled (legacy behaviour)
`1`	compress ARC data

Change:

Dynamic

Tags:

ABD, ARC, compression

Disable compressed ARC buffers

When to change: Testing ARC compression feature

Verification: raw ARC statistics are observable in /proc/spl/kstat/zfs/arcstats and ARC hit ratios can be observed using zarcstat (arcstat before 2.4.0)

Notes: When compression is enabled for a dataset, later reads of the data can store the blocks in ARC in their on-disk, compressed state. This can increase the effective size of the ARC, as counted in blocks, and thus improve the ARC hit ratio.

zfs_condense_indirect_commit_entry_delay_ms

Versions:: v0.8 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 0
Range:: 0 to MAX_INT
Change:: Dynamic
Tags:: condense, delay, vdev, vdev_indirect, vdev_removal

Used by tests to ensure certain actions happen in the middle of a condense. A maximum value of 1 should be sufficient.

When to change: do not change

Notes: During vdev removal, the vdev indirection layer sleeps for zfs_condense_indirect_commit_entry_delay_ms milliseconds during mapping generation. This parameter is used during automated testing of the ZFS code to improve test coverage.

zfs_condense_indirect_obsolete_pct

Versions:: v2.0 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 25
Change:: Dynamic
Tags:: condense, vdev, vdev_indirect

Minimum obsolete percent of bytes in the mapping to attempt condensing

zfs_condense_indirect_vdevs_enable

Versions:

v0.8 - master

Platforms:

Linux, FreeBSD

Type:

int

Default:

1

Range:

`0`	do not save memory
`1`	save memory by condensing obsolete mapping after vdev removal

Change:

Dynamic

Tags:

condense, vdev, vdev_indirect, vdev_removal

Whether to attempt condensing indirect vdev mappings

Notes: During vdev removal, condensing process is an attempt to save memory by removing obsolete mappings. zfs_condense_indirect_vdevs_enable enables condensing indirect vdev mappings. When set, ZFS attempts to condense indirect vdev mappings if the mapping uses more than zfs_condense_min_mapping_bytes bytes of memory and if the obsolete space map object uses more than zfs_condense_max_obsolete_bytes bytes on disk.

zfs_condense_max_obsolete_bytes

Versions:: v0.8 - master
Platforms:: Linux, FreeBSD
Type:: u64
Default:: 1073741824
Range:: 0 to MAX_ULONG
Change:: Dynamic
Tags:: condense, vdev, vdev_indirect, vdev_removal

Minimum size obsolete spacemap to attempt condensing

When to change: no not change

Notes: After vdev removal, zfs_condense_max_obsolete_bytes sets the limit for beginning the condensing process. Condensing begins if the obsolete space map takes up more than zfs_condense_max_obsolete_bytes of space on disk (logically). The default of 1 GiB is small enough relative to a typical pool that the space consumed by the obsolete space map is minimal. See also zfs_condense_indirect_vdevs_enable

zfs_condense_min_mapping_bytes

Versions:: v0.8 - master
Platforms:: Linux, FreeBSD
Type:: u64
Default:: 131072
Range:: 0 to MAX_ULONG
Change:: Dynamic
Tags:: condense, vdev, vdev_indirect, vdev_removal

Don’t bother condensing if the mapping uses less than this amount of memory

When to change: do not change

Notes: After vdev removal, zfs_condense_min_mapping_bytes is the lower limit for determining when to condense the in-memory obsolete space map. The condensing process will not continue unless a minimum of zfs_condense_min_mapping_bytes of memory can be freed. See also zfs_condense_indirect_vdevs_enable

zfs_dbgmsg_enable

Versions:

v0.6 - master

Platforms:

FreeBSD

Type:

int

Default:

v2.1 - master	`1`
v0.6 - v2.0	`0`

Range:

`0`	do not log debug messages
`1`	log debug messages

Change:

Dynamic

Tags:

debug

Enable ZFS debug message log

When to change: To view ZFS internal debug log

Notes: Internally ZFS keeps a small log to facilitate debugging. The contents of the log are in the /proc/spl/kstat/zfs/dbgmsg file. Writing 0 to /proc/spl/kstat/zfs/dbgmsg file clears the log. See also zfs_dbgmsg_maxsize

zfs_dbgmsg_maxsize

Versions:

v0.6 - master

Platforms:

FreeBSD

Type:

uint

Default:

v2.1 - master	`4194304`
v0.6 - v2.0	`4M`

Range:

0 to INT_MAX

Change:

Dynamic

Tags:

debug

Maximum ZFS debug log size

Notes: The /proc/spl/kstat/zfs/dbgmsg file size limit is set by zfs_dbgmsg_maxsize. See also zfs_dbgmsg_enable

zfs_dbuf_state_index

Versions:: v0.6 - master
Platforms:: Linux, FreeBSD
Type:: int
Default:: 0
Change:: Dynamic
Tags:: dbuf, debug

Calculate arc header index

When to change: Do not change

Notes: The zfs_dbuf_state_index feature is currently unused. It is normally used for controlling values in the /proc/spl/kstat/zfs/dbufs file.

zfs_ddt_data_is_special

Versions:

v0.8 - master

Platforms:

Linux, FreeBSD

Type:

int

Default:

1

Range:

`0`	do not use special vdevs to store DDT
`1`	store DDT in special vdevs

Change:

Dynamic

Tags:

dedup, SPA, special_vdev

Place DDT data into the special class

When to change: when using a special top-level vdev and no dedup top-level vdev and it is desired to store the DDT in the main pool top-level vdevs

Notes: zfs_ddt_data_is_special enables the deduplication table (DDT) to reside on a special top-level vdev.

zfs_deadman_checktime_ms

Versions:

v0.7 - master

Platforms:

Linux, FreeBSD

Type:

u64

Default:

v0.8 - master	`60,000`
v0.7	`5,000`

Range:

1 to ULONG_MAX

Change:

Dynamic

Tags:

Dead I/O check interval in milliseconds

When to change: When debugging slow I/O

Notes: Once a pool sync operation has taken longer than zfs_deadman_synctime_ms milliseconds, continue to check for slow operations every zfs_deadman_checktime_ms milliseconds.

zfs_deadman_enabled

Versions:

v0.6 - master

Platforms:

Linux, FreeBSD

Type:

int

Default:

1

Range:

`0`	do not log slow I/O
`1`	log slow I/O

Change:

Dynamic

Tags:

Enable deadman timer

When to change: To disable logging of slow I/O

Notes: When a pool sync operation takes longer than zfs_deadman_synctime_ms milliseconds, a “slow spa_sync” message is logged to the debug log (see zfs_dbgmsg_enable). If zfs_deadman_enabled is set to 1, then all pending IO operations are also checked and if any haven’t completed within zfs_deadman_synctime_ms milliseconds, a “SLOW IO” message is logged to the debug log and a “deadman” system event (see zpool events command) with the details of the hung IO is posted.

zfs_deadman_events_per_second

Versions:: v2.2 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 1
Change:: Dynamic
Tags:: vdev

Rate limit hung IO (deadman) events to this many per second

zfs_deadman_failmode

Versions:

v0.8 - master

Platforms:

Linux, FreeBSD

Type:

charp

Default:

wait

Range:

`wait`	wait for the “hung” I/O (default)
`continue`	attempt to recover from the “hung” I/O
`panic`	panic the system

Change:

Dynamic

Tags:

Failmode for deadman timer

When to change: In some cluster cases, panic can be appropriate

Notes: zfs_deadman_failmode controls the behavior of the I/O deadman timer when it detects a “hung” I/O..

zfs_deadman_synctime_ms

Versions:

v0.6 - master

Platforms:

Linux, FreeBSD

Type:

u64

Default:

v0.8 - master	`600,000`
v0.6 - v0.7	`1,000,000`

Range:

1 to ULONG_MAX

Change:

Dynamic

Tags:

Pool sync expiration time in milliseconds

When to change: When debugging slow I/O

Notes: The I/O deadman timer expiration time has two meanings 1. determines when the spa_deadman() logic should fire, indicating the txg sync has not completed in a timely manner 2. determines if an I/O is considered “hung” In version v0.8.0, any I/O that has not completed in zfs_deadman_synctime_ms is considered “hung” resulting in one of three behaviors controlled by the zfs_deadman_failmode parameter. zfs_deadman_synctime_ms takes effect if zfs_deadman_enabled = 1.

zfs_deadman_ziotime_ms

Versions:: v0.8 - master
Platforms:: Linux, FreeBSD
Type:: u64
Default:: 300000
Range:: 1 to ULONG_MAX
Change:: Dynamic
Tags:: deadman, debug, SPA

IO expiration time in milliseconds

When to change: Testing ABD features

Notes: When an individual I/O takes longer than zfs_deadman_ziotime_ms milliseconds, then the operation is considered to be “hung”. If zfs_deadman_enabled is set then the deadman behaviour is invoked as described by the zfs_deadman_failmode option.

zfs_dedup_log_cap

Versions:: v2.3 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: UINT_MAX
Change:: Dynamic
Tags:: DDT, dedup

Soft cap for the size of the current dedup log

zfs_dedup_log_flush_entries_max

Versions:: v2.3 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: UINT_MAX
Change:: Dynamic
Tags:: DDT, dedup

Max number of log entries to flush each transaction

zfs_dedup_log_flush_entries_min

Versions:: v2.3 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 100
Change:: Dynamic
Tags:: DDT, dedup

Min number of log entries to flush each transaction

zfs_dedup_log_flush_flow_rate_txgs

Versions:: v2.3 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 10
Change:: Dynamic
Tags:: DDT, dedup

Number of txgs to average flow rates across

zfs_dedup_log_flush_min_time_ms

Versions:: v2.3 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 1000
Change:: Dynamic
Tags:: DDT, dedup

Min time to spend on incremental dedup log flush each transaction

zfs_dedup_log_flush_txgs

Versions:: v2.3 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 100
Change:: Dynamic
Tags:: DDT, dedup

Number of TXGs to try to rotate the log in

zfs_dedup_log_hard_cap

Versions:: v2.3 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 0
Change:: Dynamic
Tags:: DDT, dedup

Whether to use the soft cap as a hard cap

zfs_dedup_log_mem_max

Versions:: v2.3 - master
Platforms:: Linux, FreeBSD
Type:: u64
Default:: 0
Change:: Prior to module load
Tags:: DDT, ddt_log, dedup

Max memory for dedup logs

zfs_dedup_log_mem_max_percent

Versions:: v2.3 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 1
Change:: Prior to module load
Tags:: DDT, ddt_log, dedup

Max memory for dedup logs, as % of total memory

zfs_dedup_log_txg_max

Versions:: v2.3 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 8
Change:: Dynamic
Tags:: DDT, ddt_log, dedup

Max transactions before starting to flush dedup logs

zfs_dedup_prefetch

Versions:

v0.6 - master

Platforms:

Linux, FreeBSD

Type:

int

Default:

0

Range:

`0`	do not prefetch
`1`	prefetch dedup table entries

Change:

Dynamic

Tags:

DDT, dedup, memory, prefetch

Enable prefetching dedup-ed blks

When to change: For systems with limited RAM using the dedup feature, disabling deduplication table prefetch can reduce memory pressure

Notes: ZFS can prefetch deduplication table (DDT) entries. zfs_dedup_prefetch allows DDT prefetches to be enabled.

zfs_default_bs

Versions:: v2.1 - master
Platforms:: Linux, FreeBSD
Type:: int
Default:: 9
Change:: Dynamic
Tags:: dnode

Default dnode block shift

zfs_default_ibs

Versions:: v2.1 - master
Platforms:: Linux, FreeBSD
Type:: int
Default:: 17
Change:: Dynamic
Tags:: dnode

Default dnode indirect block shift

zfs_delay_min_dirty_percent

Versions:: v0.6 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 60
Range:: 0 to 100
Change:: Dynamic
Tags:: delay, DSL, dsl_pool, write_throttle

Transaction delay threshold

When to change: See ZFS Transaction Delay

Notes: The ZFS write throttle begins to delay each transaction when the amount of dirty data reaches the threshold zfs_delay_min_dirty_percent of zfs_dirty_data_max. This value should be >= zfs_vdev_async_write_active_max_dirty_percent.

zfs_delay_scale

Versions:: v0.6 - master
Platforms:: Linux, FreeBSD
Type:: u64
Default:: 500000
Range:: 0 to ULONG_MAX
Change:: Dynamic
Tags:: delay, DSL, dsl_pool, write_throttle

How quickly delay approaches infinity

When to change: See ZFS Transaction Delay

Notes: zfs_delay_scale controls how quickly the ZFS write throttle transaction delay approaches infinity. Larger values cause longer delays for a given amount of dirty data. For the smoothest delay, this value should be about 1 billion divided by the maximum number of write operations per second the pool can sustain. The throttle will smoothly handle between 10x and 1/10th zfs_delay_scale. Note: zfs_delay_scale * zfs_dirty_data_max must be < 2^64.

zfs_delays_per_second

Versions:: v0.7
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 20
Change:: Dynamic
Tags:: delay, vdev, zed

Rate limit delay events to this many IO delays per second

When to change: If processing delay events at a higher rate is desired

Notes: Renamed to zfs_slow_io_events_per_second in v0.8.0 (commit ad796b8a3). The ZFS Event Daemon (zed) processes events from ZFS. However, it can be overwhelmed by high rates of error reports which can be generated by failing, high-performance devices. zfs_delays_per_second limits the rate of delay events reported to zed.

zfs_delete_blocks

Versions:: v0.7 - master
Platforms:: Linux
Type:: ulong
Default:: 20480
Range:: 1 to ULONG_MAX
Change:: Dynamic
Tags:: delete, filesystem, vnops

Delete files larger than N blocks async

When to change: If applications delete large files and blocking on unlink(2) is not desired

Notes: zfs_delete_blocks defines a large file for the purposes of delete. Files containing more than zfs_delete_blocks will be deleted asynchronously while smaller files are deleted synchronously. Decreasing this value reduces the time spent in an unlink(2) system call at the expense of a longer delay before the freed space is available. The zfs_delete_blocks value is specified in blocks, not bytes. The size of blocks can vary and is ultimately limited by the filesystem’s recordsize property.

zfs_delete_dentry

Versions:: v2.3 - master
Platforms:: Linux
Type:: int
Default:: 0
Change:: Dynamic
Tags:: delete, super

Delete dentries from dentry cache as soon as the last reference is released.

zfs_delete_inode

Versions:: v2.3 - master
Platforms:: Linux
Type:: int
Default:: 0
Change:: Dynamic
Tags:: delete, super

Delete inodes as soon as the last reference is released.

zfs_dio_enabled

Versions:

v2.3 - master

Platforms:

Linux, FreeBSD

Type:

int

Default:

v2.4 - master	`1`
v2.3	`0`

Change:

Dynamic

Tags:

vnops

Enable Direct I/O

zfs_dio_strict

Versions:: v2.4 - master
Platforms:: Linux, FreeBSD
Type:: int
Default:: 0
Change:: Dynamic
Tags:: vnops

Return errors on misaligned Direct I/O

zfs_dio_write_verify_events_per_second

Versions:: v2.3 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 20
Change:: Dynamic
Tags:: vdev

Rate Direct I/O write verify events to this many per second

zfs_dirty_data_max

Versions:: v0.6 - master
Platforms:: Linux, FreeBSD
Type:: u64
Range:: 1 to zfs_dirty_data_max_max
Change:: Dynamic
Tags:: DSL, dsl_pool, write_throttle

Determines the dirty space limit

When to change: See ZFS Transaction Delay

Notes: zfs_dirty_data_max is the ZFS write throttle dirty space limit. Once this limit is exceeded, new writes are delayed until space is freed by writes being committed to the pool. zfs_dirty_data_max takes precedence over zfs_dirty_data_max_percent.

zfs_dirty_data_max_max

Versions:: v0.6 - master
Platforms:: Linux, FreeBSD
Type:: u64
Range:: 1 to physical RAM size
Change:: Prior to module load
Tags:: DSL, dsl_pool, write_throttle

zfs_dirty_data_max upper bound in bytes

When to change: See ZFS Transaction Delay

Notes: zfs_dirty_data_max_max is the maximum allowable value of zfs_dirty_data_max. zfs_dirty_data_max_max takes precedence over zfs_dirty_data_max_max_percent.

zfs_dirty_data_max_max_percent

Versions:: v0.6 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 25
Range:: 1 to 100
Change:: Prior to module load
Tags:: DSL, dsl_pool, write_throttle

zfs_dirty_data_max upper bound as % of RAM

When to change: See ZFS Transaction Delay

Notes: zfs_dirty_data_max_max_percent an alternative to zfs_dirty_data_max_max for setting the maximum allowable value of zfs_dirty_data_max zfs_dirty_data_max_max takes precedence over zfs_dirty_data_max_max_percent

zfs_dirty_data_max_percent

Versions:: v0.6 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 10
Range:: 1 to 100
Change:: Prior to module load
Tags:: DSL, dsl_pool, write_throttle

Max percent of RAM allowed to be dirty

When to change: See ZFS Transaction Delay

Notes: zfs_dirty_data_max_percent is an alternative method of specifying zfs_dirty_data_max, the ZFS write throttle dirty space limit. Once this limit is exceeded, new writes are delayed until space is freed by writes being committed to the pool. zfs_dirty_data_max takes precedence over zfs_dirty_data_max_percent.

zfs_dirty_data_sync

Versions:: v0.6 - v0.7
Platforms:: Linux, FreeBSD
Type:: ulong
Default:: 67,108,864
Range:: 1 to ULONG_MAX
Change:: Dynamic
Tags:: DSL, dsl_pool, write_throttle, ZIO_scheduler

sync txg when this much dirty data

Notes: When there is at least zfs_dirty_data_sync dirty data, a transaction group sync is started. This allows a transaction group sync to occur more frequently than the transaction group timeout interval (see zfs_txg_timeout) when there is dirty data to be written.

zfs_dirty_data_sync_percent

Versions:: v0.8 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 20
Range:: 1 to zfs_vdev_async_write_active_min_dirty_percent
Change:: Dynamic
Tags:: DSL, dsl_pool, write_throttle, ZIO_scheduler

Dirty data txg sync threshold as a percentage of zfs_dirty_data_max

Notes: When there is at least zfs_dirty_data_sync_percent of zfs_dirty_data_max dirty data, a transaction group sync is started. This allows a transaction group sync to occur more frequently than the transaction group timeout interval (see zfs_txg_timeout) when there is dirty data to be written.

zfs_disable_dup_eviction

Versions:

v0.6

Platforms:

Linux, FreeBSD

Type:

int

Range:

`0`	duplicate buffers can be evicted
`1`	do not evict duplicate buffers

Change:

Dynamic

Tags:

ARC, dedup

disable duplicate buffer eviction

Notes: Removed in v0.7.0 by commit d3c2ae1c0 (“OpenZFS 6950 - ARC should cache compressed data”). Duplicate buffer handling was reworked as part of the compressed ARC changes. No replacement parameter. Disable duplicate buffer eviction from ARC.

zfs_disable_ivset_guid_check

Versions:

v0.8 - master

Platforms:

Linux, FreeBSD

Type:

int

Default:

0

Range:

`0`	check IVset guid
`1`	do not check IVset guid

Change:

Dynamic

Tags:

DSL, receive

Set to allow raw receives without IVset guids

When to change: debugging pre-release ZFS raw sends

Notes: zfs_disable_ivset_guid_check disables requirement for IVset guids to be present and match when doing a raw receive of encrypted datasets. Intended for users whose pools were created with ZFS on Linux pre-release versions and now have compatibility issues. For a ZFS raw receive, from a send stream created by zfs send --raw, the crypt_keydata nvlist includes a to_ivset_guid to be set on the new snapshot. This value will override the value generated by the snapshot code. However, this value may not be present, because older implementations of the raw send code did not include this value. When zfs_disable_ivset_guid_check is enabled, the receive proceeds and a newly-generated value is used.

zfs_dmu_offset_next_sync

Versions:

v0.7 - master

Platforms:

Linux, FreeBSD

Type:

int

Default:

1

Range:

`0`	do not force txg sync to find holes
`1`	force txg sync to find holes

Change:

Dynamic

Tags:

DMU

Enable forcing txg sync to find holes

When to change: to exchange strict hole reporting for performance

Notes: zfs_dmu_offset_next_sync enables forcing txg sync to find holes. This causes ZFS to act like older versions when SEEK_HOLE or SEEK_DATA flags are used: when a dirty dnode causes txgs to be synced so the previous data can be found.

zfs_embedded_slog_min_ms

Versions:: v2.1 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 64
Change:: Dynamic
Tags:: vdev

Minimum number of metaslabs required to dedicate one for log blocks

zfs_expire_snapshot

Versions:: v0.6 - master
Platforms:: Linux
Type:: int
Default:: 300
Range:: 0 disables automatic unmounting, maximum time is INT_MAX
Change:: Dynamic
Tags:: ctldir, filesystem, snapshot

Seconds to expire .zfs/snapshot

Notes: Snapshots of filesystems are normally automounted under the filesystem’s .zfs/snapshot subdirectory. When not in use, snapshots are unmounted after zfs_expire_snapshot seconds.

zfs_fallocate_reserve_percent

Versions:: v2.0 - master
Platforms:: Linux
Type:: uint
Default:: 110
Change:: Dynamic
Tags:: file

Percentage of length to use for the available capacity check

zfs_flags

Versions:: v0.6 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 0
Change:: Dynamic
Tags:: debug, SPA

Set additional debugging flags

When to change: When debugging ZFS

zfs_fletcher_4_impl

Versions:: v0.7 - v0.8
Platforms:: Linux, FreeBSD
Type:: string
Default:: fastest
Range:: fastest, scalar, superscalar, superscalar4, sse2, ssse3, avx2, avx512f, or aarch64_neon depending on hardware support
Change:: Dynamic
Tags:: checksum, CPU

Select fletcher 4 implementation.

When to change: Testing Fletcher-4 algorithms

Notes: Fletcher-4 is the default checksum algorithm for metadata and data. When the zfs kernel module is loaded, a set of microbenchmarks are run to determine the fastest algorithm for the current hardware. The zfs_fletcher_4_impl parameter allows a specific implementation to be specified other than the default (fastest). Selectors other than fastest and scalar require instruction set extensions to be available and will only appear if ZFS detects their presence. The scalar implementation works on all processors. The results of the microbenchmark are visible in the /proc/spl/kstat/zfs/fletcher_4_bench file. Larger numbers indicate better performance. Since ZFS is processor endian-independent, the microbenchmark is run against both big and little-endian transformation.

zfs_free_bpobj_enabled

Versions:

v0.7 - master

Platforms:

Linux, FreeBSD

Type:

int

Default:

1

Range:

`0`	do not process free_bpobj objects
`1`	process free_bpobj objects

Change:

Dynamic

Tags:

delete, DSL, dsl_scan

Enable processing of the free_bpobj

When to change: If there’s a problem with processing free_bpobj (e.g. i/o error or bug)

zfs_free_leak_on_eio

Versions:

v0.6 - master

Platforms:

Linux, FreeBSD

Type:

int

Default:

0

Range:

`0`	normal behavior
`1`	ignore error and permanently leak space

Change:

Dynamic

Tags:

debug, SPA

Set to ignore IO errors during free and permanently leak the space

When to change: When debugging I/O errors during destroy

Notes: If destroy encounters an I/O error (EIO) while reading metadata (eg indirect blocks), space referenced by the missing metadata cannot be freed. Normally, this causes the background destroy to become “stalled”, as the destroy is unable to make forward progress. While in this stalled state, all remaining space to free from the error-encountering filesystem is temporarily leaked. Set zfs_free_leak_on_eio = 1 to ignore the EIO, permanently leak the space from indirect blocks that can not be read, and continue to free everything else that it can. The default, stalling behavior is useful if the storage partially fails (eg some but not all I/Os fail), and then later recovers. In this case, we will be able to continue pool operations while it is partially failed, and when it recovers, we can continue to free the space, with no leaks. However, note that this case is rare. Typically pools either: 1. fail completely (but perhaps temporarily (eg a top-level vdev going offline) 2. have localized, permanent errors (eg disk returns the wrong data due to bit flip or firmware bug) In case (1), the zfs_free_leak_on_eio setting does not matter because the pool will be suspended and the sync thread will not be able to make forward progress. In case (2), because the error is permanent, the best effort do is leak the minimum amount of space. Therefore, it is reasonable for zfs_free_leak_on_eio be set, but by default the more conservative approach is taken, so that there is no possibility of leaking space in the “partial temporary” failure case.

zfs_free_max_blocks

Versions:: v0.6 - v0.7
Platforms:: Linux, FreeBSD
Type:: ulong
Default:: 100,000
Range:: 1 to ULONG_MAX
Change:: Dynamic
Tags:: delete, DSL, dsl_scan, filesystem

Max number of blocks freed in one txg

When to change: For workloads that delete large files, zfs_free_max_blocks can be adjusted to meet performance requirements while reducing the impacts of deletion

Notes: zfs_free_max_blocks sets the maximum number of blocks to be freed in a single transaction group (txg). For workloads that delete (free) large numbers of blocks in a short period of time, the processing of the frees can negatively impact other operations, including txg commits. zfs_free_max_blocks acts as a limit to reduce the impact.

zfs_free_min_time_ms

Versions:

v0.6 - master

Platforms:

Linux, FreeBSD

Type:

uint

Default:

v2.4 - master	`500`
v0.6 - v2.3	`1,000`

Range:

1 to (zfs_txg_timeout * 1000)

Change:

Dynamic

Tags:

delete, DSL, dsl_scan

Min millisecs to free per txg

Notes: During a zfs destroy operation using feature@async_destroy a minimum of zfs_free_min_time_ms time will be spent working on freeing blocks per txg commit.

zfs_history_output_max

Versions:: v2.0 - master
Platforms:: Linux, FreeBSD
Type:: u64
Default:: 1048576
Change:: Dynamic
Tags:: ioctl

Maximum size in bytes of ZFS ioctl output that will be logged

zfs_immediate_write_sz

Versions:: v0.6 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 32768
Range:: 512 to 16,777,216 (valid block sizes)
Change:: Dynamic
Tags:: ZIL

Largest write size to store data into ZIL

Verification: Data blocks that exceed zfs_immediate_write_sz or are written as logbias=throughput increment the zil_itx_indirect_count entry in /proc/spl/kstat/zfs/zil

Notes: If a pool does not have a log device, data blocks equal to or larger than zfs_immediate_write_sz are treated as if the dataset being written to had the property setting logbias=throughput Terminology note: logbias=throughput writes the blocks in “indirect mode” to the ZIL where the data is written to the pool and a pointer to the data is written to the ZIL.

zfs_import_defer_txgs

Versions:: v2.4 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 5
Change:: Dynamic
Tags:: DSL, dsl_scan

Number of TXGs to defer background work after pool import

zfs_initialize_chunk_size

Versions:: v2.0 - master
Platforms:: Linux, FreeBSD
Type:: u64
Default:: 1048576
Change:: Dynamic
Tags:: vdev

Size in bytes of writes by zpool initialize

zfs_initialize_value

Versions:: v0.8 - master
Platforms:: Linux, FreeBSD
Type:: u64
Default:: 16045690984833335022
Change:: Dynamic
Tags:: vdev, vdev_initialize

Value written during zpool initialize

When to change: when debugging initialization code

Notes: When initializing a vdev, ZFS writes patterns of zfs_initialize_value bytes to the device.

zfs_keep_log_spacemaps_at_export

Versions:: v2.0 - master
Platforms:: Linux, FreeBSD
Type:: int
Default:: 0
Change:: Dynamic
Tags:: SPA, spa_log_spacemap

Prevent the log spacemaps from being flushed and destroyed during pool export/destroy

zfs_key_max_salt_uses

Versions:: v0.8 - master
Platforms:: Linux
Type:: ulong
Default:: 400000000
Range:: 1 to ULONG_MAX
Change:: Dynamic
Tags:: encryption, ZIO

Max number of times a salt value can be used for generating encryption keys before it is rotated

When to change: Testing encryption features

Notes: For encrypted datasets, the salt is regenerated every zfs_key_max_salt_uses blocks. This automatic regeneration reduces the probability of collisions due to the Birthday problem. When set to the default (400,000,000) the probability of collision is approximately 1 in 1 trillion.

zfs_livelist_condense_new_alloc

Versions:: v2.0 - master
Platforms:: Linux, FreeBSD
Type:: int
Default:: 0
Change:: Dynamic
Tags:: livelist, livelist_condense, SPA

Whether extra ALLOC blkptrs were added to a livelist entry while it was being condensed

zfs_livelist_condense_sync_cancel

Versions:: v2.0 - master
Platforms:: Linux, FreeBSD
Type:: int
Default:: 0
Change:: Dynamic
Tags:: livelist, livelist_condense, SPA

Whether livelist condensing was canceled in the synctask

zfs_livelist_condense_sync_pause

Versions:: v2.0 - master
Platforms:: Linux, FreeBSD
Type:: int
Default:: 0
Change:: Dynamic
Tags:: livelist, livelist_condense, SPA

Set the livelist condense synctask to pause

zfs_livelist_condense_zthr_cancel

Versions:: v2.0 - master
Platforms:: Linux, FreeBSD
Type:: int
Default:: 0
Change:: Dynamic
Tags:: livelist, livelist_condense, SPA

Whether livelist condensing was canceled in the zthr function

zfs_livelist_condense_zthr_pause

Versions:: v2.0 - master
Platforms:: Linux, FreeBSD
Type:: int
Default:: 0
Change:: Dynamic
Tags:: livelist, livelist_condense, SPA

Set the livelist condense zthr to pause

zfs_livelist_max_entries

Versions:: v2.0 - master
Platforms:: Linux, FreeBSD
Type:: u64
Default:: 500000
Change:: Dynamic
Tags:: DSL, dsl_deadlist, livelist

Size to start the next sub-livelist in a livelist

zfs_livelist_min_percent_shared

Versions:: v2.0 - master
Platforms:: Linux, FreeBSD
Type:: int
Default:: 75
Change:: Dynamic
Tags:: DSL, dsl_deadlist, livelist

Threshold at which livelist is disabled

zfs_lua_max_instrlimit

Versions:: v0.8 - master
Platforms:: Linux, FreeBSD
Type:: u64
Default:: 100000000
Range:: 0 to MAX_ULONG
Change:: Dynamic
Tags:: channel_programs, lua, zcp

Max instruction limit that can be specified for a channel program

When to change: to enforce a CPU usage limit on ZFS channel programs

zfs_lua_max_memlimit

Versions:: v0.8 - master
Platforms:: Linux, FreeBSD
Type:: u64
Default:: 104857600
Range:: 0 to MAX_ULONG
Change:: Dynamic
Tags:: channel_programs, lua, zcp

Max memory limit that can be specified for a channel program

zfs_max_async_dedup_frees

Versions:

v2.0 - master

Platforms:

Linux, FreeBSD

Type:

u64

Default:

v2.4 - master	`250000`
v2.0 - v2.3	`100,000`

Change:

Dynamic

Tags:

dedup, DSL, dsl_scan

Max number of dedup, clone or gang blocks freed in one txg

zfs_max_dataset_nesting

Versions:: v0.8 - master
Platforms:: Linux, FreeBSD
Type:: int
Default:: 50
Range:: 0 to MAX_INT
Change:: Dynamic
Tags:: dataset

Limit to the amount of nesting a path can have. Defaults to 50.

When to change: can be tuned temporarily to fix existing datasets that exceed the predefined limit

Notes: zfs_max_dataset_nesting limits the depth of nested datasets. Deeply nested datasets can overflow the stack. The maximum stack depth depends on kernel compilation options, so it is impractical to predict the possible limits. For kernels compiled with small stack sizes, zfs_max_dataset_nesting may require changes.

zfs_max_log_walking

Versions:: v2.0 - master
Platforms:: Linux, FreeBSD
Type:: u64
Default:: 5
Change:: Dynamic
Tags:: SPA, spa_log_spacemap

The number of past TXGs that the flushing algorithm of the log spacemap feature uses to estimate incoming log blocks

zfs_max_logsm_summary_length

Versions:: v2.0 - master
Platforms:: Linux, FreeBSD
Type:: u64
Default:: 10
Change:: Dynamic
Tags:: SPA, spa_log_spacemap

Maximum number of rows allowed in the summary of the spacemap log

zfs_max_missing_tvds

Versions:: v0.8 - master
Platforms:: Linux, FreeBSD
Type:: u64
Default:: 0
Range:: 0 to MAX_INT
Change:: Dynamic
Tags:: import, SPA

Allow importing pool with up to this number of missing top-level vdevs (in read-only mode)

When to change: troubleshooting pools with missing devices

Notes: When importing a pool in readonly mode (zpool import -o readonly=on ...) then up to zfs_max_missing_tvds top-level vdevs can be missing, but the import can attempt to progress. Note: This is strictly intended for advanced pool recovery cases since missing data is almost inevitable. Pools with missing devices can only be imported read-only for safety reasons, and the pool’s failmode property is automatically set to continue The expected use case is to recover pool data immediately after accidentally adding a non-protected vdev to a protected pool. - With 1 missing top-level vdev, ZFS should be able to import the pool and mount all datasets. User data that was not modified after the missing device has been added should be recoverable. Thus snapshots created prior to the addition of that device should be completely intact. - With 2 missing top-level vdevs, some datasets may fail to mount since there are dataset statistics that are stored as regular metadata. Some data might be recoverable if those vdevs were added recently. - With 3 or more top-level missing vdevs, the pool is severely damaged and MOS entries may be missing entirely. Chances of data recovery are very low. Note that there are also risks of performing an inadvertent rewind as we might be missing all the vdevs with the latest uberblocks.

zfs_max_missing_tvds_cachefile

Versions:: master
Platforms:: Linux, FreeBSD
Type:: u64
Default:: 2
Change:: Dynamic
Tags:: SPA

Allow importing pools with missing top-level vdevs in cache file

zfs_max_missing_tvds_scan

Versions:: master
Platforms:: Linux, FreeBSD
Type:: u64
Default:: 0
Change:: Dynamic
Tags:: SPA

Allow importing pools with missing top-level vdevs during scan

zfs_max_nvlist_src_size

Versions:: v2.0 - master
Platforms:: Linux, FreeBSD
Type:: u64
Change:: Dynamic
Tags:: ioctl

Maximum size in bytes allowed for src nvlist passed with ZFS ioctls

zfs_max_recordsize

Versions:

v0.6 - master

Platforms:

Linux, FreeBSD

Type:

uint

Default:

v2.2 - master	`16777216`
v0.6 - v2.1	`1,048,576`

Range:

512 to 16,777,216 (valid block sizes)

Change:

Dynamic

Tags:

DSL, dsl_dataset, filesystem, memory, volume

Max allowed record size

When to change: To create datasets with larger volblocksize or recordsize

Notes: ZFS supports logical record (block) sizes from 512 bytes to 16 MiB. The benefits of larger blocks, and thus larger average I/O sizes, can be weighed against the cost of copy-on-write of large block to modify one byte. Additionally, very large blocks can have a negative impact on both I/O latency at the device level and the memory allocator. The zfs_max_recordsize parameter limits the upper bound of the dataset volblocksize and recordsize properties. Larger blocks can be created by enabling zpool large_blocks feature and changing this zfs_max_recordsize. Pools with larger blocks can always be imported and used, regardless of the value of zfs_max_recordsize. For 32-bit systems, zfs_max_recordsize also limits the size of kernel virtual memory caches used in the ZFS I/O pipeline (zio_buf_* and zio_data_buf_*). See also the zpool large_blocks feature.

zfs_mdcomp_disable

Versions:

v0.6 - v0.7

Platforms:

Linux, FreeBSD

Type:

int

Range:

`0`	compress metadata
`1`	do not compress metadata

Change:

Dynamic

Tags:

CPU, DMU, metadata

Disable meta data compression

When to change: When CPU cycles cost less than I/O

zfs_metaslab_condense_pct

Versions:: master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 200
Change:: Dynamic
Tags:: metaslab

Condense on-disk spacemap when it is more than this many percents of in-memory counterpart

zfs_metaslab_find_max_tries

Versions:: v2.1 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 100
Change:: Dynamic
Tags:: metaslab

Normally only consider this many of the best metaslabs in each vdev

zfs_metaslab_fragmentation_threshold

Versions:

v0.6 - master

Platforms:

Linux, FreeBSD

Type:

uint

Default:

v2.3 - master	`77`
v0.6 - v2.2	`70`

Range:

1 to 100

Change:

Dynamic

Tags:

allocation, fragmentation, metaslab, vdev

Fragmentation for metaslab to allow allocation

When to change: Testing metaslab allocation

Notes: Allow metaslabs to keep their active state as long as their fragmentation percentage is less than or equal to this value. When writing, an active metaslab whose fragmentation percentage exceeds zfs_metaslab_fragmentation_threshold is avoided allowing metaslabs with less fragmentation to be preferred. Metaslab fragmentation is used to calculate the overall pool fragmentation property value. However, individual metaslab fragmentation levels are observable using the zdb with the -mm option. zfs_metaslab_fragmentation_threshold works at the metaslab level and each top-level vdev has approximately zfs_vdev_default_ms_count metaslabs. See also zfs_mg_fragmentation_threshold

zfs_metaslab_max_size_cache_sec

Versions:: v2.0 - master
Platforms:: Linux, FreeBSD
Type:: u64
Default:: 3600
Change:: Dynamic
Tags:: metaslab

How long to trust the cached max chunk size of a metaslab

zfs_metaslab_mem_limit

Versions:: v2.0 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 25
Change:: Dynamic
Tags:: metaslab

Percentage of memory that can be used to store metaslab range trees

zfs_metaslab_segment_weight_enabled

Versions:

v0.7 - master

Platforms:

Linux, FreeBSD

Type:

int

Default:

1

Range:

`0`	do not consider metaslab fragmentation
`1`	avoid metaslabs where free space is highly fragmented

Change:

Dynamic

Tags:

allocation, metaslab

Enable segment-based metaslab selection

When to change: When testing allocation and fragmentation

Notes: Enables metaslab allocation based on largest free segment rather than total amount of free space. The goal is to avoid metaslabs that exhibit free space fragmentation: when there is a lot of small free spaces, but few larger free spaces. If zfs_metaslab_segment_weight_enabled is enabled, then metaslab_fragmentation_factor_enabled is ignored.

zfs_metaslab_sm_blksz_no_log

Versions:: master
Platforms:: Linux, FreeBSD
Type:: int
Default:: 16384
Change:: Dynamic
Tags:: metaslab

Block size for space map in pools with log space map disabled. Power of 2 greater than 4096.

zfs_metaslab_sm_blksz_with_log

Versions:: master
Platforms:: Linux, FreeBSD
Type:: int
Default:: 131072
Change:: Dynamic
Tags:: metaslab

Block size for space map in pools with log space map enabled. Power of 2 greater than 4096.

zfs_metaslab_switch_threshold

Versions:: v0.7 - master
Platforms:: Linux, FreeBSD
Type:: int
Default:: 2
Range:: 0 to UINT64_MAX
Change:: Dynamic
Tags:: allocation, metaslab

Segment-based metaslab selection maximum buckets before switching

When to change: When testing allocation and fragmentation

Notes: When using segment-based metaslab selection (see zfs_metaslab_segment_weight_enabled), continue allocating from the active metaslab until zfs_metaslab_switch_threshold worth of free space buckets have been exhausted.

zfs_metaslab_try_hard_before_gang

Versions:: v2.1 - master
Platforms:: Linux, FreeBSD
Type:: int
Default:: 0
Change:: Dynamic
Tags:: metaslab

Try hard to allocate before ganging

zfs_mg_fragmentation_threshold

Versions:

v0.6 - master

Platforms:

Linux, FreeBSD

Type:

uint

Default:

v0.8 - master	`95`
v0.6 - v0.7	`85`

Range:

1 to 100

Change:

Dynamic

Tags:

allocation, fragmentation, metaslab, vdev

Percentage of metaslab group size that should be considered eligible for allocations unless all metaslab groups within the metaslab class have also crossed this threshold

When to change: Testing metaslab allocation

Notes: Metaslab groups (top-level vdevs) are considered eligible for allocations if their fragmentation percentage metric is less than or equal to zfs_mg_fragmentation_threshold. If a metaslab group exceeds this threshold then it will be skipped unless all metaslab groups within the metaslab class have also crossed the zfs_mg_fragmentation_threshold threshold.

zfs_mg_noalloc_threshold

Versions:: v0.6 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 0
Range:: 0 to 100
Change:: Dynamic
Tags:: allocation, fragmentation, metaslab, vdev

Percentage of metaslab group size that should be free to make it eligible for allocation

When to change: To force rebalancing as top-level vdevs are added or expanded

Notes: Metaslab groups (top-level vdevs) with free space percentage greater than zfs_mg_noalloc_threshold are eligible for new allocations. If a metaslab group’s free space is less than or equal to the threshold, the allocator avoids allocating to that group unless all groups in the pool have reached the threshold. Once all metaslab groups have reached the threshold, all metaslab groups are allowed to accept allocations. The default value of 0 disables the feature and causes all metaslab groups to be eligible for allocations. This parameter allows one to deal with pools having heavily imbalanced vdevs such as would be the case when a new vdev has been added. Setting the threshold to a non-zero percentage will stop allocations from being made to vdevs that aren’t filled to the specified percentage and allow lesser filled vdevs to acquire more allocations than they otherwise would under the older zfs_mg_alloc_failures facility.

zfs_min_metaslabs_to_condense_pct

Versions:: master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 5
Change:: Dynamic
Tags:: SPA, spa_log_spacemap

Minimum number of metaslabs to flush per TXG when condensing, as a percent of the number of dirty metaslabs at condense start.

zfs_min_metaslabs_to_flush

Versions:: v2.0 - master
Platforms:: Linux, FreeBSD
Type:: u64
Default:: 1
Change:: Dynamic
Tags:: SPA, spa_log_spacemap

Minimum number of metaslabs to flush per dirty TXG

zfs_multihost_fail_intervals

Versions:

v0.7 - master

Platforms:

Linux, FreeBSD

Type:

uint

Default:

v0.8 - master	`10`
v0.7	`5`

Change:

Dynamic

Tags:

import, MMP, multihost

Max allowed period without a successful mmp write

Notes: zfs_multihost_fail_intervals controls the behavior of the pool when write failures are detected in the multihost multimodifier protection (MMP) subsystem. If zfs_multihost_fail_intervals = 0 then multihost write failures are ignored. The write failures are reported to the ZFS event daemon (zed) which can take action such as suspending the pool or offlining a device. If zfs_multihost_fail_intervals > 0 then sequential multihost write failures will cause the pool to be suspended. This occurs when (zfs_multihost_fail_intervals * zfs_multihost_interval) milliseconds have passed since the last successful multihost write. This guarantees the activity test will see multihost writes if the pool is attempted to be imported by another system.

zfs_multihost_history

Versions:: v0.7 - master
Platforms:: Linux, FreeBSD
Type:: uint
Default:: 0
Range:: 0 to INT_MAX
Change:: Dynamic
Tags:: import, MMP, multihost, SPA, spa_stats

Historical statistics for last N multihost writes

When to change: When testing multihost feature

Notes: The pool multihost multimodifier protection (MMP) subsystem can record historical updates in the /proc/spl/kstat/zfs/POOL_NAME/multihost file for debugging purposes. The number of lines of history is determined by zfs_multihost_history.

zfs_multihost_import_intervals

Versions:

v0.7 - master

Platforms:

Linux, FreeBSD

Type:

uint

Default:

v0.8 - master	`20`
v0.7	`10`

Range:

1 to UINT_MAX

Change:

Dynamic

Tags:

import, MMP, multihost

Number of zfs_multihost_interval periods to wait for activity

Notes: zfs_multihost_import_intervals controls the duration of the activity test on pool import for the multihost multimodifier protection (MMP) subsystem. The activity test can be expected to take a minimum time of (zfs_multihost_import_intervals * zfs_multihost_interval * random(25%)) milliseconds. The random period of up to 25% improves simultaneous import detection. For example, if two hosts are rebooted at the same time and automatically attempt to import the pool, then is is highly probable that one host will win. Smaller values of zfs_multihost_import_intervals reduces the import time but increases the risk of failing to detect an active pool. The total activity check time is never allowed to drop below one second. Note: the multihost protection feature applies to storage devices that can be shared between multiple systems.

zfs_multihost_interval

Versions:: v0.7 - master
Platforms:: Linux, FreeBSD
Type:: u64
Default:: 1000
Range:: 100 to ULONG_MAX
Change:: Dynamic
Tags:: import, MMP, multihost, vdev

Milliseconds between mmp writes to each leaf

When to change: To optimize pool import time against possibility of simultaneous import by another system

Notes: zfs_multihost_interval controls the frequency of multihost writes performed by the pool multihost multimodifier protection (MMP) subsystem. The multihost write period is (zfs_multihost_interval / number of leaf-vdevs) milliseconds. Thus on average a multihost write will be issued for each leaf vdev every zfs_multihost_interval milliseconds. In practice, the observed period can vary with the I/O load and this observed value is the delay which is stored in the uberblock. On import the multihost activity check waits a minimum amount of time determined by (zfs_multihost_interval * zfs_multihost_import_intervals) with a lower bound of 1 second. The activity check time may be further extended if the value of mmp delay found in the best uberblock indicates actual multihost updates happened at longer intervals than zfs_multihost_interval Note: the multihost protection feature applies to storage devices that can be shared between multiple systems.

zfs_multilist_num_sublists

Versions:

v0.7 - master

Platforms:

Linux, FreeBSD

Type:

uint

Default:

v2.1 - master	`0`
v0.7 - v2.0	`4 or the number of online CPUs, whichever is greater`

Range:

1 to INT_MAX

Change:

Dynamic

Tags: