ZFS on Linux Module Parameters

Most of the ZFS kernel module parameters are accessible in the SysFS /sys/module/zfs/parameters directory. Current value can be observed by

cat /sys/module/zfs/parameters/PARAMETER

Many of these can be changed by writing new values. These are denoted by Change|Dynamic in the PARAMETER details below.

echo NEWVALUE >> /sys/module/zfs/parameters/PARAMETER

If the parameter is not dynamically adjustable, an error can occur and the value will not be set. It can be helpful to check the permissions for the PARAMETER file in SysFS.

In some cases, the parameter must be set prior to loading the kernel modules or it is desired to have the parameters set automatically at boot time. For many distros, this can be accomplished by creating a file named /etc/modprobe.d/zfs.conf containing a text line for each module parameter using the format:

# change PARAMETER for workload XZY to solve problem PROBLEM_DESCRIPTION
# changed by YOUR_NAME on DATE
options zfs PARAMETER=VALUE

Some parameters related to ZFS operations are located in module parameters other than in the zfs kernel module. These are documented in the individual parameter description. Unless otherwise noted, the tunable applies to the zfs kernel module. For example, the icp kernel module parameters are visible in the /sys/module/icp/parameters directory and can be set by default at boot time by changing the /etc/modprobe.d/icp.conf file.

See the man page for modprobe.d for more information.

zfs-module-parameters Manual Page

The zfs-module-parameters(5) man page contains brief descriptions of the module parameters. Alas, man pages are not as suitable for quick reference as wiki pages. This wiki page is intended to be a better cross-reference and capture some of the wisdom of ZFS developers and practitioners.

ZFS Module Parameters

The ZFS kernel module, zfs.ko, parameters are detailed below.

To observe the list of parameters along with a short synopsis of each parameter, use the modinfo command:

modinfo zfs

Tags

The list of parameters is quite large and resists hierarchical representation. To assist in quickly finding relevant information quickly, each module parameter has a “Tags” row with keywords for frequent searches.

vdev_initialize

Index

Module Parameters

ignore_hole_birth

When set, the hole_birth optimization will not be used and all holes will always be sent by zfs send In the source code, ignore_hole_birth is an alias for and SysFS PARAMETER for send_holes_without_birth_time.

ignore_hole_birth

Notes

Tags

send

When to change

Enable if you suspect your datasets are affected by a bug in hole_birth during zfs send operations

Data Type

boolean

Range

0=disabled, 1=enabled

Default

1 (hole birth optimization is ignored)

Change

Dynamic

Versions Affected

TBD

l2arc_feed_again

Turbo L2ARC cache warm-up. When the L2ARC is cold the fill interval will be set to aggressively fill as fast as possible.

l2arc_feed_again

Notes

Tags

ARC, L2ARC

When to change

If cache devices exist and it is desired to fill them as fast as possible

Data Type

boolean

Range

0=disabled, 1=enabled

Default

1

Change

Dynamic

Versions Affected

TBD

l2arc_feed_min_ms

Minimum time period for aggressively feeding the L2ARC. The L2ARC feed thread wakes up once per second (see l2arc_feed_secs) to look for data to feed into the L2ARC. l2arc_feed_min_ms only affects the turbo L2ARC cache warm-up and allows the aggressiveness to be adjusted.

l2arc_feed_min_ms

Notes

Tags

ARC, L2ARC

When to change

If cache devices exist and l2arc_feed_again and the feed is too aggressive, then this tunable can be adjusted to reduce the impact of the fill

Data Type

uint64

Units

milliseconds

Range

0 to (1000 * l2arc_feed_secs)

Default

200

Change

Dynamic

Versions Affected

0.6 and later

l2arc_feed_secs

Seconds between waking the L2ARC feed thread. One feed thread works for all cache devices in turn.

If the pool that owns a cache device is imported readonly, then the feed thread is delayed 5 * l2arc_feed_secs before moving onto the next cache device. If multiple pools are imported with cache devices and one pool with cache is imported readonly, the L2ARC feed rate to all caches can be slowed.

l2arc_feed_secs

Notes

Tags

ARC, L2ARC

When to change

Do not change

Data Type

uint64

Units

seconds

Range

1 to UINT64_MAX

Default

1

Change

Dynamic

Versions Affected

0.6 and later

l2arc_headroom

How far through the ARC lists to search for L2ARC cacheable content, expressed as a multiplier of l2arc_write_max

l2arc_headroom

Notes

Tags

ARC, L2ARC

When to change

If the rate of change in the ARC is faster than the overall L2ARC feed rate, then increasing l2arc_headroom can increase L2ARC efficiency. Setting the value too large can cause the L2ARC feed thread to consume more CPU time looking for data to feed.

Data Type

uint64

Units

unit

Range

0 to UINT64_MAX

Default

2

Change

Dynamic

Versions Affected

0.6 and later

l2arc_headroom_boost

Percentage scale for l2arc_headroom when L2ARC contents are being successfully compressed before writing.

l2arc_headroom_boost

Notes

Tags

ARC, L2ARC

When to change

If average compression efficiency is greater than 2:1, then increasing l2a rc_headroom_boost can increase the L2ARC feed rate

Data Type

uint64

Units

percent

Range

100 to UINT64_MAX, when set to 100, the L2ARC headroom boost feature is effectively disabled

Default

200

Change

Dynamic

Versions Affected

all

l2arc_nocompress

Disable writing compressed data to cache devices. Disabling allows the legacy behavior of writing decompressed data to cache devices.

l2arc_nocompress

Notes

Tags

ARC, L2ARC

When to change

When testing compressed L2ARC feature

Data Type

boolean

Range

0=store compressed blocks in cache device, 1=store uncompressed blocks in cache device

Default

0

Change

Dynamic

Versions Affected

deprecated in v0.7.0 by new compressed ARC design

l2arc_meta_percent

Percent of ARC size allowed for L2ARC-only headers. Since L2ARC buffers are not evicted on memory pressure, too large amount of headers on system with irrationaly large L2ARC can render it slow or unusable. This parameter limits L2ARC writes and rebuild to achieve it.

l2arc_nocompress

Notes

Tags

ARC, L2ARC

When to change

When workload really require enormous L2ARC.

Data Type

int

Range

0 to 100

Default

33

Change

Dynamic

Versions Affected

v2.0 and later

l2arc_noprefetch

Disables writing prefetched, but unused, buffers to cache devices.

l2arc_noprefetch

Notes

Tags

ARC, L2ARC, prefetch

When to change

Setting to 0 can increase L2ARC hit rates for workloads where the ARC is too small for a read workload that benefits from prefetching. Also, if the main pool devices are very slow, setting to 0 can improve some workloads such as backups.

Data Type

boolean

Range

0=write prefetched but unused buffers to cache devices, 1=do not write prefetched but unused buffers to cache devices

Default

1

Change

Dynamic

Versions Affected

v0.6.0 and later

l2arc_norw

Disables writing to cache devices while they are being read.

l2arc_norw

Notes

Tags

ARC, L2ARC

When to change

In the early days of SSDs, some devices did not perform well when reading and writing simultaneously. Modern SSDs do not have these issues.

Data Type

boolean

Range

0=read and write simultaneously, 1=avoid writes when reading for antique SSDs

Default

0

Change

Dynamic

Versions Affected

all

l2arc_write_boost

Until the ARC fills, increases the L2ARC fill rate l2arc_write_max by l2arc_write_boost.

l2arc_write_boost

Notes

Tags

ARC, L2ARC

When to change

To fill the cache devices more aggressively after pool import.

Data Type

uint64

Units

bytes

Range

0 to UINT64_MAX

Default

8,388,608

Change

Dynamic

Versions Affected

all

l2arc_write_max

Maximum number of bytes to be written to each cache device for each L2ARC feed thread interval (see l2arc_feed_secs). The actual limit can be adjusted by l2arc_write_boost. By default l2arc_feed_secs is 1 second, delivering a maximum write workload to cache devices of 8 MiB/sec.

l2arc_write_max

Notes

Tags

ARC, L2ARC

When to change

If the cache devices can sustain the write workload, increasing the rate of cache device fill when workloads generate new data at a rate higher than l2arc_write_max can increase L2ARC hit rate

Data Type

uint64

Units

bytes

Range

1 to UINT64_MAX

Default

8,388,608

Change

Dynamic

Versions Affected

all

metaslab_aliquot

Sets the metaslab granularity. Nominally, ZFS will try to allocate this amount of data to a top-level vdev before moving on to the next top-level vdev. This is roughly similar to what would be referred to as the “stripe size” in traditional RAID arrays.

When tuning for HDDs, it can be more efficient to have a few larger, sequential writes to a device rather than switching to the next device. Monitoring the size of contiguous writes to the disks relative to the write throughput can be used to determine if increasing metaslab_aliquot can help. For modern devices, it is unlikely that decreasing metaslab_aliquot from the default will help.

If there is only one top-level vdev, this tunable is not used.

metaslab_aliquot

Notes

Tags

allocation, metaslab, vdev

When to change

If write performance increases as devices more efficiently write larger, contiguous blocks

Data Type

uint64

Units

bytes

Range

0 to UINT64_MAX

Default

524,288

Change

Dynamic

Versions Affected

all

metaslab_bias_enabled

Enables metaslab group biasing based on a top-level vdev’s utilization relative to the pool. Nominally, all top-level devs are the same size and the allocation is spread evenly. When the top-level vdevs are not of the same size, for example if a new (empty) top-level is added to the pool, this allows the new top-level vdev to get a larger portion of new allocations.

metaslab_bias_enabled

Notes

Tags

allocation, metaslab, vdev

When to change

If a new top-level vdev is added and you do not want to bias new allocations to the new top-level vdev

Data Type

boolean

Range

0=spread evenly across top-level vdevs, 1=bias spread to favor less full top-level vdevs

Default

1

Change

Dynamic

Versions Affected

v0.6.4 and later

zfs_metaslab_segment_weight_enabled

Enables metaslab allocation based on largest free segment rather than total amount of free space. The goal is to avoid metaslabs that exhibit free space fragmentation: when there is a lot of small free spaces, but few larger free spaces.

If zfs_metaslab_segment_weight_enabled is enabled, then metaslab_fragmentation_factor_enabled is ignored.

zfs _metaslab_segment_weight_enabled

Notes

Tags

allocation, metaslab

When to change

When testing allocation and fragmentation

Data Type

boolean

Range

0=do not consider metaslab fragmentation, 1=avoid metaslabs where free space is highly fragmented

Default

1

Change

Dynamic

Versions Affected

v0.7.0 and later

zfs_metaslab_switch_threshold

When using segment-based metaslab selection (see zfs_metaslab_segment_weight_enabled), continue allocating from the active metaslab until zfs_metaslab_switch_threshold worth of free space buckets have been exhausted.

zfs_metaslab_switch_threshold

Notes

Tags

allocation, metaslab

When to change

When testing allocation and fragmentation

Data Type

uint64

Units

free spaces

Range

0 to UINT64_MAX

Default

2

Change

Dynamic

Versions Affected

v0.7.0 and later

metaslab_debug_load

When enabled, all metaslabs are loaded into memory during pool import. Nominally, metaslab space map information is loaded and unloaded as needed (see metaslab_debug_unload)

It is difficult to predict how much RAM is required to store a space map. An empty or completely full metaslab has a small space map. However, a highly fragmented space map can consume significantly more memory.

Enabling metaslab_debug_load can increase pool import time.

metaslab_debug_load

Notes

Tags

allocation, memory, metaslab

When to change

When RAM is plentiful and pool import time is not a consideration

Data Type

boolean

Range

0=do not load all metaslab info at pool import, 1=dynamically load metaslab info as needed

Default

0

Change

Dynamic

Versions Affected

v0.6.4 and later

metaslab_debug_unload

When enabled, prevents metaslab information from being dynamically unloaded from RAM. Nominally, metaslab space map information is loaded and unloaded as needed (see metaslab_debug_load)

It is difficult to predict how much RAM is required to store a space map. An empty or completely full metaslab has a small space map. However, a highly fragmented space map can consume significantly more memory.

Enabling metaslab_debug_unload consumes RAM that would otherwise be freed.

metaslab_debug_unload

Notes

Tags

allocation, memory, metaslab

When to change

When RAM is plentiful and the penalty for dynamically reloading metaslab info from the pool is high

Data Type

boolean

Range

0=dynamically unload metaslab info, 1=unload metaslab info only upon pool export

Default

0

Change

Dynamic

Versions Affected

v0.6.4 and later

metaslab_fragmentation_factor_enabled

Enable use of the fragmentation metric in computing metaslab weights.

In version v0.7.0, if zfs_metaslab_segment_weight_enabled is enabled, then metaslab_fragmentation_factor_enabled is ignored.

metas lab_fragmentation_factor_enabled

Notes

Tags

allocation, metaslab

When to change

To test metaslab fragmentation

Data Type

boolean

Range

0=do not consider metaslab free space fragmentation, 1=try to avoid fragmented metaslabs

Default

1

Change

Dynamic

Versions Affected

v0.6.4 and later

metaslabs_per_vdev

When a vdev is added, it will be divided into approximately, but no more than, this number of metaslabs.

metaslabs_per_vdev

Notes

Tags

allocation, metaslab, vdev

When to change

When testing metaslab allocation

Data Type

uint64

Units

metaslabs

Range

16 to UINT64_MAX

Default

200

Change

Prior to pool creation or adding new top-level vdevs

Versions Affected

all

metaslab_preload_enabled

Enable metaslab group preloading. Each top-level vdev has a metaslab group. By default, up to 3 copies of metadata can exist and are distributed across multiple top-level vdevs. metaslab_preload_enabled allows the corresponding metaslabs to be preloaded, thus improving allocation efficiency.

metaslab_preload_enabled

Notes

Tags

allocation, metaslab

When to change

When testing metaslab allocation

Data Type

boolean

Range

0=do not preload metaslab info, 1=preload up to 3 metaslabs

Default

1

Change

Dynamic

Versions Affected

v0.6.4 and later

metaslab_lba_weighting_enabled

Modern HDDs have uniform bit density and constant angular velocity. Therefore, the outer recording zones are faster (higher bandwidth) than the inner zones by the ratio of outer to inner track diameter. The difference in bandwidth can be 2:1, and is often available in the HDD detailed specifications or drive manual. For HDDs when metaslab_lba_weighting_enabled is true, write allocation preference is given to the metaslabs representing the outer recording zones. Thus the allocation to metaslabs prefers faster bandwidth over free space.

If the devices are not rotational, yet misrepresent themselves to the OS as rotational, then disabling metaslab_lba_weighting_enabled can result in more even, free-space-based allocation.

metaslab_lba_weighting_enabled

Notes

Tags

allocation, metaslab, HDD, SSD

When to change

disable if using only SSDs and version v0.6.4 or earlier

Data Type

boolean

Range

0=do not use LBA weighting, 1=use LBA weighting

Default

1

Change

Dynamic

Verification

The rotational setting described by a block device in sysfs by observing /sys/ block/DISK_NAME/queue/rotational

Versions Affected

prior to v0.6.5, the check for non-rotation media did not exist

spa_config_path

By default, the zpool import command searches for pool information in the zpool.cache file. If the pool to be imported has an entry in zpool.cache then the devices do not have to be scanned to determine if they are pool members. The path to the cache file is spa_config_path.

For more information on zpool import and the -o cachefile and -d options, see the man page for zpool(8)

See also zfs_autoimport_disable

spa_config_path

Notes

Tags

import

When to change

If creating a non-standard distribution and the cachefile property is inconvenient

Data Type

string

Default

/etc/zfs/zpool.cache

Change

Dynamic, applies only to the next invocation of zpool import

Versions Affected

all

spa_asize_inflation

Multiplication factor used to estimate actual disk consumption from the size of data being written. The default value is a worst case estimate, but lower values may be valid for a given pool depending on its configuration. Pool administrators who understand the factors involved may wish to specify a more realistic inflation factor, particularly if they operate close to quota or capacity limits.

The worst case space requirement for allocation is single-sector max-parity RAIDZ blocks, in which case the space requirement is exactly 4 times the size, accounting for a maximum of 3 parity blocks. This is added to the maximum number of ZFS copies parameter (copies max=3). Additional space is required if the block could impact deduplication tables. Altogether, the worst case is 24.

If the estimation is not correct, then quotas or out-of-space conditions can lead to optimistic expectations of the ability to allocate. Applications are typically not prepared to deal with such failures and can misbehave.

spa_asize_inflation

Notes

Tags

allocation, SPA

When to change

If the allocation requirements for the workload are well known and quotas are used

Data Type

uint64

Units

unit

Range

1 to 24

Default

24

Change

Dynamic

Versions Affected

v0.6.3 and later

spa_load_verify_data

An extreme rewind import (see zpool import -X) normally performs a full traversal of all blocks in the pool for verification. If this parameter is set to 0, the traversal skips non-metadata blocks. It can be toggled once the import has started to stop or start the traversal of non-metadata blocks. See also spa_load_verify_metadata.

spa_load_verify_data

Notes

Tags

allocation, SPA

When to change

At the risk of data integrity, to speed extreme import of large pool

Data Type

boolean

Range

0=do not verify data upon pool import, 1=verify pool data upon import

Default

1

Change

Dynamic

Versions Affected

v0.6.4 and later

spa_load_verify_metadata

An extreme rewind import (see zpool import -X) normally performs a full traversal of all blocks in the pool for verification. If this parameter is set to 0, the traversal is not performed. It can be toggled once the import has started to stop or start the traversal. See spa_load_verify_data

spa_load_verify_metadata

Notes

Tags

import

When to change

At the risk of data integrity, to speed extreme import of large pool

Data Type

boolean

Range

0=do not verify metadata upon pool import, 1=verify pool metadata upon import

Default

1

Change

Dynamic

Versions Affected

v0.6.4 and later

spa_load_verify_maxinflight

Maximum number of concurrent I/Os during the data verification performed during an extreme rewind import (see zpool import -X)

spa_load_verify_maxinflight

Notes

Tags

import

When to change

During an extreme rewind import, to match the concurrent I/O capabilities of the pool devices

Data Type

int

Units

I/Os

Range

1 to MAX_INT

Default

10,000

Change

Dynamic

Versions Affected

v0.6.4 and later

spa_slop_shift

Normally, the last 3.2% (1/(2^spa_slop_shift)) of pool space is reserved to ensure the pool doesn’t run completely out of space, due to unaccounted changes (e.g. to the MOS). This also limits the worst-case time to allocate space. When less than this amount of free space exists, most ZPL operations (e.g. write, create) return error:no space (ENOSPC).

Changing spa_slop_shift affects the currently loaded ZFS module and all imported pools. spa_slop_shift is not stored on disk. Beware when importing full pools on systems with larger spa_slop_shift can lead to over-full conditions.

The minimum SPA slop space is limited to 128 MiB.

spa_slop_shift

Notes

Tags

allocation, SPA

When to change

For large pools, when 3.2% may be too conservative and more usable space is desired, consider increasing spa_slop_shift

Data Type

int

Units

shift

Range

1 to MAX_INT, however the practical upper limit is 15 for a system with 4TB of RAM

Default

5

Change

Dynamic

Versions Affected

v0.6.5 and later

zfetch_array_rd_sz

If prefetching is enabled, do not prefetch blocks larger than zfetch_array_rd_sz size.

zfetch_array_rd_sz

Notes

Tags

prefetch

When to change

To allow prefetching when using large block sizes

Data Type

unsigned long

Units

bytes

Range

0 to MAX_ULONG

Default

1,048,576 (1 MiB)

Change

Dynamic

Versions Affected

all

zfetch_max_distance

Limits the maximum number of bytes to prefetch per stream.

zfetch_max_distance

Notes

Tags

prefetch

When to change

Consider increasing read workloads that use large blocks and exhibit high prefetch hit ratios

Data Type

uint

Units

bytes

Range

0 to UINT_MAX

Default

8,388,608

Change

Dynamic

Versions Affected

v0.7.0

zfetch_max_streams

Maximum number of prefetch streams per file.

For version v0.7.0 and later, when prefetching small files the number of prefetch streams is automatically reduced below to prevent the streams from overlapping.

zfetch_max_streams

Notes

Tags

prefetch

When to change

If the workload benefits from prefetching and has more than zfetch_max_streams concurrent reader threads

Data Type

uint

Units

streams

Range

1 to MAX_UINT

Default

8

Change

Dynamic

Versions Affected

all

zfetch_min_sec_reap

Prefetch streams that have been accessed in zfetch_min_sec_reap seconds are automatically stopped.

zfetch_min_sec_reap

Notes

Tags

prefetch

When to change

To test prefetch efficiency

Data Type

uint

Units

seconds

Range

0 to MAX_UINT

Default

2

Change

Dynamic

Versions Affected

all

zfs_arc_dnode_limit_percent

Percentage of ARC metadata space that can be used for dnodes.

The value calculated for zfs_arc_dnode_limit_percent can be overridden by zfs_arc_dnode_limit.

zfs_arc_dnode_limit_percent

Notes

Tags

ARC

When to change

Consider increasing if arc_prune is using excessive system time and /proc/spl/kstat/zfs/arcstats shows arc_dnode_size is near or over arc_dnode_limit

Data Type

int

Units

percent of arc_meta_limit

Range

0 to 100

Default

10

Change

Dynamic

Versions Affected

v0.7.0 and later

zfs_arc_dnode_limit

When the number of bytes consumed by dnodes in the ARC exceeds zfs_arc_dnode_limit bytes, demand for new metadata can take from the space consumed by dnodes.

The default value 0, indicates that a percent which is based on zfs_arc_dnode_limit_percent of the ARC meta buffers that may be used for dnodes.

zfs_arc_dnode_limit is similar to zfs_arc_meta_prune which serves a similar purpose for metadata.

zfs_arc_dnode_limit

Notes

Tags

ARC

When to change

Consider increasing if arc_prune is using excessive system time and /proc/spl/kstat/zfs/arcstats shows arc_dnode_size is near or over arc_dnode_limit

Data Type

uint64

Units

bytes

Range

0 to MAX_UINT64

Default

0 (uses zfs_arc_dnode_lim it_percent)

Change

Dynamic

Versions Affected

v0.7.0 and later

zfs_arc_dnode_reduce_percent

Percentage of ARC dnodes to try to evict in response to demand for non-metadata when the number of bytes consumed by dnodes exceeds zfs_arc_dnode_limit.

zfs_arc_dnode_reduce_percent

Notes

Tags

ARC

When to change

Testing dnode cache efficiency

Data Type

uint64

Units

percent of size of dnode space used above zfs_arc_d node_limit

Range

0 to 100

Default

10

Change

Dynamic

Versions Affected

v0.7.0 and later

zfs_arc_average_blocksize

The ARC’s buffer hash table is sized based on the assumption of an average block size of zfs_arc_average_blocksize. The default of 8 KiB uses approximately 1 MiB of hash table per 1 GiB of physical memory with 8-byte pointers.

zfs_arc_average_blocksize

Notes

Tags

ARC, memory

When to change

For workloads where the known average blocksize is larger, increasing zfs_arc_average_blocksize can reduce memory usage

Data Type

int

Units

bytes

Range

512 to 16,777,216

Default

8,192

Change

Prior to zfs module load

Versions Affected

all

zfs_arc_evict_batch_limit

Number ARC headers to evict per sublist before proceeding to another sublist. This batch-style operation prevents entire sublists from being evicted at once but comes at a cost of additional unlocking and locking.

zfs_arc_evict_batch_limit

Notes

Tags

ARC

When to change

Testing ARC multilist features

Data Type

int

Units

count of ARC headers

Range

1 to INT_MAX

Default

10

Change

Dynamic

Versions Affected

v0.6.5 and later

zfs_arc_grow_retry

When the ARC is shrunk due to memory demand, do not retry growing the ARC for zfs_arc_grow_retry seconds. This operates as a damper to prevent oscillating grow/shrink cycles when there is memory pressure.

If zfs_arc_grow_retry = 0, the internal default of 5 seconds is used.

zfs_arc_grow_retry

Notes

Tags

ARC, memory

When to change

TBD

Data Type

int

Units

seconds

Range

1 to MAX_INT

Default

0

Change

Dynamic

Versions Affected

v0.6.5 and later

zfs_arc_lotsfree_percent

Throttle ARC memory consumption, effectively throttling I/O, when free system memory drops below this percentage of total system memory. Setting zfs_arc_lotsfree_percent to 0 disables the throttle.

The arcstat_memory_throttle_count counter in /proc/spl/kstat/arcstats can indicate throttle activity.

zfs_arc_lotsfree_percent

Notes

Tags

ARC, memory

When to change

TBD

Data Type

int

Units

percent

Range

0 to 100

Default

10

Change

Dynamic

Versions Affected

v0.6.5 and later

zfs_arc_max

Maximum size of ARC in bytes. If set to 0 then the maximum ARC size is set to 1/2 of system RAM.

zfs_arc_max can be changed dynamically with some caveats. It cannot be set back to 0 while running and reducing it below the current ARC size will not cause the ARC to shrink without memory pressure to induce shrinking.

zfs_arc_max

Notes

Tags

ARC, memory

When to change

Reduce if ARC competes too much with other applications, increase if ZFS is the primary application and can use more RAM

Data Type

uint64

Units

bytes

Range

67,108,864 to RAM size in bytes

Default

0 (uses default of RAM size in bytes / 2)

Change

Dynamic (see description above)

Verification

c column in arcstats.py or /proc/spl/kstat/zfs/arcstats entry c_max

Versions Affected

all

zfs_arc_meta_adjust_restarts

The number of restart passes to make while scanning the ARC attempting the free buffers in order to stay below the zfs_arc_meta_limit.

zfs_arc_meta_adjust_restarts

Notes

Tags

ARC

When to change

Testing ARC metadata adjustment feature

Data Type

int

Units

restarts

Range

0 to INT_MAX

Default

4,096

Change

Dynamic

Versions Affected

v0.6.5 and later

zfs_arc_meta_limit

Sets the maximum allowed size metadata buffers in the ARC. When zfs_arc_meta_limit is reached metadata buffers are reclaimed, even if the overall c_max has not been reached.

In version v0.7.0, with a default value = 0, zfs_arc_meta_limit_percent is used to set arc_meta_limit

zfs_arc_meta_limit

Notes

Tags

ARC

When to change

For workloads where the metadata to data ratio in the ARC can be changed to improve ARC hit rates

Data Type

uint64

Units

bytes

Range

0 to c_max

Default

0

Change

Dynamic, except that it cannot be set back to 0 for a specific percent of the ARC; it must be set to an explicit value

Verification

/proc/spl/kstat/zfs/arcstats entry arc_meta_limit

Versions Affected

all

zfs_arc_meta_limit_percent

Sets the limit to ARC metadata, arc_meta_limit, as a percentage of the maximum size target of the ARC, c_max

Prior to version v0.7.0, the zfs_arc_meta_limit was used to set the limit as a fixed size. zfs_arc_meta_limit_percent provides a more convenient interface for setting the limit.

zfs_arc_meta_limit_percent

Notes

Tags

ARC

When to change

For workloads where the metadata to data ratio in the ARC can be changed to improve ARC hit rates

Data Type

uint64

Units

percent of c_max

Range

0 to 100

Default

75

Change

Dynamic

Verification

/proc/spl/kstat/zfs/arcstats entry arc_meta_limit

Versions Affected

v0.7.0 and later

zfs_arc_meta_min

The minimum allowed size in bytes that metadata buffers may consume in the ARC. This value defaults to 0 which disables a floor on the amount of the ARC devoted meta data.

When evicting data from the ARC, if the metadata_size is less than arc_meta_min then data is evicted instead of metadata.

zfs_arc_meta_min

Notes

Tags

ARC

When to change

Data Type

uint64

Units

bytes

Range

16,777,216 to c_max

Default

0 (use internal default 16 MiB)

Change

Dynamic

Verification

/proc/spl/kstat/zfs/arcstats entry arc_meta_min

Versions Affected

all

zfs_arc_meta_prune

zfs_arc_meta_prune sets the number of dentries and znodes to be scanned looking for entries which can be dropped. This provides a mechanism to ensure the ARC can honor the arc_meta_limit and reclaim otherwise pinned ARC buffers. Pruning may be required when the ARC size drops to arc_meta_limit because dentries and znodes can pin buffers in the ARC. Increasing this value will cause to dentry and znode caches to be pruned more aggressively and the arc_prune thread becomes more active. Setting zfs_arc_meta_prune to 0 will disable pruning.

zfs_arc_meta_prune

Notes

Tags

ARC

When to change

TBD

Data Type

uint64

Units

entries

Range

0 to INT_MAX

Default

10,000

Change

Dynamic

! Verification

Prune activity is counted by the /proc/spl/kstat/zfs/arcstats entry arc_prune

Versions Affected

v0.6.5 and later

zfs_arc_meta_strategy

Defines the strategy for ARC metadata eviction (meta reclaim strategy). A value of 0 (META_ONLY) will evict only the ARC metadata. A value of 1 (BALANCED) indicates that additional data may be evicted if required in order to evict the requested amount of metadata.

zfs_arc_meta_strategy

Notes

Tags

ARC

When to change

Testing ARC metadata eviction

Data Type

int

Units

enum

Range

0=evict metadata only, 1=also evict data buffers if they can free metadata buffers for eviction

Default

1 (BALANCED)

Change

Dynamic

Versions Affected

v0.6.5 and later

zfs_arc_min

Minimum ARC size limit. When the ARC is asked to shrink, it will stop shrinking at c_min as tuned by zfs_arc_min.

zfs_arc_min

Notes

Tags

ARC

When to change

If the primary focus of the system is ZFS, then increasing can ensure the ARC gets a minimum amount of RAM

Data Type

uint64

Units

bytes

Range

33,554,432 to c_max

Default

For kernel: greater of 33,554,432 (32 MiB) and memory size / 32. For user-land: greater of 33,554,432 (32 MiB) and c_max / 2.

Change

Dynamic

Verification

/proc/spl/kstat/zfs/arcstats entry c_min

Versions Affected

all

zfs_arc_min_prefetch_ms

Minimum time prefetched blocks are locked in the ARC.

A value of 0 represents the default of 1 second. However, once changed, dynamically setting to 0 will not return to the default.

zfs_arc_min_prefetch_ms

Notes

Tags

ARC, prefetch

When to change

TBD

Data Type

int

Units

milliseconds

Range

1 to INT_MAX

Default

0 (use internal default of 1000 ms)

Change

Dynamic

Versions Affected

v0.8.0 and later

zfs_arc_min_prescient_prefetch_ms

Minimum time “prescient prefetched” blocks are locked in the ARC. These blocks are meant to be prefetched fairly aggressively ahead of the code that may use them.

A value of 0 represents the default of 6 seconds. However, once changed, dynamically setting to 0 will not return to the default.

z fs_arc_min_prescient_prefetch_ms

Notes

Tags

ARC, prefetch

When to change

TBD

Data Type

int

Units

milliseconds

Range

1 to INT_MAX

Default

0 (use internal default of 6000 ms)

Change

Dynamic

Versions Affected

v0.8.0 and later

zfs_multilist_num_sublists

To allow more fine-grained locking, each ARC state contains a series of lists (sublists) for both data and metadata objects. Locking is performed at the sublist level. This parameters controls the number of sublists per ARC state, and also applies to other uses of the multilist data structure.

zfs_multilist_num_sublists

Notes

Tags

ARC

When to change

TBD

Data Type

int

Units

lists

Range

1 to INT_MAX

Default

0 (internal value is greater of number of online CPUs or 4)

Change

Prior to zfs module load

Versions Affected

v0.7.0 and later

zfs_arc_overflow_shift

The ARC size is considered to be overflowing if it exceeds the current ARC target size (/proc/spl/kstat/zfs/arcstats entry c) by a threshold determined by zfs_arc_overflow_shift. The threshold is calculated as a fraction of c using the formula: (ARC target size) c >> zfs_arc_overflow_shift

The default value of 8 causes the ARC to be considered to be overflowing if it exceeds the target size by 1/256th (0.3%) of the target size.

When the ARC is overflowing, new buffer allocations are stalled until the reclaim thread catches up and the overflow condition no longer exists.

zfs_arc_overflow_shift

Notes

Tags

ARC

When to change

TBD

Data Type

int

Units

shift

Range

1 to INT_MAX

Default

8

Change

Dynamic

Versions Affected

v0.6.5 and later

zfs_arc_p_min_shift

arc_p_min_shift is used to shift of ARC target size (/proc/spl/kstat/zfs/arcstats entry c) for calculating both minimum and maximum most recently used (MRU) target size (/proc/spl/kstat/zfs/arcstats entry p)

A value of 0 represents the default setting of arc_p_min_shift = 4. However, once changed, dynamically setting zfs_arc_p_min_shift to 0 will not return to the default.

zfs_arc_p_min_shift

Notes

Tags

ARC

When to change

TBD

Data Type

int

Units

shift

Range

1 to INT_MAX

Default

0 (internal default = 4)

Change

Dynamic

Verification

Observe changes to /proc/spl/kstat/zfs/arcstats entry p

Versions Affected

all

zfs_arc_p_dampener_disable

When data is being added to the ghost lists, the MRU target size is adjusted. The amount of adjustment is based on the ratio of the MRU/MFU sizes. When enabled, the ratio is capped to 10, avoiding large adjustments.

zfs_arc_p_dampener_disable

Notes

Tags

ARC

When to change

Testing ARC ghost list behaviour

Data Type

boolean

Range

0=avoid large adjustments, 1=permit large adjustments

Default

1

Change

Dynamic

Versions Affected

v0.6.4 and later

zfs_arc_shrink_shift

arc_shrink_shift is used to adjust the ARC target sizes when large reduction is required. The current ARC target size, c, and MRU size p can be reduced by by the current size >> arc_shrink_shift. For the default value of 7, this reduces the target by approximately 0.8%.

A value of 0 represents the default setting of arc_shrink_shift = 7. However, once changed, dynamically setting arc_shrink_shift to 0 will not return to the default.

zfs_arc_shrink_shift

Notes

Tags

ARC, memory

When to change

During memory shortfall, reducing zfs_arc_shrink_shift increases the rate of ARC shrinkage

Data Type

int

Units

shift

Range

1 to INT_MAX

Default

0 (arc_shrink_shift = 7)

Change

Dynamic

Versions Affected

all

zfs_arc_pc_percent

zfs_arc_pc_percent allows ZFS arc to play more nicely with the kernel’s LRU pagecache. It can guarantee that the arc size won’t collapse under scanning pressure on the pagecache, yet still allows arc to be reclaimed down to zfs_arc_min if necessary. This value is specified as percent of pagecache size (as measured by NR_FILE_PAGES) where that percent may exceed 100. This only operates during memory pressure/reclaim.

zfs_arc_pc_percent

Notes

Tags

ARC, memory

When to change

When using file systems under memory shortfall, if the page scanner causes the ARC to shrink too fast, then adjusting zfs_arc_pc_percent can reduce the shrink rate

Data Type

int

Units

percent

Range

0 to 100

Default

0 (disabled)

Change

Dynamic

Versions Affected

v0.7.0 and later

zfs_arc_sys_free

zfs_arc_sys_free is the target number of bytes the ARC should leave as free memory on the system. Defaults to the larger of 1/64 of physical memory or 512K. Setting this option to a non-zero value will override the default.

A value of 0 represents the default setting of larger of 1/64 of physical memory or 512 KiB. However, once changed, dynamically setting zfs_arc_sys_free to 0 will not return to the default.

zfs_arc_sys_free

Notes

Tags

ARC, memory

When to change

Change if more free memory is desired as a margin against memory demand by applications

Data Type

ulong

Units

bytes

Range

0 to ULONG_MAX

Default

0 (default to larger of 1/64 of physical memory or 512 KiB)

Change

Dynamic

Versions Affected

v0.6.5 and later

zfs_autoimport_disable

Disable reading zpool.cache file (see spa_config_path) when loading the zfs module.

zfs_autoimport_disable

Notes

Tags

import

When to change

Leave as default so that zfs behaves as other Linux kernel modules

Data Type

boolean

Range

0=read zpool.cache at module load, 1=do not read zpool.cache at module load

Default

1

Change

Dynamic

Versions Affected

v0.6.4 and later

zfs_commit_timeout_pct

zfs_commit_timeout_pct controls the amount of time that a log (ZIL) write block (lwb) remains “open” when it isn’t “full” and it has a thread waiting to commit to stable storage. The timeout is scaled based on a percentage of the last lwb latency to avoid significantly impacting the latency of each individual intent log transaction (itx).

zfs_commit_timeout_pct

Notes

Tags

ZIL

When to change

TBD

Data Type

int

Units

percent

Range

1 to 100

Default

5

Change

Dynamic

Versions Affected

v0.8.0

zfs_dbgmsg_enable

Internally ZFS keeps a small log to facilitate debugging. The contents of the log are in the /proc/spl/kstat/zfs/dbgmsg file.
Writing 0 to /proc/spl/kstat/zfs/dbgmsg file clears the log.

See also zfs_dbgmsg_maxsize

zfs_dbgmsg_enable

Notes

Tags

debug

When to change

To view ZFS internal debug log

Data Type

boolean

Range

0=do not log debug messages, 1=log debug messages

Default

0 (1 for debug builds)

Change

Dynamic

Versions Affected

v0.6.5 and later

zfs_dbgmsg_maxsize

The /proc/spl/kstat/zfs/dbgmsg file size limit is set by zfs_dbgmsg_maxsize.

See also zfs_dbgmsg_enable

zfs_dbgmsg_maxsize

Notes

Tags

debug

When to change

TBD

Data Type

int

Units

bytes

Range

0 to INT_MAX

Default

4 MiB

Change

Dynamic

Versions Affected

v0.6.5 and later

zfs_dbuf_state_index

The zfs_dbuf_state_index feature is currently unused. It is normally used for controlling values in the /proc/spl/kstat/zfs/dbufs file.

zfs_dbuf_state_index

Notes

Tags

debug

When to change

Do not change

Data Type

int

Units

TBD

Range

TBD

Default

0

Change

Dynamic

Versions Affected

v0.6.5 and later

zfs_deadman_enabled

When a pool sync operation takes longer than zfs_deadman_synctime_ms milliseconds, a “slow spa_sync” message is logged to the debug log (see zfs_dbgmsg_enable). If zfs_deadman_enabled is set to 1, then all pending IO operations are also checked and if any haven’t completed within zfs_deadman_synctime_ms milliseconds, a “SLOW IO” message is logged to the debug log and a “deadman” system event (see zpool events command) with the details of the hung IO is posted.

zfs_deadman_enabled

Notes

Tags

debug

When to change

To disable logging of slow I/O

Data Type

boolean

Range

0=do not log slow I/O, 1=log slow I/O

Default

1

Change

Dynamic

Versions Affected

v0.8.0

zfs_deadman_checktime_ms

Once a pool sync operation has taken longer than zfs_deadman_synctime_ms milliseconds, continue to check for slow operations every zfs_deadman_checktime_ms milliseconds.

zfs_deadman_checktime_ms

Notes

Tags

debug

When to change

When debugging slow I/O

Data Type

ulong

Units

milliseconds

Range

1 to ULONG_MAX

Default

60,000 (1 minute)

Change

Dynamic

Versions Affected

v0.8.0

zfs_deadman_ziotime_ms

When an individual I/O takes longer than zfs_deadman_ziotime_ms milliseconds, then the operation is considered to be “hung”. If zfs_deadman_enabled is set then the deadman behaviour is invoked as described by the zfs_deadman_failmode option.

zfs_deadman_ziotime_ms

Notes

Tags

debug

When to change

Testing ABD features

Data Type

ulong

Units

milliseconds

Range

1 to ULONG_MAX

Default

300,000 (5 minutes)

Change

Dynamic

Versions Affected

v0.8.0

zfs_deadman_synctime_ms

The I/O deadman timer expiration time has two meanings

  1. determines when the spa_deadman() logic should fire, indicating the txg sync has not completed in a timely manner

  2. determines if an I/O is considered “hung”

In version v0.8.0, any I/O that has not completed in zfs_deadman_synctime_ms is considered “hung” resulting in one of three behaviors controlled by the zfs_deadman_failmode parameter.

zfs_deadman_synctime_ms takes effect if zfs_deadman_enabled = 1.

zfs_deadman_synctime_ms

Notes

Tags

debug

When to change

When debugging slow I/O

Data Type

ulong

Units

milliseconds

Range

1 to ULONG_MAX

Default

600,000 (10 minutes)

Change

Dynamic

Versions Affected

v0.6.5 and later

zfs_deadman_failmode

zfs_deadman_failmode controls the behavior of the I/O deadman timer when it detects a “hung” I/O. Valid values are:

  • wait - Wait for the “hung” I/O (default)

  • continue - Attempt to recover from a “hung” I/O

  • panic - Panic the system

zfs_deadman_failmode

Notes

Tags

debug

When to change

In some cluster cases, panic can be appropriate

Data Type

string

Range

wait, continue, or panic

Default

wait

Change

Dynamic

Versions Affected

v0.8.0

zfs_dedup_prefetch

ZFS can prefetch deduplication table (DDT) entries. zfs_dedup_prefetch allows DDT prefetches to be enabled.

zfs_dedup_prefetch

Notes

Tags

prefetch, memory

When to change

For systems with limited RAM using the dedup feature, disabling deduplication table prefetch can reduce memory pressure

Data Type

boolean

Range

0=do not prefetch, 1=prefetch dedup table entries

Default

0

Change

Dynamic

Versions Affected

v0.6.5 and later

zfs_delete_blocks

zfs_delete_blocks defines a large file for the purposes of delete. Files containing more than zfs_delete_blocks will be deleted asynchronously while smaller files are deleted synchronously. Decreasing this value reduces the time spent in an unlink(2) system call at the expense of a longer delay before the freed space is available.

The zfs_delete_blocks value is specified in blocks, not bytes. The size of blocks can vary and is ultimately limited by the filesystem’s recordsize property.

zfs_delete_blocks

Notes

Tags

filesystem, delete

When to change

If applications delete large files and blocking on unlink(2) is not desired

Data Type

ulong

Units

blocks

Range

1 to ULONG_MAX

Default

20,480

Change

Dynamic

Versions Affected

all

zfs_delay_min_dirty_percent

The ZFS write throttle begins to delay each transaction when the amount of dirty data reaches the threshold zfs_delay_min_dirty_percent of zfs_dirty_data_max. This value should be >= zfs_vdev_async_write_active_max_dirty_percent.

zfs_delay_min_dirty_percent

Notes

Tags

write_throttle

When to change

See section “ZFS TRANSACTION DELAY”

Data Type

int

Units

percent

Range

0 to 100

Default

60

Change

Dynamic

Versions Affected

v0.6.4 and later

zfs_delay_scale

zfs_delay_scale controls how quickly the ZFS write throttle transaction delay approaches infinity. Larger values cause longer delays for a given amount of dirty data.

For the smoothest delay, this value should be about 1 billion divided by the maximum number of write operations per second the pool can sustain. The throttle will smoothly handle between 10x and 1/10th zfs_delay_scale.

Note: zfs_delay_scale * zfs_dirty_data_max must be < 2^64.

zfs_delay_scale

Notes

Tags

write_throttle

When to change

See section “ZFS TRANSACTION DELAY”

Data Type

ulong

Units

scalar (nanoseconds)

Range

0 to ULONG_MAX

Default

500,000

Change

Dynamic

Versions Affected

v0.6.4 and later

zfs_dirty_data_max

zfs_dirty_data_max is the ZFS write throttle dirty space limit. Once this limit is exceeded, new writes are delayed until space is freed by writes being committed to the pool.

zfs_dirty_data_max takes precedence over zfs_dirty_data_max_percent.

zfs_dirty_data_max

Notes

Tags

write_throttle

When to change

See section “ZFS TRANSACTION DELAY”

Data Type

ulong

Units

bytes

Range

1 to zfs_d irty_data_max_max

Default

10% of physical RAM

Change

Dynamic

Versions Affected

v0.6.4 and later

zfs_dirty_data_max_percent

zfs_dirty_data_max_percent is an alternative method of specifying zfs_dirty_data_max, the ZFS write throttle dirty space limit. Once this limit is exceeded, new writes are delayed until space is freed by writes being committed to the pool.

zfs_dirty_data_max takes precedence over zfs_dirty_data_max_percent.

zfs_dirty_data_max_percent

Notes

Tags

write_throttle

When to change

See section “ZFS TRANSACTION DELAY”

Data Type

int

Units

percent

Range

1 to 100

Default

10% of physical RAM

Change

Prior to zfs module load or a memory hot plug event

Versions Affected

v0.6.4 and later

zfs_dirty_data_max_max

zfs_dirty_data_max_max is the maximum allowable value of zfs_dirty_data_max.

zfs_dirty_data_max_max takes precedence over zfs_dirty_data_max_max_percent.

zfs_dirty_data_max_max

Notes

Tags

write_throttle

When to change

See section “ZFS TRANSACTION DELAY”

Data Type

ulong

Units

bytes

Range

1 to physical RAM size

Default

25% of physical RAM

Change

Prior to zfs module load

Versions Affected

v0.6.4 and later

zfs_dirty_data_max_max_percent

zfs_dirty_data_max_max_percent an alternative to zfs_dirty_data_max_max for setting the maximum allowable value of zfs_dirty_data_max

zfs_dirty_data_max_max takes precedence over zfs_dirty_data_max_max_percent

zfs_dirty_data_max_max_percent

Notes

Tags

write_throttle

When to change

See section “ZFS TRANSACTION DELAY”

Data Type

int

Units

percent

Range

1 to 100

Default

25% of physical RAM

Change

Prior to zfs module load

Versions Affected

v0.6.4 and later

zfs_dirty_data_sync

When there is at least zfs_dirty_data_sync dirty data, a transaction group sync is started. This allows a transaction group sync to occur more frequently than the transaction group timeout interval (see zfs_txg_timeout) when there is dirty data to be written.

zfs_dirty_data_sync

Notes

Tags

write_throttle, ZIO_scheduler

When to change

TBD

Data Type

ulong

Units

bytes

Range

1 to ULONG_MAX

Default

67,108,864 (64 MiB)

Change

Dynamic

Versions Affected

v0.6.4 through v0.8.x, deprecation planned for v2

zfs_dirty_data_sync_percent

When there is at least zfs_dirty_data_sync_percent of zfs_dirty_data_max dirty data, a transaction group sync is started. This allows a transaction group sync to occur more frequently than the transaction group timeout interval (see zfs_txg_timeout) when there is dirty data to be written.

zfs_dirty_data_sync_percent

Notes

Tags

write_throttle, ZIO_scheduler

When to change

TBD

Data Type

int

Units

percent

Range

1 to zfs_vdev_async_write_ac tive_min_dirty_percent

Default

20

Change

Dynamic

Versions Affected

planned for v2, deprecates zfs_dirt y_data_sync

zfs_fletcher_4_impl

Fletcher-4 is the default checksum algorithm for metadata and data. When the zfs kernel module is loaded, a set of microbenchmarks are run to determine the fastest algorithm for the current hardware. The zfs_fletcher_4_impl parameter allows a specific implementation to be specified other than the default (fastest). Selectors other than fastest and scalar require instruction set extensions to be available and will only appear if ZFS detects their presence. The scalar implementation works on all processors.

The results of the microbenchmark are visible in the /proc/spl/kstat/zfs/fletcher_4_bench file. Larger numbers indicate better performance. Since ZFS is processor endian-independent, the microbenchmark is run against both big and little-endian transformation.

zfs_fletcher_4_impl

Notes

Tags

CPU, checksum

When to change

Testing Fletcher-4 algorithms

Data Type

string

Range

fastest, scalar, superscalar, superscalar4, sse2, ssse3, avx2, avx512f, or aarch64_neon depending on hardware support

Default

fastest

Change

Dynamic

Versions Affected

v0.7.0 and later

zfs_free_bpobj_enabled

The processing of the free_bpobj object can be enabled by zfs_free_bpobj_enabled

zfs_free_bpobj_enabled

Notes

Tags

delete

When to change

If there’s a problem with processing free_bpobj (e.g. i/o error or bug)

Data Type

boolean

Range

0=do not process free_bpobj objects, 1=process free_bpobj objects

Default

1

Change

Dynamic

Versions Affected

v0.7.0 and later

zfs_free_max_blocks

zfs_free_max_blocks sets the maximum number of blocks to be freed in a single transaction group (txg). For workloads that delete (free) large numbers of blocks in a short period of time, the processing of the frees can negatively impact other operations, including txg commits. zfs_free_max_blocks acts as a limit to reduce the impact.

zfs_free_max_blocks

Notes

Tags

filesystem, delete

When to change

For workloads that delete large files, zfs_free_max_blocks can be adjusted to meet performance requirements while reducing the impacts of deletion

Data Type

ulong

Units

blocks

Range

1 to ULONG_MAX

Default

100,000

Change

Dynamic

Versions Affected

v0.7.0 and later

zfs_vdev_async_read_max_active

Maximum asynchronous read I/Os active to each device.

zfs_vdev_async_read_max_active

Notes

Tags

vdev, ZIO_scheduler

When to change

See ZFS I/O Scheduler

Data Type

uint32

Units

I/O operations

Range

1 to zfs_vdev_ma x_active

Default

3

Change

Dynamic

Versions Affected

v0.6.4 and later

zfs_vdev_async_read_min_active

Minimum asynchronous read I/Os active to each device.

zfs_vdev_async_read_min_active

Notes

Tags

vdev, ZIO_scheduler

When to change

See ZFS I/O Scheduler

Data Type

uint32

Units

I/O operations

Range

1 to ( zfs_vdev_async_read_max_active - 1)

Default

1

Change

Dynamic

Versions Affected

v0.6.4 and later

zfs_vdev_async_write_active_max_dirty_percent

When the amount of dirty data exceeds the threshold zfs_vdev_async_write_active_max_dirty_percent of zfs_dirty_data_max dirty data, then zfs_vdev_async_write_max_active is used to limit active async writes. If the dirty data is between zfs_vdev_async_write_active_min_dirty_percent and zfs_vdev_async_write_active_max_dirty_percent, the active I/O limit is linearly interpolated between zfs_vdev_async_write_min_active and zfs_vdev_async_write_max_active

zfs_vdev_asyn c_write_active_max_dirty_percent

Notes

Tags

vdev, Z IO_scheduler

When to change

See ZFS I/O Sch eduler

Data Type

int

Units

percent of zfs_dirty_d ata_max

Range

0 to 100

Default

60

Change

Dynamic

Versions Affected

v0.6.4 and later

zfs_vdev_async_write_active_min_dirty_percent

If the amount of dirty data is between zfs_vdev_async_write_active_min_dirty_percent and zfs_vdev_async_write_active_max_dirty_percent of zfs_dirty_data_max, the active I/O limit is linearly interpolated between zfs_vdev_async_write_min_active and zfs_vdev_async_write_max_active

zfs_vdev_asyn c_write_active_min_dirty_percent

Notes

Tags

vdev, Z IO_scheduler

When to change

See ZFS I/O Sch eduler

Data Type

int

Units

percent of zfs_dirty_data_max

Range

0 to (z fs_vdev_async_write_active_max_d irty_percent - 1)

Default

30

Change

Dynamic

Versions Affected

v0.6.4 and later

zfs_vdev_async_write_max_active

zfs_vdev_async_write_max_active sets the maximum asynchronous write I/Os active to each device.

zfs_vdev_async_write_max_active

Notes

Tags

vdev, ` ZIO_scheduler <#zio-scheduler>`__

When to change

See ZFS I/O S cheduler

Data Type

uint32

Units

I/O operations

Range

1 to zfs_vdev_max _active

Default

10

Change

Dynamic

Versions Affected

v0.6.4 and later

zfs_vdev_async_write_min_active

zfs_vdev_async_write_min_active sets the minimum asynchronous write I/Os active to each device.

Lower values are associated with better latency on rotational media but poorer resilver performance. The default value of 2 was chosen as a compromise. A value of 3 has been shown to improve resilver performance further at a cost of further increasing latency.

zfs_vdev_async_write_min_active

Notes

Tags

vdev, ` ZIO_scheduler <#zio-scheduler>`__

When to change

See ZFS I/O S cheduler

Data Type

uint32

Units

I/O operations

Range

1 to zfs _vdev_async_write_max_active

Default

1 for v0.6.x, 2 for v0.7.0 and later

Change

Dynamic

Versions Affected

v0.6.4 and later

zfs_vdev_max_active

The maximum number of I/Os active to each device. Ideally, zfs_vdev_max_active >= the sum of each queue’s max_active.

Once queued to the device, the ZFS I/O scheduler is no longer able to prioritize I/O operations. The underlying device drivers have their own scheduler and queue depth limits. Values larger than the device’s maximum queue depth can have the affect of increased latency as the I/Os are queued in the intervening device driver layers.

zfs_vdev_max_active

Notes

Tags

vdev, ZIO_scheduler

When to change

See ZFS I/O Scheduler

Data Type

uint32

Units

I/O operations

Range

sum of each queue’s min_active to UINT32_MAX

Default

1,000

Change

Dynamic

Versions Affected

v0.6.4 and later

zfs_vdev_scrub_max_active

zfs_vdev_scrub_max_active sets the maximum scrub or scan read I/Os active to each device.

zfs_vdev_scrub_max_active

Notes

Tags

vdev, ZIO_scheduler, scrub, resilver

When to change

See ZFS I/O Scheduler

Data Type

uint32

Units

I/O operations

Range

1 to zfs_vd ev_max_active

Default

2

Change

Dynamic

Versions Affected

v0.6.4 and later

zfs_vdev_scrub_min_active

zfs_vdev_scrub_min_active sets the minimum scrub or scan read I/Os active to each device.

zfs_vdev_scrub_min_active

Notes

Tags

vdev, ZIO_scheduler, scrub, resilver

When to change

See ZFS I/O Scheduler

Data Type

uint32

Units

I/O operations

Range

1 to zfs_vdev_scrub_max _active

Default

1

Change

Dynamic

Versions Affected

v0.6.4 and later

zfs_vdev_sync_read_max_active

Maximum synchronous read I/Os active to each device.

zfs_vdev_sync_read_max_active

Notes

Tags

vdev, ZIO_scheduler

When to change

See ZFS I/O Scheduler

Data Type

uint32

Units

I/O operations

Range

1 to zfs_vdev_m ax_active

Default

10

Change

Dynamic

Versions Affected

v0.6.4 and later

zfs_vdev_sync_read_min_active

zfs_vdev_sync_read_min_active sets the minimum synchronous read I/Os active to each device.

zfs_vdev_sync_read_min_active

Notes

Tags

vdev, ZIO_scheduler

When to change

See ZFS I/O Scheduler

Data Type

uint32

Units

I/O operations

Range

1 to zfs_vdev_sync_read_max_active

Default

10

Change

Dynamic

Versions Affected

v0.6.4 and later

zfs_vdev_sync_write_max_active

zfs_vdev_sync_write_max_active sets the maximum synchronous write I/Os active to each device.

zfs_vdev_sync_write_max_active

Notes

Tags

vdev, ZIO_scheduler

When to change

See ZFS I/O Scheduler

Data Type

uint32

Units

I/O operations

Range

1 to zfs_vdev_ma x_active

Default

10

Change

Dynamic

Versions Affected

v0.6.4 and later

zfs_vdev_sync_write_min_active

zfs_vdev_sync_write_min_active sets the minimum synchronous write I/Os active to each device.

zfs_vdev_sync_write_min_active

Notes

Tags

vdev, ZIO_scheduler

When to change

See ZFS I/O Scheduler

Data Type

uint32

Units

I/O operations

Range

1 to zfs_vdev_sync_write_max_active

Default

10

Change

Dynamic

Versions Affected

v0.6.4 and later

zfs_vdev_queue_depth_pct

Maximum number of queued allocations per top-level vdev expressed as a percentage of zfs_vdev_async_write_max_active. This allows the system to detect devices that are more capable of handling allocations and to allocate more blocks to those devices. It also allows for dynamic allocation distribution when devices are imbalanced as fuller devices will tend to be slower than empty devices. Once the queue depth reaches (zfs_vdev_queue_depth_pct * zfs_vdev_async_write_max_active / 100) then allocator will stop allocating blocks on that top-level device and switch to the next.

See also zio_dva_throttle_enabled

zfs_vdev_queue_depth_pct

Notes

Tags

vdev, ZIO_scheduler

When to change

See ZFS I/O Scheduler

Data Type

uint32

Units

I/O operations

Range

1 to UINT32_MAX

Default

1,000

Change

Dynamic

Versions Affected

v0.7.0 and later

zfs_disable_dup_eviction

Disable duplicate buffer eviction from ARC.

zfs_disable_dup_eviction

Notes

Tags

ARC, dedup

When to change

TBD

Data Type

boolean

Range

0=duplicate buffers can be evicted, 1=do not evict duplicate buffers

Default

0

Change

Dynamic

Versions Affected

v0.6.5, deprecated in v0.7.0

zfs_expire_snapshot

Snapshots of filesystems are normally automounted under the filesystem’s .zfs/snapshot subdirectory. When not in use, snapshots are unmounted after zfs_expire_snapshot seconds.

zfs_expire_snapshot

Notes

Tags

filesystem, snapshot

When to change

TBD

Data Type

int

Units

seconds

Range

0 disables automatic unmounting, maximum time is INT_MAX

Default

300

Change

Dynamic

Versions Affected

v0.6.1 and later

zfs_admin_snapshot

Allow the creation, removal, or renaming of entries in the .zfs/snapshot subdirectory to cause the creation, destruction, or renaming of snapshots. When enabled this functionality works both locally and over NFS exports which have the “no_root_squash” option set.

zfs_admin_snapshot

Notes

Tags

filesystem, snapshot

When to change

TBD

Data Type

boolean

Range

0=do not allow snapshot manipulation via the filesystem, 1=allow snapshot manipulation via the filesystem

Default

1

Change

Dynamic

Versions Affected

v0.6.5 and later

zfs_flags

Set additional debugging flags (see zfs_dbgmsg_enable)

flag value

symbolic name

description

0x1

ZFS_DEBUG_DPRINTF

Enable dprintf entries in the debug log

0x2

ZFS_DEBUG_DBUF_VERIFY

Enable extra dnode verifications

0x4

ZFS_DEBUG_DNODE_VERIFY

Enable extra dnode verifications

0x8

ZFS_DEBUG_SNAPNAMES

Enable snapshot name verification

0x10

ZFS_DEBUG_MODIFY

Check for illegally modified ARC buffers

0x20

ZFS_DEBUG_SPA

Enable spa_dbgmsg entries in the debug log

0x40

ZFS_DEBUG_ZIO_FREE

Enable verification of block frees

0x80

Z FS_DEBUG_HISTOGRAM_VERIFY

Enable extra spacemap histogram verifications

0x100

ZFS_DEBUG_METASLAB_VERIFY

Verify space accounting on disk matches in-core range_trees

0x200

ZFS_DEBUG_SET_ERROR

Enable SET_ERROR and dprintf entries in the debug log

zfs_flags

Notes

Tags

debug

When to change

When debugging ZFS

Data Type

int

Default

0 no debug flags set, for debug builds: all except ZFS_DEBUG_DPRINTF and ZFS_DEBUG_SPA

Change

Dynamic

Versions Affected

v0.6.4 and later

zfs_free_leak_on_eio

If destroy encounters an I/O error (EIO) while reading metadata (eg indirect blocks), space referenced by the missing metadata cannot be freed. Normally, this causes the background destroy to become “stalled”, as the destroy is unable to make forward progress. While in this stalled state, all remaining space to free from the error-encountering filesystem is temporarily leaked. Set zfs_free_leak_on_eio = 1 to ignore the EIO, permanently leak the space from indirect blocks that can not be read, and continue to free everything else that it can.

The default, stalling behavior is useful if the storage partially fails (eg some but not all I/Os fail), and then later recovers. In this case, we will be able to continue pool operations while it is partially failed, and when it recovers, we can continue to free the space, with no leaks. However, note that this case is rare.

Typically pools either:

  1. fail completely (but perhaps temporarily (eg a top-level vdev going offline)

  2. have localized, permanent errors (eg disk returns the wrong data due to bit flip or firmware bug)

In case (1), the zfs_free_leak_on_eio setting does not matter because the pool will be suspended and the sync thread will not be able to make forward progress. In case (2), because the error is permanent, the best effort do is leak the minimum amount of space. Therefore, it is reasonable for zfs_free_leak_on_eio be set, but by default the more conservative approach is taken, so that there is no possibility of leaking space in the “partial temporary” failure case.

zfs_free_leak_on_eio

Notes

Tags

debug

When to change

When debugging I/O errors during destroy

Data Type

boolean

Range

0=normal behavior, 1=ignore error and permanently leak space

Default

0

Change

Dynamic

Versions Affected

v0.6.5 and later

zfs_free_min_time_ms

During a zfs destroy operation using feature@async_destroy a minimum of zfs_free_min_time_ms time will be spent working on freeing blocks per txg commit.

zfs_free_min_time_ms

Notes

Tags

delete

When to change

TBD

Data Type

int

Units

milliseconds

Range

1 to (zfs_txg_timeout * 1000)

Default

1,000

Change

Dynamic

Versions Affected

v0.6.0 and later

zfs_immediate_write_sz

If a pool does not have a log device, data blocks equal to or larger than zfs_immediate_write_sz are treated as if the dataset being written to had the property setting logbias=throughput

Terminology note: logbias=throughput writes the blocks in “indirect mode” to the ZIL where the data is written to the pool and a pointer to the data is written to the ZIL.

zfs_immediate_write_sz

Notes

Tags

ZIL

When to change

TBD

Data Type

long

Units

bytes

Range

512 to 16,777,216 (valid block sizes)

Default

32,768 (32 KiB)

Change

Dynamic

Verification

Data blocks that exceed zfs_immediate_write_sz or are written as logbias=throughput increment the zil_itx_indirect_count entry in /proc/spl/kstat/zfs/zil

Versions Affected

all

zfs_max_recordsize

ZFS supports logical record (block) sizes from 512 bytes to 16 MiB. The benefits of larger blocks, and thus larger average I/O sizes, can be weighed against the cost of copy-on-write of large block to modify one byte. Additionally, very large blocks can have a negative impact on both I/O latency at the device level and the memory allocator. The zfs_max_recordsize parameter limits the upper bound of the dataset volblocksize and recordsize properties.

Larger blocks can be created by enabling zpool large_blocks feature and changing this zfs_max_recordsize. Pools with larger blocks can always be imported and used, regardless of the value of zfs_max_recordsize.

For 32-bit systems, zfs_max_recordsize also limits the size of kernel virtual memory caches used in the ZFS I/O pipeline (zio_buf_* and zio_data_buf_*).

See also the zpool large_blocks feature.

zfs_max_recordsize

Notes

Tags

filesystem, memory, volume

When to change

To create datasets with larger volblocksize or recordsize

Data Type

int

Units

bytes

Range

512 to 16,777,216 (valid block sizes)

Default

1,048,576

Change

Dynamic, set prior to creating volumes or changing filesystem recordsize

Versions Affected

v0.6.5 and later

zfs_mdcomp_disable

zfs_mdcomp_disable allows metadata compression to be disabled.

zfs_mdcomp_disable

Notes

Tags

CPU, metadata

When to change

When CPU cycles cost less than I/O

Data Type

boolean

Range

0=compress metadata, 1=do not compress metadata

Default

0

Change

Dynamic

Versions Affected

from v0.6.0 to v0.8.0

zfs_metaslab_fragmentation_threshold

Allow metaslabs to keep their active state as long as their fragmentation percentage is less than or equal to this value. When writing, an active metaslab whose fragmentation percentage exceeds zfs_metaslab_fragmentation_threshold is avoided allowing metaslabs with less fragmentation to be preferred.

Metaslab fragmentation is used to calculate the overall pool fragmentation property value. However, individual metaslab fragmentation levels are observable using the zdb with the -mm option.

zfs_metaslab_fragmentation_threshold works at the metaslab level and each top-level vdev has approximately metaslabs_per_vdev metaslabs. See also zfs_mg_fragmentation_threshold

zfs_metaslab_fragmentation_thresh old

Notes

Tags

allocation, fr agmentation, vdev

When to change

Testing metaslab allocation

Data Type

int

Units

percent

Range

1 to 100

Default

70

Change

Dynamic

Versions Affected

v0.6.4 and later

zfs_mg_fragmentation_threshold

Metaslab groups (top-level vdevs) are considered eligible for allocations if their fragmentation percentage metric is less than or equal to zfs_mg_fragmentation_threshold. If a metaslab group exceeds this threshold then it will be skipped unless all metaslab groups within the metaslab class have also crossed the zfs_mg_fragmentation_threshold threshold.

zfs_mg_fragmentation_threshold

Notes

Tags

allocation, ` fragmentation <#fragmentation>`__, vdev

When to change

Testing metaslab allocation

Data Type

int

Units

percent

Range

1 to 100

Default

85

Change

Dynamic

Versions Affected

v0.6.4 and later

zfs_mg_noalloc_threshold

Metaslab groups (top-level vdevs) with free space percentage greater than zfs_mg_noalloc_threshold are eligible for new allocations. If a metaslab group’s free space is less than or equal to the threshold, the allocator avoids allocating to that group unless all groups in the pool have reached the threshold. Once all metaslab groups have reached the threshold, all metaslab groups are allowed to accept allocations. The default value of 0 disables the feature and causes all metaslab groups to be eligible for allocations.

This parameter allows one to deal with pools having heavily imbalanced vdevs such as would be the case when a new vdev has been added. Setting the threshold to a non-zero percentage will stop allocations from being made to vdevs that aren’t filled to the specified percentage and allow lesser filled vdevs to acquire more allocations than they otherwise would under the older zfs_mg_alloc_failures facility.

zfs_mg_noalloc_threshold

Notes

Tags

allocation, fragmentation, vdev

When to change

To force rebalancing as top-level vdevs are added or expanded

Data Type

int

Units

percent

Range

0 to 100

Default

0 (disabled)

Change

Dynamic

Versions Affected

v0.7.0 and later

zfs_multihost_history

The pool multihost multimodifier protection (MMP) subsystem can record historical updates in the /proc/spl/kstat/zfs/POOL_NAME/multihost file for debugging purposes. The number of lines of history is determined by zfs_multihost_history.

zfs_multihost_history

Notes

Tags

MMP, import

When to change

When testing multihost feature

Data Type

int

Units

lines

Range

0 to INT_MAX

Default

0

Change

Dynamic

Versions Affected

v0.7.0 and later

zfs_multihost_interval

zfs_multihost_interval controls the frequency of multihost writes performed by the pool multihost multimodifier protection (MMP) subsystem. The multihost write period is (zfs_multihost_interval / number of leaf-vdevs) milliseconds. Thus on average a multihost write will be issued for each leaf vdev every zfs_multihost_interval milliseconds. In practice, the observed period can vary with the I/O load and this observed value is the delay which is stored in the uberblock.

On import the multihost activity check waits a minimum amount of time determined by (zfs_multihost_interval * zfs_multihost_import_intervals) with a lower bound of 1 second. The activity check time may be further extended if the value of mmp delay found in the best uberblock indicates actual multihost updates happened at longer intervals than zfs_multihost_interval

Note: the multihost protection feature applies to storage devices that can be shared between multiple systems.

zfs_multihost_interval

Notes

Tags

MMP, import, vdev

When to change

To optimize pool import time against possibility of simultaneous import by another system

Data Type

ulong

Units

milliseconds

Range

100 to ULONG_MAX

Default

1000

Change

Dynamic

Versions Affected

v0.7.0 and later

zfs_multihost_import_intervals

zfs_multihost_import_intervals controls the duration of the activity test on pool import for the multihost multimodifier protection (MMP) subsystem. The activity test can be expected to take a minimum time of (zfs_multihost_import_intervals * zfs_multihost_interval * random(25%)) milliseconds. The random period of up to 25% improves simultaneous import detection. For example, if two hosts are rebooted at the same time and automatically attempt to import the pool, then is is highly probable that one host will win.

Smaller values of zfs_multihost_import_intervals reduces the import time but increases the risk of failing to detect an active pool. The total activity check time is never allowed to drop below one second.

Note: the multihost protection feature applies to storage devices that can be shared between multiple systems.

zfs_multihost_import_intervals

Notes

Tags

MMP, import

When to change

TBD

Data Type

uint

Units

intervals

Range

1 to UINT_MAX

Default

10

Change

Dynamic

Versions Affected

v0.7.0 and later

zfs_multihost_fail_intervals

zfs_multihost_fail_intervals controls the behavior of the pool when write failures are detected in the multihost multimodifier protection (MMP) subsystem.

If zfs_multihost_fail_intervals = 0 then multihost write failures are ignored. The write failures are reported to the ZFS event daemon (zed) which can take action such as suspending the pool or offlining a device.

If zfs_multihost_fail_intervals > 0 then sequential multihost write failures will cause the pool to be suspended. This occurs when (zfs_multihost_fail_intervals * zfs_multihost_interval) milliseconds have passed since the last successful multihost write.
This guarantees the activity test will see multihost writes if the pool is attempted to be imported by another system.

zfs_multihost_fail_intervals

Notes

Tags

MMP, import

When to change

TBD

Data Type

uint

Units

intervals

Range

0 to UINT_MAX

Default

5

Change

Dynamic

Versions Affected

v0.7.0 and later

zfs_delays_per_second

The ZFS Event Daemon (zed) processes events from ZFS. However, it can be overwhelmed by high rates of error reports which can be generated by failing, high-performance devices. zfs_delays_per_second limits the rate of delay events reported to zed.

zfs_delays_per_second

Notes

Tags

zed, delay

When to change

If processing delay events at a higher rate is desired

Data Type

uint

Units

events per second

Range

0 to UINT_MAX

Default

20

Change

Dynamic

Versions Affected

v0.7.7 and later

zfs_checksums_per_second

The ZFS Event Daemon (zed) processes events from ZFS. However, it can be overwhelmed by high rates of error reports which can be generated by failing, high-performance devices. zfs_checksums_per_second limits the rate of checksum events reported to zed.

Note: do not set this value lower than the SERD limit for checksum in zed. By default, checksum_N = 10 and checksum_T = 10 minutes, resulting in a practical lower limit of 1.

zfs_checksums_per_second

Notes

Tags

zed, checksum

When to change

If processing checksum error events at a higher rate is desired

Data Type

uint

Units

events per second

Range

0 to UINT_MAX

Default

20

Change

Dynamic

Versions Affected

v0.7.7 and later

zfs_no_scrub_io

When zfs_no_scrub_io = 1 scrubs do not actually scrub data and simply doing a metadata crawl of the pool instead.

zfs_no_scrub_io

Notes

Tags

scrub

When to change

Testing scrub feature

Data Type

boolean

Range

0=perform scrub I/O, 1=do not perform scrub I/O

Default

0

Change

Dynamic

Versions Affected

v0.6.0 and later

zfs_no_scrub_prefetch

When zfs_no_scrub_prefetch = 1, prefetch is disabled for scrub I/Os.

zfs_no_scrub_prefetch

Notes

Tags

prefetch, scrub

When to change

Testing scrub feature

Data Type

boolean

Range

0=prefetch scrub I/Os, 1=do not prefetch scrub I/Os

Default

0

Change

Dynamic

Versions Affected

v0.6.4 and later

zfs_nocacheflush

ZFS uses barriers (volatile cache flush commands) to ensure data is committed to permanent media by devices. This ensures consistent on-media state for devices where caches are volatile (eg HDDs).

For devices with nonvolatile caches, the cache flush operation can be a no-op. However, in some RAID arrays, cache flushes can cause the entire cache to be flushed to the backing devices.

To ensure on-media consistency, keep cache flush enabled.

zfs_nocacheflush

Notes

Tags

disks

When to change

If the storage device has nonvolatile cache, then disabling cache flush can save the cost of occasional cache flush commands

Data Type

boolean

Range

0=send cache flush commands, 1=do not send cache flush commands

Default

0

Change

Dynamic

Versions Affected

all

zfs_nopwrite_enabled

The NOP-write feature is enabled by default when a crytographically-secure checksum algorithm is in use by the dataset. zfs_nopwrite_enabled allows the NOP-write feature to be completely disabled.

zfs_nopwrite_enabled

Notes

Tags

checksum, debug

When to change

TBD

Data Type

boolean

Range

0=disable NOP-write feature, 1=enable NOP-write feature

Default

1

Change

Dynamic

Versions Affected

v0.6.0 and later

zfs_dmu_offset_next_sync

zfs_dmu_offset_next_sync enables forcing txg sync to find holes. This causes ZFS to act like older versions when SEEK_HOLE or SEEK_DATA flags are used: when a dirty dnode causes txgs to be synced so the previous data can be found.

zfs_dmu_offset_next_sync

Notes

Tags

DMU

When to change

TBD

Data Type

boolean

Range

0=do not force txg sync to find holes, 1=force txg sync to find holes

Default

0

Change

Dynamic

Versions Affected

v0.7.0 and later

zfs_pd_bytes_max

zfs_pd_bytes_max limits the number of bytes prefetched during a pool traversal (eg zfs send or other data crawling operations). These prefetches are referred to as “prescient prefetches” and are always 100% hit rate. The traversal operations do not use the default data or metadata prefetcher.

zfs_pd_bytes_max

Notes

Tags

prefetch, send

When to change

TBD

Data Type

int32

Units

bytes

Range

0 to INT32_MAX

Default

52,428,800 (50 MiB)

Change

Dynamic

Versions Affected

TBD

zfs_per_txg_dirty_frees_percent

zfs_per_txg_dirty_frees_percent as a percentage of zfs_dirty_data_max controls the percentage of dirtied blocks from frees in one txg. After the threshold is crossed, additional dirty blocks from frees wait until the next txg. Thus, when deleting large files, filling consecutive txgs with deletes/frees, does not throttle other, perhaps more important, writes.

A side effect of this throttle can impact zfs receive workloads that contain a large number of frees and the ignore_hole_birth optimization is disabled. The symptom is that the receive workload causes an increase in the frequency of txg commits. The frequency of txg commits is observable via the otime column of /proc/spl/kstat/zfs/POOLNAME/txgs. Since txg commits also flush data from volatile caches in HDDs to media, HDD performance can be negatively impacted. Also, since the frees do not consume much bandwidth over the pipe, the pipe can appear to stall. Thus the overall progress of receives is slower than expected.

A value of zero will disable this throttle.

zfs_per_txg_dirty_frees_percent

Notes

Tags

delete

When to change

For zfs receive workloads, consider increasing or disabling. See section ZFS I/O S cheduler

Data Type

ulong

Units

percent

Range

0 to 100

Default

30

Change

Dynamic

Versions Affected

v0.7.0 and later

zfs_prefetch_disable

zfs_prefetch_disable controls the predictive prefetcher.

Note that it leaves “prescient” prefetch (eg prefetch for zfs send) intact (see zfs_pd_bytes_max)

zfs_prefetch_disable

Notes

Tags

prefetch

When to change

In some case where the workload is completely random reads, overall performance can be better if prefetch is disabled

Data Type

boolean

Range

0=prefetch enabled, 1=prefetch disabled

Default

0

Change

Dynamic

Verification

prefetch efficacy is observed by arcstat, arc_summary, and the relevant entries in /proc/spl/kstat/zfs/arcstats

Versions Affected

all

zfs_read_chunk_size

zfs_read_chunk_size is the limit for ZFS filesystem reads. If an application issues a read() larger than zfs_read_chunk_size, then the read() is divided into multiple operations no larger than zfs_read_chunk_size

zfs_read_chunk_size

Notes

Tags

filesystem

When to change

TBD

Data Type

ulong

Units

bytes

Range

512 to ULONG_MAX

Default

1,048,576

Change

Dynamic

Versions Affected

all

zfs_read_history

Historical statistics for the last zfs_read_history reads are available in /proc/spl/kstat/zfs/POOL_NAME/reads

zfs_read_history

Notes

Tags

debug

When to change

To observe read operation details

Data Type

int

Units

lines

Range

0 to INT_MAX

Default

0

Change

Dynamic

Versions Affected

all

zfs_read_history_hits

When zfs_read_history> 0, zfs_read_history_hits controls whether ARC hits are displayed in the read history file, /proc/spl/kstat/zfs/POOL_NAME/reads

zfs_read_history_hits

Notes

Tags

debug

When to change

To observe read operation details with ARC hits

Data Type

boolean

Range

0=do not include data for ARC hits, 1=include ARC hit data

Default

0

Change

Dynamic

Versions Affected

all

zfs_recover

zfs_recover can be set to true (1) to attempt to recover from otherwise-fatal errors, typically caused by on-disk corruption. When set, calls to zfs_panic_recover() will turn into warning messages rather than calling panic()

zfs_recover

Notes

Tags

import

When to change

zfs_recover should only be used as a last resort, as it typically results in leaked space, or worse

Data Type

boolean

Range

0=normal operation, 1=attempt recovery zpool import

Default

0

Change

Dynamic

Verification

check output of dmesg and other logs for details

Versions Affected

v0.6.4 or later

zfs_resilver_min_time_ms

Resilvers are processed by the sync thread in syncing context. While resilvering, ZFS spends at least zfs_resilver_min_time_ms time working on a resilver between txg commits.

The zfs_txg_timeout tunable sets a nominal timeout value for the txg commits. By default, this timeout is 5 seconds and the zfs_resilver_min_time_ms is 3 seconds. However, many variables contribute to changing the actual txg times. The measured txg interval is observed as the otime column (in nanoseconds) in the /proc/spl/kstat/zfs/POOL_NAME/txgs file.

See also zfs_txg_timeout and zfs_scan_min_time_ms

zfs_resilver_min_time_ms

Notes

Tags

resilver

When to change

In some resilvering cases, increasing zfs_resilver_min_time_ms can result in faster completion

Data Type

int

Units

milliseconds

Range

1 to zfs_txg_timeout converted to milliseconds

Default

3,000

Change

Dynamic

Versions Affected

all

zfs_scan_min_time_ms

Scrubs are processed by the sync thread in syncing context. While scrubbing, ZFS spends at least zfs_scan_min_time_ms time working on a scrub between txg commits.

See also zfs_txg_timeout and zfs_resilver_min_time_ms

zfs_scan_min_time_ms

Notes

Tags

scrub

When to change

In some scrub cases, increasing zfs_scan_min_time_ms can result in faster completion

Data Type

int

Units

milliseconds

Range

1 to zfs_txg_timeout converted to milliseconds

Default

1,000

Change

Dynamic

Versions Affected

all

zfs_scan_checkpoint_intval

To preserve progress across reboots the sequential scan algorithm periodically needs to stop metadata scanning and issue all the verifications I/Os to disk every zfs_scan_checkpoint_intval seconds.

zfs_scan_checkpoint_intval

Notes

Tags

resilver, scrub

When to change

TBD

Data Type

int

Units

seconds

Range

1 to INT_MAX

Default

7,200 (2 hours)

Change

Dynamic

Versions Affected

v0.8.0 and later

zfs_scan_fill_weight

This tunable affects how scrub and resilver I/O segments are ordered. A higher number indicates that we care more about how filled in a segment is, while a lower number indicates we care more about the size of the extent without considering the gaps within a segment.

zfs_scan_fill_weight

Notes

Tags

resilver, scrub

When to change

Testing sequential scrub and resilver

Data Type

int

Units

scalar

Range

0 to INT_MAX

Default

3

Change

Prior to zfs module load

Versions Affected

v0.8.0 and later

zfs_scan_issue_strategy

zfs_scan_issue_strategy controls the order of data verification while scrubbing or resilvering.

value

description

0

fs will use strategy 1 during normal verification and strategy 2 while taking a checkpoint

1

data is verified as sequentially as possible, given the amount of memory reserved for scrubbing (see zfs_scan_mem_lim_fact). This can improve scrub performance if the pool’s data is heavily fragmented.

2

the largest mostly-contiguous chunk of found data is verified first. By deferring scrubbing of small segments, we may later find adjacent data to coalesce and increase the segment size.

zfs_scan_issue_strategy

Notes

Tags

resilver, scrub

When to change

TBD

Data Type

enum

Range

0 to 2

Default

0

Change

Dynamic

Versions Affected

TBD

zfs_scan_legacy

Setting zfs_scan_legacy = 1 enables the legacy scan and scrub behavior instead of the newer sequential behavior.

zfs_scan_legacy

Notes

Tags

resilver, scrub

When to change

In some cases, the new scan mode can consumer more memory as it collects and sorts I/Os; using the legacy algorithm can be more memory efficient at the expense of HDD read efficiency

Data Type

boolean

Range

0=use new method: scrubs and resilvers will gather metadata in memory before issuing sequential I/O, 1=use legacy algorithm will be used where I/O is initiated as soon as it is discovered

Default

0

Change

Dynamic, however changing to 0 does not affect in-progress scrubs or resilvers

Versions Affected

v0.8.0 and later

zfs_scan_max_ext_gap

zfs_scan_max_ext_gap limits the largest gap in bytes between scrub and resilver I/Os that will still be considered sequential for sorting purposes.

zfs_scan_max_ext_gap

Notes

Tags

resilver, scrub

When to change

TBD

Data Type

ulong

Units

bytes

Range

512 to ULONG_MAX

Default

2,097,152 (2 MiB)

Change

Dynamic, however changing to 0 does not affect in-progress scrubs or resilvers

Versions Affected

v0.8.0 and later

zfs_scan_mem_lim_fact

zfs_scan_mem_lim_fact limits the maximum fraction of RAM used for I/O sorting by sequential scan algorithm. When the limit is reached scanning metadata is stopped and data verification I/O is started. Data verification I/O continues until the memory used by the sorting algorithm drops below below zfs_scan_mem_lim_soft_fact

Memory used by the sequential scan algorithm can be observed as the kmem sio_cache. This is visible from procfs as grep sio_cache /proc/slabinfo and can be monitored using slab-monitoring tools such as slabtop

zfs_scan_mem_lim_fact

Notes

Tags

memory, resilver, scrub

When to change

TBD

Data Type

int

Units

divisor of physical RAM

Range

TBD

Default

20 (physical RAM / 20 or 5%)

Change

Dynamic

Versions Affected

v0.8.0 and later

zfs_scan_mem_lim_soft_fact

zfs_scan_mem_lim_soft_fact sets the fraction of the hard limit, zfs_scan_mem_lim_fact, used to determined the RAM soft limit for I/O sorting by the sequential scan algorithm. After zfs_scan_mem_lim_fact has been reached, metadata scanning is stopped until the RAM usage drops below zfs_scan_mem_lim_soft_fact

zfs_scan_mem_lim_soft_fact

Notes

Tags

resilver, scrub

When to change

TBD

Data Type

int

Units

divisor of (physical RAM / zfs_scan_mem _lim_fact)

Range

1 to INT_MAX

Default

20 (for default zfs_scan_mem _lim_fact, 0.25% of physical RAM)

Change

Dynamic

Versions Affected

v0.8.0 and later

zfs_scan_vdev_limit

zfs_scan_vdev_limit is the maximum amount of data that can be concurrently issued at once for scrubs and resilvers per leaf vdev. zfs_scan_vdev_limit attempts to strike a balance between keeping the leaf vdev queues full of I/Os while not overflowing the queues causing high latency resulting in long txg sync times. While zfs_scan_vdev_limit represents a bandwidth limit, the existing I/O limit of zfs_vdev_scrub_max_active remains in effect, too.

zfs_scan_vdev_limit

Notes

Tags

resilver, scrub, vdev

When to change

TBD

Data Type

ulong

Units

bytes

Range

512 to ULONG_MAX

Default

4,194,304 (4 MiB)

Change

Dynamic

Versions Affected

v0.8.0 and later

zfs_send_corrupt_data

zfs_send_corrupt_data enables zfs send to send of corrupt data by ignoring read and checksum errors. The corrupted or unreadable blocks are replaced with the value 0x2f5baddb10c (ZFS bad block)

zfs_send_corrupt_data

Notes

Tags

send

When to change

When data corruption exists and an attempt to recover at least some data via zfs send is needed

Data Type

boolean

Range

0=do not send corrupt data, 1=replace corrupt data with cookie

Default

0

Change

Dynamic

Versions Affected

v0.6.0 and later

zfs_sync_pass_deferred_free

The SPA sync process is performed in multiple passes. Once the pass number reaches zfs_sync_pass_deferred_free, frees are no long processed and must wait for the next SPA sync.

The zfs_sync_pass_deferred_free value is expected to be removed as a tunable once the optimal value is determined during field testing.

The zfs_sync_pass_deferred_free pass must be greater than 1 to ensure that regular blocks are not deferred.

zfs_sync_pass_deferred_free

Notes

Tags

SPA

When to change

Testing SPA sync process

Data Type

int

Units

SPA sync passes

Range

1 to INT_MAX

Default

2

Change

Dynamic

Versions Affected

all

zfs_sync_pass_dont_compress

The SPA sync process is performed in multiple passes. Once the pass number reaches zfs_sync_pass_dont_compress, data block compression is no longer processed and must wait for the next SPA sync.

The zfs_sync_pass_dont_compress value is expected to be removed as a tunable once the optimal value is determined during field testing.

zfs_sync_pass_dont_compress

Notes

Tags

SPA

When to change

Testing SPA sync process

Data Type

int

Units

SPA sync passes

Range

1 to INT_MAX

Default

5

Change

Dynamic

Versions Affected

all

zfs_sync_pass_rewrite

The SPA sync process is performed in multiple passes. Once the pass number reaches zfs_sync_pass_rewrite, blocks can be split into gang blocks.

The zfs_sync_pass_rewrite value is expected to be removed as a tunable once the optimal value is determined during field testing.

zfs_sync_pass_rewrite

Notes

Tags

SPA

When to change

Testing SPA sync process

Data Type

int

Units

SPA sync passes

Range

1 to INT_MAX

Default

2

Change

Dynamic

Versions Affected

all

zfs_sync_taskq_batch_pct

zfs_sync_taskq_batch_pct controls the number of threads used by the DSL pool sync taskq, dp_sync_taskq

zfs_sync_taskq_batch_pct

Notes

Tags

SPA

When to change

to adjust the number of dp_sync_taskq threads

Data Type

int

Units

percent of number of online CPUs

Range

1 to 100

Default

75

Change

Prior to zfs module load

Versions Affected

v0.7.0 and later

zfs_txg_history

Historical statistics for the last zfs_txg_history txg commits are available in /proc/spl/kstat/zfs/POOL_NAME/txgs

The work required to measure the txg commit (SPA statistics) is low. However, for debugging purposes, it can be useful to observe the SPA statistics.

zfs_txg_history

Notes

Tags

debug

When to change

To observe details of SPA sync behavior.

Data Type

int

Units

lines

Range

0 to INT_MAX

Default

0 for version v0.6.0 to v0.7.6, 100 for version v0.8.0

Change

Dynamic

Versions Affected

all

zfs_txg_timeout

The open txg is committed to the pool periodically (SPA sync) and zfs_txg_timeout represents the default target upper limit.

txg commits can occur more frequently and a rapid rate of txg commits often indicates a busy write workload, quota limits reached, or the free space is critically low.

Many variables contribute to changing the actual txg times. txg commits can also take longer than zfs_txg_timeout if the ZFS write throttle is not properly tuned or the time to sync is otherwise delayed (eg slow device). Shorter txg commit intervals can occur due to zfs_dirty_data_sync for write-intensive workloads. The measured txg interval is observed as the otime column (in nanoseconds) in the /proc/spl/kstat/zfs/POOL_NAME/txgs file.

See also zfs_dirty_data_sync and zfs_txg_history

zfs_txg_timeout

Notes

Tags

SPA, ZIO_scheduler

When to change

To optimize the work done by txg commit relative to the pool requirements. See also section ZFS I/O Scheduler

Data Type

int

Units

seconds

Range

1 to INT_MAX

Default

5

Change

Dynamic

Versions Affected

all

zfs_vdev_aggregation_limit

To reduce IOPs, small, adjacent I/Os can be aggregated (coalesced) into a large I/O. For reads, aggregations occur across small adjacency gaps. For writes, aggregation can occur at the ZFS or disk level. zfs_vdev_aggregation_limit is the upper bound on the size of the larger, aggregated I/O.

Setting zfs_vdev_aggregation_limit = 0 effectively disables aggregation by ZFS. However, the block device scheduler can still merge (aggregate) I/Os. Also, many devices, such as modern HDDs, contain schedulers that can aggregate I/Os.

In general, I/O aggregation can improve performance for devices, such as HDDs, where ordering I/O operations for contiguous LBAs is a benefit. For random access devices, such as SSDs, aggregation might not improve performance relative to the CPU cycles needed to aggregate. For devices that represent themselves as having no rotation, the zfs_vdev_aggregation_limit_non_rotating parameter is used instead of zfs_vdev_aggregation_limit

zfs_vdev_aggregation_limit

Notes

Tags

vdev, ZIO_scheduler

When to change

If the workload does not benefit from aggregation, the zfs_vdev_aggregation_limit can be reduced to avoid aggregation attempts

Data Type

int

Units

bytes

Range

0 to 1,048,576 (default) or 16,777,216 (if zpool large_blocks feature is enabled)

Default

1,048,576, or 131,072 for <v0.8

Change

Dynamic

Verification

ZFS aggregation is observed with zpool iostat -r and the block scheduler merging is observed with iostat -x

Versions Affected

all

zfs_vdev_cache_size

Note: with the current ZFS code, the vdev cache is not helpful and in some cases actually harmful. Thusit is disabled by setting the zfs_vdev_cache_size = 0

zfs_vdev_cache_size is the size of the vdev cache.

zfs_vdev_cache_size

Notes

Tags

vdev, vdev_cache

When to change

Do not change

Data Type

int

Units

bytes

Range

0 to MAX_INT

Default

0 (vdev cache is disabled)

Change

Dynamic

Verification

vdev cache statistics are available in the /proc/spl/kstat/zfs/vdev_cache_stats file

Versions Affected

all

zfs_vdev_cache_bshift

Note: with the current ZFS code, the vdev cache is not helpful and in some cases actually harmful. Thus it is disabled by setting the zfs_vdev_cache_size to zero. This related tunable is, by default, inoperative.

All read I/Os smaller than zfs_vdev_cache_max are turned into (1 << zfs_vdev_cache_bshift) byte reads by the vdev cache. At most zfs_vdev_cache_size bytes will be kept in each vdev’s cache.

zfs_vdev_cache_bshift

Notes

Tags

vdev, vdev_cache

When to change

Do not change

Data Type

int

Units

shift

Range

1 to INT_MAX

Default

16 (65,536 bytes)

Change

Dynamic

Versions Affected

all

zfs_vdev_cache_max

Note: with the current ZFS code, the vdev cache is not helpful and in some cases actually harmful. Thus it is disabled by setting the zfs_vdev_cache_size to zero. This related tunable is, by default, inoperative.

All read I/Os smaller than zfs_vdev_cache_max will be turned into (1 <<zfs_vdev_cache_bshift byte reads by the vdev cache. At most zfs_vdev_cache_size bytes will be kept in each vdev’s cache.

zfs_vdev_cache_max

Notes

Tags

vdev, vdev_cache

When to change

Do not change

Data Type

int

Units

bytes

Range

512 to INT_MAX

Default

16,384 (16 KiB)

Change

Dynamic

Versions Affected

all

zfs_vdev_mirror_rotating_inc

The mirror read algorithm uses current load and an incremental weighting value to determine the vdev to service a read operation. Lower values determine the preferred vdev. The weighting value is zfs_vdev_mirror_rotating_inc for rotating media and zfs_vdev_mirror_non_rotating_inc for nonrotating media.

Verify the rotational setting described by a block device in sysfs by observing /sys/block/DISK_NAME/queue/rotational

zfs_vdev_mirror_rotating_inc

Notes

Tags

vdev, mirror, HDD

When to change

Increasing for mirrors with both rotating and nonrotating media more strongly favors the nonrotating media

Data Type

int

Units

scalar

Range

0 to MAX_INT

Default

0

Change

Dynamic

Versions Affected

v0.7.0 and later

zfs_vdev_mirror_non_rotating_inc

The mirror read algorithm uses current load and an incremental weighting value to determine the vdev to service a read operation. Lower values determine the preferred vdev. The weighting value is zfs_vdev_mirror_rotating_inc for rotating media and zfs_vdev_mirror_non_rotating_inc for nonrotating media.

Verify the rotational setting described by a block device in sysfs by observing /sys/block/DISK_NAME/queue/rotational

zfs_vdev_mirror_non_rotating_inc

Notes

Tags

vdev, mirror, SSD

When to change

TBD

Data Type

int

Units

scalar

Range

0 to INT_MAX

Default

0

Change

Dynamic

Versions Affected

v0.7.0 and later

zfs_vdev_mirror_rotating_seek_inc

For rotating media in a mirror, if the next I/O offset is within zfs_vdev_mirror_rotating_seek_offset then the weighting factor is incremented by (zfs_vdev_mirror_rotating_seek_inc / 2). Otherwise the weighting factor is increased by zfs_vdev_mirror_rotating_seek_inc. This algorithm prefers rotating media with lower seek distance.

Verify the rotational setting described by a block device in sysfs by observing /sys/block/DISK_NAME/queue/rotational

z fs_vdev_mirror_rotating_seek_inc

Notes

Tags

vdev, mirror, HDD

When to change

TBD

Data Type

int

Units

scalar

Range

0 to INT_MAX

Default

5

Change

Dynamic

Versions Affected

v0.7.0 and later

zfs_vdev_mirror_rotating_seek_offset

For rotating media in a mirror, if the next I/O offset is within zfs_vdev_mirror_rotating_seek_offset then the weighting factor is incremented by (zfs_vdev_mirror_rotating_seek_inc/ 2). Otherwise the weighting factor is increased by zfs_vdev_mirror_rotating_seek_inc. This algorithm prefers rotating media with lower seek distance.

Verify the rotational setting described by a block device in sysfs by observing /sys/block/DISK_NAME/queue/rotational

zfs_vdev_mirror_rotating_seek_off set

Notes

Tags

vdev, mirror, HDD

When to change

TBD

Data Type

int

Units

bytes

Range

0 to INT_MAX

Default

1,048,576 (1 MiB)

Change

Dynamic

Versions Affected

v0.7.0 and later

zfs_vdev_mirror_non_rotating_seek_inc

For nonrotating media in a mirror, a seek penalty is applied as sequential I/O’s can be aggregated into fewer operations, avoiding unnecessary per-command overhead, often boosting performance.

Verify the rotational setting described by a block device in SysFS by observing /sys/block/DISK_NAME/queue/rotational

zfs_v dev_mirror_non_rotating_seek_inc

Notes

Tags

vdev, mirror, SSD

When to change

TBD

Data Type

int

Units

scalar

Range

0 to INT_MAX

Default

1

Change

Dynamic

Versions Affected

v0.7.0 and later

zfs_vdev_read_gap_limit

To reduce IOPs, small, adjacent I/Os are aggregated (coalesced) into into a large I/O. For reads, aggregations occur across small adjacency gaps where the gap is less than zfs_vdev_read_gap_limit

zfs_vdev_read_gap_limit

Notes

Tags

vdev, ZIO_scheduler

When to change

TBD

Data Type

int

Units

bytes

Range

0 to INT_MAX

Default

32,768 (32 KiB)

Change

Dynamic

Versions Affected

all

zfs_vdev_write_gap_limit

To reduce IOPs, small, adjacent I/Os are aggregated (coalesced) into into a large I/O. For writes, aggregations occur across small adjacency gaps where the gap is less than zfs_vdev_write_gap_limit

zfs_vdev_write_gap_limit

Notes

Tags

vdev, ZIO_scheduler

When to change

TBD

Data Type

int

Units

bytes

Range

0 to INT_MAX

Default

4,096 (4 KiB)

Change

Dynamic

Versions Affected

all

zfs_vdev_scheduler

When the pool is imported, for whole disk vdevs, the block device I/O scheduler is set to zfs_vdev_scheduler. The most common schedulers are: noop, cfq, bfq, and deadline.

In some cases, the scheduler is not changeable using this method. Known schedulers that cannot be changed are: scsi_mq and none. In these cases, the scheduler is unchanged and an error message can be reported to logs.

zfs_vdev_scheduler

Notes

Tags

vdev, ZIO_scheduler

When to change

since ZFS has its own I/O scheduler, using a simple scheduler can result in more consistent performance

Data Type

string

Range

expected: noop, cfq, bfq, and deadline

Default

noop

Change

Dynamic, but takes effect upon pool creation or import

Versions Affected

all

zfs_vdev_raidz_impl

zfs_vdev_raidz_impl overrides the raidz parity algorithm. By default, the algorithm is selected at zfs module load time by the results of a microbenchmark of algorithms based on the current hardware.

Once the module is loaded, the content of /sys/module/zfs/parameters/zfs_vdev_raidz_impl shows available options with the currently selected enclosed in []. Details of the results of the microbenchmark are observable in the /proc/spl/kstat/zfs/vdev_raidz_bench file.

algorithm

architecture

description

fastest

all

fastest implementation selected by microbenchmark

original

all

original raidz implementation

scalar

all

scalar raidz implementation

sse2

64-bit x86

uses SSE2 instruction set

ssse3

64-bit x86

uses SSSE3 instruction set

avx2

64-bit x86

uses AVX2 instruction set

avx512f

64-bit x86

uses AVX512F instruction set

avx512bw

64-bit x86

uses AVX512F & AVX512BW instruction sets

aarch64_neon

aarch64/64 bit ARMv8

uses NEON

aarch64_neonx2

aarch64/64 bit ARMv8

uses NEON with more unrolling

zfs_vdev_raidz_impl

Notes

Tags

CPU, raidz, vdev

When to change

testing raidz algorithms

Data Type

string

Range

see table above

Default

fastest

Change

Dynamic

Versions Affected

v0.7.0 and later

zfs_zevent_cols

zfs_zevent_cols is a soft wrap limit in columns (characters) for ZFS events logged to the console.

zfs_zevent_cols

Notes

Tags

debug

When to change

if 80 columns isn’t enough

Data Type

int

Units

characters

Range

1 to INT_MAX

Default

80

Change

Dynamic

Versions Affected

all

zfs_zevent_console

If zfs_zevent_console is true (1), then ZFS events are logged to the console.

More logging and log filtering capabilities are provided by zed

zfs_zevent_console

Notes

Tags

debug

When to change

to log ZFS events to the console

Data Type

boolean

Range

0=do not log to console, 1=log to console

Default

0

Change

Dynamic

Versions Affected

all

zfs_zevent_len_max

zfs_zevent_len_max is the maximum ZFS event queue length. A value of 0 results in a calculated value (16 * number of CPUs) with a minimum of 64. Events in the queue can be viewed with the zpool events command.

zfs_zevent_len_max

Notes

Tags

debug

When to change

increase to see more ZFS events

Data Type

int

Units

events

Range

0 to INT_MAX

Default

0 (calculate as described above)

Change

Dynamic

Versions Affected

all

zfs_zil_clean_taskq_maxalloc

During a SPA sync, intent log transaction groups (itxg) are cleaned. The cleaning work is dispatched to the DSL pool ZIL clean taskq (dp_zil_clean_taskq). zfs_zil_clean_taskq_minalloc is the minimum and zfs_zil_clean_taskq_maxalloc is the maximum number of cached taskq entries for dp_zil_clean_taskq. The actual number of taskq entries dynamically varies between these values.

When zfs_zil_clean_taskq_maxalloc is exceeded transaction records (itxs) are cleaned synchronously with possible negative impact to the performance of SPA sync.

Ideally taskq entries are pre-allocated prior to being needed by zil_clean(), thus avoiding dynamic allocation of new taskq entries.

zfs_zil_clean_taskq_maxalloc

Notes

Tags

ZIL

When to change

If more dp_zil_clean_taskq entries are needed to prevent the itxs from being synchronously cleaned

Data Type

int

Units

dp_zil_clean_taskq taskq entries

Range

zfs_zil_clean_taskq_minallo c to INT_MAX

Default

1,048,576

Change

Dynamic, takes effect per-pool when the pool is imported

Versions Affected

v0.8.0

zfs_zil_clean_taskq_minalloc

During a SPA sync, intent log transaction groups (itxg) are cleaned. The cleaning work is dispatched to the DSL pool ZIL clean taskq (dp_zil_clean_taskq). zfs_zil_clean_taskq_minalloc is the minimum and zfs_zil_clean_taskq_maxalloc is the maximum number of cached taskq entries for dp_zil_clean_taskq. The actual number of taskq entries dynamically varies between these values.

zfs_zil_clean_taskq_minalloc is the minimum number of ZIL transaction records (itxs).

Ideally taskq entries are pre-allocated prior to being needed by zil_clean(), thus avoiding dynamic allocation of new taskq entries.

zfs_zil_clean_taskq_minalloc

Notes

Tags

ZIL

When to change

TBD

Data Type

int

Units

dp_zil_clean_taskq taskq entries

Range

1 to zfs_zil_clean_taskq_maxallo c

Default

1,024

Change

Dynamic, takes effect per-pool when the pool is imported

Versions Affected

v0.8.0

zfs_zil_clean_taskq_nthr_pct

zfs_zil_clean_taskq_nthr_pct controls the number of threads used by the DSL pool ZIL clean taskq (dp_zil_clean_taskq). The default value of 100% will create a maximum of one thread per cpu.

zfs_zil_clean_taskq_nthr_pct

Notes

Tags

taskq, ZIL

When to change

Testing ZIL clean and SPA sync performance

Data Type

int

Units

percent of number of CPUs

Range

1 to 100

Default

100

Change

Dynamic, takes effect per-pool when the pool is imported

Versions Affected

v0.8.0

zil_replay_disable

If zil_replay_disable = 1, then when a volume or filesystem is brought online, no attempt to replay the ZIL is made and any existing ZIL is destroyed. This can result in loss of data without notice.

zil_replay_disable

Notes

Tags

debug, ZIL

When to change

Do not change

Data Type

boolean

Range

0=replay ZIL, 1=destroy ZIL

Default

0

Change

Dynamic

Versions Affected

v0.6.5

zil_slog_bulk

zil_slog_bulk is the log device write size limit per commit executed with synchronous priority. Writes below zil_slog_bulk are executed with synchronous priority. Writes above zil_slog_bulk are executed with lower (asynchronous) priority to reduct potential log device abuse by a single active ZIL writer.

zil_slog_bulk

Notes

Tags

ZIL

When to change

See ZFS I/O Scheduler

Data Type

ulong

Units

bytes

Range

0 to ULONG_MAX

Default

786,432

Change

Dynamic

Versions Affected

v0.8.0

zio_delay_max

If a ZFS I/O operation takes more than zio_delay_max milliseconds to complete, then an event is logged. Note that this is only a logging facility, not a timeout on operations. See also zpool events

zio_delay_max

Notes

Tags

debug

When to change

when debugging slow I/O

Data Type

int

Units

milliseconds

Range

1 to INT_MAX

Default

30,000 (30 seconds)

Change

Dynamic

Versions Affected

all

zio_dva_throttle_enabled

zio_dva_throttle_enabled controls throttling of block allocations in the ZFS I/O (ZIO) pipeline. When enabled, the maximum number of pending allocations per top-level vdev is limited by zfs_vdev_queue_depth_pct

zio_dva_throttle_enabled

Notes

Tags

vdev, ZIO_scheduler

When to change

Testing ZIO block allocation algorithms

Data Type

boolean

Range

0=do not throttle ZIO block allocations, 1=throttle ZIO block allocations

Default

1

Change

Dynamic

Versions Affected

v0.7.0 and later

zio_requeue_io_start_cut_in_line

zio_requeue_io_start_cut_in_line controls prioritization of a re-queued ZFS I/O (ZIO) in the ZIO pipeline by the ZIO taskq.

zio_requeue_io_start_cut_in_line

Notes

Tags

Z IO_scheduler

When to change

Do not change

Data Type

boolean

Range

0=don’t prioritize re-queued I/Os, 1=prioritize re-queued I/Os

Default

1

Change

Dynamic

Versions Affected

all

zio_taskq_batch_pct

zio_taskq_batch_pct sets the number of I/O worker threads as a percentage of online CPUs. These workers threads are responsible for IO work such as compression and checksum calculations.

Each block is handled by one worker thread, so maximum overall worker thread throughput is function of the number of concurrent blocks being processed, the number of worker threads, and the algorithms used. The default value of 75% is chosen to avoid using all CPUs which can result in latency issues and inconsistent application performance, especially when high compression is enabled.

The taskq batch processes are:

taskq

process name

Notes

Write issue

z_wr_iss[_#]

Can be CPU intensive, runs at lower priority than other taskqs

Other taskqs exist, but most have fixed numbers of instances and therefore require recompiling the kernel module to adjust.

zio_taskq_batch_pct

Notes

Tags

taskq, ZIO_scheduler

When to change

To tune parallelism in multiprocessor systems

Data Type

int

Units

percent of number of CPUs

Range

1 to 100, fractional number of CPUs are rounded down

Default

75

Change

Prior to zfs module load

Verification

The number of taskqs for each batch group can be observed using ps and counting the threads

Versions Affected

TBD

zvol_inhibit_dev

zvol_inhibit_dev controls the creation of volume device nodes upon pool import.

zvol_inhibit_dev

Notes

Tags

import, volume

When to change

Inhibiting can slightly improve startup time on systems with a very large number of volumes

Data Type

boolean

Range

0=create volume device nodes, 1=do not create volume device nodes

Default

0

Change

Dynamic, takes effect per-pool when the pool is imported

Versions Affected

v0.6.0 and later

zvol_major

zvol_major is the default major number for volume devices.

zvol_major

Notes

Tags

volume

When to change

Do not change

Data Type

uint

Default

230

Change

Dynamic, takes effect per-pool when the pool is imported or volumes are created

Versions Affected

all

zvol_max_discard_blocks

Discard (aka ATA TRIM or SCSI UNMAP) operations done on volumes are done in batches zvol_max_discard_blocks blocks. The block size is determined by the volblocksize property of a volume.

Some applications, such as mkfs, discard the whole volume at once using the maximum possible discard size. As a result, many gigabytes of discard requests are not uncommon. Unfortunately, if a large amount of data is already allocated in the volume, ZFS can be quite slow to process discard requests. This is especially true if the volblocksize is small (eg default=8KB). As a result, very large discard requests can take a very long time (perhaps minutes under heavy load) to complete. This can cause a number of problems, most notably if the volume is accessed remotely (eg via iSCSI), in which case the client has a high probability of timing out on the request.

Limiting the zvol_max_discard_blocks can decrease the amount of discard workload request by setting the discard_max_bytes and discard_max_hw_bytes for the volume’s block device in SysFS. This value is readable by volume device consumers.

zvol_max_discard_blocks

Notes

Tags

discard, volume

When to change

if volume discard activity severely impacts other workloads

Data Type

ulong

Units

number of blocks of size volblocksize

Range

0 to ULONG_MAX

Default

16,384

Change

Dynamic, takes effect per-pool when the pool is imported or volumes are created

Verification

Observe value of /sys/block/ VOLUME_INSTANCE/queue/discard_max_bytes

Versions Affected

v0.6.0 and later

zvol_prefetch_bytes

When importing a pool with volumes or adding a volume to a pool, zvol_prefetch_bytes are prefetch from the start and end of the volume. Prefetching these regions of the volume is desirable because they are likely to be accessed immediately by blkid(8) or by the kernel scanning for a partition table.

zvol_prefetch_bytes

Notes

Tags

prefetch, volume

When to change

TBD

Data Type

uint

Units

bytes

Range

0 to UINT_MAX

Default

131,072

Change

Dynamic

Versions Affected

v0.6.5 and later

zvol_request_sync

When processing I/O requests for a volume submit them synchronously. This effectively limits the queue depth to 1 for each I/O submitter. When set to 0 requests are handled asynchronously by the “zvol” thread pool.

See also zvol_threads

zvol_request_sync

Notes

Tags

volume

When to change

Testing concurrent volume requests

Data Type

boolean

Range

0=do concurrent (async) volume requests, 1=do sync volume requests

Default

0

Change

Dynamic

Versions Affected

v0.7.2 and later

zvol_threads

zvol_threads controls the maximum number of threads handling concurrent volume I/O requests.

The default of 32 threads behaves similarly to a disk with a 32-entry command queue. The actual number of threads required can vary widely by workload and available CPUs. If lock analysis shows high contention in the zvol taskq threads, then reducing the number of zvol_threads or workload queue depth can improve overall throughput.

See also zvol_request_sync

zvol_threads

Notes

Tags

volume

When to change

Matching the number of concurrent volume requests with workload requirements can improve concurrency

Data Type

uint

Units

threads

Range

1 to UINT_MAX

Default

32

Change

Dynamic, takes effect per-volume when the pool is imported or volumes are created

Verification

iostat using avgqu-sz or aqu-sz results

Versions Affected

v0.7.0 and later

zvol_volmode

zvol_volmode defines volume block devices behaviour when the volmode property is set to default

Note: to maintain compatibility with ZFS on BSD, “geom” is synonymous with “full”

value

volmode

Description

1

full

legacy fully functional behaviour (default)

2

dev

hide partitions on volume block devices

3

none

not exposing volumes outside ZFS

zvol_volmode

Notes

Tags

volume

When to change

TBD

Data Type

enum

Range

1, 2, or 3

Default

1

Change

Dynamic

Versions Affected

v0.7.0 and later

zfs_qat_disable

zfs_qat_disable controls the Intel QuickAssist Technology (QAT) driver providing hardware acceleration for gzip compression. When the QAT hardware is present and qat driver available, the default behaviour is to enable QAT.

zfs_qat_disable

Notes

Tags

compression, QAT

When to change

Testing QAT functionality

Data Type

boolean

Range

0=use QAT acceleration if available, 1=do not use QAT acceleration

Default

0

Change

Dynamic

Versions Affected

v0.7, renamed to zfs_qat_ compress_disable in v0.8

zfs_qat_checksum_disable

zfs_qat_checksum_disable controls the Intel QuickAssist Technology (QAT) driver providing hardware acceleration for checksums. When the QAT hardware is present and qat driver available, the default behaviour is to enable QAT.

zfs_qat_checksum_disable

Notes

Tags

checksum, QAT

When to change

Testing QAT functionality

Data Type

boolean

Range

0=use QAT acceleration if available, 1=do not use QAT acceleration

Default

0

Change

Dynamic

Versions Affected

v0.8.0

zfs_qat_compress_disable

zfs_qat_compress_disable controls the Intel QuickAssist Technology (QAT) driver providing hardware acceleration for gzip compression. When the QAT hardware is present and qat driver available, the default behaviour is to enable QAT.

zfs_qat_compress_disable

Notes

Tags

compression, QAT

When to change

Testing QAT functionality

Data Type

boolean

Range

0=use QAT acceleration if available, 1=do not use QAT acceleration

Default

0

Change

Dynamic

Versions Affected

v0.8.0

zfs_qat_encrypt_disable

zfs_qat_encrypt_disable controls the Intel QuickAssist Technology (QAT) driver providing hardware acceleration for encryption. When the QAT hardware is present and qat driver available, the default behaviour is to enable QAT.

zfs_qat_encrypt_disable

Notes

Tags

encryption, QAT

When to change

Testing QAT functionality

Data Type

boolean

Range

0=use QAT acceleration if available, 1=do not use QAT acceleration

Default

0

Change

Dynamic

Versions Affected

v0.8.0

dbuf_cache_hiwater_pct

The dbuf_cache_hiwater_pct and dbuf_cache_lowater_pct define the operating range for dbuf cache evict thread. The hiwater and lowater are percentages of the dbuf_cache_max_bytes value. When the dbuf cache grows above ((100% + dbuf_cache_hiwater_pct) * dbuf_cache_max_bytes) then the dbuf cache thread begins evicting. When the dbug cache falls below ((100% - dbuf_cache_lowater_pct) * dbuf_cache_max_bytes) then the dbuf cache thread stops evicting.

dbuf_cache_hiwater_pct

Notes

Tags

dbuf_cache

When to change

Testing dbuf cache algorithms

Data Type

uint

Units

percent

Range

0 to UINT_MAX

Default

10

Change

Dynamic

Versions Affected

v0.7.0 and later

dbuf_cache_lowater_pct

The dbuf_cache_hiwater_pct and dbuf_cache_lowater_pct define the operating range for dbuf cache evict thread. The hiwater and lowater are percentages of the dbuf_cache_max_bytes value. When the dbuf cache grows above ((100% + dbuf_cache_hiwater_pct) * dbuf_cache_max_bytes) then the dbuf cache thread begins evicting. When the dbug cache falls below ((100% - dbuf_cache_lowater_pct) * dbuf_cache_max_bytes) then the dbuf cache thread stops evicting.

dbuf_cache_lowater_pct

Notes

Tags

dbuf_cache

When to change

Testing dbuf cache algorithms

Data Type

uint

Units

percent

Range

0 to UINT_MAX

Default

10

Change

Dynamic

Versions Affected

v0.7.0 and later

dbuf_cache_max_bytes

The dbuf cache maintains a list of dbufs that are not currently held but have been recently released. These dbufs are not eligible for ARC eviction until they are aged out of the dbuf cache. Dbufs are added to the dbuf cache once the last hold is released. If a dbuf is later accessed and still exists in the dbuf cache, then it will be removed from the cache and later re-added to the head of the cache. Dbufs that are aged out of the cache will be immediately destroyed and become eligible for ARC eviction.

The size of the dbuf cache is set by dbuf_cache_max_bytes. The actual size is dynamically adjusted to the minimum of current ARC target size (c) >> dbuf_cache_max_shift and the default dbuf_cache_max_bytes

dbuf_cache_max_bytes

Notes

Tags

dbuf_cache

When to change

Testing dbuf cache algorithms

Data Type

ulong

Units

bytes

Range

16,777,216 to ULONG_MAX

Default

104,857,600 (100 MiB)

Change

Dynamic

Versions Affected

v0.7.0 and later

dbuf_cache_max_shift

The dbuf_cache_max_bytes minimum is the lesser of dbuf_cache_max_bytes and the current ARC target size (c) >> dbuf_cache_max_shift

dbuf_cache_max_shift

Notes

Tags

dbuf_cache

When to change

Testing dbuf cache algorithms

Data Type

int

Units

shift

Range

1 to 63

Default

5

Change

Dynamic

Versions Affected

v0.7.0 and later

dmu_object_alloc_chunk_shift

Each of the concurrent object allocators grabs 2^dmu_object_alloc_chunk_shift dnode slots at a time. The default is to grab 128 slots, or 4 blocks worth. This default value was experimentally determined to be the lowest value that eliminates the measurable effect of lock contention in the DMU object allocation code path.

dmu_object_alloc_chunk_shift

Notes

Tags

allocation, DMU

When to change

If the workload creates many files concurrently on a system with many CPUs, then increasing dmu_object_alloc_chunk_shift can decrease lock contention

Data Type

int

Units

shift

Range

7 to 9

Default

7

Change

Dynamic

Versions Affected

v0.7.0 and later

send_holes_without_birth_time

Alias for ignore_hole_birth

zfs_abd_scatter_enabled

zfs_abd_scatter_enabled controls the ARC Buffer Data (ABD) scatter/gather feature.

When disabled, the legacy behaviour is selected using linear buffers. For linear buffers, all the data in the ABD is stored in one contiguous buffer in memory (from a zio_[data_]buf_* kmem cache).

When enabled (default), the data in the ABD is split into equal-sized chunks (from the abd_chunk_cache kmem_cache), with pointers to the chunks recorded in an array at the end of the ABD structure. This allows more efficient memory allocation for buffers, especially when large recordsizes are used.

zfs_abd_scatter_enabled

Notes

Tags

ABD, memory

When to change

Testing ABD

Data Type

boolean

Range

0=use linear allocation only, 1=allow scatter/gather

Default

1

Change

Dynamic

Verification

ABD statistics are observable in /proc/spl/kstat/zfs/abdstats. Slab allocations are observable in /proc/slabinfo

Versions Affected

v0.7.0 and later

zfs_abd_scatter_max_order

zfs_abd_scatter_max_order sets the maximum order for physical page allocation when ABD is enabled (see zfs_abd_scatter_enabled)

See also Buddy Memory Allocation in the Linux kernel documentation.

zfs_abd_scatter_max_order

Notes

Tags

ABD, memory

When to change

Testing ABD features

Data Type

int

Units

orders

Range

1 to 10 (upper limit is hardware-dependent)

Default

10

Change

Dynamic

Verification

ABD statistics are observable in /proc/spl/kstat/zfs/abdstats

Versions Affected

v0.7.0 and later

zfs_compressed_arc_enabled

When compression is enabled for a dataset, later reads of the data can store the blocks in ARC in their on-disk, compressed state. This can increse the effective size of the ARC, as counted in blocks, and thus improve the ARC hit ratio.

zfs_compressed_arc_enabled

Notes

Tags

ABD, compression

When to change

Testing ARC compression feature

Data Type

boolean

Range

0=compressed ARC disabled (legacy behaviour), 1=compress ARC data

Default

1

Change

Dynamic

Verification

raw ARC statistics are observable in /proc/spl/kstat/zfs/arcstats and ARC hit ratios can be observed using arcstat

Versions Affected

v0.7.0 and later

zfs_key_max_salt_uses

For encrypted datasets, the salt is regenerated every zfs_key_max_salt_uses blocks. This automatic regeneration reduces the probability of collisions due to the Birthday problem. When set to the default (400,000,000) the probability of collision is approximately 1 in 1 trillion.

zfs_key_max_salt_uses

Notes

Tags

encryption

When to change

Testing encryption features

Data Type

ulong

Units

blocks encrypted

Range

1 to ULONG_MAX

Default

400,000,000

Change

Dynamic

Versions Affected

v0.8.0 and later

zfs_object_mutex_size

zfs_object_mutex_size facilitates resizing the the per-dataset znode mutex array for testing deadlocks therein.

zfs_object_mutex_size

Notes

Tags

debug

When to change

Testing znode mutex array deadlocks

Data Type

uint

Units

orders

Range

1 to UINT_MAX

Default

64

Change

Dynamic

Versions Affected

v0.7.0 and later

zfs_scan_strict_mem_lim

When scrubbing or resilvering, by default, ZFS checks to ensure it is not over the hard memory limit before each txg commit. If finer-grained control of this is needed zfs_scan_strict_mem_lim can be set to 1 to enable checking before scanning each block.

zfs_scan_strict_mem_lim

Notes

Tags

memory, resilver, scrub

When to change

Do not change

Data Type

boolean

Range

0=normal scan behaviour, 1=check hard memory limit strictly during scan

Default

0

Change

Dynamic

Versions Affected

v0.8.0

zfs_send_queue_length

zfs_send_queue_length is the maximum number of bytes allowed in the zfs send queue.

zfs_send_queue_length

Notes

Tags

send

When to change

When using the largest recordsize or volblocksize (16 MiB), increasing can improve send efficiency

Data Type

int

Units

bytes

Range

Must be at least twice the maximum recordsize or volblocksize in use

Default

16,777,216 bytes (16 MiB)

Change

Dynamic

Versions Affected

v0.8.1

zfs_recv_queue_length

zfs_recv_queue_length is the maximum number of bytes allowed in the zfs receive queue.

zfs_recv_queue_length

Notes

Tags

receive

When to change

When using the largest recordsize or volblocksize (16 MiB), increasing can improve receive efficiency

Data Type

int

Units

bytes

Range

Must be at least twice the maximum recordsize or volblocksize in use

Default

16,777,216 bytes (16 MiB)

Change

Dynamic

Versions Affected

v0.8.1

zfs_arc_min_prefetch_lifespan

arc_min_prefetch_lifespan is the minimum time for a prefetched block to remain in ARC before it is eligible for eviction.

zfs_arc_min_prefetch_lifespan

Notes

Tags

ARC

When to change

TBD

Data Type

int

Units

clock ticks

Range

0 = use default value

Default

1 second (as expressed in clock ticks)

Change

Dynamic

Versions Affected

v0.7.0

zfs_scan_ignore_errors

zfs_scan_ignore_errors allows errors discovered during scrub or resilver to be ignored. This can be tuned as a workaround to remove the dirty time list (DTL) when completing a pool scan. It is intended to be used during pool repair or recovery to prevent resilvering when the pool is imported.

zfs_scan_ignore_errors

Notes

Tags

resilver

When to change

See description above

Data Type

boolean

Range

0 = do not ignore errors, 1 = ignore errors during pool scrub or resilver

Default

0

Change

Dynamic

Versions Affected

v0.8.1

zfs_top_maxinflight

zfs_top_maxinflight is used to limit the maximum number of I/Os queued to top-level vdevs during scrub or resilver operations. The actual top-level vdev limit is calculated by multiplying the number of child vdevs by zfs_top_maxinflight This limit is an additional cap over and above the scan limits

zfs_top_maxinflight

Notes

Tags

resilver, scrub, ZIO_scheduler

When to change

for modern ZFS versions, the ZIO scheduler limits usually take precedence

Data Type

int

Units

I/O operations

Range

1 to MAX_INT

Default

32

Change

Dynamic

Versions Affected

v0.6.0

zfs_resilver_delay

zfs_resilver_delay sets a time-based delay for resilver I/Os. This delay is in addition to the ZIO scheduler’s treatment of scrub workloads. See also zfs_scan_idle

zfs_resilver_delay

Notes

Tags

resilver, ZIO_scheduler

When to change

increasing can reduce impact of resilver workload on dynamic workloads

Data Type

int

Units

clock ticks

Range

0 to MAX_INT

Default

2

Change

Dynamic

Versions Affected

v0.6.0

zfs_scrub_delay

zfs_scrub_delay sets a time-based delay for scrub I/Os. This delay is in addition to the ZIO scheduler’s treatment of scrub workloads. See also zfs_scan_idle

zfs_scrub_delay

Notes

Tags

scrub, ZIO_scheduler

When to change

increasing can reduce impact of scrub workload on dynamic workloads

Data Type

int

Units

clock ticks

Range

0 to MAX_INT

Default

4

Change

Dynamic

Versions Affected

v0.6.0

zfs_scan_idle

When a non-scan I/O has occurred in the past zfs_scan_idle clock ticks, then zfs_resilver_delay or zfs_scrub_delay are enabled.

zfs_scan_idle

Notes

Tags

resilver, scrub, ZIO_scheduler

When to change

as part of a resilver/scrub tuning effort

Data Type

int

Units

clock ticks

Range

0 to MAX_INT

Default

50

Change

Dynamic

Versions Affected

v0.6.0

icp_aes_impl

By default, ZFS will choose the highest performance, hardware-optimized implementation of the AES encryption algorithm. The icp_aes_impl tunable overrides this automatic choice.

Note: icp_aes_impl is set in the icp kernel module, not the zfs kernel module.

To observe the available options cat /sys/module/icp/parameters/icp_aes_impl The default option is shown in brackets ‘[]’

icp_aes_impl

Notes

Tags

encryption

Kernel module

icp

When to change

debugging ZFS encryption on hardware

Data Type

string

Range

varies by hardware

Default

automatic, depends on the hardware

Change

dynamic

Versions Affected

planned for v2

icp_gcm_impl

By default, ZFS will choose the highest performance, hardware-optimized implementation of the GCM encryption algorithm. The icp_gcm_impl tunable overrides this automatic choice.

Note: icp_gcm_impl is set in the icp kernel module, not the zfs kernel module.

To observe the available options cat /sys/module/icp/parameters/icp_gcm_impl The default option is shown in brackets ‘[]’

icp_gcm_impl

Notes

Tags

encryption

Kernel module

icp

When to change

debugging ZFS encryption on hardware

Data Type

string

Range

varies by hardware

Default

automatic, depends on the hardware

Change

Dynamic

Versions Affected

planned for v2

zfs_abd_scatter_min_size

zfs_abd_scatter_min_size changes the ARC buffer data (ABD) allocator’s threshold for using linear or page-based scatter buffers. Allocations smaller than zfs_abd_scatter_min_size use linear ABDs.

Scatter ABD’s use at least one page each, so sub-page allocations waste some space when allocated as scatter allocations. For example, 2KB scatter allocation wastes half of each page. Using linear ABD’s for small allocations results in slabs containing many allocations. This can improve memory efficiency, at the expense of more work for ARC evictions attempting to free pages, because all the buffers on one slab need to be freed in order to free the slab and its underlying pages.

Typically, 512B and 1KB kmem caches have 16 buffers per slab, so it’s possible for them to actually waste more memory than scatter allocations:

  • one page per buf = wasting 3/4 or 7/8

  • one buf per slab = wasting 15/16

Spill blocks are typically 512B and are heavily used on systems running selinux with the default dnode size and the xattr=sa property set.

By default, linear allocations for 512B and 1KB, and scatter allocations for larger (>= 1.5KB) allocation requests.

zfs_abd_scatter_min_size

Notes

Tags

ARC

When to change

debugging memory allocation, especially for large pages

Data Type

int

Units

bytes

Range

0 to MAX_INT

Default

1536 (512B and 1KB allocations will be linear)

Change

Dynamic

Versions Affected

planned for v2

spa_load_verify_shift

spa_load_verify_shift sets the fraction of ARC that can be used by inflight I/Os when verifying the pool during import. This value is a “shift” representing the fraction of ARC target size (grep -w c /proc/spl/kstat/zfs/arcstats). The ARC target size is shifted to the right. Thus a value of ‘2’ results in the fraction = 1/4, while a value of ‘4’ results in the fraction = 1/8.

For large memory machines, pool import can consume large amounts of ARC: much larger than the value of maxinflight. This can result in spa_load_verify_maxinflight having a value of 0 causing the system to hang. Setting spa_load_verify_shift can reduce this limit and allow importing without hanging.

spa_load_verify_shift

Notes

Tags

import, ARC, SPA

When to change

troubleshooting pool import on large memory machines

Data Type

int

Units

shift

Range

1 to MAX_INT

Default

4

Change

prior to importing a pool

Versions Affected

planned for v2

spa_load_print_vdev_tree

spa_load_print_vdev_tree enables printing of the attempted pool import’s vdev tree to kernel message to the ZFS debug message log /proc/spl/kstat/zfs/dbgmsg Both the provided vdev tree and MOS vdev tree are printed, which can be useful for debugging problems with the zpool cachefile

spa_load_print_vdev_tree

Notes

Tags

import, SPA

When to change

troubleshooting pool import failures

Data Type

boolean

Range

0 = do not print pool configuration in logs, 1 = print pool configuration in logs

Default

0

Change

prior to pool import

Versions Affected

planned for v2

zfs_max_missing_tvds

When importing a pool in readonly mode (zpool import -o readonly=on ...) then up to zfs_max_missing_tvds top-level vdevs can be missing, but the import can attempt to progress.

Note: This is strictly intended for advanced pool recovery cases since missing data is almost inevitable. Pools with missing devices can only be imported read-only for safety reasons, and the pool’s failmode property is automatically set to continue

The expected use case is to recover pool data immediately after accidentally adding a non-protected vdev to a protected pool.

  • With 1 missing top-level vdev, ZFS should be able to import the pool and mount all datasets. User data that was not modified after the missing device has been added should be recoverable. Thus snapshots created prior to the addition of that device should be completely intact.

  • With 2 missing top-level vdevs, some datasets may fail to mount since there are dataset statistics that are stored as regular metadata. Some data might be recoverable if those vdevs were added recently.

  • With 3 or more top-level missing vdevs, the pool is severely damaged and MOS entries may be missing entirely. Chances of data recovery are very low. Note that there are also risks of performing an inadvertent rewind as we might be missing all the vdevs with the latest uberblocks.

zfs_max_missing_tvds

Notes

Tags

import

When to change

troubleshooting pools with missing devices

Data Type

int

Units

missing top-level vdevs

Range

0 to MAX_INT

Default

0

Change

prior to pool import

Versions Affected

planned for v2

dbuf_metadata_cache_shift

dbuf_metadata_cache_shift sets the size of the dbuf metadata cache as a fraction of ARC target size. This is an alternate method for setting dbuf metadata cache size than dbuf_metadata_cache_max_bytes.

dbuf_metadata_cache_max_bytes overrides dbuf_metadata_cache_shift

This value is a “shift” representing the fraction of ARC target size (grep -w c /proc/spl/kstat/zfs/arcstats). The ARC target size is shifted to the right. Thus a value of ‘2’ results in the fraction = 1/4, while a value of ‘6’ results in the fraction = 1/64.

dbuf_metadata_cache_shift

Notes

Tags

ARC, dbuf_cache

When to change

Data Type

int

Units

shift

Range

practical range is (` dbuf_cache_shift <#dbuf-cache-shift>`__ + 1) to MAX_INT

Default

6

Change

Dynamic

Versions Affected

planned for v2

dbuf_metadata_cache_max_bytes

dbuf_metadata_cache_max_bytes sets the size of the dbuf metadata cache as a number of bytes. This is an alternate method for setting dbuf metadata cache size than dbuf_metadata_cache_shift

dbuf_metadata_cache_max_bytes overrides dbuf_metadata_cache_shift

dbuf_metadata_cache_max_bytes

Notes

Tags

dbuf_cache

When to change

Data Type

int

Units

bytes

Range

0 = use dbuf_metadata_cache_sh ift to ARC c_max

Default

0

Change

Dynamic

Versions Affected

planned for v2

dbuf_cache_shift

dbuf_cache_shift sets the size of the dbuf cache as a fraction of ARC target size. This is an alternate method for setting dbuf cache size than dbuf_cache_max_bytes.

dbuf_cache_max_bytes overrides dbuf_cache_shift

This value is a “shift” representing the fraction of ARC target size (grep -w c /proc/spl/kstat/zfs/arcstats). The ARC target size is shifted to the right. Thus a value of ‘2’ results in the fraction = 1/4, while a value of ‘5’ results in the fraction = 1/32.

Performance tuning of dbuf cache can be monitored using:

  • dbufstat command

  • node_exporter ZFS module for prometheus environments

  • telegraf ZFS plugin for general-purpose metric collection

  • /proc/spl/kstat/zfs/dbufstats kstat

dbuf_cache_shift

Notes

Tags

ARC, dbuf_cache

When to change

to improve performance of read-intensive channel programs

Data Type

int

Units

shift

Range

5 to MAX_INT

Default

5

Change

Dynamic

Versions Affected

planned for v2

dbuf_cache_max_bytes

dbuf_cache_max_bytes sets the size of the dbuf cache in bytes. This is an alternate method for setting dbuf cache size than dbuf_cache_shift

Performance tuning of dbuf cache can be monitored using:

  • dbufstat command

  • node_exporter ZFS module for prometheus environments

  • telegraf ZFS plugin for general-purpose metric collection

  • /proc/spl/kstat/zfs/dbufstats kstat

dbuf_cache_max_bytes

Notes

Tags

ARC, dbuf_cache

When to change

Data Type

int

Units

bytes

Range

0 = use dbuf_cache_shift to ARC c_max

Default

0

Change

Dynamic

Versions Affected

planned for v2

metaslab_force_ganging

When testing allocation code, metaslab_force_ganging forces blocks above the specified size to be ganged.

metaslab_force_ganging

Notes

Tags

allocation

When to change

for development testing purposes only

Data Type

ulong

Units

bytes

Range

SPA_MINBLOCKSIZE to (SPA_MAXBLOCKSIZE + 1)

Default

SPA_MAXBLOCKSIZE + 1 (16,777,217 bytes)

Change

Dynamic

Versions Affected

planned for v2

zfs_vdev_default_ms_count

When adding a top-level vdev, zfs_vdev_default_ms_count is the target number of metaslabs.

zfs_vdev_default_ms_count

Notes

Tags

allocation

When to change

for development testing purposes only

Data Type

int

Range

16 to MAX_INT

Default

200

Change

prior to creating a pool or adding a top-level vdev

Versions Affected

planned for v2

vdev_removal_max_span

During top-level vdev removal, chunks of data are copied from the vdev which may include free space in order to trade bandwidth for IOPS. vdev_removal_max_span sets the maximum span of free space included as unnecessary data in a chunk of copied data.

vdev_removal_max_span

Notes

Tags

vdev_removal

When to change

TBD

Data Type

int

Units

bytes

Range

0 to MAX_INT

Default

32,768 (32 MiB)

Change

Dynamic

Versions Affected

planned for v2

zfs_removal_ignore_errors

When removing a device, zfs_removal_ignore_errors controls the process for handling hard I/O errors. When set, if a device encounters a hard IO error during the removal process the removal will not be cancelled. This can result in a normally recoverable block becoming permanently damaged and is not recommended. This should only be used as a last resort when the pool cannot be returned to a healthy state prior to removing the device.

zfs_removal_ignore_errors

Notes

Tags

vdev_removal

When to change

See description for caveat

Data Type

boolean

Range

during device removal: 0 = hard errors are not ignored, 1 = hard errors are ignored

Default

0

Change

Dynamic

Versions Affected

planned for v2

zfs_removal_suspend_progress

zfs_removal_suspend_progress is used during automated testing of the ZFS code to incease test coverage.

zfs_removal_suspend_progress

Notes

Tags

vdev_removal

When to change

do not change

Data Type

boolean

Range

0 = do not suspend during vdev removal

Default

0

Change

Dynamic

Versions Affected

planned for v2

zfs_condense_indirect_commit_entry_delay_ms

During vdev removal, the vdev indirection layer sleeps for zfs_condense_indirect_commit_entry_delay_ms milliseconds during mapping generation. This parameter is used during automated testing of the ZFS code to improve test coverage.

zfs_condens e_indirect_commit_entry_delay_ms

Notes

Tags

vdev_removal

When to change

do not change

Data Type

int

Units

milliseconds

Range

0 to MAX_INT

Default

0

Change

Dynamic

Versions Affected

planned for v2

zfs_condense_indirect_vdevs_enable

During vdev removal, condensing process is an attempt to save memory by removing obsolete mappings. zfs_condense_indirect_vdevs_enable enables condensing indirect vdev mappings. When set, ZFS attempts to condense indirect vdev mappings if the mapping uses more than zfs_condense_min_mapping_bytes bytes of memory and if the obsolete space map object uses more than zfs_condense_max_obsolete_bytes bytes on disk.

zf s_condense_indirect_vdevs_enable

Notes

Tags

vdev_removal

When to change

TBD

Data Type

boolean

Range

0 = do not save memory, 1 = save memory by condensing obsolete mapping after vdev removal

Default

1

Change

Dynamic

Versions Affected

planned for v2

zfs_condense_max_obsolete_bytes

After vdev removal, zfs_condense_max_obsolete_bytes sets the limit for beginning the condensing process. Condensing begins if the obsolete space map takes up more than zfs_condense_max_obsolete_bytes of space on disk (logically). The default of 1 GiB is small enough relative to a typical pool that the space consumed by the obsolete space map is minimal.

See also zfs_condense_indirect_vdevs_enable

zfs_condense_max_obsolete_bytes

Notes

Tags

vdev_removal

When to change

no not change

Data Type

ulong

Units

bytes

Range

0 to MAX_ULONG

Default

1,073,741,824 (1 GiB)

Change

Dynamic

Versions Affected

planned for v2

zfs_condense_min_mapping_bytes

After vdev removal, zfs_condense_min_mapping_bytes is the lower limit for determining when to condense the in-memory obsolete space map. The condensing process will not continue unless a minimum of zfs_condense_min_mapping_bytes of memory can be freed.

See also zfs_condense_indirect_vdevs_enable

zfs_condense_min_mapping_bytes

Notes

Tags

vdev_removal

When to change

do not change

Data Type

ulong

Units

bytes

Range

0 to MAX_ULONG

Default

128 KiB

Change

Dynamic

Versions Affected

planned for v2

zfs_vdev_initializing_max_active

zfs_vdev_initializing_max_active sets the maximum initializing I/Os active to each device.

zfs_vdev_initializing_max_active

Notes

Tags

vdev, Z IO_scheduler

When to change

See ZFS I/O Sch eduler

Data Type

uint32

Units

I/O operations

Range

1 to zfs_vdev_max_ active

Default

1

Change

Dynamic

Versions Affected

planned for v2

zfs_vdev_initializing_min_active

zfs_vdev_initializing_min_active sets the minimum initializing I/Os active to each device.

zfs_vdev_initializing_min_active

Notes

Tags

vdev, Z IO_scheduler

When to change

See ZFS I/O Sch eduler

Data Type

uint32

Units

I/O operations

Range

1 to zfs_vde v_initializing_max_active

Default

1

Change

Dynamic

Versions Affected

planned for v2

zfs_vdev_removal_max_active

zfs_vdev_removal_max_active sets the maximum top-level vdev removal I/Os active to each device.

zfs_vdev_removal_max_active

Notes

Tags

vdev, ZIO_scheduler

When to change

See ZFS I/O Scheduler

Data Type

uint32

Units

I/O operations

Range

1 to zfs_vdev _max_active

Default

2

Change

Dynamic

Versions Affected

planned for v2

zfs_vdev_removal_min_active

zfs_vdev_removal_min_active sets the minimum top-level vdev removal I/Os active to each device.

zfs_vdev_removal_min_active

Notes

Tags

vdev, ZIO_scheduler

When to change

See ZFS I/O Scheduler

Data Type

uint32

Units

I/O operations

Range

1 to zfs_vdev_removal_max_act ive

Default

1

Change

Dynamic

Versions Affected

planned for v2

zfs_vdev_trim_max_active

zfs_vdev_trim_max_active sets the maximum trim I/Os active to each device.

zfs_vdev_trim_max_active

Notes

Tags

vdev, ZIO_scheduler

When to change

See ZFS I/O Scheduler

Data Type

uint32

Units

I/O operations

Range

1 to zfs_v dev_max_active

Default

2

Change

Dynamic

Versions Affected

planned for v2

zfs_vdev_trim_min_active

zfs_vdev_trim_min_active sets the minimum trim I/Os active to each device.

zfs_vdev_trim_min_active

Notes

Tags

vdev, ZIO_scheduler

When to change

See ZFS I/O Scheduler

Data Type

uint32

Units

I/O operations

Range

1 to zfs_vdev_trim_m ax_active

Default

1

Change

Dynamic

Versions Affected

planned for v2

zfs_initialize_value

When initializing a vdev, ZFS writes patterns of zfs_initialize_value bytes to the device.

zfs_initialize_value

Notes

Tags

vdev_initialize

When to change

when debugging initialization code

Data Type

uint32 or uint64

Default

0xdeadbeef for 32-bit systems, 0xdeadbeefdeadbeee for 64-bit systems

Change

prior to running zpool initialize

Versions Affected

planned for v2

zfs_lua_max_instrlimit

zfs_lua_max_instrlimit limits the maximum time for a ZFS channel program to run.

zfs_lua_max_instrlimit

Notes

Tags

channel_programs

When to change

to enforce a CPU usage limit on ZFS channel programs

Data Type

ulong

Units

LUA instructions

Range

0 to MAX_ULONG

Default

100,000,000

Change

Dynamic

Versions Affected

planned for v2

zfs_lua_max_memlimit

‘zfs_lua_max_memlimit’ is the maximum memory limit for a ZFS channel program.

zfs_lua_max_memlimit

Notes

Tags

channel_programs

When to change

Data Type

ulong

Units

bytes

Range

0 to MAX_ULONG

Default

104,857,600 (100 MiB)

Change

Dynamic

Versions Affected

planned for v2

zfs_max_dataset_nesting

zfs_max_dataset_nesting limits the depth of nested datasets. Deeply nested datasets can overflow the stack. The maximum stack depth depends on kernel compilation options, so it is impractical to predict the possible limits. For kernels compiled with small stack sizes, zfs_max_dataset_nesting may require changes.

zfs_max_dataset_nesting

Notes

Tags

dataset

When to change

can be tuned temporarily to fix existing datasets that exceed the predefined limit

Data Type

int

Units

datasets

Range

0 to MAX_INT

Default

50

Change

Dynamic, though once on-disk the value for the pool is set

Versions Affected

planned for v2

zfs_ddt_data_is_special

zfs_ddt_data_is_special enables the deduplication table (DDT) to reside on a special top-level vdev.

zfs_ddt_data_is_special

Notes

Tags

dedup, special_vdev

When to change

when using a special top-level vdev and no dedup top-level vdev and it is desired to store the DDT in the main pool top-level vdevs

Data Type

boolean

Range

0=do not use special vdevs to store DDT, 1=store DDT in special vdevs

Default

1

Change

Dynamic

Versions Affected

planned for v2

zfs_user_indirect_is_special

If special vdevs are in use, zfs_user_indirect_is_special enables user data indirect blocks (a form of metadata) to be written to the special vdevs.

zfs_user_indirect_is_special

Notes

Tags

special_vdev

When to change

to force user data indirect blocks to remain in the main pool top-level vdevs

Data Type

boolean

Range

0=do not write user indirect blocks to a special vdev, 1=write user indirect blocks to a special vdev

Default

1

Change

Dynamic

Versions Affected

planned for v2

zfs_reconstruct_indirect_combinations_max

After device removal, if an indirect split block contains more than zfs_reconstruct_indirect_combinations_max many possible unique combinations when being reconstructed, it can be considered too computationally expensive to check them all. Instead, at most zfs_reconstruct_indirect_combinations_max randomly-selected combinations are attempted each time the block is accessed. This allows all segment copies to participate fairly in the reconstruction when all combinations cannot be checked and prevents repeated use of one bad copy.

zfs_recon struct_indirect_combinations_max

Notes

Tags

vdev_removal

When to change

TBD

Data Type

int

Units

attempts

Range

0=do not limit attempts, 1 to MAX_INT = limit for attempts

Default

4096

Change

Dynamic

Versions Affected

planned for v2

zfs_send_unmodified_spill_blocks

zfs_send_unmodified_spill_blocks enables sending of unmodified spill blocks in the send stream. Under certain circumstances, previous versions of ZFS could incorrectly remove the spill block from an existing object. Including unmodified copies of the spill blocks creates a backwards compatible stream which will recreate a spill block if it was incorrectly removed.

zfs_send_unmodified_spill_blocks

Notes

Tags

send

When to change

TBD

Data Type

boolean

Range

0=do not send unmodified spill blocks, 1=send unmodified spill blocks

Default

1

Change

Dynamic

Versions Affected

planned for v2

zfs_spa_discard_memory_limit

zfs_spa_discard_memory_limit sets the limit for maximum memory used for prefetching a pool’s checkpoint space map on each vdev while discarding a pool checkpoint.

zfs_spa_discard_memory_limit

Notes

Tags

checkpoint

When to change

TBD

Data Type

int

Units

bytes

Range

0 to MAX_INT

Default

16,777,216 (16 MiB)

Change

Dynamic

Versions Affected

planned for v2

zfs_special_class_metadata_reserve_pct

zfs_special_class_metadata_reserve_pct sets a threshold for space in special vdevs to be reserved exclusively for metadata. This prevents small blocks or dedup table from completely consuming a special vdev.

zfs_special_class_metadata_reserve_pct

Notes

Tags

special_vdev

When to change

TBD

Data Type

int

Units

percent

Range

0 to 100

Default

25

Change

Dynamic

Versions Affected

planned for v2

zfs_trim_extent_bytes_max

zfs_trim_extent_bytes_max sets the maximum size of a trim (aka discard, scsi unmap) command. Ranges larger than zfs_trim_extent_bytes_max are split in to chunks no larger than zfs_trim_extent_bytes_max bytes prior to being issued to the device. Use zpool iostat -w to observe the latency of trim commands.

zfs_trim_extent_bytes_max

Notes

Tags

trim

When to change

if the device can efficiently handle larger trim requests

Data Type

uint

Units

bytes

Range

zfs_trim_extent_by tes_min to MAX_UINT

Default

134,217,728 (128 MiB)

Change

Dynamic

Versions Affected

planned for v2

zfs_trim_extent_bytes_min

zfs_trim_extent_bytes_min sets the minimum size of trim (aka discard, scsi unmap) commands. Trim ranges smaller than zfs_trim_extent_bytes_min are skipped unless they’re part of a larger range which was broken in to chunks. Some devices have performance degradation during trim operations, so using a larger zfs_trim_extent_bytes_min can reduce the total amount of space trimmed. Use zpool iostat -w to observe the latency of trim commands.

zfs_trim_extent_bytes_min

Notes

Tags

trim

When to change

when trim is in use and device performance suffers from trimming small allocations

Data Type

uint

Units

bytes

Range

0=trim all unallocated space, otherwise minimum physical block size to MAX_

Default

32,768 (32 KiB)

Change

Dynamic

Versions Affected

planned for v2

zfs_trim_metaslab_skip

zfs_trim_metaslab_skip enables uninitialized metaslabs to be skipped during the trim (aka discard, scsi unmap) process. zfs_trim_metaslab_skip can be useful for pools constructed from large thinly-provisioned devices where trim operations perform slowly.
As a pool ages an increasing fraction of the pool’s metaslabs are initialized, progressively degrading the usefulness of this option. This setting is stored when starting a manual trim and persists for the duration of the requested trim. Use zpool iostat -w to observe the latency of trim commands.

zfs_trim_metaslab_skip

Notes

Tags

trim

When to change

Data Type

boolean

Range

0=do not skip uninitialized metaslabs during trim, 1=skip uninitialized metaslabs during trim

Default

0

Change

Dynamic

Versions Affected

planned for v2

zfs_trim_queue_limit

zfs_trim_queue_limit sets the maximum queue depth for leaf vdevs. See also zfs_vdev_trim_max_active and zfs_trim_extent_bytes_max Use zpool iostat -q to observe trim queue depth.

zfs_trim_queue_limit

Notes

Tags

trim

When to change

to restrict the number of trim commands in the queue

Data Type

uint

Units

I/O operations

Range

1 to MAX_UINT

Default

10

Change

Dynamic

Versions Affected

planned for v2

zfs_trim_txg_batch

zfs_trim_txg_batch sets the number of transaction groups worth of frees which should be aggregated before trim (aka discard, scsi unmap) commands are issued to a device. This setting represents a trade-off between issuing larger, more efficient trim commands and the delay before the recently trimmed space is available for use by the device.

Increasing this value will allow frees to be aggregated for a longer time. This will result is larger trim operations and potentially increased memory usage. Decreasing this value will have the opposite effect. The default value of 32 was empirically determined to be a reasonable compromise.

zfs_trim_txg_batch

Notes

Tags

trim

When to change

TBD

Data Type

uint

Units

metaslabs to stride

Range

1 to MAX_UINT

Default

32

Change

Dynamic

Versions Affected

planned for v2

zfs_vdev_aggregate_trim

zfs_vdev_aggregate_trim allows trim I/Os to be aggregated. This is normally not helpful because the extents to be trimmed will have been already been aggregated by the metaslab.

zfs_vdev_aggregate_trim

Notes

Tags

trim, vdev, ZIO_scheduler

When to change

when debugging trim code or trim performance issues

Data Type

boolean

Range

0=do not attempt to aggregate trim commands, 1=attempt to aggregate trim commands

Default

0

Change

Dynamic

Versions Affected

planned for v2

zfs_vdev_aggregation_limit_non_rotating

zfs_vdev_aggregation_limit_non_rotating is the equivalent of zfs_vdev_aggregation_limit for devices which represent themselves as non-rotating to the Linux blkdev interfaces. Such devices have a value of 0 in /sys/block/DEVICE/queue/rotational and are expected to be SSDs.

zfs_vde v_aggregation_limit_non_rotating

Notes

Tags

vdev, Z IO_scheduler

When to change

see zfs_vdev_aggregation_limit

Data Type

int

Units

bytes

Range

0 to MAX_INT

Default

131,072 bytes (128 KiB)

Change

Dynamic

Versions Affected

planned for v2

zil_nocacheflush

ZFS uses barriers (volatile cache flush commands) to ensure data is committed to permanent media by devices. This ensures consistent on-media state for devices where caches are volatile (eg HDDs).

zil_nocacheflush disables the cache flush commands that are normally sent to devices by the ZIL after a log write has completed.

The difference between zil_nocacheflush and zfs_nocacheflush is zil_nocacheflush applies to ZIL writes while zfs_nocacheflush disables barrier writes to the pool devices at the end of transaction group syncs.

WARNING: setting this can cause ZIL corruption on power loss if the device has a volatile write cache.

zil_nocacheflush

Notes

Tags

disks, ZIL

When to change

If the storage device has nonvolatile cache, then disabling cache flush can save the cost of occasional cache flush commands

Data Type

boolean

Range

0=send cache flush commands, 1=do not send cache flush commands

Default

0

Change

Dynamic

Versions Affected

planned for v2

zio_deadman_log_all

zio_deadman_log_all enables debugging messages for all ZFS I/Os, rather than only for leaf ZFS I/Os for a vdev. This is meant to be used by developers to gain diagnostic information for hang conditions which don’t involve a mutex or other locking primitive. Typically these are conditions where a thread in the zio pipeline is looping indefinitely.

See also zfs_dbgmsg_enable

zio_deadman_log_all

Notes

Tags

debug

When to change

when debugging ZFS I/O pipeline

Data Type

boolean

Range

0=do not log all deadman events, 1=log all deadman events

Default

0

Change

Dynamic

Versions Affected

planned for v2

zio_decompress_fail_fraction

If non-zero, zio_decompress_fail_fraction represents the denominator of the probability that ZFS should induce a decompression failure. For instance, for a 5% decompression failure rate, this value should be set to 20.

zio_decompress_fail_fraction

Notes

Tags

debug

When to change

when debugging ZFS internal compressed buffer code

Data Type

ulong

Units

probability of induced decompression failure is 1/zio_decompress_fail_fraction

Range

0 = do not induce failures, or 1 to MAX_ULONG

Default

0

Change

Dynamic

Versions Affected

planned for v2

zio_slow_io_ms

An I/O operation taking more than zio_slow_io_ms milliseconds to complete is marked as a slow I/O. Slow I/O counters can be observed with zpool status -s. Each slow I/O causes a delay zevent, observable using zpool events. See also zfs-events(5).

zio_slow_io_ms

Notes

Tags

vdev, zed

When to change

when debugging slow devices and the default value is inappropriate

Data Type

int

Units

milliseconds

Range

0 to MAX_INT

Default

30,000 (30 seconds)

Change

Dynamic

Versions Affected

planned for v2

vdev_validate_skip

vdev_validate_skip disables label validation steps during pool import. Changing is not recommended unless you know what you are doing and are recovering a damaged label.

vdev_validate_skip

Notes

Tags

vdev

When to change

do not change

Data Type

boolean

Range

0=validate labels during pool import, 1=do not validate vdev labels during pool import

Default

0

Change

prior to pool import

Versions Affected

planned for v2

zfs_async_block_max_blocks

zfs_async_block_max_blocks limits the number of blocks freed in a single transaction group commit. During deletes of large objects, such as snapshots, the number of freed blocks can cause the DMU to extend txg sync times well beyond zfs_txg_timeout. zfs_async_block_max_blocks is used to limit these effects.

zfs_async_block_max_blocks

Notes

Tags

delete, DMU

When to change

TBD

Data Type

ulong

Units

blocks

Range

1 to MAX_ULONG

Default

MAX_ULONG (do not limit)

Change

Dynamic

Versions Affected

planned for v2

zfs_checksum_events_per_second

zfs_checksum_events_per_second is a rate limit for checksum events. Note that this should not be set below the zed thresholds (currently 10 checksums over 10 sec) or else zed may not trigger any action.

zfs_checksum_events_per_second

Notes

Tags

vdev

When to change

TBD

Data Type

uint

Units

checksum events

Range

zed threshold to MAX_UINT

Default

20

Change

Dynamic

Versions Affected

planned for v2

zfs_disable_ivset_guid_check

zfs_disable_ivset_guid_check disables requirement for IVset guids to be present and match when doing a raw receive of encrypted datasets. Intended for users whose pools were created with ZFS on Linux pre-release versions and now have compatibility issues.

For a ZFS raw receive, from a send stream created by zfs send --raw, the crypt_keydata nvlist includes a to_ivset_guid to be set on the new snapshot. This value will override the value generated by the snapshot code. However, this value may not be present, because older implementations of the raw send code did not include this value. When zfs_disable_ivset_guid_check is enabled, the receive proceeds and a newly-generated value is used.

zfs_disable_ivset_guid_check

Notes

Tags

receive

When to change

debugging pre-release ZFS raw sends

Data Type

boolean

Range

0=check IVset guid, 1=do not check IVset guid

Default

0

Change

Dynamic

Versions Affected

planned for v2

zfs_obsolete_min_time_ms

zfs_obsolete_min_time_ms is similar to zfs_free_min_time_ms and used for cleanup of old indirection records for vdevs removed using the zpool remove command.

zfs_obsolete_min_time_ms

Notes

Tags

delete, remove

When to change

TBD

Data Type

int

Units

milliseconds

Range

0 to MAX_INT

Default

500

Change

Dynamic

Versions Affected

planned for v2

zfs_override_estimate_recordsize

zfs_override_estimate_recordsize overrides the default logic for estimating block sizes when doing a zfs send. The default heuristic is that the average block size will be the current recordsize.

zfs_override_estimate_recordsize

Notes

Tags

send

When to change

if most data in your dataset is not of the current recordsize and you require accurate zfs send size estimates

Data Type

ulong

Units

bytes

Range

0=do not override, 1 to MAX_ULONG

Default

0

Change

Dynamic

Versions Affected

planned for v2

zfs_remove_max_segment

zfs_remove_max_segment sets the largest contiguous segment that ZFS attempts to allocate when removing a vdev. This can be no larger than 16MB. If there is a performance problem with attempting to allocate large blocks, consider decreasing this. The value is rounded up to a power-of-2.

zfs_remove_max_segment

Notes

Tags

remove

When to change

after removing a top-level vdev, consider decreasing if there is a performance degradation when attempting to allocate large blocks

Data Type

int

Units

bytes

Range

maximum of the physical block size of all vdevs in the pool to 16,777,216 bytes (16 MiB)

Default

16,777,216 bytes (16 MiB)

Change

Dynamic

Versions Affected

planned for v2

zfs_resilver_disable_defer

zfs_resilver_disable_defer disables the resilver_defer pool feature. The resilver_defer feature allows ZFS to postpone new resilvers if an existing resilver is in progress.

zfs_resilver_disable_defer

Notes

Tags

resilver

When to change

if resilver postponement is not desired due to overall resilver time constraints

Data Type

boolean

Range

0=allow resilver_defer to postpone new resilver operations, 1=immediately restart resilver when needed

Default

0

Change

Dynamic

Versions Affected

planned for v2

zfs_scan_suspend_progress

zfs_scan_suspend_progress causes a scrub or resilver scan to freeze without actually pausing.

zfs_scan_suspend_progress

Notes

Tags

resilver, scrub

When to change

testing or debugging scan code

Data Type

boolean

Range

0=do not freeze scans, 1=freeze scans

Default

0

Change

Dynamic

Versions Affected

planned for v2

zfs_scrub_min_time_ms

Scrubs are processed by the sync thread. While scrubbing at least zfs_scrub_min_time_ms time is spent working on a scrub between txg syncs.

zfs_scrub_min_time_ms

Notes

Tags

scrub

When to change

Data Type

int

Units

milliseconds

Range

1 to (zfs_txg_timeout - 1)

Default

1,000

Change

Dynamic

Versions Affected

planned for v2

zfs_slow_io_events_per_second

zfs_slow_io_events_per_second is a rate limit for slow I/O events. Note that this should not be set below the zed thresholds (currently 10 checksums over 10 sec) or else zed may not trigger any action.

zfs_slow_io_events_per_second

Notes

Tags

vdev

When to change

TBD

Data Type

uint

Units

slow I/O events

Range

zed threshold to MAX_UINT

Default

20

Change

Dynamic

Versions Affected

planned for v2

zfs_vdev_min_ms_count

zfs_vdev_min_ms_count is the minimum number of metaslabs to create in a top-level vdev.

zfs_vdev_min_ms_count

Notes

Tags

metaslab, vdev

When to change

TBD

Data Type

int

Units

metaslabs

Range

16 to zfs_vdev_m s_count_limit

Default

16

Change

prior to creating a pool or adding a top-level vdev

Versions Affected

planned for v2

zfs_vdev_ms_count_limit

zfs_vdev_ms_count_limit is the practical upper limit for the number of metaslabs per top-level vdev.

zfs_vdev_ms_count_limit

Notes

Tags

metaslab, vdev

When to change

TBD

Data Type

int

Units

metaslabs

Range

zfs_vdev _min_ms_count to 131,072

Default

131,072

Change

prior to creating a pool or adding a top-level vdev

Versions Affected

planned for v2

spl_hostid

spl_hostid is a unique system id number. It originated in Sun’s products where most systems had a unique id assigned at the factory. This assignment does not exist in modern hardware.
In ZFS, the hostid is stored in the vdev label and can be used to determine if another system had imported the pool. When set spl_hostid can be used to uniquely identify a system. By default this value is set to zero which indicates the hostid is disabled. It can be explicitly enabled by placing a unique non-zero value in the file shown in spl_hostid_path

spl_hostid

Notes

Tags

hostid, MMP

Kernel module

spl

When to change

to uniquely identify a system when vdevs can be shared across multiple systems

Data Type

ulong

Range

0=ignore hostid, 1 to 4,294,967,295 (32-bits or 0xffffffff)

Default

0

Change

prior to importing pool

Versions Affected

v0.6.1

spl_hostid_path

spl_hostid_path is the path name for a file that can contain a unique hostid. For testing purposes, spl_hostid_path can be overridden by the ZFS_HOSTID environment variable.

spl_hostid_path

Notes

Tags

hostid, MMP

Kernel module

spl

When to change

when creating a new ZFS distribution where the default value is inappropriate

Data Type

string

Default

“/etc/hostid”

Change

read-only, can only be changed prior to spl module load

Versions Affected

v0.6.1

spl_kmem_alloc_max

Large kmem_alloc() allocations fail if they exceed KMALLOC_MAX_SIZE, as determined by the kernel source. Allocations which are marginally smaller than this limit may succeed but should still be avoided due to the expense of locating a contiguous range of free pages. Therefore, a maximum kmem size with reasonable safely margin of 4x is set. kmem_alloc() allocations larger than this maximum will quickly fail. vmem_alloc() allocations less than or equal to this value will use kmalloc(), but shift to vmalloc() when exceeding this value.

spl_kmem_alloc_max

Notes

Tags

memory

Kernel module

spl

When to change

TBD

Data Type

uint

Units

bytes

Range

TBD

Default

KMALLOC_MAX_SIZE / 4

Change

Dynamic

Versions Affected

v0.7.0

spl_kmem_alloc_warn

As a general rule kmem_alloc() allocations should be small, preferably just a few pages since they must by physically contiguous. Therefore, a rate limited warning is printed to the console for any kmem_alloc() which exceeds the threshold spl_kmem_alloc_warn

The default warning threshold is set to eight pages but capped at 32K to accommodate systems using large pages. This value was selected to be small enough to ensure the largest allocations are quickly noticed and fixed. But large enough to avoid logging any warnings when a allocation size is larger than optimal but not a serious concern. Since this value is tunable, developers are encouraged to set it lower when testing so any new largish allocations are quickly caught. These warnings may be disabled by setting the threshold to zero.

spl_kmem_alloc_warn

Notes

Tags

memory

Kernel module

spl

When to change

developers are encouraged lower when testing so any new, large allocations are quickly caught

Data Type

uint

Units

bytes

Range

0=disable the warnings,

Default

32,768 (32 KiB)

Change

Dynamic

Versions Affected

v0.7.0

spl_kmem_cache_expire

Cache expiration is part of default illumos cache behavior. The idea is that objects in magazines which have not been recently accessed should be returned to the slabs periodically. This is known as cache aging and when enabled objects will be typically returned after 15 seconds.

On the other hand Linux slabs are designed to never move objects back to the slabs unless there is memory pressure. This is possible because under Linux the cache will be notified when memory is low and objects can be released.

By default only the Linux method is enabled. It has been shown to improve responsiveness on low memory systems and not negatively impact the performance of systems with more memory. This policy may be changed by setting the spl_kmem_cache_expire bit mask as follows, both policies may be enabled concurrently.

spl_kmem_cache_expire

Notes

Tags

memory

Kernel module

spl

When to change

TBD

Data Type

bitmask

Range

0x01 - Aging (illumos), 0x02 - Low memory (Linux)

Default

0x02

Change

Dynamic

Versions Affected

v0.6.1

spl_kmem_cache_kmem_limit

Depending on the size of a memory cache object it may be backed by kmalloc() or vmalloc() memory. This is because the size of the required allocation greatly impacts the best way to allocate the memory.

When objects are small and only a small number of memory pages need to be allocated, ideally just one, then kmalloc() is very efficient. However, allocating multiple pages with kmalloc() gets increasingly expensive because the pages must be physically contiguous.

For this reason we shift to vmalloc() for slabs of large objects which which removes the need for contiguous pages. vmalloc() cannot be used in all cases because there is significant locking overhead involved. This function takes a single global lock over the entire virtual address range which serializes all allocations. Using slightly different allocation functions for small and large objects allows us to handle a wide range of object sizes.

The spl_kmem_cache_kmem_limit value is used to determine this cutoff size. One quarter of the kernel’s compiled PAGE_SIZE is used as the default value because spl_kmem_cache_obj_per_slab defaults to 16. With these default values, at most four contiguous pages are allocated.

spl_kmem_cache_kmem_limit

Notes

Tags

memory

Kernel module

spl

When to change

TBD

Data Type

uint

Units

pages

Range

TBD

Default

PAGE_SIZE / 4

Change

Dynamic

Versions Affected

v0.7.0

spl_kmem_cache_max_size

spl_kmem_cache_max_size is the maximum size of a kmem cache slab in MiB. This effectively limits the maximum cache object size to spl_kmem_cache_max_size / spl_kmem_cache_obj_per_slab Kmem caches may not be created with object sized larger than this limit.

spl_kmem_cache_max_size

Notes

Tags

memory

Kernel module

spl

When to change

TBD

Data Type

uint

Units

MiB

Range

TBD

Default

4 for 32-bit kernel, 32 for 64-bit kernel

Change

Dynamic

Versions Affected

v0.7.0

spl_kmem_cache_obj_per_slab

spl_kmem_cache_obj_per_slab is the preferred number of objects per slab in the kmem cache. In general, a larger value will increase the caches memory footprint while decreasing the time required to perform an allocation. Conversely, a smaller value will minimize the footprint and improve cache reclaim time but individual allocations may take longer.

spl_kmem_cache_obj_per_slab

Notes

Tags

memory

Kernel module

spl

When to change

TBD

Data Type

uint

Units

kmem cache objects

Range

TBD

Default

8

Change

Dynamic

Versions Affected

v0.7.0

spl_kmem_cache_obj_per_slab_min

spl_kmem_cache_obj_per_slab_min is the minimum number of objects allowed per slab. Normally slabs will contain spl_kmem_cache_obj_per_slab objects but for caches that contain very large objects it’s desirable to only have a few, or even just one, object per slab.

spl_kmem_cache_obj_per_slab_min

Notes

Tags

memory

Kernel module

spl

When to change

debugging kmem cache operations

Data Type

uint

Units

kmem cache objects

Range

TBD

Default

1

Change

Dynamic

Versions Affected

v0.7.0

spl_kmem_cache_reclaim

spl_kmem_cache_reclaim prevents Linux from being able to rapidly reclaim all the memory held by the kmem caches. This may be useful in circumstances where it’s preferable that Linux reclaim memory from some other subsystem first. Setting spl_kmem_cache_reclaim increases the likelihood out of memory events on a memory constrained system.

spl_kmem_cache_reclaim

Notes

Tags

memory

Kernel module

spl

When to change

TBD

Data Type

boolean

Range

0=enable rapid memory reclaim from kmem caches, 1=disable rapid memory reclaim from kmem caches

Default

0

Change

Dynamic

Versions Affected

v0.7.0

spl_kmem_cache_slab_limit

For small objects the Linux slab allocator should be used to make the most efficient use of the memory. However, large objects are not supported by the Linux slab allocator and therefore the SPL implementation is preferred. spl_kmem_cache_slab_limit is used to determine the cutoff between a small and large object.

Objects of spl_kmem_cache_slab_limit or smaller will be allocated using the Linux slab allocator, large objects use the SPL allocator. A cutoff of 16 KiB was determined to be optimal for architectures using 4 KiB pages.

spl_kmem_cache_slab_limit

Notes

Tags

memory

Kernel module

spl

When to change

TBD

Data Type

uint

Units

bytes

Range

TBD

Default

16,384 (16 KiB) when kernel PAGE_SIZE = 4KiB, 0 for other PAGE_SIZE values

Change

Dynamic

Versions Affected

v0.7.0

spl_max_show_tasks

spl_max_show_tasks is the limit of tasks per pending list in each taskq shown in /proc/spl/taskq and /proc/spl/taskq-all. Reading the ProcFS files walks the lists with lock held and it could cause a lock up if the list grow too large. If the list is larger than the limit, the string `”(truncated)” is printed.

spl_max_show_tasks

Notes

Tags

taskq

Kernel module

spl

When to change

TBD

Data Type

uint

Units

tasks reported

Range

0 disables the limit, 1 to MAX_UINT

Default

512

Change

Dynamic

Versions Affected

v0.7.0

spl_panic_halt

spl_panic_halt enables kernel panic upon assertion failures. When not enabled, the asserting thread is halted to facilitate further debugging.

spl_panic_halt

Notes

Tags

debug, panic

Kernel module

spl

When to change

when debugging assertions and kernel core dumps are desired

Data Type

boolean

Range

0=halt thread upon assertion, 1=panic kernel upon assertion

Default

0

Change

Dynamic

Versions Affected

v0.7.0

spl_taskq_kick

Upon writing a non-zero value to spl_taskq_kick, all taskqs are scanned. If any taskq has a pending task more than 5 seconds old, the taskq spawns more threads. This can be useful in rare deadlock situations caused by one or more taskqs not spawning a thread when it should.

spl_taskq_kick

Notes

Tags

taskq

Kernel module

spl

When to change

See description above

Data Type

uint

Units

N/A

Default

0

Change

Dynamic

Versions Affected

v0.7.0

spl_taskq_thread_bind

spl_taskq_thread_bind enables binding taskq threads to specific CPUs, distributed evenly over the available CPUs. By default, this behavior is disabled to allow the Linux scheduler the maximum flexibility to determine where a thread should run.

spl_taskq_thread_bind

Notes

Tags

CPU, taskq

Kernel module

spl

When to change

when debugging CPU scheduling options

Data Type

boolean

Range

0=taskqs are not bound to specific CPUs, 1=taskqs are bound to CPUs

Default

0

Change

prior to loading spl kernel module

Versions Affected

v0.7.0

spl_taskq_thread_dynamic

spl_taskq_thread_dynamic enables taskqs to set the TASKQ_DYNAMIC flag will by default create only a single thread. New threads will be created on demand up to a maximum allowed number to facilitate the completion of outstanding tasks. Threads which are no longer needed are promptly destroyed. By default this behavior is enabled but it can be d.

See also zfs_zil_clean_taskq_nthr_pct, zio_taskq_batch_pct

spl_taskq_thread_dynamic

Notes

Tags

taskq

Kernel module

spl

When to change

disable for performance analysis or troubleshooting

Data Type

boolean

Range

0=taskq threads are not dynamic, 1=taskq threads are dynamically created and destroyed

Default

1

Change

prior to loading spl kernel module

Versions Affected

v0.7.0

spl_taskq_thread_priority

spl_taskq_thread_priority allows newly created taskq threads to set a non-default scheduler priority. When enabled the priority specified when a taskq is created will be applied to all threads created by that taskq.
When disabled all threads will use the default Linux kernel thread priority.

spl_taskq_thread_priority

Notes

Tags

CPU, taskq

Kernel module

spl

When to change

when troubleshooting CPU scheduling-related performance issues

Data Type

boolean

Range

0=taskq threads use the default Linux kernel thread priority, 1=

Default

1

Change

prior to loading spl kernel module

Versions Affected

v0.7.0

spl_taskq_thread_sequential

spl_taskq_thread_sequential is the number of items a taskq worker thread must handle without interruption before requesting a new worker thread be spawned. spl_taskq_thread_sequential controls how quickly taskqs ramp up the number of threads processing the queue. Because Linux thread creation and destruction are relatively inexpensive a small default value has been selected. Thus threads are created aggressively, which is typically desirable. Increasing this value results in a slower thread creation rate which may be preferable for some configurations.

spl_taskq_thread_sequential

Notes

Tags

CPU, taskq

Kernel module

spl

When to change

TBD

Data Type

int

Units

taskq items

Range

1 to MAX_INT

Default

4

Change

Dynamic

Versions Affected

v0.7.0

spl_kmem_cache_kmem_threads

spl_kmem_cache_kmem_threads shows the current number of spl_kmem_cache threads. This task queue is responsible for allocating new slabs for use by the kmem caches. For the majority of systems and workloads only a small number of threads are required.

spl_kmem_cache_kmem_threads

Notes

Tags

CPU, memory

Kernel module

spl

When to change

read-only

Data Type

int

Range

1 to MAX_INT

Units

threads

Default

4

Change

read-only, can only be changed prior to spl module load

Versions Affected

v0.7.0

spl_kmem_cache_magazine_size

spl_kmem_cache_magazine_size shows the current . Cache magazines are an optimization designed to minimize the cost of allocating memory. They do this by keeping a per-cpu cache of recently freed objects, which can then be reallocated without taking a lock. This can improve performance on highly contended caches. However, because objects in magazines will prevent otherwise empty slabs from being immediately released this may not be ideal for low memory machines.

For this reason spl_kmem_cache_magazine_size can be used to set a maximum magazine size. When this value is set to 0 the magazine size will be automatically determined based on the object size. Otherwise magazines will be limited to 2-256 objects per magazine (eg per CPU). Magazines cannot be disabled entirely in this implementation.

spl_kmem_cache_magazine_size

Notes

Tags

CPU, memory

Kernel module

spl

When to change

Data Type

int

Units

threads

Range

0=automatically scale magazine size, otherwise 2 to 256

Default

0

Change

read-only, can only be changed prior to spl module load

Versions Affected

v0.7.0