OpenZFS Patches

The ZFS on Linux project is an adaptation of the upstream OpenZFS repository designed to work in a Linux environment. This upstream repository acts as a location where new features, bug fixes, and performance improvements from all the OpenZFS platforms can be integrated. Each platform is responsible for tracking the OpenZFS repository and merging the relevant improvements back in to their release.

For the ZFS on Linux project this tracking is managed through an OpenZFS tracking page. The page is updated regularly and shows a list of OpenZFS commits and their status in regard to the ZFS on Linux master branch.

This page describes the process of applying outstanding OpenZFS commits to ZFS on Linux and submitting those changes for inclusion. As a developer this is a great way to familiarize yourself with ZFS on Linux and to begin quickly making a valuable contribution to the project. The following guide assumes you have a github account, are familiar with git, and are used to developing in a Linux environment.

Porting OpenZFS changes to ZFS on Linux

Setup the Environment

Clone the source. Start by making a local clone of the spl and zfs repositories.

$ git clone -o zfsonlinux https://github.com/zfsonlinux/spl.git
$ git clone -o zfsonlinux https://github.com/zfsonlinux/zfs.git

Add remote repositories. Using the GitHub web interface fork the zfs repository in to your personal GitHub account. Add your new zfs fork and the openzfs repository as remotes and then fetch both repositories. The OpenZFS repository is large and the initial fetch may take some time over a slow connection.

$ cd zfs
$ git remote add <your-github-account> git@github.com:<your-github-account>/zfs.git
$ git remote add openzfs https://github.com/openzfs/openzfs.git
$ git fetch --all

Build the source. Compile the spl and zfs master branches. These branches are always kept stable and this is a useful verification that you have a full build environment installed and all the required dependencies are available. This may also speed up the compile time latter for small patches where incremental builds are an option.

$ cd ../spl
$ sh autogen.sh && ./configure --enable-debug && make -s -j$(nproc)
$
$ cd ../zfs
$ sh autogen.sh && ./configure --enable-debug && make -s -j$(nproc)

Pick a patch

Consult the OpenZFS tracking page and select a patch which has not yet been applied. For your first patch you will want to select a small patch to familiarize yourself with the process.

Porting a Patch

There are 2 methods:

Please read about manual merge first to learn the whole process.

Cherry-pick

You can start to cherry-pick by your own, but we have made a special script, which tries to cherry-pick the patch automatically and generates the description.

  1. Prepare environment:

Mandatory git settings (add to ~/.gitconfig):

[merge]
    renameLimit = 999999
[user]
    email = mail@yourmail.com
    name = Your Name

Download the script:

wget https://raw.githubusercontent.com/zfsonlinux/zfs-buildbot/master/scripts/openzfs-merge.sh

  1. Run:

./openzfs-merge.sh -d path_to_zfs_folder -c openzfs_commit_hash

This command will fetch all repositories, create a new branch autoport-ozXXXX (XXXX - OpenZFS issue number), try to cherry-pick, compile and check cstyle on success.

If it succeeds without any merge conflicts - go to autoport-ozXXXX branch, it will have ready to pull commit. Congratulations, you can go to step 7!

Otherwise you should go to step 2.

  1. Resolve all merge conflicts manually. Easy method - install Meld or any other diff tool and run git mergetool.

  2. Check all compile and cstyle errors (See Testing a patch).

  3. Commit your changes with any description.

  4. Update commit description (last commit will be changed):

./openzfs-merge.sh -d path_to_zfs_folder -g openzfs_commit_hash

  1. Add any porting notes (if you have modified something): git commit --amend

  2. Push your commit to github: git push <your-github-account> autoport-ozXXXX

  3. Create a pull request to ZoL master branch.

  4. Go to Testing a patch section.

Manual merge

Create a new branch. It is important to create a new branch for every commit you port to ZFS on Linux. This will allow you to easily submit your work as a GitHub pull request and it makes it possible to work on multiple OpenZFS changes concurrently. All development branches need to be based off of the ZFS master branch and it’s helpful to name the branches after the issue number you’re working on.

$ git checkout -b openzfs-<issue-nr> master

Generate a patch. One of the first things you’ll notice about the ZFS on Linux repository is that it is laid out differently than the OpenZFS repository. Organizationally it is much flatter, this is possible because it only contains the code for OpenZFS not an entire OS. That means that in order to apply a patch from OpenZFS the path names in the patch must be changed. A script called zfs2zol-patch.sed has been provided to perform this translation. Use the git format-patch command and this script to generate a patch.

$ git format-patch --stdout <commit-hash>^..<commit-hash> | \
    ./scripts/zfs2zol-patch.sed >openzfs-<issue-nr>.diff

Apply the patch. In many cases the generated patch will apply cleanly to the repository. However, it’s important to keep in mind the zfs2zol-patch.sed script only translates the paths. There are often additional reasons why a patch might not apply. In some cases hunks of the patch may not be applicable to Linux and should be dropped. In other cases a patch may depend on other changes which must be applied first. The changes may also conflict with Linux specific modifications. In all of these cases the patch will need to be manually modified to apply cleanly while preserving the its original intent.

$ git am ./openzfs-<commit-nr>.diff

Update the commit message. By using git format-patch to generate the patch and then git am to apply it the original comment and authorship will be preserved. However, due to the formatting of the OpenZFS commit you will likely find that the entire commit comment has been squashed in to the subject line. Use git commit --amend to cleanup the comment and be careful to follow these standard guidelines.

The summary line of an OpenZFS commit is often very long and you should truncate it to 50 characters. This is useful because it preserves the correct formatting of git log --pretty=oneline command. Make sure to leave a blank line between the summary and body of the commit. Then include the full OpenZFS commit message wrapping any lines which exceed 72 characters. Finally, add a Ported-by tag with your contact information and both a OpenZFS-issue and OpenZFS-commit tag with appropriate links. You’ll want to verify your commit contains all of the following information:

  • The subject line from the original OpenZFS patch in the form: “OpenZFS <issue-nr> - short description”.

  • The original patch authorship should be preserved.

  • The OpenZFS commit message.

  • The following tags:

    • Authored by: Original patch author

    • Reviewed by: All OpenZFS reviewers from the original patch.

    • Approved by: All OpenZFS reviewers from the original patch.

    • Ported-by: Your name and email address.

    • OpenZFS-issue: https ://www.illumos.org/issues/issue

    • OpenZFS-commit: https ://github.com/openzfs/openzfs/commit/hash

  • Porting Notes: An optional section describing any changes required when porting.

For example, OpenZFS issue 6873 was applied to Linux from this upstream OpenZFS commit.

OpenZFS 6873 - zfs_destroy_snaps_nvl leaks errlist

Authored by: Chris Williamson <chris.williamson@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Paul Dagnelie <pcd@delphix.com>
Ported-by: Denys Rtveliashvili <denys@rtveliashvili.name>

lzc_destroy_snaps() returns an nvlist in errlist.
zfs_destroy_snaps_nvl() should nvlist_free() it before returning.

OpenZFS-issue: https://www.illumos.org/issues/6873
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/ee06391

Testing a Patch

Build the source. Verify the patched source compiles without errors and all warnings are resolved.

$ make -s -j$(nproc)

Run the style checker. Verify the patched source passes the style checker, the command should return without printing any output.

$ make cstyle

Open a Pull Request. When your patch builds cleanly and passes the style checks open a new pull request. The pull request will be queued for automated testing. As part of the testing the change is built for a wide range of Linux distributions and a battery of functional and stress tests are run to detect regressions.

$ git push <your-github-account> openzfs-<issue-nr>

Fix any issues. Testing takes approximately 2 hours to fully complete and the results are posted in the GitHub pull request. All the tests are expected to pass and you should investigate and resolve any test failures. The test scripts are all available and designed to run locally in order reproduce an issue. Once you’ve resolved the issue force update the pull request to trigger a new round of testing. Iterate until all the tests are passing.

# Fix issue, amend commit, force update branch.
$ git commit --amend
$ git push --force <your-github-account> openzfs-<issue-nr>

Merging the Patch

Review. Lastly one of the ZFS on Linux maintainers will make a final review of the patch and may request additional changes. Once the maintainer is happy with the final version of the patch they will add their signed-off-by, merge it to the master branch, mark it complete on the tracking page, and thank you for your contribution to the project!

Porting ZFS on Linux changes to OpenZFS

Often an issue will be first fixed in ZFS on Linux or a new feature developed. Changes which are not Linux specific should be submitted upstream to the OpenZFS GitHub repository for review. The process for this is described in the OpenZFS README.