-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

=============================================================================
FreeBSD-EN-24:09.zfs                                            Errata Notice
                                                          The FreeBSD Project

Topic:          High CPU usage by kernel threads related to ZFS

Category:       contrib
Module:         zfs
Announced:      2024-04-24
Affects:        FreeBSD 13.3
Corrected:      2024-04-12 13:00:11 UTC (stable/13, 13-STABLE)
                2024-04-24 20:21:10 UTC (releng/13.3, 13.3-RELEASE-p2)

For general information regarding FreeBSD Errata Notices and Security
Advisories, including descriptions of the fields above, security
branches, and the following sections, please visit
<URL:https://security.FreeBSD.org/>.

I.   Background

ZFS is an advanced and scalable file system originally developed by Sun
Microsystems for its Solaris operating system. ZFS was integrated as part of
the FreeBSD starting with FreeBSD 7.0, and it has since become a prominent
and preferred choice for storage management.

II.  Problem Description

Because ZFS may consume large amounts of RAM to cache various types of
filesystem objects, it continuously monitors system RAM available to decide
whether to shrink its caches.  Some caches are shrunk using a dedicated
thread, to which work is dispatched asynchronously.

In some cases, the cache shrinking logic may dispatch excessive amounts of
work to the "ARC pruning" thread, causing it to continue attempting to shrink
caches even after resource shortages are resolved.

III. Impact

The bug manifests as a kernel thread, "arc_prune", consuming 100% of a CPU core
for indefinite periods, even while the system is otherwise idle.  This behavior
also impacts workloads running on the system, by reducing available CPU
resources and by triggering lock contention in the kernel, in particular with
the "vnlru" process whose function is to recycle vnodes (structures representing
files, whether opened or cached), a mechanism frequently triggered by intensive
filesystem workloads.

IV.  Workaround

No workaround is available.  Systems not using ZFS are unaffected.

V.   Solution

Upgrade your system to a supported FreeBSD stable or release / security branch
(releng) dated after the correction date.  A reboot is required following the
upgrade.

Perform one of the following:

1) To update your system via a binary patch:

Systems running a RELEASE version of FreeBSD on the amd64 or arm64 platforms,
or the i386 platform on FreeBSD 13 and earlier, can be updated via
the freebsd-update(8) utility:

# freebsd-update fetch
# freebsd-update install
# reboot

2) To update your system via a source code patch:

The following patches have been verified to apply to the applicable
FreeBSD release branches.

a) Download the relevant patch from the location below, and verify the
detached PGP signature using your PGP utility.

# fetch https://security.FreeBSD.org/patches/EN-24:09/zfs.patch
# fetch https://security.FreeBSD.org/patches/EN-24:09/zfs.patch.asc
# gpg --verify zfs.patch.asc

b) Apply the patch.  Execute the following commands as root:

# cd /usr/src
# patch < /path/to/patch

c) Recompile your kernel as described in
<URL:https://www.FreeBSD.org/handbook/kernelconfig.html> and reboot the
system.

VI.  Correction details

This issue is corrected as of the corresponding Git commit hash or Subversion
revision number in the following stable and release branches:

Branch/path                             Hash                     Revision
- -------------------------------------------------------------------------
stable/13/                              330954bdb822    stable/13-n257698
releng/13.3/                            266b3bd3f26d  releng/13.3-n257432
- -------------------------------------------------------------------------

Run the following command to see which files were modified by a
particular commit:

# git show --stat <commit hash>

Or visit the following URL, replacing NNNNNN with the hash:

<URL:https://cgit.freebsd.org/src/commit/?id=NNNNNN>

To determine the commit count in a working tree (for comparison against
nNNNNNN in the table above), run:

# git rev-list --count --first-parent HEAD

VII. References

See problem reports
<URL:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=274698> and
<URL:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=275594>.

See also the previous, similar errata notice issued for FreeBSD 14.0:
<URL:https://security.FreeBSD.org/advisories/FreeBSD-EN-23:18.openzfs.asc>.

The latest revision of this advisory is available at
<URL:https://security.FreeBSD.org/advisories/FreeBSD-EN-24:09.zfs.asc>
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEthUnfoEIffdcgYM7bljekB8AGu8FAmYpapQACgkQbljekB8A
Gu8gBxAAiuUNqeGaKNQ1XbV0kSucwnae5uOrQmthHQBY98PJJKUZpm1RTt/FnBB7
qPxEY5vFRcGgZ43GVlnmfmH/EmqOg6WPpsgKfdq1XTy/ERU815JOsD+wKUWa/9Ia
g67pnl8HPMSF5eZ1FreWfzNsWmxakiDLg2VXtFx7x3+qocifD/WwGvDTjdDBzzyK
+cIrBqvTlbOCRdHzl49wmNLz46ha5bmxTb7MzXB3jIQ1v+PZ71biyQxBZTrZgR6S
La8oVe4Kj2lJTJw5S2xvsoyo5PzqmPCyD1m22fzgKTyaAUCXiioUUQDuFTxu9rhW
I3lSvqdIRw28yRFjGslxlq9x1vShQTw3ILcH31ucxKUNow7hlDz4Ow2NzqXhSjxN
RMGamxLTA5BcNCR4/DexAjfeh6OKnCG7n0ntlhxI0LWGr4ceT3/ySck7xhCNCSm1
Ze/Gf9/j4+zR2jyauRANkITPkVHUV79/Sgjn1IlcMDLpzegH+QfQsX6CosG5uSWS
UlpK2hhCv2g3lE7XuBItz7E/8i5Nx9RZgnh047Nj3ZB/6dCauAeUYKnY5X3xJa5X
OKJWIGyJAyrCoFIg+LdBS47ggg8wswyyb1XBF2rZgZNqVmzZrJd7lBV/sjDaEC1H
13lHhIIwtpTagDAT1Nbji++IT+2DatjhLZnMQwvALno0tIE19mg=
=IgLQ
-----END PGP SIGNATURE-----