What the ZFS ARC does with your RAM, and what a cap costs

For three and a half years I ran ZFS on my homelab server without ever asking what its cache was allowed to do. When I finally looked, the answer was: almost all of the RAM. This post is what I learned about the ARC since then, with ten days of data from before and after I capped it.

The machine is small: an i5-6500T with 4 cores, 31 GiB of RAM, and one Samsung 850 EVO that holds the root filesystem, the ZFS pool and the swap. Around seventy containers run on it, each with its own memory limit.

What made me look was an outage. On September 21 Traefik went from 85 to 371 MB in less than two minutes and hit its container limit of 384 MB, and with the proxy dead every web service was unreachable. That kill came from Traefik’s own cgroup limit; the ARC had nothing to do with it. But it was the afternoon I went through the memory of the whole box, and the ARC was the biggest surprise.

What the ARC is

The ARC, Adaptive Replacement Cache, is the read cache ZFS keeps in RAM. “Adaptive” because it tracks two kinds of blocks, the ones read recently and the ones read often, and keeps moving the boundary between them depending on which kind pays off.

Three things about it I did not know:

• It lives outside the kernel’s page cache. ZFS manages that memory itself. The kernel can ask for it back under memory pressure, and the ARC then shrinks its target in small steps: by default each step takes off about 0.8% of its size.
• Its default limit is huge. With zfs_arc_max left at 0, OpenZFS uses the larger of RAM minus 1 GiB and 5/8 of RAM. On my 31 GiB that was an arc_c_max of 30.24 GiB, 97% of the RAM.
• Lowering the limit at runtime does not shrink it. The man page says it plainly: reducing zfs_arc_max below the current size “will not cause the ARC to shrink without memory pressure”.

Why it matters on this box

Seventy containers share 31 GiB, and each has a memory limit that covers its own processes. The ARC is not inside any of those limits. When several containers need memory at the same time, the kernel first has to win it back from a cache that returns it slowly, and until then it swaps, on the same SSD that holds everything else, or kills something.

Most of the time the ARC was harmless. Its median size was 8.4 GiB. But its peaks reached 15 GiB, and peaks are exactly when the containers need memory too.

The cap

The limit is a module option, so it survives a reboot. Because changing it at runtime does not shrink the cache, the Ansible role also writes the new value to /sys/module/zfs/parameters/zfs_arc_max, and the cache gives memory back the next time the kernel asks:

options zfs zfs_arc_max={{ zfs_arc_max_bytes }}

zfs_arc_max_bytes is 8589934592: 8 GiB, just under the old median and about a quarter of the RAM.

What the cap changed

I compared September 11 to 20 with September 22 to October 1, skipping the day of the change. The numbers come from node-exporter’s ZFS metrics, exported from Prometheus before its 30-day retention dropped them:

The cap did what I wanted: the peaks dropped by about 4 GiB, and the typical size did not move.

It also has a price, and I did not expect it to be this visible. The workload read the same amount, about 22,000 hits a second, but misses went from 20 to almost 300 a second. A hit ratio of 98.67% sounds fine; it means fifteen times more reads going to disk. Reads do not wear an SSD the way writes do, and I did not notice any difference in daily use, so for now I keep the cap. It is a trade, not a free fix.

The ARC still goes over its limit

This one I have not solved. After the cap, the ARC’s target (arc_c) stays between 6.3 and 8 GiB, as it should. Its real size does not: it sits above 8 GiB in almost every sample, and peaked at 11 GiB.

According to the man page the ARC starts evicting as soon as it is 0.2% over its target, and blocks new allocations at 0.6%. Sitting above 8 GiB for days, up to 37% over, is something else. My guess is memory the ARC cannot evict at that moment, such as metadata for files that are open. I have not checked it yet: the next step is a breakdown with arc_summary, and I would rather report a guess as a guess than dress it up.

How I measured

Prometheus keeps 30 days, so three weeks after the change I exported every series I needed through its HTTP API (/api/v1/query_range, one sample every five minutes) into gzipped JSON. A small Python script compares the two windows. Gauges get an average and percentiles. Counters, like hits and misses, get an average rate, and a drop between two samples counts as a counter reset:

def counter_rate(points, period):
    """Average per-second increase, skipping counter resets."""
    pts = window(points, period)
    total = sum(max(b - a, 0) for (_, a), (_, b) in zip(pts, pts[1:]))
    span = pts[-1][0] - pts[0][0] if len(pts) > 1 else float("nan")
    return total / span

The rest of that afternoon

The ARC was one of ten changes I made on the same day. That was a mistake for the measurement: I wanted an aggressive plan, and I got data that cannot separate most of them. Briefly:

• TRIM clearly works. fstrim never reached the pool, because ZFS does not implement the FITRIM ioctl it uses, and the SSD had gone untrimmed for three and a half years. With autotrim=on and a monthly zpool trim, discards went from 5.5 to 140 GB a day.
• The performance governor raised the average clock from 2.30 to 2.84 GHz. I did not feel it.
• I/O stalls dropped from 5.3% to 3.0% of the time. The scheduler, TRIM, atime and recordsize all touch the same disk, so I cannot say which one did it.
• Writes to the SSD stayed at about 280 GB a day: turning off atime and capping the journal did nothing visible. What writes that much is a question for another post.

Load and swap went up in the same days, for a reason unrelated to this tuning. That is another post too.

What I would do differently

Next time I change one thing, wait for a few days of data, check it, and only then move to the next. And I will read the defaults of anything that holds on to memory before I run it for three years.