devncode

What the ZFS ARC does with your RAM, and what a cap costs

Tue, 13 Oct 2026 00:00:00 GMT

For three and a half years I ran ZFS on my homelab server without ever asking what its cache was allowed to do. When I finally looked, the answer was: almost all of the RAM. This post is what I learned about the ARC since then, with ten days of data from before and after I capped it.

Achtung If you never set zfs_arc_max, current OpenZFS on Linux lets the ARC grow to the larger of "all RAM minus 1 GiB" and "5/8 of RAM". On a box that also runs containers, pick that number yourself.

The machine is small: an i5-6500T with 4 cores, 31 GiB of RAM, and one Samsung 850 EVO that holds the root filesystem, the ZFS pool and the swap. Around seventy containers run on it, each with its own memory limit.

What made me look was an outage. On September 21 Traefik went from 85 to 371 MB in less than two minutes and hit its container limit of 384 MB, and with the proxy dead every web service was unreachable. That kill came from Traefik's own cgroup limit; the ARC had nothing to do with it. But it was the afternoon I went through the memory of the whole box, and the ARC was the biggest surprise.

What the ARC is

The ARC, Adaptive Replacement Cache, is the read cache ZFS keeps in RAM. "Adaptive" because it tracks two kinds of blocks, the ones read recently and the ones read often, and keeps moving the boundary between them depending on which kind pays off.

Three things about it I did not know:

It lives outside the kernel's page cache. ZFS manages that memory itself. The kernel can ask for it back under memory pressure, and the ARC then shrinks its target in small steps: by default each step takes off about 0.8% of its size.
Its default limit is huge. With zfs_arc_max left at 0, OpenZFS uses the larger of RAM minus 1 GiB and 5/8 of RAM. On my 31 GiB that was an arc_c_max of 30.24 GiB, 97% of the RAM.
Lowering the limit at runtime does not shrink it. The man page says it plainly: reducing zfs_arc_max below the current size "will not cause the ARC to shrink without memory pressure".

Why it matters on this box

Seventy containers share 31 GiB, and each has a memory limit that covers its own processes. The ARC is not inside any of those limits. When several containers need memory at the same time, the kernel first has to win it back from a cache that returns it slowly, and until then it swaps, on the same SSD that holds everything else, or kills something.

Most of the time the ARC was harmless. Its median size was 8.4 GiB. But its peaks reached 15 GiB, and peaks are exactly when the containers need memory too.

The cap

The limit is a module option, so it survives a reboot. Because changing it at runtime does not shrink the cache, the Ansible role also writes the new value to /sys/module/zfs/parameters/zfs_arc_max, and the cache gives memory back the next time the kernel asks:

options zfs zfs_arc_max={{ zfs_arc_max_bytes }}

zfs_arc_max_bytes is 8589934592: 8 GiB, just under the old median and about a quarter of the RAM.

What the cap changed

I compared September 11 to 20 with September 22 to October 1, skipping the day of the change. The numbers come from node-exporter's ZFS metrics, exported from Prometheus before its 30-day retention dropped them:

ARC	Before the cap	After the cap
Median size	8.44 GiB	8.81 GiB
99th percentile	13.86 GiB	10.18 GiB
Maximum	14.99 GiB	10.96 GiB
Hits per second	21,464	22,062
Misses per second	19.6	298
Hit ratio	99.91%	98.67%

The cap did what I wanted: the peaks dropped by about 4 GiB, and the typical size did not move.

It also has a price, and I did not expect it to be this visible. The workload read the same amount, about 22,000 hits a second, but misses went from 20 to almost 300 a second. A hit ratio of 98.67% sounds fine; it means fifteen times more reads going to disk. Reads do not wear an SSD the way writes do, and I did not notice any difference in daily use, so for now I keep the cap. It is a trade, not a free fix.

The ARC still goes over its limit

This one I have not solved. After the cap, the ARC's target (arc_c) stays between 6.3 and 8 GiB, as it should. Its real size does not: it sits above 8 GiB in almost every sample, and peaked at 11 GiB.

According to the man page the ARC starts evicting as soon as it is 0.2% over its target, and blocks new allocations at 0.6%. Sitting above 8 GiB for days, up to 37% over, is something else. My guess is memory the ARC cannot evict at that moment, such as metadata for files that are open. I have not checked it yet: the next step is a breakdown with arc_summary, and I would rather report a guess as a guess than dress it up.

How I measured

Prometheus keeps 30 days, so three weeks after the change I exported every series I needed through its HTTP API (/api/v1/query_range, one sample every five minutes) into gzipped JSON. A small Python script compares the two windows. Gauges get an average and percentiles. Counters, like hits and misses, get an average rate, and a drop between two samples counts as a counter reset:

def counter_rate(points, period):
    """Average per-second increase, skipping counter resets."""
    pts = window(points, period)
    total = sum(max(b - a, 0) for (_, a), (_, b) in zip(pts, pts[1:]))
    span = pts[-1][0] - pts[0][0] if len(pts) > 1 else float("nan")
    return total / span

The rest of that afternoon

The ARC was one of ten changes I made on the same day. That was a mistake for the measurement: I wanted an aggressive plan, and I got data that cannot separate most of them. Briefly:

TRIM clearly works. fstrim never reached the pool, because ZFS does not implement the FITRIM ioctl it uses, and the SSD had gone untrimmed for three and a half years. With autotrim=on and a monthly zpool trim, discards went from 5.5 to 140 GB a day.
The performance governor raised the average clock from 2.30 to 2.84 GHz. I did not feel it.
I/O stalls dropped from 5.3% to 3.0% of the time. The scheduler, TRIM, atime and recordsize all touch the same disk, so I cannot say which one did it.
Writes to the SSD stayed at about 280 GB a day: turning off atime and capping the journal did nothing visible. What writes that much is a question for another post.

Load and swap went up in the same days, for a reason unrelated to this tuning. That is another post too.

What I would do differently

Next time I change one thing, wait for a few days of data, check it, and only then move to the next. And I will read the defaults of anything that holds on to memory before I run it for three years.

Time Machine backups on a Samba share in Docker

Mon, 12 Oct 2026 00:00:00 GMT

I found out my Time Machine backups were broken right around my upgrade to macOS 27: every attempt was failing. That one is on me. I set the backup up in April, saw the first one complete, and never tested it again.

My homelab server is also my NAS, so the Mac backs up to it over the LAN. Samba runs in Docker on that server and exports a Time Machine share that lives on a USB disk. This post covers the share, what it took for the Mac to find it, the two faults that stopped it in September, and the alerts I run now.

Achtung fruit:time machine max size must leave room for a full backup, not only for the next incremental. Mine did not, and the day macOS decided it needed a full copy, the copy never fit.

The share

The image is ghcr.io/servercontainers/samba. It runs Samba, Avahi and wsdd2 in one container and builds smb.conf from environment variables. A Time Machine share needs the fruit VFS module, which speaks Apple's SMB extensions, stacked with streams_xattr, plus fruit:time machine = yes to advertise it as a backup target.

These are the lines of the compose file that matter for Time Machine:

services:
  samba:
    image: ghcr.io/servercontainers/samba:latest
    environment:
      MODEL: "TimeCapsule"
      SAMBA_GLOBAL_CONFIG_interfaces: "wan0"
      SAMBA_GLOBAL_CONFIG_bind_SPACE_interfaces_SPACE_only: "yes"
      SAMBA_VOLUME_CONFIG_timemachine: |
        [TimeMachine]
          path = /shares/timemachine
          read only = no
          valid users = {{ samba_user }}
          vfs objects = fruit streams_xattr
          fruit:time machine = yes
          fruit:time machine max size = {{ samba_timemachine_max_size }}
    cap_add:
      - NET_ADMIN
      - NET_RAW
    volumes:
      - "{{ samba_timemachine_path }}:/shares/timemachine"
      - "{{ samba_config_dir }}/avahi-daemon.conf:/etc/avahi/avahi-daemon.conf:ro"
    network_mode: host

Two details of the image cost me a fix each. Option names with a space are written with _SPACE_: my first version used double underscores, and the day after I corrected them. And network_mode: host is not optional: Avahi and wsdd2 announce the share with multicast on the LAN, and from a bridged container nobody hears them. MODEL is only the model Avahi announces, which decides the icon Finder shows.

Making the Mac find it

Samba was up on March 30, but the Mac listed the share only on April 18. Three problems, in the order I hit them:

Avahi was off. The image disables it when AVAHI_DISABLE is set, whatever the value. I had the variable in the compose file, so mDNS never started, and without mDNS the share never shows up in Time Machine's list of disks.
The dbus socket. To get Avahi running I bind-mounted the host's /var/run/dbus into the container. Avahi started, the Mac found the share on April 18, and the first full backup ran that same day. Five days later something else on the host broke: hostnamectl failed, and so did the Ansible tasks that go through dbus. Sharing that directory had let the container delete the host's system bus socket.
Docker interfaces. With host networking Avahi sees every veth interface Docker creates and removes. Every new container brought a new interface, Avahi reconfigured itself each time, and the announcement on the LAN never settled.

The fix for the last two is a small avahi-daemon.conf, mounted over the image's own:

[server]
use-ipv4=yes
use-ipv6=yes
allow-interfaces={{ router_lan_nic | default('wan0') }}
enable-dbus=no
ratelimit-interval-usec=1000000
ratelimit-burst=1000

[publish]
publish-hinfo=no
publish-workstation=no

allow-interfaces pins Avahi to the LAN interface, enable-dbus=no lets it run without the host's dbus. The two capabilities in the compose file belong to wsdd2, the Windows discovery daemon: without NET_ADMIN and NET_RAW it cannot set up its multicast sockets.

What broke in September

Date	What happened
2026-04-18	First full backup. macOS verifies the new bundle.
April to September	Hourly incrementals succeed.
2026-09-14, 04:03 UTC	Last successful backup.
2026-09-15	I notice that every backup fails.

backupd stayed in PreparingSourceVolumes: no CPU use, nothing in the logs. Two separate problems were behind it.

The verification came due. macOS re-verifies network backups now and then. The last verification of my bundle dated from April 18, almost five months earlier, and the bundle held 720G of history. Running that check over SMB, on a USB disk with shingled (SMR) recording, never finished, and no backup could start until it did.

The full copy did not fit. The bundle held 720 GiB and the cap was 1 TiB, which left the Mac about 326 GB of free space (macOS counts in decimal units). The full copy backupd had planned was 413 GB. Instead of copying, it kept deleting old snapshots to make room, at about 700 KB/s on that disk, and the copy never started.

One wrong turn is worth telling. When I wrote the monitoring, I first read VerificationState = 1 in the bundle's MachineID.plist as "this bundle is damaged" and planned to alert on it. Then the collector ran on the real share, and the bundle created that morning reported 1 too. What separates a healthy bundle from mine is the date of the last verification, not the flag. The check on the empty replacement bundle finished in 25 seconds. So the old bundle was intact. At 720G, verifying it over SMB on this disk never finished.

Raising the cap

The cap does not live in the bundle. Samba uses it to report a smaller disk to the Mac (the vfs_fruit man page says it "limits the reported disksize"), so changing it needs no work on the bundle: the Mac reads the new size the next time it mounts the share.

samba_timemachine_max_size: 1T
samba_timemachine_max_size: 2T

I stopped at 2T because of the disk: it had 1.9T free with the bundle already taking 720G, so 2T is a cap the disk can actually honour. The man page also warns that the cap is an estimate built from the bundle's band files only. Anything else stored on the same disk is invisible to it, so keep the cap below the space the disk really has.

I deleted the old bundle and let the Mac start a new one.

Alerts that would have caught it

Everything else on the server had monitoring. The Time Machine share had none. Now a systemd timer runs a Python collector every five minutes:

[Unit]
Description=Refresh Time Machine backup metrics every 5 minutes

[Timer]
OnBootSec=2min
OnUnitActiveSec=5min
AccuracySec=30s

[Install]
WantedBy=timers.target

The collector reads the plists inside each sparsebundle and writes a file for node-exporter's textfile collector. The completion dates of the snapshots sit at different key paths depending on the macOS release, so it walks the whole plist instead of trusting one path:

def collect_dates(node, out):
    if isinstance(node, dict):
        for key, value in node.items():
            if key.endswith("SnapshotCompletionDate") and hasattr(value, "timestamp"):
                out.append(value.timestamp())
            else:
                collect_dates(value, out)
    elif isinstance(node, list):
        for item in node:
            collect_dates(item, out)

The bundle size comes from st_blocks, not st_size: band files are sparse, and their size would overcount.

One trap in this script. Ansible renders it through Jinja before copying it to the server, to fill in the share path and the cap. Two opening braces in a row, anywhere in the file and even in a comment, start a Jinja expression and break the deploy, and doubled braces are what str.format uses to print a literal brace. The script sticks to %-formatting, and a test rejects any {{ other than the variables Ansible is meant to fill.

Prometheus turns the metrics into six alerts, each aimed at one of the ways this went wrong:

Alert	Fires when	What it would have caught
`TimeMachineBackupMissed`	no completed snapshot for 48 hours	the September outage, without me finding it by hand
`TimeMachineNeverCompleted`	a bundle holds no snapshot for 24 hours	a first backup that never finishes
`TimeMachineVerificationOverdue`	the last verification is older than 90 days	the real cause, about two months before it stalled anything
`TimeMachineBundleNearCap`	the bundle passes 70% of the cap	the 720G-of-1T situation
`TimeMachineNoBundle`	the share holds no bundle at all	an empty share
`TimeMachineMetricsStale`	the collector's last run is over 30 minutes old, for another 30 minutes	a collector that stopped, leaving numbers that look fine forever

The last one matters more than it looks. The collector replaces its file atomically, and node-exporter keeps serving the last file it finds, so a stopped timer makes nothing disappear: the numbers freeze. The rules built on time() still fire on a frozen timestamp, but a frozen size or snapshot count looks healthy forever. The freshness check catches that case, and absent() covers a file that was never written:

- alert: TimeMachineVerificationOverdue
  expr: time() - timemachine_verification_timestamp_seconds > 7776000
  for: 12h
  labels:
    severity: warning

- alert: TimeMachineMetricsStale
  expr: absent(timemachine_collector_last_run_timestamp_seconds) or time() - timemachine_collector_last_run_timestamp_seconds > 1800
  for: 30m
  labels:
    severity: warning

The test I skipped in April

I recently formatted the Mac and restored it from this share, and it went smoothly. That restore is the test I should have run in April, right after the first backup completed. The alerts tell me when backups stop. Only a restore tells me they are usable.

Structuring an Ansible repo for a one-server homelab

Sun, 11 Oct 2026 00:00:00 GMT

The server from part 1 runs about a hundred containers, and one Ansible repo describes all of them. I picked Ansible because I have used it for years and I have always been happy with it. This part is about the shape of that repo: where a service lives, where its values live, and how one bad loading order overrode production for months without anyone noticing.

One role per service

Every service is a role, and every role has its own tag in the playbook. That is the whole reason for the layout: a deploy can touch one service and nothing else. A role is small and always looks the same:

roles/<service>/
  tasks/main.yml           create the directories, render the files, compose up
  templates/
    docker-compose.yml.j2  the container, with limits from the resource classes
    <config files>.j2      whatever the service reads at start
  handlers/main.yml        restart the container when a rendered file changes

The playbook runs the roles in order: first the base system (common, router, VPN, ZFS, Docker), then the services. A full deploy goes through all of them. A normal one does not. When I moved the temp directory of Plex into RAM, it was one variable, one line in a template and make deploy-tags TAGS=plex, and only Plex restarted.

Almost no role has a defaults/ directory, and none has vars/. A role holds the logic, not the values.

Every value in two files

The values live in two files. inventory/group_vars/all/common.yml has everything shared: datasets, ports, hostnames, resource limits. inventory/production/group_vars/all/vars.yml has what belongs to the real server: addresses, interfaces, and references to the secrets, which are in an encrypted vault.yml next to it.

vault.yml.example lists the keys of the vault with fake values, so I can read what exists without decrypting anything. make vault-check compares the two lists and fails when a key is missing from the example or left there after removal.

Resource classes instead of numbers

This server was born for Plex and the *arr stack, and nothing else. Everything that came after shares the same 4 cores and 32 GB, and the CPU is the tight part. Giving every service the right resources is critical, and Docker gives three knobs per container:

cpu_shares is a weight, not a cap. It only matters when the CPU is busy, and Docker's default is 1024.
mem_limit is a hard cap. A container that goes over it gets a process killed by the kernel, inside that container, and the rest of the server is not involved.
oom_score_adj, from -1000 to 1000, decides who the kernel kills first when the whole host runs out of memory.

Three numbers for each of a hundred services is a lot of numbers to keep consistent. So the numbers live in a few named classes, and a service picks one:

resource_class:
  critical:
    cpu_shares: 4096
    mem_limit: "0"
    oom_score_adj: -500
  high:
    cpu_shares: 768
    mem_limit: 768m
    oom_score_adj: -100
  high_light:
    cpu_shares: 768
    mem_limit: 384m
    oom_score_adj: -100
  medium:
    cpu_shares: 384
    mem_limit: 768m
    oom_score_adj: 0
  low_light:
    cpu_shares: 192
    mem_limit: 384m
    oom_score_adj: 100

The name says the priority, the suffix says the memory: high and high_light get the same CPU weight and the same place in the kill order, and differ only in the cap. critical has no memory cap, four times Docker's default weight, and is the last thing the kernel kills. Only two services are in it: Vaultwarden, and Plex, the reason the server exists. A few classes exist for one kind of workload, like the containers that drive a real Chrome: a single page of results costs about 400 MB, so they get 2 GB.

The mapping is one line per service:

container_resources:
  plex: "{{ resource_class.critical }}"
  traefik: "{{ resource_class.high }}"
  adguard: "{{ resource_class.high_light }}"
  sonarr: "{{ resource_class.medium }}"
  transmission: "{{ resource_class.high }}"
  vaultwarden: "{{ resource_class.critical }}"

Each compose template reads its three values from there, so no number is ever written in a role:

    cpu_shares: {{ container_resources.plex.cpu_shares }}
    mem_limit: {{ container_resources.plex.mem_limit }}
    oom_score_adj: {{ container_resources.plex.oom_score_adj }}

The classes change after incidents. A service that needs more moves to another class, and the comment next to its line says why. Traefik was high_light, until one afternoon it went from 85 to 371 MB in under two minutes: the kernel killed it at its 384 MB cap, and every site behind the proxy went down with it. Now it is high, with the same weight and twice the memory. Transmission was medium, until two Plex transcodes at once left it so little CPU that 41% of the calls to its web interface took more than 5 seconds. Now it is high, with the same memory and twice the weight. Both fixes were a one-word change.

The loading order that overrode production

For months the playbook loaded common.yml with vars_files. That is the wrong place for shared defaults. In Ansible's variable precedence, play vars_files rank above inventory group_vars. So when a key was in both files, the shared default won over production.

It happened with the DHCP reservations. common.yml had an empty router_dhcp_static_leases: [] as a default, vars.yml had the real list with the access point in it, and dnsmasq got the empty one. The access point had no reserved address, and I never noticed. It was not about that key either: every key present in both files lost to the default, including the ones nobody had added yet.

Now common.yml is part of the inventory. The environment's group_vars/all/ has a symlink to it, next to vars.yml, and Ansible reads the files of a group_vars directory in lexicographical order: c before v, shared before production.

group_vars/all/
  common.yml              the shared file, the only copy
production/
  hosts.yml
  group_vars/all/
    common.yml -> ../../../group_vars/all/common.yml
    vars.yml              read after common.yml, so it wins
    vault.yml

Achtung The order is alphabetical. Rename vars.yml to something that sorts before common.yml, and the shared defaults win again, without a warning.

Something this easy to break gets a test. tests/test_var_precedence.sh has two halves. The first reads the repo: no vars_files in the playbook, a symlink that resolves to the one shared file, common.yml sorting before vars.yml. The second asks Ansible for the values with ansible -m debug -c local, which renders them without touching the server, and expects production to win: the access point is in the lease list, and every key that differs between the two files has the value from vars.yml. CI has no vault, so there only the first half runs. The second runs on my machine.

Docs that point to the code

The last rule is about the docs/ folder. Docs explain why. They never list services, datasets, networks, databases, tags or versions: those live in common.yml and site.yml, and the docs link there. I made it a rule because static lists were impossible to keep up to date. When I applied it, some docs no longer matched the code at all: one still described HTTP-01 certificates, while Traefik was using the DNS-01 challenge with Route53.

Lint is a rule too: yamllint and ansible-lint, with the production profile, on every push, and the result must be zero warnings. Exceptions go in the linter config, where they are visible. It is worth it, but it is a detail next to the rest.

Part 3 follows one change from the commit to the running container: the CI, the Semaphore webhook, and the checks after a deploy.

Running a one-server homelab like production

Sat, 10 Oct 2026 00:00:00 GMT

Everything my family uses at home runs on one machine: an i5-6500T with 4 cores and 32 GB of RAM, Debian 13, a 250 GB SSD with the system and a ZFS pool, and two USB disks, 6 TB for media and 4 TB for backups. About a hundred containers run on it. It is also the router, so when it goes down, the house goes offline.

This series is about how I keep that machine boring. This first part is the why, and a map of the pieces. The other parts go into each piece.

Kubernetes was too much

Before the current setup I had a Kubernetes cluster with Argo CD. Exploring GitOps was fun, but for us it was overkill: we have one node and no shared filesystem.

In March 2026 I started again from an empty repo, with Ansible. Its README states the rule: the repo is the source of truth, and the server is disposable. Everything that matters lives on ZFS datasets that are backed up every night, and a fresh Debian install plus one playbook rebuilds the rest. There is a runbook for that rebuild in the repo.

Why the server became the router

Until April the router was a TP-Link AX18. A NAT router keeps a connection tracking table: one entry for every connection between a device inside and a host outside, so it knows where to send the replies. Every entry costs memory, and the AX18 has 256 MB of RAM for everything it does. When the table is full, new connections are dropped. With the devices of a whole family and a torrent client, ours was full, and the symptom was the upload: it went to almost zero.

First I tried to make the problem smaller: I lowered Transmission's peer limit from 200 to 80, and started pinging three targets every 15 seconds, the gateway and two public DNS servers, so the next outage would show whether the router or the provider was at fault. Two days later I moved NAT, DHCP and the firewall to the server, and the AX18 became a Wi-Fi access point.

	AX18 doing NAT	Server doing NAT
Upload	almost 0 Mbps	887 Mbps
Download	355 Mbps	941 Mbps
Connection tracking	256 MB of RAM for the whole router	room for 262,144 entries, about 2,800 in use as I write

Probably a good router would have been enough. But we like to make our lives complicated, and this way we got two things for one. The server now hands out the addresses on the network, and it gives itself as the DNS server, so every device goes through AdGuard and the ad and tracker filters live in one place.

The price is that the server is the single point of failure for the whole family. It is not ideal. One day we will fix it.

How the pieces fit

The diagram is the whole series in one picture. The laptop and Semaphore change the server; Prometheus, Loki and restic watch it and copy it. Forgejo and Semaphore run on the same server, which is less circular than it looks: a deploy from the laptop needs neither of them.

<figure style="margin:2rem 0"> <svg viewBox="0 0 440 520" role="img" aria-labelledby="hlp-title hlp-desc" style="display:block;width:100%;max-width:440px;height:auto;margin:0 auto;font-family:var(--font-sans)"> <title id="hlp-title">How the homelab is built, deployed and watched</title> <desc id="hlp-desc">A laptop pushes to Forgejo, which lints and tests every push. A release triggers Semaphore. The laptop or Semaphore applies Ansible roles to the server. The server sends metrics and logs to Prometheus, Loki and Grafana, which alert on Telegram, and restic copies its data to S3 every night.</desc> <defs><marker id="hlp-arrow" viewBox="0 0 8 8" refX="7" refY="4" markerWidth="7" markerHeight="7" orient="auto"><path d="M0,0 L8,4 L0,8 z" style="fill:var(--muted)"/></marker></defs> <g style="fill:var(--code-bg);stroke:var(--rule);stroke-width:1"> <rect x="20" y="20" width="190" height="80" rx="6"/> <rect x="230" y="20" width="190" height="80" rx="6"/> <rect x="230" y="150" width="190" height="80" rx="6"/> <rect x="20" y="410" width="190" height="96" rx="6"/> <rect x="230" y="410" width="190" height="96" rx="6"/> </g> <rect x="20" y="280" width="400" height="80" rx="6" style="fill:var(--code-bg);stroke:var(--accent);stroke-width:1.5"/> <g style="stroke:var(--muted);stroke-width:1.5;fill:none" marker-end="url(#hlp-arrow)"> <line x1="210" y1="60" x2="228" y2="60" marker-end="url(#hlp-arrow)"/> <line x1="325" y1="100" x2="325" y2="148" marker-end="url(#hlp-arrow)"/> <line x1="115" y1="100" x2="115" y2="278" marker-end="url(#hlp-arrow)"/> <line x1="325" y1="230" x2="325" y2="278" marker-end="url(#hlp-arrow)"/> <line x1="115" y1="360" x2="115" y2="408" marker-end="url(#hlp-arrow)"/> <line x1="325" y1="360" x2="325" y2="408" marker-end="url(#hlp-arrow)"/> </g> <g text-anchor="middle" style="fill:var(--fg);font-size:15px;font-weight:600"> <text x="115" y="46">Laptop</text> <text x="325" y="46">Forgejo</text> <text x="325" y="176">Semaphore</text> <text x="220" y="306">Server</text> <text x="115" y="436">Grafana</text> <text x="325" y="436">restic → S3</text> </g> <g text-anchor="middle" style="fill:var(--muted);font-size:13px"> <text x="115" y="68">git push</text> <text x="115" y="88">make deploy-tags</text> <text x="325" y="68">git repos</text> <text x="325" y="88">Actions: lint, tests</text> <text x="325" y="198">webhook on release</text> <text x="325" y="218">runs one role</text> <text x="220" y="328">Debian 13 · ZFS · ~100 containers</text> <text x="220" y="348">one Ansible role per service</text> <text x="115" y="458">Prometheus, Loki</text> <text x="115" y="478">a dashboard per service</text> <text x="115" y="498">alerts on Telegram</text> <text x="325" y="458">every night</text> <text x="325" y="478">checked every week</text> <text x="325" y="498">restored every month</text> </g> <g style="fill:var(--muted);font-size:12px"> <text x="123" y="194">deploy</text> <text x="333" y="128">release</text> <text x="333" y="258">apply</text> <text x="123" y="390">metrics, logs</text> <text x="333" y="390">data</text> </g> </svg> </figure>

One repo, one playbook

One Ansible role per service, all run in order by one playbook. Two variable files hold every setting: one shared, and one for production with the real addresses and the references to the vault, where the secrets are. In broad lines the repo looks like this:

inventory/
  group_vars/all/common.yml     shared settings: datasets, limits, hostnames
  production/
    hosts.yml                   the one server
    group_vars/all/
      vars.yml                  production values, pointers into the vault
      vault.yml                 secrets, encrypted with ansible-vault
playbooks/
  site.yml                      one play, every role in order
roles/
  <service>/
    tasks/main.yml              directories, config files, compose up
    templates/                  docker-compose.yml.j2 and the config files
    handlers/main.yml           restart when the config changes
  grafana/files/dashboards/     one dashboard per service
scripts/
  deploy-verify.sh              checks after a deploy
docs/                           the why: architecture, runbooks
Makefile                        lint, test, deploy-check, deploy-tags

The playbook starts by refusing to run on anything that is not Debian 13, then goes through the roles from the base system up:

  pre_tasks:
    - name: Verify Debian 13 (Trixie)
      ansible.builtin.assert:
        that:
          - ansible_facts['distribution'] == "Debian"
          - ansible_facts['distribution_major_version'] == "13"
        fail_msg: "This playbook requires Debian 13 (Trixie)"

  roles:
    - role: common
      tags: [common, security]

    - role: router
      tags: [router, network]
      when: router_enabled | default(false)

    - role: wireguard
      tags: [wireguard, vpn, network]
      when: wireguard_port is defined

    - role: zfs
      tags: [zfs, storage]

    - role: docker
      tags: [docker]

Part 2 is about how the repo is organized and why the docs never list services.

Deploy one role at a time

I don't change containers on the server by hand. A change is a commit, and a deploy usually touches one role, with a dry run first when the change is risky:

deploy-check: galaxy ## Dry-run deployment on production
	uv run ansible-playbook $(PLAYBOOK) -i $(INVENTORY) --check --diff

deploy: galaxy ## Deploy to production
	uv run ansible-playbook $(PLAYBOOK) -i $(INVENTORY)

deploy-tags: galaxy ## Deploy specific tags to production (usage: make deploy-tags TAGS=router)
	uv run ansible-playbook $(PLAYBOOK) -i $(INVENTORY) --tags $(TAGS)

After the deploy, a script checks the container status, the HTTP answer over the real domain and the last minutes of logs. Every push runs yamllint, ansible-lint and the offline tests on Forgejo Actions, and lint has to finish with zero warnings. The apps I write myself have their own pipeline: a tagged release runs test, publish, release and deploy, and the last step calls a Semaphore webhook that applies the app's role. Part 3 is the deploy path, step by step.

Alerts with the threshold on the graph

Every service gets a Grafana dashboard, and its alert rules live in the repo with everything else. They arrive on Telegram. The rule is that a panel with an alert draws the alert's threshold as a line, so the number on the graph and the number that sends the message are the same number. Part 4 is about how I decide what deserves an alert.

Backups that get restored

Restic copies the container data to S3 every night and keeps 30 days. Once a week a job runs restic check and reads the data back. Once a month a drill restores every database into a throwaway copy. I learned the hard way that a backup has to be tested: part 5 is that story.

What it caught, and what it didn't

The discipline doesn't mean that the monitoring finds everything. Three examples from the last months:

Time Machine. The verification of the backup bundle had not run for five months, and no alert said it. I found it by hand, and the alert on the age of the last verification came after (part 5).
The ZFS ARC. It was allowed 97% of the RAM, next to all those containers. I found it while going through the memory of the server after an outage (part 6).
authentik. One day the Docker daemon restarted. A minute before, the authentik worker had died: it had started before DNS was up and could not find its database. The unless-stopped policy does not bring back a container that was already stopped when the daemon starts again, so the worker stayed down, and half of authentik with it. No alert fired.

None of the three was caught by an alert the first time. Each one ended up in the repo: an alert for Time Machine, a cap in the ZFS role, and restart: always for the critical containers.

What it costs, and the weak spot

Day to day, it is low effort. Most changes are one role and one make deploy-tags.

The weak spot is the storage. Two USB disks are amateur stuff: they are hard to expand, and the backup disk is six years old. The next step is probably a real DAS.

The rest of the series, in the order it makes sense to read it, is on the series page.