Running a one-server homelab like production

Everything my family uses at home runs on one machine: an i5-6500T with 4 cores and 32 GB of RAM, Debian 13, a 250 GB SSD with the system and a ZFS pool, and two USB disks, 6 TB for media and 4 TB for backups. About a hundred containers run on it. It is also the router, so when it goes down, the house goes offline.

This series is about how I keep that machine boring. This first part is the why, and a map of the pieces. The other parts go into each piece.

Kubernetes was too much

Before the current setup I had a Kubernetes cluster with Argo CD. Exploring GitOps was fun, but for us it was overkill: we have one node and no shared filesystem.

In March 2026 I started again from an empty repo, with Ansible. Its README states the rule: the repo is the source of truth, and the server is disposable. Everything that matters lives on ZFS datasets that are backed up every night, and a fresh Debian install plus one playbook rebuilds the rest. There is a runbook for that rebuild in the repo.

Why the server became the router

Until April the router was a TP-Link AX18. A NAT router keeps a connection tracking table: one entry for every connection between a device inside and a host outside, so it knows where to send the replies. Every entry costs memory, and the AX18 has 256 MB of RAM for everything it does. When the table is full, new connections are dropped. With the devices of a whole family and a torrent client, ours was full, and the symptom was the upload: it went to almost zero.

First I tried to make the problem smaller: I lowered Transmission’s peer limit from 200 to 80, and started pinging three targets every 15 seconds, the gateway and two public DNS servers, so the next outage would show whether the router or the provider was at fault. Two days later I moved NAT, DHCP and the firewall to the server, and the AX18 became a Wi-Fi access point.

Probably a good router would have been enough. But we like to make our lives complicated, and this way we got two things for one. The server now hands out the addresses on the network, and it gives itself as the DNS server, so every device goes through AdGuard and the ad and tracker filters live in one place.

The price is that the server is the single point of failure for the whole family. It is not ideal. One day we will fix it.

How the pieces fit

The diagram is the whole series in one picture. The laptop and Semaphore change the server; Prometheus, Loki and restic watch it and copy it. Forgejo and Semaphore run on the same server, which is less circular than it looks: a deploy from the laptop needs neither of them.

One repo, one playbook

One Ansible role per service, all run in order by one playbook. Two variable files hold every setting: one shared, and one for production with the real addresses and the references to the vault, where the secrets are. In broad lines the repo looks like this:

inventory/
  group_vars/all/common.yml     shared settings: datasets, limits, hostnames
  production/
    hosts.yml                   the one server
    group_vars/all/
      vars.yml                  production values, pointers into the vault
      vault.yml                 secrets, encrypted with ansible-vault
playbooks/
  site.yml                      one play, every role in order
roles/
  <service>/
    tasks/main.yml              directories, config files, compose up
    templates/                  docker-compose.yml.j2 and the config files
    handlers/main.yml           restart when the config changes
  grafana/files/dashboards/     one dashboard per service
scripts/
  deploy-verify.sh              checks after a deploy
docs/                           the why: architecture, runbooks
Makefile                        lint, test, deploy-check, deploy-tags

The playbook starts by refusing to run on anything that is not Debian 13, then goes through the roles from the base system up:

  pre_tasks:
    - name: Verify Debian 13 (Trixie)
      ansible.builtin.assert:
        that:
          - ansible_facts['distribution'] == "Debian"
          - ansible_facts['distribution_major_version'] == "13"
        fail_msg: "This playbook requires Debian 13 (Trixie)"

  roles:
    - role: common
      tags: [common, security]

    - role: router
      tags: [router, network]
      when: router_enabled | default(false)

    - role: wireguard
      tags: [wireguard, vpn, network]
      when: wireguard_port is defined

    - role: zfs
      tags: [zfs, storage]

    - role: docker
      tags: [docker]

Part 2 is about how the repo is organized and why the docs never list services.

Deploy one role at a time

I don’t change containers on the server by hand. A change is a commit, and a deploy usually touches one role, with a dry run first when the change is risky:

deploy-check: galaxy ## Dry-run deployment on production
  uv run ansible-playbook $(PLAYBOOK) -i $(INVENTORY) --check --diff

deploy: galaxy ## Deploy to production
  uv run ansible-playbook $(PLAYBOOK) -i $(INVENTORY)

deploy-tags: galaxy ## Deploy specific tags to production (usage: make deploy-tags TAGS=router)
  uv run ansible-playbook $(PLAYBOOK) -i $(INVENTORY) --tags $(TAGS)

After the deploy, a script checks the container status, the HTTP answer over the real domain and the last minutes of logs. Every push runs yamllint, ansible-lint and the offline tests on Forgejo Actions, and lint has to finish with zero warnings. The apps I write myself have their own pipeline: a tagged release runs test, publish, release and deploy, and the last step calls a Semaphore webhook that applies the app’s role. Part 3 is the deploy path, step by step.

Alerts with the threshold on the graph

Every service gets a Grafana dashboard, and its alert rules live in the repo with everything else. They arrive on Telegram. The rule is that a panel with an alert draws the alert’s threshold as a line, so the number on the graph and the number that sends the message are the same number. Part 4 is about how I decide what deserves an alert.

Backups that get restored

Restic copies the container data to S3 every night and keeps 30 days. Once a week a job runs restic check and reads the data back. Once a month a drill restores every database into a throwaway copy. I learned the hard way that a backup has to be tested: part 5 is that story.

What it caught, and what it didn’t

The discipline doesn’t mean that the monitoring finds everything. Three examples from the last months:

• Time Machine. The verification of the backup bundle had not run for five months, and no alert said it. I found it by hand, and the alert on the age of the last verification came after (part 5).
• The ZFS ARC. It was allowed 97% of the RAM, next to all those containers. I found it while going through the memory of the server after an outage (part 6).
• authentik. One day the Docker daemon restarted. A minute before, the authentik worker had died: it had started before DNS was up and could not find its database. The unless-stopped policy does not bring back a container that was already stopped when the daemon starts again, so the worker stayed down, and half of authentik with it. No alert fired.

None of the three was caught by an alert the first time. Each one ended up in the repo: an alert for Time Machine, a cap in the ZFS role, and restart: always for the critical containers.

What it costs, and the weak spot

Day to day, it is low effort. Most changes are one role and one make deploy-tags.

The weak spot is the storage. Two USB disks are amateur stuff: they are hard to expand, and the backup disk is six years old. The next step is probably a real DAS.

The rest of the series, in the order it makes sense to read it, is on the series page.