Structuring an Ansible repo for a one-server homelab

The server from part 1 runs about a hundred containers, and one Ansible repo describes all of them. I picked Ansible because I have used it for years and I have always been happy with it. This part is about the shape of that repo: where a service lives, where its values live, and how one bad loading order overrode production for months without anyone noticing.

One role per service

Every service is a role, and every role has its own tag in the playbook. That is the whole reason for the layout: a deploy can touch one service and nothing else. A role is small and always looks the same:

roles/<service>/
  tasks/main.yml           create the directories, render the files, compose up
  templates/
    docker-compose.yml.j2  the container, with limits from the resource classes
    <config files>.j2      whatever the service reads at start
  handlers/main.yml        restart the container when a rendered file changes

The playbook runs the roles in order: first the base system (common, router, VPN, ZFS, Docker), then the services. A full deploy goes through all of them. A normal one does not. When I moved the temp directory of Plex into RAM, it was one variable, one line in a template and make deploy-tags TAGS=plex, and only Plex restarted.

Almost no role has a defaults/ directory, and none has vars/. A role holds the logic, not the values.

Every value in two files

The values live in two files. inventory/group_vars/all/common.yml has everything shared: datasets, ports, hostnames, resource limits. inventory/production/group_vars/all/vars.yml has what belongs to the real server: addresses, interfaces, and references to the secrets, which are in an encrypted vault.yml next to it.

vault.yml.example lists the keys of the vault with fake values, so I can read what exists without decrypting anything. make vault-check compares the two lists and fails when a key is missing from the example or left there after removal.

Resource classes instead of numbers

This server was born for Plex and the *arr stack, and nothing else. Everything that came after shares the same 4 cores and 32 GB, and the CPU is the tight part. Giving every service the right resources is critical, and Docker gives three knobs per container:

cpu_shares is a weight, not a cap. It only matters when the CPU is busy, and Docker’s default is 1024.
mem_limit is a hard cap. A container that goes over it gets a process killed by the kernel, inside that container, and the rest of the server is not involved.
oom_score_adj, from -1000 to 1000, decides who the kernel kills first when the whole host runs out of memory.

Three numbers for each of a hundred services is a lot of numbers to keep consistent. So the numbers live in a few named classes, and a service picks one:

resource_class:
  critical:
    cpu_shares: 4096
    mem_limit: "0"
    oom_score_adj: -500
  high:
    cpu_shares: 768
    mem_limit: 768m
    oom_score_adj: -100
  high_light:
    cpu_shares: 768
    mem_limit: 384m
    oom_score_adj: -100
  medium:
    cpu_shares: 384
    mem_limit: 768m
    oom_score_adj: 0
  low_light:
    cpu_shares: 192
    mem_limit: 384m
    oom_score_adj: 100

The name says the priority, the suffix says the memory: high and high_light get the same CPU weight and the same place in the kill order, and differ only in the cap. critical has no memory cap, four times Docker’s default weight, and is the last thing the kernel kills. Only two services are in it: Vaultwarden, and Plex, the reason the server exists. A few classes exist for one kind of workload, like the containers that drive a real Chrome: a single page of results costs about 400 MB, so they get 2 GB.

The mapping is one line per service:

container_resources:
  plex: "{{ resource_class.critical }}"
  traefik: "{{ resource_class.high }}"
  adguard: "{{ resource_class.high_light }}"
  sonarr: "{{ resource_class.medium }}"
  transmission: "{{ resource_class.high }}"
  vaultwarden: "{{ resource_class.critical }}"

Each compose template reads its three values from there, so no number is ever written in a role:

    cpu_shares: {{ container_resources.plex.cpu_shares }}
    mem_limit: {{ container_resources.plex.mem_limit }}
    oom_score_adj: {{ container_resources.plex.oom_score_adj }}

The classes change after incidents. A service that needs more moves to another class, and the comment next to its line says why. Traefik was high_light, until one afternoon it went from 85 to 371 MB in under two minutes: the kernel killed it at its 384 MB cap, and every site behind the proxy went down with it. Now it is high, with the same weight and twice the memory. Transmission was medium, until two Plex transcodes at once left it so little CPU that 41% of the calls to its web interface took more than 5 seconds. Now it is high, with the same memory and twice the weight. Both fixes were a one-word change.

The loading order that overrode production

For months the playbook loaded common.yml with vars_files. That is the wrong place for shared defaults. In Ansible’s variable precedence, play vars_files rank above inventory group_vars. So when a key was in both files, the shared default won over production.

It happened with the DHCP reservations. common.yml had an empty router_dhcp_static_leases: [] as a default, vars.yml had the real list with the access point in it, and dnsmasq got the empty one. The access point had no reserved address, and I never noticed. It was not about that key either: every key present in both files lost to the default, including the ones nobody had added yet.

Now common.yml is part of the inventory. The environment’s group_vars/all/ has a symlink to it, next to vars.yml, and Ansible reads the files of a group_vars directory in lexicographical order: c before v, shared before production.

group_vars/all/
  common.yml              the shared file, the only copy
production/
  hosts.yml
  group_vars/all/
    common.yml -> ../../../group_vars/all/common.yml
    vars.yml              read after common.yml, so it wins
    vault.yml

Something this easy to break gets a test. tests/test_var_precedence.sh has two halves. The first reads the repo: no vars_files in the playbook, a symlink that resolves to the one shared file, common.yml sorting before vars.yml. The second asks Ansible for the values with ansible -m debug -c local, which renders them without touching the server, and expects production to win: the access point is in the lease list, and every key that differs between the two files has the value from vars.yml. CI has no vault, so there only the first half runs. The second runs on my machine.

Docs that point to the code

The last rule is about the docs/ folder. Docs explain why. They never list services, datasets, networks, databases, tags or versions: those live in common.yml and site.yml, and the docs link there. I made it a rule because static lists were impossible to keep up to date. When I applied it, some docs no longer matched the code at all: one still described HTTP-01 certificates, while Traefik was using the DNS-01 challenge with Route53.

Lint is a rule too: yamllint and ansible-lint, with the production profile, on every push, and the result must be zero warnings. Exceptions go in the linter config, where they are visible. It is worth it, but it is a detail next to the rest.

Part 3 follows one change from the commit to the running container: the CI, the Semaphore webhook, and the checks after a deploy.