devncode

Homelab deploys with Forgejo Actions and Semaphore

How do I deploy a homelab from git with Forgejo Actions and Semaphore?

Published HomelabPart 3 of 5 in Running a homelab like production

The repo from part 2 has two kinds of things to deploy. The infrastructure: Traefik, Plex, the databases, the monitoring, everything the house runs on. And the apps we write ourselves, like the backend of TreniTardi, a free iPhone app that tells you how late your regional train will be before it leaves. They take two different roads to the server, on purpose.

Two deploy paths to one server On the left, the infrastructure: from the laptop, a dry run and then a deploy of one Ansible tag, followed by a check script. On the right, our apps: a git tag starts Forgejo Actions, which tests, builds the image, makes a release and calls a Semaphore webhook; Semaphore runs the app's Ansible role on the server. INFRASTRUCTURE OUR APPS Laptop git tag Forgejo Actions Semaphore Server deploy-verify.sh make deploy-check make deploy-tags pushed to Forgejo test, build the image release, call the webhook runs the app's role ansible-playbook --tags <role> status, HTTP, logs

The infrastructure: by hand, one tag at a time

For now we prefer to deploy the base infrastructure by hand, from the laptop. Nothing in the infrastructure deploys itself: a push to the repo only runs the checks on Forgejo Actions, yamllint, ansible-lint and the offline tests, and stops there.

A deploy is three commands. The dry run first, when the change is risky: --check --diff shows what Ansible would change, file by file, without changing it. Then the deploy of one role through its tag, as in part 2. Then the check:

Terminal window
make deploy-check
make deploy-tags TAGS=plex
scripts/deploy-verify.sh plex

deploy-verify.sh asks the server three things. Are the containers running? Does the service answer over its real domain, with a 200, 302 or 401, and not a 500, a 502 or a refused connection? Did any container log an error in the last five minutes?

Our apps: a git tag is a deploy

The apps live in their own repos, and pushing a tag is the whole release. One workflow on Forgejo Actions runs in steps, and each one starts only if the previous one passed: the tests, the build of the image and its push to Forgejo’s own registry, the release with the changelog, and the deploy. The deploy step is just a call to Semaphore:

.forgejo/workflows/release.yaml
  deploy:
    name: Deploy via Semaphore
    needs: publish
    runs-on: ubuntu-latest
    container:
      image: docker:cli
    steps:
      - name: Trigger Semaphore deploy
        run: |
          apk add --no-cache curl
          curl -fsS -X POST "${{ secrets.SEMAPHORE_WEBHOOK_URL }}" \
            -H "X-Semaphore-Token: ${{ secrets.SEMAPHORE_WEBHOOK_TOKEN }}" \
            -H "Content-Type: application/json" \
            -d "{\"ref\":\"${{ github.ref }}\"}"

Semaphore is a web UI that runs Ansible. It lives on the same server, and it has one template per app: the same playbook as the laptop, limited to that app’s tag. Its own Ansible role deploys it, together with an encrypted copy of the vault and the password to open it, so a webhook deploy runs the same playbook and the same role that make deploy-tags runs from the laptop.

It fits our infrastructure well, but I’m not convinced by it. For our case it may be overkill, and the infrastructure doesn’t use it at all.

The CI runs on the server it deploys to

Forgejo and its Actions runner live on the same 4-core machine as everything else. Every CI job is a handful of containers: the job, and the services it asks for, like a throwaway PostgreSQL for the tests. And the runner creates those containers itself, outside of compose, so the resource classes from part 2 never reached them.

They ran with Docker’s default CPU weight, 1024. Every class except critical is below it, so when the CPU was busy, the CI won over the services. With four jobs at a time the load average went to about 40, and the CPU pressure said that two thirds of the time something was waiting for a core. With two jobs, the CI still took 7 core-hours a day.

Now the runner takes one job at a time, and every job container gets the weight of the low class, at most one and a half cores, and 5 GB of memory with no swap, so a runaway test cannot push the rest of the server out of RAM:

roles/forgejo/templates/runner-config.yaml.j2
runner:
  capacity: 1
container:
  network: ""
  options: "--memory=5g --memory-swap=5g --cpus=1.5 --cpu-shares=192"

A job still uses every idle cycle. It just steps aside when the CPU is short, and a pipeline waits in the queue for a few minutes instead of slowing down Plex. The empty network gives each job a fresh network of its own: with a shared one, every job’s PostgreSQL had the same name, and the tests sometimes connected to another job’s database.

Running the CI next to the services is not ideal, and it is one of the things we will fix sooner or later.

Next

Part 4 is about what happens after the deploy: the alerts that should tell me when something breaks, including the ones a deploy causes.