chore(deps): update renovate/renovate docker tag to v44.132.4

.gitea/deploy-dependencies.json

@@ -1,3 +0,0 @@
-{
-  "postgres": ["authentik", "gitea", "immich", "n8n", "netbox", "netronome"]
-}

.gitea/runner/README.md

@@ -1,122 +0,0 @@
-# Homelab CI/CD
-
-The native Gitea runner runs on **vps**; production runs on **workstation**.
-Jobs run on `homelab:host`, one at a time. No job images or Kubernetes credentials
-are needed on the VPS. Builds use one pinned BuildKit helper container. CI and deploy are separate workflows.
-
-## Runner installation
-
-Install Docker Engine with Compose and Buildx, Git, Python 3.11+, Bash, curl,
-GNU tar/xz, flock and systemd using the host's package manager. Keep the existing
-Gitea runner 3.0.2 binary at `/usr/local/bin/gitea-runner`.
-
-From this checkout on the VPS:
-
-```sh
-sudo bash .gitea/runner/setup-runner.sh
-```
-
-The installer reuses `/var/lib/gitea-runner/.runner` and the existing service.
-For a new host, install the same runner binary and register as `gitea-runner`
-using the registration token interactively, label `homelab:host`, and working
-directory `/var/lib/gitea-runner`; then rerun the installer. Tokens never belong
-in this repository or command-line examples.
-
-Pinned tools live in the runner user's `~/.cache/homelab-ci`; CI repairs version
-drift there. Installations are locked. Buildx uses only the `homelab-ci` builder,
-pushes directly to the registry, and caps retained local cache at 1 GiB with a
-2 GiB free-space target. This is not a hard limit on peak build disk usage.
-Nothing runs `docker system prune`, removes unrelated images, or deletes volumes.
-
-## Workstation setup
-
-As the existing SSH deploy user on workstation:
-
-```sh
-sudo loginctl enable-linger forust
-bash .gitea/runner/setup-workstation.sh
-```
-
-The controller uses `/srv/homelab` as the persistent configuration tree and makes
-a detached source worktree for each SHA. It never resets `/srv/homelab`, moves
-local configuration, renames Compose projects, or changes volume names.
-The installer records the current Kubernetes context and cluster UID in
-`~/.config/homelab-deploy/environment`. Check these before installing.
-
-Configure Gitea Actions Variables:
-
-- `DEPLOY_HOST`, `DEPLOY_USER`, `DEPLOY_PORT`: the existing VPS-to-workstation SSH endpoint.
-- `DEPLOY_KNOWN_HOSTS`: workstation's verified SSH host key entry for that endpoint.
-- `AUTODEPLOY`: `false` initially; `true` enables deployment after successful main CI.
-
-Keep `DEPLOY_SSH_KEY`, `REGISTRY_USERNAME` and `REGISTRY_PASSWORD` in Actions
-Secrets. Legacy endpoint secrets remain accepted during migration. The Actions
-token must have repository read and Actions read access for release downloads.
-The deploy user's existing Docker registry authentication remains necessary.
-
-## Releases and deployment
-
-CI publishes `release-<full SHA>` as a Gitea artifact with all three owned image
-digests and build input fingerprints. Unchanged images are reused only from a
-successful main CI artifact, never from `:prod`. EDU images remain pinned to the
-digests released by their application repository. Expired artifacts cause CI to
-rebuild images; they block deployment until CI is rerun.
-
-Run deploy from main with `deploy_ref=main` or a checked SHA:
-
-- `full`: required for the first baseline; reconcile all active components.
-- `changed`: compare with the last fully successful production deploy.
-- `plan`: validate configuration and show selection without changing production resources.
-- `refresh_images=true`: explicitly refresh mutable third-party Compose tags.
-
-The manual and automatic paths both require successful CI, a successful build
-job and the exact SHA's release artifact. PRs cannot publish images or deploy.
-Removed resources are reported and require explicit removal; no automatic prune.
-Service dependencies are listed in `.gitea/deploy-dependencies.json`.
-
-A workstation user systemd service holds the deploy lock across validation,
-sequential apply, verification and smoke checks. SSH clients only submit/follow:
-disconnecting or cancelling the Actions client does not kill production apply.
-Retrying the same run ID does not start another apply. `ExecStopPost` recovers
-interrupted runs before the unit finishes. Kubernetes rolls back to captured
-revisions; configuration and persistent data are not reverted.
-
-## Status and recovery
-
-`--retry` repeats failed verification and smoke checks, never apply. Recovery
-keeps a failed deploy out of the successful baseline, even after rollback.
-
-On workstation (replace the numeric ID with Actions run ID and attempt):
-
-```sh
-python3 ~/.local/lib/homelab-deploy/controller.py status 123-1
-python3 ~/.local/lib/homelab-deploy/controller.py recover 123-1 --retry
-journalctl --user -u homelab-deploy@123-1
-```
-
-Runs live in `~/.local/state/homelab-deploy/runs`. Compose stores resolved configs
-with restricted permissions; these may contain credentials and must never be
-uploaded as CI artifacts. Stage logs print the exact manual recovery command
-using `compose-before/<stack>.json`, the original project directory and project
-name. Compose does not automatically roll back, and Nextcloud AIO's child
-containers remain managed by AIO. Preserve its own backups for data recovery.
-
-The controller retains twenty successful/planned runs and preserves failures.
-Update the workstation dispatcher only when no deploy is running.
-
-## Validation and migration rollback
-
-```sh
-python3 -m unittest discover -s tests -v
-bash .gitea/tests/deploy-validation.sh
-```
-
-Test on a separate namespace before the initial production `full` run. Check a
-failed rollout, interrupted SSH and repeated run ID, and verify that an isolated
-service change does not upgrade unrelated Helm releases or Compose stacks.
-
-To roll back the migration, disable autodeploy and finish or recover the remote
-run first. Restore the runner config/unit from `.before-<timestamp>` backups,
-reload systemd and restart the runner. Restore the prior workflows from Git.
-Production data and persistent volumes stay where they were. Do not remove run
-state or Compose recovery files until recovery is confirmed.

.gitea/runner/buildkitd.toml

@@ -1,11 +0,0 @@
-[worker.oci]
-  gc = true
-  reservedSpace = "256MB"
-  maxUsedSpace = "1GB"
-  minFreeSpace = "2GB"
-
-[[worker.oci.gcpolicy]]
-  reservedSpace = "256MB"
-  maxUsedSpace = "1GB"
-  minFreeSpace = "2GB"
-  all = true

.gitea/runner/config.yaml

@@ -1,10 +0,0 @@
-runner:
-  file: /var/lib/gitea-runner/.runner
-  capacity: 1
-  timeout: 5h
-  labels:
-    - homelab:host
-cache:
-  enabled: false
-container:
-  docker_host: unix:///var/run/docker.sock

.gitea/runner/gitea-runner.service

@@ -1,18 +0,0 @@
-[Unit]
-Description=Gitea Actions runner
-After=network-online.target docker.service
-Wants=network-online.target
-
-[Service]
-User=gitea-runner
-Group=gitea-runner
-SupplementaryGroups=docker
-WorkingDirectory=/var/lib/gitea-runner
-Environment=PATH=/var/lib/gitea-runner/.cache/homelab-ci/bin:/usr/local/bin:/usr/bin:/bin
-ExecStart=/usr/local/bin/gitea-runner daemon --config /etc/gitea-runner/config.yaml
-Restart=on-failure
-RestartSec=5
-UMask=0077
-
-[Install]
-WantedBy=multi-user.target

.gitea/runner/homelab-deploy@.service

@@ -1,12 +0,0 @@
-[Unit]
-Description=Homelab deploy %i
-
-[Service]
-Type=exec
-EnvironmentFile=%h/.config/homelab-deploy/environment
-ExecStart=/usr/bin/python3 %h/.local/lib/homelab-deploy/controller.py execute %i
-ExecStopPost=/usr/bin/python3 %h/.local/lib/homelab-deploy/controller.py recover %i
-RuntimeMaxSec=5h
-TimeoutStopSec=135min
-KillMode=control-group
-UMask=0077

.gitea/runner/setup-runner.sh

@@ -1,48 +0,0 @@
-#!/usr/bin/env bash
-# Native host runner, with pinned user-space tools and no extra CI images.
-set -euo pipefail
-here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-[ "$(id -u)" -eq 0 ] || { echo 'Run with sudo on the runner host' >&2; exit 1; }
-for tool in docker curl python3 git tar xz flock runuser systemctl; do
-  command -v "$tool" >/dev/null || { echo "Install missing prerequisite: $tool" >&2; exit 1; }
-done
-docker info >/dev/null
-docker compose version >/dev/null
-docker buildx version >/dev/null
-id gitea-runner >/dev/null 2>&1 || useradd --system --create-home --home-dir /var/lib/gitea-runner --shell /usr/bin/bash gitea-runner
-# Reuse the established service account and runner registration.
-runner_home="$(getent passwd gitea-runner | cut -d: -f6)"
-[ "$runner_home" = /var/lib/gitea-runner ] || { echo 'Unexpected runner home; inspect the existing service first' >&2; exit 1; }
-runuser -u gitea-runner -- docker info >/dev/null || { echo "The runner user needs access to Docker before setup" >&2; exit 1; }
-command -v gitea-runner >/dev/null || { echo 'Install gitea-runner 3.0.2 at /usr/local/bin/gitea-runner first' >&2; exit 1; }
-mkdir -p /etc/gitea-runner
-stamp="$(date -u +%Y%m%dT%H%M%SZ)"
-for existing in /etc/gitea-runner/config.yaml /etc/systemd/system/gitea-runner.service; do
-  [ ! -f "$existing" ] || cp -p "$existing" "$existing.before-$stamp"
-done
-scratch="$(mktemp -d)"
-trap 'rm -rf "$scratch"' EXIT
-chmod 755 "$scratch"
-install -m 0644 "$here/../workflows/install-ci-tools.sh" "$here/../workflows/tool-versions.env" "$scratch/"
-runuser -u gitea-runner -- bash "$scratch/install-ci-tools.sh"
-install -m 0644 "$here/config.yaml" /etc/gitea-runner/config.yaml
-python3 - <<'PYLABELS'
-import json
-from pathlib import Path
-registration = Path('/var/lib/gitea-runner/.runner')
-if registration.exists():
-    labels = json.loads(registration.read_text()).get('labels', [])
-    labels = [label for label in labels if isinstance(label, str) and label.split(':')[0] != 'homelab']
-    labels.append('homelab:host')
-    config = Path('/etc/gitea-runner/config.yaml')
-    config.write_text(config.read_text().replace('    - homelab:host', '\n'.join('    - ' + json.dumps(label) for label in labels)))
-PYLABELS
-install -m 0644 "$here/gitea-runner.service" /etc/systemd/system/gitea-runner.service
-if [ ! -f /var/lib/gitea-runner/.runner ]; then
-  echo 'Register once as gitea-runner with homelab:host before starting the service.'
-  exit 0
-fi
-systemctl daemon-reload
-systemctl enable --now gitea-runner.service
-systemctl restart gitea-runner.service
-echo "Runner ready. Configuration backups: *.before-$stamp"

.gitea/runner/setup-workstation.sh

@@ -1,32 +0,0 @@
-#!/usr/bin/env bash
-# Run as the existing deploy user on workstation. Never resets the working tree.
-set -euo pipefail
-here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-repo="${HOMELAB_REPO:-/srv/homelab}"
-for tool in python3 git kubectl helm docker flock timeout; do
-  command -v "$tool" >/dev/null || { echo "Install missing dependency: $tool" >&2; exit 1; }
-done
-[ -d "$repo/.git" ] || { echo "Missing deploy checkout: $repo" >&2; exit 1; }
-[[ "$repo" =~ ^/[A-Za-z0-9_./-]+$ ]] || { echo 'Deploy path must be absolute and contain no whitespace' >&2; exit 1; }
-if [ "$(loginctl show-user "$USER" -p Linger --value)" != yes ]; then
-  echo "Run once: sudo loginctl enable-linger $USER" >&2
-  exit 1
-fi
-config="${XDG_CONFIG_HOME:-$HOME/.config}/homelab-deploy"
-mkdir -p "$config" "$HOME/.local/lib/homelab-deploy" "$HOME/.config/systemd/user"
-chmod 700 "$config"
-if [ ! -f "$config/environment" ]; then
-  context="$(kubectl config current-context)"
-  cluster_uid="$(kubectl get namespace kube-system -o jsonpath='{.metadata.uid}')"
-  printf 'HOMELAB_REPO=%s\nKUBE_CONTEXT=%s\nEXPECTED_CLUSTER_UID=%s\n' "$repo" "$context" "$cluster_uid" >"$config/environment"
-  chmod 600 "$config/environment"
-fi
-# Do not replace a dispatcher while an existing deploy uses it.
-if systemctl --user list-units 'homelab-deploy@*' --state=running --no-legend | grep -q .; then
-  echo 'An existing deploy is running; wait before updating the controller' >&2
-  exit 1
-fi
-install -m 0755 "$here/../workflows/deploy-controller.py" "$HOME/.local/lib/homelab-deploy/controller.py"
-install -m 0644 "$here/homelab-deploy@.service" "$HOME/.config/systemd/user/homelab-deploy@.service"
-systemctl --user daemon-reload
-echo 'Controller ready. Run a checked main SHA in full mode for the initial baseline.'

.gitea/tests/deploy-validation.sh

@@ -1,94 +0,0 @@
-#!/usr/bin/env bash
-# Local regressions only: kubectl is mocked and Docker is used for config parsing.
-set -euo pipefail
-repo="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
-scratch="$(mktemp -d)"
-trap 'rm -rf "$scratch"' EXIT
-
-mkdir -p "$scratch/repo/app" "$scratch/repo/postgres" "$scratch/repo/netbird" "$scratch/repo/renovate"
-git -C "$scratch/repo" init -q
-for file in app/compose.yaml postgres/shared-compose.yaml netbird/client.compose.yaml renovate/renovate-compose.yaml; do
-  touch "$scratch/repo/$file"
-done
-git -C "$scratch/repo" add .
-# shellcheck source=../workflows/compose-lint.sh
-source "$repo/.gitea/workflows/compose-lint.sh"
-actual="$(cd "$scratch/repo" && compose_files)"
-expected=$'app/compose.yaml\nnetbird/client.compose.yaml\npostgres/shared-compose.yaml\nrenovate/renovate-compose.yaml'
-[ "$actual" = "$expected" ] || { echo 'Compose discovery missed a file' >&2; exit 1; }
-
-cat >"$scratch/compose.yaml" <<'YAML'
-services:
-  example:
-    image: busybox:1.37.0
-    environment:
-      REQUIRED: ${HOMELAB_TEST_REQUIRED:?required for this regression}
-YAML
-unset HOMELAB_TEST_REQUIRED
-if validate_compose_file "$scratch/compose.yaml" >"$scratch/config.log" 2>&1; then
-  echo 'Full Compose validation accepted a missing variable' >&2
-  exit 1
-fi
-grep -q 'required for this regression' "$scratch/config.log"
-HOMELAB_TEST_REQUIRED=present validate_compose_file "$scratch/compose.yaml"
-
-cat >"$scratch/resources.json" <<'JSON'
-{"kind":"List","items":[
-  {"kind":"Deployment","metadata":{"namespace":"app"},"spec":{"template":{"spec":{
-    "containers":[{"envFrom":[{"secretRef":{"name":"credentials"}},{"secretRef":{"name":"optional","optional":true}}],"env":[{"valueFrom":{"secretKeyRef":{"name":"credentials","key":"password"}}}]}],
-    "initContainers":[{"envFrom":[{"secretRef":{"name":"init"}}]}],
-    "imagePullSecrets":[{"name":"registry"}],
-    "volumes":[{"secret":{"secretName":"mounted"}},{"projected":{"sources":[{"secret":{"name":"projected"}},{"secret":{"name":"optional-projected","optional":true}}]}}]
-  }}}},
-  {"kind":"CronJob","metadata":{},"spec":{"jobTemplate":{"spec":{"template":{"spec":{"containers":[{"envFrom":[{"secretRef":{"name":"cron"}}]}]}}}}}},
-  {"kind":"IngressRoute","metadata":{"namespace":"app"},"spec":{"tls":{"secretName":"controller-issued-tls"}}}
-]}
-JSON
-actual="$(jq -r -f "$repo/.gitea/workflows/secret-references.jq" "$scratch/resources.json" | sort)"
-expected=$'app credentials\napp init\napp mounted\napp projected\napp registry\ndefault cron'
-[ "$actual" = "$expected" ] || { echo "Unexpected Secret references: $actual" >&2; exit 1; }
-
-REPO="$repo"
-# shellcheck source=../workflows/deploy-lib.sh
-source "$repo/.gitea/workflows/deploy-lib.sh"
-K8S_MANIFESTS=("$scratch/resources.json")
-KUSTOMIZE_APPS=()
-# No live cluster access. Reject credentials in app even if they exist elsewhere.
-kubectl() {
-  case "$1" in
-    create) cat "$scratch/resources.json" ;;
-    get)
-      if [ "$3" = credentials ] && [ "$5" = app ]; then
-        return 1
-      fi
-      return 0
-      ;;
-    *) echo "Unexpected kubectl invocation: $*" >&2; return 1 ;;
-  esac
-}
-if check_referenced_secrets >"$scratch/secrets.log"; then
-  echo 'Namespace-scoped Secret check accepted a missing Secret' >&2
-  exit 1
-fi
-grep -q 'MISSING OR UNREADABLE: app/credentials' "$scratch/secrets.log"
-# API/rendering errors must not produce an empty reference list and pass.
-kubectl() { return 1; }
-if ! skip_uninstalled_vmagent_crd "$REPO/prometheus-stack/k8s/vmagent.yaml"; then
-  echo 'VMAgent preflight did not skip an uninstalled CRD' >&2
-  exit 1
-fi
-kubectl() { return 0; }
-if skip_uninstalled_vmagent_crd "$REPO/prometheus-stack/k8s/vmagent.yaml"; then
-  echo 'VMAgent preflight skipped an installed CRD' >&2
-  exit 1
-fi
-if skip_uninstalled_vmagent_crd "$REPO/prometheus-stack/k8s/victoria.yaml"; then
-  echo 'VMAgent preflight skipped an unrelated manifest' >&2
-  exit 1
-fi
-kubectl() { return 1; }
-if check_referenced_secrets >"$scratch/secrets.log"; then
-  echo 'Secret check accepted a failed manifest render' >&2
-  exit 1
-fi
-printf '%s\n' 'Deploy validation regressions passed.'

.gitea/workflows/ci.yaml

@@ -1,29 +1,38 @@
 name: ci
-"on":
+
+on:
   push:
     branches:
-      - main
+      - "**"
-  pull_request: null
+  pull_request:
-  workflow_dispatch: null
+  workflow_dispatch:
+
+# Every job here is checkout plus local tools. The token needs to read the tree
+# and nothing else, and saying so keeps a future step that reaches for the API
+# from quietly holding a token that can write to the repository.
 permissions:
   contents: read
-  actions: read
+
 concurrency:
   group: ci-${{ github.ref }}
   cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
+
+env:
+  REGISTRY: gcr.forust.xyz
+
 jobs:
-  checks:
+  lint-compose:
-    runs-on: homelab
+    runs-on: [self-hosted, linux, arch, homelab]
-    timeout-minutes: 30
+    timeout-minutes: 10
     steps:
       - name: Checkout repository
-        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
-      - name: Prepare pinned tools
+
-        shell: bash
+      # Structure check for every committed Compose file, active or not.
-        run: |
+      # Interpolation, env-file and bind-mount resolution are all switched off,
-          set -euo pipefail
+      # because inactive stacks have no .env here and would only fail on their
-          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh)"
+      # ${VAR:?} guards. Active stacks get the full check with interpolation in
-          echo "$tools_dir" >> "$GITHUB_PATH"
+      # the deploy workflow, where the real .env files live.
       - name: Validate Compose files
         shell: bash
         run: |
@@ -52,15 +61,35 @@ jobs:
             exit 1
           fi
           echo "checked ${#files[@]} Compose file(s)"
+
+  lint-actionlint:
+    runs-on: [self-hosted, linux, arch, homelab]
+    timeout-minutes: 10
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+
       - name: Lint Gitea Actions workflows with actionlint
         shell: bash
         run: |
           set -euo pipefail
+          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh actionlint)"
+          export PATH="$tools_dir:$PATH"
           actionlint -config-file .gitea/actionlint.yaml -color .gitea/workflows/*.yaml
+
+  lint-shellcheck:
+    runs-on: [self-hosted, linux, arch, homelab]
+    timeout-minutes: 10
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+
       - name: Lint shell scripts with ShellCheck
         shell: bash
         run: |
           set -euo pipefail
+          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh shellcheck)"
+          export PATH="$tools_dir:$PATH"
           mapfile -t scripts < <(
             git ls-files '*.sh' ':(glob)**/*.bash'
           )
@@ -69,11 +98,20 @@ jobs:
             exit 0
           fi
           shellcheck --external-sources --source-path=SCRIPTDIR --severity=style "${scripts[@]}"
-          bash .gitea/tests/deploy-validation.sh
+
+  lint-prettier:
+    runs-on: [self-hosted, linux, arch, homelab]
+    timeout-minutes: 10
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+
       - name: Check formatting with Prettier
         shell: bash
         run: |
           set -euo pipefail
+          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh prettier)"
+          export PATH="$tools_dir:$PATH"
 
           mapfile -t prettier_files < <(
             git ls-files \
@@ -87,17 +125,36 @@ jobs:
           fi
 
           prettier --check --ignore-unknown "${prettier_files[@]}"
+
+  lint-ruff:
+    runs-on: [self-hosted, linux, arch, homelab]
+    timeout-minutes: 10
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+
       - name: Lint and format-check Python with Ruff
         shell: bash
         run: |
           set -euo pipefail
-          ruff check . .gitea/workflows
+          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh ruff)"
-          ruff format --check . .gitea/workflows
+          export PATH="$tools_dir:$PATH"
-          python3 -m unittest discover -s tests -v
+          ruff check .
+          ruff format --check .
+
+  lint-yaml:
+    runs-on: [self-hosted, linux, arch, homelab]
+    timeout-minutes: 10
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+
       - name: Lint YAML syntax
         shell: bash
         run: |
           set -euo pipefail
+          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh yamllint)"
+          export PATH="$tools_dir:$PATH"
 
           mapfile -t yaml_files < <(
             git ls-files '*.yaml' '*.yml' \
@@ -111,10 +168,20 @@ jobs:
           fi
 
           yamllint -c .yamllint "${yaml_files[@]}"
+
+  lint-dockerfiles:
+    runs-on: [self-hosted, linux, arch, homelab]
+    timeout-minutes: 10
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+
       - name: Lint Dockerfiles
         shell: bash
         run: |
           set -euo pipefail
+          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh hadolint)"
+          export PATH="$tools_dir:$PATH"
 
           mapfile -t dockerfiles < <(
             git ls-files ':(glob)**/Dockerfile' ':(glob)**/Dockerfile.*'
@@ -126,10 +193,20 @@ jobs:
           fi
 
           hadolint -c .hadolint.yaml "${dockerfiles[@]}"
+
+  validate:
+    runs-on: [self-hosted, linux, arch, homelab]
+    timeout-minutes: 20
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+
       - name: Validate Kubernetes manifests against JSON schemas
         shell: bash
         run: |
           set -euo pipefail
+          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh kubeconform)"
+          export PATH="$tools_dir:$PATH"
 
           mapfile -t manifests < <(
             git ls-files ':(glob)**/k8s/**/*.yaml' ':(glob)**/k8s/**/*.yml' \
@@ -146,27 +223,327 @@ jobs:
             -ignore-missing-schemas \
             -summary \
             "${manifests[@]}"
+
+      # kubeconform has no schemas for CRDs, so every IngressRoute, Certificate,
+      # PrometheusRule, Middleware, ServersTransport and ServiceMonitor is silently
+      # skipped above. The live API server knows the real CRD schemas (and runs the
+      # cert-manager / Traefik admission webhooks), so validate there too.
+      #
+      # Only services marked with a k8s/active marker are checked: server-side
+      # dry-run needs the target namespace to exist, and inactive services are not
+      # deployed. Services being enabled for the first time are still covered by
+      # the JSON-schema pass above.
+      #
+      # Main pushes only. `--dry-run=server` persists nothing, but it does execute
+      # the admission webhooks of the production API server, so anyone able to open
+      # a pull request would be able to run arbitrary manifest content through
+      # cert-manager and Traefik. A pull request has nothing to gain from it either:
+      # only main is ever deployed, and this job runs to completion before the
+      # deploy workflow is allowed to start, so a bad CRD is still caught before
+      # anything reaches the cluster -- just on the push rather than on the PR.
+      - name: Note the server-side check is not running here
+        if: github.event_name == 'pull_request' || github.ref != 'refs/heads/main'
+        shell: bash
+        run: |
+          echo "::notice::Skipping the server-side dry-run. It executes the cert-manager and" \
+            "Traefik admission webhooks against the production API server, so it is limited" \
+            "to pushes to main. CRDs are still schema-checked by kubeconform above, and the" \
+            "server-side pass still runs on main before the deploy."
+
+      - name: Validate active manifests against the live API server
+        if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main'
+        shell: bash
+        run: |
+          set -euo pipefail
+
+          if ! kubectl get --raw='/readyz' --request-timeout=10s >/dev/null 2>&1; then
+            echo "::warning::Cluster unreachable — skipped server-side validation of CRDs (IngressRoute, Certificate, PrometheusRule). Review manifest changes manually."
+            exit 0
+          fi
+
+          mapfile -t k8s_dirs < <(
+            git ls-files '*.yaml' '*.yml' \
+              | grep -E '(^|/)k8s/' \
+              | sed -E 's#((^|.*/)k8s)/.*#\1#' \
+              | sort -u
+          )
+
+          manifests=()
+          kustomize_apps=()
+          for dir in "${k8s_dirs[@]}"; do
+            if [ ! -f "${dir}/active" ]; then
+              echo "skip (no k8s/active): ${dir}"
+              continue
+            fi
+            if [ -f "${dir}/overlays/prod/kustomization.yaml" ]; then
+              kustomize_apps+=("${dir}/overlays/prod")
+            elif [ -f "${dir}/base/kustomization.yaml" ]; then
+              kustomize_apps+=("${dir}/base")
+            else
+              while IFS= read -r f; do
+                [ -n "$f" ] && manifests+=("$f")
+              done < <(
+                git ls-files "${dir}/*.yaml" "${dir}/*.yml" \
+                  | grep -Ev '(^|/)(kustomization\.ya?ml|.*\.example\.ya?ml|.*values\.ya?ml|patch-.*\.ya?ml)$'
+              )
+            fi
+          done
+
+          echo "server-side dry-run: ${#manifests[@]} manifests, ${#kustomize_apps[@]} kustomize apps"
+          failed=0
+          for m in ${manifests[@]+"${manifests[@]}"}; do
+            if ! out="$(kubectl apply --dry-run=server -f "$m" 2>&1)"; then
+              failed=1
+              echo "::error file=${m}::$(printf '%s' "$out" | head -1)"
+            fi
+          done
+          for k in ${kustomize_apps[@]+"${kustomize_apps[@]}"}; do
+            if ! out="$(kubectl apply -k "$k" --dry-run=server 2>&1)"; then
+              failed=1
+              echo "::error file=${k}::$(printf '%s' "$out" | head -1)"
+            fi
+          done
+
+          if [ "$failed" -ne 0 ]; then
+            echo "Server-side validation failed. The API server (or an admission webhook) rejected these manifests."
+            exit 1
+          fi
+          echo "server-side dry-run: all active manifests accepted by the API server"
+
   build:
     needs:
-      - checks
+      # The panel's scan-deps/test-backend/test-frontend jobs gated here until
-    if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main'
+      # userbot moved to its own repo; upstream's code is upstream's gate now.
-    runs-on: homelab
+      # The rule is unchanged: publishing and passing the checks are the same
+      # gate, so a commit that fails any of these still cannot move :prod.
+      [lint-actionlint, lint-shellcheck, lint-compose, lint-prettier, lint-ruff, lint-yaml, lint-dockerfiles, validate]
+    if: github.event_name != 'pull_request' && (github.ref_name == 'main' || github.ref_name == 'dev') && !startsWith(github.ref_name, 'renovate/')
+    runs-on: [self-hosted, linux, arch, homelab]
     timeout-minutes: 60
+    outputs:
+      services: ${{ steps.services.outputs.services }}
     steps:
       - name: Checkout repository
-        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
         with:
           fetch-depth: 0
-      - name: Build changed images and write release
+
+      - name: Detect changed docker-built services
+        id: services
+        shell: bash
+        run: |
+          set -euo pipefail
+          base="${{ github.event.before }}"
+          if [ -z "$base" ] || [ "$base" = "0000000000000000000000000000000000000000" ]; then
+            base="$(git rev-list --max-parents=0 HEAD)"
+          fi
+
+          # A failed diff used to leave changed_files empty, which reads exactly
+          # like "nothing to build": the job went green having built nothing and
+          # the tag never moved. The status is checked, not assumed.
+          if ! changed="$(git diff --name-only "$base" "${GITHUB_SHA}")"; then
+            echo "::error::cannot diff ${base}..${GITHUB_SHA}"
+            exit 1
+          fi
+          mapfile -t changed_files <<<"$changed"
+
+          services=()
+
+          add_service() {
+            local name="$1"
+            local seen=0
+            for existing in "${services[@]}"; do
+              if [ "$existing" = "$name" ]; then
+                seen=1
+                break
+              fi
+            done
+            if [ "$seen" -eq 0 ]; then
+              services+=("$name")
+            fi
+          }
+
+          for file in "${changed_files[@]}"; do
+            case "$file" in
+              errorpages/*)
+                add_service errorpages
+                ;;
+              homepages/*)
+                add_service homepages
+                ;;
+              edu_master/phpsessid-bot/*|edu_master/webinar-checker/*|edu_master/compose.yaml)
+                add_service edu_master
+                ;;
+            esac
+          done
+
+          if [ "${#services[@]}" -eq 0 ]; then
+            echo "No docker-built services changed."
+            echo "services=" >> "$GITHUB_OUTPUT"
+            exit 0
+          fi
+
+          printf '%s\n' "${services[@]}" | tee /tmp/services.txt
+          echo "services=$(paste -sd, /tmp/services.txt)" >> "$GITHUB_OUTPUT"
+
+      - name: Log in to registry
+        # The pin step below also writes (manifest PUTs), and it runs on every
+        # main push — including manifest-only ones where services is empty. A
+        # stale persistent login on the old runner used to mask this; a clean
+        # runner pushes anonymously and gets 401.
+        if: steps.services.outputs.services != '' || github.ref_name == 'main'
+        shell: bash
+        # Through env, not by substitution into the script. A secret written
+        # into a run: block is pasted into the shell source before bash parses
+        # it, so a password containing a quote, a backtick or $(...) becomes
+        # code that runs. Masking the value in the log does not prevent that.
         env:
-          GITEA_TOKEN: ${{ github.token }}
           REGISTRY_USERNAME: ${{ secrets.REGISTRY_USERNAME }}
           REGISTRY_PASSWORD: ${{ secrets.REGISTRY_PASSWORD }}
-        run: python3 .gitea/workflows/release.py build
+        run: |
-      - name: Store commit release
+          set -euo pipefail
-        uses: actions/upload-artifact@c6a366c94c3e0affe28c06c8df20a878f24da3cf
+          printf '%s' "$REGISTRY_PASSWORD" | docker login "${REGISTRY}" \
-        with:
+            -u "$REGISTRY_USERNAME" \
-          name: release-${{ github.sha }}
+            --password-stdin
-          path: release.json
+
-          if-no-files-found: error
+      - name: Build and push changed images
-          retention-days: 30
+        if: steps.services.outputs.services != ''
+        shell: bash
+        run: |
+          # This step was the one run: block in the workflow without it, and it
+          # is the one that cannot afford it: a docker push that failed partway
+          # through the loop used to be followed by more pushes, the loop's exit
+          # status came from the last one, and the job went green with half the
+          # images missing from the registry.
+          set -euo pipefail
+          IFS=, read -r -a services <<< "${{ steps.services.outputs.services }}"
+
+          # Tags for this push. The commit-pinned name is the point of this
+          # step: the deploy resolves it in preference to :prod, so a deploy
+          # that sat in the queue behind a later push still gets the build of
+          # the commit CI validated, instead of whatever :prod points at by the
+          # time it runs. See render_pinned in deploy-lib.sh.
+          commit_tag=""
+          if [ "${GITHUB_REF_NAME}" = "main" ]; then
+            commit_tag="sha-${GITHUB_SHA:0:12}"
+          fi
+
+          set_tags() {
+            tags=()
+            case "${GITHUB_REF_NAME}" in
+              main) tags+=("main" "prod") ;;
+              dev) tags+=("dev") ;;
+            esac
+            if [ -n "$commit_tag" ]; then
+              tags+=("$commit_tag")
+            fi
+          }
+
+          for service in "${services[@]}"; do
+            case "$service" in
+              errorpages)
+                image="${REGISTRY}/forust/error-pages"
+                set_tags
+                build_args=()
+                for tag in "${tags[@]}"; do
+                  build_args+=(-t "${image}:${tag}")
+                done
+                docker build \
+                  --cache-from "type=registry,ref=${image}:buildcache" \
+                  --cache-to "type=registry,ref=${image}:buildcache,mode=max" \
+                  "${build_args[@]}" errorpages
+                for tag in "${tags[@]}"; do
+                  docker push "${image}:${tag}"
+                done
+                ;;
+              homepages)
+                for variant in forust xdfnx; do
+                  case "$variant" in
+                    forust)
+                      image="${REGISTRY}/forust/forust-homepage"
+                      ;;
+                    xdfnx)
+                      image="${REGISTRY}/forust/xdfnx-homepage"
+                      ;;
+                  esac
+                  set_tags
+                  build_args=()
+                  for tag in "${tags[@]}"; do
+                    build_args+=(-t "${image}:${tag}")
+                  done
+                  docker build \
+                    --cache-from "type=registry,ref=${image}:buildcache" \
+                    --cache-to "type=registry,ref=${image}:buildcache,mode=max" \
+                    "${build_args[@]}" -f "homepages/Dockerfile.${variant}" homepages
+                  for tag in "${tags[@]}"; do
+                    docker push "${image}:${tag}"
+                  done
+                done
+                ;;
+              edu_master)
+                for variant in session-keeper webinar-checker; do
+                  case "$variant" in
+                    session-keeper)
+                      context="edu_master/phpsessid-bot"
+                      image="${REGISTRY}/forust/session-keeper"
+                      ;;
+                    webinar-checker)
+                      context="edu_master/webinar-checker"
+                      image="${REGISTRY}/forust/webinar-checker"
+                      ;;
+                  esac
+                  set_tags
+                  build_args=()
+                  for tag in "${tags[@]}"; do
+                    build_args+=(-t "${image}:${tag}")
+                  done
+                  docker build \
+                    --cache-from "type=registry,ref=${image}:buildcache" \
+                    --cache-to "type=registry,ref=${image}:buildcache,mode=max" \
+                    "${build_args[@]}" "$context"
+                  for tag in "${tags[@]}"; do
+                    docker push "${image}:${tag}"
+                  done
+                done
+                ;;
+            esac
+          done
+
+      # Every image the tree names has to carry the commit-pinned name, not only
+      # the ones this push rebuilt. A push that touches nothing but manifests
+      # builds nothing, and its deploy would then find no commit-pinned tag to
+      # resolve and quietly fall back to the moving :prod - which is the whole
+      # failure the commit-pinned name exists to remove.
+      #
+      # Re-tagging copies the manifest list and transfers no layers, so pinning
+      # six images that already exist costs six registry writes.
+      #
+      # The list is derived from the tree rather than written out here, so an
+      # image added to a manifest is covered without a second place to update.
+      - name: Pin the commit name on the images this push did not rebuild
+        if: github.ref_name == 'main'
+        shell: bash
+        run: |
+          set -euo pipefail
+          commit_tag="sha-${GITHUB_SHA:0:12}"
+          mapfile -t repos < <(
+            git grep -hoE 'gcr\.forust\.xyz/forust/[A-Za-z0-9._-]+' -- '*.yaml' '*.yml' \
+              | sort -u
+          )
+          if [ "${#repos[@]}" -eq 0 ]; then
+            echo "No own images referenced by the tree."
+            exit 0
+          fi
+          echo "pinning ${#repos[@]} image(s) to $commit_tag"
+          for repo in "${repos[@]}"; do
+            if docker buildx imagetools inspect "$repo:$commit_tag" >/dev/null 2>&1; then
+              echo "  already built by this push: ${repo##*/}"
+              continue
+            fi
+            if ! docker buildx imagetools inspect "$repo:prod" >/dev/null 2>&1; then
+              echo "  WARNING: ${repo##*/} has no :prod to pin and no build produced it"
+              continue
+            fi
+            docker buildx imagetools create --tag "$repo:$commit_tag" "$repo:prod"
+            echo "  pinned ${repo##*/}"
+          done

.gitea/workflows/compose-lint.sh

@@ -21,7 +21,8 @@
 # All committed Compose files, including the ones deploy never starts.
 compose_files() {
   git ls-files \
-    '*compose.yaml' '*compose.yml'
+    '*/compose.yaml' '*/compose.yml' 'compose.yaml' 'compose.yml' \
+    '*/docker-compose.yaml' '*/docker-compose.yml'
 }
 
 # Prints the flags that turn `docker compose config` into the general check.

.gitea/workflows/compose-release.py

@@ -1,93 +0,0 @@
-#!/usr/bin/env python3
-"""Resolve Compose images without changing project names or local bind paths."""
-
-import json
-import os
-import re
-import subprocess
-import sys
-from pathlib import Path
-
-
-def output(*args, **kwargs):
-    return subprocess.check_output(args, text=True, **kwargs).strip()  # noqa: S603
-
-
-def resolve(reference):
-    if '@sha256:' in reference:
-        return reference
-    descriptor = json.loads(
-        output('docker', 'buildx', 'imagetools', 'inspect', reference, '--format', '{{json .Manifest}}')
-    )
-    digest = descriptor['digest']
-    if not re.fullmatch(r'sha256:[0-9a-f]{64}', digest):
-        raise ValueError(f'Invalid registry digest for {reference}')
-    # Strip tag only from the final path segment (registry ports are preserved).
-    repository = reference.rsplit('/', 1)
-    repository[-1] = repository[-1].split(':')[0]
-    return '/'.join(repository) + '@' + digest
-
-
-def prepare(source_file):
-    config_repo = Path(os.environ['CONFIG_REPO'])
-    source_repo = Path(os.environ['REPO'])
-    directory = Path(os.environ['RUN_DIR'])
-    relative = source_file.relative_to(source_repo)
-    project_dir = config_repo / relative.parent
-    base = ['docker', 'compose', '--project-directory', str(project_dir), '-f', str(source_file)]
-    config = json.loads(output(*base, 'config', '--format', 'json', cwd=config_repo))
-    project = config['name']
-    previous_file = directory / 'previous.json'
-    previous = json.loads(previous_file.read_text()) if previous_file.exists() else {}
-    images_file = directory / 'compose-images.json'
-    locks = json.loads(images_file.read_text()) if images_file.exists() else previous.get('compose-images', {})
-    release = json.loads((directory / 'release.json').read_text())
-    before = json.loads(json.dumps(config))
-    for service, settings in config['services'].items():
-        reference = settings.get('image')
-        if not reference or settings.get('build'):
-            raise ValueError(f'{project}/{service}: Compose deploy requires a published image')
-        image_repo = reference.split('@')[0].rsplit('/', 1)
-        image_repo[-1] = image_repo[-1].split(':')[0]
-        image_repo = '/'.join(image_repo)
-        if image_repo in release['images']:
-            pinned = image_repo + '@' + release['images'][image_repo]
-        elif os.environ.get('REFRESH_IMAGES') != 'true' and reference in locks:
-            pinned = locks[reference]
-        else:
-            pinned = resolve(reference)
-        settings['image'] = pinned
-        locks[reference] = pinned
-        # Capture what is running, not the current value of its mutable tag.
-        ids = output(
-            'docker',
-            'ps',
-            '-aq',
-            '--filter',
-            f'label=com.docker.compose.project={project}',
-            '--filter',
-            f'label=com.docker.compose.service={service}',
-        ).splitlines()
-        actual = set()
-        for container in ids:
-            image_id = output('docker', 'inspect', container, '--format', '{{.Image}}')
-            digests = json.loads(output('docker', 'image', 'inspect', image_id, '--format', '{{json .RepoDigests}}'))
-            actual.add(next((d for d in digests or [] if d.split('@')[0] == image_repo), image_id))
-        if len(actual) > 1:
-            raise ValueError(f'{project}/{service}: mixed running images, cannot capture one recovery config')
-        before['services'][service]['image'] = next(iter(actual)) if actual else reference
-    for name, data in (('compose', config), ('compose-before', before)):
-        folder = directory / name
-        folder.mkdir(mode=0o700, exist_ok=True)
-        destination = folder / f'{relative.parent.name}.json'
-        destination.write_text(json.dumps(data, indent=2) + '\n')
-        destination.chmod(0o600)
-    images_file.write_text(json.dumps(locks, indent=2) + '\n')
-    print(f'Compose {project}: images pinned; local paths preserved')
-    print(
-        f'Recovery: docker compose --project-directory {project_dir} -p {project} -f {directory}/compose-before/{relative.parent.name}.json up -d --pull never'
-    )
-
-
-if __name__ == '__main__':
-    prepare(Path(sys.argv[1]))

.gitea/workflows/deploy-controller.py

@@ -1,323 +0,0 @@
-#!/usr/bin/env python3
-"""Durable workstation deployment controller. Install with setup-workstation.sh."""
-
-import argparse
-import contextlib
-import fcntl
-import importlib.util
-import json
-import math
-import os
-import re
-import shutil
-import subprocess
-import sys
-import time
-from pathlib import Path
-
-STATE = Path(os.environ.get('HOMELAB_STATE', Path.home() / '.local/state/homelab-deploy'))
-CONFIG_REPO = Path(os.environ.get('HOMELAB_REPO', '/srv/homelab'))
-RUN_ID = re.compile(r'[0-9]+-[0-9]+')
-
-
-def command(*args, **kwargs):
-    return subprocess.check_output(args, text=True, **kwargs).strip()  # noqa: S603, S607
-
-
-def atomic_json(path, data):
-    temporary = path.with_suffix('.tmp')
-    temporary.write_text(json.dumps(data, indent=2) + '\n')
-    temporary.chmod(0o600)
-    temporary.replace(path)
-
-
-@contextlib.contextmanager
-def lock(name):
-    STATE.mkdir(mode=0o700, parents=True, exist_ok=True)
-    with (STATE / name).open('a') as stream:
-        fcntl.flock(stream, fcntl.LOCK_EX)
-        yield
-
-
-def load_module(name, path):
-    spec = importlib.util.spec_from_file_location(name, path)
-    module = importlib.util.module_from_spec(spec)
-    spec.loader.exec_module(module)
-    return module
-
-
-def run_directory(run_id):
-    if not RUN_ID.fullmatch(run_id):
-        raise ValueError('Run ID must be numeric workflow-id and attempt')
-    return STATE / 'runs' / run_id
-
-
-def start(run_id):
-    payload = sys.stdin.buffer.read(256 * 1024 + 1)
-    if len(payload) > 256 * 1024:
-        raise ValueError('Deploy request exceeds 256 KiB')
-    request = json.loads(payload)
-    sha = request['release']['sha']
-    if not re.fullmatch(r'[0-9a-f]{40}', sha) or request['mode'] not in ('changed', 'full', 'plan'):
-        raise ValueError('Invalid deploy SHA or mode')
-    if not isinstance(request['refresh_images'], bool):
-        raise ValueError('refresh_images must be boolean')
-    directory = run_directory(run_id)
-    with lock('prepare.lock'):
-        if (directory / 'request.json').exists():
-            if json.loads((directory / 'request.json').read_text()) != request:
-                raise ValueError('Run ID already belongs to a different request')
-        else:
-            directory.mkdir(mode=0o700, parents=True, exist_ok=True)
-            command('git', '-C', str(CONFIG_REPO), 'fetch', '--quiet', 'origin', 'main')
-            command('git', '-C', str(CONFIG_REPO), 'merge-base', '--is-ancestor', sha, 'origin/main')
-            if not (directory / 'source').exists():
-                command('git', '-C', str(CONFIG_REPO), 'worktree', 'add', '--detach', str(directory / 'source'), sha)
-            if command('git', '-C', str(directory / 'source'), 'rev-parse', 'HEAD') != sha:
-                raise ValueError('Prepared source does not match deploy SHA')
-            release_module = load_module('release', directory / 'source/.gitea/workflows/release.py')
-            release_module.validate_release(request['release'], sha)
-            atomic_json(directory / 'release.json', request['release'])
-            atomic_json(directory / 'request.json', request)
-        if not (directory / 'status.json').exists():
-            atomic_json(directory / 'status.json', {'state': 'queued', 'stages': {}})
-    # Starting an existing active or finished ID is idempotent; never re-apply it.
-    if json.loads((directory / 'status.json').read_text())['state'] == 'queued':
-        command('systemctl', '--user', 'start', '--no-block', f'homelab-deploy@{run_id}.service')
-    print(f'Accepted deploy {run_id} ({sha})')
-
-
-def environment(directory):
-    request = json.loads((directory / 'request.json').read_text())
-    return {
-        **os.environ,
-        'REPO': str(directory / 'source'),
-        'CONFIG_REPO': str(CONFIG_REPO),
-        'RUN_DIR': str(directory),
-        'DEPLOY_SHA': request['release']['sha'],
-        'RELEASE_FILE': str(directory / 'release.json'),
-        'DEPLOY_PLAN': str(directory / 'plan.json'),
-        'DEPLOY_SNAPSHOT_DIR': str(directory / 'snapshot'),
-        'REFRESH_IMAGES': str(request['refresh_images']).lower(),
-        'ROLLOUT_PARALLELISM': '4',
-    }
-
-
-def stage(directory, name, budget):
-    status = json.loads((directory / 'status.json').read_text())
-    if name in status['stages'] and status['stages'][name].get('result') in ('success', 'failure'):
-        return status['stages'][name]['result'] == 'success'
-    started = time.time()
-    status['stages'][name] = {'result': 'running', 'started': started}
-    atomic_json(directory / 'status.json', status)
-    script = directory / 'source/.gitea/workflows/deploy-stage.sh'
-    with (directory / f'{name}.log').open('a') as log:
-        # timeout kills the whole stage process group, including children, before recovery.
-        result = subprocess.run(  # noqa: S603, S607
-            [
-                shutil.which('timeout') or '/usr/bin/timeout',
-                '--signal=TERM',
-                '--kill-after=30s',
-                str(budget),
-                'bash',
-                str(script),
-                name,
-            ],
-            env=environment(directory),
-            stdout=log,
-            stderr=subprocess.STDOUT,
-            check=False,
-        ).returncode
-    status = json.loads((directory / 'status.json').read_text())
-    status['stages'][name].update(
-        result='success' if result == 0 else 'failure', exit_code=result, seconds=round(time.time() - started)
-    )
-    atomic_json(directory / 'status.json', status)
-    return result == 0
-
-
-def make_plan(directory):
-    source = directory / 'source'
-    planner = load_module('deploy_plan', source / '.gitea/workflows/deploy-plan.py')
-    request = json.loads((directory / 'request.json').read_text())
-    previous = json.loads((STATE / 'last-success.json').read_text()) if (STATE / 'last-success.json').exists() else None
-    helm = json.loads(command('helm', 'list', '--all', '-A', '-o', 'json'))
-    plan = planner.make_plan(source, CONFIG_REPO, request['release'], previous, request['mode'], helm)
-    if request['refresh_images']:
-        plan['selected']['compose'] = plan['active']['compose']
-    atomic_json(directory / 'plan.json', plan)
-    if previous:
-        atomic_json(directory / 'previous.json', previous)
-    # Local config is deliberately separate from the immutable Git source.
-    return plan
-
-
-def finish_success(directory, plan):
-    # Repeating finalization after a crash is safe while holding deploy.lock.
-    plan['run_id'] = directory.name
-    path = directory / 'compose-images.json'
-    previous = directory / 'previous.json'
-    plan['compose-images'] = (
-        json.loads(path.read_text())
-        if path.exists()
-        else json.loads(previous.read_text()).get('compose-images', {})
-        if previous.exists()
-        else {}
-    )
-    atomic_json(STATE / 'last-success.json', plan)
-    status = json.loads((directory / 'status.json').read_text())
-    status['state'] = 'success'
-    atomic_json(directory / 'status.json', status)
-    try:
-        retain_completed(directory)
-    except (OSError, subprocess.CalledProcessError) as error:
-        print(f'Retention deferred: {error}', flush=True)
-
-
-def recover(directory, retry=False):
-    status = json.loads((directory / 'status.json').read_text())
-    if status['state'] in ('success', 'planned'):
-        return
-    completed = ('doctor', 'validate', 'apply-k8s', 'apply-compose', 'verify-k8s', 'smoke')
-    if all(status['stages'].get(name, {}).get('result') == 'success' for name in completed):
-        finish_success(directory, json.loads((directory / 'plan.json').read_text()))
-        return
-    if retry:
-        for name in ('verify-k8s', 'smoke'):
-            if status['stages'].get(name, {}).get('result') == 'failure':
-                del status['stages'][name]
-        atomic_json(directory / 'status.json', status)
-    snapshot = directory / 'snapshot/current'
-    if snapshot.exists():
-        stage(directory, 'verify-k8s', 7200)
-        stage(directory, 'smoke', 600)
-    status = json.loads((directory / 'status.json').read_text())
-    status['state'] = 'failure'
-    atomic_json(directory / 'status.json', status)
-
-
-def execute(run_id):
-    directory = run_directory(run_id)
-    with lock('deploy.lock'):
-        status = json.loads((directory / 'status.json').read_text())
-        if status['state'] != 'queued':
-            return
-        # A crashed predecessor must be recovered before another apply begins.
-        for other in (STATE / 'runs').iterdir():
-            if (
-                other != directory
-                and (other / 'status.json').exists()
-                and json.loads((other / 'status.json').read_text())['state'] == 'running'
-            ):
-                raise ValueError(f'Interrupted deploy {other.name}; run recover first')
-        status['state'] = 'running'
-        atomic_json(directory / 'status.json', status)
-        try:
-            plan = make_plan(directory)
-            print(
-                json.dumps({'selected': plan['selected'], 'helm': plan['helm'], 'manual_removals': plan['removed']}),
-                flush=True,
-            )
-            if not stage(directory, 'doctor', 600):
-                raise RuntimeError('Preflight failed')
-            if not stage(directory, 'validate', 1200):
-                raise RuntimeError('Validation failed')
-            if json.loads((directory / 'request.json').read_text())['mode'] == 'plan':
-                status = json.loads((directory / 'status.json').read_text())
-                status['state'] = 'planned'
-                atomic_json(directory / 'status.json', status)
-                return
-            # Budget includes both rollout checks and rollback waves, plus API overhead.
-            count = int(
-                command(
-                    'bash',
-                    str(directory / 'source/.gitea/workflows/deploy-stage.sh'),
-                    'workload-count',
-                    env=environment(directory),
-                )
-            )
-            verify_budget = max(600, 2 * math.ceil(count / 4) * 300 + 120)
-            if verify_budget > 7200:
-                raise ValueError('More than two hours of recovery required; split this deploy')
-            k8s_ok = stage(directory, 'apply-k8s', 2700)
-            compose_ok = stage(directory, 'apply-compose', 1800) if k8s_ok else False
-            verify_ok = stage(directory, 'verify-k8s', verify_budget)
-            smoke_ok = stage(directory, 'smoke', 600)
-            if not all((k8s_ok, compose_ok, verify_ok, smoke_ok)):
-                raise RuntimeError('Deploy failed; inspect stage logs and recovery report')
-            finish_success(directory, plan)
-        except Exception as error:
-            with (directory / 'controller.log').open('a') as stream:
-                stream.write(f'{error}\n')
-            recover(directory)
-            raise
-
-
-def retain_completed(current):
-    finished = []
-    for directory in (STATE / 'runs').iterdir():
-        status_file = directory / 'status.json'
-        if status_file.exists() and json.loads(status_file.read_text())['state'] in ('success', 'planned'):
-            finished.append(directory)
-    for directory in sorted(finished, key=lambda p: p.stat().st_mtime, reverse=True)[20:]:
-        if directory == current:
-            continue
-        command('git', '-C', str(CONFIG_REPO), 'worktree', 'remove', '--force', str(directory / 'source'))
-        shutil.rmtree(directory)
-
-
-def follow(run_id, phase):
-    directory = run_directory(run_id)
-    groups = {
-        'apply': ('doctor', 'validate', 'apply-k8s', 'apply-compose'),
-        'verify': ('verify-k8s',),
-        'smoke': ('smoke',),
-    }
-    names = groups[phase]
-    offsets = {}
-    while True:
-        status = json.loads((directory / 'status.json').read_text())
-        for name in (*names, 'controller'):
-            path = directory / f'{name}.log'
-            if path.exists():
-                with path.open() as stream:
-                    stream.seek(offsets.get(name, 0))
-                    content = stream.read()
-                    if content:
-                        print(content, end='', flush=True)
-                    offsets[name] = stream.tell()
-        stages = status['stages']
-        if all(stages.get(name, {}).get('result') in ('success', 'failure') for name in names):
-            return all(stages[name]['result'] == 'success' for name in names)
-        if status['state'] in ('success', 'failure', 'planned'):
-            return status['state'] in ('success', 'planned')
-        time.sleep(3)
-
-
-def main():
-    os.umask(0o077)
-    parser = argparse.ArgumentParser(description=__doc__)
-    parser.add_argument('action', choices=('start', 'execute', 'recover', 'status', 'follow'))
-    parser.add_argument('run_id')
-    parser.add_argument('phase', nargs='?', choices=('apply', 'verify', 'smoke'))
-    parser.add_argument('--retry', action='store_true', help='Retry failed recovery checks; never repeat apply')
-    args = parser.parse_args()
-    directory = run_directory(args.run_id)
-    if args.action == 'start':
-        start(args.run_id)
-    elif args.action == 'execute':
-        execute(args.run_id)
-    elif args.action == 'recover':
-        with lock('deploy.lock'):
-            recover(directory, retry=args.retry)
-    elif args.action == 'status':
-        print((directory / 'status.json').read_text())
-        if (directory / 'plan.json').exists():
-            plan = json.loads((directory / 'plan.json').read_text())
-            print(json.dumps({k: plan[k] for k in ('sha', 'selected', 'helm', 'removed')}, indent=2))
-    elif not follow(args.run_id, args.phase):
-        sys.exit(1)
-
-
-if __name__ == '__main__':
-    main()

.gitea/workflows/deploy-lib.sh

@@ -1,5 +1,5 @@
 #!/usr/bin/env bash
-# Workstation deploy stages; invoked by the durable controller against pinned source.
+# Shared stages for the deploy workflow. Runs on the workstation, invoked as:
 #   REPO=/srv/homelab APPLY_PRUNE=false bash -se <<'EOF'
 #   source "$REPO/.gitea/workflows/deploy-lib.sh"
 #   run_stage "$STAGE"
@@ -8,8 +8,8 @@ set -euo pipefail
 
 : "${REPO:?REPO must be set}"
 APPLY_PRUNE="${APPLY_PRUNE:-false}"
-CONFIG_REPO="${CONFIG_REPO:-$REPO}"
+# Commit CI validated. Empty for a manual workflow_dispatch, which falls back to
-# Exact SHA accepted by the CI gate for both manual and automatic deploys.
+# the current origin/main.
 DEPLOY_SHA="${DEPLOY_SHA:-}"
 # Handoff point between the apply stage (writes) and the verify stage (reads).
 # Under the deploy user's own XDG state directory rather than /var/backups: the
@@ -20,7 +20,7 @@ DEPLOY_SNAPSHOT_DIR="${DEPLOY_SNAPSHOT_DIR:-${XDG_STATE_HOME:-$HOME/.local/state
 # Per-workload rollout budget and how many workloads to watch at once. The whole
 # apply job has its own timeout-minutes as a backstop.
 ROLLOUT_TIMEOUT="${ROLLOUT_TIMEOUT:-300}"
-ROLLOUT_PARALLELISM="${ROLLOUT_PARALLELISM:-4}"
+ROLLOUT_PARALLELISM="${ROLLOUT_PARALLELISM:-8}"
 WORKLOAD_KINDS="deployments.apps,statefulsets.apps,daemonsets.apps"
 
 log() {
@@ -31,16 +31,6 @@ warn() {
   echo "WARNING: $*" >&2
 }
 
-# Prune needs the complete desired set in one invocation. Per-file pruning
-# treats resources from the other files as absent and can delete them.
-check_prune_mode() {
-  if [ "$APPLY_PRUNE" = "true" ]; then
-    echo "ERROR: APPLY_PRUNE=true is unsupported by the per-file deploy loop." >&2
-    echo "Disable it; remove obsolete resources explicitly after review." >&2
-    return 1
-  fi
-}
-
 collect_k8s() {
   git -C "$REPO" ls-files -- "$1" \
     | grep -E '\.ya?ml$' \
@@ -60,25 +50,6 @@ kustomize_overlay() {
   fi
 }
 
-selected_service() {
-  local kind="$1" service="$2" section=selected
-  [ -n "${DEPLOY_PLAN:-}" ] || return 0
-  if [ "${DEPLOY_SMOKE_ALL:-false}" = true ]; then section=active; fi
-  jq -e --arg kind "$kind" --arg service "$service" --arg section "$section" \
-    '.[$section][$kind] | index($service) != null' "$DEPLOY_PLAN" >/dev/null
-}
-
-# Resolve .env and relative binds on the persistent workstation tree. Locked
-# JSON configs keep the same Compose project name and volume names.
-compose() {
-  local cf="$1" locked project_dir
-  project_dir="$CONFIG_REPO/$(basename "$(dirname "$cf")")"
-  shift
-  locked="${RUN_DIR:-/nonexistent}/compose/$(basename "$(dirname "$cf")").json"
-  if [ -f "$locked" ]; then cf="$locked"; fi
-  (cd "$CONFIG_REPO" && docker compose --project-directory "$project_dir" -f "$cf" "$@")
-}
-
 select_manifests() {
   K8S_MANIFESTS=()
   KUSTOMIZE_APPS=()
@@ -86,7 +57,6 @@ select_manifests() {
   local kd_rel kd overlay cf_rel cf f
   while IFS= read -r kd_rel; do
     kd="$REPO/$kd_rel"
-    selected_service k8s "${kd_rel%/k8s}" || continue
     if [ ! -f "$kd/active" ]; then
       echo "skip (no k8s/active): $kd_rel"
       continue
@@ -108,7 +78,6 @@ select_manifests() {
   )
   while IFS= read -r cf_rel; do
     cf="$REPO/$cf_rel"
-    selected_service compose "$(dirname "$cf_rel")" || continue
     if [ -f "$(dirname "$cf")/active" ]; then
       echo "compose: $cf_rel"
       COMPOSE_STACKS+=("$cf")
@@ -125,10 +94,16 @@ select_manifests() {
 # on failure roll them back to the revision that was running before, so a bad
 # push to main cannot leave a service crash-looping.
 #
-# The workstation controller runs apply and verification as separate durable
+# Verification lives in its own workflow job, not at the end of the apply stage.
-# stages. Runner jobs only follow their logs. ExecStopPost recovers interrupted
+# Inside a single process it is worthless exactly when it is needed most: a job
-# runs using the per-run snapshot, even when the SSH connection has gone away.
+# killed by timeout-minutes or cancelled mid-apply never reaches the rollback
+# code, and leaves a half-applied cluster behind. Split out, the apply job can
+# die in any way and the verify job still runs.
 #
+# That split needs a handoff point on the workstation, because the two stages are
+# separate processes on separate runner jobs: DEPLOY_SNAPSHOT_DIR/current, written
+# before anything is applied, read by the verify stage afterwards.
+
 # Creates this run's snapshot directory and publishes it as the handoff point for
 # the verify stage. Fails hard by design: a deploy that cannot record what it is
 # about to change must not start, because then nothing can be rolled back for it
@@ -158,35 +133,18 @@ snapshot_dir() {
 }
 
 save_snapshot() {
-  local dir="$1" releases revision status
+  local dir="$1"
   log "Saving pre-apply snapshot to $dir"
-  workload_generations >"$dir/generations.before" || return 1
+  workload_generations >"$dir/generations.before" 2>/dev/null \
-  kubectl get "$WORKLOAD_KINDS" -A -o json >"$dir/workloads.json" || return 1
+    || warn "could not snapshot workload generations"
-  kubectl get controllerrevisions.apps -A -o json >"$dir/controller-revisions.json" || return 1
+  kubectl get "$WORKLOAD_KINDS" -A -o yaml >"$dir/workloads.yaml" 2>/dev/null \
-  jq --slurpfile revisions "$dir/controller-revisions.json" '
+    || warn "could not snapshot workloads"
-    [.items[] | . as $w | {
+  for release in prometheus-stack loki alloy; do
-      kind: (.kind | ascii_downcase), namespace: .metadata.namespace, name: .metadata.name, uid: .metadata.uid,
+    if helm status "$release" -n prometheus >/dev/null 2>&1; then
-      revision: (if .kind == "Deployment" then (.metadata.annotations["deployment.kubernetes.io/revision"] // "0" | tonumber)
+      {
-        else ([$revisions[0].items[] | select(.metadata.namespace == $w.metadata.namespace)
+        echo "revision: $(helm history "$release" -n prometheus -o json 2>/dev/null)"
-          | select(any(.metadata.ownerReferences[]?; .uid == $w.metadata.uid))
+        helm get values "$release" -n prometheus --all 2>/dev/null
-          | select($w.kind != "StatefulSet" or .metadata.name == $w.status.currentRevision) | .revision] | max // 0) end)
+      } >"$dir/helm-$release.txt"
-    }]' "$dir/workloads.json" >"$dir/revisions.json" || return 1
-  releases="$(helm list --all -A -o json)" || return 1
-  for entry in "${HELM_RELEASES[@]}"; do
-    IFS='|' read -r release _ namespace _ _ _ <<<"$entry"
-    if ! jq -e --arg r "$release" --arg n "$namespace" \
-      'any(.[]; .name == $r and .namespace == $n)' <<<"$releases" >/dev/null; then
-      continue
-    fi
-    helm status "$release" -n "$namespace" -o json >"$dir/helm-$release.json" || return 1
-    status="$(jq -r '.info.status' "$dir/helm-$release.json")"
-    if [ "$status" != deployed ]; then
-      # Never capture a pending/failed revision as the recovery target.
-      helm history "$release" -n "$namespace" -o json >"$dir/helm-$release.history.json" || return 1
-      revision="$(jq '[.[] | select(.status == "deployed" or .status == "superseded") | .revision] | max // 0' \
-        "$dir/helm-$release.history.json")"
-      jq --argjson revision "$revision" '.version = $revision' "$dir/helm-$release.json" >"$dir/helm-$release.tmp"
-      mv "$dir/helm-$release.tmp" "$dir/helm-$release.json"
     fi
   done
   # The verify stage compares this against the commit it is deploying, to refuse
@@ -210,24 +168,294 @@ workload_generations() {
 # moved since the snapshot, i.e. the ones this apply actually touched.
 changed_workloads() {
   local before="$1"
-  local ns name kind gen old current
+  local ns name kind gen old
-  current="$(workload_generations)" || return 1
   while read -r ns name kind gen; do
     [ -n "${gen:-}" ] || continue
-    old="$(awk -v want_ns="$ns" -v want_name="$name" -v want_kind="$kind" \
+    old="$(awk -v want_ns="$ns" -v want_name="$name" \
-      '$1 == want_ns && $2 == want_name && $3 == want_kind { print $4; exit }' "$before" 2>/dev/null || true)"
+      '$1 == want_ns && $2 == want_name { print $4; exit }' "$before" 2>/dev/null || true)"
-    if [ -n "${RUN_DIR:-}" ] && ! grep -qxF "$kind $ns $name" "$RUN_DIR/workload-refs"; then
-      continue
-    fi
     if [ "$old" != "$gen" ]; then
       printf '%s %s %s\n' "$kind" "$ns" "$name"
     fi
-  done <<<"$current"
+  done < <(workload_generations)
 }
 
-# Resolve owned image references exclusively from the checked CI artifact.
+# Prints "<ns> <kind>/<name> <image>" for every workload this repository owns that
+# runs an image from our own registry.
+#
+# The repository is the scope, deliberately. The cluster also holds workloads on
+# our registry that no manifest here declares (they are applied out of band), and
+# those are somebody else's to deploy. Walking the manifests rather than the
+# cluster means those can never be restarted by this pipeline, now or later.
+owned_registry_workloads() {
+  local kd_rel f
+  while IFS= read -r kd_rel; do
+    [ -f "$REPO/$kd_rel/active" ] || continue
+    while IFS= read -r f; do
+      [ -n "$f" ] || continue
+      # A file that does not mention the registry cannot declare a workload on it,
+      # and parsing costs ~2.5s per file against a millisecond for the grep. The
+      # filter keeps this at a handful of parses instead of one per manifest.
+      grep -q 'gcr\.forust\.xyz/forust/' "$REPO/$f" 2>/dev/null || continue
+      # kubectl prints a bare object for a single-document file and a List for a
+      # multi-document one, so normalise both shapes before filtering.
+      kubectl apply --dry-run=client -f "$REPO/$f" -o json 2>/dev/null \
+        | jq -r '
+            (if .items then .items[] else . end)
+            | select(.kind | test("^(Deployment|StatefulSet|DaemonSet)$"))
+            | select(any((.spec.template.spec.containers // [])[]?;
+                       (.image // "") | test("^gcr\\.forust\\.xyz/forust/")))
+            | (.metadata.namespace // "default") as $ns
+            | ([.spec.template.spec.containers[].image
+                | select(test("^gcr\\.forust\\.xyz/forust/"))][0]) as $img
+            | "\($ns) \(.kind | ascii_downcase)/\(.metadata.name) \($img)"
+          ' 2>/dev/null || true
+    done < <(collect_k8s "$kd_rel" || true)
+  done < <(
+    git -C "$REPO" ls-files '*.yaml' '*.yml' \
+      | grep -E '(^|/)k8s/' \
+      | sed -E 's#((^|.*/)k8s)/.*#\1#' \
+      | sort -u
+  )
+}
+
+# Prints the digest an image tag resolves to for this cluster's architecture, or
+# nothing when it cannot be resolved.
+#
+# Only the manifest entry matching the node architecture counts. A multi-arch tag
+# also carries `unknown/unknown` entries for the build attestation, and a pod's
+# imageID is always the per-platform digest, so comparing the wrong entry would
+# mark every workload stale forever and restart the whole cluster on every deploy.
+registry_digest() {
+  local arch
+  arch="$(kubectl get nodes -o jsonpath='{.items[0].status.nodeInfo.architecture}' 2>/dev/null || true)"
+  [ -n "$arch" ] || arch=amd64
+  # The || true is load-bearing. Every caller runs under set -euo pipefail, and
+  # pipefail reports the rightmost non-zero stage, so a ref the registry does not
+  # have would abort the caller at the assignment instead of yielding an empty
+  # string. The callers check for empty themselves and report it by name.
+  #
+  # Retried with a hard timeout because the registry has a known hang mode (and
+  # a known blink mode: a single failed lookup aborts the whole apply file in
+  # render_pinned). A short sleep between attempts lets a restarting registry
+  # come back instead of failing the deploy on one bad second.
+  local attempt=0 digest=""
+  while [ "$attempt" -lt 3 ]; do
+    digest="$(timeout 25s docker manifest inspect "$1" 2>/dev/null \
+      | jq -r --arg arch "$arch" '
+          .manifests[]?
+          | select(.platform.os == "linux" and .platform.architecture == $arch)
+          | .digest
+        ' 2>/dev/null \
+      | head -1 || true)"
+    [ -n "$digest" ] && break
+    attempt=$((attempt + 1))
+    if [ "$attempt" -lt 3 ]; then
+      echo "WARNING: registry lookup of $1 failed (attempt $attempt/3), retrying in 5s" >&2
+      sleep 5
+    fi
+  done
+  printf '%s' "$digest"
+}
+
+# The commit this deploy is for: what CI validated, or - on a manual dispatch,
+# whatever stage_preflight just checked out.
+deploy_commit() {
+  local c="${DEPLOY_SHA:-}"
+  [ -n "$c" ] || c="$(git -C "$REPO" rev-parse HEAD 2>/dev/null || true)"
+  printf '%.12s' "${c:-}"
+}
+
+# Resolves one of our image refs to the digest THIS commit's build produced.
+#
+# A manifest naming `:prod` names a pointer, not a version, and the deploy
+# resolves it when the apply runs - which is not when CI ran it. Deploy runs are
+# queued rather than cancelled (see deploy.yaml), so two pushes in a row leave
+# the first deploy resolving the second push's build: the right manifests with
+# the wrong code, and nothing anywhere reports it. ci therefore publishes every
+# image it ships under `sha-<commit12>`, a name that cannot move, and that is
+# the name resolved here.
+#
+# The fallback to the plain tag is for an image this pipeline never built. It
+# reports itself, because a fallback nobody sees is the failure this removes.
+pinned_digest() {
+  local ref="$1" commit pinned
+  commit="$(deploy_commit)"
+  if [ -n "$commit" ]; then
+    pinned="$(registry_digest "${ref%:*}:sha-$commit")"
+    if [ -n "$pinned" ]; then
+      printf '%s' "$pinned"
+      return 0
+    fi
+  fi
+  pinned="$(registry_digest "$ref")"
+  if [ -n "$pinned" ]; then
+    echo "WARNING: ${ref} carries no sha-${commit:-<unknown>} tag; resolved the moving tag instead" >&2
+  fi
+  printf '%s' "$pinned"
+}
+
+# Rewrites our own images to immutable digests on the way into the cluster.
+# Reads a manifest stream on stdin, writes the pinned stream to stdout.
+#
+# A digest is not knowable when a manifest is written, so it is never committed:
+# git keeps a readable `:prod` tag and the exact bytes are chosen here, at apply
+# time, from the tag ci published for the commit being deployed. That is what
+# makes rollback mean something. `kubectl rollout undo` restores the previous
+# ReplicaSet's pod template verbatim, and a template naming a digest restores the
+# exact bytes that were serving before. A template naming a moving tag does not —
+# the tag has already moved by the time the rollback runs, so the "rollback"
+# re-pulls the very image that just failed and the cluster stays broken.
+#
+# imagePullPolicy is deliberately left alone. The manifests no longer set it, and a
+# reference that is not `:latest` defaults to IfNotPresent, which is what the
+# Kubernetes docs ask for alongside a digest: the bytes under a digest cannot
+# change, so pulling again buys nothing.
+#
+# An image that cannot be resolved is fatal. Carrying on would quietly apply a
+# mutable tag again, which is the exact failure this function exists to remove.
 render_pinned() {
-  python3 "$REPO/.gitea/workflows/release.py" render
+  local src refs map ref digest missing=0
+  src="$(mktemp)"
+  refs="$(mktemp)"
+  map="$(mktemp)"
+
+  cat >"$src"
+  grep -oE 'gcr\.forust\.xyz/forust/[A-Za-z0-9._-]+:[A-Za-z0-9._-]+' "$src" | sort -u >"$refs" || true
+
+  while read -r ref; do
+    [ -n "$ref" ] || continue
+    digest="$(pinned_digest "$ref")"
+    if [ -z "$digest" ]; then
+      echo "ERROR: cannot resolve ${ref} in the registry; applying nothing." >&2
+      echo "       The build job has to push that tag before the deploy resolves it." >&2
+      missing=$((missing + 1))
+      continue
+    fi
+    printf '%s\t%s\n' "$ref" "$digest" >>"$map"
+  done <"$refs"
+  if [ "$missing" -gt 0 ]; then
+    rm -f "$src" "$refs" "$map"
+    return 1
+  fi
+
+  awk -v mapfile="$map" '
+    BEGIN {
+      while ((getline line < mapfile) > 0) {
+        i = index(line, "\t")
+        d[substr(line, 1, i - 1)] = substr(line, i + 1)
+      }
+    }
+    {
+      if (match($0, /^[[:space:]]*image:[[:space:]]*gcr\.forust\.xyz\/forust\/[A-Za-z0-9._-]+:[A-Za-z0-9._-]+[[:space:]]*$/)) {
+        name = $0
+        sub(/^[[:space:]]*image:[[:space:]]*/, "", name)
+        sub(/[[:space:]]*$/, "", name)
+        if (name in d) {
+          pad = $0
+          sub(/image:.*/, "", pad)
+          # Drop the tag: the canonical form used in the docs is repo@sha256:...,
+          # and leaving :prod next to the digest reads like it still matters.
+          repo = name
+          sub(/:[A-Za-z0-9._-]+$/, "", repo)
+          print pad "image: " repo "@" d[name]
+          next
+        }
+      }
+      print
+    }
+  ' "$src"
+  rm -f "$src" "$refs" "$map"
+}
+
+# Restarts every owned workload whose running image is not the one its tag
+# resolves to now.
+#
+# This used to be how a rebuild reached the cluster at all: the manifests pinned
+# `:latest`, so a rebuild left the pod template byte-identical, `kubectl apply`
+# decided there was nothing to do, and the cluster served the previous build
+# indefinitely. The apply now pins digests via render_pinned, so a rebuild moves
+# the pod template and rolls out on its own.
+#
+# What is left is the drift check: a hand-run `kubectl set image`, or anything
+# else that edits a live workload behind the deploy's back, is the only way to end
+# up serving a digest the tag has moved past. It stays idempotent, so a redeploy
+# that changed no image still does not bounce healthy services.
+#
+# The container is matched on its repository rather than on the exact reference:
+# once render_pinned has run, a pod's status reports `repo@sha256:...` while this
+# still reads the repository's `:prod` tag out of the manifest.
+restart_stale_images() {
+  local ns target image want selector running entry one
+  local unchecked=0
+  local -A digests=()
+  local -a stale=()
+  while read -r ns target image; do
+    [ -n "${target:-}" ] || continue
+    if [ -z "${digests[$image]:-}" ]; then
+      digests[$image]="$(pinned_digest "$image")"
+    fi
+    want="${digests[$image]}"
+    if [ -z "$want" ]; then
+      warn "cannot resolve ${image##*/} in the registry, leaving $target alone"
+      unchecked=$((unchecked + 1))
+      continue
+    fi
+    selector="$(kubectl get "$target" -n "$ns" -o jsonpath='{.spec.selector.matchLabels}' 2>/dev/null \
+      | jq -r 'to_entries | map("\(.key)=\(.value)") | join(",")' 2>/dev/null)"
+    if [ -z "$selector" ]; then
+      warn "cannot read the pod selector of $target, skipping"
+      unchecked=$((unchecked + 1))
+      continue
+    fi
+    running="$(kubectl get pods -n "$ns" -l "$selector" -o json 2>/dev/null \
+      | jq -r --arg repo "${image%%:*}" '
+          .items[] | .status.containerStatuses[]?
+          | select(.image == $repo
+                   or (.image | startswith($repo + ":"))
+                   or (.image | startswith($repo + "@")))
+          | .imageID
+        ' 2>/dev/null)"
+    if [ -z "$running" ]; then
+      # Scaled to zero. Nothing is serving stale code, and imagePullPolicy
+      # resolves the tag when it is scaled back up.
+      continue
+    fi
+    entry=""
+    while IFS= read -r one; do
+      [ -n "$one" ] || continue
+      entry="${one##*@}"
+      if [ "$entry" != "$want" ]; then
+        stale+=("$ns $target")
+        break
+      fi
+    done <<<"$running"
+  done < <(owned_registry_workloads)
+  if [ "${#stale[@]}" -eq 0 ]; then
+    if [ "$unchecked" -gt 0 ]; then
+      # Say so plainly. Reporting "everything is current" after checking nothing
+      # would tell the operator the deploy is fine when it may not be.
+      warn "No workload needed a restart, but $unchecked could not be checked"
+    else
+      log "All owned workloads already run the image their tag points at"
+    fi
+    return 0
+  fi
+  log "Restarting ${#stale[@]} workload(s) running an image their tag has moved past"
+  for ref in "${stale[@]}"; do
+    log "  $ref"
+  done
+  local failed=()
+  for ref in "${stale[@]}"; do
+    ns="${ref%% *}"
+    target="${ref#* }"
+    if ! kubectl rollout restart "$target" -n "$ns" >/dev/null 2>&1; then
+      failed+=("$ref")
+    fi
+  done
+  if [ "${#failed[@]}" -gt 0 ]; then
+    warn "could not restart: ${failed[*]}"
+    return 1
+  fi
 }
 
 # verify_workloads <failed-file> <kind> <ns> <name> ...
@@ -271,44 +499,31 @@ verify_workloads() {
 # settle. Prints a report and returns non-zero if any workload is still unhealthy,
 # so the operator knows manual recovery is required.
 rollback_workloads() {
-  local failed_file="$1" snapshot kind ns name index=0 running=0 pid revision uid
+  local failed_file="$1"
-  local -a pids=()
+  local kind ns name unrecovered=()
-  snapshot="$(cat "$DEPLOY_SNAPSHOT_DIR/current")"
+  local -a recovered=()
   while read -r kind ns name; do
-    [[ "$kind" =~ ^(deployment|statefulset|daemonset)$ ]] || continue
+    [ -n "${kind:-}" ] || continue
-    index=$((index + 1))
+    # Helm-owned workloads are already rolled back by the release's --rollback-on-failure
-    (
+    # upgrade. `rollout undo` here would step back to the revision Helm just
-      if kubectl get "$kind/$name" -n "$ns" -o jsonpath='{.metadata.annotations}' | grep -q 'meta.helm.sh/release-name'; then
+    # escaped (the failed one), so leave them for the operator instead.
-        echo "  skip (Helm recovery owns this workload): $kind/$ns/$name"
+    if kubectl get "${kind}/${name}" -n "$ns" -o jsonpath='{.metadata.annotations}' 2>/dev/null | grep -q 'meta.helm.sh/release-name'; then
-        exit 1
+      echo "  skip (helm-managed, needs manual check): ${kind}/${ns}/${name}"
-      fi
+      unrecovered+=("${kind}/${ns}/${name} (helm-managed)")
-      revision="$(jq -r --arg ns "$ns" --arg name "$name" --arg kind "$kind" \
+      continue
-        '.[] | select(.namespace == $ns and .name == $name and .kind == $kind) | .revision' "$snapshot/revisions.json")"
+    fi
-      uid="$(jq -r --arg ns "$ns" --arg name "$name" --arg kind "$kind" \
+    if kubectl rollout undo "${kind}/${name}" -n "$ns" >/dev/null 2>&1 \
-        '.[] | select(.namespace == $ns and .name == $name and .kind == $kind) | .uid' "$snapshot/revisions.json")"
+      && kubectl rollout status "${kind}/${name}" -n "$ns" --timeout="${ROLLOUT_TIMEOUT}s" >/dev/null 2>&1; then
-      if [[ ! "$revision" =~ ^[1-9][0-9]*$ ]] || [ "$uid" != "$(kubectl get "$kind/$name" -n "$ns" -o jsonpath='{.metadata.uid}')" ]; then
+      echo "  rolled back: ${kind}/${ns}/${name}"
-        echo "  no safe previous revision: $kind/$ns/$name (new or replaced workload)"
+      recovered+=("${kind}/${ns}/${name}")
-        exit 1
+    else
-      fi
+      echo "  NOT RECOVERED: ${kind}/${ns}/${name}"
-      kubectl rollout undo "$kind/$name" -n "$ns" --to-revision="$revision" \
+      unrecovered+=("${kind}/${ns}/${name}")
-        && kubectl rollout status "$kind/$name" -n "$ns" --timeout="${ROLLOUT_TIMEOUT}s"
-    ) >"$snapshot/rollback-$index.log" 2>&1 &
-    pids+=($!)
-    running=$((running + 1))
-    if [ "$running" -ge "$ROLLOUT_PARALLELISM" ]; then
-      wait -n 2>/dev/null || true
-      running=$((running - 1))
     fi
   done <"$failed_file"
-  local recovered=0 unrecovered=0 i=0
+  echo "ROLLED_BACK=${#recovered[@]}" >>"$failed_file"
-  for pid in "${pids[@]}"; do
+  echo "UNRECOVERED=${#unrecovered[@]}" >>"$failed_file"
-    i=$((i + 1))
+  [ "${#unrecovered[@]}" -eq 0 ]
-    if wait "$pid"; then recovered=$((recovered + 1)); else unrecovered=$((unrecovered + 1)); fi
-    cat "$snapshot/rollback-$i.log"
-  done
-  echo "ROLLED_BACK=$recovered" >>"$failed_file"
-  echo "UNRECOVERED=$unrecovered" >>"$failed_file"
-  [ "$unrecovered" -eq 0 ]
 }
 
 # Helm releases owned by this stage, one line each:
@@ -321,7 +536,6 @@ rollback_workloads() {
 # have to be declared as custom.regex managers in renovate/renovate.json.
 HELM_RELEASES=(
   "prometheus-stack|prometheus-community/kube-prometheus-stack|prometheus|86.2.3|prometheus-stack/k8s/grafana-values.yaml|prometheus-stack/k8s/active"
-  "victoria-operator|victoriametrics/victoria-metrics-operator|prometheus|0.68.1|prometheus-stack/k8s/victoria-operator-values.yaml|prometheus-stack/k8s/active"
   "loki|grafana/loki|prometheus|7.3.0|loki/k8s/loki-values.yaml|loki/k8s/active"
   "alloy|grafana/alloy|prometheus|1.12.1|loki/k8s/alloy-values.yaml|loki/k8s/active"
   "reloader|stakater/reloader|reloader|2.2.17|reloader/k8s/reloader-values.yaml|reloader/k8s/active"
@@ -333,7 +547,6 @@ helm_repo_for() {
     prometheus-community/*) echo "prometheus-community https://prometheus-community.github.io/helm-charts" ;;
     grafana/*) echo "grafana https://grafana.github.io/helm-charts" ;;
     stakater/*) echo "stakater https://stakater.github.io/stakater-charts" ;;
-    victoriametrics/*) echo "victoriametrics https://victoriametrics.github.io/helm-charts" ;;
   esac
 }
 
@@ -343,12 +556,8 @@ helm_repo_for() {
 helm_release_status() {
   local out
   if ! out="$(helm status "$1" -n "$2" 2>&1)"; then
-    if [[ "$out" == *"release: not found"* ]]; then
+    echo "not-found"
-      echo "not-found"
+    return 0
-      return 0
-    fi
-    printf 'ERROR: cannot read Helm status: %s\n' "$out" >&2
-    return 1
   fi
   awk '/^STATUS:/{print $2}' <<<"$out" | tr '[:upper:]' '[:lower:]'
 }
@@ -360,27 +569,16 @@ helm_release_status() {
 # pending (deployed, failed, not-found). Returns non-zero when the release is
 # still not recoverable, so the pipeline fails loud instead of wedging.
 recover_pending_release() {
-  local release="$1" namespace="$2" status revision snapshot
+  local release="$1" namespace="$2" status
-  status="$(helm_release_status "$release" "$namespace")" || return 1
+  status="$(helm_release_status "$release" "$namespace")"
   case "$status" in
     pending-upgrade|pending-rollback|pending-install)
       log "Release $release is $status, rolling back to the last deployed revision"
-      revision=""
+      if ! helm rollback "$release" -n "$namespace" --wait --timeout 10m >/dev/null 2>&1; then
-      if [ -s "$DEPLOY_SNAPSHOT_DIR/current" ]; then
-        snapshot="$(cat "$DEPLOY_SNAPSHOT_DIR/current")"
-        if [ -s "$snapshot/helm-$release.json" ]; then
-          revision="$(jq -r '.version' "$snapshot/helm-$release.json")"
-        fi
-      fi
-      if [[ ! "$revision" =~ ^[1-9][0-9]*$ ]]; then
-        echo "ERROR: no captured Helm revision for $release; manual recovery required"
-        return 1
-      fi
-      if ! helm rollback "$release" "$revision" -n "$namespace" --wait --timeout 10m; then
         echo "WARN: helm rollback of $release did not complete"
         return 1
       fi
-      status="$(helm_release_status "$release" "$namespace")" || return 1
+      status="$(helm_release_status "$release" "$namespace")"
       if [ "$status" != "deployed" ]; then
         echo "WARN: $release is $status after rollback"
         return 1
@@ -416,16 +614,11 @@ upgrade_helm_releases() {
   local entry release chart namespace version values marker repo
   for entry in ${HELM_RELEASES[@]+"${HELM_RELEASES[@]}"}; do
     IFS='|' read -r release chart namespace version values marker <<<"$entry"
-    if [ -n "${DEPLOY_PLAN:-}" ] && ! jq -e --arg name "$release" '.helm | index($name) != null' "$DEPLOY_PLAN" >/dev/null; then
-      echo "skip (unchanged Helm release): $release"
-      continue
-    fi
-    if [ ! -f "$REPO/$values" ] && [ -f "$CONFIG_REPO/$values" ]; then values="$CONFIG_REPO/$values"; else values="$REPO/$values"; fi
     if [ ! -f "$REPO/$marker" ]; then
       echo "skip (no $marker): $release"
       continue
     fi
-    if [ ! -f "$values" ]; then
+    if [ ! -f "$REPO/$values" ]; then
       echo "ERROR: $values is gitignored but missing on the workstation, restore it first."
       return 1
     fi
@@ -434,8 +627,8 @@ upgrade_helm_releases() {
       echo "ERROR: no Helm repository configured for chart $chart"
       return 1
     fi
-    helm repo add "${repo%% *}" "${repo#* }" >/dev/null
+    helm repo add "${repo%% *}" "${repo#* }" >/dev/null 2>&1 || true
-    helm repo update "${repo%% *}" >/dev/null
+    helm repo update "${repo%% *}" >/dev/null 2>&1 || true
     log "Upgrading $release ($chart $version)"
     wait_for_calm "helm $release"
     # A previous run with --rollback-on-failure whose own rollback never finished leaves the
@@ -451,7 +644,7 @@ upgrade_helm_releases() {
     if ! helm upgrade --install "$release" "$chart" \
       --namespace "$namespace" \
       --version "$version" \
-      --values "$values" \
+      --values "$REPO/$values" \
       --wait --rollback-on-failure --cleanup-on-fail --timeout 10m; then
       echo "WARN: upgrade of $release failed, checking release state"
       # --rollback-on-failure already attempted its own rollback; finish the job when that
@@ -467,112 +660,56 @@ upgrade_helm_releases() {
   done
 }
 
-stage_doctor() {
+stage_preflight() {
-  local tool entry release chart namespace version values marker
+  if [ ! -d "$REPO/.git" ]; then
-  for tool in git docker kubectl helm jq curl timeout flock python3; do
+    echo "Repository not found at $REPO"
-    command -v "$tool" >/dev/null || { echo "Missing workstation tool: $tool"; return 1; }
+    exit 1
-  done
-  docker compose version >/dev/null
-  docker buildx version >/dev/null
-  [ "$(kubectl config current-context)" = "${KUBE_CONTEXT:?configure KUBE_CONTEXT}" ] || { echo "Unexpected Kubernetes context"; return 1; }
-  [ "$(kubectl get namespace kube-system -o jsonpath='{.metadata.uid}')" = "${EXPECTED_CLUSTER_UID:?configure EXPECTED_CLUSTER_UID}" ] || { echo "Unexpected Kubernetes cluster"; return 1; }
-  kubectl get --raw=/readyz --request-timeout=10s >/dev/null
-  [ "$(git -C "$REPO" rev-parse HEAD)" = "$DEPLOY_SHA" ] || return 1
-  select_manifests
-  for entry in "${HELM_RELEASES[@]}"; do
-    IFS='|' read -r release chart namespace version values marker <<<"$entry"
-    [ -f "$REPO/$marker" ] || continue
-    [ -f "$REPO/$values" ] || [ -f "$CONFIG_REPO/$values" ] || { echo "Missing values: $values"; return 1; }
-  done
-  jq '{sha, selected, helm, removed}' "$DEPLOY_PLAN"
-  local cf
-  for cf in "${COMPOSE_STACKS[@]}"; do
-    compose "$cf" config --quiet
-    while IFS= read -r network; do
-      docker network inspect "$network" >/dev/null || return 1
-    done < <(compose "$cf" config --format json | jq -r '.networks // {} | to_entries[] | select(.value.external == true) | .value.name')
-    python3 "$REPO/.gitea/workflows/compose-release.py" "$cf"
-  done
-  local image refs m k
-  refs="$(
-    for m in "${K8S_MANIFESTS[@]}"; do render_pinned <"$m" || return 1; done
-    for k in "${KUSTOMIZE_APPS[@]}"; do kubectl kustomize "$k" | render_pinned || return 1; done
-  )" || return 1
-  refs="$(grep -oE 'gcr\.forust\.xyz/forust/[A-Za-z0-9._-]+@sha256:[0-9a-f]{64}' <<<"$refs" | sort -u || true)"
-  while IFS= read -r image; do
-    [ -n "$image" ] || continue
-    timeout 60s docker buildx imagetools inspect "$image" >/dev/null
-  done <<<"$refs"
-}
-
-# Required pod Secrets, scoped to the resource namespace. TLS route Secrets are
-# created by cert-manager and are not prerequisites for applying a Certificate.
-check_referenced_secrets() {
-  local m k objects refs extracted ns name
-  local missing=()
-  refs=""
-  for m in ${K8S_MANIFESTS[@]+"${K8S_MANIFESTS[@]}"}; do
-    if skip_uninstalled_vmagent_crd "$m"; then
-      continue
-    fi
-    objects="$(kubectl create --dry-run=client --validate=false -f "$m" -o json)" || return 1
-    extracted="$(printf '%s' "$objects" | jq -r -f "$REPO/.gitea/workflows/secret-references.jq")" || return 1
-    refs+="$extracted"$'\n'
-  done
-  for k in ${KUSTOMIZE_APPS[@]+"${KUSTOMIZE_APPS[@]}"}; do
-    objects="$(kubectl kustomize "$k" | kubectl create --dry-run=client --validate=false -f - -o json)" || return 1
-    extracted="$(printf '%s' "$objects" | jq -r -f "$REPO/.gitea/workflows/secret-references.jq")" || return 1
-    refs+="$extracted"$'\n'
-  done
-  while read -r ns name; do
-    [ -n "${name:-}" ] || continue
-    if kubectl get secret "$name" -n "$ns" -o name >/dev/null 2>&1; then
-      echo "  ok: $ns/$name"
-    else
-      echo "  MISSING OR UNREADABLE: $ns/$name"
-      missing+=("$ns/$name")
-    fi
-  done < <(printf '%s' "$refs" | sort -u)
-  if [ "${#missing[@]}" -gt 0 ]; then
-    echo "ERROR: required pod Secrets are missing or unreadable:"
-    printf '  - %s\n' "${missing[@]}"
-    echo "Create them in the listed namespaces from the service's secret example."
-    return 1
   fi
-}
+  if [ -n "$DEPLOY_SHA" ]; then
-
+    log "Checking out the commit CI validated ($DEPLOY_SHA)"
-# The VMAgent CRD is installed by the VictoriaMetrics Operator Helm release in
+    git -C "$REPO" fetch origin --quiet "$DEPLOY_SHA" 2>/dev/null \
-# stage_apply_k8s, after this preflight stage. Skip only its dry-run until then.
+      || git -C "$REPO" fetch origin main
-skip_uninstalled_vmagent_crd() {
+  else
-  local manifest="$1"
+    git -C "$REPO" fetch origin main
-  if [[ "$manifest" == "$REPO/prometheus-stack/k8s/vmagent.yaml" ]] \
-    && ! kubectl get crd vmagents.operator.victoriametrics.com >/dev/null 2>&1; then
-    echo "  skip: VMAgent CRD is installed by Helm during apply: ${manifest#"$REPO"/}"
-    return 0
   fi
-  return 1
+  target="${DEPLOY_SHA:-origin/main}"
+  log "Workstation state"
+  echo "  local:  $(git -C "$REPO" rev-parse --short HEAD)"
+  echo "  target: $(git -C "$REPO" rev-parse --short "$target")"
+  if [ -n "$(git -C "$REPO" status --porcelain --untracked-files=no)" ]; then
+    echo "ERROR: workstation has local tracked modifications, refusing reset:"
+    git -C "$REPO" status --porcelain --untracked-files=no
+    git -C "$REPO" diff --stat
+    echo "Fix it on the workstation (commit, or 'git restore .'), then re-run the deploy."
+    exit 1
+  fi
+  git -C "$REPO" reset --hard "$target"
 }
 
 stage_validate() {
-  check_prune_mode || return 1
   cd "$REPO"
   select_manifests
   local m k cf
-  # The deploy host has the local .env and secret files. Resolve them here so
+  # Compose .env files and secret files are gitignored by design, so the
-  # missing configuration fails before either apply job changes workloads.
+  # workstation never has real values for the inactive stacks. This stage only
-  # CI keeps the structure-only check for inactive stacks.
+  # runs the full check on active stacks; the general structure check for every
+  # committed Compose file (active or not) lives in the ci workflow, which has no
+  # .env at all.
+  #
+  # Active stacks are still validated with interpolation and env-file resolution
+  # off, so required-variable guards (:?) and missing local files do not fail the
+  # deploy. Normalization and consistency checks stay enabled.
   # shellcheck source=compose-lint.sh
   source "$REPO/.gitea/workflows/compose-lint.sh"
+  local compose_validate_flags=()
+  mapfile -t compose_validate_flags < <(compose_safe_flags)
   log "Validate compose stacks"
   for cf in ${COMPOSE_STACKS[@]+"${COMPOSE_STACKS[@]}"}; do
     echo "  config: $cf"
-    compose "$cf" config --quiet
+    validate_compose_file "$cf" ${compose_validate_flags[@]+"${compose_validate_flags[@]}"}
   done
   log "Validate k8s manifests (kubectl dry-run=client)"
   for m in ${K8S_MANIFESTS[@]+"${K8S_MANIFESTS[@]}"}; do
-    if skip_uninstalled_vmagent_crd "$m"; then
-      continue
-    fi
     kubectl apply --dry-run=client -f "$m" >/dev/null
   done
   for k in ${KUSTOMIZE_APPS[@]+"${KUSTOMIZE_APPS[@]}"}; do
@@ -580,9 +717,6 @@ stage_validate() {
   done
   log "Validate k8s manifests (kubectl dry-run=server)"
   for m in ${K8S_MANIFESTS[@]+"${K8S_MANIFESTS[@]}"}; do
-    if skip_uninstalled_vmagent_crd "$m"; then
-      continue
-    fi
     kubectl apply --dry-run=server -f "$m" >/dev/null
   done
   for k in ${KUSTOMIZE_APPS[@]+"${KUSTOMIZE_APPS[@]}"}; do
@@ -590,43 +724,54 @@ stage_validate() {
   done
   log "Checking referenced Secrets exist"
   echo "  (deploy never applies *secret*.yaml; create missing ones manually)"
-  check_referenced_secrets
+  local ref_secrets=() missing_secrets=() all_secrets s
-}
+  if [ "${#K8S_MANIFESTS[@]}" -gt 0 ]; then
-
+    while IFS= read -r s; do
-selected_workload_refs() {
+      [ -n "$s" ] && ref_secrets+=("$s")
-  local m k
+    done < <(
-  for m in "${K8S_MANIFESTS[@]}"; do
+      {
-    if skip_uninstalled_vmagent_crd "$m" >/dev/null; then continue; fi
+        grep -h -A1 -E 'secretRef:|secretKeyRef:' "${K8S_MANIFESTS[@]}" 2>/dev/null || true
-    kubectl create --dry-run=client --validate=false -f "$m" -o json | jq -r '
+        grep -h -E 'secretName:' "${K8S_MANIFESTS[@]}" 2>/dev/null || true
-      (if .kind == "List" then .items[] else . end) | select(.kind | test("^(Deployment|StatefulSet|DaemonSet)$"))
+      } | grep -E 'name:' | sed -E 's/.*name:[[:space:]]*//' | tr -d '"'"'"' "'"'" | sed -E 's/[[:space:]]*#.*//' | awk 'NF' | sort -u || true
-      | "\(.kind | ascii_downcase) \(.metadata.namespace // "default") \(.metadata.name)"'
+    )
-  done
+  fi
-  for k in "${KUSTOMIZE_APPS[@]}"; do
+  all_secrets="$(kubectl get secrets -A --no-headers -o custom-columns=:metadata.name 2>/dev/null || true)"
-    kubectl kustomize "$k" | kubectl create --dry-run=client --validate=false -f - -o json | jq -r '
+  for s in ${ref_secrets[@]+"${ref_secrets[@]}"}; do
-      (if .kind == "List" then .items[] else . end) | select(.kind | test("^(Deployment|StatefulSet|DaemonSet)$"))
+    if printf '%s\n' "$all_secrets" | grep -qx "$s"; then
-      | "\(.kind | ascii_downcase) \(.metadata.namespace // "default") \(.metadata.name)"'
+      echo "  ok: $s"
+    else
+      echo "  MISSING: $s"
+      missing_secrets+=("$s")
+    fi
   done
+  if [ "${#missing_secrets[@]}" -gt 0 ]; then
+    echo "ERROR: ${#missing_secrets[@]} referenced Secret(s) not found in the cluster:"
+    printf '  - %s\n' "${missing_secrets[@]}"
+    echo "Create them manually from the laptop, e.g.:"
+    echo "  kubectl apply -f SERVICE/k8s/secrets.yaml  # see SERVICE/k8s/secrets.yaml.example"
+    exit 1
+  fi
 }
 
 stage_apply_k8s() {
-  check_prune_mode || return 1
   cd "$REPO"
   select_manifests >/dev/null
-  local ns_files=() other_files=() m k
+  local ns_files=() other_files=() m k prune_opts=()
   for m in ${K8S_MANIFESTS[@]+"${K8S_MANIFESTS[@]}"}; do
     case "$m" in
-      */namespace.yaml|*/namespace.yml) ns_files+=("$m") ;;
+      */namespace.y?ml) ns_files+=("$m") ;;
       *) other_files+=("$m") ;;
     esac
   done
+  if [ "$APPLY_PRUNE" = "true" ]; then
+    prune_opts=(--prune -l app.kubernetes.io/managed-by=homelab-deploy)
+  fi
 
   # Record what is about to change, and publish it for the verify job, before
   # the first apply. Both are fatal on failure: see snapshot_dir.
-  selected_workload_refs >"$RUN_DIR/workload-refs"
   local snapshot
   snapshot="$(snapshot_dir)" || return 1
   save_snapshot "$snapshot" || return 1
-  touch "$snapshot/ready"
 
   if [ "${#ns_files[@]}" -gt 0 ]; then
     log "Applying namespaces (${#ns_files[@]} files)"
@@ -634,8 +779,8 @@ stage_apply_k8s() {
       kubectl apply -f "$m"
     done
   fi
-  if selected_service k8s prometheus-stack && [ -f "$REPO/prometheus-stack/k8s/active" ]; then
+  if [ -f "$REPO/prometheus-stack/k8s/active" ]; then
-    if [ ! -f "$CONFIG_REPO/prometheus-stack/k8s/grafana-values.yaml" ]; then
+    if [ ! -f "$REPO/prometheus-stack/k8s/grafana-values.yaml" ]; then
       echo "ERROR: prometheus-stack/k8s/grafana-values.yaml (gitignored) missing on workstation, restore it first."
       exit 1
     fi
@@ -645,8 +790,7 @@ stage_apply_k8s() {
   if [ "${#other_files[@]}" -gt 0 ]; then
     log "Applying resources (${#other_files[@]} files, our images pinned to digests)"
     for m in "${other_files[@]}"; do
-      log "Applying ${m#"$REPO"/}"
+      if ! render_pinned <"$m" | kubectl apply "${prune_opts[@]}" -f -; then
-      if ! render_pinned <"$m" | kubectl apply -f -; then
         echo "ERROR: apply failed for ${m#"$REPO"/}" >&2
         exit 1
       fi
@@ -659,6 +803,7 @@ stage_apply_k8s() {
       exit 1
     fi
   done
+  restart_stale_images
 
   # No verification here on purpose. This stage may be killed at any point by
   # timeout-minutes, by the runner cancelling the job, or by a dropped SSH
@@ -673,7 +818,7 @@ stage_apply_k8s() {
 # and rolls back the ones that never became healthy.
 stage_verify_k8s() {
   local pointer="$DEPLOY_SNAPSHOT_DIR/current"
-  local snapshot want have generations changed
+  local snapshot want have generations
   local -a touched=()
 
   if [ ! -s "$pointer" ]; then
@@ -684,7 +829,7 @@ stage_verify_k8s() {
     return 1
   fi
   snapshot="$(head -1 "$pointer")"
-  if [ ! -d "$snapshot" ] || [ ! -f "$snapshot/ready" ]; then
+  if [ ! -d "$snapshot" ]; then
     echo "ERROR: snapshot pointer refers to a missing directory: $snapshot"
     return 1
   fi
@@ -707,13 +852,6 @@ stage_verify_k8s() {
   fi
   echo "  snapshot: $snapshot (commit ${have:0:12})"
 
-  local entry release chart namespace version values marker
-  for entry in "${HELM_RELEASES[@]}"; do
-    IFS='|' read -r release chart namespace version values marker <<<"$entry"
-    jq -e --arg name "$release" '.helm | index($name) != null' "$DEPLOY_PLAN" >/dev/null || continue
-    recover_pending_release "$release" "$namespace" || return 1
-  done
-
   generations="$snapshot/generations.before"
   if [ ! -s "$generations" ]; then
     # Without a baseline we cannot tell which workloads the apply touched, so
@@ -722,10 +860,9 @@ stage_verify_k8s() {
     : >"$generations"
   fi
 
-  changed="$(changed_workloads "$generations")" || return 1
   while read -r kind ns name; do
     [ -n "${kind:-}" ] && touched+=("$kind $ns $name")
-  done <<<"$changed"
+  done < <(changed_workloads "$generations")
 
   log "Verifying ${#touched[@]} changed workload(s) (timeout ${ROLLOUT_TIMEOUT}s each)"
   if [ "${#touched[@]}" -eq 0 ]; then
@@ -764,19 +901,21 @@ stage_verify_k8s() {
 verify_compose_stack() {
   local cf="$1"
   local expected running missing=()
-  expected="$(compose "$cf" config --format json | jq -r ' .services | to_entries[] | select(.value.restart != "no") | .key' | sort)" || return 1
+  expected="$(docker compose -f "$cf" config --services 2>/dev/null | sort || true)"
-  running="$(compose "$cf" ps --status running --services | sort)" || return 1
+  running="$(docker compose -f "$cf" ps --status running --services 2>/dev/null | sort || true)"
   [ -n "$expected" ] || return 0
   while IFS= read -r svc; do
     [ -n "$svc" ] || continue
     # restart:"no" services are allowed to have exited.
-    if ! printf '%s\n' "$running" | grep -qx "$svc"; then
+    if ! printf '%s\n' "$running" | grep -qx "$svc" \
+      && ! docker compose -f "$cf" config 2>/dev/null \
+      | grep -A5 "^  ${svc}:" | grep -qE 'restart:\s*"?no"?'; then
       missing+=("$svc")
     fi
   done <<<"$expected"
   if [ "${#missing[@]}" -gt 0 ]; then
     echo "  NOT RUNNING: ${missing[*]}"
-    compose "$cf" ps --all 2>/dev/null | sed 's/^/    /' || true
+    docker compose -f "$cf" ps --all 2>/dev/null | sed 's/^/    /' || true
     return 1
   fi
   echo "  all ${#expected} service(s) running"
@@ -860,11 +999,10 @@ traefik_routed_hosts() {
 # cases, so ask Traefik which routes it built and fail on the difference.
 stage_smoke() {
   cd "$REPO"
-  if [ -n "${DEPLOY_PLAN:-}" ] && jq -e '.full_smoke' "$DEPLOY_PLAN" >/dev/null; then
-    DEPLOY_SMOKE_ALL=true
-  fi
   select_manifests >/dev/null
   local -a hosts=()
+  # Not named failed: an array of that name already exists in restart_stale_images
+  # above, and a scalar shadowing an array is a trap rather than a shadow.
   local h code rc bad=0
   while IFS= read -r h; do
     [ -n "$h" ] && hosts+=("$h")
@@ -872,8 +1010,8 @@ stage_smoke() {
 
   if [ "${#hosts[@]}" -eq 0 ]; then
     # Nothing to probe means the extraction broke, not that the cluster is empty.
-    echo "No public routes in the selected components"
+    echo "ERROR: no public hostnames found in active manifests, refusing to report success"
-    return 0
+    return 1
   fi
 
   log "Probing ${#hosts[@]} public route(s)"
@@ -957,17 +1095,37 @@ stage_apply_compose() {
   cd "$REPO"
   select_manifests >/dev/null
   local cf
-  for cf in "${COMPOSE_STACKS[@]}"; do
+  log "Redeploying docker compose stacks (${#COMPOSE_STACKS[@]} stacks)"
-    log "Applying Compose ${cf#"$REPO"/}"
+  for cf in ${COMPOSE_STACKS[@]+"${COMPOSE_STACKS[@]}"}; do
-    compose "$cf" up -d --wait --wait-timeout 180 --pull missing --remove-orphans
+    echo "  compose: $cf"
-    verify_compose_stack "$cf"
+    if grep -Eq '^\s+pull_policy:\s*build\b' "$cf"; then
+      docker compose -f "$cf" build
+      docker compose -f "$cf" push
+    fi
+    docker compose -f "$cf" up -d --pull always --remove-orphans
   done
-  echo "Compose recovery files: $RUN_DIR/compose-before (manual recovery only)"
+
+  local -a broken=()
+  for cf in ${COMPOSE_STACKS[@]+"${COMPOSE_STACKS[@]}"}; do
+    echo "  verifying: $cf"
+    if ! verify_compose_stack "$cf"; then
+      broken+=("$cf")
+    fi
+  done
+  if [ "${#broken[@]}" -gt 0 ]; then
+    echo
+    echo "ERROR: ${#broken[@]} compose stack(s) did not come up:"
+    printf '  - %s\n' "${broken[@]}"
+    echo "Compose stacks are not rolled back automatically: their images use mutable"
+    echo "':latest' tags, so there is no previous version to return to. Check the logs"
+    echo "above, then re-run the deploy once the cause is fixed."
+    return 1
+  fi
 }
 
 run_stage() {
   case "${1:?stage required}" in
-    doctor) stage_doctor ;;
+    preflight) stage_preflight ;;
     validate) stage_validate ;;
     apply-k8s) stage_apply_k8s ;;
     verify-k8s) stage_verify_k8s ;;

.gitea/workflows/deploy-plan.py

@@ -1,124 +0,0 @@
-#!/usr/bin/env python3
-"""Calculate selected components against the last fully successful deploy."""
-
-import hashlib
-import json
-import re
-import subprocess
-from pathlib import Path
-
-
-def output(*args, **kwargs):
-    return subprocess.check_output(args, text=True, **kwargs).strip()  # noqa: S603
-
-
-def tracked(repo):
-    return output('git', '-C', str(repo), 'ls-files').splitlines()
-
-
-def helm_releases(repo):
-    text = (repo / '.gitea/workflows/deploy-lib.sh').read_text()
-    return [line.split('|') for line in re.findall(r'^  "([^"\n]+\|[^"\n]+)"$', text, re.MULTILINE)]
-
-
-def inventory(repo):
-    files = tracked(repo)
-    k8s = sorted(
-        {f.split('/k8s/')[0] for f in files if '/k8s/' in f and (repo / f.split('/k8s/')[0] / 'k8s/active').is_file()}
-    )
-    compose = sorted(
-        {
-            str(Path(f).parent)
-            for f in files
-            if Path(f).name in ('compose.yaml', 'compose.yml') and (repo / Path(f).parent / 'active').is_file()
-        }
-    )
-    return {'k8s': k8s, 'compose': compose}
-
-
-def file_hash(path):
-    return hashlib.sha256(path.read_bytes()).hexdigest() if path.is_file() else 'missing'
-
-
-def make_plan(repo, config_repo, release, previous, mode, live_helm):
-    active = inventory(repo)
-    all_services = set(active['k8s'] + active['compose'])
-    helm_inputs = {}
-    helm_selected = []
-    for name, chart, namespace, version, values, marker in helm_releases(repo):
-        if not (repo / marker).is_file():
-            continue
-        value_path = repo / values if (repo / values).is_file() else config_repo / values
-        if not value_path.is_file():
-            raise ValueError(f'Missing Helm values: {values}')
-        stamp = hashlib.sha256(f'{chart}|{version}|{file_hash(value_path)}'.encode()).hexdigest()
-        helm_inputs[name] = stamp
-        live = next((h for h in live_helm if h['name'] == name and h['namespace'] == namespace), None)
-        if (
-            mode == 'full'
-            or previous is None
-            or previous.get('helm_inputs', {}).get(name) != stamp
-            or live is None
-            or live.get('status') != 'deployed'
-            or live.get('chart') != f'{chart.split("/")[-1]}-{version}'
-        ):
-            helm_selected.append(name)
-    local_inputs = {}
-    for service in all_services:
-        candidates = [config_repo / service / '.env']
-        if service in active['compose']:
-            candidates.append(config_repo / '.env')
-            cfg = config_repo / service / 'config'
-            if cfg.is_dir():
-                candidates.extend(
-                    p for p in cfg.rglob('*') if p.is_file() and p.suffix in ('.yaml', '.yml', '.json', '.conf')
-                )
-        local_inputs[service] = hashlib.sha256(
-            '\n'.join(f'{p.relative_to(config_repo)}:{file_hash(p)}' for p in sorted(candidates)).encode()
-        ).hexdigest()
-    if previous is None:
-        if mode == 'changed':
-            raise ValueError('No successful baseline; run deploy in full mode first')
-        changed = set(all_services)
-        removed = []
-    else:
-        paths = output('git', '-C', str(repo), 'diff', '--name-only', previous['sha'], release['sha']).splitlines()
-        changed = {path.split('/')[0] for path in paths}
-        if any(path.startswith('.gitea/') for path in paths):
-            changed |= all_services
-        changed |= {s for s in all_services if previous.get('local_inputs', {}).get(s) != local_inputs[s]}
-        for file in tracked(repo):
-            service = file.split('/')[0]
-            if service not in all_services or not file.endswith(('.yaml', '.yml')):
-                continue
-            text = (repo / file).read_text()
-            if any(
-                image in text and previous.get('images', {}).get(image) != digest
-                for image, digest in release['images'].items()
-            ):
-                changed.add(service)
-        removed = sorted(
-            set(previous.get('active', {}).get('k8s', []) + previous.get('active', {}).get('compose', []))
-            - all_services
-        )
-        removed += [path for path in paths if '/k8s/' in path and not (repo / path).exists()]
-    if mode == 'full':
-        changed = set(all_services)
-    dependencies = json.loads((repo / '.gitea/deploy-dependencies.json').read_text())
-    while True:
-        expanded = changed | {dependent for service in changed for dependent in dependencies.get(service, [])}
-        if expanded == changed:
-            break
-        changed = expanded
-    return {
-        'version': 1,
-        'sha': release['sha'],
-        'images': release['images'],
-        'active': active,
-        'selected': {kind: sorted(set(services) & changed) for kind, services in active.items()},
-        'helm': helm_selected,
-        'helm_inputs': helm_inputs,
-        'local_inputs': local_inputs,
-        'removed': sorted(set(removed)),
-        'full_smoke': mode == 'full' or 'traefik' in changed,
-    }

.gitea/workflows/deploy-stage.sh

@@ -1,10 +0,0 @@
-#!/usr/bin/env bash
-set -euo pipefail
-source "${REPO:?}/.gitea/workflows/deploy-lib.sh"
-case "${1:?stage required}" in
-  workload-count)
-    select_manifests >/dev/null
-    selected_workload_refs | sort -u | wc -l
-    ;;
-  *) run_stage "$1" ;;
-esac

.gitea/workflows/deploy.yaml

@@ -1,104 +1,197 @@
 name: deploy
 
 on:
+  # Deploy only what CI already validated. workflow_run is used instead of
+  # workflow_dispatch so a red lint/validate run can never reach the cluster.
   workflow_run:
     workflows: [ci]
-    branches: [main]
     types: [completed]
   workflow_dispatch:
-    inputs:
-      deploy_ref:
-        description: "Commit already checked by successful main CI (main or SHA)"
-        default: main
-        required: true
-      deploy_mode:
-        description: "First deploy requires full; plan changes no production resources"
-        type: choice
-        options: [changed, full, plan]
-        default: changed
-      refresh_images:
-        description: "Explicitly refresh mutable third-party Compose tags"
-        type: boolean
-        default: false
 
+# The deploy jobs read the tree, then reach the cluster over SSH with the
+# deploy key. The Actions token itself is not part of that path, so it gets
+# read-only contents and no more.
 permissions:
   contents: read
-  actions: read
 
 concurrency:
   group: deploy-main
+  # Queue instead of cancelling. Cancelling a run kills the apply job mid-loop and
+  # takes the verify job down with it, so a superseded deploy would leave the
+  # cluster half-applied and unchecked — the exact failure the verify job exists
+  # to catch. kubectl apply and docker compose up are both idempotent, so letting
+  # the older run finish and then deploying the newer commit costs little.
   cancel-in-progress: false
 
 env:
-  DEPLOY_HOST: ${{ vars.DEPLOY_HOST || secrets.DEPLOY_HOST }}
+  DEPLOY_HOST: ${{ secrets.DEPLOY_HOST }}
-  DEPLOY_PORT: ${{ vars.DEPLOY_PORT || secrets.DEPLOY_PORT }}
+  DEPLOY_PORT: ${{ secrets.DEPLOY_PORT }}
-  DEPLOY_USER: ${{ vars.DEPLOY_USER || secrets.DEPLOY_USER }}
+  DEPLOY_USER: ${{ secrets.DEPLOY_USER }}
+  DEPLOY_PATH: ${{ secrets.DEPLOY_PATH }}
   DEPLOY_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
-  DEPLOY_KNOWN_HOSTS: ${{ vars.DEPLOY_KNOWN_HOSTS }}
+  APPLY_PRUNE: ${{ vars.APPLY_PRUNE }}
-  DEPLOY_RUN_ID: ${{ github.run_id }}-${{ github.run_attempt || 1 }}
+  # workflow_run's own GITHUB_SHA points at the branch head, not at the commit the
-  DEPLOY_MODE: ${{ inputs.deploy_mode || 'changed' }}
+  # finished ci run checked. Pin the exact validated commit instead, so a push
-  REFRESH_IMAGES: ${{ inputs.refresh_images && 'true' || 'false' }}
+  # landing mid-deploy cannot make the workstation deploy something else. Also
+  # what the verify job checks the snapshot against. Empty for workflow_dispatch,
+  # which falls back to the current origin/main.
+  DEPLOY_SHA: ${{ github.event.workflow_run.head_sha }}
 
 jobs:
-  gate:
+  preflight:
+    # Autodeploy defaults to OFF: pushes deploy only when the AUTODEPLOY repo
+    # variable is set to 'true' (Settings -> Actions -> Variables). A manual
+    # Run workflow always bypasses the switch: dispatching it is the explicit
+    # intent to deploy.
     if: >-
-      github.ref == 'refs/heads/main' &&
       (vars.AUTODEPLOY == 'true' || github.event_name == 'workflow_dispatch') &&
       (github.event_name != 'workflow_run' ||
-       (github.event.workflow_run.conclusion == 'success' && github.event.workflow_run.head_branch == 'main'))
+      (github.event.workflow_run.conclusion == 'success' &&
-    runs-on: homelab
+       github.event.workflow_run.head_branch == 'main'))
+    runs-on: [self-hosted, linux, arch, homelab, prod]
     timeout-minutes: 10
-    outputs:
-      sha: ${{ steps.release.outputs.sha }}
     steps:
       - name: Checkout repository
         uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
-        with:
-          fetch-depth: 0
-      - name: Check successful CI and download the exact commit release
-        id: release
-        env:
-          GITEA_TOKEN: ${{ github.token }}
-          DEPLOY_REF: ${{ inputs.deploy_ref || 'main' }}
-          EVENT_SHA: ${{ github.event.workflow_run.head_sha }}
-        run: python3 .gitea/workflows/release.py gate --ref "$DEPLOY_REF" --event-sha "$EVENT_SHA"
-      - name: Submit durable deploy to workstation
-        run: bash .gitea/workflows/ssh-run.sh start
 
-  apply:
+      - name: Fetch and reset workstation
-    needs: [gate]
+        shell: bash
-    runs-on: homelab
+        run: |
-    timeout-minutes: 100
+          set -euo pipefail
+          ./.gitea/workflows/ssh-run.sh preflight
+
+  validate:
+    needs: [preflight]
+    runs-on: [self-hosted, linux, arch, homelab, prod]
+    timeout-minutes: 20
     steps:
-      - name: Checkout checked commit
+      - name: Checkout repository
         uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
-        with:
-          ref: ${{ needs.gate.outputs.sha }}
-      - name: Follow validation and sequential Kubernetes / Compose apply
-        run: bash .gitea/workflows/ssh-run.sh apply
 
-  verify:
+      - name: Dry-run manifests and check Secrets
-    needs: [gate, apply]
+        shell: bash
-    if: always() && needs.gate.result == 'success'
+        run: |
-    runs-on: homelab
+          set -euo pipefail
-    timeout-minutes: 130
+          ./.gitea/workflows/ssh-run.sh validate
+
+  apply-k8s:
+    needs: [validate]
+    runs-on: [self-hosted, linux, arch, homelab, prod]
+    # Apply only, no verification, so this is just the work itself: snapshot,
+    # then sequential `helm upgrade --install --wait --rollback-on-failure --timeout 10m`, then the apply loop.
+    # Verification has its own job and its own budget.
+    #
+    # 45 is roughly four times the measured cost of the stage, which is
+    # deliberately not raised on a theory:
+    #
+    #   helm, healthy          3 no-op upgrades              ~3-5 min
+    #   helm, one release bad  rollback-on-failure spends its 10m,      ~10-15 min
+    #                            then rolls that one back
+    #   apply loop             ~40 manifests, 4 of which     ~1 min
+    #                           resolve an image digest
+    #   restart_stale_images   7.6s to find 8 workloads,     ~0.5 min
+    #                           9.8s to resolve their digests
+    #
+    # The helm figure is one release, not three: `set -e` aborts
+    # upgrade_helm_releases on the first failure, so a broken release costs
+    # 10m and the other two are never attempted. Multiplying 10m by three
+    # overstates the worst case by 20 minutes.
+    #
+    # The 45 minutes this was last raised to 45 were still not enough, and the
+    # job logs for those runs no longer exist, so what actually consumed the
+    # budget is not known - the two measurable candidates above account for
+    # ~15 of it. The unbounded `docker manifest inspect` against the registry's
+    # known hang mode is now bounded inside registry_digest (25s timeout, 3
+    # attempts): a dead registry fails each owned image after ~85s instead of
+    # hanging the stage, and a blinking one is retried instead of failing the
+    # whole apply file. Still open: make the stage announce which manifest it
+    # is working on, so a killed run leaves a diagnosable last line.
+    timeout-minutes: 45
     steps:
-      - name: Checkout checked commit
+      - name: Checkout repository
         uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
-        with:
-          ref: ${{ needs.gate.outputs.sha }}
-      - name: Follow workload verification and recovery
-        run: bash .gitea/workflows/ssh-run.sh verify
 
+      - name: Apply Kubernetes manifests
+        shell: bash
+        run: |
+          set -euo pipefail
+          ./.gitea/workflows/ssh-run.sh apply-k8s
+
+  apply-compose:
+    needs: [validate]
+    runs-on: [self-hosted, linux, arch, homelab, prod]
+    timeout-minutes: 30
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+
+      - name: Redeploy docker compose stacks
+        shell: bash
+        run: |
+          set -euo pipefail
+          ./.gitea/workflows/ssh-run.sh apply-compose
+
+  # Watches the workloads this deploy changed and rolls back the ones that never
+  # became healthy. Runs even when the apply jobs failed, timed out or were
+  # cancelled — that is the whole point of splitting it out. `always()` is what
+  # lets it start after a failed dependency; the needs on apply-compose are a
+  # barrier, so verification begins only once both applies are done.
+  verify-k8s:
+    needs: [apply-k8s, apply-compose]
+    if: >-
+      always() &&
+      needs.apply-k8s.result != 'skipped' &&
+      needs.apply-compose.result != 'skipped'
+    runs-on: [self-hosted, linux, arch, homelab, prod]
+    # Not raised, because the arithmetic does not close.
+    #
+    # 32 workloads are under management and the wave width is 8, so the verify
+    # itself is 4 waves of ROLLOUT_TIMEOUT (300s) = 20 minutes worst case, when
+    # every rollout times out rather than converging. That is already 20 of 30.
+    #
+    # The other 10 would have to absorb rollback, and rollback_workloads is a
+    # serial `while read` loop at 300s per failed workload. 10 minutes buys two.
+    # Any larger number is buying a bigger multiple of an unbounded term rather
+    # than covering a known cost: 60 minutes buys eight, and 60 minutes is
+    # therefore not a bound, it is a guess with two digits.
+    #
+    # The number becomes derivable the moment rollback uses the same wave width
+    # as the verify: 32 failures then cost 4 waves = 20 minutes instead of 160,
+    # and 45 covers verify plus rollback at full width. That change is to the
+    # recovery path and is not folded into a timeout edit.
+    timeout-minutes: 30
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+
+      - name: Verify workloads and roll back on failure
+        shell: bash
+        run: |
+          set -euo pipefail
+          ./.gitea/workflows/ssh-run.sh verify-k8s
+
+  # Asks the public route of every active service whether it is actually
+  # serving, which the rollout check above structurally cannot: a pod can
+  # converge and still be crash-looping, or be listening on a port no Service
+  # points at, or answer 500.
+  #
+  # `always()` for the same reason verify-k8s has it, and it runs after that job
+  # specifically because a rollback is when a route most needs re-checking. The
+  # needs is a barrier, not a filter: whether verify-k8s passed, failed or was
+  # cancelled, the probes are what say whether the cluster is serving, and
+  # suppressing them on a rollback would hide the one run where the answer
+  # matters most.
   smoke:
-    needs: [gate, verify]
+    needs: [verify-k8s]
-    if: always() && needs.gate.result == 'success'
+    if: always() && needs.verify-k8s.result != 'skipped'
-    runs-on: homelab
+    runs-on: [self-hosted, linux, arch, homelab, prod]
-    timeout-minutes: 15
+    timeout-minutes: 10
     steps:
-      - name: Checkout checked commit
+      - name: Checkout repository
         uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
-        with:
+
-          ref: ${{ needs.gate.outputs.sha }}
+      - name: Probe the public route of every active service
-      - name: Follow public route checks
+        shell: bash
-        run: bash .gitea/workflows/ssh-run.sh smoke
+        run: |
+          set -euo pipefail
+          ./.gitea/workflows/ssh-run.sh smoke

.gitea/workflows/install-ci-tools.sh

@@ -13,14 +13,9 @@ here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 # shellcheck source=tool-versions.env
 . "$here/tool-versions.env"
 
-TOOLS_DIR="${TOOLS_DIR:-${XDG_CACHE_HOME:-$HOME/.cache}/homelab-ci}"
+TOOLS_DIR="${TOOLS_DIR:-${RUNNER_TEMP:-/tmp}/homelab-tools}"
 BIN_DIR="$TOOLS_DIR/bin"
 mkdir -p "$BIN_DIR"
-# A runner may accept overlapping workflows even though each workflow is sequential.
-exec 9>"$TOOLS_DIR/install.lock"
-flock -w 300 9
-export UV_TOOL_DIR="$TOOLS_DIR/uv-tools"
-export UV_CACHE_DIR="$TOOLS_DIR/uv-cache"
 # The just-installed tools must resolve inside this script too: callers only
 # prepend BIN_DIR to PATH after the script exits, so a bare `uv` below would
 # miss the binary install_uv just placed (exit 127 on a clean runner).
@@ -53,7 +48,7 @@ esac
 fetch() {
   # fetch <url> <dest>
   if command -v curl >/dev/null 2>&1; then
-    curl -sSLf --connect-timeout 15 --max-time 120 --retry 3 -o "$2" "$1"
+    curl -sSLf --retry 3 -o "$2" "$1"
   elif command -v wget >/dev/null 2>&1; then
     wget -q -O "$2" "$1"
   else
@@ -93,13 +88,10 @@ installed_version() {
 
 # at_version <command> <expected>
 at_version() {
-  local version expected="${2#v}"
+  case "$(installed_version "$1")" in
-  version="$(installed_version "$1")"
+    *"$2"*) return 0 ;;
-  if [[ "$version" =~ (^|[^0-9.])v?([0-9]+(\.[0-9]+)+) ]]; then
+    *) return 1 ;;
-    [ "${BASH_REMATCH[2]}" = "$expected" ]
+  esac
-  else
-    return 1
-  fi
 }
 
 install_kubeconform() {
@@ -128,15 +120,6 @@ install_shellcheck() {
   rm -rf "$tmp"
 }
 
-install_jq() {
-  if at_version jq "${JQ_VERSION}"; then
-    return 0
-  fi
-  fetch "https://github.com/jqlang/jq/releases/download/jq-${JQ_VERSION}/jq-linux-${goarch}" \
-    "$BIN_DIR/jq"
-  chmod 0755 "$BIN_DIR/jq"
-}
-
 install_uv() {
   if at_version uv "${UV_VERSION}"; then
     return 0
@@ -184,7 +167,6 @@ install_pip_audit() {
 }
 
 install_prettier() {
-  install_node
   if at_version prettier "${PRETTIER_VERSION}"; then
     return 0
   fi
@@ -245,41 +227,28 @@ install_actionlint() {
   rm -rf "$tmp"
 }
 
-main() {
+wanted=("$@")
-  wanted=("$@")
+if [ "${#wanted[@]}" -eq 0 ]; then
-  if [ "${#wanted[@]}" -eq 0 ]; then
+  wanted=(kubeconform shellcheck actionlint prettier ruff yamllint hadolint)
-    wanted=(node jq kubeconform shellcheck actionlint prettier ruff yamllint hadolint)
+fi
-  fi
 
-  for tool in "${wanted[@]}"; do
+for tool in "${wanted[@]}"; do
-    case "$tool" in
+  case "$tool" in
-      kubeconform) install_kubeconform ;;
+    kubeconform) install_kubeconform ;;
-      shellcheck) install_shellcheck ;;
+    shellcheck) install_shellcheck ;;
-      jq) install_jq ;;
+    actionlint) install_actionlint ;;
-      actionlint) install_actionlint ;;
+    prettier) install_prettier ;;
-      prettier) install_prettier ;;
+    ruff) install_ruff ;;
-      ruff) install_ruff ;;
+    yamllint) install_yamllint ;;
-      yamllint) install_yamllint ;;
+    pip-audit) install_pip_audit ;;
-      pip-audit) install_pip_audit ;;
+    hadolint) install_hadolint ;;
-      hadolint) install_hadolint ;;
+    node) install_node ;;
-      node) install_node ;;
+    uv) install_uv ;;
-      uv) install_uv ;;
+    *)
-      *)
+      echo "install-ci-tools: unknown tool: $tool" >&2
-        echo "install-ci-tools: unknown tool: $tool" >&2
+      exit 1
-        exit 1
+      ;;
-        ;;
+  esac
-    esac
+done
-  done
 
-  for old in "$BIN_DIR"/node-* "$BIN_DIR"/prettier-*; do
+printf '%s\n' "$BIN_DIR"
-    [ -d "$old" ] || continue
-    case "$(basename "$old")" in
-      "node-$NODE_VERSION"|"prettier-$PRETTIER_VERSION") ;;
-      *) rm -rf "$old" ;;
-    esac
-  done
-  if [ -x "$BIN_DIR/uv" ]; then "$BIN_DIR/uv" cache prune >/dev/null; fi
-  printf '%s\n' "$BIN_DIR"
-}
-
-if [ "${BASH_SOURCE[0]}" = "$0" ]; then main "$@"; fi

.gitea/workflows/release.py

@@ -1,350 +0,0 @@
-#!/usr/bin/env python3
-"""CI release artifacts and the SHA-specific Gitea deployment gate (stdlib only)."""
-
-import argparse
-import hashlib
-import io
-import itertools
-import json
-import os
-import re
-import shutil
-import subprocess
-import sys
-import tempfile
-import urllib.error
-import urllib.parse
-import urllib.request
-import zipfile
-from pathlib import Path
-
-SHA = re.compile(r'[0-9a-f]{40}')
-DIGEST = re.compile(r'sha256:[0-9a-f]{64}')
-IMAGES = {
-    'error-pages': ('errorpages', 'errorpages/Dockerfile'),
-    'forust-homepage': ('homepages', 'homepages/Dockerfile.forust'),
-    'xdfnx-homepage': ('homepages', 'homepages/Dockerfile.xdfnx'),
-}
-
-# These images are released by the EDU application repository.
-EXTERNAL_IMAGES = {'gcr.forust.xyz/forust/session-keeper', 'gcr.forust.xyz/forust/webinar-checker'}
-
-
-def command(*args, **kwargs):
-    """Arguments are passed directly to the executable, never to a shell."""
-    return subprocess.check_output(args, text=True, **kwargs).strip()  # noqa: S603, S607
-
-
-def validate_release(data, sha=None):
-    if data.get('version') != 1 or not SHA.fullmatch(data.get('sha', '')):
-        raise ValueError('Invalid release version or SHA')
-    if sha is not None and data['sha'] != sha:
-        raise ValueError('Release SHA does not match the checked CI commit')
-    expected = {f'gcr.forust.xyz/forust/{name}' for name in IMAGES}
-    if set(data.get('images', {})) != expected:
-        raise ValueError('Release must contain all owned images')
-    if not all(DIGEST.fullmatch(value) for value in data['images'].values()):
-        raise ValueError('Release has an invalid image digest')
-    if set(data.get('inputs', {})) != expected or not all(
-        re.fullmatch(r'[0-9a-f]{64}', value) for value in data['inputs'].values()
-    ):
-        raise ValueError('Release has invalid build input fingerprints')
-    return data
-
-
-class NoRedirect(urllib.request.HTTPRedirectHandler):
-    def redirect_request(self, _req, _fp, _code, _msg, _headers, _newurl):
-        return None
-
-
-class Gitea:
-    def __init__(self):
-        self.origin = os.environ['GITHUB_SERVER_URL'].rstrip('/')
-        if urllib.parse.urlsplit(self.origin).scheme != 'https':
-            raise ValueError('Gitea API must use HTTPS')
-        self.repository = os.environ['GITHUB_REPOSITORY']
-        if not re.fullmatch(r'[\w.-]+/[\w.-]+', self.repository):
-            raise ValueError('Invalid Gitea repository')
-        self.token = os.environ['GITEA_TOKEN']
-        self.base = f'{self.origin}/api/v1/repos/{self.repository}'
-
-    def request(self, url, *, archive=False):
-        if not url.startswith(self.base + '/'):
-            raise ValueError('Refusing to send the Actions token to another origin')
-        req = urllib.request.Request(url, headers={'Authorization': f'token {self.token}'})  # noqa: S310 -- HTTPS origin validated above
-        opener = urllib.request.build_opener(NoRedirect())
-        try:
-            response = opener.open(req, timeout=30)  # noqa: S310
-        except urllib.error.HTTPError as error:
-            if not archive or error.code not in (301, 302, 303, 307, 308):
-                raise RuntimeError(f'Gitea API returned HTTP {error.code}') from None
-            target = urllib.parse.urljoin(url, error.headers['Location'])
-            if urllib.parse.urlsplit(target).scheme != 'https':
-                raise ValueError('Artifact redirect must use HTTPS') from None
-            # Signed storage redirects must never receive the Gitea token.
-            response = urllib.request.urlopen(target, timeout=30)  # noqa: S310
-        with response:
-            payload = response.read(8 * 1024 * 1024 + 1)
-        if len(payload) > 8 * 1024 * 1024:
-            raise ValueError('Gitea response exceeds 8 MiB')
-        return payload if archive else json.loads(payload)
-
-    def pages(self, path, key, **params):
-        for page in range(1, 101):
-            query = urllib.parse.urlencode({**params, 'page': page, 'limit': 50})
-            data = self.request(f'{self.base}/{path}?{query}')
-            entries = data[key]
-            yield from entries
-            if len(entries) < 50:
-                return
-        raise RuntimeError('Gitea pagination limit exceeded')
-
-    def successful_runs(self, sha=None):
-        params = {'branch': 'main', 'status': 'success', 'exclude_pull_requests': 'true'}
-        if sha:
-            params['head_sha'] = sha
-        for run in self.pages('actions/workflows/ci.yaml/runs', 'workflow_runs', **params):
-            if (
-                run.get('status') == 'completed'
-                and run.get('conclusion') == 'success'
-                and run.get('head_branch') == 'main'
-                and run.get('event') in ('push', 'workflow_dispatch')
-                and (run.get('repository') or {}).get('full_name') == self.repository
-                and (run.get('head_repository') or run.get('repository') or {}).get('full_name') == self.repository
-                and (sha is None or run.get('head_sha') == sha)
-            ):
-                yield run
-
-    def release(self, run):
-        sha = run['head_sha']
-        jobs = list(self.pages(f'actions/runs/{run["id"]}/jobs', 'jobs'))
-        # A green workflow with a skipped build must not authorize a deploy.
-        if not any(job.get('name') == 'build' and job.get('conclusion') == 'success' for job in jobs):
-            raise ValueError('CI build job did not succeed')
-        artifacts = self.request(f'{self.base}/actions/runs/{run["id"]}/artifacts')['artifacts']
-        matching = [a for a in artifacts if a['name'] == f'release-{sha}' and not a.get('expired')]
-        if len(matching) != 1:
-            raise ValueError('CI release artifact is missing, expired or ambiguous; rerun CI')
-        blob = self.request(f'{self.base}/actions/artifacts/{matching[0]["id"]}/zip', archive=True)
-        with zipfile.ZipFile(io.BytesIO(blob)) as archive:
-            files = [entry for entry in archive.infolist() if not entry.is_dir()]
-            if len(files) != 1 or files[0].filename != 'release.json' or files[0].file_size > 256 * 1024:
-                raise ValueError('Unexpected release archive contents')
-            return validate_release(json.loads(archive.read(files[0])), sha)
-
-
-def fingerprint(context, dockerfile):
-    tree = command('git', 'ls-tree', '-r', 'HEAD', '--', context, dockerfile, '.gitea/workflows/release.py')
-    return hashlib.sha256(tree.encode()).hexdigest()
-
-
-def gate(output, requested_ref, event_sha):
-    command('git', 'fetch', '--quiet', 'origin', 'main')
-    if event_sha:
-        if not SHA.fullmatch(event_sha):
-            raise ValueError('Invalid workflow_run SHA')
-        sha = event_sha
-    else:
-        if requested_ref == 'main':
-            requested_ref = 'origin/main'
-        sha = command('git', 'rev-parse', '--verify', '--end-of-options', f'{requested_ref}^{{commit}}')
-    if not SHA.fullmatch(sha):
-        raise ValueError('Invalid deploy SHA')
-    command('git', 'merge-base', '--is-ancestor', sha, 'origin/main')
-    api = Gitea()
-    runs = list(api.successful_runs(sha))
-    if not runs:
-        raise ValueError(f'No successful main CI for {sha}; run CI before deploying')
-    release = api.release(max(runs, key=lambda run: run['id']))
-    output.write_text(json.dumps(release, indent=2) + '\n')
-    if os.environ.get('GITHUB_OUTPUT'):
-        with Path(os.environ['GITHUB_OUTPUT']).open('a') as stream:
-            stream.write(f'sha={sha}\n')
-    print(f'CI gate accepted {sha}')
-
-
-def build(output):
-    sha = command('git', 'rev-parse', 'HEAD')
-    if sha != os.environ['GITHUB_SHA'] or not SHA.fullmatch(sha):
-        raise ValueError('Build checkout does not match GITHUB_SHA')
-    api = Gitea()
-    previous = None
-    for run in sorted(itertools.islice(api.successful_runs(), 50), key=lambda item: item['id'], reverse=True):
-        if str(run['id']) == os.environ.get('GITHUB_RUN_ID'):
-            continue
-        try:
-            previous = api.release(run)
-            break
-        except ValueError:
-            # Expired artifacts only cost a rebuild; mutable tags are never a fallback.
-            continue
-    docker_config = tempfile.mkdtemp(prefix='homelab-registry-')
-    builder_config = Path.home() / '.cache/homelab-ci/buildx'
-    builder_config.mkdir(parents=True, exist_ok=True)
-    env = {**os.environ, 'DOCKER_CONFIG': docker_config, 'BUILDX_CONFIG': str(builder_config)}
-    try:
-        subprocess.run(  # noqa: S603, S607
-            [
-                shutil.which('docker') or '/usr/bin/docker',
-                'login',
-                'gcr.forust.xyz',
-                '-u',
-                os.environ['REGISTRY_USERNAME'],
-                '--password-stdin',
-            ],
-            input=os.environ['REGISTRY_PASSWORD'],
-            text=True,
-            check=True,
-            env=env,
-        )
-        builder = 'homelab-ci'
-        versions = dict(
-            re.findall(r'^([A-Z_]+)="([^"\n]+)"$', Path('.gitea/workflows/tool-versions.env').read_text(), re.MULTILINE)
-        )
-        image = versions['BUILDKIT_IMAGE']
-        signature = builder_config / 'homelab-ci-image'
-        exists = (
-            subprocess.run(  # noqa: S603
-                [shutil.which('docker') or '/usr/bin/docker', 'buildx', 'inspect', builder],
-                capture_output=True,
-                env=env,
-            ).returncode
-            == 0
-        )
-        if exists and (not signature.exists() or signature.read_text().strip() != image):
-            command('docker', 'buildx', 'rm', '--keep-state', builder, env=env)
-            exists = False
-        if not exists:
-            command(
-                'docker',
-                'buildx',
-                'create',
-                '--name',
-                builder,
-                '--driver',
-                'docker-container',
-                '--driver-opt',
-                f'image={image}',
-                '--buildkitd-config',
-                '.gitea/runner/buildkitd.toml',
-                env=env,
-            )
-            signature.write_text(image + '\n')
-        release = {'version': 1, 'sha': sha, 'images': {}, 'inputs': {}}
-        for name, (context, dockerfile) in IMAGES.items():
-            image = f'gcr.forust.xyz/forust/{name}'
-            inputs = fingerprint(context, dockerfile)
-            old_digest = (previous or {}).get('images', {}).get(image)
-            exists = False
-            if old_digest and previous['inputs'].get(image) == inputs:
-                exists = (
-                    subprocess.run(  # noqa: S603, S607
-                        [
-                            shutil.which('docker') or '/usr/bin/docker',
-                            'buildx',
-                            'imagetools',
-                            'inspect',
-                            f'{image}@{old_digest}',
-                        ],
-                        capture_output=True,
-                        env=env,
-                        timeout=60,
-                    ).returncode
-                    == 0
-                )
-            if exists:
-                print(f'Reuse {name}: inputs unchanged')
-                digest = old_digest
-            else:
-                print(f'Build {name}', flush=True)
-                metadata = Path(docker_config) / 'metadata.json'
-                command(
-                    'docker',
-                    'buildx',
-                    'build',
-                    '--builder',
-                    builder,
-                    '--push',
-                    '--platform',
-                    'linux/amd64',
-                    '--provenance=false',
-                    '--cache-from',
-                    f'type=registry,ref={image}:buildcache',
-                    '--cache-to',
-                    f'type=registry,ref={image}:buildcache,mode=max',
-                    '--tag',
-                    f'{image}:sha-{sha}',
-                    '--metadata-file',
-                    str(metadata),
-                    '--file',
-                    dockerfile,
-                    context,
-                    env=env,
-                )
-                digest = json.loads(metadata.read_text())['containerimage.digest']
-            release['images'][image] = digest
-            release['inputs'][image] = inputs
-        validate_release(release, sha)
-        output.write_text(json.dumps(release, indent=2) + '\n')
-    finally:
-        # Cleanup errors must neither leak credentials nor mask the original build error.
-        try:
-            subprocess.run(  # noqa: S603
-                [
-                    shutil.which('docker') or '/usr/bin/docker',
-                    'buildx',
-                    'prune',
-                    '--builder',
-                    'homelab-ci',
-                    '--force',
-                    '--max-used-space',
-                    '1gb',
-                ],
-                env=env,
-                timeout=60,
-            )
-        except (OSError, subprocess.TimeoutExpired):
-            print('CI builder cache cleanup deferred', flush=True)
-        finally:
-            shutil.rmtree(docker_config)
-
-
-def render(stream, destination):
-    release = validate_release(json.loads(Path(os.environ['RELEASE_FILE']).read_text()), os.environ['DEPLOY_SHA'])
-    image_line = re.compile(
-        r"^(\s*(?:-\s*)?image:\s*)(['\"]?)(gcr\.forust\.xyz/forust/[\w.-]+)(?::[\w.-]+|@sha256:[0-9a-f]{64})\2(\s*(?:#.*)?)$"
-    )
-    rendered = []
-    for line in stream:
-        match = image_line.fullmatch(line.rstrip('\n'))
-        if match:
-            prefix, quote, image, tail = match.groups()
-            if image in EXTERNAL_IMAGES and f'{image}@sha256:' in line:
-                rendered.append(line)
-                continue
-            if image not in release['images']:
-                raise ValueError(f'Owned image missing from checked release: {image}')
-            line = f'{prefix}{quote}{image}@{release["images"][image]}{quote}{tail}\n'
-        elif re.match(r'\s*(?:-\s*)?image:', line) and 'gcr.forust.xyz/forust/' in line:
-            raise ValueError('Unsupported owned image syntax; refusing to apply a mutable tag')
-        rendered.append(line)
-    destination.writelines(rendered)
-
-
-def main():
-    parser = argparse.ArgumentParser(description=__doc__)
-    parser.add_argument('action', choices=('build', 'gate', 'render'))
-    parser.add_argument('--output', type=Path, default=Path('release.json'))
-    parser.add_argument('--ref', default='main')
-    parser.add_argument('--event-sha', default='')
-    args = parser.parse_args()
-    if args.action == 'render':
-        render(sys.stdin, sys.stdout)
-    elif args.action == 'gate':
-        gate(args.output, args.ref, args.event_sha)
-    else:
-        build(args.output)
-
-
-if __name__ == '__main__':
-    main()

.gitea/workflows/renovate-ci.yaml

@@ -2,23 +2,9 @@ name: renovate-ci
 
 on:
   pull_request:
-    paths:
-      - "renovate/**"
-      - ".gitea/workflows/renovate-ci.yaml"
-      - ".gitea/workflows/sync-renovate-configmap.sh"
-      - ".gitea/workflows/compose-lint.sh"
-      - ".gitea/workflows/install-ci-tools.sh"
-      - ".gitea/workflows/tool-versions.env"
   push:
     branches:
       - main
-    paths:
-      - "renovate/**"
-      - ".gitea/workflows/renovate-ci.yaml"
-      - ".gitea/workflows/sync-renovate-configmap.sh"
-      - ".gitea/workflows/compose-lint.sh"
-      - ".gitea/workflows/install-ci-tools.sh"
-      - ".gitea/workflows/tool-versions.env"
   workflow_dispatch:
 
 permissions:
@@ -26,7 +12,7 @@ permissions:
 
 jobs:
   validate-renovate:
-    runs-on: homelab
+    runs-on: [self-hosted, linux, arch, homelab]
     timeout-minutes: 20
     steps:
       - name: Checkout repository

.gitea/workflows/renovate-run.yaml

@@ -32,7 +32,7 @@ concurrency:
 
 jobs:
   run-renovate:
-    runs-on: homelab
+    runs-on: [self-hosted, linux, arch, homelab]
     timeout-minutes: 60
     steps:
       - name: Checkout repository
@@ -81,7 +81,7 @@ jobs:
           docker run --rm \
             -v "$PWD/renovate/renovate.json:/opt/renovate/renovate.json:ro" \
             -e RENOVATE_PLATFORM=gitea \
-            -e RENOVATE_ENDPOINT=https://git.forust.xyz/api/v1 \
+            -e RENOVATE_ENDPOINT=https://gitea.forust.xyz/api/v1 \
             -e RENOVATE_TOKEN="$RENOVATE_TOKEN" \
             -e RENOVATE_GITHUB_COM_TOKEN="${RENOVATE_GITHUB_COM_TOKEN:-}" \
             -e RENOVATE_REPOSITORIES="${RENOVATE_REPOSITORIES:-forust/homelab}" \

.gitea/workflows/secret-references.jq

@@ -1,13 +0,0 @@
-# kubectl emits a List for files containing multiple resources.
-(if .kind == "List" then .items[] else . end)
-| (.metadata.namespace // "default") as $ns
-| [
-    (.. | objects
-      | (.secretRef? // empty), (.secretKeyRef? // empty), (.secret? // empty)
-      | select(.optional != true)
-      | .name // .secretName // empty),
-    (.. | objects | .imagePullSecrets[]?.name)
-  ]
-| unique[]
-| select(. != null and . != "")
-| "\($ns) \(.)"

.gitea/workflows/ssh-run.sh

@@ -1,56 +1,71 @@
 #!/usr/bin/env bash
-# The SSH client submits once and follows durable stages on workstation.
+# usage: ssh-run.sh <stage>
+# Runs one deploy-lib.sh stage on the workstation over SSH.
 set -euo pipefail
+
 : "${DEPLOY_HOST:?missing DEPLOY_HOST}"
 : "${DEPLOY_USER:?missing DEPLOY_USER}"
 : "${DEPLOY_KEY:?missing DEPLOY_SSH_KEY}"
-: "${DEPLOY_KNOWN_HOSTS:?configure pinned DEPLOY_KNOWN_HOSTS}"
+
-: "${DEPLOY_RUN_ID:?missing DEPLOY_RUN_ID}"
+deploy_port="${DEPLOY_PORT:-22}"
-[[ "$DEPLOY_USER" =~ ^[A-Za-z_][A-Za-z0-9_.-]*$ ]] || exit 1
+deploy_path="${DEPLOY_PATH:-/srv/homelab}"
-[[ "$DEPLOY_HOST" =~ ^[A-Za-z0-9_.:-]+$ ]] || exit 1
+deploy_path="$(printf '%s' "$deploy_path" | tr -d '\"' | tr -d '\r' | xargs)"
-[[ "$DEPLOY_RUN_ID" =~ ^[0-9]+-[0-9]+$ ]] || exit 1
+
-[[ "${DEPLOY_PORT:-22}" =~ ^[0-9]+$ ]] || exit 1
+# The private key is written to a per-run directory that is removed on exit, so a
+# failed or cancelled job cannot leave deploy credentials in the runner's temp
+# directory. Do not use a fixed path: apply-k8s and apply-compose run in parallel.
 key_dir="$(mktemp -d "${RUNNER_TEMP:-/tmp}/homelab-deploy-key.XXXXXXXX")"
-trap 'rm -rf "$key_dir"' EXIT
+trap 'rm -rf "$key_dir"' EXIT INT TERM
-chmod 700 "$key_dir"
+
-printf '%s\n' "$DEPLOY_KEY" >"$key_dir/key"
+ssh_key="$key_dir/deploy_key"
-printf '%s\n' "$DEPLOY_KNOWN_HOSTS" >"$key_dir/known_hosts"
+printf '%s\n' "$DEPLOY_KEY" > "$ssh_key"
-chmod 600 "$key_dir/key" "$key_dir/known_hosts"
+chmod 600 "$ssh_key"
-ssh_opts=(-i "$key_dir/key" -p "${DEPLOY_PORT:-22}" -o BatchMode=yes -o StrictHostKeyChecking=yes
+
-  -o "UserKnownHostsFile=$key_dir/known_hosts" -o ConnectTimeout=15
+# A connection that died silently used to hang until the job timeout, and the
-  -o ServerAliveInterval=15 -o ServerAliveCountMax=4)
+# stage was never re-run: one flaky TCP session cost a whole 45-minute apply.
-controller=.local/lib/homelab-deploy/controller.py
+# ServerAlive* bounds how long a dead peer goes unnoticed, ConnectTimeout bounds
-case "${1:?start, apply, verify or smoke required}" in
+# setup. Only exit 255 - ssh's own transport failures - is retried. A stage that
-  start)
+# fails on its own merits exits with the remote's status, so a real failure
-    python3 - <<'PY' >"$key_dir/request.json"
+# still surfaces its own log instead of burning three attempts. The stages are
-import json
+# declarative applies, so re-running one that had already committed is harmless.
-import os
+ssh_opts=(
-from pathlib import Path
+  -i "$ssh_key" -p "$deploy_port"
-release = json.loads(Path('release.json').read_text())
+  -o BatchMode=yes -o StrictHostKeyChecking=accept-new
-print(json.dumps({'release': release, 'mode': os.environ.get('DEPLOY_MODE', 'changed'),
+  -o ConnectTimeout=15
-                  'refresh_images': os.environ.get('REFRESH_IMAGES', 'false') == 'true'}))
+  -o ServerAliveInterval=15 -o ServerAliveCountMax=4
-PY
+)
-    for attempt in 1 2 3; do
+
-      rc=0
+rc=0
-      # shellcheck disable=SC2029 # The run ID and operation are validated local arguments, not remote variables.
+# apply-k8s and apply-compose are separate workflow jobs so the graph stays
-      ssh "${ssh_opts[@]}" "$DEPLOY_USER@$DEPLOY_HOST" python3 "$controller" start "$DEPLOY_RUN_ID" <"$key_dir/request.json" || rc=$?
+# intact for the verify job, but on a single node they must not run at once:
-      [ "$rc" -eq 0 ] && exit 0
+# host docker churn on top of cluster churn is what melts the node (load 40+,
-      [ "$rc" -eq 255 ] || exit "$rc"
+# netbird/ssh die, helm is left pending-*). Serialize them on the workstation
-      sleep 5
+# with a shared lock; whoever arrives second waits.
-    done
+remote_cmd=(bash -se)
-    exit "$rc"
+case "$1" in
+  apply-k8s | apply-compose)
+    remote_cmd=(flock -w 5400 /tmp/homelab-apply.lock bash -se)
     ;;
-  apply|verify|smoke)
-    for attempt in 1 2 3; do
-      rc=0
-      # shellcheck disable=SC2029 # The run ID and operation are validated local arguments, not remote variables.
-      ssh "${ssh_opts[@]}" "$DEPLOY_USER@$DEPLOY_HOST" python3 "$controller" follow "$DEPLOY_RUN_ID" "$1" || rc=$?
-      [ "$rc" -eq 0 ] && exit 0
-      [ "$rc" -eq 255 ] || exit "$rc"
-      echo "SSH disconnected; reconnecting to the existing deploy ($attempt/3)"
-      sleep 5
-    done
-    exit "$rc"
-    ;;
-  *) echo "Unknown SSH operation: $1" >&2; exit 1 ;;
 esac
+for attempt in 1 2 3; do
+  if [ "$attempt" -gt 1 ]; then
+    echo ":: warning::ssh transport failed, retrying (${attempt}/3)"
+    sleep $((attempt * 5))
+  fi
+  rc=0
+  # shellcheck disable=SC2029 # remote_cmd/ssh_opts expand on the client on purpose: they select the local ssh invocation, only the heredoc runs remotely.
+  ssh "${ssh_opts[@]}" "${DEPLOY_USER}@${DEPLOY_HOST}" \
+    env "REPO=$deploy_path" "APPLY_PRUNE=${APPLY_PRUNE:-false}" \
+    "DEPLOY_SHA=${DEPLOY_SHA:-}" "DEPLOY_SNAPSHOT_DIR=${DEPLOY_SNAPSHOT_DIR:-}" \
+    "STAGE=$1" "${remote_cmd[@]}" <<'EOF' || rc=$?
+source "$REPO/.gitea/workflows/deploy-lib.sh"
+run_stage "$STAGE"
+EOF
+  [ "$rc" -eq 0 ] && break
+  [ "$rc" -ne 255 ] && break
+done
+
+if [ "$rc" -ne 0 ]; then
+  echo ":: error::stage $1 failed over ssh (exit $rc)"
+fi
+exit "$rc"

.gitea/workflows/tool-versions.env

@@ -31,9 +31,3 @@ UV_VERSION="0.12.17"
 # so the tree that gets tested is the tree that gets built. Renovate keeps this
 # in step with the Dockerfile's node: tag via the "node runtime" group.
 NODE_VERSION="22.23.3"
-
-# Secret-reference regression tests parse rendered Kubernetes objects.
-JQ_VERSION="1.8.1"
-
-# BuildKit is the only auxiliary CI container; jobs themselves stay on the host.
-BUILDKIT_IMAGE="moby/buildkit:v0.33.1"

adguardhome/compose.yaml

@@ -31,7 +31,7 @@ services:
       - "traefik.http.routers.adguard-dev.entrypoints=websecure"
       - "traefik.http.routers.adguard-dev.tls=true"
       # DoH Router
-      - "traefik.http.routers.dns-over-https.rule=(Host(`dns.forust.xyz`) || Host(`adguard.forust.xyz`)) && PathPrefix(`/dns-query`)"
+      - "traefik.http.routers.dns-over-https.rule=(Host(`dns.forust.xyz` || Host(`adguard.forust.xyz`)) && PathPrefix(`/dns-query`))"
       - "traefik.http.routers.dns-over-https.entrypoints=websecure"
       - "traefik.http.routers.dns-over-https.tls.certresolver=letsencrypt"
 

adguardhome/k8s/adguard.yaml

@@ -51,8 +51,6 @@ spec:
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: adguard-deployment
   namespace: adguard
 spec:
@@ -66,8 +64,39 @@ spec:
     metadata:
       labels:
         app: adguard
+      annotations:
+        reloader.stakater.com/auto: "true"
     spec:
       containers:
+        - name: netbird
+          image: netbirdio/netbird:0.80.0
+          envFrom:
+            - configMapRef:
+                name: adguard-config
+          env:
+            - name: NB_SETUP_KEY
+              valueFrom:
+                secretKeyRef:
+                  name: adguard-netbird-secrets
+                  key: NB_SETUP_KEY
+          securityContext:
+            capabilities:
+              add:
+                - NET_ADMIN
+                - SYS_ADMIN
+                - SYS_RESOURCE
+          resources:
+            requests:
+              memory: "64Mi"
+              cpu: "50m"
+            limits:
+              memory: "256Mi"
+              cpu: "200m"
+          volumeMounts:
+            - name: netbird-state
+              mountPath: /var/lib/netbird
+            - name: dev-tun
+              mountPath: /dev/net/tun
         - name: adguard
           image: adguard/adguardhome:v0.107.79
           resources:
@@ -84,13 +113,6 @@ spec:
               name: dns
             - containerPort: 853
               name: dot
-          readinessProbe:
-            tcpSocket:
-              port: dns
-            initialDelaySeconds: 5
-            periodSeconds: 5
-            successThreshold: 1
-            failureThreshold: 3
           volumeMounts:
             - name: adguard-data
               mountPath: /opt/adguardhome/work
@@ -102,6 +124,12 @@ spec:
               mountPath: /certs
               readOnly: true
       volumes:
+        - name: netbird-state
+          emptyDir: {}
+        - name: dev-tun
+          hostPath:
+            path: /dev/net/tun
+            type: CharDevice
         - name: adguard-data
           persistentVolumeClaim:
             claimName: adguard-pvc

adguardhome/k8s/config.yaml

@@ -0,0 +1,10 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: adguard-config
+  namespace: adguard
+data:
+  NB_MANAGEMENT_URL: "https://nb.forust.xyz"
+  NB_HOSTNAME: "adguard"
+  NB_LOG_LEVEL: "info"
+  NB_DISABLE_DNS: "true"

adguardhome/k8s/netbird-secrets.yaml.example

@@ -0,0 +1,8 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: adguard-netbird-secrets
+  namespace: adguard
+type: Opaque
+stringData:
+  NB_SETUP_KEY: "REPLACE_ME"

authentik/k8s/authentik.yaml

@@ -27,8 +27,6 @@ spec:
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: authentik-server-deployment
   namespace: authentik
 spec:
@@ -65,8 +63,6 @@ spec:
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: authentik-worker-deployment
   namespace: authentik
 spec:

cfddns/k8s/deployment.yaml

@@ -1,8 +1,6 @@
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: cfddns
   labels:
     app: cfddns

checkmk/k8s/checkmk.yaml

@@ -17,8 +17,6 @@ spec:
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: checkmk-deployment
   namespace: checkmk
 spec:

cloudflared/k8s/deployment.yaml

@@ -1,8 +1,6 @@
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: cloudflared
   labels:
     app: cloudflared
@@ -20,7 +18,7 @@ spec:
     spec:
       containers:
         - name: cloudflared
-          image: cloudflare/cloudflared:2026.10.0
+          image: cloudflare/cloudflared:2026.9.3
           imagePullPolicy: IfNotPresent
           args:
             - tunnel

converters/k8s/convertx.yaml

@@ -13,8 +13,6 @@ spec:
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: convertx-deployment
   namespace: converters
 spec:

edu_master/README.md

@@ -1,14 +0,0 @@
-# EDU deployment ownership
-
-Application source and release builds: `forust/edu-master`.
-The homelab pipeline deploys `edu_master/k8s` and preserves explicit image digests.
-The application copies in this directory are legacy and are not build inputs.
-Do not publish EDU `prod` images from homelab or resolve releases from moving tags.
-
-For an EDU release, validate both images, select their digests in the keeper and
-checker manifests, and run the existing homelab validation/apply/verification
-helpers against this service. Keep the existing Secret and Redis PVC.
-Coordinate Redis authentication changes with both clients and all init/probes;
-keep a pre-rollout Redis backup and both previous compatible image references.
-The current HTTP checker does not depend on Playwright; check other consumers
-before removing the separate browser service.

edu_master/k8s/alerts.yaml

@@ -50,36 +50,7 @@ spec:
             severity: critical
           annotations:
             summary: "Webinar checker failing consecutively"
-            description: 'edu-master/webinar-checker: {{ $value }} consecutive webinar check failures (timeout / http error / page error). Check pod logs (Loki: {namespace="edu-master", container="webinar-checker"}).'
+            description: 'edu-master/webinar-checker: {{ $value }} consecutive webinar check failures (timeout / playwright error / page error). Check pod logs (Loki: {namespace="edu-master", container="webinar-checker"}).'
-
-        - alert: WebinarCheckerNeverStarted
-          expr: |
-            (time() - edu_process_start > 120)
-              and (webinar_check_last_run_timestamp_seconds == 0)
-          for: 2m
-          labels:
-            severity: critical
-          annotations:
-            summary: "Webinar checker job has not started"
-            description: "The process exposes metrics but its webinar job has never started."
-
-        - alert: WebinarDeliveryPending
-          expr: edu_delivery_pending > 0
-          for: 5m
-          labels:
-            severity: warning
-          annotations:
-            summary: "Webinar notifications await delivery"
-            description: "Telegram delivery has pending recipients. Check delivery failures and retry status."
-
-        - alert: EduRedisUnavailable
-          expr: edu_redis_connected == 0
-          for: 2m
-          labels:
-            severity: critical
-          annotations:
-            summary: "EDU checker cannot reach Redis"
-            description: "Redis health checks are failing; checker commands and delivery may be unavailable."
 
         # Metrics endpoint not scraped for 10m: pod down, metrics server dead, or ServiceMonitor broken.
         - alert: WebinarCheckerScrapeDown
@@ -103,7 +74,7 @@ spec:
             summary: "EDU_PHPSESSID missing"
             description: "edu-master: EDU_PHPSESSID absent from redis for 10m. Webinar/diari/schedule checks are all skipped. Check session-keeper logs and EDU credentials."
 
-        # Hard deps: checker deployment unavailable.
+        # Hard deps: checker and playwright deployments unavailable.
         - alert: WebinarCheckerDeploymentDown
           expr: |
             kube_deployment_status_replicas_unavailable{deployment="webinar-checker", namespace="edu-master"} > 0
@@ -113,3 +84,13 @@ spec:
           annotations:
             summary: "Webinar checker deployment unavailable"
             description: "edu-master/webinar-checker deployment has {{ $value }} unavailable replica(s) for 10m."
+
+        - alert: PlaywrightServiceDown
+          expr: |
+            kube_deployment_status_replicas_unavailable{deployment="playwright-service", namespace="edu-master"} > 0
+          for: 10m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Playwright service unavailable"
+            description: "edu-master/playwright-service deployment has {{ $value }} unavailable replica(s) for 10m. All webinar/diari/schedule checks fail without it."

edu_master/k8s/redis-networkpolicy.yaml

@@ -1,22 +0,0 @@
-apiVersion: networking.k8s.io/v1
-kind: NetworkPolicy
-metadata:
-  name: redis-clients-only
-  namespace: edu-master
-spec:
-  podSelector:
-    matchLabels:
-      app: edu-master-redis
-  policyTypes:
-    - Ingress
-  ingress:
-    - from:
-        - podSelector:
-            matchLabels:
-              app: edu-master-session-keeper
-        - podSelector:
-            matchLabels:
-              app: edu-master-webinar-checker
-      ports:
-        - protocol: TCP
-          port: 6379

edu_master/k8s/redis.yaml

@@ -20,27 +20,6 @@ spec:
         - name: redis
           image: redis:8.10.2-alpine
           imagePullPolicy: IfNotPresent
-          env:
-            - name: REDIS_PASSWORD
-              valueFrom:
-                secretKeyRef:
-                  name: edu-master-secrets
-                  key: REDIS_PASSWORD
-            - name: REDISCLI_AUTH
-              valueFrom:
-                secretKeyRef:
-                  name: edu-master-secrets
-                  key: REDIS_PASSWORD
-          command:
-            - /bin/sh
-            - -ec
-            - |
-              case "$REDIS_PASSWORD" in *[!0-9a-fA-F]*|'') echo 'REDIS_PASSWORD must be 64 hex characters' >&2; exit 1;; esac
-              [ "${#REDIS_PASSWORD}" -eq 64 ] || { echo 'REDIS_PASSWORD must be 64 hex characters' >&2; exit 1; }
-              umask 077
-              printf 'requirepass "%s"\n' "$REDIS_PASSWORD" > /tmp/redis-auth.conf
-              chown redis:redis /tmp/redis-auth.conf
-              exec docker-entrypoint.sh redis-server /tmp/redis-auth.conf
           ports:
             - containerPort: 6379
           volumeMounts:

edu_master/k8s/session-keeper.yaml

@@ -1,8 +1,6 @@
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: session-keeper
   namespace: edu-master
   labels:
@@ -16,20 +14,12 @@ spec:
     type: Recreate
   template:
     metadata:
-      annotations:
-        edu.forust.xyz/source-commit: "90829d6c8080b9928f9da23587678e640939e10a"
       labels:
         app: edu-master-session-keeper
     spec:
       initContainers:
         - name: wait-redis
           image: redis:8.10.2-alpine
-          env:
-            - name: REDISCLI_AUTH
-              valueFrom:
-                secretKeyRef:
-                  name: edu-master-secrets
-                  key: REDIS_PASSWORD
           command:
             - /bin/sh
             - -ec
@@ -43,16 +33,10 @@ spec:
               echo "redis is ready"
       containers:
         - name: session-keeper
-          image: gcr.forust.xyz/forust/session-keeper@sha256:49285e87cc5bc4cf4ffe190813d87927916c2df8a206daac0aeb7d227c636450
+          image: gcr.forust.xyz/forust/session-keeper:prod
           envFrom:
             - secretRef:
                 name: edu-master-secrets
-          env:
-            - name: REDISCLI_AUTH
-              valueFrom:
-                secretKeyRef:
-                  name: edu-master-secrets
-                  key: REDIS_PASSWORD
           resources:
             requests:
               cpu: 25m

edu_master/k8s/webinar-checker.yaml

@@ -1,8 +1,6 @@
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: webinar-checker
   namespace: edu-master
   labels:
@@ -16,22 +14,14 @@ spec:
     type: Recreate
   template:
     metadata:
-      annotations:
-        edu.forust.xyz/source-commit: "90829d6c8080b9928f9da23587678e640939e10a"
       labels:
         app: edu-master-webinar-checker
     spec:
       # Enforces dependency order like compose depends_on:
-      # redis healthy -> session-keeper healthy (EXISTS EDU_PHPSESSID)
+      # redis healthy -> session-keeper healthy (EXISTS EDU_PHPSESSID) -> playwright started
       initContainers:
         - name: wait-deps
           image: redis:8.10.2-alpine
-          env:
-            - name: REDISCLI_AUTH
-              valueFrom:
-                secretKeyRef:
-                  name: edu-master-secrets
-                  key: REDIS_PASSWORD
           command:
             - /bin/sh
             - -ec
@@ -49,9 +39,15 @@ spec:
                 sleep 2
               done
               echo "PHPSESSID ok"
+              until nc -z playwright-service 3000; do
+                i=$((i+1))
+                [ "$i" -ge 300 ] && echo "TIMEOUT: playwright-service not reachable" && exit 1
+                sleep 2
+              done
+              echo "playwright ok"
       containers:
         - name: webinar-checker
-          image: gcr.forust.xyz/forust/webinar-checker@sha256:66c146f7b43cb9f0dc31ba9aa36d217e01df42ddafba5971b79c12ec215b2c01
+          image: gcr.forust.xyz/forust/webinar-checker:prod
           ports:
             - name: metrics
               containerPort: 8000
@@ -64,14 +60,6 @@ spec:
             timeoutSeconds: 3
             failureThreshold: 12
             initialDelaySeconds: 10
-          livenessProbe:
-            httpGet:
-              path: /live
-              port: metrics
-            initialDelaySeconds: 60
-            periodSeconds: 15
-            timeoutSeconds: 3
-            failureThreshold: 4
           envFrom:
             - secretRef:
                 name: edu-master-secrets

edu_master/phpsessid-bot/Dockerfile

@@ -1,4 +1,4 @@
-FROM python:3.14-slim
+FROM python:3.11-slim
 
 WORKDIR /app
 

edu_master/webinar-checker/Dockerfile

@@ -1,4 +1,4 @@
-FROM python:3.14-slim
+FROM python:3.11-slim
 
 WORKDIR /app
 

gitea/compose.yaml

@@ -13,12 +13,9 @@ services:
       - GITEA__database__PASSWD=gitea
       - GITEA__database__NAME=gitea
       # Server
-      - GITEA__server__ROOT_URL=https://git.forust.xyz
+      - GITEA__server__ROOT_URL=https://gitea.forust.xyz
       - GITEA__server__SSH_DOMAIN=gitssh.forust.xyz
       - GITEA__server__SSH_PORT=2221
-      # Pin 28.0 defaults explicitly (see k8s/config.yaml for rationale)
-      - GITEA__service__DISABLE_REGISTRATION=true
-      - GITEA__actions__RUN_RETENTION_DAYS=90
       # Mailer
       - GITEA__mailer__ENABLED=true
       - GITEA__mailer__FROM=${SERVICE_EMAIL}
@@ -37,7 +34,7 @@ services:
       - "traefik.http.services.gitea.loadbalancer.server.port=3000"
 
       # Prod Router
-      - "traefik.http.routers.gitea.rule=Host(`git.forust.xyz`) || Host(`gitea.forust.xyz`)"
+      - "traefik.http.routers.gitea.rule=Host(`gitea.forust.xyz`)"
       - "traefik.http.routers.gitea.entrypoints=websecure"
       - "traefik.http.routers.gitea.tls.certresolver"
       # Local Router

gitea/k8s/config.yaml

@@ -4,14 +4,11 @@ metadata:
   name: gitea-config
   namespace: gitea
 data:
-  GITEA__server__ROOT_URL: "https://git.forust.xyz"
+  GITEA__server__DOMAIN: "gitea.forust.xyz"
+  GITEA__server__ROOT_URL: "https://gitea.forust.xyz"
   GITEA__server__SSH_DOMAIN: "gitssh.forust.xyz"
   GITEA__server__SSH_PORT: "2221"
 
-  GITEA__service__DISABLE_REGISTRATION: "true"
-
-  GITEA__actions__RUN_RETENTION_DAYS: "90"
-
   GITEA__database__DB_TYPE: "postgres"
   GITEA__database__HOST: "postgres.database.svc.cluster.local:5432"
   GITEA__database__NAME: "gitea"

gitea/k8s/gitea.yaml

@@ -17,8 +17,6 @@ spec:
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: gitea-deployment
   namespace: gitea
 spec:

glance/k8s/glance-config.yaml

@@ -177,8 +177,8 @@ data:
                   # url: https://gitssh.forust.xyz
                 # - title: gcr.forust.xyz
                   # url: https://gcr.forust.xyz/v2/
-                - title: git.forust.xyz
+                - title: gitea.forust.xyz
-                  url: https://git.forust.xyz
+                  url: https://gitea.forust.xyz
                 - title: nextcloud.forust.xyz
                   url: https://nextcloud.forust.xyz
                 - title: mc.forust.xyz

glance/k8s/glance.yaml

@@ -13,8 +13,6 @@ spec:
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: glance-deployment
   namespace: glance
 spec:
@@ -71,7 +69,7 @@ spec:
             name: glance-config
         - name: glance-assets
           configMap:
-            name: glance-assets
+            name: glance-config
         - name: docker-socket
           hostPath:
             path: /var/run/docker.sock

homarr/.env.example

@@ -1,4 +0,0 @@
-SECRET_ENCRYPTION_KEY="REPLACE_ME"
-TZ="Europe/Bratislava"
-PUID="1000"
-PGID="1000"

homarr/compose.yaml

@@ -1,38 +0,0 @@
-services:
-  homarr:
-    container_name: homarr
-    image: ghcr.io/homarr-labs/homarr:v2.2.0
-    restart: unless-stopped
-    volumes:
-      - ./appdata:/appdata
-      - /var/run/docker.sock:/var/run/docker.sock:ro
-      - ./kubeconfig:/app/config/kubeconfig:ro
-    env_file: .env
-    ports:
-      - 80:7575
-      - 81:3000
-    environment:
-      - TZ=${TZ:-Europe/Bratislava}
-      - TURBO_TELEMETRY_DISABLED=1
-      - KUBECONFIG=/app/config/kubeconfig
-    labels:
-      - "traefik.enable=true"
-      - "traefik.http.services.homarr.loadbalancer.server.port=7575"
-
-      # Prod Router
-      - "traefik.http.routers.homarr.rule=Host(`homarr.forust.xyz`)"
-      - "traefik.http.routers.homarr.entrypoints=websecure"
-      - "traefik.http.routers.homarr.tls.certresolver=letsencrypt"
-      # Local Router
-      - "traefik.http.routers.homarr-local.rule=Host(`homarr.workstation.internal`)"
-      - "traefik.http.routers.homarr-local.entrypoints=websecure"
-      - "traefik.http.routers.homarr-local.tls=true"
-      # Dev Router
-      - "traefik.http.routers.homarr-dev.rule=Host(`homarr.gigaforust.internal`)"
-      - "traefik.http.routers.homarr-dev.entrypoints=websecure"
-      - "traefik.http.routers.homarr-dev.tls=true"
-    networks:
-      - proxy
-networks:
-  proxy:
-    external: true

homarr/k8s/certificates.yaml

@@ -1,28 +0,0 @@
-# apiVersion: cert-manager.io/v1
-# kind: Certificate
-# metadata:
-#   name: home-prod-tls
-#   namespace: homarr
-# spec:
-#   secretName: home-prod-tls
-#   dnsNames:
-#     - home.forust.xyz
-#   issuerRef:
-#     name: letsencrypt-prod
-#     kind: ClusterIssuer
-# ---
-apiVersion: cert-manager.io/v1
-kind: Certificate
-metadata:
-  name: internal-wildcard-tls
-  namespace: homarr
-spec:
-  secretName: internal-wildcard-tls
-  dnsNames:
-    - "*.workstation.internal"
-    - "*.gigaforust.internal"
-    - workstation.internal
-    - gigaforust.internal
-  issuerRef:
-    name: internal-ca
-    kind: ClusterIssuer

homarr/k8s/config.yaml

@@ -1,9 +0,0 @@
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: homarr-config
-  namespace: homarr
-data:
-  TZ: "Europe/Bratislava"
-  TURBO_TELEMETRY_DISABLED: "1"
-  ENABLE_KUBERNETES: "true"

homarr/k8s/homarr.yaml

@@ -1,83 +0,0 @@
-apiVersion: v1
-kind: Service
-metadata:
-  name: homarr-service
-  namespace: homarr
-spec:
-  selector:
-    app: homarr
-  ports:
-    - port: 7575
-      targetPort: 7575
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
-  name: homarr-deployment
-  namespace: homarr
-spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      app: homarr
-  strategy:
-    type: Recreate
-  template:
-    metadata:
-      labels:
-        app: homarr
-    spec:
-      serviceAccountName: homarr
-      containers:
-        - name: homarr
-          image: ghcr.io/homarr-labs/homarr:v2.2.0
-          envFrom:
-            - configMapRef:
-                name: homarr-config
-            - secretRef:
-                name: homarr-secrets
-          ports:
-            - containerPort: 7575
-          readinessProbe:
-            httpGet:
-              path: /
-              port: 7575
-            initialDelaySeconds: 30
-            periodSeconds: 10
-            failureThreshold: 6
-          livenessProbe:
-            httpGet:
-              path: /
-              port: 7575
-            initialDelaySeconds: 60
-            periodSeconds: 30
-            failureThreshold: 3
-          volumeMounts:
-            - name: homarr-data
-              mountPath: /appdata
-          resources:
-            requests:
-              cpu: "250m"
-              memory: "350Mi"
-            limits:
-              cpu: "500m"
-              memory: "700Mi"
-      volumes:
-        - name: homarr-data
-          persistentVolumeClaim:
-            claimName: homarr-pvc
----
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: homarr-pvc
-  namespace: homarr
-spec:
-  resources:
-    requests:
-      storage: 2Gi
-  volumeMode: Filesystem
-  accessModes:
-    - ReadWriteOnce

homarr/k8s/ingress.yaml

@@ -1,33 +0,0 @@
-# apiVersion: traefik.io/v1alpha1
-# kind: IngressRoute
-# metadata:
-#   name: homarr-prod
-#   namespace: homarr
-# spec:
-#   entryPoints:
-#     - websecure
-#   routes:
-#     - match: Host(`home.forust.xyz`)
-#       kind: Rule
-#       services:
-#         - name: homarr-service
-#           port: 7575
-#   tls:
-#     secretName: home-prod-tls
-# ---
-apiVersion: traefik.io/v1alpha1
-kind: IngressRoute
-metadata:
-  name: homarr-local
-  namespace: homarr
-spec:
-  entryPoints:
-    - websecure
-  routes:
-    - match: Host(`home.workstation.internal`) || Host(`home.gigaforust.internal`)
-      kind: Rule
-      services:
-        - name: homarr-service
-          port: 7575
-  tls:
-    secretName: internal-wildcard-tls

homarr/k8s/namespace.yaml

@@ -1,4 +0,0 @@
-apiVersion: v1
-kind: Namespace
-metadata:
-  name: homarr

homarr/k8s/rbac.yaml

@@ -1,58 +0,0 @@
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: homarr
-  namespace: homarr
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRole
-metadata:
-  name: homarr-readonly
-rules:
-  - apiGroups: [""]
-    resources:
-      - pods
-      - services
-      - endpoints
-      - namespaces
-      - nodes
-      - configmaps
-      - persistentvolumeclaims
-      - events
-    verbs: ["get", "list", "watch"]
-  - apiGroups: ["apps"]
-    resources:
-      - deployments
-      - statefulsets
-      - daemonsets
-      - replicasets
-    verbs: ["get", "list", "watch"]
-  - apiGroups: ["networking.k8s.io"]
-    resources:
-      - ingresses
-    verbs: ["get", "list", "watch"]
-  - apiGroups: ["traefik.io"]
-    resources:
-      - ingressroutes
-      - ingressroutetcps
-      - ingressrouteudps
-      - middlewares
-    verbs: ["get", "list", "watch"]
-  - apiGroups: ["metrics.k8s.io"]
-    resources:
-      - pods
-      - nodes
-    verbs: ["get", "list"]
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRoleBinding
-metadata:
-  name: homarr-readonly
-roleRef:
-  apiGroup: rbac.authorization.k8s.io
-  kind: ClusterRole
-  name: homarr-readonly
-subjects:
-  - kind: ServiceAccount
-    name: homarr
-    namespace: homarr

homarr/k8s/secrets.yaml.example

@@ -1,9 +0,0 @@
-apiVersion: v1
-kind: Secret
-metadata:
-  name: homarr-secrets
-  namespace: homarr
-type: Opaque
-stringData:
-  # openssl rand -hex 32
-  SECRET_ENCRYPTION_KEY: "REPLACE_ME"

homepages/forust_files/index.html

@@ -174,7 +174,7 @@
         <h2>./projects</h2>
         <ul class="repo-list">
           <li>
-            <a href="https://git.forust.xyz/forust/gosleep" target="_blank">forust/gosleep</a>
+            <a href="https://gitea.forust.xyz/forust/gosleep" target="_blank">forust/gosleep</a>
             <span class="comment">// linux sleep timer written in rust (originally in go)</span>
           </li>
         </ul>

immich/k8s/immich.yaml

@@ -14,8 +14,6 @@ spec:
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: immich-deployment
   namespace: immich
   labels:

immich/k8s/machine-learning.yaml

@@ -14,8 +14,6 @@ spec:
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: immich-machine-learning-deployment
   namespace: immich
   labels:

immich/k8s/valkey.yaml

@@ -17,8 +17,6 @@ spec:
 apiVersion: apps/v1
 kind: StatefulSet
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: immich-valkey
   namespace: immich
   labels:

kener/k8s/kener.yaml

@@ -13,8 +13,6 @@ spec:
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: kener-deployment
   namespace: kener
 spec:

metube/k8s/metube.yaml

@@ -13,8 +13,6 @@ spec:
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: metube-deployment
   namespace: metube
 spec:

n8n/compose.yaml

@@ -1,6 +1,6 @@
 services:
   n8n:
-    image: docker.n8n.io/n8nio/n8n:2.43.0
+    image: docker.n8n.io/n8nio/n8n:2.42.2
     container_name: n8n
     restart: unless-stopped
     environment:

n8n/k8s/n8n.yaml

@@ -13,8 +13,6 @@ spec:
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: n8n-deployment
   namespace: n8n
 spec:
@@ -31,7 +29,7 @@ spec:
     spec:
       containers:
         - name: n8n
-          image: docker.n8n.io/n8nio/n8n:2.43.0
+          image: docker.n8n.io/n8nio/n8n:2.42.2
           envFrom:
             - configMapRef:
                 name: n8n-config

netbird/entrypoint.sh

@@ -1,109 +0,0 @@
-#!/bin/sh
-set -eu
-
-umask 077
-
-TEMPLATE_PATH=/opt/netbird/config.template.yaml
-RENDERED_PATH=/run/netbird/config.yaml
-RELAY_SECRET_PATH=/run/secrets/relay_auth_secret
-ENCRYPTION_KEY_PATH=/run/secrets/datastore_encryption_key
-
-is_valid_proxy_subnet() {
-  candidate="$1"
-  case "$candidate" in
-    0.0.0.0/0)
-      return 1
-      ;;
-    */*)
-      address="${candidate%%/*}"
-      prefix="${candidate#*/}"
-      ;;
-    *)
-      return 1
-      ;;
-  esac
-
-  case "$prefix" in
-    0|[1-9]|[1-2][0-9]|3[0-2]) ;;
-    *)
-      return 1
-      ;;
-  esac
-
-  old_ifs="$IFS"
-  IFS=.
-  # shellcheck disable=SC2086
-  set -- $address
-  IFS="$old_ifs"
-  [ "$#" -eq 4 ] || return 1
-
-  for octet do
-    case "$octet" in
-      0|[1-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5]) ;;
-      *)
-        return 1
-        ;;
-    esac
-  done
-}
-
-read_secret() {
-  secret_path="$1"
-
-  if [ ! -r "$secret_path" ]; then
-    echo "Required secret is not readable: $secret_path" >&2
-    exit 1
-  fi
-
-  secret_value="$(cat "$secret_path")"
-  if [ -z "$secret_value" ]; then
-    echo "Required secret is empty: $secret_path" >&2
-    exit 1
-  fi
-
-  printf '%s' "$secret_value"
-}
-
-if [ -z "${NETBIRD_DOMAIN:-}" ]; then
-  echo "NETBIRD_DOMAIN must be set" >&2
-  exit 1
-fi
-
-case "$NETBIRD_DOMAIN" in
-  *[!A-Za-z0-9.-]*)
-    echo "NETBIRD_DOMAIN contains unsupported characters" >&2
-    exit 1
-    ;;
-esac
-
-if [ -z "${NETBIRD_PROXY_SUBNET:-}" ] || [ "$NETBIRD_PROXY_SUBNET" = "auto" ]; then
-  echo "NETBIRD_PROXY_SUBNET must be an explicit IPv4 CIDR; run netbird/setup.sh first" >&2
-  exit 1
-fi
-if ! is_valid_proxy_subnet "$NETBIRD_PROXY_SUBNET"; then
-  echo "NETBIRD_PROXY_SUBNET must be a non-default IPv4 CIDR, for example 172.20.0.0/16" >&2
-  exit 1
-fi
-
-if [ "$#" -ne 2 ] || [ "$1" != "--config" ] || [ "$2" != "$RENDERED_PATH" ]; then
-  echo "Expected: --config $RENDERED_PATH" >&2
-  exit 1
-fi
-
-relay_secret="$(read_secret "$RELAY_SECRET_PATH")"
-encryption_key="$(read_secret "$ENCRYPTION_KEY_PATH")"
-
-mkdir -p "$(dirname "$RENDERED_PATH")"
-sed \
-  -e "s|__NETBIRD_DOMAIN__|${NETBIRD_DOMAIN}|g" \
-  -e "s|__NETBIRD_AUTH_SECRET__|${relay_secret}|g" \
-  -e "s|__NETBIRD_ENCRYPTION_KEY__|${encryption_key}|g" \
-  -e "s|__NETBIRD_PROXY_SUBNET__|${NETBIRD_PROXY_SUBNET}|g" \
-  "$TEMPLATE_PATH" >"$RENDERED_PATH"
-
-if grep -q '__NETBIRD_' "$RENDERED_PATH"; then
-  echo "Rendered NetBird configuration still contains unresolved placeholders" >&2
-  exit 1
-fi
-
-exec /go/bin/netbird-server "$@"

netbird/k8s/netbird.yaml

@@ -32,8 +32,6 @@ spec:
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: netbird-server-deployment
   namespace: netbird
 spec:
@@ -128,8 +126,6 @@ spec:
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: netbird-dashboard-deployment
   namespace: netbird
 spec:

netbird/setup.sh

@@ -1,38 +0,0 @@
-#!/usr/bin/env bash
-# Prepare local Compose configuration without replacing existing credentials.
-set -euo pipefail
-cd "$(dirname "${BASH_SOURCE[0]}")"
-umask 077
-if [ ! -f .env ]; then
-  cp .env.example .env
-fi
-
-if grep -q '^NETBIRD_PROXY_SUBNET=auto$' .env; then
-  subnet="$(docker network inspect proxy --format '{{range .IPAM.Config}}{{println .Subnet}}{{end}}' | awk '/^[0-9]+\./ { print; exit }')"
-  if [ -z "$subnet" ]; then
-    echo "No IPv4 subnet found on the Docker proxy network. Set NETBIRD_PROXY_SUBNET in .env." >&2
-    exit 1
-  fi
-  # The detected value must be safe to substitute into the env file.
-  if [[ ! "$subnet" =~ ^[0-9.]+/[0-9]+$ ]]; then
-    echo "Unexpected Docker network subnet: $subnet" >&2
-    exit 1
-  fi
-  sed -i "s|^NETBIRD_PROXY_SUBNET=auto$|NETBIRD_PROXY_SUBNET=$subnet|" .env
-fi
-
-mkdir -p secrets
-chmod 700 secrets
-for name in relay-auth-secret datastore-encryption-key; do
-  path="secrets/$name"
-  if [ -e "$path" ]; then
-    if [ ! -s "$path" ]; then
-      echo "Existing secret is empty: $path. Restore it before continuing." >&2
-      exit 1
-    fi
-  else
-    openssl rand -base64 32 >"$path"
-  fi
-  chmod 600 "$path"
-done
-printf '%s\n' 'Local files are ready. Review .env, then run docker compose config --quiet.'

netbox/k8s/netbox.yaml

@@ -14,8 +14,6 @@ spec:
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: netbox-deployment
   namespace: netbox
   labels:
@@ -120,8 +118,6 @@ spec:
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: netbox-worker-deployment
   namespace: netbox
   labels:

netbox/k8s/valkey.yaml

@@ -17,8 +17,6 @@ spec:
 apiVersion: apps/v1
 kind: StatefulSet
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: netbox-valkey
   namespace: netbox
   labels:

netronome/compose.yaml

@@ -1,6 +1,6 @@
 services:
   netronome:
-    image: ghcr.io/autobrr/netronome:v0.16.0
+    image: ghcr.io/autobrr/netronome:v0.15.0
     restart: unless-stopped
     container_name: netronome
     ports:

netronome/k8s/netronome.yaml

@@ -14,8 +14,6 @@ spec:
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: netronome-deployment
   namespace: netronome
   labels:
@@ -34,7 +32,7 @@ spec:
     spec:
       containers:
         - name: netronome
-          image: ghcr.io/autobrr/netronome:v0.16.0
+          image: ghcr.io/autobrr/netronome:v0.15.0
           ports:
             - name: netronome-port
               protocol: TCP

postgres/.env.example

@@ -1,7 +1,6 @@
 POSTGRES_ADMIN_PASSWORD=
 AUTHENTIK_DB_PASSWORD=
 GITEA_DB_PASSWORD=
-NETBOX_DB_PASSWORD=
 NETRONOME_DB_PASSWORD=
 PENPOT_DB_PASSWORD=
 STATUSPAGE_DB_PASSWORD=

prometheus-stack/k8s/README.md

@@ -1,17 +0,0 @@
-# VictoriaMetrics
-
-The `victoria-operator` Helm release converts Prometheus Operator
-`ServiceMonitor` resources into owned `VMServiceScrape` resources. The
-`VMAgent` selects converted scrapes labeled `release: prometheus-stack` in all
-namespaces and writes them to the existing single-node VictoriaMetrics
-instance. Changes to selected `ServiceMonitor` resources are reconciled
-automatically; there is no copied Prometheus scrape-config blob to regenerate.
-
-The agent drops targets for the Prometheus server service to avoid duplicating
-its self-scrape. `scraper: victoria` identifies the samples ingested by this
-VMAgent.
-
-The VictoriaMetrics Operator chart and its CRDs are installed before the
-Kubernetes manifests by the normal deploy workflow. On a cluster where the
-operator CRDs are not installed yet, CI skips the server-side dry-run of the
-`VMAgent` resource; the deploy installs the chart before applying that resource.

prometheus-stack/k8s/grafana-values.yaml

@@ -38,8 +38,6 @@ grafana:
 
   # One block covers both the dashboards and datasources sidecars (p95 91M / 80M).
   sidecar:
-    datasources:
-      defaultDatasourceEnabled: false
     resources:
       requests:
         memory: "96Mi"
@@ -52,18 +50,9 @@ grafana:
       type: loki
       url: http://loki-gateway.prometheus.svc.cluster.local
       access: proxy
-    - name: VictoriaMetrics
-      type: prometheus
-      url: http://victoria-metrics.prometheus.svc.cluster.local:8428
-      access: proxy
-      isDefault: true
 
 prometheus:
   prometheusSpec:
-    # VM trial: vmagent scrapes and remote-writes to VictoriaMetrics, so the
-    # Prometheus server itself stands down. Encoded here (not a kubectl patch)
-    # so helm keeps owning spec.replicas and upgrades do not conflict on it.
-    replicas: 0
     retention: 60d
     retentionSize: 32GB
     storageSpec:

prometheus-stack/k8s/ingress.yaml

@@ -31,105 +31,3 @@ spec:
           port: 80
   tls:
     secretName: internal-wildcard-tls
----
-apiVersion: traefik.io/v1alpha1
-kind: IngressRoute
-metadata:
-  name: prometheus-local
-  namespace: prometheus
-spec:
-  entryPoints:
-    - websecure
-  routes:
-    - match: Host(`prom.workstation.internal`) || Host(`prom.gigaforust.internal`)
-      kind: Rule
-      services:
-        - name: prometheus-stack-kube-prom-prometheus
-          port: 9090
-  tls:
-    secretName: internal-wildcard-tls
----
-apiVersion: traefik.io/v1alpha1
-kind: IngressRoute
-metadata:
-  name: alertmanager-local
-  namespace: prometheus
-spec:
-  entryPoints:
-    - websecure
-  routes:
-    - match: Host(`am.workstation.internal`) || Host(`am.gigaforust.internal`)
-      kind: Rule
-      services:
-        - name: prometheus-stack-kube-prom-alertmanager
-          port: 9093
-  tls:
-    secretName: internal-wildcard-tls
----
-apiVersion: traefik.io/v1alpha1
-kind: IngressRoute
-metadata:
-  name: loki-local
-  namespace: prometheus
-spec:
-  entryPoints:
-    - websecure
-  routes:
-    - match: Host(`loki.workstation.internal`) || Host(`loki.gigaforust.internal`)
-      kind: Rule
-      services:
-        - name: loki-gateway
-          port: 80
-  tls:
-    secretName: internal-wildcard-tls
----
-apiVersion: traefik.io/v1alpha1
-kind: IngressRoute
-metadata:
-  name: alloy-local
-  namespace: prometheus
-spec:
-  entryPoints:
-    - websecure
-  routes:
-    - match: Host(`alloy.workstation.internal`) || Host(`alloy.gigaforust.internal`)
-      kind: Rule
-      services:
-        - name: alloy
-          port: 12345
-  tls:
-    secretName: internal-wildcard-tls
----
-apiVersion: traefik.io/v1alpha1
-kind: IngressRoute
-metadata:
-  name: victoria-local
-  namespace: prometheus
-spec:
-  entryPoints:
-    - websecure
-  routes:
-    - match: Host(`victoria.workstation.internal`) || Host(`victoria.gigaforust.internal`)
-      kind: Rule
-      services:
-        - name: victoria-metrics
-          port: 8428
-  tls:
-    secretName: internal-wildcard-tls
----
-apiVersion: traefik.io/v1alpha1
-kind: IngressRoute
-metadata:
-  name: vmalert-local
-  namespace: prometheus
-spec:
-  entryPoints:
-    - websecure
-  routes:
-    - match: Host(`vmalert.workstation.internal`) || Host(`vmalert.gigaforust.internal`)
-      kind: Rule
-      services:
-        - name: vmalert
-          port: 8880
-  tls:
-    secretName: internal-wildcard-tls

prometheus-stack/k8s/victoria-operator-values.yaml

@@ -1,12 +0,0 @@
-nameOverride: victoria-operator
-
-operator:
-  enable_converter_ownership: true
-
-resources:
-  requests:
-    cpu: 50m
-    memory: 96Mi
-  limits:
-    cpu: 200m
-    memory: 256Mi

prometheus-stack/k8s/victoria.yaml

@@ -1,79 +0,0 @@
-apiVersion: v1
-kind: Service
-metadata:
-  name: victoria-metrics
-  namespace: prometheus
-spec:
-  selector:
-    app: victoria-metrics
-  ports:
-    - port: 8428
-      targetPort: 8428
----
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: victoria-pvc
-  namespace: prometheus
-spec:
-  resources:
-    requests:
-      storage: 10Gi
-  volumeMode: Filesystem
-  accessModes:
-    - ReadWriteOnce
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: victoria-deployment
-  namespace: prometheus
-spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      app: victoria-metrics
-  strategy:
-    type: Recreate
-  template:
-    metadata:
-      labels:
-        app: victoria-metrics
-    spec:
-      containers:
-        - name: victoria
-          image: victoriametrics/victoria-metrics:v1.153.0-scratch
-          args:
-            - -storageDataPath=/vmdata
-            - -retentionPeriod=30d
-            - -httpListenAddr=:8428
-          ports:
-            - containerPort: 8428
-          readinessProbe:
-            httpGet:
-              path: /health
-              port: 8428
-            initialDelaySeconds: 15
-            periodSeconds: 10
-            failureThreshold: 6
-          livenessProbe:
-            httpGet:
-              path: /health
-              port: 8428
-            initialDelaySeconds: 60
-            periodSeconds: 30
-            failureThreshold: 3
-          volumeMounts:
-            - name: vmdata
-              mountPath: /vmdata
-          resources:
-            requests:
-              cpu: "100m"
-              memory: "256Mi"
-            limits:
-              cpu: "1000m"
-              memory: "1Gi"
-      volumes:
-        - name: vmdata
-          persistentVolumeClaim:
-            claimName: victoria-pvc

prometheus-stack/k8s/vmagent.yaml

@@ -1,29 +0,0 @@
-apiVersion: operator.victoriametrics.com/v1beta1
-kind: VMAgent
-metadata:
-  name: vmagent
-  namespace: prometheus
-spec:
-  image:
-    tag: v1.153.0
-  scrapeInterval: 30s
-  externalLabels:
-    scraper: victoria
-  serviceScrapeNamespaceSelector: {}
-  serviceScrapeSelector:
-    matchLabels:
-      release: prometheus-stack
-  globalScrapeRelabelConfigs:
-    - action: drop
-      source_labels:
-        - __meta_kubernetes_service_name
-      regex: prometheus-stack-kube-prom-prometheus
-  remoteWrite:
-    - url: http://victoria-metrics.prometheus.svc.cluster.local:8428/api/v1/write
-  resources:
-    requests:
-      cpu: 100m
-      memory: 256Mi
-    limits:
-      cpu: "1000m"
-      memory: 1Gi

prometheus-stack/k8s/vmalert.yaml

@@ -1,70 +0,0 @@
-apiVersion: v1
-kind: Service
-metadata:
-  name: vmalert
-  namespace: prometheus
-spec:
-  selector:
-    app: vmalert
-  ports:
-    - port: 8880
-      targetPort: 8880
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: vmalert-deployment
-  namespace: prometheus
-spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      app: vmalert
-  strategy:
-    type: Recreate
-  template:
-    metadata:
-      labels:
-        app: vmalert
-    spec:
-      containers:
-        - name: vmalert
-          image: victoriametrics/vmalert:v1.153.0
-          args:
-            - -datasource.url=http://victoria-metrics.prometheus.svc.cluster.local:8428
-            - -remoteWrite.url=http://victoria-metrics.prometheus.svc.cluster.local:8428
-            - -notifier.url=http://prometheus-stack-kube-prom-alertmanager.prometheus.svc.cluster.local:9093
-            - -rule=/etc/vm/rules/*.yaml
-            - -evaluationInterval=60s
-            - -httpListenAddr=:8880
-          ports:
-            - containerPort: 8880
-          readinessProbe:
-            httpGet:
-              path: /metrics
-              port: 8880
-            initialDelaySeconds: 15
-            periodSeconds: 10
-            failureThreshold: 6
-          livenessProbe:
-            httpGet:
-              path: /metrics
-              port: 8880
-            initialDelaySeconds: 60
-            periodSeconds: 30
-            failureThreshold: 3
-          volumeMounts:
-            - name: rules
-              mountPath: /etc/vm/rules
-              readOnly: true
-          resources:
-            requests:
-              cpu: "50m"
-              memory: "64Mi"
-            limits:
-              cpu: "200m"
-              memory: "256Mi"
-      volumes:
-        - name: rules
-          configMap:
-            name: prometheus-prometheus-stack-kube-prom-prometheus-rulefiles-0

reloader/k8s/reloader-values.yaml

@@ -1,17 +1,11 @@
 # Pinned chart: stakater/reloader 2.2.17 (app v1.4.22).
 # Deployed by the deploy workflow, namespace reloader.
 # Restarts pods when a ConfigMap or Secret they consume changes. Opt-in per workload
-# via the reloader.stakater.com/auto: "true" workload annotation; watchGlobally because
+# via the reloader.stakater.com/auto: "true" pod annotation; watchGlobally because
 # the workloads that need it are spread across a few dozen namespaces.
 
 reloader:
   watchGlobally: true
-  # Only opted-in workloads are restarted. Keep scheduled jobs on their schedule.
-  autoReloadAll: false
-  ignoreJobs: true
-  ignoreCronJobs: true
-  # Change pod-template annotations rather than injecting STAKATER_* env vars.
-  reloadStrategy: annotations
 
   deployment:
     replicas: 1

renovate/.env.example

@@ -1,4 +1,4 @@
-RENOVATE_ENDPOINT=https://git.forust.xyz/api/v1
+RENOVATE_ENDPOINT=https://gitea.forust.xyz/api/v1
 RENOVATE_TOKEN=
 RENOVATE_REPOSITORIES=forust/homelab
 LOG_LEVEL=info

renovate/k8s/configmap.yaml

@@ -19,9 +19,6 @@ data:
       "dependencyDashboard": true,
       "prCreation": "immediate",
       "labels": ["dependencies", "automated"],
-      "docker-compose": {
-        "managerFilePatterns": ["renovate/renovate-compose.yaml"]
-      },
       "helm-values": {
         "managerFilePatterns": ["/k8s/.+values\\.ya?ml$/"]
       },
@@ -32,7 +29,7 @@ data:
         {
           "customType": "regex",
           "description": "singlesource: playwright npm version pinned in npx command (k8s + compose)",
-          "managerFilePatterns": ["edu_master/k8s/playwright.yaml", "edu_master/compose.yaml"],
+          "managerFilePatterns": ["^edu_master/k8s/playwright\\.yaml$", "^edu_master/compose\\.yaml$"],
           "matchStrings": ["playwright@(?<currentValue>\\d+\\.\\d+\\.\\d+)"],
           "datasourceTemplate": "npm",
           "depNameTemplate": "playwright"
@@ -40,42 +37,24 @@ data:
         {
           "customType": "regex",
           "description": "singlesource: PLAYWRIGHT_VERSION file",
-          "managerFilePatterns": ["edu_master/PLAYWRIGHT_VERSION"],
+          "managerFilePatterns": ["^edu_master/PLAYWRIGHT_VERSION$"],
-          "matchStrings": ["^(?<currentValue>\\d+\\.\\d+\\.\\d+)(?:\\r?\\n)?$"],
+          "matchStrings": ["^(?<currentValue>\\d+\\.\\d+\\.\\d+)$"],
           "datasourceTemplate": "pypi",
           "depNameTemplate": "playwright"
         },
-        {
-          "customType": "regex",
-          "description": "singlesource: playwright Python client version pinned in Dockerfile ARG",
-          "managerFilePatterns": ["edu_master/webinar-checker/Dockerfile"],
-          "matchStrings": ["(?:^|\\n)ARG PLAYWRIGHT_VERSION=(?<currentValue>\\d+\\.\\d+\\.\\d+)(?:\\r?\\n|$)"],
-          "datasourceTemplate": "pypi",
-          "depNameTemplate": "playwright",
-          "versioningTemplate": "pep440"
-        },
         {
           "customType": "regex",
           "description": "kube-prometheus-stack chart version pinned in the deploy workflow",
-          "managerFilePatterns": [".gitea/workflows/deploy-lib.sh"],
+          "managerFilePatterns": ["^\\.gitea/workflows/deploy-lib\\.sh$"],
           "matchStrings": ["\\|prometheus-community/kube-prometheus-stack\\|prometheus\\|(?<currentValue>[0-9.]+)\\|"],
           "datasourceTemplate": "helm",
           "depNameTemplate": "kube-prometheus-stack",
           "registryUrlTemplate": "https://prometheus-community.github.io/helm-charts"
         },
-        {
-          "customType": "regex",
-          "description": "VictoriaMetrics Operator chart version pinned in the deploy workflow",
-          "managerFilePatterns": [".gitea/workflows/deploy-lib.sh"],
-          "matchStrings": ["\\|victoriametrics/victoria-metrics-operator\\|prometheus\\|(?<currentValue>[0-9.]+)\\|"],
-          "datasourceTemplate": "helm",
-          "depNameTemplate": "victoria-metrics-operator",
-          "registryUrlTemplate": "https://victoriametrics.github.io/helm-charts"
-        },
         {
           "customType": "regex",
           "description": "grafana/loki chart version pinned in the deploy workflow",
-          "managerFilePatterns": [".gitea/workflows/deploy-lib.sh"],
+          "managerFilePatterns": ["^\\.gitea/workflows/deploy-lib\\.sh$"],
           "matchStrings": ["\\|grafana/loki\\|prometheus\\|(?<currentValue>[0-9.]+)\\|"],
           "datasourceTemplate": "helm",
           "depNameTemplate": "loki",
@@ -84,7 +63,7 @@ data:
         {
           "customType": "regex",
           "description": "grafana/alloy chart version pinned in the deploy workflow",
-          "managerFilePatterns": [".gitea/workflows/deploy-lib.sh"],
+          "managerFilePatterns": ["^\\.gitea/workflows/deploy-lib\\.sh$"],
           "matchStrings": ["\\|grafana/alloy\\|prometheus\\|(?<currentValue>[0-9.]+)\\|"],
           "datasourceTemplate": "helm",
           "depNameTemplate": "alloy",
@@ -93,7 +72,7 @@ data:
         {
           "customType": "regex",
           "description": "actionlint version used by the ci workflow",
-          "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
+          "managerFilePatterns": ["^\\.gitea/workflows/tool-versions\\.env$"],
           "matchStrings": ["(?:^|\\n)ACTIONLINT_VERSION=\"(?<currentValue>[0-9.]+)\""],
           "datasourceTemplate": "github-tags",
           "depNameTemplate": "rhysd/actionlint"
@@ -101,7 +80,7 @@ data:
         {
           "customType": "regex",
           "description": "shellcheck version used by the ci workflow",
-          "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
+          "managerFilePatterns": ["^\\.gitea/workflows/tool-versions\\.env$"],
           "matchStrings": ["(?:^|\\n)SHELLCHECK_VERSION=\"(?<currentValue>[0-9.]+)\""],
           "datasourceTemplate": "github-tags",
           "depNameTemplate": "koalaman/shellcheck"
@@ -109,7 +88,7 @@ data:
         {
           "customType": "regex",
           "description": "kubeconform version used by the ci workflow",
-          "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
+          "managerFilePatterns": ["^\\.gitea/workflows/tool-versions\\.env$"],
           "matchStrings": ["(?:^|\\n)KUBECONFORM_VERSION=\"(?<currentValue>[0-9.]+)\""],
           "datasourceTemplate": "github-tags",
           "depNameTemplate": "yannh/kubeconform"
@@ -117,7 +96,7 @@ data:
         {
           "customType": "regex",
           "description": "uv version used to build the pytest venv",
-          "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
+          "managerFilePatterns": ["^\\.gitea/workflows/tool-versions\\.env$"],
           "matchStrings": ["(?:^|\\n)UV_VERSION=\"(?<currentValue>[0-9.]+)\""],
           "datasourceTemplate": "github-tags",
           "depNameTemplate": "astral-sh/uv"
@@ -125,7 +104,7 @@ data:
         {
           "customType": "regex",
           "description": "prettier version used by the ci workflow",
-          "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
+          "managerFilePatterns": ["^\\.gitea/workflows/tool-versions\\.env$"],
           "matchStrings": ["(?:^|\\n)PRETTIER_VERSION=\"(?<currentValue>[0-9.]+)\""],
           "datasourceTemplate": "npm",
           "depNameTemplate": "prettier"
@@ -133,7 +112,7 @@ data:
         {
           "customType": "regex",
           "description": "ruff version used by the ci workflow",
-          "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
+          "managerFilePatterns": ["^\\.gitea/workflows/tool-versions\\.env$"],
           "matchStrings": ["(?:^|\\n)RUFF_VERSION=\"(?<currentValue>[0-9.]+)\""],
           "datasourceTemplate": "pypi",
           "depNameTemplate": "ruff"
@@ -141,7 +120,7 @@ data:
         {
           "customType": "regex",
           "description": "pip-audit version used by the ci workflow",
-          "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
+          "managerFilePatterns": ["^\\.gitea/workflows/tool-versions\\.env$"],
           "matchStrings": ["(?:^|\\n)PIP_AUDIT_VERSION=\"(?<currentValue>[0-9.]+)\""],
           "datasourceTemplate": "pypi",
           "depNameTemplate": "pip-audit"
@@ -149,7 +128,7 @@ data:
         {
           "customType": "regex",
           "description": "yamllint version used by the ci workflow",
-          "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
+          "managerFilePatterns": ["^\\.gitea/workflows/tool-versions\\.env$"],
           "matchStrings": ["(?:^|\\n)YAMLLINT_VERSION=\"(?<currentValue>[0-9.]+)\""],
           "datasourceTemplate": "pypi",
           "depNameTemplate": "yamllint"
@@ -157,7 +136,7 @@ data:
         {
           "customType": "regex",
           "description": "hadolint version used by the ci workflow",
-          "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
+          "managerFilePatterns": ["^\\.gitea/workflows/tool-versions\\.env$"],
           "matchStrings": ["(?:^|\\n)HADOLINT_VERSION=\"(?<currentValue>[0-9.]+)\""],
           "datasourceTemplate": "github-tags",
           "depNameTemplate": "hadolint/hadolint"
@@ -165,7 +144,7 @@ data:
         {
           "customType": "regex",
           "description": "node version the ci workflow runs npm with",
-          "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
+          "managerFilePatterns": ["^\\.gitea/workflows/tool-versions\\.env$"],
           "matchStrings": ["(?:^|\\n)NODE_VERSION=\"(?<currentValue>[0-9.]+)\""],
           "datasourceTemplate": "node",
           "depNameTemplate": "node"
@@ -173,23 +152,16 @@ data:
         {
           "customType": "regex",
           "description": "stakater/reloader chart version pinned in the deploy workflow",
-          "managerFilePatterns": [".gitea/workflows/deploy-lib.sh"],
+          "managerFilePatterns": ["^\\.gitea/workflows/deploy-lib\\.sh$"],
           "matchStrings": ["\\|stakater/reloader\\|reloader\\|(?<currentValue>[0-9.]+)\\|"],
           "datasourceTemplate": "helm",
           "depNameTemplate": "reloader",
           "registryUrlTemplate": "https://stakater.github.io/stakater-charts"
-        },
-        {
-          "customType": "regex",
-          "description": "Pinned CI BuildKit helper image",
-          "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
-          "matchStrings": ["BUILDKIT_IMAGE=\"(?<depName>moby/buildkit):(?<currentValue>[^\"\\n]+)\""],
-          "datasourceTemplate": "docker"
         }
       ],
       "packageRules": [
         {
-          "description": "Automerge ordinary digest and patch updates after successful checks; specific manual-review rules below override this.",
+          "description": "Automerge digest and patch updates - safe by definition, review adds nothing, keeps the renovate queue and the deploy line short. Specific no-automerge rules below still override this for playwright, helm and majors.",
           "matchUpdateTypes": ["digest", "patch"],
           "automerge": true
         },
@@ -200,12 +172,6 @@ data:
           "groupSlug": "all-minor",
           "automerge": false
         },
-        {
-          "description": "Group ordinary patch updates; the specific groups and manual-review rules below take precedence",
-          "matchUpdateTypes": ["patch"],
-          "groupName": "all patch updates",
-          "groupSlug": "all-patch"
-        },
         {
           "description": "Keep private homelab images unchanged",
           "matchDatasources": ["docker"],
@@ -230,7 +196,7 @@ data:
           "automerge": false
         },
         {
-          "description": "Keep CI Node runtime updates in a separate, manually reviewed group",
+          "description": "CI runs npm on the node the panel image is built from - the NODE_VERSION pin in tool-versions.env and node:22-alpine in the Dockerfile are the same dependency and move as one",
           "matchPackageNames": ["node"],
           "groupName": "node runtime",
           "groupSlug": "node",
@@ -239,8 +205,6 @@ data:
         {
           "description": "Helm chart bumps change PVC fields and admission behaviour, keep them reviewable",
           "matchDatasources": ["helm"],
-          "groupName": "Helm chart {{depName}}",
-          "groupSlug": "helm-{{depName}}",
           "automerge": false
         },
         {
@@ -250,32 +214,10 @@ data:
           "automerge": false
         },
         {
-          "description": "Python Y-bumps break compat (3.11->3.12->3.13->3.14) - keep the base image out of the shared minor/patch groups, review every bump separately. Placed last so its groupName wins.",
+          "description": "Group patch updates from all sources - automerge still applies via the digest/patch rule above (helm/playwright stay manual via their own rules)",
-          "matchDatasources": ["docker"],
+          "matchUpdateTypes": ["patch"],
-          "matchPackageNames": ["python"],
+          "groupName": "all patch updates",
-          "groupName": "python base image",
+          "groupSlug": "all-patch"
-          "groupSlug": "python",
-          "automerge": false
-        },
-        {
-          "description": "Rolling/floating tags (streaming stack, nextcloud beta, kubectl latest) - never automerge, every bump is a manual review. Placed last so automerge:false wins over the shared digest/patch rule.",
-          "matchDatasources": ["docker"],
-          "matchPackageNames": [
-            "lscr.io/linuxserver/jellyfin",
-            "lscr.io/linuxserver/qbittorrent",
-            "lscr.io/linuxserver/sonarr",
-            "lscr.io/linuxserver/radarr",
-            "lscr.io/linuxserver/prowlarr",
-            "lscr.io/linuxserver/bazarr",
-            "ghcr.io/seerr-team/seerr",
-            "fallenbagel/jellyseerr",
-            "ghcr.io/lampac-nextgen/lampac",
-            "ghcr.io/nextcloud-releases/all-in-one",
-            "alpine/kubectl"
-          ],
-          "groupName": "floating images - manual",
-          "groupSlug": "floating-manual",
-          "automerge": false
         }
       ]
     }

renovate/k8s/cronjob.yaml

@@ -19,7 +19,7 @@ spec:
           restartPolicy: Never
           containers:
             - name: renovate
-              image: renovate/renovate:44.140.0
+              image: renovate/renovate:44.132.4
               env:
                 - name: RENOVATE_PLATFORM
                   value: gitea

renovate/k8s/secrets.yaml.example

@@ -6,6 +6,6 @@ metadata:
 type: Opaque
 stringData:
   RENOVATE_TOKEN: ""
-  RENOVATE_ENDPOINT: "https://git.forust.xyz/api/v1"
+  RENOVATE_ENDPOINT: "https://gitea.forust.xyz/api/v1"
   RENOVATE_REPOSITORIES: "forust/homelab"
   RENOVATE_GITHUB_COM_TOKEN: ""

renovate/renovate-compose.yaml

@@ -2,7 +2,7 @@ services:
   renovate:
     # Kept in step with renovate/k8s/cronjob.yaml by the "renovate self-update"
     # package rule in renovate/renovate.json.
-    image: renovate/renovate:44.136.0
+    image: renovate/renovate:44.115.9
     container_name: renovate
     restart: "no"
     env_file:

renovate/renovate.json

@@ -8,9 +8,6 @@
   "dependencyDashboard": true,
   "prCreation": "immediate",
   "labels": ["dependencies", "automated"],
-  "docker-compose": {
-    "managerFilePatterns": ["renovate/renovate-compose.yaml"]
-  },
   "helm-values": {
     "managerFilePatterns": ["/k8s/.+values\\.ya?ml$/"]
   },
@@ -21,7 +18,7 @@
     {
       "customType": "regex",
       "description": "singlesource: playwright npm version pinned in npx command (k8s + compose)",
-      "managerFilePatterns": ["edu_master/k8s/playwright.yaml", "edu_master/compose.yaml"],
+      "managerFilePatterns": ["^edu_master/k8s/playwright\\.yaml$", "^edu_master/compose\\.yaml$"],
       "matchStrings": ["playwright@(?<currentValue>\\d+\\.\\d+\\.\\d+)"],
       "datasourceTemplate": "npm",
       "depNameTemplate": "playwright"
@@ -29,42 +26,24 @@
     {
       "customType": "regex",
       "description": "singlesource: PLAYWRIGHT_VERSION file",
-      "managerFilePatterns": ["edu_master/PLAYWRIGHT_VERSION"],
+      "managerFilePatterns": ["^edu_master/PLAYWRIGHT_VERSION$"],
-      "matchStrings": ["^(?<currentValue>\\d+\\.\\d+\\.\\d+)(?:\\r?\\n)?$"],
+      "matchStrings": ["^(?<currentValue>\\d+\\.\\d+\\.\\d+)$"],
       "datasourceTemplate": "pypi",
       "depNameTemplate": "playwright"
     },
-    {
-      "customType": "regex",
-      "description": "singlesource: playwright Python client version pinned in Dockerfile ARG",
-      "managerFilePatterns": ["edu_master/webinar-checker/Dockerfile"],
-      "matchStrings": ["(?:^|\\n)ARG PLAYWRIGHT_VERSION=(?<currentValue>\\d+\\.\\d+\\.\\d+)(?:\\r?\\n|$)"],
-      "datasourceTemplate": "pypi",
-      "depNameTemplate": "playwright",
-      "versioningTemplate": "pep440"
-    },
     {
       "customType": "regex",
       "description": "kube-prometheus-stack chart version pinned in the deploy workflow",
-      "managerFilePatterns": [".gitea/workflows/deploy-lib.sh"],
+      "managerFilePatterns": ["^\\.gitea/workflows/deploy-lib\\.sh$"],
       "matchStrings": ["\\|prometheus-community/kube-prometheus-stack\\|prometheus\\|(?<currentValue>[0-9.]+)\\|"],
       "datasourceTemplate": "helm",
       "depNameTemplate": "kube-prometheus-stack",
       "registryUrlTemplate": "https://prometheus-community.github.io/helm-charts"
     },
-    {
-      "customType": "regex",
-      "description": "VictoriaMetrics Operator chart version pinned in the deploy workflow",
-      "managerFilePatterns": [".gitea/workflows/deploy-lib.sh"],
-      "matchStrings": ["\\|victoriametrics/victoria-metrics-operator\\|prometheus\\|(?<currentValue>[0-9.]+)\\|"],
-      "datasourceTemplate": "helm",
-      "depNameTemplate": "victoria-metrics-operator",
-      "registryUrlTemplate": "https://victoriametrics.github.io/helm-charts"
-    },
     {
       "customType": "regex",
       "description": "grafana/loki chart version pinned in the deploy workflow",
-      "managerFilePatterns": [".gitea/workflows/deploy-lib.sh"],
+      "managerFilePatterns": ["^\\.gitea/workflows/deploy-lib\\.sh$"],
       "matchStrings": ["\\|grafana/loki\\|prometheus\\|(?<currentValue>[0-9.]+)\\|"],
       "datasourceTemplate": "helm",
       "depNameTemplate": "loki",
@@ -73,7 +52,7 @@
     {
       "customType": "regex",
       "description": "grafana/alloy chart version pinned in the deploy workflow",
-      "managerFilePatterns": [".gitea/workflows/deploy-lib.sh"],
+      "managerFilePatterns": ["^\\.gitea/workflows/deploy-lib\\.sh$"],
       "matchStrings": ["\\|grafana/alloy\\|prometheus\\|(?<currentValue>[0-9.]+)\\|"],
       "datasourceTemplate": "helm",
       "depNameTemplate": "alloy",
@@ -82,7 +61,7 @@
     {
       "customType": "regex",
       "description": "actionlint version used by the ci workflow",
-      "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
+      "managerFilePatterns": ["^\\.gitea/workflows/tool-versions\\.env$"],
       "matchStrings": ["(?:^|\\n)ACTIONLINT_VERSION=\"(?<currentValue>[0-9.]+)\""],
       "datasourceTemplate": "github-tags",
       "depNameTemplate": "rhysd/actionlint"
@@ -90,7 +69,7 @@
     {
       "customType": "regex",
       "description": "shellcheck version used by the ci workflow",
-      "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
+      "managerFilePatterns": ["^\\.gitea/workflows/tool-versions\\.env$"],
       "matchStrings": ["(?:^|\\n)SHELLCHECK_VERSION=\"(?<currentValue>[0-9.]+)\""],
       "datasourceTemplate": "github-tags",
       "depNameTemplate": "koalaman/shellcheck"
@@ -98,7 +77,7 @@
     {
       "customType": "regex",
       "description": "kubeconform version used by the ci workflow",
-      "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
+      "managerFilePatterns": ["^\\.gitea/workflows/tool-versions\\.env$"],
       "matchStrings": ["(?:^|\\n)KUBECONFORM_VERSION=\"(?<currentValue>[0-9.]+)\""],
       "datasourceTemplate": "github-tags",
       "depNameTemplate": "yannh/kubeconform"
@@ -106,7 +85,7 @@
     {
       "customType": "regex",
       "description": "uv version used to build the pytest venv",
-      "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
+      "managerFilePatterns": ["^\\.gitea/workflows/tool-versions\\.env$"],
       "matchStrings": ["(?:^|\\n)UV_VERSION=\"(?<currentValue>[0-9.]+)\""],
       "datasourceTemplate": "github-tags",
       "depNameTemplate": "astral-sh/uv"
@@ -114,7 +93,7 @@
     {
       "customType": "regex",
       "description": "prettier version used by the ci workflow",
-      "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
+      "managerFilePatterns": ["^\\.gitea/workflows/tool-versions\\.env$"],
       "matchStrings": ["(?:^|\\n)PRETTIER_VERSION=\"(?<currentValue>[0-9.]+)\""],
       "datasourceTemplate": "npm",
       "depNameTemplate": "prettier"
@@ -122,7 +101,7 @@
     {
       "customType": "regex",
       "description": "ruff version used by the ci workflow",
-      "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
+      "managerFilePatterns": ["^\\.gitea/workflows/tool-versions\\.env$"],
       "matchStrings": ["(?:^|\\n)RUFF_VERSION=\"(?<currentValue>[0-9.]+)\""],
       "datasourceTemplate": "pypi",
       "depNameTemplate": "ruff"
@@ -130,7 +109,7 @@
     {
       "customType": "regex",
       "description": "pip-audit version used by the ci workflow",
-      "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
+      "managerFilePatterns": ["^\\.gitea/workflows/tool-versions\\.env$"],
       "matchStrings": ["(?:^|\\n)PIP_AUDIT_VERSION=\"(?<currentValue>[0-9.]+)\""],
       "datasourceTemplate": "pypi",
       "depNameTemplate": "pip-audit"
@@ -138,7 +117,7 @@
     {
       "customType": "regex",
       "description": "yamllint version used by the ci workflow",
-      "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
+      "managerFilePatterns": ["^\\.gitea/workflows/tool-versions\\.env$"],
       "matchStrings": ["(?:^|\\n)YAMLLINT_VERSION=\"(?<currentValue>[0-9.]+)\""],
       "datasourceTemplate": "pypi",
       "depNameTemplate": "yamllint"
@@ -146,7 +125,7 @@
     {
       "customType": "regex",
       "description": "hadolint version used by the ci workflow",
-      "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
+      "managerFilePatterns": ["^\\.gitea/workflows/tool-versions\\.env$"],
       "matchStrings": ["(?:^|\\n)HADOLINT_VERSION=\"(?<currentValue>[0-9.]+)\""],
       "datasourceTemplate": "github-tags",
       "depNameTemplate": "hadolint/hadolint"
@@ -154,7 +133,7 @@
     {
       "customType": "regex",
       "description": "node version the ci workflow runs npm with",
-      "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
+      "managerFilePatterns": ["^\\.gitea/workflows/tool-versions\\.env$"],
       "matchStrings": ["(?:^|\\n)NODE_VERSION=\"(?<currentValue>[0-9.]+)\""],
       "datasourceTemplate": "node",
       "depNameTemplate": "node"
@@ -162,23 +141,16 @@
     {
       "customType": "regex",
       "description": "stakater/reloader chart version pinned in the deploy workflow",
-      "managerFilePatterns": [".gitea/workflows/deploy-lib.sh"],
+      "managerFilePatterns": ["^\\.gitea/workflows/deploy-lib\\.sh$"],
       "matchStrings": ["\\|stakater/reloader\\|reloader\\|(?<currentValue>[0-9.]+)\\|"],
       "datasourceTemplate": "helm",
       "depNameTemplate": "reloader",
       "registryUrlTemplate": "https://stakater.github.io/stakater-charts"
-    },
-    {
-      "customType": "regex",
-      "description": "Pinned CI BuildKit helper image",
-      "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
-      "matchStrings": ["BUILDKIT_IMAGE=\"(?<depName>moby/buildkit):(?<currentValue>[^\"\\n]+)\""],
-      "datasourceTemplate": "docker"
     }
   ],
   "packageRules": [
     {
-      "description": "Automerge ordinary digest and patch updates after successful checks; specific manual-review rules below override this.",
+      "description": "Automerge digest and patch updates - safe by definition, review adds nothing, keeps the renovate queue and the deploy line short. Specific no-automerge rules below still override this for playwright, helm and majors.",
       "matchUpdateTypes": ["digest", "patch"],
       "automerge": true
     },
@@ -189,12 +161,6 @@
       "groupSlug": "all-minor",
       "automerge": false
     },
-    {
-      "description": "Group ordinary patch updates; the specific groups and manual-review rules below take precedence",
-      "matchUpdateTypes": ["patch"],
-      "groupName": "all patch updates",
-      "groupSlug": "all-patch"
-    },
     {
       "description": "Keep private homelab images unchanged",
       "matchDatasources": ["docker"],
@@ -219,7 +185,7 @@
       "automerge": false
     },
     {
-      "description": "Keep CI Node runtime updates in a separate, manually reviewed group",
+      "description": "CI runs npm on the node the panel image is built from - the NODE_VERSION pin in tool-versions.env and node:22-alpine in the Dockerfile are the same dependency and move as one",
       "matchPackageNames": ["node"],
       "groupName": "node runtime",
       "groupSlug": "node",
@@ -228,8 +194,6 @@
     {
       "description": "Helm chart bumps change PVC fields and admission behaviour, keep them reviewable",
       "matchDatasources": ["helm"],
-      "groupName": "Helm chart {{depName}}",
-      "groupSlug": "helm-{{depName}}",
       "automerge": false
     },
     {
@@ -239,32 +203,10 @@
       "automerge": false
     },
     {
-      "description": "Python Y-bumps break compat (3.11->3.12->3.13->3.14) - keep the base image out of the shared minor/patch groups, review every bump separately. Placed last so its groupName wins.",
+      "description": "Group patch updates from all sources - automerge still applies via the digest/patch rule above (helm/playwright stay manual via their own rules)",
-      "matchDatasources": ["docker"],
+      "matchUpdateTypes": ["patch"],
-      "matchPackageNames": ["python"],
+      "groupName": "all patch updates",
-      "groupName": "python base image",
+      "groupSlug": "all-patch"
-      "groupSlug": "python",
-      "automerge": false
-    },
-    {
-      "description": "Rolling/floating tags (streaming stack, nextcloud beta, kubectl latest) - never automerge, every bump is a manual review. Placed last so automerge:false wins over the shared digest/patch rule.",
-      "matchDatasources": ["docker"],
-      "matchPackageNames": [
-        "lscr.io/linuxserver/jellyfin",
-        "lscr.io/linuxserver/qbittorrent",
-        "lscr.io/linuxserver/sonarr",
-        "lscr.io/linuxserver/radarr",
-        "lscr.io/linuxserver/prowlarr",
-        "lscr.io/linuxserver/bazarr",
-        "ghcr.io/seerr-team/seerr",
-        "fallenbagel/jellyseerr",
-        "ghcr.io/lampac-nextgen/lampac",
-        "ghcr.io/nextcloud-releases/all-in-one",
-        "alpine/kubectl"
-      ],
-      "groupName": "floating images - manual",
-      "groupSlug": "floating-manual",
-      "automerge": false
     }
   ]
 }

searxng/compose.yaml

@@ -12,11 +12,11 @@ services:
       - "traefik.enable=true"
       - "traefik.http.services.searxng.loadbalancer.server.port=8080"
       # Prod Router
-      - "traefik.http.routers.searxng.rule=Host(`s.forust.xyz`) || Host(`search.forust.xyz`)"
+      - "traefik.http.routers.searxng.rule=Host(`s.forust.xyz` || `search.forust.xyz`)"
       - "traefik.http.routers.searxng.entrypoints=websecure"
       - "traefik.http.routers.searxng.tls.certresolver=letsencrypt"
       # Local Router
-      - "traefik.http.routers.searxng-local.rule=Host(`s.workstation.internal`) || Host(`searxng.workstation.internal`)"
+      - "traefik.http.routers.searxng-local.rule=Host(`s.workstation.internal` || `searxng.workstation.internal`)"
       - "traefik.http.routers.searxng-local.entrypoints=websecure"
       - "traefik.http.routers.searxng-local.tls=true"
       # Dev Router

searxng/k8s/searxng.yaml

@@ -13,8 +13,6 @@ spec:
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: searxng-deployment
   namespace: searxng
 spec:

streaming/.env.example

@@ -1,3 +0,0 @@
-PUID=1000
-PGID=1000
-TZ=Europe/Berlin

streaming/compose.yaml

@@ -1,133 +0,0 @@
-services:
-  jellyfin:
-    image: lscr.io/linuxserver/jellyfin:version-12.1ubu2604
-    container_name: jellyfin
-    restart: unless-stopped
-    environment:
-      - PUID=${PUID:-1000}
-      - PGID=${PGID:-1000}
-      - TZ=${TZ:-Europe/Berlin}
-    volumes:
-      - jellyfin-cfg:/config
-      - movies:/media/movies
-      - tv:/media/tv
-    devices:
-      - /dev/dri:/dev/dri
-    ports:
-      - "18096:8096"
-    networks:
-      - streaming
-
-  qbittorrent:
-    image: lscr.io/linuxserver/qbittorrent:20.04.1
-    container_name: qbittorrent
-    restart: unless-stopped
-    environment:
-      - PUID=${PUID:-1000}
-      - PGID=${PGID:-1000}
-      - TZ=${TZ:-Europe/Berlin}
-      - WEBUI_PORT=8080
-    volumes:
-      - qbittorrent-cfg:/config
-      - downloads:/downloads
-    ports:
-      - "18180:8080"
-      - "6881:6881"
-      - "6881:6881/udp"
-    networks:
-      - streaming
-
-  sonarr:
-    image: lscr.io/linuxserver/sonarr:4.0.20
-    container_name: sonarr
-    restart: unless-stopped
-    environment:
-      - PUID=${PUID:-1000}
-      - PGID=${PGID:-1000}
-      - TZ=${TZ:-Europe/Berlin}
-    volumes:
-      - sonarr-cfg:/config
-      - downloads:/downloads
-      - tv:/tv
-    ports:
-      - "18989:8989"
-    networks:
-      - streaming
-
-  radarr:
-    image: lscr.io/linuxserver/radarr:6.4.4
-    container_name: radarr
-    restart: unless-stopped
-    environment:
-      - PUID=${PUID:-1000}
-      - PGID=${PGID:-1000}
-      - TZ=${TZ:-Europe/Berlin}
-    volumes:
-      - radarr-cfg:/config
-      - downloads:/downloads
-      - movies:/movies
-    ports:
-      - "17878:7878"
-    networks:
-      - streaming
-
-  prowlarr:
-    image: lscr.io/linuxserver/prowlarr:2.6.5
-    container_name: prowlarr
-    restart: unless-stopped
-    environment:
-      - PUID=${PUID:-1000}
-      - PGID=${PGID:-1000}
-      - TZ=${TZ:-Europe/Berlin}
-    volumes:
-      - prowlarr-cfg:/config
-    ports:
-      - "19696:9696"
-    networks:
-      - streaming
-
-  jellyseerr:
-    image: fallenbagel/jellyseerr:latest
-    container_name: jellyseerr
-    restart: unless-stopped
-    environment:
-      - TZ=${TZ:-Europe/Berlin}
-    volumes:
-      - jellyseerr-cfg:/app/config
-    ports:
-      - "15055:5055"
-    networks:
-      - streaming
-
-  bazarr:
-    image: lscr.io/linuxserver/bazarr:1.6.2
-    container_name: bazarr
-    restart: unless-stopped
-    environment:
-      - PUID=${PUID:-1000}
-      - PGID=${PGID:-1000}
-      - TZ=${TZ:-Europe/Berlin}
-    volumes:
-      - bazarr-cfg:/config
-      - movies:/movies
-      - tv:/tv
-    ports:
-      - "16767:6767"
-    networks:
-      - streaming
-
-volumes:
-  jellyfin-cfg:
-  qbittorrent-cfg:
-  sonarr-cfg:
-  radarr-cfg:
-  prowlarr-cfg:
-  jellyseerr-cfg:
-  bazarr-cfg:
-  downloads:
-  movies:
-  tv:
-
-networks:
-  streaming:
-    name: streaming

streaming/k8s/namespace.yaml

@@ -1,4 +0,0 @@
-apiVersion: v1
-kind: Namespace
-metadata:
-  name: streaming

streaming/k8s/routing/certificates.yaml

@@ -1,93 +0,0 @@
-apiVersion: cert-manager.io/v1
-kind: Certificate
-metadata:
-  name: internal-wildcard-tls
-  namespace: streaming
-spec:
-  secretName: internal-wildcard-tls
-  dnsNames:
-    - "*.workstation.internal"
-    - "*.gigaforust.internal"
-    - workstation.internal
-    - gigaforust.internal
-  issuerRef:
-    name: internal-ca
-    kind: ClusterIssuer
-# ---
-# apiVersion: cert-manager.io/v1
-# kind: Certificate
-# metadata:
-#   name: jellyfin-prod-tls
-#   namespace: streaming
-# spec:
-#   secretName: jellyfin-prod-tls
-#   dnsNames:
-#     - jellyfin.forust.xyz
-#   issuerRef:
-#     name: letsencrypt-prod
-#     kind: ClusterIssuer
-# ---
-# apiVersion: cert-manager.io/v1
-# kind: Certificate
-# metadata:
-#   name: qbittorrent-prod-tls
-#   namespace: streaming
-# spec:
-#   secretName: qbittorrent-prod-tls
-#   dnsNames:
-#     - qbittorrent.forust.xyz
-#   issuerRef:
-#     name: letsencrypt-prod
-#     kind: ClusterIssuer
-# ---
-# apiVersion: cert-manager.io/v1
-# kind: Certificate
-# metadata:
-#   name: sonarr-prod-tls
-#   namespace: streaming
-# spec:
-#   secretName: sonarr-prod-tls
-#   dnsNames:
-#     - sonarr.forust.xyz
-#   issuerRef:
-#     name: letsencrypt-prod
-#     kind: ClusterIssuer
-# ---
-# apiVersion: cert-manager.io/v1
-# kind: Certificate
-# metadata:
-#   name: radarr-prod-tls
-#   namespace: streaming
-# spec:
-#   secretName: radarr-prod-tls
-#   dnsNames:
-#     - radarr.forust.xyz
-#   issuerRef:
-#     name: letsencrypt-prod
-#     kind: ClusterIssuer
-# ---
-# apiVersion: cert-manager.io/v1
-# kind: Certificate
-# metadata:
-#   name: prowlarr-prod-tls
-#   namespace: streaming
-# spec:
-#   secretName: prowlarr-prod-tls
-#   dnsNames:
-#     - prowlarr.forust.xyz
-#   issuerRef:
-#     name: letsencrypt-prod
-#     kind: ClusterIssuer
-# ---
-# apiVersion: cert-manager.io/v1
-# kind: Certificate
-# metadata:
-#   name: jellyseerr-prod-tls
-#   namespace: streaming
-# spec:
-#   secretName: jellyseerr-prod-tls
-#   dnsNames:
-#     - jellyseerr.forust.xyz
-#   issuerRef:
-#     name: letsencrypt-prod
-#     kind: ClusterIssuer

streaming/k8s/routing/external-service.yaml

@@ -1,188 +0,0 @@
-apiVersion: v1
-kind: Service
-metadata:
-  name: jellyfin
-  namespace: streaming
-spec:
-  ports:
-    - port: 18096
-      targetPort: 18096
----
-apiVersion: discovery.k8s.io/v1
-kind: EndpointSlice
-metadata:
-  name: jellyfin
-  namespace: streaming
-  labels:
-    kubernetes.io/service-name: jellyfin
-addressType: IPv4
-ports:
-  - port: 18096
-    protocol: TCP
-endpoints:
-  - addresses:
-      - "192.168.88.100"
-    conditions:
-      ready: true
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: qbittorrent
-  namespace: streaming
-spec:
-  ports:
-    - port: 18180
-      targetPort: 18180
----
-apiVersion: discovery.k8s.io/v1
-kind: EndpointSlice
-metadata:
-  name: qbittorrent
-  namespace: streaming
-  labels:
-    kubernetes.io/service-name: qbittorrent
-addressType: IPv4
-ports:
-  - port: 18180
-    protocol: TCP
-endpoints:
-  - addresses:
-      - "192.168.88.100"
-    conditions:
-      ready: true
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: sonarr
-  namespace: streaming
-spec:
-  ports:
-    - port: 18989
-      targetPort: 18989
----
-apiVersion: discovery.k8s.io/v1
-kind: EndpointSlice
-metadata:
-  name: sonarr
-  namespace: streaming
-  labels:
-    kubernetes.io/service-name: sonarr
-addressType: IPv4
-ports:
-  - port: 18989
-    protocol: TCP
-endpoints:
-  - addresses:
-      - "192.168.88.100"
-    conditions:
-      ready: true
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: radarr
-  namespace: streaming
-spec:
-  ports:
-    - port: 17878
-      targetPort: 17878
----
-apiVersion: discovery.k8s.io/v1
-kind: EndpointSlice
-metadata:
-  name: radarr
-  namespace: streaming
-  labels:
-    kubernetes.io/service-name: radarr
-addressType: IPv4
-ports:
-  - port: 17878
-    protocol: TCP
-endpoints:
-  - addresses:
-      - "192.168.88.100"
-    conditions:
-      ready: true
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: prowlarr
-  namespace: streaming
-spec:
-  ports:
-    - port: 19696
-      targetPort: 19696
----
-apiVersion: discovery.k8s.io/v1
-kind: EndpointSlice
-metadata:
-  name: prowlarr
-  namespace: streaming
-  labels:
-    kubernetes.io/service-name: prowlarr
-addressType: IPv4
-ports:
-  - port: 19696
-    protocol: TCP
-endpoints:
-  - addresses:
-      - "192.168.88.100"
-    conditions:
-      ready: true
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: jellyseerr
-  namespace: streaming
-spec:
-  ports:
-    - port: 15055
-      targetPort: 15055
----
-apiVersion: discovery.k8s.io/v1
-kind: EndpointSlice
-metadata:
-  name: jellyseerr
-  namespace: streaming
-  labels:
-    kubernetes.io/service-name: jellyseerr
-addressType: IPv4
-ports:
-  - port: 15055
-    protocol: TCP
-endpoints:
-  - addresses:
-      - "192.168.88.100"
-    conditions:
-      ready: true
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: bazarr
-  namespace: streaming
-spec:
-  ports:
-    - port: 16767
-      targetPort: 16767
----
-apiVersion: discovery.k8s.io/v1
-kind: EndpointSlice
-metadata:
-  name: bazarr
-  namespace: streaming
-  labels:
-    kubernetes.io/service-name: bazarr
-addressType: IPv4
-ports:
-  - port: 16767
-    protocol: TCP
-endpoints:
-  - addresses:
-      - "192.168.88.100"
-    conditions:
-      ready: true

streaming/k8s/routing/ingress.yaml

@@ -1,118 +0,0 @@
-apiVersion: traefik.io/v1alpha1
-kind: IngressRoute
-metadata:
-  name: jellyfin-local
-  namespace: streaming
-spec:
-  entryPoints:
-    - websecure
-  routes:
-    - match: Host(`jellyfin.workstation.internal`)
-      kind: Rule
-      services:
-        - name: jellyfin
-          port: 18096
-  tls:
-    secretName: internal-wildcard-tls
----
-apiVersion: traefik.io/v1alpha1
-kind: IngressRoute
-metadata:
-  name: qbittorrent-local
-  namespace: streaming
-spec:
-  entryPoints:
-    - websecure
-  routes:
-    - match: Host(`qbittorrent.workstation.internal`)
-      kind: Rule
-      services:
-        - name: qbittorrent
-          port: 18180
-  tls:
-    secretName: internal-wildcard-tls
----
-apiVersion: traefik.io/v1alpha1
-kind: IngressRoute
-metadata:
-  name: sonarr-local
-  namespace: streaming
-spec:
-  entryPoints:
-    - websecure
-  routes:
-    - match: Host(`sonarr.workstation.internal`)
-      kind: Rule
-      services:
-        - name: sonarr
-          port: 18989
-  tls:
-    secretName: internal-wildcard-tls
----
-apiVersion: traefik.io/v1alpha1
-kind: IngressRoute
-metadata:
-  name: radarr-local
-  namespace: streaming
-spec:
-  entryPoints:
-    - websecure
-  routes:
-    - match: Host(`radarr.workstation.internal`)
-      kind: Rule
-      services:
-        - name: radarr
-          port: 17878
-  tls:
-    secretName: internal-wildcard-tls
----
-apiVersion: traefik.io/v1alpha1
-kind: IngressRoute
-metadata:
-  name: prowlarr-local
-  namespace: streaming
-spec:
-  entryPoints:
-    - websecure
-  routes:
-    - match: Host(`prowlarr.workstation.internal`)
-      kind: Rule
-      services:
-        - name: prowlarr
-          port: 19696
-  tls:
-    secretName: internal-wildcard-tls
----
-apiVersion: traefik.io/v1alpha1
-kind: IngressRoute
-metadata:
-  name: jellyseerr-local
-  namespace: streaming
-spec:
-  entryPoints:
-    - websecure
-  routes:
-    - match: Host(`jellyseerr.workstation.internal`)
-      kind: Rule
-      services:
-        - name: jellyseerr
-          port: 15055
-  tls:
-    secretName: internal-wildcard-tls
----
-apiVersion: traefik.io/v1alpha1
-kind: IngressRoute
-metadata:
-  name: bazarr-local
-  namespace: streaming
-spec:
-  entryPoints:
-    - websecure
-  routes:
-    - match: Host(`bazarr.workstation.internal`)
-      kind: Rule
-      services:
-        - name: bazarr
-          port: 16767
-  tls:
-    secretName: internal-wildcard-tls

termix/compose.yaml

@@ -1,6 +1,6 @@
 services:
   termix:
-    image: ghcr.io/lukegus/termix:2.9.2
+    image: ghcr.io/lukegus/termix:2.9.0
     container_name: termix
     restart: unless-stopped
     # ports:

termix/k8s/termix.yaml

@@ -13,8 +13,6 @@ spec:
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: termix-deployment
   namespace: termix
 spec:
@@ -31,7 +29,7 @@ spec:
     spec:
       containers:
         - name: termix
-          image: ghcr.io/lukegus/termix:2.9.2
+          image: ghcr.io/lukegus/termix:2.9.0
           envFrom:
             - configMapRef:
                 name: termix-config

tests/test_cicd.py

@@ -1,267 +0,0 @@
-"""CI gate, selection, persistent configuration and recovery regression tests."""
-
-import importlib.util
-import json
-import os
-import subprocess
-import tempfile
-import unittest
-from pathlib import Path
-from unittest.mock import patch
-
-ROOT = Path(__file__).resolve().parents[1]
-
-
-def module(name, filename):
-    spec = importlib.util.spec_from_file_location(name, ROOT / '.gitea/workflows' / filename)
-    loaded = importlib.util.module_from_spec(spec)
-    spec.loader.exec_module(loaded)
-    return loaded
-
-
-release_module = module('release_test', 'release.py')
-planner = module('plan_test', 'deploy-plan.py')
-compose_module = module('compose_test', 'compose-release.py')
-controller = module('controller_test', 'deploy-controller.py')
-
-
-def release(sha='a' * 40):
-    return {
-        'version': 1,
-        'sha': sha,
-        'images': {f'gcr.forust.xyz/forust/{name}': 'sha256:' + 'b' * 64 for name in release_module.IMAGES},
-        'inputs': {f'gcr.forust.xyz/forust/{name}': 'c' * 64 for name in release_module.IMAGES},
-    }
-
-
-class ReleaseGateTests(unittest.TestCase):
-    def test_release_rejects_wrong_sha_missing_images_and_mutable_tags(self):
-        for mutation in ('sha', 'missing', 'tag'):
-            data = release()
-            if mutation == 'sha':
-                data['sha'] = 'd' * 40
-            elif mutation == 'missing':
-                data['images'].pop(next(iter(data['images'])))
-            else:
-                data['images'][next(iter(data['images']))] = 'prod'
-            with self.assertRaises(ValueError):
-                release_module.validate_release(data, 'a' * 40)
-
-    def test_gate_excludes_pr_wrong_branch_and_failed_runs(self):
-        api = object.__new__(release_module.Gitea)
-        api.repository = 'forust/homelab'
-        good = {
-            'id': 1,
-            'status': 'completed',
-            'conclusion': 'success',
-            'head_branch': 'main',
-            'event': 'push',
-            'head_sha': 'a' * 40,
-            'repository': {'full_name': api.repository},
-        }
-        entries = [
-            good,
-            {**good, 'event': 'pull_request'},
-            {**good, 'head_branch': 'dev'},
-            {**good, 'conclusion': 'failure'},
-            {**good, 'head_sha': 'b' * 40},
-            {**good, 'head_repository': {'full_name': 'attacker/fork'}},
-        ]
-        with patch.object(api, 'pages', return_value=iter(entries)):
-            self.assertEqual(list(api.successful_runs('a' * 40)), [good])
-
-    def test_green_ci_with_skipped_build_is_rejected(self):
-        api = object.__new__(release_module.Gitea)
-        with (
-            patch.object(api, 'pages', return_value=iter([{'name': 'build', 'conclusion': 'skipped'}])),
-            self.assertRaisesRegex(ValueError, 'build job'),
-        ):
-            api.release({'id': 1, 'head_sha': 'a' * 40})
-
-    def test_expired_or_ambiguous_artifacts_are_rejected(self):
-        api = object.__new__(release_module.Gitea)
-        api.base = 'https://example.test/api/v1/repos/a/b'
-        artifact = {'id': 1, 'name': 'release-' + 'a' * 40}
-        for artifacts in ([{**artifact, 'expired': True}], [artifact, artifact], []):
-            with (
-                patch.object(api, 'pages', return_value=iter([{'name': 'build', 'conclusion': 'success'}])),
-                patch.object(api, 'request', return_value={'artifacts': artifacts}),
-                self.assertRaisesRegex(ValueError, 'artifact'),
-            ):
-                api.release({'id': 1, 'head_sha': 'a' * 40})
-
-
-class SelectionTests(unittest.TestCase):
-    def setUp(self):
-        self.scratch = tempfile.TemporaryDirectory()
-        self.addCleanup(self.scratch.cleanup)
-        self.repo = Path(self.scratch.name)
-        self.git('init', '-q')
-        self.git('config', 'user.email', 'test@example.test')
-        self.git('config', 'user.name', 'CI Test')
-        for service in ('one', 'two', 'postgres'):
-            directory = self.repo / service / 'k8s'
-            directory.mkdir(parents=True)
-            (directory / 'active').touch()
-            (directory / 'app.yaml').write_text('kind: Deployment\n')
-        (self.repo / '.gitea/workflows').mkdir(parents=True)
-        (self.repo / '.gitea/workflows/deploy-lib.sh').write_text('HELM_RELEASES=(\n)\n')
-        (self.repo / '.gitea/deploy-dependencies.json').write_text('{"postgres": ["one", "two"]}')
-        self.sha = self.commit()
-        self.initial = planner.make_plan(self.repo, self.repo, release(self.sha), None, 'full', [])
-
-    def git(self, *args):
-        return planner.output('git', '-C', str(self.repo), *args)
-
-    def commit(self):
-        self.git('add', '.')
-        self.git('commit', '-qm', 'Test state')
-        return self.git('rev-parse', 'HEAD')
-
-    def test_first_changed_deploy_requires_explicit_full(self):
-        with self.assertRaisesRegex(ValueError, 'full'):
-            planner.make_plan(self.repo, self.repo, release(self.sha), None, 'changed', [])
-
-    def test_only_changed_service_is_selected(self):
-        (self.repo / 'one/k8s/app.yaml').write_text('kind: StatefulSet\n')
-        result = planner.make_plan(self.repo, self.repo, release(self.commit()), self.initial, 'changed', [])
-        self.assertEqual(result['selected']['k8s'], ['one'])
-        self.assertEqual(result['helm'], [])
-
-    def test_failed_intermediate_deploy_does_not_lose_changes(self):
-        (self.repo / 'one/k8s/app.yaml').write_text('kind: StatefulSet\n')
-        self.commit()  # This commit failed deploy: baseline must remain initial.
-        (self.repo / 'two/k8s/app.yaml').write_text('kind: StatefulSet\n')
-        result = planner.make_plan(self.repo, self.repo, release(self.commit()), self.initial, 'changed', [])
-        self.assertEqual(result['selected']['k8s'], ['one', 'two'])
-
-    def test_dependencies_and_removals_are_reported(self):
-        (self.repo / 'postgres/k8s/app.yaml').write_text('kind: StatefulSet\n')
-        (self.repo / 'two/k8s/active').unlink()
-        result = planner.make_plan(self.repo, self.repo, release(self.commit()), self.initial, 'changed', [])
-        self.assertEqual(result['selected']['k8s'], ['one', 'postgres'])
-        self.assertIn('two', result['removed'])
-
-    def test_local_configuration_change_selects_service(self):
-        (self.repo / 'one/.env').write_text('TEST_VALUE=changed\n')
-        result = planner.make_plan(self.repo, self.repo, release(self.sha), self.initial, 'changed', [])
-        self.assertEqual(result['selected']['k8s'], ['one'])
-
-    def test_redeploy_is_noop_and_full_includes_all(self):
-        result = planner.make_plan(self.repo, self.repo, release(self.sha), self.initial, 'changed', [])
-        self.assertEqual(result['selected']['k8s'], [])
-        result = planner.make_plan(self.repo, self.repo, release(self.sha), self.initial, 'full', [])
-        self.assertEqual(result['selected']['k8s'], ['one', 'postgres', 'two'])
-
-
-class ComposeConfigurationTests(unittest.TestCase):
-    def test_pin_preserves_project_volumes_paths_and_previous_image(self):
-        with tempfile.TemporaryDirectory() as scratch:
-            root = Path(scratch)
-            run = root / 'run'
-            source = run / 'source'
-            config_repo = root / 'persistent'
-            (source / 'headscale').mkdir(parents=True)
-            config_repo.mkdir()
-            (run / 'release.json').write_text(json.dumps(release()))
-            old = 'busybox@sha256:' + 'd' * 64
-            new = 'busybox@sha256:' + 'e' * 64
-            config = {
-                'name': 'headscale',
-                'services': {
-                    'app': {
-                        'image': 'busybox:latest',
-                        'volumes': [
-                            {'type': 'bind', 'source': str(config_repo / 'headscale/config.yaml'), 'target': '/config'},
-                            {'type': 'volume', 'source': 'data', 'target': '/data'},
-                        ],
-                    }
-                },
-                'volumes': {'data': {'name': 'headscale_data'}},
-            }
-
-            def fake_output(*args, **kwargs):
-                if args[:2] == ('docker', 'compose'):
-                    self.assertEqual(kwargs['cwd'], config_repo)
-                    self.assertIn(str(config_repo / 'headscale'), args)
-                    return json.dumps(config)
-                if args[:2] == ('docker', 'ps'):
-                    return 'container'
-                if args[:2] == ('docker', 'inspect'):
-                    return 'sha256:' + 'f' * 64
-                return json.dumps([old])
-
-            with (
-                patch.dict(os.environ, {'CONFIG_REPO': str(config_repo), 'REPO': str(source), 'RUN_DIR': str(run)}),
-                patch.object(compose_module, 'output', side_effect=fake_output),
-                patch.object(compose_module, 'resolve', return_value=new),
-            ):
-                compose_module.prepare(source / 'headscale/compose.yaml')
-            pinned = json.loads((run / 'compose/headscale.json').read_text())
-            before = json.loads((run / 'compose-before/headscale.json').read_text())
-            self.assertEqual(pinned['name'], 'headscale')
-            self.assertEqual(pinned['volumes'], config['volumes'])
-            self.assertEqual(pinned['services']['app']['volumes'], config['services']['app']['volumes'])
-            self.assertEqual(pinned['services']['app']['image'], new)
-            self.assertEqual(before['services']['app']['image'], old)
-            self.assertEqual((run / 'compose/headscale.json').stat().st_mode & 0o777, 0o600)
-
-    def test_registry_index_and_single_image_descriptors(self):
-        for digest in ('a' * 64, 'b' * 64):
-            with patch.object(compose_module, 'output', return_value=json.dumps({'digest': 'sha256:' + digest})):
-                self.assertEqual(
-                    compose_module.resolve('registry.test:5000/repo:latest'), f'registry.test:5000/repo@sha256:{digest}'
-                )
-
-
-class ControllerTests(unittest.TestCase):
-    def test_completed_stage_cannot_apply_again(self):
-        with tempfile.TemporaryDirectory() as scratch:
-            directory = Path(scratch)
-            controller.atomic_json(
-                directory / 'status.json', {'state': 'success', 'stages': {'apply-k8s': {'result': 'success'}}}
-            )
-            with patch.object(subprocess, 'run') as execute:
-                self.assertTrue(controller.stage(directory, 'apply-k8s', 60))
-                execute.assert_not_called()
-
-    def test_run_id_is_not_shell_or_path_input(self):
-        for invalid in ('../123', '-1', '1;touch bad', 'abc', '1/2'):
-            with self.assertRaises(ValueError):
-                controller.run_directory(invalid)
-
-    def test_exact_previous_revision_is_used_for_rollback(self):
-        with tempfile.TemporaryDirectory() as scratch:
-            root = Path(scratch)
-            (root / 'current').write_text(str(root))
-            (root / 'revisions.json').write_text(
-                json.dumps([{'kind': 'deployment', 'namespace': 'app', 'name': 'web', 'uid': 'same', 'revision': 7}])
-            )
-            (root / 'failed').write_text('deployment app web\n')
-            script = """set -euo pipefail
-source "$LIB"
-kubectl() {
-  case "$*" in
-    *metadata.annotations*) printf '{}' ;;
-    *metadata.uid*) printf same ;;
-    'rollout undo'*) printf '%s\\n' "$*" >>"$CALLS" ;;
-    'rollout status'*) return 0 ;;
-    *) return 1 ;;
-  esac
-}
-rollback_workloads "$FAILED"
-"""
-            env = {
-                **os.environ,
-                'REPO': str(ROOT),
-                'LIB': str(ROOT / '.gitea/workflows/deploy-lib.sh'),
-                'DEPLOY_SNAPSHOT_DIR': str(root),
-                'CALLS': str(root / 'calls'),
-                'FAILED': str(root / 'failed'),
-            }
-            subprocess.run(['/usr/bin/bash', '-c', script], env=env, check=True)  # noqa: S603
-            self.assertIn('--to-revision=7', (root / 'calls').read_text())
-
-
-if __name__ == '__main__':
-    unittest.main()

tests/test_cicd_lifecycle.py

@@ -1,237 +0,0 @@
-"""Release publication and controller recovery tests without a live server."""
-
-import io
-import json
-import os
-import subprocess
-import tempfile
-import unittest
-import zipfile
-from pathlib import Path
-from unittest.mock import Mock, patch
-
-from test_cicd import ROOT, controller, release, release_module
-
-
-class ArtifactTests(unittest.TestCase):
-    def test_archive_rejects_nested_or_extra_files(self):
-        api = object.__new__(release_module.Gitea)
-        api.base = 'https://example.test/api/v1/repos/a/b'
-        for names in (['../release.json'], ['release.json', 'credentials']):
-            blob = io.BytesIO()
-            with zipfile.ZipFile(blob, 'w') as archive:
-                for name in names:
-                    archive.writestr(name, json.dumps(release()))
-            replies = [{'artifacts': [{'id': 1, 'name': 'release-' + 'a' * 40}]}, blob.getvalue()]
-            with (
-                patch.object(api, 'pages', return_value=iter([{'name': 'build', 'conclusion': 'success'}])),
-                patch.object(api, 'request', side_effect=replies),
-                self.assertRaisesRegex(ValueError, 'archive'),
-            ):
-                api.release({'id': 1, 'head_sha': 'a' * 40})
-
-    def test_quoted_and_single_platform_images_render_from_checked_release(self):
-        with tempfile.TemporaryDirectory() as scratch:
-            path = Path(scratch) / 'release.json'
-            path.write_text(json.dumps(release()))
-            result = io.StringIO()
-            image = 'gcr.forust.xyz/forust/error-pages'
-            with patch.dict(os.environ, {'RELEASE_FILE': str(path), 'DEPLOY_SHA': 'a' * 40}):
-                release_module.render(io.StringIO(f'  image: "{image}:prod" # note\n'), result)
-            self.assertEqual(result.getvalue(), f'  image: "{image}@sha256:{"b" * 64}" # note\n')
-
-    def test_edu_release_digest_is_preserved(self):
-        with tempfile.TemporaryDirectory() as scratch:
-            path = Path(scratch) / 'release.json'
-            path.write_text(json.dumps(release()))
-            image = 'gcr.forust.xyz/forust/session-keeper'
-            line = f'image: {image}@sha256:{"e" * 64}\n'
-            result = io.StringIO()
-            with patch.dict(os.environ, {'RELEASE_FILE': str(path), 'DEPLOY_SHA': 'a' * 40}):
-                release_module.render(io.StringIO(line), result)
-                self.assertEqual(result.getvalue(), line)
-                with self.assertRaises(ValueError):
-                    release_module.render(io.StringIO(f'image: {image}:prod\n'), io.StringIO())
-
-    def test_unknown_image_cannot_emit_partial_manifest(self):
-        with tempfile.TemporaryDirectory() as scratch:
-            path = Path(scratch) / 'release.json'
-            path.write_text(json.dumps(release()))
-            result = io.StringIO()
-            with (
-                patch.dict(os.environ, {'RELEASE_FILE': str(path), 'DEPLOY_SHA': 'a' * 40}),
-                self.assertRaisesRegex(ValueError, 'missing'),
-            ):
-                release_module.render(
-                    io.StringIO('kind: Deployment\nimage: gcr.forust.xyz/forust/unknown:prod\n'), result
-                )
-            self.assertEqual(result.getvalue(), '')
-
-    def test_only_changed_image_is_built_and_credentials_are_removed(self):
-        with tempfile.TemporaryDirectory() as scratch:
-            root = Path(scratch)
-            built = []
-            auth_directories = []
-            old = release()
-
-            def fake_command(*args, **kwargs):
-                if args[:2] == ('git', 'rev-parse'):
-                    return 'e' * 40
-                if args[:3] == ('docker', 'buildx', 'build'):
-                    built.append(args[args.index('--file') + 1])
-                    metadata = Path(args[args.index('--metadata-file') + 1])
-                    metadata.write_text(json.dumps({'containerimage.digest': 'sha256:' + 'f' * 64}))
-                    auth_directories.append(Path(kwargs['env']['DOCKER_CONFIG']))
-                return ''
-
-            api = Mock()
-            api.successful_runs.return_value = iter([{'id': 1}])
-            api.release.return_value = old
-            with (
-                patch.dict(
-                    os.environ,
-                    {
-                        'GITHUB_SHA': 'e' * 40,
-                        'GITHUB_RUN_ID': '2',
-                        'REGISTRY_USERNAME': 'test',
-                        'REGISTRY_PASSWORD': 'placeholder',
-                    },
-                ),
-                patch.object(release_module.Path, 'home', return_value=root),
-                patch.object(release_module, 'Gitea', return_value=api),
-                patch.object(
-                    release_module,
-                    'fingerprint',
-                    side_effect=lambda context, _file: ('d' if context == 'errorpages' else 'c') * 64,
-                ),
-                patch.object(release_module, 'command', side_effect=fake_command),
-                patch.object(subprocess, 'run', return_value=subprocess.CompletedProcess([], 0)),
-            ):
-                release_module.build(root / 'release.json')
-            self.assertEqual(built, ['errorpages/Dockerfile'])
-            self.assertTrue(all(not directory.exists() for directory in auth_directories))
-            self.assertEqual(json.loads((root / 'release.json').read_text())['sha'], 'e' * 40)
-
-
-class DurableRunTests(unittest.TestCase):
-    def test_duplicate_start_only_reattaches(self):
-        with tempfile.TemporaryDirectory() as scratch:
-            state = Path(scratch)
-            directory = state / 'runs/123-1'
-            directory.mkdir(parents=True)
-            request = {'release': release(), 'mode': 'full', 'refresh_images': False}
-            controller.atomic_json(directory / 'request.json', request)
-            controller.atomic_json(directory / 'status.json', {'state': 'running', 'stages': {}})
-            with (
-                patch.object(controller, 'STATE', state),
-                patch.object(controller.sys, 'stdin', io.TextIOWrapper(io.BytesIO(json.dumps(request).encode()))),
-                patch.object(controller, 'command') as execute,
-            ):
-                controller.start('123-1')
-                execute.assert_not_called()
-
-    def test_failed_apply_still_verifies_and_does_not_advance_baseline(self):
-        with tempfile.TemporaryDirectory() as scratch:
-            state = Path(scratch)
-            directory = state / 'runs/123-1'
-            directory.mkdir(parents=True)
-            controller.atomic_json(
-                directory / 'request.json', {'release': release(), 'mode': 'full', 'refresh_images': False}
-            )
-            controller.atomic_json(directory / 'status.json', {'state': 'queued', 'stages': {}})
-            called = []
-
-            def fake_stage(folder, name, _budget):
-                called.append(name)
-                status = json.loads((folder / 'status.json').read_text())
-                status['stages'][name] = {'result': 'failure' if name == 'apply-k8s' else 'success'}
-                controller.atomic_json(folder / 'status.json', status)
-                return name != 'apply-k8s'
-
-            with (
-                patch.object(controller, 'STATE', state),
-                patch.object(controller, 'make_plan', return_value={'selected': {}, 'helm': [], 'removed': []}),
-                patch.object(controller, 'stage', side_effect=fake_stage),
-                patch.object(controller, 'command', return_value='1'),
-                self.assertRaises(RuntimeError),
-            ):
-                controller.execute('123-1')
-            self.assertIn('verify-k8s', called)
-            self.assertIn('smoke', called)
-            self.assertNotIn('apply-compose', called)
-            self.assertFalse((state / 'last-success.json').exists())
-            self.assertEqual(json.loads((directory / 'status.json').read_text())['state'], 'failure')
-
-    def test_recovery_finishes_baseline_after_all_stages_completed(self):
-        with tempfile.TemporaryDirectory() as scratch:
-            state = Path(scratch)
-            directory = state / 'runs/123-1'
-            directory.mkdir(parents=True)
-            names = ('doctor', 'validate', 'apply-k8s', 'apply-compose', 'verify-k8s', 'smoke')
-            controller.atomic_json(
-                directory / 'status.json',
-                {'state': 'running', 'stages': {name: {'result': 'success'} for name in names}},
-            )
-            controller.atomic_json(directory / 'plan.json', {'sha': 'a' * 40})
-            with patch.object(controller, 'STATE', state), patch.object(controller, 'stage') as execute:
-                controller.recover(directory)
-                execute.assert_not_called()
-            self.assertEqual(json.loads((state / 'last-success.json').read_text())['run_id'], '123-1')
-            self.assertEqual(json.loads((directory / 'status.json').read_text())['state'], 'success')
-
-    def test_manual_recovery_retries_checks_without_repeating_apply(self):
-        with tempfile.TemporaryDirectory() as scratch:
-            state = Path(scratch)
-            directory = state / 'runs/123-1'
-            (directory / 'snapshot').mkdir(parents=True)
-            (directory / 'snapshot/current').write_text('snapshot')
-            controller.atomic_json(
-                directory / 'status.json',
-                {
-                    'state': 'failure',
-                    'stages': {
-                        'apply-k8s': {'result': 'failure'},
-                        'verify-k8s': {'result': 'failure'},
-                        'smoke': {'result': 'failure'},
-                    },
-                },
-            )
-            called = []
-
-            def checks(folder, name, _budget):
-                status = json.loads((folder / 'status.json').read_text())
-                self.assertNotIn(name, status['stages'])
-                called.append(name)
-                status['stages'][name] = {'result': 'success'}
-                controller.atomic_json(folder / 'status.json', status)
-                return True
-
-            with patch.object(controller, 'STATE', state), patch.object(controller, 'stage', side_effect=checks):
-                controller.recover(directory, retry=True)
-            self.assertEqual(called, ['verify-k8s', 'smoke'])
-            self.assertFalse((state / 'last-success.json').exists())
-            self.assertEqual(json.loads((directory / 'status.json').read_text())['state'], 'failure')
-
-
-class InstallerTests(unittest.TestCase):
-    def test_version_comparison_is_exact_without_network_or_host_packages(self):
-        with tempfile.TemporaryDirectory() as scratch:
-            root = Path(scratch)
-            binary = root / 'bin/fake'
-            binary.parent.mkdir()
-            binary.write_text('#!/bin/sh\necho fake-v1.7.70\n')
-            binary.chmod(0o755)
-            script = """set -euo pipefail
-source "$LIB"
-if at_version fake 1.7.7; then exit 1; fi
-at_version fake 1.7.70
-"""
-            subprocess.run(  # noqa: S603
-                ['/usr/bin/bash', '-c', script],
-                check=True,
-                env={**os.environ, 'TOOLS_DIR': str(root), 'LIB': str(ROOT / '.gitea/workflows/install-ci-tools.sh')},
-            )
-
-
-if __name__ == '__main__':
-    unittest.main()