fix(edu-master): bound session refreshes and expire stale cookies

.gitea/EDU_HANDOFF.md

@@ -1,42 +0,0 @@
-# EDU ownership handoff
-
-## Current status
-
-EDU PR #1 merged at 2026-10-07 08:04:30 UTC. Main release `5094952464ce315130839303985fd04d721bc1f2` passed CI run 1585 and deploy run 1586. The workstation checkout `/srv/edu-master` is at that SHA. The release changed the application image digests:
-
-- Session keeper: `sha256:1e59473bd40fe4c22622017d808a8927a68788275fe073dc23d718c44b2fd5dd`
-- Webinar checker: `sha256:987d9bf0770272766523ea5b94c7f3f849175d737551d46591ae55e058cf9f12`
-
-The live workloads remain healthy in context `Default`, namespace `edu-master`. Both health and live probes return 200. Redis AUTH passes, session TTL is 1178 seconds, the delivery backlog is zero, all nine EDU alert rules are healthy, and the scrape target is UP. The unauthorized-pod Redis check passed. The Redis PVC UID and Secret UID and values, including the Fernet key, match their pre-release state.
-
-The homelab EDU active marker was present after the EDU deployment. It was moved to the private snapshot as `homelab-k8s-active.marker` while holding `/tmp/homelab-apply.lock`. The homelab checkout at `/srv/homelab` is at `5f9354b` and has the tracked marker deletion. Its deploy preflight blocks a dirty checkout until this removal is reconciled. Preserve private ignored configuration when syncing that checkout.
-
-The remaining homelab change is PR #105, branch `feat/edu-handoff-matrix`, based on `codex/ci-visible-checks`. Its eight protected checks passed. Renovate runs 1587 and 1588 passed. Image publishing was skipped for the PR. The EDU runtime changes are in PR #3 from `fix/handoff-runtime` to `main`; CI run 1589 is in progress. Those runtime changes have not been released.
-
-## Approval gate and next steps
-
-PR #99 must merge before PR #105 can target `main`. A merge attempt for PR #99 returned HTTP 405 because it needs one approval; the protected branch has `required_approvals=1` and whitelist approval is enabled. This approval gate prevents the remaining transfer steps.
-
-After the required approval:
-
-1. Merge PR #99.
-2. Retarget PR #105 to `main`. Complete CI and review, then approve and merge it.
-3. Under the homelab apply lock, sync `/srv/homelab` to the merged removal. Preserve private ignored configuration and keep the active marker removed. Confirm the deploy preflight is clean.
-4. Merge the EDU runtime PR after its CI and review pass. The main-push CI run must complete successfully before its exact SHA can deploy.
-5. Verify the new release SHA, image digests, workload health, Redis AUTH and TTL, backlog, PVC and Secret identity, and monitoring. Record the results in the EDU PR.
-
-`AUTODEPLOY=false` is explicitly configured. The EDU repository path and port secrets are confirmed, and `EDU_KUBE_CONTEXT=Default` is configured as a repository variable. Keep deployment and registry credentials outside Git. Never run both homelab and EDU deployment paths at the same time.
-
-## Change summary
-
-The homelab PR removes the EDU subtree, deployment and image selection, rollback and verification cases, route probes, Renovate references, and external-image exceptions. It adds a serial dynamic matrix for the three homelab images. Each job builds an image or reuses a matching immutable digest. The final job checks all image results and publishes full-SHA tags and the existing release artifact only after they pass. PRs do not publish images. The protected check names from PR #99 are preserved. PR #100's service-metrics work is independent of this handoff.
-
-The EDU runtime PR adds the Playwright service manifest, reconciles Redis storage and Secret reload annotations, and adds pre-apply Redis backup and identity checks. It verifies application endpoints, Redis AUTH, session TTL, metrics, and all nine vmalert rules. Rollback checks workload and application health and reports when manual recovery is needed. Its deployment guard rejects an unexpected or dirty checkout and refuses deployment while either legacy homelab EDU marker exists.
-
-## Rollback and limits
-
-The private snapshot is `/home/forust/.local/state/edu-master-deploy/handoff-20261007T080838Z` on the workstation. It contains the pre-handoff Redis RDB and recovery data. RDB checksum verification confirmed twelve keys. Keep the snapshot outside Git. For an EDU release failure, restore the saved Kubernetes resources and inspect application health. The rollback does not automatically restore the Redis RDB; restore old Redis data only when recovery requires it.
-
-For an ownership rollback, stop EDU deployment triggers first, restore the reviewed homelab source and marker, then reapply recorded immutable image digests. Verify both workload and application health. Never delete or recreate the Redis PVC.
-
-The initial EDU release and the homelab marker move are complete. PR #99 approval and merge, PR #105 retarget and merge, homelab checkout reconciliation, EDU runtime PR merge, and release of those runtime changes remain pending. Synthetic Telegram delivery and Alertmanager-to-Telegram notification were not tested.

.gitea/deploy-dependencies.json

@@ -1,3 +0,0 @@
-{
-  "postgres": ["authentik", "gitea", "immich", "n8n", "netbox", "netronome"]
-}

.gitea/runner/README.md

@@ -1,130 +0,0 @@
-# Homelab CI/CD
-
-The native Gitea runner runs on **vps**; production runs on **workstation**.
-Compose, workflow, shell, Python, formatting, YAML, Dockerfile and Kubernetes
-checks appear as separate jobs. Jobs run on `homelab:host`, one at a time; the
-build waits for every check to pass. CI and deploy runs also show a summary with
-the release SHA, image build or reuse results, deploy mode, selected services,
-and image digests. Failed runs keep a summary of completed image builds, stage
-results, apply results, and recorded Kubernetes recovery. The final deploy
-summary is in the smoke job; earlier jobs show the state observed at that time.
-Apply success is separate from health and recovery. Update the installed
-workstation controller with `setup-workstation.sh` when no deploy is running.
-No job images or Kubernetes credentials
-are needed on the VPS. Builds use one pinned BuildKit helper container. CI and deploy are separate workflows.
-
-## Runner installation
-
-Install Docker Engine with Compose and Buildx, Git, Python 3.11+, Bash, curl,
-GNU tar/xz, flock and systemd using the host's package manager. Keep the existing
-Gitea runner 3.0.2 binary at `/usr/local/bin/gitea-runner`.
-
-From this checkout on the VPS:
-
-```sh
-sudo bash .gitea/runner/setup-runner.sh
-```
-
-The installer reuses `/var/lib/gitea-runner/.runner` and the existing service.
-For a new host, install the same runner binary and register as `gitea-runner`
-using the registration token interactively, label `homelab:host`, and working
-directory `/var/lib/gitea-runner`; then rerun the installer. Tokens never belong
-in this repository or command-line examples.
-
-Pinned tools live in the runner user's `~/.cache/homelab-ci`; CI repairs version
-drift there. Installations are locked. Buildx uses only the `homelab-ci` builder,
-pushes directly to the registry, and caps retained local cache at 1 GiB with a
-2 GiB free-space target. This is not a hard limit on peak build disk usage.
-Nothing runs `docker system prune`, removes unrelated images, or deletes volumes.
-
-## Workstation setup
-
-As the existing SSH deploy user on workstation:
-
-```sh
-sudo loginctl enable-linger forust
-bash .gitea/runner/setup-workstation.sh
-```
-
-The controller uses `/srv/homelab` as the persistent configuration tree and makes
-a detached source worktree for each SHA. It never resets `/srv/homelab`, moves
-local configuration, renames Compose projects, or changes volume names.
-The installer records the current Kubernetes context and cluster UID in
-`~/.config/homelab-deploy/environment`. Check these before installing.
-
-Configure Gitea Actions Variables:
-
-- `DEPLOY_HOST`, `DEPLOY_USER`, `DEPLOY_PORT`: the existing VPS-to-workstation SSH endpoint.
-- `DEPLOY_KNOWN_HOSTS`: workstation's verified SSH host key entry for that endpoint.
-- `AUTODEPLOY`: `false` initially; `true` enables deployment after successful main CI.
-
-Keep `DEPLOY_SSH_KEY`, `REGISTRY_USERNAME` and `REGISTRY_PASSWORD` in Actions
-Secrets. Legacy endpoint secrets remain accepted during migration. The Actions
-token must have repository read and Actions read access for release downloads.
-The deploy user's existing Docker registry authentication remains necessary.
-
-## Releases and deployment
-
-CI publishes `release-<full SHA>` as a Gitea artifact with all three owned image
-digests and build input fingerprints. Unchanged images are reused only from a
-successful main CI artifact, never from `:prod`. Expired artifacts cause CI to
-rebuild images; they block deployment until CI is rerun.
-
-Run deploy from main with `deploy_ref=main` or a checked SHA:
-
-- `full`: required for the first baseline; reconcile all active components.
-- `changed`: compare with the last fully successful production deploy.
-- `plan`: validate configuration and show selection without changing production resources.
-- `refresh_images=true`: explicitly refresh mutable third-party Compose tags.
-
-The manual and automatic paths both require successful CI, a successful build
-job and the exact SHA's release artifact. PRs cannot publish images or deploy.
-Removed resources are reported and require explicit removal; no automatic prune.
-Service dependencies are listed in `.gitea/deploy-dependencies.json`.
-
-A workstation user systemd service holds the deploy lock across validation,
-sequential apply, verification and smoke checks. SSH clients only submit/follow:
-disconnecting or cancelling the Actions client does not kill production apply.
-Retrying the same run ID does not start another apply. `ExecStopPost` recovers
-interrupted runs before the unit finishes. Kubernetes rolls back to captured
-revisions; configuration and persistent data are not reverted.
-
-## Status and recovery
-
-`--retry` repeats failed verification and smoke checks, never apply. Recovery
-keeps a failed deploy out of the successful baseline, even after rollback.
-
-On workstation (replace the numeric ID with Actions run ID and attempt):
-
-```sh
-python3 ~/.local/lib/homelab-deploy/controller.py status 123-1
-python3 ~/.local/lib/homelab-deploy/controller.py recover 123-1 --retry
-journalctl --user -u homelab-deploy@123-1
-```
-
-Runs live in `~/.local/state/homelab-deploy/runs`. Compose stores resolved configs
-with restricted permissions; these may contain credentials and must never be
-uploaded as CI artifacts. Stage logs print the exact manual recovery command
-using `compose-before/<stack>.json`, the original project directory and project
-name. Compose does not automatically roll back, and Nextcloud AIO's child
-containers remain managed by AIO. Preserve its own backups for data recovery.
-
-The controller retains twenty successful/planned runs and preserves failures.
-Update the workstation dispatcher only when no deploy is running.
-
-## Validation and migration rollback
-
-```sh
-python3 -m unittest discover -s tests -v
-bash .gitea/tests/deploy-validation.sh
-```
-
-Test on a separate namespace before the initial production `full` run. Check a
-failed rollout, interrupted SSH and repeated run ID, and verify that an isolated
-service change does not upgrade unrelated Helm releases or Compose stacks.
-
-To roll back the migration, disable autodeploy and finish or recover the remote
-run first. Restore the runner config/unit from `.before-<timestamp>` backups,
-reload systemd and restart the runner. Restore the prior workflows from Git.
-Production data and persistent volumes stay where they were. Do not remove run
-state or Compose recovery files until recovery is confirmed.

.gitea/runner/buildkitd.toml

@@ -1,11 +0,0 @@
-[worker.oci]
-  gc = true
-  reservedSpace = "256MB"
-  maxUsedSpace = "1GB"
-  minFreeSpace = "2GB"
-
-[[worker.oci.gcpolicy]]
-  reservedSpace = "256MB"
-  maxUsedSpace = "1GB"
-  minFreeSpace = "2GB"
-  all = true

.gitea/runner/config.yaml

@@ -1,10 +0,0 @@
-runner:
-  file: /var/lib/gitea-runner/.runner
-  capacity: 1
-  timeout: 5h
-  labels:
-    - homelab:host
-cache:
-  enabled: false
-container:
-  docker_host: unix:///var/run/docker.sock

.gitea/runner/gitea-runner.service

@@ -1,18 +0,0 @@
-[Unit]
-Description=Gitea Actions runner
-After=network-online.target docker.service
-Wants=network-online.target
-
-[Service]
-User=gitea-runner
-Group=gitea-runner
-SupplementaryGroups=docker
-WorkingDirectory=/var/lib/gitea-runner
-Environment=PATH=/var/lib/gitea-runner/.cache/homelab-ci/bin:/usr/local/bin:/usr/bin:/bin
-ExecStart=/usr/local/bin/gitea-runner daemon --config /etc/gitea-runner/config.yaml
-Restart=on-failure
-RestartSec=5
-UMask=0077
-
-[Install]
-WantedBy=multi-user.target

.gitea/runner/homelab-deploy@.service

@@ -1,12 +0,0 @@
-[Unit]
-Description=Homelab deploy %i
-
-[Service]
-Type=exec
-EnvironmentFile=%h/.config/homelab-deploy/environment
-ExecStart=/usr/bin/python3 %h/.local/lib/homelab-deploy/controller.py execute %i
-ExecStopPost=/usr/bin/python3 %h/.local/lib/homelab-deploy/controller.py recover %i
-RuntimeMaxSec=5h
-TimeoutStopSec=135min
-KillMode=control-group
-UMask=0077

.gitea/runner/setup-runner.sh

@@ -1,48 +0,0 @@
-#!/usr/bin/env bash
-# Native host runner, with pinned user-space tools and no extra CI images.
-set -euo pipefail
-here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-[ "$(id -u)" -eq 0 ] || { echo 'Run with sudo on the runner host' >&2; exit 1; }
-for tool in docker curl python3 git tar xz flock runuser systemctl; do
-  command -v "$tool" >/dev/null || { echo "Install missing prerequisite: $tool" >&2; exit 1; }
-done
-docker info >/dev/null
-docker compose version >/dev/null
-docker buildx version >/dev/null
-id gitea-runner >/dev/null 2>&1 || useradd --system --create-home --home-dir /var/lib/gitea-runner --shell /usr/bin/bash gitea-runner
-# Reuse the established service account and runner registration.
-runner_home="$(getent passwd gitea-runner | cut -d: -f6)"
-[ "$runner_home" = /var/lib/gitea-runner ] || { echo 'Unexpected runner home; inspect the existing service first' >&2; exit 1; }
-runuser -u gitea-runner -- docker info >/dev/null || { echo "The runner user needs access to Docker before setup" >&2; exit 1; }
-command -v gitea-runner >/dev/null || { echo 'Install gitea-runner 3.0.2 at /usr/local/bin/gitea-runner first' >&2; exit 1; }
-mkdir -p /etc/gitea-runner
-stamp="$(date -u +%Y%m%dT%H%M%SZ)"
-for existing in /etc/gitea-runner/config.yaml /etc/systemd/system/gitea-runner.service; do
-  [ ! -f "$existing" ] || cp -p "$existing" "$existing.before-$stamp"
-done
-scratch="$(mktemp -d)"
-trap 'rm -rf "$scratch"' EXIT
-chmod 755 "$scratch"
-install -m 0644 "$here/../workflows/install-ci-tools.sh" "$here/../workflows/tool-versions.env" "$scratch/"
-runuser -u gitea-runner -- bash "$scratch/install-ci-tools.sh"
-install -m 0644 "$here/config.yaml" /etc/gitea-runner/config.yaml
-python3 - <<'PYLABELS'
-import json
-from pathlib import Path
-registration = Path('/var/lib/gitea-runner/.runner')
-if registration.exists():
-    labels = json.loads(registration.read_text()).get('labels', [])
-    labels = [label for label in labels if isinstance(label, str) and label.split(':')[0] != 'homelab']
-    labels.append('homelab:host')
-    config = Path('/etc/gitea-runner/config.yaml')
-    config.write_text(config.read_text().replace('    - homelab:host', '\n'.join('    - ' + json.dumps(label) for label in labels)))
-PYLABELS
-install -m 0644 "$here/gitea-runner.service" /etc/systemd/system/gitea-runner.service
-if [ ! -f /var/lib/gitea-runner/.runner ]; then
-  echo 'Register once as gitea-runner with homelab:host before starting the service.'
-  exit 0
-fi
-systemctl daemon-reload
-systemctl enable --now gitea-runner.service
-systemctl restart gitea-runner.service
-echo "Runner ready. Configuration backups: *.before-$stamp"

.gitea/runner/setup-workstation.sh

@@ -1,32 +0,0 @@
-#!/usr/bin/env bash
-# Run as the existing deploy user on workstation. Never resets the working tree.
-set -euo pipefail
-here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-repo="${HOMELAB_REPO:-/srv/homelab}"
-for tool in python3 git kubectl helm docker flock timeout; do
-  command -v "$tool" >/dev/null || { echo "Install missing dependency: $tool" >&2; exit 1; }
-done
-[ -d "$repo/.git" ] || { echo "Missing deploy checkout: $repo" >&2; exit 1; }
-[[ "$repo" =~ ^/[A-Za-z0-9_./-]+$ ]] || { echo 'Deploy path must be absolute and contain no whitespace' >&2; exit 1; }
-if [ "$(loginctl show-user "$USER" -p Linger --value)" != yes ]; then
-  echo "Run once: sudo loginctl enable-linger $USER" >&2
-  exit 1
-fi
-config="${XDG_CONFIG_HOME:-$HOME/.config}/homelab-deploy"
-mkdir -p "$config" "$HOME/.local/lib/homelab-deploy" "$HOME/.config/systemd/user"
-chmod 700 "$config"
-if [ ! -f "$config/environment" ]; then
-  context="$(kubectl config current-context)"
-  cluster_uid="$(kubectl get namespace kube-system -o jsonpath='{.metadata.uid}')"
-  printf 'HOMELAB_REPO=%s\nKUBE_CONTEXT=%s\nEXPECTED_CLUSTER_UID=%s\n' "$repo" "$context" "$cluster_uid" >"$config/environment"
-  chmod 600 "$config/environment"
-fi
-# Do not replace a dispatcher while an existing deploy uses it.
-if systemctl --user list-units 'homelab-deploy@*' --state=running --no-legend | grep -q .; then
-  echo 'An existing deploy is running; wait before updating the controller' >&2
-  exit 1
-fi
-install -m 0755 "$here/../workflows/deploy-controller.py" "$HOME/.local/lib/homelab-deploy/controller.py"
-install -m 0644 "$here/homelab-deploy@.service" "$HOME/.config/systemd/user/homelab-deploy@.service"
-systemctl --user daemon-reload
-echo 'Controller ready. Run a checked main SHA in full mode for the initial baseline.'

.gitea/tests/deploy-validation.sh

@@ -1,94 +0,0 @@
-#!/usr/bin/env bash
-# Local regressions only: kubectl is mocked and Docker is used for config parsing.
-set -euo pipefail
-repo="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
-scratch="$(mktemp -d)"
-trap 'rm -rf "$scratch"' EXIT
-
-mkdir -p "$scratch/repo/app" "$scratch/repo/postgres" "$scratch/repo/netbird" "$scratch/repo/renovate"
-git -C "$scratch/repo" init -q
-for file in app/compose.yaml postgres/shared-compose.yaml netbird/client.compose.yaml renovate/renovate-compose.yaml; do
-  touch "$scratch/repo/$file"
-done
-git -C "$scratch/repo" add .
-# shellcheck source=../workflows/compose-lint.sh
-source "$repo/.gitea/workflows/compose-lint.sh"
-actual="$(cd "$scratch/repo" && compose_files)"
-expected=$'app/compose.yaml\nnetbird/client.compose.yaml\npostgres/shared-compose.yaml\nrenovate/renovate-compose.yaml'
-[ "$actual" = "$expected" ] || { echo 'Compose discovery missed a file' >&2; exit 1; }
-
-cat >"$scratch/compose.yaml" <<'YAML'
-services:
-  example:
-    image: busybox:1.37.0
-    environment:
-      REQUIRED: ${HOMELAB_TEST_REQUIRED:?required for this regression}
-YAML
-unset HOMELAB_TEST_REQUIRED
-if validate_compose_file "$scratch/compose.yaml" >"$scratch/config.log" 2>&1; then
-  echo 'Full Compose validation accepted a missing variable' >&2
-  exit 1
-fi
-grep -q 'required for this regression' "$scratch/config.log"
-HOMELAB_TEST_REQUIRED=present validate_compose_file "$scratch/compose.yaml"
-
-cat >"$scratch/resources.json" <<'JSON'
-{"kind":"List","items":[
-  {"kind":"Deployment","metadata":{"namespace":"app"},"spec":{"template":{"spec":{
-    "containers":[{"envFrom":[{"secretRef":{"name":"credentials"}},{"secretRef":{"name":"optional","optional":true}}],"env":[{"valueFrom":{"secretKeyRef":{"name":"credentials","key":"password"}}}]}],
-    "initContainers":[{"envFrom":[{"secretRef":{"name":"init"}}]}],
-    "imagePullSecrets":[{"name":"registry"}],
-    "volumes":[{"secret":{"secretName":"mounted"}},{"projected":{"sources":[{"secret":{"name":"projected"}},{"secret":{"name":"optional-projected","optional":true}}]}}]
-  }}}},
-  {"kind":"CronJob","metadata":{},"spec":{"jobTemplate":{"spec":{"template":{"spec":{"containers":[{"envFrom":[{"secretRef":{"name":"cron"}}]}]}}}}}},
-  {"kind":"IngressRoute","metadata":{"namespace":"app"},"spec":{"tls":{"secretName":"controller-issued-tls"}}}
-]}
-JSON
-actual="$(jq -r -f "$repo/.gitea/workflows/secret-references.jq" "$scratch/resources.json" | sort)"
-expected=$'app credentials\napp init\napp mounted\napp projected\napp registry\ndefault cron'
-[ "$actual" = "$expected" ] || { echo "Unexpected Secret references: $actual" >&2; exit 1; }
-
-REPO="$repo"
-# shellcheck source=../workflows/deploy-lib.sh
-source "$repo/.gitea/workflows/deploy-lib.sh"
-K8S_MANIFESTS=("$scratch/resources.json")
-KUSTOMIZE_APPS=()
-# No live cluster access. Reject credentials in app even if they exist elsewhere.
-kubectl() {
-  case "$1" in
-    create) cat "$scratch/resources.json" ;;
-    get)
-      if [ "$3" = credentials ] && [ "$5" = app ]; then
-        return 1
-      fi
-      return 0
-      ;;
-    *) echo "Unexpected kubectl invocation: $*" >&2; return 1 ;;
-  esac
-}
-if check_referenced_secrets >"$scratch/secrets.log"; then
-  echo 'Namespace-scoped Secret check accepted a missing Secret' >&2
-  exit 1
-fi
-grep -q 'MISSING OR UNREADABLE: app/credentials' "$scratch/secrets.log"
-# API/rendering errors must not produce an empty reference list and pass.
-kubectl() { return 1; }
-if ! skip_uninstalled_vmagent_crd "$REPO/prometheus-stack/k8s/vmagent.yaml"; then
-  echo 'VMAgent preflight did not skip an uninstalled CRD' >&2
-  exit 1
-fi
-kubectl() { return 0; }
-if skip_uninstalled_vmagent_crd "$REPO/prometheus-stack/k8s/vmagent.yaml"; then
-  echo 'VMAgent preflight skipped an installed CRD' >&2
-  exit 1
-fi
-if skip_uninstalled_vmagent_crd "$REPO/prometheus-stack/k8s/victoria.yaml"; then
-  echo 'VMAgent preflight skipped an unrelated manifest' >&2
-  exit 1
-fi
-kubectl() { return 1; }
-if check_referenced_secrets >"$scratch/secrets.log"; then
-  echo 'Secret check accepted a failed manifest render' >&2
-  exit 1
-fi
-printf '%s\n' 'Deploy validation regressions passed.'

.gitea/workflows/ci.yaml

@@ -1,25 +1,38 @@
 name: ci
-"on":
+
+on:
   push:
     branches:
-      - main
-  pull_request: null
-  workflow_dispatch: null
+      - "**"
+  pull_request:
+  workflow_dispatch:
+
+# Every job here is checkout plus local tools. The token needs to read the tree
+# and nothing else, and saying so keeps a future step that reaches for the API
+# from quietly holding a token that can write to the repository.
 permissions:
   contents: read
-  actions: read
+
 concurrency:
   group: ci-${{ github.ref }}
   cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
+
+env:
+  REGISTRY: gcr.forust.xyz
+
 jobs:
-  compose:
-    name: Compose
-    runs-on: homelab
-    timeout-minutes: 15
+  lint-compose:
+    runs-on: [self-hosted, linux, arch, homelab]
+    timeout-minutes: 10
     steps:
       - name: Checkout repository
-        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
-        id: source
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+
+      # Structure check for every committed Compose file, active or not.
+      # Interpolation, env-file and bind-mount resolution are all switched off,
+      # because inactive stacks have no .env here and would only fail on their
+      # ${VAR:?} guards. Active stacks get the full check with interpolation in
+      # the deploy workflow, where the real .env files live.
       - name: Validate Compose files
         shell: bash
         run: |
@@ -48,77 +61,35 @@ jobs:
             exit 1
           fi
           echo "checked ${#files[@]} Compose file(s)"
-        id: check
-      - name: Write the job result
-        if: always()
-        env:
-          SUMMARY_CHECK: Compose
-          SUMMARY_RESULT: ${{ job.status }}
-          SUMMARY_FAILED_STEP:
-            ${{ steps.check.conclusion == 'failure' && 'Check or image build' || steps.source.conclusion == 'failure'
-            && 'Source checkout' || '' }}
-        shell: bash
-        run: |
-          if [ -f .gitea/workflows/release.py ]; then
-            python3 .gitea/workflows/release.py check-summary
-          elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
-            printf '## %s\n\n- Result: **%s**\n- Failed step: %s\n' "$SUMMARY_CHECK" "$SUMMARY_RESULT" "$SUMMARY_FAILED_STEP" >>"$GITHUB_STEP_SUMMARY" || true
-          fi
-  workflows:
-    name: Workflows
-    runs-on: homelab
-    timeout-minutes: 15
+
+  lint-actionlint:
+    runs-on: [self-hosted, linux, arch, homelab]
+    timeout-minutes: 10
     steps:
       - name: Checkout repository
-        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
-        id: source
-      - name: Prepare pinned tools
-        shell: bash
-        run: |
-          set -euo pipefail
-          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh actionlint shellcheck)"
-          echo "$tools_dir" >> "$GITHUB_PATH"
-        id: tools
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+
       - name: Lint Gitea Actions workflows with actionlint
         shell: bash
         run: |
           set -euo pipefail
+          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh actionlint)"
+          export PATH="$tools_dir:$PATH"
           actionlint -config-file .gitea/actionlint.yaml -color .gitea/workflows/*.yaml
-        id: check
-      - name: Write the job result
-        if: always()
-        env:
-          SUMMARY_CHECK: Workflows
-          SUMMARY_RESULT: ${{ job.status }}
-          SUMMARY_FAILED_STEP:
-            ${{ steps.check.conclusion == 'failure' && 'Check or image build' || steps.tools.conclusion == 'failure'
-            && 'Tool setup' || steps.source.conclusion == 'failure' && 'Source checkout' || '' }}
-        shell: bash
-        run: |
-          if [ -f .gitea/workflows/release.py ]; then
-            python3 .gitea/workflows/release.py check-summary
-          elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
-            printf '## %s\n\n- Result: **%s**\n- Failed step: %s\n' "$SUMMARY_CHECK" "$SUMMARY_RESULT" "$SUMMARY_FAILED_STEP" >>"$GITHUB_STEP_SUMMARY" || true
-          fi
-  shell:
-    name: Shell
-    runs-on: homelab
-    timeout-minutes: 15
+
+  lint-shellcheck:
+    runs-on: [self-hosted, linux, arch, homelab]
+    timeout-minutes: 10
     steps:
       - name: Checkout repository
-        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
-        id: source
-      - name: Prepare pinned tools
-        shell: bash
-        run: |
-          set -euo pipefail
-          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh shellcheck jq)"
-          echo "$tools_dir" >> "$GITHUB_PATH"
-        id: tools
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+
       - name: Lint shell scripts with ShellCheck
         shell: bash
         run: |
           set -euo pipefail
+          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh shellcheck)"
+          export PATH="$tools_dir:$PATH"
           mapfile -t scripts < <(
             git ls-files '*.sh' ':(glob)**/*.bash'
           )
@@ -127,42 +98,20 @@ jobs:
             exit 0
           fi
           shellcheck --external-sources --source-path=SCRIPTDIR --severity=style "${scripts[@]}"
-          bash .gitea/tests/deploy-validation.sh
-        id: check
-      - name: Write the job result
-        if: always()
-        env:
-          SUMMARY_CHECK: Shell
-          SUMMARY_RESULT: ${{ job.status }}
-          SUMMARY_FAILED_STEP:
-            ${{ steps.check.conclusion == 'failure' && 'Check or image build' || steps.tools.conclusion == 'failure'
-            && 'Tool setup' || steps.source.conclusion == 'failure' && 'Source checkout' || '' }}
-        shell: bash
-        run: |
-          if [ -f .gitea/workflows/release.py ]; then
-            python3 .gitea/workflows/release.py check-summary
-          elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
-            printf '## %s\n\n- Result: **%s**\n- Failed step: %s\n' "$SUMMARY_CHECK" "$SUMMARY_RESULT" "$SUMMARY_FAILED_STEP" >>"$GITHUB_STEP_SUMMARY" || true
-          fi
-  formatting:
-    name: Formatting
-    runs-on: homelab
-    timeout-minutes: 15
+
+  lint-prettier:
+    runs-on: [self-hosted, linux, arch, homelab]
+    timeout-minutes: 10
     steps:
       - name: Checkout repository
-        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
-        id: source
-      - name: Prepare pinned tools
-        shell: bash
-        run: |
-          set -euo pipefail
-          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh prettier)"
-          echo "$tools_dir" >> "$GITHUB_PATH"
-        id: tools
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+
       - name: Check formatting with Prettier
         shell: bash
         run: |
           set -euo pipefail
+          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh prettier)"
+          export PATH="$tools_dir:$PATH"
 
           mapfile -t prettier_files < <(
             git ls-files \
@@ -176,79 +125,37 @@ jobs:
           fi
 
           prettier --check --ignore-unknown "${prettier_files[@]}"
-        id: check
-      - name: Write the job result
-        if: always()
-        env:
-          SUMMARY_CHECK: Formatting
-          SUMMARY_RESULT: ${{ job.status }}
-          SUMMARY_FAILED_STEP:
-            ${{ steps.check.conclusion == 'failure' && 'Check or image build' || steps.tools.conclusion == 'failure'
-            && 'Tool setup' || steps.source.conclusion == 'failure' && 'Source checkout' || '' }}
-        shell: bash
-        run: |
-          if [ -f .gitea/workflows/release.py ]; then
-            python3 .gitea/workflows/release.py check-summary
-          elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
-            printf '## %s\n\n- Result: **%s**\n- Failed step: %s\n' "$SUMMARY_CHECK" "$SUMMARY_RESULT" "$SUMMARY_FAILED_STEP" >>"$GITHUB_STEP_SUMMARY" || true
-          fi
-  python:
-    name: Python and tests
-    runs-on: homelab
-    timeout-minutes: 15
+
+  lint-ruff:
+    runs-on: [self-hosted, linux, arch, homelab]
+    timeout-minutes: 10
     steps:
       - name: Checkout repository
-        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
-        id: source
-      - name: Prepare pinned tools
-        shell: bash
-        run: |
-          set -euo pipefail
-          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh ruff jq)"
-          echo "$tools_dir" >> "$GITHUB_PATH"
-        id: tools
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+
       - name: Lint and format-check Python with Ruff
         shell: bash
         run: |
           set -euo pipefail
-          ruff check . .gitea/workflows
-          ruff format --check . .gitea/workflows
+          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh ruff)"
+          export PATH="$tools_dir:$PATH"
+          ruff check .
+          ruff format --check .
           python3 -m unittest discover -s tests -v
-        id: check
-      - name: Write the job result
-        if: always()
-        env:
-          SUMMARY_CHECK: Python and tests
-          SUMMARY_RESULT: ${{ job.status }}
-          SUMMARY_FAILED_STEP:
-            ${{ steps.check.conclusion == 'failure' && 'Check or image build' || steps.tools.conclusion == 'failure'
-            && 'Tool setup' || steps.source.conclusion == 'failure' && 'Source checkout' || '' }}
-        shell: bash
-        run: |
-          if [ -f .gitea/workflows/release.py ]; then
-            python3 .gitea/workflows/release.py check-summary
-          elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
-            printf '## %s\n\n- Result: **%s**\n- Failed step: %s\n' "$SUMMARY_CHECK" "$SUMMARY_RESULT" "$SUMMARY_FAILED_STEP" >>"$GITHUB_STEP_SUMMARY" || true
-          fi
-  yaml:
-    name: YAML
-    runs-on: homelab
-    timeout-minutes: 15
+
+  lint-yaml:
+    runs-on: [self-hosted, linux, arch, homelab]
+    timeout-minutes: 10
     steps:
       - name: Checkout repository
-        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
-        id: source
-      - name: Prepare pinned tools
-        shell: bash
-        run: |
-          set -euo pipefail
-          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh yamllint)"
-          echo "$tools_dir" >> "$GITHUB_PATH"
-        id: tools
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+
       - name: Lint YAML syntax
         shell: bash
         run: |
           set -euo pipefail
+          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh yamllint)"
+          export PATH="$tools_dir:$PATH"
 
           mapfile -t yaml_files < <(
             git ls-files '*.yaml' '*.yml' \
@@ -262,41 +169,20 @@ jobs:
           fi
 
           yamllint -c .yamllint "${yaml_files[@]}"
-        id: check
-      - name: Write the job result
-        if: always()
-        env:
-          SUMMARY_CHECK: YAML
-          SUMMARY_RESULT: ${{ job.status }}
-          SUMMARY_FAILED_STEP:
-            ${{ steps.check.conclusion == 'failure' && 'Check or image build' || steps.tools.conclusion == 'failure'
-            && 'Tool setup' || steps.source.conclusion == 'failure' && 'Source checkout' || '' }}
-        shell: bash
-        run: |
-          if [ -f .gitea/workflows/release.py ]; then
-            python3 .gitea/workflows/release.py check-summary
-          elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
-            printf '## %s\n\n- Result: **%s**\n- Failed step: %s\n' "$SUMMARY_CHECK" "$SUMMARY_RESULT" "$SUMMARY_FAILED_STEP" >>"$GITHUB_STEP_SUMMARY" || true
-          fi
-  dockerfiles:
-    name: Dockerfiles
-    runs-on: homelab
-    timeout-minutes: 15
+
+  lint-dockerfiles:
+    runs-on: [self-hosted, linux, arch, homelab]
+    timeout-minutes: 10
     steps:
       - name: Checkout repository
-        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
-        id: source
-      - name: Prepare pinned tools
-        shell: bash
-        run: |
-          set -euo pipefail
-          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh hadolint)"
-          echo "$tools_dir" >> "$GITHUB_PATH"
-        id: tools
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+
       - name: Lint Dockerfiles
         shell: bash
         run: |
           set -euo pipefail
+          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh hadolint)"
+          export PATH="$tools_dir:$PATH"
 
           mapfile -t dockerfiles < <(
             git ls-files ':(glob)**/Dockerfile' ':(glob)**/Dockerfile.*'
@@ -308,41 +194,20 @@ jobs:
           fi
 
           hadolint -c .hadolint.yaml "${dockerfiles[@]}"
-        id: check
-      - name: Write the job result
-        if: always()
-        env:
-          SUMMARY_CHECK: Dockerfiles
-          SUMMARY_RESULT: ${{ job.status }}
-          SUMMARY_FAILED_STEP:
-            ${{ steps.check.conclusion == 'failure' && 'Check or image build' || steps.tools.conclusion == 'failure'
-            && 'Tool setup' || steps.source.conclusion == 'failure' && 'Source checkout' || '' }}
-        shell: bash
-        run: |
-          if [ -f .gitea/workflows/release.py ]; then
-            python3 .gitea/workflows/release.py check-summary
-          elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
-            printf '## %s\n\n- Result: **%s**\n- Failed step: %s\n' "$SUMMARY_CHECK" "$SUMMARY_RESULT" "$SUMMARY_FAILED_STEP" >>"$GITHUB_STEP_SUMMARY" || true
-          fi
-  kubernetes:
-    name: Kubernetes
-    runs-on: homelab
-    timeout-minutes: 15
+
+  validate:
+    runs-on: [self-hosted, linux, arch, homelab]
+    timeout-minutes: 20
     steps:
       - name: Checkout repository
-        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
-        id: source
-      - name: Prepare pinned tools
-        shell: bash
-        run: |
-          set -euo pipefail
-          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh kubeconform)"
-          echo "$tools_dir" >> "$GITHUB_PATH"
-        id: tools
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+
       - name: Validate Kubernetes manifests against JSON schemas
         shell: bash
         run: |
           set -euo pipefail
+          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh kubeconform)"
+          export PATH="$tools_dir:$PATH"
 
           mapfile -t manifests < <(
             git ls-files ':(glob)**/k8s/**/*.yaml' ':(glob)**/k8s/**/*.yml' \
@@ -359,162 +224,327 @@ jobs:
             -ignore-missing-schemas \
             -summary \
             "${manifests[@]}"
-        id: check
-      - name: Write the job result
-        if: always()
-        env:
-          SUMMARY_CHECK: Kubernetes
-          SUMMARY_RESULT: ${{ job.status }}
-          SUMMARY_FAILED_STEP:
-            ${{ steps.check.conclusion == 'failure' && 'Check or image build' || steps.tools.conclusion == 'failure'
-            && 'Tool setup' || steps.source.conclusion == 'failure' && 'Source checkout' || '' }}
+
+      # kubeconform has no schemas for CRDs, so every IngressRoute, Certificate,
+      # PrometheusRule, Middleware, ServersTransport and ServiceMonitor is silently
+      # skipped above. The live API server knows the real CRD schemas (and runs the
+      # cert-manager / Traefik admission webhooks), so validate there too.
+      #
+      # Only services marked with a k8s/active marker are checked: server-side
+      # dry-run needs the target namespace to exist, and inactive services are not
+      # deployed. Services being enabled for the first time are still covered by
+      # the JSON-schema pass above.
+      #
+      # Main pushes only. `--dry-run=server` persists nothing, but it does execute
+      # the admission webhooks of the production API server, so anyone able to open
+      # a pull request would be able to run arbitrary manifest content through
+      # cert-manager and Traefik. A pull request has nothing to gain from it either:
+      # only main is ever deployed, and this job runs to completion before the
+      # deploy workflow is allowed to start, so a bad CRD is still caught before
+      # anything reaches the cluster -- just on the push rather than on the PR.
+      - name: Note the server-side check is not running here
+        if: github.event_name == 'pull_request' || github.ref != 'refs/heads/main'
         shell: bash
         run: |
-          if [ -f .gitea/workflows/release.py ]; then
-            python3 .gitea/workflows/release.py check-summary
-          elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
-            printf '## %s\n\n- Result: **%s**\n- Failed step: %s\n' "$SUMMARY_CHECK" "$SUMMARY_RESULT" "$SUMMARY_FAILED_STEP" >>"$GITHUB_STEP_SUMMARY" || true
+          echo "::notice::Skipping the server-side dry-run. It executes the cert-manager and" \
+            "Traefik admission webhooks against the production API server, so it is limited" \
+            "to pushes to main. CRDs are still schema-checked by kubeconform above, and the" \
+            "server-side pass still runs on main before the deploy."
+
+      - name: Validate active manifests against the live API server
+        if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main'
+        shell: bash
+        run: |
+          set -euo pipefail
+
+          if ! kubectl get --raw='/readyz' --request-timeout=10s >/dev/null 2>&1; then
+            echo "::warning::Cluster unreachable — skipped server-side validation of CRDs (IngressRoute, Certificate, PrometheusRule). Review manifest changes manually."
+            exit 0
           fi
-  image-plan:
-    needs: [compose, workflows, shell, formatting, python, yaml, dockerfiles, kubernetes]
-    if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main'
-    runs-on: homelab
-    timeout-minutes: 10
+
+          mapfile -t k8s_dirs < <(
+            git ls-files '*.yaml' '*.yml' \
+              | grep -E '(^|/)k8s/' \
+              | sed -E 's#((^|.*/)k8s)/.*#\1#' \
+              | sort -u
+          )
+
+          manifests=()
+          kustomize_apps=()
+          for dir in "${k8s_dirs[@]}"; do
+            if [ ! -f "${dir}/active" ]; then
+              echo "skip (no k8s/active): ${dir}"
+              continue
+            fi
+            if [ -f "${dir}/overlays/prod/kustomization.yaml" ]; then
+              kustomize_apps+=("${dir}/overlays/prod")
+            elif [ -f "${dir}/base/kustomization.yaml" ]; then
+              kustomize_apps+=("${dir}/base")
+            else
+              while IFS= read -r f; do
+                [ -n "$f" ] && manifests+=("$f")
+              done < <(
+                git ls-files "${dir}/*.yaml" "${dir}/*.yml" \
+                  | grep -Ev '(^|/)(kustomization\.ya?ml|.*\.example\.ya?ml|.*values\.ya?ml|patch-.*\.ya?ml)$'
+              )
+            fi
+          done
+
+          echo "server-side dry-run: ${#manifests[@]} manifests, ${#kustomize_apps[@]} kustomize apps"
+          failed=0
+          for m in ${manifests[@]+"${manifests[@]}"}; do
+            if ! out="$(kubectl apply --dry-run=server -f "$m" 2>&1)"; then
+              failed=1
+              echo "::error file=${m}::$(printf '%s' "$out" | head -1)"
+            fi
+          done
+          for k in ${kustomize_apps[@]+"${kustomize_apps[@]}"}; do
+            if ! out="$(kubectl apply -k "$k" --dry-run=server 2>&1)"; then
+              failed=1
+              echo "::error file=${k}::$(printf '%s' "$out" | head -1)"
+            fi
+          done
+
+          if [ "$failed" -ne 0 ]; then
+            echo "Server-side validation failed. The API server (or an admission webhook) rejected these manifests."
+            exit 1
+          fi
+          echo "server-side dry-run: all active manifests accepted by the API server"
+
+  build:
+    needs:
+      # The panel's scan-deps/test-backend/test-frontend jobs gated here until
+      # userbot moved to its own repo; upstream's code is upstream's gate now.
+      # The rule is unchanged: publishing and passing the checks are the same
+      # gate, so a commit that fails any of these still cannot move :prod.
+      [lint-actionlint, lint-shellcheck, lint-compose, lint-prettier, lint-ruff, lint-yaml, lint-dockerfiles, validate]
+    if: github.event_name != 'pull_request' && (github.ref_name == 'main' || github.ref_name == 'dev') && !startsWith(github.ref_name, 'renovate/')
+    runs-on: [self-hosted, linux, arch, homelab]
+    timeout-minutes: 60
     outputs:
-      matrix: ${{ steps.plan.outputs.matrix }}
+      services: ${{ steps.services.outputs.services }}
     steps:
       - name: Checkout repository
-        id: source
-        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
         with:
           fetch-depth: 0
-      - name: Detect build inputs against successful CI
-        id: plan
-        env:
-          GITEA_TOKEN: ${{ github.token }}
-        run: python3 .gitea/workflows/release.py prepare --output build-plan.json
-      - name: Store the image plan
-        id: artifact
-        uses: actions/upload-artifact@c6a366c94c3e0affe28c06c8df20a878f24da3cf
-        with:
-          name: build-plan
-          path: build-plan.json
-          if-no-files-found: error
-          retention-days: 30
 
-      - name: Write the plan result
-        if: always()
-        env:
-          SUMMARY_CHECK: Image plan
-          SUMMARY_RESULT: ${{ job.status }}
-          SUMMARY_FAILED_STEP: >-
-            ${{ steps.plan.conclusion == 'failure' && 'Build input detection' ||
-                steps.artifact.conclusion == 'failure' && 'Plan upload' ||
-                steps.source.conclusion == 'failure' && 'Source checkout' || '' }}
+      - name: Detect changed docker-built services
+        id: services
         shell: bash
         run: |
-          if [ -f .gitea/workflows/release.py ]; then
-            python3 .gitea/workflows/release.py check-summary
-          elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
-            printf '## Image plan\n\nResult: %s\n' "$SUMMARY_RESULT" >>"$GITHUB_STEP_SUMMARY" || true
+          set -euo pipefail
+          base="${{ github.event.before }}"
+          if [ -z "$base" ] || [ "$base" = "0000000000000000000000000000000000000000" ]; then
+            base="$(git rev-list --max-parents=0 HEAD)"
           fi
 
-  images:
-    name: Image (${{ matrix.name }})
-    needs: [image-plan]
-    if: needs.image-plan.result == 'success'
-    runs-on: homelab
-    timeout-minutes: 60
-    strategy:
-      max-parallel: 1
-      fail-fast: false
-      matrix: ${{ fromJSON(needs.image-plan.outputs.matrix || '{"include":[{"name":"inactive"}]}') }}
-    steps:
-      - name: Checkout repository
-        id: source
-        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
-      - name: Download the checked image plan
-        id: inputs
-        uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a
-        with:
-          name: build-plan
-      - name: Build or reuse this image
-        id: check
-        env:
-          IMAGE_NAME: ${{ matrix.name }}
-          REGISTRY_USERNAME: ${{ secrets.REGISTRY_USERNAME }}
-          REGISTRY_PASSWORD: ${{ secrets.REGISTRY_PASSWORD }}
-        run: python3 .gitea/workflows/release.py image --image "$IMAGE_NAME" --output image.json
-      - name: Store the image result
-        id: artifact
-        uses: actions/upload-artifact@c6a366c94c3e0affe28c06c8df20a878f24da3cf
-        with:
-          name: image-${{ matrix.name }}
-          path: image.json
-          if-no-files-found: error
-          retention-days: 30
-      - name: Write the job result
-        if: always()
-        env:
-          SUMMARY_CHECK: Image (${{ matrix.name }})
-          SUMMARY_RESULT: ${{ job.status }}
-          SUMMARY_FAILED_STEP: >-
-            ${{ steps.check.conclusion == 'failure' && 'Build or tag images' ||
-                steps.artifact.conclusion == 'failure' && 'Artifact upload' ||
-                steps.inputs.conclusion == 'failure' && 'Artifact download' ||
-                steps.source.conclusion == 'failure' && 'Source checkout' || '' }}
+          # A failed diff used to leave changed_files empty, which reads exactly
+          # like "nothing to build": the job went green having built nothing and
+          # the tag never moved. The status is checked, not assumed.
+          if ! changed="$(git diff --name-only "$base" "${GITHUB_SHA}")"; then
+            echo "::error::cannot diff ${base}..${GITHUB_SHA}"
+            exit 1
+          fi
+          mapfile -t changed_files <<<"$changed"
+
+          services=()
+
+          add_service() {
+            local name="$1"
+            local seen=0
+            for existing in "${services[@]}"; do
+              if [ "$existing" = "$name" ]; then
+                seen=1
+                break
+              fi
+            done
+            if [ "$seen" -eq 0 ]; then
+              services+=("$name")
+            fi
+          }
+
+          for file in "${changed_files[@]}"; do
+            case "$file" in
+              errorpages/*)
+                add_service errorpages
+                ;;
+              homepages/*)
+                add_service homepages
+                ;;
+              edu_master/phpsessid-bot/*|edu_master/webinar-checker/*|edu_master/compose.yaml)
+                add_service edu_master
+                ;;
+            esac
+          done
+
+          if [ "${#services[@]}" -eq 0 ]; then
+            echo "No docker-built services changed."
+            echo "services=" >> "$GITHUB_OUTPUT"
+            exit 0
+          fi
+
+          printf '%s\n' "${services[@]}" | tee /tmp/services.txt
+          echo "services=$(paste -sd, /tmp/services.txt)" >> "$GITHUB_OUTPUT"
+
+      - name: Log in to registry
+        # The pin step below also writes (manifest PUTs), and it runs on every
+        # main push — including manifest-only ones where services is empty. A
+        # stale persistent login on the old runner used to mask this; a clean
+        # runner pushes anonymously and gets 401.
+        if: steps.services.outputs.services != '' || github.ref_name == 'main'
         shell: bash
-        run: |
-          if [ -f .gitea/workflows/release.py ]; then
-            python3 .gitea/workflows/release.py check-summary
-          elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
-            printf '## %s\n\n- Result: **%s**\n- Failed step: %s\n' "$SUMMARY_CHECK" "$SUMMARY_RESULT" "$SUMMARY_FAILED_STEP" >>"$GITHUB_STEP_SUMMARY" || true
-          fi
-
-  # Retain the build job name required by the immutable release deployment gate.
-  build:
-    needs: [image-plan, images]
-    runs-on: homelab
-    timeout-minutes: 15
-    steps:
-      - name: Checkout repository
-        id: source
-        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
-      - name: Download all image results
-        id: inputs
-        uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a
-        with:
-          path: artifacts
-      - name: Pin SHA tags and write the complete release
-        id: check
+        # Through env, not by substitution into the script. A secret written
+        # into a run: block is pasted into the shell source before bash parses
+        # it, so a password containing a quote, a backtick or $(...) becomes
+        # code that runs. Masking the value in the log does not prevent that.
         env:
           REGISTRY_USERNAME: ${{ secrets.REGISTRY_USERNAME }}
           REGISTRY_PASSWORD: ${{ secrets.REGISTRY_PASSWORD }}
-        run: >-
-          python3 .gitea/workflows/release.py finalize
-          --plan artifacts/build-plan/build-plan.json
-      - name: Store commit release
-        id: artifact
-        uses: actions/upload-artifact@c6a366c94c3e0affe28c06c8df20a878f24da3cf
-        with:
-          name: release-${{ github.sha }}
-          path: release.json
-          if-no-files-found: error
-          retention-days: 30
-      - name: Write the job result
-        if: always()
-        env:
-          SUMMARY_CHECK: Image release and SHA tags
-          SUMMARY_RESULT: ${{ job.status }}
-          SUMMARY_FAILED_STEP: >-
-            ${{ steps.check.conclusion == 'failure' && 'Build or tag images' ||
-                steps.artifact.conclusion == 'failure' && 'Artifact upload' ||
-                steps.inputs.conclusion == 'failure' && 'Artifact download' ||
-                steps.source.conclusion == 'failure' && 'Source checkout' || '' }}
+        run: |
+          set -euo pipefail
+          printf '%s' "$REGISTRY_PASSWORD" | docker login "${REGISTRY}" \
+            -u "$REGISTRY_USERNAME" \
+            --password-stdin
+
+      - name: Build and push changed images
+        if: steps.services.outputs.services != ''
         shell: bash
         run: |
-          if [ -f .gitea/workflows/release.py ]; then
-            python3 .gitea/workflows/release.py check-summary
-          elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
-            printf '## %s\n\n- Result: **%s**\n- Failed step: %s\n' "$SUMMARY_CHECK" "$SUMMARY_RESULT" "$SUMMARY_FAILED_STEP" >>"$GITHUB_STEP_SUMMARY" || true
+          # This step was the one run: block in the workflow without it, and it
+          # is the one that cannot afford it: a docker push that failed partway
+          # through the loop used to be followed by more pushes, the loop's exit
+          # status came from the last one, and the job went green with half the
+          # images missing from the registry.
+          set -euo pipefail
+          IFS=, read -r -a services <<< "${{ steps.services.outputs.services }}"
+
+          # Tags for this push. The commit-pinned name is the point of this
+          # step: the deploy resolves it in preference to :prod, so a deploy
+          # that sat in the queue behind a later push still gets the build of
+          # the commit CI validated, instead of whatever :prod points at by the
+          # time it runs. See render_pinned in deploy-lib.sh.
+          commit_tag=""
+          if [ "${GITHUB_REF_NAME}" = "main" ]; then
+            commit_tag="sha-${GITHUB_SHA:0:12}"
           fi
+
+          set_tags() {
+            tags=()
+            case "${GITHUB_REF_NAME}" in
+              main) tags+=("main" "prod") ;;
+              dev) tags+=("dev") ;;
+            esac
+            if [ -n "$commit_tag" ]; then
+              tags+=("$commit_tag")
+            fi
+          }
+
+          for service in "${services[@]}"; do
+            case "$service" in
+              errorpages)
+                image="${REGISTRY}/forust/error-pages"
+                set_tags
+                build_args=()
+                for tag in "${tags[@]}"; do
+                  build_args+=(-t "${image}:${tag}")
+                done
+                docker build \
+                  --cache-from "type=registry,ref=${image}:buildcache" \
+                  --cache-to "type=registry,ref=${image}:buildcache,mode=max" \
+                  "${build_args[@]}" errorpages
+                for tag in "${tags[@]}"; do
+                  docker push "${image}:${tag}"
+                done
+                ;;
+              homepages)
+                for variant in forust xdfnx; do
+                  case "$variant" in
+                    forust)
+                      image="${REGISTRY}/forust/forust-homepage"
+                      ;;
+                    xdfnx)
+                      image="${REGISTRY}/forust/xdfnx-homepage"
+                      ;;
+                  esac
+                  set_tags
+                  build_args=()
+                  for tag in "${tags[@]}"; do
+                    build_args+=(-t "${image}:${tag}")
+                  done
+                  docker build \
+                    --cache-from "type=registry,ref=${image}:buildcache" \
+                    --cache-to "type=registry,ref=${image}:buildcache,mode=max" \
+                    "${build_args[@]}" -f "homepages/Dockerfile.${variant}" homepages
+                  for tag in "${tags[@]}"; do
+                    docker push "${image}:${tag}"
+                  done
+                done
+                ;;
+              edu_master)
+                for variant in session-keeper webinar-checker; do
+                  case "$variant" in
+                    session-keeper)
+                      context="edu_master/phpsessid-bot"
+                      image="${REGISTRY}/forust/session-keeper"
+                      ;;
+                    webinar-checker)
+                      context="edu_master/webinar-checker"
+                      image="${REGISTRY}/forust/webinar-checker"
+                      ;;
+                  esac
+                  set_tags
+                  build_args=()
+                  for tag in "${tags[@]}"; do
+                    build_args+=(-t "${image}:${tag}")
+                  done
+                  docker build \
+                    --cache-from "type=registry,ref=${image}:buildcache" \
+                    --cache-to "type=registry,ref=${image}:buildcache,mode=max" \
+                    "${build_args[@]}" "$context"
+                  for tag in "${tags[@]}"; do
+                    docker push "${image}:${tag}"
+                  done
+                done
+                ;;
+            esac
+          done
+
+      # Every image the tree names has to carry the commit-pinned name, not only
+      # the ones this push rebuilt. A push that touches nothing but manifests
+      # builds nothing, and its deploy would then find no commit-pinned tag to
+      # resolve and quietly fall back to the moving :prod - which is the whole
+      # failure the commit-pinned name exists to remove.
+      #
+      # Re-tagging copies the manifest list and transfers no layers, so pinning
+      # six images that already exist costs six registry writes.
+      #
+      # The list is derived from the tree rather than written out here, so an
+      # image added to a manifest is covered without a second place to update.
+      - name: Pin the commit name on the images this push did not rebuild
+        if: github.ref_name == 'main'
+        shell: bash
+        run: |
+          set -euo pipefail
+          commit_tag="sha-${GITHUB_SHA:0:12}"
+          mapfile -t repos < <(
+            git grep -hoE 'gcr\.forust\.xyz/forust/[A-Za-z0-9._-]+' -- '*.yaml' '*.yml' \
+              | sort -u
+          )
+          if [ "${#repos[@]}" -eq 0 ]; then
+            echo "No own images referenced by the tree."
+            exit 0
+          fi
+          echo "pinning ${#repos[@]} image(s) to $commit_tag"
+          for repo in "${repos[@]}"; do
+            if docker buildx imagetools inspect "$repo:$commit_tag" >/dev/null 2>&1; then
+              echo "  already built by this push: ${repo##*/}"
+              continue
+            fi
+            if ! docker buildx imagetools inspect "$repo:prod" >/dev/null 2>&1; then
+              echo "  WARNING: ${repo##*/} has no :prod to pin and no build produced it"
+              continue
+            fi
+            docker buildx imagetools create --tag "$repo:$commit_tag" "$repo:prod"
+            echo "  pinned ${repo##*/}"
+          done

.gitea/workflows/compose-lint.sh

@@ -21,7 +21,8 @@
 # All committed Compose files, including the ones deploy never starts.
 compose_files() {
   git ls-files \
-    '*compose.yaml' '*compose.yml'
+    '*/compose.yaml' '*/compose.yml' 'compose.yaml' 'compose.yml' \
+    '*/docker-compose.yaml' '*/docker-compose.yml'
 }
 
 # Prints the flags that turn `docker compose config` into the general check.

.gitea/workflows/compose-release.py

@@ -1,93 +0,0 @@
-#!/usr/bin/env python3
-"""Resolve Compose images without changing project names or local bind paths."""
-
-import json
-import os
-import re
-import subprocess
-import sys
-from pathlib import Path
-
-
-def output(*args, **kwargs):
-    return subprocess.check_output(args, text=True, **kwargs).strip()  # noqa: S603
-
-
-def resolve(reference):
-    if '@sha256:' in reference:
-        return reference
-    descriptor = json.loads(
-        output('docker', 'buildx', 'imagetools', 'inspect', reference, '--format', '{{json .Manifest}}')
-    )
-    digest = descriptor['digest']
-    if not re.fullmatch(r'sha256:[0-9a-f]{64}', digest):
-        raise ValueError(f'Invalid registry digest for {reference}')
-    # Strip tag only from the final path segment (registry ports are preserved).
-    repository = reference.rsplit('/', 1)
-    repository[-1] = repository[-1].split(':')[0]
-    return '/'.join(repository) + '@' + digest
-
-
-def prepare(source_file):
-    config_repo = Path(os.environ['CONFIG_REPO'])
-    source_repo = Path(os.environ['REPO'])
-    directory = Path(os.environ['RUN_DIR'])
-    relative = source_file.relative_to(source_repo)
-    project_dir = config_repo / relative.parent
-    base = ['docker', 'compose', '--project-directory', str(project_dir), '-f', str(source_file)]
-    config = json.loads(output(*base, 'config', '--format', 'json', cwd=config_repo))
-    project = config['name']
-    previous_file = directory / 'previous.json'
-    previous = json.loads(previous_file.read_text()) if previous_file.exists() else {}
-    images_file = directory / 'compose-images.json'
-    locks = json.loads(images_file.read_text()) if images_file.exists() else previous.get('compose-images', {})
-    release = json.loads((directory / 'release.json').read_text())
-    before = json.loads(json.dumps(config))
-    for service, settings in config['services'].items():
-        reference = settings.get('image')
-        if not reference or settings.get('build'):
-            raise ValueError(f'{project}/{service}: Compose deploy requires a published image')
-        image_repo = reference.split('@')[0].rsplit('/', 1)
-        image_repo[-1] = image_repo[-1].split(':')[0]
-        image_repo = '/'.join(image_repo)
-        if image_repo in release['images']:
-            pinned = image_repo + '@' + release['images'][image_repo]
-        elif os.environ.get('REFRESH_IMAGES') != 'true' and reference in locks:
-            pinned = locks[reference]
-        else:
-            pinned = resolve(reference)
-        settings['image'] = pinned
-        locks[reference] = pinned
-        # Capture what is running, not the current value of its mutable tag.
-        ids = output(
-            'docker',
-            'ps',
-            '-aq',
-            '--filter',
-            f'label=com.docker.compose.project={project}',
-            '--filter',
-            f'label=com.docker.compose.service={service}',
-        ).splitlines()
-        actual = set()
-        for container in ids:
-            image_id = output('docker', 'inspect', container, '--format', '{{.Image}}')
-            digests = json.loads(output('docker', 'image', 'inspect', image_id, '--format', '{{json .RepoDigests}}'))
-            actual.add(next((d for d in digests or [] if d.split('@')[0] == image_repo), image_id))
-        if len(actual) > 1:
-            raise ValueError(f'{project}/{service}: mixed running images, cannot capture one recovery config')
-        before['services'][service]['image'] = next(iter(actual)) if actual else reference
-    for name, data in (('compose', config), ('compose-before', before)):
-        folder = directory / name
-        folder.mkdir(mode=0o700, exist_ok=True)
-        destination = folder / f'{relative.parent.name}.json'
-        destination.write_text(json.dumps(data, indent=2) + '\n')
-        destination.chmod(0o600)
-    images_file.write_text(json.dumps(locks, indent=2) + '\n')
-    print(f'Compose {project}: images pinned; local paths preserved')
-    print(
-        f'Recovery: docker compose --project-directory {project_dir} -p {project} -f {directory}/compose-before/{relative.parent.name}.json up -d --pull never'
-    )
-
-
-if __name__ == '__main__':
-    prepare(Path(sys.argv[1]))

.gitea/workflows/deploy-controller.py

@@ -1,422 +0,0 @@
-#!/usr/bin/env python3
-"""Durable workstation deployment controller. Install with setup-workstation.sh."""
-
-import argparse
-import contextlib
-import fcntl
-import importlib.util
-import json
-import math
-import os
-import re
-import shutil
-import subprocess
-import sys
-import time
-from pathlib import Path
-
-STATE = Path(os.environ.get('HOMELAB_STATE', Path.home() / '.local/state/homelab-deploy'))
-CONFIG_REPO = Path(os.environ.get('HOMELAB_REPO', '/srv/homelab'))
-RUN_ID = re.compile(r'[0-9]+-[0-9]+')
-
-
-def command(*args, **kwargs):
-    return subprocess.check_output(args, text=True, **kwargs).strip()  # noqa: S603, S607
-
-
-def atomic_json(path, data):
-    temporary = path.with_suffix('.tmp')
-    temporary.write_text(json.dumps(data, indent=2) + '\n')
-    temporary.chmod(0o600)
-    temporary.replace(path)
-
-
-@contextlib.contextmanager
-def lock(name):
-    STATE.mkdir(mode=0o700, parents=True, exist_ok=True)
-    with (STATE / name).open('a') as stream:
-        fcntl.flock(stream, fcntl.LOCK_EX)
-        yield
-
-
-def load_module(name, path):
-    spec = importlib.util.spec_from_file_location(name, path)
-    module = importlib.util.module_from_spec(spec)
-    spec.loader.exec_module(module)
-    return module
-
-
-def run_directory(run_id):
-    if not RUN_ID.fullmatch(run_id):
-        raise ValueError('Run ID must be numeric workflow-id and attempt')
-    return STATE / 'runs' / run_id
-
-
-def start(run_id):
-    payload = sys.stdin.buffer.read(256 * 1024 + 1)
-    if len(payload) > 256 * 1024:
-        raise ValueError('Deploy request exceeds 256 KiB')
-    request = json.loads(payload)
-    sha = request['release']['sha']
-    if not re.fullmatch(r'[0-9a-f]{40}', sha) or request['mode'] not in ('changed', 'full', 'plan'):
-        raise ValueError('Invalid deploy SHA or mode')
-    if not isinstance(request['refresh_images'], bool):
-        raise ValueError('refresh_images must be boolean')
-    directory = run_directory(run_id)
-    with lock('prepare.lock'):
-        if (directory / 'request.json').exists():
-            if json.loads((directory / 'request.json').read_text()) != request:
-                raise ValueError('Run ID already belongs to a different request')
-        else:
-            directory.mkdir(mode=0o700, parents=True, exist_ok=True)
-            command('git', '-C', str(CONFIG_REPO), 'fetch', '--quiet', 'origin', 'main')
-            command('git', '-C', str(CONFIG_REPO), 'merge-base', '--is-ancestor', sha, 'origin/main')
-            if not (directory / 'source').exists():
-                command('git', '-C', str(CONFIG_REPO), 'worktree', 'add', '--detach', str(directory / 'source'), sha)
-            if command('git', '-C', str(directory / 'source'), 'rev-parse', 'HEAD') != sha:
-                raise ValueError('Prepared source does not match deploy SHA')
-            release_module = load_module('release', directory / 'source/.gitea/workflows/release.py')
-            release_module.validate_release(request['release'], sha)
-            atomic_json(directory / 'release.json', request['release'])
-            atomic_json(directory / 'request.json', request)
-        if not (directory / 'status.json').exists():
-            atomic_json(directory / 'status.json', {'state': 'queued', 'stages': {}})
-    # Starting an existing active or finished ID is idempotent; never re-apply it.
-    if json.loads((directory / 'status.json').read_text())['state'] == 'queued':
-        command('systemctl', '--user', 'start', '--no-block', f'homelab-deploy@{run_id}.service')
-    print(f'Accepted deploy {run_id} ({sha})')
-
-
-def environment(directory):
-    request = json.loads((directory / 'request.json').read_text())
-    return {
-        **os.environ,
-        'REPO': str(directory / 'source'),
-        'CONFIG_REPO': str(CONFIG_REPO),
-        'RUN_DIR': str(directory),
-        'DEPLOY_SHA': request['release']['sha'],
-        'RELEASE_FILE': str(directory / 'release.json'),
-        'DEPLOY_PLAN': str(directory / 'plan.json'),
-        'DEPLOY_SNAPSHOT_DIR': str(directory / 'snapshot'),
-        'REFRESH_IMAGES': str(request['refresh_images']).lower(),
-        'ROLLOUT_PARALLELISM': '4',
-    }
-
-
-def stage(directory, name, budget):
-    status = json.loads((directory / 'status.json').read_text())
-    if name in status['stages'] and status['stages'][name].get('result') in ('success', 'failure'):
-        return status['stages'][name]['result'] == 'success'
-    started = time.time()
-    status['stages'][name] = {'result': 'running', 'started': started}
-    atomic_json(directory / 'status.json', status)
-    script = directory / 'source/.gitea/workflows/deploy-stage.sh'
-    with (directory / f'{name}.log').open('a') as log:
-        # timeout kills the whole stage process group, including children, before recovery.
-        result = subprocess.run(  # noqa: S603, S607
-            [
-                shutil.which('timeout') or '/usr/bin/timeout',
-                '--signal=TERM',
-                '--kill-after=30s',
-                str(budget),
-                'bash',
-                str(script),
-                name,
-            ],
-            env=environment(directory),
-            stdout=log,
-            stderr=subprocess.STDOUT,
-            check=False,
-        ).returncode
-    status = json.loads((directory / 'status.json').read_text())
-    status['stages'][name].update(
-        result='success' if result == 0 else 'failure', exit_code=result, seconds=round(time.time() - started)
-    )
-    atomic_json(directory / 'status.json', status)
-    return result == 0
-
-
-def make_plan(directory):
-    source = directory / 'source'
-    planner = load_module('deploy_plan', source / '.gitea/workflows/deploy-plan.py')
-    request = json.loads((directory / 'request.json').read_text())
-    previous = json.loads((STATE / 'last-success.json').read_text()) if (STATE / 'last-success.json').exists() else None
-    helm = json.loads(command('helm', 'list', '--all', '-A', '-o', 'json'))
-    plan = planner.make_plan(source, CONFIG_REPO, request['release'], previous, request['mode'], helm)
-    if request['refresh_images']:
-        plan['selected']['compose'] = plan['active']['compose']
-    atomic_json(directory / 'plan.json', plan)
-    if previous:
-        atomic_json(directory / 'previous.json', previous)
-    # Local config is deliberately separate from the immutable Git source.
-    return plan
-
-
-def finish_success(directory, plan):
-    # Repeating finalization after a crash is safe while holding deploy.lock.
-    plan['run_id'] = directory.name
-    path = directory / 'compose-images.json'
-    previous = directory / 'previous.json'
-    plan['compose-images'] = (
-        json.loads(path.read_text())
-        if path.exists()
-        else json.loads(previous.read_text()).get('compose-images', {})
-        if previous.exists()
-        else {}
-    )
-    atomic_json(STATE / 'last-success.json', plan)
-    status = json.loads((directory / 'status.json').read_text())
-    status['state'] = 'success'
-    atomic_json(directory / 'status.json', status)
-    try:
-        retain_completed(directory)
-    except (OSError, subprocess.CalledProcessError) as error:
-        print(f'Retention deferred: {error}', flush=True)
-
-
-def recover(directory, retry=False):
-    status = json.loads((directory / 'status.json').read_text())
-    if status['state'] in ('success', 'planned'):
-        return
-    completed = ('doctor', 'validate', 'apply-k8s', 'apply-compose', 'verify-k8s', 'smoke')
-    if all(status['stages'].get(name, {}).get('result') == 'success' for name in completed):
-        finish_success(directory, json.loads((directory / 'plan.json').read_text()))
-        return
-    if retry:
-        for name in ('verify-k8s', 'smoke'):
-            if status['stages'].get(name, {}).get('result') == 'failure':
-                del status['stages'][name]
-        atomic_json(directory / 'status.json', status)
-    snapshot = directory / 'snapshot/current'
-    if snapshot.exists():
-        stage(directory, 'verify-k8s', 7200)
-        stage(directory, 'smoke', 600)
-    status = json.loads((directory / 'status.json').read_text())
-    status['state'] = 'failure'
-    atomic_json(directory / 'status.json', status)
-
-
-def execute(run_id):
-    directory = run_directory(run_id)
-    with lock('deploy.lock'):
-        status = json.loads((directory / 'status.json').read_text())
-        if status['state'] != 'queued':
-            return
-        # A crashed predecessor must be recovered before another apply begins.
-        for other in (STATE / 'runs').iterdir():
-            if (
-                other != directory
-                and (other / 'status.json').exists()
-                and json.loads((other / 'status.json').read_text())['state'] == 'running'
-            ):
-                raise ValueError(f'Interrupted deploy {other.name}; run recover first')
-        status['state'] = 'running'
-        atomic_json(directory / 'status.json', status)
-        phase = 'plan'
-        try:
-            plan = make_plan(directory)
-            print(
-                json.dumps({'selected': plan['selected'], 'helm': plan['helm'], 'manual_removals': plan['removed']}),
-                flush=True,
-            )
-            phase = 'doctor'
-            if not stage(directory, 'doctor', 600):
-                raise RuntimeError('Preflight failed')
-            phase = 'validate'
-            if not stage(directory, 'validate', 1200):
-                raise RuntimeError('Validation failed')
-            if json.loads((directory / 'request.json').read_text())['mode'] == 'plan':
-                status = json.loads((directory / 'status.json').read_text())
-                status['state'] = 'planned'
-                atomic_json(directory / 'status.json', status)
-                return
-            # Budget includes both rollout checks and rollback waves, plus API overhead.
-            phase = 'Recovery budget'
-            count = int(
-                command(
-                    'bash',
-                    str(directory / 'source/.gitea/workflows/deploy-stage.sh'),
-                    'workload-count',
-                    env=environment(directory),
-                )
-            )
-            verify_budget = max(600, 2 * math.ceil(count / 4) * 300 + 120)
-            if verify_budget > 7200:
-                raise ValueError('More than two hours of recovery required; split this deploy')
-            phase = 'apply-k8s'
-            k8s_ok = stage(directory, 'apply-k8s', 2700)
-            phase = 'apply-compose'
-            compose_ok = stage(directory, 'apply-compose', 1800) if k8s_ok else False
-            phase = 'verify-k8s'
-            verify_ok = stage(directory, 'verify-k8s', verify_budget)
-            phase = 'smoke'
-            smoke_ok = stage(directory, 'smoke', 600)
-            if not all((k8s_ok, compose_ok, verify_ok, smoke_ok)):
-                raise RuntimeError('Deploy failed; inspect stage logs and recovery report')
-            phase = 'Save the successful baseline'
-            finish_success(directory, plan)
-        except Exception as error:
-            status = json.loads((directory / 'status.json').read_text())
-            status['failure_stage'] = next(
-                (name for name, result in status['stages'].items() if result.get('result') == 'failure'), phase
-            )
-            atomic_json(directory / 'status.json', status)
-            with (directory / 'controller.log').open('a') as stream:
-                stream.write(f'{error}\n')
-            recover(directory)
-            raise
-
-
-def retain_completed(current):
-    finished = []
-    for directory in (STATE / 'runs').iterdir():
-        status_file = directory / 'status.json'
-        if status_file.exists() and json.loads(status_file.read_text())['state'] in ('success', 'planned'):
-            finished.append(directory)
-    for directory in sorted(finished, key=lambda p: p.stat().st_mtime, reverse=True)[20:]:
-        if directory == current:
-            continue
-        command('git', '-C', str(CONFIG_REPO), 'worktree', 'remove', '--force', str(directory / 'source'))
-        shutil.rmtree(directory)
-
-
-def follow(run_id, phase):
-    directory = run_directory(run_id)
-    groups = {
-        'apply': ('doctor', 'validate', 'apply-k8s', 'apply-compose'),
-        'verify': ('verify-k8s',),
-        'smoke': ('smoke',),
-    }
-    names = groups[phase]
-    offsets = {}
-    while True:
-        status = json.loads((directory / 'status.json').read_text())
-        for name in (*names, 'controller'):
-            path = directory / f'{name}.log'
-            if path.exists():
-                with path.open() as stream:
-                    stream.seek(offsets.get(name, 0))
-                    content = stream.read()
-                    if content:
-                        print(content, end='', flush=True)
-                    offsets[name] = stream.tell()
-        stages = status['stages']
-        if all(stages.get(name, {}).get('result') in ('success', 'failure') for name in names):
-            return all(stages[name]['result'] == 'success' for name in names)
-        if status['state'] in ('success', 'failure', 'planned'):
-            return status['state'] in ('success', 'planned')
-        time.sleep(3)
-
-
-def summary(run_id):
-    directory = run_directory(run_id)
-    request = json.loads((directory / 'request.json').read_text())
-    release = request['release']
-    plan_file = directory / 'plan.json'
-    lines = [
-        f'## Deploy `{release["sha"]}`',
-        '',
-        f'- Mode: `{request["mode"]}`',
-        f'- Refresh third-party images: `{request["refresh_images"]}`',
-    ]
-    status = json.loads((directory / 'status.json').read_text())
-    if status.get('failure_stage'):
-        lines.append(f'- Failed stage: **{status["failure_stage"]}**')
-    lines.extend(
-        [
-            '',
-            f'- Observed run state: **{status["state"]}**',
-            '',
-            '### Stage results',
-            '| Stage | Result | Exit code |',
-            '| --- | --- | --- |',
-        ]
-    )
-    for name in ('doctor', 'validate', 'apply-k8s', 'apply-compose', 'verify-k8s', 'smoke'):
-        stage_result = status['stages'].get(name, {})
-        lines.append(f'| {name} | {stage_result.get("result", "not started")} | {stage_result.get("exit_code", "—")} |')
-    lines.extend(['', '### Apply and Helm recovery results'])
-    events_file = directory / 'apply-events.jsonl'
-    events = []
-    if events_file.exists():
-        for line in events_file.read_text().splitlines():
-            try:
-                events.append(json.loads(line))
-            except json.JSONDecodeError:
-                lines.append('- An operation record is incomplete. Check the stage log.')
-    latest = {(event['action'], event['target']): event['result'] for event in events}
-    lines.extend(f'- `{action}` `{target}`: **{result}**' for (action, target), result in latest.items())
-    if not latest:
-        lines.append('- No apply results were recorded.')
-    lines.append('- A completed apply does not confirm health. See verification and smoke results.')
-    lines.extend(['', '### Kubernetes recovery'])
-    pointer = directory / 'snapshot/current'
-    failed = Path(pointer.read_text().strip()) / 'failed-workloads' if pointer.exists() else None
-    if failed and failed.exists():
-        contents = failed.read_text()
-        counts = dict(re.findall(r'^(ROLLED_BACK|UNRECOVERED)=([0-9]+)$', contents, re.MULTILINE))
-        if not contents.strip():
-            lines.append('- No failed workloads were recorded. See the verification result above.')
-        elif counts:
-            lines.append(f'- Workloads restored: **{counts.get("ROLLED_BACK", "unknown")}**')
-            lines.append(f'- Workloads that need manual recovery: **{counts.get("UNRECOVERED", "unknown")}**')
-        else:
-            lines.append('- Rollback has no recorded result yet. Check the verification log.')
-    else:
-        lines.append('- No workload rollback was recorded. This does not confirm health.')
-    lines.append('- Compose requires manual recovery. Use the saved command in the apply log.')
-    if not plan_file.exists():
-        lines.extend(['', 'Plan was not created. Check the controller log.'])
-        print('\n'.join(lines))
-        return
-    plan = json.loads(plan_file.read_text())
-    lines.extend(['', '### Selected services'])
-    count = 0
-    for kind, services in plan['selected'].items():
-        for service in services:
-            lines.append(f'- `{kind}`: `{service}`')
-            count += 1
-    if not count:
-        lines.append('- None')
-    lines.extend(['', '### Selected Helm releases'])
-    lines.extend(f'- `{release}`' for release in plan.get('helm', []))
-    if not plan.get('helm'):
-        lines.append('- None')
-    lines.extend(['', '### Images pinned in the checked release'])
-    lines.extend(f'- `{image}@{digest}`' for image, digest in sorted(release['images'].items()))
-    lines.extend(['', '### Removed resources requiring manual review'])
-    lines.extend(f'- `{item}`' for item in plan.get('removed', []))
-    if not plan.get('removed'):
-        lines.append('- None')
-    print('\n'.join(lines))
-
-
-def main():
-    os.umask(0o077)
-    parser = argparse.ArgumentParser(description=__doc__)
-    parser.add_argument('action', choices=('start', 'execute', 'recover', 'status', 'follow', 'summary'))
-    parser.add_argument('run_id')
-    parser.add_argument('phase', nargs='?', choices=('apply', 'verify', 'smoke'))
-    parser.add_argument('--retry', action='store_true', help='Retry failed recovery checks; never repeat apply')
-    args = parser.parse_args()
-    directory = run_directory(args.run_id)
-    if args.action == 'start':
-        start(args.run_id)
-    elif args.action == 'execute':
-        execute(args.run_id)
-    elif args.action == 'recover':
-        with lock('deploy.lock'):
-            recover(directory, retry=args.retry)
-    elif args.action == 'status':
-        print((directory / 'status.json').read_text())
-        if (directory / 'plan.json').exists():
-            plan = json.loads((directory / 'plan.json').read_text())
-            print(json.dumps({k: plan[k] for k in ('sha', 'selected', 'helm', 'removed')}, indent=2))
-    elif args.action == 'summary':
-        summary(args.run_id)
-    elif not follow(args.run_id, args.phase):
-        sys.exit(1)
-
-
-if __name__ == '__main__':
-    main()

.gitea/workflows/deploy-plan.py

@@ -1,124 +0,0 @@
-#!/usr/bin/env python3
-"""Calculate selected components against the last fully successful deploy."""
-
-import hashlib
-import json
-import re
-import subprocess
-from pathlib import Path
-
-
-def output(*args, **kwargs):
-    return subprocess.check_output(args, text=True, **kwargs).strip()  # noqa: S603
-
-
-def tracked(repo):
-    return output('git', '-C', str(repo), 'ls-files').splitlines()
-
-
-def helm_releases(repo):
-    text = (repo / '.gitea/workflows/deploy-lib.sh').read_text()
-    return [line.split('|') for line in re.findall(r'^  "([^"\n]+\|[^"\n]+)"$', text, re.MULTILINE)]
-
-
-def inventory(repo):
-    files = tracked(repo)
-    k8s = sorted(
-        {f.split('/k8s/')[0] for f in files if '/k8s/' in f and (repo / f.split('/k8s/')[0] / 'k8s/active').is_file()}
-    )
-    compose = sorted(
-        {
-            str(Path(f).parent)
-            for f in files
-            if Path(f).name in ('compose.yaml', 'compose.yml') and (repo / Path(f).parent / 'active').is_file()
-        }
-    )
-    return {'k8s': k8s, 'compose': compose}
-
-
-def file_hash(path):
-    return hashlib.sha256(path.read_bytes()).hexdigest() if path.is_file() else 'missing'
-
-
-def make_plan(repo, config_repo, release, previous, mode, live_helm):
-    active = inventory(repo)
-    all_services = set(active['k8s'] + active['compose'])
-    helm_inputs = {}
-    helm_selected = []
-    for name, chart, namespace, version, values, marker in helm_releases(repo):
-        if not (repo / marker).is_file():
-            continue
-        value_path = repo / values if (repo / values).is_file() else config_repo / values
-        if not value_path.is_file():
-            raise ValueError(f'Missing Helm values: {values}')
-        stamp = hashlib.sha256(f'{chart}|{version}|{file_hash(value_path)}'.encode()).hexdigest()
-        helm_inputs[name] = stamp
-        live = next((h for h in live_helm if h['name'] == name and h['namespace'] == namespace), None)
-        if (
-            mode == 'full'
-            or previous is None
-            or previous.get('helm_inputs', {}).get(name) != stamp
-            or live is None
-            or live.get('status') != 'deployed'
-            or live.get('chart') != f'{chart.split("/")[-1]}-{version}'
-        ):
-            helm_selected.append(name)
-    local_inputs = {}
-    for service in all_services:
-        candidates = [config_repo / service / '.env']
-        if service in active['compose']:
-            candidates.append(config_repo / '.env')
-            cfg = config_repo / service / 'config'
-            if cfg.is_dir():
-                candidates.extend(
-                    p for p in cfg.rglob('*') if p.is_file() and p.suffix in ('.yaml', '.yml', '.json', '.conf')
-                )
-        local_inputs[service] = hashlib.sha256(
-            '\n'.join(f'{p.relative_to(config_repo)}:{file_hash(p)}' for p in sorted(candidates)).encode()
-        ).hexdigest()
-    if previous is None:
-        if mode == 'changed':
-            raise ValueError('No successful baseline; run deploy in full mode first')
-        changed = set(all_services)
-        removed = []
-    else:
-        paths = output('git', '-C', str(repo), 'diff', '--name-only', previous['sha'], release['sha']).splitlines()
-        changed = {path.split('/')[0] for path in paths}
-        if any(path.startswith('.gitea/') for path in paths):
-            changed |= all_services
-        changed |= {s for s in all_services if previous.get('local_inputs', {}).get(s) != local_inputs[s]}
-        for file in tracked(repo):
-            service = file.split('/')[0]
-            if service not in all_services or not file.endswith(('.yaml', '.yml')):
-                continue
-            text = (repo / file).read_text()
-            if any(
-                image in text and previous.get('images', {}).get(image) != digest
-                for image, digest in release['images'].items()
-            ):
-                changed.add(service)
-        removed = sorted(
-            set(previous.get('active', {}).get('k8s', []) + previous.get('active', {}).get('compose', []))
-            - all_services
-        )
-        removed += [path for path in paths if '/k8s/' in path and not (repo / path).exists()]
-    if mode == 'full':
-        changed = set(all_services)
-    dependencies = json.loads((repo / '.gitea/deploy-dependencies.json').read_text())
-    while True:
-        expanded = changed | {dependent for service in changed for dependent in dependencies.get(service, [])}
-        if expanded == changed:
-            break
-        changed = expanded
-    return {
-        'version': 1,
-        'sha': release['sha'],
-        'images': release['images'],
-        'active': active,
-        'selected': {kind: sorted(set(services) & changed) for kind, services in active.items()},
-        'helm': helm_selected,
-        'helm_inputs': helm_inputs,
-        'local_inputs': local_inputs,
-        'removed': sorted(set(removed)),
-        'full_smoke': mode == 'full' or 'traefik' in changed,
-    }

.gitea/workflows/deploy-stage.sh

@@ -1,10 +0,0 @@
-#!/usr/bin/env bash
-set -euo pipefail
-source "${REPO:?}/.gitea/workflows/deploy-lib.sh"
-case "${1:?stage required}" in
-  workload-count)
-    select_manifests >/dev/null
-    selected_workload_refs | sort -u | wc -l
-    ;;
-  *) run_stage "$1" ;;
-esac

.gitea/workflows/deploy.yaml

@@ -1,140 +1,197 @@
 name: deploy
 
 on:
+  # Deploy only what CI already validated. workflow_run is used instead of
+  # workflow_dispatch so a red lint/validate run can never reach the cluster.
   workflow_run:
     workflows: [ci]
-    branches: [main]
     types: [completed]
   workflow_dispatch:
-    inputs:
-      deploy_ref:
-        description: "Commit already checked by successful main CI (main or SHA)"
-        default: main
-        required: true
-      deploy_mode:
-        description: "First deploy requires full; plan changes no production resources"
-        type: choice
-        options: [changed, full, plan]
-        default: changed
-      refresh_images:
-        description: "Explicitly refresh mutable third-party Compose tags"
-        type: boolean
-        default: false
 
+# The deploy jobs read the tree, then reach the cluster over SSH with the
+# deploy key. The Actions token itself is not part of that path, so it gets
+# read-only contents and no more.
 permissions:
   contents: read
-  actions: read
 
 concurrency:
   group: deploy-main
+  # Queue instead of cancelling. Cancelling a run kills the apply job mid-loop and
+  # takes the verify job down with it, so a superseded deploy would leave the
+  # cluster half-applied and unchecked — the exact failure the verify job exists
+  # to catch. kubectl apply and docker compose up are both idempotent, so letting
+  # the older run finish and then deploying the newer commit costs little.
   cancel-in-progress: false
 
 env:
-  DEPLOY_HOST: ${{ vars.DEPLOY_HOST || secrets.DEPLOY_HOST }}
-  DEPLOY_PORT: ${{ vars.DEPLOY_PORT || secrets.DEPLOY_PORT }}
-  DEPLOY_USER: ${{ vars.DEPLOY_USER || secrets.DEPLOY_USER }}
+  DEPLOY_HOST: ${{ secrets.DEPLOY_HOST }}
+  DEPLOY_PORT: ${{ secrets.DEPLOY_PORT }}
+  DEPLOY_USER: ${{ secrets.DEPLOY_USER }}
+  DEPLOY_PATH: ${{ secrets.DEPLOY_PATH }}
   DEPLOY_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
-  DEPLOY_KNOWN_HOSTS: ${{ vars.DEPLOY_KNOWN_HOSTS }}
-  DEPLOY_RUN_ID: ${{ github.run_id }}-${{ github.run_attempt || 1 }}
-  DEPLOY_MODE: ${{ inputs.deploy_mode || 'changed' }}
-  REFRESH_IMAGES: ${{ inputs.refresh_images && 'true' || 'false' }}
+  APPLY_PRUNE: ${{ vars.APPLY_PRUNE }}
+  # workflow_run's own GITHUB_SHA points at the branch head, not at the commit the
+  # finished ci run checked. Pin the exact validated commit instead, so a push
+  # landing mid-deploy cannot make the workstation deploy something else. Also
+  # what the verify job checks the snapshot against. Empty for workflow_dispatch,
+  # which falls back to the current origin/main.
+  DEPLOY_SHA: ${{ github.event.workflow_run.head_sha }}
 
 jobs:
-  gate:
+  preflight:
+    # Autodeploy defaults to OFF: pushes deploy only when the AUTODEPLOY repo
+    # variable is set to 'true' (Settings -> Actions -> Variables). A manual
+    # Run workflow always bypasses the switch: dispatching it is the explicit
+    # intent to deploy.
     if: >-
-      github.ref == 'refs/heads/main' &&
       (vars.AUTODEPLOY == 'true' || github.event_name == 'workflow_dispatch') &&
       (github.event_name != 'workflow_run' ||
-       (github.event.workflow_run.conclusion == 'success' && github.event.workflow_run.head_branch == 'main'))
-    runs-on: homelab
+      (github.event.workflow_run.conclusion == 'success' &&
+       github.event.workflow_run.head_branch == 'main'))
+    runs-on: [self-hosted, linux, arch, homelab, prod]
     timeout-minutes: 10
-    outputs:
-      sha: ${{ steps.release.outputs.sha }}
     steps:
       - name: Checkout repository
         uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
-        with:
-          fetch-depth: 0
-      - name: Check successful CI and download the exact commit release
-        id: release
-        env:
-          GITEA_TOKEN: ${{ github.token }}
-          DEPLOY_REF: ${{ inputs.deploy_ref || 'main' }}
-          EVENT_SHA: ${{ github.event.workflow_run.head_sha }}
-        run: python3 .gitea/workflows/release.py gate --ref "$DEPLOY_REF" --event-sha "$EVENT_SHA"
-      - name: Submit durable deploy to workstation
-        run: bash .gitea/workflows/ssh-run.sh start
-      - name: Write the request result
-        if: always()
-        env:
-          REQUEST_RESULT: ${{ job.status }}
-          CHECKED_SHA: ${{ steps.release.outputs.sha }}
-        run: |
-          if [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
-            printf '## Deploy request\n\n- Result: **%s**\n- Checked commit: %s\n- Mode: %s\n' "$REQUEST_RESULT" "${CHECKED_SHA:-not checked}" "$DEPLOY_MODE" >>"$GITHUB_STEP_SUMMARY"
-            if [ "$REQUEST_RESULT" != success ]; then
-              echo 'Open the failed step log. If SSH submission failed, check the remote controller state.' >>"$GITHUB_STEP_SUMMARY"
-            fi
-          fi
 
-  apply:
-    needs: [gate]
-    runs-on: homelab
-    timeout-minutes: 100
+      - name: Fetch and reset workstation
+        shell: bash
+        run: |
+          set -euo pipefail
+          ./.gitea/workflows/ssh-run.sh preflight
+
+  validate:
+    needs: [preflight]
+    runs-on: [self-hosted, linux, arch, homelab, prod]
+    timeout-minutes: 20
     steps:
-      - name: Checkout checked commit
+      - name: Checkout repository
         uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
-        with:
-          ref: ${{ needs.gate.outputs.sha }}
-      - name: Follow validation and sequential Kubernetes / Compose apply
-        run: bash .gitea/workflows/ssh-run.sh apply
-      - name: Write the deploy result
-        if: always()
-        run: |
-          if [ -f .gitea/workflows/ssh-run.sh ]; then
-            bash .gitea/workflows/ssh-run.sh summary
-          elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
-            echo 'Source checkout failed. The remote deploy state is unknown. Check the job log.' >>"$GITHUB_STEP_SUMMARY"
-          fi
 
-  verify:
-    needs: [gate, apply]
-    if: always() && needs.gate.result == 'success'
-    runs-on: homelab
-    timeout-minutes: 130
+      - name: Dry-run manifests and check Secrets
+        shell: bash
+        run: |
+          set -euo pipefail
+          ./.gitea/workflows/ssh-run.sh validate
+
+  apply-k8s:
+    needs: [validate]
+    runs-on: [self-hosted, linux, arch, homelab, prod]
+    # Apply only, no verification, so this is just the work itself: snapshot,
+    # then sequential `helm upgrade --install --wait --rollback-on-failure --timeout 10m`, then the apply loop.
+    # Verification has its own job and its own budget.
+    #
+    # 45 is roughly four times the measured cost of the stage, which is
+    # deliberately not raised on a theory:
+    #
+    #   helm, healthy          3 no-op upgrades              ~3-5 min
+    #   helm, one release bad  rollback-on-failure spends its 10m,      ~10-15 min
+    #                            then rolls that one back
+    #   apply loop             ~40 manifests, 4 of which     ~1 min
+    #                           resolve an image digest
+    #   restart_stale_images   7.6s to find 8 workloads,     ~0.5 min
+    #                           9.8s to resolve their digests
+    #
+    # The helm figure is one release, not three: `set -e` aborts
+    # upgrade_helm_releases on the first failure, so a broken release costs
+    # 10m and the other two are never attempted. Multiplying 10m by three
+    # overstates the worst case by 20 minutes.
+    #
+    # The 45 minutes this was last raised to 45 were still not enough, and the
+    # job logs for those runs no longer exist, so what actually consumed the
+    # budget is not known - the two measurable candidates above account for
+    # ~15 of it. The unbounded `docker manifest inspect` against the registry's
+    # known hang mode is now bounded inside registry_digest (25s timeout, 3
+    # attempts): a dead registry fails each owned image after ~85s instead of
+    # hanging the stage, and a blinking one is retried instead of failing the
+    # whole apply file. Still open: make the stage announce which manifest it
+    # is working on, so a killed run leaves a diagnosable last line.
+    timeout-minutes: 45
     steps:
-      - name: Checkout checked commit
+      - name: Checkout repository
         uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
-        with:
-          ref: ${{ needs.gate.outputs.sha }}
-      - name: Follow workload verification and recovery
-        run: bash .gitea/workflows/ssh-run.sh verify
-      - name: Write the deploy result
-        if: always()
-        run: |
-          if [ -f .gitea/workflows/ssh-run.sh ]; then
-            bash .gitea/workflows/ssh-run.sh summary
-          elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
-            echo 'Source checkout failed. The remote deploy state is unknown. Check the job log.' >>"$GITHUB_STEP_SUMMARY"
-          fi
 
+      - name: Apply Kubernetes manifests
+        shell: bash
+        run: |
+          set -euo pipefail
+          ./.gitea/workflows/ssh-run.sh apply-k8s
+
+  apply-compose:
+    needs: [validate]
+    runs-on: [self-hosted, linux, arch, homelab, prod]
+    timeout-minutes: 30
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+
+      - name: Redeploy docker compose stacks
+        shell: bash
+        run: |
+          set -euo pipefail
+          ./.gitea/workflows/ssh-run.sh apply-compose
+
+  # Watches the workloads this deploy changed and rolls back the ones that never
+  # became healthy. Runs even when the apply jobs failed, timed out or were
+  # cancelled — that is the whole point of splitting it out. `always()` is what
+  # lets it start after a failed dependency; the needs on apply-compose are a
+  # barrier, so verification begins only once both applies are done.
+  verify-k8s:
+    needs: [apply-k8s, apply-compose]
+    if: >-
+      always() &&
+      needs.apply-k8s.result != 'skipped' &&
+      needs.apply-compose.result != 'skipped'
+    runs-on: [self-hosted, linux, arch, homelab, prod]
+    # Not raised, because the arithmetic does not close.
+    #
+    # 32 workloads are under management and the wave width is 8, so the verify
+    # itself is 4 waves of ROLLOUT_TIMEOUT (300s) = 20 minutes worst case, when
+    # every rollout times out rather than converging. That is already 20 of 30.
+    #
+    # The other 10 would have to absorb rollback, and rollback_workloads is a
+    # serial `while read` loop at 300s per failed workload. 10 minutes buys two.
+    # Any larger number is buying a bigger multiple of an unbounded term rather
+    # than covering a known cost: 60 minutes buys eight, and 60 minutes is
+    # therefore not a bound, it is a guess with two digits.
+    #
+    # The number becomes derivable the moment rollback uses the same wave width
+    # as the verify: 32 failures then cost 4 waves = 20 minutes instead of 160,
+    # and 45 covers verify plus rollback at full width. That change is to the
+    # recovery path and is not folded into a timeout edit.
+    timeout-minutes: 30
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+
+      - name: Verify workloads and roll back on failure
+        shell: bash
+        run: |
+          set -euo pipefail
+          ./.gitea/workflows/ssh-run.sh verify-k8s
+
+  # Asks the public route of every active service whether it is actually
+  # serving, which the rollout check above structurally cannot: a pod can
+  # converge and still be crash-looping, or be listening on a port no Service
+  # points at, or answer 500.
+  #
+  # `always()` for the same reason verify-k8s has it, and it runs after that job
+  # specifically because a rollback is when a route most needs re-checking. The
+  # needs is a barrier, not a filter: whether verify-k8s passed, failed or was
+  # cancelled, the probes are what say whether the cluster is serving, and
+  # suppressing them on a rollback would hide the one run where the answer
+  # matters most.
   smoke:
-    needs: [gate, verify]
-    if: always() && needs.gate.result == 'success'
-    runs-on: homelab
-    timeout-minutes: 15
+    needs: [verify-k8s]
+    if: always() && needs.verify-k8s.result != 'skipped'
+    runs-on: [self-hosted, linux, arch, homelab, prod]
+    timeout-minutes: 10
     steps:
-      - name: Checkout checked commit
+      - name: Checkout repository
         uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
-        with:
-          ref: ${{ needs.gate.outputs.sha }}
-      - name: Follow public route checks
-        run: bash .gitea/workflows/ssh-run.sh smoke
-      - name: Write the deploy result
-        if: always()
+
+      - name: Probe the public route of every active service
+        shell: bash
         run: |
-          if [ -f .gitea/workflows/ssh-run.sh ]; then
-            bash .gitea/workflows/ssh-run.sh summary
-          elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
-            echo 'Source checkout failed. The remote deploy state is unknown. Check the job log.' >>"$GITHUB_STEP_SUMMARY"
-          fi
+          set -euo pipefail
+          ./.gitea/workflows/ssh-run.sh smoke

.gitea/workflows/install-ci-tools.sh

@@ -13,14 +13,9 @@ here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 # shellcheck source=tool-versions.env
 . "$here/tool-versions.env"
 
-TOOLS_DIR="${TOOLS_DIR:-${XDG_CACHE_HOME:-$HOME/.cache}/homelab-ci}"
+TOOLS_DIR="${TOOLS_DIR:-${RUNNER_TEMP:-/tmp}/homelab-tools}"
 BIN_DIR="$TOOLS_DIR/bin"
 mkdir -p "$BIN_DIR"
-# A runner may accept overlapping workflows even though each workflow is sequential.
-exec 9>"$TOOLS_DIR/install.lock"
-flock -w 300 9
-export UV_TOOL_DIR="$TOOLS_DIR/uv-tools"
-export UV_CACHE_DIR="$TOOLS_DIR/uv-cache"
 # The just-installed tools must resolve inside this script too: callers only
 # prepend BIN_DIR to PATH after the script exits, so a bare `uv` below would
 # miss the binary install_uv just placed (exit 127 on a clean runner).
@@ -53,7 +48,7 @@ esac
 fetch() {
   # fetch <url> <dest>
   if command -v curl >/dev/null 2>&1; then
-    curl -sSLf --connect-timeout 15 --max-time 120 --retry 3 -o "$2" "$1"
+    curl -sSLf --retry 3 -o "$2" "$1"
   elif command -v wget >/dev/null 2>&1; then
     wget -q -O "$2" "$1"
   else
@@ -93,13 +88,10 @@ installed_version() {
 
 # at_version <command> <expected>
 at_version() {
-  local version expected="${2#v}"
-  version="$(installed_version "$1")"
-  if [[ "$version" =~ (^|[^0-9.])v?([0-9]+(\.[0-9]+)+) ]]; then
-    [ "${BASH_REMATCH[2]}" = "$expected" ]
-  else
-    return 1
-  fi
+  case "$(installed_version "$1")" in
+    *"$2"*) return 0 ;;
+    *) return 1 ;;
+  esac
 }
 
 install_kubeconform() {
@@ -128,15 +120,6 @@ install_shellcheck() {
   rm -rf "$tmp"
 }
 
-install_jq() {
-  if at_version jq "${JQ_VERSION}"; then
-    return 0
-  fi
-  fetch "https://github.com/jqlang/jq/releases/download/jq-${JQ_VERSION}/jq-linux-${goarch}" \
-    "$BIN_DIR/jq"
-  chmod 0755 "$BIN_DIR/jq"
-}
-
 install_uv() {
   if at_version uv "${UV_VERSION}"; then
     return 0
@@ -184,7 +167,6 @@ install_pip_audit() {
 }
 
 install_prettier() {
-  install_node
   if at_version prettier "${PRETTIER_VERSION}"; then
     return 0
   fi
@@ -245,41 +227,28 @@ install_actionlint() {
   rm -rf "$tmp"
 }
 
-main() {
-  wanted=("$@")
-  if [ "${#wanted[@]}" -eq 0 ]; then
-    wanted=(node jq kubeconform shellcheck actionlint prettier ruff yamllint hadolint)
-  fi
+wanted=("$@")
+if [ "${#wanted[@]}" -eq 0 ]; then
+  wanted=(kubeconform shellcheck actionlint prettier ruff yamllint hadolint)
+fi
 
-  for tool in "${wanted[@]}"; do
-    case "$tool" in
-      kubeconform) install_kubeconform ;;
-      shellcheck) install_shellcheck ;;
-      jq) install_jq ;;
-      actionlint) install_actionlint ;;
-      prettier) install_prettier ;;
-      ruff) install_ruff ;;
-      yamllint) install_yamllint ;;
-      pip-audit) install_pip_audit ;;
-      hadolint) install_hadolint ;;
-      node) install_node ;;
-      uv) install_uv ;;
-      *)
-        echo "install-ci-tools: unknown tool: $tool" >&2
-        exit 1
-        ;;
-    esac
-  done
+for tool in "${wanted[@]}"; do
+  case "$tool" in
+    kubeconform) install_kubeconform ;;
+    shellcheck) install_shellcheck ;;
+    actionlint) install_actionlint ;;
+    prettier) install_prettier ;;
+    ruff) install_ruff ;;
+    yamllint) install_yamllint ;;
+    pip-audit) install_pip_audit ;;
+    hadolint) install_hadolint ;;
+    node) install_node ;;
+    uv) install_uv ;;
+    *)
+      echo "install-ci-tools: unknown tool: $tool" >&2
+      exit 1
+      ;;
+  esac
+done
 
-  for old in "$BIN_DIR"/node-* "$BIN_DIR"/prettier-*; do
-    [ -d "$old" ] || continue
-    case "$(basename "$old")" in
-      "node-$NODE_VERSION"|"prettier-$PRETTIER_VERSION") ;;
-      *) rm -rf "$old" ;;
-    esac
-  done
-  if [ -x "$BIN_DIR/uv" ]; then "$BIN_DIR/uv" cache prune >/dev/null; fi
-  printf '%s\n' "$BIN_DIR"
-}
-
-if [ "${BASH_SOURCE[0]}" = "$0" ]; then main "$@"; fi
+printf '%s\n' "$BIN_DIR"

.gitea/workflows/release.py

@@ -1,516 +0,0 @@
-#!/usr/bin/env python3
-"""CI release artifacts and the SHA-specific Gitea deployment gate (stdlib only)."""
-
-import argparse
-import hashlib
-import io
-import itertools
-import json
-import os
-import re
-import shutil
-import subprocess
-import sys
-import tempfile
-import urllib.error
-import urllib.parse
-import urllib.request
-import zipfile
-from pathlib import Path
-
-SHA = re.compile(r'[0-9a-f]{40}')
-DIGEST = re.compile(r'sha256:[0-9a-f]{64}')
-IMAGES = {
-    'error-pages': ('errorpages', 'errorpages/Dockerfile'),
-    'forust-homepage': ('homepages', 'homepages/Dockerfile.forust'),
-    'xdfnx-homepage': ('homepages', 'homepages/Dockerfile.xdfnx'),
-}
-
-
-def command(*args, **kwargs):
-    """Arguments are passed directly to the executable, never to a shell."""
-    return subprocess.check_output(args, text=True, **kwargs).strip()  # noqa: S603, S607
-
-
-def validate_release(data, sha=None):
-    if data.get('version') != 1 or not SHA.fullmatch(data.get('sha', '')):
-        raise ValueError('Invalid release version or SHA')
-    if sha is not None and data['sha'] != sha:
-        raise ValueError('Release SHA does not match the checked CI commit')
-    expected = {f'gcr.forust.xyz/forust/{name}' for name in IMAGES}
-    if set(data.get('images', {})) != expected:
-        raise ValueError('Release must contain all owned images')
-    if not all(DIGEST.fullmatch(value) for value in data['images'].values()):
-        raise ValueError('Release has an invalid image digest')
-    if set(data.get('inputs', {})) != expected or not all(
-        re.fullmatch(r'[0-9a-f]{64}', value) for value in data['inputs'].values()
-    ):
-        raise ValueError('Release has invalid build input fingerprints')
-    return data
-
-
-class NoRedirect(urllib.request.HTTPRedirectHandler):
-    def redirect_request(self, _req, _fp, _code, _msg, _headers, _newurl):
-        return None
-
-
-class Gitea:
-    def __init__(self):
-        self.origin = os.environ['GITHUB_SERVER_URL'].rstrip('/')
-        if urllib.parse.urlsplit(self.origin).scheme != 'https':
-            raise ValueError('Gitea API must use HTTPS')
-        self.repository = os.environ['GITHUB_REPOSITORY']
-        if not re.fullmatch(r'[\w.-]+/[\w.-]+', self.repository):
-            raise ValueError('Invalid Gitea repository')
-        self.token = os.environ['GITEA_TOKEN']
-        self.base = f'{self.origin}/api/v1/repos/{self.repository}'
-
-    def request(self, url, *, archive=False):
-        if not url.startswith(self.base + '/'):
-            raise ValueError('Refusing to send the Actions token to another origin')
-        req = urllib.request.Request(url, headers={'Authorization': f'token {self.token}'})  # noqa: S310 -- HTTPS origin validated above
-        opener = urllib.request.build_opener(NoRedirect())
-        try:
-            response = opener.open(req, timeout=30)  # noqa: S310
-        except urllib.error.HTTPError as error:
-            if not archive or error.code not in (301, 302, 303, 307, 308):
-                raise RuntimeError(f'Gitea API returned HTTP {error.code}') from None
-            target = urllib.parse.urljoin(url, error.headers['Location'])
-            if urllib.parse.urlsplit(target).scheme != 'https':
-                raise ValueError('Artifact redirect must use HTTPS') from None
-            # Signed storage redirects must never receive the Gitea token.
-            response = urllib.request.urlopen(target, timeout=30)  # noqa: S310
-        with response:
-            payload = response.read(8 * 1024 * 1024 + 1)
-        if len(payload) > 8 * 1024 * 1024:
-            raise ValueError('Gitea response exceeds 8 MiB')
-        return payload if archive else json.loads(payload)
-
-    def pages(self, path, key, **params):
-        for page in range(1, 101):
-            query = urllib.parse.urlencode({**params, 'page': page, 'limit': 50})
-            data = self.request(f'{self.base}/{path}?{query}')
-            entries = data[key]
-            yield from entries
-            if len(entries) < 50:
-                return
-        raise RuntimeError('Gitea pagination limit exceeded')
-
-    def successful_runs(self, sha=None):
-        params = {'branch': 'main', 'status': 'success', 'exclude_pull_requests': 'true'}
-        if sha:
-            params['head_sha'] = sha
-        for run in self.pages('actions/workflows/ci.yaml/runs', 'workflow_runs', **params):
-            if (
-                run.get('status') == 'completed'
-                and run.get('conclusion') == 'success'
-                and run.get('head_branch') == 'main'
-                and run.get('event') in ('push', 'workflow_dispatch')
-                and (run.get('repository') or {}).get('full_name') == self.repository
-                and (run.get('head_repository') or run.get('repository') or {}).get('full_name') == self.repository
-                and (sha is None or run.get('head_sha') == sha)
-            ):
-                yield run
-
-    def release(self, run):
-        sha = run['head_sha']
-        jobs = list(self.pages(f'actions/runs/{run["id"]}/jobs', 'jobs'))
-        # A green workflow with a skipped build must not authorize a deploy.
-        if not any(job.get('name') == 'build' and job.get('conclusion') == 'success' for job in jobs):
-            raise ValueError('CI build job did not succeed')
-        artifacts = self.request(f'{self.base}/actions/runs/{run["id"]}/artifacts')['artifacts']
-        matching = [a for a in artifacts if a['name'] == f'release-{sha}' and not a.get('expired')]
-        if len(matching) != 1:
-            raise ValueError('CI release artifact is missing, expired or ambiguous; rerun CI')
-        blob = self.request(f'{self.base}/actions/artifacts/{matching[0]["id"]}/zip', archive=True)
-        with zipfile.ZipFile(io.BytesIO(blob)) as archive:
-            files = [entry for entry in archive.infolist() if not entry.is_dir()]
-            if len(files) != 1 or files[0].filename != 'release.json' or files[0].file_size > 256 * 1024:
-                raise ValueError('Unexpected release archive contents')
-            return validate_release(json.loads(archive.read(files[0])), sha)
-
-
-def fingerprint(context, dockerfile):
-    tree = command('git', 'ls-tree', '-r', 'HEAD', '--', context, dockerfile, '.gitea/workflows/release.py')
-    return hashlib.sha256(tree.encode()).hexdigest()
-
-
-def gate(output, requested_ref, event_sha):
-    command('git', 'fetch', '--quiet', 'origin', 'main')
-    if event_sha:
-        if not SHA.fullmatch(event_sha):
-            raise ValueError('Invalid workflow_run SHA')
-        sha = event_sha
-    else:
-        if requested_ref == 'main':
-            requested_ref = 'origin/main'
-        sha = command('git', 'rev-parse', '--verify', '--end-of-options', f'{requested_ref}^{{commit}}')
-    if not SHA.fullmatch(sha):
-        raise ValueError('Invalid deploy SHA')
-    command('git', 'merge-base', '--is-ancestor', sha, 'origin/main')
-    api = Gitea()
-    runs = list(api.successful_runs(sha))
-    if not runs:
-        raise ValueError(f'No successful main CI for {sha}; run CI before deploying')
-    release = api.release(max(runs, key=lambda run: run['id']))
-    output.write_text(json.dumps(release, indent=2) + '\n')
-    if os.environ.get('GITHUB_OUTPUT'):
-        with Path(os.environ['GITHUB_OUTPUT']).open('a') as stream:
-            stream.write(f'sha={sha}\n')
-    print(f'CI gate accepted {sha}')
-
-
-def prepare_images(output):
-    sha = command('git', 'rev-parse', 'HEAD')
-    if sha != os.environ['GITHUB_SHA'] or not SHA.fullmatch(sha):
-        raise ValueError('Build checkout does not match GITHUB_SHA')
-    api = Gitea()
-    previous = None
-    for run in sorted(itertools.islice(api.successful_runs(), 50), key=lambda item: item['id'], reverse=True):
-        if str(run['id']) == os.environ.get('GITHUB_RUN_ID'):
-            continue
-        try:
-            previous = api.release(run)
-            break
-        except ValueError:
-            # Expired artifacts only cost a rebuild; mutable tags are never a fallback.
-            continue
-    targets = []
-    for name, (context, dockerfile) in IMAGES.items():
-        image = f'gcr.forust.xyz/forust/{name}'
-        inputs = fingerprint(context, dockerfile)
-        old_digest = (previous or {}).get('images', {}).get(image)
-        targets.append(
-            {
-                'name': name,
-                'image': image,
-                'context': context,
-                'dockerfile': dockerfile,
-                'inputs': inputs,
-                'reuse_digest': old_digest if (previous or {}).get('inputs', {}).get(image) == inputs else None,
-            }
-        )
-    output.write_text(json.dumps({'sha': sha, 'targets': targets}, indent=2) + '\n')
-    if os.environ.get('GITHUB_OUTPUT'):
-        with Path(os.environ['GITHUB_OUTPUT']).open('a') as stream:
-            stream.write('matrix=' + json.dumps({'include': targets}, separators=(',', ':')) + '\n')
-    print(f'Prepared {len(targets)} image jobs; {sum(t["reuse_digest"] is None for t in targets)} require builds')
-
-
-def checked_plan(path):
-    data = json.loads(path.read_text())
-    sha = command('git', 'rev-parse', 'HEAD')
-    if data.get('sha') != sha or sha != os.environ['GITHUB_SHA'] or not SHA.fullmatch(sha):
-        raise ValueError('Image plan does not match the checked source commit')
-    targets = data.get('targets', [])
-    if sorted(t['name'] for t in targets) != sorted(IMAGES):
-        raise ValueError('Image plan must contain each owned image once')
-    for target in targets:
-        name = target['name']
-        context, dockerfile = IMAGES[name]
-        if (target['context'], target['dockerfile'], target['image']) != (
-            context,
-            dockerfile,
-            f'gcr.forust.xyz/forust/{name}',
-        ) or target['inputs'] != fingerprint(context, dockerfile):
-            raise ValueError('Image plan has invalid build inputs')
-        if target['reuse_digest'] is not None and not DIGEST.fullmatch(target['reuse_digest']):
-            raise ValueError('Image plan has an invalid reuse digest')
-    return data
-
-
-def build_images(output, report, name, plan):
-    data = checked_plan(plan)
-    sha = data['sha']
-    target = next(t for t in data['targets'] if t['name'] == name)
-    context, dockerfile = IMAGES[name]
-    docker_config = tempfile.mkdtemp(prefix='homelab-registry-')
-    builder_config = Path.home() / '.cache/homelab-ci/buildx'
-    builder_config.mkdir(parents=True, exist_ok=True)
-    env = {**os.environ, 'DOCKER_CONFIG': docker_config, 'BUILDX_CONFIG': str(builder_config)}
-    try:
-        report['phase'] = 'Registry login'
-        subprocess.run(  # noqa: S603, S607
-            [
-                shutil.which('docker') or '/usr/bin/docker',
-                'login',
-                'gcr.forust.xyz',
-                '-u',
-                os.environ['REGISTRY_USERNAME'],
-                '--password-stdin',
-            ],
-            input=os.environ['REGISTRY_PASSWORD'],
-            text=True,
-            check=True,
-            env=env,
-        )
-        report['phase'] = 'Prepare the builder'
-        builder = 'homelab-ci'
-        versions = dict(
-            re.findall(r'^([A-Z_]+)="([^"\n]+)"$', Path('.gitea/workflows/tool-versions.env').read_text(), re.MULTILINE)
-        )
-        image = versions['BUILDKIT_IMAGE']
-        signature = builder_config / 'homelab-ci-image'
-        exists = (
-            subprocess.run(  # noqa: S603
-                [shutil.which('docker') or '/usr/bin/docker', 'buildx', 'inspect', builder],
-                capture_output=True,
-                env=env,
-            ).returncode
-            == 0
-        )
-        if exists and (not signature.exists() or signature.read_text().strip() != image):
-            command('docker', 'buildx', 'rm', '--keep-state', builder, env=env)
-            exists = False
-        if not exists:
-            command(
-                'docker',
-                'buildx',
-                'create',
-                '--name',
-                builder,
-                '--driver',
-                'docker-container',
-                '--driver-opt',
-                f'image={image}',
-                '--buildkitd-config',
-                '.gitea/runner/buildkitd.toml',
-                env=env,
-            )
-            signature.write_text(image + '\n')
-        release = {'version': 1, 'sha': sha, 'images': {}, 'inputs': {}}
-        report['images'] = release['images']
-        report['phase'] = f'Build or reuse {name}'
-        report['current'] = name
-        image = f'gcr.forust.xyz/forust/{name}'
-        inputs = target['inputs']
-        old_digest = target['reuse_digest']
-        exists = False
-        if old_digest:
-            exists = (
-                subprocess.run(  # noqa: S603, S607
-                    [
-                        shutil.which('docker') or '/usr/bin/docker',
-                        'buildx',
-                        'imagetools',
-                        'inspect',
-                        f'{image}@{old_digest}',
-                    ],
-                    capture_output=True,
-                    env=env,
-                    timeout=60,
-                ).returncode
-                == 0
-            )
-        if exists:
-            print(f'Reuse {name}: inputs unchanged')
-            digest = old_digest
-            report['reused'].append(name)
-        else:
-            print(f'Build {name}', flush=True)
-            metadata = Path(docker_config) / 'metadata.json'
-            command(
-                'docker',
-                'buildx',
-                'build',
-                '--builder',
-                builder,
-                '--platform',
-                'linux/amd64',
-                '--provenance=false',
-                '--cache-from',
-                f'type=registry,ref={image}:buildcache',
-                '--cache-to',
-                f'type=registry,ref={image}:buildcache,mode=max',
-                '--output',
-                f'type=image,name={image},push-by-digest=true,name-canonical=true,push=true',
-                '--metadata-file',
-                str(metadata),
-                '--file',
-                dockerfile,
-                context,
-                env=env,
-            )
-            digest = json.loads(metadata.read_text())['containerimage.digest']
-            report['built'].append(name)
-        release['images'][image] = digest
-        release['inputs'][image] = inputs
-        if not DIGEST.fullmatch(digest):
-            raise ValueError('Image job returned an invalid digest')
-        output.write_text(json.dumps(release, indent=2) + '\n')
-        report['current'] = None
-        report['phase'] = 'Release file saved'
-    finally:
-        # Cleanup errors must neither leak credentials nor mask the original build error.
-        try:
-            subprocess.run(  # noqa: S603
-                [
-                    shutil.which('docker') or '/usr/bin/docker',
-                    'buildx',
-                    'prune',
-                    '--builder',
-                    'homelab-ci',
-                    '--force',
-                    '--max-used-space',
-                    '1gb',
-                ],
-                env=env,
-                timeout=60,
-            )
-        except (OSError, subprocess.TimeoutExpired):
-            print('CI builder cache cleanup deferred', flush=True)
-        finally:
-            shutil.rmtree(docker_config)
-
-
-def write_summary(lines):
-    path = os.environ.get('GITHUB_STEP_SUMMARY')
-    if path:
-        try:
-            with Path(path).open('a') as stream:
-                stream.write('\n'.join(lines) + '\n\n')
-        except OSError:
-            print('WARNING: cannot write the job summary')
-
-
-def check_summary():
-    lines = [
-        f'## {os.environ["SUMMARY_CHECK"]}',
-        '',
-        f'- Commit: `{os.environ.get("GITHUB_SHA", "unknown")}`',
-        f'- Result: **{os.environ["SUMMARY_RESULT"]}**',
-    ]
-    if os.environ.get('SUMMARY_FAILED_STEP'):
-        lines.append(f'- Failed step: {os.environ["SUMMARY_FAILED_STEP"]}')
-    if os.environ['SUMMARY_RESULT'] != 'success':
-        lines.append('- Open the failed step log for the error details.')
-    write_summary(lines)
-
-
-def build(output, name, plan):
-    report = {'phase': 'Check the source commit', 'current': None, 'built': [], 'reused': [], 'images': {}}
-    result = 'failure'
-    try:
-        build_images(output, report, name, plan)
-        result = 'success'
-    finally:
-        lines = [
-            f'## Image release `{os.environ.get("GITHUB_SHA", "unknown")}`',
-            '',
-            f'- Result: **{result}**',
-            f'- Last stage: {report["phase"]}',
-        ]
-        if result == 'failure':
-            lines.append('- No release from this build can be deployed. Open the failed step log.')
-        if report['current']:
-            lines.append(f'- Image at the failure: `{report["current"]}`')
-        for title, key in (('Built', 'built'), ('Reused from successful CI', 'reused')):
-            lines.extend(['', f'### {title}'])
-            lines.extend(f'- `{name}`' for name in report[key])
-            if not report[key]:
-                lines.append('- None')
-        lines.extend(['', '### Completed image digests'])
-        lines.extend(f'- `{image}@{digest}`' for image, digest in report['images'].items())
-        if not report['images']:
-            lines.append('- None')
-        write_summary(lines)
-
-
-def render(stream, destination):
-    release = validate_release(json.loads(Path(os.environ['RELEASE_FILE']).read_text()), os.environ['DEPLOY_SHA'])
-    image_line = re.compile(
-        r"^(\s*(?:-\s*)?image:\s*)(['\"]?)(gcr\.forust\.xyz/forust/[\w.-]+)(?::[\w.-]+|@sha256:[0-9a-f]{64})\2(\s*(?:#.*)?)$"
-    )
-    rendered = []
-    for line in stream:
-        match = image_line.fullmatch(line.rstrip('\n'))
-        if match:
-            prefix, quote, image, tail = match.groups()
-            if image not in release['images']:
-                raise ValueError(f'Owned image missing from checked release: {image}')
-            line = f'{prefix}{quote}{image}@{release["images"][image]}{quote}{tail}\n'
-        elif re.match(r'\s*(?:-\s*)?image:', line) and 'gcr.forust.xyz/forust/' in line:
-            raise ValueError('Unsupported owned image syntax; refusing to apply a mutable tag')
-        rendered.append(line)
-    destination.writelines(rendered)
-
-
-def finalize_images(output, fragments, plan):
-    data = checked_plan(plan)
-    sha = data['sha']
-    release = {'version': 1, 'sha': sha, 'images': {}, 'inputs': {}}
-    for name in IMAGES:
-        fragment = json.loads((fragments / f'image-{name}' / 'image.json').read_text())
-        image = f'gcr.forust.xyz/forust/{name}'
-        if fragment.get('sha') != sha or fragment.get('version') != 1 or set(fragment.get('images', {})) != {image}:
-            raise ValueError('Image job artifact is missing or belongs to another commit')
-        target = next(t for t in data['targets'] if t['name'] == name)
-        if fragment.get('inputs') != {image: target['inputs']}:
-            raise ValueError('Image artifact does not match the build plan')
-        release['images'].update(fragment['images'])
-        release['inputs'].update(fragment['inputs'])
-    validate_release(release, sha)
-    # Only a complete set of successful image jobs can publish the release tags.
-    docker_config = tempfile.mkdtemp(prefix='homelab-registry-')
-    env = {**os.environ, 'DOCKER_CONFIG': docker_config}
-    try:
-        subprocess.run(  # noqa: S603, S607
-            [
-                shutil.which('docker') or '/usr/bin/docker',
-                'login',
-                'gcr.forust.xyz',
-                '-u',
-                os.environ['REGISTRY_USERNAME'],
-                '--password-stdin',
-            ],
-            input=os.environ['REGISTRY_PASSWORD'],
-            text=True,
-            check=True,
-            env=env,
-        )
-        for image, digest in release['images'].items():
-            command(
-                'docker',
-                'buildx',
-                'imagetools',
-                'create',
-                '--prefer-index=false',
-                '--tag',
-                f'{image}:sha-{sha}',
-                f'{image}@{digest}',
-                env=env,
-                timeout=90,
-            )
-        output.write_text(json.dumps(release, indent=2) + '\n')
-    finally:
-        shutil.rmtree(docker_config)
-
-
-def main():
-    parser = argparse.ArgumentParser(description=__doc__)
-    parser.add_argument('action', choices=('prepare', 'image', 'finalize', 'gate', 'render', 'check-summary'))
-    parser.add_argument('--output', type=Path, default=Path('release.json'))
-    parser.add_argument('--ref', default='main')
-    parser.add_argument('--event-sha', default='')
-    parser.add_argument('--image', choices=IMAGES)
-    parser.add_argument('--plan', type=Path, default=Path('build-plan.json'))
-    parser.add_argument('--fragments', type=Path, default=Path('artifacts'))
-    args = parser.parse_args()
-    if args.action == 'check-summary':
-        check_summary()
-    elif args.action == 'render':
-        render(sys.stdin, sys.stdout)
-    elif args.action == 'gate':
-        gate(args.output, args.ref, args.event_sha)
-    elif args.action == 'prepare':
-        prepare_images(args.output)
-    elif args.action == 'image':
-        if not args.image:
-            parser.error('--image is required')
-        build(args.output, args.image, args.plan)
-    else:
-        finalize_images(args.output, args.fragments, args.plan)
-
-
-if __name__ == '__main__':
-    main()

.gitea/workflows/renovate-ci.yaml

@@ -2,23 +2,9 @@ name: renovate-ci
 
 on:
   pull_request:
-    paths:
-      - "renovate/**"
-      - ".gitea/workflows/renovate-ci.yaml"
-      - ".gitea/workflows/sync-renovate-configmap.sh"
-      - ".gitea/workflows/compose-lint.sh"
-      - ".gitea/workflows/install-ci-tools.sh"
-      - ".gitea/workflows/tool-versions.env"
   push:
     branches:
       - main
-    paths:
-      - "renovate/**"
-      - ".gitea/workflows/renovate-ci.yaml"
-      - ".gitea/workflows/sync-renovate-configmap.sh"
-      - ".gitea/workflows/compose-lint.sh"
-      - ".gitea/workflows/install-ci-tools.sh"
-      - ".gitea/workflows/tool-versions.env"
   workflow_dispatch:
 
 permissions:
@@ -26,7 +12,7 @@ permissions:
 
 jobs:
   validate-renovate:
-    runs-on: homelab
+    runs-on: [self-hosted, linux, arch, homelab]
     timeout-minutes: 20
     steps:
       - name: Checkout repository

.gitea/workflows/renovate-run.yaml

@@ -32,7 +32,7 @@ concurrency:
 
 jobs:
   run-renovate:
-    runs-on: homelab
+    runs-on: [self-hosted, linux, arch, homelab]
     timeout-minutes: 60
     steps:
       - name: Checkout repository

.gitea/workflows/secret-references.jq

@@ -1,13 +0,0 @@
-# kubectl emits a List for files containing multiple resources.
-(if .kind == "List" then .items[] else . end)
-| (.metadata.namespace // "default") as $ns
-| [
-    (.. | objects
-      | (.secretRef? // empty), (.secretKeyRef? // empty), (.secret? // empty)
-      | select(.optional != true)
-      | .name // .secretName // empty),
-    (.. | objects | .imagePullSecrets[]?.name)
-  ]
-| unique[]
-| select(. != null and . != "")
-| "\($ns) \(.)"

.gitea/workflows/ssh-run.sh

@@ -1,70 +1,71 @@
 #!/usr/bin/env bash
-# The SSH client submits once and follows durable stages on workstation.
+# usage: ssh-run.sh <stage>
+# Runs one deploy-lib.sh stage on the workstation over SSH.
 set -euo pipefail
+
 : "${DEPLOY_HOST:?missing DEPLOY_HOST}"
 : "${DEPLOY_USER:?missing DEPLOY_USER}"
 : "${DEPLOY_KEY:?missing DEPLOY_SSH_KEY}"
-: "${DEPLOY_KNOWN_HOSTS:?configure pinned DEPLOY_KNOWN_HOSTS}"
-: "${DEPLOY_RUN_ID:?missing DEPLOY_RUN_ID}"
-[[ "$DEPLOY_USER" =~ ^[A-Za-z_][A-Za-z0-9_.-]*$ ]] || exit 1
-[[ "$DEPLOY_HOST" =~ ^[A-Za-z0-9_.:-]+$ ]] || exit 1
-[[ "$DEPLOY_RUN_ID" =~ ^[0-9]+-[0-9]+$ ]] || exit 1
-[[ "${DEPLOY_PORT:-22}" =~ ^[0-9]+$ ]] || exit 1
+
+deploy_port="${DEPLOY_PORT:-22}"
+deploy_path="${DEPLOY_PATH:-/srv/homelab}"
+deploy_path="$(printf '%s' "$deploy_path" | tr -d '\"' | tr -d '\r' | xargs)"
+
+# The private key is written to a per-run directory that is removed on exit, so a
+# failed or cancelled job cannot leave deploy credentials in the runner's temp
+# directory. Do not use a fixed path: apply-k8s and apply-compose run in parallel.
 key_dir="$(mktemp -d "${RUNNER_TEMP:-/tmp}/homelab-deploy-key.XXXXXXXX")"
-trap 'rm -rf "$key_dir"' EXIT
-chmod 700 "$key_dir"
-printf '%s\n' "$DEPLOY_KEY" >"$key_dir/key"
-printf '%s\n' "$DEPLOY_KNOWN_HOSTS" >"$key_dir/known_hosts"
-chmod 600 "$key_dir/key" "$key_dir/known_hosts"
-ssh_opts=(-i "$key_dir/key" -p "${DEPLOY_PORT:-22}" -o BatchMode=yes -o StrictHostKeyChecking=yes
-  -o "UserKnownHostsFile=$key_dir/known_hosts" -o ConnectTimeout=15
-  -o ServerAliveInterval=15 -o ServerAliveCountMax=4)
-controller=.local/lib/homelab-deploy/controller.py
-case "${1:?start, apply, verify, smoke or summary required}" in
-  start)
-    python3 - <<'PY' >"$key_dir/request.json"
-import json
-import os
-from pathlib import Path
-release = json.loads(Path('release.json').read_text())
-print(json.dumps({'release': release, 'mode': os.environ.get('DEPLOY_MODE', 'changed'),
-                  'refresh_images': os.environ.get('REFRESH_IMAGES', 'false') == 'true'}))
-PY
-    for attempt in 1 2 3; do
-      rc=0
-      # shellcheck disable=SC2029 # The run ID and operation are validated local arguments, not remote variables.
-      ssh "${ssh_opts[@]}" "$DEPLOY_USER@$DEPLOY_HOST" python3 "$controller" start "$DEPLOY_RUN_ID" <"$key_dir/request.json" || rc=$?
-      [ "$rc" -eq 0 ] && exit 0
-      [ "$rc" -eq 255 ] || exit "$rc"
-      sleep 5
-    done
-    exit "$rc"
+trap 'rm -rf "$key_dir"' EXIT INT TERM
+
+ssh_key="$key_dir/deploy_key"
+printf '%s\n' "$DEPLOY_KEY" > "$ssh_key"
+chmod 600 "$ssh_key"
+
+# A connection that died silently used to hang until the job timeout, and the
+# stage was never re-run: one flaky TCP session cost a whole 45-minute apply.
+# ServerAlive* bounds how long a dead peer goes unnoticed, ConnectTimeout bounds
+# setup. Only exit 255 - ssh's own transport failures - is retried. A stage that
+# fails on its own merits exits with the remote's status, so a real failure
+# still surfaces its own log instead of burning three attempts. The stages are
+# declarative applies, so re-running one that had already committed is harmless.
+ssh_opts=(
+  -i "$ssh_key" -p "$deploy_port"
+  -o BatchMode=yes -o StrictHostKeyChecking=accept-new
+  -o ConnectTimeout=15
+  -o ServerAliveInterval=15 -o ServerAliveCountMax=4
+)
+
+rc=0
+# apply-k8s and apply-compose are separate workflow jobs so the graph stays
+# intact for the verify job, but on a single node they must not run at once:
+# host docker churn on top of cluster churn is what melts the node (load 40+,
+# netbird/ssh die, helm is left pending-*). Serialize them on the workstation
+# with a shared lock; whoever arrives second waits.
+remote_cmd=(bash -se)
+case "$1" in
+  apply-k8s | apply-compose)
+    remote_cmd=(flock -w 5400 /tmp/homelab-apply.lock bash -se)
     ;;
-  apply|verify|smoke)
-    result=0
-    for attempt in 1 2 3; do
-      rc=0
-      # shellcheck disable=SC2029 # The run ID and operation are validated local arguments, not remote variables.
-      ssh "${ssh_opts[@]}" "$DEPLOY_USER@$DEPLOY_HOST" python3 "$controller" follow "$DEPLOY_RUN_ID" "$1" || rc=$?
-      [ "$rc" -eq 0 ] && break
-      [ "$rc" -eq 255 ] || { result="$rc"; break; }
-      echo "SSH disconnected; reconnecting to the existing deploy ($attempt/3)"
-      if [ "$attempt" -eq 3 ]; then result=255; break; fi
-      sleep 5
-    done
-    exit "$result"
-    ;;
-  summary)
-    if [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
-      rc=0
-      # shellcheck disable=SC2029 # The run ID is validated above.
-      ssh "${ssh_opts[@]}" "$DEPLOY_USER@$DEPLOY_HOST" python3 "$controller" summary "$DEPLOY_RUN_ID" >"$key_dir/deploy-summary.md" || rc=$?
-      if [ "$rc" -eq 0 ]; then
-        cat "$key_dir/deploy-summary.md" >>"$GITHUB_STEP_SUMMARY" || echo "WARNING: cannot write the deploy summary"
-      else
-        echo 'Deploy summary is unavailable. The SSH connection failed or the controller did not respond. Check the job log.' >>"$GITHUB_STEP_SUMMARY" || true
-      fi
-    fi
-    ;;
-  *) echo "Unknown SSH operation: $1" >&2; exit 1 ;;
 esac
+for attempt in 1 2 3; do
+  if [ "$attempt" -gt 1 ]; then
+    echo ":: warning::ssh transport failed, retrying (${attempt}/3)"
+    sleep $((attempt * 5))
+  fi
+  rc=0
+  # shellcheck disable=SC2029 # remote_cmd/ssh_opts expand on the client on purpose: they select the local ssh invocation, only the heredoc runs remotely.
+  ssh "${ssh_opts[@]}" "${DEPLOY_USER}@${DEPLOY_HOST}" \
+    env "REPO=$deploy_path" "APPLY_PRUNE=${APPLY_PRUNE:-false}" \
+    "DEPLOY_SHA=${DEPLOY_SHA:-}" "DEPLOY_SNAPSHOT_DIR=${DEPLOY_SNAPSHOT_DIR:-}" \
+    "STAGE=$1" "${remote_cmd[@]}" <<'EOF' || rc=$?
+source "$REPO/.gitea/workflows/deploy-lib.sh"
+run_stage "$STAGE"
+EOF
+  [ "$rc" -eq 0 ] && break
+  [ "$rc" -ne 255 ] && break
+done
+
+if [ "$rc" -ne 0 ]; then
+  echo ":: error::stage $1 failed over ssh (exit $rc)"
+fi
+exit "$rc"

.gitea/workflows/tool-versions.env

@@ -31,9 +31,3 @@ UV_VERSION="0.12.17"
 # so the tree that gets tested is the tree that gets built. Renovate keeps this
 # in step with the Dockerfile's node: tag via the "node runtime" group.
 NODE_VERSION="22.23.3"
-
-# Secret-reference regression tests parse rendered Kubernetes objects.
-JQ_VERSION="1.8.1"
-
-# BuildKit is the only auxiliary CI container; jobs themselves stay on the host.
-BUILDKIT_IMAGE="moby/buildkit:v0.33.1"

.gitignore

@@ -94,6 +94,7 @@ replacements.txt
 .idea
 
 # Temp files
+edu_master/temp/
 temp/*
 # Local-only tooling scratch space (pinned CI tools, verification scripts)
 tmp/

adguardhome/compose.yaml

@@ -31,7 +31,7 @@ services:
       - "traefik.http.routers.adguard-dev.entrypoints=websecure"
       - "traefik.http.routers.adguard-dev.tls=true"
       # DoH Router
-      - "traefik.http.routers.dns-over-https.rule=(Host(`dns.forust.xyz`) || Host(`adguard.forust.xyz`)) && PathPrefix(`/dns-query`)"
+      - "traefik.http.routers.dns-over-https.rule=(Host(`dns.forust.xyz` || Host(`adguard.forust.xyz`)) && PathPrefix(`/dns-query`))"
       - "traefik.http.routers.dns-over-https.entrypoints=websecure"
       - "traefik.http.routers.dns-over-https.tls.certresolver=letsencrypt"
 

adguardhome/k8s/adguard.yaml

@@ -51,8 +51,6 @@ spec:
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: adguard-deployment
   namespace: adguard
 spec:
@@ -66,6 +64,8 @@ spec:
     metadata:
       labels:
         app: adguard
+      annotations:
+        reloader.stakater.com/auto: "true"
     spec:
       containers:
         - name: adguard

authentik/k8s/authentik.yaml

@@ -27,8 +27,6 @@ spec:
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: authentik-server-deployment
   namespace: authentik
 spec:
@@ -65,8 +63,6 @@ spec:
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: authentik-worker-deployment
   namespace: authentik
 spec:

cfddns/k8s/deployment.yaml

@@ -1,8 +1,6 @@
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: cfddns
   labels:
     app: cfddns

checkmk/k8s/checkmk.yaml

@@ -17,8 +17,6 @@ spec:
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: checkmk-deployment
   namespace: checkmk
 spec:

cloudflared/k8s/deployment.yaml

@@ -1,8 +1,6 @@
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: cloudflared
   labels:
     app: cloudflared
@@ -20,7 +18,7 @@ spec:
     spec:
       containers:
         - name: cloudflared
-          image: cloudflare/cloudflared:2026.10.0
+          image: cloudflare/cloudflared:2026.9.3
           imagePullPolicy: IfNotPresent
           args:
             - tunnel

converters/k8s/convertx.yaml

@@ -13,8 +13,6 @@ spec:
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: convertx-deployment
   namespace: converters
 spec:

edu_master/.env.example

@@ -0,0 +1,15 @@
+KEEPER_LOGIN=your_edu_login_here
+KEEPER_PASSWORD=your_edu_password_here
+EDU_URL_BASE=https://edu.edu.vn.ua
+EDU_URL_LOGIN=/user/login
+EDU_URL_COURSES=/course/userlist
+KEEPER_INTERVAL=10
+USER_AGENT="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/142.0.0.0 Safari/537.36"
+EDU_URL_WEBINAR=/webinar/useractive
+WEBINAR_CHECK_INTERVAL=60
+REDIS_HOST=redis
+REDIS_PORT=6379
+PLAYWRIGHT_WS=ws://playwright-service:3000/ws
+TZ=Europe/Kyiv
+WEBINAR_TELEGRAM_TOKEN=your_telegram_bot_token_here
+WEBINAR_ADMIN_ID=123456789

edu_master/PLAYWRIGHT_VERSION

@@ -0,0 +1 @@
+1.56.0

edu_master/compose.yaml

@@ -0,0 +1,49 @@
+services:
+  redis:
+    image: redis:8.10.2-alpine
+    restart: unless-stopped
+    volumes:
+      - redis-data:/data
+    healthcheck:
+      test: ["CMD", "redis-cli", "ping"]
+      interval: 5s
+      timeout: 3s
+      retries: 5
+
+  playwright-service:
+    image: mcr.microsoft.com/playwright:v1.56.0-jammy
+    restart: unless-stopped
+    command: npx -y playwright@1.56.0 run-server --port 3000 --path /ws
+
+  session-keeper:
+    build: ./phpsessid-bot
+    image: gcr.forust.xyz/forust/session-keeper:prod
+    pull_policy: build
+    env_file: .env
+    restart: unless-stopped
+    depends_on:
+      redis:
+        condition: service_healthy
+    healthcheck:
+      test: ["CMD-SHELL", "redis-cli -h redis EXISTS EDU_PHPSESSID | grep -q 1"]
+      interval: 30s
+      timeout: 5s
+      retries: 10
+      start_period: 60s
+
+  webinar-checker:
+    build: ./webinar-checker
+    image: gcr.forust.xyz/forust/webinar-checker:prod
+    pull_policy: build
+    env_file: .env
+    restart: unless-stopped
+    depends_on:
+      redis:
+        condition: service_healthy
+      session-keeper:
+        condition: service_healthy
+      playwright-service:
+        condition: service_started
+
+volumes:
+  redis-data:

edu_master/k8s/alerts.yaml

@@ -0,0 +1,96 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: edu-master-webinar
+  namespace: edu-master
+  labels:
+    release: prometheus-stack
+spec:
+  groups:
+    - name: edu_master.webinar
+      rules:
+        # No successful webinar check for 5m (~2-3 missed 2-min checks).
+        # Catches: playwright hangs/timeouts, version skew, site changes, hung job.
+        # The last_success > 0 guard is mandatory: checker.py initialises
+        # last_success to 0, so without it `time() - 0` equals the current epoch
+        # and humanizeDuration renders ~20722d on every pod restart. Keep the
+        # duration expression on the left so $value stays the real gap.
+        - alert: WebinarCheckerNoSuccessfulCheck
+          expr: |
+            ((time() - webinar_check_last_success_timestamp_seconds) > 300)
+              and (webinar_check_last_success_timestamp_seconds > 0)
+              and (webinar_check_last_run_timestamp_seconds > 0)
+          for: 2m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Webinar checker has no successful check for 5m"
+            description: "edu-master/webinar-checker: last successful webinar check was {{ $value | humanizeDuration }} ago. Checks are failing or hanging (see consecutive failures alert). Notifications about new webinars are NOT being sent."
+
+        # Checks are running but none has ever succeeded since pod start.
+        # Split out from the rule above so a zeroed gauge never feeds
+        # humanizeDuration.
+        - alert: WebinarCheckerNeverSucceeded
+          expr: |
+            (webinar_check_last_success_timestamp_seconds == 0)
+              and (webinar_check_last_run_timestamp_seconds > 0)
+          for: 10m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Webinar checker has never completed a successful check"
+            description: 'edu-master/webinar-checker: checks have been running for 10m but not one has ever succeeded since the pod started, so every check is failing. Check pod logs (Loki: {namespace="edu-master", container="webinar-checker"}).'
+
+        # Fast path: 3 consecutive failures (~6+ min at 2-min interval).
+        - alert: WebinarCheckerConsecutiveFailures
+          expr: |
+            webinar_check_consecutive_failures >= 3
+          for: 5m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Webinar checker failing consecutively"
+            description: 'edu-master/webinar-checker: {{ $value }} consecutive webinar check failures (timeout / playwright error / page error). Check pod logs (Loki: {namespace="edu-master", container="webinar-checker"}).'
+
+        # Metrics endpoint not scraped for 10m: pod down, metrics server dead, or ServiceMonitor broken.
+        - alert: WebinarCheckerScrapeDown
+          expr: |
+            absent(webinar_check_last_run_timestamp_seconds) == 1
+          for: 10m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Webinar checker metrics missing"
+            description: "edu-master/webinar-checker: no metrics series for 10m. Pod may be down, metrics server dead, or ServiceMonitor/Service broken. Webinar checks are unobserved."
+
+        # EDU session lost: session-keeper down or credentials expired. Without PHPSESSID every check is skipped.
+        - alert: EduPhpsessidMissing
+          expr: |
+            edu_phpsessid_present == 0
+          for: 10m
+          labels:
+            severity: critical
+          annotations:
+            summary: "EDU_PHPSESSID missing"
+            description: "edu-master: EDU_PHPSESSID absent from redis for 10m. Webinar/diari/schedule checks are all skipped. Check session-keeper logs and EDU credentials."
+
+        # Hard deps: checker and playwright deployments unavailable.
+        - alert: WebinarCheckerDeploymentDown
+          expr: |
+            kube_deployment_status_replicas_unavailable{deployment="webinar-checker", namespace="edu-master"} > 0
+          for: 10m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Webinar checker deployment unavailable"
+            description: "edu-master/webinar-checker deployment has {{ $value }} unavailable replica(s) for 10m."
+
+        - alert: PlaywrightServiceDown
+          expr: |
+            kube_deployment_status_replicas_unavailable{deployment="playwright-service", namespace="edu-master"} > 0
+          for: 10m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Playwright service unavailable"
+            description: "edu-master/playwright-service deployment has {{ $value }} unavailable replica(s) for 10m. All webinar/diari/schedule checks fail without it."

edu_master/k8s/namespace.yaml

@@ -0,0 +1,4 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: edu-master

edu_master/k8s/playwright.yaml

@@ -0,0 +1,69 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: playwright-service
+  namespace: edu-master
+  labels:
+    app: edu-master-playwright
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: edu-master-playwright
+  strategy:
+    type: Recreate
+  template:
+    metadata:
+      labels:
+        app: edu-master-playwright
+    spec:
+      containers:
+        - name: playwright
+          # renovate: datasource=docker depName=mcr.microsoft.com/playwright versioning=docker
+          image: mcr.microsoft.com/playwright:v1.56.0-jammy
+          imagePullPolicy: IfNotPresent
+          # p95 412M, max 478M over 7 days, no limit before. Request is set at p95
+          # so the pod is not an eviction candidate; the limit stays above 2x the
+          # request because browser page lifetimes are unpredictable.
+          resources:
+            requests:
+              cpu: "200m"
+              memory: "416Mi"
+            limits:
+              memory: "1Gi"
+          command:
+            - npx
+            - -y
+            - playwright@1.56.0
+            - run-server
+            - --port
+            - "3000"
+            - --path
+            - /ws
+          ports:
+            - containerPort: 3000
+          readinessProbe:
+            tcpSocket:
+              port: 3000
+            initialDelaySeconds: 5
+            periodSeconds: 10
+            timeoutSeconds: 3
+          livenessProbe:
+            tcpSocket:
+              port: 3000
+            initialDelaySeconds: 15
+            periodSeconds: 20
+            timeoutSeconds: 3
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: playwright-service
+  namespace: edu-master
+spec:
+  selector:
+    app: edu-master-playwright
+  ports:
+    - name: ws
+      port: 3000
+      targetPort: 3000

edu_master/k8s/redis.yaml

@@ -0,0 +1,75 @@
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  name: redis
+  namespace: edu-master
+  labels:
+    app: edu-master-redis
+spec:
+  serviceName: redis
+  replicas: 1
+  selector:
+    matchLabels:
+      app: edu-master-redis
+  template:
+    metadata:
+      labels:
+        app: edu-master-redis
+    spec:
+      containers:
+        - name: redis
+          image: redis:8.10.2-alpine
+          imagePullPolicy: IfNotPresent
+          ports:
+            - containerPort: 6379
+          volumeMounts:
+            - name: redis-data
+              mountPath: /data
+          resources:
+            requests:
+              cpu: 25m
+              memory: 32Mi
+            limits:
+              cpu: 250m
+              memory: 128Mi
+          readinessProbe:
+            exec:
+              command: ["redis-cli", "ping"]
+            initialDelaySeconds: 5
+            periodSeconds: 5
+            timeoutSeconds: 3
+          livenessProbe:
+            exec:
+              command: ["redis-cli", "ping"]
+            initialDelaySeconds: 10
+            periodSeconds: 10
+            timeoutSeconds: 3
+      volumes:
+        - name: redis-data
+          persistentVolumeClaim:
+            claimName: redis-data-pvc
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: redis-data-pvc
+  namespace: edu-master
+spec:
+  accessModes:
+    - ReadWriteOnce
+  resources:
+    requests:
+      storage: 1Gi
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: redis
+  namespace: edu-master
+spec:
+  selector:
+    app: edu-master-redis
+  ports:
+    - name: redis
+      port: 6379
+      targetPort: 6379

edu_master/k8s/restore-seed-job.yaml.example

@@ -0,0 +1,50 @@
+# One-time Job to migrate redis state from docker compose to k8s (maintenance window).
+# The .example file is not applied by the deploy pipeline (mask *.example.yaml).
+#
+# Runbook:
+#   1. docker compose -f <repo>/edu_master/compose.yaml stop        # SIGTERM -> redis will flush dump.rdb
+#   2. docker run --rm -v edu_master_redis-data:/data \
+#        -v /tmp/edu-master-backup:/backup \
+#        redis:alpine sh -c "cp /data/dump.rdb /backup/ && ls -la /backup"
+#   3. kubectl apply -f edu_master/k8s/namespace.yaml
+#   4. kubectl apply -f <only the PVC from redis.yaml>              # seed must come BEFORE redis pod starts
+#   5. kubectl apply -f edu_master/k8s/restore-seed-job.yaml.example
+#      kubectl wait --for=condition=complete job/redis-restore-seed -n edu-master --timeout=120s
+#   6. kubectl delete job redis-restore-seed -n edu-master
+#   7. kubectl apply -f edu_master/k8s/ -R                           # apply remaining manifests
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: redis-restore-seed
+  namespace: edu-master
+spec:
+  backoffLimit: 2
+  ttlSecondsAfterFinished: 3600
+  template:
+    spec:
+      restartPolicy: Never
+      containers:
+        - name: seed
+          image: redis:alpine
+          command:
+            - /bin/sh
+            - -ec
+            - |
+              ls -la /backup
+              cp /backup/dump.rdb /data/dump.rdb
+              chmod 644 /data/dump.rdb
+              ls -la /data
+          volumeMounts:
+            - name: redis-data
+              mountPath: /data
+            - name: backup
+              mountPath: /backup
+              readOnly: true
+      volumes:
+        - name: redis-data
+          persistentVolumeClaim:
+            claimName: redis-data-pvc
+        - name: backup
+          hostPath:
+            path: /tmp/edu-master-backup
+            type: DirectoryOrCreate

edu_master/k8s/secrets.yaml.example

@@ -0,0 +1,29 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: edu-master-secrets
+  namespace: edu-master
+type: Opaque
+stringData:
+  # Session keeper credentials
+  KEEPER_LOGIN: ""
+  KEEPER_PASSWORD: ""
+  KEEPER_INTERVAL: "10"
+  # EDU links
+  EDU_URL_BASE: "https://edu.edu.vn.ua"
+  EDU_URL_LOGIN: "/user/login"
+  EDU_URL_COURSES: "/course/userlist"
+  EDU_URL_WEBINAR: "/webinar/useractive"
+  # Playwright
+  USER_AGENT: ""
+  PLAYWRIGHT_WS: "ws://playwright-service:3000/ws"
+  # Webinar-checker
+  WEBINAR_TELEGRAM_TOKEN: ""
+  WEBINAR_ADMIN_ID: ""
+  WEBINAR_CHECK_INTERVAL: "60"
+  # Prometheus metrics endpoint (scraped via ServiceMonitor, alerts in k8s/alerts.yaml)
+  METRICS_PORT: "8000"
+  # Database
+  REDIS_HOST: "redis"
+  REDIS_PORT: "6379"
+  TZ: "Europe/Kyiv"

edu_master/k8s/service.yaml

@@ -0,0 +1,15 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: webinar-checker
+  namespace: edu-master
+  labels:
+    app: edu-master-webinar-checker
+spec:
+  selector:
+    app: edu-master-webinar-checker
+  ports:
+    - name: metrics
+      port: 8000
+      targetPort: metrics
+      protocol: TCP

edu_master/k8s/servicemonitor.yaml

@@ -0,0 +1,16 @@
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: webinar-checker
+  namespace: edu-master
+  labels:
+    release: prometheus-stack
+spec:
+  selector:
+    matchLabels:
+      app: edu-master-webinar-checker
+  endpoints:
+    - port: metrics
+      path: /metrics
+      interval: 30s
+      scrapeTimeout: 10s

edu_master/k8s/session-keeper.yaml

@@ -0,0 +1,53 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: session-keeper
+  namespace: edu-master
+  labels:
+    app: edu-master-session-keeper
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: edu-master-session-keeper
+  strategy:
+    type: Recreate
+  template:
+    metadata:
+      labels:
+        app: edu-master-session-keeper
+    spec:
+      initContainers:
+        - name: wait-redis
+          image: redis:8.10.2-alpine
+          command:
+            - /bin/sh
+            - -ec
+            - |
+              i=0
+              until redis-cli -h redis ping | grep -q PONG; do
+                i=$((i+1))
+                [ "$i" -ge 300 ] && echo "TIMEOUT: redis not ready" && exit 1
+                sleep 2
+              done
+              echo "redis is ready"
+      containers:
+        - name: session-keeper
+          image: gcr.forust.xyz/forust/session-keeper:prod
+          envFrom:
+            - secretRef:
+                name: edu-master-secrets
+          resources:
+            requests:
+              cpu: 25m
+              memory: 32Mi
+            limits:
+              cpu: 250m
+              memory: 128Mi
+          readinessProbe:
+            exec:
+              command: ["/bin/sh", "-ec", "redis-cli -h redis EXISTS EDU_PHPSESSID | grep -q 1"]
+            initialDelaySeconds: 15
+            periodSeconds: 30
+            timeoutSeconds: 5
+            failureThreshold: 10

edu_master/k8s/webinar-checker.yaml

@@ -0,0 +1,75 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: webinar-checker
+  namespace: edu-master
+  labels:
+    app: edu-master-webinar-checker
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: edu-master-webinar-checker
+  strategy:
+    type: Recreate
+  template:
+    metadata:
+      labels:
+        app: edu-master-webinar-checker
+    spec:
+      # Enforces dependency order like compose depends_on:
+      # redis healthy -> session-keeper healthy (EXISTS EDU_PHPSESSID) -> playwright started
+      initContainers:
+        - name: wait-deps
+          image: redis:8.10.2-alpine
+          command:
+            - /bin/sh
+            - -ec
+            - |
+              i=0
+              until redis-cli -h redis ping | grep -q PONG; do
+                i=$((i+1))
+                [ "$i" -ge 300 ] && echo "TIMEOUT: redis not ready" && exit 1
+                sleep 2
+              done
+              echo "redis ok"
+              until [ "$(redis-cli -h redis EXISTS EDU_PHPSESSID)" = "1" ]; do
+                i=$((i+1))
+                [ "$i" -ge 300 ] && echo "TIMEOUT: no PHPSESSID (session-keeper down?)" && exit 1
+                sleep 2
+              done
+              echo "PHPSESSID ok"
+              until nc -z playwright-service 3000; do
+                i=$((i+1))
+                [ "$i" -ge 300 ] && echo "TIMEOUT: playwright-service not reachable" && exit 1
+                sleep 2
+              done
+              echo "playwright ok"
+      containers:
+        - name: webinar-checker
+          image: gcr.forust.xyz/forust/webinar-checker:prod
+          ports:
+            - name: metrics
+              containerPort: 8000
+              protocol: TCP
+          readinessProbe:
+            httpGet:
+              path: /health
+              port: metrics
+            periodSeconds: 10
+            timeoutSeconds: 3
+            failureThreshold: 12
+            initialDelaySeconds: 10
+          envFrom:
+            - secretRef:
+                name: edu-master-secrets
+          env:
+            - name: TZ
+              value: "Europe/Kyiv"
+          resources:
+            requests:
+              cpu: "50m"
+              memory: "192Mi"
+            limits:
+              cpu: "600m"
+              memory: "384Mi"

edu_master/phpsessid-bot/Dockerfile

@@ -0,0 +1,15 @@
+FROM python:3.11-slim
+
+WORKDIR /app
+
+# Install system dependencies
+RUN apt-get update && apt-get install -y --no-install-recommends redis-tools && rm -rf /var/lib/apt/lists/*
+
+# Install dependencies
+RUN pip install --no-cache-dir requests==2.32.3 redis==5.2.1
+
+# Copy application code
+COPY . .
+
+# Run the bot
+CMD ["python", "bot.py"]

edu_master/phpsessid-bot/bot.py

@@ -0,0 +1,121 @@
+import logging
+import os
+import time
+from datetime import datetime
+
+import redis
+import requests
+
+# Configure logging
+logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
+logger = logging.getLogger(__name__)
+
+
+# Load configuration (adapted to .env keys)
+def _env(key, default=None):
+    v = os.getenv(key, default)
+    if isinstance(v, str) and len(v) >= 2 and ((v[0] == '"' and v[-1] == '"') or (v[0] == "'" and v[-1] == "'")):
+        return v[1:-1]
+    return v
+
+
+LOGIN = _env('KEEPER_LOGIN')
+PASSWORD = _env('KEEPER_PASSWORD')
+
+EDU_BASE = _env('EDU_URL_BASE', 'https://edu.edu.vn.ua')
+EDU_LOGIN_PATH = _env('EDU_URL_LOGIN', '/user/login')
+EDU_COURSES_PATH = _env('EDU_URL_COURSES', '/course/userlist')
+URL_LOGIN = f'{EDU_BASE.rstrip("/")}/{EDU_LOGIN_PATH.lstrip("/")}'
+URL_VERIFY = f'{EDU_BASE.rstrip("/")}/{EDU_COURSES_PATH.lstrip("/")}'
+
+INTERVAL = int(_env('KEEPER_INTERVAL', 10))
+USER_AGENT = _env(
+    'USER_AGENT',
+    'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/142.0.0.0 Safari/537.36',
+)
+REDIS_HOST = _env('REDIS_HOST', 'redis')
+REDIS_PORT = int(_env('REDIS_PORT', 6379))
+
+SUCCESS_FILE = '/tmp/last_success'  # noqa: S108
+
+
+def touch_success_file():
+    """Updates the timestamp of the success file for healthchecks."""
+    try:
+        with open(SUCCESS_FILE, 'w') as f:
+            f.write(str(datetime.now().timestamp()))
+    except Exception as e:
+        logger.error(f'Failed to touch success file: {e}')
+
+
+def refresh_session(session, redis_client):
+    """Publish a verified cookie with a lifetime tied to the refresh interval."""
+    login_response = session.post(
+        URL_LOGIN, data={'login': LOGIN, 'password': PASSWORD}, allow_redirects=True, timeout=(10, 30)
+    )
+    login_response.raise_for_status()
+    verify_response = session.get(URL_VERIFY, allow_redirects=False, timeout=(10, 30))
+    if verify_response.status_code != 200:
+        logger.warning('Session verification failed (HTTP %s)', verify_response.status_code)
+        return False
+    phpsessid = session.cookies.get('PHPSESSID')
+    if not phpsessid:
+        logger.warning('Verified response did not provide a PHPSESSID cookie')
+        return False
+    redis_client.set('EDU_PHPSESSID', phpsessid, ex=INTERVAL * 120)
+    touch_success_file()
+    logger.info('Verified session saved to Redis')
+    return True
+
+
+def main():
+    if not LOGIN or not PASSWORD:
+        raise ValueError('KEEPER_LOGIN and KEEPER_PASSWORD must be set')
+    if INTERVAL <= 0:
+        raise ValueError('KEEPER_INTERVAL must be a positive number of minutes')
+    logger.info('Starting Session Keeper Bot')
+
+    # Connect to Redis
+    try:
+        redis_client = redis.Redis(
+            host=REDIS_HOST, port=REDIS_PORT, decode_responses=True, socket_connect_timeout=5, socket_timeout=5
+        )
+        redis_client.ping()
+        logger.info(f'Connected to Redis at {REDIS_HOST}:{REDIS_PORT}')
+    except Exception as e:
+        logger.error(f'Failed to connect to Redis: {e}')
+        raise
+
+    session = requests.Session()
+
+    # Set headers
+    headers = {
+        'User-Agent': USER_AGENT,
+        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',
+        'Accept-Language': 'en-US,en;q=0.9',
+        'Cache-Control': 'max-age=0',
+        'Upgrade-Insecure-Requests': '1',
+        'Sec-Fetch-Site': 'same-origin',
+        'Sec-Fetch-Mode': 'navigate',
+        'Sec-Fetch-User': '?1',
+        'Sec-Fetch-Dest': 'document',
+        'Sec-Ch-Ua': '"Not_A Brand";v="99", "Chromium";v="142"',
+        'Sec-Ch-Ua-Mobile': '?0',
+        'Sec-Ch-Ua-Platform': '"Linux"',
+        'Accept-Encoding': 'gzip, deflate, br',
+        'Priority': 'u=0, i',
+    }
+    session.headers.update(headers)
+
+    while True:
+        try:
+            refresh_session(session, redis_client)
+        except Exception as e:
+            logger.error(f'An error occurred: {e}')
+
+        logger.info(f'Sleeping for {INTERVAL} minutes...')
+        time.sleep(INTERVAL * 60)
+
+
+if __name__ == '__main__':
+    main()

edu_master/webinar-checker/Dockerfile

@@ -0,0 +1,13 @@
+FROM python:3.11-slim
+
+WORKDIR /app
+
+# renovate: datasource=pypi depName=playwright versioning=pep440
+ARG PLAYWRIGHT_VERSION=1.56.0
+
+# Install dependencies - PLAYWRIGHT_VERSION is single-source, renovate updates ARG above and all other places via regexManagers
+RUN pip install --no-cache-dir pip==25.0.1 && pip install --no-cache-dir playwright==${PLAYWRIGHT_VERSION} redis==5.2.1 requests==2.32.3 "python-telegram-bot[job-queue]==21.10"
+
+COPY checker.py .
+
+CMD ["python", "checker.py"]

gitea/k8s/gitea.yaml

@@ -17,8 +17,6 @@ spec:
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: gitea-deployment
   namespace: gitea
 spec:

glance/k8s/glance.yaml

@@ -13,8 +13,6 @@ spec:
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: glance-deployment
   namespace: glance
 spec:
@@ -71,7 +69,7 @@ spec:
             name: glance-config
         - name: glance-assets
           configMap:
-            name: glance-assets
+            name: glance-config
         - name: docker-socket
           hostPath:
             path: /var/run/docker.sock

homarr/compose.yaml

@@ -1,7 +1,7 @@
 services:
   homarr:
     container_name: homarr
-    image: ghcr.io/homarr-labs/homarr:v2.2.0
+    image: ghcr.io/homarr-labs/homarr:v2.1.2
     restart: unless-stopped
     volumes:
       - ./appdata:/appdata

homarr/k8s/homarr.yaml

@@ -13,8 +13,6 @@ spec:
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: homarr-deployment
   namespace: homarr
 spec:
@@ -32,7 +30,7 @@ spec:
       serviceAccountName: homarr
       containers:
         - name: homarr
-          image: ghcr.io/homarr-labs/homarr:v2.2.0
+          image: ghcr.io/homarr-labs/homarr:v2.1.2
           envFrom:
             - configMapRef:
                 name: homarr-config

immich/k8s/immich.yaml

@@ -14,8 +14,6 @@ spec:
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: immich-deployment
   namespace: immich
   labels:

immich/k8s/machine-learning.yaml

@@ -14,8 +14,6 @@ spec:
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: immich-machine-learning-deployment
   namespace: immich
   labels:

immich/k8s/valkey.yaml

@@ -17,8 +17,6 @@ spec:
 apiVersion: apps/v1
 kind: StatefulSet
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: immich-valkey
   namespace: immich
   labels:

kener/k8s/kener.yaml

@@ -13,8 +13,6 @@ spec:
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: kener-deployment
   namespace: kener
 spec:

metube/k8s/metube.yaml

@@ -13,8 +13,6 @@ spec:
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: metube-deployment
   namespace: metube
 spec:

n8n/compose.yaml

@@ -1,6 +1,6 @@
 services:
   n8n:
-    image: docker.n8n.io/n8nio/n8n:2.43.0
+    image: docker.n8n.io/n8nio/n8n:2.42.3
     container_name: n8n
     restart: unless-stopped
     environment:

n8n/k8s/n8n.yaml

@@ -13,8 +13,6 @@ spec:
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: n8n-deployment
   namespace: n8n
 spec:
@@ -31,7 +29,7 @@ spec:
     spec:
       containers:
         - name: n8n
-          image: docker.n8n.io/n8nio/n8n:2.43.0
+          image: docker.n8n.io/n8nio/n8n:2.42.3
           envFrom:
             - configMapRef:
                 name: n8n-config

netbird/entrypoint.sh

@@ -1,109 +0,0 @@
-#!/bin/sh
-set -eu
-
-umask 077
-
-TEMPLATE_PATH=/opt/netbird/config.template.yaml
-RENDERED_PATH=/run/netbird/config.yaml
-RELAY_SECRET_PATH=/run/secrets/relay_auth_secret
-ENCRYPTION_KEY_PATH=/run/secrets/datastore_encryption_key
-
-is_valid_proxy_subnet() {
-  candidate="$1"
-  case "$candidate" in
-    0.0.0.0/0)
-      return 1
-      ;;
-    */*)
-      address="${candidate%%/*}"
-      prefix="${candidate#*/}"
-      ;;
-    *)
-      return 1
-      ;;
-  esac
-
-  case "$prefix" in
-    0|[1-9]|[1-2][0-9]|3[0-2]) ;;
-    *)
-      return 1
-      ;;
-  esac
-
-  old_ifs="$IFS"
-  IFS=.
-  # shellcheck disable=SC2086
-  set -- $address
-  IFS="$old_ifs"
-  [ "$#" -eq 4 ] || return 1
-
-  for octet do
-    case "$octet" in
-      0|[1-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5]) ;;
-      *)
-        return 1
-        ;;
-    esac
-  done
-}
-
-read_secret() {
-  secret_path="$1"
-
-  if [ ! -r "$secret_path" ]; then
-    echo "Required secret is not readable: $secret_path" >&2
-    exit 1
-  fi
-
-  secret_value="$(cat "$secret_path")"
-  if [ -z "$secret_value" ]; then
-    echo "Required secret is empty: $secret_path" >&2
-    exit 1
-  fi
-
-  printf '%s' "$secret_value"
-}
-
-if [ -z "${NETBIRD_DOMAIN:-}" ]; then
-  echo "NETBIRD_DOMAIN must be set" >&2
-  exit 1
-fi
-
-case "$NETBIRD_DOMAIN" in
-  *[!A-Za-z0-9.-]*)
-    echo "NETBIRD_DOMAIN contains unsupported characters" >&2
-    exit 1
-    ;;
-esac
-
-if [ -z "${NETBIRD_PROXY_SUBNET:-}" ] || [ "$NETBIRD_PROXY_SUBNET" = "auto" ]; then
-  echo "NETBIRD_PROXY_SUBNET must be an explicit IPv4 CIDR; run netbird/setup.sh first" >&2
-  exit 1
-fi
-if ! is_valid_proxy_subnet "$NETBIRD_PROXY_SUBNET"; then
-  echo "NETBIRD_PROXY_SUBNET must be a non-default IPv4 CIDR, for example 172.20.0.0/16" >&2
-  exit 1
-fi
-
-if [ "$#" -ne 2 ] || [ "$1" != "--config" ] || [ "$2" != "$RENDERED_PATH" ]; then
-  echo "Expected: --config $RENDERED_PATH" >&2
-  exit 1
-fi
-
-relay_secret="$(read_secret "$RELAY_SECRET_PATH")"
-encryption_key="$(read_secret "$ENCRYPTION_KEY_PATH")"
-
-mkdir -p "$(dirname "$RENDERED_PATH")"
-sed \
-  -e "s|__NETBIRD_DOMAIN__|${NETBIRD_DOMAIN}|g" \
-  -e "s|__NETBIRD_AUTH_SECRET__|${relay_secret}|g" \
-  -e "s|__NETBIRD_ENCRYPTION_KEY__|${encryption_key}|g" \
-  -e "s|__NETBIRD_PROXY_SUBNET__|${NETBIRD_PROXY_SUBNET}|g" \
-  "$TEMPLATE_PATH" >"$RENDERED_PATH"
-
-if grep -q '__NETBIRD_' "$RENDERED_PATH"; then
-  echo "Rendered NetBird configuration still contains unresolved placeholders" >&2
-  exit 1
-fi
-
-exec /go/bin/netbird-server "$@"

netbird/k8s/netbird.yaml

@@ -32,8 +32,6 @@ spec:
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: netbird-server-deployment
   namespace: netbird
 spec:
@@ -128,8 +126,6 @@ spec:
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: netbird-dashboard-deployment
   namespace: netbird
 spec:

netbird/setup.sh

@@ -1,38 +0,0 @@
-#!/usr/bin/env bash
-# Prepare local Compose configuration without replacing existing credentials.
-set -euo pipefail
-cd "$(dirname "${BASH_SOURCE[0]}")"
-umask 077
-if [ ! -f .env ]; then
-  cp .env.example .env
-fi
-
-if grep -q '^NETBIRD_PROXY_SUBNET=auto$' .env; then
-  subnet="$(docker network inspect proxy --format '{{range .IPAM.Config}}{{println .Subnet}}{{end}}' | awk '/^[0-9]+\./ { print; exit }')"
-  if [ -z "$subnet" ]; then
-    echo "No IPv4 subnet found on the Docker proxy network. Set NETBIRD_PROXY_SUBNET in .env." >&2
-    exit 1
-  fi
-  # The detected value must be safe to substitute into the env file.
-  if [[ ! "$subnet" =~ ^[0-9.]+/[0-9]+$ ]]; then
-    echo "Unexpected Docker network subnet: $subnet" >&2
-    exit 1
-  fi
-  sed -i "s|^NETBIRD_PROXY_SUBNET=auto$|NETBIRD_PROXY_SUBNET=$subnet|" .env
-fi
-
-mkdir -p secrets
-chmod 700 secrets
-for name in relay-auth-secret datastore-encryption-key; do
-  path="secrets/$name"
-  if [ -e "$path" ]; then
-    if [ ! -s "$path" ]; then
-      echo "Existing secret is empty: $path. Restore it before continuing." >&2
-      exit 1
-    fi
-  else
-    openssl rand -base64 32 >"$path"
-  fi
-  chmod 600 "$path"
-done
-printf '%s\n' 'Local files are ready. Review .env, then run docker compose config --quiet.'

netbox/k8s/netbox.yaml

@@ -14,8 +14,6 @@ spec:
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: netbox-deployment
   namespace: netbox
   labels:
@@ -120,8 +118,6 @@ spec:
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: netbox-worker-deployment
   namespace: netbox
   labels:

netbox/k8s/valkey.yaml

@@ -17,8 +17,6 @@ spec:
 apiVersion: apps/v1
 kind: StatefulSet
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: netbox-valkey
   namespace: netbox
   labels:

netronome/compose.yaml

@@ -1,6 +1,6 @@
 services:
   netronome:
-    image: ghcr.io/autobrr/netronome:v0.16.0
+    image: ghcr.io/autobrr/netronome:v0.15.0
     restart: unless-stopped
     container_name: netronome
     ports:

netronome/k8s/netronome.yaml

@@ -14,8 +14,6 @@ spec:
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: netronome-deployment
   namespace: netronome
   labels:
@@ -34,7 +32,7 @@ spec:
     spec:
       containers:
         - name: netronome
-          image: ghcr.io/autobrr/netronome:v0.16.0
+          image: ghcr.io/autobrr/netronome:v0.15.0
           ports:
             - name: netronome-port
               protocol: TCP

postgres/.env.example

@@ -1,7 +1,6 @@
 POSTGRES_ADMIN_PASSWORD=
 AUTHENTIK_DB_PASSWORD=
 GITEA_DB_PASSWORD=
-NETBOX_DB_PASSWORD=
 NETRONOME_DB_PASSWORD=
 PENPOT_DB_PASSWORD=
 STATUSPAGE_DB_PASSWORD=

prometheus-stack/k8s/README.md

@@ -1,17 +0,0 @@
-# VictoriaMetrics
-
-The `victoria-operator` Helm release converts Prometheus Operator
-`ServiceMonitor` resources into owned `VMServiceScrape` resources. The
-`VMAgent` selects converted scrapes labeled `release: prometheus-stack` in all
-namespaces and writes them to the existing single-node VictoriaMetrics
-instance. Changes to selected `ServiceMonitor` resources are reconciled
-automatically; there is no copied Prometheus scrape-config blob to regenerate.
-
-The agent drops targets for the Prometheus server service to avoid duplicating
-its self-scrape. `scraper: victoria` identifies the samples ingested by this
-VMAgent.
-
-The VictoriaMetrics Operator chart and its CRDs are installed before the
-Kubernetes manifests by the normal deploy workflow. On a cluster where the
-operator CRDs are not installed yet, CI skips the server-side dry-run of the
-`VMAgent` resource; the deploy installs the chart before applying that resource.

prometheus-stack/k8s/grafana-values.yaml

@@ -38,8 +38,6 @@ grafana:
 
   # One block covers both the dashboards and datasources sidecars (p95 91M / 80M).
   sidecar:
-    datasources:
-      defaultDatasourceEnabled: false
     resources:
       requests:
         memory: "96Mi"
@@ -52,18 +50,9 @@ grafana:
       type: loki
       url: http://loki-gateway.prometheus.svc.cluster.local
       access: proxy
-    - name: VictoriaMetrics
-      type: prometheus
-      url: http://victoria-metrics.prometheus.svc.cluster.local:8428
-      access: proxy
-      isDefault: true
 
 prometheus:
   prometheusSpec:
-    # VM trial: vmagent scrapes and remote-writes to VictoriaMetrics, so the
-    # Prometheus server itself stands down. Encoded here (not a kubectl patch)
-    # so helm keeps owning spec.replicas and upgrades do not conflict on it.
-    replicas: 0
     retention: 60d
     retentionSize: 32GB
     storageSpec:

prometheus-stack/k8s/ingress.yaml

@@ -31,105 +31,3 @@ spec:
           port: 80
   tls:
     secretName: internal-wildcard-tls
----
-apiVersion: traefik.io/v1alpha1
-kind: IngressRoute
-metadata:
-  name: prometheus-local
-  namespace: prometheus
-spec:
-  entryPoints:
-    - websecure
-  routes:
-    - match: Host(`prom.workstation.internal`) || Host(`prom.gigaforust.internal`)
-      kind: Rule
-      services:
-        - name: prometheus-stack-kube-prom-prometheus
-          port: 9090
-  tls:
-    secretName: internal-wildcard-tls
----
-apiVersion: traefik.io/v1alpha1
-kind: IngressRoute
-metadata:
-  name: alertmanager-local
-  namespace: prometheus
-spec:
-  entryPoints:
-    - websecure
-  routes:
-    - match: Host(`am.workstation.internal`) || Host(`am.gigaforust.internal`)
-      kind: Rule
-      services:
-        - name: prometheus-stack-kube-prom-alertmanager
-          port: 9093
-  tls:
-    secretName: internal-wildcard-tls
----
-apiVersion: traefik.io/v1alpha1
-kind: IngressRoute
-metadata:
-  name: loki-local
-  namespace: prometheus
-spec:
-  entryPoints:
-    - websecure
-  routes:
-    - match: Host(`loki.workstation.internal`) || Host(`loki.gigaforust.internal`)
-      kind: Rule
-      services:
-        - name: loki-gateway
-          port: 80
-  tls:
-    secretName: internal-wildcard-tls
----
-apiVersion: traefik.io/v1alpha1
-kind: IngressRoute
-metadata:
-  name: alloy-local
-  namespace: prometheus
-spec:
-  entryPoints:
-    - websecure
-  routes:
-    - match: Host(`alloy.workstation.internal`) || Host(`alloy.gigaforust.internal`)
-      kind: Rule
-      services:
-        - name: alloy
-          port: 12345
-  tls:
-    secretName: internal-wildcard-tls
----
-apiVersion: traefik.io/v1alpha1
-kind: IngressRoute
-metadata:
-  name: victoria-local
-  namespace: prometheus
-spec:
-  entryPoints:
-    - websecure
-  routes:
-    - match: Host(`victoria.workstation.internal`) || Host(`victoria.gigaforust.internal`)
-      kind: Rule
-      services:
-        - name: victoria-metrics
-          port: 8428
-  tls:
-    secretName: internal-wildcard-tls
----
-apiVersion: traefik.io/v1alpha1
-kind: IngressRoute
-metadata:
-  name: vmalert-local
-  namespace: prometheus
-spec:
-  entryPoints:
-    - websecure
-  routes:
-    - match: Host(`vmalert.workstation.internal`) || Host(`vmalert.gigaforust.internal`)
-      kind: Rule
-      services:
-        - name: vmalert
-          port: 8880
-  tls:
-    secretName: internal-wildcard-tls

prometheus-stack/k8s/victoria-operator-values.yaml

@@ -1,12 +0,0 @@
-nameOverride: victoria-operator
-
-operator:
-  enable_converter_ownership: true
-
-resources:
-  requests:
-    cpu: 50m
-    memory: 96Mi
-  limits:
-    cpu: 200m
-    memory: 256Mi

prometheus-stack/k8s/victoria.yaml

@@ -1,79 +0,0 @@
-apiVersion: v1
-kind: Service
-metadata:
-  name: victoria-metrics
-  namespace: prometheus
-spec:
-  selector:
-    app: victoria-metrics
-  ports:
-    - port: 8428
-      targetPort: 8428
----
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: victoria-pvc
-  namespace: prometheus
-spec:
-  resources:
-    requests:
-      storage: 10Gi
-  volumeMode: Filesystem
-  accessModes:
-    - ReadWriteOnce
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: victoria-deployment
-  namespace: prometheus
-spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      app: victoria-metrics
-  strategy:
-    type: Recreate
-  template:
-    metadata:
-      labels:
-        app: victoria-metrics
-    spec:
-      containers:
-        - name: victoria
-          image: victoriametrics/victoria-metrics:v1.153.0-scratch
-          args:
-            - -storageDataPath=/vmdata
-            - -retentionPeriod=30d
-            - -httpListenAddr=:8428
-          ports:
-            - containerPort: 8428
-          readinessProbe:
-            httpGet:
-              path: /health
-              port: 8428
-            initialDelaySeconds: 15
-            periodSeconds: 10
-            failureThreshold: 6
-          livenessProbe:
-            httpGet:
-              path: /health
-              port: 8428
-            initialDelaySeconds: 60
-            periodSeconds: 30
-            failureThreshold: 3
-          volumeMounts:
-            - name: vmdata
-              mountPath: /vmdata
-          resources:
-            requests:
-              cpu: "100m"
-              memory: "256Mi"
-            limits:
-              cpu: "1000m"
-              memory: "1Gi"
-      volumes:
-        - name: vmdata
-          persistentVolumeClaim:
-            claimName: victoria-pvc

prometheus-stack/k8s/vmagent.yaml

@@ -1,29 +0,0 @@
-apiVersion: operator.victoriametrics.com/v1beta1
-kind: VMAgent
-metadata:
-  name: vmagent
-  namespace: prometheus
-spec:
-  image:
-    tag: v1.153.0
-  scrapeInterval: 30s
-  externalLabels:
-    scraper: victoria
-  serviceScrapeNamespaceSelector: {}
-  serviceScrapeSelector:
-    matchLabels:
-      release: prometheus-stack
-  globalScrapeRelabelConfigs:
-    - action: drop
-      source_labels:
-        - __meta_kubernetes_service_name
-      regex: prometheus-stack-kube-prom-prometheus
-  remoteWrite:
-    - url: http://victoria-metrics.prometheus.svc.cluster.local:8428/api/v1/write
-  resources:
-    requests:
-      cpu: 100m
-      memory: 256Mi
-    limits:
-      cpu: "1000m"
-      memory: 1Gi

prometheus-stack/k8s/vmalert.yaml

@@ -1,70 +0,0 @@
-apiVersion: v1
-kind: Service
-metadata:
-  name: vmalert
-  namespace: prometheus
-spec:
-  selector:
-    app: vmalert
-  ports:
-    - port: 8880
-      targetPort: 8880
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: vmalert-deployment
-  namespace: prometheus
-spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      app: vmalert
-  strategy:
-    type: Recreate
-  template:
-    metadata:
-      labels:
-        app: vmalert
-    spec:
-      containers:
-        - name: vmalert
-          image: victoriametrics/vmalert:v1.153.0
-          args:
-            - -datasource.url=http://victoria-metrics.prometheus.svc.cluster.local:8428
-            - -remoteWrite.url=http://victoria-metrics.prometheus.svc.cluster.local:8428
-            - -notifier.url=http://prometheus-stack-kube-prom-alertmanager.prometheus.svc.cluster.local:9093
-            - -rule=/etc/vm/rules/*.yaml
-            - -evaluationInterval=60s
-            - -httpListenAddr=:8880
-          ports:
-            - containerPort: 8880
-          readinessProbe:
-            httpGet:
-              path: /metrics
-              port: 8880
-            initialDelaySeconds: 15
-            periodSeconds: 10
-            failureThreshold: 6
-          livenessProbe:
-            httpGet:
-              path: /metrics
-              port: 8880
-            initialDelaySeconds: 60
-            periodSeconds: 30
-            failureThreshold: 3
-          volumeMounts:
-            - name: rules
-              mountPath: /etc/vm/rules
-              readOnly: true
-          resources:
-            requests:
-              cpu: "50m"
-              memory: "64Mi"
-            limits:
-              cpu: "200m"
-              memory: "256Mi"
-      volumes:
-        - name: rules
-          configMap:
-            name: prometheus-prometheus-stack-kube-prom-prometheus-rulefiles-0

reloader/k8s/reloader-values.yaml

@@ -1,17 +1,11 @@
 # Pinned chart: stakater/reloader 2.2.17 (app v1.4.22).
 # Deployed by the deploy workflow, namespace reloader.
 # Restarts pods when a ConfigMap or Secret they consume changes. Opt-in per workload
-# via the reloader.stakater.com/auto: "true" workload annotation; watchGlobally because
+# via the reloader.stakater.com/auto: "true" pod annotation; watchGlobally because
 # the workloads that need it are spread across a few dozen namespaces.
 
 reloader:
   watchGlobally: true
-  # Only opted-in workloads are restarted. Keep scheduled jobs on their schedule.
-  autoReloadAll: false
-  ignoreJobs: true
-  ignoreCronJobs: true
-  # Change pod-template annotations rather than injecting STAKATER_* env vars.
-  reloadStrategy: annotations
 
   deployment:
     replicas: 1

renovate/k8s/configmap.yaml

@@ -19,9 +19,6 @@ data:
       "dependencyDashboard": true,
       "prCreation": "immediate",
       "labels": ["dependencies", "automated"],
-      "docker-compose": {
-        "managerFilePatterns": ["renovate/renovate-compose.yaml"]
-      },
       "helm-values": {
         "managerFilePatterns": ["/k8s/.+values\\.ya?ml$/"]
       },
@@ -29,28 +26,35 @@ data:
         "managerFilePatterns": ["/k8s/.+\\.ya?ml$/"]
       },
       "customManagers": [
+        {
+          "customType": "regex",
+          "description": "singlesource: playwright npm version pinned in npx command (k8s + compose)",
+          "managerFilePatterns": ["^edu_master/k8s/playwright\\.yaml$", "^edu_master/compose\\.yaml$"],
+          "matchStrings": ["playwright@(?<currentValue>\\d+\\.\\d+\\.\\d+)"],
+          "datasourceTemplate": "npm",
+          "depNameTemplate": "playwright"
+        },
+        {
+          "customType": "regex",
+          "description": "singlesource: PLAYWRIGHT_VERSION file",
+          "managerFilePatterns": ["^edu_master/PLAYWRIGHT_VERSION$"],
+          "matchStrings": ["^(?<currentValue>\\d+\\.\\d+\\.\\d+)$"],
+          "datasourceTemplate": "pypi",
+          "depNameTemplate": "playwright"
+        },
         {
           "customType": "regex",
           "description": "kube-prometheus-stack chart version pinned in the deploy workflow",
-          "managerFilePatterns": [".gitea/workflows/deploy-lib.sh"],
+          "managerFilePatterns": ["^\\.gitea/workflows/deploy-lib\\.sh$"],
           "matchStrings": ["\\|prometheus-community/kube-prometheus-stack\\|prometheus\\|(?<currentValue>[0-9.]+)\\|"],
           "datasourceTemplate": "helm",
           "depNameTemplate": "kube-prometheus-stack",
           "registryUrlTemplate": "https://prometheus-community.github.io/helm-charts"
         },
-        {
-          "customType": "regex",
-          "description": "VictoriaMetrics Operator chart version pinned in the deploy workflow",
-          "managerFilePatterns": [".gitea/workflows/deploy-lib.sh"],
-          "matchStrings": ["\\|victoriametrics/victoria-metrics-operator\\|prometheus\\|(?<currentValue>[0-9.]+)\\|"],
-          "datasourceTemplate": "helm",
-          "depNameTemplate": "victoria-metrics-operator",
-          "registryUrlTemplate": "https://victoriametrics.github.io/helm-charts"
-        },
         {
           "customType": "regex",
           "description": "grafana/loki chart version pinned in the deploy workflow",
-          "managerFilePatterns": [".gitea/workflows/deploy-lib.sh"],
+          "managerFilePatterns": ["^\\.gitea/workflows/deploy-lib\\.sh$"],
           "matchStrings": ["\\|grafana/loki\\|prometheus\\|(?<currentValue>[0-9.]+)\\|"],
           "datasourceTemplate": "helm",
           "depNameTemplate": "loki",
@@ -59,7 +63,7 @@ data:
         {
           "customType": "regex",
           "description": "grafana/alloy chart version pinned in the deploy workflow",
-          "managerFilePatterns": [".gitea/workflows/deploy-lib.sh"],
+          "managerFilePatterns": ["^\\.gitea/workflows/deploy-lib\\.sh$"],
           "matchStrings": ["\\|grafana/alloy\\|prometheus\\|(?<currentValue>[0-9.]+)\\|"],
           "datasourceTemplate": "helm",
           "depNameTemplate": "alloy",
@@ -68,7 +72,7 @@ data:
         {
           "customType": "regex",
           "description": "actionlint version used by the ci workflow",
-          "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
+          "managerFilePatterns": ["^\\.gitea/workflows/tool-versions\\.env$"],
           "matchStrings": ["(?:^|\\n)ACTIONLINT_VERSION=\"(?<currentValue>[0-9.]+)\""],
           "datasourceTemplate": "github-tags",
           "depNameTemplate": "rhysd/actionlint"
@@ -76,7 +80,7 @@ data:
         {
           "customType": "regex",
           "description": "shellcheck version used by the ci workflow",
-          "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
+          "managerFilePatterns": ["^\\.gitea/workflows/tool-versions\\.env$"],
           "matchStrings": ["(?:^|\\n)SHELLCHECK_VERSION=\"(?<currentValue>[0-9.]+)\""],
           "datasourceTemplate": "github-tags",
           "depNameTemplate": "koalaman/shellcheck"
@@ -84,7 +88,7 @@ data:
         {
           "customType": "regex",
           "description": "kubeconform version used by the ci workflow",
-          "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
+          "managerFilePatterns": ["^\\.gitea/workflows/tool-versions\\.env$"],
           "matchStrings": ["(?:^|\\n)KUBECONFORM_VERSION=\"(?<currentValue>[0-9.]+)\""],
           "datasourceTemplate": "github-tags",
           "depNameTemplate": "yannh/kubeconform"
@@ -92,7 +96,7 @@ data:
         {
           "customType": "regex",
           "description": "uv version used to build the pytest venv",
-          "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
+          "managerFilePatterns": ["^\\.gitea/workflows/tool-versions\\.env$"],
           "matchStrings": ["(?:^|\\n)UV_VERSION=\"(?<currentValue>[0-9.]+)\""],
           "datasourceTemplate": "github-tags",
           "depNameTemplate": "astral-sh/uv"
@@ -100,7 +104,7 @@ data:
         {
           "customType": "regex",
           "description": "prettier version used by the ci workflow",
-          "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
+          "managerFilePatterns": ["^\\.gitea/workflows/tool-versions\\.env$"],
           "matchStrings": ["(?:^|\\n)PRETTIER_VERSION=\"(?<currentValue>[0-9.]+)\""],
           "datasourceTemplate": "npm",
           "depNameTemplate": "prettier"
@@ -108,7 +112,7 @@ data:
         {
           "customType": "regex",
           "description": "ruff version used by the ci workflow",
-          "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
+          "managerFilePatterns": ["^\\.gitea/workflows/tool-versions\\.env$"],
           "matchStrings": ["(?:^|\\n)RUFF_VERSION=\"(?<currentValue>[0-9.]+)\""],
           "datasourceTemplate": "pypi",
           "depNameTemplate": "ruff"
@@ -116,7 +120,7 @@ data:
         {
           "customType": "regex",
           "description": "pip-audit version used by the ci workflow",
-          "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
+          "managerFilePatterns": ["^\\.gitea/workflows/tool-versions\\.env$"],
           "matchStrings": ["(?:^|\\n)PIP_AUDIT_VERSION=\"(?<currentValue>[0-9.]+)\""],
           "datasourceTemplate": "pypi",
           "depNameTemplate": "pip-audit"
@@ -124,7 +128,7 @@ data:
         {
           "customType": "regex",
           "description": "yamllint version used by the ci workflow",
-          "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
+          "managerFilePatterns": ["^\\.gitea/workflows/tool-versions\\.env$"],
           "matchStrings": ["(?:^|\\n)YAMLLINT_VERSION=\"(?<currentValue>[0-9.]+)\""],
           "datasourceTemplate": "pypi",
           "depNameTemplate": "yamllint"
@@ -132,7 +136,7 @@ data:
         {
           "customType": "regex",
           "description": "hadolint version used by the ci workflow",
-          "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
+          "managerFilePatterns": ["^\\.gitea/workflows/tool-versions\\.env$"],
           "matchStrings": ["(?:^|\\n)HADOLINT_VERSION=\"(?<currentValue>[0-9.]+)\""],
           "datasourceTemplate": "github-tags",
           "depNameTemplate": "hadolint/hadolint"
@@ -140,7 +144,7 @@ data:
         {
           "customType": "regex",
           "description": "node version the ci workflow runs npm with",
-          "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
+          "managerFilePatterns": ["^\\.gitea/workflows/tool-versions\\.env$"],
           "matchStrings": ["(?:^|\\n)NODE_VERSION=\"(?<currentValue>[0-9.]+)\""],
           "datasourceTemplate": "node",
           "depNameTemplate": "node"
@@ -148,23 +152,16 @@ data:
         {
           "customType": "regex",
           "description": "stakater/reloader chart version pinned in the deploy workflow",
-          "managerFilePatterns": [".gitea/workflows/deploy-lib.sh"],
+          "managerFilePatterns": ["^\\.gitea/workflows/deploy-lib\\.sh$"],
           "matchStrings": ["\\|stakater/reloader\\|reloader\\|(?<currentValue>[0-9.]+)\\|"],
           "datasourceTemplate": "helm",
           "depNameTemplate": "reloader",
           "registryUrlTemplate": "https://stakater.github.io/stakater-charts"
-        },
-        {
-          "customType": "regex",
-          "description": "Pinned CI BuildKit helper image",
-          "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
-          "matchStrings": ["BUILDKIT_IMAGE=\"(?<depName>moby/buildkit):(?<currentValue>[^\"\\n]+)\""],
-          "datasourceTemplate": "docker"
         }
       ],
       "packageRules": [
         {
-          "description": "Automerge ordinary digest and patch updates after successful checks; specific manual-review rules below override this.",
+          "description": "Automerge digest and patch updates - safe by definition, review adds nothing, keeps the renovate queue and the deploy line short. Specific no-automerge rules below still override this for playwright, helm and majors.",
           "matchUpdateTypes": ["digest", "patch"],
           "automerge": true
         },
@@ -175,18 +172,23 @@ data:
           "groupSlug": "all-minor",
           "automerge": false
         },
-        {
-          "description": "Group ordinary patch updates; the specific groups and manual-review rules below take precedence",
-          "matchUpdateTypes": ["patch"],
-          "groupName": "all patch updates",
-          "groupSlug": "all-patch"
-        },
         {
           "description": "Keep private homelab images unchanged",
           "matchDatasources": ["docker"],
           "matchPackageNames": ["/gcr\\.forust\\.xyz\\/forust\\/.+/"],
           "enabled": false
         },
+        {
+          "description": "singlesource playwright - use whichever version is found, keep docker+pypi+npm in sync",
+          "matchPackageNames": ["playwright", "mcr.microsoft.com/playwright"],
+          "groupName": "playwright singlesource",
+          "groupSlug": "playwright"
+        },
+        {
+          "description": "playwright must not automerge - version skew breaks the WS handshake (checker.py:1523 vs playwright.yaml:20)",
+          "matchPackageNames": ["playwright", "mcr.microsoft.com/playwright"],
+          "automerge": false
+        },
         {
           "description": "Renovate updates itself in lockstep across the CronJob and the Compose file",
           "matchPackageNames": ["renovate/renovate"],
@@ -194,7 +196,7 @@ data:
           "automerge": false
         },
         {
-          "description": "Keep CI Node runtime updates in a separate, manually reviewed group",
+          "description": "CI runs npm on the node the panel image is built from - the NODE_VERSION pin in tool-versions.env and node:22-alpine in the Dockerfile are the same dependency and move as one",
           "matchPackageNames": ["node"],
           "groupName": "node runtime",
           "groupSlug": "node",
@@ -203,8 +205,6 @@ data:
         {
           "description": "Helm chart bumps change PVC fields and admission behaviour, keep them reviewable",
           "matchDatasources": ["helm"],
-          "groupName": "Helm chart {{depName}}",
-          "groupSlug": "helm-{{depName}}",
           "automerge": false
         },
         {
@@ -213,6 +213,12 @@ data:
           "dependencyDashboardApproval": true,
           "automerge": false
         },
+        {
+          "description": "Group patch updates from all sources - automerge still applies via the digest/patch rule above (helm/playwright stay manual via their own rules)",
+          "matchUpdateTypes": ["patch"],
+          "groupName": "all patch updates",
+          "groupSlug": "all-patch"
+        },
         {
           "description": "Python Y-bumps break compat (3.11->3.12->3.13->3.14) - keep the base image out of the shared minor/patch groups, review every bump separately. Placed last so its groupName wins.",
           "matchDatasources": ["docker"],

renovate/k8s/cronjob.yaml

@@ -19,7 +19,7 @@ spec:
           restartPolicy: Never
           containers:
             - name: renovate
-              image: renovate/renovate:44.140.0
+              image: renovate/renovate:44.136.0
               env:
                 - name: RENOVATE_PLATFORM
                   value: gitea

renovate/renovate-compose.yaml

@@ -2,7 +2,7 @@ services:
   renovate:
     # Kept in step with renovate/k8s/cronjob.yaml by the "renovate self-update"
     # package rule in renovate/renovate.json.
-    image: renovate/renovate:44.136.0
+    image: renovate/renovate:44.115.9
     container_name: renovate
     restart: "no"
     env_file:

renovate/renovate.json

@@ -8,9 +8,6 @@
   "dependencyDashboard": true,
   "prCreation": "immediate",
   "labels": ["dependencies", "automated"],
-  "docker-compose": {
-    "managerFilePatterns": ["renovate/renovate-compose.yaml"]
-  },
   "helm-values": {
     "managerFilePatterns": ["/k8s/.+values\\.ya?ml$/"]
   },
@@ -18,28 +15,35 @@
     "managerFilePatterns": ["/k8s/.+\\.ya?ml$/"]
   },
   "customManagers": [
+    {
+      "customType": "regex",
+      "description": "singlesource: playwright npm version pinned in npx command (k8s + compose)",
+      "managerFilePatterns": ["^edu_master/k8s/playwright\\.yaml$", "^edu_master/compose\\.yaml$"],
+      "matchStrings": ["playwright@(?<currentValue>\\d+\\.\\d+\\.\\d+)"],
+      "datasourceTemplate": "npm",
+      "depNameTemplate": "playwright"
+    },
+    {
+      "customType": "regex",
+      "description": "singlesource: PLAYWRIGHT_VERSION file",
+      "managerFilePatterns": ["^edu_master/PLAYWRIGHT_VERSION$"],
+      "matchStrings": ["^(?<currentValue>\\d+\\.\\d+\\.\\d+)$"],
+      "datasourceTemplate": "pypi",
+      "depNameTemplate": "playwright"
+    },
     {
       "customType": "regex",
       "description": "kube-prometheus-stack chart version pinned in the deploy workflow",
-      "managerFilePatterns": [".gitea/workflows/deploy-lib.sh"],
+      "managerFilePatterns": ["^\\.gitea/workflows/deploy-lib\\.sh$"],
       "matchStrings": ["\\|prometheus-community/kube-prometheus-stack\\|prometheus\\|(?<currentValue>[0-9.]+)\\|"],
       "datasourceTemplate": "helm",
       "depNameTemplate": "kube-prometheus-stack",
       "registryUrlTemplate": "https://prometheus-community.github.io/helm-charts"
     },
-    {
-      "customType": "regex",
-      "description": "VictoriaMetrics Operator chart version pinned in the deploy workflow",
-      "managerFilePatterns": [".gitea/workflows/deploy-lib.sh"],
-      "matchStrings": ["\\|victoriametrics/victoria-metrics-operator\\|prometheus\\|(?<currentValue>[0-9.]+)\\|"],
-      "datasourceTemplate": "helm",
-      "depNameTemplate": "victoria-metrics-operator",
-      "registryUrlTemplate": "https://victoriametrics.github.io/helm-charts"
-    },
     {
       "customType": "regex",
       "description": "grafana/loki chart version pinned in the deploy workflow",
-      "managerFilePatterns": [".gitea/workflows/deploy-lib.sh"],
+      "managerFilePatterns": ["^\\.gitea/workflows/deploy-lib\\.sh$"],
       "matchStrings": ["\\|grafana/loki\\|prometheus\\|(?<currentValue>[0-9.]+)\\|"],
       "datasourceTemplate": "helm",
       "depNameTemplate": "loki",
@@ -48,7 +52,7 @@
     {
       "customType": "regex",
       "description": "grafana/alloy chart version pinned in the deploy workflow",
-      "managerFilePatterns": [".gitea/workflows/deploy-lib.sh"],
+      "managerFilePatterns": ["^\\.gitea/workflows/deploy-lib\\.sh$"],
       "matchStrings": ["\\|grafana/alloy\\|prometheus\\|(?<currentValue>[0-9.]+)\\|"],
       "datasourceTemplate": "helm",
       "depNameTemplate": "alloy",
@@ -57,7 +61,7 @@
     {
       "customType": "regex",
       "description": "actionlint version used by the ci workflow",
-      "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
+      "managerFilePatterns": ["^\\.gitea/workflows/tool-versions\\.env$"],
       "matchStrings": ["(?:^|\\n)ACTIONLINT_VERSION=\"(?<currentValue>[0-9.]+)\""],
       "datasourceTemplate": "github-tags",
       "depNameTemplate": "rhysd/actionlint"
@@ -65,7 +69,7 @@
     {
       "customType": "regex",
       "description": "shellcheck version used by the ci workflow",
-      "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
+      "managerFilePatterns": ["^\\.gitea/workflows/tool-versions\\.env$"],
       "matchStrings": ["(?:^|\\n)SHELLCHECK_VERSION=\"(?<currentValue>[0-9.]+)\""],
       "datasourceTemplate": "github-tags",
       "depNameTemplate": "koalaman/shellcheck"
@@ -73,7 +77,7 @@
     {
       "customType": "regex",
       "description": "kubeconform version used by the ci workflow",
-      "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
+      "managerFilePatterns": ["^\\.gitea/workflows/tool-versions\\.env$"],
       "matchStrings": ["(?:^|\\n)KUBECONFORM_VERSION=\"(?<currentValue>[0-9.]+)\""],
       "datasourceTemplate": "github-tags",
       "depNameTemplate": "yannh/kubeconform"
@@ -81,7 +85,7 @@
     {
       "customType": "regex",
       "description": "uv version used to build the pytest venv",
-      "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
+      "managerFilePatterns": ["^\\.gitea/workflows/tool-versions\\.env$"],
       "matchStrings": ["(?:^|\\n)UV_VERSION=\"(?<currentValue>[0-9.]+)\""],
       "datasourceTemplate": "github-tags",
       "depNameTemplate": "astral-sh/uv"
@@ -89,7 +93,7 @@
     {
       "customType": "regex",
       "description": "prettier version used by the ci workflow",
-      "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
+      "managerFilePatterns": ["^\\.gitea/workflows/tool-versions\\.env$"],
       "matchStrings": ["(?:^|\\n)PRETTIER_VERSION=\"(?<currentValue>[0-9.]+)\""],
       "datasourceTemplate": "npm",
       "depNameTemplate": "prettier"
@@ -97,7 +101,7 @@
     {
       "customType": "regex",
       "description": "ruff version used by the ci workflow",
-      "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
+      "managerFilePatterns": ["^\\.gitea/workflows/tool-versions\\.env$"],
       "matchStrings": ["(?:^|\\n)RUFF_VERSION=\"(?<currentValue>[0-9.]+)\""],
       "datasourceTemplate": "pypi",
       "depNameTemplate": "ruff"
@@ -105,7 +109,7 @@
     {
       "customType": "regex",
       "description": "pip-audit version used by the ci workflow",
-      "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
+      "managerFilePatterns": ["^\\.gitea/workflows/tool-versions\\.env$"],
       "matchStrings": ["(?:^|\\n)PIP_AUDIT_VERSION=\"(?<currentValue>[0-9.]+)\""],
       "datasourceTemplate": "pypi",
       "depNameTemplate": "pip-audit"
@@ -113,7 +117,7 @@
     {
       "customType": "regex",
       "description": "yamllint version used by the ci workflow",
-      "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
+      "managerFilePatterns": ["^\\.gitea/workflows/tool-versions\\.env$"],
       "matchStrings": ["(?:^|\\n)YAMLLINT_VERSION=\"(?<currentValue>[0-9.]+)\""],
       "datasourceTemplate": "pypi",
       "depNameTemplate": "yamllint"
@@ -121,7 +125,7 @@
     {
       "customType": "regex",
       "description": "hadolint version used by the ci workflow",
-      "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
+      "managerFilePatterns": ["^\\.gitea/workflows/tool-versions\\.env$"],
       "matchStrings": ["(?:^|\\n)HADOLINT_VERSION=\"(?<currentValue>[0-9.]+)\""],
       "datasourceTemplate": "github-tags",
       "depNameTemplate": "hadolint/hadolint"
@@ -129,7 +133,7 @@
     {
       "customType": "regex",
       "description": "node version the ci workflow runs npm with",
-      "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
+      "managerFilePatterns": ["^\\.gitea/workflows/tool-versions\\.env$"],
       "matchStrings": ["(?:^|\\n)NODE_VERSION=\"(?<currentValue>[0-9.]+)\""],
       "datasourceTemplate": "node",
       "depNameTemplate": "node"
@@ -137,23 +141,16 @@
     {
       "customType": "regex",
       "description": "stakater/reloader chart version pinned in the deploy workflow",
-      "managerFilePatterns": [".gitea/workflows/deploy-lib.sh"],
+      "managerFilePatterns": ["^\\.gitea/workflows/deploy-lib\\.sh$"],
       "matchStrings": ["\\|stakater/reloader\\|reloader\\|(?<currentValue>[0-9.]+)\\|"],
       "datasourceTemplate": "helm",
       "depNameTemplate": "reloader",
       "registryUrlTemplate": "https://stakater.github.io/stakater-charts"
-    },
-    {
-      "customType": "regex",
-      "description": "Pinned CI BuildKit helper image",
-      "managerFilePatterns": [".gitea/workflows/tool-versions.env"],
-      "matchStrings": ["BUILDKIT_IMAGE=\"(?<depName>moby/buildkit):(?<currentValue>[^\"\\n]+)\""],
-      "datasourceTemplate": "docker"
     }
   ],
   "packageRules": [
     {
-      "description": "Automerge ordinary digest and patch updates after successful checks; specific manual-review rules below override this.",
+      "description": "Automerge digest and patch updates - safe by definition, review adds nothing, keeps the renovate queue and the deploy line short. Specific no-automerge rules below still override this for playwright, helm and majors.",
       "matchUpdateTypes": ["digest", "patch"],
       "automerge": true
     },
@@ -164,18 +161,23 @@
       "groupSlug": "all-minor",
       "automerge": false
     },
-    {
-      "description": "Group ordinary patch updates; the specific groups and manual-review rules below take precedence",
-      "matchUpdateTypes": ["patch"],
-      "groupName": "all patch updates",
-      "groupSlug": "all-patch"
-    },
     {
       "description": "Keep private homelab images unchanged",
       "matchDatasources": ["docker"],
       "matchPackageNames": ["/gcr\\.forust\\.xyz\\/forust\\/.+/"],
       "enabled": false
     },
+    {
+      "description": "singlesource playwright - use whichever version is found, keep docker+pypi+npm in sync",
+      "matchPackageNames": ["playwright", "mcr.microsoft.com/playwright"],
+      "groupName": "playwright singlesource",
+      "groupSlug": "playwright"
+    },
+    {
+      "description": "playwright must not automerge - version skew breaks the WS handshake (checker.py:1523 vs playwright.yaml:20)",
+      "matchPackageNames": ["playwright", "mcr.microsoft.com/playwright"],
+      "automerge": false
+    },
     {
       "description": "Renovate updates itself in lockstep across the CronJob and the Compose file",
       "matchPackageNames": ["renovate/renovate"],
@@ -183,7 +185,7 @@
       "automerge": false
     },
     {
-      "description": "Keep CI Node runtime updates in a separate, manually reviewed group",
+      "description": "CI runs npm on the node the panel image is built from - the NODE_VERSION pin in tool-versions.env and node:22-alpine in the Dockerfile are the same dependency and move as one",
       "matchPackageNames": ["node"],
       "groupName": "node runtime",
       "groupSlug": "node",
@@ -192,8 +194,6 @@
     {
       "description": "Helm chart bumps change PVC fields and admission behaviour, keep them reviewable",
       "matchDatasources": ["helm"],
-      "groupName": "Helm chart {{depName}}",
-      "groupSlug": "helm-{{depName}}",
       "automerge": false
     },
     {
@@ -202,6 +202,12 @@
       "dependencyDashboardApproval": true,
       "automerge": false
     },
+    {
+      "description": "Group patch updates from all sources - automerge still applies via the digest/patch rule above (helm/playwright stay manual via their own rules)",
+      "matchUpdateTypes": ["patch"],
+      "groupName": "all patch updates",
+      "groupSlug": "all-patch"
+    },
     {
       "description": "Python Y-bumps break compat (3.11->3.12->3.13->3.14) - keep the base image out of the shared minor/patch groups, review every bump separately. Placed last so its groupName wins.",
       "matchDatasources": ["docker"],

searxng/compose.yaml

@@ -12,11 +12,11 @@ services:
       - "traefik.enable=true"
       - "traefik.http.services.searxng.loadbalancer.server.port=8080"
       # Prod Router
-      - "traefik.http.routers.searxng.rule=Host(`s.forust.xyz`) || Host(`search.forust.xyz`)"
+      - "traefik.http.routers.searxng.rule=Host(`s.forust.xyz` || `search.forust.xyz`)"
       - "traefik.http.routers.searxng.entrypoints=websecure"
       - "traefik.http.routers.searxng.tls.certresolver=letsencrypt"
       # Local Router
-      - "traefik.http.routers.searxng-local.rule=Host(`s.workstation.internal`) || Host(`searxng.workstation.internal`)"
+      - "traefik.http.routers.searxng-local.rule=Host(`s.workstation.internal` || `searxng.workstation.internal`)"
       - "traefik.http.routers.searxng-local.entrypoints=websecure"
       - "traefik.http.routers.searxng-local.tls=true"
       # Dev Router

searxng/k8s/searxng.yaml

@@ -13,8 +13,6 @@ spec:
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: searxng-deployment
   namespace: searxng
 spec:

streaming/compose.yaml

@@ -19,7 +19,7 @@ services:
       - streaming
 
   qbittorrent:
-    image: lscr.io/linuxserver/qbittorrent:20.04.1
+    image: lscr.io/linuxserver/qbittorrent:5.2.4
     container_name: qbittorrent
     restart: unless-stopped
     environment:

termix/compose.yaml

@@ -1,6 +1,6 @@
 services:
   termix:
-    image: ghcr.io/lukegus/termix:2.9.2
+    image: ghcr.io/lukegus/termix:2.9.1
     container_name: termix
     restart: unless-stopped
     # ports:

termix/k8s/termix.yaml

@@ -13,8 +13,6 @@ spec:
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: termix-deployment
   namespace: termix
 spec:
@@ -31,7 +29,7 @@ spec:
     spec:
       containers:
         - name: termix
-          image: ghcr.io/lukegus/termix:2.9.2
+          image: ghcr.io/lukegus/termix:2.9.1
           envFrom:
             - configMapRef:
                 name: termix-config

tests/test_cicd.py

@@ -1,267 +0,0 @@
-"""CI gate, selection, persistent configuration and recovery regression tests."""
-
-import importlib.util
-import json
-import os
-import subprocess
-import tempfile
-import unittest
-from pathlib import Path
-from unittest.mock import patch
-
-ROOT = Path(__file__).resolve().parents[1]
-
-
-def module(name, filename):
-    spec = importlib.util.spec_from_file_location(name, ROOT / '.gitea/workflows' / filename)
-    loaded = importlib.util.module_from_spec(spec)
-    spec.loader.exec_module(loaded)
-    return loaded
-
-
-release_module = module('release_test', 'release.py')
-planner = module('plan_test', 'deploy-plan.py')
-compose_module = module('compose_test', 'compose-release.py')
-controller = module('controller_test', 'deploy-controller.py')
-
-
-def release(sha='a' * 40):
-    return {
-        'version': 1,
-        'sha': sha,
-        'images': {f'gcr.forust.xyz/forust/{name}': 'sha256:' + 'b' * 64 for name in release_module.IMAGES},
-        'inputs': {f'gcr.forust.xyz/forust/{name}': 'c' * 64 for name in release_module.IMAGES},
-    }
-
-
-class ReleaseGateTests(unittest.TestCase):
-    def test_release_rejects_wrong_sha_missing_images_and_mutable_tags(self):
-        for mutation in ('sha', 'missing', 'tag'):
-            data = release()
-            if mutation == 'sha':
-                data['sha'] = 'd' * 40
-            elif mutation == 'missing':
-                data['images'].pop(next(iter(data['images'])))
-            else:
-                data['images'][next(iter(data['images']))] = 'prod'
-            with self.assertRaises(ValueError):
-                release_module.validate_release(data, 'a' * 40)
-
-    def test_gate_excludes_pr_wrong_branch_and_failed_runs(self):
-        api = object.__new__(release_module.Gitea)
-        api.repository = 'forust/homelab'
-        good = {
-            'id': 1,
-            'status': 'completed',
-            'conclusion': 'success',
-            'head_branch': 'main',
-            'event': 'push',
-            'head_sha': 'a' * 40,
-            'repository': {'full_name': api.repository},
-        }
-        entries = [
-            good,
-            {**good, 'event': 'pull_request'},
-            {**good, 'head_branch': 'dev'},
-            {**good, 'conclusion': 'failure'},
-            {**good, 'head_sha': 'b' * 40},
-            {**good, 'head_repository': {'full_name': 'attacker/fork'}},
-        ]
-        with patch.object(api, 'pages', return_value=iter(entries)):
-            self.assertEqual(list(api.successful_runs('a' * 40)), [good])
-
-    def test_green_ci_with_skipped_build_is_rejected(self):
-        api = object.__new__(release_module.Gitea)
-        with (
-            patch.object(api, 'pages', return_value=iter([{'name': 'build', 'conclusion': 'skipped'}])),
-            self.assertRaisesRegex(ValueError, 'build job'),
-        ):
-            api.release({'id': 1, 'head_sha': 'a' * 40})
-
-    def test_expired_or_ambiguous_artifacts_are_rejected(self):
-        api = object.__new__(release_module.Gitea)
-        api.base = 'https://example.test/api/v1/repos/a/b'
-        artifact = {'id': 1, 'name': 'release-' + 'a' * 40}
-        for artifacts in ([{**artifact, 'expired': True}], [artifact, artifact], []):
-            with (
-                patch.object(api, 'pages', return_value=iter([{'name': 'build', 'conclusion': 'success'}])),
-                patch.object(api, 'request', return_value={'artifacts': artifacts}),
-                self.assertRaisesRegex(ValueError, 'artifact'),
-            ):
-                api.release({'id': 1, 'head_sha': 'a' * 40})
-
-
-class SelectionTests(unittest.TestCase):
-    def setUp(self):
-        self.scratch = tempfile.TemporaryDirectory()
-        self.addCleanup(self.scratch.cleanup)
-        self.repo = Path(self.scratch.name)
-        self.git('init', '-q')
-        self.git('config', 'user.email', 'test@example.test')
-        self.git('config', 'user.name', 'CI Test')
-        for service in ('one', 'two', 'postgres'):
-            directory = self.repo / service / 'k8s'
-            directory.mkdir(parents=True)
-            (directory / 'active').touch()
-            (directory / 'app.yaml').write_text('kind: Deployment\n')
-        (self.repo / '.gitea/workflows').mkdir(parents=True)
-        (self.repo / '.gitea/workflows/deploy-lib.sh').write_text('HELM_RELEASES=(\n)\n')
-        (self.repo / '.gitea/deploy-dependencies.json').write_text('{"postgres": ["one", "two"]}')
-        self.sha = self.commit()
-        self.initial = planner.make_plan(self.repo, self.repo, release(self.sha), None, 'full', [])
-
-    def git(self, *args):
-        return planner.output('git', '-C', str(self.repo), *args)
-
-    def commit(self):
-        self.git('add', '.')
-        self.git('commit', '-qm', 'Test state')
-        return self.git('rev-parse', 'HEAD')
-
-    def test_first_changed_deploy_requires_explicit_full(self):
-        with self.assertRaisesRegex(ValueError, 'full'):
-            planner.make_plan(self.repo, self.repo, release(self.sha), None, 'changed', [])
-
-    def test_only_changed_service_is_selected(self):
-        (self.repo / 'one/k8s/app.yaml').write_text('kind: StatefulSet\n')
-        result = planner.make_plan(self.repo, self.repo, release(self.commit()), self.initial, 'changed', [])
-        self.assertEqual(result['selected']['k8s'], ['one'])
-        self.assertEqual(result['helm'], [])
-
-    def test_failed_intermediate_deploy_does_not_lose_changes(self):
-        (self.repo / 'one/k8s/app.yaml').write_text('kind: StatefulSet\n')
-        self.commit()  # This commit failed deploy: baseline must remain initial.
-        (self.repo / 'two/k8s/app.yaml').write_text('kind: StatefulSet\n')
-        result = planner.make_plan(self.repo, self.repo, release(self.commit()), self.initial, 'changed', [])
-        self.assertEqual(result['selected']['k8s'], ['one', 'two'])
-
-    def test_dependencies_and_removals_are_reported(self):
-        (self.repo / 'postgres/k8s/app.yaml').write_text('kind: StatefulSet\n')
-        (self.repo / 'two/k8s/active').unlink()
-        result = planner.make_plan(self.repo, self.repo, release(self.commit()), self.initial, 'changed', [])
-        self.assertEqual(result['selected']['k8s'], ['one', 'postgres'])
-        self.assertIn('two', result['removed'])
-
-    def test_local_configuration_change_selects_service(self):
-        (self.repo / 'one/.env').write_text('TEST_VALUE=changed\n')
-        result = planner.make_plan(self.repo, self.repo, release(self.sha), self.initial, 'changed', [])
-        self.assertEqual(result['selected']['k8s'], ['one'])
-
-    def test_redeploy_is_noop_and_full_includes_all(self):
-        result = planner.make_plan(self.repo, self.repo, release(self.sha), self.initial, 'changed', [])
-        self.assertEqual(result['selected']['k8s'], [])
-        result = planner.make_plan(self.repo, self.repo, release(self.sha), self.initial, 'full', [])
-        self.assertEqual(result['selected']['k8s'], ['one', 'postgres', 'two'])
-
-
-class ComposeConfigurationTests(unittest.TestCase):
-    def test_pin_preserves_project_volumes_paths_and_previous_image(self):
-        with tempfile.TemporaryDirectory() as scratch:
-            root = Path(scratch)
-            run = root / 'run'
-            source = run / 'source'
-            config_repo = root / 'persistent'
-            (source / 'headscale').mkdir(parents=True)
-            config_repo.mkdir()
-            (run / 'release.json').write_text(json.dumps(release()))
-            old = 'busybox@sha256:' + 'd' * 64
-            new = 'busybox@sha256:' + 'e' * 64
-            config = {
-                'name': 'headscale',
-                'services': {
-                    'app': {
-                        'image': 'busybox:latest',
-                        'volumes': [
-                            {'type': 'bind', 'source': str(config_repo / 'headscale/config.yaml'), 'target': '/config'},
-                            {'type': 'volume', 'source': 'data', 'target': '/data'},
-                        ],
-                    }
-                },
-                'volumes': {'data': {'name': 'headscale_data'}},
-            }
-
-            def fake_output(*args, **kwargs):
-                if args[:2] == ('docker', 'compose'):
-                    self.assertEqual(kwargs['cwd'], config_repo)
-                    self.assertIn(str(config_repo / 'headscale'), args)
-                    return json.dumps(config)
-                if args[:2] == ('docker', 'ps'):
-                    return 'container'
-                if args[:2] == ('docker', 'inspect'):
-                    return 'sha256:' + 'f' * 64
-                return json.dumps([old])
-
-            with (
-                patch.dict(os.environ, {'CONFIG_REPO': str(config_repo), 'REPO': str(source), 'RUN_DIR': str(run)}),
-                patch.object(compose_module, 'output', side_effect=fake_output),
-                patch.object(compose_module, 'resolve', return_value=new),
-            ):
-                compose_module.prepare(source / 'headscale/compose.yaml')
-            pinned = json.loads((run / 'compose/headscale.json').read_text())
-            before = json.loads((run / 'compose-before/headscale.json').read_text())
-            self.assertEqual(pinned['name'], 'headscale')
-            self.assertEqual(pinned['volumes'], config['volumes'])
-            self.assertEqual(pinned['services']['app']['volumes'], config['services']['app']['volumes'])
-            self.assertEqual(pinned['services']['app']['image'], new)
-            self.assertEqual(before['services']['app']['image'], old)
-            self.assertEqual((run / 'compose/headscale.json').stat().st_mode & 0o777, 0o600)
-
-    def test_registry_index_and_single_image_descriptors(self):
-        for digest in ('a' * 64, 'b' * 64):
-            with patch.object(compose_module, 'output', return_value=json.dumps({'digest': 'sha256:' + digest})):
-                self.assertEqual(
-                    compose_module.resolve('registry.test:5000/repo:latest'), f'registry.test:5000/repo@sha256:{digest}'
-                )
-
-
-class ControllerTests(unittest.TestCase):
-    def test_completed_stage_cannot_apply_again(self):
-        with tempfile.TemporaryDirectory() as scratch:
-            directory = Path(scratch)
-            controller.atomic_json(
-                directory / 'status.json', {'state': 'success', 'stages': {'apply-k8s': {'result': 'success'}}}
-            )
-            with patch.object(subprocess, 'run') as execute:
-                self.assertTrue(controller.stage(directory, 'apply-k8s', 60))
-                execute.assert_not_called()
-
-    def test_run_id_is_not_shell_or_path_input(self):
-        for invalid in ('../123', '-1', '1;touch bad', 'abc', '1/2'):
-            with self.assertRaises(ValueError):
-                controller.run_directory(invalid)
-
-    def test_exact_previous_revision_is_used_for_rollback(self):
-        with tempfile.TemporaryDirectory() as scratch:
-            root = Path(scratch)
-            (root / 'current').write_text(str(root))
-            (root / 'revisions.json').write_text(
-                json.dumps([{'kind': 'deployment', 'namespace': 'app', 'name': 'web', 'uid': 'same', 'revision': 7}])
-            )
-            (root / 'failed').write_text('deployment app web\n')
-            script = """set -euo pipefail
-source "$LIB"
-kubectl() {
-  case "$*" in
-    *metadata.annotations*) printf '{}' ;;
-    *metadata.uid*) printf same ;;
-    'rollout undo'*) printf '%s\\n' "$*" >>"$CALLS" ;;
-    'rollout status'*) return 0 ;;
-    *) return 1 ;;
-  esac
-}
-rollback_workloads "$FAILED"
-"""
-            env = {
-                **os.environ,
-                'REPO': str(ROOT),
-                'LIB': str(ROOT / '.gitea/workflows/deploy-lib.sh'),
-                'DEPLOY_SNAPSHOT_DIR': str(root),
-                'CALLS': str(root / 'calls'),
-                'FAILED': str(root / 'failed'),
-            }
-            subprocess.run(['/usr/bin/bash', '-c', script], env=env, check=True)  # noqa: S603
-            self.assertIn('--to-revision=7', (root / 'calls').read_text())
-
-
-if __name__ == '__main__':
-    unittest.main()

tests/test_cicd_lifecycle.py

@@ -1,285 +0,0 @@
-"""Release publication and controller recovery tests without a live server."""
-
-import io
-import json
-import os
-import subprocess
-import tempfile
-import unittest
-import zipfile
-from pathlib import Path
-from unittest.mock import Mock, patch
-
-from test_cicd import ROOT, controller, release, release_module
-
-
-class ArtifactTests(unittest.TestCase):
-    def test_archive_rejects_nested_or_extra_files(self):
-        api = object.__new__(release_module.Gitea)
-        api.base = 'https://example.test/api/v1/repos/a/b'
-        for names in (['../release.json'], ['release.json', 'credentials']):
-            blob = io.BytesIO()
-            with zipfile.ZipFile(blob, 'w') as archive:
-                for name in names:
-                    archive.writestr(name, json.dumps(release()))
-            replies = [{'artifacts': [{'id': 1, 'name': 'release-' + 'a' * 40}]}, blob.getvalue()]
-            with (
-                patch.object(api, 'pages', return_value=iter([{'name': 'build', 'conclusion': 'success'}])),
-                patch.object(api, 'request', side_effect=replies),
-                self.assertRaisesRegex(ValueError, 'archive'),
-            ):
-                api.release({'id': 1, 'head_sha': 'a' * 40})
-
-    def test_quoted_and_single_platform_images_render_from_checked_release(self):
-        with tempfile.TemporaryDirectory() as scratch:
-            path = Path(scratch) / 'release.json'
-            path.write_text(json.dumps(release()))
-            result = io.StringIO()
-            image = 'gcr.forust.xyz/forust/error-pages'
-            with patch.dict(os.environ, {'RELEASE_FILE': str(path), 'DEPLOY_SHA': 'a' * 40}):
-                release_module.render(io.StringIO(f'  image: "{image}:prod" # note\n'), result)
-            self.assertEqual(result.getvalue(), f'  image: "{image}@sha256:{"b" * 64}" # note\n')
-
-    def test_unknown_image_cannot_emit_partial_manifest(self):
-        with tempfile.TemporaryDirectory() as scratch:
-            path = Path(scratch) / 'release.json'
-            path.write_text(json.dumps(release()))
-            result = io.StringIO()
-            with (
-                patch.dict(os.environ, {'RELEASE_FILE': str(path), 'DEPLOY_SHA': 'a' * 40}),
-                self.assertRaisesRegex(ValueError, 'missing'),
-            ):
-                release_module.render(
-                    io.StringIO('kind: Deployment\nimage: gcr.forust.xyz/forust/unknown:prod\n'), result
-                )
-            self.assertEqual(result.getvalue(), '')
-
-    def test_only_changed_image_is_built_and_credentials_are_removed(self):
-        with tempfile.TemporaryDirectory() as scratch:
-            root = Path(scratch)
-            built = []
-            auth_directories = []
-            old = release()
-
-            def fake_command(*args, **kwargs):
-                if args[:2] == ('git', 'rev-parse'):
-                    return 'e' * 40
-                if args[:3] == ('docker', 'buildx', 'build'):
-                    built.append(args[args.index('--file') + 1])
-                    metadata = Path(args[args.index('--metadata-file') + 1])
-                    metadata.write_text(json.dumps({'containerimage.digest': 'sha256:' + 'f' * 64}))
-                    auth_directories.append(Path(kwargs['env']['DOCKER_CONFIG']))
-                return ''
-
-            api = Mock()
-            api.successful_runs.return_value = iter([{'id': 1}])
-            api.release.return_value = old
-            with (
-                patch.dict(
-                    os.environ,
-                    {
-                        'GITHUB_SHA': 'e' * 40,
-                        'GITHUB_RUN_ID': '2',
-                        'REGISTRY_USERNAME': 'test',
-                        'REGISTRY_PASSWORD': 'placeholder',
-                    },
-                ),
-                patch.object(release_module.Path, 'home', return_value=root),
-                patch.object(release_module, 'Gitea', return_value=api),
-                patch.object(
-                    release_module,
-                    'fingerprint',
-                    side_effect=lambda context, _file: ('d' if context == 'errorpages' else 'c') * 64,
-                ),
-                patch.object(release_module, 'command', side_effect=fake_command),
-                patch.object(subprocess, 'run', return_value=subprocess.CompletedProcess([], 0)),
-            ):
-                plan = root / 'plan.json'
-                release_module.prepare_images(plan)
-                for name in release_module.IMAGES:
-                    release_module.build(root / f'{name}.json', name, plan)
-            self.assertEqual(built, ['errorpages/Dockerfile'])
-            self.assertTrue(all(not directory.exists() for directory in auth_directories))
-            self.assertEqual(json.loads((root / 'error-pages.json').read_text())['sha'], 'e' * 40)
-
-
-class DurableRunTests(unittest.TestCase):
-    def test_duplicate_start_only_reattaches(self):
-        with tempfile.TemporaryDirectory() as scratch:
-            state = Path(scratch)
-            directory = state / 'runs/123-1'
-            directory.mkdir(parents=True)
-            request = {'release': release(), 'mode': 'full', 'refresh_images': False}
-            controller.atomic_json(directory / 'request.json', request)
-            controller.atomic_json(directory / 'status.json', {'state': 'running', 'stages': {}})
-            with (
-                patch.object(controller, 'STATE', state),
-                patch.object(controller.sys, 'stdin', io.TextIOWrapper(io.BytesIO(json.dumps(request).encode()))),
-                patch.object(controller, 'command') as execute,
-            ):
-                controller.start('123-1')
-                execute.assert_not_called()
-
-    def test_failed_apply_still_verifies_and_does_not_advance_baseline(self):
-        with tempfile.TemporaryDirectory() as scratch:
-            state = Path(scratch)
-            directory = state / 'runs/123-1'
-            directory.mkdir(parents=True)
-            controller.atomic_json(
-                directory / 'request.json', {'release': release(), 'mode': 'full', 'refresh_images': False}
-            )
-            controller.atomic_json(directory / 'status.json', {'state': 'queued', 'stages': {}})
-            called = []
-
-            def fake_stage(folder, name, _budget):
-                called.append(name)
-                status = json.loads((folder / 'status.json').read_text())
-                status['stages'][name] = {'result': 'failure' if name == 'apply-k8s' else 'success'}
-                controller.atomic_json(folder / 'status.json', status)
-                return name != 'apply-k8s'
-
-            with (
-                patch.object(controller, 'STATE', state),
-                patch.object(controller, 'make_plan', return_value={'selected': {}, 'helm': [], 'removed': []}),
-                patch.object(controller, 'stage', side_effect=fake_stage),
-                patch.object(controller, 'command', return_value='1'),
-                self.assertRaises(RuntimeError),
-            ):
-                controller.execute('123-1')
-            self.assertIn('verify-k8s', called)
-            self.assertIn('smoke', called)
-            self.assertNotIn('apply-compose', called)
-            self.assertFalse((state / 'last-success.json').exists())
-            self.assertEqual(json.loads((directory / 'status.json').read_text())['state'], 'failure')
-
-    def test_recovery_finishes_baseline_after_all_stages_completed(self):
-        with tempfile.TemporaryDirectory() as scratch:
-            state = Path(scratch)
-            directory = state / 'runs/123-1'
-            directory.mkdir(parents=True)
-            names = ('doctor', 'validate', 'apply-k8s', 'apply-compose', 'verify-k8s', 'smoke')
-            controller.atomic_json(
-                directory / 'status.json',
-                {'state': 'running', 'stages': {name: {'result': 'success'} for name in names}},
-            )
-            controller.atomic_json(directory / 'plan.json', {'sha': 'a' * 40})
-            with patch.object(controller, 'STATE', state), patch.object(controller, 'stage') as execute:
-                controller.recover(directory)
-                execute.assert_not_called()
-            self.assertEqual(json.loads((state / 'last-success.json').read_text())['run_id'], '123-1')
-            self.assertEqual(json.loads((directory / 'status.json').read_text())['state'], 'success')
-
-    def test_manual_recovery_retries_checks_without_repeating_apply(self):
-        with tempfile.TemporaryDirectory() as scratch:
-            state = Path(scratch)
-            directory = state / 'runs/123-1'
-            (directory / 'snapshot').mkdir(parents=True)
-            (directory / 'snapshot/current').write_text('snapshot')
-            controller.atomic_json(
-                directory / 'status.json',
-                {
-                    'state': 'failure',
-                    'stages': {
-                        'apply-k8s': {'result': 'failure'},
-                        'verify-k8s': {'result': 'failure'},
-                        'smoke': {'result': 'failure'},
-                    },
-                },
-            )
-            called = []
-
-            def checks(folder, name, _budget):
-                status = json.loads((folder / 'status.json').read_text())
-                self.assertNotIn(name, status['stages'])
-                called.append(name)
-                status['stages'][name] = {'result': 'success'}
-                controller.atomic_json(folder / 'status.json', status)
-                return True
-
-            with patch.object(controller, 'STATE', state), patch.object(controller, 'stage', side_effect=checks):
-                controller.recover(directory, retry=True)
-            self.assertEqual(called, ['verify-k8s', 'smoke'])
-            self.assertFalse((state / 'last-success.json').exists())
-            self.assertEqual(json.loads((directory / 'status.json').read_text())['state'], 'failure')
-
-
-class FailureSummaryTests(unittest.TestCase):
-    def test_build_failure_keeps_progress_and_does_not_expose_exception_text(self):
-        with tempfile.TemporaryDirectory() as scratch:
-            summary = Path(scratch) / 'summary.md'
-
-            def failed_build(_output, report, _name, _plan):
-                report.update(phase='Build or reuse xdfnx-homepage', current='xdfnx-homepage', built=['error-pages'])
-                report['images']['gcr.forust.xyz/forust/error-pages'] = 'sha256:' + 'b' * 64
-                raise RuntimeError('private value must not appear in the summary')
-
-            with (
-                patch.dict(os.environ, {'GITHUB_STEP_SUMMARY': str(summary), 'GITHUB_SHA': 'a' * 40}),
-                patch.object(release_module, 'build_images', side_effect=failed_build),
-                self.assertRaises(RuntimeError),
-            ):
-                release_module.build(Path(scratch) / 'release.json', 'xdfnx-homepage', Path('plan.json'))
-            content = summary.read_text()
-            self.assertIn('**failure**', content)
-            self.assertIn('error-pages', content)
-            self.assertIn('xdfnx-homepage', content)
-            self.assertNotIn('private value', content)
-
-    def test_deploy_failure_reports_completed_apply_and_rollback_result(self):
-        with tempfile.TemporaryDirectory() as scratch:
-            state = Path(scratch)
-            directory = state / 'runs/123-1'
-            snapshot = directory / 'snapshot/before'
-            snapshot.mkdir(parents=True)
-            (directory / 'snapshot/current').write_text(str(snapshot))
-            (snapshot / 'failed-workloads').write_text('deployment app api\nROLLED_BACK=1\nUNRECOVERED=0\n')
-            controller.atomic_json(
-                directory / 'request.json', {'release': release(), 'mode': 'changed', 'refresh_images': False}
-            )
-            controller.atomic_json(
-                directory / 'status.json',
-                {
-                    'state': 'failure',
-                    'stages': {
-                        'apply-k8s': {'result': 'success', 'exit_code': 0},
-                        'verify-k8s': {'result': 'failure', 'exit_code': 1},
-                    },
-                },
-            )
-            controller.atomic_json(directory / 'plan.json', {'selected': {'k8s': ['app'], 'compose': []}})
-            (directory / 'apply-events.jsonl').write_text(
-                json.dumps({'action': 'kubectl', 'target': 'app/k8s/api.yaml', 'result': 'success'}) + '\n'
-            )
-            output = io.StringIO()
-            with patch.object(controller, 'STATE', state), patch('sys.stdout', output):
-                controller.summary('123-1')
-            content = output.getvalue()
-            self.assertIn('verify-k8s | failure | 1', content)
-            self.assertIn('app/k8s/api.yaml', content)
-            self.assertIn('Workloads restored: **1**', content)
-            self.assertIn('manual recovery: **0**', content)
-            self.assertIn('Compose requires manual recovery', content)
-
-
-class InstallerTests(unittest.TestCase):
-    def test_version_comparison_is_exact_without_network_or_host_packages(self):
-        with tempfile.TemporaryDirectory() as scratch:
-            root = Path(scratch)
-            binary = root / 'bin/fake'
-            binary.parent.mkdir()
-            binary.write_text('#!/bin/sh\necho fake-v1.7.70\n')
-            binary.chmod(0o755)
-            script = """set -euo pipefail
-source "$LIB"
-if at_version fake 1.7.7; then exit 1; fi
-at_version fake 1.7.70
-"""
-            subprocess.run(  # noqa: S603
-                ['/usr/bin/bash', '-c', script],
-                check=True,
-                env={**os.environ, 'TOOLS_DIR': str(root), 'LIB': str(ROOT / '.gitea/workflows/install-ci-tools.sh')},
-            )
-
-
-if __name__ == '__main__':
-    unittest.main()

tests/test_image_matrix.py

@@ -1,144 +0,0 @@
-"""Matrix release contracts and failure gates without a registry."""
-
-import json
-import os
-import tempfile
-import unittest
-from pathlib import Path
-from unittest.mock import Mock, call, patch
-
-from test_cicd import release, release_module
-
-
-def plan_data(changed):
-    targets = []
-    for name, (context, dockerfile) in release_module.IMAGES.items():
-        targets.append(
-            {
-                'name': name,
-                'image': f'gcr.forust.xyz/forust/{name}',
-                'context': context,
-                'dockerfile': dockerfile,
-                'inputs': ('d' if name in changed else 'c') * 64,
-                'reuse_digest': None if name in changed else 'sha256:' + 'b' * 64,
-            }
-        )
-    return {'sha': 'a' * 40, 'targets': targets}
-
-
-class MatrixTests(unittest.TestCase):
-    def test_no_change_one_image_all_images_and_missing_baseline(self):
-        for changed in (set(), {'error-pages'}, set(release_module.IMAGES)):
-            with self.subTest(changed=changed), tempfile.TemporaryDirectory() as scratch:
-                api = Mock()
-                api.successful_runs.return_value = iter([{'id': 1}])
-                api.release.return_value = release()
-                expected = plan_data(changed)
-                fingerprints = {t['dockerfile']: t['inputs'] for t in expected['targets']}
-                output = Path(scratch) / 'plan.json'
-                with (
-                    patch.dict(os.environ, {'GITHUB_SHA': 'a' * 40, 'GITHUB_RUN_ID': '2'}),
-                    patch.object(release_module, 'command', return_value='a' * 40),
-                    patch.object(release_module, 'Gitea', return_value=api),
-                    patch.object(
-                        release_module, 'fingerprint', side_effect=lambda _c, f, mapping=fingerprints: mapping[f]
-                    ),
-                ):
-                    release_module.prepare_images(output)
-                self.assertEqual(json.loads(output.read_text()), expected)
-        api.successful_runs.return_value = iter([])
-        with (
-            tempfile.TemporaryDirectory() as scratch,
-            patch.dict(os.environ, {'GITHUB_SHA': 'a' * 40}),
-            patch.object(release_module, 'command', return_value='a' * 40),
-            patch.object(release_module, 'Gitea', return_value=api),
-            patch.object(release_module, 'fingerprint', return_value='c' * 64),
-        ):
-            output = Path(scratch) / 'plan.json'
-            release_module.prepare_images(output)
-            self.assertTrue(all(t['reuse_digest'] is None for t in json.loads(output.read_text())['targets']))
-
-    def test_incomplete_or_wrong_sha_fragments_cannot_publish_tags(self):
-        for wrong_sha in (False, True):
-            with self.subTest(wrong_sha=wrong_sha), tempfile.TemporaryDirectory() as scratch:
-                root = Path(scratch)
-                plan = root / 'plan.json'
-                plan.write_text(json.dumps(plan_data(set())))
-                for name in release_module.IMAGES:
-                    if name == 'xdfnx-homepage' and not wrong_sha:
-                        continue
-                    folder = root / f'image-{name}'
-                    folder.mkdir()
-                    image = f'gcr.forust.xyz/forust/{name}'
-                    folder.joinpath('image.json').write_text(
-                        json.dumps(
-                            {
-                                'version': 1,
-                                'sha': ('e' if wrong_sha else 'a') * 40,
-                                'images': {image: 'sha256:' + 'b' * 64},
-                                'inputs': {image: 'c' * 64},
-                            }
-                        )
-                    )
-                with (
-                    patch.object(release_module, 'checked_plan', return_value=plan_data(set())),
-                    patch.object(release_module.subprocess, 'run') as execute,
-                    self.assertRaises((ValueError, FileNotFoundError)),
-                ):
-                    release_module.finalize_images(root / 'release.json', root, plan)
-                execute.assert_not_called()
-                self.assertFalse((root / 'release.json').exists())
-
-    def test_manifest_only_release_pins_each_successful_digest(self):
-        with tempfile.TemporaryDirectory() as scratch:
-            root = Path(scratch)
-            data = plan_data(set())
-            for target in data['targets']:
-                folder = root / f'image-{target["name"]}'
-                folder.mkdir()
-                image = target['image']
-                folder.joinpath('image.json').write_text(
-                    json.dumps(
-                        {
-                            'version': 1,
-                            'sha': data['sha'],
-                            'images': {image: target['reuse_digest']},
-                            'inputs': {image: target['inputs']},
-                        }
-                    )
-                )
-            with (
-                patch.object(release_module, 'checked_plan', return_value=data),
-                patch.dict(os.environ, {'REGISTRY_USERNAME': 'test', 'REGISTRY_PASSWORD': 'placeholder'}),
-                patch.object(release_module.subprocess, 'run'),
-                patch.object(release_module, 'command') as execute,
-            ):
-                release_module.finalize_images(root / 'release.json', root, root / 'plan.json')
-            self.assertEqual(
-                [entry.args for entry in execute.call_args_list],
-                [
-                    call(
-                        'docker',
-                        'buildx',
-                        'imagetools',
-                        'create',
-                        '--prefer-index=false',
-                        '--tag',
-                        f'{t["image"]}:sha-{data["sha"]}',
-                        f'{t["image"]}@{t["reuse_digest"]}',
-                    ).args
-                    for t in data['targets']
-                ],
-            )
-            self.assertEqual(json.loads((root / 'release.json').read_text()), release())
-
-    def test_checkout_mismatch_cannot_build(self):
-        with tempfile.TemporaryDirectory() as scratch:
-            plan = Path(scratch) / 'plan.json'
-            plan.write_text(json.dumps(plan_data(set())))
-            with (
-                patch.dict(os.environ, {'GITHUB_SHA': 'e' * 40}),
-                patch.object(release_module, 'command', return_value='a' * 40),
-                self.assertRaisesRegex(ValueError, 'source commit'),
-            ):
-                release_module.checked_plan(plan)

tests/test_netbird_runtime.py

@@ -1,87 +0,0 @@
-"""Exercise local setup and config rendering without a Docker daemon."""
-
-import os
-import shutil
-import subprocess
-import tempfile
-import unittest
-from pathlib import Path
-
-ROOT = Path(__file__).resolve().parents[1]
-
-
-class NetbirdRuntimeTests(unittest.TestCase):
-    def setUp(self):
-        self.temp = tempfile.TemporaryDirectory()
-        self.addCleanup(self.temp.cleanup)
-        self.root = Path(self.temp.name)
-        self.stack = self.root / 'netbird'
-        self.stack.mkdir()
-        for name in ('setup.sh', '.env.example', 'config.template.yaml'):
-            shutil.copy(ROOT / 'netbird' / name, self.stack / name)
-        binary = self.root / 'bin'
-        binary.mkdir()
-        docker = binary / 'docker'
-        docker.write_text('#!/bin/sh\nprintf "%s\\n" 172.20.0.0/16\n')
-        docker.chmod(0o755)
-        self.env = dict(os.environ, PATH=f'{binary}:{os.environ["PATH"]}')
-
-    def setup(self):
-        return subprocess.run(  # noqa: S603 - executes the repository script copied into this test's temp dir
-            ['/bin/bash', str(self.stack / 'setup.sh')], env=self.env, capture_output=True, check=False
-        )
-
-    def test_setup_preserves_existing_secrets_and_env(self):
-        self.assertEqual(self.setup().returncode, 0)
-        paths = [self.stack / '.env', *sorted((self.stack / 'secrets').iterdir())]
-        before = [p.read_bytes() for p in paths]
-        self.assertIn(b'NETBIRD_PROXY_SUBNET=172.20.0.0/16', before[0])
-        self.assertEqual(self.setup().returncode, 0)
-        self.assertEqual(before, [p.read_bytes() for p in paths])
-        for p in paths[1:]:
-            self.assertEqual(p.stat().st_mode & 0o777, 0o600)
-
-    def test_setup_rejects_empty_existing_secret(self):
-        (self.stack / 'secrets').mkdir()
-        secret = self.stack / 'secrets/datastore-encryption-key'
-        secret.touch()
-        self.assertNotEqual(self.setup().returncode, 0)
-        self.assertEqual(secret.read_bytes(), b'')
-
-    def render(self, subnet):
-        self.assertEqual(self.setup().returncode, 0)
-        rendered = self.root / 'run/config.yaml'
-        script = (ROOT / 'netbird/entrypoint.sh').read_text()
-        replacements = {
-            '/opt/netbird/config.template.yaml': str(self.stack / 'config.template.yaml'),
-            '/run/netbird/config.yaml': str(rendered),
-            '/run/secrets/relay_auth_secret': str(self.stack / 'secrets/relay-auth-secret'),
-            '/run/secrets/datastore_encryption_key': str(self.stack / 'secrets/datastore-encryption-key'),
-            '/go/bin/netbird-server': '/bin/true',
-        }
-        for original, local in replacements.items():
-            script = script.replace(original, local)
-        result = subprocess.run(  # noqa: S603 - repository renderer, with test-local paths
-            ['/bin/sh', '-c', script, 'entrypoint', '--config', str(rendered)],
-            env=dict(self.env, NETBIRD_DOMAIN='nb.example.com', NETBIRD_PROXY_SUBNET=subnet),
-            capture_output=True,
-            check=False,
-        )
-        return result, rendered
-
-    def test_renderer_replaces_placeholders_and_restricts_file_permissions(self):
-        result, rendered = self.render('172.20.0.0/16')
-        self.assertEqual(result.returncode, 0, result.stderr)
-        self.assertNotIn('__NETBIRD_', rendered.read_text())
-        self.assertIn('nb.example.com', rendered.read_text())
-        self.assertEqual(rendered.stat().st_mode & 0o777, 0o600)
-
-    def test_renderer_rejects_auto_and_default_route(self):
-        for subnet in ('auto', '0.0.0.0/0', '999.1.1.1/24'):
-            with self.subTest(subnet=subnet):
-                result, _ = self.render(subnet)
-                self.assertNotEqual(result.returncode, 0)
-
-
-if __name__ == '__main__':
-    unittest.main()

tests/test_session_keeper.py

@@ -0,0 +1,74 @@
+"""Session publication checks without Redis, the EDU website, or credentials."""
+
+import importlib.util
+import sys
+import unittest
+from pathlib import Path
+from unittest.mock import Mock, patch
+
+SCRIPT = Path(__file__).resolve().parents[1] / 'edu_master/phpsessid-bot/bot.py'
+spec = importlib.util.spec_from_file_location('session_keeper', SCRIPT)
+bot = importlib.util.module_from_spec(spec)
+with patch.dict(sys.modules, {'redis': Mock(), 'requests': Mock()}):
+    spec.loader.exec_module(bot)
+
+
+class SessionKeeperTests(unittest.TestCase):
+    def setUp(self):
+        self.session = Mock()
+        self.session.get.return_value.status_code = 200
+        self.session.cookies.get.return_value = 'test-cookie'
+        self.redis = Mock()
+        self.touch = patch.object(bot, 'touch_success_file').start()
+        self.addCleanup(patch.stopall)
+
+    def test_verified_cookie_expires_and_is_not_logged(self):
+        with self.assertLogs(bot.logger, level='INFO') as logs:
+            self.assertTrue(bot.refresh_session(self.session, self.redis))
+        self.redis.set.assert_called_once_with('EDU_PHPSESSID', 'test-cookie', ex=bot.INTERVAL * 120)
+        self.touch.assert_called_once()
+        self.assertNotIn('test-cookie', '\n'.join(logs.output))
+        self.assertEqual(self.session.post.call_args.kwargs['timeout'], (10, 30))
+        self.assertEqual(self.session.get.call_args.kwargs['timeout'], (10, 30))
+
+    def test_redirect_does_not_publish(self):
+        self.session.get.return_value.status_code = 302
+        self.assertFalse(bot.refresh_session(self.session, self.redis))
+        self.redis.set.assert_not_called()
+        self.touch.assert_not_called()
+
+    def test_missing_cookie_does_not_mark_success(self):
+        self.session.cookies.get.return_value = None
+        self.assertFalse(bot.refresh_session(self.session, self.redis))
+        self.redis.set.assert_not_called()
+        self.touch.assert_not_called()
+
+    def test_redis_failure_does_not_mark_success(self):
+        self.redis.set.side_effect = OSError('redis unavailable')
+        with self.assertRaises(OSError):
+            bot.refresh_session(self.session, self.redis)
+        self.touch.assert_not_called()
+
+    def test_http_timeout_does_not_publish(self):
+        self.session.post.side_effect = TimeoutError('EDU unavailable')
+        with self.assertRaises(TimeoutError):
+            bot.refresh_session(self.session, self.redis)
+        self.redis.set.assert_not_called()
+        self.touch.assert_not_called()
+
+    def test_missing_credentials_fail_before_network_access(self):
+        with patch.object(bot, 'LOGIN', None), self.assertRaises(ValueError):
+            bot.main()
+
+    def test_nonpositive_interval_fails_before_network_access(self):
+        with (
+            patch.object(bot, 'LOGIN', 'test'),
+            patch.object(bot, 'PASSWORD', 'test'),
+            patch.object(bot, 'INTERVAL', 0),
+            self.assertRaises(ValueError),
+        ):
+            bot.main()
+
+
+if __name__ == '__main__':
+    unittest.main()

vaultwarden/compose.yaml

@@ -1,7 +1,7 @@
 services:
   server:
     container_name: vaultwarden-server
-    image: vaultwarden/server:1.37.4
+    image: vaultwarden/server:1.37.3
     restart: unless-stopped
     ports:
       - 9993:80

vaultwarden/k8s/vaultwarden.yaml

@@ -13,8 +13,6 @@ spec:
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: vaultwarden-deployment
   namespace: vaultwarden
 spec:
@@ -31,7 +29,7 @@ spec:
     spec:
       containers:
         - name: vaultwarden
-          image: vaultwarden/server:1.37.4
+          image: vaultwarden/server:1.37.3
           ports:
             - containerPort: 80
           envFrom:

vpn/xui/k8s/xui.yaml

@@ -20,8 +20,6 @@ spec:
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
   name: xui-deployment
   namespace: xui
 spec:
@@ -38,7 +36,7 @@ spec:
     spec:
       containers:
         - name: xui
-          image: ghcr.io/mhsanaei/3x-ui:v3.9.0
+          image: ghcr.io/mhsanaei/3x-ui:v3.8.5
           envFrom:
             - configMapRef:
                 name: xui-config