chore(deps): update container patch updates

Every bouncer-protected request blocks on a synchronous
GET /v1/decisions against the LAPI, and the plugin fails CLOSED with 403
when that lookup exceeds its timeout. The LAPI was capped at 400m/500Mi
on a single replica, so idle lookups measured 1.3-7.4s and a deploy
burst pushed them past the fork's implicit 10s default: gitea answered
403 for ten seconds straight, and containerd turned those 403s on
gcr.forust.xyz into ErrImagePull/ImagePullBackOff on freshly rolled
pods.

Three changes, plus the 403 feedback loop that made it sticky:

* gitea/k8s/ingress.yaml - drop the bouncer from the /v2 registry route.
  A deploy fires hundreds of parallel authenticated OCI requests
  (runner Action API, manifest inspect per own image, containerd pulls,
  smoke probes) and scanners gain nothing from a registry that already
  does its own token auth. The web UI route keeps the bouncer.
* crowdsec-values.yaml - LAPI to 1500m/1Gi. Replicas stay at 1 on
  purpose: LAPI is stateful (BoltDB plus credentials on two RWO PVCs),
  so a second replica would corrupt the decision store.
* crowdsec-middleware.yaml - CrowdsecLapiTimeout: 2s instead of the
  implicit 10s, so a slow LAPI costs a fast 403 rather than a 10s hang.

LePresidente/http-generic-403-bf then banned us for our own 403s: five
POSTs answered 403 within 10s earn a 4h ban, and the hairpin-NAT
address 192.168.88.1 that the Gitea Actions runner presents to Traefik
is not covered by the home-dynamic-IP whitelist. That scenario cannot
be dropped per-scenario - it is baked into the hub item
crowdsecurity/http-generic-bf v0.9, and disabling the whole
base-http-scenarios collection would cost ~40 useful detections. So the
janitor now deletes its decisions hourly and a new forust/lan
whitelist postoverflow covers 192.168.88.0/24. First janitor run
removed 165 decisions; none of the remaining ones are local.

Measured after: gitea 200 in 25-148ms (was 403 at 10001ms), a 60-way
parallel burst all 200 with a 422ms max, gcr /v2 back to its 401 auth
challenge, LAPI at 60m CPU with no throttling.

.docs/storage-audit-instruction.md

@@ -0,0 +1,191 @@
+# Инструкция: Анализ хранилища Kubernetes и настройка NFS
+
+## Цель
+Проанализировать текущую конфигурацию хранилища Kubernetes и подготовить план внедрения NFS StorageClass для сохранения данных при удалении namespace.
+
+## 1. Собрать информацию о кластере
+
+### 1.1. Версия Kubernetes и тип дистрибутива
+```bash
+kubectl version --short
+# или
+kubectl version
+```
+
+Определить, используется ли k3s, k8s, microk8s и т.д.:
+```bash
+# Проверить наличие k3s
+which k3s
+# Проверить процесс
+ps aux | grep -E 'kube|k3s'
+```
+
+### 1.2. StorageClass
+```bash
+kubectl get storageclass -o wide
+```
+
+Запомнить:
+- `PROVISIONER` — какой драйвер используется
+- `RECLAIMPOLICY` — Delete или Retain
+- Какой StorageClass помечен как `(default)`
+
+### 1.3. Существующие PV и PVC
+```bash
+kubectl get pv -o wide
+kubectl get pvc --all-namespaces
+```
+
+Посмотреть, какие PVC привязаны к каким PV, и какой reclaimPolicy у PV.
+
+### 1.4. Нода и диски
+```bash
+# Список нод
+kubectl get nodes -o wide
+
+# На каждой ноде (через ssh или локально):
+lsblk
+df -h
+cat /etc/fstab
+```
+
+Определить:
+- Есть ли отдельный раздел/диск для данных
+- Куда смонтированы разделы
+- Сколько свободного места
+- Есть ли монтирование NTFS-разделов (как `/media/forust/Programs`)
+
+### 1.5. Где local-path хранит данные (для k3s)
+```bash
+ls -la /var/lib/rancher/k3s/storage/ 2>/dev/null
+# или для microk8s
+ls -la /var/snap/microk8s/common/ 2>/dev/null
+```
+
+## 2. Анализ: сохраняются ли данные при удалении namespace?
+
+| Сценарий | Результат |
+|---|---|
+| `kubectl delete ns <ns>` | Все PVC в namespace удаляются |
+| PVC → PV c `reclaimPolicy: Delete` | PV и данные удалены |
+| PVC → PV c `reclaimPolicy: Retain` | PV остаётся (статус Released), данные целы |
+
+**Вывод:** Если reclaimPolicy в StorageClass = `Delete`, то данные **пропадут**. Если `Retain` — сохранятся.
+
+## 3. План внедрения NFS
+
+### 3.1. Проверить, установлен ли NFS
+```bash
+which nfsstat exportfs mount.nfs
+systemctl status nfs-server 2>/dev/null || systemctl status nfs-kernel-server 2>/dev/null
+```
+
+### 3.2. Выбрать директорию для NFS-экспорта
+
+Варианты (выбрать подходящий):
+- `/var/lib/k8s-nfs/` — на корневом разделе
+- `<путь к отдельному разделу>/k8s-nfs/` — если есть отдельный диск/раздел
+- Не рекомендуется использовать NTFS-раздел (проблемы с правами и производительностью)
+
+Требования:
+- Файловая система: ext4 или xfs (не ntfs!)
+- Достаточно свободного места
+- Права: `755`, владелец root
+
+### 3.3. Установить NFS-сервер
+
+```bash
+# Debian/Ubuntu
+apt update && apt install -y nfs-kernel-server
+
+# RHEL/Fedora
+dnf install -y nfs-utils
+```
+
+### 3.4. Настроить экспорт
+
+Создать директорию:
+```bash
+mkdir -p /var/lib/k8s-nfs
+chmod 755 /var/lib/k8s-nfs
+```
+
+Добавить в `/etc/exports`:
+```
+/var/lib/k8s-nfs *(rw,sync,no_subtree_check,no_root_squash)
+```
+
+Применить:
+```bash
+exportfs -rav
+```
+
+Проверить:
+```bash
+showmount -e localhost
+```
+
+### 3.5. Выбрать способ интеграции с Kubernetes
+
+#### Вариант A: nfs-subdir-external-provisioner (проще)
+```bash
+helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/
+helm install nfs-provisioner nfs-subdir-external-provisioner/nfs-subdir-external-provisioner \
+  --namespace kube-system \
+  --set nfs.server=127.0.0.1 \
+  --set nfs.path=/var/lib/k8s-nfs \
+  --set storageClass.name=nfs \
+  --set storageClass.defaultClass=false \
+  --set storageClass.reclaimPolicy=Retain
+```
+
+#### Вариант B: NFS CSI Driver
+```bash
+helm repo add csi-driver-nfs https://raw.githubusercontent.com/kubernetes-csi/csi-driver-nfs/master/charts
+helm install csi-driver-nfs csi-driver-nfs/csi-driver-nfs --namespace kube-system
+```
+
+После установки CSI драйвера создать StorageClass:
+```yaml
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata:
+  name: nfs
+provisioner: nfs.csi.k8s.io
+parameters:
+  server: 127.0.0.1
+  share: /var/lib/k8s-nfs
+reclaimPolicy: Retain
+volumeBindingMode: Immediate
+```
+
+### 3.6. Проверить результат
+```bash
+kubectl get storageclass
+kubectl get pods -n kube-system | grep -E 'nfs|provisioner'
+```
+
+## 4. Итоговая конфигурация
+
+После внедрения в кластере будет два StorageClass:
+
+| Имя | Provisioner | ReclaimPolicy | Назначение |
+|---|---|---|---|
+| `local-path` (default) | rancher.io/local-path | Delete | Временные данные, stateless |
+| `nfs` | nfs-subdir-external-provisioner или nfs.csi.k8s.io | Retain | Данные, которые нужно сохранять |
+
+**Главное преимущество:** PVC c `storageClassName: nfs` при удалении namespace сохраняют данные на диске, так как NFS-провизор использует `reclaimPolicy: Retain` или файлы физически остаются в NFS-экспорте.
+
+## 5. Ответы на частые вопросы
+
+**В:** Не упадёт ли local-path при установке NFS?
+**О:** Нет, они независимы. local-path продолжает работать как обычно.
+
+**В:** Данные NFS и local-path будут на одном диске?
+**О:** Да, можно настроить оба на одном разделе, в разных каталогах.
+
+**В:** Что если у меня несколько нод?
+**О:** NFS сервер нужно поднять на одной ноде, а с других нод должна быть доступна шари. Для multi-node лучше использовать отдельный сервер или distributed storage (Longhorn, Rook/Ceph).
+
+**В:** Можно ли использовать существующий NTFS-раздел для NFS?
+**О:** Не рекомендуется — NTFS не поддерживает права Linux (no_root_squash не сработает корректно), возможны проблемы с блокировками и производительностью.

.editorconfig

@@ -0,0 +1,24 @@
+root = true
+
+[*]
+indent_style = space
+indent_size = 2
+end_of_line = lf
+charset = utf-8
+trim_trailing_whitespace = true
+insert_final_newline = true
+
+[*.{yml,yaml}]
+indent_size = 2
+
+[*.{json,jsonc}]
+indent_size = 2
+
+[*.md]
+trim_trailing_whitespace = false
+
+[*.py]
+indent_size = 4
+
+[{Makefile,makefile}]
+indent_style = tab

.gitea/actionlint.yaml

@@ -0,0 +1,10 @@
+# actionlint configuration. Passed explicitly from the ci workflow:
+#   actionlint -config-file .gitea/actionlint.yaml .gitea/workflows/*.yaml
+#
+# The self-hosted act_runner registers custom labels that actionlint cannot know
+# about, so declare them here instead of silencing the whole runner-label check.
+self-hosted-runner:
+  labels:
+    - arch
+    - homelab
+    - prod

.gitea/workflows/ci.yaml

@@ -0,0 +1,698 @@
+name: ci
+
+on:
+  push:
+    branches:
+      - "**"
+  pull_request:
+  workflow_dispatch:
+
+# Every job here is checkout plus local tools. The token needs to read the tree
+# and nothing else, and saying so keeps a future step that reaches for the API
+# from quietly holding a token that can write to the repository.
+permissions:
+  contents: read
+
+concurrency:
+  group: ci-${{ github.ref }}
+  cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
+
+env:
+  REGISTRY: gcr.forust.xyz
+
+jobs:
+  lint-compose:
+    runs-on: [self-hosted, linux, arch, homelab]
+    timeout-minutes: 10
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+
+      # Structure check for every committed Compose file, active or not.
+      # Interpolation, env-file and bind-mount resolution are all switched off,
+      # because inactive stacks have no .env here and would only fail on their
+      # ${VAR:?} guards. Active stacks get the full check with interpolation in
+      # the deploy workflow, where the real .env files live.
+      - name: Validate Compose files
+        shell: bash
+        run: |
+          set -euo pipefail
+          source .gitea/workflows/compose-lint.sh
+
+          mapfile -t safe_flags < <(compose_safe_flags)
+          echo "docker compose config ${safe_flags[*]-}"
+
+          mapfile -t files < <(compose_files)
+          if [ "${#files[@]}" -eq 0 ]; then
+            echo "No Compose files found."
+            exit 0
+          fi
+
+          failed=0
+          for f in "${files[@]}"; do
+            if ! out="$(validate_compose_file "$f" ${safe_flags[@]+"${safe_flags[@]}"} 2>&1)"; then
+              failed=1
+              echo "::error file=${f}::$(printf '%s' "$out" | head -1)"
+            fi
+          done
+
+          if [ "$failed" -ne 0 ]; then
+            echo "Compose validation failed."
+            exit 1
+          fi
+          echo "checked ${#files[@]} Compose file(s)"
+
+  lint-actionlint:
+    runs-on: [self-hosted, linux, arch, homelab]
+    timeout-minutes: 10
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+
+      - name: Lint Gitea Actions workflows with actionlint
+        shell: bash
+        run: |
+          set -euo pipefail
+          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh actionlint)"
+          export PATH="$tools_dir:$PATH"
+          actionlint -config-file .gitea/actionlint.yaml -color .gitea/workflows/*.yaml
+
+  lint-shellcheck:
+    runs-on: [self-hosted, linux, arch, homelab]
+    timeout-minutes: 10
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+
+      - name: Lint shell scripts with ShellCheck
+        shell: bash
+        run: |
+          set -euo pipefail
+          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh shellcheck)"
+          export PATH="$tools_dir:$PATH"
+          # userbot/ is a git subtree synced from forust/userbot, so its shell
+          # scripts are upstream's to maintain, not ours. Linting them would let a
+          # routine subtree pull turn the deploy gate red on code we do not own.
+          mapfile -t scripts < <(
+            git ls-files '*.sh' ':(glob)**/*.bash' ':!userbot/**'
+          )
+          if [ "${#scripts[@]}" -eq 0 ]; then
+            echo "No shell scripts found."
+            exit 0
+          fi
+          shellcheck --external-sources --source-path=SCRIPTDIR --severity=style "${scripts[@]}"
+
+  lint-prettier:
+    runs-on: [self-hosted, linux, arch, homelab]
+    timeout-minutes: 10
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+
+      - name: Check formatting with Prettier
+        shell: bash
+        run: |
+          set -euo pipefail
+          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh prettier)"
+          export PATH="$tools_dir:$PATH"
+
+          mapfile -t prettier_files < <(
+            git ls-files \
+              | grep -E '\.(md|json|ya?ml|html|css)$' \
+              | grep -Ev '^(\.docs/|\.zed/|errorpages/html/|homepages/(forust_files|xdfnx_files)/)'
+          )
+
+          if [ "${#prettier_files[@]}" -eq 0 ]; then
+            echo "No Prettier-managed files found."
+            exit 0
+          fi
+
+          prettier --check --ignore-unknown "${prettier_files[@]}"
+
+  lint-ruff:
+    runs-on: [self-hosted, linux, arch, homelab]
+    timeout-minutes: 10
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+
+      - name: Lint and format-check Python with Ruff
+        shell: bash
+        run: |
+          set -euo pipefail
+          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh ruff)"
+          export PATH="$tools_dir:$PATH"
+          ruff check .
+          ruff format --check .
+
+  lint-yaml:
+    runs-on: [self-hosted, linux, arch, homelab]
+    timeout-minutes: 10
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+
+      - name: Lint YAML syntax
+        shell: bash
+        run: |
+          set -euo pipefail
+          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh yamllint)"
+          export PATH="$tools_dir:$PATH"
+
+          mapfile -t yaml_files < <(
+            git ls-files '*.yaml' '*.yml' \
+              ':!node_modules/**' \
+              ':!**/.venv/**'
+          )
+
+          if [ "${#yaml_files[@]}" -eq 0 ]; then
+            echo "No YAML files found."
+            exit 0
+          fi
+
+          yamllint -c .yamllint "${yaml_files[@]}"
+
+  lint-dockerfiles:
+    runs-on: [self-hosted, linux, arch, homelab]
+    timeout-minutes: 10
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+
+      - name: Lint Dockerfiles
+        shell: bash
+        run: |
+          set -euo pipefail
+          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh hadolint)"
+          export PATH="$tools_dir:$PATH"
+
+          mapfile -t dockerfiles < <(
+            git ls-files ':(glob)**/Dockerfile' ':(glob)**/Dockerfile.*'
+          )
+
+          if [ "${#dockerfiles[@]}" -eq 0 ]; then
+            echo "No Dockerfiles found."
+            exit 0
+          fi
+
+          hadolint -c .hadolint.yaml "${dockerfiles[@]}"
+
+  # Known, accepted, and recorded. Each line is a real advisory against a
+  # package we build into the panel image, kept in this workflow rather than in
+  # the package manifest so that a subtree sync from forust/userbot cannot
+  # silently widen the exemption.
+  #
+  # starlette is the reason this job is not simply "fail on everything":
+  # fastapi 0.115.12 pins `starlette<0.47.0`, and the fixes for the last four
+  # below need 0.49.1 through 1.3.1, so clearing them means a jump from fastapi
+  # 0.115.12 to 0.141.x. That is upstream's call, not a drive-by in a lint
+  # commit. Of the seven, four are reachable here in principle: 1942 is a
+  # crafted Range header hitting FileResponse, and the panel serves its built
+  # SPA through exactly that; 249 is request.form() ignoring max_fields for
+  # x-www-form-urlencoded, which is the login form; 1941 is a large multipart
+  # body blocking the event loop; 161 and 248 are unvalidated Host and request
+  # path reaching request.url. 2280 needs HTTPEndpoint, which the panel does
+  # not use, and 2281 is Windows-only, and this deploys on Linux.
+  #
+  # The panel answers on userbot.workstation.internal and has no public
+  # forust.xyz route, which is what keeps the four reachable ones from being
+  # an internet-facing DoS. It still manages Telegram credentials.
+  #
+  # Deleting an entry here is how you accept a new advisory, so the diff says
+  # so out loud.
+  scan-deps:
+    runs-on: [self-hosted, linux, arch, homelab]
+    timeout-minutes: 15
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+
+      - name: Audit the Python dependencies that ship in the image
+        shell: bash
+        run: |
+          set -euo pipefail
+          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh pip-audit)"
+          export PATH="$tools_dir:$PATH"
+          # requirements.txt, not requirements-dev.txt: this is what the image
+          # installs, and the test tooling is not a shipped attack surface.
+          pip-audit -r userbot/panel/backend/requirements.txt --strict \
+            --ignore-vuln CVE-2025-67720 \
+            --ignore-vuln PYSEC-2026-161 \
+            --ignore-vuln PYSEC-2026-1941 \
+            --ignore-vuln PYSEC-2026-1942 \
+            --ignore-vuln PYSEC-2026-2280 \
+            --ignore-vuln PYSEC-2026-2281 \
+            --ignore-vuln PYSEC-2026-248 \
+            --ignore-vuln PYSEC-2026-249
+
+      # devDependencies are excluded on purpose. `npm audit` on the full tree
+      # reports 7 findings, and every one of them is a build- or test-time
+      # package: the esbuild CORS advisory needs a vite dev server serving to
+      # the internet, and nanoid's infinite loop needs a custom generator
+      # called with size 0, which postcss does not do. None of them are in the
+      # 91 kB bundle the panel serves. The one production finding, devalue
+      # via svelte, is moderate, which is where --audit-level draws the line;
+      # this fails on the next high or critical one.
+      - name: Audit the production npm dependencies
+        shell: bash
+        run: |
+          set -euo pipefail
+          # The pinned node, not whatever the runner has. Its system node is a
+          # rolling Arch package: during this very push its npm was missing
+          # entirely, and an hour later it was npm 12 on node 26. Both are the
+          # wrong major anyway — the panel image is node:22-alpine.
+          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh node)"
+          export PATH="$tools_dir:$PATH"
+          cd userbot/panel/frontend
+          npm ci
+          npm audit --omit=dev --audit-level=high
+
+  test-backend:
+    runs-on: [self-hosted, linux, arch, homelab]
+    timeout-minutes: 15
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+
+      # 25 tests over the panel's pydantic models, its auth flow, the SPA
+      # fallback and the Kubernetes client it shells out with. They existed and
+      # had never been executed by anything.
+      #
+      # Note that userbot/ is a subtree synced from forust/userbot, so a routine
+      # sync can turn this red on upstream's code. Unlike the shellcheck job,
+      # which skips that tree because style disagreements there are ours to
+      # lose, a failing test here is a real defect in a service we deploy.
+      - name: Run the panel backend test suite
+        shell: bash
+        run: |
+          set -euo pipefail
+          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh uv)"
+          export PATH="$tools_dir:$PATH"
+
+          # A venv in a temp dir rather than a checked-out one: the runner is
+          # shared, and a leftover .venv would let a dependency the
+          # requirements no longer pin still satisfy an import.
+          #
+          # --python is not optional. uv otherwise takes whatever interpreter it
+          # finds first, and which one that is depends on the machine: this
+          # runner runs jobs on the host, where the only interpreter is 3.14,
+          # and pyrogram's sync.py calls the bare asyncio.get_event_loop() that
+          # 3.14 no longer auto-creates, so three tests fail at collection. The
+          # image is python:3.13-slim, so 3.13 is also the version worth
+          # testing: uv fetches a managed build of it when the host has none,
+          # which is what makes this job independent of the runner.
+          venv="$(mktemp -d)/venv"
+          uv venv --python 3.13 --quiet "$venv"
+          uv pip install --quiet --python "$venv/bin/python" \
+            -r userbot/panel/backend/requirements-dev.txt
+
+          # `python -m`, not bare `pytest`: the tests import `app.*` relative to
+          # the backend directory, which only works if the cwd is on sys.path,
+          # and only `python -m` puts it there.
+          cd userbot/panel/backend
+          "$venv/bin/python" -m pytest tests/ -q
+
+  test-frontend:
+    runs-on: [self-hosted, linux, arch, homelab]
+    timeout-minutes: 15
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+
+      # One `npm ci` for both checks below: it is by far the slowest part of
+      # this job, and a second one would learn nothing the first did not.
+      #
+      # `npm ci`, not `npm install`, for the same reason the Dockerfile uses it:
+      # the lockfile is what makes the tree that gets checked the tree that
+      # gets shipped.
+      - name: Type-check and test the panel frontend
+        shell: bash
+        run: |
+          set -euo pipefail
+          # The pinned node, not whatever the runner has. Its system node is a
+          # rolling Arch package: during this very push its npm was missing
+          # entirely, and an hour later it was npm 12 on node 26. Both are the
+          # wrong major anyway — the panel image is node:22-alpine.
+          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh node)"
+          export PATH="$tools_dir:$PATH"
+          cd userbot/panel/frontend
+          npm ci
+
+          # svelte-check has been a devDependency all along with no script
+          # pointing at it, so the type errors it reports had nowhere to
+          # surface. It is clean today, which is the only reason it can be a
+          # gate: it stops at whatever upstream introduces rather than
+          # reporting a backlog we inherited.
+          npm run check
+          npm test
+
+  validate:
+    runs-on: [self-hosted, linux, arch, homelab]
+    timeout-minutes: 20
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+
+      - name: Validate Kubernetes manifests against JSON schemas
+        shell: bash
+        run: |
+          set -euo pipefail
+          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh kubeconform)"
+          export PATH="$tools_dir:$PATH"
+
+          mapfile -t manifests < <(
+            git ls-files ':(glob)**/k8s/**/*.yaml' ':(glob)**/k8s/**/*.yml' \
+              | grep -Ev '(^|/)(kustomization\.ya?ml|.*\.example\.ya?ml|.*values\.ya?ml|patch-.*\.ya?ml)$'
+          )
+
+          if [ "${#manifests[@]}" -eq 0 ]; then
+            echo "No Kubernetes manifests found."
+            exit 0
+          fi
+
+          kubeconform \
+            -strict \
+            -ignore-missing-schemas \
+            -summary \
+            "${manifests[@]}"
+
+      # kubeconform has no schemas for CRDs, so every IngressRoute, Certificate,
+      # PrometheusRule, Middleware, ServersTransport and ServiceMonitor is silently
+      # skipped above. The live API server knows the real CRD schemas (and runs the
+      # cert-manager / Traefik admission webhooks), so validate there too.
+      #
+      # Only services marked with a k8s/active marker are checked: server-side
+      # dry-run needs the target namespace to exist, and inactive services are not
+      # deployed. Services being enabled for the first time are still covered by
+      # the JSON-schema pass above.
+      #
+      # Main pushes only. `--dry-run=server` persists nothing, but it does execute
+      # the admission webhooks of the production API server, so anyone able to open
+      # a pull request would be able to run arbitrary manifest content through
+      # cert-manager and Traefik. A pull request has nothing to gain from it either:
+      # only main is ever deployed, and this job runs to completion before the
+      # deploy workflow is allowed to start, so a bad CRD is still caught before
+      # anything reaches the cluster -- just on the push rather than on the PR.
+      - name: Note the server-side check is not running here
+        if: github.event_name == 'pull_request' || github.ref != 'refs/heads/main'
+        shell: bash
+        run: |
+          echo "::notice::Skipping the server-side dry-run. It executes the cert-manager and" \
+            "Traefik admission webhooks against the production API server, so it is limited" \
+            "to pushes to main. CRDs are still schema-checked by kubeconform above, and the" \
+            "server-side pass still runs on main before the deploy."
+
+      - name: Validate active manifests against the live API server
+        if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main'
+        shell: bash
+        run: |
+          set -euo pipefail
+
+          if ! kubectl get --raw='/readyz' --request-timeout=10s >/dev/null 2>&1; then
+            echo "::warning::Cluster unreachable — skipped server-side validation of CRDs (IngressRoute, Certificate, PrometheusRule). Review manifest changes manually."
+            exit 0
+          fi
+
+          mapfile -t k8s_dirs < <(
+            git ls-files '*.yaml' '*.yml' \
+              | grep -E '(^|/)k8s/' \
+              | sed -E 's#((^|.*/)k8s)/.*#\1#' \
+              | sort -u
+          )
+
+          manifests=()
+          kustomize_apps=()
+          for dir in "${k8s_dirs[@]}"; do
+            if [ ! -f "${dir}/active" ]; then
+              echo "skip (no k8s/active): ${dir}"
+              continue
+            fi
+            if [ -f "${dir}/overlays/prod/kustomization.yaml" ]; then
+              kustomize_apps+=("${dir}/overlays/prod")
+            elif [ -f "${dir}/base/kustomization.yaml" ]; then
+              kustomize_apps+=("${dir}/base")
+            else
+              while IFS= read -r f; do
+                [ -n "$f" ] && manifests+=("$f")
+              done < <(
+                git ls-files "${dir}/*.yaml" "${dir}/*.yml" \
+                  | grep -Ev '(^|/)(kustomization\.ya?ml|.*\.example\.ya?ml|.*values\.ya?ml|patch-.*\.ya?ml)$'
+              )
+            fi
+          done
+
+          echo "server-side dry-run: ${#manifests[@]} manifests, ${#kustomize_apps[@]} kustomize apps"
+          failed=0
+          for m in ${manifests[@]+"${manifests[@]}"}; do
+            if ! out="$(kubectl apply --dry-run=server -f "$m" 2>&1)"; then
+              failed=1
+              echo "::error file=${m}::$(printf '%s' "$out" | head -1)"
+            fi
+          done
+          for k in ${kustomize_apps[@]+"${kustomize_apps[@]}"}; do
+            if ! out="$(kubectl apply -k "$k" --dry-run=server 2>&1)"; then
+              failed=1
+              echo "::error file=${k}::$(printf '%s' "$out" | head -1)"
+            fi
+          done
+
+          if [ "$failed" -ne 0 ]; then
+            echo "Server-side validation failed. The API server (or an admission webhook) rejected these manifests."
+            exit 1
+          fi
+          echo "server-side dry-run: all active manifests accepted by the API server"
+
+  build:
+    needs:
+      [lint-actionlint, lint-shellcheck, lint-compose, lint-prettier, lint-ruff, lint-yaml, lint-dockerfiles, validate]
+    if: github.event_name != 'pull_request' && (github.ref_name == 'main' || github.ref_name == 'dev')
+    runs-on: [self-hosted, linux, arch, homelab]
+    timeout-minutes: 60
+    outputs:
+      services: ${{ steps.services.outputs.services }}
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+        with:
+          fetch-depth: 0
+
+      - name: Detect changed docker-built services
+        id: services
+        shell: bash
+        run: |
+          base="${{ github.event.before }}"
+          if [ -z "$base" ] || [ "$base" = "0000000000000000000000000000000000000000" ]; then
+            base="$(git rev-list --max-parents=0 HEAD)"
+          fi
+
+          mapfile -t changed_files < <(git diff --name-only "$base" "${GITHUB_SHA}")
+
+          services=()
+
+          add_service() {
+            local name="$1"
+            local seen=0
+            for existing in "${services[@]}"; do
+              if [ "$existing" = "$name" ]; then
+                seen=1
+                break
+              fi
+            done
+            if [ "$seen" -eq 0 ]; then
+              services+=("$name")
+            fi
+          }
+
+          for file in "${changed_files[@]}"; do
+            case "$file" in
+              dtek_notif/*)
+                add_service dtek_notif
+                ;;
+              errorpages/*)
+                add_service errorpages
+                ;;
+              userbot/*)
+                add_service userbot
+                ;;
+              homepages/*)
+                add_service homepages
+                ;;
+              edu_master/phpsessid-bot/*|edu_master/webinar-checker/*|edu_master/compose.yaml)
+                add_service edu_master
+                ;;
+            esac
+          done
+
+          if [ "${#services[@]}" -eq 0 ]; then
+            echo "No docker-built services changed."
+            echo "services=" >> "$GITHUB_OUTPUT"
+            exit 0
+          fi
+
+          printf '%s\n' "${services[@]}" | tee /tmp/services.txt
+          echo "services=$(paste -sd, /tmp/services.txt)" >> "$GITHUB_OUTPUT"
+
+      - name: Log in to registry
+        if: steps.services.outputs.services != ''
+        shell: bash
+        run: |
+          echo "${{ secrets.REGISTRY_PASSWORD }}" | docker login "${REGISTRY}" \
+            -u "${{ secrets.REGISTRY_USERNAME }}" \
+            --password-stdin
+
+      - name: Build and push changed images
+        if: steps.services.outputs.services != ''
+        shell: bash
+        run: |
+          IFS=, read -r -a services <<< "${{ steps.services.outputs.services }}"
+
+          for service in "${services[@]}"; do
+            case "$service" in
+              dtek_notif)
+                image="${REGISTRY}/forust/dtek-notif"
+                tags=()
+                case "${GITHUB_REF_NAME}" in
+                  main)
+                    tags+=("main" "prod")
+                    ;;
+                  dev)
+                    tags+=("dev")
+                    ;;
+                esac
+                build_args=()
+                for tag in "${tags[@]}"; do
+                  build_args+=(-t "${image}:${tag}")
+                done
+                docker build \
+                  --cache-from "type=registry,ref=${image}:buildcache" \
+                  --cache-to "type=registry,ref=${image}:buildcache,mode=max" \
+                  "${build_args[@]}" dtek_notif
+                for tag in "${tags[@]}"; do
+                  docker push "${image}:${tag}"
+                done
+                ;;
+              errorpages)
+                image="${REGISTRY}/forust/error-pages"
+                tags=()
+                case "${GITHUB_REF_NAME}" in
+                  main)
+                    tags+=("main" "prod")
+                    ;;
+                  dev)
+                    tags+=("dev")
+                    ;;
+                esac
+                build_args=()
+                for tag in "${tags[@]}"; do
+                  build_args+=(-t "${image}:${tag}")
+                done
+                docker build \
+                  --cache-from "type=registry,ref=${image}:buildcache" \
+                  --cache-to "type=registry,ref=${image}:buildcache,mode=max" \
+                  "${build_args[@]}" errorpages
+                for tag in "${tags[@]}"; do
+                  docker push "${image}:${tag}"
+                done
+                ;;
+              userbot)
+                tags=()
+                case "${GITHUB_REF_NAME}" in
+                  main)
+                    tags+=("main" "prod")
+                    ;;
+                  dev)
+                    tags+=("dev")
+                    ;;
+                esac
+                for target in runtime panel; do
+                  case "$target" in
+                    runtime)
+                      context="userbot"
+                      image="${REGISTRY}/forust/userbot"
+                      ;;
+                    panel)
+                      context="userbot/panel"
+                      image="${REGISTRY}/forust/userbot-panel"
+                      ;;
+                  esac
+                  build_args=()
+                  for tag in "${tags[@]}"; do
+                    build_args+=(-t "${image}:${tag}")
+                  done
+                  docker build \
+                    --cache-from "type=registry,ref=${image}:buildcache" \
+                    --cache-to "type=registry,ref=${image}:buildcache,mode=max" \
+                    "${build_args[@]}" "$context"
+                  for tag in "${tags[@]}"; do
+                    docker push "${image}:${tag}"
+                  done
+                done
+                ;;
+              homepages)
+                for variant in forust xdfnx; do
+                  case "$variant" in
+                    forust)
+                      image="${REGISTRY}/forust/forust-homepage"
+                      ;;
+                    xdfnx)
+                      image="${REGISTRY}/forust/xdfnx-homepage"
+                      ;;
+                  esac
+                  tags=()
+                  case "${GITHUB_REF_NAME}" in
+                    main)
+                      tags+=("main" "prod")
+                      ;;
+                    dev)
+                      tags+=("dev")
+                      ;;
+                  esac
+                  build_args=()
+                  for tag in "${tags[@]}"; do
+                    build_args+=(-t "${image}:${tag}")
+                  done
+                  docker build \
+                    --cache-from "type=registry,ref=${image}:buildcache" \
+                    --cache-to "type=registry,ref=${image}:buildcache,mode=max" \
+                    "${build_args[@]}" -f "homepages/Dockerfile.${variant}" homepages
+                  for tag in "${tags[@]}"; do
+                    docker push "${image}:${tag}"
+                  done
+                done
+                ;;
+              edu_master)
+                for variant in session-keeper webinar-checker; do
+                  case "$variant" in
+                    session-keeper)
+                      context="edu_master/phpsessid-bot"
+                      image="${REGISTRY}/forust/session-keeper"
+                      ;;
+                    webinar-checker)
+                      context="edu_master/webinar-checker"
+                      image="${REGISTRY}/forust/webinar-checker"
+                      ;;
+                  esac
+                  tags=()
+                  case "${GITHUB_REF_NAME}" in
+                    main)
+                      tags+=("main" "prod")
+                      ;;
+                    dev)
+                      tags+=("dev")
+                      ;;
+                  esac
+                  build_args=()
+                  for tag in "${tags[@]}"; do
+                    build_args+=(-t "${image}:${tag}")
+                  done
+                  docker build \
+                    --cache-from "type=registry,ref=${image}:buildcache" \
+                    --cache-to "type=registry,ref=${image}:buildcache,mode=max" \
+                    "${build_args[@]}" "$context"
+                  for tag in "${tags[@]}"; do
+                    docker push "${image}:${tag}"
+                  done
+                done
+                ;;
+            esac
+          done

.gitea/workflows/compose-lint.sh

@@ -0,0 +1,46 @@
+#!/usr/bin/env bash
+# Shared helpers for validating Compose files. Sourced both by steps in
+# .gitea/workflows/ci.yaml and by deploy-lib.sh on the workstation.
+#
+# Two levels of checking, matching how the repo is structured:
+#
+#   general  every committed Compose file, active or not. Pure structure check:
+#            no ${VAR} interpolation, no .env lookup, no bind-mount path
+#            resolution. Disabled stacks deliberately have no .env in the repo
+#            and no values on the CI runner, so a full `config` run would fail on
+#            their `${VAR:?}` guards for reasons that have nothing to do with the
+#            change under review.
+#
+#   full     active stacks only, with interpolation and env-file resolution, so
+#            required variables and referenced files are actually resolved. Needs
+#            the gitignored .env files, so this only runs in the deploy workflow
+#            on the workstation.
+#
+# This file is meant to be sourced, not executed.
+
+# All committed Compose files, including the ones deploy never starts.
+compose_files() {
+  git ls-files \
+    '*/compose.yaml' '*/compose.yml' 'compose.yaml' 'compose.yml' \
+    '*/docker-compose.yaml' '*/docker-compose.yml'
+}
+
+# Prints the flags that turn `docker compose config` into the general check.
+# Probed rather than hardcoded so an older Compose without --no-env-resolution
+# still gets the flags it does support.
+compose_safe_flags() {
+  local help flag
+  help="$(docker compose config --help 2>/dev/null || true)"
+  for flag in --no-interpolate --no-env-resolution --no-path-resolution; do
+    if printf '%s' "$help" | grep -q -- "$flag"; then
+      printf '%s\n' "$flag"
+    fi
+  done
+}
+
+# validate_compose_file <file> [extra docker compose config flags...]
+validate_compose_file() {
+  local file="$1"
+  shift
+  docker compose -f "$file" config --quiet "$@"
+}

.gitea/workflows/deploy-lib.sh

@@ -0,0 +1,937 @@
+#!/usr/bin/env bash
+# Shared stages for the deploy workflow. Runs on the workstation, invoked as:
+#   REPO=/srv/homelab APPLY_PRUNE=false bash -se <<'EOF'
+#   source "$REPO/.gitea/workflows/deploy-lib.sh"
+#   run_stage "$STAGE"
+#   EOF
+set -euo pipefail
+
+: "${REPO:?REPO must be set}"
+APPLY_PRUNE="${APPLY_PRUNE:-false}"
+# Commit CI validated. Empty for a manual workflow_dispatch, which falls back to
+# the current origin/main.
+DEPLOY_SHA="${DEPLOY_SHA:-}"
+# Handoff point between the apply stage (writes) and the verify stage (reads).
+# Under the deploy user's own XDG state directory rather than /var/backups: the
+# deploy is unprivileged, /var/backups does not exist on a minimal Arch host, and
+# creating it would need root — which is why the first real deploy died here with
+# "is not writable" before touching a single workload. $HOME comes from sshd.
+DEPLOY_SNAPSHOT_DIR="${DEPLOY_SNAPSHOT_DIR:-${XDG_STATE_HOME:-$HOME/.local/state}/homelab-deploy}"
+# Per-workload rollout budget and how many workloads to watch at once. The whole
+# apply job has its own timeout-minutes as a backstop.
+ROLLOUT_TIMEOUT="${ROLLOUT_TIMEOUT:-300}"
+ROLLOUT_PARALLELISM="${ROLLOUT_PARALLELISM:-8}"
+WORKLOAD_KINDS="deployments.apps,statefulsets.apps,daemonsets.apps"
+
+log() {
+  echo "== $* =="
+}
+
+warn() {
+  echo "WARNING: $*" >&2
+}
+
+collect_k8s() {
+  git -C "$REPO" ls-files -- "$1" \
+    | grep -E '\.ya?ml$' \
+    | grep -Ev '/overlays/' \
+    | grep -Ev '(^|/)(kustomization\.ya?ml|.*\.example\.ya?ml|.*values\.ya?ml|patch-.*\.ya?ml)$' \
+    | grep -Ev '(^|/)[^/]*secret[^/]*\.ya?ml$' \
+    | sort
+}
+
+kustomize_overlay() {
+  if [ -f "$1/overlays/prod/kustomization.yaml" ]; then
+    echo "$1/overlays/prod"
+  elif [ -f "$1/base/kustomization.yaml" ]; then
+    echo "$1/base"
+  elif [ -f "$1/kustomization.yaml" ]; then
+    echo "$1"
+  fi
+}
+
+select_manifests() {
+  K8S_MANIFESTS=()
+  KUSTOMIZE_APPS=()
+  COMPOSE_STACKS=()
+  local kd_rel kd overlay cf_rel cf f
+  while IFS= read -r kd_rel; do
+    kd="$REPO/$kd_rel"
+    if [ ! -f "$kd/active" ]; then
+      echo "skip (no k8s/active): $kd_rel"
+      continue
+    fi
+    overlay="$(kustomize_overlay "$kd" || true)"
+    if [ -n "${overlay:-}" ]; then
+      echo "kustomize app: ${overlay#"$REPO"/}"
+      KUSTOMIZE_APPS+=("$overlay")
+    else
+      while IFS= read -r f; do
+        [ -n "$f" ] && K8S_MANIFESTS+=("$REPO/$f")
+      done < <(collect_k8s "$kd_rel" || true)
+    fi
+  done < <(
+    git -C "$REPO" ls-files '*.yaml' '*.yml' \
+      | grep -E '(^|/)k8s/' \
+      | sed -E 's#((^|.*/)k8s)/.*#\1#' \
+      | sort -u
+  )
+  while IFS= read -r cf_rel; do
+    cf="$REPO/$cf_rel"
+    if [ -f "$(dirname "$cf")/active" ]; then
+      echo "compose: $cf_rel"
+      COMPOSE_STACKS+=("$cf")
+    else
+      echo "skip (no root active): $cf_rel"
+    fi
+  done < <(git -C "$REPO" ls-files '*/compose.yaml' '*/compose.yml' compose.yaml compose.yml | sort)
+}
+
+# --- post-apply verification and rollback -------------------------------------
+#
+# A green `kubectl apply` says nothing about the cluster being healthy. These
+# helpers watch exactly the workloads whose spec changed during this apply, and
+# on failure roll them back to the revision that was running before, so a bad
+# push to main cannot leave a service crash-looping.
+#
+# Verification lives in its own workflow job, not at the end of the apply stage.
+# Inside a single process it is worthless exactly when it is needed most: a job
+# killed by timeout-minutes or cancelled mid-apply never reaches the rollback
+# code, and leaves a half-applied cluster behind. Split out, the apply job can
+# die in any way and the verify job still runs.
+#
+# That split needs a handoff point on the workstation, because the two stages are
+# separate processes on separate runner jobs: DEPLOY_SNAPSHOT_DIR/current, written
+# before anything is applied, read by the verify stage afterwards.
+
+# Creates this run's snapshot directory and publishes it as the handoff point for
+# the verify stage. Fails hard by design: a deploy that cannot record what it is
+# about to change must not start, because then nothing can be rolled back for it
+# automatically. Publishing happens before the first apply, so an apply killed
+# mid-flight still leaves a usable baseline behind.
+snapshot_dir() {
+  local stamp dir
+  stamp="$(date -u +%Y%m%dT%H%M%SZ)-${DEPLOY_SHA:-$(git -C "$REPO" rev-parse --short HEAD 2>/dev/null || echo unknown)}"
+  dir="$DEPLOY_SNAPSHOT_DIR/$stamp"
+
+  if ! mkdir -p "$DEPLOY_SNAPSHOT_DIR" 2>/dev/null || [ ! -w "$DEPLOY_SNAPSHOT_DIR" ]; then
+    echo "ERROR: $DEPLOY_SNAPSHOT_DIR is not writable." >&2
+    echo "The verify job needs it to learn which workloads this deploy touches." >&2
+    echo "Refusing to deploy without a way to roll back." >&2
+    return 1
+  fi
+  if ! mkdir -p "$dir" 2>/dev/null || [ ! -w "$dir" ]; then
+    echo "ERROR: cannot create snapshot dir $dir" >&2
+    return 1
+  fi
+  if ! printf '%s\n' "$dir" >"$DEPLOY_SNAPSHOT_DIR/current" 2>/dev/null; then
+    echo "ERROR: cannot publish the snapshot pointer at $DEPLOY_SNAPSHOT_DIR/current" >&2
+    return 1
+  fi
+
+  printf '%s\n' "$dir"
+}
+
+save_snapshot() {
+  local dir="$1"
+  log "Saving pre-apply snapshot to $dir"
+  workload_generations >"$dir/generations.before" 2>/dev/null \
+    || warn "could not snapshot workload generations"
+  kubectl get "$WORKLOAD_KINDS" -A -o yaml >"$dir/workloads.yaml" 2>/dev/null \
+    || warn "could not snapshot workloads"
+  for release in prometheus-stack loki alloy; do
+    if helm status "$release" -n prometheus >/dev/null 2>&1; then
+      {
+        echo "revision: $(helm history "$release" -n prometheus -o json 2>/dev/null)"
+        helm get values "$release" -n prometheus --all 2>/dev/null
+      } >"$dir/helm-$release.txt"
+    fi
+  done
+  # The verify stage compares this against the commit it is deploying, to refuse
+  # rolling back against a baseline left by an earlier run. A snapshot we cannot
+  # attribute to a commit is unusable for that, so fail before anything is applied.
+  if ! git -C "$REPO" rev-parse HEAD >"$dir/commit" 2>/dev/null; then
+    echo "ERROR: cannot record the deploy commit in $dir/commit" >&2
+    return 1
+  fi
+}
+
+# Prints "<ns> <name> <kind> <generation>" for every workload in the cluster.
+workload_generations() {
+  kubectl get "$WORKLOAD_KINDS" -A \
+    -o 'custom-columns=NS:.metadata.namespace,NAME:.metadata.name,KIND:.kind,GEN:.metadata.generation' \
+    --no-headers 2>/dev/null \
+    | awk 'NF >= 4 { printf "%s %s %s %s\n", $1, $2, tolower($3), $4 }'
+}
+
+# Prints "<kind> <ns> <name>" for every workload that is new or whose generation
+# moved since the snapshot, i.e. the ones this apply actually touched.
+changed_workloads() {
+  local before="$1"
+  local ns name kind gen old
+  while read -r ns name kind gen; do
+    [ -n "${gen:-}" ] || continue
+    old="$(awk -v want_ns="$ns" -v want_name="$name" \
+      '$1 == want_ns && $2 == want_name { print $4; exit }' "$before" 2>/dev/null || true)"
+    if [ "$old" != "$gen" ]; then
+      printf '%s %s %s\n' "$kind" "$ns" "$name"
+    fi
+  done < <(workload_generations)
+}
+
+# Prints "<ns> <kind>/<name> <image>" for every workload this repository owns that
+# runs an image from our own registry.
+#
+# The repository is the scope, deliberately. The cluster also holds workloads on
+# our registry that no manifest here declares (they are applied out of band), and
+# those are somebody else's to deploy. Walking the manifests rather than the
+# cluster means those can never be restarted by this pipeline, now or later.
+owned_registry_workloads() {
+  local kd_rel f
+  while IFS= read -r kd_rel; do
+    [ -f "$REPO/$kd_rel/active" ] || continue
+    while IFS= read -r f; do
+      [ -n "$f" ] || continue
+      # A file that does not mention the registry cannot declare a workload on it,
+      # and parsing costs ~2.5s per file against a millisecond for the grep. The
+      # filter keeps this at a handful of parses instead of one per manifest.
+      grep -q 'gcr\.forust\.xyz/forust/' "$REPO/$f" 2>/dev/null || continue
+      # kubectl prints a bare object for a single-document file and a List for a
+      # multi-document one, so normalise both shapes before filtering.
+      kubectl apply --dry-run=client -f "$REPO/$f" -o json 2>/dev/null \
+        | jq -r '
+            (if .items then .items[] else . end)
+            | select(.kind | test("^(Deployment|StatefulSet|DaemonSet)$"))
+            | select(any((.spec.template.spec.containers // [])[]?;
+                       (.image // "") | test("^gcr\\.forust\\.xyz/forust/")))
+            | (.metadata.namespace // "default") as $ns
+            | ([.spec.template.spec.containers[].image
+                | select(test("^gcr\\.forust\\.xyz/forust/"))][0]) as $img
+            | "\($ns) \(.kind | ascii_downcase)/\(.metadata.name) \($img)"
+          ' 2>/dev/null || true
+    done < <(collect_k8s "$kd_rel" || true)
+  done < <(
+    git -C "$REPO" ls-files '*.yaml' '*.yml' \
+      | grep -E '(^|/)k8s/' \
+      | sed -E 's#((^|.*/)k8s)/.*#\1#' \
+      | sort -u
+  )
+}
+
+# Prints the digest an image tag resolves to for this cluster's architecture, or
+# nothing when it cannot be resolved.
+#
+# Only the manifest entry matching the node architecture counts. A multi-arch tag
+# also carries `unknown/unknown` entries for the build attestation, and a pod's
+# imageID is always the per-platform digest, so comparing the wrong entry would
+# mark every workload stale forever and restart the whole cluster on every deploy.
+registry_digest() {
+  local arch
+  arch="$(kubectl get nodes -o jsonpath='{.items[0].status.nodeInfo.architecture}' 2>/dev/null || true)"
+  [ -n "$arch" ] || arch=amd64
+  # The || true is load-bearing. Every caller runs under set -euo pipefail, and
+  # pipefail reports the rightmost non-zero stage, so a ref the registry does not
+  # have would abort the caller at the assignment instead of yielding an empty
+  # string. The callers check for empty themselves and report it by name.
+  docker manifest inspect "$1" 2>/dev/null \
+    | jq -r --arg arch "$arch" '
+        .manifests[]?
+        | select(.platform.os == "linux" and .platform.architecture == $arch)
+        | .digest
+      ' 2>/dev/null \
+    | head -1 || true
+}
+
+# Rewrites our own images to immutable digests on the way into the cluster.
+# Reads a manifest stream on stdin, writes the pinned stream to stdout.
+#
+# A digest is not knowable when a manifest is written, so it is resolved here, at
+# apply time, and never committed: git keeps a readable `:prod` tag. That is what
+# makes rollback mean something. `kubectl rollout undo` restores the previous
+# ReplicaSet's pod template verbatim, and a template naming a digest restores the
+# exact bytes that were serving before. A template naming a moving tag does not —
+# the tag has already moved by the time the rollback runs, so the "rollback"
+# re-pulls the very image that just failed and the cluster stays broken.
+#
+# imagePullPolicy is deliberately left alone. The manifests no longer set it, and a
+# reference that is not `:latest` defaults to IfNotPresent, which is what the
+# Kubernetes docs ask for alongside a digest: the bytes under a digest cannot
+# change, so pulling again buys nothing.
+#
+# An image that cannot be resolved is fatal. Carrying on would quietly apply a
+# mutable tag again, which is the exact failure this function exists to remove.
+render_pinned() {
+  local src refs map ref digest missing=0
+  src="$(mktemp)"
+  refs="$(mktemp)"
+  map="$(mktemp)"
+
+  cat >"$src"
+  grep -oE 'gcr\.forust\.xyz/forust/[A-Za-z0-9._-]+:[A-Za-z0-9._-]+' "$src" | sort -u >"$refs" || true
+
+  while read -r ref; do
+    [ -n "$ref" ] || continue
+    digest="$(registry_digest "$ref")"
+    if [ -z "$digest" ]; then
+      echo "ERROR: cannot resolve ${ref} in the registry; applying nothing." >&2
+      echo "       The build job has to push that tag before the deploy resolves it." >&2
+      missing=$((missing + 1))
+      continue
+    fi
+    printf '%s\t%s\n' "$ref" "$digest" >>"$map"
+  done <"$refs"
+  if [ "$missing" -gt 0 ]; then
+    rm -f "$src" "$refs" "$map"
+    return 1
+  fi
+
+  awk -v mapfile="$map" '
+    BEGIN {
+      while ((getline line < mapfile) > 0) {
+        i = index(line, "\t")
+        d[substr(line, 1, i - 1)] = substr(line, i + 1)
+      }
+    }
+    {
+      if (match($0, /^[[:space:]]*image:[[:space:]]*gcr\.forust\.xyz\/forust\/[A-Za-z0-9._-]+:[A-Za-z0-9._-]+[[:space:]]*$/)) {
+        name = $0
+        sub(/^[[:space:]]*image:[[:space:]]*/, "", name)
+        sub(/[[:space:]]*$/, "", name)
+        if (name in d) {
+          pad = $0
+          sub(/image:.*/, "", pad)
+          # Drop the tag: the canonical form used in the docs is repo@sha256:...,
+          # and leaving :prod next to the digest reads like it still matters.
+          repo = name
+          sub(/:[A-Za-z0-9._-]+$/, "", repo)
+          print pad "image: " repo "@" d[name]
+          next
+        }
+      }
+      print
+    }
+  ' "$src"
+  rm -f "$src" "$refs" "$map"
+}
+
+# Restarts every owned workload whose running image is not the one its tag
+# resolves to now.
+#
+# This used to be how a rebuild reached the cluster at all: the manifests pinned
+# `:latest`, so a rebuild left the pod template byte-identical, `kubectl apply`
+# decided there was nothing to do, and the cluster served the previous build
+# indefinitely. The apply now pins digests via render_pinned, so a rebuild moves
+# the pod template and rolls out on its own.
+#
+# What is left is the drift check: a hand-run `kubectl set image`, or anything
+# else that edits a live workload behind the deploy's back, is the only way to end
+# up serving a digest the tag has moved past. It stays idempotent, so a redeploy
+# that changed no image still does not bounce healthy services.
+#
+# The container is matched on its repository rather than on the exact reference:
+# once render_pinned has run, a pod's status reports `repo@sha256:...` while this
+# still reads the repository's `:prod` tag out of the manifest.
+restart_stale_images() {
+  local ns target image want selector running entry one
+  local unchecked=0
+  local -A digests=()
+  local -a stale=()
+  while read -r ns target image; do
+    [ -n "${target:-}" ] || continue
+    if [ -z "${digests[$image]:-}" ]; then
+      digests[$image]="$(registry_digest "$image")"
+    fi
+    want="${digests[$image]}"
+    if [ -z "$want" ]; then
+      warn "cannot resolve ${image##*/} in the registry, leaving $target alone"
+      unchecked=$((unchecked + 1))
+      continue
+    fi
+    selector="$(kubectl get "$target" -n "$ns" -o jsonpath='{.spec.selector.matchLabels}' 2>/dev/null \
+      | jq -r 'to_entries | map("\(.key)=\(.value)") | join(",")' 2>/dev/null)"
+    if [ -z "$selector" ]; then
+      warn "cannot read the pod selector of $target, skipping"
+      unchecked=$((unchecked + 1))
+      continue
+    fi
+    running="$(kubectl get pods -n "$ns" -l "$selector" -o json 2>/dev/null \
+      | jq -r --arg repo "${image%%:*}" '
+          .items[] | .status.containerStatuses[]?
+          | select(.image == $repo
+                   or (.image | startswith($repo + ":"))
+                   or (.image | startswith($repo + "@")))
+          | .imageID
+        ' 2>/dev/null)"
+    if [ -z "$running" ]; then
+      # Scaled to zero. Nothing is serving stale code, and imagePullPolicy
+      # resolves the tag when it is scaled back up.
+      continue
+    fi
+    entry=""
+    while IFS= read -r one; do
+      [ -n "$one" ] || continue
+      entry="${one##*@}"
+      if [ "$entry" != "$want" ]; then
+        stale+=("$ns $target")
+        break
+      fi
+    done <<<"$running"
+  done < <(owned_registry_workloads)
+  if [ "${#stale[@]}" -eq 0 ]; then
+    if [ "$unchecked" -gt 0 ]; then
+      # Say so plainly. Reporting "everything is current" after checking nothing
+      # would tell the operator the deploy is fine when it may not be.
+      warn "No workload needed a restart, but $unchecked could not be checked"
+    else
+      log "All owned workloads already run the image their tag points at"
+    fi
+    return 0
+  fi
+  log "Restarting ${#stale[@]} workload(s) running an image their tag has moved past"
+  for ref in "${stale[@]}"; do
+    log "  $ref"
+  done
+  local failed=()
+  for ref in "${stale[@]}"; do
+    ns="${ref%% *}"
+    target="${ref#* }"
+    if ! kubectl rollout restart "$target" -n "$ns" >/dev/null 2>&1; then
+      failed+=("$ref")
+    fi
+  done
+  if [ "${#failed[@]}" -gt 0 ]; then
+    warn "could not restart: ${failed[*]}"
+    return 1
+  fi
+}
+
+# verify_workloads <failed-file> <kind> <ns> <name> ...
+# Watches every workload in parallel and records the ones that never became
+# healthy. Returns non-zero if any of them failed.
+verify_workloads() {
+  local failed_file="$1"
+  shift
+  [ "$#" -gt 0 ] || return 0
+  : >"$failed_file"
+  local running=0 pid kind ns name
+  local -a pids=()
+  for entry in "$@"; do
+    read -r kind ns name <<<"$entry"
+    (
+      if kubectl rollout status "${kind}/${name}" -n "$ns" --timeout="${ROLLOUT_TIMEOUT}s" >/dev/null 2>&1; then
+        echo "  ok: ${kind}/${ns}/${name}"
+      else
+        echo "  FAILED: ${kind}/${ns}/${name}"
+        printf '%s %s %s\n' "$kind" "$ns" "$name" >>"$failed_file"
+      fi
+    ) &
+    pids+=($!)
+    running=$((running + 1))
+    if [ "$running" -ge "$ROLLOUT_PARALLELISM" ]; then
+      wait -n 2>/dev/null || true
+      running=$((running - 1))
+    fi
+  done
+  for pid in ${pids[@]+"${pids[@]}"}; do
+    wait "$pid" || true
+  done
+  # Non-zero when the file holds at least one failure, i.e. a workload never
+  # became healthy. `[ -s ]` alone is the opposite test and silently disabled
+  # every rollback this stage is meant to perform.
+  [ ! -s "$failed_file" ]
+}
+
+# rollback_workloads <failed-file>
+# Restores the previous revision of every failed workload and waits for it to
+# settle. Prints a report and returns non-zero if any workload is still unhealthy,
+# so the operator knows manual recovery is required.
+rollback_workloads() {
+  local failed_file="$1"
+  local kind ns name unrecovered=()
+  local -a recovered=()
+  while read -r kind ns name; do
+    [ -n "${kind:-}" ] || continue
+    if kubectl rollout undo "${kind}/${name}" -n "$ns" >/dev/null 2>&1 \
+      && kubectl rollout status "${kind}/${name}" -n "$ns" --timeout="${ROLLOUT_TIMEOUT}s" >/dev/null 2>&1; then
+      echo "  rolled back: ${kind}/${ns}/${name}"
+      recovered+=("${kind}/${ns}/${name}")
+    else
+      echo "  NOT RECOVERED: ${kind}/${ns}/${name}"
+      unrecovered+=("${kind}/${ns}/${name}")
+    fi
+  done <"$failed_file"
+  echo "ROLLED_BACK=${#recovered[@]}" >>"$failed_file"
+  echo "UNRECOVERED=${#unrecovered[@]}" >>"$failed_file"
+  [ "${#unrecovered[@]}" -eq 0 ]
+}
+
+# Helm releases owned by this stage, one line each:
+#
+#   release|chart|namespace|chart version|values file (rel. to $REPO)|active marker
+#
+# The chart version is the field Renovate keeps current. The helmv3 manager only
+# understands Chart.yaml and the helm-values manager only values files, so a pin
+# written straight into a `helm upgrade` command would never be updated: these
+# have to be declared as custom.regex managers in renovate/renovate.json.
+HELM_RELEASES=(
+  "prometheus-stack|prometheus-community/kube-prometheus-stack|prometheus|86.2.3|prometheus-stack/k8s/grafana-values.yaml|prometheus-stack/k8s/active"
+  "loki|grafana/loki|prometheus|7.3.0|loki/k8s/loki-values.yaml|loki/k8s/active"
+  "alloy|grafana/alloy|prometheus|1.12.1|loki/k8s/alloy-values.yaml|loki/k8s/active"
+  "reloader|stakater/reloader|reloader|2.2.17|reloader/k8s/reloader-values.yaml|reloader/k8s/active"
+)
+
+# "name url" for the Helm repository hosting a chart, empty if unknown.
+helm_repo_for() {
+  case "$1" in
+    prometheus-community/*) echo "prometheus-community https://prometheus-community.github.io/helm-charts" ;;
+    grafana/*) echo "grafana https://grafana.github.io/helm-charts" ;;
+    stakater/*) echo "stakater https://stakater.github.io/stakater-charts" ;;
+  esac
+}
+
+upgrade_helm_releases() {
+  local entry release chart namespace version values marker repo
+  for entry in ${HELM_RELEASES[@]+"${HELM_RELEASES[@]}"}; do
+    IFS='|' read -r release chart namespace version values marker <<<"$entry"
+    if [ ! -f "$REPO/$marker" ]; then
+      echo "skip (no $marker): $release"
+      continue
+    fi
+    if [ ! -f "$REPO/$values" ]; then
+      echo "ERROR: $values is gitignored but missing on the workstation, restore it first."
+      return 1
+    fi
+    repo="$(helm_repo_for "$chart")"
+    if [ -z "$repo" ]; then
+      echo "ERROR: no Helm repository configured for chart $chart"
+      return 1
+    fi
+    helm repo add "${repo%% *}" "${repo#* }" >/dev/null 2>&1 || true
+    helm repo update "${repo%% *}" >/dev/null 2>&1 || true
+    log "Upgrading $release ($chart $version)"
+    # --atomic rolls the release back when the upgrade times out or the workloads
+    # it touches never become ready, so a bad chart bump is not left half applied.
+    helm upgrade --install "$release" "$chart" \
+      --namespace "$namespace" \
+      --version "$version" \
+      --values "$REPO/$values" \
+      --atomic --cleanup-on-fail --timeout 10m
+  done
+}
+
+stage_preflight() {
+  if [ ! -d "$REPO/.git" ]; then
+    echo "Repository not found at $REPO"
+    exit 1
+  fi
+  if [ -n "$DEPLOY_SHA" ]; then
+    log "Checking out the commit CI validated ($DEPLOY_SHA)"
+    git -C "$REPO" fetch origin --quiet "$DEPLOY_SHA" 2>/dev/null \
+      || git -C "$REPO" fetch origin main
+  else
+    git -C "$REPO" fetch origin main
+  fi
+  target="${DEPLOY_SHA:-origin/main}"
+  log "Workstation state"
+  echo "  local:  $(git -C "$REPO" rev-parse --short HEAD)"
+  echo "  target: $(git -C "$REPO" rev-parse --short "$target")"
+  if [ -n "$(git -C "$REPO" status --porcelain --untracked-files=no)" ]; then
+    echo "ERROR: workstation has local tracked modifications, refusing reset:"
+    git -C "$REPO" status --porcelain --untracked-files=no
+    git -C "$REPO" diff --stat
+    echo "Fix it on the workstation (commit, or 'git restore .'), then re-run the deploy."
+    exit 1
+  fi
+  git -C "$REPO" reset --hard "$target"
+}
+
+stage_validate() {
+  cd "$REPO"
+  select_manifests
+  local m k cf
+  # Compose .env files and secret files are gitignored by design, so the
+  # workstation never has real values for the inactive stacks. This stage only
+  # runs the full check on active stacks; the general structure check for every
+  # committed Compose file (active or not) lives in the ci workflow, which has no
+  # .env at all.
+  #
+  # Active stacks are still validated with interpolation and env-file resolution
+  # off, so required-variable guards (:?) and missing local files do not fail the
+  # deploy. Normalization and consistency checks stay enabled.
+  # shellcheck source=compose-lint.sh
+  source "$REPO/.gitea/workflows/compose-lint.sh"
+  local compose_validate_flags=()
+  mapfile -t compose_validate_flags < <(compose_safe_flags)
+  log "Validate compose stacks"
+  for cf in ${COMPOSE_STACKS[@]+"${COMPOSE_STACKS[@]}"}; do
+    echo "  config: $cf"
+    validate_compose_file "$cf" ${compose_validate_flags[@]+"${compose_validate_flags[@]}"}
+  done
+  log "Validate k8s manifests (kubectl dry-run=client)"
+  for m in ${K8S_MANIFESTS[@]+"${K8S_MANIFESTS[@]}"}; do
+    kubectl apply --dry-run=client -f "$m" >/dev/null
+  done
+  for k in ${KUSTOMIZE_APPS[@]+"${KUSTOMIZE_APPS[@]}"}; do
+    kubectl apply -k "$k" --dry-run=client >/dev/null
+  done
+  log "Validate k8s manifests (kubectl dry-run=server)"
+  for m in ${K8S_MANIFESTS[@]+"${K8S_MANIFESTS[@]}"}; do
+    kubectl apply --dry-run=server -f "$m" >/dev/null
+  done
+  for k in ${KUSTOMIZE_APPS[@]+"${KUSTOMIZE_APPS[@]}"}; do
+    kubectl apply -k "$k" --dry-run=server >/dev/null
+  done
+  log "Checking referenced Secrets exist"
+  echo "  (deploy never applies *secret*.yaml; create missing ones manually)"
+  local ref_secrets=() missing_secrets=() all_secrets s
+  if [ "${#K8S_MANIFESTS[@]}" -gt 0 ]; then
+    while IFS= read -r s; do
+      [ -n "$s" ] && ref_secrets+=("$s")
+    done < <(
+      {
+        grep -h -A1 -E 'secretRef:|secretKeyRef:' "${K8S_MANIFESTS[@]}" 2>/dev/null || true
+        grep -h -E 'secretName:' "${K8S_MANIFESTS[@]}" 2>/dev/null || true
+      } | grep -E 'name:' | sed -E 's/.*name:[[:space:]]*//' | tr -d '"'"'"' "'"'" | sed -E 's/[[:space:]]*#.*//' | awk 'NF' | sort -u || true
+    )
+  fi
+  all_secrets="$(kubectl get secrets -A --no-headers -o custom-columns=:metadata.name 2>/dev/null || true)"
+  for s in ${ref_secrets[@]+"${ref_secrets[@]}"}; do
+    if printf '%s\n' "$all_secrets" | grep -qx "$s"; then
+      echo "  ok: $s"
+    else
+      echo "  MISSING: $s"
+      missing_secrets+=("$s")
+    fi
+  done
+  if [ "${#missing_secrets[@]}" -gt 0 ]; then
+    echo "ERROR: ${#missing_secrets[@]} referenced Secret(s) not found in the cluster:"
+    printf '  - %s\n' "${missing_secrets[@]}"
+    echo "Create them manually from the laptop, e.g.:"
+    echo "  kubectl apply -f SERVICE/k8s/secrets.yaml  # see SERVICE/k8s/secrets.yaml.example"
+    exit 1
+  fi
+}
+
+stage_apply_k8s() {
+  cd "$REPO"
+  select_manifests >/dev/null
+  local ns_files=() other_files=() m k prune_opts=()
+  for m in ${K8S_MANIFESTS[@]+"${K8S_MANIFESTS[@]}"}; do
+    case "$m" in
+      */namespace.y?ml) ns_files+=("$m") ;;
+      *) other_files+=("$m") ;;
+    esac
+  done
+  if [ "$APPLY_PRUNE" = "true" ]; then
+    prune_opts=(--prune -l app.kubernetes.io/managed-by=homelab-deploy)
+  fi
+
+  # Record what is about to change, and publish it for the verify job, before
+  # the first apply. Both are fatal on failure: see snapshot_dir.
+  local snapshot
+  snapshot="$(snapshot_dir)" || return 1
+  save_snapshot "$snapshot" || return 1
+
+  if [ "${#ns_files[@]}" -gt 0 ]; then
+    log "Applying namespaces (${#ns_files[@]} files)"
+    for m in "${ns_files[@]}"; do
+      kubectl apply -f "$m"
+    done
+  fi
+  if [ -f "$REPO/prometheus-stack/k8s/active" ]; then
+    if [ ! -f "$REPO/prometheus-stack/k8s/grafana-values.yaml" ]; then
+      echo "ERROR: prometheus-stack/k8s/grafana-values.yaml (gitignored) missing on workstation, restore it first."
+      exit 1
+    fi
+  fi
+  upgrade_helm_releases
+  if [ "${#other_files[@]}" -gt 0 ]; then
+    log "Applying resources (${#other_files[@]} files, our images pinned to digests)"
+    for m in "${other_files[@]}"; do
+      if ! render_pinned <"$m" | kubectl apply "${prune_opts[@]}" -f -; then
+        echo "ERROR: apply failed for ${m#"$REPO"/}" >&2
+        exit 1
+      fi
+    done
+  fi
+  for k in ${KUSTOMIZE_APPS[@]+"${KUSTOMIZE_APPS[@]}"}; do
+    log "Applying kustomize app: ${k#"$REPO"/} (our images pinned to digests)"
+    if ! kubectl kustomize "$k" | render_pinned | kubectl apply -f -; then
+      echo "ERROR: apply failed for kustomize app ${k#"$REPO"/}" >&2
+      exit 1
+    fi
+  done
+  if [ -f "$REPO/userbot/k8s/active" ]; then
+    log "userbot panel hook"
+    if kubectl get secret userbot-common-secrets -n userbot >/dev/null 2>&1; then
+      echo "  userbot-common-secrets already present in userbot ns, not touching"
+    elif kubectl get secret userbot-common-secrets -n default >/dev/null 2>&1; then
+      echo "  bootstrapping userbot-common-secrets into userbot ns"
+      kubectl get secret userbot-common-secrets -n default -o json \
+        | jq 'del(.metadata.annotations,.metadata.creationTimestamp,.metadata.resourceVersion,.metadata.uid,.metadata.managedFields) | .metadata.namespace = "userbot"' \
+        | kubectl apply -f -
+    else
+      echo "  WARNING: userbot-common-secrets missing in both default and userbot ns; create it manually from the laptop"
+    fi
+  fi
+  restart_stale_images
+
+  # No verification here on purpose. This stage may be killed at any point by
+  # timeout-minutes, by the runner cancelling the job, or by a dropped SSH
+  # connection, and any code below that line would simply not run. stage_verify_k8s
+  # picks the work up from the snapshot instead.
+  log "Applied. Verification and rollback are the verify job's job, not this one's."
+}
+
+# Runs as its own workflow job, after apply-k8s (and apply-compose) are done —
+# including when they failed, timed out or were cancelled. Reads the baseline the
+# apply stage published and works out what it changed, watches those workloads,
+# and rolls back the ones that never became healthy.
+stage_verify_k8s() {
+  local pointer="$DEPLOY_SNAPSHOT_DIR/current"
+  local snapshot want have generations
+  local -a touched=()
+
+  if [ ! -s "$pointer" ]; then
+    echo "ERROR: no snapshot pointer at $pointer."
+    echo "The apply stage died before publishing any state, so there is no baseline to"
+    echo "tell which workloads it touched. Nothing can be rolled back automatically —"
+    echo "inspect the cluster by hand."
+    return 1
+  fi
+  snapshot="$(head -1 "$pointer")"
+  if [ ! -d "$snapshot" ]; then
+    echo "ERROR: snapshot pointer refers to a missing directory: $snapshot"
+    return 1
+  fi
+
+  # Never trust the pointer blindly. If the apply stage was killed before it
+  # published its own snapshot, `current` still points at the previous deploy's
+  # baseline. Verifying against that would watch the wrong workloads and the
+  # rollback would revert the wrong revisions, so refuse instead.
+  want="${DEPLOY_SHA:-}"
+  if [ -z "$want" ]; then
+    want="$(git -C "$REPO" rev-parse HEAD 2>/dev/null || true)"
+  fi
+  have="$(cat "$snapshot/commit" 2>/dev/null || true)"
+  if [ -z "$want" ] || [ "$have" != "$want" ]; then
+    echo "ERROR: refusing to verify or roll back against a stale snapshot."
+    echo "  snapshot:        $snapshot"
+    echo "  snapshot commit: ${have:-<missing>}"
+    echo "  deploy commit:   ${want:-<unknown>}"
+    return 1
+  fi
+  echo "  snapshot: $snapshot (commit ${have:0:12})"
+
+  generations="$snapshot/generations.before"
+  if [ ! -s "$generations" ]; then
+    # Without a baseline we cannot tell which workloads the apply touched, so
+    # fall back to watching everything rather than silently skipping the check.
+    warn "no pre-apply baseline, verifying every workload in the cluster"
+    : >"$generations"
+  fi
+
+  while read -r kind ns name; do
+    [ -n "${kind:-}" ] && touched+=("$kind $ns $name")
+  done < <(changed_workloads "$generations")
+
+  log "Verifying ${#touched[@]} changed workload(s) (timeout ${ROLLOUT_TIMEOUT}s each)"
+  if [ "${#touched[@]}" -eq 0 ]; then
+    echo "  nothing to verify"
+    return 0
+  fi
+  printf '  watching: %s\n' "${touched[@]/#/    }"
+
+  local failed_file="$snapshot/failed-workloads"
+  if ! verify_workloads "$failed_file" ${touched[@]+"${touched[@]}"}; then
+    echo
+    echo "ERROR: ${#touched[@]} workload(s) changed by this deploy, and these never became healthy:"
+    grep -v -E '^(ROLLED_BACK|UNRECOVERED)=' "$failed_file" | sed 's/^/  - /'
+    echo
+    log "Rolling back to the previous revision"
+    if rollback_workloads "$failed_file"; then
+      echo
+      echo "Rolled back successfully. The cluster is back on the pre-deploy revision."
+      echo "Nothing else was reverted: Git holds desired state only, so config changes, PVCs and"
+      echo "externally created resources from this commit are still in place. Review the failed"
+      echo "workload, then re-run the deploy (Actions -> deploy -> Run workflow)."
+    else
+      echo
+      echo "Rollback did NOT fully recover the cluster. Manual intervention required:"
+      grep -E '^(ROLLED_BACK|UNRECOVERED)=' "$failed_file" | sed 's/^/  /'
+      echo "Pre-apply snapshot: $snapshot"
+    fi
+    return 1
+  fi
+}
+
+# verify_compose_stack <compose-file>
+# `docker compose up -d` exits 0 as soon as containers are created, so a stack can
+# come back broken with a green pipeline. Require every long-running service to
+# actually be running.
+verify_compose_stack() {
+  local cf="$1"
+  local expected running missing=()
+  expected="$(docker compose -f "$cf" config --services 2>/dev/null | sort || true)"
+  running="$(docker compose -f "$cf" ps --status running --services 2>/dev/null | sort || true)"
+  [ -n "$expected" ] || return 0
+  while IFS= read -r svc; do
+    [ -n "$svc" ] || continue
+    # restart:"no" services are allowed to have exited.
+    if ! printf '%s\n' "$running" | grep -qx "$svc" \
+      && ! docker compose -f "$cf" config 2>/dev/null \
+      | grep -A5 "^  ${svc}:" | grep -qE 'restart:\s*"?no"?'; then
+      missing+=("$svc")
+    fi
+  done <<<"$expected"
+  if [ "${#missing[@]}" -gt 0 ]; then
+    echo "  NOT RUNNING: ${missing[*]}"
+    docker compose -f "$cf" ps --all 2>/dev/null | sed 's/^/    /' || true
+    return 1
+  fi
+  echo "  all ${#expected} service(s) running"
+  return 0
+}
+
+# The public hostname of every active service, one per line.
+#
+# Comments are stripped first, and deliberately so: a route that someone
+# disabled by commenting it out is not a service to probe, and naio and xui are
+# both still in the tree that way. A `#` only starts a comment when it is at the
+# start of a line or after whitespace, so `s/#.*//` alone would also cut a
+# legitimate value in half.
+#
+# Only the public names. The *.internal names are the same Traefik and the same
+# Services, reached by a different label, so probing both would double the run
+# to learn the same thing. The public name is also the one a user types.
+smoke_hosts() {
+  local m k
+  # The backticks below are literal. They are Traefik's Host() delimiter, and the
+  # single quotes are precisely what keeps the shell from reading them as a
+  # command substitution, so the warning is the opposite of a real problem.
+  # shellcheck disable=SC2016
+  {
+    for m in ${K8S_MANIFESTS[@]+"${K8S_MANIFESTS[@]}"}; do
+      [ -f "$m" ] && cat "$m"
+    done
+    for k in ${KUSTOMIZE_APPS[@]+"${KUSTOMIZE_APPS[@]}"}; do
+      kubectl kustomize "$k" 2>/dev/null || true
+    done
+  } | sed -E 's/(^|[[:space:]])#.*$//' \
+    | grep -oE 'Host\(`[^`]+`\)' \
+    | sed -E 's/^Host\(`//; s/`\)$//' \
+    | grep -E '(^|\.)forust\.xyz$' \
+    | grep -v '\${' \
+    | sort -u
+}
+
+# stage_verify_k8s watches the rollout, which reports that the pods converged.
+# It cannot tell a converged pod from a serving one: a route pointing at the
+# wrong port, a Service selector that matches nothing the app listens on, a 500
+# from the app itself, an OOMKill loop that still counts as Available for long
+# enough to pass. All of those are green at the rollout level.
+#
+# So ask the thing users ask. Any HTTP response proves Traefik matched the
+# host, the Service resolved to a pod and the pod answered -- a 302 to a login
+# or a 404 from a path the service does not serve still means the chain is
+# intact. Only a transport failure (no DNS, refused, timeout) or a 5xx means
+# the service is not serving, and only those fail the run.
+stage_smoke() {
+  cd "$REPO"
+  select_manifests >/dev/null
+  local -a hosts=()
+  # Not named failed: an array of that name already exists in restart_stale_images
+  # above, and a scalar shadowing an array is a trap rather than a shadow.
+  local h code rc bad=0
+  while IFS= read -r h; do
+    [ -n "$h" ] && hosts+=("$h")
+  done < <(smoke_hosts)
+
+  if [ "${#hosts[@]}" -eq 0 ]; then
+    # Nothing to probe means the extraction broke, not that the cluster is empty.
+    echo "ERROR: no public hostnames found in active manifests, refusing to report success"
+    return 1
+  fi
+
+  log "Probing ${#hosts[@]} public route(s)"
+  for h in "${hosts[@]}"; do
+    code="$(curl -sS -o /dev/null --max-time 20 -w '%{http_code}' "https://$h/" 2>/dev/null)" && rc=0 || rc=$?
+    if [ "$rc" -ne 0 ]; then
+      echo "  UNREACHABLE $h (curl exit $rc)"
+      bad=1
+      continue
+    fi
+    # A glob, not a string compare. `case` on the leading digit is the only one
+    # of these that survives a three-digit code, and the obvious expansion to
+    # try first -- ${code%%[0-9]*} -- is empty for every input, so it silently
+    # reports a 500 as healthy.
+    case "$code" in
+      5*)
+        echo "  SERVER ERROR $h $code"
+        bad=1
+        ;;
+      000)
+        # curl exited 0 and still no status, so nothing on the far end replied.
+        # Not a pass, whatever the transport thought.
+        echo "  NO RESPONSE $h"
+        bad=1
+        ;;
+      *)
+        echo "  ok $h $code"
+        ;;
+    esac
+  done
+
+  if [ "$bad" -ne 0 ]; then
+    echo "ERROR: at least one active service is not serving over its public route"
+    return 1
+  fi
+  echo "all ${#hosts[@]} route(s) answered"
+}
+
+stage_apply_compose() {
+  cd "$REPO"
+  select_manifests >/dev/null
+  local cf
+  log "Redeploying docker compose stacks (${#COMPOSE_STACKS[@]} stacks)"
+  for cf in ${COMPOSE_STACKS[@]+"${COMPOSE_STACKS[@]}"}; do
+    echo "  compose: $cf"
+    if grep -Eq '^\s+pull_policy:\s*build\b' "$cf"; then
+      docker compose -f "$cf" build
+      docker compose -f "$cf" push
+    fi
+    docker compose -f "$cf" up -d --pull always --remove-orphans
+  done
+
+  local -a broken=()
+  for cf in ${COMPOSE_STACKS[@]+"${COMPOSE_STACKS[@]}"}; do
+    echo "  verifying: $cf"
+    if ! verify_compose_stack "$cf"; then
+      broken+=("$cf")
+    fi
+  done
+  if [ "${#broken[@]}" -gt 0 ]; then
+    echo
+    echo "ERROR: ${#broken[@]} compose stack(s) did not come up:"
+    printf '  - %s\n' "${broken[@]}"
+    echo "Compose stacks are not rolled back automatically: their images use mutable"
+    echo "':latest' tags, so there is no previous version to return to. Check the logs"
+    echo "above, then re-run the deploy once the cause is fixed."
+    return 1
+  fi
+}
+
+run_stage() {
+  case "${1:?stage required}" in
+    preflight) stage_preflight ;;
+    validate) stage_validate ;;
+    apply-k8s) stage_apply_k8s ;;
+    verify-k8s) stage_verify_k8s ;;
+    smoke) stage_smoke ;;
+    apply-compose) stage_apply_compose ;;
+    *)
+      echo "ERROR: unknown stage: $1"
+      exit 1
+      ;;
+  esac
+}

.gitea/workflows/deploy.yaml

@@ -0,0 +1,151 @@
+name: deploy
+
+on:
+  # Deploy only what CI already validated. workflow_run is used instead of
+  # workflow_dispatch so a red lint/validate run can never reach the cluster.
+  workflow_run:
+    workflows: [ci]
+    types: [completed]
+  workflow_dispatch:
+
+# The deploy jobs read the tree, then reach the cluster over SSH with the
+# deploy key. The Actions token itself is not part of that path, so it gets
+# read-only contents and no more.
+permissions:
+  contents: read
+
+concurrency:
+  group: deploy-main
+  # Queue instead of cancelling. Cancelling a run kills the apply job mid-loop and
+  # takes the verify job down with it, so a superseded deploy would leave the
+  # cluster half-applied and unchecked — the exact failure the verify job exists
+  # to catch. kubectl apply and docker compose up are both idempotent, so letting
+  # the older run finish and then deploying the newer commit costs little.
+  cancel-in-progress: false
+
+env:
+  DEPLOY_HOST: ${{ secrets.DEPLOY_HOST }}
+  DEPLOY_PORT: ${{ secrets.DEPLOY_PORT }}
+  DEPLOY_USER: ${{ secrets.DEPLOY_USER }}
+  DEPLOY_PATH: ${{ secrets.DEPLOY_PATH }}
+  DEPLOY_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
+  APPLY_PRUNE: ${{ vars.APPLY_PRUNE }}
+  # workflow_run's own GITHUB_SHA points at the branch head, not at the commit the
+  # finished ci run checked. Pin the exact validated commit instead, so a push
+  # landing mid-deploy cannot make the workstation deploy something else. Also
+  # what the verify job checks the snapshot against. Empty for workflow_dispatch,
+  # which falls back to the current origin/main.
+  DEPLOY_SHA: ${{ github.event.workflow_run.head_sha }}
+
+jobs:
+  preflight:
+    if: >-
+      github.event_name != 'workflow_run' ||
+      (github.event.workflow_run.conclusion == 'success' &&
+       github.event.workflow_run.head_branch == 'main')
+    runs-on: [self-hosted, linux, arch, homelab, prod]
+    timeout-minutes: 10
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+
+      - name: Fetch and reset workstation
+        shell: bash
+        run: |
+          set -euo pipefail
+          ./.gitea/workflows/ssh-run.sh preflight
+
+  validate:
+    needs: [preflight]
+    runs-on: [self-hosted, linux, arch, homelab, prod]
+    timeout-minutes: 20
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+
+      - name: Dry-run manifests and check Secrets
+        shell: bash
+        run: |
+          set -euo pipefail
+          ./.gitea/workflows/ssh-run.sh validate
+
+  apply-k8s:
+    needs: [validate]
+    runs-on: [self-hosted, linux, arch, homelab, prod]
+    # Apply only, no verification, so this is just the work itself: snapshot,
+    # then up to three sequential `helm upgrade --atomic --timeout 10m`, then the
+    # apply loop. Verification has its own job and its own budget.
+    timeout-minutes: 45
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+
+      - name: Apply Kubernetes manifests
+        shell: bash
+        run: |
+          set -euo pipefail
+          ./.gitea/workflows/ssh-run.sh apply-k8s
+
+  apply-compose:
+    needs: [validate]
+    runs-on: [self-hosted, linux, arch, homelab, prod]
+    timeout-minutes: 30
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+
+      - name: Redeploy docker compose stacks
+        shell: bash
+        run: |
+          set -euo pipefail
+          ./.gitea/workflows/ssh-run.sh apply-compose
+
+  # Watches the workloads this deploy changed and rolls back the ones that never
+  # became healthy. Runs even when the apply jobs failed, timed out or were
+  # cancelled — that is the whole point of splitting it out. `always()` is what
+  # lets it start after a failed dependency; the needs on apply-compose are a
+  # barrier, so verification begins only once both applies are done.
+  verify-k8s:
+    needs: [apply-k8s, apply-compose]
+    if: >-
+      always() &&
+      needs.apply-k8s.result != 'skipped' &&
+      needs.apply-compose.result != 'skipped'
+    runs-on: [self-hosted, linux, arch, homelab, prod]
+    # ceil(changed_workloads / 8) waves of ROLLOUT_TIMEOUT each, plus rollback.
+    timeout-minutes: 30
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+
+      - name: Verify workloads and roll back on failure
+        shell: bash
+        run: |
+          set -euo pipefail
+          ./.gitea/workflows/ssh-run.sh verify-k8s
+
+  # Asks the public route of every active service whether it is actually
+  # serving, which the rollout check above structurally cannot: a pod can
+  # converge and still be crash-looping, or be listening on a port no Service
+  # points at, or answer 500.
+  #
+  # `always()` for the same reason verify-k8s has it, and it runs after that job
+  # specifically because a rollback is when a route most needs re-checking. The
+  # needs is a barrier, not a filter: whether verify-k8s passed, failed or was
+  # cancelled, the probes are what say whether the cluster is serving, and
+  # suppressing them on a rollback would hide the one run where the answer
+  # matters most.
+  smoke:
+    needs: [verify-k8s]
+    if: always() && needs.verify-k8s.result != 'skipped'
+    runs-on: [self-hosted, linux, arch, homelab, prod]
+    timeout-minutes: 10
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+
+      - name: Probe the public route of every active service
+        shell: bash
+        run: |
+          set -euo pipefail
+          ./.gitea/workflows/ssh-run.sh smoke

.gitea/workflows/install-ci-tools.sh

@@ -0,0 +1,233 @@
+#!/usr/bin/env bash
+# Installs the pinned CI tools into "$TOOLS_DIR/bin" and echoes that directory
+# on stdout, so callers can do:
+#
+#   export PATH="$(bash .gitea/workflows/install-ci-tools.sh kubeconform shellcheck):$PATH"
+#
+# Versions come from tool-versions.env next to this script and are kept fresh by
+# Renovate. Re-running is cheap: an already-installed tool at the pinned version
+# is left alone.
+set -euo pipefail
+
+here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=tool-versions.env
+. "$here/tool-versions.env"
+
+TOOLS_DIR="${TOOLS_DIR:-${RUNNER_TEMP:-/tmp}/homelab-tools}"
+BIN_DIR="$TOOLS_DIR/bin"
+mkdir -p "$BIN_DIR"
+
+arch="$(uname -m)"
+# Upstream projects disagree on arch spelling: kubeconform and actionlint use
+# Go names (amd64/arm64), shellcheck uses uname names (x86_64/aarch64), node
+# uses neither (x64/arm64), and hadolint mixes the two in a single release
+# (x86_64 but arm64).
+case "$arch" in
+  x86_64 | amd64)
+    goarch=amd64
+    sharch=x86_64
+    nodearch=x64
+    hadolintarch=x86_64
+    ;;
+  aarch64 | arm64)
+    goarch=arm64
+    sharch=aarch64
+    nodearch=arm64
+    hadolintarch=arm64
+    ;;
+  *)
+    echo "install-ci-tools: unsupported architecture: $arch" >&2
+    exit 1
+    ;;
+esac
+
+fetch() {
+  # fetch <url> <dest>
+  if command -v curl >/dev/null 2>&1; then
+    curl -sSLf --retry 3 -o "$2" "$1"
+  elif command -v wget >/dev/null 2>&1; then
+    wget -q -O "$2" "$1"
+  else
+    echo "install-ci-tools: neither curl nor wget is available" >&2
+    exit 1
+  fi
+}
+
+# installed_version <command>
+# Prints the version of an already-installed tool, or nothing. Each tool spells
+# its version flag differently, hence the case.
+installed_version() {
+  local out
+  case "$1" in
+    kubeconform) out="$("$1" -v 2>/dev/null | head -1 || true)" ;;
+    *) out="$("$1" --version 2>/dev/null | head -1 || true)" ;;
+  esac
+  printf '%s' "$out"
+}
+
+# at_version <command> <expected>
+at_version() {
+  case "$(installed_version "$1")" in
+    *"$2"*) return 0 ;;
+    *) return 1 ;;
+  esac
+}
+
+install_kubeconform() {
+  if at_version kubeconform "v${KUBECONFORM_VERSION}"; then
+    return 0
+  fi
+  local tmp
+  tmp="$(mktemp -d)"
+  fetch "https://github.com/yannh/kubeconform/releases/download/v${KUBECONFORM_VERSION}/kubeconform-linux-${goarch}.tar.gz" \
+    "$tmp/kubeconform.tar.gz"
+  tar -xzf "$tmp/kubeconform.tar.gz" -C "$tmp" kubeconform
+  install -m 0755 "$tmp/kubeconform" "$BIN_DIR/kubeconform"
+  rm -rf "$tmp"
+}
+
+install_shellcheck() {
+  if at_version shellcheck "${SHELLCHECK_VERSION}"; then
+    return 0
+  fi
+  local tmp
+  tmp="$(mktemp -d)"
+  fetch "https://github.com/koalaman/shellcheck/releases/download/v${SHELLCHECK_VERSION}/shellcheck-v${SHELLCHECK_VERSION}.linux.${sharch}.tar.xz" \
+    "$tmp/shellcheck.tar.xz"
+  tar -xJf "$tmp/shellcheck.tar.xz" -C "$tmp" --strip-components=1 "shellcheck-v${SHELLCHECK_VERSION}/shellcheck"
+  install -m 0755 "$tmp/shellcheck" "$BIN_DIR/shellcheck"
+  rm -rf "$tmp"
+}
+
+install_uv() {
+  if at_version uv "${UV_VERSION}"; then
+    return 0
+  fi
+  local tmp
+  tmp="$(mktemp -d)"
+  # uv release tags carry no leading v, unlike every other tool installed here.
+  fetch "https://github.com/astral-sh/uv/releases/download/${UV_VERSION}/uv-${sharch}-unknown-linux-gnu.tar.gz" \
+    "$tmp/uv.tar.gz"
+  tar -xzf "$tmp/uv.tar.gz" -C "$tmp" --strip-components=1 "uv-${sharch}-unknown-linux-gnu/uv"
+  install -m 0755 "$tmp/uv" "$BIN_DIR/uv"
+  rm -rf "$tmp"
+}
+
+install_hadolint() {
+  if at_version hadolint "${HADOLINT_VERSION}"; then
+    return 0
+  fi
+  # A bare binary, no archive: hadolint ships one file per platform.
+  fetch "https://github.com/hadolint/hadolint/releases/download/v${HADOLINT_VERSION}/hadolint-linux-${hadolintarch}" \
+    "$BIN_DIR/hadolint"
+  chmod 0755 "$BIN_DIR/hadolint"
+}
+
+# ruff and yamllint both come from PyPI as wheels, which uv unpacks for us.
+install_uv_tool() {
+  # <package> <pinned version>
+  if at_version "$1" "$2"; then
+    return 0
+  fi
+  install_uv
+  UV_TOOL_BIN_DIR="$BIN_DIR" uv tool install --force "$1==$2" >/dev/null
+}
+
+install_ruff() {
+  install_uv_tool ruff "${RUFF_VERSION}"
+}
+
+install_yamllint() {
+  install_uv_tool yamllint "${YAMLLINT_VERSION}"
+}
+
+install_pip_audit() {
+  install_uv_tool pip-audit "${PIP_AUDIT_VERSION}"
+}
+
+install_prettier() {
+  if at_version prettier "${PRETTIER_VERSION}"; then
+    return 0
+  fi
+  # Not a standalone binary: prettier's entry point requires ../package.json
+  # relative to its own real path, so the package directory has to survive
+  # next to it. Hence a versioned directory plus a relative symlink, rather
+  # than copying the one file out as the other installers do.
+  local dir="$BIN_DIR/prettier-${PRETTIER_VERSION}"
+  if [ ! -f "$dir/package/package.json" ]; then
+    rm -rf "$dir"
+    mkdir -p "$dir"
+    fetch "https://registry.npmjs.org/prettier/-/prettier-${PRETTIER_VERSION}.tgz" "$dir/prettier.tgz"
+    tar -xzf "$dir/prettier.tgz" -C "$dir"
+    rm -f "$dir/prettier.tgz"
+    # npm strips the exec bit from bin/ on the way into the tarball.
+    chmod 0755 "$dir/package/bin/prettier.cjs"
+  fi
+  # Relative, so the whole tree stays valid if TOOLS_DIR is relocated.
+  ln -sfn "prettier-${PRETTIER_VERSION}/package/bin/prettier.cjs" "$BIN_DIR/prettier"
+}
+
+install_node() {
+  # npm gets checked by running it, not by looking it up: what matters is that
+  # it answers, so a stub, a half-removed Arch package or a name that resolves
+  # to something broken all have to read as "not installed". The runner's npm
+  # is a symlink into /usr/lib/node_modules/npm, which is exactly the kind of
+  # thing that disappears between runs.
+  if at_version node "v${NODE_VERSION}" && [ -n "$(installed_version npm)" ]; then
+    return 0
+  fi
+  # Same shape as prettier above: the tarball's bin/npm and bin/npx are links
+  # into lib/node_modules, so the whole tree has to survive next to them.
+  local dir="$BIN_DIR/node-${NODE_VERSION}"
+  if [ ! -x "$dir/bin/node" ]; then
+    rm -rf "$dir"
+    mkdir -p "$dir"
+    fetch "https://nodejs.org/dist/v${NODE_VERSION}/node-v${NODE_VERSION}-linux-${nodearch}.tar.xz" \
+      "$dir/node.tar.xz"
+    tar -xJf "$dir/node.tar.xz" -C "$dir" --strip-components=1 "node-v${NODE_VERSION}-linux-${nodearch}"
+    rm -f "$dir/node.tar.xz"
+  fi
+  # Relative, so the whole tree stays valid if TOOLS_DIR is relocated.
+  for bin in node npm npx; do
+    ln -sfn "node-${NODE_VERSION}/bin/${bin}" "$BIN_DIR/${bin}"
+  done
+}
+
+install_actionlint() {
+  if at_version actionlint "${ACTIONLINT_VERSION}"; then
+    return 0
+  fi
+  local tmp
+  tmp="$(mktemp -d)"
+  fetch "https://github.com/rhysd/actionlint/releases/download/v${ACTIONLINT_VERSION}/actionlint_${ACTIONLINT_VERSION}_linux_${goarch}.tar.gz" \
+    "$tmp/actionlint.tar.gz"
+  tar -xzf "$tmp/actionlint.tar.gz" -C "$tmp" actionlint
+  install -m 0755 "$tmp/actionlint" "$BIN_DIR/actionlint"
+  rm -rf "$tmp"
+}
+
+wanted=("$@")
+if [ "${#wanted[@]}" -eq 0 ]; then
+  wanted=(kubeconform shellcheck actionlint prettier ruff yamllint hadolint)
+fi
+
+for tool in "${wanted[@]}"; do
+  case "$tool" in
+    kubeconform) install_kubeconform ;;
+    shellcheck) install_shellcheck ;;
+    actionlint) install_actionlint ;;
+    prettier) install_prettier ;;
+    ruff) install_ruff ;;
+    yamllint) install_yamllint ;;
+    pip-audit) install_pip_audit ;;
+    hadolint) install_hadolint ;;
+    node) install_node ;;
+    uv) install_uv ;;
+    *)
+      echo "install-ci-tools: unknown tool: $tool" >&2
+      exit 1
+      ;;
+  esac
+done
+
+printf '%s\n' "$BIN_DIR"

.gitea/workflows/renovate-ci.yaml

@@ -0,0 +1,76 @@
+name: renovate-ci
+
+on:
+  pull_request:
+  push:
+    branches:
+      - main
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  validate-renovate:
+    runs-on: [self-hosted, linux, arch, homelab]
+    timeout-minutes: 20
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+
+      # renovate/k8s/cronjob.yaml is the single source of truth for the image tag,
+      # so the same version that runs in the cluster is the one validated here.
+      - name: Resolve the deployed Renovate image
+        id: image
+        shell: bash
+        run: |
+          set -euo pipefail
+          image="$(sed -n 's|.*image:[[:space:]]*\(renovate/renovate:[^[:space:]]*\).*|\1|p' \
+            renovate/k8s/cronjob.yaml | head -1)"
+          if [ -z "$image" ]; then
+            echo "::error::no renovate/renovate image found in renovate/k8s/cronjob.yaml"
+            exit 1
+          fi
+          echo "using $image"
+          echo "image=$image" >> "$GITHUB_OUTPUT"
+
+      - name: Validate Renovate repository config
+        shell: bash
+        run: |
+          set -euo pipefail
+          docker run --rm \
+            -v "$PWD/renovate:/opt/renovate:ro" \
+            -e RENOVATE_CONFIG_FILE=/opt/renovate/renovate.json \
+            "${{ steps.image.outputs.image }}" \
+            renovate-config-validator /opt/renovate/renovate.json
+
+      # The CronJob cannot read the repository, so renovate/k8s/configmap.yaml
+      # carries an inlined copy of the config. Fail if it no longer matches.
+      - name: Check the generated Renovate ConfigMap
+        shell: bash
+        run: |
+          set -euo pipefail
+          ./.gitea/workflows/sync-renovate-configmap.sh --check
+
+      - name: Validate Renovate Kubernetes manifests
+        shell: bash
+        run: |
+          set -euo pipefail
+          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh kubeconform)"
+          export PATH="$tools_dir:$PATH"
+          kubeconform \
+            -strict \
+            -ignore-missing-schemas \
+            -summary \
+            renovate/k8s/namespace.yaml \
+            renovate/k8s/configmap.yaml \
+            renovate/k8s/cronjob.yaml
+
+      - name: Validate Renovate Compose file
+        shell: bash
+        run: |
+          set -euo pipefail
+          source .gitea/workflows/compose-lint.sh
+          mapfile -t safe_flags < <(compose_safe_flags)
+          validate_compose_file renovate/renovate-compose.yaml \
+            ${safe_flags[@]+"${safe_flags[@]}"}

.gitea/workflows/renovate-run.yaml

@@ -0,0 +1,92 @@
+name: renovate-run
+
+on:
+  workflow_dispatch:
+    inputs:
+      repositories:
+        description: "Repositories to scan (comma-separated)"
+        required: false
+        default: "forust/homelab"
+      log_level:
+        description: "Renovate log level"
+        required: false
+        default: "info"
+        type: choice
+        options:
+          - info
+          - debug
+      dry_run:
+        description: "Plan only, do not open or update PRs"
+        required: false
+        default: false
+        type: boolean
+
+# Renovate writes through its own bot PAT, passed in as RENOVATE_TOKEN, so the
+# Actions token is only ever used to read the checkout.
+permissions:
+  contents: read
+
+concurrency:
+  group: renovate-run
+  cancel-in-progress: false
+
+jobs:
+  run-renovate:
+    runs-on: [self-hosted, linux, arch, homelab]
+    timeout-minutes: 60
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+
+      # renovate/k8s/cronjob.yaml is the single source of truth for the image tag.
+      # Reading it here means this workflow validates and runs the exact version
+      # that is deployed, instead of a copy that silently goes stale.
+      - name: Resolve the deployed Renovate image
+        id: image
+        shell: bash
+        run: |
+          set -euo pipefail
+          image="$(sed -n 's|.*image:[[:space:]]*\(renovate/renovate:[^[:space:]]*\).*|\1|p' \
+            renovate/k8s/cronjob.yaml | head -1)"
+          if [ -z "$image" ]; then
+            echo "::error::no renovate/renovate image found in renovate/k8s/cronjob.yaml"
+            exit 1
+          fi
+          echo "using $image"
+          echo "image=$image" >> "$GITHUB_OUTPUT"
+
+      - name: Validate Renovate config
+        shell: bash
+        run: |
+          set -euo pipefail
+          docker run --rm \
+            -v "$PWD/renovate/renovate.json:/opt/renovate/renovate.json:ro" \
+            -e RENOVATE_CONFIG_FILE=/opt/renovate/renovate.json \
+            "${{ steps.image.outputs.image }}" \
+            renovate-config-validator
+
+      - name: Run Renovate
+        shell: bash
+        env:
+          RENOVATE_TOKEN: ${{ secrets.RENOVATE_TOKEN }}
+          RENOVATE_GITHUB_COM_TOKEN: ${{ secrets.RENOVATE_GITHUB_COM_TOKEN }}
+          RENOVATE_REPOSITORIES: ${{ inputs.repositories }}
+          RENOVATE_DRY_RUN: ${{ inputs.dry_run && 'full' || '' }}
+          LOG_LEVEL: ${{ inputs.log_level }}
+        run: |
+          set -euo pipefail
+
+          : "${RENOVATE_TOKEN:?missing RENOVATE_TOKEN secret — add a renovate-bot PAT in repo/org Actions secrets}"
+
+          docker run --rm \
+            -v "$PWD/renovate/renovate.json:/opt/renovate/renovate.json:ro" \
+            -e RENOVATE_PLATFORM=gitea \
+            -e RENOVATE_ENDPOINT=https://gitea.forust.xyz/api/v1 \
+            -e RENOVATE_TOKEN="$RENOVATE_TOKEN" \
+            -e RENOVATE_GITHUB_COM_TOKEN="${RENOVATE_GITHUB_COM_TOKEN:-}" \
+            -e RENOVATE_REPOSITORIES="${RENOVATE_REPOSITORIES:-forust/homelab}" \
+            -e RENOVATE_DRY_RUN="${RENOVATE_DRY_RUN:-}" \
+            -e RENOVATE_CONFIG_FILE=/opt/renovate/renovate.json \
+            -e RENOVATE_BASE_DIR=/tmp/renovate \
+            -e LOG_LEVEL="${LOG_LEVEL:-info}" \
+            "${{ steps.image.outputs.image }}"

.gitea/workflows/ssh-run.sh

@@ -0,0 +1,30 @@
+#!/usr/bin/env bash
+# usage: ssh-run.sh <stage>
+# Runs one deploy-lib.sh stage on the workstation over SSH.
+set -euo pipefail
+
+: "${DEPLOY_HOST:?missing DEPLOY_HOST}"
+: "${DEPLOY_USER:?missing DEPLOY_USER}"
+: "${DEPLOY_KEY:?missing DEPLOY_SSH_KEY}"
+
+deploy_port="${DEPLOY_PORT:-22}"
+deploy_path="${DEPLOY_PATH:-/srv/homelab}"
+deploy_path="$(printf '%s' "$deploy_path" | tr -d '\"' | tr -d '\r' | xargs)"
+
+# The private key is written to a per-run directory that is removed on exit, so a
+# failed or cancelled job cannot leave deploy credentials in the runner's temp
+# directory. Do not use a fixed path: apply-k8s and apply-compose run in parallel.
+key_dir="$(mktemp -d "${RUNNER_TEMP:-/tmp}/homelab-deploy-key.XXXXXXXX")"
+trap 'rm -rf "$key_dir"' EXIT INT TERM
+
+ssh_key="$key_dir/deploy_key"
+printf '%s\n' "$DEPLOY_KEY" > "$ssh_key"
+chmod 600 "$ssh_key"
+
+ssh -i "$ssh_key" -p "$deploy_port" \
+  -o BatchMode=yes -o StrictHostKeyChecking=accept-new \
+  "${DEPLOY_USER}@${DEPLOY_HOST}" \
+  "REPO=$deploy_path APPLY_PRUNE=${APPLY_PRUNE:-false} DEPLOY_SHA=${DEPLOY_SHA:-} DEPLOY_SNAPSHOT_DIR=${DEPLOY_SNAPSHOT_DIR:-} STAGE=$1 bash -se" <<'EOF'
+source "$REPO/.gitea/workflows/deploy-lib.sh"
+run_stage "$STAGE"
+EOF

.gitea/workflows/sync-renovate-configmap.sh

@@ -0,0 +1,55 @@
+#!/usr/bin/env bash
+# Regenerates renovate/k8s/configmap.yaml from renovate/renovate.json.
+#
+# renovate/renovate.json is the single source of truth: the CronJob, the Compose
+# file and the renovate-run workflow all mount that exact file. A ConfigMap cannot
+# read a file from the repository, so the same bytes are inlined here as a literal
+# block. This script keeps the copy honest:
+#
+#   .gitea/workflows/sync-renovate-configmap.sh          # rewrite in place
+#   .gitea/workflows/sync-renovate-configmap.sh --check  # fail if out of date
+#
+# renovate-ci runs the --check form on every PR and push, so a config change that
+# forgets to regenerate the ConfigMap cannot be merged.
+set -euo pipefail
+
+here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+repo="$(git -C "$here" rev-parse --show-toplevel)"
+
+src="$repo/renovate/renovate.json"
+dst="$repo/renovate/k8s/configmap.yaml"
+[ -f "$src" ] || {
+  echo "missing $src" >&2
+  exit 1
+}
+
+render() {
+  cat <<'HEADER'
+# GENERATED FILE - do not edit by hand.
+# Source: renovate/renovate.json
+# Regenerate: .gitea/workflows/sync-renovate-configmap.sh
+# Verify:     .gitea/workflows/sync-renovate-configmap.sh --check
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: renovate-config
+  namespace: renovate
+data:
+  renovate.json: |
+HEADER
+  sed 's/^/    /' "$src"
+}
+
+if [ "${1:-}" = "--check" ]; then
+  if ! diff -u "$dst" <(render) >/dev/null 2>&1; then
+    echo "ERROR: $dst is out of sync with renovate/renovate.json"
+    echo "Run: .gitea/workflows/sync-renovate-configmap.sh"
+    diff -u "$dst" <(render) || true
+    exit 1
+  fi
+  echo "renovate/k8s/configmap.yaml is in sync with renovate/renovate.json"
+  exit 0
+fi
+
+render >"$dst"
+echo "wrote $dst"

.gitea/workflows/tool-versions.env

@@ -0,0 +1,33 @@
+# Pinned versions of the CI tools installed by install-ci-tools.sh.
+# Renovate keeps these up to date (see customManagers in renovate/renovate.json).
+#
+# Every version here except NODE_VERSION matches what was already installed on
+# the runner, so pinning them changes what CI does not at all. It changes what
+# CI does when the runner is rebuilt with something else: today
+# install-ci-tools.sh finds the pinned version already on PATH and installs
+# nothing, and a runner that drifts gets the pinned one installed over it.
+#
+# The renovate image version is NOT pinned here: renovate/k8s/cronjob.yaml is the
+# single source of truth and the workflows read the tag from it, so there is
+# nothing to drift.
+ACTIONLINT_VERSION="1.7.7"
+SHELLCHECK_VERSION="0.11.0"
+KUBECONFORM_VERSION="0.8.0"
+PRETTIER_VERSION="3.8.1"
+RUFF_VERSION="0.16.8"
+YAMLLINT_VERSION="1.38.0"
+HADOLINT_VERSION="2.14.0"
+# pip-audit reads the advisory database over the network, so a floating version
+# would make the same commit report different things on different days. Pin it
+# like the rest: the advisories themselves are the moving part, not the tool.
+PIP_AUDIT_VERSION="2.10.1"
+# uv builds the throwaway venv the pytest job runs in, and unpacks the PyPI
+# wheels for ruff, yamllint and pip-audit.
+UV_VERSION="0.12.17"
+# node runs `npm ci` for the frontend tests and the npm audit, and it is the one
+# pin here that does NOT come from the runner: the runner's system node is a
+# rolling Arch package (it was node 26 with no npm at all when this was pinned),
+# and the panel image is node:22-alpine. Pinned to the image's major on purpose,
+# so the tree that gets tested is the tree that gets built. Renovate keeps this
+# in step with the Dockerfile's node: tag via the "node runtime" group.
+NODE_VERSION="22.23.3"

.gitignore

@@ -2,19 +2,15 @@
 sync.ffs_lock
 .sync.ffs_db
 
-# Environment 
+# Copyparty
-.env
+*.hist/
-.env.anna
-.env.forust
-.env.*
-!.env.*example
 
-# Volumes and data directories
+# Volumes, configs and data directories
 gitea/gitea-db/
 gitea/gitea-data/*
 n8n/n8n-data/*
 n8n/n8n-node-data/*
-adguardhome/data/*
+adguardhome/conf/*
 dockmon/data/*
 portainer/portainer_data/*
 metube/MeTube_downloads
@@ -22,6 +18,13 @@ uptime-kuma/data/
 termix/termix-data/*
 cfddns/config.json
 checkmk/checkmk/*
+downtify/Downtify_downloads
+headscale/config/*
+headscale/data/*
+# NetBird local hostnames and generated secrets
+netbird/.env
+netbird/secrets/
+searxng/core-config/*
 
 # Steaming services files
 streaming/jellyfin/*
@@ -33,13 +36,22 @@ streaming/qbittorrent/*
 streaming/prowlarr/*
 
 # Homepage
-homepages/forust_files/assets/images/team/*
+homepages/forust_files/.well-known/*
-
 
 # Traefik files
 traefik/letsencrypt/acme.json
+traefik/dynamic/fileservers.yml
+traefik/dynamic/*.local.y*ml.*
+traefik/dynamic/*.external.y*ml
+traefik/k8s/fileservers.y*ml
+traefik/k8s/aliasHeadersStrategy.md
+
 traefik/logs/*
+
+# SSL Certificates
+adguardhome/certs/*
 traefik/certs/*
+certs/
 
 # Monitoring
 monitoring/prometheus.yml
@@ -74,6 +86,8 @@ replacements.txt
 
 # Git
 .gitattributes
+# Gitea/github Runners
+.runner
 
 # Misc
 .DS_Store
@@ -81,3 +95,24 @@ replacements.txt
 
 # Temp files
 edu_master/temp/
+temp/*
+# Local-only tooling scratch space (pinned CI tools, verification scripts)
+tmp/
+
+# Environment
+.env
+.env.anna
+.env.forust
+.env.*
+!*example
+
+# kubernetes
+*/k8s/*secret*
+!*/k8s/*secret*.example
+**/k8s/*secret*
+!**/k8s/*secret*.example
+# Local-only tweaks, not for upstream
+prometheus-stack/k8s/grafana-values.yaml
+traefik/k8s/local-tls.yaml
+converters/k8s/config.yaml
+convertx/k8s/config.yaml

.hadolint.yaml

@@ -0,0 +1,10 @@
+ignored:
+  - DL3008
+  - DL3042
+  - DL3018
+  - DL3059
+trustedRegistries:
+  - docker.io
+  - ghcr.io
+  - quay.io
+  - gcr.forust.xyz

.markdownlint.json

@@ -0,0 +1,8 @@
+{
+  "default": true,
+  "MD013": false,
+  "MD024": false,
+  "MD033": false,
+  "MD041": false,
+  "MD046": false
+}

.prettierrc.yaml

@@ -0,0 +1,7 @@
+bracketSameLine: true
+htmlWhitespaceSensitivity: css
+printWidth: 120
+tabWidth: 2
+trailingComma: all
+proseWrap: preserve
+endOfLine: lf

.yamllint

@@ -0,0 +1,22 @@
+extends: default
+
+rules:
+  comments:
+    min-spaces-from-content: 1
+  comments-indentation: false
+  document-start: disable
+  line-length: disable
+  braces:
+    min-spaces-inside: 0
+    max-spaces-inside: 1
+  brackets:
+    min-spaces-inside: 0
+    max-spaces-inside: 1
+  indentation:
+    spaces: 2
+    indent-sequences: consistent
+  truthy:
+    allowed-values:
+      - "true"
+      - "false"
+      - "on"

.zed/settings.json

@@ -0,0 +1,40 @@
+{
+  "tab_size": 2,
+  "soft_wrap": "prefer_line",
+  "preferred_line_length": 120,
+  "format_on_save": "on",
+  "languages": {
+    "YAML": {
+      "tab_size": 2,
+      "hard_tabs": false,
+      "format_on_save": "on",
+      "formatter": {
+        "language_server": { "name": "yaml-language-server" },
+      },
+    },
+    "Python": {
+      "tab_size": 4,
+      "format_on_save": "on",
+      "language_servers": ["pyright", "ruff"],
+      "formatter": {
+        "language_server": { "name": "ruff" },
+      },
+    },
+  },
+  "lsp": {
+    "yaml-language-server": {
+      "settings": {
+        "yaml": {
+          "schemas": {
+            "kubernetes": ["**/k8s/*.yaml", "**/k8s/*.yml"],
+          },
+          "validate": true,
+          "completion": true,
+          "format": {
+            "enable": true,
+          },
+        },
+      },
+    },
+  },
+}

adguardhome/compose.yaml

@@ -1,50 +1,48 @@
 services:
   adguard:
-    image: adguard/adguardhome:latest
+    image: adguard/adguardhome:v0.107.79
     container_name: adguardhome
     restart: unless-stopped
-    environment:
-      - TZ=${TZ}
     ports:
       - "53:53/tcp"
       - "53:53/udp"
+      - "853:853/tcp" # DNS over TLS
       # - "67:67/udp" # DHCP
       # - "68:68/tcp" # DHCP
-      - "3000:3000/tcp"
+      # - "3000:3000/tcp"
     volumes:
-      - ./data/work:/opt/adguardhome/work
+      - data:/opt/adguardhome/work
-      - ./data/conf:/opt/adguardhome/conf
+      - ./conf:/opt/adguardhome/conf
-    networks:
+      - ./certs:/certs:ro
-      - traefik-proxy
     labels:
       - "traefik.enable=true"
-      - "traefik.docker.network=traefik-proxy"
-
-      # Prod Router
-      - "traefik.http.routers.adguard.rule=Host(`adguard.forust.xyz`)"
-      - "traefik.http.routers.adguard.entrypoints=websecure"
-      - "traefik.http.routers.adguard.middlewares=security-headers@file"
-      - "traefik.http.routers.adguard.service=adguard"
-      - "traefik.http.routers.adguard.tls=true"
       - "traefik.http.services.adguard.loadbalancer.server.port=3000"
 
+      # Prod Router
+      - "traefik.http.routers.adguard.rule=Host(`dns.forust.xyz`) || Host(`adguard.forust.xyz`)"
+      - "traefik.http.routers.adguard.entrypoints=websecure"
+      - "traefik.http.routers.adguard.tls.certresolver=letsencrypt"
       # Local Router
-      - "traefik.http.routers.adguard-local.rule=Host(`adguard.workstation.internal`) || Host(`adguard.internal`)"
+      - "traefik.http.routers.adguard-local.rule=Host(`adguard.workstation.internal`) || Host(`dns.workstation.internal`)"
       - "traefik.http.routers.adguard-local.entrypoints=websecure"
-      - "traefik.http.routers.adguard-local.middlewares=security-headers@file"
-      - "traefik.http.routers.adguard-local.service=adguard"
       - "traefik.http.routers.adguard-local.tls=true"
-
       # Dev Router
-      - "traefik.http.routers.adguard-dev.rule=Host(`adguard.gigaforust.internal`)"
+      - "traefik.http.routers.adguard-dev.rule=Host(`adguard.gigaforust.internal`) || Host(`dns.gigaforust.internal`)"
       - "traefik.http.routers.adguard-dev.entrypoints=websecure"
-      - "traefik.http.routers.adguard-dev.middlewares=security-headers@file"
-      - "traefik.http.routers.adguard-dev.service=adguard"
       - "traefik.http.routers.adguard-dev.tls=true"
+      # DoH Router
+      - "traefik.http.routers.dns-over-https.rule=(Host(`dns.forust.xyz` || Host(`adguard.forust.xyz`)) && PathPrefix(`/dns-query`))"
+      - "traefik.http.routers.dns-over-https.entrypoints=websecure"
+      - "traefik.http.routers.dns-over-https.tls.certresolver=letsencrypt"
 
+      # Glance Metadata
       - glance.name=adguard
       - glance.url=https://adguard.forust.xyz/
       - glance.description=AdGuard Home is a network-wide software for blocking ads.
+    networks:
+      - proxy
+volumes:
+  data:
 networks:
-  traefik-proxy:
+  proxy:
     external: true

adguardhome/k8s/adguard.yaml

@@ -0,0 +1,118 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: adguard-lb-service
+  namespace: adguard
+  annotations:
+    metallb.io/loadBalancerIPs: "192.168.80.3"
+spec:
+  type: LoadBalancer
+  externalTrafficPolicy: Local
+  selector:
+    app: adguard
+  ports:
+    - name: dns-udp
+      port: 53
+      targetPort: 53
+      protocol: UDP
+    - name: dns-tcp
+      port: 53
+      targetPort: 53
+      protocol: TCP
+    - name: dot
+      port: 853
+      targetPort: 853
+      protocol: TCP
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: adguard-service
+  namespace: adguard
+spec:
+  selector:
+    app: adguard
+  ports:
+    - port: 3000
+      name: webui
+      targetPort: 3000
+    - port: 53
+      name: dns
+      targetPort: 53
+      protocol: UDP
+    - port: 53
+      name: dns-tcp
+      targetPort: 53
+      protocol: TCP
+    - port: 853
+      name: dot
+      targetPort: 853
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: adguard-deployment
+  namespace: adguard
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: adguard
+  template:
+    metadata:
+      labels:
+        app: adguard
+      annotations:
+        reloader.stakater.com/auto: "true"
+    spec:
+      containers:
+        - name: adguard
+          image: adguard/adguardhome:v0.107.79
+          resources:
+            limits:
+              memory: "1.5Gi"
+              cpu: "300m"
+            requests:
+              memory: "500Mi"
+              cpu: "50m"
+          ports:
+            - containerPort: 3000
+              name: webui
+            - containerPort: 53
+              name: dns
+            - containerPort: 853
+              name: dot
+          volumeMounts:
+            - name: adguard-data
+              mountPath: /opt/adguardhome/work
+              subPath: work
+            - name: adguard-data
+              mountPath: /opt/adguardhome/conf
+              subPath: conf
+            - name: adguard-certs
+              mountPath: /certs
+              readOnly: true
+      volumes:
+        - name: adguard-data
+          persistentVolumeClaim:
+            claimName: adguard-pvc
+        - name: adguard-certs
+          secret:
+            secretName: adguard-certs
+            items:
+              - key: tls.crt
+                path: fullchain.pem
+              - key: tls.key
+                path: privkey.pem
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: adguard-pvc
+  namespace: adguard
+spec:
+  accessModes:
+    - ReadWriteOnce
+  resources:
+    requests:
+      storage: 2Gi

adguardhome/k8s/certificate.yaml

@@ -0,0 +1,12 @@
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: adguard-certs
+  namespace: adguard
+spec:
+  secretName: adguard-certs
+  dnsNames:
+    - dns.forust.xyz
+  issuerRef:
+    name: letsencrypt-prod
+    kind: ClusterIssuer

adguardhome/k8s/ingress.yaml

@@ -0,0 +1,46 @@
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
+metadata:
+  name: adguard-prod
+  namespace: adguard
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`dns.forust.xyz`)
+      kind: Rule
+      middlewares:
+        - name: crowdsec-bouncer
+          namespace: crowdsec
+      services:
+        - name: adguard-service
+          port: 3000
+    - match: (Host(`dns.forust.xyz`)) && PathPrefix(`/dns-query`)
+      kind: Rule
+      services:
+        - name: adguard-service
+          port: 3000
+  tls:
+    secretName: adguard-certs
+---
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
+metadata:
+  name: adguard-local
+  namespace: adguard
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`adguard.workstation.internal`) || Host(`dns.workstation.internal`) || Host(`adguard.gigaforust.internal`) || Host(`dns.gigaforust.internal`)
+      kind: Rule
+      services:
+        - name: adguard-service
+          port: 3000
+    - match: (Host(`adguard.workstation.internal`) || Host(`dns.workstation.internal`) || Host(`adguard.gigaforust.internal`) || Host(`dns.gigaforust.internal`)) && PathPrefix(`/dns-query`)
+      kind: Rule
+      services:
+        - name: adguard-service
+          port: 3000
+  tls:
+    secretName: internal-wildcard-tls

adguardhome/k8s/internal-certificate.yaml

@@ -0,0 +1,15 @@
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: internal-wildcard-tls
+  namespace: adguard
+spec:
+  secretName: internal-wildcard-tls
+  dnsNames:
+    - "*.workstation.internal"
+    - "*.gigaforust.internal"
+    - workstation.internal
+    - gigaforust.internal
+  issuerRef:
+    name: internal-ca
+    kind: ClusterIssuer

adguardhome/k8s/namespace.yaml

@@ -0,0 +1,4 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: adguard

adguardhome/k8s/secrets.yaml.example

@@ -0,0 +1,10 @@
+kubectl apply -f k8s/namespace.yaml && \
+kubectl create secret tls adguard-certs -n adguard \
+  --cert=certs/fullchain.pem \
+  --key=certs/privkey.pem --dry-run=client -o yaml > \
+  k8s/secrets.yaml
+
+#  OR WITH NO FILE CREATION:
+kubectl create secret tls adguard-certs -n adguard \
+  --cert=certs/fullchain.pem --key=certs/privkey.pem \
+  --save-config

authentik/compose.yaml

@@ -1,6 +1,6 @@
 services:
   postgresql:
-    image: docker.io/library/postgres:15-alpine
+    image: docker.io/library/postgres:15.19-alpine
     restart: unless-stopped
     env_file:
       - .env
@@ -26,9 +26,9 @@ services:
     command: server
     container_name: authentik-server
     restart: unless-stopped
-    ports:
+    # ports:
-      - ${PORT_HTTP:-9000}:9000
+    #   - ${PORT_HTTP:-9000}:9000
-      - ${PORT_HTTPS:-9443}:9443
+    #   - ${PORT_HTTPS:-9443}:9443
     env_file:
       - .env
     environment:
@@ -37,39 +37,28 @@ services:
       AUTHENTIK_POSTGRESQL__PASSWORD: ${PG_PASS}
       AUTHENTIK_POSTGRESQL__USER: ${PG_USER:-authentik}
       AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY:?secret key required}
-
+    volumes:
+      - ./media:/media
+      - ./custom-templates:/templates
     labels:
       - "traefik.enable=true"
-      - "traefik.docker.network=traefik-proxy"
-      # Services
-      # - "traefik.http.services.authentik-server.loadbalancer.server.port=9443"
       - "traefik.http.services.authentik-server.loadbalancer.server.port=9000"
 
       # Prod Router
       - "traefik.http.routers.authentik-server.rule=Host(`auth.forust.xyz`)"
       - "traefik.http.routers.authentik-server.entrypoints=websecure"
-      - "traefik.http.routers.authentik-server.middlewares=security-headers@file"
+      - "traefik.http.routers.authentik-server.tls.certresolver=letsencrypt"
-      - "traefik.http.routers.authentik-server.service=authentik-server"
-      - "traefik.http.routers.authentik-server.tls=true"
-
       # Local Router
-      - "traefik.http.routers.authentik-server-local.rule=Host(`auth.workstation.internal`) || Host(`auth-dashboard.internal`)"
+      - "traefik.http.routers.authentik-server-local.rule=Host(`auth.workstation.internal`)"
       - "traefik.http.routers.authentik-server-local.entrypoints=websecure"
-      - "traefik.http.routers.authentik-server-local.middlewares=security-headers@file"
-      - "traefik.http.routers.authentik-server-local.service=authentik-server"
       - "traefik.http.routers.authentik-server-local.tls=true"
-
       # Dev Router
       - "traefik.http.routers.authentik-server-dev.rule=Host(`auth.gigaforust.internal`)"
       - "traefik.http.routers.authentik-server-dev.entrypoints=websecure"
       - "traefik.http.routers.authentik-server-dev.middlewares=security-headers@file"
-      - "traefik.http.routers.authentik-server-dev.service=authentik-server"
       - "traefik.http.routers.authentik-server-dev.tls=true"
-    volumes:
-      - ./media:/media
-      - ./custom-templates:/templates
     networks:
-      - traefik-proxy
+      - proxy
       - authentik
     depends_on:
       postgresql:
@@ -102,5 +91,5 @@ volumes:
     driver: local
 networks:
   authentik:
-  traefik-proxy:
+  proxy:
     external: true

authentik/k8s/authentik.yaml

@@ -0,0 +1,93 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: authentik-server-service
+  namespace: authentik
+spec:
+  type: ClusterIP
+  selector:
+    app: authentik-server
+  ports:
+    - port: 9000
+      targetPort: 9000
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: authentik-worker-service
+  namespace: authentik
+spec:
+  type: ClusterIP
+  selector:
+    app: authentik-worker
+  ports:
+    - port: 9000
+      targetPort: 9000
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: authentik-server-deployment
+  namespace: authentik
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: authentik-server
+  template:
+    metadata:
+      labels:
+        app: authentik-server
+    spec:
+      containers:
+        - name: authentik-server
+          image: ghcr.io/goauthentik/server:2026.8.3
+          args: ["server"]
+          envFrom:
+            - configMapRef:
+                name: authentik-config
+            - secretRef:
+                name: authentik-secrets
+          ports:
+            - containerPort: 9000
+          resources:
+            requests:
+              memory: "700Mi"
+              cpu: "300m"
+            limits:
+              memory: "1.5Gi"
+              cpu: "1000m"
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: authentik-worker-deployment
+  namespace: authentik
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: authentik-worker
+  template:
+    metadata:
+      labels:
+        app: authentik-worker
+    spec:
+      containers:
+        - name: authentik-worker
+          image: ghcr.io/goauthentik/server:2026.8.3
+          args: ["worker"]
+          securityContext:
+            runAsUser: 0
+          envFrom:
+            - configMapRef:
+                name: authentik-config
+            - secretRef:
+                name: authentik-secrets
+          resources:
+            requests:
+              memory: "512Mi"
+              cpu: "300m"
+            limits:
+              memory: "1Gi"
+              cpu: "700m"

authentik/k8s/certificates.yaml

@@ -0,0 +1,28 @@
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: authentik-prod-tls
+  namespace: authentik
+spec:
+  secretName: authentik-prod-tls
+  dnsNames:
+    - auth.forust.xyz
+  issuerRef:
+    name: letsencrypt-prod
+    kind: ClusterIssuer
+---
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: internal-wildcard-tls
+  namespace: authentik
+spec:
+  secretName: internal-wildcard-tls
+  dnsNames:
+    - "*.workstation.internal"
+    - "*.gigaforust.internal"
+    - workstation.internal
+    - gigaforust.internal
+  issuerRef:
+    name: internal-ca
+    kind: ClusterIssuer

authentik/k8s/config.yaml

@@ -0,0 +1,11 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: authentik-config
+  namespace: authentik
+data:
+  AUTHENTIK_IMAGE: ghcr.io/goauthentik/server
+  AUTHENTIK_TAG: "2025.10.2"
+  AUTHENTIK_POSTGRESQL__HOST: postgres.database.svc.cluster.local
+  AUTHENTIK_POSTGRESQL__NAME: authentik
+  AUTHENTIK_ERROR_REPORTING__ENABLED: "true"

authentik/k8s/ingress.yaml

@@ -0,0 +1,36 @@
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
+metadata:
+  name: authentik-prod
+  namespace: authentik
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`auth.forust.xyz`)
+      kind: Rule
+      middlewares:
+        - name: crowdsec-bouncer
+          namespace: crowdsec
+      services:
+        - name: authentik-server-service
+          port: 9000
+  tls:
+    secretName: authentik-prod-tls
+---
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
+metadata:
+  name: authentik-local
+  namespace: authentik
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`auth.workstation.internal`) || Host(`auth.gigaforust.internal`)
+      kind: Rule
+      services:
+        - name: authentik-server-service
+          port: 9000
+  tls:
+    secretName: internal-wildcard-tls

authentik/k8s/namespace.yaml

@@ -0,0 +1,4 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: authentik

authentik/k8s/secrets.yaml.example

@@ -0,0 +1,11 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: authentik-secrets
+  namespace: authentik
+type: Opaque
+stringData:
+  AUTHENTIK_SECRET_KEY: ""
+  AUTHENTIK_POSTGRESQL__PASSWORD: ""
+  AUTHENTIK_POSTGRESQL__USER: authentik
+  AUTHENTIK_BOOTSTRAP_PASSWORD: authentik

cert-manager/k8s/cert-manager-values.yaml

@@ -0,0 +1,9 @@
+crds:
+  enabled: true
+prometheus:
+  servicemonitor:
+    enabled: true
+    interval: 60s
+    scrapeTimeout: 30s
+    labels:
+      release: prometheus-stack

cert-manager/k8s/clusterissuer.yaml

@@ -0,0 +1,29 @@
+apiVersion: cert-manager.io/v1
+kind: ClusterIssuer
+metadata:
+  name: letsencrypt-staging
+spec:
+  acme:
+    email: bobrovod@national.shitposting.agency
+    server: https://acme-staging-v02.api.letsencrypt.org/directory
+    privateKeySecretRef:
+      name: letsencrypt-staging-account-key
+    solvers:
+      - http01:
+          ingress:
+            class: traefik
+---
+apiVersion: cert-manager.io/v1
+kind: ClusterIssuer
+metadata:
+  name: letsencrypt-prod
+spec:
+  acme:
+    email: bobrovod@national.shitposting.agency
+    server: https://acme-v02.api.letsencrypt.org/directory
+    privateKeySecretRef:
+      name: letsencrypt-prod-account-key
+    solvers:
+      - http01:
+          ingress:
+            class: traefik

cert-manager/k8s/homelab-internal-root-ca.crt

@@ -0,0 +1,30 @@
+-----BEGIN CERTIFICATE-----
+MIIFFjCCAv6gAwIBAgIUetKpTfEDOn2985FFMu6G26itT+wwDQYJKoZIhvcNAQEN
+BQAwIzEhMB8GA1UEAxMYaG9tZWxhYiBpbnRlcm5hbCByb290IENBMB4XDTI2MDky
+MzEyNDA0N1oXDTM2MDkyMDEyNDA0N1owIzEhMB8GA1UEAxMYaG9tZWxhYiBpbnRl
+cm5hbCByb290IENBMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAvmNP
+ZCOoD8NtNuYJKVXBlTPjX7D7sJCSK5neH7ZbYV5+lmUlEErY8Mik7j37V5k5NfpF
+Ig85pOjP7RckTPz5V6ek3yaN40s4AL053sN5ZPauDVYjalaEHTgj5sEMqlLACQWI
+yZmJOZspZykae8dIpQnqCoFpRT4FurJ78v4a0ylnFVLMQn/lyCHedwTjkEdtYWYr
+ccJy8vQwqkzs/rWvEH1lDqZhennLOrmcCjfonG7D/pruMn4z+6E28p4+ejkRrI6x
+luak3KnpT1XMeHtgU21hiRGaMDBchHMFgAhnY1qosymKenXvfTZItwgjZbwa1hJI
+GAiDm+jQDKMjzRZ3rH6Xfc0auUcykNz73PpNu1NGm78nndXwCXcXn1LFKNQJ+r1U
+sJiyAmUZmXVn4aM4OMf2F38k7wTYIKg7nRGaUkNeKDlNkjA4HvgWw+jwO1KmdHQ/
+mOem1rosDWHRK01wg+Gga9mQCnhNhxglg3t/UeSic6uOaRsvaz4qkzHq8MbCujVz
+DpKQjqdikYOAXZOs4KlBLWrS7NaK4NzfSD02pBUErh54ruJfY/bWz9KyXzBD/lQZ
+VUTKyvUVB0bkVHEdf1jJmX3H4IZRQSF5JPqOBotW6bJI5fEGNBvj9Zxy4nm2WWGz
+yyP3uWsQz8U/Wdx9nXZLHInTZBsvgLYtKUAWA30CAwEAAaNCMEAwDgYDVR0PAQH/
+BAQDAgKkMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFEkKm2rxPaK6+O9WD80z
+BLC6F9QsMA0GCSqGSIb3DQEBDQUAA4ICAQAnFyHz97Umf5VIu+dKTJid7C73VugJ
+TIar/xJBs/4CxP+znBxhJjXygRoyIfzoVGWcB2ZSL//vL78Qlts79K/Imc9a4RFF
+wMvCxsRXAEQ4TpeWi3ophPNcs4rhsP+gQKQFtnyKP9519bqpfxp0bTqwOV2o18fn
+za7rlQViiEnNV58j7CVoM9+mJvVVfBEX1Km+GyJL9GadzbIQ7FxClVJZefCbft93
+zHVk9gDOw8ys1XGSR2OUCyCLinXO6mqS16CmBb2MAKXq/YyH7E0N8iotAPGtfA8V
+M/0ddy947rY0xCrtECfWwvGQpJS7NRv/Z9b2jCfXrI5LXmL2nfQRg0y9GE4Vjwr+
+WxtGU5jOeFt0jQ+xRzcgG0Op+qK3x55l5LSo2hOcOVYbiHxcHEJFgwNi1ADeBFwb
+q/HdysfURSOghqjIpMMAUabBp+DBUg2EUF7pIaUqbdqExFYcr9EYisEMiNsmKmN+
+8ZbcOeerFKDQj+t/R0bFXa7UBn2UWsjI8zlR74aa2kLDXwtyz/XlO/FlYm66eBFo
+2/eYUSeU+S4ej+wUAs/dvjF7f190/DUQGuwOTlLTahqWDztmhCk7qzbECu56CwKT
+E5Ect2P72UleYwdblkVOVd352AmiwEzdOaziIRrPh8uenEknH6JBYPO3mjk7cCg+
+GPGcNIBctjdXhg==
+-----END CERTIFICATE-----

cert-manager/k8s/internal-ca.yaml

@@ -0,0 +1,33 @@
+apiVersion: cert-manager.io/v1
+kind: ClusterIssuer
+metadata:
+  name: selfsigned
+spec:
+  selfSigned: {}
+---
+# Homelab internal root CA (10y). Install the .crt on clients (see below).
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: internal-ca-root
+  namespace: cert-manager
+spec:
+  isCA: true
+  commonName: homelab internal root CA
+  duration: 87600h
+  renewBefore: 7200h
+  secretName: internal-ca-root
+  privateKey:
+    algorithm: RSA
+    size: 4096
+  issuerRef:
+    name: selfsigned
+    kind: ClusterIssuer
+---
+apiVersion: cert-manager.io/v1
+kind: ClusterIssuer
+metadata:
+  name: internal-ca
+spec:
+  ca:
+    secretName: internal-ca-root

cert-manager/k8s/namespace.yaml

@@ -0,0 +1,4 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: cert-manager

cfddns/.env.example

@@ -0,0 +1,15 @@
+CLOUDFLARE_API_TOKEN=YOUR_CLOUDFLARE_API_TOKEN
+DOMAINS=example.com,dns.example.com,mc.example.com,auth.example.com,ssh.example.com
+IP4_DOMAINS=
+IP6_DOMAINS=
+IP4_PROVIDER=cloudflare.trace
+IP6_PROVIDER=none # change if you want to update AAAA
+UPDATE_CRON=@every 5m
+UPDATE_ON_START=true
+DELETE_ON_STOP=false
+DELETE_ON_FAILURE=true
+TTL=1
+PROXIED=!is(dns.example.com) && !is(mc.example.com) && !is(ssh.example.com)
+EMOJI=true
+UPTIMEKUMA=https://uptime-kuma.example.com/api/push/AsaSDFGFkfklaFALSKffkfFKfkfkfkFK?status=up&msg=OK&ping=
+REJECT_CLOUDFLARE_IPS=true

cfddns/compose.yaml

@@ -1,13 +1,30 @@
 services:
   cloudflare-ddns:
-    image: timothyjmiller/cloudflare-ddns:latest
+    image: timothyjmiller/cloudflare-ddns:2.2.0
     container_name: cloudflare-ddns
+    restart: unless-stopped
     security_opt:
       - no-new-privileges:true
-    network_mode: 'host'
+    network_mode: "host"
+    # https://github.com/timothymiller/cloudflare-ddns#-quick-start
     environment:
-      - PUID=1000
+      - CLOUDFLARE_API_TOKEN=${CLOUDFLARE_API_TOKEN:?Cloudflare API token is required}
-      - PGID=1000
+      - DOMAINS=${DOMAINS:-}
-    volumes:
+      - IP4_DOMAINS=${IP4_DOMAINS:-}
-      - ./config.json:/config.json
+      - IP6_DOMAINS=${IP6_DOMAINS:-}
-    restart: unless-stopped
+      - IP4_PROVIDER=${IP4_PROVIDER:-cloudflare.trace}
+      - IP6_PROVIDER=${IP6_PROVIDER:-none}
+      - UPDATE_CRON=${UPDATE_CRON:-@every 5m}
+      - UPDATE_ON_START=${UPDATE_ON_START:-true}
+      - DELETE_ON_STOP=${DELETE_ON_STOP:-false}
+      - DELETE_ON_FAILURE=${DELETE_ON_FAILURE:-true}
+      - TTL=${TTL:-1} # 1=auto
+      # to proxy only "dns.example.com" and "wfs.example.com" use "!is(dns.domain.com) && !is (wfs.domain.com)"
+      - PROXIED=${PROXIED:-true}
+      - EMOJI=${EMOJI:-true}
+      - UPTIMEKUMA=${UPTIMEKUMA:-}
+      - HEALTHCHECKS=${HEALTHCHECKS:-}
+      - REJECT_CLOUDFLARE_IPS=${REJECT_CLOUDFLARE_IPS:-true}
+      # volumes:
+      # Prefer using environment variables for configuration, config.json legacy support
+      # - ./config.json:/config.json

cfddns/config.json.example

@@ -7,7 +7,7 @@
       "api_key": {
         "api_key": "api_key_here",
         "account_email": "your_email_here"
-      }
+      },
       "zone_id": "your_zone-id",
       "subdomains": [
         { "name": "", "proxied": true },

cfddns/k8s/.gitignore

@@ -0,0 +1 @@
+secret.yaml

cfddns/k8s/deployment.yaml

@@ -0,0 +1,32 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: cfddns
+  labels:
+    app: cfddns
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: cfddns
+  template:
+    metadata:
+      labels:
+        app: cfddns
+    spec:
+      hostNetwork: true
+      dnsPolicy: ClusterFirstWithHostNet
+      containers:
+        - name: cloudflare-ddns
+          image: timothyjmiller/cloudflare-ddns:2.2.0
+          imagePullPolicy: Always
+          resources:
+            requests:
+              memory: "20Mi"
+              cpu: "30m"
+            limits:
+              memory: "64Mi"
+              cpu: "50m"
+          envFrom:
+            - secretRef:
+                name: cfddns-secrets

cfddns/k8s/secret.yaml.example

@@ -0,0 +1,18 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: cfddns-secrets
+type: Opaque
+stringData:
+  CLOUDFLARE_API_TOKEN: your_token
+  DOMAINS: "example.com,www.example.com"
+  IP4_PROVIDER: cloudflare.trace
+  IP6_PROVIDER: none
+  UPDATE_CRON: "@every 5m"
+  UPDATE_ON_START: "true"
+  DELETE_ON_STOP: "false"
+  DELETE_ON_FAILURE: "true"
+  TTL: "1"
+  PROXIED: "true"
+  EMOJI: "true"
+  REJECT_CLOUDFLARE_IPS: "true"

checkmk/.env.example

@@ -0,0 +1,2 @@
+CMK_PASSWORD=password
+TZ=Europe/Berlin

checkmk/compose.yaml

@@ -0,0 +1,39 @@
+services:
+  checkmk:
+    image: "checkmk/check-mk-raw:2.4.0-2026.09.14"
+    container_name: "checkmk"
+    restart: unless-stopped
+    # ports:
+    #   - 5000:5000
+    #   - 6776:8000
+    volumes:
+      - sites:/omd/sites
+    tmpfs:
+      - /opt/omd/sites/cmk/tmp:uid=1000,gid=1000
+    environment:
+      - CMK_PASSWORD=${CMK_PASSWORD:-password}
+      - CMK_SITE_ID=cmk
+      - TZ=${TZ:-Etc/UTC}
+    labels:
+      - "traefik.enable=true"
+      - "traefik.http.services.checkmk.loadbalancer.server.port=5000"
+
+      # Prod Router
+      - "traefik.http.routers.checkmk.rule=Host(`cmk.forust.xyz`)"
+      - "traefik.http.routers.checkmk.entrypoints=websecure"
+      - "traefik.http.routers.checkmk.tls.certresolver=letsencrypt"
+      # Local Router
+      - "traefik.http.routers.checkmk-local.rule=Host(`cmk.workstation.internal`)"
+      - "traefik.http.routers.checkmk-local.entrypoints=websecure"
+      - "traefik.http.routers.checkmk-local.tls=true"
+      # Dev Router
+      - "traefik.http.routers.checkmk-dev.rule=Host(`cmk.gigaforust.internal`)"
+      - "traefik.http.routers.checkmk-dev.entrypoints=websecure"
+      - "traefik.http.routers.checkmk-dev.tls=true"
+    networks:
+      - proxy
+networks:
+  proxy:
+    external: true
+volumes:
+  sites:

checkmk/k8s/certificates.yaml

@@ -0,0 +1,28 @@
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: checkmk-prod-tls
+  namespace: checkmk
+spec:
+  secretName: checkmk-prod-tls
+  dnsNames:
+    - cmk.forust.xyz
+  issuerRef:
+    name: letsencrypt-prod
+    kind: ClusterIssuer
+---
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: internal-wildcard-tls
+  namespace: checkmk
+spec:
+  secretName: internal-wildcard-tls
+  dnsNames:
+    - "*.workstation.internal"
+    - "*.gigaforust.internal"
+    - workstation.internal
+    - gigaforust.internal
+  issuerRef:
+    name: internal-ca
+    kind: ClusterIssuer

checkmk/k8s/checkmk.yaml

@@ -0,0 +1,75 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: checkmk-service
+  namespace: checkmk
+spec:
+  selector:
+    app: checkmk
+  ports:
+    - name: web
+      port: 5000
+      targetPort: 5000
+    - name: agent-receiver
+      port: 8000
+      targetPort: 8000
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: checkmk-deployment
+  namespace: checkmk
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: checkmk
+  template:
+    metadata:
+      labels:
+        app: checkmk
+    spec:
+      containers:
+        - name: checkmk
+          image: checkmk/check-mk-raw:2.4.0-2026.09.14
+          envFrom:
+            - secretRef:
+                name: checkmk-secrets
+            - configMapRef:
+                name: checkmk-config
+          ports:
+            - name: web
+              containerPort: 5000
+            - name: agent-receiver
+              containerPort: 8000
+          volumeMounts:
+            - name: sites
+              mountPath: /omd/sites
+            - name: tmp
+              mountPath: /opt/omd/sites/cmk/tmp
+          resources:
+            requests:
+              memory: "2Gi"
+              cpu: "600m"
+            limits:
+              memory: "5Gi"
+              cpu: "4"
+      volumes:
+        - name: sites
+          persistentVolumeClaim:
+            claimName: checkmk-sites-pvc
+        - name: tmp
+          emptyDir:
+            medium: Memory
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: checkmk-sites-pvc
+  namespace: checkmk
+spec:
+  accessModes:
+    - ReadWriteOnce
+  resources:
+    requests:
+      storage: 5Gi

checkmk/k8s/config.yaml

@@ -0,0 +1,8 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: checkmk-config
+  namespace: checkmk
+data:
+  TZ: Europe/Bratislava
+  CMK_SITE_ID: cmk

checkmk/k8s/ingress.yaml

@@ -0,0 +1,52 @@
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
+metadata:
+  name: checkmk-prod
+  namespace: checkmk
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`cmk.forust.xyz`)
+      kind: Rule
+      middlewares:
+        - name: crowdsec-bouncer
+          namespace: crowdsec
+      services:
+        - name: checkmk-service
+          port: 5000
+  tls:
+    secretName: checkmk-prod-tls
+---
+apiVersion: traefik.io/v1alpha1
+kind: IngressRouteTCP
+metadata:
+  name: checkmk-agent-receiver
+  namespace: checkmk
+spec:
+  entryPoints:
+    - checkmk-agent
+  routes:
+    - match: HostSNI(`*`)
+      services:
+        - name: checkmk-service
+          port: 8000
+  tls:
+    passthrough: true
+---
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
+metadata:
+  name: checkmk-local
+  namespace: checkmk
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`cmk.workstation.internal`) || Host(`cmk.gigaforust.internal`)
+      kind: Rule
+      services:
+        - name: checkmk-service
+          port: 5000
+  tls:
+    secretName: internal-wildcard-tls

checkmk/k8s/namespace.yaml

@@ -0,0 +1,4 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: checkmk

checkmk/k8s/secrets.yaml.example

@@ -0,0 +1,8 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: checkmk-secrets
+  namespace: checkmk
+type: Opaque
+stringData:
+  CMK_PASSWORD: "password"

cloudflared/k8s/.gitignore

@@ -0,0 +1 @@
+secret.yaml

cloudflared/k8s/deployment.yaml

@@ -0,0 +1,37 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: cloudflared
+  labels:
+    app: cloudflared
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: cloudflared
+  template:
+    metadata:
+      labels:
+        app: cloudflared
+    spec:
+      containers:
+        - name: cloudflared
+          image: cloudflare/cloudflared:2026.9.3
+          imagePullPolicy: IfNotPresent
+          args:
+            - tunnel
+            - --no-autoupdate
+            - run
+          env:
+            - name: TUNNEL_TOKEN
+              valueFrom:
+                secretKeyRef:
+                  name: cloudflared-secrets
+                  key: TUNNEL_TOKEN
+          resources:
+            requests:
+              memory: "32Mi"
+              cpu: "30m"
+            limits:
+              memory: "128Mi"
+              cpu: "200m"

cloudflared/k8s/secret.yaml.example

@@ -0,0 +1,7 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: cloudflared-secrets
+type: Opaque
+stringData:
+  TUNNEL_TOKEN: your_tunnel_token_here

converters/.env.example

@@ -0,0 +1,9 @@
+ACCOUNT_REGISTRATION=false
+HTTP_ALLOWED=false
+ALLOW_UNAUTHENTICAED=false
+AUTO_DELETE_EVERY_N_HOURS=24
+WEBROOT=/convert
+HIDE_HISTORY=false
+LANGUAGE=en
+UNAUTHED_USER_SHARING=false
+MAX_CONVERT_PROCESS=0

converters/compose.yaml

@@ -0,0 +1,70 @@
+services:
+  convertx:
+    container_name: convertx
+    image: ghcr.io/c4illin/convertx:v0.18.0
+    restart: unless-stopped
+    ports:
+      - "9992:3000"
+    # https://github.com/C4illin/ConvertX#environment-variables
+    environment:
+      - JWT_SECRET=$(JWT_SECRET)
+      - ACCOUNT_REGISTRATION=$(ACCOUNT_REGISTRATION:-false)
+      - HTTP_ALLOWED=$(HTTP_ALLOWED:-false)
+      - ALLOW_UNAUTHENTICATED=$(ALLOW_UNAUTHENTICATED:-false)
+      - AUTO_DELETE_EVERY_N_HOURS=$(AUTO_DELETE_EVERY_N_HOURS:-24)
+      - WEBROOT=$(WEBROOT)
+      - HIDE_HISTORY=$(HIDE_HISTORY:-false)
+      - LANGUAGE=$(LANGUAGE:-en)
+      - UNAUTHENTICATED_USER_SHARING=$(UNAUTHENTICATED_USER_SHARING:-false)
+      - MAX_CONVERT_PROCESS=$(MAX_CONVERT_PROCESS:-0)
+    labels:
+      - "traefik.enable=true"
+      - "traefik.http.services.convertx.loadbalancer.server.port=3000"
+      # Prod Router
+      - "traefik.http.routers.convertx.rule=(Host(`forust.xyz`) || Host(`www.forust.xyz`)) && PathPrefix(`/convert`)"
+      - "traefik.http.routers.convertx.entrypoints=websecure"
+      - "traefik.http.routers.convertx.priority=50"
+      - "traefik.http.routers.convertx.tls.certresolver=letsencrypt"
+      # Local Router
+      - "traefik.http.routers.convertx-local.rule=Host(`workstation.internal`) && PathPrefix(`/convert`)"
+      - "traefik.http.routers.convertx-local.entrypoints=websecure"
+      - "traefik.http.routers.convertx-local.priority=50"
+      - "traefik.http.routers.convertx-local.tls=true"
+      # Dev Router
+      - "traefik.http.routers.convertx-dev.rule=Host(`gigaforust.internal`) && PathPrefix(`/convert`)"
+      - "traefik.http.routers.convertx-dev.entrypoints=websecure"
+      - "traefik.http.routers.convertx-dev.priority=50"
+      - "traefik.http.routers.convertx-dev.tls=true"
+    networks:
+      - proxy
+    volumes:
+      - data:/app/data
+
+  bentopdf:
+    container_name: bentopdf
+    image: bentopdf/bentopdf@sha256:4eb4ec8f5030faf87c29a73d3d5a2781f28a597cf440c3ab111eb96aee550871
+    restart: unless-stopped
+    labels:
+      - "traefik.enable=true"
+      - "traefik.http.services.bentopdf.loadbalancer.server.port=8080"
+
+      # Prod router
+      - "traefik.http.routers.bentopdf.rule=Host(`pdf.forust.xyz`)"
+      - "traefik.http.routers.bentopdf.entrypoints=websecure"
+      - "traefik.http.routers.bentopdf.tls.certresolver=letsencrypt"
+      - "traefik.http.routers.bentopdf.tls=true"
+      # Local router
+      - "traefik.http.routers.bentopdf-local.rule=Host(`pdf.workstation.internal`)"
+      - "traefik.http.routers.bentopdf-local.entrypoints=websecure"
+      - "traefik.http.routers.bentopdf-local.tls=true"
+      # Dev router
+      - "traefik.http.routers.bentopdf-dev.rule=Host(`pdf.gigaforust.internal`)"
+      - "traefik.http.routers.bentopdf-dev.entrypoints=websecure"
+      - "traefik.http.routers.bentopdf-dev.tls=true"
+    networks:
+      - proxy
+networks:
+  proxy:
+    external: true
+volumes:
+  data:

converters/k8s/bentopdf.yaml

@@ -0,0 +1,42 @@
+kind: Service
+apiVersion: v1
+metadata:
+  name: bentopdf-service
+  namespace: converters
+spec:
+  selector:
+    app: bentopdf
+  ports:
+    - port: 8080
+      targetPort: 8080
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: bentopdf-deployment
+  namespace: converters
+spec:
+  replicas: 2
+  selector:
+    matchLabels:
+      app: bentopdf
+  template:
+    metadata:
+      labels:
+        app: bentopdf
+    spec:
+      containers:
+        - image: bentopdf/bentopdf@sha256:4eb4ec8f5030faf87c29a73d3d5a2781f28a597cf440c3ab111eb96aee550871
+          imagePullPolicy: Always
+          name: bentopdf
+          ports:
+            - containerPort: 8080
+          resources:
+            requests:
+              memory: "50Mi"
+              cpu: "50m"
+              ephemeral-storage: "100Mi"
+            limits:
+              memory: "700Mi"
+              cpu: "700m"
+              ephemeral-storage: "5Gi"

converters/k8s/certificates.yaml

@@ -0,0 +1,42 @@
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: convertx-prod-tls
+  namespace: converters
+spec:
+  secretName: convertx-prod-tls
+  dnsNames:
+    - forust.xyz
+    - www.forust.xyz
+  issuerRef:
+    name: letsencrypt-prod
+    kind: ClusterIssuer
+---
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: bentopdf-prod-tls
+  namespace: converters
+spec:
+  secretName: bentopdf-prod-tls
+  dnsNames:
+    - pdf.forust.xyz
+  issuerRef:
+    name: letsencrypt-prod
+    kind: ClusterIssuer
+---
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: internal-wildcard-tls
+  namespace: converters
+spec:
+  secretName: internal-wildcard-tls
+  dnsNames:
+    - "*.workstation.internal"
+    - "*.gigaforust.internal"
+    - workstation.internal
+    - gigaforust.internal
+  issuerRef:
+    name: internal-ca
+    kind: ClusterIssuer

converters/k8s/config.yaml.example

@@ -0,0 +1,16 @@
+# test manifest with docker and k8s config keys mismatch
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: convertx-config
+  namespace: converters
+data:
+  ACCOUNT_REGISTRATION: "false"
+  HTTP_ALLOWED: "false"
+  ALLOW_UNAUTHENTICAED: "false"
+  AUTO_DELETE_EVERY_N_HOURS: "24"
+  WEBROOT: "/convert"
+  HIDE_HISTORY: "false"
+  LANGUAGE: "en"
+  UNAUTHED_USER_SHARING: "false"
+  MAX_CONVERT_PROCESS: "0"

converters/k8s/convertx.yaml

@@ -0,0 +1,63 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: convertx-service
+  namespace: converters
+spec:
+  selector:
+    app: convertx
+  ports:
+    - port: 3000
+      targetPort: 3000
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: convertx-deployment
+  namespace: converters
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: convertx
+  template:
+    metadata:
+      labels:
+        app: convertx
+    spec:
+      containers:
+        - image: ghcr.io/c4illin/convertx:v0.18.0
+          name: convertx
+          envFrom:
+            - configMapRef:
+                name: convertx-config
+            - secretRef:
+                name: convertx-secrets
+          ports:
+            - containerPort: 3000
+          volumeMounts:
+            - mountPath: /data
+              name: data
+          resources:
+            requests:
+              memory: "250Mi"
+              cpu: "100m"
+            limits:
+              cpu: "1500m"
+              memory: "1.5Gi"
+      volumes:
+        - name: data
+          persistentVolumeClaim:
+            claimName: convertx-pvc
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: convertx-pvc
+  namespace: converters
+spec:
+  accessModes:
+    - ReadWriteOnce
+  resources:
+    requests:
+      storage: 2Gi

converters/k8s/ingress.yaml

@@ -0,0 +1,70 @@
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
+metadata:
+  name: convertx-prod
+  namespace: converters
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: (Host(`forust.xyz`) || Host(`www.forust.xyz`)) && PathPrefix(`/convert`)
+      kind: Rule
+      priority: 50
+      services:
+        - name: convertx-service
+          port: 3000
+  tls:
+    secretName: convertx-prod-tls
+---
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
+metadata:
+  name: convertx-local
+  namespace: converters
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: (Host(`workstation.internal`) || Host(`gigaforust.internal`)) && PathPrefix(`/convert`)
+      kind: Rule
+      priority: 50
+      services:
+        - name: convertx-service
+          port: 3000
+  tls:
+    secretName: internal-wildcard-tls
+
+---
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
+metadata:
+  name: bentopdf-prod
+  namespace: converters
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`pdf.forust.xyz`)
+      kind: Rule
+      services:
+        - name: bentopdf-service
+          port: 8080
+  tls:
+    secretName: bentopdf-prod-tls
+---
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
+metadata:
+  name: bentopdf-local
+  namespace: converters
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`pdf.workstation.internal`) || Host(`pdf.gigaforust.internal`)
+      kind: Rule
+      services:
+        - name: bentopdf-service
+          port: 8080
+  tls:
+    secretName: internal-wildcard-tls

converters/k8s/namespace.yaml

@@ -0,0 +1,4 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: converters

converters/k8s/secrets.yaml.example

@@ -0,0 +1,8 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: convertx-secrets
+  namespace: converters
+type: Opaque
+stringData:
+  jwt-secret: ""

crowdsec/k8s/crowdsec-middleware.yaml

@@ -0,0 +1,24 @@
+apiVersion: traefik.io/v1alpha1
+kind: Middleware
+metadata:
+  name: crowdsec-bouncer
+  namespace: crowdsec
+spec:
+  plugin:
+    crowdsec-bouncer:
+      enabled: true
+      LogLevel: INFO
+      CrowdsecMode: live
+      CrowdsecLapiScheme: http
+      CrowdsecLapiHost: crowdsec-service.crowdsec.svc.cluster.local:8080
+      CrowdsecLapiKeyFile: "/etc/traefik/secrets/traefik-api-key"
+      # LAPI lookup is SYNCHRONOUS and per-request: the plugin blocks on
+      # `GET /v1/decisions?ip=...&banned=true` before the request reaches
+      # the backend, and fails CLOSED (403) if the lookup exceeds the
+      # timeout. Unset, the fork defaults to 10s, which is an eternity for
+      # a request path: a single slow LAPI (idle 1.3-7.4s here) turned
+      # every request into a 10s hang and then a self-inflicted 403.
+      # 2s keeps the fail-closed path fast and bounded; with the LAPI
+      # resourced properly (see crowdsec-values.yaml) the lookup is
+      # sub-100ms and this budget is never hit.
+      CrowdsecLapiTimeout: "2s"

crowdsec/k8s/crowdsec-values.yaml

@@ -0,0 +1,126 @@
+container_runtime: containerd
+
+agent:
+  acquisition: []
+  additionalAcquisition:
+    - labels:
+        type: traefik
+      limit: 1000
+      query: |
+        {namespace="traefik"}
+      source: loki
+      url: http://loki.prometheus.svc.cluster.local:3100/
+      wait_for_ready: 30s
+  env:
+    - name: COLLECTIONS
+      value: crowdsecurity/traefik crowdsecurity/base-http-scenarios
+    - name: DISABLE_COLLECTIONS
+      value: crowdsecurity/sshd
+  metrics:
+    enabled: true
+    serviceMonitor:
+      additionalLabels:
+        release: prometheus-stack
+      enabled: true
+  # Static machine identity: agent pods mount pre-created LAPI credentials
+  # (Secret crowdsec-agent-credentials, key local_api_credentials.yaml)
+  # at the exact path the agent entrypoint expects. Together with the
+  # patched register-init (enforced by janitor-cronjob.yaml) the agent
+  # never calls `cscli lapi register` in steady state, so pod names,
+  # restarts and reboots can no longer break it.
+  extraVolumes:
+    - name: static-creds
+      secret:
+        secretName: crowdsec-agent-credentials
+        items:
+          - key: local_api_credentials.yaml
+            path: local_api_credentials.yaml
+  extraVolumeMounts:
+    - name: static-creds
+      mountPath: /tmp_config/local_api_credentials.yaml
+      subPath: local_api_credentials.yaml
+      readOnly: true
+  resources:
+    limits:
+      cpu: 200m
+      memory: 500Mi
+    requests:
+      cpu: 50m
+      memory: 100Mi
+
+config:
+  parsers:
+    s02-enrich:
+      mobile-whitelist.yaml: |
+        name: forust/mobile-whitelist
+        description: "Whitelist SWAN/4ka mobile network"
+        whitelist:
+          reason: "Mobile IP whitelist"
+          cidr:
+            - "84.245.64.0/18"
+
+  postoverflows:
+    s01-whitelist:
+      home-dynamic-ip.yaml: |
+        name: forust/home-dynamic-ip
+        description: "Whitelist home dynamic IP"
+        whitelist:
+          reason: "Home dynamic IP"
+          expression:
+            - evt.Overflow.Alert.Source.IP in LookupHost("ddns.forust.xyz")
+      # The hairpin-NAT address of the router (192.168.88.1) is what the
+      # Gitea Actions runner presents to Traefik - it is NOT the home
+      # dynamic IP, so the whitelist above did not cover it. During a
+      # deploy the runner POSTs to the Actions API many times a second;
+      # a single 403 storm was enough to earn it a 4h ban and break every
+      # later job. Whitelisting the whole LAN also covers phones and
+      # tablets browsing over 192.168.88.0/24.
+      lan.yaml: |
+        name: forust/lan
+        description: "Whitelist local network"
+        whitelist:
+          reason: "Local network"
+          cidr:
+            - "127.0.0.0/8"
+            - "10.0.0.0/8"
+            - "172.16.0.0/12"
+            - "192.168.0.0/16"
+
+lapi:
+  env:
+    - name: COLLECTIONS
+      value: crowdsecurity/traefik crowdsecurity/base-http-scenarios
+    - name: DISABLE_COLLECTIONS
+      value: crowdsecurity/linux crowdsecurity/sshd
+  metrics:
+    enabled: true
+    serviceMonitor:
+      additionalLabels:
+        release: prometheus-stack
+      enabled: true
+  persistentVolume:
+    config:
+      enabled: true
+      size: 100Mi
+      storageClassName: local-path-retain
+    data:
+      enabled: true
+      size: 1Gi
+      storageClassName: local-path-retain
+  # LAPI answers a blocking /v1/decisions lookup for EVERY bouncer-protected
+  # request (whole Traefik front door), so it is the hot path of the proxy.
+  # At 400m/500Mi it went CPU-throttled and idle lookups measured 1.3-7.4s,
+  # which pushed requests into the bouncer's fail-closed 403.
+  # Single replica on purpose: LAPI is stateful (BoltDB on the `data` PVC,
+  # credentials on the `config` PVC) - two replicas sharing those RWO
+  # volumes would corrupt the decision store. Scale up CPU, not replicas.
+  resources:
+    limits:
+      cpu: 1500m
+      memory: 1Gi
+    requests:
+      cpu: 250m
+      memory: 500Mi
+  service:
+    type: ClusterIP
+  storeLAPICscliCredentialsInSecret: true

crowdsec/k8s/grafana-dashboards.yaml

@@ -0,0 +1,32 @@
+apiVersion: v1
+data:
+  crowdsec-overview.json: "{\n  \"__inputs\": [\n    {\n      \"name\": \"DS_PROMETHEUS\",\n      \"label\": \"Prometheus\",\n      \"description\": \"\",\n      \"type\": \"datasource\",\n      \"pluginId\": \"prometheus\",\n      \"pluginName\": \"Prometheus\"\n    }\n  ],\n  \"__requires\": [\n    {\n      \"type\": \"grafana\",\n      \"id\": \"grafana\",\n      \"name\": \"Grafana\",\n      \"version\": \"8.1.2\"\n    },\n    {\n      \"type\": \"panel\",\n      \"id\": \"graph\",\n      \"name\": \"Graph (old)\",\n      \"version\": \"\"\n    },\n    {\n      \"type\": \"datasource\",\n      \"id\": \"prometheus\",\n      \"name\": \"Prometheus\",\n      \"version\": \"1.0.0\"\n    },\n    {\n      \"type\": \"panel\",\n      \"id\": \"stat\",\n      \"name\": \"Stat\",\n      \"version\": \"\"\n    },\n    {\n      \"type\": \"panel\",\n      \"id\": \"timeseries\",\n      \"name\": \"Time series\",\n      \"version\": \"\"\n    }\n  ],\n  \"annotations\": {\n    \"list\": [\n      {\n        \"builtIn\": 1,\n        \"datasource\": \"-- Grafana --\",\n        \"enable\": true,\n        \"hide\": true,\n        \"iconColor\": \"rgba(0, 211, 255, 1)\",\n        \"name\": \"Annotations & Alerts\",\n        \"target\": {\n          \"limit\": 100,\n          \"matchAny\": false,\n          \"tags\": [],\n          \"type\": \"dashboard\"\n        },\n        \"type\": \"dashboard\"\n      }\n    ]\n  },\n  \"editable\": true,\n  \"gnetId\": null,\n  \"graphTooltip\": 0,\n  \"id\": null,\n  \"links\": [],\n  \"panels\": [\n    {\n      \"collapsed\": false,\n      \"datasource\": null,\n      \"gridPos\": {\n        \"h\": 1,\n        \"w\": 24,\n        \"x\": 0,\n        \"y\": 0\n      },\n      \"id\": 24,\n      \"panels\": [],\n      \"title\": \"Summary\",\n      \"type\": \"row\"\n    },\n    {\n      \"cacheTimeout\": null,\n      \"datasource\": \"${DS_PROMETHEUS}\",\n      \"fieldConfig\": {\n        \"defaults\": {\n          \"color\": {\n            \"mode\": \"thresholds\"\n          },\n          \"mappings\": [\n            {\n              \"options\": {\n                \"match\": \"null\",\n                \"result\": {\n                  \"text\": \"N/A\"\n                }\n              },\n              \"type\": \"special\"\n            }\n          ],\n          \"thresholds\": {\n            \"mode\": \"absolute\",\n            \"steps\": [\n              {\n                \"color\": \"#E02F44\",\n                \"value\": null\n              },\n              {\n                \"color\": \"#E02F44\",\n                \"value\": 10\n              },\n              {\n                \"color\": \"#299c46\",\n                \"value\": 10\n              }\n            ]\n          },\n          \"unit\": \"none\"\n        },\n        \"overrides\": []\n      },\n      \"gridPos\": {\n        \"h\": 8,\n        \"w\": 6,\n        \"x\": 0,\n        \"y\": 1\n      },\n      \"id\": 2,\n      \"interval\": null,\n      \"links\": [],\n      \"maxDataPoints\": 100,\n      \"options\": {\n        \"colorMode\": \"background\",\n        \"graphMode\": \"none\",\n        \"justifyMode\": \"auto\",\n        \"orientation\": \"horizontal\",\n        \"reduceOptions\": {\n          \"calcs\": [\n            \"lastNotNull\"\n          ],\n          \"fields\": \"\",\n          \"values\": false\n        },\n        \"text\": {},\n        \"textMode\": \"auto\"\n      },\n      \"pluginVersion\": \"8.1.2\",\n      \"targets\": [\n        {\n          \"exemplar\": true,\n          \"expr\": \"count(cs_info)\",\n          \"interval\": \"\",\n          \"legendFormat\": \"\",\n          \"refId\": \"A\"\n        }\n      ],\n      \"timeFrom\": null,\n      \"timeShift\": null,\n      \"title\": \"Running Crowdsec\",\n      \"transparent\": true,\n      \"type\": \"stat\"\n    },\n    {\n      \"aliasColors\": {},\n      \"bars\": false,\n      \"dashLength\": 10,\n      \"dashes\": false,\n      \"datasource\": \"${DS_PROMETHEUS}\",\n      \"decimals\": 1,\n      \"fieldConfig\": {\n        \"defaults\": {\n          \"links\": []\n        },\n        \"overrides\": []\n      },\n      \"fill\": 1,\n      \"fillGradient\": 0,\n      \"gridPos\": {\n        \"h\": 8,\n        \"w\": 18,\n        \"x\": 6,\n        \"y\": 1\n      },\n      \"hiddenSeries\": false,\n      \"id\": 8,\n      \"legend\": {\n        \"alignAsTable\": true,\n        \"avg\": false,\n        \"current\": false,\n        \"max\": false,\n        \"min\": false,\n        \"rightSide\": true,\n        \"show\": true,\n        \"sort\": \"total\",\n        \"sortDesc\": true,\n        \"total\": true,\n        \"values\": true\n      },\n      \"lines\": true,\n      \"linewidth\": 1,\n      \"nullPointMode\": \"null\",\n      \"options\": {\n        \"alertThreshold\": true\n      },\n      \"percentage\": false,\n      \"pluginVersion\": \"8.1.2\",\n      \"pointradius\": 2,\n      \"points\": false,\n      \"renLine truncated
+kind: ConfigMap
+metadata:
+  labels:
+    app.kubernetes.io/managed-by: manual
+    grafana_dashboard: "1"
+  name: crowdsec-crowdsec-overview
+  namespace: prometheus
+---
+apiVersion: v1
+data:
+  crowdsec-lapi-metrics.json: "{\n  \"__inputs\": [\n    {\n      \"name\": \"DS_PROMETHEUS\",\n      \"label\": \"Prometheus\",\n      \"description\": \"\",\n      \"type\": \"datasource\",\n      \"pluginId\": \"prometheus\",\n      \"pluginName\": \"Prometheus\"\n    }\n  ],\n  \"__requires\": [\n    {\n      \"type\": \"panel\",\n      \"id\": \"bargauge\",\n      \"name\": \"Bar gauge\",\n      \"version\": \"\"\n    },\n    {\n      \"type\": \"grafana\",\n      \"id\": \"grafana\",\n      \"name\": \"Grafana\",\n      \"version\": \"8.1.2\"\n    },\n    {\n      \"type\": \"datasource\",\n      \"id\": \"prometheus\",\n      \"name\": \"Prometheus\",\n      \"version\": \"1.0.0\"\n    }\n  ],\n  \"annotations\": {\n    \"list\": [\n      {\n        \"builtIn\": 1,\n        \"datasource\": \"-- Grafana --\",\n        \"enable\": true,\n        \"hide\": true,\n        \"iconColor\": \"rgba(0, 211, 255, 1)\",\n        \"name\": \"Annotations & Alerts\",\n        \"target\": {\n          \"limit\": 100,\n          \"matchAny\": false,\n          \"tags\": [],\n          \"type\": \"dashboard\"\n        },\n        \"type\": \"dashboard\"\n      }\n    ]\n  },\n  \"editable\": true,\n  \"gnetId\": null,\n  \"graphTooltip\": 0,\n  \"id\": null,\n  \"iteration\": 1655915193937,\n  \"links\": [],\n  \"panels\": [\n    {\n      \"collapsed\": false,\n      \"datasource\": null,\n      \"gridPos\": {\n        \"h\": 1,\n        \"w\": 24,\n        \"x\": 0,\n        \"y\": 0\n      },\n      \"id\": 10,\n      \"panels\": [],\n      \"title\": \"Agents\",\n      \"type\": \"row\"\n    },\n    {\n      \"datasource\": \"${DS_PROMETHEUS}\",\n      \"fieldConfig\": {\n        \"defaults\": {\n          \"color\": {\n            \"mode\": \"thresholds\"\n          },\n          \"mappings\": [],\n          \"thresholds\": {\n            \"mode\": \"absolute\",\n            \"steps\": [\n              {\n                \"color\": \"green\",\n                \"value\": null\n              },\n              {\n                \"color\": \"red\",\n                \"value\": 80\n              }\n            ]\n          }\n        },\n        \"overrides\": []\n      },\n      \"gridPos\": {\n        \"h\": 8,\n        \"w\": 12,\n        \"x\": 0,\n        \"y\": 1\n      },\n      \"id\": 2,\n      \"options\": {\n        \"displayMode\": \"gradient\",\n        \"orientation\": \"vertical\",\n        \"reduceOptions\": {\n          \"calcs\": [\n            \"lastNotNull\"\n          ],\n          \"fields\": \"\",\n          \"values\": false\n        },\n        \"showUnfilled\": false,\n        \"text\": {}\n      },\n      \"pluginVersion\": \"8.1.2\",\n      \"repeat\": \"query0\",\n      \"repeatDirection\": \"h\",\n      \"targets\": [\n        {\n          \"exemplar\": false,\n          \"expr\": \"sum(rate(cs_lapi_request_duration_seconds_bucket{endpoint=\\\"/v1/watchers/login\\\", instance=\\\"$lapi\\\"}[$__rate_interval])) by (le)\",\n          \"format\": \"heatmap\",\n          \"interval\": \"\",\n          \"legendFormat\": \"{{le}}\",\n          \"refId\": \"A\"\n        }\n      ],\n      \"title\": \"Agents Login\",\n      \"type\": \"heatmap\"\n    },\n    {\n      \"datasource\": \"${DS_PROMETHEUS}\",\n      \"fieldConfig\": {\n        \"defaults\": {\n          \"color\": {\n            \"mode\": \"thresholds\"\n          },\n          \"mappings\": [],\n          \"thresholds\": {\n            \"mode\": \"absolute\",\n            \"steps\": [\n              {\n                \"color\": \"green\",\n                \"value\": null\n              }\n            ]\n          },\n          \"unit\": \"none\"\n        },\n        \"overrides\": []\n      },\n      \"gridPos\": {\n        \"h\": 8,\n        \"w\": 12,\n        \"x\": 12,\n        \"y\": 1\n      },\n      \"id\": 6,\n      \"options\": {\n        \"displayMode\": \"gradient\",\n        \"orientation\": \"auto\",\n        \"reduceOptions\": {\n          \"calcs\": [\n            \"lastNotNull\"\n          ],\n          \"fields\": \"\",\n          \"values\": false\n        },\n        \"showUnfilled\": false,\n        \"text\": {}\n      },\n      \"pluginVersion\": \"8.1.2\",\n      \"targets\": [\n        {\n          \"exemplar\": true,\n          \"expr\": \"sum(rate(cs_lapi_request_duration_seconds_bucket{endpoint=\\\"/v1/watchers/login\\\"}[$__rate_interval])) by (le)\",\n          \"format\": \"heatmap\",\n          \"interval\": \"\",\n          \"legendFormat\": \"{{le}}\",\n          \"refId\": \"A\"\n        }\n      ],\n      \"title\": \"Heartbeat\",\n      \"type\": \"heatmap\"\n    },\n    {\n      \"collapsed\": false,\n      \"datasource\": null,\n      \"gridPos\": {\n        \"h\": 1,\n        \"w\": 24,\n        \"x\": 0,\n        \"y\": 9\n      },\n      \"id\": 12,\n      \"panels\": [],\n      \"title\": \"Decisions\",\n      \"type\": \"row\"\n    },\n    {\n      \"datasource\": \"${DS_PROMETHEUS}\",\n  Line truncated
+kind: ConfigMap
+metadata:
+  labels:
+    app.kubernetes.io/managed-by: manual
+    grafana_dashboard: "1"
+  name: crowdsec-crowdsec-lapi-metrics
+  namespace: prometheus
+---
+apiVersion: v1
+data:
+  crowdsec-insight.json: "{\n  \"__inputs\": [\n    {\n      \"name\": \"DS_PROMETHEUS\",\n      \"label\": \"Prometheus\",\n      \"description\": \"\",\n      \"type\": \"datasource\",\n      \"pluginId\": \"prometheus\",\n      \"pluginName\": \"Prometheus\"\n    }\n  ],\n  \"__requires\": [\n    {\n      \"type\": \"panel\",\n      \"id\": \"bargauge\",\n      \"name\": \"Bar gauge\",\n      \"version\": \"\"\n    },\n    {\n      \"type\": \"panel\",\n      \"id\": \"gauge\",\n      \"name\": \"Gauge\",\n      \"version\": \"\"\n    },\n    {\n      \"type\": \"grafana\",\n      \"id\": \"grafana\",\n      \"name\": \"Grafana\",\n      \"version\": \"8.1.2\"\n    },\n    {\n      \"type\": \"datasource\",\n      \"id\": \"prometheus\",\n      \"name\": \"Prometheus\",\n      \"version\": \"1.0.0\"\n    },\n    {\n      \"type\": \"panel\",\n      \"id\": \"stat\",\n      \"name\": \"Stat\",\n      \"version\": \"\"\n    }\n  ],\n  \"annotations\": {\n    \"list\": [\n      {\n        \"builtIn\": 1,\n        \"datasource\": \"-- Grafana --\",\n        \"enable\": true,\n        \"hide\": true,\n        \"iconColor\": \"rgba(0, 211, 255, 1)\",\n        \"name\": \"Annotations & Alerts\",\n        \"target\": {\n          \"limit\": 100,\n          \"matchAny\": false,\n          \"tags\": [],\n          \"type\": \"dashboard\"\n        },\n        \"type\": \"dashboard\"\n      }\n    ]\n  },\n  \"editable\": true,\n  \"gnetId\": null,\n  \"graphTooltip\": 0,\n  \"id\": null,\n  \"iteration\": 1655915159751,\n  \"links\": [],\n  \"panels\": [\n    {\n      \"collapsed\": true,\n      \"datasource\": null,\n      \"gridPos\": {\n        \"h\": 1,\n        \"w\": 24,\n        \"x\": 0,\n        \"y\": 0\n      },\n      \"id\": 22,\n      \"panels\": [\n        {\n          \"cacheTimeout\": null,\n          \"datasource\": \"${DS_PROMETHEUS}\",\n          \"fieldConfig\": {\n            \"defaults\": {\n              \"color\": {\n                \"mode\": \"thresholds\"\n              },\n              \"mappings\": [\n                {\n                  \"options\": {\n                    \"match\": \"null\",\n                    \"result\": {\n                      \"text\": \"N/A\"\n                    }\n                  },\n                  \"type\": \"special\"\n                }\n              ],\n              \"thresholds\": {\n                \"mode\": \"absolute\",\n                \"steps\": [\n                  {\n                    \"color\": \"green\",\n                    \"value\": null\n                  },\n                  {\n                    \"color\": \"red\",\n                    \"value\": 80\n                  }\n                ]\n              },\n              \"unit\": \"dateTimeAsIso\"\n            },\n            \"overrides\": []\n          },\n          \"gridPos\": {\n            \"h\": 9,\n            \"w\": 5,\n            \"x\": 2,\n            \"y\": 1\n          },\n          \"id\": 2,\n          \"interval\": null,\n          \"links\": [],\n          \"maxDataPoints\": 100,\n          \"options\": {\n            \"colorMode\": \"none\",\n            \"graphMode\": \"none\",\n            \"justifyMode\": \"auto\",\n            \"orientation\": \"horizontal\",\n            \"reduceOptions\": {\n              \"calcs\": [\n                \"lastNotNull\"\n              ],\n              \"fields\": \"\",\n              \"values\": false\n            },\n            \"text\": {},\n            \"textMode\": \"auto\"\n          },\n          \"pluginVersion\": \"8.1.2\",\n          \"targets\": [\n            {\n              \"exemplar\": true,\n              \"expr\": \"(process_start_time_seconds{instance=\\\"$instance\\\"})*1000\",\n              \"interval\": \"\",\n              \"legendFormat\": \"{{instance}}\",\n              \"refId\": \"A\"\n            }\n          ],\n          \"timeFrom\": null,\n          \"timeShift\": null,\n          \"title\": \"Up since\",\n          \"type\": \"stat\"\n        },\n        {\n          \"datasource\": \"${DS_PROMETHEUS}\",\n          \"fieldConfig\": {\n            \"defaults\": {\n              \"displayName\": \"\",\n              \"mappings\": [],\n              \"thresholds\": {\n                \"mode\": \"absolute\",\n                \"steps\": [\n                  {\n                    \"color\": \"green\",\n                    \"value\": null\n                  }\n                ]\n              },\n              \"unit\": \"decbytes\"\n            },\n            \"overrides\": []\n          },\n          \"gridPos\": {\n            \"h\": 9,\n            \"w\": 5,\n            \"x\": 7,\n            \"y\": 1\n          },\n          \"id\": 4,\n          \"options\": {\n            \"orientation\": \"auto\",\n            \"reduceOptions\": {\n              \"calcs\": [\n                \"mean\"\n              ],\n              \"fields\": \"\",\n              \"values\": false\n            },\Line truncated
+kind: ConfigMap
+metadata:
+  labels:
+    app.kubernetes.io/managed-by: manual
+    grafana_dashboard: "1"
+  name: crowdsec-crowdsec-insight
+  namespace: prometheus

crowdsec/k8s/janitor-cronjob.yaml

@@ -0,0 +1,214 @@
+# CrowdSec self-healing: static machine identity + enforcement loops.
+#
+# Problem it fixes: the chart's agent init container runs
+#   `cscli lapi register --machine "$POD_NAME" ...`
+# unconditionally. Credentials live in an emptyDir, the machine row lives
+# in LAPI's persistent DB. Any init re-run for an already-known pod name
+# (kubelet restart, node reboot) dies with
+#   403 Forbidden: user '<pod>' already exist
+# and the DaemonSet pod sticks in Init forever. Every DS restart also
+# leaves an orphan machine row that is never cleaned.
+#
+# Design (name-independent):
+#   * Agent identity is a STATIC machine `crowdsec-agent-workstation`
+#     whose password lives in Secret `crowdsec-agent-credentials`
+#     (created once, manually - like all other secrets in this repo).
+#     The secret is mounted into agent pods at
+#     /tmp_config/local_api_credentials.yaml (see extraVolumeMounts in
+#     crowdsec-values.yaml), which is exactly the path the agent's main
+#     container copies into place at startup.
+#   * The DS init command is patched (strategic merge, by container name)
+#     to SKIP registration when that file exists, keeping the legacy
+#     register path only as fallback. Detection marker in the patched
+#     command: `[ -s /tmp_config`.
+#   * This CronJob enforces the desired state hourly, so recovery is
+#     automatic even after `helm upgrade` reverts the DS patch or the
+#     LAPI database is wiped:
+#       1. patch DS init if it still has the unconditional register
+#          (no-op otherwise - no restart churn);
+#       2. prune machines with no heartbeat for 2h (orphan hygiene);
+#       3. ensure the static machine exists, recreating it with the
+#          Secret password if missing (agent retry loops reconnect
+#          on their own - same name + same password);
+#       4. prune bouncer entries idle for 30d;
+#       5. delete decisions from LePresidente/http-generic-403-bf, a hub
+#          scenario that bans an IP for 4h after 5 POST-403s in 10s and
+#          therefore bans us for our own bouncer's fail-closed 403s.
+#
+# Manual apply (crowdsec/k8s is NOT managed by deploy.yaml):
+#   kubectl apply -f crowdsec/k8s/janitor-cronjob.yaml
+# Force a run:
+#   kubectl create job -n crowdsec --from=cronjob/crowdsec-janitor janitor-now
+#
+# Helm upgrades: the janitor's strategic patch puts the DS field under
+# the `kubectl-patch` field manager, so a plain `helm upgrade` FAILS
+# with an SSA conflict on initContainers[].command. Procedure:
+#   1. revert init to chart state (kills the conflict):
+#        helm template crowdsec crowdsec/crowdsec --version <ver> \
+#          -n crowdsec -f crowdsec/k8s/crowdsec-values.yaml > /tmp/r.yaml
+#        python3 -c "import yaml,json; ..." # build revert patch from
+#        the rendered DaemonSet init command, then
+#        kubectl patch ds crowdsec-agent -n crowdsec \
+#          --type strategic -p "\$(cat /tmp/revert_patch.json)"
+#   2. helm upgrade --install crowdsec ... (no --force needed)
+#   3. janitor-now right away (upgrade reverts init; new pods would
+#      sit in Init until the next hourly run otherwise).
+#
+# One-time bootstrap (order matters):
+#   1. Create Secret + static machine (see commands in chat).
+#   2. Apply this file, trigger janitor-now, wait for agent 1/1.
+#   3. One-time orphan cleanup:
+#        kubectl exec -n crowdsec deploy/crowdsec-lapi -- \
+#          cscli machines prune --duration 1h --force
+#   4. Only then `helm upgrade` crowdsec with the extraVolumes values.
+#      Upgrade reverts the DS patch; trigger janitor-now right after it
+#      (otherwise new pods sit in Init until the next hourly run, then
+#      self-heal anyway).
+#
+# Password rotation: update the Secret, delete the machine
+# (`cscli machines delete crowdsec-agent-workstation`), trigger
+# janitor-now (recreates it), then `kubectl rollout restart
+# ds/crowdsec-agent -n crowdsec` (agent reads the file at startup only).
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: crowdsec-janitor
+  namespace: crowdsec
+  labels:
+    app.kubernetes.io/part-of: crowdsec
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: crowdsec-janitor
+  namespace: crowdsec
+  labels:
+    app.kubernetes.io/part-of: crowdsec
+rules:
+  - apiGroups: [""]
+    resources: ["pods"]
+    verbs: ["get", "list"]
+  - apiGroups: [""]
+    resources: ["pods/exec"]
+    verbs: ["create"]
+  - apiGroups: ["apps"]
+    resources: ["daemonsets"]
+    verbs: ["get", "patch"]
+  # `kubectl exec deploy/<name>` resolves deploy -> replicaset -> pod,
+  # which needs read access to these (exec itself is pods/exec above).
+  - apiGroups: ["apps"]
+    resources: ["deployments", "replicasets"]
+    verbs: ["get", "list"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: crowdsec-janitor
+  namespace: crowdsec
+  labels:
+    app.kubernetes.io/part-of: crowdsec
+subjects:
+  - kind: ServiceAccount
+    name: crowdsec-janitor
+    namespace: crowdsec
+roleRef:
+  kind: Role
+  name: crowdsec-janitor
+  apiGroup: rbac.authorization.k8s.io
+---
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: crowdsec-janitor
+  namespace: crowdsec
+  labels:
+    app.kubernetes.io/part-of: crowdsec
+spec:
+  schedule: "17 * * * *"
+  concurrencyPolicy: Forbid
+  successfulJobsHistoryLimit: 3
+  failedJobsHistoryLimit: 3
+  jobTemplate:
+    spec:
+      activeDeadlineSeconds: 300
+      template:
+        metadata:
+          labels:
+            app.kubernetes.io/part-of: crowdsec
+        spec:
+          serviceAccountName: crowdsec-janitor
+          restartPolicy: OnFailure
+          containers:
+            - name: janitor
+              # Same image the chart itself uses for registration jobs;
+              # IfNotPresent so it works while the node is offline
+              # (layer cached from the chart install).
+              image: alpine/kubectl:latest
+              imagePullPolicy: IfNotPresent
+              env:
+                - name: AGENT_PASSWORD
+                  valueFrom:
+                    secretKeyRef:
+                      name: crowdsec-agent-credentials
+                      key: password
+              command:
+                - /bin/sh
+                - -c
+                - |
+                  set -eu
+                  LAPI_EXEC="kubectl exec -n crowdsec deploy/crowdsec-lapi --"
+                  echo "== 1. enforce patched agent init =="
+                  CUR=$(kubectl get ds crowdsec-agent -n crowdsec \
+                    -o jsonpath='{.spec.template.spec.initContainers[0].command[2]}')
+                  case "$CUR" in
+                    *'-s /tmp_config'*)
+                      echo "init already patched"
+                      ;;
+                    *)
+                      echo "patching init"
+                      WAIT='until nc "$LAPI_HOST" "$LAPI_PORT" -z'
+                      WAIT="$WAIT; do echo waiting for lapi to start; sleep 5; done"
+                      LINK='ln -s /staging/etc/crowdsec /etc/crowdsec'
+                      REG='cscli lapi register --machine "$USERNAME"'
+                      REG="$REG -u \"\$LAPI_URL\" --token \"\$REGISTRATION_TOKEN\""
+                      CREDS=/tmp_config/local_api_credentials.yaml
+                      CMD="$WAIT; $LINK; [ -s $CREDS ] || {"
+                      CMD="$CMD $REG && cp"
+                      CMD="$CMD /etc/crowdsec/local_api_credentials.yaml $CREDS; }"
+                      ESC=$(printf '%s' "$CMD" | sed 's/"/\\"/g')
+                      PATCH='{"spec":{"template":{"spec":{"initContainers":'
+                      PATCH=$PATCH'[{"name":"wait-for-lapi-and-register",'
+                      PATCH=$PATCH'"command":["sh","-c","'$ESC'"]}]}}}}'
+                      kubectl patch ds crowdsec-agent -n crowdsec \
+                        --type strategic -p "$PATCH"
+                      ;;
+                  esac
+                  echo "== 2. prune orphan machines (no heartbeat for 2h) =="
+                  $LAPI_EXEC cscli machines prune --duration 2h --force
+                  echo "== 3. ensure static machine exists =="
+                  if $LAPI_EXEC cscli machines inspect \
+                    crowdsec-agent-workstation >/dev/null 2>&1; then
+                    echo "static machine present"
+                  else
+                    echo "recreating static machine"
+                    $LAPI_EXEC cscli machines add crowdsec-agent-workstation \
+                      --password "$AGENT_PASSWORD" --force
+                  fi
+                  echo "== 4. prune stale bouncers (no pull for 30d) =="
+                  $LAPI_EXEC cscli bouncers prune -d 720h --force
+                  echo "== 5. drop http-403-bf decisions (4h self-bans) =="
+                  # `LePresidente/http-generic-403-bf` (hub item
+                  # crowdsecurity/http-generic-bf v0.9) bans any source IP
+                  # after 5 POSTs answered 403 within 10s, for 4h. That
+                  # includes 403s this homelab generates ITSELF (any
+                  # bouncer fail-closed, any app CSRF/rate-limit 403), and a
+                  # 4h ban on the runner/home IP silently breaks deploys and
+                  # browsing. The scenario cannot be removed per-scenario -
+                  # it is baked into a hub item, and disabling the whole
+                  # base-http-scenarios collection would drop ~40 useful
+                  # detections. Instead we keep the detection and drop its
+                  # decisions hourly; the LAN/home whitelists in
+                  # crowdsec-values.yaml handle the legit sources, so this
+                  # only ever hits real scanners (who are re-banned anyway).
+                  $LAPI_EXEC cscli decisions delete \
+                    --scenario LePresidente/http-generic-403-bf --all || true

crowdsec/k8s/namespace.yaml

@@ -0,0 +1,6 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: crowdsec
+  labels:
+    app.kubernetes.io/part-of: crowdsec

crowdsec/k8s/network-policy.yaml

@@ -0,0 +1,34 @@
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: crowdsec-lapi
+  namespace: crowdsec
+spec:
+  podSelector:
+    matchLabels:
+      k8s-app: crowdsec
+      type: lapi
+  policyTypes:
+    - Ingress
+  ingress:
+    - from:
+        - namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: traefik
+          podSelector:
+            matchLabels:
+              app.kubernetes.io/name: traefik
+        - podSelector:
+            matchLabels:
+              k8s-app: crowdsec
+              type: agent
+      ports:
+        - protocol: TCP
+          port: 8080
+    - from:
+        - namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: prometheus
+      ports:
+        - protocol: TCP
+          port: 6060

dockmon/compose.yaml

@@ -1,54 +1,46 @@
 services:
   dockmon:
-      image: darthnorse/dockmon:latest
+    image: darthnorse/dockmon:2.5.0
     container_name: dockmon
     restart: unless-stopped
-      ports:
+    # ports:
-         - 8000:443
+    #   - 8000:443
-      environment:
-         - TZ=Europe/Bratislava
     volumes:
-         - ./data:/app/data
+      - data:/app/data
       - /var/run/docker.sock:/var/run/docker.sock
     healthcheck:
-         test: [ "CMD", "curl", "-k", "-f", "https://localhost:443/health" ]
+      test: ["CMD", "curl", "-k", "-f", "https://localhost:443/health"]
       interval: 30s
       timeout: 10s
       retries: 3
-      networks:
-         - traefik-proxy
     labels:
       - "traefik.enable=true"
-         - "traefik.docker.network=traefik-proxy"
-
-         # Prod Router
-         - "traefik.http.routers.dockmon.rule=Host(`dockmon.forust.xyz`)"
-         - "traefik.http.routers.dockmon.entrypoints=websecure"
-         - "traefik.http.routers.dockmon.middlewares=security-chain@file"
-         - "traefik.http.routers.dockmon.service=dockmon"
-         - "traefik.http.routers.dockmon.tls=true"
       - "traefik.http.services.dockmon.loadbalancer.server.port=443"
       - "traefik.http.services.dockmon.loadbalancer.server.scheme=https"
       - "traefik.http.services.dockmon.loadbalancer.serverstransport=insecureTransport@file"
 
+      # Prod Router
+      - "traefik.http.routers.dockmon.rule=Host(`dockmon.forust.xyz`)"
+      - "traefik.http.routers.dockmon.entrypoints=websecure"
+      - "traefik.http.routers.dockmon.middlewares=security-headers@file"
+      - "traefik.http.routers.dockmon.tls.certresolver=letsencrypt"
       # Local Router
-         - "traefik.http.routers.dockmon-local.rule=Host(`dockmon.workstation.internal`) || Host(`dockmon.internal`)"
+      - "traefik.http.routers.dockmon-local.rule=Host(`dockmon.workstation.internal`)"
       - "traefik.http.routers.dockmon-local.entrypoints=websecure"
-         - "traefik.http.routers.dockmon-local.middlewares=security-headers@file"
-         - "traefik.http.routers.dockmon-local.service=dockmon"
       - "traefik.http.routers.dockmon-local.tls=true"
-
       # Dev Router
       - "traefik.http.routers.dockmon-dev.rule=Host(`dockmon.gigaforust.internal`)"
       - "traefik.http.routers.dockmon-dev.entrypoints=websecure"
-         - "traefik.http.routers.dockmon-dev.middlewares=security-chain@file"
-         - "traefik.http.routers.dockmon-dev.service=dockmon"
       - "traefik.http.routers.dockmon-dev.tls=true"
 
+      # Glance Metadata
       - glance.name=dockmon
       - glance.url=https://dockmon.forust.xyz/
       - glance.description=Dockmon is a lightweight Docker container monitoring and management tool with a user-friendly web interface.
-
+    networks:
+      - proxy
+volumes:
+  data:
 networks:
-   traefik-proxy:
+  proxy:
     external: true

dockmon/k8s/certificates.yaml

@@ -0,0 +1,28 @@
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: dockmon-prod-tls
+  namespace: dockmon
+spec:
+  secretName: dockmon-prod-tls
+  dnsNames:
+    - dockmon.forust.xyz
+  issuerRef:
+    name: letsencrypt-prod
+    kind: ClusterIssuer
+---
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: internal-wildcard-tls
+  namespace: dockmon
+spec:
+  secretName: internal-wildcard-tls
+  dnsNames:
+    - "*.workstation.internal"
+    - "*.gigaforust.internal"
+    - workstation.internal
+    - gigaforust.internal
+  issuerRef:
+    name: internal-ca
+    kind: ClusterIssuer

dockmon/k8s/dockmon.yaml

@@ -0,0 +1,68 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: dockmon-service
+  namespace: dockmon
+spec:
+  clusterIP: None
+  selector:
+    app: dockmon
+  ports:
+    - port: 443
+      targetPort: 443
+---
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  name: dockmon-statefulset
+  namespace: dockmon
+spec:
+  serviceName: dockmon-service
+  replicas: 1
+  selector:
+    matchLabels:
+      app: dockmon
+  template:
+    metadata:
+      labels:
+        app: dockmon
+    spec:
+      containers:
+        - name: dockmon
+          image: darthnorse/dockmon:2.5.0
+          ports:
+            - containerPort: 443
+          volumeMounts:
+            - name: data
+              mountPath: /app/data
+            - name: docker-sock
+              mountPath: /var/run/docker.sock
+          livenessProbe:
+            httpGet:
+              path: /health
+              port: 443
+              scheme: HTTPS
+            initialDelaySeconds: 30
+            periodSeconds: 30
+            timeoutSeconds: 10
+            failureThreshold: 3
+          resources:
+            requests:
+              memory: "512Mi"
+              cpu: "200m"
+            limits:
+              memory: "1.5Gi"
+              cpu: "700m "
+      volumes:
+        - name: docker-sock
+          hostPath:
+            path: /var/run/docker.sock
+            type: Socket
+  volumeClaimTemplates:
+    - metadata:
+        name: data
+      spec:
+        accessModes: ["ReadWriteOnce"]
+        resources:
+          requests:
+            storage: 1Gi

dockmon/k8s/ingress.yaml

@@ -0,0 +1,47 @@
+apiVersion: traefik.io/v1alpha1
+kind: ServersTransport
+metadata:
+  name: dockmon-transport
+  namespace: dockmon
+spec:
+  insecureSkipVerify: true
+---
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
+metadata:
+  name: dockmon-prod
+  namespace: dockmon
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`dockmon.forust.xyz`)
+      kind: Rule
+      middlewares:
+        - name: crowdsec-bouncer
+          namespace: crowdsec
+        - name: security-headers@file
+      services:
+        - name: dockmon-service
+          port: 443
+          serversTransport: dockmon-transport
+  tls:
+    secretName: dockmon-prod-tls
+---
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
+metadata:
+  name: dockmon-local
+  namespace: dockmon
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`dockmon.workstation.internal`) || Host(`dockmon.gigaforust.internal`)
+      kind: Rule
+      services:
+        - name: dockmon-service
+          port: 443
+          serversTransport: dockmon-transport
+  tls:
+    secretName: internal-wildcard-tls

dockmon/k8s/namespace.yaml

@@ -0,0 +1,4 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: dockmon

downtify/compose.yaml

@@ -0,0 +1,31 @@
+services:
+  downtify:
+    container_name: downtify
+    image: ghcr.io/henriquesebastiao/downtify:3.1.0
+    restart: unless-stopped
+    # ports:
+    #   - '7077:8000'
+    volumes:
+      - ./Downtify_downloads:/downloads
+    labels:
+      - "traefik.enable=true"
+      - "traefik.http.services.downtify.loadbalancer.server.port=8000"
+
+      # Prod Router
+      - "traefik.http.routers.downtify.rule=Host(`downtify.forust.xyz`)"
+      - "traefik.http.routers.downtify.entrypoints=websecure"
+      - "traefik.http.routers.downtify.middlewares=security-chain@file"
+      - "traefik.http.routers.downtify.tls.certresolver=letsencrypt"
+      # Local Router
+      - "traefik.http.routers.downtify-local.rule=Host(`downtify.workstation.internal`)"
+      - "traefik.http.routers.downtify-local.entrypoints=websecure"
+      - "traefik.http.routers.downtify-local.tls=true"
+      # Dev Router
+      - "traefik.http.routers.downtify-dev.rule=Host(`downtify.gigaforust.internal`)"
+      - "traefik.http.routers.downtify-dev.entrypoints=websecure"
+      - "traefik.http.routers.downtify-dev.tls=true"
+    networks:
+      - proxy
+networks:
+  proxy:
+    external: true

downtify/k8s/downtify.yaml

@@ -0,0 +1,58 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: downtify-service
+  namespace: downtify
+spec:
+  selector:
+    app: downtify
+  ports:
+    - port: 8000
+      targetPort: 8000
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: downtify-deployment
+  namespace: downtify
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: downtify
+  template:
+    metadata:
+      labels:
+        app: downtify
+    spec:
+      containers:
+        - name: downtify
+          image: ghcr.io/henriquesebastiao/downtify:3.1.0
+          ports:
+            - containerPort: 8000
+          volumeMounts:
+            - name: downloads
+              mountPath: /downloads
+          resources:
+            requests:
+              memory: "128Mi"
+              cpu: "200m"
+            limits:
+              memory: "1Gi"
+              cpu: "1"
+      volumes:
+        - name: downloads
+          persistentVolumeClaim:
+            claimName: downtify-downloads-pvc
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: downtify-downloads-pvc
+  namespace: downtify
+spec:
+  accessModes:
+    - ReadWriteOnce
+  resources:
+    requests:
+      storage: 10Gi

downtify/k8s/ingress.yaml

@@ -0,0 +1,37 @@
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
+metadata:
+  name: downtify-prod
+  namespace: downtify
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`downtify.forust.xyz`)
+      kind: Rule
+      middlewares:
+        - name: crowdsec-bouncer
+          namespace: crowdsec
+        - name: security-chain@file
+      services:
+        - name: downtify-service
+          port: 8000
+  tls:
+    secretName: downtify-prod-tls
+---
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
+metadata:
+  name: downtify-local
+  namespace: downtify
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`downtify.workstation.internal`) || Host(`downtify.gigaforust.internal`)
+      kind: Rule
+      services:
+        - name: downtify-service
+          port: 8000
+  tls:
+    secretName: internal-wildcard-tls

downtify/k8s/namespace.yaml

@@ -0,0 +1,4 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: downtify

dtek_notif/docker-compose.yaml

@@ -3,10 +3,11 @@ services:
     build:
       context: .
       dockerfile: Dockerfile
+    image: gcr.forust.xyz/forust/dtek-notif:latest
+    pull_policy: build
     restart: unless-stopped
     environment:
       - TZ=Europe/Kyiv
-    
     dns:
       - 1.1.1.1
       - 8.8.8.8

edu_master/.env.example

@@ -9,5 +9,6 @@ WEBINAR_CHECK_INTERVAL=60
 REDIS_HOST=redis
 REDIS_PORT=6379
 PLAYWRIGHT_WS=ws://playwright-service:3000/ws
+TZ=Europe/Kyiv
 WEBINAR_TELEGRAM_TOKEN=your_telegram_bot_token_here
 WEBINAR_ADMIN_ID=123456789

edu_master/PLAYWRIGHT_VERSION

@@ -0,0 +1 @@
+1.56.0

edu_master/compose.yaml

@@ -1,11 +1,11 @@
 services:
   redis:
-    image: redis:alpine
+    image: redis:8.10.2-alpine
     restart: unless-stopped
     volumes:
       - redis-data:/data
     healthcheck:
-      test: [ "CMD", "redis-cli", "ping" ]
+      test: ["CMD", "redis-cli", "ping"]
       interval: 5s
       timeout: 3s
       retries: 5
@@ -17,13 +17,15 @@ services:
 
   session-keeper:
     build: ./phpsessid-bot
+    image: gcr.forust.xyz/forust/session-keeper:latest
+    pull_policy: build
     env_file: .env
     restart: unless-stopped
     depends_on:
       redis:
         condition: service_healthy
     healthcheck:
-      test: [ "CMD-SHELL", "redis-cli -h redis EXISTS EDU_PHPSESSID | grep -q 1" ]
+      test: ["CMD-SHELL", "redis-cli -h redis EXISTS EDU_PHPSESSID | grep -q 1"]
       interval: 30s
       timeout: 5s
       retries: 10
@@ -31,6 +33,8 @@ services:
 
   webinar-checker:
     build: ./webinar-checker
+    image: gcr.forust.xyz/forust/webinar-checker:latest
+    pull_policy: build
     env_file: .env
     restart: unless-stopped
     depends_on:

edu_master/k8s/alerts.yaml

@@ -0,0 +1,96 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: edu-master-webinar
+  namespace: edu-master
+  labels:
+    release: prometheus-stack
+spec:
+  groups:
+    - name: edu_master.webinar
+      rules:
+        # No successful webinar check for 5m (~2-3 missed 2-min checks).
+        # Catches: playwright hangs/timeouts, version skew, site changes, hung job.
+        # The last_success > 0 guard is mandatory: checker.py initialises
+        # last_success to 0, so without it `time() - 0` equals the current epoch
+        # and humanizeDuration renders ~20722d on every pod restart. Keep the
+        # duration expression on the left so $value stays the real gap.
+        - alert: WebinarCheckerNoSuccessfulCheck
+          expr: |
+            ((time() - webinar_check_last_success_timestamp_seconds) > 300)
+              and (webinar_check_last_success_timestamp_seconds > 0)
+              and (webinar_check_last_run_timestamp_seconds > 0)
+          for: 2m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Webinar checker has no successful check for 5m"
+            description: "edu-master/webinar-checker: last successful webinar check was {{ $value | humanizeDuration }} ago. Checks are failing or hanging (see consecutive failures alert). Notifications about new webinars are NOT being sent."
+
+        # Checks are running but none has ever succeeded since pod start.
+        # Split out from the rule above so a zeroed gauge never feeds
+        # humanizeDuration.
+        - alert: WebinarCheckerNeverSucceeded
+          expr: |
+            (webinar_check_last_success_timestamp_seconds == 0)
+              and (webinar_check_last_run_timestamp_seconds > 0)
+          for: 10m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Webinar checker has never completed a successful check"
+            description: 'edu-master/webinar-checker: checks have been running for 10m but not one has ever succeeded since the pod started, so every check is failing. Check pod logs (Loki: {namespace="edu-master", container="webinar-checker"}).'
+
+        # Fast path: 3 consecutive failures (~6+ min at 2-min interval).
+        - alert: WebinarCheckerConsecutiveFailures
+          expr: |
+            webinar_check_consecutive_failures >= 3
+          for: 5m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Webinar checker failing consecutively"
+            description: 'edu-master/webinar-checker: {{ $value }} consecutive webinar check failures (timeout / playwright error / page error). Check pod logs (Loki: {namespace="edu-master", container="webinar-checker"}).'
+
+        # Metrics endpoint not scraped for 10m: pod down, metrics server dead, or ServiceMonitor broken.
+        - alert: WebinarCheckerScrapeDown
+          expr: |
+            absent(webinar_check_last_run_timestamp_seconds) == 1
+          for: 10m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Webinar checker metrics missing"
+            description: "edu-master/webinar-checker: no metrics series for 10m. Pod may be down, metrics server dead, or ServiceMonitor/Service broken. Webinar checks are unobserved."
+
+        # EDU session lost: session-keeper down or credentials expired. Without PHPSESSID every check is skipped.
+        - alert: EduPhpsessidMissing
+          expr: |
+            edu_phpsessid_present == 0
+          for: 10m
+          labels:
+            severity: critical
+          annotations:
+            summary: "EDU_PHPSESSID missing"
+            description: "edu-master: EDU_PHPSESSID absent from redis for 10m. Webinar/diari/schedule checks are all skipped. Check session-keeper logs and EDU credentials."
+
+        # Hard deps: checker and playwright deployments unavailable.
+        - alert: WebinarCheckerDeploymentDown
+          expr: |
+            kube_deployment_status_replicas_unavailable{deployment="webinar-checker", namespace="edu-master"} > 0
+          for: 10m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Webinar checker deployment unavailable"
+            description: "edu-master/webinar-checker deployment has {{ $value }} unavailable replica(s) for 10m."
+
+        - alert: PlaywrightServiceDown
+          expr: |
+            kube_deployment_status_replicas_unavailable{deployment="playwright-service", namespace="edu-master"} > 0
+          for: 10m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Playwright service unavailable"
+            description: "edu-master/playwright-service deployment has {{ $value }} unavailable replica(s) for 10m. All webinar/diari/schedule checks fail without it."

edu_master/k8s/namespace.yaml

@@ -0,0 +1,4 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: edu-master

edu_master/k8s/playwright.yaml

@@ -0,0 +1,58 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: playwright-service
+  namespace: edu-master
+  labels:
+    app: edu-master-playwright
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: edu-master-playwright
+  template:
+    metadata:
+      labels:
+        app: edu-master-playwright
+    spec:
+      containers:
+        - name: playwright
+          # renovate: datasource=docker depName=mcr.microsoft.com/playwright versioning=docker
+          image: mcr.microsoft.com/playwright:v1.56.0-jammy
+          imagePullPolicy: IfNotPresent
+          command:
+            - npx
+            - -y
+            - playwright@1.56.0
+            - run-server
+            - --port
+            - "3000"
+            - --path
+            - /ws
+          ports:
+            - containerPort: 3000
+          readinessProbe:
+            tcpSocket:
+              port: 3000
+            initialDelaySeconds: 5
+            periodSeconds: 10
+            timeoutSeconds: 3
+          livenessProbe:
+            tcpSocket:
+              port: 3000
+            initialDelaySeconds: 15
+            periodSeconds: 20
+            timeoutSeconds: 3
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: playwright-service
+  namespace: edu-master
+spec:
+  selector:
+    app: edu-master-playwright
+  ports:
+    - name: ws
+      port: 3000
+      targetPort: 3000

edu_master/k8s/redis.yaml

@@ -0,0 +1,75 @@
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  name: redis
+  namespace: edu-master
+  labels:
+    app: edu-master-redis
+spec:
+  serviceName: redis
+  replicas: 1
+  selector:
+    matchLabels:
+      app: edu-master-redis
+  template:
+    metadata:
+      labels:
+        app: edu-master-redis
+    spec:
+      containers:
+        - name: redis
+          image: redis:8.10.2-alpine
+          imagePullPolicy: IfNotPresent
+          ports:
+            - containerPort: 6379
+          volumeMounts:
+            - name: redis-data
+              mountPath: /data
+          resources:
+            requests:
+              cpu: 25m
+              memory: 64Mi
+            limits:
+              cpu: 250m
+              memory: 256Mi
+          readinessProbe:
+            exec:
+              command: ["redis-cli", "ping"]
+            initialDelaySeconds: 5
+            periodSeconds: 5
+            timeoutSeconds: 3
+          livenessProbe:
+            exec:
+              command: ["redis-cli", "ping"]
+            initialDelaySeconds: 10
+            periodSeconds: 10
+            timeoutSeconds: 3
+      volumes:
+        - name: redis-data
+          persistentVolumeClaim:
+            claimName: redis-data-pvc
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: redis-data-pvc
+  namespace: edu-master
+spec:
+  accessModes:
+    - ReadWriteOnce
+  resources:
+    requests:
+      storage: 1Gi
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: redis
+  namespace: edu-master
+spec:
+  selector:
+    app: edu-master-redis
+  ports:
+    - name: redis
+      port: 6379
+      targetPort: 6379

edu_master/k8s/restore-seed-job.yaml.example

@@ -0,0 +1,50 @@
+# One-time Job to migrate redis state from docker compose to k8s (maintenance window).
+# The .example file is not applied by the deploy pipeline (mask *.example.yaml).
+#
+# Runbook:
+#   1. docker compose -f <repo>/edu_master/compose.yaml stop        # SIGTERM -> redis will flush dump.rdb
+#   2. docker run --rm -v edu_master_redis-data:/data \
+#        -v /tmp/edu-master-backup:/backup \
+#        redis:alpine sh -c "cp /data/dump.rdb /backup/ && ls -la /backup"
+#   3. kubectl apply -f edu_master/k8s/namespace.yaml
+#   4. kubectl apply -f <only the PVC from redis.yaml>              # seed must come BEFORE redis pod starts
+#   5. kubectl apply -f edu_master/k8s/restore-seed-job.yaml.example
+#      kubectl wait --for=condition=complete job/redis-restore-seed -n edu-master --timeout=120s
+#   6. kubectl delete job redis-restore-seed -n edu-master
+#   7. kubectl apply -f edu_master/k8s/ -R                           # apply remaining manifests
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: redis-restore-seed
+  namespace: edu-master
+spec:
+  backoffLimit: 2
+  ttlSecondsAfterFinished: 3600
+  template:
+    spec:
+      restartPolicy: Never
+      containers:
+        - name: seed
+          image: redis:alpine
+          command:
+            - /bin/sh
+            - -ec
+            - |
+              ls -la /backup
+              cp /backup/dump.rdb /data/dump.rdb
+              chmod 644 /data/dump.rdb
+              ls -la /data
+          volumeMounts:
+            - name: redis-data
+              mountPath: /data
+            - name: backup
+              mountPath: /backup
+              readOnly: true
+      volumes:
+        - name: redis-data
+          persistentVolumeClaim:
+            claimName: redis-data-pvc
+        - name: backup
+          hostPath:
+            path: /tmp/edu-master-backup
+            type: DirectoryOrCreate

edu_master/k8s/secrets.yaml.example

@@ -0,0 +1,29 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: edu-master-secrets
+  namespace: edu-master
+type: Opaque
+stringData:
+  # Session keeper credentials
+  KEEPER_LOGIN: ""
+  KEEPER_PASSWORD: ""
+  KEEPER_INTERVAL: "10"
+  # EDU links
+  EDU_URL_BASE: "https://edu.edu.vn.ua"
+  EDU_URL_LOGIN: "/user/login"
+  EDU_URL_COURSES: "/course/userlist"
+  EDU_URL_WEBINAR: "/webinar/useractive"
+  # Playwright
+  USER_AGENT: ""
+  PLAYWRIGHT_WS: "ws://playwright-service:3000/ws"
+  # Webinar-checker
+  WEBINAR_TELEGRAM_TOKEN: ""
+  WEBINAR_ADMIN_ID: ""
+  WEBINAR_CHECK_INTERVAL: "60"
+  # Prometheus metrics endpoint (scraped via ServiceMonitor, alerts in k8s/alerts.yaml)
+  METRICS_PORT: "8000"
+  # Database
+  REDIS_HOST: "redis"
+  REDIS_PORT: "6379"
+  TZ: "Europe/Kyiv"

edu_master/k8s/service.yaml

@@ -0,0 +1,15 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: webinar-checker
+  namespace: edu-master
+  labels:
+    app: edu-master-webinar-checker
+spec:
+  selector:
+    app: edu-master-webinar-checker
+  ports:
+    - name: metrics
+      port: 8000
+      targetPort: metrics
+      protocol: TCP

edu_master/k8s/servicemonitor.yaml

@@ -0,0 +1,16 @@
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: webinar-checker
+  namespace: edu-master
+  labels:
+    release: prometheus-stack
+spec:
+  selector:
+    matchLabels:
+      app: edu-master-webinar-checker
+  endpoints:
+    - port: metrics
+      path: /metrics
+      interval: 30s
+      scrapeTimeout: 10s

edu_master/k8s/session-keeper.yaml

@@ -0,0 +1,51 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: session-keeper
+  namespace: edu-master
+  labels:
+    app: edu-master-session-keeper
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: edu-master-session-keeper
+  template:
+    metadata:
+      labels:
+        app: edu-master-session-keeper
+    spec:
+      initContainers:
+        - name: wait-redis
+          image: redis:8.10.2-alpine
+          command:
+            - /bin/sh
+            - -ec
+            - |
+              i=0
+              until redis-cli -h redis ping | grep -q PONG; do
+                i=$((i+1))
+                [ "$i" -ge 300 ] && echo "TIMEOUT: redis not ready" && exit 1
+                sleep 2
+              done
+              echo "redis is ready"
+      containers:
+        - name: session-keeper
+          image: gcr.forust.xyz/forust/session-keeper:prod
+          envFrom:
+            - secretRef:
+                name: edu-master-secrets
+          resources:
+            requests:
+              cpu: 25m
+              memory: 96Mi
+            limits:
+              cpu: 250m
+              memory: 256Mi
+          readinessProbe:
+            exec:
+              command: ["/bin/sh", "-ec", "redis-cli -h redis EXISTS EDU_PHPSESSID | grep -q 1"]
+            initialDelaySeconds: 15
+            periodSeconds: 30
+            timeoutSeconds: 5
+            failureThreshold: 10

edu_master/k8s/webinar-checker.yaml

@@ -0,0 +1,73 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: webinar-checker
+  namespace: edu-master
+  labels:
+    app: edu-master-webinar-checker
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: edu-master-webinar-checker
+  template:
+    metadata:
+      labels:
+        app: edu-master-webinar-checker
+    spec:
+      # Enforces dependency order like compose depends_on:
+      # redis healthy -> session-keeper healthy (EXISTS EDU_PHPSESSID) -> playwright started
+      initContainers:
+        - name: wait-deps
+          image: redis:8.10.2-alpine
+          command:
+            - /bin/sh
+            - -ec
+            - |
+              i=0
+              until redis-cli -h redis ping | grep -q PONG; do
+                i=$((i+1))
+                [ "$i" -ge 300 ] && echo "TIMEOUT: redis not ready" && exit 1
+                sleep 2
+              done
+              echo "redis ok"
+              until [ "$(redis-cli -h redis EXISTS EDU_PHPSESSID)" = "1" ]; do
+                i=$((i+1))
+                [ "$i" -ge 300 ] && echo "TIMEOUT: no PHPSESSID (session-keeper down?)" && exit 1
+                sleep 2
+              done
+              echo "PHPSESSID ok"
+              until nc -z playwright-service 3000; do
+                i=$((i+1))
+                [ "$i" -ge 300 ] && echo "TIMEOUT: playwright-service not reachable" && exit 1
+                sleep 2
+              done
+              echo "playwright ok"
+      containers:
+        - name: webinar-checker
+          image: gcr.forust.xyz/forust/webinar-checker:prod
+          ports:
+            - name: metrics
+              containerPort: 8000
+              protocol: TCP
+          readinessProbe:
+            httpGet:
+              path: /health
+              port: metrics
+            periodSeconds: 10
+            timeoutSeconds: 3
+            failureThreshold: 12
+            initialDelaySeconds: 10
+          envFrom:
+            - secretRef:
+                name: edu-master-secrets
+          env:
+            - name: TZ
+              value: "Europe/Kyiv"
+          resources:
+            requests:
+              cpu: "50m"
+              memory: "128Mi"
+            limits:
+              cpu: "600m"
+              memory: "512Mi"