Use plain text for the deploy request details

Reviewed-on: #98

.docs/storage-audit-instruction.md

@@ -0,0 +1,191 @@
+# Инструкция: Анализ хранилища Kubernetes и настройка NFS
+
+## Цель
+Проанализировать текущую конфигурацию хранилища Kubernetes и подготовить план внедрения NFS StorageClass для сохранения данных при удалении namespace.
+
+## 1. Собрать информацию о кластере
+
+### 1.1. Версия Kubernetes и тип дистрибутива
+```bash
+kubectl version --short
+# или
+kubectl version
+```
+
+Определить, используется ли k3s, k8s, microk8s и т.д.:
+```bash
+# Проверить наличие k3s
+which k3s
+# Проверить процесс
+ps aux | grep -E 'kube|k3s'
+```
+
+### 1.2. StorageClass
+```bash
+kubectl get storageclass -o wide
+```
+
+Запомнить:
+- `PROVISIONER` — какой драйвер используется
+- `RECLAIMPOLICY` — Delete или Retain
+- Какой StorageClass помечен как `(default)`
+
+### 1.3. Существующие PV и PVC
+```bash
+kubectl get pv -o wide
+kubectl get pvc --all-namespaces
+```
+
+Посмотреть, какие PVC привязаны к каким PV, и какой reclaimPolicy у PV.
+
+### 1.4. Нода и диски
+```bash
+# Список нод
+kubectl get nodes -o wide
+
+# На каждой ноде (через ssh или локально):
+lsblk
+df -h
+cat /etc/fstab
+```
+
+Определить:
+- Есть ли отдельный раздел/диск для данных
+- Куда смонтированы разделы
+- Сколько свободного места
+- Есть ли монтирование NTFS-разделов (как `/media/forust/Programs`)
+
+### 1.5. Где local-path хранит данные (для k3s)
+```bash
+ls -la /var/lib/rancher/k3s/storage/ 2>/dev/null
+# или для microk8s
+ls -la /var/snap/microk8s/common/ 2>/dev/null
+```
+
+## 2. Анализ: сохраняются ли данные при удалении namespace?
+
+| Сценарий | Результат |
+|---|---|
+| `kubectl delete ns <ns>` | Все PVC в namespace удаляются |
+| PVC → PV c `reclaimPolicy: Delete` | PV и данные удалены |
+| PVC → PV c `reclaimPolicy: Retain` | PV остаётся (статус Released), данные целы |
+
+**Вывод:** Если reclaimPolicy в StorageClass = `Delete`, то данные **пропадут**. Если `Retain` — сохранятся.
+
+## 3. План внедрения NFS
+
+### 3.1. Проверить, установлен ли NFS
+```bash
+which nfsstat exportfs mount.nfs
+systemctl status nfs-server 2>/dev/null || systemctl status nfs-kernel-server 2>/dev/null
+```
+
+### 3.2. Выбрать директорию для NFS-экспорта
+
+Варианты (выбрать подходящий):
+- `/var/lib/k8s-nfs/` — на корневом разделе
+- `<путь к отдельному разделу>/k8s-nfs/` — если есть отдельный диск/раздел
+- Не рекомендуется использовать NTFS-раздел (проблемы с правами и производительностью)
+
+Требования:
+- Файловая система: ext4 или xfs (не ntfs!)
+- Достаточно свободного места
+- Права: `755`, владелец root
+
+### 3.3. Установить NFS-сервер
+
+```bash
+# Debian/Ubuntu
+apt update && apt install -y nfs-kernel-server
+
+# RHEL/Fedora
+dnf install -y nfs-utils
+```
+
+### 3.4. Настроить экспорт
+
+Создать директорию:
+```bash
+mkdir -p /var/lib/k8s-nfs
+chmod 755 /var/lib/k8s-nfs
+```
+
+Добавить в `/etc/exports`:
+```
+/var/lib/k8s-nfs *(rw,sync,no_subtree_check,no_root_squash)
+```
+
+Применить:
+```bash
+exportfs -rav
+```
+
+Проверить:
+```bash
+showmount -e localhost
+```
+
+### 3.5. Выбрать способ интеграции с Kubernetes
+
+#### Вариант A: nfs-subdir-external-provisioner (проще)
+```bash
+helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/
+helm install nfs-provisioner nfs-subdir-external-provisioner/nfs-subdir-external-provisioner \
+  --namespace kube-system \
+  --set nfs.server=127.0.0.1 \
+  --set nfs.path=/var/lib/k8s-nfs \
+  --set storageClass.name=nfs \
+  --set storageClass.defaultClass=false \
+  --set storageClass.reclaimPolicy=Retain
+```
+
+#### Вариант B: NFS CSI Driver
+```bash
+helm repo add csi-driver-nfs https://raw.githubusercontent.com/kubernetes-csi/csi-driver-nfs/master/charts
+helm install csi-driver-nfs csi-driver-nfs/csi-driver-nfs --namespace kube-system
+```
+
+После установки CSI драйвера создать StorageClass:
+```yaml
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata:
+  name: nfs
+provisioner: nfs.csi.k8s.io
+parameters:
+  server: 127.0.0.1
+  share: /var/lib/k8s-nfs
+reclaimPolicy: Retain
+volumeBindingMode: Immediate
+```
+
+### 3.6. Проверить результат
+```bash
+kubectl get storageclass
+kubectl get pods -n kube-system | grep -E 'nfs|provisioner'
+```
+
+## 4. Итоговая конфигурация
+
+После внедрения в кластере будет два StorageClass:
+
+| Имя | Provisioner | ReclaimPolicy | Назначение |
+|---|---|---|---|
+| `local-path` (default) | rancher.io/local-path | Delete | Временные данные, stateless |
+| `nfs` | nfs-subdir-external-provisioner или nfs.csi.k8s.io | Retain | Данные, которые нужно сохранять |
+
+**Главное преимущество:** PVC c `storageClassName: nfs` при удалении namespace сохраняют данные на диске, так как NFS-провизор использует `reclaimPolicy: Retain` или файлы физически остаются в NFS-экспорте.
+
+## 5. Ответы на частые вопросы
+
+**В:** Не упадёт ли local-path при установке NFS?
+**О:** Нет, они независимы. local-path продолжает работать как обычно.
+
+**В:** Данные NFS и local-path будут на одном диске?
+**О:** Да, можно настроить оба на одном разделе, в разных каталогах.
+
+**В:** Что если у меня несколько нод?
+**О:** NFS сервер нужно поднять на одной ноде, а с других нод должна быть доступна шари. Для multi-node лучше использовать отдельный сервер или distributed storage (Longhorn, Rook/Ceph).
+
+**В:** Можно ли использовать существующий NTFS-раздел для NFS?
+**О:** Не рекомендуется — NTFS не поддерживает права Linux (no_root_squash не сработает корректно), возможны проблемы с блокировками и производительностью.

.editorconfig

@@ -0,0 +1,24 @@
+root = true
+
+[*]
+indent_style = space
+indent_size = 2
+end_of_line = lf
+charset = utf-8
+trim_trailing_whitespace = true
+insert_final_newline = true
+
+[*.{yml,yaml}]
+indent_size = 2
+
+[*.{json,jsonc}]
+indent_size = 2
+
+[*.md]
+trim_trailing_whitespace = false
+
+[*.py]
+indent_size = 4
+
+[{Makefile,makefile}]
+indent_style = tab

.gitea/actionlint.yaml

@@ -0,0 +1,10 @@
+# actionlint configuration. Passed explicitly from the ci workflow:
+#   actionlint -config-file .gitea/actionlint.yaml .gitea/workflows/*.yaml
+#
+# The self-hosted act_runner registers custom labels that actionlint cannot know
+# about, so declare them here instead of silencing the whole runner-label check.
+self-hosted-runner:
+  labels:
+    - arch
+    - homelab
+    - prod

.gitea/deploy-dependencies.json

@@ -0,0 +1,3 @@
+{
+  "postgres": ["authentik", "gitea", "immich", "n8n", "netbox", "netronome"]
+}

.gitea/runner/README.md

@@ -0,0 +1,131 @@
+# Homelab CI/CD
+
+The native Gitea runner runs on **vps**; production runs on **workstation**.
+Compose, workflow, shell, Python, formatting, YAML, Dockerfile and Kubernetes
+checks appear as separate jobs. Jobs run on `homelab:host`, one at a time; the
+build waits for every check to pass. CI and deploy runs also show a summary with
+the release SHA, image build or reuse results, deploy mode, selected services,
+and image digests. Failed runs keep a summary of completed image builds, stage
+results, apply results, and recorded Kubernetes recovery. The final deploy
+summary is in the smoke job; earlier jobs show the state observed at that time.
+Apply success is separate from health and recovery. Update the installed
+workstation controller with `setup-workstation.sh` when no deploy is running.
+No job images or Kubernetes credentials
+are needed on the VPS. Builds use one pinned BuildKit helper container. CI and deploy are separate workflows.
+
+## Runner installation
+
+Install Docker Engine with Compose and Buildx, Git, Python 3.11+, Bash, curl,
+GNU tar/xz, flock and systemd using the host's package manager. Keep the existing
+Gitea runner 3.0.2 binary at `/usr/local/bin/gitea-runner`.
+
+From this checkout on the VPS:
+
+```sh
+sudo bash .gitea/runner/setup-runner.sh
+```
+
+The installer reuses `/var/lib/gitea-runner/.runner` and the existing service.
+For a new host, install the same runner binary and register as `gitea-runner`
+using the registration token interactively, label `homelab:host`, and working
+directory `/var/lib/gitea-runner`; then rerun the installer. Tokens never belong
+in this repository or command-line examples.
+
+Pinned tools live in the runner user's `~/.cache/homelab-ci`; CI repairs version
+drift there. Installations are locked. Buildx uses only the `homelab-ci` builder,
+pushes directly to the registry, and caps retained local cache at 1 GiB with a
+2 GiB free-space target. This is not a hard limit on peak build disk usage.
+Nothing runs `docker system prune`, removes unrelated images, or deletes volumes.
+
+## Workstation setup
+
+As the existing SSH deploy user on workstation:
+
+```sh
+sudo loginctl enable-linger forust
+bash .gitea/runner/setup-workstation.sh
+```
+
+The controller uses `/srv/homelab` as the persistent configuration tree and makes
+a detached source worktree for each SHA. It never resets `/srv/homelab`, moves
+local configuration, renames Compose projects, or changes volume names.
+The installer records the current Kubernetes context and cluster UID in
+`~/.config/homelab-deploy/environment`. Check these before installing.
+
+Configure Gitea Actions Variables:
+
+- `DEPLOY_HOST`, `DEPLOY_USER`, `DEPLOY_PORT`: the existing VPS-to-workstation SSH endpoint.
+- `DEPLOY_KNOWN_HOSTS`: workstation's verified SSH host key entry for that endpoint.
+- `AUTODEPLOY`: `false` initially; `true` enables deployment after successful main CI.
+
+Keep `DEPLOY_SSH_KEY`, `REGISTRY_USERNAME` and `REGISTRY_PASSWORD` in Actions
+Secrets. Legacy endpoint secrets remain accepted during migration. The Actions
+token must have repository read and Actions read access for release downloads.
+The deploy user's existing Docker registry authentication remains necessary.
+
+## Releases and deployment
+
+CI publishes `release-<full SHA>` as a Gitea artifact with all three owned image
+digests and build input fingerprints. Unchanged images are reused only from a
+successful main CI artifact, never from `:prod`. EDU images remain pinned to the
+digests released by their application repository. Expired artifacts cause CI to
+rebuild images; they block deployment until CI is rerun.
+
+Run deploy from main with `deploy_ref=main` or a checked SHA:
+
+- `full`: required for the first baseline; reconcile all active components.
+- `changed`: compare with the last fully successful production deploy.
+- `plan`: validate configuration and show selection without changing production resources.
+- `refresh_images=true`: explicitly refresh mutable third-party Compose tags.
+
+The manual and automatic paths both require successful CI, a successful build
+job and the exact SHA's release artifact. PRs cannot publish images or deploy.
+Removed resources are reported and require explicit removal; no automatic prune.
+Service dependencies are listed in `.gitea/deploy-dependencies.json`.
+
+A workstation user systemd service holds the deploy lock across validation,
+sequential apply, verification and smoke checks. SSH clients only submit/follow:
+disconnecting or cancelling the Actions client does not kill production apply.
+Retrying the same run ID does not start another apply. `ExecStopPost` recovers
+interrupted runs before the unit finishes. Kubernetes rolls back to captured
+revisions; configuration and persistent data are not reverted.
+
+## Status and recovery
+
+`--retry` repeats failed verification and smoke checks, never apply. Recovery
+keeps a failed deploy out of the successful baseline, even after rollback.
+
+On workstation (replace the numeric ID with Actions run ID and attempt):
+
+```sh
+python3 ~/.local/lib/homelab-deploy/controller.py status 123-1
+python3 ~/.local/lib/homelab-deploy/controller.py recover 123-1 --retry
+journalctl --user -u homelab-deploy@123-1
+```
+
+Runs live in `~/.local/state/homelab-deploy/runs`. Compose stores resolved configs
+with restricted permissions; these may contain credentials and must never be
+uploaded as CI artifacts. Stage logs print the exact manual recovery command
+using `compose-before/<stack>.json`, the original project directory and project
+name. Compose does not automatically roll back, and Nextcloud AIO's child
+containers remain managed by AIO. Preserve its own backups for data recovery.
+
+The controller retains twenty successful/planned runs and preserves failures.
+Update the workstation dispatcher only when no deploy is running.
+
+## Validation and migration rollback
+
+```sh
+python3 -m unittest discover -s tests -v
+bash .gitea/tests/deploy-validation.sh
+```
+
+Test on a separate namespace before the initial production `full` run. Check a
+failed rollout, interrupted SSH and repeated run ID, and verify that an isolated
+service change does not upgrade unrelated Helm releases or Compose stacks.
+
+To roll back the migration, disable autodeploy and finish or recover the remote
+run first. Restore the runner config/unit from `.before-<timestamp>` backups,
+reload systemd and restart the runner. Restore the prior workflows from Git.
+Production data and persistent volumes stay where they were. Do not remove run
+state or Compose recovery files until recovery is confirmed.

.gitea/runner/buildkitd.toml

@@ -0,0 +1,11 @@
+[worker.oci]
+  gc = true
+  reservedSpace = "256MB"
+  maxUsedSpace = "1GB"
+  minFreeSpace = "2GB"
+
+[[worker.oci.gcpolicy]]
+  reservedSpace = "256MB"
+  maxUsedSpace = "1GB"
+  minFreeSpace = "2GB"
+  all = true

.gitea/runner/config.yaml

@@ -0,0 +1,10 @@
+runner:
+  file: /var/lib/gitea-runner/.runner
+  capacity: 1
+  timeout: 5h
+  labels:
+    - homelab:host
+cache:
+  enabled: false
+container:
+  docker_host: unix:///var/run/docker.sock

.gitea/runner/gitea-runner.service

@@ -0,0 +1,18 @@
+[Unit]
+Description=Gitea Actions runner
+After=network-online.target docker.service
+Wants=network-online.target
+
+[Service]
+User=gitea-runner
+Group=gitea-runner
+SupplementaryGroups=docker
+WorkingDirectory=/var/lib/gitea-runner
+Environment=PATH=/var/lib/gitea-runner/.cache/homelab-ci/bin:/usr/local/bin:/usr/bin:/bin
+ExecStart=/usr/local/bin/gitea-runner daemon --config /etc/gitea-runner/config.yaml
+Restart=on-failure
+RestartSec=5
+UMask=0077
+
+[Install]
+WantedBy=multi-user.target

.gitea/runner/homelab-deploy@.service

@@ -0,0 +1,12 @@
+[Unit]
+Description=Homelab deploy %i
+
+[Service]
+Type=exec
+EnvironmentFile=%h/.config/homelab-deploy/environment
+ExecStart=/usr/bin/python3 %h/.local/lib/homelab-deploy/controller.py execute %i
+ExecStopPost=/usr/bin/python3 %h/.local/lib/homelab-deploy/controller.py recover %i
+RuntimeMaxSec=5h
+TimeoutStopSec=135min
+KillMode=control-group
+UMask=0077

.gitea/runner/setup-runner.sh

@@ -0,0 +1,48 @@
+#!/usr/bin/env bash
+# Native host runner, with pinned user-space tools and no extra CI images.
+set -euo pipefail
+here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+[ "$(id -u)" -eq 0 ] || { echo 'Run with sudo on the runner host' >&2; exit 1; }
+for tool in docker curl python3 git tar xz flock runuser systemctl; do
+  command -v "$tool" >/dev/null || { echo "Install missing prerequisite: $tool" >&2; exit 1; }
+done
+docker info >/dev/null
+docker compose version >/dev/null
+docker buildx version >/dev/null
+id gitea-runner >/dev/null 2>&1 || useradd --system --create-home --home-dir /var/lib/gitea-runner --shell /usr/bin/bash gitea-runner
+# Reuse the established service account and runner registration.
+runner_home="$(getent passwd gitea-runner | cut -d: -f6)"
+[ "$runner_home" = /var/lib/gitea-runner ] || { echo 'Unexpected runner home; inspect the existing service first' >&2; exit 1; }
+runuser -u gitea-runner -- docker info >/dev/null || { echo "The runner user needs access to Docker before setup" >&2; exit 1; }
+command -v gitea-runner >/dev/null || { echo 'Install gitea-runner 3.0.2 at /usr/local/bin/gitea-runner first' >&2; exit 1; }
+mkdir -p /etc/gitea-runner
+stamp="$(date -u +%Y%m%dT%H%M%SZ)"
+for existing in /etc/gitea-runner/config.yaml /etc/systemd/system/gitea-runner.service; do
+  [ ! -f "$existing" ] || cp -p "$existing" "$existing.before-$stamp"
+done
+scratch="$(mktemp -d)"
+trap 'rm -rf "$scratch"' EXIT
+chmod 755 "$scratch"
+install -m 0644 "$here/../workflows/install-ci-tools.sh" "$here/../workflows/tool-versions.env" "$scratch/"
+runuser -u gitea-runner -- bash "$scratch/install-ci-tools.sh"
+install -m 0644 "$here/config.yaml" /etc/gitea-runner/config.yaml
+python3 - <<'PYLABELS'
+import json
+from pathlib import Path
+registration = Path('/var/lib/gitea-runner/.runner')
+if registration.exists():
+    labels = json.loads(registration.read_text()).get('labels', [])
+    labels = [label for label in labels if isinstance(label, str) and label.split(':')[0] != 'homelab']
+    labels.append('homelab:host')
+    config = Path('/etc/gitea-runner/config.yaml')
+    config.write_text(config.read_text().replace('    - homelab:host', '\n'.join('    - ' + json.dumps(label) for label in labels)))
+PYLABELS
+install -m 0644 "$here/gitea-runner.service" /etc/systemd/system/gitea-runner.service
+if [ ! -f /var/lib/gitea-runner/.runner ]; then
+  echo 'Register once as gitea-runner with homelab:host before starting the service.'
+  exit 0
+fi
+systemctl daemon-reload
+systemctl enable --now gitea-runner.service
+systemctl restart gitea-runner.service
+echo "Runner ready. Configuration backups: *.before-$stamp"

.gitea/runner/setup-workstation.sh

@@ -0,0 +1,32 @@
+#!/usr/bin/env bash
+# Run as the existing deploy user on workstation. Never resets the working tree.
+set -euo pipefail
+here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+repo="${HOMELAB_REPO:-/srv/homelab}"
+for tool in python3 git kubectl helm docker flock timeout; do
+  command -v "$tool" >/dev/null || { echo "Install missing dependency: $tool" >&2; exit 1; }
+done
+[ -d "$repo/.git" ] || { echo "Missing deploy checkout: $repo" >&2; exit 1; }
+[[ "$repo" =~ ^/[A-Za-z0-9_./-]+$ ]] || { echo 'Deploy path must be absolute and contain no whitespace' >&2; exit 1; }
+if [ "$(loginctl show-user "$USER" -p Linger --value)" != yes ]; then
+  echo "Run once: sudo loginctl enable-linger $USER" >&2
+  exit 1
+fi
+config="${XDG_CONFIG_HOME:-$HOME/.config}/homelab-deploy"
+mkdir -p "$config" "$HOME/.local/lib/homelab-deploy" "$HOME/.config/systemd/user"
+chmod 700 "$config"
+if [ ! -f "$config/environment" ]; then
+  context="$(kubectl config current-context)"
+  cluster_uid="$(kubectl get namespace kube-system -o jsonpath='{.metadata.uid}')"
+  printf 'HOMELAB_REPO=%s\nKUBE_CONTEXT=%s\nEXPECTED_CLUSTER_UID=%s\n' "$repo" "$context" "$cluster_uid" >"$config/environment"
+  chmod 600 "$config/environment"
+fi
+# Do not replace a dispatcher while an existing deploy uses it.
+if systemctl --user list-units 'homelab-deploy@*' --state=running --no-legend | grep -q .; then
+  echo 'An existing deploy is running; wait before updating the controller' >&2
+  exit 1
+fi
+install -m 0755 "$here/../workflows/deploy-controller.py" "$HOME/.local/lib/homelab-deploy/controller.py"
+install -m 0644 "$here/homelab-deploy@.service" "$HOME/.config/systemd/user/homelab-deploy@.service"
+systemctl --user daemon-reload
+echo 'Controller ready. Run a checked main SHA in full mode for the initial baseline.'

.gitea/tests/deploy-validation.sh

@@ -0,0 +1,94 @@
+#!/usr/bin/env bash
+# Local regressions only: kubectl is mocked and Docker is used for config parsing.
+set -euo pipefail
+repo="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
+scratch="$(mktemp -d)"
+trap 'rm -rf "$scratch"' EXIT
+
+mkdir -p "$scratch/repo/app" "$scratch/repo/postgres" "$scratch/repo/netbird" "$scratch/repo/renovate"
+git -C "$scratch/repo" init -q
+for file in app/compose.yaml postgres/shared-compose.yaml netbird/client.compose.yaml renovate/renovate-compose.yaml; do
+  touch "$scratch/repo/$file"
+done
+git -C "$scratch/repo" add .
+# shellcheck source=../workflows/compose-lint.sh
+source "$repo/.gitea/workflows/compose-lint.sh"
+actual="$(cd "$scratch/repo" && compose_files)"
+expected=$'app/compose.yaml\nnetbird/client.compose.yaml\npostgres/shared-compose.yaml\nrenovate/renovate-compose.yaml'
+[ "$actual" = "$expected" ] || { echo 'Compose discovery missed a file' >&2; exit 1; }
+
+cat >"$scratch/compose.yaml" <<'YAML'
+services:
+  example:
+    image: busybox:1.37.0
+    environment:
+      REQUIRED: ${HOMELAB_TEST_REQUIRED:?required for this regression}
+YAML
+unset HOMELAB_TEST_REQUIRED
+if validate_compose_file "$scratch/compose.yaml" >"$scratch/config.log" 2>&1; then
+  echo 'Full Compose validation accepted a missing variable' >&2
+  exit 1
+fi
+grep -q 'required for this regression' "$scratch/config.log"
+HOMELAB_TEST_REQUIRED=present validate_compose_file "$scratch/compose.yaml"
+
+cat >"$scratch/resources.json" <<'JSON'
+{"kind":"List","items":[
+  {"kind":"Deployment","metadata":{"namespace":"app"},"spec":{"template":{"spec":{
+    "containers":[{"envFrom":[{"secretRef":{"name":"credentials"}},{"secretRef":{"name":"optional","optional":true}}],"env":[{"valueFrom":{"secretKeyRef":{"name":"credentials","key":"password"}}}]}],
+    "initContainers":[{"envFrom":[{"secretRef":{"name":"init"}}]}],
+    "imagePullSecrets":[{"name":"registry"}],
+    "volumes":[{"secret":{"secretName":"mounted"}},{"projected":{"sources":[{"secret":{"name":"projected"}},{"secret":{"name":"optional-projected","optional":true}}]}}]
+  }}}},
+  {"kind":"CronJob","metadata":{},"spec":{"jobTemplate":{"spec":{"template":{"spec":{"containers":[{"envFrom":[{"secretRef":{"name":"cron"}}]}]}}}}}},
+  {"kind":"IngressRoute","metadata":{"namespace":"app"},"spec":{"tls":{"secretName":"controller-issued-tls"}}}
+]}
+JSON
+actual="$(jq -r -f "$repo/.gitea/workflows/secret-references.jq" "$scratch/resources.json" | sort)"
+expected=$'app credentials\napp init\napp mounted\napp projected\napp registry\ndefault cron'
+[ "$actual" = "$expected" ] || { echo "Unexpected Secret references: $actual" >&2; exit 1; }
+
+REPO="$repo"
+# shellcheck source=../workflows/deploy-lib.sh
+source "$repo/.gitea/workflows/deploy-lib.sh"
+K8S_MANIFESTS=("$scratch/resources.json")
+KUSTOMIZE_APPS=()
+# No live cluster access. Reject credentials in app even if they exist elsewhere.
+kubectl() {
+  case "$1" in
+    create) cat "$scratch/resources.json" ;;
+    get)
+      if [ "$3" = credentials ] && [ "$5" = app ]; then
+        return 1
+      fi
+      return 0
+      ;;
+    *) echo "Unexpected kubectl invocation: $*" >&2; return 1 ;;
+  esac
+}
+if check_referenced_secrets >"$scratch/secrets.log"; then
+  echo 'Namespace-scoped Secret check accepted a missing Secret' >&2
+  exit 1
+fi
+grep -q 'MISSING OR UNREADABLE: app/credentials' "$scratch/secrets.log"
+# API/rendering errors must not produce an empty reference list and pass.
+kubectl() { return 1; }
+if ! skip_uninstalled_vmagent_crd "$REPO/prometheus-stack/k8s/vmagent.yaml"; then
+  echo 'VMAgent preflight did not skip an uninstalled CRD' >&2
+  exit 1
+fi
+kubectl() { return 0; }
+if skip_uninstalled_vmagent_crd "$REPO/prometheus-stack/k8s/vmagent.yaml"; then
+  echo 'VMAgent preflight skipped an installed CRD' >&2
+  exit 1
+fi
+if skip_uninstalled_vmagent_crd "$REPO/prometheus-stack/k8s/victoria.yaml"; then
+  echo 'VMAgent preflight skipped an unrelated manifest' >&2
+  exit 1
+fi
+kubectl() { return 1; }
+if check_referenced_secrets >"$scratch/secrets.log"; then
+  echo 'Secret check accepted a failed manifest render' >&2
+  exit 1
+fi
+printf '%s\n' 'Deploy validation regressions passed.'

.gitea/workflows/ci.yaml

@@ -0,0 +1,426 @@
+name: ci
+"on":
+  push:
+    branches:
+      - main
+  pull_request: null
+  workflow_dispatch: null
+permissions:
+  contents: read
+  actions: read
+concurrency:
+  group: ci-${{ github.ref }}
+  cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
+jobs:
+  compose:
+    name: Compose
+    runs-on: homelab
+    timeout-minutes: 15
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
+        id: source
+      - name: Validate Compose files
+        shell: bash
+        run: |
+          set -euo pipefail
+          source .gitea/workflows/compose-lint.sh
+
+          mapfile -t safe_flags < <(compose_safe_flags)
+          echo "docker compose config ${safe_flags[*]-}"
+
+          mapfile -t files < <(compose_files)
+          if [ "${#files[@]}" -eq 0 ]; then
+            echo "No Compose files found."
+            exit 0
+          fi
+
+          failed=0
+          for f in "${files[@]}"; do
+            if ! out="$(validate_compose_file "$f" ${safe_flags[@]+"${safe_flags[@]}"} 2>&1)"; then
+              failed=1
+              echo "::error file=${f}::$(printf '%s' "$out" | head -1)"
+            fi
+          done
+
+          if [ "$failed" -ne 0 ]; then
+            echo "Compose validation failed."
+            exit 1
+          fi
+          echo "checked ${#files[@]} Compose file(s)"
+        id: check
+      - name: Write the job result
+        if: always()
+        env:
+          SUMMARY_CHECK: Compose
+          SUMMARY_RESULT: ${{ job.status }}
+          SUMMARY_FAILED_STEP:
+            ${{ steps.check.conclusion == 'failure' && 'Check or image build' || steps.source.conclusion == 'failure'
+            && 'Source checkout' || '' }}
+        shell: bash
+        run: |
+          if [ -f .gitea/workflows/release.py ]; then
+            python3 .gitea/workflows/release.py check-summary
+          elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
+            printf '## %s\n\n- Result: **%s**\n- Failed step: %s\n' "$SUMMARY_CHECK" "$SUMMARY_RESULT" "$SUMMARY_FAILED_STEP" >>"$GITHUB_STEP_SUMMARY" || true
+          fi
+  workflows:
+    name: Workflows
+    runs-on: homelab
+    timeout-minutes: 15
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
+        id: source
+      - name: Prepare pinned tools
+        shell: bash
+        run: |
+          set -euo pipefail
+          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh actionlint shellcheck)"
+          echo "$tools_dir" >> "$GITHUB_PATH"
+        id: tools
+      - name: Lint Gitea Actions workflows with actionlint
+        shell: bash
+        run: |
+          set -euo pipefail
+          actionlint -config-file .gitea/actionlint.yaml -color .gitea/workflows/*.yaml
+        id: check
+      - name: Write the job result
+        if: always()
+        env:
+          SUMMARY_CHECK: Workflows
+          SUMMARY_RESULT: ${{ job.status }}
+          SUMMARY_FAILED_STEP:
+            ${{ steps.check.conclusion == 'failure' && 'Check or image build' || steps.tools.conclusion == 'failure'
+            && 'Tool setup' || steps.source.conclusion == 'failure' && 'Source checkout' || '' }}
+        shell: bash
+        run: |
+          if [ -f .gitea/workflows/release.py ]; then
+            python3 .gitea/workflows/release.py check-summary
+          elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
+            printf '## %s\n\n- Result: **%s**\n- Failed step: %s\n' "$SUMMARY_CHECK" "$SUMMARY_RESULT" "$SUMMARY_FAILED_STEP" >>"$GITHUB_STEP_SUMMARY" || true
+          fi
+  shell:
+    name: Shell
+    runs-on: homelab
+    timeout-minutes: 15
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
+        id: source
+      - name: Prepare pinned tools
+        shell: bash
+        run: |
+          set -euo pipefail
+          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh shellcheck jq)"
+          echo "$tools_dir" >> "$GITHUB_PATH"
+        id: tools
+      - name: Lint shell scripts with ShellCheck
+        shell: bash
+        run: |
+          set -euo pipefail
+          mapfile -t scripts < <(
+            git ls-files '*.sh' ':(glob)**/*.bash'
+          )
+          if [ "${#scripts[@]}" -eq 0 ]; then
+            echo "No shell scripts found."
+            exit 0
+          fi
+          shellcheck --external-sources --source-path=SCRIPTDIR --severity=style "${scripts[@]}"
+          bash .gitea/tests/deploy-validation.sh
+        id: check
+      - name: Write the job result
+        if: always()
+        env:
+          SUMMARY_CHECK: Shell
+          SUMMARY_RESULT: ${{ job.status }}
+          SUMMARY_FAILED_STEP:
+            ${{ steps.check.conclusion == 'failure' && 'Check or image build' || steps.tools.conclusion == 'failure'
+            && 'Tool setup' || steps.source.conclusion == 'failure' && 'Source checkout' || '' }}
+        shell: bash
+        run: |
+          if [ -f .gitea/workflows/release.py ]; then
+            python3 .gitea/workflows/release.py check-summary
+          elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
+            printf '## %s\n\n- Result: **%s**\n- Failed step: %s\n' "$SUMMARY_CHECK" "$SUMMARY_RESULT" "$SUMMARY_FAILED_STEP" >>"$GITHUB_STEP_SUMMARY" || true
+          fi
+  formatting:
+    name: Formatting
+    runs-on: homelab
+    timeout-minutes: 15
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
+        id: source
+      - name: Prepare pinned tools
+        shell: bash
+        run: |
+          set -euo pipefail
+          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh prettier)"
+          echo "$tools_dir" >> "$GITHUB_PATH"
+        id: tools
+      - name: Check formatting with Prettier
+        shell: bash
+        run: |
+          set -euo pipefail
+
+          mapfile -t prettier_files < <(
+            git ls-files \
+              | grep -E '\.(md|json|ya?ml|html|css)$' \
+              | grep -Ev '^(\.docs/|\.zed/|errorpages/html/|homepages/(forust_files|xdfnx_files)/)'
+          )
+
+          if [ "${#prettier_files[@]}" -eq 0 ]; then
+            echo "No Prettier-managed files found."
+            exit 0
+          fi
+
+          prettier --check --ignore-unknown "${prettier_files[@]}"
+        id: check
+      - name: Write the job result
+        if: always()
+        env:
+          SUMMARY_CHECK: Formatting
+          SUMMARY_RESULT: ${{ job.status }}
+          SUMMARY_FAILED_STEP:
+            ${{ steps.check.conclusion == 'failure' && 'Check or image build' || steps.tools.conclusion == 'failure'
+            && 'Tool setup' || steps.source.conclusion == 'failure' && 'Source checkout' || '' }}
+        shell: bash
+        run: |
+          if [ -f .gitea/workflows/release.py ]; then
+            python3 .gitea/workflows/release.py check-summary
+          elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
+            printf '## %s\n\n- Result: **%s**\n- Failed step: %s\n' "$SUMMARY_CHECK" "$SUMMARY_RESULT" "$SUMMARY_FAILED_STEP" >>"$GITHUB_STEP_SUMMARY" || true
+          fi
+  python:
+    name: Python and tests
+    runs-on: homelab
+    timeout-minutes: 15
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
+        id: source
+      - name: Prepare pinned tools
+        shell: bash
+        run: |
+          set -euo pipefail
+          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh ruff jq)"
+          echo "$tools_dir" >> "$GITHUB_PATH"
+        id: tools
+      - name: Lint and format-check Python with Ruff
+        shell: bash
+        run: |
+          set -euo pipefail
+          ruff check . .gitea/workflows
+          ruff format --check . .gitea/workflows
+          python3 -m unittest discover -s tests -v
+        id: check
+      - name: Write the job result
+        if: always()
+        env:
+          SUMMARY_CHECK: Python and tests
+          SUMMARY_RESULT: ${{ job.status }}
+          SUMMARY_FAILED_STEP:
+            ${{ steps.check.conclusion == 'failure' && 'Check or image build' || steps.tools.conclusion == 'failure'
+            && 'Tool setup' || steps.source.conclusion == 'failure' && 'Source checkout' || '' }}
+        shell: bash
+        run: |
+          if [ -f .gitea/workflows/release.py ]; then
+            python3 .gitea/workflows/release.py check-summary
+          elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
+            printf '## %s\n\n- Result: **%s**\n- Failed step: %s\n' "$SUMMARY_CHECK" "$SUMMARY_RESULT" "$SUMMARY_FAILED_STEP" >>"$GITHUB_STEP_SUMMARY" || true
+          fi
+  yaml:
+    name: YAML
+    runs-on: homelab
+    timeout-minutes: 15
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
+        id: source
+      - name: Prepare pinned tools
+        shell: bash
+        run: |
+          set -euo pipefail
+          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh yamllint)"
+          echo "$tools_dir" >> "$GITHUB_PATH"
+        id: tools
+      - name: Lint YAML syntax
+        shell: bash
+        run: |
+          set -euo pipefail
+
+          mapfile -t yaml_files < <(
+            git ls-files '*.yaml' '*.yml' \
+              ':!node_modules/**' \
+              ':!**/.venv/**'
+          )
+
+          if [ "${#yaml_files[@]}" -eq 0 ]; then
+            echo "No YAML files found."
+            exit 0
+          fi
+
+          yamllint -c .yamllint "${yaml_files[@]}"
+        id: check
+      - name: Write the job result
+        if: always()
+        env:
+          SUMMARY_CHECK: YAML
+          SUMMARY_RESULT: ${{ job.status }}
+          SUMMARY_FAILED_STEP:
+            ${{ steps.check.conclusion == 'failure' && 'Check or image build' || steps.tools.conclusion == 'failure'
+            && 'Tool setup' || steps.source.conclusion == 'failure' && 'Source checkout' || '' }}
+        shell: bash
+        run: |
+          if [ -f .gitea/workflows/release.py ]; then
+            python3 .gitea/workflows/release.py check-summary
+          elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
+            printf '## %s\n\n- Result: **%s**\n- Failed step: %s\n' "$SUMMARY_CHECK" "$SUMMARY_RESULT" "$SUMMARY_FAILED_STEP" >>"$GITHUB_STEP_SUMMARY" || true
+          fi
+  dockerfiles:
+    name: Dockerfiles
+    runs-on: homelab
+    timeout-minutes: 15
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
+        id: source
+      - name: Prepare pinned tools
+        shell: bash
+        run: |
+          set -euo pipefail
+          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh hadolint)"
+          echo "$tools_dir" >> "$GITHUB_PATH"
+        id: tools
+      - name: Lint Dockerfiles
+        shell: bash
+        run: |
+          set -euo pipefail
+
+          mapfile -t dockerfiles < <(
+            git ls-files ':(glob)**/Dockerfile' ':(glob)**/Dockerfile.*'
+          )
+
+          if [ "${#dockerfiles[@]}" -eq 0 ]; then
+            echo "No Dockerfiles found."
+            exit 0
+          fi
+
+          hadolint -c .hadolint.yaml "${dockerfiles[@]}"
+        id: check
+      - name: Write the job result
+        if: always()
+        env:
+          SUMMARY_CHECK: Dockerfiles
+          SUMMARY_RESULT: ${{ job.status }}
+          SUMMARY_FAILED_STEP:
+            ${{ steps.check.conclusion == 'failure' && 'Check or image build' || steps.tools.conclusion == 'failure'
+            && 'Tool setup' || steps.source.conclusion == 'failure' && 'Source checkout' || '' }}
+        shell: bash
+        run: |
+          if [ -f .gitea/workflows/release.py ]; then
+            python3 .gitea/workflows/release.py check-summary
+          elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
+            printf '## %s\n\n- Result: **%s**\n- Failed step: %s\n' "$SUMMARY_CHECK" "$SUMMARY_RESULT" "$SUMMARY_FAILED_STEP" >>"$GITHUB_STEP_SUMMARY" || true
+          fi
+  kubernetes:
+    name: Kubernetes
+    runs-on: homelab
+    timeout-minutes: 15
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
+        id: source
+      - name: Prepare pinned tools
+        shell: bash
+        run: |
+          set -euo pipefail
+          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh kubeconform)"
+          echo "$tools_dir" >> "$GITHUB_PATH"
+        id: tools
+      - name: Validate Kubernetes manifests against JSON schemas
+        shell: bash
+        run: |
+          set -euo pipefail
+
+          mapfile -t manifests < <(
+            git ls-files ':(glob)**/k8s/**/*.yaml' ':(glob)**/k8s/**/*.yml' \
+              | grep -Ev '(^|/)(kustomization\.ya?ml|.*\.example\.ya?ml|.*values\.ya?ml|patch-.*\.ya?ml)$'
+          )
+
+          if [ "${#manifests[@]}" -eq 0 ]; then
+            echo "No Kubernetes manifests found."
+            exit 0
+          fi
+
+          kubeconform \
+            -strict \
+            -ignore-missing-schemas \
+            -summary \
+            "${manifests[@]}"
+        id: check
+      - name: Write the job result
+        if: always()
+        env:
+          SUMMARY_CHECK: Kubernetes
+          SUMMARY_RESULT: ${{ job.status }}
+          SUMMARY_FAILED_STEP:
+            ${{ steps.check.conclusion == 'failure' && 'Check or image build' || steps.tools.conclusion == 'failure'
+            && 'Tool setup' || steps.source.conclusion == 'failure' && 'Source checkout' || '' }}
+        shell: bash
+        run: |
+          if [ -f .gitea/workflows/release.py ]; then
+            python3 .gitea/workflows/release.py check-summary
+          elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
+            printf '## %s\n\n- Result: **%s**\n- Failed step: %s\n' "$SUMMARY_CHECK" "$SUMMARY_RESULT" "$SUMMARY_FAILED_STEP" >>"$GITHUB_STEP_SUMMARY" || true
+          fi
+  build:
+    needs:
+      - compose
+      - workflows
+      - shell
+      - formatting
+      - python
+      - yaml
+      - dockerfiles
+      - kubernetes
+    if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main'
+    runs-on: homelab
+    timeout-minutes: 60
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
+        with:
+          fetch-depth: 0
+        id: source
+      - name: Build changed images and write release
+        env:
+          GITEA_TOKEN: ${{ github.token }}
+          REGISTRY_USERNAME: ${{ secrets.REGISTRY_USERNAME }}
+          REGISTRY_PASSWORD: ${{ secrets.REGISTRY_PASSWORD }}
+        run: python3 .gitea/workflows/release.py build
+        id: check
+      - name: Store commit release
+        uses: actions/upload-artifact@c6a366c94c3e0affe28c06c8df20a878f24da3cf
+        with:
+          name: release-${{ github.sha }}
+          path: release.json
+          if-no-files-found: error
+          retention-days: 30
+        id: artifact
+      - name: Write the job result
+        if: always()
+        env:
+          SUMMARY_CHECK: Image build and release artifact
+          SUMMARY_RESULT: ${{ job.status }}
+          SUMMARY_FAILED_STEP:
+            ${{ steps.check.conclusion == 'failure' && 'Check or image build' || steps.artifact.conclusion == 'failure'
+            && 'Release artifact upload' || steps.source.conclusion == 'failure' && 'Source checkout' || '' }}
+        shell: bash
+        run: |
+          if [ -f .gitea/workflows/release.py ]; then
+            python3 .gitea/workflows/release.py check-summary
+          elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
+            printf '## %s\n\n- Result: **%s**\n- Failed step: %s\n' "$SUMMARY_CHECK" "$SUMMARY_RESULT" "$SUMMARY_FAILED_STEP" >>"$GITHUB_STEP_SUMMARY" || true
+          fi

.gitea/workflows/compose-lint.sh

@@ -0,0 +1,45 @@
+#!/usr/bin/env bash
+# Shared helpers for validating Compose files. Sourced both by steps in
+# .gitea/workflows/ci.yaml and by deploy-lib.sh on the workstation.
+#
+# Two levels of checking, matching how the repo is structured:
+#
+#   general  every committed Compose file, active or not. Pure structure check:
+#            no ${VAR} interpolation, no .env lookup, no bind-mount path
+#            resolution. Disabled stacks deliberately have no .env in the repo
+#            and no values on the CI runner, so a full `config` run would fail on
+#            their `${VAR:?}` guards for reasons that have nothing to do with the
+#            change under review.
+#
+#   full     active stacks only, with interpolation and env-file resolution, so
+#            required variables and referenced files are actually resolved. Needs
+#            the gitignored .env files, so this only runs in the deploy workflow
+#            on the workstation.
+#
+# This file is meant to be sourced, not executed.
+
+# All committed Compose files, including the ones deploy never starts.
+compose_files() {
+  git ls-files \
+    '*compose.yaml' '*compose.yml'
+}
+
+# Prints the flags that turn `docker compose config` into the general check.
+# Probed rather than hardcoded so an older Compose without --no-env-resolution
+# still gets the flags it does support.
+compose_safe_flags() {
+  local help flag
+  help="$(docker compose config --help 2>/dev/null || true)"
+  for flag in --no-interpolate --no-env-resolution --no-path-resolution; do
+    if printf '%s' "$help" | grep -q -- "$flag"; then
+      printf '%s\n' "$flag"
+    fi
+  done
+}
+
+# validate_compose_file <file> [extra docker compose config flags...]
+validate_compose_file() {
+  local file="$1"
+  shift
+  docker compose -f "$file" config --quiet "$@"
+}

.gitea/workflows/compose-release.py

@@ -0,0 +1,93 @@
+#!/usr/bin/env python3
+"""Resolve Compose images without changing project names or local bind paths."""
+
+import json
+import os
+import re
+import subprocess
+import sys
+from pathlib import Path
+
+
+def output(*args, **kwargs):
+    return subprocess.check_output(args, text=True, **kwargs).strip()  # noqa: S603
+
+
+def resolve(reference):
+    if '@sha256:' in reference:
+        return reference
+    descriptor = json.loads(
+        output('docker', 'buildx', 'imagetools', 'inspect', reference, '--format', '{{json .Manifest}}')
+    )
+    digest = descriptor['digest']
+    if not re.fullmatch(r'sha256:[0-9a-f]{64}', digest):
+        raise ValueError(f'Invalid registry digest for {reference}')
+    # Strip tag only from the final path segment (registry ports are preserved).
+    repository = reference.rsplit('/', 1)
+    repository[-1] = repository[-1].split(':')[0]
+    return '/'.join(repository) + '@' + digest
+
+
+def prepare(source_file):
+    config_repo = Path(os.environ['CONFIG_REPO'])
+    source_repo = Path(os.environ['REPO'])
+    directory = Path(os.environ['RUN_DIR'])
+    relative = source_file.relative_to(source_repo)
+    project_dir = config_repo / relative.parent
+    base = ['docker', 'compose', '--project-directory', str(project_dir), '-f', str(source_file)]
+    config = json.loads(output(*base, 'config', '--format', 'json', cwd=config_repo))
+    project = config['name']
+    previous_file = directory / 'previous.json'
+    previous = json.loads(previous_file.read_text()) if previous_file.exists() else {}
+    images_file = directory / 'compose-images.json'
+    locks = json.loads(images_file.read_text()) if images_file.exists() else previous.get('compose-images', {})
+    release = json.loads((directory / 'release.json').read_text())
+    before = json.loads(json.dumps(config))
+    for service, settings in config['services'].items():
+        reference = settings.get('image')
+        if not reference or settings.get('build'):
+            raise ValueError(f'{project}/{service}: Compose deploy requires a published image')
+        image_repo = reference.split('@')[0].rsplit('/', 1)
+        image_repo[-1] = image_repo[-1].split(':')[0]
+        image_repo = '/'.join(image_repo)
+        if image_repo in release['images']:
+            pinned = image_repo + '@' + release['images'][image_repo]
+        elif os.environ.get('REFRESH_IMAGES') != 'true' and reference in locks:
+            pinned = locks[reference]
+        else:
+            pinned = resolve(reference)
+        settings['image'] = pinned
+        locks[reference] = pinned
+        # Capture what is running, not the current value of its mutable tag.
+        ids = output(
+            'docker',
+            'ps',
+            '-aq',
+            '--filter',
+            f'label=com.docker.compose.project={project}',
+            '--filter',
+            f'label=com.docker.compose.service={service}',
+        ).splitlines()
+        actual = set()
+        for container in ids:
+            image_id = output('docker', 'inspect', container, '--format', '{{.Image}}')
+            digests = json.loads(output('docker', 'image', 'inspect', image_id, '--format', '{{json .RepoDigests}}'))
+            actual.add(next((d for d in digests or [] if d.split('@')[0] == image_repo), image_id))
+        if len(actual) > 1:
+            raise ValueError(f'{project}/{service}: mixed running images, cannot capture one recovery config')
+        before['services'][service]['image'] = next(iter(actual)) if actual else reference
+    for name, data in (('compose', config), ('compose-before', before)):
+        folder = directory / name
+        folder.mkdir(mode=0o700, exist_ok=True)
+        destination = folder / f'{relative.parent.name}.json'
+        destination.write_text(json.dumps(data, indent=2) + '\n')
+        destination.chmod(0o600)
+    images_file.write_text(json.dumps(locks, indent=2) + '\n')
+    print(f'Compose {project}: images pinned; local paths preserved')
+    print(
+        f'Recovery: docker compose --project-directory {project_dir} -p {project} -f {directory}/compose-before/{relative.parent.name}.json up -d --pull never'
+    )
+
+
+if __name__ == '__main__':
+    prepare(Path(sys.argv[1]))

.gitea/workflows/deploy-controller.py

@@ -0,0 +1,422 @@
+#!/usr/bin/env python3
+"""Durable workstation deployment controller. Install with setup-workstation.sh."""
+
+import argparse
+import contextlib
+import fcntl
+import importlib.util
+import json
+import math
+import os
+import re
+import shutil
+import subprocess
+import sys
+import time
+from pathlib import Path
+
+STATE = Path(os.environ.get('HOMELAB_STATE', Path.home() / '.local/state/homelab-deploy'))
+CONFIG_REPO = Path(os.environ.get('HOMELAB_REPO', '/srv/homelab'))
+RUN_ID = re.compile(r'[0-9]+-[0-9]+')
+
+
+def command(*args, **kwargs):
+    return subprocess.check_output(args, text=True, **kwargs).strip()  # noqa: S603, S607
+
+
+def atomic_json(path, data):
+    temporary = path.with_suffix('.tmp')
+    temporary.write_text(json.dumps(data, indent=2) + '\n')
+    temporary.chmod(0o600)
+    temporary.replace(path)
+
+
+@contextlib.contextmanager
+def lock(name):
+    STATE.mkdir(mode=0o700, parents=True, exist_ok=True)
+    with (STATE / name).open('a') as stream:
+        fcntl.flock(stream, fcntl.LOCK_EX)
+        yield
+
+
+def load_module(name, path):
+    spec = importlib.util.spec_from_file_location(name, path)
+    module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(module)
+    return module
+
+
+def run_directory(run_id):
+    if not RUN_ID.fullmatch(run_id):
+        raise ValueError('Run ID must be numeric workflow-id and attempt')
+    return STATE / 'runs' / run_id
+
+
+def start(run_id):
+    payload = sys.stdin.buffer.read(256 * 1024 + 1)
+    if len(payload) > 256 * 1024:
+        raise ValueError('Deploy request exceeds 256 KiB')
+    request = json.loads(payload)
+    sha = request['release']['sha']
+    if not re.fullmatch(r'[0-9a-f]{40}', sha) or request['mode'] not in ('changed', 'full', 'plan'):
+        raise ValueError('Invalid deploy SHA or mode')
+    if not isinstance(request['refresh_images'], bool):
+        raise ValueError('refresh_images must be boolean')
+    directory = run_directory(run_id)
+    with lock('prepare.lock'):
+        if (directory / 'request.json').exists():
+            if json.loads((directory / 'request.json').read_text()) != request:
+                raise ValueError('Run ID already belongs to a different request')
+        else:
+            directory.mkdir(mode=0o700, parents=True, exist_ok=True)
+            command('git', '-C', str(CONFIG_REPO), 'fetch', '--quiet', 'origin', 'main')
+            command('git', '-C', str(CONFIG_REPO), 'merge-base', '--is-ancestor', sha, 'origin/main')
+            if not (directory / 'source').exists():
+                command('git', '-C', str(CONFIG_REPO), 'worktree', 'add', '--detach', str(directory / 'source'), sha)
+            if command('git', '-C', str(directory / 'source'), 'rev-parse', 'HEAD') != sha:
+                raise ValueError('Prepared source does not match deploy SHA')
+            release_module = load_module('release', directory / 'source/.gitea/workflows/release.py')
+            release_module.validate_release(request['release'], sha)
+            atomic_json(directory / 'release.json', request['release'])
+            atomic_json(directory / 'request.json', request)
+        if not (directory / 'status.json').exists():
+            atomic_json(directory / 'status.json', {'state': 'queued', 'stages': {}})
+    # Starting an existing active or finished ID is idempotent; never re-apply it.
+    if json.loads((directory / 'status.json').read_text())['state'] == 'queued':
+        command('systemctl', '--user', 'start', '--no-block', f'homelab-deploy@{run_id}.service')
+    print(f'Accepted deploy {run_id} ({sha})')
+
+
+def environment(directory):
+    request = json.loads((directory / 'request.json').read_text())
+    return {
+        **os.environ,
+        'REPO': str(directory / 'source'),
+        'CONFIG_REPO': str(CONFIG_REPO),
+        'RUN_DIR': str(directory),
+        'DEPLOY_SHA': request['release']['sha'],
+        'RELEASE_FILE': str(directory / 'release.json'),
+        'DEPLOY_PLAN': str(directory / 'plan.json'),
+        'DEPLOY_SNAPSHOT_DIR': str(directory / 'snapshot'),
+        'REFRESH_IMAGES': str(request['refresh_images']).lower(),
+        'ROLLOUT_PARALLELISM': '4',
+    }
+
+
+def stage(directory, name, budget):
+    status = json.loads((directory / 'status.json').read_text())
+    if name in status['stages'] and status['stages'][name].get('result') in ('success', 'failure'):
+        return status['stages'][name]['result'] == 'success'
+    started = time.time()
+    status['stages'][name] = {'result': 'running', 'started': started}
+    atomic_json(directory / 'status.json', status)
+    script = directory / 'source/.gitea/workflows/deploy-stage.sh'
+    with (directory / f'{name}.log').open('a') as log:
+        # timeout kills the whole stage process group, including children, before recovery.
+        result = subprocess.run(  # noqa: S603, S607
+            [
+                shutil.which('timeout') or '/usr/bin/timeout',
+                '--signal=TERM',
+                '--kill-after=30s',
+                str(budget),
+                'bash',
+                str(script),
+                name,
+            ],
+            env=environment(directory),
+            stdout=log,
+            stderr=subprocess.STDOUT,
+            check=False,
+        ).returncode
+    status = json.loads((directory / 'status.json').read_text())
+    status['stages'][name].update(
+        result='success' if result == 0 else 'failure', exit_code=result, seconds=round(time.time() - started)
+    )
+    atomic_json(directory / 'status.json', status)
+    return result == 0
+
+
+def make_plan(directory):
+    source = directory / 'source'
+    planner = load_module('deploy_plan', source / '.gitea/workflows/deploy-plan.py')
+    request = json.loads((directory / 'request.json').read_text())
+    previous = json.loads((STATE / 'last-success.json').read_text()) if (STATE / 'last-success.json').exists() else None
+    helm = json.loads(command('helm', 'list', '--all', '-A', '-o', 'json'))
+    plan = planner.make_plan(source, CONFIG_REPO, request['release'], previous, request['mode'], helm)
+    if request['refresh_images']:
+        plan['selected']['compose'] = plan['active']['compose']
+    atomic_json(directory / 'plan.json', plan)
+    if previous:
+        atomic_json(directory / 'previous.json', previous)
+    # Local config is deliberately separate from the immutable Git source.
+    return plan
+
+
+def finish_success(directory, plan):
+    # Repeating finalization after a crash is safe while holding deploy.lock.
+    plan['run_id'] = directory.name
+    path = directory / 'compose-images.json'
+    previous = directory / 'previous.json'
+    plan['compose-images'] = (
+        json.loads(path.read_text())
+        if path.exists()
+        else json.loads(previous.read_text()).get('compose-images', {})
+        if previous.exists()
+        else {}
+    )
+    atomic_json(STATE / 'last-success.json', plan)
+    status = json.loads((directory / 'status.json').read_text())
+    status['state'] = 'success'
+    atomic_json(directory / 'status.json', status)
+    try:
+        retain_completed(directory)
+    except (OSError, subprocess.CalledProcessError) as error:
+        print(f'Retention deferred: {error}', flush=True)
+
+
+def recover(directory, retry=False):
+    status = json.loads((directory / 'status.json').read_text())
+    if status['state'] in ('success', 'planned'):
+        return
+    completed = ('doctor', 'validate', 'apply-k8s', 'apply-compose', 'verify-k8s', 'smoke')
+    if all(status['stages'].get(name, {}).get('result') == 'success' for name in completed):
+        finish_success(directory, json.loads((directory / 'plan.json').read_text()))
+        return
+    if retry:
+        for name in ('verify-k8s', 'smoke'):
+            if status['stages'].get(name, {}).get('result') == 'failure':
+                del status['stages'][name]
+        atomic_json(directory / 'status.json', status)
+    snapshot = directory / 'snapshot/current'
+    if snapshot.exists():
+        stage(directory, 'verify-k8s', 7200)
+        stage(directory, 'smoke', 600)
+    status = json.loads((directory / 'status.json').read_text())
+    status['state'] = 'failure'
+    atomic_json(directory / 'status.json', status)
+
+
+def execute(run_id):
+    directory = run_directory(run_id)
+    with lock('deploy.lock'):
+        status = json.loads((directory / 'status.json').read_text())
+        if status['state'] != 'queued':
+            return
+        # A crashed predecessor must be recovered before another apply begins.
+        for other in (STATE / 'runs').iterdir():
+            if (
+                other != directory
+                and (other / 'status.json').exists()
+                and json.loads((other / 'status.json').read_text())['state'] == 'running'
+            ):
+                raise ValueError(f'Interrupted deploy {other.name}; run recover first')
+        status['state'] = 'running'
+        atomic_json(directory / 'status.json', status)
+        phase = 'plan'
+        try:
+            plan = make_plan(directory)
+            print(
+                json.dumps({'selected': plan['selected'], 'helm': plan['helm'], 'manual_removals': plan['removed']}),
+                flush=True,
+            )
+            phase = 'doctor'
+            if not stage(directory, 'doctor', 600):
+                raise RuntimeError('Preflight failed')
+            phase = 'validate'
+            if not stage(directory, 'validate', 1200):
+                raise RuntimeError('Validation failed')
+            if json.loads((directory / 'request.json').read_text())['mode'] == 'plan':
+                status = json.loads((directory / 'status.json').read_text())
+                status['state'] = 'planned'
+                atomic_json(directory / 'status.json', status)
+                return
+            # Budget includes both rollout checks and rollback waves, plus API overhead.
+            phase = 'Recovery budget'
+            count = int(
+                command(
+                    'bash',
+                    str(directory / 'source/.gitea/workflows/deploy-stage.sh'),
+                    'workload-count',
+                    env=environment(directory),
+                )
+            )
+            verify_budget = max(600, 2 * math.ceil(count / 4) * 300 + 120)
+            if verify_budget > 7200:
+                raise ValueError('More than two hours of recovery required; split this deploy')
+            phase = 'apply-k8s'
+            k8s_ok = stage(directory, 'apply-k8s', 2700)
+            phase = 'apply-compose'
+            compose_ok = stage(directory, 'apply-compose', 1800) if k8s_ok else False
+            phase = 'verify-k8s'
+            verify_ok = stage(directory, 'verify-k8s', verify_budget)
+            phase = 'smoke'
+            smoke_ok = stage(directory, 'smoke', 600)
+            if not all((k8s_ok, compose_ok, verify_ok, smoke_ok)):
+                raise RuntimeError('Deploy failed; inspect stage logs and recovery report')
+            phase = 'Save the successful baseline'
+            finish_success(directory, plan)
+        except Exception as error:
+            status = json.loads((directory / 'status.json').read_text())
+            status['failure_stage'] = next(
+                (name for name, result in status['stages'].items() if result.get('result') == 'failure'), phase
+            )
+            atomic_json(directory / 'status.json', status)
+            with (directory / 'controller.log').open('a') as stream:
+                stream.write(f'{error}\n')
+            recover(directory)
+            raise
+
+
+def retain_completed(current):
+    finished = []
+    for directory in (STATE / 'runs').iterdir():
+        status_file = directory / 'status.json'
+        if status_file.exists() and json.loads(status_file.read_text())['state'] in ('success', 'planned'):
+            finished.append(directory)
+    for directory in sorted(finished, key=lambda p: p.stat().st_mtime, reverse=True)[20:]:
+        if directory == current:
+            continue
+        command('git', '-C', str(CONFIG_REPO), 'worktree', 'remove', '--force', str(directory / 'source'))
+        shutil.rmtree(directory)
+
+
+def follow(run_id, phase):
+    directory = run_directory(run_id)
+    groups = {
+        'apply': ('doctor', 'validate', 'apply-k8s', 'apply-compose'),
+        'verify': ('verify-k8s',),
+        'smoke': ('smoke',),
+    }
+    names = groups[phase]
+    offsets = {}
+    while True:
+        status = json.loads((directory / 'status.json').read_text())
+        for name in (*names, 'controller'):
+            path = directory / f'{name}.log'
+            if path.exists():
+                with path.open() as stream:
+                    stream.seek(offsets.get(name, 0))
+                    content = stream.read()
+                    if content:
+                        print(content, end='', flush=True)
+                    offsets[name] = stream.tell()
+        stages = status['stages']
+        if all(stages.get(name, {}).get('result') in ('success', 'failure') for name in names):
+            return all(stages[name]['result'] == 'success' for name in names)
+        if status['state'] in ('success', 'failure', 'planned'):
+            return status['state'] in ('success', 'planned')
+        time.sleep(3)
+
+
+def summary(run_id):
+    directory = run_directory(run_id)
+    request = json.loads((directory / 'request.json').read_text())
+    release = request['release']
+    plan_file = directory / 'plan.json'
+    lines = [
+        f'## Deploy `{release["sha"]}`',
+        '',
+        f'- Mode: `{request["mode"]}`',
+        f'- Refresh third-party images: `{request["refresh_images"]}`',
+    ]
+    status = json.loads((directory / 'status.json').read_text())
+    if status.get('failure_stage'):
+        lines.append(f'- Failed stage: **{status["failure_stage"]}**')
+    lines.extend(
+        [
+            '',
+            f'- Observed run state: **{status["state"]}**',
+            '',
+            '### Stage results',
+            '| Stage | Result | Exit code |',
+            '| --- | --- | --- |',
+        ]
+    )
+    for name in ('doctor', 'validate', 'apply-k8s', 'apply-compose', 'verify-k8s', 'smoke'):
+        stage_result = status['stages'].get(name, {})
+        lines.append(f'| {name} | {stage_result.get("result", "not started")} | {stage_result.get("exit_code", "—")} |')
+    lines.extend(['', '### Apply and Helm recovery results'])
+    events_file = directory / 'apply-events.jsonl'
+    events = []
+    if events_file.exists():
+        for line in events_file.read_text().splitlines():
+            try:
+                events.append(json.loads(line))
+            except json.JSONDecodeError:
+                lines.append('- An operation record is incomplete. Check the stage log.')
+    latest = {(event['action'], event['target']): event['result'] for event in events}
+    lines.extend(f'- `{action}` `{target}`: **{result}**' for (action, target), result in latest.items())
+    if not latest:
+        lines.append('- No apply results were recorded.')
+    lines.append('- A completed apply does not confirm health. See verification and smoke results.')
+    lines.extend(['', '### Kubernetes recovery'])
+    pointer = directory / 'snapshot/current'
+    failed = Path(pointer.read_text().strip()) / 'failed-workloads' if pointer.exists() else None
+    if failed and failed.exists():
+        contents = failed.read_text()
+        counts = dict(re.findall(r'^(ROLLED_BACK|UNRECOVERED)=([0-9]+)$', contents, re.MULTILINE))
+        if not contents.strip():
+            lines.append('- No failed workloads were recorded. See the verification result above.')
+        elif counts:
+            lines.append(f'- Workloads restored: **{counts.get("ROLLED_BACK", "unknown")}**')
+            lines.append(f'- Workloads that need manual recovery: **{counts.get("UNRECOVERED", "unknown")}**')
+        else:
+            lines.append('- Rollback has no recorded result yet. Check the verification log.')
+    else:
+        lines.append('- No workload rollback was recorded. This does not confirm health.')
+    lines.append('- Compose requires manual recovery. Use the saved command in the apply log.')
+    if not plan_file.exists():
+        lines.extend(['', 'Plan was not created. Check the controller log.'])
+        print('\n'.join(lines))
+        return
+    plan = json.loads(plan_file.read_text())
+    lines.extend(['', '### Selected services'])
+    count = 0
+    for kind, services in plan['selected'].items():
+        for service in services:
+            lines.append(f'- `{kind}`: `{service}`')
+            count += 1
+    if not count:
+        lines.append('- None')
+    lines.extend(['', '### Selected Helm releases'])
+    lines.extend(f'- `{release}`' for release in plan.get('helm', []))
+    if not plan.get('helm'):
+        lines.append('- None')
+    lines.extend(['', '### Images pinned in the checked release'])
+    lines.extend(f'- `{image}@{digest}`' for image, digest in sorted(release['images'].items()))
+    lines.extend(['', '### Removed resources requiring manual review'])
+    lines.extend(f'- `{item}`' for item in plan.get('removed', []))
+    if not plan.get('removed'):
+        lines.append('- None')
+    print('\n'.join(lines))
+
+
+def main():
+    os.umask(0o077)
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument('action', choices=('start', 'execute', 'recover', 'status', 'follow', 'summary'))
+    parser.add_argument('run_id')
+    parser.add_argument('phase', nargs='?', choices=('apply', 'verify', 'smoke'))
+    parser.add_argument('--retry', action='store_true', help='Retry failed recovery checks; never repeat apply')
+    args = parser.parse_args()
+    directory = run_directory(args.run_id)
+    if args.action == 'start':
+        start(args.run_id)
+    elif args.action == 'execute':
+        execute(args.run_id)
+    elif args.action == 'recover':
+        with lock('deploy.lock'):
+            recover(directory, retry=args.retry)
+    elif args.action == 'status':
+        print((directory / 'status.json').read_text())
+        if (directory / 'plan.json').exists():
+            plan = json.loads((directory / 'plan.json').read_text())
+            print(json.dumps({k: plan[k] for k in ('sha', 'selected', 'helm', 'removed')}, indent=2))
+    elif args.action == 'summary':
+        summary(args.run_id)
+    elif not follow(args.run_id, args.phase):
+        sys.exit(1)
+
+
+if __name__ == '__main__':
+    main()

.gitea/workflows/deploy-plan.py

@@ -0,0 +1,124 @@
+#!/usr/bin/env python3
+"""Calculate selected components against the last fully successful deploy."""
+
+import hashlib
+import json
+import re
+import subprocess
+from pathlib import Path
+
+
+def output(*args, **kwargs):
+    return subprocess.check_output(args, text=True, **kwargs).strip()  # noqa: S603
+
+
+def tracked(repo):
+    return output('git', '-C', str(repo), 'ls-files').splitlines()
+
+
+def helm_releases(repo):
+    text = (repo / '.gitea/workflows/deploy-lib.sh').read_text()
+    return [line.split('|') for line in re.findall(r'^  "([^"\n]+\|[^"\n]+)"$', text, re.MULTILINE)]
+
+
+def inventory(repo):
+    files = tracked(repo)
+    k8s = sorted(
+        {f.split('/k8s/')[0] for f in files if '/k8s/' in f and (repo / f.split('/k8s/')[0] / 'k8s/active').is_file()}
+    )
+    compose = sorted(
+        {
+            str(Path(f).parent)
+            for f in files
+            if Path(f).name in ('compose.yaml', 'compose.yml') and (repo / Path(f).parent / 'active').is_file()
+        }
+    )
+    return {'k8s': k8s, 'compose': compose}
+
+
+def file_hash(path):
+    return hashlib.sha256(path.read_bytes()).hexdigest() if path.is_file() else 'missing'
+
+
+def make_plan(repo, config_repo, release, previous, mode, live_helm):
+    active = inventory(repo)
+    all_services = set(active['k8s'] + active['compose'])
+    helm_inputs = {}
+    helm_selected = []
+    for name, chart, namespace, version, values, marker in helm_releases(repo):
+        if not (repo / marker).is_file():
+            continue
+        value_path = repo / values if (repo / values).is_file() else config_repo / values
+        if not value_path.is_file():
+            raise ValueError(f'Missing Helm values: {values}')
+        stamp = hashlib.sha256(f'{chart}|{version}|{file_hash(value_path)}'.encode()).hexdigest()
+        helm_inputs[name] = stamp
+        live = next((h for h in live_helm if h['name'] == name and h['namespace'] == namespace), None)
+        if (
+            mode == 'full'
+            or previous is None
+            or previous.get('helm_inputs', {}).get(name) != stamp
+            or live is None
+            or live.get('status') != 'deployed'
+            or live.get('chart') != f'{chart.split("/")[-1]}-{version}'
+        ):
+            helm_selected.append(name)
+    local_inputs = {}
+    for service in all_services:
+        candidates = [config_repo / service / '.env']
+        if service in active['compose']:
+            candidates.append(config_repo / '.env')
+            cfg = config_repo / service / 'config'
+            if cfg.is_dir():
+                candidates.extend(
+                    p for p in cfg.rglob('*') if p.is_file() and p.suffix in ('.yaml', '.yml', '.json', '.conf')
+                )
+        local_inputs[service] = hashlib.sha256(
+            '\n'.join(f'{p.relative_to(config_repo)}:{file_hash(p)}' for p in sorted(candidates)).encode()
+        ).hexdigest()
+    if previous is None:
+        if mode == 'changed':
+            raise ValueError('No successful baseline; run deploy in full mode first')
+        changed = set(all_services)
+        removed = []
+    else:
+        paths = output('git', '-C', str(repo), 'diff', '--name-only', previous['sha'], release['sha']).splitlines()
+        changed = {path.split('/')[0] for path in paths}
+        if any(path.startswith('.gitea/') for path in paths):
+            changed |= all_services
+        changed |= {s for s in all_services if previous.get('local_inputs', {}).get(s) != local_inputs[s]}
+        for file in tracked(repo):
+            service = file.split('/')[0]
+            if service not in all_services or not file.endswith(('.yaml', '.yml')):
+                continue
+            text = (repo / file).read_text()
+            if any(
+                image in text and previous.get('images', {}).get(image) != digest
+                for image, digest in release['images'].items()
+            ):
+                changed.add(service)
+        removed = sorted(
+            set(previous.get('active', {}).get('k8s', []) + previous.get('active', {}).get('compose', []))
+            - all_services
+        )
+        removed += [path for path in paths if '/k8s/' in path and not (repo / path).exists()]
+    if mode == 'full':
+        changed = set(all_services)
+    dependencies = json.loads((repo / '.gitea/deploy-dependencies.json').read_text())
+    while True:
+        expanded = changed | {dependent for service in changed for dependent in dependencies.get(service, [])}
+        if expanded == changed:
+            break
+        changed = expanded
+    return {
+        'version': 1,
+        'sha': release['sha'],
+        'images': release['images'],
+        'active': active,
+        'selected': {kind: sorted(set(services) & changed) for kind, services in active.items()},
+        'helm': helm_selected,
+        'helm_inputs': helm_inputs,
+        'local_inputs': local_inputs,
+        'removed': sorted(set(removed)),
+        'full_smoke': mode == 'full' or 'traefik' in changed,
+    }

.gitea/workflows/deploy-stage.sh

@@ -0,0 +1,10 @@
+#!/usr/bin/env bash
+set -euo pipefail
+source "${REPO:?}/.gitea/workflows/deploy-lib.sh"
+case "${1:?stage required}" in
+  workload-count)
+    select_manifests >/dev/null
+    selected_workload_refs | sort -u | wc -l
+    ;;
+  *) run_stage "$1" ;;
+esac

.gitea/workflows/deploy.yaml

@@ -0,0 +1,140 @@
+name: deploy
+
+on:
+  workflow_run:
+    workflows: [ci]
+    branches: [main]
+    types: [completed]
+  workflow_dispatch:
+    inputs:
+      deploy_ref:
+        description: "Commit already checked by successful main CI (main or SHA)"
+        default: main
+        required: true
+      deploy_mode:
+        description: "First deploy requires full; plan changes no production resources"
+        type: choice
+        options: [changed, full, plan]
+        default: changed
+      refresh_images:
+        description: "Explicitly refresh mutable third-party Compose tags"
+        type: boolean
+        default: false
+
+permissions:
+  contents: read
+  actions: read
+
+concurrency:
+  group: deploy-main
+  cancel-in-progress: false
+
+env:
+  DEPLOY_HOST: ${{ vars.DEPLOY_HOST || secrets.DEPLOY_HOST }}
+  DEPLOY_PORT: ${{ vars.DEPLOY_PORT || secrets.DEPLOY_PORT }}
+  DEPLOY_USER: ${{ vars.DEPLOY_USER || secrets.DEPLOY_USER }}
+  DEPLOY_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
+  DEPLOY_KNOWN_HOSTS: ${{ vars.DEPLOY_KNOWN_HOSTS }}
+  DEPLOY_RUN_ID: ${{ github.run_id }}-${{ github.run_attempt || 1 }}
+  DEPLOY_MODE: ${{ inputs.deploy_mode || 'changed' }}
+  REFRESH_IMAGES: ${{ inputs.refresh_images && 'true' || 'false' }}
+
+jobs:
+  gate:
+    if: >-
+      github.ref == 'refs/heads/main' &&
+      (vars.AUTODEPLOY == 'true' || github.event_name == 'workflow_dispatch') &&
+      (github.event_name != 'workflow_run' ||
+       (github.event.workflow_run.conclusion == 'success' && github.event.workflow_run.head_branch == 'main'))
+    runs-on: homelab
+    timeout-minutes: 10
+    outputs:
+      sha: ${{ steps.release.outputs.sha }}
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+        with:
+          fetch-depth: 0
+      - name: Check successful CI and download the exact commit release
+        id: release
+        env:
+          GITEA_TOKEN: ${{ github.token }}
+          DEPLOY_REF: ${{ inputs.deploy_ref || 'main' }}
+          EVENT_SHA: ${{ github.event.workflow_run.head_sha }}
+        run: python3 .gitea/workflows/release.py gate --ref "$DEPLOY_REF" --event-sha "$EVENT_SHA"
+      - name: Submit durable deploy to workstation
+        run: bash .gitea/workflows/ssh-run.sh start
+      - name: Write the request result
+        if: always()
+        env:
+          REQUEST_RESULT: ${{ job.status }}
+          CHECKED_SHA: ${{ steps.release.outputs.sha }}
+        run: |
+          if [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
+            printf '## Deploy request\n\n- Result: **%s**\n- Checked commit: %s\n- Mode: %s\n' "$REQUEST_RESULT" "${CHECKED_SHA:-not checked}" "$DEPLOY_MODE" >>"$GITHUB_STEP_SUMMARY"
+            if [ "$REQUEST_RESULT" != success ]; then
+              echo 'Open the failed step log. If SSH submission failed, check the remote controller state.' >>"$GITHUB_STEP_SUMMARY"
+            fi
+          fi
+
+  apply:
+    needs: [gate]
+    runs-on: homelab
+    timeout-minutes: 100
+    steps:
+      - name: Checkout checked commit
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+        with:
+          ref: ${{ needs.gate.outputs.sha }}
+      - name: Follow validation and sequential Kubernetes / Compose apply
+        run: bash .gitea/workflows/ssh-run.sh apply
+      - name: Write the deploy result
+        if: always()
+        run: |
+          if [ -f .gitea/workflows/ssh-run.sh ]; then
+            bash .gitea/workflows/ssh-run.sh summary
+          elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
+            echo 'Source checkout failed. The remote deploy state is unknown. Check the job log.' >>"$GITHUB_STEP_SUMMARY"
+          fi
+
+  verify:
+    needs: [gate, apply]
+    if: always() && needs.gate.result == 'success'
+    runs-on: homelab
+    timeout-minutes: 130
+    steps:
+      - name: Checkout checked commit
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+        with:
+          ref: ${{ needs.gate.outputs.sha }}
+      - name: Follow workload verification and recovery
+        run: bash .gitea/workflows/ssh-run.sh verify
+      - name: Write the deploy result
+        if: always()
+        run: |
+          if [ -f .gitea/workflows/ssh-run.sh ]; then
+            bash .gitea/workflows/ssh-run.sh summary
+          elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
+            echo 'Source checkout failed. The remote deploy state is unknown. Check the job log.' >>"$GITHUB_STEP_SUMMARY"
+          fi
+
+  smoke:
+    needs: [gate, verify]
+    if: always() && needs.gate.result == 'success'
+    runs-on: homelab
+    timeout-minutes: 15
+    steps:
+      - name: Checkout checked commit
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+        with:
+          ref: ${{ needs.gate.outputs.sha }}
+      - name: Follow public route checks
+        run: bash .gitea/workflows/ssh-run.sh smoke
+      - name: Write the deploy result
+        if: always()
+        run: |
+          if [ -f .gitea/workflows/ssh-run.sh ]; then
+            bash .gitea/workflows/ssh-run.sh summary
+          elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
+            echo 'Source checkout failed. The remote deploy state is unknown. Check the job log.' >>"$GITHUB_STEP_SUMMARY"
+          fi

.gitea/workflows/install-ci-tools.sh

@@ -0,0 +1,285 @@
+#!/usr/bin/env bash
+# Installs the pinned CI tools into "$TOOLS_DIR/bin" and echoes that directory
+# on stdout, so callers can do:
+#
+#   export PATH="$(bash .gitea/workflows/install-ci-tools.sh kubeconform shellcheck):$PATH"
+#
+# Versions come from tool-versions.env next to this script and are kept fresh by
+# Renovate. Re-running is cheap: an already-installed tool at the pinned version
+# is left alone.
+set -euo pipefail
+
+here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=tool-versions.env
+. "$here/tool-versions.env"
+
+TOOLS_DIR="${TOOLS_DIR:-${XDG_CACHE_HOME:-$HOME/.cache}/homelab-ci}"
+BIN_DIR="$TOOLS_DIR/bin"
+mkdir -p "$BIN_DIR"
+# A runner may accept overlapping workflows even though each workflow is sequential.
+exec 9>"$TOOLS_DIR/install.lock"
+flock -w 300 9
+export UV_TOOL_DIR="$TOOLS_DIR/uv-tools"
+export UV_CACHE_DIR="$TOOLS_DIR/uv-cache"
+# The just-installed tools must resolve inside this script too: callers only
+# prepend BIN_DIR to PATH after the script exits, so a bare `uv` below would
+# miss the binary install_uv just placed (exit 127 on a clean runner).
+export PATH="$BIN_DIR:$PATH"
+
+arch="$(uname -m)"
+# Upstream projects disagree on arch spelling: kubeconform and actionlint use
+# Go names (amd64/arm64), shellcheck uses uname names (x86_64/aarch64), node
+# uses neither (x64/arm64), and hadolint mixes the two in a single release
+# (x86_64 but arm64).
+case "$arch" in
+  x86_64 | amd64)
+    goarch=amd64
+    sharch=x86_64
+    nodearch=x64
+    hadolintarch=x86_64
+    ;;
+  aarch64 | arm64)
+    goarch=arm64
+    sharch=aarch64
+    nodearch=arm64
+    hadolintarch=arm64
+    ;;
+  *)
+    echo "install-ci-tools: unsupported architecture: $arch" >&2
+    exit 1
+    ;;
+esac
+
+fetch() {
+  # fetch <url> <dest>
+  if command -v curl >/dev/null 2>&1; then
+    curl -sSLf --connect-timeout 15 --max-time 120 --retry 3 -o "$2" "$1"
+  elif command -v wget >/dev/null 2>&1; then
+    wget -q -O "$2" "$1"
+  else
+    echo "install-ci-tools: neither curl nor wget is available" >&2
+    exit 1
+  fi
+}
+
+# resolve <command>
+# Absolute path to use for invoking a tool: the copy in BIN_DIR when present,
+# otherwise the name for PATH lookup. Every version check and every in-script
+# invocation goes through this, so a tool missing from both places reads as
+# "not installed" instead of dying with 127 under `set -e`.
+resolve() {
+  if [ -x "$BIN_DIR/$1" ]; then
+    printf '%s' "$BIN_DIR/$1"
+  else
+    printf '%s' "$1"
+  fi
+}
+
+# installed_version <command>
+# Prints the version of an already-installed tool, or nothing. Each tool spells
+# its version flag differently, hence the case.
+installed_version() {
+  local bin out
+  bin="$(resolve "$1")"
+  if ! command -v "$bin" >/dev/null 2>&1; then
+    return 0
+  fi
+  case "$1" in
+    kubeconform) out="$("$bin" -v 2>/dev/null | head -1 || true)" ;;
+    *) out="$("$bin" --version 2>/dev/null | head -1 || true)" ;;
+  esac
+  printf '%s' "$out"
+}
+
+# at_version <command> <expected>
+at_version() {
+  local version expected="${2#v}"
+  version="$(installed_version "$1")"
+  if [[ "$version" =~ (^|[^0-9.])v?([0-9]+(\.[0-9]+)+) ]]; then
+    [ "${BASH_REMATCH[2]}" = "$expected" ]
+  else
+    return 1
+  fi
+}
+
+install_kubeconform() {
+  if at_version kubeconform "v${KUBECONFORM_VERSION}"; then
+    return 0
+  fi
+  local tmp
+  tmp="$(mktemp -d)"
+  fetch "https://github.com/yannh/kubeconform/releases/download/v${KUBECONFORM_VERSION}/kubeconform-linux-${goarch}.tar.gz" \
+    "$tmp/kubeconform.tar.gz"
+  tar -xzf "$tmp/kubeconform.tar.gz" -C "$tmp" kubeconform
+  install -m 0755 "$tmp/kubeconform" "$BIN_DIR/kubeconform"
+  rm -rf "$tmp"
+}
+
+install_shellcheck() {
+  if at_version shellcheck "${SHELLCHECK_VERSION}"; then
+    return 0
+  fi
+  local tmp
+  tmp="$(mktemp -d)"
+  fetch "https://github.com/koalaman/shellcheck/releases/download/v${SHELLCHECK_VERSION}/shellcheck-v${SHELLCHECK_VERSION}.linux.${sharch}.tar.xz" \
+    "$tmp/shellcheck.tar.xz"
+  tar -xJf "$tmp/shellcheck.tar.xz" -C "$tmp" --strip-components=1 "shellcheck-v${SHELLCHECK_VERSION}/shellcheck"
+  install -m 0755 "$tmp/shellcheck" "$BIN_DIR/shellcheck"
+  rm -rf "$tmp"
+}
+
+install_jq() {
+  if at_version jq "${JQ_VERSION}"; then
+    return 0
+  fi
+  fetch "https://github.com/jqlang/jq/releases/download/jq-${JQ_VERSION}/jq-linux-${goarch}" \
+    "$BIN_DIR/jq"
+  chmod 0755 "$BIN_DIR/jq"
+}
+
+install_uv() {
+  if at_version uv "${UV_VERSION}"; then
+    return 0
+  fi
+  local tmp
+  tmp="$(mktemp -d)"
+  # uv release tags carry no leading v, unlike every other tool installed here.
+  fetch "https://github.com/astral-sh/uv/releases/download/${UV_VERSION}/uv-${sharch}-unknown-linux-gnu.tar.gz" \
+    "$tmp/uv.tar.gz"
+  tar -xzf "$tmp/uv.tar.gz" -C "$tmp" --strip-components=1 "uv-${sharch}-unknown-linux-gnu/uv"
+  install -m 0755 "$tmp/uv" "$BIN_DIR/uv"
+  rm -rf "$tmp"
+}
+
+install_hadolint() {
+  if at_version hadolint "${HADOLINT_VERSION}"; then
+    return 0
+  fi
+  # A bare binary, no archive: hadolint ships one file per platform.
+  fetch "https://github.com/hadolint/hadolint/releases/download/v${HADOLINT_VERSION}/hadolint-linux-${hadolintarch}" \
+    "$BIN_DIR/hadolint"
+  chmod 0755 "$BIN_DIR/hadolint"
+}
+
+# ruff and yamllint both come from PyPI as wheels, which uv unpacks for us.
+install_uv_tool() {
+  # <package> <pinned version>
+  if at_version "$1" "$2"; then
+    return 0
+  fi
+  install_uv
+  UV_TOOL_BIN_DIR="$BIN_DIR" "$BIN_DIR/uv" tool install --force "$1==$2" >/dev/null
+}
+
+install_ruff() {
+  install_uv_tool ruff "${RUFF_VERSION}"
+}
+
+install_yamllint() {
+  install_uv_tool yamllint "${YAMLLINT_VERSION}"
+}
+
+install_pip_audit() {
+  install_uv_tool pip-audit "${PIP_AUDIT_VERSION}"
+}
+
+install_prettier() {
+  install_node
+  if at_version prettier "${PRETTIER_VERSION}"; then
+    return 0
+  fi
+  # Not a standalone binary: prettier's entry point requires ../package.json
+  # relative to its own real path, so the package directory has to survive
+  # next to it. Hence a versioned directory plus a relative symlink, rather
+  # than copying the one file out as the other installers do.
+  local dir="$BIN_DIR/prettier-${PRETTIER_VERSION}"
+  if [ ! -f "$dir/package/package.json" ]; then
+    rm -rf "$dir"
+    mkdir -p "$dir"
+    fetch "https://registry.npmjs.org/prettier/-/prettier-${PRETTIER_VERSION}.tgz" "$dir/prettier.tgz"
+    tar -xzf "$dir/prettier.tgz" -C "$dir"
+    rm -f "$dir/prettier.tgz"
+    # npm strips the exec bit from bin/ on the way into the tarball.
+    chmod 0755 "$dir/package/bin/prettier.cjs"
+  fi
+  # Relative, so the whole tree stays valid if TOOLS_DIR is relocated.
+  ln -sfn "prettier-${PRETTIER_VERSION}/package/bin/prettier.cjs" "$BIN_DIR/prettier"
+}
+
+install_node() {
+  # npm gets checked by running it, not by looking it up: what matters is that
+  # it answers, so a stub, a half-removed Arch package or a name that resolves
+  # to something broken all have to read as "not installed". The runner's npm
+  # is a symlink into /usr/lib/node_modules/npm, which is exactly the kind of
+  # thing that disappears between runs.
+  if at_version node "v${NODE_VERSION}" && [ -n "$(installed_version npm)" ]; then
+    return 0
+  fi
+  # Same shape as prettier above: the tarball's bin/npm and bin/npx are links
+  # into lib/node_modules, so the whole tree has to survive next to them.
+  local dir="$BIN_DIR/node-${NODE_VERSION}"
+  if [ ! -x "$dir/bin/node" ]; then
+    rm -rf "$dir"
+    mkdir -p "$dir"
+    fetch "https://nodejs.org/dist/v${NODE_VERSION}/node-v${NODE_VERSION}-linux-${nodearch}.tar.xz" \
+      "$dir/node.tar.xz"
+    tar -xJf "$dir/node.tar.xz" -C "$dir" --strip-components=1 "node-v${NODE_VERSION}-linux-${nodearch}"
+    rm -f "$dir/node.tar.xz"
+  fi
+  # Relative, so the whole tree stays valid if TOOLS_DIR is relocated.
+  for bin in node npm npx; do
+    ln -sfn "node-${NODE_VERSION}/bin/${bin}" "$BIN_DIR/${bin}"
+  done
+}
+
+install_actionlint() {
+  if at_version actionlint "${ACTIONLINT_VERSION}"; then
+    return 0
+  fi
+  local tmp
+  tmp="$(mktemp -d)"
+  fetch "https://github.com/rhysd/actionlint/releases/download/v${ACTIONLINT_VERSION}/actionlint_${ACTIONLINT_VERSION}_linux_${goarch}.tar.gz" \
+    "$tmp/actionlint.tar.gz"
+  tar -xzf "$tmp/actionlint.tar.gz" -C "$tmp" actionlint
+  install -m 0755 "$tmp/actionlint" "$BIN_DIR/actionlint"
+  rm -rf "$tmp"
+}
+
+main() {
+  wanted=("$@")
+  if [ "${#wanted[@]}" -eq 0 ]; then
+    wanted=(node jq kubeconform shellcheck actionlint prettier ruff yamllint hadolint)
+  fi
+
+  for tool in "${wanted[@]}"; do
+    case "$tool" in
+      kubeconform) install_kubeconform ;;
+      shellcheck) install_shellcheck ;;
+      jq) install_jq ;;
+      actionlint) install_actionlint ;;
+      prettier) install_prettier ;;
+      ruff) install_ruff ;;
+      yamllint) install_yamllint ;;
+      pip-audit) install_pip_audit ;;
+      hadolint) install_hadolint ;;
+      node) install_node ;;
+      uv) install_uv ;;
+      *)
+        echo "install-ci-tools: unknown tool: $tool" >&2
+        exit 1
+        ;;
+    esac
+  done
+
+  for old in "$BIN_DIR"/node-* "$BIN_DIR"/prettier-*; do
+    [ -d "$old" ] || continue
+    case "$(basename "$old")" in
+      "node-$NODE_VERSION"|"prettier-$PRETTIER_VERSION") ;;
+      *) rm -rf "$old" ;;
+    esac
+  done
+  if [ -x "$BIN_DIR/uv" ]; then "$BIN_DIR/uv" cache prune >/dev/null; fi
+  printf '%s\n' "$BIN_DIR"
+}
+
+if [ "${BASH_SOURCE[0]}" = "$0" ]; then main "$@"; fi

.gitea/workflows/release.py

@@ -0,0 +1,415 @@
+#!/usr/bin/env python3
+"""CI release artifacts and the SHA-specific Gitea deployment gate (stdlib only)."""
+
+import argparse
+import hashlib
+import io
+import itertools
+import json
+import os
+import re
+import shutil
+import subprocess
+import sys
+import tempfile
+import urllib.error
+import urllib.parse
+import urllib.request
+import zipfile
+from pathlib import Path
+
+SHA = re.compile(r'[0-9a-f]{40}')
+DIGEST = re.compile(r'sha256:[0-9a-f]{64}')
+IMAGES = {
+    'error-pages': ('errorpages', 'errorpages/Dockerfile'),
+    'forust-homepage': ('homepages', 'homepages/Dockerfile.forust'),
+    'xdfnx-homepage': ('homepages', 'homepages/Dockerfile.xdfnx'),
+}
+
+# These images are released by the EDU application repository.
+EXTERNAL_IMAGES = {'gcr.forust.xyz/forust/session-keeper', 'gcr.forust.xyz/forust/webinar-checker'}
+
+
+def command(*args, **kwargs):
+    """Arguments are passed directly to the executable, never to a shell."""
+    return subprocess.check_output(args, text=True, **kwargs).strip()  # noqa: S603, S607
+
+
+def validate_release(data, sha=None):
+    if data.get('version') != 1 or not SHA.fullmatch(data.get('sha', '')):
+        raise ValueError('Invalid release version or SHA')
+    if sha is not None and data['sha'] != sha:
+        raise ValueError('Release SHA does not match the checked CI commit')
+    expected = {f'gcr.forust.xyz/forust/{name}' for name in IMAGES}
+    if set(data.get('images', {})) != expected:
+        raise ValueError('Release must contain all owned images')
+    if not all(DIGEST.fullmatch(value) for value in data['images'].values()):
+        raise ValueError('Release has an invalid image digest')
+    if set(data.get('inputs', {})) != expected or not all(
+        re.fullmatch(r'[0-9a-f]{64}', value) for value in data['inputs'].values()
+    ):
+        raise ValueError('Release has invalid build input fingerprints')
+    return data
+
+
+class NoRedirect(urllib.request.HTTPRedirectHandler):
+    def redirect_request(self, _req, _fp, _code, _msg, _headers, _newurl):
+        return None
+
+
+class Gitea:
+    def __init__(self):
+        self.origin = os.environ['GITHUB_SERVER_URL'].rstrip('/')
+        if urllib.parse.urlsplit(self.origin).scheme != 'https':
+            raise ValueError('Gitea API must use HTTPS')
+        self.repository = os.environ['GITHUB_REPOSITORY']
+        if not re.fullmatch(r'[\w.-]+/[\w.-]+', self.repository):
+            raise ValueError('Invalid Gitea repository')
+        self.token = os.environ['GITEA_TOKEN']
+        self.base = f'{self.origin}/api/v1/repos/{self.repository}'
+
+    def request(self, url, *, archive=False):
+        if not url.startswith(self.base + '/'):
+            raise ValueError('Refusing to send the Actions token to another origin')
+        req = urllib.request.Request(url, headers={'Authorization': f'token {self.token}'})  # noqa: S310 -- HTTPS origin validated above
+        opener = urllib.request.build_opener(NoRedirect())
+        try:
+            response = opener.open(req, timeout=30)  # noqa: S310
+        except urllib.error.HTTPError as error:
+            if not archive or error.code not in (301, 302, 303, 307, 308):
+                raise RuntimeError(f'Gitea API returned HTTP {error.code}') from None
+            target = urllib.parse.urljoin(url, error.headers['Location'])
+            if urllib.parse.urlsplit(target).scheme != 'https':
+                raise ValueError('Artifact redirect must use HTTPS') from None
+            # Signed storage redirects must never receive the Gitea token.
+            response = urllib.request.urlopen(target, timeout=30)  # noqa: S310
+        with response:
+            payload = response.read(8 * 1024 * 1024 + 1)
+        if len(payload) > 8 * 1024 * 1024:
+            raise ValueError('Gitea response exceeds 8 MiB')
+        return payload if archive else json.loads(payload)
+
+    def pages(self, path, key, **params):
+        for page in range(1, 101):
+            query = urllib.parse.urlencode({**params, 'page': page, 'limit': 50})
+            data = self.request(f'{self.base}/{path}?{query}')
+            entries = data[key]
+            yield from entries
+            if len(entries) < 50:
+                return
+        raise RuntimeError('Gitea pagination limit exceeded')
+
+    def successful_runs(self, sha=None):
+        params = {'branch': 'main', 'status': 'success', 'exclude_pull_requests': 'true'}
+        if sha:
+            params['head_sha'] = sha
+        for run in self.pages('actions/workflows/ci.yaml/runs', 'workflow_runs', **params):
+            if (
+                run.get('status') == 'completed'
+                and run.get('conclusion') == 'success'
+                and run.get('head_branch') == 'main'
+                and run.get('event') in ('push', 'workflow_dispatch')
+                and (run.get('repository') or {}).get('full_name') == self.repository
+                and (run.get('head_repository') or run.get('repository') or {}).get('full_name') == self.repository
+                and (sha is None or run.get('head_sha') == sha)
+            ):
+                yield run
+
+    def release(self, run):
+        sha = run['head_sha']
+        jobs = list(self.pages(f'actions/runs/{run["id"]}/jobs', 'jobs'))
+        # A green workflow with a skipped build must not authorize a deploy.
+        if not any(job.get('name') == 'build' and job.get('conclusion') == 'success' for job in jobs):
+            raise ValueError('CI build job did not succeed')
+        artifacts = self.request(f'{self.base}/actions/runs/{run["id"]}/artifacts')['artifacts']
+        matching = [a for a in artifacts if a['name'] == f'release-{sha}' and not a.get('expired')]
+        if len(matching) != 1:
+            raise ValueError('CI release artifact is missing, expired or ambiguous; rerun CI')
+        blob = self.request(f'{self.base}/actions/artifacts/{matching[0]["id"]}/zip', archive=True)
+        with zipfile.ZipFile(io.BytesIO(blob)) as archive:
+            files = [entry for entry in archive.infolist() if not entry.is_dir()]
+            if len(files) != 1 or files[0].filename != 'release.json' or files[0].file_size > 256 * 1024:
+                raise ValueError('Unexpected release archive contents')
+            return validate_release(json.loads(archive.read(files[0])), sha)
+
+
+def fingerprint(context, dockerfile):
+    tree = command('git', 'ls-tree', '-r', 'HEAD', '--', context, dockerfile, '.gitea/workflows/release.py')
+    return hashlib.sha256(tree.encode()).hexdigest()
+
+
+def gate(output, requested_ref, event_sha):
+    command('git', 'fetch', '--quiet', 'origin', 'main')
+    if event_sha:
+        if not SHA.fullmatch(event_sha):
+            raise ValueError('Invalid workflow_run SHA')
+        sha = event_sha
+    else:
+        if requested_ref == 'main':
+            requested_ref = 'origin/main'
+        sha = command('git', 'rev-parse', '--verify', '--end-of-options', f'{requested_ref}^{{commit}}')
+    if not SHA.fullmatch(sha):
+        raise ValueError('Invalid deploy SHA')
+    command('git', 'merge-base', '--is-ancestor', sha, 'origin/main')
+    api = Gitea()
+    runs = list(api.successful_runs(sha))
+    if not runs:
+        raise ValueError(f'No successful main CI for {sha}; run CI before deploying')
+    release = api.release(max(runs, key=lambda run: run['id']))
+    output.write_text(json.dumps(release, indent=2) + '\n')
+    if os.environ.get('GITHUB_OUTPUT'):
+        with Path(os.environ['GITHUB_OUTPUT']).open('a') as stream:
+            stream.write(f'sha={sha}\n')
+    print(f'CI gate accepted {sha}')
+
+
+def build_images(output, report):
+    sha = command('git', 'rev-parse', 'HEAD')
+    if sha != os.environ['GITHUB_SHA'] or not SHA.fullmatch(sha):
+        raise ValueError('Build checkout does not match GITHUB_SHA')
+    report['phase'] = 'Find a successful CI release'
+    api = Gitea()
+    previous = None
+    for run in sorted(itertools.islice(api.successful_runs(), 50), key=lambda item: item['id'], reverse=True):
+        if str(run['id']) == os.environ.get('GITHUB_RUN_ID'):
+            continue
+        try:
+            previous = api.release(run)
+            break
+        except ValueError:
+            # Expired artifacts only cost a rebuild; mutable tags are never a fallback.
+            continue
+    docker_config = tempfile.mkdtemp(prefix='homelab-registry-')
+    builder_config = Path.home() / '.cache/homelab-ci/buildx'
+    builder_config.mkdir(parents=True, exist_ok=True)
+    env = {**os.environ, 'DOCKER_CONFIG': docker_config, 'BUILDX_CONFIG': str(builder_config)}
+    try:
+        report['phase'] = 'Registry login'
+        subprocess.run(  # noqa: S603, S607
+            [
+                shutil.which('docker') or '/usr/bin/docker',
+                'login',
+                'gcr.forust.xyz',
+                '-u',
+                os.environ['REGISTRY_USERNAME'],
+                '--password-stdin',
+            ],
+            input=os.environ['REGISTRY_PASSWORD'],
+            text=True,
+            check=True,
+            env=env,
+        )
+        report['phase'] = 'Prepare the builder'
+        builder = 'homelab-ci'
+        versions = dict(
+            re.findall(r'^([A-Z_]+)="([^"\n]+)"$', Path('.gitea/workflows/tool-versions.env').read_text(), re.MULTILINE)
+        )
+        image = versions['BUILDKIT_IMAGE']
+        signature = builder_config / 'homelab-ci-image'
+        exists = (
+            subprocess.run(  # noqa: S603
+                [shutil.which('docker') or '/usr/bin/docker', 'buildx', 'inspect', builder],
+                capture_output=True,
+                env=env,
+            ).returncode
+            == 0
+        )
+        if exists and (not signature.exists() or signature.read_text().strip() != image):
+            command('docker', 'buildx', 'rm', '--keep-state', builder, env=env)
+            exists = False
+        if not exists:
+            command(
+                'docker',
+                'buildx',
+                'create',
+                '--name',
+                builder,
+                '--driver',
+                'docker-container',
+                '--driver-opt',
+                f'image={image}',
+                '--buildkitd-config',
+                '.gitea/runner/buildkitd.toml',
+                env=env,
+            )
+            signature.write_text(image + '\n')
+        release = {'version': 1, 'sha': sha, 'images': {}, 'inputs': {}}
+        report['images'] = release['images']
+        for name, (context, dockerfile) in IMAGES.items():
+            report['phase'] = f'Build or reuse {name}'
+            report['current'] = name
+            image = f'gcr.forust.xyz/forust/{name}'
+            inputs = fingerprint(context, dockerfile)
+            old_digest = (previous or {}).get('images', {}).get(image)
+            exists = False
+            if old_digest and previous['inputs'].get(image) == inputs:
+                exists = (
+                    subprocess.run(  # noqa: S603, S607
+                        [
+                            shutil.which('docker') or '/usr/bin/docker',
+                            'buildx',
+                            'imagetools',
+                            'inspect',
+                            f'{image}@{old_digest}',
+                        ],
+                        capture_output=True,
+                        env=env,
+                        timeout=60,
+                    ).returncode
+                    == 0
+                )
+            if exists:
+                print(f'Reuse {name}: inputs unchanged')
+                digest = old_digest
+                report['reused'].append(name)
+            else:
+                print(f'Build {name}', flush=True)
+                metadata = Path(docker_config) / 'metadata.json'
+                command(
+                    'docker',
+                    'buildx',
+                    'build',
+                    '--builder',
+                    builder,
+                    '--push',
+                    '--platform',
+                    'linux/amd64',
+                    '--provenance=false',
+                    '--cache-from',
+                    f'type=registry,ref={image}:buildcache',
+                    '--cache-to',
+                    f'type=registry,ref={image}:buildcache,mode=max',
+                    '--tag',
+                    f'{image}:sha-{sha}',
+                    '--metadata-file',
+                    str(metadata),
+                    '--file',
+                    dockerfile,
+                    context,
+                    env=env,
+                )
+                digest = json.loads(metadata.read_text())['containerimage.digest']
+                report['built'].append(name)
+            release['images'][image] = digest
+            release['inputs'][image] = inputs
+        validate_release(release, sha)
+        output.write_text(json.dumps(release, indent=2) + '\n')
+        report['current'] = None
+        report['phase'] = 'Release file saved'
+    finally:
+        # Cleanup errors must neither leak credentials nor mask the original build error.
+        try:
+            subprocess.run(  # noqa: S603
+                [
+                    shutil.which('docker') or '/usr/bin/docker',
+                    'buildx',
+                    'prune',
+                    '--builder',
+                    'homelab-ci',
+                    '--force',
+                    '--max-used-space',
+                    '1gb',
+                ],
+                env=env,
+                timeout=60,
+            )
+        except (OSError, subprocess.TimeoutExpired):
+            print('CI builder cache cleanup deferred', flush=True)
+        finally:
+            shutil.rmtree(docker_config)
+
+
+def write_summary(lines):
+    path = os.environ.get('GITHUB_STEP_SUMMARY')
+    if path:
+        try:
+            with Path(path).open('a') as stream:
+                stream.write('\n'.join(lines) + '\n\n')
+        except OSError:
+            print('WARNING: cannot write the job summary')
+
+
+def check_summary():
+    lines = [
+        f'## {os.environ["SUMMARY_CHECK"]}',
+        '',
+        f'- Commit: `{os.environ.get("GITHUB_SHA", "unknown")}`',
+        f'- Result: **{os.environ["SUMMARY_RESULT"]}**',
+    ]
+    if os.environ.get('SUMMARY_FAILED_STEP'):
+        lines.append(f'- Failed step: {os.environ["SUMMARY_FAILED_STEP"]}')
+    if os.environ['SUMMARY_RESULT'] != 'success':
+        lines.append('- Open the failed step log for the error details.')
+    write_summary(lines)
+
+
+def build(output):
+    report = {'phase': 'Check the source commit', 'current': None, 'built': [], 'reused': [], 'images': {}}
+    result = 'failure'
+    try:
+        build_images(output, report)
+        result = 'success'
+    finally:
+        lines = [
+            f'## Image release `{os.environ.get("GITHUB_SHA", "unknown")}`',
+            '',
+            f'- Result: **{result}**',
+            f'- Last stage: {report["phase"]}',
+        ]
+        if result == 'failure':
+            lines.append('- No release from this build can be deployed. Open the failed step log.')
+        if report['current']:
+            lines.append(f'- Image at the failure: `{report["current"]}`')
+        for title, key in (('Built', 'built'), ('Reused from successful CI', 'reused')):
+            lines.extend(['', f'### {title}'])
+            lines.extend(f'- `{name}`' for name in report[key])
+            if not report[key]:
+                lines.append('- None')
+        lines.extend(['', '### Completed image digests'])
+        lines.extend(f'- `{image}@{digest}`' for image, digest in report['images'].items())
+        if not report['images']:
+            lines.append('- None')
+        write_summary(lines)
+
+
+def render(stream, destination):
+    release = validate_release(json.loads(Path(os.environ['RELEASE_FILE']).read_text()), os.environ['DEPLOY_SHA'])
+    image_line = re.compile(
+        r"^(\s*(?:-\s*)?image:\s*)(['\"]?)(gcr\.forust\.xyz/forust/[\w.-]+)(?::[\w.-]+|@sha256:[0-9a-f]{64})\2(\s*(?:#.*)?)$"
+    )
+    rendered = []
+    for line in stream:
+        match = image_line.fullmatch(line.rstrip('\n'))
+        if match:
+            prefix, quote, image, tail = match.groups()
+            if image in EXTERNAL_IMAGES and f'{image}@sha256:' in line:
+                rendered.append(line)
+                continue
+            if image not in release['images']:
+                raise ValueError(f'Owned image missing from checked release: {image}')
+            line = f'{prefix}{quote}{image}@{release["images"][image]}{quote}{tail}\n'
+        elif re.match(r'\s*(?:-\s*)?image:', line) and 'gcr.forust.xyz/forust/' in line:
+            raise ValueError('Unsupported owned image syntax; refusing to apply a mutable tag')
+        rendered.append(line)
+    destination.writelines(rendered)
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument('action', choices=('build', 'gate', 'render', 'check-summary'))
+    parser.add_argument('--output', type=Path, default=Path('release.json'))
+    parser.add_argument('--ref', default='main')
+    parser.add_argument('--event-sha', default='')
+    args = parser.parse_args()
+    if args.action == 'check-summary':
+        check_summary()
+    elif args.action == 'render':
+        render(sys.stdin, sys.stdout)
+    elif args.action == 'gate':
+        gate(args.output, args.ref, args.event_sha)
+    else:
+        build(args.output)
+
+
+if __name__ == '__main__':
+    main()

.gitea/workflows/renovate-ci.yaml

@@ -0,0 +1,90 @@
+name: renovate-ci
+
+on:
+  pull_request:
+    paths:
+      - "renovate/**"
+      - ".gitea/workflows/renovate-ci.yaml"
+      - ".gitea/workflows/sync-renovate-configmap.sh"
+      - ".gitea/workflows/compose-lint.sh"
+      - ".gitea/workflows/install-ci-tools.sh"
+      - ".gitea/workflows/tool-versions.env"
+  push:
+    branches:
+      - main
+    paths:
+      - "renovate/**"
+      - ".gitea/workflows/renovate-ci.yaml"
+      - ".gitea/workflows/sync-renovate-configmap.sh"
+      - ".gitea/workflows/compose-lint.sh"
+      - ".gitea/workflows/install-ci-tools.sh"
+      - ".gitea/workflows/tool-versions.env"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  validate-renovate:
+    runs-on: homelab
+    timeout-minutes: 20
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+
+      # renovate/k8s/cronjob.yaml is the single source of truth for the image tag,
+      # so the same version that runs in the cluster is the one validated here.
+      - name: Resolve the deployed Renovate image
+        id: image
+        shell: bash
+        run: |
+          set -euo pipefail
+          image="$(sed -n 's|.*image:[[:space:]]*\(renovate/renovate:[^[:space:]]*\).*|\1|p' \
+            renovate/k8s/cronjob.yaml | head -1)"
+          if [ -z "$image" ]; then
+            echo "::error::no renovate/renovate image found in renovate/k8s/cronjob.yaml"
+            exit 1
+          fi
+          echo "using $image"
+          echo "image=$image" >> "$GITHUB_OUTPUT"
+
+      - name: Validate Renovate repository config
+        shell: bash
+        run: |
+          set -euo pipefail
+          docker run --rm \
+            -v "$PWD/renovate:/opt/renovate:ro" \
+            -e RENOVATE_CONFIG_FILE=/opt/renovate/renovate.json \
+            "${{ steps.image.outputs.image }}" \
+            renovate-config-validator /opt/renovate/renovate.json
+
+      # The CronJob cannot read the repository, so renovate/k8s/configmap.yaml
+      # carries an inlined copy of the config. Fail if it no longer matches.
+      - name: Check the generated Renovate ConfigMap
+        shell: bash
+        run: |
+          set -euo pipefail
+          ./.gitea/workflows/sync-renovate-configmap.sh --check
+
+      - name: Validate Renovate Kubernetes manifests
+        shell: bash
+        run: |
+          set -euo pipefail
+          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh kubeconform)"
+          export PATH="$tools_dir:$PATH"
+          kubeconform \
+            -strict \
+            -ignore-missing-schemas \
+            -summary \
+            renovate/k8s/namespace.yaml \
+            renovate/k8s/configmap.yaml \
+            renovate/k8s/cronjob.yaml
+
+      - name: Validate Renovate Compose file
+        shell: bash
+        run: |
+          set -euo pipefail
+          source .gitea/workflows/compose-lint.sh
+          mapfile -t safe_flags < <(compose_safe_flags)
+          validate_compose_file renovate/renovate-compose.yaml \
+            ${safe_flags[@]+"${safe_flags[@]}"}

.gitea/workflows/renovate-run.yaml

@@ -0,0 +1,92 @@
+name: renovate-run
+
+on:
+  workflow_dispatch:
+    inputs:
+      repositories:
+        description: "Repositories to scan (comma-separated)"
+        required: false
+        default: "forust/homelab"
+      log_level:
+        description: "Renovate log level"
+        required: false
+        default: "info"
+        type: choice
+        options:
+          - info
+          - debug
+      dry_run:
+        description: "Plan only, do not open or update PRs"
+        required: false
+        default: false
+        type: boolean
+
+# Renovate writes through its own bot PAT, passed in as RENOVATE_TOKEN, so the
+# Actions token is only ever used to read the checkout.
+permissions:
+  contents: read
+
+concurrency:
+  group: renovate-run
+  cancel-in-progress: false
+
+jobs:
+  run-renovate:
+    runs-on: homelab
+    timeout-minutes: 60
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+
+      # renovate/k8s/cronjob.yaml is the single source of truth for the image tag.
+      # Reading it here means this workflow validates and runs the exact version
+      # that is deployed, instead of a copy that silently goes stale.
+      - name: Resolve the deployed Renovate image
+        id: image
+        shell: bash
+        run: |
+          set -euo pipefail
+          image="$(sed -n 's|.*image:[[:space:]]*\(renovate/renovate:[^[:space:]]*\).*|\1|p' \
+            renovate/k8s/cronjob.yaml | head -1)"
+          if [ -z "$image" ]; then
+            echo "::error::no renovate/renovate image found in renovate/k8s/cronjob.yaml"
+            exit 1
+          fi
+          echo "using $image"
+          echo "image=$image" >> "$GITHUB_OUTPUT"
+
+      - name: Validate Renovate config
+        shell: bash
+        run: |
+          set -euo pipefail
+          docker run --rm \
+            -v "$PWD/renovate/renovate.json:/opt/renovate/renovate.json:ro" \
+            -e RENOVATE_CONFIG_FILE=/opt/renovate/renovate.json \
+            "${{ steps.image.outputs.image }}" \
+            renovate-config-validator
+
+      - name: Run Renovate
+        shell: bash
+        env:
+          RENOVATE_TOKEN: ${{ secrets.RENOVATE_TOKEN }}
+          RENOVATE_GITHUB_COM_TOKEN: ${{ secrets.RENOVATE_GITHUB_COM_TOKEN }}
+          RENOVATE_REPOSITORIES: ${{ inputs.repositories }}
+          RENOVATE_DRY_RUN: ${{ inputs.dry_run && 'full' || '' }}
+          LOG_LEVEL: ${{ inputs.log_level }}
+        run: |
+          set -euo pipefail
+
+          : "${RENOVATE_TOKEN:?missing RENOVATE_TOKEN secret — add a renovate-bot PAT in repo/org Actions secrets}"
+
+          docker run --rm \
+            -v "$PWD/renovate/renovate.json:/opt/renovate/renovate.json:ro" \
+            -e RENOVATE_PLATFORM=gitea \
+            -e RENOVATE_ENDPOINT=https://git.forust.xyz/api/v1 \
+            -e RENOVATE_TOKEN="$RENOVATE_TOKEN" \
+            -e RENOVATE_GITHUB_COM_TOKEN="${RENOVATE_GITHUB_COM_TOKEN:-}" \
+            -e RENOVATE_REPOSITORIES="${RENOVATE_REPOSITORIES:-forust/homelab}" \
+            -e RENOVATE_DRY_RUN="${RENOVATE_DRY_RUN:-}" \
+            -e RENOVATE_CONFIG_FILE=/opt/renovate/renovate.json \
+            -e RENOVATE_BASE_DIR=/tmp/renovate \
+            -e LOG_LEVEL="${LOG_LEVEL:-info}" \
+            "${{ steps.image.outputs.image }}"

.gitea/workflows/secret-references.jq

@@ -0,0 +1,13 @@
+# kubectl emits a List for files containing multiple resources.
+(if .kind == "List" then .items[] else . end)
+| (.metadata.namespace // "default") as $ns
+| [
+    (.. | objects
+      | (.secretRef? // empty), (.secretKeyRef? // empty), (.secret? // empty)
+      | select(.optional != true)
+      | .name // .secretName // empty),
+    (.. | objects | .imagePullSecrets[]?.name)
+  ]
+| unique[]
+| select(. != null and . != "")
+| "\($ns) \(.)"

.gitea/workflows/ssh-run.sh

@@ -0,0 +1,70 @@
+#!/usr/bin/env bash
+# The SSH client submits once and follows durable stages on workstation.
+set -euo pipefail
+: "${DEPLOY_HOST:?missing DEPLOY_HOST}"
+: "${DEPLOY_USER:?missing DEPLOY_USER}"
+: "${DEPLOY_KEY:?missing DEPLOY_SSH_KEY}"
+: "${DEPLOY_KNOWN_HOSTS:?configure pinned DEPLOY_KNOWN_HOSTS}"
+: "${DEPLOY_RUN_ID:?missing DEPLOY_RUN_ID}"
+[[ "$DEPLOY_USER" =~ ^[A-Za-z_][A-Za-z0-9_.-]*$ ]] || exit 1
+[[ "$DEPLOY_HOST" =~ ^[A-Za-z0-9_.:-]+$ ]] || exit 1
+[[ "$DEPLOY_RUN_ID" =~ ^[0-9]+-[0-9]+$ ]] || exit 1
+[[ "${DEPLOY_PORT:-22}" =~ ^[0-9]+$ ]] || exit 1
+key_dir="$(mktemp -d "${RUNNER_TEMP:-/tmp}/homelab-deploy-key.XXXXXXXX")"
+trap 'rm -rf "$key_dir"' EXIT
+chmod 700 "$key_dir"
+printf '%s\n' "$DEPLOY_KEY" >"$key_dir/key"
+printf '%s\n' "$DEPLOY_KNOWN_HOSTS" >"$key_dir/known_hosts"
+chmod 600 "$key_dir/key" "$key_dir/known_hosts"
+ssh_opts=(-i "$key_dir/key" -p "${DEPLOY_PORT:-22}" -o BatchMode=yes -o StrictHostKeyChecking=yes
+  -o "UserKnownHostsFile=$key_dir/known_hosts" -o ConnectTimeout=15
+  -o ServerAliveInterval=15 -o ServerAliveCountMax=4)
+controller=.local/lib/homelab-deploy/controller.py
+case "${1:?start, apply, verify, smoke or summary required}" in
+  start)
+    python3 - <<'PY' >"$key_dir/request.json"
+import json
+import os
+from pathlib import Path
+release = json.loads(Path('release.json').read_text())
+print(json.dumps({'release': release, 'mode': os.environ.get('DEPLOY_MODE', 'changed'),
+                  'refresh_images': os.environ.get('REFRESH_IMAGES', 'false') == 'true'}))
+PY
+    for attempt in 1 2 3; do
+      rc=0
+      # shellcheck disable=SC2029 # The run ID and operation are validated local arguments, not remote variables.
+      ssh "${ssh_opts[@]}" "$DEPLOY_USER@$DEPLOY_HOST" python3 "$controller" start "$DEPLOY_RUN_ID" <"$key_dir/request.json" || rc=$?
+      [ "$rc" -eq 0 ] && exit 0
+      [ "$rc" -eq 255 ] || exit "$rc"
+      sleep 5
+    done
+    exit "$rc"
+    ;;
+  apply|verify|smoke)
+    result=0
+    for attempt in 1 2 3; do
+      rc=0
+      # shellcheck disable=SC2029 # The run ID and operation are validated local arguments, not remote variables.
+      ssh "${ssh_opts[@]}" "$DEPLOY_USER@$DEPLOY_HOST" python3 "$controller" follow "$DEPLOY_RUN_ID" "$1" || rc=$?
+      [ "$rc" -eq 0 ] && break
+      [ "$rc" -eq 255 ] || { result="$rc"; break; }
+      echo "SSH disconnected; reconnecting to the existing deploy ($attempt/3)"
+      if [ "$attempt" -eq 3 ]; then result=255; break; fi
+      sleep 5
+    done
+    exit "$result"
+    ;;
+  summary)
+    if [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
+      rc=0
+      # shellcheck disable=SC2029 # The run ID is validated above.
+      ssh "${ssh_opts[@]}" "$DEPLOY_USER@$DEPLOY_HOST" python3 "$controller" summary "$DEPLOY_RUN_ID" >"$key_dir/deploy-summary.md" || rc=$?
+      if [ "$rc" -eq 0 ]; then
+        cat "$key_dir/deploy-summary.md" >>"$GITHUB_STEP_SUMMARY" || echo "WARNING: cannot write the deploy summary"
+      else
+        echo 'Deploy summary is unavailable. The SSH connection failed or the controller did not respond. Check the job log.' >>"$GITHUB_STEP_SUMMARY" || true
+      fi
+    fi
+    ;;
+  *) echo "Unknown SSH operation: $1" >&2; exit 1 ;;
+esac

.gitea/workflows/sync-renovate-configmap.sh

@@ -0,0 +1,55 @@
+#!/usr/bin/env bash
+# Regenerates renovate/k8s/configmap.yaml from renovate/renovate.json.
+#
+# renovate/renovate.json is the single source of truth: the CronJob, the Compose
+# file and the renovate-run workflow all mount that exact file. A ConfigMap cannot
+# read a file from the repository, so the same bytes are inlined here as a literal
+# block. This script keeps the copy honest:
+#
+#   .gitea/workflows/sync-renovate-configmap.sh          # rewrite in place
+#   .gitea/workflows/sync-renovate-configmap.sh --check  # fail if out of date
+#
+# renovate-ci runs the --check form on every PR and push, so a config change that
+# forgets to regenerate the ConfigMap cannot be merged.
+set -euo pipefail
+
+here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+repo="$(git -C "$here" rev-parse --show-toplevel)"
+
+src="$repo/renovate/renovate.json"
+dst="$repo/renovate/k8s/configmap.yaml"
+[ -f "$src" ] || {
+  echo "missing $src" >&2
+  exit 1
+}
+
+render() {
+  cat <<'HEADER'
+# GENERATED FILE - do not edit by hand.
+# Source: renovate/renovate.json
+# Regenerate: .gitea/workflows/sync-renovate-configmap.sh
+# Verify:     .gitea/workflows/sync-renovate-configmap.sh --check
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: renovate-config
+  namespace: renovate
+data:
+  renovate.json: |
+HEADER
+  sed 's/^/    /' "$src"
+}
+
+if [ "${1:-}" = "--check" ]; then
+  if ! diff -u "$dst" <(render) >/dev/null 2>&1; then
+    echo "ERROR: $dst is out of sync with renovate/renovate.json"
+    echo "Run: .gitea/workflows/sync-renovate-configmap.sh"
+    diff -u "$dst" <(render) || true
+    exit 1
+  fi
+  echo "renovate/k8s/configmap.yaml is in sync with renovate/renovate.json"
+  exit 0
+fi
+
+render >"$dst"
+echo "wrote $dst"

.gitea/workflows/tool-versions.env

@@ -0,0 +1,39 @@
+# Pinned versions of the CI tools installed by install-ci-tools.sh.
+# Renovate keeps these up to date (see customManagers in renovate/renovate.json).
+#
+# Every version here except NODE_VERSION matches what was already installed on
+# the runner, so pinning them changes what CI does not at all. It changes what
+# CI does when the runner is rebuilt with something else: today
+# install-ci-tools.sh finds the pinned version already on PATH and installs
+# nothing, and a runner that drifts gets the pinned one installed over it.
+#
+# The renovate image version is NOT pinned here: renovate/k8s/cronjob.yaml is the
+# single source of truth and the workflows read the tag from it, so there is
+# nothing to drift.
+ACTIONLINT_VERSION="1.7.7"
+SHELLCHECK_VERSION="0.11.0"
+KUBECONFORM_VERSION="0.8.0"
+PRETTIER_VERSION="3.8.1"
+RUFF_VERSION="0.16.8"
+YAMLLINT_VERSION="1.38.0"
+HADOLINT_VERSION="2.14.0"
+# pip-audit reads the advisory database over the network, so a floating version
+# would make the same commit report different things on different days. Pin it
+# like the rest: the advisories themselves are the moving part, not the tool.
+PIP_AUDIT_VERSION="2.10.1"
+# uv builds the throwaway venv the pytest job runs in, and unpacks the PyPI
+# wheels for ruff, yamllint and pip-audit.
+UV_VERSION="0.12.17"
+# node runs `npm ci` for the frontend tests and the npm audit, and it is the one
+# pin here that does NOT come from the runner: the runner's system node is a
+# rolling Arch package (it was node 26 with no npm at all when this was pinned),
+# and the panel image is node:22-alpine. Pinned to the image's major on purpose,
+# so the tree that gets tested is the tree that gets built. Renovate keeps this
+# in step with the Dockerfile's node: tag via the "node runtime" group.
+NODE_VERSION="22.23.3"
+
+# Secret-reference regression tests parse rendered Kubernetes objects.
+JQ_VERSION="1.8.1"
+
+# BuildKit is the only auxiliary CI container; jobs themselves stay on the host.
+BUILDKIT_IMAGE="moby/buildkit:v0.33.1"

.gitignore

@@ -2,14 +2,15 @@
 sync.ffs_lock
 .sync.ffs_db
 
+# Copyparty
+*.hist/
 
-
-# Volumes and data directories
+# Volumes, configs and data directories
 gitea/gitea-db/
 gitea/gitea-data/*
 n8n/n8n-data/*
 n8n/n8n-node-data/*
-adguardhome/data/*
+adguardhome/conf/*
 dockmon/data/*
 portainer/portainer_data/*
 metube/MeTube_downloads
@@ -20,6 +21,10 @@ checkmk/checkmk/*
 downtify/Downtify_downloads
 headscale/config/*
 headscale/data/*
+# NetBird local hostnames and generated secrets
+netbird/.env
+netbird/secrets/
+searxng/core-config/*
 
 # Steaming services files
 streaming/jellyfin/*
@@ -30,22 +35,28 @@ streaming/data/*
 streaming/qbittorrent/*
 streaming/prowlarr/*
 
-# Homepage 
-homepages/forust_files/assets/images/team/*
-
+# Homepage
+homepages/forust_files/.well-known/*
 
 # Traefik files
 traefik/letsencrypt/acme.json
+traefik/dynamic/fileservers.yml
+traefik/dynamic/*.local.y*ml.*
+traefik/dynamic/*.external.y*ml
+traefik/k8s/fileservers.y*ml
+traefik/k8s/aliasHeadersStrategy.md
+
 traefik/logs/*
 
 # SSL Certificates
 adguardhome/certs/*
 traefik/certs/*
+certs/
 
 # Monitoring
 monitoring/prometheus.yml
 
-# Python 
+# Python
 .python-version
 venv/
 pyc
@@ -75,6 +86,8 @@ replacements.txt
 
 # Git
 .gitattributes
+# Gitea/github Runners
+.runner
 
 # Misc
 .DS_Store
@@ -83,10 +96,23 @@ replacements.txt
 # Temp files
 edu_master/temp/
 temp/*
+# Local-only tooling scratch space (pinned CI tools, verification scripts)
+tmp/
 
-# Environment 
+# Environment
 .env
 .env.anna
 .env.forust
 .env.*
 !*example
+
+# kubernetes
+*/k8s/*secret*
+!*/k8s/*secret*.example
+**/k8s/*secret*
+!**/k8s/*secret*.example
+# Local-only tweaks, not for upstream
+prometheus-stack/k8s/grafana-values.yaml
+traefik/k8s/local-tls.yaml
+converters/k8s/config.yaml
+convertx/k8s/config.yaml

.hadolint.yaml

@@ -0,0 +1,10 @@
+ignored:
+  - DL3008
+  - DL3042
+  - DL3018
+  - DL3059
+trustedRegistries:
+  - docker.io
+  - ghcr.io
+  - quay.io
+  - gcr.forust.xyz

.markdownlint.json

@@ -0,0 +1,8 @@
+{
+  "default": true,
+  "MD013": false,
+  "MD024": false,
+  "MD033": false,
+  "MD041": false,
+  "MD046": false
+}

.prettierrc.yaml

@@ -0,0 +1,7 @@
+bracketSameLine: true
+htmlWhitespaceSensitivity: css
+printWidth: 120
+tabWidth: 2
+trailingComma: all
+proseWrap: preserve
+endOfLine: lf

.yamllint

@@ -0,0 +1,22 @@
+extends: default
+
+rules:
+  comments:
+    min-spaces-from-content: 1
+  comments-indentation: false
+  document-start: disable
+  line-length: disable
+  braces:
+    min-spaces-inside: 0
+    max-spaces-inside: 1
+  brackets:
+    min-spaces-inside: 0
+    max-spaces-inside: 1
+  indentation:
+    spaces: 2
+    indent-sequences: consistent
+  truthy:
+    allowed-values:
+      - "true"
+      - "false"
+      - "on"

.zed/settings.json

@@ -0,0 +1,40 @@
+{
+  "tab_size": 2,
+  "soft_wrap": "prefer_line",
+  "preferred_line_length": 120,
+  "format_on_save": "on",
+  "languages": {
+    "YAML": {
+      "tab_size": 2,
+      "hard_tabs": false,
+      "format_on_save": "on",
+      "formatter": {
+        "language_server": { "name": "yaml-language-server" },
+      },
+    },
+    "Python": {
+      "tab_size": 4,
+      "format_on_save": "on",
+      "language_servers": ["pyright", "ruff"],
+      "formatter": {
+        "language_server": { "name": "ruff" },
+      },
+    },
+  },
+  "lsp": {
+    "yaml-language-server": {
+      "settings": {
+        "yaml": {
+          "schemas": {
+            "kubernetes": ["**/k8s/*.yaml", "**/k8s/*.yml"],
+          },
+          "validate": true,
+          "completion": true,
+          "format": {
+            "enable": true,
+          },
+        },
+      },
+    },
+  },
+}

adguardhome/compose.yaml

@@ -1,6 +1,6 @@
 services:
   adguard:
-    image: adguard/adguardhome:latest
+    image: adguard/adguardhome:v0.107.79
     container_name: adguardhome
     restart: unless-stopped
     ports:
@@ -11,46 +11,38 @@ services:
       # - "68:68/tcp" # DHCP
       # - "3000:3000/tcp"
     volumes:
-      - ./data/work:/opt/adguardhome/work
-      - ./data/conf:/opt/adguardhome/conf
+      - data:/opt/adguardhome/work
+      - ./conf:/opt/adguardhome/conf
       - ./certs:/certs:ro
-    networks:
-      - proxy
     labels:
       - "traefik.enable=true"
-      - "traefik.docker.network=proxy"
       - "traefik.http.services.adguard.loadbalancer.server.port=3000"
 
       # Prod Router
       - "traefik.http.routers.adguard.rule=Host(`dns.forust.xyz`) || Host(`adguard.forust.xyz`)"
       - "traefik.http.routers.adguard.entrypoints=websecure"
-      - "traefik.http.routers.adguard.middlewares=security-headers@file"
-      - "traefik.http.routers.adguard.service=adguard"
-      - "traefik.http.routers.adguard.tls=true"
-
+      - "traefik.http.routers.adguard.tls.certresolver=letsencrypt"
       # Local Router
-      - "traefik.http.routers.adguard-local.rule=Host(`dns.workstation.internal`) || Host(`adguard.internal`) || Host(`adguard.workstation.internal`)"
+      - "traefik.http.routers.adguard-local.rule=Host(`adguard.workstation.internal`) || Host(`dns.workstation.internal`)"
       - "traefik.http.routers.adguard-local.entrypoints=websecure"
-      - "traefik.http.routers.adguard-local.middlewares=security-headers@file"
-      - "traefik.http.routers.adguard-local.service=adguard"
       - "traefik.http.routers.adguard-local.tls=true"
-
       # Dev Router
-      - "traefik.http.routers.adguard-dev.rule=Host(`dns.gigaforust.internal`) || Host(`adguard.gigaforust.internal`)"
+      - "traefik.http.routers.adguard-dev.rule=Host(`adguard.gigaforust.internal`) || Host(`dns.gigaforust.internal`)"
       - "traefik.http.routers.adguard-dev.entrypoints=websecure"
-      - "traefik.http.routers.adguard-dev.middlewares=security-headers@file"
-      - "traefik.http.routers.adguard-dev.service=adguard"
       - "traefik.http.routers.adguard-dev.tls=true"
+      # DoH Router
+      - "traefik.http.routers.dns-over-https.rule=(Host(`dns.forust.xyz`) || Host(`adguard.forust.xyz`)) && PathPrefix(`/dns-query`)"
+      - "traefik.http.routers.dns-over-https.entrypoints=websecure"
+      - "traefik.http.routers.dns-over-https.tls.certresolver=letsencrypt"
 
-      # DoH
-      - "traefik.http.routers.dns.rule=(Host(`dns.forust.xyz`) && PathPrefix(`/dns-query`))"
-      - "traefik.http.routers.dns.entrypoints=websecure"
-      - "traefik.http.routers.dns.service=adguard"
-      - "traefik.http.routers.dns.tls.certresolver=letsencrypt"
-
+      # Glance Metadata
       - glance.name=adguard
       - glance.url=https://adguard.forust.xyz/
       - glance.description=AdGuard Home is a network-wide software for blocking ads.
+    networks:
+      - proxy
+volumes:
+  data:
 networks:
   proxy:
     external: true

adguardhome/k8s/adguard.yaml

@@ -0,0 +1,127 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: adguard-lb-service
+  namespace: adguard
+  annotations:
+    metallb.io/loadBalancerIPs: "192.168.80.3"
+spec:
+  type: LoadBalancer
+  externalTrafficPolicy: Local
+  selector:
+    app: adguard
+  ports:
+    - name: dns-udp
+      port: 53
+      targetPort: 53
+      protocol: UDP
+    - name: dns-tcp
+      port: 53
+      targetPort: 53
+      protocol: TCP
+    - name: dot
+      port: 853
+      targetPort: 853
+      protocol: TCP
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: adguard-service
+  namespace: adguard
+spec:
+  selector:
+    app: adguard
+  ports:
+    - port: 3000
+      name: webui
+      targetPort: 3000
+    - port: 53
+      name: dns
+      targetPort: 53
+      protocol: UDP
+    - port: 53
+      name: dns-tcp
+      targetPort: 53
+      protocol: TCP
+    - port: 853
+      name: dot
+      targetPort: 853
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  annotations:
+    reloader.stakater.com/auto: "true"
+  name: adguard-deployment
+  namespace: adguard
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: adguard
+  strategy:
+    type: Recreate
+  template:
+    metadata:
+      labels:
+        app: adguard
+    spec:
+      containers:
+        - name: adguard
+          image: adguard/adguardhome:v0.107.79
+          resources:
+            limits:
+              memory: "1.5Gi"
+              cpu: "300m"
+            requests:
+              memory: "512Mi"
+              cpu: "50m"
+          ports:
+            - containerPort: 3000
+              name: webui
+            - containerPort: 53
+              name: dns
+            - containerPort: 853
+              name: dot
+          readinessProbe:
+            tcpSocket:
+              port: dns
+            initialDelaySeconds: 5
+            periodSeconds: 5
+            successThreshold: 1
+            failureThreshold: 3
+          volumeMounts:
+            - name: adguard-data
+              mountPath: /opt/adguardhome/work
+              subPath: work
+            - name: adguard-data
+              mountPath: /opt/adguardhome/conf
+              subPath: conf
+            - name: adguard-certs
+              mountPath: /certs
+              readOnly: true
+      volumes:
+        - name: adguard-data
+          persistentVolumeClaim:
+            claimName: adguard-pvc
+        - name: adguard-certs
+          secret:
+            secretName: adguard-certs
+            items:
+              - key: tls.crt
+                path: fullchain.pem
+              - key: tls.key
+                path: privkey.pem
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: adguard-pvc
+  namespace: adguard
+spec:
+  accessModes:
+    - ReadWriteOnce
+  resources:
+    requests:
+      storage: 2Gi

adguardhome/k8s/certificate.yaml

@@ -0,0 +1,12 @@
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: adguard-certs
+  namespace: adguard
+spec:
+  secretName: adguard-certs
+  dnsNames:
+    - dns.forust.xyz
+  issuerRef:
+    name: letsencrypt-prod
+    kind: ClusterIssuer

adguardhome/k8s/ingress.yaml

@@ -0,0 +1,43 @@
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
+metadata:
+  name: adguard-prod
+  namespace: adguard
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`dns.forust.xyz`)
+      kind: Rule
+      services:
+        - name: adguard-service
+          port: 3000
+    - match: (Host(`dns.forust.xyz`)) && PathPrefix(`/dns-query`)
+      kind: Rule
+      services:
+        - name: adguard-service
+          port: 3000
+  tls:
+    secretName: adguard-certs
+---
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
+metadata:
+  name: adguard-local
+  namespace: adguard
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`adguard.workstation.internal`) || Host(`dns.workstation.internal`) || Host(`adguard.gigaforust.internal`) || Host(`dns.gigaforust.internal`)
+      kind: Rule
+      services:
+        - name: adguard-service
+          port: 3000
+    - match: (Host(`adguard.workstation.internal`) || Host(`dns.workstation.internal`) || Host(`adguard.gigaforust.internal`) || Host(`dns.gigaforust.internal`)) && PathPrefix(`/dns-query`)
+      kind: Rule
+      services:
+        - name: adguard-service
+          port: 3000
+  tls:
+    secretName: internal-wildcard-tls

adguardhome/k8s/internal-certificate.yaml

@@ -0,0 +1,15 @@
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: internal-wildcard-tls
+  namespace: adguard
+spec:
+  secretName: internal-wildcard-tls
+  dnsNames:
+    - "*.workstation.internal"
+    - "*.gigaforust.internal"
+    - workstation.internal
+    - gigaforust.internal
+  issuerRef:
+    name: internal-ca
+    kind: ClusterIssuer

adguardhome/k8s/namespace.yaml

@@ -0,0 +1,4 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: adguard

adguardhome/k8s/secrets.yaml.example

@@ -0,0 +1,10 @@
+kubectl apply -f k8s/namespace.yaml && \
+kubectl create secret tls adguard-certs -n adguard \
+  --cert=certs/fullchain.pem \
+  --key=certs/privkey.pem --dry-run=client -o yaml > \
+  k8s/secrets.yaml
+
+#  OR WITH NO FILE CREATION:
+kubectl create secret tls adguard-certs -n adguard \
+  --cert=certs/fullchain.pem --key=certs/privkey.pem \
+  --save-config

authentik/compose.yaml

@@ -1,6 +1,6 @@
 services:
   postgresql:
-    image: docker.io/library/postgres:15-alpine
+    image: docker.io/library/postgres:15.19-alpine
     restart: unless-stopped
     env_file:
       - .env
@@ -37,37 +37,26 @@ services:
       AUTHENTIK_POSTGRESQL__PASSWORD: ${PG_PASS}
       AUTHENTIK_POSTGRESQL__USER: ${PG_USER:-authentik}
       AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY:?secret key required}
-
+    volumes:
+      - ./media:/media
+      - ./custom-templates:/templates
     labels:
       - "traefik.enable=true"
-      - "traefik.docker.network=proxy"
-      # Services
-      # - "traefik.http.services.authentik-server.loadbalancer.server.port=9443"
       - "traefik.http.services.authentik-server.loadbalancer.server.port=9000"
 
       # Prod Router
       - "traefik.http.routers.authentik-server.rule=Host(`auth.forust.xyz`)"
       - "traefik.http.routers.authentik-server.entrypoints=websecure"
-      - "traefik.http.routers.authentik-server.middlewares=security-headers@file"
-      - "traefik.http.routers.authentik-server.service=authentik-server"
-      - "traefik.http.routers.authentik-server.tls=true"
-
+      - "traefik.http.routers.authentik-server.tls.certresolver=letsencrypt"
       # Local Router
-      - "traefik.http.routers.authentik-server-local.rule=Host(`auth.workstation.internal`) || Host(`auth-dashboard.internal`)"
+      - "traefik.http.routers.authentik-server-local.rule=Host(`auth.workstation.internal`)"
       - "traefik.http.routers.authentik-server-local.entrypoints=websecure"
-      - "traefik.http.routers.authentik-server-local.middlewares=security-headers@file"
-      - "traefik.http.routers.authentik-server-local.service=authentik-server"
       - "traefik.http.routers.authentik-server-local.tls=true"
-
       # Dev Router
       - "traefik.http.routers.authentik-server-dev.rule=Host(`auth.gigaforust.internal`)"
       - "traefik.http.routers.authentik-server-dev.entrypoints=websecure"
       - "traefik.http.routers.authentik-server-dev.middlewares=security-headers@file"
-      - "traefik.http.routers.authentik-server-dev.service=authentik-server"
       - "traefik.http.routers.authentik-server-dev.tls=true"
-    volumes:
-      - ./media:/media
-      - ./custom-templates:/templates
     networks:
       - proxy
       - authentik

authentik/k8s/authentik.yaml

@@ -0,0 +1,101 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: authentik-server-service
+  namespace: authentik
+spec:
+  type: ClusterIP
+  selector:
+    app: authentik-server
+  ports:
+    - port: 9000
+      targetPort: 9000
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: authentik-worker-service
+  namespace: authentik
+spec:
+  type: ClusterIP
+  selector:
+    app: authentik-worker
+  ports:
+    - port: 9000
+      targetPort: 9000
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  annotations:
+    reloader.stakater.com/auto: "true"
+  name: authentik-server-deployment
+  namespace: authentik
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: authentik-server
+  strategy:
+    type: Recreate
+  template:
+    metadata:
+      labels:
+        app: authentik-server
+    spec:
+      containers:
+        - name: authentik-server
+          image: ghcr.io/goauthentik/server:2026.8.3
+          args: ["server"]
+          envFrom:
+            - configMapRef:
+                name: authentik-config
+            - secretRef:
+                name: authentik-secrets
+          ports:
+            - containerPort: 9000
+          resources:
+            requests:
+              memory: "768Mi"
+              cpu: "100m"
+            limits:
+              memory: "1.5Gi"
+              cpu: "1000m"
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  annotations:
+    reloader.stakater.com/auto: "true"
+  name: authentik-worker-deployment
+  namespace: authentik
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: authentik-worker
+  strategy:
+    type: Recreate
+  template:
+    metadata:
+      labels:
+        app: authentik-worker
+    spec:
+      containers:
+        - name: authentik-worker
+          image: ghcr.io/goauthentik/server:2026.8.3
+          args: ["worker"]
+          securityContext:
+            runAsUser: 0
+          envFrom:
+            - configMapRef:
+                name: authentik-config
+            - secretRef:
+                name: authentik-secrets
+          resources:
+            requests:
+              memory: "320Mi"
+              cpu: "100m"
+            limits:
+              memory: "768Mi"
+              cpu: "700m"

authentik/k8s/certificates.yaml

@@ -0,0 +1,28 @@
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: authentik-prod-tls
+  namespace: authentik
+spec:
+  secretName: authentik-prod-tls
+  dnsNames:
+    - auth.forust.xyz
+  issuerRef:
+    name: letsencrypt-prod
+    kind: ClusterIssuer
+---
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: internal-wildcard-tls
+  namespace: authentik
+spec:
+  secretName: internal-wildcard-tls
+  dnsNames:
+    - "*.workstation.internal"
+    - "*.gigaforust.internal"
+    - workstation.internal
+    - gigaforust.internal
+  issuerRef:
+    name: internal-ca
+    kind: ClusterIssuer

authentik/k8s/config.yaml

@@ -0,0 +1,11 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: authentik-config
+  namespace: authentik
+data:
+  AUTHENTIK_IMAGE: ghcr.io/goauthentik/server
+  AUTHENTIK_TAG: "2025.10.2"
+  AUTHENTIK_POSTGRESQL__HOST: postgres.database.svc.cluster.local
+  AUTHENTIK_POSTGRESQL__NAME: authentik
+  AUTHENTIK_ERROR_REPORTING__ENABLED: "true"

authentik/k8s/ingress.yaml

@@ -0,0 +1,33 @@
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
+metadata:
+  name: authentik-prod
+  namespace: authentik
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`auth.forust.xyz`)
+      kind: Rule
+      services:
+        - name: authentik-server-service
+          port: 9000
+  tls:
+    secretName: authentik-prod-tls
+---
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
+metadata:
+  name: authentik-local
+  namespace: authentik
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`auth.workstation.internal`) || Host(`auth.gigaforust.internal`)
+      kind: Rule
+      services:
+        - name: authentik-server-service
+          port: 9000
+  tls:
+    secretName: internal-wildcard-tls

authentik/k8s/namespace.yaml

@@ -0,0 +1,4 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: authentik

authentik/k8s/secrets.yaml.example

@@ -0,0 +1,11 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: authentik-secrets
+  namespace: authentik
+type: Opaque
+stringData:
+  AUTHENTIK_SECRET_KEY: ""
+  AUTHENTIK_POSTGRESQL__PASSWORD: ""
+  AUTHENTIK_POSTGRESQL__USER: authentik
+  AUTHENTIK_BOOTSTRAP_PASSWORD: authentik

cert-manager/k8s/cert-manager-values.yaml

@@ -0,0 +1,9 @@
+crds:
+  enabled: true
+prometheus:
+  servicemonitor:
+    enabled: true
+    interval: 60s
+    scrapeTimeout: 30s
+    labels:
+      release: prometheus-stack

cert-manager/k8s/clusterissuer.yaml

@@ -0,0 +1,29 @@
+apiVersion: cert-manager.io/v1
+kind: ClusterIssuer
+metadata:
+  name: letsencrypt-staging
+spec:
+  acme:
+    email: bobrovod@national.shitposting.agency
+    server: https://acme-staging-v02.api.letsencrypt.org/directory
+    privateKeySecretRef:
+      name: letsencrypt-staging-account-key
+    solvers:
+      - http01:
+          ingress:
+            class: traefik
+---
+apiVersion: cert-manager.io/v1
+kind: ClusterIssuer
+metadata:
+  name: letsencrypt-prod
+spec:
+  acme:
+    email: bobrovod@national.shitposting.agency
+    server: https://acme-v02.api.letsencrypt.org/directory
+    privateKeySecretRef:
+      name: letsencrypt-prod-account-key
+    solvers:
+      - http01:
+          ingress:
+            class: traefik

cert-manager/k8s/homelab-internal-root-ca.crt

@@ -0,0 +1,30 @@
+-----BEGIN CERTIFICATE-----
+MIIFFjCCAv6gAwIBAgIUetKpTfEDOn2985FFMu6G26itT+wwDQYJKoZIhvcNAQEN
+BQAwIzEhMB8GA1UEAxMYaG9tZWxhYiBpbnRlcm5hbCByb290IENBMB4XDTI2MDky
+MzEyNDA0N1oXDTM2MDkyMDEyNDA0N1owIzEhMB8GA1UEAxMYaG9tZWxhYiBpbnRl
+cm5hbCByb290IENBMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAvmNP
+ZCOoD8NtNuYJKVXBlTPjX7D7sJCSK5neH7ZbYV5+lmUlEErY8Mik7j37V5k5NfpF
+Ig85pOjP7RckTPz5V6ek3yaN40s4AL053sN5ZPauDVYjalaEHTgj5sEMqlLACQWI
+yZmJOZspZykae8dIpQnqCoFpRT4FurJ78v4a0ylnFVLMQn/lyCHedwTjkEdtYWYr
+ccJy8vQwqkzs/rWvEH1lDqZhennLOrmcCjfonG7D/pruMn4z+6E28p4+ejkRrI6x
+luak3KnpT1XMeHtgU21hiRGaMDBchHMFgAhnY1qosymKenXvfTZItwgjZbwa1hJI
+GAiDm+jQDKMjzRZ3rH6Xfc0auUcykNz73PpNu1NGm78nndXwCXcXn1LFKNQJ+r1U
+sJiyAmUZmXVn4aM4OMf2F38k7wTYIKg7nRGaUkNeKDlNkjA4HvgWw+jwO1KmdHQ/
+mOem1rosDWHRK01wg+Gga9mQCnhNhxglg3t/UeSic6uOaRsvaz4qkzHq8MbCujVz
+DpKQjqdikYOAXZOs4KlBLWrS7NaK4NzfSD02pBUErh54ruJfY/bWz9KyXzBD/lQZ
+VUTKyvUVB0bkVHEdf1jJmX3H4IZRQSF5JPqOBotW6bJI5fEGNBvj9Zxy4nm2WWGz
+yyP3uWsQz8U/Wdx9nXZLHInTZBsvgLYtKUAWA30CAwEAAaNCMEAwDgYDVR0PAQH/
+BAQDAgKkMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFEkKm2rxPaK6+O9WD80z
+BLC6F9QsMA0GCSqGSIb3DQEBDQUAA4ICAQAnFyHz97Umf5VIu+dKTJid7C73VugJ
+TIar/xJBs/4CxP+znBxhJjXygRoyIfzoVGWcB2ZSL//vL78Qlts79K/Imc9a4RFF
+wMvCxsRXAEQ4TpeWi3ophPNcs4rhsP+gQKQFtnyKP9519bqpfxp0bTqwOV2o18fn
+za7rlQViiEnNV58j7CVoM9+mJvVVfBEX1Km+GyJL9GadzbIQ7FxClVJZefCbft93
+zHVk9gDOw8ys1XGSR2OUCyCLinXO6mqS16CmBb2MAKXq/YyH7E0N8iotAPGtfA8V
+M/0ddy947rY0xCrtECfWwvGQpJS7NRv/Z9b2jCfXrI5LXmL2nfQRg0y9GE4Vjwr+
+WxtGU5jOeFt0jQ+xRzcgG0Op+qK3x55l5LSo2hOcOVYbiHxcHEJFgwNi1ADeBFwb
+q/HdysfURSOghqjIpMMAUabBp+DBUg2EUF7pIaUqbdqExFYcr9EYisEMiNsmKmN+
+8ZbcOeerFKDQj+t/R0bFXa7UBn2UWsjI8zlR74aa2kLDXwtyz/XlO/FlYm66eBFo
+2/eYUSeU+S4ej+wUAs/dvjF7f190/DUQGuwOTlLTahqWDztmhCk7qzbECu56CwKT
+E5Ect2P72UleYwdblkVOVd352AmiwEzdOaziIRrPh8uenEknH6JBYPO3mjk7cCg+
+GPGcNIBctjdXhg==
+-----END CERTIFICATE-----

cert-manager/k8s/internal-ca.yaml

@@ -0,0 +1,33 @@
+apiVersion: cert-manager.io/v1
+kind: ClusterIssuer
+metadata:
+  name: selfsigned
+spec:
+  selfSigned: {}
+---
+# Homelab internal root CA (10y). Install the .crt on clients (see below).
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: internal-ca-root
+  namespace: cert-manager
+spec:
+  isCA: true
+  commonName: homelab internal root CA
+  duration: 87600h
+  renewBefore: 7200h
+  secretName: internal-ca-root
+  privateKey:
+    algorithm: RSA
+    size: 4096
+  issuerRef:
+    name: selfsigned
+    kind: ClusterIssuer
+---
+apiVersion: cert-manager.io/v1
+kind: ClusterIssuer
+metadata:
+  name: internal-ca
+spec:
+  ca:
+    secretName: internal-ca-root

cert-manager/k8s/namespace.yaml

@@ -0,0 +1,4 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: cert-manager

cfddns/.env.example

@@ -0,0 +1,15 @@
+CLOUDFLARE_API_TOKEN=YOUR_CLOUDFLARE_API_TOKEN
+DOMAINS=example.com,dns.example.com,mc.example.com,auth.example.com,ssh.example.com
+IP4_DOMAINS=
+IP6_DOMAINS=
+IP4_PROVIDER=cloudflare.trace
+IP6_PROVIDER=none # change if you want to update AAAA
+UPDATE_CRON=@every 5m
+UPDATE_ON_START=true
+DELETE_ON_STOP=false
+DELETE_ON_FAILURE=true
+TTL=1
+PROXIED=!is(dns.example.com) && !is(mc.example.com) && !is(ssh.example.com)
+EMOJI=true
+UPTIMEKUMA=https://uptime-kuma.example.com/api/push/AsaSDFGFkfklaFALSKffkfFKfkfkfkFK?status=up&msg=OK&ping=
+REJECT_CLOUDFLARE_IPS=true

cfddns/compose.yaml

@@ -1,13 +1,30 @@
 services:
   cloudflare-ddns:
-    image: timothyjmiller/cloudflare-ddns:latest
+    image: timothyjmiller/cloudflare-ddns:2.2.0
     container_name: cloudflare-ddns
+    restart: unless-stopped
     security_opt:
       - no-new-privileges:true
-    network_mode: 'host'
+    network_mode: "host"
+    # https://github.com/timothymiller/cloudflare-ddns#-quick-start
     environment:
-      - PUID=1000
-      - PGID=1000
-    volumes:
-      - ./config.json:/config.json
-    restart: unless-stopped
+      - CLOUDFLARE_API_TOKEN=${CLOUDFLARE_API_TOKEN:?Cloudflare API token is required}
+      - DOMAINS=${DOMAINS:-}
+      - IP4_DOMAINS=${IP4_DOMAINS:-}
+      - IP6_DOMAINS=${IP6_DOMAINS:-}
+      - IP4_PROVIDER=${IP4_PROVIDER:-cloudflare.trace}
+      - IP6_PROVIDER=${IP6_PROVIDER:-none}
+      - UPDATE_CRON=${UPDATE_CRON:-@every 5m}
+      - UPDATE_ON_START=${UPDATE_ON_START:-true}
+      - DELETE_ON_STOP=${DELETE_ON_STOP:-false}
+      - DELETE_ON_FAILURE=${DELETE_ON_FAILURE:-true}
+      - TTL=${TTL:-1} # 1=auto
+      # to proxy only "dns.example.com" and "wfs.example.com" use "!is(dns.domain.com) && !is (wfs.domain.com)"
+      - PROXIED=${PROXIED:-true}
+      - EMOJI=${EMOJI:-true}
+      - UPTIMEKUMA=${UPTIMEKUMA:-}
+      - HEALTHCHECKS=${HEALTHCHECKS:-}
+      - REJECT_CLOUDFLARE_IPS=${REJECT_CLOUDFLARE_IPS:-true}
+      # volumes:
+      # Prefer using environment variables for configuration, config.json legacy support
+      # - ./config.json:/config.json

cfddns/config.json.example

@@ -7,7 +7,7 @@
       "api_key": {
         "api_key": "api_key_here",
         "account_email": "your_email_here"
-      }
+      },
       "zone_id": "your_zone-id",
       "subdomains": [
         { "name": "", "proxied": true },

cfddns/k8s/.gitignore

@@ -0,0 +1 @@
+secret.yaml

cfddns/k8s/deployment.yaml

@@ -0,0 +1,36 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  annotations:
+    reloader.stakater.com/auto: "true"
+  name: cfddns
+  labels:
+    app: cfddns
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: cfddns
+  strategy:
+    type: Recreate
+  template:
+    metadata:
+      labels:
+        app: cfddns
+    spec:
+      hostNetwork: true
+      dnsPolicy: ClusterFirstWithHostNet
+      containers:
+        - name: cloudflare-ddns
+          image: timothyjmiller/cloudflare-ddns:2.2.0
+          imagePullPolicy: Always
+          resources:
+            requests:
+              memory: "32Mi"
+              cpu: "30m"
+            limits:
+              memory: "128Mi"
+              cpu: "50m"
+          envFrom:
+            - secretRef:
+                name: cfddns-secrets

cfddns/k8s/secret.yaml.example

@@ -0,0 +1,18 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: cfddns-secrets
+type: Opaque
+stringData:
+  CLOUDFLARE_API_TOKEN: your_token
+  DOMAINS: "example.com,www.example.com"
+  IP4_PROVIDER: cloudflare.trace
+  IP6_PROVIDER: none
+  UPDATE_CRON: "@every 5m"
+  UPDATE_ON_START: "true"
+  DELETE_ON_STOP: "false"
+  DELETE_ON_FAILURE: "true"
+  TTL: "1"
+  PROXIED: "true"
+  EMOJI: "true"
+  REJECT_CLOUDFLARE_IPS: "true"

checkmk/.env.example

@@ -0,0 +1,2 @@
+CMK_PASSWORD=password
+TZ=Europe/Berlin

checkmk/compose.yaml

@@ -0,0 +1,39 @@
+services:
+  checkmk:
+    image: "checkmk/check-mk-raw:2.4.0-2026.09.14"
+    container_name: "checkmk"
+    restart: unless-stopped
+    # ports:
+    #   - 5000:5000
+    #   - 6776:8000
+    volumes:
+      - sites:/omd/sites
+    tmpfs:
+      - /opt/omd/sites/cmk/tmp:uid=1000,gid=1000
+    environment:
+      - CMK_PASSWORD=${CMK_PASSWORD:-password}
+      - CMK_SITE_ID=cmk
+      - TZ=${TZ:-Etc/UTC}
+    labels:
+      - "traefik.enable=true"
+      - "traefik.http.services.checkmk.loadbalancer.server.port=5000"
+
+      # Prod Router
+      - "traefik.http.routers.checkmk.rule=Host(`cmk.forust.xyz`)"
+      - "traefik.http.routers.checkmk.entrypoints=websecure"
+      - "traefik.http.routers.checkmk.tls.certresolver=letsencrypt"
+      # Local Router
+      - "traefik.http.routers.checkmk-local.rule=Host(`cmk.workstation.internal`)"
+      - "traefik.http.routers.checkmk-local.entrypoints=websecure"
+      - "traefik.http.routers.checkmk-local.tls=true"
+      # Dev Router
+      - "traefik.http.routers.checkmk-dev.rule=Host(`cmk.gigaforust.internal`)"
+      - "traefik.http.routers.checkmk-dev.entrypoints=websecure"
+      - "traefik.http.routers.checkmk-dev.tls=true"
+    networks:
+      - proxy
+networks:
+  proxy:
+    external: true
+volumes:
+  sites:

checkmk/k8s/certificates.yaml

@@ -0,0 +1,28 @@
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: checkmk-prod-tls
+  namespace: checkmk
+spec:
+  secretName: checkmk-prod-tls
+  dnsNames:
+    - cmk.forust.xyz
+  issuerRef:
+    name: letsencrypt-prod
+    kind: ClusterIssuer
+---
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: internal-wildcard-tls
+  namespace: checkmk
+spec:
+  secretName: internal-wildcard-tls
+  dnsNames:
+    - "*.workstation.internal"
+    - "*.gigaforust.internal"
+    - workstation.internal
+    - gigaforust.internal
+  issuerRef:
+    name: internal-ca
+    kind: ClusterIssuer

checkmk/k8s/checkmk.yaml

@@ -0,0 +1,79 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: checkmk-service
+  namespace: checkmk
+spec:
+  selector:
+    app: checkmk
+  ports:
+    - name: web
+      port: 5000
+      targetPort: 5000
+    - name: agent-receiver
+      port: 8000
+      targetPort: 8000
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  annotations:
+    reloader.stakater.com/auto: "true"
+  name: checkmk-deployment
+  namespace: checkmk
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: checkmk
+  strategy:
+    type: Recreate
+  template:
+    metadata:
+      labels:
+        app: checkmk
+    spec:
+      containers:
+        - name: checkmk
+          image: checkmk/check-mk-raw:2.4.0-2026.09.14
+          envFrom:
+            - secretRef:
+                name: checkmk-secrets
+            - configMapRef:
+                name: checkmk-config
+          ports:
+            - name: web
+              containerPort: 5000
+            - name: agent-receiver
+              containerPort: 8000
+          volumeMounts:
+            - name: sites
+              mountPath: /omd/sites
+            - name: tmp
+              mountPath: /opt/omd/sites/cmk/tmp
+          resources:
+            requests:
+              memory: "2Gi"
+              cpu: "600m"
+            limits:
+              memory: "5Gi"
+              cpu: "4"
+      volumes:
+        - name: sites
+          persistentVolumeClaim:
+            claimName: checkmk-sites-pvc
+        - name: tmp
+          emptyDir:
+            medium: Memory
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: checkmk-sites-pvc
+  namespace: checkmk
+spec:
+  accessModes:
+    - ReadWriteOnce
+  resources:
+    requests:
+      storage: 5Gi

checkmk/k8s/config.yaml

@@ -0,0 +1,8 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: checkmk-config
+  namespace: checkmk
+data:
+  TZ: Europe/Bratislava
+  CMK_SITE_ID: cmk

checkmk/k8s/ingress.yaml

@@ -0,0 +1,49 @@
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
+metadata:
+  name: checkmk-prod
+  namespace: checkmk
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`cmk.forust.xyz`)
+      kind: Rule
+      services:
+        - name: checkmk-service
+          port: 5000
+  tls:
+    secretName: checkmk-prod-tls
+---
+apiVersion: traefik.io/v1alpha1
+kind: IngressRouteTCP
+metadata:
+  name: checkmk-agent-receiver
+  namespace: checkmk
+spec:
+  entryPoints:
+    - checkmk-agent
+  routes:
+    - match: HostSNI(`*`)
+      services:
+        - name: checkmk-service
+          port: 8000
+  tls:
+    passthrough: true
+---
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
+metadata:
+  name: checkmk-local
+  namespace: checkmk
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`cmk.workstation.internal`) || Host(`cmk.gigaforust.internal`)
+      kind: Rule
+      services:
+        - name: checkmk-service
+          port: 5000
+  tls:
+    secretName: internal-wildcard-tls

checkmk/k8s/namespace.yaml

@@ -0,0 +1,4 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: checkmk

checkmk/k8s/secrets.yaml.example

@@ -0,0 +1,8 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: checkmk-secrets
+  namespace: checkmk
+type: Opaque
+stringData:
+  CMK_PASSWORD: "password"

cloudflared/k8s/.gitignore

@@ -0,0 +1 @@
+secret.yaml

cloudflared/k8s/deployment.yaml

@@ -0,0 +1,41 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  annotations:
+    reloader.stakater.com/auto: "true"
+  name: cloudflared
+  labels:
+    app: cloudflared
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: cloudflared
+  strategy:
+    type: Recreate
+  template:
+    metadata:
+      labels:
+        app: cloudflared
+    spec:
+      containers:
+        - name: cloudflared
+          image: cloudflare/cloudflared:2026.10.0
+          imagePullPolicy: IfNotPresent
+          args:
+            - tunnel
+            - --no-autoupdate
+            - run
+          env:
+            - name: TUNNEL_TOKEN
+              valueFrom:
+                secretKeyRef:
+                  name: cloudflared-secrets
+                  key: TUNNEL_TOKEN
+          resources:
+            requests:
+              memory: "128Mi"
+              cpu: "30m"
+            limits:
+              memory: "256Mi"
+              cpu: "200m"

cloudflared/k8s/secret.yaml.example

@@ -0,0 +1,7 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: cloudflared-secrets
+type: Opaque
+stringData:
+  TUNNEL_TOKEN: your_tunnel_token_here

converters/.env.example

@@ -0,0 +1,9 @@
+ACCOUNT_REGISTRATION=false
+HTTP_ALLOWED=false
+ALLOW_UNAUTHENTICAED=false
+AUTO_DELETE_EVERY_N_HOURS=24
+WEBROOT=/convert
+HIDE_HISTORY=false
+LANGUAGE=en
+UNAUTHED_USER_SHARING=false
+MAX_CONVERT_PROCESS=0

converters/compose.yaml

@@ -0,0 +1,70 @@
+services:
+  convertx:
+    container_name: convertx
+    image: ghcr.io/c4illin/convertx:v0.19.0
+    restart: unless-stopped
+    ports:
+      - "9992:3000"
+    # https://github.com/C4illin/ConvertX#environment-variables
+    environment:
+      - JWT_SECRET=$(JWT_SECRET)
+      - ACCOUNT_REGISTRATION=$(ACCOUNT_REGISTRATION:-false)
+      - HTTP_ALLOWED=$(HTTP_ALLOWED:-false)
+      - ALLOW_UNAUTHENTICATED=$(ALLOW_UNAUTHENTICATED:-false)
+      - AUTO_DELETE_EVERY_N_HOURS=$(AUTO_DELETE_EVERY_N_HOURS:-24)
+      - WEBROOT=$(WEBROOT)
+      - HIDE_HISTORY=$(HIDE_HISTORY:-false)
+      - LANGUAGE=$(LANGUAGE:-en)
+      - UNAUTHENTICATED_USER_SHARING=$(UNAUTHENTICATED_USER_SHARING:-false)
+      - MAX_CONVERT_PROCESS=$(MAX_CONVERT_PROCESS:-0)
+    labels:
+      - "traefik.enable=true"
+      - "traefik.http.services.convertx.loadbalancer.server.port=3000"
+      # Prod Router
+      - "traefik.http.routers.convertx.rule=(Host(`forust.xyz`) || Host(`www.forust.xyz`)) && PathPrefix(`/convert`)"
+      - "traefik.http.routers.convertx.entrypoints=websecure"
+      - "traefik.http.routers.convertx.priority=50"
+      - "traefik.http.routers.convertx.tls.certresolver=letsencrypt"
+      # Local Router
+      - "traefik.http.routers.convertx-local.rule=Host(`workstation.internal`) && PathPrefix(`/convert`)"
+      - "traefik.http.routers.convertx-local.entrypoints=websecure"
+      - "traefik.http.routers.convertx-local.priority=50"
+      - "traefik.http.routers.convertx-local.tls=true"
+      # Dev Router
+      - "traefik.http.routers.convertx-dev.rule=Host(`gigaforust.internal`) && PathPrefix(`/convert`)"
+      - "traefik.http.routers.convertx-dev.entrypoints=websecure"
+      - "traefik.http.routers.convertx-dev.priority=50"
+      - "traefik.http.routers.convertx-dev.tls=true"
+    networks:
+      - proxy
+    volumes:
+      - data:/app/data
+
+  bentopdf:
+    container_name: bentopdf
+    image: bentopdf/bentopdf@sha256:4eb4ec8f5030faf87c29a73d3d5a2781f28a597cf440c3ab111eb96aee550871
+    restart: unless-stopped
+    labels:
+      - "traefik.enable=true"
+      - "traefik.http.services.bentopdf.loadbalancer.server.port=8080"
+
+      # Prod router
+      - "traefik.http.routers.bentopdf.rule=Host(`pdf.forust.xyz`)"
+      - "traefik.http.routers.bentopdf.entrypoints=websecure"
+      - "traefik.http.routers.bentopdf.tls.certresolver=letsencrypt"
+      - "traefik.http.routers.bentopdf.tls=true"
+      # Local router
+      - "traefik.http.routers.bentopdf-local.rule=Host(`pdf.workstation.internal`)"
+      - "traefik.http.routers.bentopdf-local.entrypoints=websecure"
+      - "traefik.http.routers.bentopdf-local.tls=true"
+      # Dev router
+      - "traefik.http.routers.bentopdf-dev.rule=Host(`pdf.gigaforust.internal`)"
+      - "traefik.http.routers.bentopdf-dev.entrypoints=websecure"
+      - "traefik.http.routers.bentopdf-dev.tls=true"
+    networks:
+      - proxy
+networks:
+  proxy:
+    external: true
+volumes:
+  data:

converters/k8s/bentopdf.yaml

@@ -0,0 +1,43 @@
+kind: Service
+apiVersion: v1
+metadata:
+  name: bentopdf-service
+  namespace: converters
+spec:
+  selector:
+    app: bentopdf
+  ports:
+    - port: 8080
+      targetPort: 8080
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: bentopdf-deployment
+  namespace: converters
+spec:
+  replicas: 2
+  selector:
+    matchLabels:
+      app: bentopdf
+  template:
+    metadata:
+      labels:
+        app: bentopdf
+    spec:
+      containers:
+        - image: bentopdf/bentopdf@sha256:4eb4ec8f5030faf87c29a73d3d5a2781f28a597cf440c3ab111eb96aee550871
+          imagePullPolicy: Always
+          name: bentopdf
+          ports:
+            - containerPort: 8080
+          # p95 4M, max 11M over 7 days. Was 50Mi/700Mi.
+          resources:
+            requests:
+              memory: "32Mi"
+              cpu: "50m"
+              ephemeral-storage: "100Mi"
+            limits:
+              memory: "128Mi"
+              cpu: "700m"
+              ephemeral-storage: "5Gi"

converters/k8s/certificates.yaml

@@ -0,0 +1,42 @@
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: convertx-prod-tls
+  namespace: converters
+spec:
+  secretName: convertx-prod-tls
+  dnsNames:
+    - forust.xyz
+    - www.forust.xyz
+  issuerRef:
+    name: letsencrypt-prod
+    kind: ClusterIssuer
+---
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: bentopdf-prod-tls
+  namespace: converters
+spec:
+  secretName: bentopdf-prod-tls
+  dnsNames:
+    - pdf.forust.xyz
+  issuerRef:
+    name: letsencrypt-prod
+    kind: ClusterIssuer
+---
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: internal-wildcard-tls
+  namespace: converters
+spec:
+  secretName: internal-wildcard-tls
+  dnsNames:
+    - "*.workstation.internal"
+    - "*.gigaforust.internal"
+    - workstation.internal
+    - gigaforust.internal
+  issuerRef:
+    name: internal-ca
+    kind: ClusterIssuer

converters/k8s/config.yaml.example

@@ -0,0 +1,16 @@
+# test manifest with docker and k8s config keys mismatch
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: convertx-config
+  namespace: converters
+data:
+  ACCOUNT_REGISTRATION: "false"
+  HTTP_ALLOWED: "false"
+  ALLOW_UNAUTHENTICAED: "false"
+  AUTO_DELETE_EVERY_N_HOURS: "24"
+  WEBROOT: "/convert"
+  HIDE_HISTORY: "false"
+  LANGUAGE: "en"
+  UNAUTHED_USER_SHARING: "false"
+  MAX_CONVERT_PROCESS: "0"

converters/k8s/convertx.yaml

@@ -0,0 +1,68 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: convertx-service
+  namespace: converters
+spec:
+  selector:
+    app: convertx
+  ports:
+    - port: 3000
+      targetPort: 3000
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  annotations:
+    reloader.stakater.com/auto: "true"
+  name: convertx-deployment
+  namespace: converters
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: convertx
+  strategy:
+    type: Recreate
+  template:
+    metadata:
+      labels:
+        app: convertx
+    spec:
+      containers:
+        - image: ghcr.io/c4illin/convertx:v0.19.0
+          name: convertx
+          envFrom:
+            - configMapRef:
+                name: convertx-config
+            - secretRef:
+                name: convertx-secrets
+          ports:
+            - containerPort: 3000
+          volumeMounts:
+            - mountPath: /data
+              name: data
+          # p95 85M, max 136M over 7 days, spikes while converting. Was 250Mi/1.5Gi.
+          resources:
+            requests:
+              memory: "128Mi"
+              cpu: "100m"
+            limits:
+              cpu: "1500m"
+              memory: "512Mi"
+      volumes:
+        - name: data
+          persistentVolumeClaim:
+            claimName: convertx-pvc
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: convertx-pvc
+  namespace: converters
+spec:
+  accessModes:
+    - ReadWriteOnce
+  resources:
+    requests:
+      storage: 2Gi

converters/k8s/ingress.yaml

@@ -0,0 +1,70 @@
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
+metadata:
+  name: convertx-prod
+  namespace: converters
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: (Host(`forust.xyz`) || Host(`www.forust.xyz`)) && PathPrefix(`/convert`)
+      kind: Rule
+      priority: 50
+      services:
+        - name: convertx-service
+          port: 3000
+  tls:
+    secretName: convertx-prod-tls
+---
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
+metadata:
+  name: convertx-local
+  namespace: converters
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: (Host(`workstation.internal`) || Host(`gigaforust.internal`)) && PathPrefix(`/convert`)
+      kind: Rule
+      priority: 50
+      services:
+        - name: convertx-service
+          port: 3000
+  tls:
+    secretName: internal-wildcard-tls
+
+---
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
+metadata:
+  name: bentopdf-prod
+  namespace: converters
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`pdf.forust.xyz`)
+      kind: Rule
+      services:
+        - name: bentopdf-service
+          port: 8080
+  tls:
+    secretName: bentopdf-prod-tls
+---
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
+metadata:
+  name: bentopdf-local
+  namespace: converters
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`pdf.workstation.internal`) || Host(`pdf.gigaforust.internal`)
+      kind: Rule
+      services:
+        - name: bentopdf-service
+          port: 8080
+  tls:
+    secretName: internal-wildcard-tls

converters/k8s/namespace.yaml

@@ -0,0 +1,4 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: converters

converters/k8s/secrets.yaml.example

@@ -0,0 +1,8 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: convertx-secrets
+  namespace: converters
+type: Opaque
+stringData:
+  jwt-secret: ""

crowdsec/k8s/crowdsec-values.yaml

@@ -0,0 +1,203 @@
+container_runtime: containerd
+
+agent:
+  acquisition: []
+  additionalAcquisition:
+    - labels:
+        type: traefik
+      limit: 1000
+      query: |
+        {namespace="traefik"}
+      source: loki
+      url: http://loki.prometheus.svc.cluster.local:3100/
+      wait_for_ready: 30s
+  env:
+    - name: COLLECTIONS
+      value: crowdsecurity/traefik crowdsecurity/base-http-scenarios
+    - name: DISABLE_COLLECTIONS
+      value: crowdsecurity/sshd
+    # Bans on 401/403 bursts hurt more than they protect: with L3 enforcement
+    # a false positive cuts the IP off everything (SSH included), and past
+    # incidents show legit automation (deploy runner, mesh peers, registry
+    # pulls) tripping this probe. Probing/XSS/SQLi/CVE scenarios stay.
+    - name: DISABLE_SCENARIOS
+      value: crowdsecurity/http-generic-bf
+  metrics:
+    enabled: true
+    serviceMonitor:
+      additionalLabels:
+        release: prometheus-stack
+      enabled: true
+  # Static machine identity: agent pods mount pre-created LAPI credentials
+  # (Secret crowdsec-agent-credentials, key local_api_credentials.yaml)
+  # at the exact path the agent entrypoint expects. Together with the
+  # patched register-init (enforced by janitor-cronjob.yaml) the agent
+  # never calls `cscli lapi register` in steady state, so pod names,
+  # restarts and reboots can no longer break it.
+  extraVolumes:
+    - name: static-creds
+      secret:
+        secretName: crowdsec-agent-credentials
+        items:
+          - key: local_api_credentials.yaml
+            path: local_api_credentials.yaml
+  extraVolumeMounts:
+    - name: static-creds
+      mountPath: /tmp_config/local_api_credentials.yaml
+      subPath: local_api_credentials.yaml
+      readOnly: true
+  resources:
+    limits:
+      cpu: 200m
+      memory: 500Mi
+    requests:
+      cpu: 50m
+      memory: 100Mi
+
+config:
+  parsers:
+    s02-enrich:
+      mobile-whitelist.yaml: |
+        name: forust/mobile-whitelist
+        description: "Whitelist SWAN/4ka mobile network"
+        whitelist:
+          reason: "Mobile IP whitelist"
+          cidr:
+            - "84.245.64.0/18"
+      # CrowdSec's own guidance: CIDR allowlisting belongs at the parser stage.
+      # A parser whitelist discards the event before it reaches a bucket, so
+      # these addresses never produce an overflow and never become a decision.
+      # A postoverflow whitelist is checked only *after* the ban exists, and
+      # the bouncer answers 403 for as long as it does - which is a window we
+      # do not want the deploy sitting in.
+      local-network.yaml: |
+        name: forust/local-network
+        description: "Whitelist loopback, private and VPN networks"
+        whitelist:
+          reason: "Local network"
+          cidr:
+            - "127.0.0.0/8"
+            - "10.0.0.0/8"
+            - "172.16.0.0/12"
+            - "192.168.0.0/16"
+            # CGNAT range (RFC 6598). The workstation and the k0s node live
+            # here on WireGuard, and 100.64.0.0/10 is not covered by the
+            # RFC 1918 blocks above.
+            - "100.64.0.0/10"
+            - "169.254.0.0/16"
+            - "fc00::/7"
+            - "fe80::/10"
+      vps-whitelist.yaml: |
+        name: forust/vps-whitelist
+        description: "Whitelist static VPS"
+        whitelist:
+          reason: "VPS"
+          ip:
+            - "193.181.211.79"
+
+  postoverflows:
+    s01-whitelist:
+      # The one whitelist that has to stay here: resolving a hostname is a
+      # network call, and the docs put expensive lookups in postoverflows on
+      # purpose - it runs only when a bucket actually overflows.
+      # ddns.forust.xyz is the public home address, not a private one, so
+      # forust/local-network does not cover it.
+      home-dynamic-ip.yaml: |
+        name: forust/home-dynamic-ip
+        description: "Whitelist home dynamic IP"
+        whitelist:
+          reason: "Home dynamic IP"
+          expression:
+            - evt.Overflow.Alert.Source.IP in LookupHost("ddns.forust.xyz")
+
+  # LAPI-only main config override, merged over config.yaml. NOTE: the
+  # chart's own default for this key is REPLACED, not merged, so its
+  # auto_registration block is repeated verbatim below - drop it and the
+  # agent can no longer register itself.
+  config.yaml.local: |
+    api:
+      server:
+        auto_registration: # Activate if not using TLS for authentication
+          enabled: true
+          token: "${REGISTRATION_TOKEN}" # /!\ Do not modify this variable (auto-generated and handled by the chart)
+          allowed_ranges: # /!\ Make sure to adapt to the pod IP ranges used by your cluster
+            - "127.0.0.1/32"
+            - "192.168.0.0/16"
+            - "10.0.0.0/8"
+            - "172.16.0.0/12"
+        # This homelab has no egress to console.crowdsec.cloud: DNS does
+        # not resolve. The LAPI kept trying anyway ("Signal push: N
+        # signals to push", "capi metrics: sending" every 10s) and each
+        # attempt sat on a resolver timeout WHILE HOLDING A WRITE
+        # TRANSACTION, which is what kept stalling per-request decision
+        # lookups even with WAL enabled. Nothing to share and nothing to
+        # pull - turn the Central API off instead of letting it block the
+        # only database writer we have.
+        online_client:
+          sharing: false
+          pull:
+            community: false
+            blocklists: false
+        disable_usage_metrics_export: true
+    db_config:
+      # SQLite without WAL serialises every reader behind the writer's
+      # rollback journal, and the LAPI writes constantly: the agent pushes
+      # Traefik alerts read from Loki, the metrics collector counts
+      # decisions, the bouncer touches "last pull" on every request.
+      # Symptom: decision lookups taking 10-30s (and a second connection
+      # that could not even open the database) while the LAPI sat at 28m
+      # CPU - the process was blocked in fsync, not computing. Every
+      # bouncer-protected request then blew through the plugin timeout and
+      # fail-closed with 403, on every site at once.
+      # The PVC is local-path-retain (hostPath), not a network share, so
+      # WAL is safe here; the crowdsec docs recommend it for exactly this
+      # ("allowing more concurrency in SQLite that will improve
+      # performances in most scenarios").
+      use_wal: true
+      # Keeps the alert table bounded. At the 5000/7d default the file
+      # reached 54MB in 15 days off the Traefik access log alone, and the
+      # metrics collector counts decisions on a timer; a smaller working
+      # set means fewer full scans. Crowdsec only prunes - SQLite never
+      # shrinks the file, so the size stays until a manual VACUUM.
+      flush:
+        max_items: 1000
+        max_age: 24h
+
+lapi:
+  env:
+    - name: COLLECTIONS
+      value: crowdsecurity/traefik crowdsecurity/base-http-scenarios
+    - name: DISABLE_COLLECTIONS
+      value: crowdsecurity/linux crowdsecurity/sshd
+  metrics:
+    enabled: true
+    serviceMonitor:
+      additionalLabels:
+        release: prometheus-stack
+      enabled: true
+  persistentVolume:
+    config:
+      enabled: true
+      size: 100Mi
+      storageClassName: local-path-retain
+    data:
+      enabled: true
+      size: 1Gi
+      storageClassName: local-path-retain
+  # LAPI answers a blocking /v1/decisions lookup for EVERY bouncer-protected
+  # request (whole Traefik front door), so it is the hot path of the proxy.
+  # At 400m/500Mi it went CPU-throttled and idle lookups measured 1.3-7.4s,
+  # which pushed requests into the bouncer's fail-closed 403.
+  # Single replica on purpose: LAPI is stateful (BoltDB on the `data` PVC,
+  # credentials on the `config` PVC) - two replicas sharing those RWO
+  # volumes would corrupt the decision store. Scale up CPU, not replicas.
+  resources:
+    limits:
+      cpu: 1500m
+      memory: 1Gi
+    requests:
+      cpu: 250m
+      memory: 500Mi
+  service:
+    type: ClusterIP
+  storeLAPICscliCredentialsInSecret: true

crowdsec/k8s/grafana-dashboards.yaml

@@ -0,0 +1,32 @@
+apiVersion: v1
+data:
+  crowdsec-overview.json: "{\n  \"__inputs\": [\n    {\n      \"name\": \"DS_PROMETHEUS\",\n      \"label\": \"Prometheus\",\n      \"description\": \"\",\n      \"type\": \"datasource\",\n      \"pluginId\": \"prometheus\",\n      \"pluginName\": \"Prometheus\"\n    }\n  ],\n  \"__requires\": [\n    {\n      \"type\": \"grafana\",\n      \"id\": \"grafana\",\n      \"name\": \"Grafana\",\n      \"version\": \"8.1.2\"\n    },\n    {\n      \"type\": \"panel\",\n      \"id\": \"graph\",\n      \"name\": \"Graph (old)\",\n      \"version\": \"\"\n    },\n    {\n      \"type\": \"datasource\",\n      \"id\": \"prometheus\",\n      \"name\": \"Prometheus\",\n      \"version\": \"1.0.0\"\n    },\n    {\n      \"type\": \"panel\",\n      \"id\": \"stat\",\n      \"name\": \"Stat\",\n      \"version\": \"\"\n    },\n    {\n      \"type\": \"panel\",\n      \"id\": \"timeseries\",\n      \"name\": \"Time series\",\n      \"version\": \"\"\n    }\n  ],\n  \"annotations\": {\n    \"list\": [\n      {\n        \"builtIn\": 1,\n        \"datasource\": \"-- Grafana --\",\n        \"enable\": true,\n        \"hide\": true,\n        \"iconColor\": \"rgba(0, 211, 255, 1)\",\n        \"name\": \"Annotations & Alerts\",\n        \"target\": {\n          \"limit\": 100,\n          \"matchAny\": false,\n          \"tags\": [],\n          \"type\": \"dashboard\"\n        },\n        \"type\": \"dashboard\"\n      }\n    ]\n  },\n  \"editable\": true,\n  \"gnetId\": null,\n  \"graphTooltip\": 0,\n  \"id\": null,\n  \"links\": [],\n  \"panels\": [\n    {\n      \"collapsed\": false,\n      \"datasource\": null,\n      \"gridPos\": {\n        \"h\": 1,\n        \"w\": 24,\n        \"x\": 0,\n        \"y\": 0\n      },\n      \"id\": 24,\n      \"panels\": [],\n      \"title\": \"Summary\",\n      \"type\": \"row\"\n    },\n    {\n      \"cacheTimeout\": null,\n      \"datasource\": \"${DS_PROMETHEUS}\",\n      \"fieldConfig\": {\n        \"defaults\": {\n          \"color\": {\n            \"mode\": \"thresholds\"\n          },\n          \"mappings\": [\n            {\n              \"options\": {\n                \"match\": \"null\",\n                \"result\": {\n                  \"text\": \"N/A\"\n                }\n              },\n              \"type\": \"special\"\n            }\n          ],\n          \"thresholds\": {\n            \"mode\": \"absolute\",\n            \"steps\": [\n              {\n                \"color\": \"#E02F44\",\n                \"value\": null\n              },\n              {\n                \"color\": \"#E02F44\",\n                \"value\": 10\n              },\n              {\n                \"color\": \"#299c46\",\n                \"value\": 10\n              }\n            ]\n          },\n          \"unit\": \"none\"\n        },\n        \"overrides\": []\n      },\n      \"gridPos\": {\n        \"h\": 8,\n        \"w\": 6,\n        \"x\": 0,\n        \"y\": 1\n      },\n      \"id\": 2,\n      \"interval\": null,\n      \"links\": [],\n      \"maxDataPoints\": 100,\n      \"options\": {\n        \"colorMode\": \"background\",\n        \"graphMode\": \"none\",\n        \"justifyMode\": \"auto\",\n        \"orientation\": \"horizontal\",\n        \"reduceOptions\": {\n          \"calcs\": [\n            \"lastNotNull\"\n          ],\n          \"fields\": \"\",\n          \"values\": false\n        },\n        \"text\": {},\n        \"textMode\": \"auto\"\n      },\n      \"pluginVersion\": \"8.1.2\",\n      \"targets\": [\n        {\n          \"exemplar\": true,\n          \"expr\": \"count(cs_info)\",\n          \"interval\": \"\",\n          \"legendFormat\": \"\",\n          \"refId\": \"A\"\n        }\n      ],\n      \"timeFrom\": null,\n      \"timeShift\": null,\n      \"title\": \"Running Crowdsec\",\n      \"transparent\": true,\n      \"type\": \"stat\"\n    },\n    {\n      \"aliasColors\": {},\n      \"bars\": false,\n      \"dashLength\": 10,\n      \"dashes\": false,\n      \"datasource\": \"${DS_PROMETHEUS}\",\n      \"decimals\": 1,\n      \"fieldConfig\": {\n        \"defaults\": {\n          \"links\": []\n        },\n        \"overrides\": []\n      },\n      \"fill\": 1,\n      \"fillGradient\": 0,\n      \"gridPos\": {\n        \"h\": 8,\n        \"w\": 18,\n        \"x\": 6,\n        \"y\": 1\n      },\n      \"hiddenSeries\": false,\n      \"id\": 8,\n      \"legend\": {\n        \"alignAsTable\": true,\n        \"avg\": false,\n        \"current\": false,\n        \"max\": false,\n        \"min\": false,\n        \"rightSide\": true,\n        \"show\": true,\n        \"sort\": \"total\",\n        \"sortDesc\": true,\n        \"total\": true,\n        \"values\": true\n      },\n      \"lines\": true,\n      \"linewidth\": 1,\n      \"nullPointMode\": \"null\",\n      \"options\": {\n        \"alertThreshold\": true\n      },\n      \"percentage\": false,\n      \"pluginVersion\": \"8.1.2\",\n      \"pointradius\": 2,\n      \"points\": false,\n      \"renLine truncated
+kind: ConfigMap
+metadata:
+  labels:
+    app.kubernetes.io/managed-by: manual
+    grafana_dashboard: "1"
+  name: crowdsec-crowdsec-overview
+  namespace: prometheus
+---
+apiVersion: v1
+data:
+  crowdsec-lapi-metrics.json: "{\n  \"__inputs\": [\n    {\n      \"name\": \"DS_PROMETHEUS\",\n      \"label\": \"Prometheus\",\n      \"description\": \"\",\n      \"type\": \"datasource\",\n      \"pluginId\": \"prometheus\",\n      \"pluginName\": \"Prometheus\"\n    }\n  ],\n  \"__requires\": [\n    {\n      \"type\": \"panel\",\n      \"id\": \"bargauge\",\n      \"name\": \"Bar gauge\",\n      \"version\": \"\"\n    },\n    {\n      \"type\": \"grafana\",\n      \"id\": \"grafana\",\n      \"name\": \"Grafana\",\n      \"version\": \"8.1.2\"\n    },\n    {\n      \"type\": \"datasource\",\n      \"id\": \"prometheus\",\n      \"name\": \"Prometheus\",\n      \"version\": \"1.0.0\"\n    }\n  ],\n  \"annotations\": {\n    \"list\": [\n      {\n        \"builtIn\": 1,\n        \"datasource\": \"-- Grafana --\",\n        \"enable\": true,\n        \"hide\": true,\n        \"iconColor\": \"rgba(0, 211, 255, 1)\",\n        \"name\": \"Annotations & Alerts\",\n        \"target\": {\n          \"limit\": 100,\n          \"matchAny\": false,\n          \"tags\": [],\n          \"type\": \"dashboard\"\n        },\n        \"type\": \"dashboard\"\n      }\n    ]\n  },\n  \"editable\": true,\n  \"gnetId\": null,\n  \"graphTooltip\": 0,\n  \"id\": null,\n  \"iteration\": 1655915193937,\n  \"links\": [],\n  \"panels\": [\n    {\n      \"collapsed\": false,\n      \"datasource\": null,\n      \"gridPos\": {\n        \"h\": 1,\n        \"w\": 24,\n        \"x\": 0,\n        \"y\": 0\n      },\n      \"id\": 10,\n      \"panels\": [],\n      \"title\": \"Agents\",\n      \"type\": \"row\"\n    },\n    {\n      \"datasource\": \"${DS_PROMETHEUS}\",\n      \"fieldConfig\": {\n        \"defaults\": {\n          \"color\": {\n            \"mode\": \"thresholds\"\n          },\n          \"mappings\": [],\n          \"thresholds\": {\n            \"mode\": \"absolute\",\n            \"steps\": [\n              {\n                \"color\": \"green\",\n                \"value\": null\n              },\n              {\n                \"color\": \"red\",\n                \"value\": 80\n              }\n            ]\n          }\n        },\n        \"overrides\": []\n      },\n      \"gridPos\": {\n        \"h\": 8,\n        \"w\": 12,\n        \"x\": 0,\n        \"y\": 1\n      },\n      \"id\": 2,\n      \"options\": {\n        \"displayMode\": \"gradient\",\n        \"orientation\": \"vertical\",\n        \"reduceOptions\": {\n          \"calcs\": [\n            \"lastNotNull\"\n          ],\n          \"fields\": \"\",\n          \"values\": false\n        },\n        \"showUnfilled\": false,\n        \"text\": {}\n      },\n      \"pluginVersion\": \"8.1.2\",\n      \"repeat\": \"query0\",\n      \"repeatDirection\": \"h\",\n      \"targets\": [\n        {\n          \"exemplar\": false,\n          \"expr\": \"sum(rate(cs_lapi_request_duration_seconds_bucket{endpoint=\\\"/v1/watchers/login\\\", instance=\\\"$lapi\\\"}[$__rate_interval])) by (le)\",\n          \"format\": \"heatmap\",\n          \"interval\": \"\",\n          \"legendFormat\": \"{{le}}\",\n          \"refId\": \"A\"\n        }\n      ],\n      \"title\": \"Agents Login\",\n      \"type\": \"heatmap\"\n    },\n    {\n      \"datasource\": \"${DS_PROMETHEUS}\",\n      \"fieldConfig\": {\n        \"defaults\": {\n          \"color\": {\n            \"mode\": \"thresholds\"\n          },\n          \"mappings\": [],\n          \"thresholds\": {\n            \"mode\": \"absolute\",\n            \"steps\": [\n              {\n                \"color\": \"green\",\n                \"value\": null\n              }\n            ]\n          },\n          \"unit\": \"none\"\n        },\n        \"overrides\": []\n      },\n      \"gridPos\": {\n        \"h\": 8,\n        \"w\": 12,\n        \"x\": 12,\n        \"y\": 1\n      },\n      \"id\": 6,\n      \"options\": {\n        \"displayMode\": \"gradient\",\n        \"orientation\": \"auto\",\n        \"reduceOptions\": {\n          \"calcs\": [\n            \"lastNotNull\"\n          ],\n          \"fields\": \"\",\n          \"values\": false\n        },\n        \"showUnfilled\": false,\n        \"text\": {}\n      },\n      \"pluginVersion\": \"8.1.2\",\n      \"targets\": [\n        {\n          \"exemplar\": true,\n          \"expr\": \"sum(rate(cs_lapi_request_duration_seconds_bucket{endpoint=\\\"/v1/watchers/login\\\"}[$__rate_interval])) by (le)\",\n          \"format\": \"heatmap\",\n          \"interval\": \"\",\n          \"legendFormat\": \"{{le}}\",\n          \"refId\": \"A\"\n        }\n      ],\n      \"title\": \"Heartbeat\",\n      \"type\": \"heatmap\"\n    },\n    {\n      \"collapsed\": false,\n      \"datasource\": null,\n      \"gridPos\": {\n        \"h\": 1,\n        \"w\": 24,\n        \"x\": 0,\n        \"y\": 9\n      },\n      \"id\": 12,\n      \"panels\": [],\n      \"title\": \"Decisions\",\n      \"type\": \"row\"\n    },\n    {\n      \"datasource\": \"${DS_PROMETHEUS}\",\n  Line truncated
+kind: ConfigMap
+metadata:
+  labels:
+    app.kubernetes.io/managed-by: manual
+    grafana_dashboard: "1"
+  name: crowdsec-crowdsec-lapi-metrics
+  namespace: prometheus
+---
+apiVersion: v1
+data:
+  crowdsec-insight.json: "{\n  \"__inputs\": [\n    {\n      \"name\": \"DS_PROMETHEUS\",\n      \"label\": \"Prometheus\",\n      \"description\": \"\",\n      \"type\": \"datasource\",\n      \"pluginId\": \"prometheus\",\n      \"pluginName\": \"Prometheus\"\n    }\n  ],\n  \"__requires\": [\n    {\n      \"type\": \"panel\",\n      \"id\": \"bargauge\",\n      \"name\": \"Bar gauge\",\n      \"version\": \"\"\n    },\n    {\n      \"type\": \"panel\",\n      \"id\": \"gauge\",\n      \"name\": \"Gauge\",\n      \"version\": \"\"\n    },\n    {\n      \"type\": \"grafana\",\n      \"id\": \"grafana\",\n      \"name\": \"Grafana\",\n      \"version\": \"8.1.2\"\n    },\n    {\n      \"type\": \"datasource\",\n      \"id\": \"prometheus\",\n      \"name\": \"Prometheus\",\n      \"version\": \"1.0.0\"\n    },\n    {\n      \"type\": \"panel\",\n      \"id\": \"stat\",\n      \"name\": \"Stat\",\n      \"version\": \"\"\n    }\n  ],\n  \"annotations\": {\n    \"list\": [\n      {\n        \"builtIn\": 1,\n        \"datasource\": \"-- Grafana --\",\n        \"enable\": true,\n        \"hide\": true,\n        \"iconColor\": \"rgba(0, 211, 255, 1)\",\n        \"name\": \"Annotations & Alerts\",\n        \"target\": {\n          \"limit\": 100,\n          \"matchAny\": false,\n          \"tags\": [],\n          \"type\": \"dashboard\"\n        },\n        \"type\": \"dashboard\"\n      }\n    ]\n  },\n  \"editable\": true,\n  \"gnetId\": null,\n  \"graphTooltip\": 0,\n  \"id\": null,\n  \"iteration\": 1655915159751,\n  \"links\": [],\n  \"panels\": [\n    {\n      \"collapsed\": true,\n      \"datasource\": null,\n      \"gridPos\": {\n        \"h\": 1,\n        \"w\": 24,\n        \"x\": 0,\n        \"y\": 0\n      },\n      \"id\": 22,\n      \"panels\": [\n        {\n          \"cacheTimeout\": null,\n          \"datasource\": \"${DS_PROMETHEUS}\",\n          \"fieldConfig\": {\n            \"defaults\": {\n              \"color\": {\n                \"mode\": \"thresholds\"\n              },\n              \"mappings\": [\n                {\n                  \"options\": {\n                    \"match\": \"null\",\n                    \"result\": {\n                      \"text\": \"N/A\"\n                    }\n                  },\n                  \"type\": \"special\"\n                }\n              ],\n              \"thresholds\": {\n                \"mode\": \"absolute\",\n                \"steps\": [\n                  {\n                    \"color\": \"green\",\n                    \"value\": null\n                  },\n                  {\n                    \"color\": \"red\",\n                    \"value\": 80\n                  }\n                ]\n              },\n              \"unit\": \"dateTimeAsIso\"\n            },\n            \"overrides\": []\n          },\n          \"gridPos\": {\n            \"h\": 9,\n            \"w\": 5,\n            \"x\": 2,\n            \"y\": 1\n          },\n          \"id\": 2,\n          \"interval\": null,\n          \"links\": [],\n          \"maxDataPoints\": 100,\n          \"options\": {\n            \"colorMode\": \"none\",\n            \"graphMode\": \"none\",\n            \"justifyMode\": \"auto\",\n            \"orientation\": \"horizontal\",\n            \"reduceOptions\": {\n              \"calcs\": [\n                \"lastNotNull\"\n              ],\n              \"fields\": \"\",\n              \"values\": false\n            },\n            \"text\": {},\n            \"textMode\": \"auto\"\n          },\n          \"pluginVersion\": \"8.1.2\",\n          \"targets\": [\n            {\n              \"exemplar\": true,\n              \"expr\": \"(process_start_time_seconds{instance=\\\"$instance\\\"})*1000\",\n              \"interval\": \"\",\n              \"legendFormat\": \"{{instance}}\",\n              \"refId\": \"A\"\n            }\n          ],\n          \"timeFrom\": null,\n          \"timeShift\": null,\n          \"title\": \"Up since\",\n          \"type\": \"stat\"\n        },\n        {\n          \"datasource\": \"${DS_PROMETHEUS}\",\n          \"fieldConfig\": {\n            \"defaults\": {\n              \"displayName\": \"\",\n              \"mappings\": [],\n              \"thresholds\": {\n                \"mode\": \"absolute\",\n                \"steps\": [\n                  {\n                    \"color\": \"green\",\n                    \"value\": null\n                  }\n                ]\n              },\n              \"unit\": \"decbytes\"\n            },\n            \"overrides\": []\n          },\n          \"gridPos\": {\n            \"h\": 9,\n            \"w\": 5,\n            \"x\": 7,\n            \"y\": 1\n          },\n          \"id\": 4,\n          \"options\": {\n            \"orientation\": \"auto\",\n            \"reduceOptions\": {\n              \"calcs\": [\n                \"mean\"\n              ],\n              \"fields\": \"\",\n              \"values\": false\n            },\Line truncated
+kind: ConfigMap
+metadata:
+  labels:
+    app.kubernetes.io/managed-by: manual
+    grafana_dashboard: "1"
+  name: crowdsec-crowdsec-insight
+  namespace: prometheus

crowdsec/k8s/janitor-cronjob.yaml

@@ -0,0 +1,202 @@
+# CrowdSec self-healing: static machine identity + enforcement loops.
+#
+# Problem it fixes: the chart's agent init container runs
+#   `cscli lapi register --machine "$POD_NAME" ...`
+# unconditionally. Credentials live in an emptyDir, the machine row lives
+# in LAPI's persistent DB. Any init re-run for an already-known pod name
+# (kubelet restart, node reboot) dies with
+#   403 Forbidden: user '<pod>' already exist
+# and the DaemonSet pod sticks in Init forever. Every DS restart also
+# leaves an orphan machine row that is never cleaned.
+#
+# Design (name-independent):
+#   * Agent identity is a STATIC machine `crowdsec-agent-workstation`
+#     whose password lives in Secret `crowdsec-agent-credentials`
+#     (created once, manually - like all other secrets in this repo).
+#     The secret is mounted into agent pods at
+#     /tmp_config/local_api_credentials.yaml (see extraVolumeMounts in
+#     crowdsec-values.yaml), which is exactly the path the agent's main
+#     container copies into place at startup.
+#   * The DS init command is patched (strategic merge, by container name)
+#     to SKIP registration when that file exists, keeping the legacy
+#     register path only as fallback. Detection marker in the patched
+#     command: `[ -s /tmp_config`.
+#   * This CronJob enforces the desired state hourly, so recovery is
+#     automatic even after `helm upgrade` reverts the DS patch or the
+#     LAPI database is wiped:
+#       1. patch DS init if it still has the unconditional register
+#          (no-op otherwise - no restart churn);
+#       2. prune machines with no heartbeat for 2h (orphan hygiene);
+#       3. ensure the static machine exists, recreating it with the
+#          Secret password if missing (agent retry loops reconnect
+#          on their own - same name + same password);
+#       4. prune bouncer entries idle for 30d.
+#
+# It used to also delete LePresidente/http-generic-403-bf decisions hourly.
+# That was a workaround for the bouncer failing closed on a slow LAPI and
+# 403-ing the deploy runner into a 4h ban. The bouncer now polls decisions
+# into a cache and never blocks on an unreachable LAPI, so it cannot
+# manufacture those 403s any more, and the scenario only fires against real
+# scanners - deleting their decisions hourly was undoing a working ban.
+#
+# Manual apply (crowdsec/k8s is NOT managed by deploy.yaml):
+#   kubectl apply -f crowdsec/k8s/janitor-cronjob.yaml
+# Force a run:
+#   kubectl create job -n crowdsec --from=cronjob/crowdsec-janitor janitor-now
+#
+# Helm upgrades: the janitor's strategic patch puts the DS field under
+# the `kubectl-patch` field manager, so a plain `helm upgrade` FAILS
+# with an SSA conflict on initContainers[].command. Procedure:
+#   1. revert init to chart state (kills the conflict):
+#        helm template crowdsec crowdsec/crowdsec --version <ver> \
+#          -n crowdsec -f crowdsec/k8s/crowdsec-values.yaml > /tmp/r.yaml
+#        python3 -c "import yaml,json; ..." # build revert patch from
+#        the rendered DaemonSet init command, then
+#        kubectl patch ds crowdsec-agent -n crowdsec \
+#          --type strategic -p "\$(cat /tmp/revert_patch.json)"
+#   2. helm upgrade --install crowdsec ... (no --force needed)
+#   3. janitor-now right away (upgrade reverts init; new pods would
+#      sit in Init until the next hourly run otherwise).
+#
+# One-time bootstrap (order matters):
+#   1. Create Secret + static machine (see commands in chat).
+#   2. Apply this file, trigger janitor-now, wait for agent 1/1.
+#   3. One-time orphan cleanup:
+#        kubectl exec -n crowdsec deploy/crowdsec-lapi -- \
+#          cscli machines prune --duration 1h --force
+#   4. Only then `helm upgrade` crowdsec with the extraVolumes values.
+#      Upgrade reverts the DS patch; trigger janitor-now right after it
+#      (otherwise new pods sit in Init until the next hourly run, then
+#      self-heal anyway).
+#
+# Password rotation: update the Secret, delete the machine
+# (`cscli machines delete crowdsec-agent-workstation`), trigger
+# janitor-now (recreates it), then `kubectl rollout restart
+# ds/crowdsec-agent -n crowdsec` (agent reads the file at startup only).
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: crowdsec-janitor
+  namespace: crowdsec
+  labels:
+    app.kubernetes.io/part-of: crowdsec
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: crowdsec-janitor
+  namespace: crowdsec
+  labels:
+    app.kubernetes.io/part-of: crowdsec
+rules:
+  - apiGroups: [""]
+    resources: ["pods"]
+    verbs: ["get", "list"]
+  - apiGroups: [""]
+    resources: ["pods/exec"]
+    verbs: ["create"]
+  - apiGroups: ["apps"]
+    resources: ["daemonsets"]
+    verbs: ["get", "patch"]
+  # `kubectl exec deploy/<name>` resolves deploy -> replicaset -> pod,
+  # which needs read access to these (exec itself is pods/exec above).
+  - apiGroups: ["apps"]
+    resources: ["deployments", "replicasets"]
+    verbs: ["get", "list"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: crowdsec-janitor
+  namespace: crowdsec
+  labels:
+    app.kubernetes.io/part-of: crowdsec
+subjects:
+  - kind: ServiceAccount
+    name: crowdsec-janitor
+    namespace: crowdsec
+roleRef:
+  kind: Role
+  name: crowdsec-janitor
+  apiGroup: rbac.authorization.k8s.io
+---
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: crowdsec-janitor
+  namespace: crowdsec
+  labels:
+    app.kubernetes.io/part-of: crowdsec
+spec:
+  schedule: "17 * * * *"
+  concurrencyPolicy: Forbid
+  successfulJobsHistoryLimit: 3
+  failedJobsHistoryLimit: 3
+  jobTemplate:
+    spec:
+      activeDeadlineSeconds: 300
+      template:
+        metadata:
+          labels:
+            app.kubernetes.io/part-of: crowdsec
+        spec:
+          serviceAccountName: crowdsec-janitor
+          restartPolicy: OnFailure
+          containers:
+            - name: janitor
+              # Same image the chart itself uses for registration jobs;
+              # IfNotPresent so it works while the node is offline
+              # (layer cached from the chart install).
+              image: alpine/kubectl:latest
+              imagePullPolicy: IfNotPresent
+              env:
+                - name: AGENT_PASSWORD
+                  valueFrom:
+                    secretKeyRef:
+                      name: crowdsec-agent-credentials
+                      key: password
+              command:
+                - /bin/sh
+                - -c
+                - |
+                  set -eu
+                  LAPI_EXEC="kubectl exec -n crowdsec deploy/crowdsec-lapi --"
+                  echo "== 1. enforce patched agent init =="
+                  CUR=$(kubectl get ds crowdsec-agent -n crowdsec \
+                    -o jsonpath='{.spec.template.spec.initContainers[0].command[2]}')
+                  case "$CUR" in
+                    *'-s /tmp_config'*)
+                      echo "init already patched"
+                      ;;
+                    *)
+                      echo "patching init"
+                      WAIT='until nc "$LAPI_HOST" "$LAPI_PORT" -z'
+                      WAIT="$WAIT; do echo waiting for lapi to start; sleep 5; done"
+                      LINK='ln -s /staging/etc/crowdsec /etc/crowdsec'
+                      REG='cscli lapi register --machine "$USERNAME"'
+                      REG="$REG -u \"\$LAPI_URL\" --token \"\$REGISTRATION_TOKEN\""
+                      CREDS=/tmp_config/local_api_credentials.yaml
+                      CMD="$WAIT; $LINK; [ -s $CREDS ] || {"
+                      CMD="$CMD $REG && cp"
+                      CMD="$CMD /etc/crowdsec/local_api_credentials.yaml $CREDS; }"
+                      ESC=$(printf '%s' "$CMD" | sed 's/"/\\"/g')
+                      PATCH='{"spec":{"template":{"spec":{"initContainers":'
+                      PATCH=$PATCH'[{"name":"wait-for-lapi-and-register",'
+                      PATCH=$PATCH'"command":["sh","-c","'$ESC'"]}]}}}}'
+                      kubectl patch ds crowdsec-agent -n crowdsec \
+                        --type strategic -p "$PATCH"
+                      ;;
+                  esac
+                  echo "== 2. prune orphan machines (no heartbeat for 2h) =="
+                  $LAPI_EXEC cscli machines prune --duration 2h --force
+                  echo "== 3. ensure static machine exists =="
+                  if $LAPI_EXEC cscli machines inspect \
+                    crowdsec-agent-workstation >/dev/null 2>&1; then
+                    echo "static machine present"
+                  else
+                    echo "recreating static machine"
+                    $LAPI_EXEC cscli machines add crowdsec-agent-workstation \
+                      --password "$AGENT_PASSWORD" --force
+                  fi
+                  echo "== 4. prune stale bouncers (no pull for 30d) =="
+                  $LAPI_EXEC cscli bouncers prune -d 720h --force

crowdsec/k8s/namespace.yaml

@@ -0,0 +1,6 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: crowdsec
+  labels:
+    app.kubernetes.io/part-of: crowdsec

crowdsec/k8s/network-policy.yaml

@@ -0,0 +1,34 @@
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: crowdsec-lapi
+  namespace: crowdsec
+spec:
+  podSelector:
+    matchLabels:
+      k8s-app: crowdsec
+      type: lapi
+  policyTypes:
+    - Ingress
+  ingress:
+    - from:
+        - namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: traefik
+          podSelector:
+            matchLabels:
+              app.kubernetes.io/name: traefik
+        - podSelector:
+            matchLabels:
+              k8s-app: crowdsec
+              type: agent
+      ports:
+        - protocol: TCP
+          port: 8080
+    - from:
+        - namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: prometheus
+      ports:
+        - protocol: TCP
+          port: 6060

dockmon/compose.yaml

@@ -1,54 +1,46 @@
 services:
-   dockmon:
-      image: darthnorse/dockmon:latest
-      container_name: dockmon
-      restart: unless-stopped
-      ports:
-         - 8000:443
-      environment:
-         - TZ=Europe/Bratislava
-      volumes:
-         - ./data:/app/data
-         - /var/run/docker.sock:/var/run/docker.sock
-      healthcheck:
-         test: [ "CMD", "curl", "-k", "-f", "https://localhost:443/health" ]
-         interval: 30s
-         timeout: 10s
-         retries: 3
-      networks:
-         - proxy
-      labels:
-         - "traefik.enable=true"
-         - "traefik.docker.network=proxy"
+  dockmon:
+    image: darthnorse/dockmon:2.5.0
+    container_name: dockmon
+    restart: unless-stopped
+    # ports:
+    #   - 8000:443
+    volumes:
+      - data:/app/data
+      - /var/run/docker.sock:/var/run/docker.sock
+    healthcheck:
+      test: ["CMD", "curl", "-k", "-f", "https://localhost:443/health"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+    labels:
+      - "traefik.enable=true"
+      - "traefik.http.services.dockmon.loadbalancer.server.port=443"
+      - "traefik.http.services.dockmon.loadbalancer.server.scheme=https"
+      - "traefik.http.services.dockmon.loadbalancer.serverstransport=insecureTransport@file"
 
-         # Prod Router
-         - "traefik.http.routers.dockmon.rule=Host(`dockmon.forust.xyz`)"
-         - "traefik.http.routers.dockmon.entrypoints=websecure"
-         - "traefik.http.routers.dockmon.middlewares=security-chain@file"
-         - "traefik.http.routers.dockmon.service=dockmon"
-         - "traefik.http.routers.dockmon.tls=true"
-         - "traefik.http.services.dockmon.loadbalancer.server.port=443"
-         - "traefik.http.services.dockmon.loadbalancer.server.scheme=https"
-         - "traefik.http.services.dockmon.loadbalancer.serverstransport=insecureTransport@file"
-
-         # Local Router
-         - "traefik.http.routers.dockmon-local.rule=Host(`dockmon.workstation.internal`) || Host(`dockmon.internal`)"
-         - "traefik.http.routers.dockmon-local.entrypoints=websecure"
-         - "traefik.http.routers.dockmon-local.middlewares=security-headers@file"
-         - "traefik.http.routers.dockmon-local.service=dockmon"
-         - "traefik.http.routers.dockmon-local.tls=true"
-
-         # Dev Router
-         - "traefik.http.routers.dockmon-dev.rule=Host(`dockmon.gigaforust.internal`)"
-         - "traefik.http.routers.dockmon-dev.entrypoints=websecure"
-         - "traefik.http.routers.dockmon-dev.middlewares=security-chain@file"
-         - "traefik.http.routers.dockmon-dev.service=dockmon"
-         - "traefik.http.routers.dockmon-dev.tls=true"
-
-         - glance.name=dockmon
-         - glance.url=https://dockmon.forust.xyz/
-         - glance.description=Dockmon is a lightweight Docker container monitoring and management tool with a user-friendly web interface.
+      # Prod Router
+      - "traefik.http.routers.dockmon.rule=Host(`dockmon.forust.xyz`)"
+      - "traefik.http.routers.dockmon.entrypoints=websecure"
+      - "traefik.http.routers.dockmon.middlewares=security-headers@file"
+      - "traefik.http.routers.dockmon.tls.certresolver=letsencrypt"
+      # Local Router
+      - "traefik.http.routers.dockmon-local.rule=Host(`dockmon.workstation.internal`)"
+      - "traefik.http.routers.dockmon-local.entrypoints=websecure"
+      - "traefik.http.routers.dockmon-local.tls=true"
+      # Dev Router
+      - "traefik.http.routers.dockmon-dev.rule=Host(`dockmon.gigaforust.internal`)"
+      - "traefik.http.routers.dockmon-dev.entrypoints=websecure"
+      - "traefik.http.routers.dockmon-dev.tls=true"
 
+      # Glance Metadata
+      - glance.name=dockmon
+      - glance.url=https://dockmon.forust.xyz/
+      - glance.description=Dockmon is a lightweight Docker container monitoring and management tool with a user-friendly web interface.
+    networks:
+      - proxy
+volumes:
+  data:
 networks:
-   proxy:
-      external: true
+  proxy:
+    external: true

dockmon/k8s/certificates.yaml

@@ -0,0 +1,28 @@
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: dockmon-prod-tls
+  namespace: dockmon
+spec:
+  secretName: dockmon-prod-tls
+  dnsNames:
+    - dockmon.forust.xyz
+  issuerRef:
+    name: letsencrypt-prod
+    kind: ClusterIssuer
+---
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: internal-wildcard-tls
+  namespace: dockmon
+spec:
+  secretName: internal-wildcard-tls
+  dnsNames:
+    - "*.workstation.internal"
+    - "*.gigaforust.internal"
+    - workstation.internal
+    - gigaforust.internal
+  issuerRef:
+    name: internal-ca
+    kind: ClusterIssuer

dockmon/k8s/dockmon.yaml

@@ -0,0 +1,68 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: dockmon-service
+  namespace: dockmon
+spec:
+  clusterIP: None
+  selector:
+    app: dockmon
+  ports:
+    - port: 443
+      targetPort: 443
+---
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  name: dockmon-statefulset
+  namespace: dockmon
+spec:
+  serviceName: dockmon-service
+  replicas: 1
+  selector:
+    matchLabels:
+      app: dockmon
+  template:
+    metadata:
+      labels:
+        app: dockmon
+    spec:
+      containers:
+        - name: dockmon
+          image: darthnorse/dockmon:2.5.0
+          ports:
+            - containerPort: 443
+          volumeMounts:
+            - name: data
+              mountPath: /app/data
+            - name: docker-sock
+              mountPath: /var/run/docker.sock
+          livenessProbe:
+            httpGet:
+              path: /health
+              port: 443
+              scheme: HTTPS
+            initialDelaySeconds: 30
+            periodSeconds: 30
+            timeoutSeconds: 10
+            failureThreshold: 3
+          resources:
+            requests:
+              memory: "512Mi"
+              cpu: "200m"
+            limits:
+              memory: "1.5Gi"
+              cpu: "700m "
+      volumes:
+        - name: docker-sock
+          hostPath:
+            path: /var/run/docker.sock
+            type: Socket
+  volumeClaimTemplates:
+    - metadata:
+        name: data
+      spec:
+        accessModes: ["ReadWriteOnce"]
+        resources:
+          requests:
+            storage: 1Gi

dockmon/k8s/ingress.yaml

@@ -0,0 +1,45 @@
+apiVersion: traefik.io/v1alpha1
+kind: ServersTransport
+metadata:
+  name: dockmon-transport
+  namespace: dockmon
+spec:
+  insecureSkipVerify: true
+---
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
+metadata:
+  name: dockmon-prod
+  namespace: dockmon
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`dockmon.forust.xyz`)
+      kind: Rule
+      middlewares:
+        - name: security-headers@file
+      services:
+        - name: dockmon-service
+          port: 443
+          serversTransport: dockmon-transport
+  tls:
+    secretName: dockmon-prod-tls
+---
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
+metadata:
+  name: dockmon-local
+  namespace: dockmon
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`dockmon.workstation.internal`) || Host(`dockmon.gigaforust.internal`)
+      kind: Rule
+      services:
+        - name: dockmon-service
+          port: 443
+          serversTransport: dockmon-transport
+  tls:
+    secretName: internal-wildcard-tls

dockmon/k8s/namespace.yaml

@@ -0,0 +1,4 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: dockmon

downtify/compose.yaml

@@ -1,39 +1,31 @@
 services:
   downtify:
     container_name: downtify
-    image: ghcr.io/henriquesebastiao/downtify:latest
+    image: ghcr.io/henriquesebastiao/downtify:3.4.0
+    restart: unless-stopped
     # ports:
     #   - '7077:8000'
-    labels:
-      - traefik.enable=true
-      - traefik.http.services.downtify.loadbalancer.server.port=8000
-
-      # Prod Router 
-      - traefik.http.routers.downtify.rule=Host(`downtify.forust.xyz`)
-      - traefik.http.routers.downtify.entrypoints=websecure
-      - traefik.http.routers.downtify.middlewares=security-chain@file
-      - traefik.http.routers.downtify.service=downtify
-      - traefik.http.routers.downtify.tls=true
-
-      # Local Router
-      - traefik.http.routers.downtify-local.rule=Host(`downtify.workstation.internal`) || Host(`downtify.internal`)
-      - traefik.http.routers.downtify-local.entrypoints=websecure
-      - traefik.http.routers.downtify-local.middlewares=security-headers@file
-      - traefik.http.routers.downtify-local.service=downtify
-      - traefik.http.routers.downtify-local.tls=true
-
-      # Dev Router
-      - traefik.http.routers.downtify-dev.rule=Host(`downtify.gigaforust.internal`)
-      - traefik.http.routers.downtify-dev.entrypoints=websecure
-      - traefik.http.routers.downtify-dev.middlewares=security-chain@file
-      - traefik.http.routers.downtify-dev.service=downtify
-      - traefik.http.routers.downtify-dev.tls=true
-    networks:
-      - proxy
-
     volumes:
       - ./Downtify_downloads:/downloads
+    labels:
+      - "traefik.enable=true"
+      - "traefik.http.services.downtify.loadbalancer.server.port=8000"
 
+      # Prod Router
+      - "traefik.http.routers.downtify.rule=Host(`downtify.forust.xyz`)"
+      - "traefik.http.routers.downtify.entrypoints=websecure"
+      - "traefik.http.routers.downtify.middlewares=security-chain@file"
+      - "traefik.http.routers.downtify.tls.certresolver=letsencrypt"
+      # Local Router
+      - "traefik.http.routers.downtify-local.rule=Host(`downtify.workstation.internal`)"
+      - "traefik.http.routers.downtify-local.entrypoints=websecure"
+      - "traefik.http.routers.downtify-local.tls=true"
+      # Dev Router
+      - "traefik.http.routers.downtify-dev.rule=Host(`downtify.gigaforust.internal`)"
+      - "traefik.http.routers.downtify-dev.entrypoints=websecure"
+      - "traefik.http.routers.downtify-dev.tls=true"
+    networks:
+      - proxy
 networks:
   proxy:
     external: true

downtify/k8s/downtify.yaml

@@ -0,0 +1,60 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: downtify-service
+  namespace: downtify
+spec:
+  selector:
+    app: downtify
+  ports:
+    - port: 8000
+      targetPort: 8000
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: downtify-deployment
+  namespace: downtify
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: downtify
+  strategy:
+    type: Recreate
+  template:
+    metadata:
+      labels:
+        app: downtify
+    spec:
+      containers:
+        - name: downtify
+          image: ghcr.io/henriquesebastiao/downtify:3.4.0
+          ports:
+            - containerPort: 8000
+          volumeMounts:
+            - name: downloads
+              mountPath: /downloads
+          resources:
+            requests:
+              memory: "128Mi"
+              cpu: "200m"
+            limits:
+              memory: "1Gi"
+              cpu: "1"
+      volumes:
+        - name: downloads
+          persistentVolumeClaim:
+            claimName: downtify-downloads-pvc
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: downtify-downloads-pvc
+  namespace: downtify
+spec:
+  accessModes:
+    - ReadWriteOnce
+  resources:
+    requests:
+      storage: 10Gi

downtify/k8s/ingress.yaml

@@ -0,0 +1,35 @@
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
+metadata:
+  name: downtify-prod
+  namespace: downtify
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`downtify.forust.xyz`)
+      kind: Rule
+      middlewares:
+        - name: security-chain@file
+      services:
+        - name: downtify-service
+          port: 8000
+  tls:
+    secretName: downtify-prod-tls
+---
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
+metadata:
+  name: downtify-local
+  namespace: downtify
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`downtify.workstation.internal`) || Host(`downtify.gigaforust.internal`)
+      kind: Rule
+      services:
+        - name: downtify-service
+          port: 8000
+  tls:
+    secretName: internal-wildcard-tls

downtify/k8s/namespace.yaml

@@ -0,0 +1,4 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: downtify

dtek_notif/Dockerfile

@@ -1,13 +0,0 @@
-FROM python:3.9-alpine
-
-WORKDIR /app
-
-# Установка зависимостей
-COPY requirements.txt .
-RUN pip install --no-cache-dir -r requirements.txt
-
-# Копирование кода
-COPY main.py .
-COPY .env .
-# Запуск бота
-CMD ["python", "-u", "main.py"]

dtek_notif/LICENSE

@@ -1,373 +0,0 @@
-Mozilla Public License Version 2.0
-==================================
-
-1. Definitions
---------------
-
-1.1. "Contributor"
-    means each individual or legal entity that creates, contributes to
-    the creation of, or owns Covered Software.
-
-1.2. "Contributor Version"
-    means the combination of the Contributions of others (if any) used
-    by a Contributor and that particular Contributor's Contribution.
-
-1.3. "Contribution"
-    means Covered Software of a particular Contributor.
-
-1.4. "Covered Software"
-    means Source Code Form to which the initial Contributor has attached
-    the notice in Exhibit A, the Executable Form of such Source Code
-    Form, and Modifications of such Source Code Form, in each case
-    including portions thereof.
-
-1.5. "Incompatible With Secondary Licenses"
-    means
-
-    (a) that the initial Contributor has attached the notice described
-        in Exhibit B to the Covered Software; or
-
-    (b) that the Covered Software was made available under the terms of
-        version 1.1 or earlier of the License, but not also under the
-        terms of a Secondary License.
-
-1.6. "Executable Form"
-    means any form of the work other than Source Code Form.
-
-1.7. "Larger Work"
-    means a work that combines Covered Software with other material, in 
-    a separate file or files, that is not Covered Software.
-
-1.8. "License"
-    means this document.
-
-1.9. "Licensable"
-    means having the right to grant, to the maximum extent possible,
-    whether at the time of the initial grant or subsequently, any and
-    all of the rights conveyed by this License.
-
-1.10. "Modifications"
-    means any of the following:
-
-    (a) any file in Source Code Form that results from an addition to,
-        deletion from, or modification of the contents of Covered
-        Software; or
-
-    (b) any new file in Source Code Form that contains any Covered
-        Software.
-
-1.11. "Patent Claims" of a Contributor
-    means any patent claim(s), including without limitation, method,
-    process, and apparatus claims, in any patent Licensable by such
-    Contributor that would be infringed, but for the grant of the
-    License, by the making, using, selling, offering for sale, having
-    made, import, or transfer of either its Contributions or its
-    Contributor Version.
-
-1.12. "Secondary License"
-    means either the GNU General Public License, Version 2.0, the GNU
-    Lesser General Public License, Version 2.1, the GNU Affero General
-    Public License, Version 3.0, or any later versions of those
-    licenses.
-
-1.13. "Source Code Form"
-    means the form of the work preferred for making modifications.
-
-1.14. "You" (or "Your")
-    means an individual or a legal entity exercising rights under this
-    License. For legal entities, "You" includes any entity that
-    controls, is controlled by, or is under common control with You. For
-    purposes of this definition, "control" means (a) the power, direct
-    or indirect, to cause the direction or management of such entity,
-    whether by contract or otherwise, or (b) ownership of more than
-    fifty percent (50%) of the outstanding shares or beneficial
-    ownership of such entity.
-
-2. License Grants and Conditions
---------------------------------
-
-2.1. Grants
-
-Each Contributor hereby grants You a world-wide, royalty-free,
-non-exclusive license:
-
-(a) under intellectual property rights (other than patent or trademark)
-    Licensable by such Contributor to use, reproduce, make available,
-    modify, display, perform, distribute, and otherwise exploit its
-    Contributions, either on an unmodified basis, with Modifications, or
-    as part of a Larger Work; and
-
-(b) under Patent Claims of such Contributor to make, use, sell, offer
-    for sale, have made, import, and otherwise transfer either its
-    Contributions or its Contributor Version.
-
-2.2. Effective Date
-
-The licenses granted in Section 2.1 with respect to any Contribution
-become effective for each Contribution on the date the Contributor first
-distributes such Contribution.
-
-2.3. Limitations on Grant Scope
-
-The licenses granted in this Section 2 are the only rights granted under
-this License. No additional rights or licenses will be implied from the
-distribution or licensing of Covered Software under this License.
-Notwithstanding Section 2.1(b) above, no patent license is granted by a
-Contributor:
-
-(a) for any code that a Contributor has removed from Covered Software;
-    or
-
-(b) for infringements caused by: (i) Your and any other third party's
-    modifications of Covered Software, or (ii) the combination of its
-    Contributions with other software (except as part of its Contributor
-    Version); or
-
-(c) under Patent Claims infringed by Covered Software in the absence of
-    its Contributions.
-
-This License does not grant any rights in the trademarks, service marks,
-or logos of any Contributor (except as may be necessary to comply with
-the notice requirements in Section 3.4).
-
-2.4. Subsequent Licenses
-
-No Contributor makes additional grants as a result of Your choice to
-distribute the Covered Software under a subsequent version of this
-License (see Section 10.2) or under the terms of a Secondary License (if
-permitted under the terms of Section 3.3).
-
-2.5. Representation
-
-Each Contributor represents that the Contributor believes its
-Contributions are its original creation(s) or it has sufficient rights
-to grant the rights to its Contributions conveyed by this License.
-
-2.6. Fair Use
-
-This License is not intended to limit any rights You have under
-applicable copyright doctrines of fair use, fair dealing, or other
-equivalents.
-
-2.7. Conditions
-
-Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted
-in Section 2.1.
-
-3. Responsibilities
--------------------
-
-3.1. Distribution of Source Form
-
-All distribution of Covered Software in Source Code Form, including any
-Modifications that You create or to which You contribute, must be under
-the terms of this License. You must inform recipients that the Source
-Code Form of the Covered Software is governed by the terms of this
-License, and how they can obtain a copy of this License. You may not
-attempt to alter or restrict the recipients' rights in the Source Code
-Form.
-
-3.2. Distribution of Executable Form
-
-If You distribute Covered Software in Executable Form then:
-
-(a) such Covered Software must also be made available in Source Code
-    Form, as described in Section 3.1, and You must inform recipients of
-    the Executable Form how they can obtain a copy of such Source Code
-    Form by reasonable means in a timely manner, at a charge no more
-    than the cost of distribution to the recipient; and
-
-(b) You may distribute such Executable Form under the terms of this
-    License, or sublicense it under different terms, provided that the
-    license for the Executable Form does not attempt to limit or alter
-    the recipients' rights in the Source Code Form under this License.
-
-3.3. Distribution of a Larger Work
-
-You may create and distribute a Larger Work under terms of Your choice,
-provided that You also comply with the requirements of this License for
-the Covered Software. If the Larger Work is a combination of Covered
-Software with a work governed by one or more Secondary Licenses, and the
-Covered Software is not Incompatible With Secondary Licenses, this
-License permits You to additionally distribute such Covered Software
-under the terms of such Secondary License(s), so that the recipient of
-the Larger Work may, at their option, further distribute the Covered
-Software under the terms of either this License or such Secondary
-License(s).
-
-3.4. Notices
-
-You may not remove or alter the substance of any license notices
-(including copyright notices, patent notices, disclaimers of warranty,
-or limitations of liability) contained within the Source Code Form of
-the Covered Software, except that You may alter any license notices to
-the extent required to remedy known factual inaccuracies.
-
-3.5. Application of Additional Terms
-
-You may choose to offer, and to charge a fee for, warranty, support,
-indemnity or liability obligations to one or more recipients of Covered
-Software. However, You may do so only on Your own behalf, and not on
-behalf of any Contributor. You must make it absolutely clear that any
-such warranty, support, indemnity, or liability obligation is offered by
-You alone, and You hereby agree to indemnify every Contributor for any
-liability incurred by such Contributor as a result of warranty, support,
-indemnity or liability terms You offer. You may include additional
-disclaimers of warranty and limitations of liability specific to any
-jurisdiction.
-
-4. Inability to Comply Due to Statute or Regulation
----------------------------------------------------
-
-If it is impossible for You to comply with any of the terms of this
-License with respect to some or all of the Covered Software due to
-statute, judicial order, or regulation then You must: (a) comply with
-the terms of this License to the maximum extent possible; and (b)
-describe the limitations and the code they affect. Such description must
-be placed in a text file included with all distributions of the Covered
-Software under this License. Except to the extent prohibited by statute
-or regulation, such description must be sufficiently detailed for a
-recipient of ordinary skill to be able to understand it.
-
-5. Termination
---------------
-
-5.1. The rights granted under this License will terminate automatically
-if You fail to comply with any of its terms. However, if You become
-compliant, then the rights granted under this License from a particular
-Contributor are reinstated (a) provisionally, unless and until such
-Contributor explicitly and finally terminates Your grants, and (b) on an
-ongoing basis, if such Contributor fails to notify You of the
-non-compliance by some reasonable means prior to 60 days after You have
-come back into compliance. Moreover, Your grants from a particular
-Contributor are reinstated on an ongoing basis if such Contributor
-notifies You of the non-compliance by some reasonable means, this is the
-first time You have received notice of non-compliance with this License
-from such Contributor, and You become compliant prior to 30 days after
-Your receipt of the notice.
-
-5.2. If You initiate litigation against any entity by asserting a patent
-infringement claim (excluding declaratory judgment actions,
-counter-claims, and cross-claims) alleging that a Contributor Version
-directly or indirectly infringes any patent, then the rights granted to
-You by any and all Contributors for the Covered Software under Section
-2.1 of this License shall terminate.
-
-5.3. In the event of termination under Sections 5.1 or 5.2 above, all
-end user license agreements (excluding distributors and resellers) which
-have been validly granted by You or Your distributors under this License
-prior to termination shall survive termination.
-
-************************************************************************
-*                                                                      *
-*  6. Disclaimer of Warranty                                           *
-*  -------------------------                                           *
-*                                                                      *
-*  Covered Software is provided under this License on an "as is"       *
-*  basis, without warranty of any kind, either expressed, implied, or  *
-*  statutory, including, without limitation, warranties that the       *
-*  Covered Software is free of defects, merchantable, fit for a        *
-*  particular purpose or non-infringing. The entire risk as to the     *
-*  quality and performance of the Covered Software is with You.        *
-*  Should any Covered Software prove defective in any respect, You     *
-*  (not any Contributor) assume the cost of any necessary servicing,   *
-*  repair, or correction. This disclaimer of warranty constitutes an   *
-*  essential part of this License. No use of any Covered Software is   *
-*  authorized under this License except under this disclaimer.         *
-*                                                                      *
-************************************************************************
-
-************************************************************************
-*                                                                      *
-*  7. Limitation of Liability                                          *
-*  --------------------------                                          *
-*                                                                      *
-*  Under no circumstances and under no legal theory, whether tort      *
-*  (including negligence), contract, or otherwise, shall any           *
-*  Contributor, or anyone who distributes Covered Software as          *
-*  permitted above, be liable to You for any direct, indirect,         *
-*  special, incidental, or consequential damages of any character      *
-*  including, without limitation, damages for lost profits, loss of    *
-*  goodwill, work stoppage, computer failure or malfunction, or any    *
-*  and all other commercial damages or losses, even if such party      *
-*  shall have been informed of the possibility of such damages. This   *
-*  limitation of liability shall not apply to liability for death or   *
-*  personal injury resulting from such party's negligence to the       *
-*  extent applicable law prohibits such limitation. Some               *
-*  jurisdictions do not allow the exclusion or limitation of           *
-*  incidental or consequential damages, so this exclusion and          *
-*  limitation may not apply to You.                                    *
-*                                                                      *
-************************************************************************
-
-8. Litigation
--------------
-
-Any litigation relating to this License may be brought only in the
-courts of a jurisdiction where the defendant maintains its principal
-place of business and such litigation shall be governed by laws of that
-jurisdiction, without reference to its conflict-of-law provisions.
-Nothing in this Section shall prevent a party's ability to bring
-cross-claims or counter-claims.
-
-9. Miscellaneous
-----------------
-
-This License represents the complete agreement concerning the subject
-matter hereof. If any provision of this License is held to be
-unenforceable, such provision shall be reformed only to the extent
-necessary to make it enforceable. Any law or regulation which provides
-that the language of a contract shall be construed against the drafter
-shall not be used to construe this License against a Contributor.
-
-10. Versions of the License
----------------------------
-
-10.1. New Versions
-
-Mozilla Foundation is the license steward. Except as provided in Section
-10.3, no one other than the license steward has the right to modify or
-publish new versions of this License. Each version will be given a
-distinguishing version number.
-
-10.2. Effect of New Versions
-
-You may distribute the Covered Software under the terms of the version
-of the License under which You originally received the Covered Software,
-or under the terms of any subsequent version published by the license
-steward.
-
-10.3. Modified Versions
-
-If you create software not governed by this License, and you want to
-create a new license for such software, you may create and use a
-modified version of this License if you rename the license and remove
-any references to the name of the license steward (except to note that
-such modified license differs from this License).
-
-10.4. Distributing Source Code Form that is Incompatible With Secondary
-Licenses
-
-If You choose to distribute Source Code Form that is Incompatible With
-Secondary Licenses under the terms of this version of the License, the
-notice described in Exhibit B of this License must be attached.
-
-Exhibit A - Source Code Form License Notice
--------------------------------------------
-
-  This Source Code Form is subject to the terms of the Mozilla Public
-  License, v. 2.0. If a copy of the MPL was not distributed with this
-  file, You can obtain one at https://mozilla.org/MPL/2.0/.
-
-If it is not possible or desirable to put the notice in a particular
-file, then You may include the notice in a location (such as a LICENSE
-file in a relevant directory) where a recipient would be likely to look
-for such a notice.
-
-You may add additional accurate notices of copyright ownership.
-
-Exhibit B - "Incompatible With Secondary Licenses" Notice
----------------------------------------------------------
-
-  This Source Code Form is "Incompatible With Secondary Licenses", as
-  defined by the Mozilla Public License, v. 2.0.

dtek_notif/docker-compose.yaml

@@ -1,14 +0,0 @@
-services:
-  dtek_notif:
-    build:
-      context: .
-      dockerfile: Dockerfile
-    restart: unless-stopped
-    environment:
-      - TZ=Europe/Kyiv
-    
-    dns:
-      - 1.1.1.1
-      - 8.8.8.8
-    networks:
-      - default