feat(paperless): add alternative Compose deployment

docs(postgres): format Paperless compatibility table
feat(paperless): deploy Paperless-ngx on shared PostgreSQL
2026-10-08 08:11:57 +02:00 · 2026-10-08 00:34:43 +02:00 · 2026-10-08 00:32:54 +02:00 · 2026-10-07 23:28:17 +02:00 · 2026-10-07 21:10:52 +00:00 · 2026-10-07 20:43:12 +02:00
70 changed files with 1928 additions and 2835 deletions
@@ -0,0 +1,50 @@
+# EDU ownership handoff
+
+## Status
+
+The EDU ownership handoff is complete. The homelab repository no longer owns
+EDU workloads, images, routes, alerts, or deployment selection. The EDU
+repository is the only deployment owner: [forust/edu-master](https://git.forust.xyz/forust/edu-master).
+
+Homelab PRs #99 and #105 are merged. PR #105 removed the EDU subtree and its
+build, deploy, rollback, verification, route-probe, and registry references.
+It also added the serial image build matrix for the homelab services. This
+handoff record is the only remaining EDU-specific file in homelab Git.
+
+The dedicated workstation checkout is `/srv/edu-master`, at release
+`4f2b2a0e37dc11ac2c75441a15076c178e219d37`. It contains `k8s/active`; root
+`active` is absent. The old untracked `/srv/homelab/edu_master` checkout was
+moved outside the homelab repository to
+`/srv/edu-master-legacy-archive-20261007/edu_master`. Its private files remain
+mode `0600` inside an archive directory with mode `0700`. The homelab deploy
+checkout has no EDU marker or tracked EDU application/deployment files.
+`AUTODEPLOY=false` remains in place for homelab deployment.
+
+## Release evidence
+
+EDU PR #4 merged after its review and CI checks. Main-push CI run 1652 passed
+all validation and both image builds. Deploy run 1653 passed for the exact main
+SHA above.
+
+The workstation rollout completed for both Deployments. The deployment
+verified `/health` and `/live` with HTTP 200, Redis AUTH, session TTL of 1058
+seconds, a delivery backlog of zero, and all nine EDU vmalert rules with
+matching expressions and healthy evaluation.
+
+The images now run by digest:
+
+- Session keeper: `sha256:998dea51aa3015fd9cabefb0f53b030157a650c3bef72e02fe84f17d5762613d`
+- Webinar checker: `sha256:92f3c1fa2bb7f9b4680a9fc76a5b33dfbea8ef3dd9c6490ebc45876fd4c54461`
+
+Redis StatefulSet was unchanged. PVC `redis-data-pvc` remains bound to PV
+`pvc-a4f2a79a-363a-4c12-ae91-92cdfc2a0d2e` with capacity 1 GiB. The existing
+runtime Secret and Fernet key were preserved during the handoff. Notification
+delivery was verified before closeout, as confirmed by the operator. The
+deployment did not record downtime.
+
+The release rollback snapshot is
+`/home/forust/.local/state/edu-master-deploy/20261007T180541Z-4f2b2a0e37dc11ac2c75441a15076c178e219d37`.
+The handoff data snapshot remains at
+`/home/forust/.local/state/edu-master-deploy/handoff-20261007T080838Z`.
+Both snapshots are outside Git. Do not restore old Redis data unless recovery
+requires it. Never delete or recreate the Redis PVC.
@@ -7,4 +7,5 @@ self-hosted-runner:
  labels:
    - arch
    - homelab
+    - homelab-pr
    - prod
@@ -1,8 +1,20 @@
 # Homelab CI/CD

-The native Gitea runner runs on **vps**; production runs on **workstation**.
-Jobs run on `homelab:host`, one at a time. No job images or Kubernetes credentials
-are needed on the VPS. Builds use one pinned BuildKit helper container. CI and deploy are separate workflows.
+The native Gitea runners run on **vps**; production runs on **workstation**.
+Main-branch checks and image builds use `homelab:host`. Pull request and
+non-main checks use `homelab-pr:host` under a separate account without Docker
+access. The `homelab-pr` runner is registered at User scope for `forust`, so
+any repository under that account can schedule jobs that request this label.
+Each runner accepts one job at a time; the build waits for every check to pass.
+CI and deploy runs also show a summary with
+the release SHA, image build or reuse results, deploy mode, selected services,
+and image digests. Failed runs keep a summary of completed image builds, stage
+results, apply results, and recorded Kubernetes recovery. The final deploy
+summary is in the smoke job; earlier jobs show the state observed at that time.
+Apply success is separate from health and recovery. Update the installed
+workstation controller with `setup-workstation.sh` when no deploy is running.
+No job images or Kubernetes credentials are needed on the VPS. Builds use one
+pinned BuildKit helper container. CI and deploy are separate workflows.

 ## Runner installation

@@ -28,6 +40,32 @@ pushes directly to the registry, and caps retained local cache at 1 GiB with a
 2 GiB free-space target. This is not a hard limit on peak build disk usage.
 Nothing runs `docker system prune`, removes unrelated images, or deletes volumes.

+### Pull request runner
+
+Install the unprivileged host runner on the VPS:
+
+```sh
+sudo bash .gitea/runner/setup-pr-runner.sh
+```
+
+Get a registration token from the user Actions runner settings. Run the
+installer in a terminal. It asks for the token without echoing it, registers the
+runner as `homelab-pr` with label `homelab-pr:host`, then enables the service.
+The work directory is `/var/lib/gitea-pr-runner`. Confirm that Gitea lists the
+runner as User scope before merging the workflow change. An unmatched label can
+fall back to the default job image.
+
+Renovate PR validation uses `pull_request_target`, which reads the workflow from
+the base branch. It checks out the PR head only after runner selection and runs
+that code on `homelab-pr`. Keep this workflow read-only and do not add secrets.
+
+The PR runner has a separate home and tool cache. Do not add it to the `docker`
+group or give it access to `/var/run/docker.sock`. It runs repository code from
+pull requests, so keep its registration and permissions separate from the
+trusted `homelab` runner. This separates users and host permissions, but both
+runners still share the VPS kernel and network. Use a disposable VM if PRs from
+untrusted external authors must be fully isolated.
+
 ## Workstation setup

 As the existing SSH deploy user on workstation:
@@ -58,8 +96,7 @@ The deploy user's existing Docker registry authentication remains necessary.

 CI publishes `release-<full SHA>` as a Gitea artifact with all three owned image
 digests and build input fingerprints. Unchanged images are reused only from a
-successful main CI artifact, never from `:prod`. EDU images remain pinned to the
-digests released by their application repository. Expired artifacts cause CI to
+successful main CI artifact, never from `:prod`. Expired artifacts cause CI to
 rebuild images; they block deployment until CI is rerun.

 Run deploy from main with `deploy_ref=main` or a checked SHA:
@@ -0,0 +1,8 @@
+runner:
+  file: /var/lib/gitea-pr-runner/.runner
+  capacity: 1
+  timeout: 5h
+  labels:
+    - homelab-pr:host
+cache:
+  enabled: false
@@ -0,0 +1,27 @@
+[Unit]
+Description=Gitea Actions untrusted pull request runner
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+User=gitea-pr-runner
+Group=gitea-pr-runner
+WorkingDirectory=/var/lib/gitea-pr-runner
+Environment=HOME=/var/lib/gitea-pr-runner
+Environment=PATH=/var/lib/gitea-pr-runner/.cache/homelab-ci/bin:/usr/local/bin:/usr/bin:/bin
+ExecStart=/usr/local/bin/gitea-runner daemon --config /etc/gitea-pr-runner/config.yaml
+Restart=on-failure
+RestartSec=5
+NoNewPrivileges=yes
+PrivateTmp=yes
+ProtectSystem=full
+ProtectHome=yes
+ProtectKernelTunables=yes
+ProtectKernelModules=yes
+ProtectControlGroups=yes
+RestrictSUIDSGID=yes
+LockPersonality=yes
+UMask=0077
+
+[Install]
+WantedBy=multi-user.target
@@ -0,0 +1,56 @@
+#!/usr/bin/env bash
+# Install a native runner for untrusted PR jobs without Docker access.
+set -euo pipefail
+here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+[ "$(id -u)" -eq 0 ] || { echo 'Run with sudo on the runner host' >&2; exit 1; }
+for tool in cp cut date getent id install runuser systemctl useradd; do
+  command -v "$tool" >/dev/null || { echo "Install missing prerequisite: $tool" >&2; exit 1; }
+done
+command -v /usr/local/bin/gitea-runner >/dev/null || {
+  echo 'Install gitea-runner 3.0.2 at /usr/local/bin/gitea-runner first' >&2
+  exit 1
+}
+
+id gitea-pr-runner >/dev/null 2>&1 || \
+  useradd --system --create-home --home-dir /var/lib/gitea-pr-runner --shell /usr/bin/bash gitea-pr-runner
+runner_home="$(getent passwd gitea-pr-runner | cut -d: -f6)"
+[ "$runner_home" = /var/lib/gitea-pr-runner ] || {
+  echo 'Unexpected PR runner home; inspect the existing service first' >&2
+  exit 1
+}
+case " $(id -nG gitea-pr-runner) " in
+  *' docker '*)
+    echo 'The PR runner account must not belong to the docker group' >&2
+    exit 1
+    ;;
+esac
+
+install -d -m 0755 /etc/gitea-pr-runner
+stamp="$(date -u +%Y%m%dT%H%M%SZ)"
+for existing in /etc/gitea-pr-runner/config.yaml /etc/systemd/system/gitea-pr-runner.service; do
+  [ ! -f "$existing" ] || cp -p "$existing" "$existing.before-$stamp"
+done
+install -m 0644 "$here/pr-config.yaml" /etc/gitea-pr-runner/config.yaml
+install -m 0644 "$here/pr-runner.service" /etc/systemd/system/gitea-pr-runner.service
+
+if [ ! -f /var/lib/gitea-pr-runner/.runner ]; then
+  read -r -s -p 'Enter the Gitea repository runner registration token: ' runner_token
+  printf '\n'
+  [ -n "$runner_token" ] || { echo 'Runner token is required' >&2; exit 1; }
+  export GITEA_RUNNER_REGISTRATION_TOKEN="$runner_token"
+  unset runner_token
+  runuser --preserve-environment -u gitea-pr-runner -- \
+    /usr/local/bin/gitea-runner register \
+    --config /etc/gitea-pr-runner/config.yaml \
+    --instance https://gitea.forust.xyz \
+    --name homelab-pr \
+    --labels homelab-pr:host \
+    --no-interactive
+  unset GITEA_RUNNER_REGISTRATION_TOKEN
+fi
+chmod 0600 /var/lib/gitea-pr-runner/.runner
+
+systemctl daemon-reload
+systemctl enable --now gitea-pr-runner.service
+systemctl restart gitea-pr-runner.service
+echo "PR runner ready. Configuration backups: *.before-$stamp"
@@ -12,18 +12,14 @@ concurrency:
  group: ci-${{ github.ref }}
  cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
 jobs:
-  checks:
-    runs-on: homelab
-    timeout-minutes: 30
+  compose:
+    name: Compose
+    runs-on: ${{ github.ref == 'refs/heads/main' && 'homelab' || 'homelab-pr' }}
+    timeout-minutes: 15
    steps:
      - name: Checkout repository
        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
-      - name: Prepare pinned tools
-        shell: bash
-        run: |
-          set -euo pipefail
-          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh)"
-          echo "$tools_dir" >> "$GITHUB_PATH"
+        id: source
      - name: Validate Compose files
        shell: bash
        run: |
@@ -52,11 +48,73 @@ jobs:
            exit 1
          fi
          echo "checked ${#files[@]} Compose file(s)"
+        id: check
+      - name: Write the job result
+        if: always()
+        env:
+          SUMMARY_CHECK: Compose
+          SUMMARY_RESULT: ${{ job.status }}
+          SUMMARY_FAILED_STEP:
+            ${{ steps.check.conclusion == 'failure' && 'Check or image build' || steps.source.conclusion == 'failure'
+            && 'Source checkout' || '' }}
+        shell: bash
+        run: |
+          if [ -f .gitea/workflows/release.py ]; then
+            python3 .gitea/workflows/release.py check-summary
+          elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
+            printf '## %s\n\n- Result: **%s**\n- Failed step: %s\n' "$SUMMARY_CHECK" "$SUMMARY_RESULT" "$SUMMARY_FAILED_STEP" >>"$GITHUB_STEP_SUMMARY" || true
+          fi
+  workflows:
+    name: Workflows
+    runs-on: ${{ github.ref == 'refs/heads/main' && 'homelab' || 'homelab-pr' }}
+    timeout-minutes: 15
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
+        id: source
+      - name: Prepare pinned tools
+        shell: bash
+        run: |
+          set -euo pipefail
+          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh actionlint shellcheck)"
+          echo "$tools_dir" >> "$GITHUB_PATH"
+        id: tools
      - name: Lint Gitea Actions workflows with actionlint
        shell: bash
        run: |
          set -euo pipefail
          actionlint -config-file .gitea/actionlint.yaml -color .gitea/workflows/*.yaml
+        id: check
+      - name: Write the job result
+        if: always()
+        env:
+          SUMMARY_CHECK: Workflows
+          SUMMARY_RESULT: ${{ job.status }}
+          SUMMARY_FAILED_STEP:
+            ${{ steps.check.conclusion == 'failure' && 'Check or image build' || steps.tools.conclusion == 'failure'
+            && 'Tool setup' || steps.source.conclusion == 'failure' && 'Source checkout' || '' }}
+        shell: bash
+        run: |
+          if [ -f .gitea/workflows/release.py ]; then
+            python3 .gitea/workflows/release.py check-summary
+          elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
+            printf '## %s\n\n- Result: **%s**\n- Failed step: %s\n' "$SUMMARY_CHECK" "$SUMMARY_RESULT" "$SUMMARY_FAILED_STEP" >>"$GITHUB_STEP_SUMMARY" || true
+          fi
+  shell:
+    name: Shell
+    runs-on: ${{ github.ref == 'refs/heads/main' && 'homelab' || 'homelab-pr' }}
+    timeout-minutes: 15
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
+        id: source
+      - name: Prepare pinned tools
+        shell: bash
+        run: |
+          set -euo pipefail
+          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh shellcheck jq)"
+          echo "$tools_dir" >> "$GITHUB_PATH"
+        id: tools
      - name: Lint shell scripts with ShellCheck
        shell: bash
        run: |
@@ -70,6 +128,37 @@ jobs:
          fi
          shellcheck --external-sources --source-path=SCRIPTDIR --severity=style "${scripts[@]}"
          bash .gitea/tests/deploy-validation.sh
+        id: check
+      - name: Write the job result
+        if: always()
+        env:
+          SUMMARY_CHECK: Shell
+          SUMMARY_RESULT: ${{ job.status }}
+          SUMMARY_FAILED_STEP:
+            ${{ steps.check.conclusion == 'failure' && 'Check or image build' || steps.tools.conclusion == 'failure'
+            && 'Tool setup' || steps.source.conclusion == 'failure' && 'Source checkout' || '' }}
+        shell: bash
+        run: |
+          if [ -f .gitea/workflows/release.py ]; then
+            python3 .gitea/workflows/release.py check-summary
+          elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
+            printf '## %s\n\n- Result: **%s**\n- Failed step: %s\n' "$SUMMARY_CHECK" "$SUMMARY_RESULT" "$SUMMARY_FAILED_STEP" >>"$GITHUB_STEP_SUMMARY" || true
+          fi
+  formatting:
+    name: Formatting
+    runs-on: ${{ github.ref == 'refs/heads/main' && 'homelab' || 'homelab-pr' }}
+    timeout-minutes: 15
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
+        id: source
+      - name: Prepare pinned tools
+        shell: bash
+        run: |
+          set -euo pipefail
+          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh prettier)"
+          echo "$tools_dir" >> "$GITHUB_PATH"
+        id: tools
      - name: Check formatting with Prettier
        shell: bash
        run: |
@@ -87,6 +176,37 @@ jobs:
          fi

          prettier --check --ignore-unknown "${prettier_files[@]}"
+        id: check
+      - name: Write the job result
+        if: always()
+        env:
+          SUMMARY_CHECK: Formatting
+          SUMMARY_RESULT: ${{ job.status }}
+          SUMMARY_FAILED_STEP:
+            ${{ steps.check.conclusion == 'failure' && 'Check or image build' || steps.tools.conclusion == 'failure'
+            && 'Tool setup' || steps.source.conclusion == 'failure' && 'Source checkout' || '' }}
+        shell: bash
+        run: |
+          if [ -f .gitea/workflows/release.py ]; then
+            python3 .gitea/workflows/release.py check-summary
+          elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
+            printf '## %s\n\n- Result: **%s**\n- Failed step: %s\n' "$SUMMARY_CHECK" "$SUMMARY_RESULT" "$SUMMARY_FAILED_STEP" >>"$GITHUB_STEP_SUMMARY" || true
+          fi
+  python:
+    name: Python and tests
+    runs-on: ${{ github.ref == 'refs/heads/main' && 'homelab' || 'homelab-pr' }}
+    timeout-minutes: 15
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
+        id: source
+      - name: Prepare pinned tools
+        shell: bash
+        run: |
+          set -euo pipefail
+          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh ruff jq)"
+          echo "$tools_dir" >> "$GITHUB_PATH"
+        id: tools
      - name: Lint and format-check Python with Ruff
        shell: bash
        run: |
@@ -94,6 +214,37 @@ jobs:
          ruff check . .gitea/workflows
          ruff format --check . .gitea/workflows
          python3 -m unittest discover -s tests -v
+        id: check
+      - name: Write the job result
+        if: always()
+        env:
+          SUMMARY_CHECK: Python and tests
+          SUMMARY_RESULT: ${{ job.status }}
+          SUMMARY_FAILED_STEP:
+            ${{ steps.check.conclusion == 'failure' && 'Check or image build' || steps.tools.conclusion == 'failure'
+            && 'Tool setup' || steps.source.conclusion == 'failure' && 'Source checkout' || '' }}
+        shell: bash
+        run: |
+          if [ -f .gitea/workflows/release.py ]; then
+            python3 .gitea/workflows/release.py check-summary
+          elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
+            printf '## %s\n\n- Result: **%s**\n- Failed step: %s\n' "$SUMMARY_CHECK" "$SUMMARY_RESULT" "$SUMMARY_FAILED_STEP" >>"$GITHUB_STEP_SUMMARY" || true
+          fi
+  yaml:
+    name: YAML
+    runs-on: ${{ github.ref == 'refs/heads/main' && 'homelab' || 'homelab-pr' }}
+    timeout-minutes: 15
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
+        id: source
+      - name: Prepare pinned tools
+        shell: bash
+        run: |
+          set -euo pipefail
+          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh yamllint)"
+          echo "$tools_dir" >> "$GITHUB_PATH"
+        id: tools
      - name: Lint YAML syntax
        shell: bash
        run: |
@@ -111,6 +262,37 @@ jobs:
          fi

          yamllint -c .yamllint "${yaml_files[@]}"
+        id: check
+      - name: Write the job result
+        if: always()
+        env:
+          SUMMARY_CHECK: YAML
+          SUMMARY_RESULT: ${{ job.status }}
+          SUMMARY_FAILED_STEP:
+            ${{ steps.check.conclusion == 'failure' && 'Check or image build' || steps.tools.conclusion == 'failure'
+            && 'Tool setup' || steps.source.conclusion == 'failure' && 'Source checkout' || '' }}
+        shell: bash
+        run: |
+          if [ -f .gitea/workflows/release.py ]; then
+            python3 .gitea/workflows/release.py check-summary
+          elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
+            printf '## %s\n\n- Result: **%s**\n- Failed step: %s\n' "$SUMMARY_CHECK" "$SUMMARY_RESULT" "$SUMMARY_FAILED_STEP" >>"$GITHUB_STEP_SUMMARY" || true
+          fi
+  dockerfiles:
+    name: Dockerfiles
+    runs-on: ${{ github.ref == 'refs/heads/main' && 'homelab' || 'homelab-pr' }}
+    timeout-minutes: 15
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
+        id: source
+      - name: Prepare pinned tools
+        shell: bash
+        run: |
+          set -euo pipefail
+          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh hadolint)"
+          echo "$tools_dir" >> "$GITHUB_PATH"
+        id: tools
      - name: Lint Dockerfiles
        shell: bash
        run: |
@@ -126,6 +308,37 @@ jobs:
          fi

          hadolint -c .hadolint.yaml "${dockerfiles[@]}"
+        id: check
+      - name: Write the job result
+        if: always()
+        env:
+          SUMMARY_CHECK: Dockerfiles
+          SUMMARY_RESULT: ${{ job.status }}
+          SUMMARY_FAILED_STEP:
+            ${{ steps.check.conclusion == 'failure' && 'Check or image build' || steps.tools.conclusion == 'failure'
+            && 'Tool setup' || steps.source.conclusion == 'failure' && 'Source checkout' || '' }}
+        shell: bash
+        run: |
+          if [ -f .gitea/workflows/release.py ]; then
+            python3 .gitea/workflows/release.py check-summary
+          elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
+            printf '## %s\n\n- Result: **%s**\n- Failed step: %s\n' "$SUMMARY_CHECK" "$SUMMARY_RESULT" "$SUMMARY_FAILED_STEP" >>"$GITHUB_STEP_SUMMARY" || true
+          fi
+  kubernetes:
+    name: Kubernetes
+    runs-on: ${{ github.ref == 'refs/heads/main' && 'homelab' || 'homelab-pr' }}
+    timeout-minutes: 15
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
+        id: source
+      - name: Prepare pinned tools
+        shell: bash
+        run: |
+          set -euo pipefail
+          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh kubeconform)"
+          echo "$tools_dir" >> "$GITHUB_PATH"
+        id: tools
      - name: Validate Kubernetes manifests against JSON schemas
        shell: bash
        run: |
@@ -146,27 +359,162 @@ jobs:
            -ignore-missing-schemas \
            -summary \
            "${manifests[@]}"
-  build:
-    needs:
-      - checks
+        id: check
+      - name: Write the job result
+        if: always()
+        env:
+          SUMMARY_CHECK: Kubernetes
+          SUMMARY_RESULT: ${{ job.status }}
+          SUMMARY_FAILED_STEP:
+            ${{ steps.check.conclusion == 'failure' && 'Check or image build' || steps.tools.conclusion == 'failure'
+            && 'Tool setup' || steps.source.conclusion == 'failure' && 'Source checkout' || '' }}
+        shell: bash
+        run: |
+          if [ -f .gitea/workflows/release.py ]; then
+            python3 .gitea/workflows/release.py check-summary
+          elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
+            printf '## %s\n\n- Result: **%s**\n- Failed step: %s\n' "$SUMMARY_CHECK" "$SUMMARY_RESULT" "$SUMMARY_FAILED_STEP" >>"$GITHUB_STEP_SUMMARY" || true
+          fi
+  image-plan:
+    needs: [compose, workflows, shell, formatting, python, yaml, dockerfiles, kubernetes]
    if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main'
    runs-on: homelab
-    timeout-minutes: 60
+    timeout-minutes: 10
+    outputs:
+      matrix: ${{ steps.plan.outputs.matrix }}
    steps:
      - name: Checkout repository
+        id: source
        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
        with:
          fetch-depth: 0
-      - name: Build changed images and write release
+      - name: Detect build inputs against successful CI
+        id: plan
        env:
          GITEA_TOKEN: ${{ github.token }}
+        run: python3 .gitea/workflows/release.py prepare --output build-plan.json
+      - name: Store the image plan
+        id: artifact
+        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
+        with:
+          name: build-plan
+          path: build-plan.json
+          if-no-files-found: error
+          retention-days: 30
+
+      - name: Write the plan result
+        if: always()
+        env:
+          SUMMARY_CHECK: Image plan
+          SUMMARY_RESULT: ${{ job.status }}
+          SUMMARY_FAILED_STEP: >-
+            ${{ steps.plan.conclusion == 'failure' && 'Build input detection' ||
+                steps.artifact.conclusion == 'failure' && 'Plan upload' ||
+                steps.source.conclusion == 'failure' && 'Source checkout' || '' }}
+        shell: bash
+        run: |
+          if [ -f .gitea/workflows/release.py ]; then
+            python3 .gitea/workflows/release.py check-summary
+          elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
+            printf '## Image plan\n\nResult: %s\n' "$SUMMARY_RESULT" >>"$GITHUB_STEP_SUMMARY" || true
+          fi
+
+  images:
+    name: Image (${{ matrix.name }})
+    needs: [image-plan]
+    if: needs.image-plan.result == 'success'
+    runs-on: homelab
+    timeout-minutes: 60
+    strategy:
+      max-parallel: 1
+      fail-fast: false
+      matrix: ${{ fromJSON(needs.image-plan.outputs.matrix || '{"include":[{"name":"inactive"}]}') }}
+    steps:
+      - name: Checkout repository
+        id: source
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
+      - name: Download the checked image plan
+        id: inputs
+        uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
+        with:
+          name: build-plan
+      - name: Build or reuse this image
+        id: check
+        env:
+          IMAGE_NAME: ${{ matrix.name }}
          REGISTRY_USERNAME: ${{ secrets.REGISTRY_USERNAME }}
          REGISTRY_PASSWORD: ${{ secrets.REGISTRY_PASSWORD }}
-        run: python3 .gitea/workflows/release.py build
+        run: python3 .gitea/workflows/release.py image --image "$IMAGE_NAME" --output image.json
+      - name: Store the image result
+        id: artifact
+        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
+        with:
+          name: image-${{ matrix.name }}
+          path: image.json
+          if-no-files-found: error
+          retention-days: 30
+      - name: Write the job result
+        if: always()
+        env:
+          SUMMARY_CHECK: Image (${{ matrix.name }})
+          SUMMARY_RESULT: ${{ job.status }}
+          SUMMARY_FAILED_STEP: >-
+            ${{ steps.check.conclusion == 'failure' && 'Build or tag images' ||
+                steps.artifact.conclusion == 'failure' && 'Artifact upload' ||
+                steps.inputs.conclusion == 'failure' && 'Artifact download' ||
+                steps.source.conclusion == 'failure' && 'Source checkout' || '' }}
+        shell: bash
+        run: |
+          if [ -f .gitea/workflows/release.py ]; then
+            python3 .gitea/workflows/release.py check-summary
+          elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
+            printf '## %s\n\n- Result: **%s**\n- Failed step: %s\n' "$SUMMARY_CHECK" "$SUMMARY_RESULT" "$SUMMARY_FAILED_STEP" >>"$GITHUB_STEP_SUMMARY" || true
+          fi
+
+  # Retain the build job name required by the immutable release deployment gate.
+  build:
+    needs: [image-plan, images]
+    runs-on: homelab
+    timeout-minutes: 15
+    steps:
+      - name: Checkout repository
+        id: source
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
+      - name: Download all image results
+        id: inputs
+        uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
+        with:
+          path: artifacts
+      - name: Pin SHA tags and write the complete release
+        id: check
+        env:
+          REGISTRY_USERNAME: ${{ secrets.REGISTRY_USERNAME }}
+          REGISTRY_PASSWORD: ${{ secrets.REGISTRY_PASSWORD }}
+        run: >-
+          python3 .gitea/workflows/release.py finalize
+          --plan artifacts/build-plan/build-plan.json
      - name: Store commit release
-        uses: actions/upload-artifact@c6a366c94c3e0affe28c06c8df20a878f24da3cf
+        id: artifact
+        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
        with:
          name: release-${{ github.sha }}
          path: release.json
          if-no-files-found: error
          retention-days: 30
+      - name: Write the job result
+        if: always()
+        env:
+          SUMMARY_CHECK: Image release and SHA tags
+          SUMMARY_RESULT: ${{ job.status }}
+          SUMMARY_FAILED_STEP: >-
+            ${{ steps.check.conclusion == 'failure' && 'Build or tag images' ||
+                steps.artifact.conclusion == 'failure' && 'Artifact upload' ||
+                steps.inputs.conclusion == 'failure' && 'Artifact download' ||
+                steps.source.conclusion == 'failure' && 'Source checkout' || '' }}
+        shell: bash
+        run: |
+          if [ -f .gitea/workflows/release.py ]; then
+            python3 .gitea/workflows/release.py check-summary
+          elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
+            printf '## %s\n\n- Result: **%s**\n- Failed step: %s\n' "$SUMMARY_CHECK" "$SUMMARY_RESULT" "$SUMMARY_FAILED_STEP" >>"$GITHUB_STEP_SUMMARY" || true
+          fi
@@ -45,12 +45,17 @@ def prepare(source_file):
    before = json.loads(json.dumps(config))
    for service, settings in config['services'].items():
        reference = settings.get('image')
+        nextcloud_aio_master = project == 'nextcloud' and service == 'nextcloud-aio-mastercontainer'
        if not reference or settings.get('build'):
            raise ValueError(f'{project}/{service}: Compose deploy requires a published image')
        image_repo = reference.split('@')[0].rsplit('/', 1)
        image_repo[-1] = image_repo[-1].split(':')[0]
        image_repo = '/'.join(image_repo)
-        if image_repo in release['images']:
+        # Nextcloud AIO validates the mastercontainer image reference and rejects
+        # a digest. Keep its configured tag so AIO can start and manage its stack.
+        if nextcloud_aio_master:
+            pinned = reference
+        elif image_repo in release['images']:
            pinned = image_repo + '@' + release['images'][image_repo]
        elif os.environ.get('REFRESH_IMAGES') != 'true' and reference in locks:
            pinned = locks[reference]
@@ -75,7 +80,12 @@ def prepare(source_file):
            actual.add(next((d for d in digests or [] if d.split('@')[0] == image_repo), image_id))
        if len(actual) > 1:
            raise ValueError(f'{project}/{service}: mixed running images, cannot capture one recovery config')
-        before['services'][service]['image'] = next(iter(actual)) if actual else reference
+        # AIO also rejects a digest in its recovery config. Preserve its tag in
+        # both deploy and recovery files.
+        if nextcloud_aio_master:
+            before['services'][service]['image'] = reference
+        else:
+            before['services'][service]['image'] = next(iter(actual)) if actual else reference
    for name, data in (('compose', config), ('compose-before', before)):
        folder = directory / name
        folder.mkdir(mode=0o700, exist_ok=True)
@@ -141,7 +141,8 @@ def make_plan(directory):
    planner = load_module('deploy_plan', source / '.gitea/workflows/deploy-plan.py')
    request = json.loads((directory / 'request.json').read_text())
    previous = json.loads((STATE / 'last-success.json').read_text()) if (STATE / 'last-success.json').exists() else None
-    helm = json.loads(command('helm', 'list', '--all', '-A', '-o', 'json'))
+    # Helm 4 lists every release status by default and removed the --all flag.
+    helm = json.loads(command('helm', 'list', '-A', '-o', 'json'))
    plan = planner.make_plan(source, CONFIG_REPO, request['release'], previous, request['mode'], helm)
    if request['refresh_images']:
        plan['selected']['compose'] = plan['active']['compose']
@@ -212,14 +213,17 @@ def execute(run_id):
                raise ValueError(f'Interrupted deploy {other.name}; run recover first')
        status['state'] = 'running'
        atomic_json(directory / 'status.json', status)
+        phase = 'plan'
        try:
            plan = make_plan(directory)
            print(
                json.dumps({'selected': plan['selected'], 'helm': plan['helm'], 'manual_removals': plan['removed']}),
                flush=True,
            )
+            phase = 'doctor'
            if not stage(directory, 'doctor', 600):
                raise RuntimeError('Preflight failed')
+            phase = 'validate'
            if not stage(directory, 'validate', 1200):
                raise RuntimeError('Validation failed')
            if json.loads((directory / 'request.json').read_text())['mode'] == 'plan':
@@ -228,6 +232,7 @@ def execute(run_id):
                atomic_json(directory / 'status.json', status)
                return
            # Budget includes both rollout checks and rollback waves, plus API overhead.
+            phase = 'Recovery budget'
            count = int(
                command(
                    'bash',
@@ -239,14 +244,24 @@ def execute(run_id):
            verify_budget = max(600, 2 * math.ceil(count / 4) * 300 + 120)
            if verify_budget > 7200:
                raise ValueError('More than two hours of recovery required; split this deploy')
+            phase = 'apply-k8s'
            k8s_ok = stage(directory, 'apply-k8s', 2700)
+            phase = 'apply-compose'
            compose_ok = stage(directory, 'apply-compose', 1800) if k8s_ok else False
+            phase = 'verify-k8s'
            verify_ok = stage(directory, 'verify-k8s', verify_budget)
+            phase = 'smoke'
            smoke_ok = stage(directory, 'smoke', 600)
            if not all((k8s_ok, compose_ok, verify_ok, smoke_ok)):
                raise RuntimeError('Deploy failed; inspect stage logs and recovery report')
+            phase = 'Save the successful baseline'
            finish_success(directory, plan)
        except Exception as error:
+            status = json.loads((directory / 'status.json').read_text())
+            status['failure_stage'] = next(
+                (name for name, result in status['stages'].items() if result.get('result') == 'failure'), phase
+            )
+            atomic_json(directory / 'status.json', status)
            with (directory / 'controller.log').open('a') as stream:
                stream.write(f'{error}\n')
            recover(directory)
@@ -294,10 +309,93 @@ def follow(run_id, phase):
        time.sleep(3)


+def summary(run_id):
+    directory = run_directory(run_id)
+    request = json.loads((directory / 'request.json').read_text())
+    release = request['release']
+    plan_file = directory / 'plan.json'
+    lines = [
+        f'## Deploy `{release["sha"]}`',
+        '',
+        f'- Mode: `{request["mode"]}`',
+        f'- Refresh third-party images: `{request["refresh_images"]}`',
+    ]
+    status = json.loads((directory / 'status.json').read_text())
+    if status.get('failure_stage'):
+        lines.append(f'- Failed stage: **{status["failure_stage"]}**')
+    lines.extend(
+        [
+            '',
+            f'- Observed run state: **{status["state"]}**',
+            '',
+            '### Stage results',
+            '| Stage | Result | Exit code |',
+            '| --- | --- | --- |',
+        ]
+    )
+    for name in ('doctor', 'validate', 'apply-k8s', 'apply-compose', 'verify-k8s', 'smoke'):
+        stage_result = status['stages'].get(name, {})
+        lines.append(f'| {name} | {stage_result.get("result", "not started")} | {stage_result.get("exit_code", "—")} |')
+    lines.extend(['', '### Apply and Helm recovery results'])
+    events_file = directory / 'apply-events.jsonl'
+    events = []
+    if events_file.exists():
+        for line in events_file.read_text().splitlines():
+            try:
+                events.append(json.loads(line))
+            except json.JSONDecodeError:
+                lines.append('- An operation record is incomplete. Check the stage log.')
+    latest = {(event['action'], event['target']): event['result'] for event in events}
+    lines.extend(f'- `{action}` `{target}`: **{result}**' for (action, target), result in latest.items())
+    if not latest:
+        lines.append('- No apply results were recorded.')
+    lines.append('- A completed apply does not confirm health. See verification and smoke results.')
+    lines.extend(['', '### Kubernetes recovery'])
+    pointer = directory / 'snapshot/current'
+    failed = Path(pointer.read_text().strip()) / 'failed-workloads' if pointer.exists() else None
+    if failed and failed.exists():
+        contents = failed.read_text()
+        counts = dict(re.findall(r'^(ROLLED_BACK|UNRECOVERED)=([0-9]+)$', contents, re.MULTILINE))
+        if not contents.strip():
+            lines.append('- No failed workloads were recorded. See the verification result above.')
+        elif counts:
+            lines.append(f'- Workloads restored: **{counts.get("ROLLED_BACK", "unknown")}**')
+            lines.append(f'- Workloads that need manual recovery: **{counts.get("UNRECOVERED", "unknown")}**')
+        else:
+            lines.append('- Rollback has no recorded result yet. Check the verification log.')
+    else:
+        lines.append('- No workload rollback was recorded. This does not confirm health.')
+    lines.append('- Compose requires manual recovery. Use the saved command in the apply log.')
+    if not plan_file.exists():
+        lines.extend(['', 'Plan was not created. Check the controller log.'])
+        print('\n'.join(lines))
+        return
+    plan = json.loads(plan_file.read_text())
+    lines.extend(['', '### Selected services'])
+    count = 0
+    for kind, services in plan['selected'].items():
+        for service in services:
+            lines.append(f'- `{kind}`: `{service}`')
+            count += 1
+    if not count:
+        lines.append('- None')
+    lines.extend(['', '### Selected Helm releases'])
+    lines.extend(f'- `{release}`' for release in plan.get('helm', []))
+    if not plan.get('helm'):
+        lines.append('- None')
+    lines.extend(['', '### Images pinned in the checked release'])
+    lines.extend(f'- `{image}@{digest}`' for image, digest in sorted(release['images'].items()))
+    lines.extend(['', '### Removed resources requiring manual review'])
+    lines.extend(f'- `{item}`' for item in plan.get('removed', []))
+    if not plan.get('removed'):
+        lines.append('- None')
+    print('\n'.join(lines))
+
+
 def main():
    os.umask(0o077)
    parser = argparse.ArgumentParser(description=__doc__)
-    parser.add_argument('action', choices=('start', 'execute', 'recover', 'status', 'follow'))
+    parser.add_argument('action', choices=('start', 'execute', 'recover', 'status', 'follow', 'summary'))
    parser.add_argument('run_id')
    parser.add_argument('phase', nargs='?', choices=('apply', 'verify', 'smoke'))
    parser.add_argument('--retry', action='store_true', help='Retry failed recovery checks; never repeat apply')
@@ -315,6 +413,8 @@ def main():
        if (directory / 'plan.json').exists():
            plan = json.loads((directory / 'plan.json').read_text())
            print(json.dumps({k: plan[k] for k in ('sha', 'selected', 'helm', 'removed')}, indent=2))
+    elif args.action == 'summary':
+        summary(args.run_id)
    elif not follow(args.run_id, args.phase):
        sys.exit(1)

@@ -27,6 +27,15 @@ log() {
  echo "== $* =="
 }

+# Store operation results without command output or local configuration values.
+record_apply() {
+  [ -n "${RUN_DIR:-}" ] || return 0
+  jq -cn --arg action "$1" --arg target "$2" --arg result "$3" \
+    '{action: $action, target: $target, result: $result}' >>"$RUN_DIR/apply-events.jsonl" \
+    || echo 'WARNING: cannot record an apply result' >&2
+  return 0
+}
+
 warn() {
  echo "WARNING: $*" >&2
 }
@@ -171,7 +180,8 @@ save_snapshot() {
          | select(any(.metadata.ownerReferences[]?; .uid == $w.metadata.uid))
          | select($w.kind != "StatefulSet" or .metadata.name == $w.status.currentRevision) | .revision] | max // 0) end)
    }]' "$dir/workloads.json" >"$dir/revisions.json" || return 1
-  releases="$(helm list --all -A -o json)" || return 1
+  # Helm 4 lists every release status by default and removed the --all flag.
+  releases="$(helm list -A -o json)" || return 1
  for entry in "${HELM_RELEASES[@]}"; do
    IFS='|' read -r release _ namespace _ _ _ <<<"$entry"
    if ! jq -e --arg r "$release" --arg n "$namespace" \
@@ -320,7 +330,7 @@ rollback_workloads() {
 # written straight into a `helm upgrade` command would never be updated: these
 # have to be declared as custom.regex managers in renovate/renovate.json.
 HELM_RELEASES=(
-  "prometheus-stack|prometheus-community/kube-prometheus-stack|prometheus|86.3.2|prometheus-stack/k8s/grafana-values.yaml|prometheus-stack/k8s/active"
+  "prometheus-stack|prometheus-community/kube-prometheus-stack|prometheus|86.2.3|prometheus-stack/k8s/grafana-values.yaml|prometheus-stack/k8s/active"
  "victoria-operator|victoriametrics/victoria-metrics-operator|prometheus|0.68.1|prometheus-stack/k8s/victoria-operator-values.yaml|prometheus-stack/k8s/active"
  "loki|grafana/loki|prometheus|7.3.0|loki/k8s/loki-values.yaml|loki/k8s/active"
  "alloy|grafana/alloy|prometheus|1.12.1|loki/k8s/alloy-values.yaml|loki/k8s/active"
@@ -376,15 +386,19 @@ recover_pending_release() {
        echo "ERROR: no captured Helm revision for $release; manual recovery required"
        return 1
      fi
+      record_apply helm-rollback "$namespace/$release" started
      if ! helm rollback "$release" "$revision" -n "$namespace" --wait --timeout 10m; then
+        record_apply helm-rollback "$namespace/$release" failure
        echo "WARN: helm rollback of $release did not complete"
        return 1
      fi
      status="$(helm_release_status "$release" "$namespace")" || return 1
      if [ "$status" != "deployed" ]; then
+        record_apply helm-rollback "$namespace/$release" failure
        echo "WARN: $release is $status after rollback"
        return 1
      fi
+      record_apply helm-rollback "$namespace/$release" success
      ;;
  esac
  return 0
@@ -448,11 +462,13 @@ upgrade_helm_releases() {
    # --rollback-on-failure (+ --wait) rolls the release back when the upgrade
    # times out or the workloads it touches never become ready, so a bad chart
    # bump is not left half applied. (--atomic was this combo; deprecated.)
+    record_apply helm-upgrade "$namespace/$release" started
    if ! helm upgrade --install "$release" "$chart" \
      --namespace "$namespace" \
      --version "$version" \
      --values "$values" \
      --wait --rollback-on-failure --cleanup-on-fail --timeout 10m; then
+      record_apply helm-upgrade "$namespace/$release" failure
      echo "WARN: upgrade of $release failed, checking release state"
      # --rollback-on-failure already attempted its own rollback; finish the job when that
      # rollback never completed, otherwise the release stays pending-* and
@@ -462,8 +478,10 @@ upgrade_helm_releases() {
      else
        echo "ERROR: upgrade of $release failed (release is back on its previous revision)."
      fi
+      record_apply helm-recovery-state "$namespace/$release" "$(helm_release_status "$release" "$namespace" || echo unknown)"
      return 1
    fi
+    record_apply helm-upgrade "$namespace/$release" success
  done
 }

@@ -631,7 +649,12 @@ stage_apply_k8s() {
  if [ "${#ns_files[@]}" -gt 0 ]; then
    log "Applying namespaces (${#ns_files[@]} files)"
    for m in "${ns_files[@]}"; do
-      kubectl apply -f "$m"
+      record_apply kubectl "${m#"$REPO"/}" started
+      if ! kubectl apply -f "$m"; then
+        record_apply kubectl "${m#"$REPO"/}" failure
+        return 1
+      fi
+      record_apply kubectl "${m#"$REPO"/}" success
    done
  fi
  if selected_service k8s prometheus-stack && [ -f "$REPO/prometheus-stack/k8s/active" ]; then
@@ -646,18 +669,24 @@ stage_apply_k8s() {
    log "Applying resources (${#other_files[@]} files, our images pinned to digests)"
    for m in "${other_files[@]}"; do
      log "Applying ${m#"$REPO"/}"
+      record_apply kubectl "${m#"$REPO"/}" started
      if ! render_pinned <"$m" | kubectl apply -f -; then
+        record_apply kubectl "${m#"$REPO"/}" failure
        echo "ERROR: apply failed for ${m#"$REPO"/}" >&2
        exit 1
      fi
+      record_apply kubectl "${m#"$REPO"/}" success
    done
  fi
  for k in ${KUSTOMIZE_APPS[@]+"${KUSTOMIZE_APPS[@]}"}; do
    log "Applying kustomize app: ${k#"$REPO"/} (our images pinned to digests)"
+    record_apply kustomize "${k#"$REPO"/}" started
    if ! kubectl kustomize "$k" | render_pinned | kubectl apply -f -; then
+      record_apply kustomize "${k#"$REPO"/}" failure
      echo "ERROR: apply failed for kustomize app ${k#"$REPO"/}" >&2
      exit 1
    fi
+    record_apply kustomize "${k#"$REPO"/}" success
  done

  # No verification here on purpose. This stage may be killed at any point by
@@ -959,7 +988,12 @@ stage_apply_compose() {
  local cf
  for cf in "${COMPOSE_STACKS[@]}"; do
    log "Applying Compose ${cf#"$REPO"/}"
-    compose "$cf" up -d --wait --wait-timeout 180 --pull missing --remove-orphans
+    record_apply compose "${cf#"$REPO"/}" started
+    if ! compose "$cf" up -d --wait --wait-timeout 180 --pull missing --remove-orphans; then
+      record_apply compose "${cf#"$REPO"/}" failure
+      return 1
+    fi
+    record_apply compose "${cf#"$REPO"/}" success
    verify_compose_stack "$cf"
  done
  echo "Compose recovery files: $RUN_DIR/compose-before (manual recovery only)"
@@ -64,11 +64,23 @@ jobs:
        run: python3 .gitea/workflows/release.py gate --ref "$DEPLOY_REF" --event-sha "$EVENT_SHA"
      - name: Submit durable deploy to workstation
        run: bash .gitea/workflows/ssh-run.sh start
+      - name: Write the request result
+        if: always()
+        env:
+          REQUEST_RESULT: ${{ job.status }}
+          CHECKED_SHA: ${{ steps.release.outputs.sha }}
+        run: |
+          if [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
+            printf '## Deploy request\n\n- Result: **%s**\n- Checked commit: %s\n- Mode: %s\n' "$REQUEST_RESULT" "${CHECKED_SHA:-not checked}" "$DEPLOY_MODE" >>"$GITHUB_STEP_SUMMARY"
+            if [ "$REQUEST_RESULT" != success ]; then
+              echo 'Open the failed step log. If SSH submission failed, check the remote controller state.' >>"$GITHUB_STEP_SUMMARY"
+            fi
+          fi

  apply:
    needs: [gate]
    runs-on: homelab
-    timeout-minutes: 100
+    timeout-minutes: 120
    steps:
      - name: Checkout checked commit
        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
@@ -76,6 +88,14 @@ jobs:
          ref: ${{ needs.gate.outputs.sha }}
      - name: Follow validation and sequential Kubernetes / Compose apply
        run: bash .gitea/workflows/ssh-run.sh apply
+      - name: Write the deploy result
+        if: always()
+        run: |
+          if [ -f .gitea/workflows/ssh-run.sh ]; then
+            bash .gitea/workflows/ssh-run.sh summary
+          elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
+            echo 'Source checkout failed. The remote deploy state is unknown. Check the job log.' >>"$GITHUB_STEP_SUMMARY"
+          fi

  verify:
    needs: [gate, apply]
@@ -89,6 +109,14 @@ jobs:
          ref: ${{ needs.gate.outputs.sha }}
      - name: Follow workload verification and recovery
        run: bash .gitea/workflows/ssh-run.sh verify
+      - name: Write the deploy result
+        if: always()
+        run: |
+          if [ -f .gitea/workflows/ssh-run.sh ]; then
+            bash .gitea/workflows/ssh-run.sh summary
+          elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
+            echo 'Source checkout failed. The remote deploy state is unknown. Check the job log.' >>"$GITHUB_STEP_SUMMARY"
+          fi

  smoke:
    needs: [gate, verify]
@@ -102,3 +130,11 @@ jobs:
          ref: ${{ needs.gate.outputs.sha }}
      - name: Follow public route checks
        run: bash .gitea/workflows/ssh-run.sh smoke
+      - name: Write the deploy result
+        if: always()
+        run: |
+          if [ -f .gitea/workflows/ssh-run.sh ]; then
+            bash .gitea/workflows/ssh-run.sh summary
+          elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
+            echo 'Source checkout failed. The remote deploy state is unknown. Check the job log.' >>"$GITHUB_STEP_SUMMARY"
+          fi
@@ -26,9 +26,6 @@ IMAGES = {
    'xdfnx-homepage': ('homepages', 'homepages/Dockerfile.xdfnx'),
 }

-# These images are released by the EDU application repository.
-EXTERNAL_IMAGES = {'gcr.forust.xyz/forust/session-keeper', 'gcr.forust.xyz/forust/webinar-checker'}
-

 def command(*args, **kwargs):
    """Arguments are passed directly to the executable, never to a shell."""
@@ -163,7 +160,7 @@ def gate(output, requested_ref, event_sha):
    print(f'CI gate accepted {sha}')


-def build(output):
+def prepare_images(output):
    sha = command('git', 'rev-parse', 'HEAD')
    if sha != os.environ['GITHUB_SHA'] or not SHA.fullmatch(sha):
        raise ValueError('Build checkout does not match GITHUB_SHA')
@@ -178,11 +175,61 @@ def build(output):
        except ValueError:
            # Expired artifacts only cost a rebuild; mutable tags are never a fallback.
            continue
+    targets = []
+    for name, (context, dockerfile) in IMAGES.items():
+        image = f'gcr.forust.xyz/forust/{name}'
+        inputs = fingerprint(context, dockerfile)
+        old_digest = (previous or {}).get('images', {}).get(image)
+        targets.append(
+            {
+                'name': name,
+                'image': image,
+                'context': context,
+                'dockerfile': dockerfile,
+                'inputs': inputs,
+                'reuse_digest': old_digest if (previous or {}).get('inputs', {}).get(image) == inputs else None,
+            }
+        )
+    output.write_text(json.dumps({'sha': sha, 'targets': targets}, indent=2) + '\n')
+    if os.environ.get('GITHUB_OUTPUT'):
+        with Path(os.environ['GITHUB_OUTPUT']).open('a') as stream:
+            stream.write('matrix=' + json.dumps({'include': targets}, separators=(',', ':')) + '\n')
+    print(f'Prepared {len(targets)} image jobs; {sum(t["reuse_digest"] is None for t in targets)} require builds')
+
+
+def checked_plan(path):
+    data = json.loads(path.read_text())
+    sha = command('git', 'rev-parse', 'HEAD')
+    if data.get('sha') != sha or sha != os.environ['GITHUB_SHA'] or not SHA.fullmatch(sha):
+        raise ValueError('Image plan does not match the checked source commit')
+    targets = data.get('targets', [])
+    if sorted(t['name'] for t in targets) != sorted(IMAGES):
+        raise ValueError('Image plan must contain each owned image once')
+    for target in targets:
+        name = target['name']
+        context, dockerfile = IMAGES[name]
+        if (target['context'], target['dockerfile'], target['image']) != (
+            context,
+            dockerfile,
+            f'gcr.forust.xyz/forust/{name}',
+        ) or target['inputs'] != fingerprint(context, dockerfile):
+            raise ValueError('Image plan has invalid build inputs')
+        if target['reuse_digest'] is not None and not DIGEST.fullmatch(target['reuse_digest']):
+            raise ValueError('Image plan has an invalid reuse digest')
+    return data
+
+
+def build_images(output, report, name, plan):
+    data = checked_plan(plan)
+    sha = data['sha']
+    target = next(t for t in data['targets'] if t['name'] == name)
+    context, dockerfile = IMAGES[name]
    docker_config = tempfile.mkdtemp(prefix='homelab-registry-')
    builder_config = Path.home() / '.cache/homelab-ci/buildx'
    builder_config.mkdir(parents=True, exist_ok=True)
    env = {**os.environ, 'DOCKER_CONFIG': docker_config, 'BUILDX_CONFIG': str(builder_config)}
    try:
+        report['phase'] = 'Registry login'
        subprocess.run(  # noqa: S603, S607
            [
                shutil.which('docker') or '/usr/bin/docker',
@@ -197,6 +244,7 @@ def build(output):
            check=True,
            env=env,
        )
+        report['phase'] = 'Prepare the builder'
        builder = 'homelab-ci'
        versions = dict(
            re.findall(r'^([A-Z_]+)="([^"\n]+)"$', Path('.gitea/workflows/tool-versions.env').read_text(), re.MULTILINE)
@@ -231,61 +279,67 @@ def build(output):
            )
            signature.write_text(image + '\n')
        release = {'version': 1, 'sha': sha, 'images': {}, 'inputs': {}}
-        for name, (context, dockerfile) in IMAGES.items():
-            image = f'gcr.forust.xyz/forust/{name}'
-            inputs = fingerprint(context, dockerfile)
-            old_digest = (previous or {}).get('images', {}).get(image)
-            exists = False
-            if old_digest and previous['inputs'].get(image) == inputs:
-                exists = (
-                    subprocess.run(  # noqa: S603, S607
-                        [
-                            shutil.which('docker') or '/usr/bin/docker',
-                            'buildx',
-                            'imagetools',
-                            'inspect',
-                            f'{image}@{old_digest}',
-                        ],
-                        capture_output=True,
-                        env=env,
-                        timeout=60,
-                    ).returncode
-                    == 0
-                )
-            if exists:
-                print(f'Reuse {name}: inputs unchanged')
-                digest = old_digest
-            else:
-                print(f'Build {name}', flush=True)
-                metadata = Path(docker_config) / 'metadata.json'
-                command(
-                    'docker',
-                    'buildx',
-                    'build',
-                    '--builder',
-                    builder,
-                    '--push',
-                    '--platform',
-                    'linux/amd64',
-                    '--provenance=false',
-                    '--cache-from',
-                    f'type=registry,ref={image}:buildcache',
-                    '--cache-to',
-                    f'type=registry,ref={image}:buildcache,mode=max',
-                    '--tag',
-                    f'{image}:sha-{sha}',
-                    '--metadata-file',
-                    str(metadata),
-                    '--file',
-                    dockerfile,
-                    context,
+        report['images'] = release['images']
+        report['phase'] = f'Build or reuse {name}'
+        report['current'] = name
+        image = f'gcr.forust.xyz/forust/{name}'
+        inputs = target['inputs']
+        old_digest = target['reuse_digest']
+        exists = False
+        if old_digest:
+            exists = (
+                subprocess.run(  # noqa: S603, S607
+                    [
+                        shutil.which('docker') or '/usr/bin/docker',
+                        'buildx',
+                        'imagetools',
+                        'inspect',
+                        f'{image}@{old_digest}',
+                    ],
+                    capture_output=True,
                    env=env,
-                )
-                digest = json.loads(metadata.read_text())['containerimage.digest']
-            release['images'][image] = digest
-            release['inputs'][image] = inputs
-        validate_release(release, sha)
+                    timeout=60,
+                ).returncode
+                == 0
+            )
+        if exists:
+            print(f'Reuse {name}: inputs unchanged')
+            digest = old_digest
+            report['reused'].append(name)
+        else:
+            print(f'Build {name}', flush=True)
+            metadata = Path(docker_config) / 'metadata.json'
+            command(
+                'docker',
+                'buildx',
+                'build',
+                '--builder',
+                builder,
+                '--platform',
+                'linux/amd64',
+                '--provenance=false',
+                '--cache-from',
+                f'type=registry,ref={image}:buildcache',
+                '--cache-to',
+                f'type=registry,ref={image}:buildcache,mode=max',
+                '--output',
+                f'type=image,name={image},push-by-digest=true,name-canonical=true,push=true',
+                '--metadata-file',
+                str(metadata),
+                '--file',
+                dockerfile,
+                context,
+                env=env,
+            )
+            digest = json.loads(metadata.read_text())['containerimage.digest']
+            report['built'].append(name)
+        release['images'][image] = digest
+        release['inputs'][image] = inputs
+        if not DIGEST.fullmatch(digest):
+            raise ValueError('Image job returned an invalid digest')
        output.write_text(json.dumps(release, indent=2) + '\n')
+        report['current'] = None
+        report['phase'] = 'Release file saved'
    finally:
        # Cleanup errors must neither leak credentials nor mask the original build error.
        try:
@@ -309,6 +363,59 @@ def build(output):
            shutil.rmtree(docker_config)


+def write_summary(lines):
+    path = os.environ.get('GITHUB_STEP_SUMMARY')
+    if path:
+        try:
+            with Path(path).open('a') as stream:
+                stream.write('\n'.join(lines) + '\n\n')
+        except OSError:
+            print('WARNING: cannot write the job summary')
+
+
+def check_summary():
+    lines = [
+        f'## {os.environ["SUMMARY_CHECK"]}',
+        '',
+        f'- Commit: `{os.environ.get("GITHUB_SHA", "unknown")}`',
+        f'- Result: **{os.environ["SUMMARY_RESULT"]}**',
+    ]
+    if os.environ.get('SUMMARY_FAILED_STEP'):
+        lines.append(f'- Failed step: {os.environ["SUMMARY_FAILED_STEP"]}')
+    if os.environ['SUMMARY_RESULT'] != 'success':
+        lines.append('- Open the failed step log for the error details.')
+    write_summary(lines)
+
+
+def build(output, name, plan):
+    report = {'phase': 'Check the source commit', 'current': None, 'built': [], 'reused': [], 'images': {}}
+    result = 'failure'
+    try:
+        build_images(output, report, name, plan)
+        result = 'success'
+    finally:
+        lines = [
+            f'## Image release `{os.environ.get("GITHUB_SHA", "unknown")}`',
+            '',
+            f'- Result: **{result}**',
+            f'- Last stage: {report["phase"]}',
+        ]
+        if result == 'failure':
+            lines.append('- No release from this build can be deployed. Open the failed step log.')
+        if report['current']:
+            lines.append(f'- Image at the failure: `{report["current"]}`')
+        for title, key in (('Built', 'built'), ('Reused from successful CI', 'reused')):
+            lines.extend(['', f'### {title}'])
+            lines.extend(f'- `{name}`' for name in report[key])
+            if not report[key]:
+                lines.append('- None')
+        lines.extend(['', '### Completed image digests'])
+        lines.extend(f'- `{image}@{digest}`' for image, digest in report['images'].items())
+        if not report['images']:
+            lines.append('- None')
+        write_summary(lines)
+
+
 def render(stream, destination):
    release = validate_release(json.loads(Path(os.environ['RELEASE_FILE']).read_text()), os.environ['DEPLOY_SHA'])
    image_line = re.compile(
@@ -319,9 +426,6 @@ def render(stream, destination):
        match = image_line.fullmatch(line.rstrip('\n'))
        if match:
            prefix, quote, image, tail = match.groups()
-            if image in EXTERNAL_IMAGES and f'{image}@sha256:' in line:
-                rendered.append(line)
-                continue
            if image not in release['images']:
                raise ValueError(f'Owned image missing from checked release: {image}')
            line = f'{prefix}{quote}{image}@{release["images"][image]}{quote}{tail}\n'
@@ -331,19 +435,81 @@ def render(stream, destination):
    destination.writelines(rendered)


+def finalize_images(output, fragments, plan):
+    data = checked_plan(plan)
+    sha = data['sha']
+    release = {'version': 1, 'sha': sha, 'images': {}, 'inputs': {}}
+    for name in IMAGES:
+        fragment = json.loads((fragments / f'image-{name}' / 'image.json').read_text())
+        image = f'gcr.forust.xyz/forust/{name}'
+        if fragment.get('sha') != sha or fragment.get('version') != 1 or set(fragment.get('images', {})) != {image}:
+            raise ValueError('Image job artifact is missing or belongs to another commit')
+        target = next(t for t in data['targets'] if t['name'] == name)
+        if fragment.get('inputs') != {image: target['inputs']}:
+            raise ValueError('Image artifact does not match the build plan')
+        release['images'].update(fragment['images'])
+        release['inputs'].update(fragment['inputs'])
+    validate_release(release, sha)
+    # Only a complete set of successful image jobs can publish the release tags.
+    docker_config = tempfile.mkdtemp(prefix='homelab-registry-')
+    env = {**os.environ, 'DOCKER_CONFIG': docker_config}
+    try:
+        subprocess.run(  # noqa: S603, S607
+            [
+                shutil.which('docker') or '/usr/bin/docker',
+                'login',
+                'gcr.forust.xyz',
+                '-u',
+                os.environ['REGISTRY_USERNAME'],
+                '--password-stdin',
+            ],
+            input=os.environ['REGISTRY_PASSWORD'],
+            text=True,
+            check=True,
+            env=env,
+        )
+        for image, digest in release['images'].items():
+            command(
+                'docker',
+                'buildx',
+                'imagetools',
+                'create',
+                '--prefer-index=false',
+                '--tag',
+                f'{image}:sha-{sha}',
+                f'{image}@{digest}',
+                env=env,
+                timeout=90,
+            )
+        output.write_text(json.dumps(release, indent=2) + '\n')
+    finally:
+        shutil.rmtree(docker_config)
+
+
 def main():
    parser = argparse.ArgumentParser(description=__doc__)
-    parser.add_argument('action', choices=('build', 'gate', 'render'))
+    parser.add_argument('action', choices=('prepare', 'image', 'finalize', 'gate', 'render', 'check-summary'))
    parser.add_argument('--output', type=Path, default=Path('release.json'))
    parser.add_argument('--ref', default='main')
    parser.add_argument('--event-sha', default='')
+    parser.add_argument('--image', choices=IMAGES)
+    parser.add_argument('--plan', type=Path, default=Path('build-plan.json'))
+    parser.add_argument('--fragments', type=Path, default=Path('artifacts'))
    args = parser.parse_args()
-    if args.action == 'render':
+    if args.action == 'check-summary':
+        check_summary()
+    elif args.action == 'render':
        render(sys.stdin, sys.stdout)
    elif args.action == 'gate':
        gate(args.output, args.ref, args.event_sha)
+    elif args.action == 'prepare':
+        prepare_images(args.output)
+    elif args.action == 'image':
+        if not args.image:
+            parser.error('--image is required')
+        build(args.output, args.image, args.plan)
    else:
-        build(args.output)
+        finalize_images(args.output, args.fragments, args.plan)


 if __name__ == '__main__':
@@ -1,7 +1,9 @@
 name: renovate-ci

 on:
-  pull_request:
+  # Read the workflow from the trusted base branch. PR code runs only on the
+  # unprivileged runner selected below.
+  pull_request_target:
    paths:
      - "renovate/**"
      - ".gitea/workflows/renovate-ci.yaml"
@@ -26,37 +28,47 @@ permissions:

 jobs:
  validate-renovate:
-    runs-on: homelab
+    runs-on: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' && 'homelab' || 'homelab-pr' }}
    timeout-minutes: 20
    steps:
      - name: Checkout repository
        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+        with:
+          ref: ${{ github.event_name == 'pull_request_target' && github.event.pull_request.head.sha || github.sha }}

-      # renovate/k8s/cronjob.yaml is the single source of truth for the image tag,
-      # so the same version that runs in the cluster is the one validated here.
-      - name: Resolve the deployed Renovate image
+      # renovate/k8s/cronjob.yaml is the single source of truth for the version.
+      - name: Resolve the deployed Renovate version
        id: image
        shell: bash
        run: |
          set -euo pipefail
          image="$(sed -n 's|.*image:[[:space:]]*\(renovate/renovate:[^[:space:]]*\).*|\1|p' \
            renovate/k8s/cronjob.yaml | head -1)"
-          if [ -z "$image" ]; then
-            echo "::error::no renovate/renovate image found in renovate/k8s/cronjob.yaml"
+          if [[ ! "$image" =~ ^renovate/renovate:([0-9]+\.[0-9]+\.[0-9]+)$ ]]; then
+            echo "::error::expected a pinned renovate/renovate semantic version in renovate/k8s/cronjob.yaml"
            exit 1
          fi
-          echo "using $image"
-          echo "image=$image" >> "$GITHUB_OUTPUT"
+          version="${BASH_REMATCH[1]}"
+          echo "using Renovate $version"
+          printf 'version=%s\n' "$version" >> "$GITHUB_OUTPUT"

-      - name: Validate Renovate repository config
+      - name: Prepare pinned validation tools
        shell: bash
        run: |
          set -euo pipefail
-          docker run --rm \
-            -v "$PWD/renovate:/opt/renovate:ro" \
-            -e RENOVATE_CONFIG_FILE=/opt/renovate/renovate.json \
-            "${{ steps.image.outputs.image }}" \
-            renovate-config-validator /opt/renovate/renovate.json
+          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh kubeconform node)"
+          echo "$tools_dir" >> "$GITHUB_PATH"
+
+      - name: Validate Renovate repository config
+        shell: bash
+        env:
+          RENOVATE_VERSION: ${{ steps.image.outputs.version }}
+        run: |
+          set -euo pipefail
+          npm_cache="$(mktemp -d "${RUNNER_TEMP:-/tmp}/renovate-npm-cache.XXXXXXXX")"
+          trap 'rm -rf "$npm_cache"' EXIT
+          NPM_CONFIG_CACHE="$npm_cache" RENOVATE_CONFIG_FILE="$PWD/renovate/renovate.json" \
+            npm exec --yes --package="renovate@${RENOVATE_VERSION}" -- renovate-config-validator

      # The CronJob cannot read the repository, so renovate/k8s/configmap.yaml
      # carries an inlined copy of the config. Fail if it no longer matches.
@@ -70,8 +82,6 @@ jobs:
        shell: bash
        run: |
          set -euo pipefail
-          tools_dir="$(bash .gitea/workflows/install-ci-tools.sh kubeconform)"
-          export PATH="$tools_dir:$PATH"
          kubeconform \
            -strict \
            -ignore-missing-schemas \
@@ -32,11 +32,14 @@ concurrency:

 jobs:
  run-renovate:
+    if: github.ref == 'refs/heads/main'
    runs-on: homelab
    timeout-minutes: 60
    steps:
      - name: Checkout repository
        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+        with:
+          ref: refs/heads/main

      # renovate/k8s/cronjob.yaml is the single source of truth for the image tag.
      # Reading it here means this workflow validates and runs the exact version
@@ -48,21 +51,23 @@ jobs:
          set -euo pipefail
          image="$(sed -n 's|.*image:[[:space:]]*\(renovate/renovate:[^[:space:]]*\).*|\1|p' \
            renovate/k8s/cronjob.yaml | head -1)"
-          if [ -z "$image" ]; then
-            echo "::error::no renovate/renovate image found in renovate/k8s/cronjob.yaml"
+          if [[ ! "$image" =~ ^renovate/renovate:[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
+            echo "::error::expected a pinned renovate/renovate semantic version in renovate/k8s/cronjob.yaml"
            exit 1
          fi
          echo "using $image"
-          echo "image=$image" >> "$GITHUB_OUTPUT"
+          printf 'image=%s\n' "$image" >> "$GITHUB_OUTPUT"

      - name: Validate Renovate config
        shell: bash
+        env:
+          RENOVATE_IMAGE: ${{ steps.image.outputs.image }}
        run: |
          set -euo pipefail
          docker run --rm \
            -v "$PWD/renovate/renovate.json:/opt/renovate/renovate.json:ro" \
            -e RENOVATE_CONFIG_FILE=/opt/renovate/renovate.json \
-            "${{ steps.image.outputs.image }}" \
+            "$RENOVATE_IMAGE" \
            renovate-config-validator

      - name: Run Renovate
@@ -73,6 +78,7 @@ jobs:
          RENOVATE_REPOSITORIES: ${{ inputs.repositories }}
          RENOVATE_DRY_RUN: ${{ inputs.dry_run && 'full' || '' }}
          LOG_LEVEL: ${{ inputs.log_level }}
+          RENOVATE_IMAGE: ${{ steps.image.outputs.image }}
        run: |
          set -euo pipefail

@@ -89,4 +95,4 @@ jobs:
            -e RENOVATE_CONFIG_FILE=/opt/renovate/renovate.json \
            -e RENOVATE_BASE_DIR=/tmp/renovate \
            -e LOG_LEVEL="${LOG_LEVEL:-info}" \
-            "${{ steps.image.outputs.image }}"
+            "$RENOVATE_IMAGE"
@@ -20,7 +20,7 @@ ssh_opts=(-i "$key_dir/key" -p "${DEPLOY_PORT:-22}" -o BatchMode=yes -o StrictHo
  -o "UserKnownHostsFile=$key_dir/known_hosts" -o ConnectTimeout=15
  -o ServerAliveInterval=15 -o ServerAliveCountMax=4)
 controller=.local/lib/homelab-deploy/controller.py
-case "${1:?start, apply, verify or smoke required}" in
+case "${1:?start, apply, verify, smoke or summary required}" in
  start)
    python3 - <<'PY' >"$key_dir/request.json"
 import json
@@ -41,16 +41,30 @@ PY
    exit "$rc"
    ;;
  apply|verify|smoke)
+    result=0
    for attempt in 1 2 3; do
      rc=0
      # shellcheck disable=SC2029 # The run ID and operation are validated local arguments, not remote variables.
      ssh "${ssh_opts[@]}" "$DEPLOY_USER@$DEPLOY_HOST" python3 "$controller" follow "$DEPLOY_RUN_ID" "$1" || rc=$?
-      [ "$rc" -eq 0 ] && exit 0
-      [ "$rc" -eq 255 ] || exit "$rc"
+      [ "$rc" -eq 0 ] && break
+      [ "$rc" -eq 255 ] || { result="$rc"; break; }
      echo "SSH disconnected; reconnecting to the existing deploy ($attempt/3)"
+      if [ "$attempt" -eq 3 ]; then result=255; break; fi
      sleep 5
    done
-    exit "$rc"
+    exit "$result"
+    ;;
+  summary)
+    if [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
+      rc=0
+      # shellcheck disable=SC2029 # The run ID is validated above.
+      ssh "${ssh_opts[@]}" "$DEPLOY_USER@$DEPLOY_HOST" python3 "$controller" summary "$DEPLOY_RUN_ID" >"$key_dir/deploy-summary.md" || rc=$?
+      if [ "$rc" -eq 0 ]; then
+        cat "$key_dir/deploy-summary.md" >>"$GITHUB_STEP_SUMMARY" || echo "WARNING: cannot write the deploy summary"
+      else
+        echo 'Deploy summary is unavailable. The SSH connection failed or the controller did not respond. Check the job log.' >>"$GITHUB_STEP_SUMMARY" || true
+      fi
+    fi
    ;;
  *) echo "Unknown SSH operation: $1" >&2; exit 1 ;;
 esac
@@ -94,7 +94,6 @@ replacements.txt
 .idea

 # Temp files
-edu_master/temp/
 temp/*
 # Local-only tooling scratch space (pinned CI tools, verification scripts)
 tmp/
@@ -1,14 +0,0 @@
-EDU_LOGIN=your_edu_login_here
-EDU_PASSWORD=your_edu_password_here
-EDU_URL_LOGIN=https://edu.edu.vn.ua/user/login
-EDU_URL_VERIFY=https://edu.edu.vn.ua/course/userlist
-PHPSESSID_INTERVAL=10
-USER_AGENT="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/142.0.0.0 Safari/537.36"
-WEBINAR_URL=https://edu.edu.vn.ua/webinar/useractive
-WEBINAR_CHECK_INTERVAL=60
-REDIS_HOST=redis
-REDIS_PORT=6379
-PLAYWRIGHT_WS=ws://playwright-service:3000/ws
-TZ=Europe/Kyiv
-WEBINAR_TELEGRAM_TOKEN=your_telegram_bot_token_here
-WEBINAR_ADMIN_ID=123456789
@@ -1 +0,0 @@
-1.56.0
@@ -1,14 +0,0 @@
-# EDU deployment ownership
-
-Application source and release builds: `forust/edu-master`.
-The homelab pipeline deploys `edu_master/k8s` and preserves explicit image digests.
-The application copies in this directory are legacy and are not build inputs.
-Do not publish EDU `prod` images from homelab or resolve releases from moving tags.
-
-For an EDU release, validate both images, select their digests in the keeper and
-checker manifests, and run the existing homelab validation/apply/verification
-helpers against this service. Keep the existing Secret and Redis PVC.
-Coordinate Redis authentication changes with both clients and all init/probes;
-keep a pre-rollout Redis backup and both previous compatible image references.
-The current HTTP checker does not depend on Playwright; check other consumers
-before removing the separate browser service.
@@ -1,49 +0,0 @@
-services:
-  redis:
-    image: redis:8.10.2-alpine
-    restart: unless-stopped
-    volumes:
-      - redis-data:/data
-    healthcheck:
-      test: ["CMD", "redis-cli", "ping"]
-      interval: 5s
-      timeout: 3s
-      retries: 5
-
-  playwright-service:
-    image: mcr.microsoft.com/playwright:v1.56.0-jammy
-    restart: unless-stopped
-    command: npx -y playwright@1.56.0 run-server --port 3000 --path /ws
-
-  session-keeper:
-    build: ./phpsessid-bot
-    image: gcr.forust.xyz/forust/session-keeper:prod
-    pull_policy: build
-    env_file: .env
-    restart: unless-stopped
-    depends_on:
-      redis:
-        condition: service_healthy
-    healthcheck:
-      test: ["CMD-SHELL", "redis-cli -h redis EXISTS EDU_PHPSESSID | grep -q 1"]
-      interval: 30s
-      timeout: 5s
-      retries: 10
-      start_period: 60s
-
-  webinar-checker:
-    build: ./webinar-checker
-    image: gcr.forust.xyz/forust/webinar-checker:prod
-    pull_policy: build
-    env_file: .env
-    restart: unless-stopped
-    depends_on:
-      redis:
-        condition: service_healthy
-      session-keeper:
-        condition: service_healthy
-      playwright-service:
-        condition: service_started
-
-volumes:
-  redis-data:
@@ -1,115 +0,0 @@
-apiVersion: monitoring.coreos.com/v1
-kind: PrometheusRule
-metadata:
-  name: edu-master-webinar
-  namespace: edu-master
-  labels:
-    release: prometheus-stack
-spec:
-  groups:
-    - name: edu_master.webinar
-      rules:
-        # No successful webinar check for 5m (~2-3 missed 2-min checks).
-        # Catches: playwright hangs/timeouts, version skew, site changes, hung job.
-        # The last_success > 0 guard is mandatory: checker.py initialises
-        # last_success to 0, so without it `time() - 0` equals the current epoch
-        # and humanizeDuration renders ~20722d on every pod restart. Keep the
-        # duration expression on the left so $value stays the real gap.
-        - alert: WebinarCheckerNoSuccessfulCheck
-          expr: |
-            ((time() - webinar_check_last_success_timestamp_seconds) > 300)
-              and (webinar_check_last_success_timestamp_seconds > 0)
-              and (webinar_check_last_run_timestamp_seconds > 0)
-          for: 2m
-          labels:
-            severity: critical
-          annotations:
-            summary: "Webinar checker has no successful check for 5m"
-            description: "edu-master/webinar-checker: last successful webinar check was {{ $value | humanizeDuration }} ago. Checks are failing or hanging (see consecutive failures alert). Notifications about new webinars are NOT being sent."
-
-        # Checks are running but none has ever succeeded since pod start.
-        # Split out from the rule above so a zeroed gauge never feeds
-        # humanizeDuration.
-        - alert: WebinarCheckerNeverSucceeded
-          expr: |
-            (webinar_check_last_success_timestamp_seconds == 0)
-              and (webinar_check_last_run_timestamp_seconds > 0)
-          for: 10m
-          labels:
-            severity: critical
-          annotations:
-            summary: "Webinar checker has never completed a successful check"
-            description: 'edu-master/webinar-checker: checks have been running for 10m but not one has ever succeeded since the pod started, so every check is failing. Check pod logs (Loki: {namespace="edu-master", container="webinar-checker"}).'
-
-        # Fast path: 3 consecutive failures (~6+ min at 2-min interval).
-        - alert: WebinarCheckerConsecutiveFailures
-          expr: |
-            webinar_check_consecutive_failures >= 3
-          for: 5m
-          labels:
-            severity: critical
-          annotations:
-            summary: "Webinar checker failing consecutively"
-            description: 'edu-master/webinar-checker: {{ $value }} consecutive webinar check failures (timeout / http error / page error). Check pod logs (Loki: {namespace="edu-master", container="webinar-checker"}).'
-
-        - alert: WebinarCheckerNeverStarted
-          expr: |
-            (time() - edu_process_start > 120)
-              and (webinar_check_last_run_timestamp_seconds == 0)
-          for: 2m
-          labels:
-            severity: critical
-          annotations:
-            summary: "Webinar checker job has not started"
-            description: "The process exposes metrics but its webinar job has never started."
-
-        - alert: WebinarDeliveryPending
-          expr: edu_delivery_pending > 0
-          for: 5m
-          labels:
-            severity: warning
-          annotations:
-            summary: "Webinar notifications await delivery"
-            description: "Telegram delivery has pending recipients. Check delivery failures and retry status."
-
-        - alert: EduRedisUnavailable
-          expr: edu_redis_connected == 0
-          for: 2m
-          labels:
-            severity: critical
-          annotations:
-            summary: "EDU checker cannot reach Redis"
-            description: "Redis health checks are failing; checker commands and delivery may be unavailable."
-
-        # Metrics endpoint not scraped for 10m: pod down, metrics server dead, or ServiceMonitor broken.
-        - alert: WebinarCheckerScrapeDown
-          expr: |
-            absent(webinar_check_last_run_timestamp_seconds) == 1
-          for: 10m
-          labels:
-            severity: critical
-          annotations:
-            summary: "Webinar checker metrics missing"
-            description: "edu-master/webinar-checker: no metrics series for 10m. Pod may be down, metrics server dead, or ServiceMonitor/Service broken. Webinar checks are unobserved."
-
-        # EDU session lost: session-keeper down or credentials expired. Without PHPSESSID every check is skipped.
-        - alert: EduPhpsessidMissing
-          expr: |
-            edu_phpsessid_present == 0
-          for: 10m
-          labels:
-            severity: critical
-          annotations:
-            summary: "EDU_PHPSESSID missing"
-            description: "edu-master: EDU_PHPSESSID absent from redis for 10m. Webinar/diari/schedule checks are all skipped. Check session-keeper logs and EDU credentials."
-
-        # Hard deps: checker deployment unavailable.
-        - alert: WebinarCheckerDeploymentDown
-          expr: |
-            kube_deployment_status_replicas_unavailable{deployment="webinar-checker", namespace="edu-master"} > 0
-          for: 10m
-          labels:
-            severity: critical
-          annotations:
-            summary: "Webinar checker deployment unavailable"
-            description: "edu-master/webinar-checker deployment has {{ $value }} unavailable replica(s) for 10m."
@@ -1,69 +0,0 @@
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: playwright-service
-  namespace: edu-master
-  labels:
-    app: edu-master-playwright
-spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      app: edu-master-playwright
-  strategy:
-    type: Recreate
-  template:
-    metadata:
-      labels:
-        app: edu-master-playwright
-    spec:
-      containers:
-        - name: playwright
-          # renovate: datasource=docker depName=mcr.microsoft.com/playwright versioning=docker
-          image: mcr.microsoft.com/playwright:v1.56.0-jammy
-          imagePullPolicy: IfNotPresent
-          # p95 412M, max 478M over 7 days, no limit before. Request is set at p95
-          # so the pod is not an eviction candidate; the limit stays above 2x the
-          # request because browser page lifetimes are unpredictable.
-          resources:
-            requests:
-              cpu: "200m"
-              memory: "416Mi"
-            limits:
-              memory: "1Gi"
-          command:
-            - npx
-            - -y
-            - playwright@1.56.0
-            - run-server
-            - --port
-            - "3000"
-            - --path
-            - /ws
-          ports:
-            - containerPort: 3000
-          readinessProbe:
-            tcpSocket:
-              port: 3000
-            initialDelaySeconds: 5
-            periodSeconds: 10
-            timeoutSeconds: 3
-          livenessProbe:
-            tcpSocket:
-              port: 3000
-            initialDelaySeconds: 15
-            periodSeconds: 20
-            timeoutSeconds: 3
---
-apiVersion: v1
-kind: Service
-metadata:
-  name: playwright-service
-  namespace: edu-master
-spec:
-  selector:
-    app: edu-master-playwright
-  ports:
-    - name: ws
-      port: 3000
-      targetPort: 3000
@@ -1,22 +0,0 @@
-apiVersion: networking.k8s.io/v1
-kind: NetworkPolicy
-metadata:
-  name: redis-clients-only
-  namespace: edu-master
-spec:
-  podSelector:
-    matchLabels:
-      app: edu-master-redis
-  policyTypes:
-    - Ingress
-  ingress:
-    - from:
-        - podSelector:
-            matchLabels:
-              app: edu-master-session-keeper
-        - podSelector:
-            matchLabels:
-              app: edu-master-webinar-checker
-      ports:
-        - protocol: TCP
-          port: 6379
@@ -1,96 +0,0 @@
-apiVersion: apps/v1
-kind: StatefulSet
-metadata:
-  name: redis
-  namespace: edu-master
-  labels:
-    app: edu-master-redis
-spec:
-  serviceName: redis
-  replicas: 1
-  selector:
-    matchLabels:
-      app: edu-master-redis
-  template:
-    metadata:
-      labels:
-        app: edu-master-redis
-    spec:
-      containers:
-        - name: redis
-          image: redis:8.10.2-alpine
-          imagePullPolicy: IfNotPresent
-          env:
-            - name: REDIS_PASSWORD
-              valueFrom:
-                secretKeyRef:
-                  name: edu-master-secrets
-                  key: REDIS_PASSWORD
-            - name: REDISCLI_AUTH
-              valueFrom:
-                secretKeyRef:
-                  name: edu-master-secrets
-                  key: REDIS_PASSWORD
-          command:
-            - /bin/sh
-            - -ec
-            - |
-              case "$REDIS_PASSWORD" in *[!0-9a-fA-F]*|'') echo 'REDIS_PASSWORD must be 64 hex characters' >&2; exit 1;; esac
-              [ "${#REDIS_PASSWORD}" -eq 64 ] || { echo 'REDIS_PASSWORD must be 64 hex characters' >&2; exit 1; }
-              umask 077
-              printf 'requirepass "%s"\n' "$REDIS_PASSWORD" > /tmp/redis-auth.conf
-              chown redis:redis /tmp/redis-auth.conf
-              exec docker-entrypoint.sh redis-server /tmp/redis-auth.conf
-          ports:
-            - containerPort: 6379
-          volumeMounts:
-            - name: redis-data
-              mountPath: /data
-          resources:
-            requests:
-              cpu: 25m
-              memory: 32Mi
-            limits:
-              cpu: 250m
-              memory: 128Mi
-          readinessProbe:
-            exec:
-              command: ["redis-cli", "ping"]
-            initialDelaySeconds: 5
-            periodSeconds: 5
-            timeoutSeconds: 3
-          livenessProbe:
-            exec:
-              command: ["redis-cli", "ping"]
-            initialDelaySeconds: 10
-            periodSeconds: 10
-            timeoutSeconds: 3
-      volumes:
-        - name: redis-data
-          persistentVolumeClaim:
-            claimName: redis-data-pvc
---
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: redis-data-pvc
-  namespace: edu-master
-spec:
-  accessModes:
-    - ReadWriteOnce
-  resources:
-    requests:
-      storage: 1Gi
---
-apiVersion: v1
-kind: Service
-metadata:
-  name: redis
-  namespace: edu-master
-spec:
-  selector:
-    app: edu-master-redis
-  ports:
-    - name: redis
-      port: 6379
-      targetPort: 6379
@@ -1,50 +0,0 @@
-# One-time Job to migrate redis state from docker compose to k8s (maintenance window).
-# The .example file is not applied by the deploy pipeline (mask *.example.yaml).
-#
-# Runbook:
-#   1. docker compose -f <repo>/edu_master/compose.yaml stop        # SIGTERM -> redis will flush dump.rdb
-#   2. docker run --rm -v edu_master_redis-data:/data \
-#        -v /tmp/edu-master-backup:/backup \
-#        redis:alpine sh -c "cp /data/dump.rdb /backup/ && ls -la /backup"
-#   3. kubectl apply -f edu_master/k8s/namespace.yaml
-#   4. kubectl apply -f <only the PVC from redis.yaml>              # seed must come BEFORE redis pod starts
-#   5. kubectl apply -f edu_master/k8s/restore-seed-job.yaml.example
-#      kubectl wait --for=condition=complete job/redis-restore-seed -n edu-master --timeout=120s
-#   6. kubectl delete job redis-restore-seed -n edu-master
-#   7. kubectl apply -f edu_master/k8s/ -R                           # apply remaining manifests
-apiVersion: batch/v1
-kind: Job
-metadata:
-  name: redis-restore-seed
-  namespace: edu-master
-spec:
-  backoffLimit: 2
-  ttlSecondsAfterFinished: 3600
-  template:
-    spec:
-      restartPolicy: Never
-      containers:
-        - name: seed
-          image: redis:alpine
-          command:
-            - /bin/sh
-            - -ec
-            - |
-              ls -la /backup
-              cp /backup/dump.rdb /data/dump.rdb
-              chmod 644 /data/dump.rdb
-              ls -la /data
-          volumeMounts:
-            - name: redis-data
-              mountPath: /data
-            - name: backup
-              mountPath: /backup
-              readOnly: true
-      volumes:
-        - name: redis-data
-          persistentVolumeClaim:
-            claimName: redis-data-pvc
-        - name: backup
-          hostPath:
-            path: /tmp/edu-master-backup
-            type: DirectoryOrCreate
@@ -1,29 +0,0 @@
-apiVersion: v1
-kind: Secret
-metadata:
-  name: edu-master-secrets
-  namespace: edu-master
-type: Opaque
-stringData:
-  # Session keeper credentials
-  KEEPER_LOGIN: ""
-  KEEPER_PASSWORD: ""
-  KEEPER_INTERVAL: "10"
-  # EDU links
-  EDU_URL_BASE: "https://edu.edu.vn.ua"
-  EDU_URL_LOGIN: "/user/login"
-  EDU_URL_COURSES: "/course/userlist"
-  EDU_URL_WEBINAR: "/webinar/useractive"
-  # Playwright
-  USER_AGENT: ""
-  PLAYWRIGHT_WS: "ws://playwright-service:3000/ws"
-  # Webinar-checker
-  WEBINAR_TELEGRAM_TOKEN: ""
-  WEBINAR_ADMIN_ID: ""
-  WEBINAR_CHECK_INTERVAL: "60"
-  # Prometheus metrics endpoint (scraped via ServiceMonitor, alerts in k8s/alerts.yaml)
-  METRICS_PORT: "8000"
-  # Database
-  REDIS_HOST: "redis"
-  REDIS_PORT: "6379"
-  TZ: "Europe/Kyiv"
@@ -1,15 +0,0 @@
-apiVersion: v1
-kind: Service
-metadata:
-  name: webinar-checker
-  namespace: edu-master
-  labels:
-    app: edu-master-webinar-checker
-spec:
-  selector:
-    app: edu-master-webinar-checker
-  ports:
-    - name: metrics
-      port: 8000
-      targetPort: metrics
-      protocol: TCP
@@ -1,69 +0,0 @@
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
-  name: session-keeper
-  namespace: edu-master
-  labels:
-    app: edu-master-session-keeper
-spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      app: edu-master-session-keeper
-  strategy:
-    type: Recreate
-  template:
-    metadata:
-      annotations:
-        edu.forust.xyz/source-commit: "90829d6c8080b9928f9da23587678e640939e10a"
-      labels:
-        app: edu-master-session-keeper
-    spec:
-      initContainers:
-        - name: wait-redis
-          image: redis:8.10.2-alpine
-          env:
-            - name: REDISCLI_AUTH
-              valueFrom:
-                secretKeyRef:
-                  name: edu-master-secrets
-                  key: REDIS_PASSWORD
-          command:
-            - /bin/sh
-            - -ec
-            - |
-              i=0
-              until redis-cli -h redis ping | grep -q PONG; do
-                i=$((i+1))
-                [ "$i" -ge 300 ] && echo "TIMEOUT: redis not ready" && exit 1
-                sleep 2
-              done
-              echo "redis is ready"
-      containers:
-        - name: session-keeper
-          image: gcr.forust.xyz/forust/session-keeper@sha256:49285e87cc5bc4cf4ffe190813d87927916c2df8a206daac0aeb7d227c636450
-          envFrom:
-            - secretRef:
-                name: edu-master-secrets
-          env:
-            - name: REDISCLI_AUTH
-              valueFrom:
-                secretKeyRef:
-                  name: edu-master-secrets
-                  key: REDIS_PASSWORD
-          resources:
-            requests:
-              cpu: 25m
-              memory: 32Mi
-            limits:
-              cpu: 250m
-              memory: 128Mi
-          readinessProbe:
-            exec:
-              command: ["/bin/sh", "-ec", "redis-cli -h redis EXISTS EDU_PHPSESSID | grep -q 1"]
-            initialDelaySeconds: 15
-            periodSeconds: 30
-            timeoutSeconds: 5
-            failureThreshold: 10
@@ -1,87 +0,0 @@
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  annotations:
-    reloader.stakater.com/auto: "true"
-  name: webinar-checker
-  namespace: edu-master
-  labels:
-    app: edu-master-webinar-checker
-spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      app: edu-master-webinar-checker
-  strategy:
-    type: Recreate
-  template:
-    metadata:
-      annotations:
-        edu.forust.xyz/source-commit: "90829d6c8080b9928f9da23587678e640939e10a"
-      labels:
-        app: edu-master-webinar-checker
-    spec:
-      # Enforces dependency order like compose depends_on:
-      # redis healthy -> session-keeper healthy (EXISTS EDU_PHPSESSID)
-      initContainers:
-        - name: wait-deps
-          image: redis:8.10.2-alpine
-          env:
-            - name: REDISCLI_AUTH
-              valueFrom:
-                secretKeyRef:
-                  name: edu-master-secrets
-                  key: REDIS_PASSWORD
-          command:
-            - /bin/sh
-            - -ec
-            - |
-              i=0
-              until redis-cli -h redis ping | grep -q PONG; do
-                i=$((i+1))
-                [ "$i" -ge 300 ] && echo "TIMEOUT: redis not ready" && exit 1
-                sleep 2
-              done
-              echo "redis ok"
-              until [ "$(redis-cli -h redis EXISTS EDU_PHPSESSID)" = "1" ]; do
-                i=$((i+1))
-                [ "$i" -ge 300 ] && echo "TIMEOUT: no PHPSESSID (session-keeper down?)" && exit 1
-                sleep 2
-              done
-              echo "PHPSESSID ok"
-      containers:
-        - name: webinar-checker
-          image: gcr.forust.xyz/forust/webinar-checker@sha256:66c146f7b43cb9f0dc31ba9aa36d217e01df42ddafba5971b79c12ec215b2c01
-          ports:
-            - name: metrics
-              containerPort: 8000
-              protocol: TCP
-          readinessProbe:
-            httpGet:
-              path: /health
-              port: metrics
-            periodSeconds: 10
-            timeoutSeconds: 3
-            failureThreshold: 12
-            initialDelaySeconds: 10
-          livenessProbe:
-            httpGet:
-              path: /live
-              port: metrics
-            initialDelaySeconds: 60
-            periodSeconds: 15
-            timeoutSeconds: 3
-            failureThreshold: 4
-          envFrom:
-            - secretRef:
-                name: edu-master-secrets
-          env:
-            - name: TZ
-              value: "Europe/Kyiv"
-          resources:
-            requests:
-              cpu: "50m"
-              memory: "192Mi"
-            limits:
-              cpu: "600m"
-              memory: "384Mi"
@@ -1,15 +0,0 @@
-FROM python:3.14-slim
-
-WORKDIR /app
-
-# Install system dependencies
-RUN apt-get update && apt-get install -y --no-install-recommends redis-tools && rm -rf /var/lib/apt/lists/*
-
-# Install dependencies
-RUN pip install --no-cache-dir requests==2.32.3 redis==5.2.1
-
-# Copy application code
-COPY . .
-
-# Run the bot
-CMD ["python", "bot.py"]
@@ -1,132 +0,0 @@
-import logging
-import os
-import time
-from datetime import datetime
-
-import redis
-import requests
-
-# Configure logging
-logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
-logger = logging.getLogger(__name__)
-
-
-# Load configuration (adapted to .env keys)
-def _env(key, default=None):
-    v = os.getenv(key, default)
-    if isinstance(v, str) and len(v) >= 2 and ((v[0] == '"' and v[-1] == '"') or (v[0] == "'" and v[-1] == "'")):
-        return v[1:-1]
-    return v
-
-
-LOGIN = _env('KEEPER_LOGIN')
-PASSWORD = _env('KEEPER_PASSWORD')
-
-EDU_BASE = _env('EDU_URL_BASE', 'https://edu.edu.vn.ua')
-EDU_LOGIN_PATH = _env('EDU_URL_LOGIN', '/user/login')
-EDU_COURSES_PATH = _env('EDU_URL_COURSES', '/course/userlist')
-URL_LOGIN = f'{EDU_BASE.rstrip("/")}/{EDU_LOGIN_PATH.lstrip("/")}'
-URL_VERIFY = f'{EDU_BASE.rstrip("/")}/{EDU_COURSES_PATH.lstrip("/")}'
-
-INTERVAL = int(_env('KEEPER_INTERVAL', 10))
-USER_AGENT = _env(
-    'USER_AGENT',
-    'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/142.0.0.0 Safari/537.36',
-)
-REDIS_HOST = _env('REDIS_HOST', 'redis')
-REDIS_PORT = int(_env('REDIS_PORT', 6379))
-
-SUCCESS_FILE = '/tmp/last_success'  # noqa: S108
-
-
-def touch_success_file():
-    """Updates the timestamp of the success file for healthchecks."""
-    try:
-        with open(SUCCESS_FILE, 'w') as f:
-            f.write(str(datetime.now().timestamp()))
-    except Exception as e:
-        logger.error(f'Failed to touch success file: {e}')
-
-
-def main():
-    logger.info('Starting Session Keeper Bot')
-
-    # Connect to Redis
-    try:
-        redis_client = redis.Redis(host=REDIS_HOST, port=REDIS_PORT, decode_responses=True)
-        redis_client.ping()
-        logger.info(f'Connected to Redis at {REDIS_HOST}:{REDIS_PORT}')
-    except Exception as e:
-        logger.error(f'Failed to connect to Redis: {e}')
-        return
-
-    session = requests.Session()
-
-    # Set headers
-    headers = {
-        'User-Agent': USER_AGENT,
-        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',
-        'Accept-Language': 'en-US,en;q=0.9',
-        'Cache-Control': 'max-age=0',
-        'Upgrade-Insecure-Requests': '1',
-        'Sec-Fetch-Site': 'same-origin',
-        'Sec-Fetch-Mode': 'navigate',
-        'Sec-Fetch-User': '?1',
-        'Sec-Fetch-Dest': 'document',
-        'Sec-Ch-Ua': '"Not_A Brand";v="99", "Chromium";v="142"',
-        'Sec-Ch-Ua-Mobile': '?0',
-        'Sec-Ch-Ua-Platform': '"Linux"',
-        'Accept-Encoding': 'gzip, deflate, br',
-        'Priority': 'u=0, i',
-    }
-    session.headers.update(headers)
-
-    while True:
-        try:
-            logger.info('Attempting login...')
-
-            # Login payload
-            payload = {'login': LOGIN, 'password': PASSWORD}
-
-            # Perform Login
-            # Note: The user request shows a POST to /user/login with form data
-            # We need to make sure we handle the PHPSESSID correctly.
-            # If we already have a PHPSESSID, requests will send it.
-
-            login_response = session.post(URL_LOGIN, data=payload, allow_redirects=True)
-
-            logger.info(f'Login Response Status: {login_response.status_code}')
-            logger.info(f'Cookies after login: {session.cookies.get_dict()}')
-
-            # Verify Session
-            logger.info('Verifying session...')
-            verify_response = session.get(URL_VERIFY, allow_redirects=False)
-
-            logger.info(f'Verify Response Status: {verify_response.status_code}')
-
-            if verify_response.status_code == 200:
-                logger.info('Session verification SUCCESS (200 OK).')
-                touch_success_file()
-
-                # Save PHPSESSID to Redis
-                phpsessid = session.cookies.get('PHPSESSID')
-                if phpsessid:
-                    try:
-                        redis_client.set('EDU_PHPSESSID', phpsessid)
-                        logger.info(f'Saved PHPSESSID to Redis: {phpsessid}')
-                    except Exception as e:
-                        logger.error(f'Failed to save PHPSESSID to Redis: {e}')
-            elif verify_response.status_code == 302:
-                logger.warning('Session verification FAILED (302 Redirect). Session might be invalid.')
-            else:
-                logger.warning(f'Session verification returned unexpected status: {verify_response.status_code}')
-
-        except Exception as e:
-            logger.error(f'An error occurred: {e}')
-
-        logger.info(f'Sleeping for {INTERVAL} minutes...')
-        time.sleep(INTERVAL * 60)
-
-
-if __name__ == '__main__':
-    main()
@@ -1,13 +0,0 @@
-FROM python:3.14-slim
-
-WORKDIR /app
-
-# renovate: datasource=pypi depName=playwright versioning=pep440
-ARG PLAYWRIGHT_VERSION=1.56.0
-
-# Install dependencies - PLAYWRIGHT_VERSION is single-source, renovate updates ARG above and all other places via regexManagers
-RUN pip install --no-cache-dir pip==25.0.1 && pip install --no-cache-dir playwright==${PLAYWRIGHT_VERSION} redis==5.2.1 requests==2.32.3 "python-telegram-bot[job-queue]==21.10"
-
-COPY checker.py .
-
-CMD ["python", "checker.py"]
@@ -20,6 +20,8 @@ data:

  GITEA__mailer__ENABLED: "false"

+  GITEA__metrics__ENABLED: "true"
+
  # No code/issue search needed: bleve reindexes the whole issue index on
  # every pod restart (cron.rebuild_issue_indexer RUN_AT_START) and hammers
  # the rotational disk for an hour. "db" serves issue search from postgres.
@@ -3,6 +3,8 @@ kind: Service
 metadata:
  name: gitea-service
  namespace: gitea
+  labels:
+    app: gitea
 spec:
  selector:
    app: gitea
@@ -7,7 +7,8 @@ spec:
  entryPoints:
    - websecure
  routes:
-    - match: Host(`gitea.forust.xyz`) || Host(`git.forust.xyz`)
+    # Metrics are scraped directly through the cluster Service.
+    - match: (Host(`gitea.forust.xyz`) || Host(`git.forust.xyz`)) && !PathPrefix(`/metrics`)
      kind: Rule
      services:
        - name: gitea-service
@@ -0,0 +1,16 @@
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: gitea
+  namespace: gitea
+  labels:
+    release: prometheus-stack
+spec:
+  selector:
+    matchLabels:
+      app: gitea
+  endpoints:
+    - port: http
+      path: /metrics
+      interval: 30s
+      scrapeTimeout: 10s
@@ -7,7 +7,7 @@
 # # Dev server_url
 # server_url: https://hs.dev_internal_domain.internal
 listen_addr: 0.0.0.0:8080
-metrics_listen_addr: 127.0.0.1:9090
+metrics_listen_addr: 0.0.0.0:9090
 grpc_listen_addr: 127.0.0.1:50443
 grpc_allow_insecure: false
 noise:
@@ -3,6 +3,8 @@ kind: Service
 metadata:
  name: headscale-server-external
  namespace: headscale
+  labels:
+    app: headscale
 spec:
  ports:
    - port: 8080
@@ -0,0 +1,18 @@
+apiVersion: operator.victoriametrics.com/v1beta1
+kind: VMServiceScrape
+metadata:
+  name: headscale
+  namespace: headscale
+  labels:
+    release: prometheus-stack
+spec:
+  # The external Service has a manually managed EndpointSlice, not Endpoints.
+  discoveryRole: endpointslice
+  selector:
+    matchLabels:
+      app: headscale
+  endpoints:
+    - port: metrics
+      path: /metrics
+      interval: 30s
+      scrapeTimeout: 10s
@@ -6,6 +6,10 @@ metadata:
 data:
  TZ: "Europe/Bratislava"

+  IMMICH_TELEMETRY_INCLUDE: "all"
+  IMMICH_API_METRICS_PORT: "8081"
+  IMMICH_MICROSERVICES_METRICS_PORT: "8082"
+
  # The database in this namespace, not the shared one in the database
  # namespace: v3 needs VectorChord, and only the dedicated image carries it.
  DB_HOSTNAME: "immich-postgres"
@@ -3,6 +3,8 @@ kind: Service
 metadata:
  name: immich-service
  namespace: immich
+  labels:
+    app: immich
 spec:
  selector:
    app: immich
@@ -10,6 +12,12 @@ spec:
    - name: http
      port: 2283
      targetPort: 2283
+    - name: api-metrics
+      port: 8081
+      targetPort: api-metrics
+    - name: worker-metrics
+      port: 8082
+      targetPort: worker-metrics
 ---
 apiVersion: apps/v1
 kind: Deployment
@@ -41,6 +49,10 @@ spec:
          ports:
            - name: http
              containerPort: 2283
+            - name: api-metrics
+              containerPort: 8081
+            - name: worker-metrics
+              containerPort: 8082
          volumeMounts:
            - name: immich-data
              mountPath: /data
@@ -0,0 +1,20 @@
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: immich
+  namespace: immich
+  labels:
+    release: prometheus-stack
+spec:
+  selector:
+    matchLabels:
+      app: immich
+  endpoints:
+    - port: api-metrics
+      path: /metrics
+      interval: 30s
+      scrapeTimeout: 10s
+    - port: worker-metrics
+      path: /metrics
+      interval: 30s
+      scrapeTimeout: 10s
@@ -3,6 +3,8 @@ kind: Service
 metadata:
  name: netbird-server-service
  namespace: netbird
+  labels:
+    app: netbird-server
 spec:
  selector:
    app: netbird-server
@@ -11,6 +13,10 @@ spec:
      name: http
      targetPort: 80
      protocol: TCP
+    - port: 9090
+      name: metrics
+      targetPort: metrics
+      protocol: TCP
    - port: 3478
      name: stun
      targetPort: 3478
@@ -59,6 +65,9 @@ spec:
            - containerPort: 80
              name: http
              protocol: TCP
+            - containerPort: 9090
+              name: metrics
+              protocol: TCP
            - containerPort: 3478
              name: stun
              protocol: UDP
@@ -1,14 +1,14 @@
 apiVersion: monitoring.coreos.com/v1
 kind: ServiceMonitor
 metadata:
-  name: webinar-checker
-  namespace: edu-master
+  name: netbird-server
+  namespace: netbird
  labels:
    release: prometheus-stack
 spec:
  selector:
    matchLabels:
-      app: edu-master-webinar-checker
+      app: netbird-server
  endpoints:
    - port: metrics
      path: /metrics
@@ -0,0 +1,16 @@
+PAPERLESS_URL=https://papers.forust.xyz
+PAPERLESS_ALLOWED_HOSTS=papers.forust.xyz,papers.workstation.internal
+PAPERLESS_CSRF_TRUSTED_ORIGINS=https://papers.forust.xyz,https://papers.workstation.internal
+PAPERLESS_TIME_ZONE=Europe/Bratislava
+PAPERLESS_REDIS=redis://valkey:6379
+PAPERLESS_DBENGINE=postgresql
+PAPERLESS_DBHOST=homelab-postgres
+PAPERLESS_DBNAME=paperless
+PAPERLESS_DBUSER=paperless
+PAPERLESS_DBPASS=<SET_THE_SAME_VALUE_AS_SHARED_POSTGRES_PAPERLESS_DB_PASSWORD>
+PAPERLESS_OCR_LANGUAGE=rus+eng
+PAPERLESS_OCR_LANGUAGES=rus
+PAPERLESS_TASK_WORKERS=1
+PAPERLESS_ADMIN_USER=admin
+PAPERLESS_SECRET_KEY=<GENERATE_WITH_python3_-c_import_secrets;_print(secrets.token_urlsafe(64))>
+PAPERLESS_ADMIN_PASSWORD=<SET_A_LONG_UNIQUE_PASSWORD>
@@ -0,0 +1,79 @@
+# Paperless-ngx
+
+Paperless-ngx runs in the `paperless` namespace. It uses the shared PostgreSQL
+service in the `database` namespace and Valkey for its task queue. The document
+library, exports, and consume folder are stored on the `local-path-retain`
+volume. The PVC size is fixed at 50 GiB because this storage class does not
+support volume expansion.
+
+The local route is `https://papers.workstation.internal`; the public route is
+`https://papers.forust.xyz`. Both use TLS. Paperless keeps its own login and
+password authentication. OCR is configured for Russian and English documents.
+Keep `papers.forust.xyz` in the existing Cloudflare DDNS `DOMAINS` setting so
+the public record follows the workstation address.
+
+## Compose alternative
+
+`compose.yaml` is an alternative to the active Kubernetes deployment. Do not
+run both at the same time: they use the same Paperless database and route
+names, but have separate document volumes.
+
+The Compose variant uses the shared Compose PostgreSQL service on the
+`homelab-database` Docker network. It does not start a PostgreSQL container.
+The shared Compose database must be running and must have the `paperless`
+database and role. Set `PAPERLESS_DBPASS` to the same password as
+`PAPERLESS_DB_PASSWORD` in the shared PostgreSQL configuration.
+
+To prepare and start the Compose variant:
+
+```sh
+cp paperless/.env.example paperless/.env
+cd paperless
+docker compose -f compose.yaml config --quiet
+docker compose -f compose.yaml up -d
+```
+
+Create unique values for `PAPERLESS_SECRET_KEY` and
+`PAPERLESS_ADMIN_PASSWORD` in `.env`. This directory has no Compose `active`
+marker, so the repository deploy workflow does not start this alternative.
+Stop the Kubernetes Paperless deployment before switching to Compose. Back up
+and migrate the media files as well as the database; the Compose named volumes
+are separate from the Kubernetes PVC.
+
+## Prepare the secret
+
+Create `k8s/secrets.yaml` on the workstation from
+`k8s/secrets.yaml.example`. Set a unique random `PAPERLESS_SECRET_KEY`, a long
+`PAPERLESS_ADMIN_PASSWORD`, and `PAPERLESS_DB_PASSWORD`.
+
+Add the same `PAPERLESS_DB_PASSWORD` value to the local
+`postgres/k8s/secrets.yaml` file. Keep both secret files out of Git. The
+database bootstrap Job creates the `paperless` role and database from the
+shared PostgreSQL secret. The job runs in the `database` namespace and needs
+that namespace's existing `postgres-shared-secrets` Secret.
+
+For example, generate a key with:
+
+```sh
+python3 -c 'import secrets; print(secrets.token_urlsafe(64))'
+```
+
+Then apply the secret before enabling the service:
+
+```sh
+kubectl apply -f paperless/k8s/namespace.yaml
+kubectl apply -f postgres/k8s/secrets.yaml
+kubectl apply -f paperless/k8s/secrets.yaml
+```
+
+The normal deploy workflow applies the remaining manifests when
+`paperless/k8s/active` is present. Verify the rollout and ingress after deploy:
+
+```sh
+kubectl -n paperless rollout status deployment/paperless
+kubectl -n paperless get pods,pvc,services
+```
+
+Back up the `paperless-data` PVC and the shared PostgreSQL database. The PVC
+contains the originals, archived PDFs, and export/consume folders. Valkey has
+no persistent volume; queued tasks are recreated after a restart.
@@ -0,0 +1,77 @@
+services:
+  paperless:
+    image: ghcr.io/paperless-ngx/paperless-ngx:3.2.1
+    container_name: paperless
+    restart: unless-stopped
+    env_file:
+      - .env
+    depends_on:
+      valkey:
+        condition: service_healthy
+    deploy:
+      resources:
+        limits:
+          cpus: "2.0"
+          memory: 2G
+        reservations:
+          cpus: "0.10"
+          memory: 512M
+    volumes:
+      - paperless-data:/usr/src/paperless/data
+      - paperless-media:/usr/src/paperless/media
+      - paperless-export:/usr/src/paperless/export
+      - paperless-consume:/usr/src/paperless/consume
+    networks:
+      - default
+      - proxy
+      - database
+    labels:
+      - "traefik.enable=true"
+      - "traefik.docker.network=proxy"
+      - "traefik.http.services.paperless-compose.loadbalancer.server.port=8000"
+      - "traefik.http.routers.paperless-compose.rule=Host(`papers.forust.xyz`)"
+      - "traefik.http.routers.paperless-compose.entrypoints=websecure"
+      - "traefik.http.routers.paperless-compose.tls.certresolver=letsencrypt"
+      - "traefik.http.routers.paperless-compose-local.rule=Host(`papers.workstation.internal`)"
+      - "traefik.http.routers.paperless-compose-local.entrypoints=websecure"
+      - "traefik.http.routers.paperless-compose-local.tls=true"
+
+  valkey:
+    image: valkey/valkey:9.0.3-alpine
+    container_name: paperless-valkey
+    restart: unless-stopped
+    command:
+      - valkey-server
+      - --save
+      - ""
+      - --appendonly
+      - "no"
+    healthcheck:
+      test: ["CMD", "valkey-cli", "ping"]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+    deploy:
+      resources:
+        limits:
+          cpus: "0.25"
+          memory: 256M
+        reservations:
+          cpus: "0.025"
+          memory: 64M
+    networks:
+      - default
+
+volumes:
+  paperless-data:
+  paperless-media:
+  paperless-export:
+  paperless-consume:
+
+networks:
+  default:
+  proxy:
+    external: true
+  database:
+    name: homelab-database
+    external: true
@@ -0,0 +1 @@
+
@@ -0,0 +1,28 @@
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: paperless-prod-tls
+  namespace: paperless
+spec:
+  secretName: paperless-prod-tls
+  dnsNames:
+    - papers.forust.xyz
+  issuerRef:
+    name: letsencrypt-prod
+    kind: ClusterIssuer
+---
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: internal-wildcard-tls
+  namespace: paperless
+spec:
+  secretName: internal-wildcard-tls
+  dnsNames:
+    - "*.workstation.internal"
+    - "*.gigaforust.internal"
+    - workstation.internal
+    - gigaforust.internal
+  issuerRef:
+    name: internal-ca
+    kind: ClusterIssuer
@@ -0,0 +1,58 @@
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: paperless-database-init
+  namespace: database
+spec:
+  backoffLimit: 5
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: paperless-database-init
+    spec:
+      restartPolicy: OnFailure
+      containers:
+        - name: create-database
+          image: postgres:17.11-alpine
+          command:
+            - /bin/sh
+            - -ec
+            - |
+              PGPASSWORD="$POSTGRES_ADMIN_PASSWORD" psql \
+                --host postgres \
+                --username postgres \
+                --dbname postgres \
+                --set ON_ERROR_STOP=1 \
+                --set paperless_password="$PAPERLESS_DB_PASSWORD" <<'SQL'
+              SELECT format(
+                'CREATE ROLE paperless LOGIN PASSWORD %L',
+                :'paperless_password'
+              )
+              WHERE NOT EXISTS (
+                SELECT FROM pg_roles WHERE rolname = 'paperless'
+              )
+              \gexec
+              SELECT format('CREATE DATABASE paperless OWNER paperless')
+              WHERE NOT EXISTS (
+                SELECT FROM pg_database WHERE datname = 'paperless'
+              )
+              \gexec
+              SQL
+          env:
+            - name: POSTGRES_ADMIN_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: postgres-shared-secrets
+                  key: POSTGRES_ADMIN_PASSWORD
+            - name: PAPERLESS_DB_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: postgres-shared-secrets
+                  key: PAPERLESS_DB_PASSWORD
+          resources:
+            requests:
+              cpu: 10m
+              memory: 32Mi
+            limits:
+              cpu: 100m
+              memory: 128Mi
@@ -0,0 +1,33 @@
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
+metadata:
+  name: paperless-prod
+  namespace: paperless
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`papers.forust.xyz`)
+      kind: Rule
+      services:
+        - name: paperless
+          port: 8000
+  tls:
+    secretName: paperless-prod-tls
+---
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
+metadata:
+  name: paperless-local
+  namespace: paperless
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`papers.workstation.internal`)
+      kind: Rule
+      services:
+        - name: paperless
+          port: 8000
+  tls:
+    secretName: internal-wildcard-tls
@@ -1,4 +1,4 @@
 apiVersion: v1
 kind: Namespace
 metadata:
-  name: edu-master
+  name: paperless
@@ -0,0 +1,39 @@
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: paperless-ingress
+  namespace: paperless
+spec:
+  podSelector:
+    matchLabels:
+      app.kubernetes.io/name: paperless
+  policyTypes:
+    - Ingress
+  ingress:
+    - from:
+        - namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: traefik
+      ports:
+        - protocol: TCP
+          port: 8000
+---
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: paperless-valkey-ingress
+  namespace: paperless
+spec:
+  podSelector:
+    matchLabels:
+      app.kubernetes.io/name: paperless-valkey
+  policyTypes:
+    - Ingress
+  ingress:
+    - from:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: paperless
+      ports:
+        - protocol: TCP
+          port: 6379
@@ -0,0 +1,210 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: paperless
+  namespace: paperless
+  labels:
+    app.kubernetes.io/name: paperless
+spec:
+  selector:
+    app.kubernetes.io/name: paperless
+  ports:
+    - name: http
+      port: 8000
+      targetPort: http
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: paperless
+  namespace: paperless
+  labels:
+    app.kubernetes.io/name: paperless
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: paperless
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: paperless
+    spec:
+      enableServiceLinks: false
+      containers:
+        - name: paperless
+          image: ghcr.io/paperless-ngx/paperless-ngx:3.2.1
+          ports:
+            - name: http
+              containerPort: 8000
+          env:
+            - name: PAPERLESS_URL
+              value: https://papers.forust.xyz
+            - name: PAPERLESS_ALLOWED_HOSTS
+              value: papers.forust.xyz,papers.workstation.internal
+            - name: PAPERLESS_CSRF_TRUSTED_ORIGINS
+              value: https://papers.forust.xyz,https://papers.workstation.internal
+            - name: PAPERLESS_TIME_ZONE
+              value: Europe/Bratislava
+            - name: PAPERLESS_REDIS
+              value: redis://paperless-valkey:6379
+            - name: PAPERLESS_DBENGINE
+              value: postgresql
+            - name: PAPERLESS_DBHOST
+              value: postgres.database.svc.cluster.local
+            - name: PAPERLESS_DBNAME
+              value: paperless
+            - name: PAPERLESS_DBUSER
+              value: paperless
+            - name: PAPERLESS_DBPASS
+              valueFrom:
+                secretKeyRef:
+                  name: paperless-secrets
+                  key: PAPERLESS_DB_PASSWORD
+            - name: PAPERLESS_OCR_LANGUAGE
+              value: rus+eng
+            - name: PAPERLESS_OCR_LANGUAGES
+              value: rus
+            - name: PAPERLESS_TASK_WORKERS
+              value: "1"
+            - name: PAPERLESS_ADMIN_USER
+              value: admin
+            - name: PAPERLESS_SECRET_KEY
+              valueFrom:
+                secretKeyRef:
+                  name: paperless-secrets
+                  key: PAPERLESS_SECRET_KEY
+            - name: PAPERLESS_ADMIN_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: paperless-secrets
+                  key: PAPERLESS_ADMIN_PASSWORD
+          volumeMounts:
+            - name: documents
+              mountPath: /usr/src/paperless/data
+              subPath: data
+            - name: documents
+              mountPath: /usr/src/paperless/media
+              subPath: media
+            - name: documents
+              mountPath: /usr/src/paperless/export
+              subPath: export
+            - name: documents
+              mountPath: /usr/src/paperless/consume
+              subPath: consume
+          startupProbe:
+            httpGet:
+              path: /
+              port: http
+              httpHeaders:
+                - name: Host
+                  value: papers.workstation.internal
+            failureThreshold: 60
+            periodSeconds: 10
+            timeoutSeconds: 5
+          readinessProbe:
+            httpGet:
+              path: /
+              port: http
+              httpHeaders:
+                - name: Host
+                  value: papers.workstation.internal
+            periodSeconds: 10
+            timeoutSeconds: 5
+          livenessProbe:
+            httpGet:
+              path: /
+              port: http
+              httpHeaders:
+                - name: Host
+                  value: papers.workstation.internal
+            periodSeconds: 30
+            timeoutSeconds: 5
+          resources:
+            requests:
+              cpu: 100m
+              memory: 512Mi
+            limits:
+              cpu: "2"
+              memory: 2Gi
+      volumes:
+        - name: documents
+          persistentVolumeClaim:
+            claimName: paperless-data
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: paperless-data
+  namespace: paperless
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: local-path-retain
+  resources:
+    requests:
+      storage: 50Gi
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: paperless-valkey
+  namespace: paperless
+  labels:
+    app.kubernetes.io/name: paperless-valkey
+spec:
+  selector:
+    app.kubernetes.io/name: paperless-valkey
+  ports:
+    - name: redis
+      port: 6379
+      targetPort: redis
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: paperless-valkey
+  namespace: paperless
+  labels:
+    app.kubernetes.io/name: paperless-valkey
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: paperless-valkey
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: paperless-valkey
+    spec:
+      containers:
+        - name: valkey
+          image: valkey/valkey:9.0.3-alpine
+          args:
+            - valkey-server
+            - --save
+            - ""
+            - --appendonly
+            - "no"
+          ports:
+            - name: redis
+              containerPort: 6379
+          readinessProbe:
+            exec:
+              command: ["valkey-cli", "ping"]
+            periodSeconds: 10
+          livenessProbe:
+            exec:
+              command: ["valkey-cli", "ping"]
+            periodSeconds: 30
+          resources:
+            requests:
+              cpu: 25m
+              memory: 64Mi
+            limits:
+              cpu: 250m
+              memory: 256Mi
@@ -0,0 +1,10 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: paperless-secrets
+  namespace: paperless
+type: Opaque
+stringData:
+  PAPERLESS_SECRET_KEY: "<GENERATE_WITH_python3_-c_import_secrets;_print(secrets.token_urlsafe(64))>"
+  PAPERLESS_ADMIN_PASSWORD: "<SET_A_LONG_UNIQUE_PASSWORD>"
+  PAPERLESS_DB_PASSWORD: "<SET_THE_SAME_VALUE_AS_database_PAPERLESS_DB_PASSWORD>"
@@ -4,4 +4,5 @@ GITEA_DB_PASSWORD=
 NETBOX_DB_PASSWORD=
 NETRONOME_DB_PASSWORD=
 PENPOT_DB_PASSWORD=
+PAPERLESS_DB_PASSWORD=
 STATUSPAGE_DB_PASSWORD=
@@ -1,20 +1,21 @@
 # Shared PostgreSQL

 This directory contains the shared PostgreSQL 17 deployment for Authentik,
-Gitea, NetBox, Netronome, and Statuspage. It creates one database and one login role
-per service. Per-service standalone databases were removed after the
-migration (Sep 2026); Penpot stays on its own compose PostgreSQL (archived,
-not part of the shared instance).
+Gitea, NetBox, Netronome, Paperless-ngx, and Statuspage. It creates one database
+and one login role per service. Per-service standalone databases were removed
+after the migration (Sep 2026); Penpot stays on its own compose PostgreSQL
+(archived, not part of the shared instance).

 ## Compatibility baseline

-| Service    | Current application | Shared PostgreSQL 17                   |
-| ---------- | ------------------- | -------------------------------------- |
-| Authentik  | 2025.10.x           | Supported (Authentik requires 14+)     |
-| Gitea      | 1.27.3              | Supported (Gitea requires 12+)         |
-| NetBox     | 4.7.x               | Supported (NetBox 4.x requires 13+)    |
-| Netronome  | 0.14.0              | Supported (upstream's example uses 17) |
-| Statuspage | custom              | Supported                              |
+| Service       | Current application | Shared PostgreSQL 17                   |
+| ------------- | ------------------- | -------------------------------------- |
+| Authentik     | 2025.10.x           | Supported (Authentik requires 14+)     |
+| Gitea         | 1.27.3              | Supported (Gitea requires 12+)         |
+| NetBox        | 4.7.x               | Supported (NetBox 4.x requires 13+)    |
+| Netronome     | 0.14.0              | Supported (upstream's example uses 17) |
+| Paperless-ngx | 3.2.1               | Supported                              |
+| Statuspage    | custom              | Supported                              |

 A major-version change must use a logical dump/restore; changing only the
 image tag while keeping a data directory is not supported.
@@ -6,6 +6,7 @@ set -euo pipefail
 : "${NETBOX_DB_PASSWORD:?NETBOX_DB_PASSWORD is required}"
 : "${NETRONOME_DB_PASSWORD:?NETRONOME_DB_PASSWORD is required}"
 : "${PENPOT_DB_PASSWORD:?PENPOT_DB_PASSWORD is required}"
+: "${PAPERLESS_DB_PASSWORD:?PAPERLESS_DB_PASSWORD is required}"
 : "${STATUSPAGE_DB_PASSWORD:?STATUSPAGE_DB_PASSWORD is required}"

 create_role_and_database() {
@@ -27,4 +28,5 @@ create_role_and_database gitea gitea "$GITEA_DB_PASSWORD"
 create_role_and_database netbox netbox "$NETBOX_DB_PASSWORD"
 create_role_and_database netronome netronome "$NETRONOME_DB_PASSWORD"
 create_role_and_database penpot penpot "$PENPOT_DB_PASSWORD"
+create_role_and_database paperless paperless "$PAPERLESS_DB_PASSWORD"
 create_role_and_database statuspage statuspage "$STATUSPAGE_DB_PASSWORD"
@@ -26,6 +26,18 @@ spec:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: penpot
+        - namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: paperless
+          podSelector:
+            matchLabels:
+              app.kubernetes.io/name: paperless
+        - namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: database
+          podSelector:
+            matchLabels:
+              app.kubernetes.io/name: paperless-database-init
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: statuspage
@@ -126,6 +126,7 @@ data:
    : "${NETBOX_DB_PASSWORD:?NETBOX_DB_PASSWORD is required}"
    : "${NETRONOME_DB_PASSWORD:?NETRONOME_DB_PASSWORD is required}"
    : "${PENPOT_DB_PASSWORD:?PENPOT_DB_PASSWORD is required}"
+    : "${PAPERLESS_DB_PASSWORD:?PAPERLESS_DB_PASSWORD is required}"
    : "${STATUSPAGE_DB_PASSWORD:?STATUSPAGE_DB_PASSWORD is required}"

    create_role_and_database() {
@@ -147,4 +148,5 @@ data:
    create_role_and_database netbox netbox "$NETBOX_DB_PASSWORD"
    create_role_and_database netronome netronome "$NETRONOME_DB_PASSWORD"
    create_role_and_database penpot penpot "$PENPOT_DB_PASSWORD"
+    create_role_and_database paperless paperless "$PAPERLESS_DB_PASSWORD"
    create_role_and_database statuspage statuspage "$STATUSPAGE_DB_PASSWORD"
@@ -11,4 +11,5 @@ stringData:
  NETBOX_DB_PASSWORD: ""
  NETRONOME_DB_PASSWORD: ""
  PENPOT_DB_PASSWORD: ""
+  PAPERLESS_DB_PASSWORD: ""
  STATUSPAGE_DB_PASSWORD: ""
@@ -15,3 +15,29 @@ The VictoriaMetrics Operator chart and its CRDs are installed before the
 Kubernetes manifests by the normal deploy workflow. On a cluster where the
 operator CRDs are not installed yet, CI skips the server-side dry-run of the
 `VMAgent` resource; the deploy installs the chart before applying that resource.
+
+## Application metrics
+
+The application ServiceMonitors use a 30s interval and a 10s timeout:
+
+- Headscale: the external Service points to the Compose host on port 19090.
+  A VMServiceScrape uses EndpointSlice discovery for this manually managed target.
+  The Compose configuration must bind metrics to `0.0.0.0:9090`.
+- NetBird: the combined server exports `/metrics` on port 9090. The existing
+  `server.metricsPort` setting enables the listener.
+- Gitea: `GITEA__metrics__ENABLED` enables `/metrics` on the HTTP port. The public
+  ingress excludes this path. The monitor uses the internal Service directly.
+- Immich: `IMMICH_TELEMETRY_INCLUDE=all` enables API and worker metrics on ports
+  8081 and 8082. The monitor scrapes both ports on each server replica.
+
+Deploy through the existing CI and deploy workflow. Gitea and Immich reload their
+ConfigMap changes through Reloader. Check the VMAgent targets after deployment
+and query `up{scraper="victoria",namespace=~"netbird|gitea|immich|headscale"}` in
+VictoriaMetrics. All targets should report 1.
+
+For rollback, revert the application metrics changes, run CI, and deploy the
+revert. Remove the three application ServiceMonitors and the Headscale VMServiceScrape explicitly: the deployment
+workflow applies manifests and does not prune removed resources.
+
+For Headscale rollback, remove its VMServiceScrape and Service label, restore the
+previous Compose metrics bind address, and restart only the Headscale service.
@@ -29,31 +29,6 @@ data:
        "managerFilePatterns": ["/k8s/.+\\.ya?ml$/"]
      },
      "customManagers": [
-        {
-          "customType": "regex",
-          "description": "singlesource: playwright npm version pinned in npx command (k8s + compose)",
-          "managerFilePatterns": ["edu_master/k8s/playwright.yaml", "edu_master/compose.yaml"],
-          "matchStrings": ["playwright@(?<currentValue>\\d+\\.\\d+\\.\\d+)"],
-          "datasourceTemplate": "npm",
-          "depNameTemplate": "playwright"
-        },
-        {
-          "customType": "regex",
-          "description": "singlesource: PLAYWRIGHT_VERSION file",
-          "managerFilePatterns": ["edu_master/PLAYWRIGHT_VERSION"],
-          "matchStrings": ["^(?<currentValue>\\d+\\.\\d+\\.\\d+)(?:\\r?\\n)?$"],
-          "datasourceTemplate": "pypi",
-          "depNameTemplate": "playwright"
-        },
-        {
-          "customType": "regex",
-          "description": "singlesource: playwright Python client version pinned in Dockerfile ARG",
-          "managerFilePatterns": ["edu_master/webinar-checker/Dockerfile"],
-          "matchStrings": ["(?:^|\\n)ARG PLAYWRIGHT_VERSION=(?<currentValue>\\d+\\.\\d+\\.\\d+)(?:\\r?\\n|$)"],
-          "datasourceTemplate": "pypi",
-          "depNameTemplate": "playwright",
-          "versioningTemplate": "pep440"
-        },
        {
          "customType": "regex",
          "description": "kube-prometheus-stack chart version pinned in the deploy workflow",
@@ -212,17 +187,6 @@ data:
          "matchPackageNames": ["/gcr\\.forust\\.xyz\\/forust\\/.+/"],
          "enabled": false
        },
-        {
-          "description": "singlesource playwright - use whichever version is found, keep docker+pypi+npm in sync",
-          "matchPackageNames": ["playwright", "mcr.microsoft.com/playwright"],
-          "groupName": "playwright singlesource",
-          "groupSlug": "playwright"
-        },
-        {
-          "description": "playwright must not automerge - version skew breaks the WS handshake (checker.py:1523 vs playwright.yaml:20)",
-          "matchPackageNames": ["playwright", "mcr.microsoft.com/playwright"],
-          "automerge": false
-        },
        {
          "description": "Renovate updates itself in lockstep across the CronJob and the Compose file",
          "matchPackageNames": ["renovate/renovate"],
@@ -18,31 +18,6 @@
    "managerFilePatterns": ["/k8s/.+\\.ya?ml$/"]
  },
  "customManagers": [
-    {
-      "customType": "regex",
-      "description": "singlesource: playwright npm version pinned in npx command (k8s + compose)",
-      "managerFilePatterns": ["edu_master/k8s/playwright.yaml", "edu_master/compose.yaml"],
-      "matchStrings": ["playwright@(?<currentValue>\\d+\\.\\d+\\.\\d+)"],
-      "datasourceTemplate": "npm",
-      "depNameTemplate": "playwright"
-    },
-    {
-      "customType": "regex",
-      "description": "singlesource: PLAYWRIGHT_VERSION file",
-      "managerFilePatterns": ["edu_master/PLAYWRIGHT_VERSION"],
-      "matchStrings": ["^(?<currentValue>\\d+\\.\\d+\\.\\d+)(?:\\r?\\n)?$"],
-      "datasourceTemplate": "pypi",
-      "depNameTemplate": "playwright"
-    },
-    {
-      "customType": "regex",
-      "description": "singlesource: playwright Python client version pinned in Dockerfile ARG",
-      "managerFilePatterns": ["edu_master/webinar-checker/Dockerfile"],
-      "matchStrings": ["(?:^|\\n)ARG PLAYWRIGHT_VERSION=(?<currentValue>\\d+\\.\\d+\\.\\d+)(?:\\r?\\n|$)"],
-      "datasourceTemplate": "pypi",
-      "depNameTemplate": "playwright",
-      "versioningTemplate": "pep440"
-    },
    {
      "customType": "regex",
      "description": "kube-prometheus-stack chart version pinned in the deploy workflow",
@@ -201,17 +176,6 @@
      "matchPackageNames": ["/gcr\\.forust\\.xyz\\/forust\\/.+/"],
      "enabled": false
    },
-    {
-      "description": "singlesource playwright - use whichever version is found, keep docker+pypi+npm in sync",
-      "matchPackageNames": ["playwright", "mcr.microsoft.com/playwright"],
-      "groupName": "playwright singlesource",
-      "groupSlug": "playwright"
-    },
-    {
-      "description": "playwright must not automerge - version skew breaks the WS handshake (checker.py:1523 vs playwright.yaml:20)",
-      "matchPackageNames": ["playwright", "mcr.microsoft.com/playwright"],
-      "automerge": false
-    },
    {
      "description": "Renovate updates itself in lockstep across the CronJob and the Compose file",
      "matchPackageNames": ["renovate/renovate"],
@@ -40,19 +40,6 @@ class ArtifactTests(unittest.TestCase):
                release_module.render(io.StringIO(f'  image: "{image}:prod" # note\n'), result)
            self.assertEqual(result.getvalue(), f'  image: "{image}@sha256:{"b" * 64}" # note\n')

-    def test_edu_release_digest_is_preserved(self):
-        with tempfile.TemporaryDirectory() as scratch:
-            path = Path(scratch) / 'release.json'
-            path.write_text(json.dumps(release()))
-            image = 'gcr.forust.xyz/forust/session-keeper'
-            line = f'image: {image}@sha256:{"e" * 64}\n'
-            result = io.StringIO()
-            with patch.dict(os.environ, {'RELEASE_FILE': str(path), 'DEPLOY_SHA': 'a' * 40}):
-                release_module.render(io.StringIO(line), result)
-                self.assertEqual(result.getvalue(), line)
-                with self.assertRaises(ValueError):
-                    release_module.render(io.StringIO(f'image: {image}:prod\n'), io.StringIO())
-
    def test_unknown_image_cannot_emit_partial_manifest(self):
        with tempfile.TemporaryDirectory() as scratch:
            path = Path(scratch) / 'release.json'
@@ -107,10 +94,13 @@ class ArtifactTests(unittest.TestCase):
                patch.object(release_module, 'command', side_effect=fake_command),
                patch.object(subprocess, 'run', return_value=subprocess.CompletedProcess([], 0)),
            ):
-                release_module.build(root / 'release.json')
+                plan = root / 'plan.json'
+                release_module.prepare_images(plan)
+                for name in release_module.IMAGES:
+                    release_module.build(root / f'{name}.json', name, plan)
            self.assertEqual(built, ['errorpages/Dockerfile'])
            self.assertTrue(all(not directory.exists() for directory in auth_directories))
-            self.assertEqual(json.loads((root / 'release.json').read_text())['sha'], 'e' * 40)
+            self.assertEqual(json.loads((root / 'error-pages.json').read_text())['sha'], 'e' * 40)


 class DurableRunTests(unittest.TestCase):
@@ -213,6 +203,64 @@ class DurableRunTests(unittest.TestCase):
            self.assertEqual(json.loads((directory / 'status.json').read_text())['state'], 'failure')


+class FailureSummaryTests(unittest.TestCase):
+    def test_build_failure_keeps_progress_and_does_not_expose_exception_text(self):
+        with tempfile.TemporaryDirectory() as scratch:
+            summary = Path(scratch) / 'summary.md'
+
+            def failed_build(_output, report, _name, _plan):
+                report.update(phase='Build or reuse xdfnx-homepage', current='xdfnx-homepage', built=['error-pages'])
+                report['images']['gcr.forust.xyz/forust/error-pages'] = 'sha256:' + 'b' * 64
+                raise RuntimeError('private value must not appear in the summary')
+
+            with (
+                patch.dict(os.environ, {'GITHUB_STEP_SUMMARY': str(summary), 'GITHUB_SHA': 'a' * 40}),
+                patch.object(release_module, 'build_images', side_effect=failed_build),
+                self.assertRaises(RuntimeError),
+            ):
+                release_module.build(Path(scratch) / 'release.json', 'xdfnx-homepage', Path('plan.json'))
+            content = summary.read_text()
+            self.assertIn('**failure**', content)
+            self.assertIn('error-pages', content)
+            self.assertIn('xdfnx-homepage', content)
+            self.assertNotIn('private value', content)
+
+    def test_deploy_failure_reports_completed_apply_and_rollback_result(self):
+        with tempfile.TemporaryDirectory() as scratch:
+            state = Path(scratch)
+            directory = state / 'runs/123-1'
+            snapshot = directory / 'snapshot/before'
+            snapshot.mkdir(parents=True)
+            (directory / 'snapshot/current').write_text(str(snapshot))
+            (snapshot / 'failed-workloads').write_text('deployment app api\nROLLED_BACK=1\nUNRECOVERED=0\n')
+            controller.atomic_json(
+                directory / 'request.json', {'release': release(), 'mode': 'changed', 'refresh_images': False}
+            )
+            controller.atomic_json(
+                directory / 'status.json',
+                {
+                    'state': 'failure',
+                    'stages': {
+                        'apply-k8s': {'result': 'success', 'exit_code': 0},
+                        'verify-k8s': {'result': 'failure', 'exit_code': 1},
+                    },
+                },
+            )
+            controller.atomic_json(directory / 'plan.json', {'selected': {'k8s': ['app'], 'compose': []}})
+            (directory / 'apply-events.jsonl').write_text(
+                json.dumps({'action': 'kubectl', 'target': 'app/k8s/api.yaml', 'result': 'success'}) + '\n'
+            )
+            output = io.StringIO()
+            with patch.object(controller, 'STATE', state), patch('sys.stdout', output):
+                controller.summary('123-1')
+            content = output.getvalue()
+            self.assertIn('verify-k8s | failure | 1', content)
+            self.assertIn('app/k8s/api.yaml', content)
+            self.assertIn('Workloads restored: **1**', content)
+            self.assertIn('manual recovery: **0**', content)
+            self.assertIn('Compose requires manual recovery', content)
+
+
 class InstallerTests(unittest.TestCase):
    def test_version_comparison_is_exact_without_network_or_host_packages(self):
        with tempfile.TemporaryDirectory() as scratch:
@@ -0,0 +1,144 @@
+"""Matrix release contracts and failure gates without a registry."""
+
+import json
+import os
+import tempfile
+import unittest
+from pathlib import Path
+from unittest.mock import Mock, call, patch
+
+from test_cicd import release, release_module
+
+
+def plan_data(changed):
+    targets = []
+    for name, (context, dockerfile) in release_module.IMAGES.items():
+        targets.append(
+            {
+                'name': name,
+                'image': f'gcr.forust.xyz/forust/{name}',
+                'context': context,
+                'dockerfile': dockerfile,
+                'inputs': ('d' if name in changed else 'c') * 64,
+                'reuse_digest': None if name in changed else 'sha256:' + 'b' * 64,
+            }
+        )
+    return {'sha': 'a' * 40, 'targets': targets}
+
+
+class MatrixTests(unittest.TestCase):
+    def test_no_change_one_image_all_images_and_missing_baseline(self):
+        for changed in (set(), {'error-pages'}, set(release_module.IMAGES)):
+            with self.subTest(changed=changed), tempfile.TemporaryDirectory() as scratch:
+                api = Mock()
+                api.successful_runs.return_value = iter([{'id': 1}])
+                api.release.return_value = release()
+                expected = plan_data(changed)
+                fingerprints = {t['dockerfile']: t['inputs'] for t in expected['targets']}
+                output = Path(scratch) / 'plan.json'
+                with (
+                    patch.dict(os.environ, {'GITHUB_SHA': 'a' * 40, 'GITHUB_RUN_ID': '2'}),
+                    patch.object(release_module, 'command', return_value='a' * 40),
+                    patch.object(release_module, 'Gitea', return_value=api),
+                    patch.object(
+                        release_module, 'fingerprint', side_effect=lambda _c, f, mapping=fingerprints: mapping[f]
+                    ),
+                ):
+                    release_module.prepare_images(output)
+                self.assertEqual(json.loads(output.read_text()), expected)
+        api.successful_runs.return_value = iter([])
+        with (
+            tempfile.TemporaryDirectory() as scratch,
+            patch.dict(os.environ, {'GITHUB_SHA': 'a' * 40}),
+            patch.object(release_module, 'command', return_value='a' * 40),
+            patch.object(release_module, 'Gitea', return_value=api),
+            patch.object(release_module, 'fingerprint', return_value='c' * 64),
+        ):
+            output = Path(scratch) / 'plan.json'
+            release_module.prepare_images(output)
+            self.assertTrue(all(t['reuse_digest'] is None for t in json.loads(output.read_text())['targets']))
+
+    def test_incomplete_or_wrong_sha_fragments_cannot_publish_tags(self):
+        for wrong_sha in (False, True):
+            with self.subTest(wrong_sha=wrong_sha), tempfile.TemporaryDirectory() as scratch:
+                root = Path(scratch)
+                plan = root / 'plan.json'
+                plan.write_text(json.dumps(plan_data(set())))
+                for name in release_module.IMAGES:
+                    if name == 'xdfnx-homepage' and not wrong_sha:
+                        continue
+                    folder = root / f'image-{name}'
+                    folder.mkdir()
+                    image = f'gcr.forust.xyz/forust/{name}'
+                    folder.joinpath('image.json').write_text(
+                        json.dumps(
+                            {
+                                'version': 1,
+                                'sha': ('e' if wrong_sha else 'a') * 40,
+                                'images': {image: 'sha256:' + 'b' * 64},
+                                'inputs': {image: 'c' * 64},
+                            }
+                        )
+                    )
+                with (
+                    patch.object(release_module, 'checked_plan', return_value=plan_data(set())),
+                    patch.object(release_module.subprocess, 'run') as execute,
+                    self.assertRaises((ValueError, FileNotFoundError)),
+                ):
+                    release_module.finalize_images(root / 'release.json', root, plan)
+                execute.assert_not_called()
+                self.assertFalse((root / 'release.json').exists())
+
+    def test_manifest_only_release_pins_each_successful_digest(self):
+        with tempfile.TemporaryDirectory() as scratch:
+            root = Path(scratch)
+            data = plan_data(set())
+            for target in data['targets']:
+                folder = root / f'image-{target["name"]}'
+                folder.mkdir()
+                image = target['image']
+                folder.joinpath('image.json').write_text(
+                    json.dumps(
+                        {
+                            'version': 1,
+                            'sha': data['sha'],
+                            'images': {image: target['reuse_digest']},
+                            'inputs': {image: target['inputs']},
+                        }
+                    )
+                )
+            with (
+                patch.object(release_module, 'checked_plan', return_value=data),
+                patch.dict(os.environ, {'REGISTRY_USERNAME': 'test', 'REGISTRY_PASSWORD': 'placeholder'}),
+                patch.object(release_module.subprocess, 'run'),
+                patch.object(release_module, 'command') as execute,
+            ):
+                release_module.finalize_images(root / 'release.json', root, root / 'plan.json')
+            self.assertEqual(
+                [entry.args for entry in execute.call_args_list],
+                [
+                    call(
+                        'docker',
+                        'buildx',
+                        'imagetools',
+                        'create',
+                        '--prefer-index=false',
+                        '--tag',
+                        f'{t["image"]}:sha-{data["sha"]}',
+                        f'{t["image"]}@{t["reuse_digest"]}',
+                    ).args
+                    for t in data['targets']
+                ],
+            )
+            self.assertEqual(json.loads((root / 'release.json').read_text()), release())
+
+    def test_checkout_mismatch_cannot_build(self):
+        with tempfile.TemporaryDirectory() as scratch:
+            plan = Path(scratch) / 'plan.json'
+            plan.write_text(json.dumps(plan_data(set())))
+            with (
+                patch.dict(os.environ, {'GITHUB_SHA': 'e' * 40}),
+                patch.object(release_module, 'command', return_value='a' * 40),
+                self.assertRaisesRegex(ValueError, 'source commit'),
+            ):
+                release_module.checked_plan(plan)