fix(ci): validate image digests and isolate test outputs

Reviewed-on: #117

.gitea/EDU_HANDOFF.md

@@ -1,42 +1,50 @@
 # EDU ownership handoff
 
-## Current status
+## Status
 
-EDU PR #1 merged at 2026-10-07 08:04:30 UTC. Main release `5094952464ce315130839303985fd04d721bc1f2` passed CI run 1585 and deploy run 1586. The workstation checkout `/srv/edu-master` is at that SHA. The release changed the application image digests:
+The EDU ownership handoff is complete. The homelab repository no longer owns
+EDU workloads, images, routes, alerts, or deployment selection. The EDU
+repository is the only deployment owner: [forust/edu-master](https://git.forust.xyz/forust/edu-master).
 
-- Session keeper: `sha256:1e59473bd40fe4c22622017d808a8927a68788275fe073dc23d718c44b2fd5dd`
+Homelab PRs #99 and #105 are merged. PR #105 removed the EDU subtree and its
-- Webinar checker: `sha256:987d9bf0770272766523ea5b94c7f3f849175d737551d46591ae55e058cf9f12`
+build, deploy, rollback, verification, route-probe, and registry references.
+It also added the serial image build matrix for the homelab services. This
+handoff record is the only remaining EDU-specific file in homelab Git.
 
-The live workloads remain healthy in context `Default`, namespace `edu-master`. Both health and live probes return 200. Redis AUTH passes, session TTL is 1178 seconds, the delivery backlog is zero, all nine EDU alert rules are healthy, and the scrape target is UP. The unauthorized-pod Redis check passed. The Redis PVC UID and Secret UID and values, including the Fernet key, match their pre-release state.
+The dedicated workstation checkout is `/srv/edu-master`, at release
+`4f2b2a0e37dc11ac2c75441a15076c178e219d37`. It contains `k8s/active`; root
+`active` is absent. The old untracked `/srv/homelab/edu_master` checkout was
+moved outside the homelab repository to
+`/srv/edu-master-legacy-archive-20261007/edu_master`. Its private files remain
+mode `0600` inside an archive directory with mode `0700`. The homelab deploy
+checkout has no EDU marker or tracked EDU application/deployment files.
+`AUTODEPLOY=false` remains in place for homelab deployment.
 
-The homelab EDU active marker was present after the EDU deployment. It was moved to the private snapshot as `homelab-k8s-active.marker` while holding `/tmp/homelab-apply.lock`. The homelab checkout at `/srv/homelab` is at `5f9354b` and has the tracked marker deletion. Its deploy preflight blocks a dirty checkout until this removal is reconciled. Preserve private ignored configuration when syncing that checkout.
+## Release evidence
 
-The remaining homelab change is PR #105, branch `feat/edu-handoff-matrix`, based on `codex/ci-visible-checks`. Its eight protected checks passed. Renovate runs 1587 and 1588 passed. Image publishing was skipped for the PR. The EDU runtime changes are in PR #3 from `fix/handoff-runtime` to `main`; CI run 1589 is in progress. Those runtime changes have not been released.
+EDU PR #4 merged after its review and CI checks. Main-push CI run 1652 passed
+all validation and both image builds. Deploy run 1653 passed for the exact main
+SHA above.
 
-## Approval gate and next steps
+The workstation rollout completed for both Deployments. The deployment
+verified `/health` and `/live` with HTTP 200, Redis AUTH, session TTL of 1058
+seconds, a delivery backlog of zero, and all nine EDU vmalert rules with
+matching expressions and healthy evaluation.
 
-PR #99 must merge before PR #105 can target `main`. A merge attempt for PR #99 returned HTTP 405 because it needs one approval; the protected branch has `required_approvals=1` and whitelist approval is enabled. This approval gate prevents the remaining transfer steps.
+The images now run by digest:
 
-After the required approval:
+- Session keeper: `sha256:998dea51aa3015fd9cabefb0f53b030157a650c3bef72e02fe84f17d5762613d`
+- Webinar checker: `sha256:92f3c1fa2bb7f9b4680a9fc76a5b33dfbea8ef3dd9c6490ebc45876fd4c54461`
 
-1. Merge PR #99.
+Redis StatefulSet was unchanged. PVC `redis-data-pvc` remains bound to PV
-2. Retarget PR #105 to `main`. Complete CI and review, then approve and merge it.
+`pvc-a4f2a79a-363a-4c12-ae91-92cdfc2a0d2e` with capacity 1 GiB. The existing
-3. Under the homelab apply lock, sync `/srv/homelab` to the merged removal. Preserve private ignored configuration and keep the active marker removed. Confirm the deploy preflight is clean.
+runtime Secret and Fernet key were preserved during the handoff. Notification
-4. Merge the EDU runtime PR after its CI and review pass. The main-push CI run must complete successfully before its exact SHA can deploy.
+delivery was verified before closeout, as confirmed by the operator. The
-5. Verify the new release SHA, image digests, workload health, Redis AUTH and TTL, backlog, PVC and Secret identity, and monitoring. Record the results in the EDU PR.
+deployment did not record downtime.
 
-`AUTODEPLOY=false` is explicitly configured. The EDU repository path and port secrets are confirmed, and `EDU_KUBE_CONTEXT=Default` is configured as a repository variable. Keep deployment and registry credentials outside Git. Never run both homelab and EDU deployment paths at the same time.
+The release rollback snapshot is
-
+`/home/forust/.local/state/edu-master-deploy/20261007T180541Z-4f2b2a0e37dc11ac2c75441a15076c178e219d37`.
-## Change summary
+The handoff data snapshot remains at
-
+`/home/forust/.local/state/edu-master-deploy/handoff-20261007T080838Z`.
-The homelab PR removes the EDU subtree, deployment and image selection, rollback and verification cases, route probes, Renovate references, and external-image exceptions. It adds a serial dynamic matrix for the three homelab images. Each job builds an image or reuses a matching immutable digest. The final job checks all image results and publishes full-SHA tags and the existing release artifact only after they pass. PRs do not publish images. The protected check names from PR #99 are preserved. PR #100's service-metrics work is independent of this handoff.
+Both snapshots are outside Git. Do not restore old Redis data unless recovery
-
+requires it. Never delete or recreate the Redis PVC.
-The EDU runtime PR adds the Playwright service manifest, reconciles Redis storage and Secret reload annotations, and adds pre-apply Redis backup and identity checks. It verifies application endpoints, Redis AUTH, session TTL, metrics, and all nine vmalert rules. Rollback checks workload and application health and reports when manual recovery is needed. Its deployment guard rejects an unexpected or dirty checkout and refuses deployment while either legacy homelab EDU marker exists.
-
-## Rollback and limits
-
-The private snapshot is `/home/forust/.local/state/edu-master-deploy/handoff-20261007T080838Z` on the workstation. It contains the pre-handoff Redis RDB and recovery data. RDB checksum verification confirmed twelve keys. Keep the snapshot outside Git. For an EDU release failure, restore the saved Kubernetes resources and inspect application health. The rollback does not automatically restore the Redis RDB; restore old Redis data only when recovery requires it.
-
-For an ownership rollback, stop EDU deployment triggers first, restore the reviewed homelab source and marker, then reapply recorded immutable image digests. Verify both workload and application health. Never delete or recreate the Redis PVC.
-
-The initial EDU release and the homelab marker move are complete. PR #99 approval and merge, PR #105 retarget and merge, homelab checkout reconciliation, EDU runtime PR merge, and release of those runtime changes remain pending. Synthetic Telegram delivery and Alertmanager-to-Telegram notification were not tested.

.gitea/runner/README.md

@@ -157,3 +157,25 @@ run first. Restore the runner config/unit from `.before-<timestamp>` backups,
 reload systemd and restart the runner. Restore the prior workflows from Git.
 Production data and persistent volumes stay where they were. Do not remove run
 state or Compose recovery files until recovery is confirmed.
+
+### Compose configuration recovery
+
+Successful deploys save the complete resolved Compose configuration in
+`~/.local/state/homelab-deploy/compose-configs/`. These files can contain secrets.
+Keep them private and do not commit or upload them.
+The next deploy uses this configuration for its recovery file, including old
+commands, environment, mounts, ports, and removed services. The recovery command
+uses `--remove-orphans` to remove services added by the failed deploy. It does
+not restore volume data or reverse database migrations.
+
+On the first run after this update, the controller can use the Compose file
+from the previous successful run. If that file is absent, it reads the persistent
+checkout and checks its service configuration hashes against existing containers.
+A mismatch stops preflight. Restore the previous configuration before retrying.
+Update the installed controller with `bash .gitea/runner/setup-workstation.sh`
+from the reviewed checkout before using this change.
+
+New namespaces are checked during preflight. Server validation of their resources
+runs after namespace creation and before application resources are applied.
+Plan mode does not create namespaces. A failed deferred check can leave an empty
+namespace; inspect it before removing it.

.gitea/tests/deploy-validation.sh

@@ -91,4 +91,66 @@ if check_referenced_secrets >"$scratch/secrets.log"; then
   echo 'Secret check accepted a failed manifest render' >&2
   exit 1
 fi
+
+# New declared namespaces defer only their own resources during preflight.
+render_selected_resources() {
+  cat <<'JSON'
+{"apiVersion":"v1","kind":"List","items":[
+ {"apiVersion":"v1","kind":"Namespace","metadata":{"name":"new"}},
+ {"apiVersion":"v1","kind":"ConfigMap","metadata":{"name":"new-config","namespace":"new"}},
+ {"apiVersion":"v1","kind":"ConfigMap","metadata":{"name":"existing-config","namespace":"default"}}
+]}
+JSON
+}
+kubectl() {
+  case "$1" in
+    get) printf '%s\n' '{"items":[{"metadata":{"name":"default"}}]}' ;;
+    apply) cat >"$scratch/server-input.json" ;;
+    *) return 1 ;;
+  esac
+}
+validate_server_resources true
+jq -e '.items | length == 2 and all(.metadata.name != "new-config")' "$scratch/server-input.json" >/dev/null
+if validate_server_resources false 2>"$scratch/deferred.log"; then
+  echo 'Post-namespace validation accepted a missing namespace' >&2
+  exit 1
+fi
+kubectl() {
+  case "$1" in
+    get) printf '%s\n' '{"items":[{"metadata":{"name":"default"}},{"metadata":{"name":"new"}}]}' ;;
+    apply) cat >"$scratch/server-input.json" ;;
+    *) return 1 ;;
+  esac
+}
+validate_server_resources false
+jq -e '.items | length == 3' "$scratch/server-input.json" >/dev/null
+render_selected_resources() {
+  printf '%s\n' '{"items":[{"kind":"ConfigMap","metadata":{"name":"bad","namespace":"undeclared"}}]}'
+}
+if validate_server_resources true 2>"$scratch/undeclared.log"; then
+  echo 'Preflight accepted an undeclared missing namespace' >&2
+  exit 1
+fi
+# Count services, not characters in the newline-separated service names.
+compose() {
+  case "$*" in
+    *'config --format json') printf '%s\n' '{"services":{"headscale":{},"headplane":{},"web":{},"init":{"restart":"no"}}}' ;;
+    *'ps --status running --services') printf '%s\n' headscale headplane web ;;
+    *) return 1 ;;
+  esac
+}
+verify_compose_stack example.yaml >"$scratch/compose-count.log"
+grep -qF 'all 3 service(s) running' "$scratch/compose-count.log"
+compose() {
+  case "$*" in
+    *'config --format json') printf '%s\n' '{"services":{"headscale":{},"headplane":{},"web":{}}}' ;;
+    *'ps --status running --services') printf '%s\n' headscale headplane ;;
+    *) return 0 ;;
+  esac
+}
+if verify_compose_stack example.yaml >"$scratch/compose-missing.log"; then
+  echo 'Compose verification accepted a missing service' >&2
+  exit 1
+fi
+grep -qF 'NOT RUNNING: web' "$scratch/compose-missing.log"
 printf '%s\n' 'Deploy validation regressions passed.'

.gitea/workflows/compose-release.py

@@ -42,15 +42,50 @@ def prepare(source_file):
     images_file = directory / 'compose-images.json'
     locks = json.loads(images_file.read_text()) if images_file.exists() else previous.get('compose-images', {})
     release = json.loads((directory / 'release.json').read_text())
-    before = json.loads(json.dumps(config))
+    state = Path(os.environ.get('HOMELAB_STATE', Path.home() / '.local/state/homelab-deploy'))
+    baseline = state / 'compose-configs' / f'{relative.parent.name}.json'
+    if not baseline.exists() and re.fullmatch(r'[0-9]+-[0-9]+', previous.get('run_id', '')):
+        baseline = state / 'runs' / previous['run_id'] / 'compose' / baseline.name
+    bootstrap = not baseline.exists()
+    if not bootstrap:
+        before = json.loads(baseline.read_text())
+    else:
+        # Bootstrap from the persistent configuration, never from the new source.
+        persistent_file = config_repo / relative
+        if persistent_file.exists():
+            before = json.loads(
+                output(
+                    'docker',
+                    'compose',
+                    '--project-directory',
+                    str(project_dir),
+                    '-f',
+                    str(persistent_file),
+                    'config',
+                    '--format',
+                    'json',
+                    cwd=config_repo,
+                )
+            )
+        elif output('docker', 'ps', '-aq', '--filter', f'label=com.docker.compose.project={project}'):
+            raise ValueError(f'{project}: no previous Compose configuration; restore it before deploy')
+        else:
+            before = {'name': project, 'services': {}}
+    if before['name'] != project:
+        raise ValueError('Compose project name changed; manual migration is required')
     for service, settings in config['services'].items():
         reference = settings.get('image')
+        nextcloud_aio_master = project == 'nextcloud' and service == 'nextcloud-aio-mastercontainer'
         if not reference or settings.get('build'):
             raise ValueError(f'{project}/{service}: Compose deploy requires a published image')
         image_repo = reference.split('@')[0].rsplit('/', 1)
         image_repo[-1] = image_repo[-1].split(':')[0]
         image_repo = '/'.join(image_repo)
-        if image_repo in release['images']:
+        # Nextcloud AIO validates the mastercontainer image reference and rejects
+        # a digest. Keep its configured tag so AIO can start and manage its stack.
+        if nextcloud_aio_master:
+            pinned = reference
+        elif image_repo in release['images']:
             pinned = image_repo + '@' + release['images'][image_repo]
         elif os.environ.get('REFRESH_IMAGES') != 'true' and reference in locks:
             pinned = locks[reference]
@@ -58,6 +93,12 @@ def prepare(source_file):
             pinned = resolve(reference)
         settings['image'] = pinned
         locks[reference] = pinned
+    for service, settings in before['services'].items():
+        reference = settings['image']
+        image_repo = reference.split('@')[0].rsplit('/', 1)
+        image_repo[-1] = image_repo[-1].split(':')[0]
+        image_repo = '/'.join(image_repo)
+        nextcloud_aio_master = project == 'nextcloud' and service == 'nextcloud-aio-mastercontainer'
         # Capture what is running, not the current value of its mutable tag.
         ids = output(
             'docker',
@@ -69,13 +110,43 @@ def prepare(source_file):
             f'label=com.docker.compose.service={service}',
         ).splitlines()
         actual = set()
+        if bootstrap and ids:
+            expected_hash = output(
+                'docker',
+                'compose',
+                '--project-directory',
+                str(project_dir),
+                '-f',
+                str(persistent_file),
+                'config',
+                '--hash',
+                service,
+                cwd=config_repo,
+            ).split()[-1]
+            for container in ids:
+                running_hash = output(
+                    'docker',
+                    'inspect',
+                    container,
+                    '--format',
+                    '{{ index .Config.Labels "com.docker.compose.config-hash" }}',
+                )
+                if running_hash != expected_hash:
+                    raise ValueError(
+                        f'{project}/{service}: persistent config differs from running config; restore the previous config'
+                    )
         for container in ids:
             image_id = output('docker', 'inspect', container, '--format', '{{.Image}}')
             digests = json.loads(output('docker', 'image', 'inspect', image_id, '--format', '{{json .RepoDigests}}'))
             actual.add(next((d for d in digests or [] if d.split('@')[0] == image_repo), image_id))
         if len(actual) > 1:
             raise ValueError(f'{project}/{service}: mixed running images, cannot capture one recovery config')
-        before['services'][service]['image'] = next(iter(actual)) if actual else reference
+        # AIO also rejects a digest in its recovery config. Preserve its tag in
+        # both deploy and recovery files.
+        if nextcloud_aio_master:
+            before['services'][service]['image'] = reference
+        else:
+            before['services'][service]['image'] = next(iter(actual)) if actual else reference
     for name, data in (('compose', config), ('compose-before', before)):
         folder = directory / name
         folder.mkdir(mode=0o700, exist_ok=True)
@@ -85,7 +156,7 @@ def prepare(source_file):
     images_file.write_text(json.dumps(locks, indent=2) + '\n')
     print(f'Compose {project}: images pinned; local paths preserved')
     print(
-        f'Recovery: docker compose --project-directory {project_dir} -p {project} -f {directory}/compose-before/{relative.parent.name}.json up -d --pull never'
+        f'Recovery: docker compose --project-directory {project_dir} -p {project} -f {directory}/compose-before/{relative.parent.name}.json up -d --pull never --remove-orphans'
     )
 
 

.gitea/workflows/deploy-controller.py

@@ -141,7 +141,8 @@ def make_plan(directory):
     planner = load_module('deploy_plan', source / '.gitea/workflows/deploy-plan.py')
     request = json.loads((directory / 'request.json').read_text())
     previous = json.loads((STATE / 'last-success.json').read_text()) if (STATE / 'last-success.json').exists() else None
-    helm = json.loads(command('helm', 'list', '--all', '-A', '-o', 'json'))
+    # Helm 4 lists every release status by default and removed the --all flag.
+    helm = json.loads(command('helm', 'list', '-A', '-o', 'json'))
     plan = planner.make_plan(source, CONFIG_REPO, request['release'], previous, request['mode'], helm)
     if request['refresh_images']:
         plan['selected']['compose'] = plan['active']['compose']
@@ -164,6 +165,10 @@ def finish_success(directory, plan):
         if previous.exists()
         else {}
     )
+    configs = STATE / 'compose-configs'
+    configs.mkdir(mode=0o700, exist_ok=True)
+    for config in (directory / 'compose').glob('*.json'):
+        atomic_json(configs / config.name, json.loads(config.read_text()))
     atomic_json(STATE / 'last-success.json', plan)
     status = json.loads((directory / 'status.json').read_text())
     status['state'] = 'success'

.gitea/workflows/deploy-lib.sh

@@ -180,7 +180,8 @@ save_snapshot() {
           | select(any(.metadata.ownerReferences[]?; .uid == $w.metadata.uid))
           | select($w.kind != "StatefulSet" or .metadata.name == $w.status.currentRevision) | .revision] | max // 0) end)
     }]' "$dir/workloads.json" >"$dir/revisions.json" || return 1
-  releases="$(helm list --all -A -o json)" || return 1
+  # Helm 4 lists every release status by default and removed the --all flag.
+  releases="$(helm list -A -o json)" || return 1
   for entry in "${HELM_RELEASES[@]}"; do
     IFS='|' read -r release _ namespace _ _ _ <<<"$entry"
     if ! jq -e --arg r "$release" --arg n "$namespace" \
@@ -570,6 +571,41 @@ skip_uninstalled_vmagent_crd() {
   return 1
 }
 
+# Render one complete resource list so new namespaces can be identified across
+# files and Kustomize apps. A missing undeclared namespace remains an error.
+render_selected_resources() {
+  local m k
+  {
+    for m in "${K8S_MANIFESTS[@]}"; do
+      if skip_uninstalled_vmagent_crd "$m" >/dev/null; then continue; fi
+      kubectl create --dry-run=client --validate=false -f "$m" -o json || return 1
+    done
+    for k in "${KUSTOMIZE_APPS[@]}"; do
+      kubectl kustomize "$k" | kubectl create --dry-run=client --validate=false -f - -o json || return 1
+    done
+  } | jq -s '{apiVersion: "v1", kind: "List", items: [ .[] | if .kind == "List" then .items[] else . end ]}'
+}
+
+validate_server_resources() {
+  local defer_new="$1" resources existing filtered
+  resources="$(render_selected_resources)" || return 1
+  existing="$(kubectl get namespaces -o json)" || return 1
+  filtered="$(jq --argjson existing "$existing" --argjson defer "$defer_new" '
+    [.items[] | select(.kind == "Namespace") | .metadata.name] as $declared
+    | [$existing.items[].metadata.name] as $present
+    | .items |= map(
+        (.metadata.namespace // "default") as $ns
+        | if .kind == "Namespace" or ($present | index($ns)) != null then .
+          elif ($declared | index($ns)) == null then error("Undeclared missing namespace: " + $ns)
+          elif $defer then empty
+          else error("Namespace still missing after namespace apply: " + $ns)
+          end)
+  ' <<<"$resources")" || return 1
+  if [ "$(jq '.items | length' <<<"$filtered")" -gt 0 ]; then
+    kubectl apply --dry-run=server -f - <<<"$filtered" >/dev/null
+  fi
+}
+
 stage_validate() {
   check_prune_mode || return 1
   cd "$REPO"
@@ -596,15 +632,7 @@ stage_validate() {
     kubectl apply -k "$k" --dry-run=client >/dev/null
   done
   log "Validate k8s manifests (kubectl dry-run=server)"
-  for m in ${K8S_MANIFESTS[@]+"${K8S_MANIFESTS[@]}"}; do
+  validate_server_resources true
-    if skip_uninstalled_vmagent_crd "$m"; then
-      continue
-    fi
-    kubectl apply --dry-run=server -f "$m" >/dev/null
-  done
-  for k in ${KUSTOMIZE_APPS[@]+"${KUSTOMIZE_APPS[@]}"}; do
-    kubectl apply -k "$k" --dry-run=server >/dev/null
-  done
   log "Checking referenced Secrets exist"
   echo "  (deploy never applies *secret*.yaml; create missing ones manually)"
   check_referenced_secrets
@@ -656,6 +684,14 @@ stage_apply_k8s() {
       record_apply kubectl "${m#"$REPO"/}" success
     done
   fi
+  # Kustomize may declare namespaces inside its rendered resources too.
+  local namespace_resources
+  namespace_resources="$(render_selected_resources | jq '.items |= map(select(.kind == "Namespace"))')" || return 1
+  if [ "$(jq '.items | length' <<<"$namespace_resources")" -gt 0 ]; then
+    kubectl apply -f - <<<"$namespace_resources" || return 1
+  fi
+  # Complete the deferred server checks before Helm or application resources change.
+  validate_server_resources false || return 1
   if selected_service k8s prometheus-stack && [ -f "$REPO/prometheus-stack/k8s/active" ]; then
     if [ ! -f "$CONFIG_REPO/prometheus-stack/k8s/grafana-values.yaml" ]; then
       echo "ERROR: prometheus-stack/k8s/grafana-values.yaml (gitignored) missing on workstation, restore it first."
@@ -791,12 +827,13 @@ stage_verify_k8s() {
 # actually be running.
 verify_compose_stack() {
   local cf="$1"
-  local expected running missing=()
+  local expected running svc missing=() service_count=0
   expected="$(compose "$cf" config --format json | jq -r ' .services | to_entries[] | select(.value.restart != "no") | .key' | sort)" || return 1
   running="$(compose "$cf" ps --status running --services | sort)" || return 1
   [ -n "$expected" ] || return 0
   while IFS= read -r svc; do
     [ -n "$svc" ] || continue
+    service_count=$((service_count + 1))
     # restart:"no" services are allowed to have exited.
     if ! printf '%s\n' "$running" | grep -qx "$svc"; then
       missing+=("$svc")
@@ -807,7 +844,7 @@ verify_compose_stack() {
     compose "$cf" ps --all 2>/dev/null | sed 's/^/    /' || true
     return 1
   fi
-  echo "  all ${#expected} service(s) running"
+  echo "  all $service_count service(s) running"
   return 0
 }
 

.gitea/workflows/deploy-plan.py

@@ -83,20 +83,20 @@ def make_plan(repo, config_repo, release, previous, mode, live_helm):
         removed = []
     else:
         paths = output('git', '-C', str(repo), 'diff', '--name-only', previous['sha'], release['sha']).splitlines()
-        changed = {path.split('/')[0] for path in paths}
+        changed = {service for service in all_services for path in paths if path.startswith(service + '/')}
         if any(path.startswith('.gitea/') for path in paths):
             changed |= all_services
         changed |= {s for s in all_services if previous.get('local_inputs', {}).get(s) != local_inputs[s]}
         for file in tracked(repo):
-            service = file.split('/')[0]
+            owners = {service for service in all_services if file.startswith(service + '/')}
-            if service not in all_services or not file.endswith(('.yaml', '.yml')):
+            if not owners or not file.endswith(('.yaml', '.yml')):
                 continue
             text = (repo / file).read_text()
             if any(
                 image in text and previous.get('images', {}).get(image) != digest
                 for image, digest in release['images'].items()
             ):
-                changed.add(service)
+                changed |= owners
         removed = sorted(
             set(previous.get('active', {}).get('k8s', []) + previous.get('active', {}).get('compose', []))
             - all_services

.gitea/workflows/deploy.yaml

@@ -80,7 +80,7 @@ jobs:
   apply:
     needs: [gate]
     runs-on: homelab
-    timeout-minutes: 100
+    timeout-minutes: 120
     steps:
       - name: Checkout checked commit
         uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4

.gitea/workflows/release.py

@@ -305,7 +305,6 @@ def build_images(output, report, name, plan):
         if exists:
             print(f'Reuse {name}: inputs unchanged')
             digest = old_digest
-            report['reused'].append(name)
         else:
             print(f'Build {name}', flush=True)
             metadata = Path(docker_config) / 'metadata.json'
@@ -332,14 +331,14 @@ def build_images(output, report, name, plan):
                 env=env,
             )
             digest = json.loads(metadata.read_text())['containerimage.digest']
-            report['built'].append(name)
+        if not isinstance(digest, str) or not DIGEST.fullmatch(digest):
+            raise ValueError('Image job returned an invalid digest')
         release['images'][image] = digest
         release['inputs'][image] = inputs
-        if not DIGEST.fullmatch(digest):
+        report['reused' if exists else 'built'].append(name)
-            raise ValueError('Image job returned an invalid digest')
         output.write_text(json.dumps(release, indent=2) + '\n')
         report['current'] = None
-        report['phase'] = 'Release file saved'
+        report['phase'] = 'Image result file saved'
     finally:
         # Cleanup errors must neither leak credentials nor mask the original build error.
         try:
@@ -395,13 +394,17 @@ def build(output, name, plan):
         result = 'success'
     finally:
         lines = [
-            f'## Image release `{os.environ.get("GITHUB_SHA", "unknown")}`',
+            f'## Image build result `{name}`',
+            '',
+            f'- Commit: `{os.environ.get("GITHUB_SHA", "unknown")}`',
             '',
             f'- Result: **{result}**',
             f'- Last stage: {report["phase"]}',
         ]
         if result == 'failure':
-            lines.append('- No release from this build can be deployed. Open the failed step log.')
+            lines.append('- This image job failed. The complete release cannot be published. Open the failed step log.')
+        if result == 'success':
+            lines.append('- This is one image result. The final build job must publish the complete release.')
         if report['current']:
             lines.append(f'- Image at the failure: `{report["current"]}`')
         for title, key in (('Built', 'built'), ('Reused from successful CI', 'reused')):

tests/ci_test_case.py

@@ -0,0 +1,24 @@
+"""Keep unit-test workflow commands out of the real CI job files."""
+
+import os
+import tempfile
+import unittest
+from pathlib import Path
+from unittest.mock import patch
+
+CI_COMMAND_FILES = ('GITHUB_STEP_SUMMARY', 'GITHUB_OUTPUT', 'GITHUB_ENV', 'GITHUB_PATH', 'GITHUB_STATE')
+
+
+class IsolatedCITestCase(unittest.TestCase):
+    def setUp(self):
+        super().setUp()
+        directory = tempfile.TemporaryDirectory(prefix='homelab-test-ci-')
+        self.addCleanup(directory.cleanup)
+        paths = {}
+        for variable in CI_COMMAND_FILES:
+            path = Path(directory.name) / variable
+            path.touch()
+            paths[variable] = str(path)
+        environment = patch.dict(os.environ, paths)
+        environment.start()
+        self.addCleanup(environment.stop)

tests/test_ci_output_isolation.py

@@ -0,0 +1,60 @@
+"""Run the real unit tests with external CI files and detect leaked writes."""
+
+import os
+import subprocess
+import sys
+import tempfile
+from pathlib import Path
+from unittest.mock import patch
+
+from ci_test_case import CI_COMMAND_FILES, IsolatedCITestCase
+
+
+class CIOutputIsolationTests(IsolatedCITestCase):
+    def test_all_command_files_are_private_and_environment_is_restored(self):
+        with tempfile.TemporaryDirectory() as scratch:
+            external = {variable: str(Path(scratch) / variable) for variable in CI_COMMAND_FILES}
+            for path in external.values():
+                Path(path).write_text('external CI file\n')
+            with patch.dict(os.environ, external):
+                probe = IsolatedCITestCase()
+                probe.setUp()
+                private = []
+                try:
+                    for variable in CI_COMMAND_FILES:
+                        self.assertNotEqual(os.environ[variable], external[variable])
+                        path = Path(os.environ[variable])
+                        private.append(path)
+                        path.write_text('test-only command\n')
+                finally:
+                    probe.doCleanups()
+                for variable in CI_COMMAND_FILES:
+                    self.assertEqual(os.environ[variable], external[variable])
+                    self.assertEqual(Path(external[variable]).read_text(), 'external CI file\n')
+                self.assertTrue(all(not path.exists() for path in private))
+
+    def test_unit_suite_preserves_external_ci_files(self):
+        tests = Path(__file__).resolve().parent
+        modules = sorted(p.stem for p in tests.glob('test_*.py') if p.name != Path(__file__).name)
+        with tempfile.TemporaryDirectory() as scratch:
+            environment = os.environ.copy()
+            environment['PYTHONPATH'] = str(tests) + os.pathsep + environment.get('PYTHONPATH', '')
+            expected = {}
+            for variable in CI_COMMAND_FILES:
+                path = Path(scratch) / variable
+                content = f'external {variable}\n'
+                path.write_text(content)
+                environment[variable] = str(path)
+                expected[path] = content
+            result = subprocess.run(  # noqa: S603 -- Run local test modules with the current Python interpreter.
+                [sys.executable, '-m', 'unittest', *modules, '-q'],
+                cwd=tests.parent,
+                env=environment,
+                capture_output=True,
+                text=True,
+                check=False,
+                timeout=60,
+            )
+            self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
+            for path, content in expected.items():
+                self.assertEqual(path.read_text(), content, f'Unit tests wrote to external {path.name}')

tests/test_cicd.py

@@ -9,6 +9,8 @@ import unittest
 from pathlib import Path
 from unittest.mock import patch
 
+from ci_test_case import IsolatedCITestCase
+
 ROOT = Path(__file__).resolve().parents[1]
 
 
@@ -34,7 +36,7 @@ def release(sha='a' * 40):
     }
 
 
-class ReleaseGateTests(unittest.TestCase):
+class ReleaseGateTests(IsolatedCITestCase):
     def test_release_rejects_wrong_sha_missing_images_and_mutable_tags(self):
         for mutation in ('sha', 'missing', 'tag'):
             data = release()
@@ -91,8 +93,9 @@ class ReleaseGateTests(unittest.TestCase):
                 api.release({'id': 1, 'head_sha': 'a' * 40})
 
 
-class SelectionTests(unittest.TestCase):
+class SelectionTests(IsolatedCITestCase):
     def setUp(self):
+        super().setUp()
         self.scratch = tempfile.TemporaryDirectory()
         self.addCleanup(self.scratch.cleanup)
         self.repo = Path(self.scratch.name)
@@ -128,6 +131,23 @@ class SelectionTests(unittest.TestCase):
         self.assertEqual(result['selected']['k8s'], ['one'])
         self.assertEqual(result['helm'], [])
 
+    def test_nested_service_change_and_owned_image_are_selected(self):
+        directory = self.repo / 'vpn/xui/k8s'
+        directory.mkdir(parents=True)
+        (directory / 'active').touch()
+        image = next(iter(release()['images']))
+        (directory / 'app.yaml').write_text('image: ' + image + ':main\n')
+        baseline_sha = self.commit()
+        baseline = planner.make_plan(self.repo, self.repo, release(baseline_sha), None, 'full', [])
+        (directory / 'app.yaml').write_text('image: ' + image + ':prod\n')
+        result = planner.make_plan(self.repo, self.repo, release(self.commit()), baseline, 'changed', [])
+        self.assertEqual(result['selected']['k8s'], ['vpn/xui'])
+        baseline = result
+        updated = release(result['sha'])
+        updated['images'][image] = 'sha256:' + 'e' * 64
+        result = planner.make_plan(self.repo, self.repo, updated, baseline, 'changed', [])
+        self.assertEqual(result['selected']['k8s'], ['vpn/xui'])
+
     def test_failed_intermediate_deploy_does_not_lose_changes(self):
         (self.repo / 'one/k8s/app.yaml').write_text('kind: StatefulSet\n')
         self.commit()  # This commit failed deploy: baseline must remain initial.
@@ -154,7 +174,7 @@ class SelectionTests(unittest.TestCase):
         self.assertEqual(result['selected']['k8s'], ['one', 'postgres', 'two'])
 
 
-class ComposeConfigurationTests(unittest.TestCase):
+class ComposeConfigurationTests(IsolatedCITestCase):
     def test_pin_preserves_project_volumes_paths_and_previous_image(self):
         with tempfile.TemporaryDirectory() as scratch:
             root = Path(scratch)
@@ -162,7 +182,8 @@ class ComposeConfigurationTests(unittest.TestCase):
             source = run / 'source'
             config_repo = root / 'persistent'
             (source / 'headscale').mkdir(parents=True)
-            config_repo.mkdir()
+            (config_repo / 'headscale').mkdir(parents=True)
+            (config_repo / 'headscale/compose.yaml').touch()
             (run / 'release.json').write_text(json.dumps(release()))
             old = 'busybox@sha256:' + 'd' * 64
             new = 'busybox@sha256:' + 'e' * 64
@@ -180,19 +201,41 @@ class ComposeConfigurationTests(unittest.TestCase):
                 'volumes': {'data': {'name': 'headscale_data'}},
             }
 
+            previous_config = json.loads(json.dumps(config))
+            previous_config['services']['app']['command'] = ['old-command']
+            previous_config['services']['app']['environment'] = {'VALUE': 'old'}
+            previous_config['services']['removed'] = {'image': 'busybox:latest'}
+            config['services']['app']['command'] = ['new-command']
+            config['services']['app']['environment'] = {'VALUE': 'new'}
+            config['services']['added'] = {'image': 'busybox:latest'}
+
             def fake_output(*args, **kwargs):
                 if args[:2] == ('docker', 'compose'):
                     self.assertEqual(kwargs['cwd'], config_repo)
                     self.assertIn(str(config_repo / 'headscale'), args)
-                    return json.dumps(config)
+                    if '--hash' in args:
+                        return 'app matching-hash'
+                    return json.dumps(
+                        previous_config if str(config_repo / 'headscale/compose.yaml') in args else config
+                    )
                 if args[:2] == ('docker', 'ps'):
                     return 'container'
                 if args[:2] == ('docker', 'inspect'):
+                    if 'com.docker.compose.config-hash' in args[-1]:
+                        return 'matching-hash'
                     return 'sha256:' + 'f' * 64
                 return json.dumps([old])
 
             with (
-                patch.dict(os.environ, {'CONFIG_REPO': str(config_repo), 'REPO': str(source), 'RUN_DIR': str(run)}),
+                patch.dict(
+                    os.environ,
+                    {
+                        'CONFIG_REPO': str(config_repo),
+                        'REPO': str(source),
+                        'RUN_DIR': str(run),
+                        'HOMELAB_STATE': str(root / 'state'),
+                    },
+                ),
                 patch.object(compose_module, 'output', side_effect=fake_output),
                 patch.object(compose_module, 'resolve', return_value=new),
             ):
@@ -204,6 +247,57 @@ class ComposeConfigurationTests(unittest.TestCase):
             self.assertEqual(pinned['services']['app']['volumes'], config['services']['app']['volumes'])
             self.assertEqual(pinned['services']['app']['image'], new)
             self.assertEqual(before['services']['app']['image'], old)
+            self.assertEqual(before['services']['app']['command'], ['old-command'])
+            self.assertEqual(before['services']['app']['environment'], {'VALUE': 'old'})
+            self.assertIn('removed', before['services'])
+            self.assertNotIn('added', before['services'])
+
+            def mismatched_output(*args, **kwargs):
+                if args[:2] == ('docker', 'inspect') and 'com.docker.compose.config-hash' in args[-1]:
+                    return 'different-hash'
+                return fake_output(*args, **kwargs)
+
+            with (
+                patch.dict(
+                    os.environ,
+                    {
+                        'CONFIG_REPO': str(config_repo),
+                        'REPO': str(source),
+                        'RUN_DIR': str(run),
+                        'HOMELAB_STATE': str(root / 'state'),
+                    },
+                ),
+                patch.object(compose_module, 'output', side_effect=mismatched_output),
+                patch.object(compose_module, 'resolve', return_value=new),
+                self.assertRaisesRegex(ValueError, 'differs from running config'),
+            ):
+                compose_module.prepare(source / 'headscale/compose.yaml')
+            state = root / 'state'
+            with patch.object(controller, 'STATE', state):
+                state.mkdir()
+                (run / 'status.json').write_text('{"state": "running", "stages": {}}')
+                with patch.object(controller, 'retain_completed'):
+                    controller.finish_success(run, {})
+            self.assertEqual(json.loads((state / 'compose-configs/headscale.json').read_text()), pinned)
+            # A stale persistent checkout must not replace the successful baseline.
+            with (
+                patch.dict(
+                    os.environ,
+                    {
+                        'CONFIG_REPO': str(config_repo),
+                        'REPO': str(source),
+                        'RUN_DIR': str(run),
+                        'HOMELAB_STATE': str(state),
+                    },
+                ),
+                patch.object(compose_module, 'output', side_effect=fake_output),
+                patch.object(compose_module, 'resolve', return_value=new),
+            ):
+                compose_module.prepare(source / 'headscale/compose.yaml')
+            before = json.loads((run / 'compose-before/headscale.json').read_text())
+            self.assertEqual(before['services']['app']['command'], ['new-command'])
+            self.assertIn('added', before['services'])
+            self.assertNotIn('removed', before['services'])
             self.assertEqual((run / 'compose/headscale.json').stat().st_mode & 0o777, 0o600)
 
     def test_registry_index_and_single_image_descriptors(self):
@@ -214,7 +308,7 @@ class ComposeConfigurationTests(unittest.TestCase):
                 )
 
 
-class ControllerTests(unittest.TestCase):
+class ControllerTests(IsolatedCITestCase):
     def test_completed_stage_cannot_apply_again(self):
         with tempfile.TemporaryDirectory() as scratch:
             directory = Path(scratch)

tests/test_cicd_lifecycle.py

@@ -10,10 +10,11 @@ import zipfile
 from pathlib import Path
 from unittest.mock import Mock, patch
 
+from ci_test_case import IsolatedCITestCase
 from test_cicd import ROOT, controller, release, release_module
 
 
-class ArtifactTests(unittest.TestCase):
+class ArtifactTests(IsolatedCITestCase):
     def test_archive_rejects_nested_or_extra_files(self):
         api = object.__new__(release_module.Gitea)
         api.base = 'https://example.test/api/v1/repos/a/b'
@@ -103,7 +104,7 @@ class ArtifactTests(unittest.TestCase):
             self.assertEqual(json.loads((root / 'error-pages.json').read_text())['sha'], 'e' * 40)
 
 
-class DurableRunTests(unittest.TestCase):
+class DurableRunTests(IsolatedCITestCase):
     def test_duplicate_start_only_reattaches(self):
         with tempfile.TemporaryDirectory() as scratch:
             state = Path(scratch)
@@ -203,7 +204,7 @@ class DurableRunTests(unittest.TestCase):
             self.assertEqual(json.loads((directory / 'status.json').read_text())['state'], 'failure')
 
 
-class FailureSummaryTests(unittest.TestCase):
+class FailureSummaryTests(IsolatedCITestCase):
     def test_build_failure_keeps_progress_and_does_not_expose_exception_text(self):
         with tempfile.TemporaryDirectory() as scratch:
             summary = Path(scratch) / 'summary.md'
@@ -225,6 +226,77 @@ class FailureSummaryTests(unittest.TestCase):
             self.assertIn('xdfnx-homepage', content)
             self.assertNotIn('private value', content)
 
+    def test_invalid_digest_is_not_reported_as_a_completed_image(self):
+        for digest in ('invalid-private-metadata', None, ['invalid']):
+            with self.subTest(digest=digest), tempfile.TemporaryDirectory() as scratch:
+                root = Path(scratch)
+                summary = root / 'summary.md'
+                name = 'error-pages'
+                context, dockerfile = release_module.IMAGES[name]
+                plan = {
+                    'sha': 'a' * 40,
+                    'targets': [
+                        {
+                            'name': name,
+                            'context': context,
+                            'dockerfile': dockerfile,
+                            'inputs': 'c' * 64,
+                            'reuse_digest': None,
+                        }
+                    ],
+                }
+
+                def fake_command(*args, digest=digest, **_kwargs):
+                    if args[:3] == ('docker', 'buildx', 'build'):
+                        Path(args[args.index('--metadata-file') + 1]).write_text(
+                            json.dumps({'containerimage.digest': digest})
+                        )
+                    return ''
+
+                with (
+                    patch.dict(
+                        os.environ,
+                        {
+                            'GITHUB_STEP_SUMMARY': str(summary),
+                            'GITHUB_SHA': 'a' * 40,
+                            'REGISTRY_USERNAME': 'test',
+                            'REGISTRY_PASSWORD': 'placeholder',
+                        },
+                    ),
+                    patch.object(release_module, 'checked_plan', return_value=plan),
+                    patch.object(release_module.Path, 'home', return_value=root),
+                    patch.object(release_module, 'command', side_effect=fake_command),
+                    patch.object(subprocess, 'run', return_value=subprocess.CompletedProcess([], 0)),
+                    self.assertRaisesRegex(ValueError, 'invalid digest'),
+                ):
+                    release_module.build(root / 'image.json', name, root / 'plan.json')
+                self.assertFalse((root / 'image.json').exists())
+                content = summary.read_text()
+                self.assertIn('**failure**', content)
+                self.assertIn('### Built\n- None', content)
+                self.assertIn('### Completed image digests\n- None', content)
+                self.assertNotIn('invalid-private-metadata', content)
+
+    def test_successful_image_result_does_not_claim_complete_release(self):
+        def complete_image(_output, report, _name, _plan):
+            report.update(phase='Image result file saved', built=['error-pages'])
+            report['images']['gcr.forust.xyz/forust/error-pages'] = 'sha256:' + 'b' * 64
+
+        with (
+            patch.dict(os.environ, {'GITHUB_SHA': 'a' * 40}),
+            patch.object(
+                release_module,
+                'build_images',
+                side_effect=complete_image,
+            ),
+        ):
+            release_module.build(Path('unused.json'), 'error-pages', Path('unused-plan.json'))
+        content = Path(os.environ['GITHUB_STEP_SUMMARY']).read_text()
+        self.assertIn('## Image build result `error-pages`', content)
+        self.assertIn('Commit: `' + 'a' * 40 + '`', content)
+        self.assertIn('final build job must publish the complete release', content)
+        self.assertNotIn('## Image release', content)
+
     def test_deploy_failure_reports_completed_apply_and_rollback_result(self):
         with tempfile.TemporaryDirectory() as scratch:
             state = Path(scratch)
@@ -261,7 +333,7 @@ class FailureSummaryTests(unittest.TestCase):
             self.assertIn('Compose requires manual recovery', content)
 
 
-class InstallerTests(unittest.TestCase):
+class InstallerTests(IsolatedCITestCase):
     def test_version_comparison_is_exact_without_network_or_host_packages(self):
         with tempfile.TemporaryDirectory() as scratch:
             root = Path(scratch)

tests/test_image_matrix.py

@@ -3,10 +3,10 @@
 import json
 import os
 import tempfile
-import unittest
 from pathlib import Path
 from unittest.mock import Mock, call, patch
 
+from ci_test_case import IsolatedCITestCase
 from test_cicd import release, release_module
 
 
@@ -26,7 +26,7 @@ def plan_data(changed):
     return {'sha': 'a' * 40, 'targets': targets}
 
 
-class MatrixTests(unittest.TestCase):
+class MatrixTests(IsolatedCITestCase):
     def test_no_change_one_image_all_images_and_missing_baseline(self):
         for changed in (set(), {'error-pages'}, set(release_module.IMAGES)):
             with self.subTest(changed=changed), tempfile.TemporaryDirectory() as scratch:

tests/test_netbird_runtime.py

@@ -7,11 +7,14 @@ import tempfile
 import unittest
 from pathlib import Path
 
+from ci_test_case import IsolatedCITestCase
+
 ROOT = Path(__file__).resolve().parents[1]
 
 
-class NetbirdRuntimeTests(unittest.TestCase):
+class NetbirdRuntimeTests(IsolatedCITestCase):
     def setUp(self):
+        super().setUp()
         self.temp = tempfile.TemporaryDirectory()
         self.addCleanup(self.temp.cleanup)
         self.root = Path(self.temp.name)