From f767f3ce1a98d6117346fc0951c7985c62ab4a42 Mon Sep 17 00:00:00 2001 From: mr-forust Date: Wed, 7 Oct 2026 10:25:17 +0200 Subject: [PATCH] docs: update EDU handoff status --- .gitea/EDU_HANDOFF.md | 59 +++++++++++++++++-------------------------- 1 file changed, 23 insertions(+), 36 deletions(-) diff --git a/.gitea/EDU_HANDOFF.md b/.gitea/EDU_HANDOFF.md index 1571ea1..01439e5 100644 --- a/.gitea/EDU_HANDOFF.md +++ b/.gitea/EDU_HANDOFF.md @@ -1,55 +1,42 @@ # EDU ownership handoff -## Review findings +## Current status -The current `main` branch can verify and roll back changed workloads across the cluster. This is unsafe when an external repository owns an application. Open PR #99 already changes this behavior to use selected workload references and immutable releases. This change is based on PR #99 branch `codex/ci-visible-checks` and keeps its protected check names: +EDU PR #1 merged at 2026-10-07 08:04:30 UTC. Main release `5094952464ce315130839303985fd04d721bc1f2` passed CI run 1585 and deploy run 1586. The workstation checkout `/srv/edu-master` is at that SHA. The release changed the application image digests: -- `ci / Compose*` -- `ci / Workflows*` -- `ci / Shell*` -- `ci / Formatting*` -- `ci / Python and tests*` -- `ci / YAML*` -- `ci / Dockerfiles*` -- `ci / Kubernetes*` +- Session keeper: `sha256:1e59473bd40fe4c22622017d808a8927a68788275fe073dc23d718c44b2fd5dd` +- Webinar checker: `sha256:987d9bf0770272766523ea5b94c7f3f849175d737551d46591ae55e058cf9f12` -Open PR #100 adds service metrics. It is independent and is not required for this handoff. +The live workloads remain healthy in context `Default`, namespace `edu-master`. Both health and live probes return 200. Redis AUTH passes, session TTL is 1178 seconds, the delivery backlog is zero, all nine EDU alert rules are healthy, and the scrape target is UP. The unauthorized-pod Redis check passed. The Redis PVC UID and Secret UID and values, including the Fernet key, match their pre-release state. -Live Gitea 28.0.0 supports dynamic job outputs, matrices, and `max-parallel`. The runner has 1 CPU, 1.7 GiB RAM, and 6 GiB free disk. Its capacity details could not be verified because the diagnostic required a sudo password. Runner concurrency capacity remains unverified. +The homelab EDU active marker was present after the EDU deployment. It was moved to the private snapshot as `homelab-k8s-active.marker` while holding `/tmp/homelab-apply.lock`. The homelab checkout at `/srv/homelab` is at `5f9354b` and has the tracked marker deletion. Its deploy preflight blocks a dirty checkout until this removal is reconciled. Preserve private ignored configuration when syncing that checkout. -The workstation checkout `/srv/edu-master` exists, is clean at `7ed537f`, and uses the SSH remote `gitssh.forust.xyz:2221`. In Kubernetes context `Default`, namespace `edu-master`, the `session-keeper` and `webinar-checker` workloads are Ready. Their health and live endpoints return 200. Their running image digests match the digests in the old homelab configuration. The Redis PVC `redis-data-pvc` is bound to PV `pvc-a4f2a79a-363a-4c12-ae91-92cdfc2a0d2e`; its reclaim policy is `Delete`. Never delete, recreate, or apply this PVC. +The remaining homelab change is PR #105, branch `feat/edu-handoff-matrix`, based on `codex/ci-visible-checks`. Its eight protected checks passed. Renovate runs 1587 and 1588 passed. Image publishing was skipped for the PR. The EDU runtime changes are in PR #3 from `fix/handoff-runtime` to `main`; CI run 1589 is in progress. Those runtime changes have not been released. -The EDU repository still needs its live `playwright-service` manifest and reconciled auto-reloader annotations. Those changes, plus backup and monitoring verification improvements, are in the separate EDU branch `fix/handoff-runtime`. +## Approval gate and next steps -## Implementation decisions +PR #99 must merge before PR #105 can target `main`. A merge attempt for PR #99 returned HTTP 405 because it needs one approval; the protected branch has `required_approvals=1` and whitelist approval is enabled. This approval gate prevents the remaining transfer steps. -The homelab change removes the complete `edu_master` subtree, EDU build and deploy selection, image pinning, rollback and verification cases, route probes, related tests, and Renovate references. It removes the external EDU image bypass from generic image scans. Application source and release ownership: [forust/edu-master](https://git.forust.xyz/forust/edu-master). +After the required approval: -The image plan compares fingerprints for all three homelab images with the last successful CI release. Each matrix job builds its image or reuses the matching immutable digest. The matrix runs one job at a time and continues after a job failure so each image has a visible result. The final build job waits for all image jobs, checks the commit, inputs, and digests, applies the full-SHA tags, and writes the existing release artifact format. This preserves the deploy gate. A manifest-only change still runs three reuse jobs and the final tag stage. Pull requests do not publish images. +1. Merge PR #99. +2. Retarget PR #105 to `main`. Complete CI and review, then approve and merge it. +3. Under the homelab apply lock, sync `/srv/homelab` to the merged removal. Preserve private ignored configuration and keep the active marker removed. Confirm the deploy preflight is clean. +4. Merge the EDU runtime PR after its CI and review pass. The main-push CI run must complete successfully before its exact SHA can deploy. +5. Verify the new release SHA, image digests, workload health, Redis AUTH and TTL, backlog, PVC and Secret identity, and monitoring. Record the results in the EDU PR. -Retain the protected check names from PR #99. Keep CI and deploy separate. `AUTODEPLOY=false` is configured explicitly. The EDU main-push deploy policy is independent of this setting. +`AUTODEPLOY=false` is explicitly configured. The EDU repository path and port secrets are confirmed, and `EDU_KUBE_CONTEXT=Default` is configured as a repository variable. Keep deployment and registry credentials outside Git. Never run both homelab and EDU deployment paths at the same time. -## Dependencies and rollout order +## Change summary -1. Merge PR #99 first, because this change uses its selected-workload and immutable-release behavior. PR #100 is not a dependency. -2. Complete the EDU runtime reconciliation on the EDU feature branch. Do not merge EDU into `main` yet. Configure and verify the EDU deployment secrets and trusted SSH host key outside Git. -3. Confirm the EDU release can pass its CI and immutable-SHA deploy gate. Keep the existing namespace, Secret, Redis data, Fernet key, and runtime credentials. -4. Stop or drain pending homelab runs that can deploy EDU. Remove the EDU active marker from the authoritative homelab source before any later homelab deploy. Confirm that the removal path does not prune resources. -5. Merge the homelab removal. Then merge EDU PR #1 into `main` to start a successful main-push release and deployment. -6. Verify that homelab no longer selects EDU and that EDU is the sole owner. Check rollout health, `/health`, `/live`, Redis AUTH, session TTL, delivery backlog, and monitoring. Record the release SHA, image digests, downtime, and any remaining limits in the relevant PR. +The homelab PR removes the EDU subtree, deployment and image selection, rollback and verification cases, route probes, Renovate references, and external-image exceptions. It adds a serial dynamic matrix for the three homelab images. Each job builds an image or reuses a matching immutable digest. The final job checks all image results and publishes full-SHA tags and the existing release artifact only after they pass. PRs do not publish images. The protected check names from PR #99 are preserved. PR #100's service-metrics work is independent of this handoff. -## Rollback +The EDU runtime PR adds the Playwright service manifest, reconciles Redis storage and Secret reload annotations, and adds pre-apply Redis backup and identity checks. It verifies application endpoints, Redis AUTH, session TTL, metrics, and all nine vmalert rules. Rollback checks workload and application health and reports when manual recovery is needed. Its deployment guard rejects an unexpected or dirty checkout and refuses deployment while either legacy homelab EDU marker exists. -For an EDU release failure, use the EDU release snapshot and reapply the last recorded immutable digests. Inspect workload and application health after recovery. Do not restore old Redis data unless recovery requires it. +## Rollback and limits -To reverse the ownership handoff, stop EDU deployment triggers and runs first. Restore the reviewed homelab configuration and active marker only after EDU is inactive. Reapply the recorded image digests and verify workload health. Never enable both deployment paths at the same time. The PVC and its PV must remain intact. +The private snapshot is `/home/forust/.local/state/edu-master-deploy/handoff-20261007T080838Z` on the workstation. It contains the pre-handoff Redis RDB and recovery data. RDB checksum verification confirmed twelve keys. Keep the snapshot outside Git. For an EDU release failure, restore the saved Kubernetes resources and inspect application health. The rollback does not automatically restore the Redis RDB; restore old Redis data only when recovery requires it. -## Verification status +For an ownership rollback, stop EDU deployment triggers first, restore the reviewed homelab source and marker, then reapply recorded immutable image digests. Verify both workload and application health. Never delete or recreate the Redis PVC. -Verified: Gitea version and matrix support; current runner CPU, memory, and free disk readings; clean EDU checkout and SSH remote; Kubernetes context and namespace; workload readiness and health endpoints; matching live image digests; and Redis PVC binding and reclaim policy. - -Not verified: runner capacity limits, merged PR state, post-merge image release, EDU deployment, or final handoff acceptance. The merge and live deployment steps remain pending. Do not report the handoff as complete until the live checks above pass. - -Live pre-handoff checks also passed: Redis AUTH (`NOAUTH` without credentials and `PONG` with them), positive session TTL, zero delivery backlog, and nine EDU vmalert rules with matching expressions and healthy evaluation. The private snapshot is `/home/forust/.local/state/edu-master-deploy/handoff-20261007T080838Z` on workstation. It includes the Redis RDB, Secret, manifests, source files, and checkout commits. Workloads have not been redeployed. - -The Redis RDB checksum passed with twelve keys. The unauthorized Redis pod test passed and the pod was removed. VictoriaMetrics reported the EDU scrape target UP. EDU Actions has the dedicated path and port secrets and `EDU_KUBE_CONTEXT=Default`. Existing deployment and registry credentials were retained. +The initial EDU release and the homelab marker move are complete. PR #99 approval and merge, PR #105 retarget and merge, homelab checkout reconciliation, EDU runtime PR merge, and release of those runtime changes remain pending. Synthetic Telegram delivery and Alertmanager-to-Telegram notification were not tested.