homelab/docs/repository-review.md

Repository review

Reviewed the tracked tree at cc9c3de and read the live workstation state on 6 October 2026. Changes are split into documentation and individual fix branches, all based on that main commit. The original local checkout and its uncommitted monitoring changes were preserved. No deployment was performed.

Confirmed problems with prepared fixes

Priority	Problem and consequence	Fix branch
High	APPLY_PRUNE=true is passed to each individual manifest apply. Each invocation sees only that file's desired objects and can delete other resources selected by the shared label.	fix/deploy-prune-guard
High	Deploy validates Compose with interpolation and env/path resolution disabled. Required settings can pass validation and then fail during apply after other workloads have changed.	fix/deploy-validation
Medium	Secret validation is text-based and compares names across all namespaces. A Secret elsewhere can hide a missing local Secret; mounted Secrets are also missed.	fix/deploy-validation
Medium	Compose CI misses postgres/shared-compose.yaml, netbird/client.compose.yaml, and renovate/renovate-compose.yaml.	fix/deploy-validation
Medium	NetBird Compose mounts entrypoint.sh, but it is absent. Its README also calls a missing setup.sh; a fresh checkout cannot start this stack as documented.	fix/netbird-compose-runtime
Medium	Glance's CSS mount uses glance-config, whose keys do not include user.css. That key is in glance-assets; the pod's subPath mount cannot be prepared correctly.	fix/glance-assets
Medium	The shared PostgreSQL initializer requires NETBOX_DB_PASSWORD, but the Compose env example omits it. Following the example leaves first initialization incomplete.	fix/postgres-env-example
Medium	EDU's Compose env example uses old credential names and full URL variables, while the code reads KEEPER_* and paths under EDU_URL_BASE.	fix/session-keeper-reliability
Medium	Session keeper HTTP calls have no timeouts. Its Redis cookie never expires, probes only check existence, and its logs include cookies. A hung or failed refresh can leave a stale session appearing ready.	fix/session-keeper-reliability
Medium	AdGuard's DoH and SearXNG's Compose rules put Boolean expressions inside Host(...). They are invalid router expressions despite valid YAML.	fix/compose-router-rules

Traefik matchers should be combined as Host(a) || Host(b); the rule syntax is described in the Traefik rules documentation. The fix retains the DoH path constraint for both hostnames.

The prune fix deliberately rejects the unsafe option. It does not introduce automatic deletion under a different implementation. Prune defaults to false, and no tracked resource currently carries the selector label, so this is a latent defect rather than evidence of a live deletion incident.

The session fix bounds HTTP and Redis calls, validates required credentials, sets a cookie lifetime of two refresh intervals, and marks success only after publishing the verified cookie. With the default ten-minute interval, an outage longer than twenty minutes will make the existing Redis-key readiness checks fail. That is an intentional change from indefinite apparent readiness.

The deployment fix extracts required pod Secret references from rendered JSON, checks their namespaces, includes init containers, image-pull credentials, and mounted/projected Secrets, and honors optional references. Ingress TLS Secrets issued by cert-manager are not treated as pre-existing pod prerequisites. It checks existence/access, not every key's contents or application validity.

Live workstation observations

The SSH alias workstation is reachable. It has one Ready control-plane node, Kubernetes v1.35.4+k0s, and a Docker daemon alongside containerd. At inspection, no pods were Pending or in another non-running, non-completed phase. This is a point-in-time observation, not a complete application health test.

The deployment checkout at /srv/homelab is on main commit 2adf17c, behind the reviewed local commit. It has untracked host configuration and a separate userbot/ directory. It was not reset or cleaned.

Observed difference	Implication
VictoriaMetrics and vmalert are running; the Prometheus StatefulSet has zero replicas.	A monitoring migration is already in progress outside committed main. Deploying the old Helm values can overwrite those settings.
Homarr, Cloudflared, and Reloader are installed without their current Git active markers.	Installed services and marker-selected services are different inventories. Missing markers do not establish that a service is stopped.
Cloudflare DDNS is running in both Docker and Kubernetes.	Confirm which instance should own DNS updates and whether their domain lists overlap before retiring either one. Secret values were not inspected.
Traefik's LoadBalancer exposes port 8080 at 192.168.80.2.	The direct API listener is deployed; its external reachability was not tested.
Default local-path has reclaim policy Delete, while many existing PVs have been changed to Retain.	Current retention is partly live state. Recreating a claim can get a different policy from the old PV.
NetBird, NetBox media/reports/scripts, EDU Redis, Homarr, and VictoriaMetrics have Delete-policy PVs.	Deleting their claims can delete important state. Plan backup and retention changes before namespace cleanup.

The monitoring files already modified in the user's local tree correspond to the live migration. They are excluded from these branches. Reconcile that work before using this review's baseline to deploy monitoring.

Remaining work

These need recovery design or infrastructure decisions rather than a small configuration correction:

• SSH apply retries can replace the rollback baseline. ssh-run.sh retries exit 255, including apply-k8s; every new invocation publishes a fresh snapshot. If the first attempt already changed workloads, the retry snapshots that partial state. Preserve a run-specific original baseline and verify it across retries.
• Rollback can exceed the job budget. Verification is parallel, but rollback_workloads is serial with a five-minute limit per workload. The thirty-minute job budget can expire before recovery finishes. Bound recovery concurrency and account for both phases before choosing a new timeout.
• Snapshot collection is allowed to fail. Generation and workload snapshot errors are warnings; verify can fall back to all workloads. A snapshot failure must not permit unrelated workloads to be selected for automatic undo.
• Rollback uses the previous revision, not the captured revision. rollout undo without an explicit revision cannot guarantee restoration to the snapshot after retries or intervening rollouts. First deployments also have no previous revision.
• Manual deploy dispatch bypasses the CI-success trigger. Either validate the target commit's successful CI run or document manual dispatch as an operator override with its own required checks.
• Direct Traefik API exposure is unauthenticated. The latest local commit explicitly added it for Homarr. Preserve that integration while choosing a cluster-internal authenticated path or a verified network restriction; do not simply disable an integration that is already in use.
• Storage retention and backup are not reproducible as a whole. Defaults and several important PV policies are Delete. There is no repository-wide backup schedule. Existing PVC StorageClass changes require migration rather than an in-place YAML edit.
• MeTube downloads are temporary on Kubernetes. /downloads is a 20 GiB emptyDir. Decide whether pod replacement should discard files or whether it should use persistent storage. Compose uses a host directory instead.
• First-time activation needs a bootstrap path. Deploy validation dry-runs namespaced resources before the apply stage creates namespaces and installs selected charts. On a fresh cluster, missing namespaces and CRDs need separate preparation; activation is not a complete installer.

Validation

Baseline lint checks passed for Python, shell, workflows, YAML, standard Compose files, and Kubernetes resources with available schemas. Kubeconform found 347 resources in 174 files: 201 valid, 146 skipped CRDs, zero invalid resources. That skip count matters: passing schema validation does not validate Traefik rule strings or other controller-specific behavior.

Fix validation covers:

• Compose discovery of manual entry points, rejection of required-variable gaps, namespace-scoped and optional Secret references, and API/render failures.
• NetBird setup idempotence, preservation of existing keys, file permissions, runtime rendering, and rejection of invalid trusted proxy CIDRs.
• Session refresh success and failure paths, timeouts, cookie expiry, log redaction, missing credentials, and nonpositive refresh intervals.
• Correct Glance ConfigMap key selection and PostgreSQL initializer/env alignment.
• YAML and Compose structure for the corrected router rules, compared with the documented Traefik grammar. They were not exercised on the live proxy.
• Prune rejection before any cluster invocation.

All seven fix branches and the documentation branch merged together without conflicts in a disposable validation worktree. The combined tree passed the CI-equivalent local checks, Markdown formatting/lint and link checks, all 35 Compose structure checks, and 11 Python regression tests plus the shell validation regressions. CRD server-side validation and live rollout tests were not run.

Runtime tests use fixtures and mocks, not production credentials. Live checks read workload metadata, storage policies, chart versions, and container state only. They did not read Secret contents or change services.

Reloader follow-up

fix/reloader-integration adds the active marker and opt-in annotations to 28 application Deployments/StatefulSets that consume runtime ConfigMaps or Secrets. It corrects AdGuard's misplaced pod-template annotation. The Helm settings use annotation-based reloads, keep global auto-reload disabled, and ignore Jobs and CronJobs. PostgreSQL workloads are excluded because their credential variables and init scripts are only effective on an empty data directory.

The controller was already running on workstation when inspected. Its live configuration is unchanged by the branch: merge and deploy the integration to apply the new policy and application annotations. Configuration reload behavior was checked against the pinned chart, with Helm rendering and manifest validation; no production configuration was changed to provoke a test restart.