homelab/docs/repository-review.md

# Repository review

Reviewed the tracked tree at `cc9c3de` and read the live workstation state on
6 October 2026. Changes are split into documentation and individual fix branches,
all based on that main commit. The original local checkout and its uncommitted
monitoring changes were preserved. No deployment was performed.

## Confirmed problems with prepared fixes

| Priority | Problem and consequence                                                                                                                                                                                    | Fix branch                       |
| -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- |
| High     | `APPLY_PRUNE=true` is passed to each individual manifest apply. Each invocation sees only that file's desired objects and can delete other resources selected by the shared label.                         | `fix/deploy-prune-guard`         |
| High     | Deploy validates Compose with interpolation and env/path resolution disabled. Required settings can pass validation and then fail during apply after other workloads have changed.                         | `fix/deploy-validation`          |
| Medium   | Secret validation is text-based and compares names across all namespaces. A Secret elsewhere can hide a missing local Secret; mounted Secrets are also missed.                                             | `fix/deploy-validation`          |
| Medium   | Compose CI misses `postgres/shared-compose.yaml`, `netbird/client.compose.yaml`, and `renovate/renovate-compose.yaml`.                                                                                     | `fix/deploy-validation`          |
| Medium   | NetBird Compose mounts `entrypoint.sh`, but it is absent. Its README also calls a missing `setup.sh`; a fresh checkout cannot start this stack as documented.                                              | `fix/netbird-compose-runtime`    |
| Medium   | Glance's CSS mount uses `glance-config`, whose keys do not include `user.css`. That key is in `glance-assets`; the pod's subPath mount cannot be prepared correctly.                                       | `fix/glance-assets`              |
| Medium   | The shared PostgreSQL initializer requires `NETBOX_DB_PASSWORD`, but the Compose env example omits it. Following the example leaves first initialization incomplete.                                       | `fix/postgres-env-example`       |
| Medium   | EDU's Compose env example uses old credential names and full URL variables, while the code reads `KEEPER_*` and paths under `EDU_URL_BASE`.                                                                | `fix/session-keeper-reliability` |
| Medium   | Session keeper HTTP calls have no timeouts. Its Redis cookie never expires, probes only check existence, and its logs include cookies. A hung or failed refresh can leave a stale session appearing ready. | `fix/session-keeper-reliability` |
| Medium   | AdGuard's DoH and SearXNG's Compose rules put Boolean expressions inside `Host(...)`. They are invalid router expressions despite valid YAML.                                                              | `fix/compose-router-rules`       |

Traefik matchers should be combined as `Host(a) || Host(b)`; the rule syntax is
described in the [Traefik rules documentation](https://doc.traefik.io/traefik/reference/routing-configuration/http/routing/rules-and-priority/).
The fix retains the DoH path constraint for both hostnames.

The prune fix deliberately rejects the unsafe option. It does not introduce
automatic deletion under a different implementation. Prune defaults to false,
and no tracked resource currently carries the selector label, so this is a
latent defect rather than evidence of a live deletion incident.

The session fix bounds HTTP and Redis calls, validates required credentials,
sets a cookie lifetime of two refresh intervals, and marks success only after
publishing the verified cookie. With the default ten-minute interval, an outage
longer than twenty minutes will make the existing Redis-key readiness checks fail.
That is an intentional change from indefinite apparent readiness.

The deployment fix extracts required pod Secret references from rendered JSON,
checks their namespaces, includes init containers, image-pull credentials, and
mounted/projected Secrets, and honors optional references. Ingress TLS Secrets
issued by cert-manager are not treated as pre-existing pod prerequisites.
It checks existence/access, not every key's contents or application validity.

## Live workstation observations

The SSH alias `workstation` is reachable. It has one Ready control-plane node,
Kubernetes `v1.35.4+k0s`, and a Docker daemon alongside containerd. At inspection,
no pods were Pending or in another non-running, non-completed phase. This is a
point-in-time observation, not a complete application health test.

The deployment checkout at `/srv/homelab` is on main commit `2adf17c`, behind the
reviewed local commit. It has untracked host configuration and a separate
`userbot/` directory. It was not reset or cleaned.

| Observed difference                                                                                   | Implication                                                                                                                                        |
| ----------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| VictoriaMetrics and vmalert are running; the Prometheus StatefulSet has zero replicas.                | A monitoring migration is already in progress outside committed main. Deploying the old Helm values can overwrite those settings.                  |
| Homarr, Cloudflared, and Reloader are installed without their current Git active markers.             | Installed services and marker-selected services are different inventories. Missing markers do not establish that a service is stopped.             |
| Cloudflare DDNS is running in both Docker and Kubernetes.                                             | Confirm which instance should own DNS updates and whether their domain lists overlap before retiring either one. Secret values were not inspected. |
| Traefik's LoadBalancer exposes port 8080 at `192.168.80.2`.                                           | The direct API listener is deployed; its external reachability was not tested.                                                                     |
| Default `local-path` has reclaim policy Delete, while many existing PVs have been changed to Retain.  | Current retention is partly live state. Recreating a claim can get a different policy from the old PV.                                             |
| NetBird, NetBox media/reports/scripts, EDU Redis, Homarr, and VictoriaMetrics have Delete-policy PVs. | Deleting their claims can delete important state. Plan backup and retention changes before namespace cleanup.                                      |

The monitoring files already modified in the user's local tree correspond to the
live migration. They are excluded from these branches. Reconcile that work before
using this review's baseline to deploy monitoring.

## Remaining work

These need recovery design or infrastructure decisions rather than a small
configuration correction:

- **SSH apply retries can replace the rollback baseline.** `ssh-run.sh` retries
  exit 255, including `apply-k8s`; every new invocation publishes a fresh snapshot.
  If the first attempt already changed workloads, the retry snapshots that partial
  state. Preserve a run-specific original baseline and verify it across retries.
- **Rollback can exceed the job budget.** Verification is parallel, but
  `rollback_workloads` is serial with a five-minute limit per workload. The
  thirty-minute job budget can expire before recovery finishes. Bound recovery
  concurrency and account for both phases before choosing a new timeout.
- **Snapshot collection is allowed to fail.** Generation and workload snapshot
  errors are warnings; verify can fall back to all workloads. A snapshot failure
  must not permit unrelated workloads to be selected for automatic undo.
- **Rollback uses the previous revision, not the captured revision.** `rollout undo`
  without an explicit revision cannot guarantee restoration to the snapshot after
  retries or intervening rollouts. First deployments also have no previous revision.
- **Manual deploy dispatch bypasses the CI-success trigger.** Either validate the
  target commit's successful CI run or document manual dispatch as an operator
  override with its own required checks.
- **Direct Traefik API exposure is unauthenticated.** The latest local commit
  explicitly added it for Homarr. Preserve that integration while choosing a
  cluster-internal authenticated path or a verified network restriction; do not
  simply disable an integration that is already in use.
- **Storage retention and backup are not reproducible as a whole.** Defaults and
  several important PV policies are Delete. There is no repository-wide backup
  schedule. Existing PVC StorageClass changes require migration rather than an
  in-place YAML edit.
- **MeTube downloads are temporary on Kubernetes.** `/downloads` is a 20 GiB
  emptyDir. Decide whether pod replacement should discard files or whether it
  should use persistent storage. Compose uses a host directory instead.
- **First-time activation needs a bootstrap path.** Deploy validation dry-runs
  namespaced resources before the apply stage creates namespaces and installs
  selected charts. On a fresh cluster, missing namespaces and CRDs need separate
  preparation; activation is not a complete installer.

## Validation

Baseline lint checks passed for Python, shell, workflows, YAML, standard Compose
files, and Kubernetes resources with available schemas. Kubeconform found 347
resources in 174 files: 201 valid, 146 skipped CRDs, zero invalid resources.
That skip count matters: passing schema validation does not validate Traefik rule
strings or other controller-specific behavior.

Fix validation covers:

- Compose discovery of manual entry points, rejection of required-variable gaps,
  namespace-scoped and optional Secret references, and API/render failures.
- NetBird setup idempotence, preservation of existing keys, file permissions,
  runtime rendering, and rejection of invalid trusted proxy CIDRs.
- Session refresh success and failure paths, timeouts, cookie expiry, log redaction,
  missing credentials, and nonpositive refresh intervals.
- Correct Glance ConfigMap key selection and PostgreSQL initializer/env alignment.
- YAML and Compose structure for the corrected router rules, compared with the
  documented Traefik grammar. They were not exercised on the live proxy.
- Prune rejection before any cluster invocation.

All seven fix branches and the documentation branch merged together without
conflicts in a disposable validation worktree. The combined tree passed the
CI-equivalent local checks, Markdown formatting/lint and link checks, all 35
Compose structure checks, and 11 Python regression tests plus the shell
validation regressions. CRD server-side validation and live rollout tests were
not run.

Runtime tests use fixtures and mocks, not production credentials. Live checks read
workload metadata, storage policies, chart versions, and container state only.
They did not read Secret contents or change services.

## Reloader follow-up

`fix/reloader-integration` adds the active marker and opt-in annotations to 28
application Deployments/StatefulSets that consume runtime ConfigMaps or Secrets.
It corrects AdGuard's misplaced pod-template annotation. The Helm settings use
annotation-based reloads, keep global auto-reload disabled, and ignore Jobs and
CronJobs. PostgreSQL workloads are excluded because their credential variables
and init scripts are only effective on an empty data directory.

The controller was already running on workstation when inspected. Its live
configuration is unchanged by the branch: merge and deploy the integration to
apply the new policy and application annotations. Configuration reload behavior
was checked against the pinned chart, with Helm rendering and manifest validation;
no production configuration was changed to provoke a test restart.