docs: document homelab services, deployment, and repository review
This commit is contained in:
forust committed
1 parent
cc9c3dea88
commit
3c4732ae20
44 files changed
+1354
-298
No files matched your search
@@ -0,0 +1,136 @@
|
||||
# Repository review
|
||||
|
||||
Reviewed the tracked tree at `cc9c3de` and read the live workstation state on
|
||||
6 October 2026. Changes are split into documentation and individual fix branches,
|
||||
all based on that main commit. The original local checkout and its uncommitted
|
||||
monitoring changes were preserved. No deployment was performed.
|
||||
|
||||
## Confirmed problems with prepared fixes
|
||||
|
||||
| Priority | Problem and consequence | Fix branch |
|
||||
| -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- |
|
||||
| High | `APPLY_PRUNE=true` is passed to each individual manifest apply. Each invocation sees only that file's desired objects and can delete other resources selected by the shared label. | `fix/deploy-prune-guard` |
|
||||
| High | Deploy validates Compose with interpolation and env/path resolution disabled. Required settings can pass validation and then fail during apply after other workloads have changed. | `fix/deploy-validation` |
|
||||
| Medium | Secret validation is text-based and compares names across all namespaces. A Secret elsewhere can hide a missing local Secret; mounted Secrets are also missed. | `fix/deploy-validation` |
|
||||
| Medium | Compose CI misses `postgres/shared-compose.yaml`, `netbird/client.compose.yaml`, and `renovate/renovate-compose.yaml`. | `fix/deploy-validation` |
|
||||
| Medium | NetBird Compose mounts `entrypoint.sh`, but it is absent. Its README also calls a missing `setup.sh`; a fresh checkout cannot start this stack as documented. | `fix/netbird-compose-runtime` |
|
||||
| Medium | Glance's CSS mount uses `glance-config`, whose keys do not include `user.css`. That key is in `glance-assets`; the pod's subPath mount cannot be prepared correctly. | `fix/glance-assets` |
|
||||
| Medium | The shared PostgreSQL initializer requires `NETBOX_DB_PASSWORD`, but the Compose env example omits it. Following the example leaves first initialization incomplete. | `fix/postgres-env-example` |
|
||||
| Medium | EDU's Compose env example uses old credential names and full URL variables, while the code reads `KEEPER_*` and paths under `EDU_URL_BASE`. | `fix/session-keeper-reliability` |
|
||||
| Medium | Session keeper HTTP calls have no timeouts. Its Redis cookie never expires, probes only check existence, and its logs include cookies. A hung or failed refresh can leave a stale session appearing ready. | `fix/session-keeper-reliability` |
|
||||
| Medium | AdGuard's DoH and SearXNG's Compose rules put Boolean expressions inside `Host(...)`. They are invalid router expressions despite valid YAML. | `fix/compose-router-rules` |
|
||||
|
||||
Traefik matchers should be combined as `Host(a) || Host(b)`; the rule syntax is
|
||||
described in the [Traefik rules documentation](https://doc.traefik.io/traefik/reference/routing-configuration/http/routing/rules-and-priority/).
|
||||
The fix retains the DoH path constraint for both hostnames.
|
||||
|
||||
The prune fix deliberately rejects the unsafe option. It does not introduce
|
||||
automatic deletion under a different implementation. Prune defaults to false,
|
||||
and no tracked resource currently carries the selector label, so this is a
|
||||
latent defect rather than evidence of a live deletion incident.
|
||||
|
||||
The session fix bounds HTTP and Redis calls, validates required credentials,
|
||||
sets a cookie lifetime of two refresh intervals, and marks success only after
|
||||
publishing the verified cookie. With the default ten-minute interval, an outage
|
||||
longer than twenty minutes will make the existing Redis-key readiness checks fail.
|
||||
That is an intentional change from indefinite apparent readiness.
|
||||
|
||||
The deployment fix extracts required pod Secret references from rendered JSON,
|
||||
checks their namespaces, includes init containers, image-pull credentials, and
|
||||
mounted/projected Secrets, and honors optional references. Ingress TLS Secrets
|
||||
issued by cert-manager are not treated as pre-existing pod prerequisites.
|
||||
It checks existence/access, not every key's contents or application validity.
|
||||
|
||||
## Live workstation observations
|
||||
|
||||
The SSH alias `workstation` is reachable. It has one Ready control-plane node,
|
||||
Kubernetes `v1.35.4+k0s`, and a Docker daemon alongside containerd. At inspection,
|
||||
no pods were Pending or in another non-running, non-completed phase. This is a
|
||||
point-in-time observation, not a complete application health test.
|
||||
|
||||
The deployment checkout at `/srv/homelab` is on main commit `2adf17c`, behind the
|
||||
reviewed local commit. It has untracked host configuration and a separate
|
||||
`userbot/` directory. It was not reset or cleaned.
|
||||
|
||||
| Observed difference | Implication |
|
||||
| ----------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| VictoriaMetrics and vmalert are running; the Prometheus StatefulSet has zero replicas. | A monitoring migration is already in progress outside committed main. Deploying the old Helm values can overwrite those settings. |
|
||||
| Homarr, Cloudflared, and Reloader are installed without their current Git active markers. | Installed services and marker-selected services are different inventories. Missing markers do not establish that a service is stopped. |
|
||||
| Cloudflare DDNS is running in both Docker and Kubernetes. | Confirm which instance should own DNS updates and whether their domain lists overlap before retiring either one. Secret values were not inspected. |
|
||||
| Traefik's LoadBalancer exposes port 8080 at `192.168.80.2`. | The direct API listener is deployed; its external reachability was not tested. |
|
||||
| Default `local-path` has reclaim policy Delete, while many existing PVs have been changed to Retain. | Current retention is partly live state. Recreating a claim can get a different policy from the old PV. |
|
||||
| NetBird, NetBox media/reports/scripts, EDU Redis, Homarr, and VictoriaMetrics have Delete-policy PVs. | Deleting their claims can delete important state. Plan backup and retention changes before namespace cleanup. |
|
||||
|
||||
The monitoring files already modified in the user's local tree correspond to the
|
||||
live migration. They are excluded from these branches. Reconcile that work before
|
||||
using this review's baseline to deploy monitoring.
|
||||
|
||||
## Remaining work
|
||||
|
||||
These need recovery design or infrastructure decisions rather than a small
|
||||
configuration correction:
|
||||
|
||||
- **SSH apply retries can replace the rollback baseline.** `ssh-run.sh` retries
|
||||
exit 255, including `apply-k8s`; every new invocation publishes a fresh snapshot.
|
||||
If the first attempt already changed workloads, the retry snapshots that partial
|
||||
state. Preserve a run-specific original baseline and verify it across retries.
|
||||
- **Rollback can exceed the job budget.** Verification is parallel, but
|
||||
`rollback_workloads` is serial with a five-minute limit per workload. The
|
||||
thirty-minute job budget can expire before recovery finishes. Bound recovery
|
||||
concurrency and account for both phases before choosing a new timeout.
|
||||
- **Snapshot collection is allowed to fail.** Generation and workload snapshot
|
||||
errors are warnings; verify can fall back to all workloads. A snapshot failure
|
||||
must not permit unrelated workloads to be selected for automatic undo.
|
||||
- **Rollback uses the previous revision, not the captured revision.** `rollout undo`
|
||||
without an explicit revision cannot guarantee restoration to the snapshot after
|
||||
retries or intervening rollouts. First deployments also have no previous revision.
|
||||
- **Manual deploy dispatch bypasses the CI-success trigger.** Either validate the
|
||||
target commit's successful CI run or document manual dispatch as an operator
|
||||
override with its own required checks.
|
||||
- **Direct Traefik API exposure is unauthenticated.** The latest local commit
|
||||
explicitly added it for Homarr. Preserve that integration while choosing a
|
||||
cluster-internal authenticated path or a verified network restriction; do not
|
||||
simply disable an integration that is already in use.
|
||||
- **Storage retention and backup are not reproducible as a whole.** Defaults and
|
||||
several important PV policies are Delete. There is no repository-wide backup
|
||||
schedule. Existing PVC StorageClass changes require migration rather than an
|
||||
in-place YAML edit.
|
||||
- **MeTube downloads are temporary on Kubernetes.** `/downloads` is a 20 GiB
|
||||
emptyDir. Decide whether pod replacement should discard files or whether it
|
||||
should use persistent storage. Compose uses a host directory instead.
|
||||
- **First-time activation needs a bootstrap path.** Deploy validation dry-runs
|
||||
namespaced resources before the apply stage creates namespaces and installs
|
||||
selected charts. On a fresh cluster, missing namespaces and CRDs need separate
|
||||
preparation; activation is not a complete installer.
|
||||
|
||||
## Validation
|
||||
|
||||
Baseline lint checks passed for Python, shell, workflows, YAML, standard Compose
|
||||
files, and Kubernetes resources with available schemas. Kubeconform found 347
|
||||
resources in 174 files: 201 valid, 146 skipped CRDs, zero invalid resources.
|
||||
That skip count matters: passing schema validation does not validate Traefik rule
|
||||
strings or other controller-specific behavior.
|
||||
|
||||
Fix validation covers:
|
||||
|
||||
- Compose discovery of manual entry points, rejection of required-variable gaps,
|
||||
namespace-scoped and optional Secret references, and API/render failures.
|
||||
- NetBird setup idempotence, preservation of existing keys, file permissions,
|
||||
runtime rendering, and rejection of invalid trusted proxy CIDRs.
|
||||
- Session refresh success and failure paths, timeouts, cookie expiry, log redaction,
|
||||
missing credentials, and nonpositive refresh intervals.
|
||||
- Correct Glance ConfigMap key selection and PostgreSQL initializer/env alignment.
|
||||
- YAML and Compose structure for the corrected router rules, compared with the
|
||||
documented Traefik grammar. They were not exercised on the live proxy.
|
||||
- Prune rejection before any cluster invocation.
|
||||
|
||||
All seven fix branches and the documentation branch merged together without
|
||||
conflicts in a disposable validation worktree. The combined tree passed the
|
||||
CI-equivalent local checks, Markdown formatting/lint and link checks, all 35
|
||||
Compose structure checks, and 11 Python regression tests plus the shell
|
||||
validation regressions. CRD server-side validation and live rollout tests were
|
||||
not run.
|
||||
|
||||
Runtime tests use fixtures and mocks, not production credentials. Live checks read
|
||||
workload metadata, storage policies, chart versions, and container state only.
|
||||
They did not read Secret contents or change services.
|
||||
Reference in new issue
Block a user