100 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
forust	6057734a4f	fix(renovate): sync generated configmap	2026-10-06 18:08:17 +02:00
forust	8eedf8b74b	Merge pull request 'feat(monitoring): add VictoriaMetrics trial stack' (#95) from feat/victoria-metrics-migration into main ... Reviewed-on: https://gitea.forust.xyz/forust/homelab/pulls/95	2026-10-06 16:05:46 +00:00
forust	8c0e36a5c0	feat(monitoring): replace scrape dump with vmagent	2026-10-06 18:03:59 +02:00
forust	78cd15f12c	chore(monitoring): remove vmctl backfill job ... One-shot Prometheus history backfill is complete; drop the Job and clean up related comments.	2026-10-06 17:53:24 +02:00
forust	18c633c242	feat(monitoring): add VictoriaMetrics trial stack	2026-10-06 17:45:44 +02:00
forust	4510394531	Merge pull request 'chore(deps): update renovate/renovate docker tag to v44.139.0' (#81) from renovate/renovate-self-update into main ... Reviewed-on: https://gitea.forust.xyz/forust/homelab/pulls/81	2026-10-06 15:33:45 +00:00
forust	8ce0b809c0	Merge pull request 'chore(deps): update all minor updates' (#77) from renovate/all-minor into main ... Reviewed-on: https://gitea.forust.xyz/forust/homelab/pulls/77	2026-10-06 15:33:28 +00:00
forust	bdcbb3d5af	Merge pull request 'chore(deps): update all patch updates' (#82) from renovate/all-patch into main ... Reviewed-on: https://gitea.forust.xyz/forust/homelab/pulls/82	2026-10-06 15:33:09 +00:00
forust	695da1308c	Merge pull request 'fix(renovate): restore custom extraction and update groups' (#93) from fix/renovate-extraction-groups into main ... Reviewed-on: https://gitea.forust.xyz/forust/homelab/pulls/93	2026-10-06 15:31:25 +00:00
forust	552b22cb66	fix(renovate): include self-update Compose filename	2026-10-06 15:31:25 +00:00
forust	3756e60c95	fix(renovate): restore custom extraction and update groups	2026-10-06 15:31:25 +00:00
forust	d0d217ddf4	Merge pull request 'fix(deploy): skip verify and smoke when deployment is disabled' (#94) from fix/deploy-skip-when-disabled into main ... Reviewed-on: https://gitea.forust.xyz/forust/homelab/pulls/94	2026-10-06 15:24:17 +00:00
forust	24f84bab2f	fix(deploy): skip verification when deployment is disabled	2026-10-06 15:24:06 +00:00
forust	0b8a3c16a7	Merge pull request 'fix(glance): mount CSS from the correct ConfigMap' (#87) from fix/glance-assets into main ... Reviewed-on: https://gitea.forust.xyz/forust/homelab/pulls/87	2026-10-06 15:17:47 +00:00
forust	e9a68aae77	fix(glance): mount CSS from the assets ConfigMap	2026-10-06 15:17:47 +00:00
forust	5359df5ed6	Merge pull request 'fix(postgres): include the required NetBox database password' (#88) from fix/postgres-env-example into main ... Reviewed-on: https://gitea.forust.xyz/forust/homelab/pulls/88	2026-10-06 15:17:19 +00:00
forust	8aea0f0d13	fix(postgres): include required NetBox password in Compose env example	2026-10-06 15:17:19 +00:00
forust	65d4f2d482	Merge pull request 'fix(traefik): correct AdGuard and SearXNG Compose rules' (#90) from fix/compose-router-rules into main ... Reviewed-on: https://gitea.forust.xyz/forust/homelab/pulls/90	2026-10-06 15:10:49 +00:00
forust	2fe9b7632f	fix(traefik): correct AdGuard and SearXNG Compose router expressions	2026-10-06 15:10:49 +00:00
forust	e436d89eef	Merge pull request 'fix(netbird): restore Compose setup and runtime renderer' (#86) from fix/netbird-compose-runtime into main ... Reviewed-on: https://gitea.forust.xyz/forust/homelab/pulls/86	2026-10-06 15:10:12 +00:00
forust	8729cb5062	fix(netbird): restore Compose setup and server entrypoint	2026-10-06 15:08:46 +00:00
forust	f1e4a1088d	Merge pull request 'fix(ci): deduplicate PR checks and filter deploy triggers' (#92) from fix/ci-trigger-dedup into main ... Reviewed-on: https://gitea.forust.xyz/forust/homelab/pulls/92	2026-10-06 15:06:47 +00:00
forust	b357ef95d8	fix(deploy): only create automatic runs for main CI	2026-10-06 17:00:05 +02:00
forust	67422663b7	fix(ci): avoid duplicate branch and PR runs	2026-10-06 16:59:26 +02:00
forust	88705fec88	Merge pull request 'fix(deploy): reject unsafe per-file pruning' (#85) from fix/deploy-prune-guard into main ... Reviewed-on: https://gitea.forust.xyz/forust/homelab/pulls/85	2026-10-06 14:57:22 +00:00
forust	c098807aa4	fix(deploy): reject destructive per-file pruning before apply	2026-10-06 14:57:11 +00:00
forust	e1eee2d3c7	Merge pull request 'feat(reloader): enable deploy and integrate configuration reloads' (#91) from fix/reloader-integration into main ... Reviewed-on: https://gitea.forust.xyz/forust/homelab/pulls/91	2026-10-06 14:55:16 +00:00
forust	ff83daed1e	feat(reloader): enable deployment and reload runtime-config consumers	2026-10-06 14:54:45 +00:00
forust	080ae343e6	Merge pull request 'fix(deploy): validate resolved Compose config and namespaced Secrets' (#84) from fix/deploy-validation into main ... Reviewed-on: https://gitea.forust.xyz/forust/homelab/pulls/84	2026-10-06 14:54:27 +00:00
forust	8d3185f8ab	fix(ci): install jq for deploy validation regressions	2026-10-06 16:31:48 +02:00
forust	ff40a71145	fix(deploy): resolve Compose config and check namespaced pod Secrets	2026-10-06 15:58:44 +02:00
forust	cc9c3dea88	feat(traefik): expose insecure API on 8080 for Homarr integration ... ClusterIP access to api@internal from Homarr pod. LAN-reachable on 192.168.80.2:8080, accepted.	2026-10-06 11:27:38 +02:00
forust	815cd85b9a	feat(homarr): deploy dashboard to k8s on home subdomain ... Local-only IngressRoute (home.workstation.internal, home.gigaforust.internal), prod commented out. Compose stack for test stand.	2026-10-06 11:27:35 +02:00
forust	9021eddbc3	chore(deps): pin streaming images, isolate python and floating tags ... Python Y-bumps arrived as minor 3.11->3.14 and floating tags (:latest/:beta) were automerged blindly. Pin the linuxserver stack to digest-verified tags and keep python plus rolling images in manual review groups.	2026-10-05 23:53:38 +02:00
forust	2adf17c307	Merge pull request 'chore(deps): update all patch updates' (#76) from renovate/all-patch into main ... Reviewed-on: #76	2026-10-05 21:41:00 +00:00
forust	34f10211ab	Merge pull request 'chore(deps): update renovate/renovate docker tag to v44.136.0' (#79) from renovate/renovate-self-update into main ... Reviewed-on: #79	2026-10-05 21:39:21 +00:00
forust	8799962b1c	Revert "feat(adguard): add netbird sidecar" ... This reverts commit 7a81b4ea8b.	2026-10-04 17:40:19 +02:00
forust	fb80024fa2	feat(streaming): add bazarr for subtitles	2026-10-04 17:25:00 +02:00
forust	0f1a788874	Merge pull request 'feat(streaming): compose *arr streaming stack (k8s routing)' (#78) from feat/streaming-compose-test into main ... Reviewed-on: #78	2026-10-04 14:04:28 +00:00
forust	39df442e60	fix(streaming): trailing newline for yamllint	2026-10-04 16:03:40 +02:00
forust	62a451773e	feat(streaming): compose *arr streaming stack (k8s routing)	2026-10-04 15:57:08 +02:00
forust	f196099491	feat(adguard): add DNS readiness probe	2026-10-03 22:39:25 +02:00
forust	66502b8279	fix(traefik): protect dashboard with security-chain	2026-10-03 22:39:25 +02:00
forust	9018c091fa	feat(uptime-kuma): add ServiceMonitor, alerts and secrets example	2026-10-03 22:39:24 +02:00
forust	114608af2f	fix(uptime-kuma): label service and name http port	2026-10-03 22:39:23 +02:00
forust	51a73fb213	chore(gitea): switch domain to git.forust.xyz (28.0.0 update)	2026-10-03 22:37:48 +02:00
forust	939fad4a23	chore(renovate,crowdsec): group minor/patch updates and whitelist VPS ... Renovate: group all minor updates and all patch updates into reviewable PRs. Crowdsec: whitelist static VPS IP. Remove stale cloudflared/k8s/active marker.	2026-10-03 11:57:46 +02:00
forust	12d5cc6202	Merge pull request 'chore(deps): update docker.gitea.com/gitea docker tag to v28' (#74) from renovate/docker.gitea.com-gitea-28.x into main ... Reviewed-on: https://gitea.forust.xyz/forust/homelab/pulls/74	2026-10-03 09:44:13 +00:00
forust	a6b84d86b4	Merge pull request 'chore(deps): update gitea/gitea docker tag to v28' (#75) from renovate/gitea-gitea-28.x into main ... Reviewed-on: https://gitea.forust.xyz/forust/homelab/pulls/75	2026-10-03 09:44:06 +00:00
forust	f3a5cedc80	Merge pull request 'chore(deps): update renovate/renovate docker tag to v44.132.2' (#69) from renovate/renovate-self-update into main ... Reviewed-on: https://gitea.forust.xyz/forust/homelab/pulls/69	2026-10-03 09:31:07 +00:00
forust	e208ccae62	Merge pull request 'chore(deps): update ghcr.io/alexta69/metube docker tag to v2026.09.29' (#70) from renovate/container-patch-updates into main ... Reviewed-on: https://gitea.forust.xyz/forust/homelab/pulls/70	2026-10-03 09:20:36 +00:00
forust	d3892d2ed5	Merge pull request 'chore(deps): update ghcr.io/henriquesebastiao/downtify docker tag to v3.4.0' (#71) from renovate/ghcr.io-henriquesebastiao-downtify-3.x into main ... Reviewed-on: https://gitea.forust.xyz/forust/homelab/pulls/71 Reviewed-by: forust <vzlomdsisma@gmail.com>	2026-10-03 09:20:06 +00:00
forust	7164b48275	Merge pull request 'chore(deps): update ghcr.io/lukegus/termix docker tag to v2.9.0' (#72) from renovate/ghcr.io-lukegus-termix-2.x into main ... Reviewed-on: https://gitea.forust.xyz/forust/homelab/pulls/72	2026-10-03 09:19:37 +00:00
forust	956596e6aa	Merge pull request 'chore(deps): update docker.n8n.io/n8nio/n8n docker tag to v2.42.2' (#68) from renovate/docker.n8n.io-n8nio-n8n-2.x into main ... Reviewed-on: https://gitea.forust.xyz/forust/homelab/pulls/68	2026-10-03 09:17:01 +00:00
forust	65709e005b	Merge pull request 'chore(deps): update netbirdio/netbird-server docker tag to v0.80.0' (#73) from renovate/netbirdio-netbird-server-0.x into main ... Reviewed-on: https://gitea.forust.xyz/forust/homelab/pulls/73	2026-10-03 09:16:28 +00:00
forust	78e6363fe2	chore(gitea): add git.forust.xyz route	2026-10-03 11:07:48 +02:00
forust	22c2f1e108	extract dtek_notif subtree to its own repo	2026-10-03 10:48:01 +02:00
forust	64ba4ce9f1	Revert "chore(prometheus): drop dead grafana routes and cert" ... This reverts commit 314ec0cda7.	2026-10-03 10:40:39 +02:00
forust	7a81b4ea8b	feat(adguard): add netbird sidecar	2026-10-03 10:36:15 +02:00
forust	0859479c0f	feat(ingress): replace traefik crowdsec plugin with firewall bouncer ... Move L3 enforcement to the host firewall-bouncer (systemd, nftables): drop the Traefik plugin, its secrets volume and the crowdsec Middleware, remove bouncer refs from all IngressRoutes. Disable the http-generic-bf scenario (403-burst bans hurt legit automation under L3 enforcement). Add a Gateway API PoC for homepages prod and CrowdSec PrometheusRule alerts.	2026-09-30 20:14:25 +02:00
forust	872f64b887	fix(ci): silence intentional SC2029 in ssh-run.sh	2026-09-29 14:54:17 +02:00
forust	a90fb19ed4	fix(traefik): size probes for HDD stalls ... Single replica is the whole ingress; liveness kills at 2s timeouts took every public service down in a loop. Same stall-sized budgets as postgres/metallb. Applied live via helm (pinned 41.5.0, values from git).	2026-09-29 14:47:53 +02:00
forust	6f4cd03f4b	feat(deploy): serialize apply stages and wait for calm node ... apply-k8s and apply-compose share a workstation flock so host docker churn never overlaps cluster churn. Each helm upgrade and the apply loop wait up to 10m for load <28 first, so a deploy never piles onto an already-hot node (the load-40/netbird-death/pending-helm spiral).	2026-09-29 14:42:56 +02:00
forust	e56662194b	fix(ci): log in to registry for manifest-only main pushes ... The pin step writes manifest PUTs on every main push, but login was gated on services != ''. Manifest-only pushes skipped login and pushed anonymously (401); this only worked before via a stale persistent login on the old runner.	2026-09-29 14:21:57 +02:00
forust	4856348a6e	chore(searxng,glance): disable services ... Workloads, services, routes and certs removed from the cluster. Configs, manifests and PVCs kept for easy re-enable via k8s/active.	2026-09-29 13:59:59 +02:00
forust	49d0da1dd0	chore(termix): disable service ... Deployment was stuck in ImagePullBackOff and only generated log churn. Workload, service, routes and certs removed from the cluster; PVC, config and manifests kept so it can be re-enabled by restoring k8s/active.	2026-09-29 13:58:27 +02:00
forust	6bc938436f	Merge pull request 'chore(deps): update grafana/grafana docker tag to v13.2.3' (#67) from renovate/grafana-monorepo into main ... Reviewed-on: https://gitea.forust.xyz/forust/homelab/pulls/67	2026-09-29 11:57:17 +00:00
forust	5dcad7eb38	fix(ci): resolve uv from BIN_DIR in install-ci-tools ... Callers prepend BIN_DIR to PATH only after the script exits, so the bare uv invocation in install_uv_tool died with 127 on clean runners. Export BIN_DIR to PATH inside the script and invoke the just-installed binary by absolute path.	2026-09-29 13:31:28 +02:00
forust	d0872bc918	feat(deploy): autodeploy off unless AUTODEPLOY=true ... Pushes no longer reach the cluster by default; set the AUTODEPLOY repo variable to 'true' to re-enable, or dispatch manually. Skips propagate through the existing needs chain.	2026-09-29 12:54:17 +02:00
forust	91dd749a29	feat(deploy): AUTODEPLOY kill-switch via repo variable ... Set AUTODEPLOY=false under Settings -> Actions -> Variables and pushes stop deploying with no commit; unset means on. Manual Run workflow always bypasses the switch. The existing needs/skipped chaining propagates the skip through verify and smoke untouched.	2026-09-29 12:52:35 +02:00
forust	b6e1dc0362	fix(immich): size postgres probes for HDD stalls ... Postmaster was SIGKILLed in a loop: 70s fsync stalls on the loaded rotational disk outlasted the 5-minute startup budget and the 60s liveness tolerance, and every kill bought another full WAL replay. Startup budget 15min, liveness 5x60s. Already applied live with kubectl; this keeps git in sync.	2026-09-29 12:36:04 +02:00
forust	a2af854867	fix(alerts): measure traefik latency per route via exported_service ... The service label the rule grouped by is the scrape target name, not the backend: with honorLabels=false the per-route value traefik emits is renamed to exported_service, so the old rule measured one global aggregate and both exclusions matched nothing. Group the latency and 5xx alerts by exported_service with exclusions in traefik's real namespace-routename-hash format. Already applied live with kubectl; this commit keeps git in sync.	2026-09-29 09:55:02 +02:00
forust	314ec0cda7	chore(prometheus): drop dead grafana routes and cert ... No grafana is deployed from this stack, so grafana-prod/grafana-local only ever served 503s. Live objects already deleted directly; this keeps git from resurrecting them on the next deploy.	2026-09-29 09:33:38 +02:00
forust	e45ae10204	feat(immich): migrate postgres 14 to 16 with dump/restore ... Reverts the v16 revert: data migrated via pg_dumpall from PG14 into a fresh PG16 data directory (same extension versions vectorchord 0.4.3/pgvectors 0.2.0). Verified 1944 assets, 2 users, 10 albums on 16.10. Note: the restore OOMed once at the 2Gi limit during index builds and recovered via WAL replay; worth watching under PG16 load.	2026-09-29 09:16:58 +02:00
forust	357ee26111	Revert "Merge pull request 'chore(deps): update ghcr.io/immich-app/postgres docker tag to v16' (#65) from renovate/ghcr.io-immich-app-postgres-16.x into main" ... This reverts commit 0c9743e241, reversing changes made to 2cb06debc5.	2026-09-29 08:25:43 +02:00
forust	436fd1ecae	chore: extract userbot subtree to its own repo ... userbot/ (bot plus panel) now lives at /home/forust/userbot as a clone of forust/userbot instead of a subtree in homelab. Cleans up the pipeline references that only existed for it: scan-deps, test-backend and test-frontend jobs, the userbot build matrix entries, the deploy panel hook, and the userbot-only pyrightconfig. Running cluster workloads are untouched; homelab just stops building and testing upstream's code.	2026-09-29 08:24:14 +02:00
forust	a564dd8b67	fix(alerts): exclude netbird streams from traefik latency alert ... SignalExchange ConnectStream holds 60s gRPC streams by design; at night they exceed 5% of samples and pin P95 to the 5.0s bucket ceiling, flapping the alert. Also fixes the stale xui exclusion pattern, which matched no real service label.	2026-09-29 08:12:07 +02:00
forust	c2c90c490d	Merge pull request 'chore(deps): update ghcr.io/autobrr/netronome docker tag to v0.15.0' (#62) from renovate/ghcr.io-autobrr-netronome-0.x into main ... Reviewed-on: https://gitea.forust.xyz/forust/homelab/pulls/62	2026-09-28 21:05:42 +00:00
forust	3ecc12300a	Merge pull request 'chore(deps): update ghcr.io/alexta69/metube docker tag to v2026.09.28' (#61) from renovate/container-patch-updates into main ... Reviewed-on: https://gitea.forust.xyz/forust/homelab/pulls/61	2026-09-28 21:05:33 +00:00
forust	2a5d690e34	Merge pull request 'chore(deps): update netbirdio/dashboard docker tag to v2.94.0' (#63) from renovate/netbirdio-dashboard-2.x into main ... Reviewed-on: https://gitea.forust.xyz/forust/homelab/pulls/63	2026-09-28 21:05:25 +00:00
forust	a8c4e9bcbe	Merge pull request 'chore(deps): update renovate/renovate docker tag to v44.117.0' (#64) from renovate/renovate-self-update into main ... Reviewed-on: https://gitea.forust.xyz/forust/homelab/pulls/64	2026-09-28 21:02:09 +00:00
forust	0c9743e241	Merge pull request 'chore(deps): update ghcr.io/immich-app/postgres docker tag to v16' (#65) from renovate/ghcr.io-immich-app-postgres-16.x into main ... Reviewed-on: https://gitea.forust.xyz/forust/homelab/pulls/65	2026-09-28 21:01:58 +00:00
forust	2cb06debc5	fix(deploy): retry registry lookups with a timeout ... A single blink of the registry failed render_pinned for the whole file and redded the apply stage. registry_digest now retries 3 times under a 25s timeout with a warning per attempt; empty still means unresolvable and callers report it by name as before.	2026-09-28 22:52:54 +02:00
forust	a6af69dca0	fix(k8s): Recreate singletons and trim requests for scheduler headroom ... RollingUpdate with default maxSurge needs a spare pod the single node does not have (99% CPU requested), so multi-workload restarts end Pending and verify times out. Recreate on all replicas:1 Deployments (immich-server and bentopdf keep RollingUpdate at replicas 2). Also trims CPU/memory requests toward measured use (adguard, authentik, gitea, netbox, uptime-kuma, netbird-server) and gives traefik requests/limits so it is no longer BestEffort.	2026-09-28 22:36:46 +02:00
forust	76f39da90c	fix(ci): skip heavy jobs on renovate branches, automerge digest and patch ... Renovate branches only carry version/digest bumps, so scan-deps, test-backend, test-frontend and build just burn runner time on the box that also serves prod. Static checks and validate still run. Digest and patch updates automerge (playwright, helm and major rules below still override to no-automerge). Also replaces deprecated helm --atomic with --wait --rollback-on-failure.	2026-09-28 22:19:57 +02:00
forust	86730ff0c4	Merge pull request 'chore(deps): update ghcr.io/c4illin/convertx docker tag to v0.19.0' (#55) from renovate/ghcr.io-c4illin-convertx-0.x into main ... Reviewed-on: https://gitea.forust.xyz/forust/homelab/pulls/55	2026-09-28 19:57:17 +00:00
forust	deeaefa695	Merge pull request 'chore(deps): update ghcr.io/henriquesebastiao/downtify docker tag to v3.2.0' (#60) from renovate/ghcr.io-henriquesebastiao-downtify-3.x into main ... Reviewed-on: https://gitea.forust.xyz/forust/homelab/pulls/60	2026-09-28 19:57:05 +00:00
forust	742d78944b	Merge pull request 'chore(deps): update ghcr.io/alexta69/metube docker tag to v2026.09.27' (#54) from renovate/container-patch-updates into main ... Reviewed-on: https://gitea.forust.xyz/forust/homelab/pulls/54	2026-09-28 19:56:55 +00:00
forust	33c54ac830	Merge pull request 'chore(deps): update docker.io/valkey/valkey:9 docker digest to 418652c' (#59) from renovate/docker.io-valkey-valkey-9 into main ... Reviewed-on: https://gitea.forust.xyz/forust/homelab/pulls/59	2026-09-28 19:56:27 +00:00
forust	b4f76373bb	fix(gitea): serve issue search from postgres instead of reindexing on boot ... cron.rebuild_issue_indexer runs at start, so every gitea pod restart reindexed the whole issue index and read ~7MB/s off the rotational disk for an hour.	2026-09-28 17:01:36 +02:00
forust	74adf38d63	feat(alerts): cover OOM kills, restart loops and evictions ... The OOMKilled container behind the immich crash loop was invisible: PodCrashLooping only fires once kubelet has already given up and started the backoff.	2026-09-28 17:01:36 +02:00
forust	f5b2f89f38	style(immich): match repo prettier quoting in compose file	2026-09-28 17:00:41 +02:00
forust	8e63284240	feat(immich): add self-hosted photo backup with dedicated postgres ... Server x2, machine learning, valkey and VectorChord postgres on local storage, Traefik routes for external and internal access.	2026-09-28 16:42:05 +02:00
forust	1d81410cd8	fix(traefik): persist plugin storage on a PVC ... Mount traefik-plugins PVC at /plugins-storage instead of the chart default emptyDir, so the crowdsec-bouncer download survives node reboots. Without this Traefik starts before the network is ready, the download from plugins.traefik.io times out, plugins get disabled and every route behind the middleware returns 404/503 until a manual restart.	2026-09-28 13:36:46 +02:00
forust	2f891a5d31	fix(postgres): give probes room on an I/O-bound single node ... pg_isready with the 1s default times out under I/O stall and kubelet kills a healthy postgres mid-recovery; each kill restarts a multi-minute fsync from zero and loops forever. readiness/liveness timeout 5s, liveness threshold 5, startup budget 15min.	2026-09-28 12:35:16 +02:00
forust	cda0022d81	fix(renovate): reap finished job pods with ttlSecondsAfterFinished ... History limits never delete manual 'create job --from' runs, so Failed pods accumulated for a week. Keep a day for debugging, reap the rest.	2026-09-28 12:03:26 +02:00
forust	dde6b1c743	fix(deploy): recover helm releases from pending-* and skip helm-owned rollbacks ... An --atomic upgrade whose own rollback never finishes leaves the release in pending-*, blocking every future run until a human rolls back (loki rev 18/21). Recover automatically before and after each upgrade, and fail loud when recovery does not land on deployed. Also skip helm-managed workloads in rollback_workloads: rollout undo there would step back to the revision --atomic just escaped.	2026-09-28 12:03:26 +02:00
forust	7c4843c88c	fix(loki): unblock gateway rollout on a single node ... Chart default is required podAntiAffinity on hostname plus RollingUpdate 25%/25%, which is maxUnavailable=0 at replicas=1: the new pod stays Unschedulable while the old one lives, and the old one never leaves while the new one is not Ready. Null the affinity (an empty map deep-merges with the default and keeps the rule) and set maxSurge/maxUnavailable to 1.	2026-09-28 11:33:38 +02:00
forust	f9e4623ade	fix(k8s): size the remaining workloads against measured use ... Finishes the sizing pass over every workload the deploy actually manages. Each request is at or above the container's p95 over the last seven days, so nothing is sized below what it is known to use, and each limit is between 1.6x and 5x the observed max, which is the figure that decides whether a burst gets an OOMKill. Some of these go up, and that is the point. adguard was holding 975M against a 500Mi request and netbox 962M against 512Mi, so both sat permanently above their own request and were standing eviction candidates on a node that has about 300M of headroom. Raising a request costs scheduler room; leaving it low costs the pod its place in the queue when the node gets tight. Others come down. loki ran with a 2Gi limit on 224M, gitea 1.5Gi on 305M, the authentik worker 1Gi on 305M, and a tail of single-purpose pods -- redis, glance, the two homepages, cfddns, session-keeper, the netbird dashboard, the loki gateway -- each reserved 4x to 16x more than they have ever touched. prometheus gets the opposite treatment: 768Mi/2560Mi, above its p95, because it compacts its TSDB in place and that is a burst worth budgeting for rather than throttling. Two of these limits are close enough to the observed max to be worth watching rather than trusting: adguard at 1.5x, and its DNS cache grows monotonically, so the ceiling is a date, not a margin. That was true before this change too; the pod sizing does not fix it and the cache needs bounding. CPU limits are untouched throughout. Leaving postgres alone as well: it sits in an uncommitted file that belongs to other work in progress. Verified: every request is at or above p95 and every limit above the observed max across all 74 containers, and 16/16 local gates pass.	2026-09-28 10:38:19 +02:00
forust	2ad4fa1b82	chore(portainer): stop deploying a container manager nothing routes to ... Portainer had been running for 111 days with a 512Mi request and a 2Gi limit against 50M of measured use, on a node that is short of memory. It is a UI over the Docker socket; nothing in the repo or the cluster depends on it. The marker goes, not the manifests. `portainer/k8s/active` is what puts these files in the deploy's manifest set, so without it the next push leaves the namespace alone and the manifests stay on disk for a one-command return. This also matters for the smoke stage: that host list is built from the active directories, so `portainer.forust.xyz` leaves it and the new router check does not go looking for a route to a service we just retired. In the cluster the Deployment, the Service and both IngressRoutes are deleted. The routes go first: leaving an IngressRoute behind a deleted Service keeps a Traefik router pointing at nothing, which answers 502 while looking perfectly healthy to the stage that just started checking for routers. Deliberately kept, so this is reversible rather than destructive: the namespace, the 2Gi `portainer-data-pvc` and both Certificates stay. Re-enabling is `git checkout HEAD~1 -- portainer/k8s/active` plus an apply, and no Let's Encrypt quota is spent reissuing the production certificate. `glance` still links to `portainer.forust.xyz` and that tile will now be a 404. Left alone on purpose. The Cloudflare record is manual and cfddns only ever creates records, so `portainer.forust.xyz` keeps resolving until it is removed in the dashboard, same as `dockmon.forust.xyz`. Verified: no Traefik router matches portainer any more, all 19 hosts left in the smoke list still have a router, and 16/16 local gates pass.	2026-09-28 10:22:51 +02:00