681 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
forust	78e6363fe2	chore(gitea): add git.forust.xyz route	2026-10-03 11:07:48 +02:00
forust	22c2f1e108	extract dtek_notif subtree to its own repo	2026-10-03 10:48:01 +02:00
forust	64ba4ce9f1	Revert "chore(prometheus): drop dead grafana routes and cert" ... This reverts commit 314ec0cda7.	2026-10-03 10:40:39 +02:00
forust	7a81b4ea8b	feat(adguard): add netbird sidecar	2026-10-03 10:36:15 +02:00
forust	0859479c0f	feat(ingress): replace traefik crowdsec plugin with firewall bouncer ... Move L3 enforcement to the host firewall-bouncer (systemd, nftables): drop the Traefik plugin, its secrets volume and the crowdsec Middleware, remove bouncer refs from all IngressRoutes. Disable the http-generic-bf scenario (403-burst bans hurt legit automation under L3 enforcement). Add a Gateway API PoC for homepages prod and CrowdSec PrometheusRule alerts.	2026-09-30 20:14:25 +02:00
forust	872f64b887	fix(ci): silence intentional SC2029 in ssh-run.sh	2026-09-29 14:54:17 +02:00
forust	a90fb19ed4	fix(traefik): size probes for HDD stalls ... Single replica is the whole ingress; liveness kills at 2s timeouts took every public service down in a loop. Same stall-sized budgets as postgres/metallb. Applied live via helm (pinned 41.5.0, values from git).	2026-09-29 14:47:53 +02:00
forust	6f4cd03f4b	feat(deploy): serialize apply stages and wait for calm node ... apply-k8s and apply-compose share a workstation flock so host docker churn never overlaps cluster churn. Each helm upgrade and the apply loop wait up to 10m for load <28 first, so a deploy never piles onto an already-hot node (the load-40/netbird-death/pending-helm spiral).	2026-09-29 14:42:56 +02:00
forust	e56662194b	fix(ci): log in to registry for manifest-only main pushes ... The pin step writes manifest PUTs on every main push, but login was gated on services != ''. Manifest-only pushes skipped login and pushed anonymously (401); this only worked before via a stale persistent login on the old runner.	2026-09-29 14:21:57 +02:00
forust	4856348a6e	chore(searxng,glance): disable services ... Workloads, services, routes and certs removed from the cluster. Configs, manifests and PVCs kept for easy re-enable via k8s/active.	2026-09-29 13:59:59 +02:00
forust	49d0da1dd0	chore(termix): disable service ... Deployment was stuck in ImagePullBackOff and only generated log churn. Workload, service, routes and certs removed from the cluster; PVC, config and manifests kept so it can be re-enabled by restoring k8s/active.	2026-09-29 13:58:27 +02:00
forust	6bc938436f	Merge pull request 'chore(deps): update grafana/grafana docker tag to v13.2.3' (#67) from renovate/grafana-monorepo into main ... Reviewed-on: https://gitea.forust.xyz/forust/homelab/pulls/67	2026-09-29 11:57:17 +00:00
renovate-bot	04b33d0736	chore(deps): update grafana/grafana docker tag to v13.2.3	2026-09-29 11:57:17 +00:00
forust	5dcad7eb38	fix(ci): resolve uv from BIN_DIR in install-ci-tools ... Callers prepend BIN_DIR to PATH only after the script exits, so the bare uv invocation in install_uv_tool died with 127 on clean runners. Export BIN_DIR to PATH inside the script and invoke the just-installed binary by absolute path.	2026-09-29 13:31:28 +02:00
forust	d0872bc918	feat(deploy): autodeploy off unless AUTODEPLOY=true ... Pushes no longer reach the cluster by default; set the AUTODEPLOY repo variable to 'true' to re-enable, or dispatch manually. Skips propagate through the existing needs chain.	2026-09-29 12:54:17 +02:00
forust	91dd749a29	feat(deploy): AUTODEPLOY kill-switch via repo variable ... Set AUTODEPLOY=false under Settings -> Actions -> Variables and pushes stop deploying with no commit; unset means on. Manual Run workflow always bypasses the switch. The existing needs/skipped chaining propagates the skip through verify and smoke untouched.	2026-09-29 12:52:35 +02:00
forust	b6e1dc0362	fix(immich): size postgres probes for HDD stalls ... Postmaster was SIGKILLed in a loop: 70s fsync stalls on the loaded rotational disk outlasted the 5-minute startup budget and the 60s liveness tolerance, and every kill bought another full WAL replay. Startup budget 15min, liveness 5x60s. Already applied live with kubectl; this keeps git in sync.	2026-09-29 12:36:04 +02:00
forust	a2af854867	fix(alerts): measure traefik latency per route via exported_service ... The service label the rule grouped by is the scrape target name, not the backend: with honorLabels=false the per-route value traefik emits is renamed to exported_service, so the old rule measured one global aggregate and both exclusions matched nothing. Group the latency and 5xx alerts by exported_service with exclusions in traefik's real namespace-routename-hash format. Already applied live with kubectl; this commit keeps git in sync.	2026-09-29 09:55:02 +02:00
forust	314ec0cda7	chore(prometheus): drop dead grafana routes and cert ... No grafana is deployed from this stack, so grafana-prod/grafana-local only ever served 503s. Live objects already deleted directly; this keeps git from resurrecting them on the next deploy.	2026-09-29 09:33:38 +02:00
forust	e45ae10204	feat(immich): migrate postgres 14 to 16 with dump/restore ... Reverts the v16 revert: data migrated via pg_dumpall from PG14 into a fresh PG16 data directory (same extension versions vectorchord 0.4.3/pgvectors 0.2.0). Verified 1944 assets, 2 users, 10 albums on 16.10. Note: the restore OOMed once at the 2Gi limit during index builds and recovered via WAL replay; worth watching under PG16 load.	2026-09-29 09:16:58 +02:00
forust	357ee26111	Revert "Merge pull request 'chore(deps): update ghcr.io/immich-app/postgres docker tag to v16' (#65) from renovate/ghcr.io-immich-app-postgres-16.x into main" ... This reverts commit 0c9743e241, reversing changes made to 2cb06debc5.	2026-09-29 08:25:43 +02:00
forust	436fd1ecae	chore: extract userbot subtree to its own repo ... userbot/ (bot plus panel) now lives at /home/forust/userbot as a clone of forust/userbot instead of a subtree in homelab. Cleans up the pipeline references that only existed for it: scan-deps, test-backend and test-frontend jobs, the userbot build matrix entries, the deploy panel hook, and the userbot-only pyrightconfig. Running cluster workloads are untouched; homelab just stops building and testing upstream's code.	2026-09-29 08:24:14 +02:00
forust	a564dd8b67	fix(alerts): exclude netbird streams from traefik latency alert ... SignalExchange ConnectStream holds 60s gRPC streams by design; at night they exceed 5% of samples and pin P95 to the 5.0s bucket ceiling, flapping the alert. Also fixes the stale xui exclusion pattern, which matched no real service label.	2026-09-29 08:12:07 +02:00
forust	c2c90c490d	Merge pull request 'chore(deps): update ghcr.io/autobrr/netronome docker tag to v0.15.0' (#62) from renovate/ghcr.io-autobrr-netronome-0.x into main ... Reviewed-on: https://gitea.forust.xyz/forust/homelab/pulls/62	2026-09-28 21:05:42 +00:00
renovate-bot	da0f9e84c3	chore(deps): update ghcr.io/autobrr/netronome docker tag to v0.15.0	2026-09-28 21:05:42 +00:00
forust	3ecc12300a	Merge pull request 'chore(deps): update ghcr.io/alexta69/metube docker tag to v2026.09.28' (#61) from renovate/container-patch-updates into main ... Reviewed-on: https://gitea.forust.xyz/forust/homelab/pulls/61	2026-09-28 21:05:33 +00:00
renovate-bot	19029fa012	chore(deps): update ghcr.io/alexta69/metube docker tag to v2026.09.28	2026-09-28 21:05:33 +00:00
forust	2a5d690e34	Merge pull request 'chore(deps): update netbirdio/dashboard docker tag to v2.94.0' (#63) from renovate/netbirdio-dashboard-2.x into main ... Reviewed-on: https://gitea.forust.xyz/forust/homelab/pulls/63	2026-09-28 21:05:25 +00:00
renovate-bot	b53ce36d89	chore(deps): update netbirdio/dashboard docker tag to v2.94.0	2026-09-28 21:05:25 +00:00
forust	a8c4e9bcbe	Merge pull request 'chore(deps): update renovate/renovate docker tag to v44.117.0' (#64) from renovate/renovate-self-update into main ... Reviewed-on: https://gitea.forust.xyz/forust/homelab/pulls/64	2026-09-28 21:02:09 +00:00
renovate-bot	761f97ef8e	chore(deps): update renovate/renovate docker tag to v44.117.0	2026-09-28 21:02:09 +00:00
forust	0c9743e241	Merge pull request 'chore(deps): update ghcr.io/immich-app/postgres docker tag to v16' (#65) from renovate/ghcr.io-immich-app-postgres-16.x into main ... Reviewed-on: https://gitea.forust.xyz/forust/homelab/pulls/65	2026-09-28 21:01:58 +00:00
renovate-bot	2b3c28a46b	chore(deps): update ghcr.io/immich-app/postgres docker tag to v16	2026-09-28 21:01:58 +00:00
forust	2cb06debc5	fix(deploy): retry registry lookups with a timeout ... A single blink of the registry failed render_pinned for the whole file and redded the apply stage. registry_digest now retries 3 times under a 25s timeout with a warning per attempt; empty still means unresolvable and callers report it by name as before.	2026-09-28 22:52:54 +02:00
forust	a6af69dca0	fix(k8s): Recreate singletons and trim requests for scheduler headroom ... RollingUpdate with default maxSurge needs a spare pod the single node does not have (99% CPU requested), so multi-workload restarts end Pending and verify times out. Recreate on all replicas:1 Deployments (immich-server and bentopdf keep RollingUpdate at replicas 2). Also trims CPU/memory requests toward measured use (adguard, authentik, gitea, netbox, uptime-kuma, netbird-server) and gives traefik requests/limits so it is no longer BestEffort.	2026-09-28 22:36:46 +02:00
forust	76f39da90c	fix(ci): skip heavy jobs on renovate branches, automerge digest and patch ... Renovate branches only carry version/digest bumps, so scan-deps, test-backend, test-frontend and build just burn runner time on the box that also serves prod. Static checks and validate still run. Digest and patch updates automerge (playwright, helm and major rules below still override to no-automerge). Also replaces deprecated helm --atomic with --wait --rollback-on-failure.	2026-09-28 22:19:57 +02:00
forust	86730ff0c4	Merge pull request 'chore(deps): update ghcr.io/c4illin/convertx docker tag to v0.19.0' (#55) from renovate/ghcr.io-c4illin-convertx-0.x into main ... Reviewed-on: https://gitea.forust.xyz/forust/homelab/pulls/55	2026-09-28 19:57:17 +00:00
renovate-bot	af66d3e7fd	chore(deps): update ghcr.io/c4illin/convertx docker tag to v0.19.0	2026-09-28 19:57:17 +00:00
forust	deeaefa695	Merge pull request 'chore(deps): update ghcr.io/henriquesebastiao/downtify docker tag to v3.2.0' (#60) from renovate/ghcr.io-henriquesebastiao-downtify-3.x into main ... Reviewed-on: https://gitea.forust.xyz/forust/homelab/pulls/60	2026-09-28 19:57:05 +00:00
renovate-bot	6999dd2728	chore(deps): update ghcr.io/henriquesebastiao/downtify docker tag to v3.2.0	2026-09-28 19:57:05 +00:00
forust	742d78944b	Merge pull request 'chore(deps): update ghcr.io/alexta69/metube docker tag to v2026.09.27' (#54) from renovate/container-patch-updates into main ... Reviewed-on: https://gitea.forust.xyz/forust/homelab/pulls/54	2026-09-28 19:56:55 +00:00
renovate-bot	e451c97dfc	chore(deps): update container patch updates	2026-09-28 19:56:45 +00:00
forust	33c54ac830	Merge pull request 'chore(deps): update docker.io/valkey/valkey:9 docker digest to 418652c' (#59) from renovate/docker.io-valkey-valkey-9 into main ... Reviewed-on: https://gitea.forust.xyz/forust/homelab/pulls/59	2026-09-28 19:56:27 +00:00
renovate-bot	b09d718310	chore(deps): update docker.io/valkey/valkey:9 docker digest to 418652c	2026-09-28 16:20:21 +00:00
forust	b4f76373bb	fix(gitea): serve issue search from postgres instead of reindexing on boot ... cron.rebuild_issue_indexer runs at start, so every gitea pod restart reindexed the whole issue index and read ~7MB/s off the rotational disk for an hour.	2026-09-28 17:01:36 +02:00
forust	74adf38d63	feat(alerts): cover OOM kills, restart loops and evictions ... The OOMKilled container behind the immich crash loop was invisible: PodCrashLooping only fires once kubelet has already given up and started the backoff.	2026-09-28 17:01:36 +02:00
forust	f5b2f89f38	style(immich): match repo prettier quoting in compose file	2026-09-28 17:00:41 +02:00
forust	8e63284240	feat(immich): add self-hosted photo backup with dedicated postgres ... Server x2, machine learning, valkey and VectorChord postgres on local storage, Traefik routes for external and internal access.	2026-09-28 16:42:05 +02:00
forust	1d81410cd8	fix(traefik): persist plugin storage on a PVC ... Mount traefik-plugins PVC at /plugins-storage instead of the chart default emptyDir, so the crowdsec-bouncer download survives node reboots. Without this Traefik starts before the network is ready, the download from plugins.traefik.io times out, plugins get disabled and every route behind the middleware returns 404/503 until a manual restart.	2026-09-28 13:36:46 +02:00
forust	2f891a5d31	fix(postgres): give probes room on an I/O-bound single node ... pg_isready with the 1s default times out under I/O stall and kubelet kills a healthy postgres mid-recovery; each kill restarts a multi-minute fsync from zero and loops forever. readiness/liveness timeout 5s, liveness threshold 5, startup budget 15min.	2026-09-28 12:35:16 +02:00