3 Commits

Include renames

Author	SHA1	Message	Date
forust	892790822d	fix(crowdsec): stop the 403 loop that banned our own VPS and runner ... Chasing why apply-k8s kept dying mid-run turned up a self-inflicted ban loop. 585 of 586 LePresidente/http-generic-403-bf alerts in the LAPI came from 193.181.211.79 - our own VPS - POSTing /management.ManagementService/GetServerKey, i.e. the NetBird client's own management call. netbird-server had never seen a single one of them, so the 403 was not NetBird's: the bouncer was rejecting the request before it got there. A banned peer keeps retrying, each retry is another 403, and the scenario turns five 403s in ten seconds into a 4h ban, so the loop kept re-arming the ban it was serving. The hourly janitor step that deleted those decisions hourly was masking all of it. * netbird/k8s/ingress.yaml - drop the bouncer from the mesh API routes (gRPC-gateway management, signal, relay, /api, /oauth2). A ban there locks a peer out of the network it needs to reach anything else, and those endpoints authenticate by NetBird token, not by a login form. The dashboard keeps the bouncer; it is a real login surface. netbird-local was already exempt, so this makes prod match. * crowdsec-middleware.yaml - CrowdsecMode stream instead of live. live blocked on GET /v1/decisions per request, so a burst saturated the LAPI and the plugin 403'd IPs that were never banned. v1.3.3 ignores UpdateMaxFailure in live, so fail-open is only reachable in stream; -1 now means an unreachable LAPI degrades to "no protection" rather than "every site 403". 15s poll instead of the 60s default, because the runner shares one public IP with the house. Also corrects the key name: HTTPTimeoutSeconds, not CrowdsecLapiTimeout, which never existed and was being silently dropped, leaving the 10s default. Back at 10, not the 2s f7cd75d guessed - a pull that times out leaves the ban cache frozen at its startup contents, so new bans would silently never apply. * crowdsec-values.yaml - CIDR allowlisting moves to parsers/s02-enrich, where CrowdSec's docs put it: a parser whitelist drops the event before it reaches a bucket, so those addresses never become a decision at all. The old postoverflow LAN list was checked only after the ban existed, which is the window the deploy kept landing in. Added 100.64.0.0/10, which the RFC 1918 blocks miss and where the workstation, the k0s node and the VPS actually live. The DDNS home-IP whitelist stays in postoverflows, because resolving a hostname is the expensive check the docs reserve that stage for. ClientTrustedIPs mirrors that list so the bouncer skips the LAPI round-trip entirely for those addresses. * janitor-cronjob.yaml - drop step 5. The bouncer can no longer manufacture 403s, so the only remaining firings of that scenario are real scanners, and deleting their decisions hourly was undoing a working ban. * Also lands the LAPI config.yaml.local (SQLite WAL, Central API off, bounded flush) that f7cd75d's comments referenced but never included: the LAPI was blocked in fsync on its rollback journal, and the CAPI resolver held a write transaction while timing out against a host this network cannot reach. Verified in-cluster: no new 403-bf alerts in the 3.5min after applying, the management endpoint answers 404 from the backend instead of 403 from the bouncer in 0.17s, and the VPS client reports Management and Signal connected with 2/2 relays.	2026-09-27 18:09:07 +02:00
forust	f7cd75d65e	fix(crowdsec): stop the bouncer from 403-ing our own deploys ... Every bouncer-protected request blocks on a synchronous GET /v1/decisions against the LAPI, and the plugin fails CLOSED with 403 when that lookup exceeds its timeout. The LAPI was capped at 400m/500Mi on a single replica, so idle lookups measured 1.3-7.4s and a deploy burst pushed them past the fork's implicit 10s default: gitea answered 403 for ten seconds straight, and containerd turned those 403s on gcr.forust.xyz into ErrImagePull/ImagePullBackOff on freshly rolled pods. Three changes, plus the 403 feedback loop that made it sticky: * gitea/k8s/ingress.yaml - drop the bouncer from the /v2 registry route. A deploy fires hundreds of parallel authenticated OCI requests (runner Action API, manifest inspect per own image, containerd pulls, smoke probes) and scanners gain nothing from a registry that already does its own token auth. The web UI route keeps the bouncer. * crowdsec-values.yaml - LAPI to 1500m/1Gi. Replicas stay at 1 on purpose: LAPI is stateful (BoltDB plus credentials on two RWO PVCs), so a second replica would corrupt the decision store. * crowdsec-middleware.yaml - CrowdsecLapiTimeout: 2s instead of the implicit 10s, so a slow LAPI costs a fast 403 rather than a 10s hang. LePresidente/http-generic-403-bf then banned us for our own 403s: five POSTs answered 403 within 10s earn a 4h ban, and the hairpin-NAT address 192.168.88.1 that the Gitea Actions runner presents to Traefik is not covered by the home-dynamic-IP whitelist. That scenario cannot be dropped per-scenario - it is baked into the hub item crowdsecurity/http-generic-bf v0.9, and disabling the whole base-http-scenarios collection would cost ~40 useful detections. So the janitor now deletes its decisions hourly and a new forust/lan whitelist postoverflow covers 192.168.88.0/24. First janitor run removed 165 decisions; none of the remaining ones are local. Measured after: gitea 200 in 25-148ms (was 403 at 10001ms), a 60-way parallel burst all 200 with a 422ms max, gcr /v2 back to its 401 auth challenge, LAPI at 60m CPU with no throttling.	2026-09-27 17:08:22 +02:00
forust	7cca330438	feat(crowdsec): switch agent to loki acquisition with static identity ... Read traefik logs from Loki instead of file tail. Static machine identity via pre-created secret stops 403 register races. Add janitor cronjob, dashboards, whitelists and metrics.	2026-09-18 19:02:43 +02:00