fix(crowdsec): stop the 403 loop that banned our own VPS and runner
Chasing why apply-k8s kept dying mid-run turned up a self-inflicted
ban loop. 585 of 586 LePresidente/http-generic-403-bf alerts in the LAPI
came from 193.181.211.79 - our own VPS - POSTing
/management.ManagementService/GetServerKey, i.e. the NetBird client's
own management call. netbird-server had never seen a single one of them,
so the 403 was not NetBird's: the bouncer was rejecting the request
before it got there. A banned peer keeps retrying, each retry is another
403, and the scenario turns five 403s in ten seconds into a 4h ban, so
the loop kept re-arming the ban it was serving. The hourly janitor step
that deleted those decisions hourly was masking all of it.
* netbird/k8s/ingress.yaml - drop the bouncer from the mesh API routes
(gRPC-gateway management, signal, relay, /api, /oauth2). A ban there
locks a peer out of the network it needs to reach anything else, and
those endpoints authenticate by NetBird token, not by a login form.
The dashboard keeps the bouncer; it is a real login surface.
netbird-local was already exempt, so this makes prod match.
* crowdsec-middleware.yaml - CrowdsecMode stream instead of live. live
blocked on GET /v1/decisions per request, so a burst saturated the
LAPI and the plugin 403'd IPs that were never banned. v1.3.3 ignores
UpdateMaxFailure in live, so fail-open is only reachable in stream;
-1 now means an unreachable LAPI degrades to "no protection" rather
than "every site 403". 15s poll instead of the 60s default, because
the runner shares one public IP with the house.
Also corrects the key name: HTTPTimeoutSeconds, not
CrowdsecLapiTimeout, which never existed and was being silently
dropped, leaving the 10s default. Back at 10, not the 2s f7cd75d
guessed - a pull that times out leaves the ban cache frozen at its
startup contents, so new bans would silently never apply.
* crowdsec-values.yaml - CIDR allowlisting moves to parsers/s02-enrich,
where CrowdSec's docs put it: a parser whitelist drops the event before
it reaches a bucket, so those addresses never become a decision at
all. The old postoverflow LAN list was checked only after the ban
existed, which is the window the deploy kept landing in. Added
100.64.0.0/10, which the RFC 1918 blocks miss and where the
workstation, the k0s node and the VPS actually live. The DDNS
home-IP whitelist stays in postoverflows, because resolving a hostname
is the expensive check the docs reserve that stage for.
ClientTrustedIPs mirrors that list so the bouncer skips the LAPI
round-trip entirely for those addresses.
* janitor-cronjob.yaml - drop step 5. The bouncer can no longer
manufacture 403s, so the only remaining firings of that scenario are
real scanners, and deleting their decisions hourly was undoing a
working ban.
* Also lands the LAPI config.yaml.local (SQLite WAL, Central API off,
bounded flush) that f7cd75d's comments referenced but never included:
the LAPI was blocked in fsync on its rollback journal, and the CAPI
resolver held a write transaction while timing out against a host
this network cannot reach.
Verified in-cluster: no new 403-bf alerts in the 3.5min after applying,
the management endpoint answers 404 from the backend instead of 403 from
the bouncer in 0.17s, and the VPS client reports Management and Signal
connected with 2/2 relays.
This commit is contained in:
1 parent
f7cd75d65e
commit
892790822d
5 files changed
+141
-66
No files matched your search
@@ -30,10 +30,14 @@
|
||||
# 3. ensure the static machine exists, recreating it with the
|
||||
# Secret password if missing (agent retry loops reconnect
|
||||
# on their own - same name + same password);
|
||||
# 4. prune bouncer entries idle for 30d;
|
||||
# 5. delete decisions from LePresidente/http-generic-403-bf, a hub
|
||||
# scenario that bans an IP for 4h after 5 POST-403s in 10s and
|
||||
# therefore bans us for our own bouncer's fail-closed 403s.
|
||||
# 4. prune bouncer entries idle for 30d.
|
||||
#
|
||||
# It used to also delete LePresidente/http-generic-403-bf decisions hourly.
|
||||
# That was a workaround for the bouncer failing closed on a slow LAPI and
|
||||
# 403-ing the deploy runner into a 4h ban. The bouncer now polls decisions
|
||||
# into a cache and never blocks on an unreachable LAPI, so it cannot
|
||||
# manufacture those 403s any more, and the scenario only fires against real
|
||||
# scanners - deleting their decisions hourly was undoing a working ban.
|
||||
#
|
||||
# Manual apply (crowdsec/k8s is NOT managed by deploy.yaml):
|
||||
# kubectl apply -f crowdsec/k8s/janitor-cronjob.yaml
|
||||
@@ -196,19 +200,3 @@ spec:
|
||||
fi
|
||||
echo "== 4. prune stale bouncers (no pull for 30d) =="
|
||||
$LAPI_EXEC cscli bouncers prune -d 720h --force
|
||||
echo "== 5. drop http-403-bf decisions (4h self-bans) =="
|
||||
# `LePresidente/http-generic-403-bf` (hub item
|
||||
# crowdsecurity/http-generic-bf v0.9) bans any source IP
|
||||
# after 5 POSTs answered 403 within 10s, for 4h. That
|
||||
# includes 403s this homelab generates ITSELF (any
|
||||
# bouncer fail-closed, any app CSRF/rate-limit 403), and a
|
||||
# 4h ban on the runner/home IP silently breaks deploys and
|
||||
# browsing. The scenario cannot be removed per-scenario -
|
||||
# it is baked into a hub item, and disabling the whole
|
||||
# base-http-scenarios collection would drop ~40 useful
|
||||
# detections. Instead we keep the detection and drop its
|
||||
# decisions hourly; the LAN/home whitelists in
|
||||
# crowdsec-values.yaml handle the legit sources, so this
|
||||
# only ever hits real scanners (who are re-banned anyway).
|
||||
$LAPI_EXEC cscli decisions delete \
|
||||
--scenario LePresidente/http-generic-403-bf --all || true
|
||||
Reference in new issue
Block a user