fix(crowdsec): stop the bouncer from 403-ing our own deploys
ci / lint-compose (push) Successful in 4s
ci / lint-actionlint (push) Successful in 1s
ci / lint-shellcheck (push) Successful in 2s
ci / lint-prettier (push) Successful in 3s
ci / validate (push) Successful in 3s
renovate-ci / validate-renovate (push) Successful in 32s
ci / lint-ruff (push) Successful in 0s
ci / lint-yaml (push) Successful in 2s
ci / lint-dockerfiles (push) Successful in 2s
ci / scan-deps (push) Successful in 17s
ci / test-backend (push) Successful in 7s
ci / test-frontend (push) Successful in 10s
ci / build (push) Successful in 1s
ci / lint-compose (push) Successful in 4s
ci / lint-actionlint (push) Successful in 1s
ci / lint-shellcheck (push) Successful in 2s
ci / lint-prettier (push) Successful in 3s
ci / validate (push) Successful in 3s
renovate-ci / validate-renovate (push) Successful in 32s
ci / lint-ruff (push) Successful in 0s
ci / lint-yaml (push) Successful in 2s
ci / lint-dockerfiles (push) Successful in 2s
ci / scan-deps (push) Successful in 17s
ci / test-backend (push) Successful in 7s
ci / test-frontend (push) Successful in 10s
ci / build (push) Successful in 1s
Every bouncer-protected request blocks on a synchronous GET /v1/decisions against the LAPI, and the plugin fails CLOSED with 403 when that lookup exceeds its timeout. The LAPI was capped at 400m/500Mi on a single replica, so idle lookups measured 1.3-7.4s and a deploy burst pushed them past the fork's implicit 10s default: gitea answered 403 for ten seconds straight, and containerd turned those 403s on gcr.forust.xyz into ErrImagePull/ImagePullBackOff on freshly rolled pods. Three changes, plus the 403 feedback loop that made it sticky: * gitea/k8s/ingress.yaml - drop the bouncer from the /v2 registry route. A deploy fires hundreds of parallel authenticated OCI requests (runner Action API, manifest inspect per own image, containerd pulls, smoke probes) and scanners gain nothing from a registry that already does its own token auth. The web UI route keeps the bouncer. * crowdsec-values.yaml - LAPI to 1500m/1Gi. Replicas stay at 1 on purpose: LAPI is stateful (BoltDB plus credentials on two RWO PVCs), so a second replica would corrupt the decision store. * crowdsec-middleware.yaml - CrowdsecLapiTimeout: 2s instead of the implicit 10s, so a slow LAPI costs a fast 403 rather than a 10s hang. LePresidente/http-generic-403-bf then banned us for our own 403s: five POSTs answered 403 within 10s earn a 4h ban, and the hairpin-NAT address 192.168.88.1 that the Gitea Actions runner presents to Traefik is not covered by the home-dynamic-IP whitelist. That scenario cannot be dropped per-scenario - it is baked into the hub item crowdsecurity/http-generic-bf v0.9, and disabling the whole base-http-scenarios collection would cost ~40 useful detections. So the janitor now deletes its decisions hourly and a new forust/lan whitelist postoverflow covers 192.168.88.0/24. First janitor run removed 165 decisions; none of the remaining ones are local. Measured after: gitea 200 in 25-148ms (was 403 at 10001ms), a 60-way parallel burst all 200 with a 422ms max, gcr /v2 back to its 401 auth challenge, LAPI at 60m CPU with no throttling.
This commit is contained in:
1 parent
6a9a460769
commit
f7cd75d65e
4 files changed
+68
-8
No files matched your search
@@ -30,7 +30,10 @@
|
||||
# 3. ensure the static machine exists, recreating it with the
|
||||
# Secret password if missing (agent retry loops reconnect
|
||||
# on their own - same name + same password);
|
||||
# 4. prune bouncer entries idle for 30d.
|
||||
# 4. prune bouncer entries idle for 30d;
|
||||
# 5. delete decisions from LePresidente/http-generic-403-bf, a hub
|
||||
# scenario that bans an IP for 4h after 5 POST-403s in 10s and
|
||||
# therefore bans us for our own bouncer's fail-closed 403s.
|
||||
#
|
||||
# Manual apply (crowdsec/k8s is NOT managed by deploy.yaml):
|
||||
# kubectl apply -f crowdsec/k8s/janitor-cronjob.yaml
|
||||
@@ -193,3 +196,19 @@ spec:
|
||||
fi
|
||||
echo "== 4. prune stale bouncers (no pull for 30d) =="
|
||||
$LAPI_EXEC cscli bouncers prune -d 720h --force
|
||||
echo "== 5. drop http-403-bf decisions (4h self-bans) =="
|
||||
# `LePresidente/http-generic-403-bf` (hub item
|
||||
# crowdsecurity/http-generic-bf v0.9) bans any source IP
|
||||
# after 5 POSTs answered 403 within 10s, for 4h. That
|
||||
# includes 403s this homelab generates ITSELF (any
|
||||
# bouncer fail-closed, any app CSRF/rate-limit 403), and a
|
||||
# 4h ban on the runner/home IP silently breaks deploys and
|
||||
# browsing. The scenario cannot be removed per-scenario -
|
||||
# it is baked into a hub item, and disabling the whole
|
||||
# base-http-scenarios collection would drop ~40 useful
|
||||
# detections. Instead we keep the detection and drop its
|
||||
# decisions hourly; the LAN/home whitelists in
|
||||
# crowdsec-values.yaml handle the legit sources, so this
|
||||
# only ever hits real scanners (who are re-banned anyway).
|
||||
$LAPI_EXEC cscli decisions delete \
|
||||
--scenario LePresidente/http-generic-403-bf --all || true
|
||||
Reference in new issue
Block a user