fix(crowdsec): stop the bouncer from 403-ing our own deploys
ci / lint-compose (push) Successful in 4s
ci / lint-actionlint (push) Successful in 1s
ci / lint-shellcheck (push) Successful in 2s
ci / lint-prettier (push) Successful in 3s
ci / validate (push) Successful in 3s
renovate-ci / validate-renovate (push) Successful in 32s
ci / lint-ruff (push) Successful in 0s
ci / lint-yaml (push) Successful in 2s
ci / lint-dockerfiles (push) Successful in 2s
ci / scan-deps (push) Successful in 17s
ci / test-backend (push) Successful in 7s
ci / test-frontend (push) Successful in 10s
ci / build (push) Successful in 1s

Every bouncer-protected request blocks on a synchronous
GET /v1/decisions against the LAPI, and the plugin fails CLOSED with 403
when that lookup exceeds its timeout. The LAPI was capped at 400m/500Mi
on a single replica, so idle lookups measured 1.3-7.4s and a deploy
burst pushed them past the fork's implicit 10s default: gitea answered
403 for ten seconds straight, and containerd turned those 403s on
gcr.forust.xyz into ErrImagePull/ImagePullBackOff on freshly rolled
pods.

Three changes, plus the 403 feedback loop that made it sticky:

* gitea/k8s/ingress.yaml - drop the bouncer from the /v2 registry route.
  A deploy fires hundreds of parallel authenticated OCI requests
  (runner Action API, manifest inspect per own image, containerd pulls,
  smoke probes) and scanners gain nothing from a registry that already
  does its own token auth. The web UI route keeps the bouncer.
* crowdsec-values.yaml - LAPI to 1500m/1Gi. Replicas stay at 1 on
  purpose: LAPI is stateful (BoltDB plus credentials on two RWO PVCs),
  so a second replica would corrupt the decision store.
* crowdsec-middleware.yaml - CrowdsecLapiTimeout: 2s instead of the
  implicit 10s, so a slow LAPI costs a fast 403 rather than a 10s hang.

LePresidente/http-generic-403-bf then banned us for our own 403s: five
POSTs answered 403 within 10s earn a 4h ban, and the hairpin-NAT
address 192.168.88.1 that the Gitea Actions runner presents to Traefik
is not covered by the home-dynamic-IP whitelist. That scenario cannot
be dropped per-scenario - it is baked into the hub item
crowdsecurity/http-generic-bf v0.9, and disabling the whole
base-http-scenarios collection would cost ~40 useful detections. So the
janitor now deletes its decisions hourly and a new forust/lan
whitelist postoverflow covers 192.168.88.0/24. First janitor run
removed 165 decisions; none of the remaining ones are local.

Measured after: gitea 200 in 25-148ms (was 403 at 10001ms), a 60-way
parallel burst all 200 with a 422ms max, gcr /v2 back to its 401 auth
challenge, LAPI at 60m CPU with no throttling.
This commit is contained in:
forust committed 2026-09-27 17:08:22 +02:00
1 parent 6a9a460769
commit f7cd75d65e
4 files changed
+68 -8

No files matched your search

+20 -1
View File
@@ -30,7 +30,10 @@
# 3. ensure the static machine exists, recreating it with the
# Secret password if missing (agent retry loops reconnect
# on their own - same name + same password);
# 4. prune bouncer entries idle for 30d.
# 4. prune bouncer entries idle for 30d;
# 5. delete decisions from LePresidente/http-generic-403-bf, a hub
# scenario that bans an IP for 4h after 5 POST-403s in 10s and
# therefore bans us for our own bouncer's fail-closed 403s.
#
# Manual apply (crowdsec/k8s is NOT managed by deploy.yaml):
# kubectl apply -f crowdsec/k8s/janitor-cronjob.yaml
@@ -193,3 +196,19 @@ spec:
fi
echo "== 4. prune stale bouncers (no pull for 30d) =="
$LAPI_EXEC cscli bouncers prune -d 720h --force
echo "== 5. drop http-403-bf decisions (4h self-bans) =="
# `LePresidente/http-generic-403-bf` (hub item
# crowdsecurity/http-generic-bf v0.9) bans any source IP
# after 5 POSTs answered 403 within 10s, for 4h. That
# includes 403s this homelab generates ITSELF (any
# bouncer fail-closed, any app CSRF/rate-limit 403), and a
# 4h ban on the runner/home IP silently breaks deploys and
# browsing. The scenario cannot be removed per-scenario -
# it is baked into a hub item, and disabling the whole
# base-http-scenarios collection would drop ~40 useful
# detections. Instead we keep the detection and drop its
# decisions hourly; the LAN/home whitelists in
# crowdsec-values.yaml handle the legit sources, so this
# only ever hits real scanners (who are re-banned anyway).
$LAPI_EXEC cscli decisions delete \
--scenario LePresidente/http-generic-403-bf --all || true