userbot/ (bot plus panel) now lives at /home/forust/userbot as a clone of forust/userbot instead of a subtree in homelab. Cleans up the pipeline references that only existed for it: scan-deps, test-backend and test-frontend jobs, the userbot build matrix entries, the deploy panel hook, and the userbot-only pyrightconfig. Running cluster workloads are untouched; homelab just stops building and testing upstream's code.
Renovate branches only carry version/digest bumps, so scan-deps, test-backend, test-frontend and build just burn runner time on the box that also serves prod. Static checks and validate still run. Digest and patch updates automerge (playwright, helm and major rules below still override to no-automerge). Also replaces deprecated helm --atomic with --wait --rollback-on-failure.
Every service tracked the mutable `:prod` tag, so a deploy applied whatever
that tag happened to name at the time rather than the commit it was
deploying. A rollback had no way to state what it was rolling back to, and
two deploys of one commit could land different images.
CI now publishes an immutable `sha-<commit12>` tag beside `:prod` on main,
and re-tags it for every image a push did not rebuild. That re-tag copies
the manifest list, so no layer moves. The deploy resolves the immutable tag
to a digest and pins the workload to it, and only falls back to the moving
tag when the immutable one cannot be resolved -- which it says out loud,
because that fallback is the deploy quietly ceasing to be reproducible from
its own commit.
The image list comes out of the tree with git grep rather than being written
out a second time, so adding a service no longer means keeping two lists in
step.
build also gains the three jobs it was skipping -- scan-deps, test-backend,
test-frontend -- so a change that breaks them cannot be tagged at all. The
two run blocks where a mid-loop failure was survivable now run under
set -euo pipefail: the build loop and the service detector both carried on
past an error and could report a green build having produced nothing.
The registry password moves from run: substitution into an env: block. A
quote, a backtick or a $(...) in the password is parsed as shell before the
command ever runs, and a login that failed that way looked exactly like a
build that failed.
The apply and verify timeouts stay at 45 and 30 minutes. The comments now
record the arithmetic that says so rather than leaving the numbers to be
raised on the next scare: three no-op helm upgrades run 3-5 minutes, one
broken release is a single 10 minute rollback because the loop aborts on
the first failure, and the apply loop itself is about a minute. That is
roughly 15 minutes of work against a 45 minute budget. verify is 32
workloads at 8 wide -- four waves of 300 seconds, 20 minutes -- which
leaves room for two serial rollbacks, and only becomes derivable at 45 once
rollback_workloads is parallelised.
The runner executes jobs on the host rather than in a container, so a
workflow that says 'python3' or 'npm' is really saying 'whatever this
machine happens to have today'. Both of the test jobs added in c00a472
were red on the first run for exactly that reason.
uv venv with no --python takes the first interpreter it finds, which is
the host's 3.14 here. pyrogram's sync.py calls the bare
asyncio.get_event_loop() that 3.14 no longer auto-creates, so three
tests died at collection. Pinned to 3.13, which is both what uv will
fetch when the host has none and what python:3.13-slim actually builds.
npm was missing outright, and turned up an hour later as npm 12 on node
26 - the same push, minutes apart. Neither is the panel image's
node:22-alpine, so node is now installed from the official tarball the
way the other tools are, at the image's major. npm is checked by running
it rather than by looking it up, so a name that resolves to something
broken reads as not installed.
Renovate keeps NODE_VERSION in step with the Dockerfile's node: tag, and
the two are one grouped dependency: CI that tests on a different major
than it builds on is a gate that can pass over a real break.
Two things, both about not finding out late.
No workflow declared `permissions`, so all eighteen jobs across the four
workflows ran on a token with the default full repository scope. Every one of
them only checks out code, and deploy reaches the cluster over SSH with the
deploy key, and Renovate writes through its own bot PAT rather than the
Actions token. So `contents: read` is all any of them needed.
The panel image ships 15 known advisories and nothing was looking. Add a
scan-deps job that fails on anything new, and record the eight current ones by
ID in the workflow. It is a list rather than a baseline count so that the diff
that accepts an advisory says so in words, and it lives in our workflow
instead of the package manifest so a subtree sync from forust/userbot cannot
quietly widen the exemption.
Both halves were checked to fail on a regression, not just to pass today:
removing one --ignore-vuln turns the Python step red, and dropping
--audit-level to moderate turns the npm one red on the devalue advisory.
npm audits production dependencies only. All seven findings in the full tree
are build- or test-time: the esbuild advisory needs a vite dev server exposed
to the internet, and nanoid's infinite loop needs a custom generator called
with size 0, which postcss does not do. None are in the 91 kB bundle the panel
serves, so failing on them would be noise that trains people to ignore the
job.
The starlette entries are the reason the job is not "fail on everything":
fastapi 0.115.12 pins starlette<0.47.0 and the last four fixes need 0.49.1
through 1.3.1, so clearing them is a jump to fastapi 0.141.x and is upstream's
call, not a drive-by. Four of the seven are reachable in principle, which the
comment on the job sets out. The panel answers only on
userbot.workstation.internal with no public route, which is what keeps those
four from being an internet-facing DoS.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
prettier, ruff, yamllint and hadolint were the only CI tools still called bare,
straight off whatever the runner happened to have installed. Pin them in
tool-versions.env like the other three and install them the same way, so the
versions Renovate moves are the versions CI runs.
Each pinned version equals what is already on the runner, so this changes what
CI does not at all today. It changes what CI does on a rebuilt runner: the
pinned one gets installed over the drift.
The four need four different mechanisms, which is why this is not one pattern:
hadolint a bare binary per platform, like actionlint
ruff,
yamllint PyPI wheels, unpacked by uv
prettier an npm tarball, unpacked by tar
prettier is the awkward one. Its entry point requires ../package.json relative
to its own real path, so copying the single file out -- which is what every
other installer here does -- yields a module-not-found at the first run. It
keeps its package directory in a versioned one next to a relative symlink, and
the tarball ships bin/ without the exec bit, so that needs chmod too.
hadolint's release names one platform uname-style and the other Go-style
(x86_64 but arm64), which 404s on the first architecture if you assume
otherwise.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
svelte-check sits in devDependencies at ^4.7.3 and nothing in the repository
ever invoked it, so the type errors it reports had no path to a human. Point a
script at it and run it in the frontend job, and it is clean: 0 errors, 0
warnings.
It shares the one `npm ci` with the test step. A second install would have
doubled the slowest part of the job to learn exactly the same thing.
The lockfile is untouched, because scripts are not part of what it pins.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The panel ships 25 pytest tests and 2 vitest tests. Nothing executed them:
there was no job, no local dev loop, and nothing that would have noticed when
one of them rotted. They pass, and they are 8 seconds of work, which is the
argument for having them.
Both jobs mirror how the image is built rather than how a developer would run
them by hand: `npm ci` because that is what the Dockerfile does, so the tree
under test is the tree that ships, and requirements-dev.txt through uv, which
is now pinned like the other CI tools.
The backend job runs `python -m pytest`, not bare `pytest`. The tests import
`app.*` relative to the backend directory, and only the `-m` form puts the
working directory on sys.path.
ruff format --check joins ruff check in the lint job. It needed 0ae0df7 to be
addable, since ten files disagreed with the style ruff.toml has always
declared.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`kubectl apply --dry-run=server` persists nothing, but it does execute the
admission webhooks of the real API server. The validate job runs on
pull_request with no branch guard, so anyone able to open a PR could run
arbitrary manifest content through cert-manager and Traefik in production.
Limit the step to pushes to main. A pull request loses nothing by it: only
main is ever deployed, and this job has to complete successfully before the
deploy workflow is allowed to start, so a bad CRD is still caught before
anything reaches the cluster -- on the push instead of on the PR.
The skip is announced rather than silent, so a missing server-side pass does
not read as a pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`kubectl rollout undo` restores the previous ReplicaSet's pod template
verbatim. While that template names a tag, the rollback does not roll back
the image: the tag has already moved, so the reverted pod pulls the very
build that just failed and the cluster stays broken. The safety net added
in 1505b63 therefore could not recover from a bad image.
Pin the digest at apply time. A digest is not knowable when a manifest is
written, so render_pinned resolves it on the way into the cluster and the
digest is never committed. Git keeps a readable `:prod`, Renovate keeps
seeing exactly the manifests it saw before, and the previous revision of
each workload now holds the digest that was actually serving, so undo
restores those exact bytes.
imagePullPolicy is dropped from the manifests rather than set to
IfNotPresent: a reference that is not `:latest` already defaults to it, and
that is what the Kubernetes docs ask for alongside a digest.
An unresolvable image is fatal instead of a warning, because carrying on
would quietly apply a mutable tag again.
restart_stale_images keeps its comparison but is no longer how a rebuild
reaches the cluster -- the pinned template rolls out on its own now. What
is left is a drift check for hand-run `kubectl set image`, so it matches
the container by repository: a pod's status now reports `repo@sha256:...`
while the manifest still says `:prod`.
The build job stops pushing `:latest` altogether, which removes the tag
that a dev branch could otherwise move under a prod deploy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds three lint jobs (actionlint, shellcheck, compose) and a server-side
dry-run of the active manifests. Previously the only k8s check was
kubeconform, which has no schemas for CRDs, so every IngressRoute,
Certificate, PrometheusRule and Middleware was silently skipped.
The server-side pass needs the live API server because that is the only
place the real CRD schemas and the cert-manager / Traefik admission
webhooks exist. It is scoped to services carrying a k8s/active marker,
since dry-run needs the target namespace to exist. userbot/ is excluded
from shellcheck: it is a git subtree, and linting upstream's scripts would
let a routine subtree pull turn the deploy gate red on code we do not own.
kubeconform, shellcheck and actionlint are now installed from pinned
versions in tool-versions.env rather than picked up from the runner's
PATH. The Compose helper is shared with the deploy workflow so both
check the same file set the same way.
Deploy workflow uses git-tracked manifests, DISABLED flag and kustomize overlays; add webinar-checker metrics with ServiceMonitor and alerts; upgrade shared postgres to 17 with statuspage DB and probes/resources.
Ruff and yamllint ship in Arch repos, drop their container pulls. Prettier keeps the node container but mounts persistent npm cache. Hadolint and kubeconform stay containerized (AUR-only / hermetic pin).
Drop duplicated .github/workflows (helm blocks ported to .gitea deploy first). Add ci concurrency with cancel on branches, pin checkout to SHA and prettier to 3.9.8, registry layer cache for image builds. renovate-run gains config validation and dry-run input.
- ensure_prerequisites runs on startup, not per-request; kube config
errors surface as 503 PanelError
- serialize provisioning with a lock; drop per-endpoint prereq checks
- guard SPA fallback against path traversal (relative_to)
- add backend tests for auth flow, k8s service, spa routing; ci comment
for legacy userbot deployments
forust
