Compare commits

..
Author SHA1 Message Date
forust 5b3ef39102 template for lampa service 2026-01-21 11:36:36 +01:00
487 changed files with 19503 additions and 18485 deletions

No files matched your search

-191
View File
@@ -1,191 +0,0 @@
# Инструкция: Анализ хранилища Kubernetes и настройка NFS
## Цель
Проанализировать текущую конфигурацию хранилища Kubernetes и подготовить план внедрения NFS StorageClass для сохранения данных при удалении namespace.
## 1. Собрать информацию о кластере
### 1.1. Версия Kubernetes и тип дистрибутива
```bash
kubectl version --short
# или
kubectl version
```
Определить, используется ли k3s, k8s, microk8s и т.д.:
```bash
# Проверить наличие k3s
which k3s
# Проверить процесс
ps aux | grep -E 'kube|k3s'
```
### 1.2. StorageClass
```bash
kubectl get storageclass -o wide
```
Запомнить:
- `PROVISIONER` — какой драйвер используется
- `RECLAIMPOLICY` — Delete или Retain
- Какой StorageClass помечен как `(default)`
### 1.3. Существующие PV и PVC
```bash
kubectl get pv -o wide
kubectl get pvc --all-namespaces
```
Посмотреть, какие PVC привязаны к каким PV, и какой reclaimPolicy у PV.
### 1.4. Нода и диски
```bash
# Список нод
kubectl get nodes -o wide
# На каждой ноде (через ssh или локально):
lsblk
df -h
cat /etc/fstab
```
Определить:
- Есть ли отдельный раздел/диск для данных
- Куда смонтированы разделы
- Сколько свободного места
- Есть ли монтирование NTFS-разделов (как `/media/forust/Programs`)
### 1.5. Где local-path хранит данные (для k3s)
```bash
ls -la /var/lib/rancher/k3s/storage/ 2>/dev/null
# или для microk8s
ls -la /var/snap/microk8s/common/ 2>/dev/null
```
## 2. Анализ: сохраняются ли данные при удалении namespace?
| Сценарий | Результат |
|---|---|
| `kubectl delete ns <ns>` | Все PVC в namespace удаляются |
| PVC → PV c `reclaimPolicy: Delete` | PV и данные удалены |
| PVC → PV c `reclaimPolicy: Retain` | PV остаётся (статус Released), данные целы |
**Вывод:** Если reclaimPolicy в StorageClass = `Delete`, то данные **пропадут**. Если `Retain` — сохранятся.
## 3. План внедрения NFS
### 3.1. Проверить, установлен ли NFS
```bash
which nfsstat exportfs mount.nfs
systemctl status nfs-server 2>/dev/null || systemctl status nfs-kernel-server 2>/dev/null
```
### 3.2. Выбрать директорию для NFS-экспорта
Варианты (выбрать подходящий):
- `/var/lib/k8s-nfs/` — на корневом разделе
- `<путь к отдельному разделу>/k8s-nfs/` — если есть отдельный диск/раздел
- Не рекомендуется использовать NTFS-раздел (проблемы с правами и производительностью)
Требования:
- Файловая система: ext4 или xfs (не ntfs!)
- Достаточно свободного места
- Права: `755`, владелец root
### 3.3. Установить NFS-сервер
```bash
# Debian/Ubuntu
apt update && apt install -y nfs-kernel-server
# RHEL/Fedora
dnf install -y nfs-utils
```
### 3.4. Настроить экспорт
Создать директорию:
```bash
mkdir -p /var/lib/k8s-nfs
chmod 755 /var/lib/k8s-nfs
```
Добавить в `/etc/exports`:
```
/var/lib/k8s-nfs *(rw,sync,no_subtree_check,no_root_squash)
```
Применить:
```bash
exportfs -rav
```
Проверить:
```bash
showmount -e localhost
```
### 3.5. Выбрать способ интеграции с Kubernetes
#### Вариант A: nfs-subdir-external-provisioner (проще)
```bash
helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/
helm install nfs-provisioner nfs-subdir-external-provisioner/nfs-subdir-external-provisioner \
--namespace kube-system \
--set nfs.server=127.0.0.1 \
--set nfs.path=/var/lib/k8s-nfs \
--set storageClass.name=nfs \
--set storageClass.defaultClass=false \
--set storageClass.reclaimPolicy=Retain
```
#### Вариант B: NFS CSI Driver
```bash
helm repo add csi-driver-nfs https://raw.githubusercontent.com/kubernetes-csi/csi-driver-nfs/master/charts
helm install csi-driver-nfs csi-driver-nfs/csi-driver-nfs --namespace kube-system
```
После установки CSI драйвера создать StorageClass:
```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: nfs
provisioner: nfs.csi.k8s.io
parameters:
server: 127.0.0.1
share: /var/lib/k8s-nfs
reclaimPolicy: Retain
volumeBindingMode: Immediate
```
### 3.6. Проверить результат
```bash
kubectl get storageclass
kubectl get pods -n kube-system | grep -E 'nfs|provisioner'
```
## 4. Итоговая конфигурация
После внедрения в кластере будет два StorageClass:
| Имя | Provisioner | ReclaimPolicy | Назначение |
|---|---|---|---|
| `local-path` (default) | rancher.io/local-path | Delete | Временные данные, stateless |
| `nfs` | nfs-subdir-external-provisioner или nfs.csi.k8s.io | Retain | Данные, которые нужно сохранять |
**Главное преимущество:** PVC c `storageClassName: nfs` при удалении namespace сохраняют данные на диске, так как NFS-провизор использует `reclaimPolicy: Retain` или файлы физически остаются в NFS-экспорте.
## 5. Ответы на частые вопросы
**В:** Не упадёт ли local-path при установке NFS?
**О:** Нет, они независимы. local-path продолжает работать как обычно.
**В:** Данные NFS и local-path будут на одном диске?
**О:** Да, можно настроить оба на одном разделе, в разных каталогах.
**В:** Что если у меня несколько нод?
**О:** NFS сервер нужно поднять на одной ноде, а с других нод должна быть доступна шари. Для multi-node лучше использовать отдельный сервер или distributed storage (Longhorn, Rook/Ceph).
**В:** Можно ли использовать существующий NTFS-раздел для NFS?
**О:** Не рекомендуется — NTFS не поддерживает права Linux (no_root_squash не сработает корректно), возможны проблемы с блокировками и производительностью.
-24
View File
@@ -1,24 +0,0 @@
root = true
[*]
indent_style = space
indent_size = 2
end_of_line = lf
charset = utf-8
trim_trailing_whitespace = true
insert_final_newline = true
[*.{yml,yaml}]
indent_size = 2
[*.{json,jsonc}]
indent_size = 2
[*.md]
trim_trailing_whitespace = false
[*.py]
indent_size = 4
[{Makefile,makefile}]
indent_style = tab
-42
View File
@@ -1,42 +0,0 @@
# EDU ownership handoff
## Current status
EDU PR #1 merged at 2026-10-07 08:04:30 UTC. Main release `5094952464ce315130839303985fd04d721bc1f2` passed CI run 1585 and deploy run 1586. The workstation checkout `/srv/edu-master` is at that SHA. The release changed the application image digests:
- Session keeper: `sha256:1e59473bd40fe4c22622017d808a8927a68788275fe073dc23d718c44b2fd5dd`
- Webinar checker: `sha256:987d9bf0770272766523ea5b94c7f3f849175d737551d46591ae55e058cf9f12`
The live workloads remain healthy in context `Default`, namespace `edu-master`. Both health and live probes return 200. Redis AUTH passes, session TTL is 1178 seconds, the delivery backlog is zero, all nine EDU alert rules are healthy, and the scrape target is UP. The unauthorized-pod Redis check passed. The Redis PVC UID and Secret UID and values, including the Fernet key, match their pre-release state.
The homelab EDU active marker was present after the EDU deployment. It was moved to the private snapshot as `homelab-k8s-active.marker` while holding `/tmp/homelab-apply.lock`. The homelab checkout at `/srv/homelab` is at `5f9354b` and has the tracked marker deletion. Its deploy preflight blocks a dirty checkout until this removal is reconciled. Preserve private ignored configuration when syncing that checkout.
The remaining homelab change is PR #105, branch `feat/edu-handoff-matrix`, based on `codex/ci-visible-checks`. Its eight protected checks passed. Renovate runs 1587 and 1588 passed. Image publishing was skipped for the PR. The EDU runtime changes are in PR #3 from `fix/handoff-runtime` to `main`; CI run 1589 is in progress. Those runtime changes have not been released.
## Approval gate and next steps
PR #99 must merge before PR #105 can target `main`. A merge attempt for PR #99 returned HTTP 405 because it needs one approval; the protected branch has `required_approvals=1` and whitelist approval is enabled. This approval gate prevents the remaining transfer steps.
After the required approval:
1. Merge PR #99.
2. Retarget PR #105 to `main`. Complete CI and review, then approve and merge it.
3. Under the homelab apply lock, sync `/srv/homelab` to the merged removal. Preserve private ignored configuration and keep the active marker removed. Confirm the deploy preflight is clean.
4. Merge the EDU runtime PR after its CI and review pass. The main-push CI run must complete successfully before its exact SHA can deploy.
5. Verify the new release SHA, image digests, workload health, Redis AUTH and TTL, backlog, PVC and Secret identity, and monitoring. Record the results in the EDU PR.
`AUTODEPLOY=false` is explicitly configured. The EDU repository path and port secrets are confirmed, and `EDU_KUBE_CONTEXT=Default` is configured as a repository variable. Keep deployment and registry credentials outside Git. Never run both homelab and EDU deployment paths at the same time.
## Change summary
The homelab PR removes the EDU subtree, deployment and image selection, rollback and verification cases, route probes, Renovate references, and external-image exceptions. It adds a serial dynamic matrix for the three homelab images. Each job builds an image or reuses a matching immutable digest. The final job checks all image results and publishes full-SHA tags and the existing release artifact only after they pass. PRs do not publish images. The protected check names from PR #99 are preserved. PR #100's service-metrics work is independent of this handoff.
The EDU runtime PR adds the Playwright service manifest, reconciles Redis storage and Secret reload annotations, and adds pre-apply Redis backup and identity checks. It verifies application endpoints, Redis AUTH, session TTL, metrics, and all nine vmalert rules. Rollback checks workload and application health and reports when manual recovery is needed. Its deployment guard rejects an unexpected or dirty checkout and refuses deployment while either legacy homelab EDU marker exists.
## Rollback and limits
The private snapshot is `/home/forust/.local/state/edu-master-deploy/handoff-20261007T080838Z` on the workstation. It contains the pre-handoff Redis RDB and recovery data. RDB checksum verification confirmed twelve keys. Keep the snapshot outside Git. For an EDU release failure, restore the saved Kubernetes resources and inspect application health. The rollback does not automatically restore the Redis RDB; restore old Redis data only when recovery requires it.
For an ownership rollback, stop EDU deployment triggers first, restore the reviewed homelab source and marker, then reapply recorded immutable image digests. Verify both workload and application health. Never delete or recreate the Redis PVC.
The initial EDU release and the homelab marker move are complete. PR #99 approval and merge, PR #105 retarget and merge, homelab checkout reconciliation, EDU runtime PR merge, and release of those runtime changes remain pending. Synthetic Telegram delivery and Alertmanager-to-Telegram notification were not tested.
-10
View File
@@ -1,10 +0,0 @@
# actionlint configuration. Passed explicitly from the ci workflow:
# actionlint -config-file .gitea/actionlint.yaml .gitea/workflows/*.yaml
#
# The self-hosted act_runner registers custom labels that actionlint cannot know
# about, so declare them here instead of silencing the whole runner-label check.
self-hosted-runner:
labels:
- arch
- homelab
- prod
-3
View File
@@ -1,3 +0,0 @@
{
"postgres": ["authentik", "gitea", "immich", "n8n", "netbox", "netronome"]
}
-130
View File
@@ -1,130 +0,0 @@
# Homelab CI/CD
The native Gitea runner runs on **vps**; production runs on **workstation**.
Compose, workflow, shell, Python, formatting, YAML, Dockerfile and Kubernetes
checks appear as separate jobs. Jobs run on `homelab:host`, one at a time; the
build waits for every check to pass. CI and deploy runs also show a summary with
the release SHA, image build or reuse results, deploy mode, selected services,
and image digests. Failed runs keep a summary of completed image builds, stage
results, apply results, and recorded Kubernetes recovery. The final deploy
summary is in the smoke job; earlier jobs show the state observed at that time.
Apply success is separate from health and recovery. Update the installed
workstation controller with `setup-workstation.sh` when no deploy is running.
No job images or Kubernetes credentials
are needed on the VPS. Builds use one pinned BuildKit helper container. CI and deploy are separate workflows.
## Runner installation
Install Docker Engine with Compose and Buildx, Git, Python 3.11+, Bash, curl,
GNU tar/xz, flock and systemd using the host's package manager. Keep the existing
Gitea runner 3.0.2 binary at `/usr/local/bin/gitea-runner`.
From this checkout on the VPS:
```sh
sudo bash .gitea/runner/setup-runner.sh
```
The installer reuses `/var/lib/gitea-runner/.runner` and the existing service.
For a new host, install the same runner binary and register as `gitea-runner`
using the registration token interactively, label `homelab:host`, and working
directory `/var/lib/gitea-runner`; then rerun the installer. Tokens never belong
in this repository or command-line examples.
Pinned tools live in the runner user's `~/.cache/homelab-ci`; CI repairs version
drift there. Installations are locked. Buildx uses only the `homelab-ci` builder,
pushes directly to the registry, and caps retained local cache at 1 GiB with a
2 GiB free-space target. This is not a hard limit on peak build disk usage.
Nothing runs `docker system prune`, removes unrelated images, or deletes volumes.
## Workstation setup
As the existing SSH deploy user on workstation:
```sh
sudo loginctl enable-linger forust
bash .gitea/runner/setup-workstation.sh
```
The controller uses `/srv/homelab` as the persistent configuration tree and makes
a detached source worktree for each SHA. It never resets `/srv/homelab`, moves
local configuration, renames Compose projects, or changes volume names.
The installer records the current Kubernetes context and cluster UID in
`~/.config/homelab-deploy/environment`. Check these before installing.
Configure Gitea Actions Variables:
- `DEPLOY_HOST`, `DEPLOY_USER`, `DEPLOY_PORT`: the existing VPS-to-workstation SSH endpoint.
- `DEPLOY_KNOWN_HOSTS`: workstation's verified SSH host key entry for that endpoint.
- `AUTODEPLOY`: `false` initially; `true` enables deployment after successful main CI.
Keep `DEPLOY_SSH_KEY`, `REGISTRY_USERNAME` and `REGISTRY_PASSWORD` in Actions
Secrets. Legacy endpoint secrets remain accepted during migration. The Actions
token must have repository read and Actions read access for release downloads.
The deploy user's existing Docker registry authentication remains necessary.
## Releases and deployment
CI publishes `release-<full SHA>` as a Gitea artifact with all three owned image
digests and build input fingerprints. Unchanged images are reused only from a
successful main CI artifact, never from `:prod`. Expired artifacts cause CI to
rebuild images; they block deployment until CI is rerun.
Run deploy from main with `deploy_ref=main` or a checked SHA:
- `full`: required for the first baseline; reconcile all active components.
- `changed`: compare with the last fully successful production deploy.
- `plan`: validate configuration and show selection without changing production resources.
- `refresh_images=true`: explicitly refresh mutable third-party Compose tags.
The manual and automatic paths both require successful CI, a successful build
job and the exact SHA's release artifact. PRs cannot publish images or deploy.
Removed resources are reported and require explicit removal; no automatic prune.
Service dependencies are listed in `.gitea/deploy-dependencies.json`.
A workstation user systemd service holds the deploy lock across validation,
sequential apply, verification and smoke checks. SSH clients only submit/follow:
disconnecting or cancelling the Actions client does not kill production apply.
Retrying the same run ID does not start another apply. `ExecStopPost` recovers
interrupted runs before the unit finishes. Kubernetes rolls back to captured
revisions; configuration and persistent data are not reverted.
## Status and recovery
`--retry` repeats failed verification and smoke checks, never apply. Recovery
keeps a failed deploy out of the successful baseline, even after rollback.
On workstation (replace the numeric ID with Actions run ID and attempt):
```sh
python3 ~/.local/lib/homelab-deploy/controller.py status 123-1
python3 ~/.local/lib/homelab-deploy/controller.py recover 123-1 --retry
journalctl --user -u homelab-deploy@123-1
```
Runs live in `~/.local/state/homelab-deploy/runs`. Compose stores resolved configs
with restricted permissions; these may contain credentials and must never be
uploaded as CI artifacts. Stage logs print the exact manual recovery command
using `compose-before/<stack>.json`, the original project directory and project
name. Compose does not automatically roll back, and Nextcloud AIO's child
containers remain managed by AIO. Preserve its own backups for data recovery.
The controller retains twenty successful/planned runs and preserves failures.
Update the workstation dispatcher only when no deploy is running.
## Validation and migration rollback
```sh
python3 -m unittest discover -s tests -v
bash .gitea/tests/deploy-validation.sh
```
Test on a separate namespace before the initial production `full` run. Check a
failed rollout, interrupted SSH and repeated run ID, and verify that an isolated
service change does not upgrade unrelated Helm releases or Compose stacks.
To roll back the migration, disable autodeploy and finish or recover the remote
run first. Restore the runner config/unit from `.before-<timestamp>` backups,
reload systemd and restart the runner. Restore the prior workflows from Git.
Production data and persistent volumes stay where they were. Do not remove run
state or Compose recovery files until recovery is confirmed.
-11
View File
@@ -1,11 +0,0 @@
[worker.oci]
gc = true
reservedSpace = "256MB"
maxUsedSpace = "1GB"
minFreeSpace = "2GB"
[[worker.oci.gcpolicy]]
reservedSpace = "256MB"
maxUsedSpace = "1GB"
minFreeSpace = "2GB"
all = true
-10
View File
@@ -1,10 +0,0 @@
runner:
file: /var/lib/gitea-runner/.runner
capacity: 1
timeout: 5h
labels:
- homelab:host
cache:
enabled: false
container:
docker_host: unix:///var/run/docker.sock
-18
View File
@@ -1,18 +0,0 @@
[Unit]
Description=Gitea Actions runner
After=network-online.target docker.service
Wants=network-online.target
[Service]
User=gitea-runner
Group=gitea-runner
SupplementaryGroups=docker
WorkingDirectory=/var/lib/gitea-runner
Environment=PATH=/var/lib/gitea-runner/.cache/homelab-ci/bin:/usr/local/bin:/usr/bin:/bin
ExecStart=/usr/local/bin/gitea-runner daemon --config /etc/gitea-runner/config.yaml
Restart=on-failure
RestartSec=5
UMask=0077
[Install]
WantedBy=multi-user.target
-12
View File
@@ -1,12 +0,0 @@
[Unit]
Description=Homelab deploy %i
[Service]
Type=exec
EnvironmentFile=%h/.config/homelab-deploy/environment
ExecStart=/usr/bin/python3 %h/.local/lib/homelab-deploy/controller.py execute %i
ExecStopPost=/usr/bin/python3 %h/.local/lib/homelab-deploy/controller.py recover %i
RuntimeMaxSec=5h
TimeoutStopSec=135min
KillMode=control-group
UMask=0077
-48
View File
@@ -1,48 +0,0 @@
#!/usr/bin/env bash
# Native host runner, with pinned user-space tools and no extra CI images.
set -euo pipefail
here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
[ "$(id -u)" -eq 0 ] || { echo 'Run with sudo on the runner host' >&2; exit 1; }
for tool in docker curl python3 git tar xz flock runuser systemctl; do
command -v "$tool" >/dev/null || { echo "Install missing prerequisite: $tool" >&2; exit 1; }
done
docker info >/dev/null
docker compose version >/dev/null
docker buildx version >/dev/null
id gitea-runner >/dev/null 2>&1 || useradd --system --create-home --home-dir /var/lib/gitea-runner --shell /usr/bin/bash gitea-runner
# Reuse the established service account and runner registration.
runner_home="$(getent passwd gitea-runner | cut -d: -f6)"
[ "$runner_home" = /var/lib/gitea-runner ] || { echo 'Unexpected runner home; inspect the existing service first' >&2; exit 1; }
runuser -u gitea-runner -- docker info >/dev/null || { echo "The runner user needs access to Docker before setup" >&2; exit 1; }
command -v gitea-runner >/dev/null || { echo 'Install gitea-runner 3.0.2 at /usr/local/bin/gitea-runner first' >&2; exit 1; }
mkdir -p /etc/gitea-runner
stamp="$(date -u +%Y%m%dT%H%M%SZ)"
for existing in /etc/gitea-runner/config.yaml /etc/systemd/system/gitea-runner.service; do
[ ! -f "$existing" ] || cp -p "$existing" "$existing.before-$stamp"
done
scratch="$(mktemp -d)"
trap 'rm -rf "$scratch"' EXIT
chmod 755 "$scratch"
install -m 0644 "$here/../workflows/install-ci-tools.sh" "$here/../workflows/tool-versions.env" "$scratch/"
runuser -u gitea-runner -- bash "$scratch/install-ci-tools.sh"
install -m 0644 "$here/config.yaml" /etc/gitea-runner/config.yaml
python3 - <<'PYLABELS'
import json
from pathlib import Path
registration = Path('/var/lib/gitea-runner/.runner')
if registration.exists():
labels = json.loads(registration.read_text()).get('labels', [])
labels = [label for label in labels if isinstance(label, str) and label.split(':')[0] != 'homelab']
labels.append('homelab:host')
config = Path('/etc/gitea-runner/config.yaml')
config.write_text(config.read_text().replace(' - homelab:host', '\n'.join(' - ' + json.dumps(label) for label in labels)))
PYLABELS
install -m 0644 "$here/gitea-runner.service" /etc/systemd/system/gitea-runner.service
if [ ! -f /var/lib/gitea-runner/.runner ]; then
echo 'Register once as gitea-runner with homelab:host before starting the service.'
exit 0
fi
systemctl daemon-reload
systemctl enable --now gitea-runner.service
systemctl restart gitea-runner.service
echo "Runner ready. Configuration backups: *.before-$stamp"
-32
View File
@@ -1,32 +0,0 @@
#!/usr/bin/env bash
# Run as the existing deploy user on workstation. Never resets the working tree.
set -euo pipefail
here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
repo="${HOMELAB_REPO:-/srv/homelab}"
for tool in python3 git kubectl helm docker flock timeout; do
command -v "$tool" >/dev/null || { echo "Install missing dependency: $tool" >&2; exit 1; }
done
[ -d "$repo/.git" ] || { echo "Missing deploy checkout: $repo" >&2; exit 1; }
[[ "$repo" =~ ^/[A-Za-z0-9_./-]+$ ]] || { echo 'Deploy path must be absolute and contain no whitespace' >&2; exit 1; }
if [ "$(loginctl show-user "$USER" -p Linger --value)" != yes ]; then
echo "Run once: sudo loginctl enable-linger $USER" >&2
exit 1
fi
config="${XDG_CONFIG_HOME:-$HOME/.config}/homelab-deploy"
mkdir -p "$config" "$HOME/.local/lib/homelab-deploy" "$HOME/.config/systemd/user"
chmod 700 "$config"
if [ ! -f "$config/environment" ]; then
context="$(kubectl config current-context)"
cluster_uid="$(kubectl get namespace kube-system -o jsonpath='{.metadata.uid}')"
printf 'HOMELAB_REPO=%s\nKUBE_CONTEXT=%s\nEXPECTED_CLUSTER_UID=%s\n' "$repo" "$context" "$cluster_uid" >"$config/environment"
chmod 600 "$config/environment"
fi
# Do not replace a dispatcher while an existing deploy uses it.
if systemctl --user list-units 'homelab-deploy@*' --state=running --no-legend | grep -q .; then
echo 'An existing deploy is running; wait before updating the controller' >&2
exit 1
fi
install -m 0755 "$here/../workflows/deploy-controller.py" "$HOME/.local/lib/homelab-deploy/controller.py"
install -m 0644 "$here/homelab-deploy@.service" "$HOME/.config/systemd/user/homelab-deploy@.service"
systemctl --user daemon-reload
echo 'Controller ready. Run a checked main SHA in full mode for the initial baseline.'
-94
View File
@@ -1,94 +0,0 @@
#!/usr/bin/env bash
# Local regressions only: kubectl is mocked and Docker is used for config parsing.
set -euo pipefail
repo="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
scratch="$(mktemp -d)"
trap 'rm -rf "$scratch"' EXIT
mkdir -p "$scratch/repo/app" "$scratch/repo/postgres" "$scratch/repo/netbird" "$scratch/repo/renovate"
git -C "$scratch/repo" init -q
for file in app/compose.yaml postgres/shared-compose.yaml netbird/client.compose.yaml renovate/renovate-compose.yaml; do
touch "$scratch/repo/$file"
done
git -C "$scratch/repo" add .
# shellcheck source=../workflows/compose-lint.sh
source "$repo/.gitea/workflows/compose-lint.sh"
actual="$(cd "$scratch/repo" && compose_files)"
expected=$'app/compose.yaml\nnetbird/client.compose.yaml\npostgres/shared-compose.yaml\nrenovate/renovate-compose.yaml'
[ "$actual" = "$expected" ] || { echo 'Compose discovery missed a file' >&2; exit 1; }
cat >"$scratch/compose.yaml" <<'YAML'
services:
example:
image: busybox:1.37.0
environment:
REQUIRED: ${HOMELAB_TEST_REQUIRED:?required for this regression}
YAML
unset HOMELAB_TEST_REQUIRED
if validate_compose_file "$scratch/compose.yaml" >"$scratch/config.log" 2>&1; then
echo 'Full Compose validation accepted a missing variable' >&2
exit 1
fi
grep -q 'required for this regression' "$scratch/config.log"
HOMELAB_TEST_REQUIRED=present validate_compose_file "$scratch/compose.yaml"
cat >"$scratch/resources.json" <<'JSON'
{"kind":"List","items":[
{"kind":"Deployment","metadata":{"namespace":"app"},"spec":{"template":{"spec":{
"containers":[{"envFrom":[{"secretRef":{"name":"credentials"}},{"secretRef":{"name":"optional","optional":true}}],"env":[{"valueFrom":{"secretKeyRef":{"name":"credentials","key":"password"}}}]}],
"initContainers":[{"envFrom":[{"secretRef":{"name":"init"}}]}],
"imagePullSecrets":[{"name":"registry"}],
"volumes":[{"secret":{"secretName":"mounted"}},{"projected":{"sources":[{"secret":{"name":"projected"}},{"secret":{"name":"optional-projected","optional":true}}]}}]
}}}},
{"kind":"CronJob","metadata":{},"spec":{"jobTemplate":{"spec":{"template":{"spec":{"containers":[{"envFrom":[{"secretRef":{"name":"cron"}}]}]}}}}}},
{"kind":"IngressRoute","metadata":{"namespace":"app"},"spec":{"tls":{"secretName":"controller-issued-tls"}}}
]}
JSON
actual="$(jq -r -f "$repo/.gitea/workflows/secret-references.jq" "$scratch/resources.json" | sort)"
expected=$'app credentials\napp init\napp mounted\napp projected\napp registry\ndefault cron'
[ "$actual" = "$expected" ] || { echo "Unexpected Secret references: $actual" >&2; exit 1; }
REPO="$repo"
# shellcheck source=../workflows/deploy-lib.sh
source "$repo/.gitea/workflows/deploy-lib.sh"
K8S_MANIFESTS=("$scratch/resources.json")
KUSTOMIZE_APPS=()
# No live cluster access. Reject credentials in app even if they exist elsewhere.
kubectl() {
case "$1" in
create) cat "$scratch/resources.json" ;;
get)
if [ "$3" = credentials ] && [ "$5" = app ]; then
return 1
fi
return 0
;;
*) echo "Unexpected kubectl invocation: $*" >&2; return 1 ;;
esac
}
if check_referenced_secrets >"$scratch/secrets.log"; then
echo 'Namespace-scoped Secret check accepted a missing Secret' >&2
exit 1
fi
grep -q 'MISSING OR UNREADABLE: app/credentials' "$scratch/secrets.log"
# API/rendering errors must not produce an empty reference list and pass.
kubectl() { return 1; }
if ! skip_uninstalled_vmagent_crd "$REPO/prometheus-stack/k8s/vmagent.yaml"; then
echo 'VMAgent preflight did not skip an uninstalled CRD' >&2
exit 1
fi
kubectl() { return 0; }
if skip_uninstalled_vmagent_crd "$REPO/prometheus-stack/k8s/vmagent.yaml"; then
echo 'VMAgent preflight skipped an installed CRD' >&2
exit 1
fi
if skip_uninstalled_vmagent_crd "$REPO/prometheus-stack/k8s/victoria.yaml"; then
echo 'VMAgent preflight skipped an unrelated manifest' >&2
exit 1
fi
kubectl() { return 1; }
if check_referenced_secrets >"$scratch/secrets.log"; then
echo 'Secret check accepted a failed manifest render' >&2
exit 1
fi
printf '%s\n' 'Deploy validation regressions passed.'
-520
View File
@@ -1,520 +0,0 @@
name: ci
"on":
push:
branches:
- main
pull_request: null
workflow_dispatch: null
permissions:
contents: read
actions: read
concurrency:
group: ci-${{ github.ref }}
cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
jobs:
compose:
name: Compose
runs-on: homelab
timeout-minutes: 15
steps:
- name: Checkout repository
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
id: source
- name: Validate Compose files
shell: bash
run: |
set -euo pipefail
source .gitea/workflows/compose-lint.sh
mapfile -t safe_flags < <(compose_safe_flags)
echo "docker compose config ${safe_flags[*]-}"
mapfile -t files < <(compose_files)
if [ "${#files[@]}" -eq 0 ]; then
echo "No Compose files found."
exit 0
fi
failed=0
for f in "${files[@]}"; do
if ! out="$(validate_compose_file "$f" ${safe_flags[@]+"${safe_flags[@]}"} 2>&1)"; then
failed=1
echo "::error file=${f}::$(printf '%s' "$out" | head -1)"
fi
done
if [ "$failed" -ne 0 ]; then
echo "Compose validation failed."
exit 1
fi
echo "checked ${#files[@]} Compose file(s)"
id: check
- name: Write the job result
if: always()
env:
SUMMARY_CHECK: Compose
SUMMARY_RESULT: ${{ job.status }}
SUMMARY_FAILED_STEP:
${{ steps.check.conclusion == 'failure' && 'Check or image build' || steps.source.conclusion == 'failure'
&& 'Source checkout' || '' }}
shell: bash
run: |
if [ -f .gitea/workflows/release.py ]; then
python3 .gitea/workflows/release.py check-summary
elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
printf '## %s\n\n- Result: **%s**\n- Failed step: %s\n' "$SUMMARY_CHECK" "$SUMMARY_RESULT" "$SUMMARY_FAILED_STEP" >>"$GITHUB_STEP_SUMMARY" || true
fi
workflows:
name: Workflows
runs-on: homelab
timeout-minutes: 15
steps:
- name: Checkout repository
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
id: source
- name: Prepare pinned tools
shell: bash
run: |
set -euo pipefail
tools_dir="$(bash .gitea/workflows/install-ci-tools.sh actionlint shellcheck)"
echo "$tools_dir" >> "$GITHUB_PATH"
id: tools
- name: Lint Gitea Actions workflows with actionlint
shell: bash
run: |
set -euo pipefail
actionlint -config-file .gitea/actionlint.yaml -color .gitea/workflows/*.yaml
id: check
- name: Write the job result
if: always()
env:
SUMMARY_CHECK: Workflows
SUMMARY_RESULT: ${{ job.status }}
SUMMARY_FAILED_STEP:
${{ steps.check.conclusion == 'failure' && 'Check or image build' || steps.tools.conclusion == 'failure'
&& 'Tool setup' || steps.source.conclusion == 'failure' && 'Source checkout' || '' }}
shell: bash
run: |
if [ -f .gitea/workflows/release.py ]; then
python3 .gitea/workflows/release.py check-summary
elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
printf '## %s\n\n- Result: **%s**\n- Failed step: %s\n' "$SUMMARY_CHECK" "$SUMMARY_RESULT" "$SUMMARY_FAILED_STEP" >>"$GITHUB_STEP_SUMMARY" || true
fi
shell:
name: Shell
runs-on: homelab
timeout-minutes: 15
steps:
- name: Checkout repository
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
id: source
- name: Prepare pinned tools
shell: bash
run: |
set -euo pipefail
tools_dir="$(bash .gitea/workflows/install-ci-tools.sh shellcheck jq)"
echo "$tools_dir" >> "$GITHUB_PATH"
id: tools
- name: Lint shell scripts with ShellCheck
shell: bash
run: |
set -euo pipefail
mapfile -t scripts < <(
git ls-files '*.sh' ':(glob)**/*.bash'
)
if [ "${#scripts[@]}" -eq 0 ]; then
echo "No shell scripts found."
exit 0
fi
shellcheck --external-sources --source-path=SCRIPTDIR --severity=style "${scripts[@]}"
bash .gitea/tests/deploy-validation.sh
id: check
- name: Write the job result
if: always()
env:
SUMMARY_CHECK: Shell
SUMMARY_RESULT: ${{ job.status }}
SUMMARY_FAILED_STEP:
${{ steps.check.conclusion == 'failure' && 'Check or image build' || steps.tools.conclusion == 'failure'
&& 'Tool setup' || steps.source.conclusion == 'failure' && 'Source checkout' || '' }}
shell: bash
run: |
if [ -f .gitea/workflows/release.py ]; then
python3 .gitea/workflows/release.py check-summary
elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
printf '## %s\n\n- Result: **%s**\n- Failed step: %s\n' "$SUMMARY_CHECK" "$SUMMARY_RESULT" "$SUMMARY_FAILED_STEP" >>"$GITHUB_STEP_SUMMARY" || true
fi
formatting:
name: Formatting
runs-on: homelab
timeout-minutes: 15
steps:
- name: Checkout repository
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
id: source
- name: Prepare pinned tools
shell: bash
run: |
set -euo pipefail
tools_dir="$(bash .gitea/workflows/install-ci-tools.sh prettier)"
echo "$tools_dir" >> "$GITHUB_PATH"
id: tools
- name: Check formatting with Prettier
shell: bash
run: |
set -euo pipefail
mapfile -t prettier_files < <(
git ls-files \
| grep -E '\.(md|json|ya?ml|html|css)$' \
| grep -Ev '^(\.docs/|\.zed/|errorpages/html/|homepages/(forust_files|xdfnx_files)/)'
)
if [ "${#prettier_files[@]}" -eq 0 ]; then
echo "No Prettier-managed files found."
exit 0
fi
prettier --check --ignore-unknown "${prettier_files[@]}"
id: check
- name: Write the job result
if: always()
env:
SUMMARY_CHECK: Formatting
SUMMARY_RESULT: ${{ job.status }}
SUMMARY_FAILED_STEP:
${{ steps.check.conclusion == 'failure' && 'Check or image build' || steps.tools.conclusion == 'failure'
&& 'Tool setup' || steps.source.conclusion == 'failure' && 'Source checkout' || '' }}
shell: bash
run: |
if [ -f .gitea/workflows/release.py ]; then
python3 .gitea/workflows/release.py check-summary
elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
printf '## %s\n\n- Result: **%s**\n- Failed step: %s\n' "$SUMMARY_CHECK" "$SUMMARY_RESULT" "$SUMMARY_FAILED_STEP" >>"$GITHUB_STEP_SUMMARY" || true
fi
python:
name: Python and tests
runs-on: homelab
timeout-minutes: 15
steps:
- name: Checkout repository
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
id: source
- name: Prepare pinned tools
shell: bash
run: |
set -euo pipefail
tools_dir="$(bash .gitea/workflows/install-ci-tools.sh ruff jq)"
echo "$tools_dir" >> "$GITHUB_PATH"
id: tools
- name: Lint and format-check Python with Ruff
shell: bash
run: |
set -euo pipefail
ruff check . .gitea/workflows
ruff format --check . .gitea/workflows
python3 -m unittest discover -s tests -v
id: check
- name: Write the job result
if: always()
env:
SUMMARY_CHECK: Python and tests
SUMMARY_RESULT: ${{ job.status }}
SUMMARY_FAILED_STEP:
${{ steps.check.conclusion == 'failure' && 'Check or image build' || steps.tools.conclusion == 'failure'
&& 'Tool setup' || steps.source.conclusion == 'failure' && 'Source checkout' || '' }}
shell: bash
run: |
if [ -f .gitea/workflows/release.py ]; then
python3 .gitea/workflows/release.py check-summary
elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
printf '## %s\n\n- Result: **%s**\n- Failed step: %s\n' "$SUMMARY_CHECK" "$SUMMARY_RESULT" "$SUMMARY_FAILED_STEP" >>"$GITHUB_STEP_SUMMARY" || true
fi
yaml:
name: YAML
runs-on: homelab
timeout-minutes: 15
steps:
- name: Checkout repository
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
id: source
- name: Prepare pinned tools
shell: bash
run: |
set -euo pipefail
tools_dir="$(bash .gitea/workflows/install-ci-tools.sh yamllint)"
echo "$tools_dir" >> "$GITHUB_PATH"
id: tools
- name: Lint YAML syntax
shell: bash
run: |
set -euo pipefail
mapfile -t yaml_files < <(
git ls-files '*.yaml' '*.yml' \
':!node_modules/**' \
':!**/.venv/**'
)
if [ "${#yaml_files[@]}" -eq 0 ]; then
echo "No YAML files found."
exit 0
fi
yamllint -c .yamllint "${yaml_files[@]}"
id: check
- name: Write the job result
if: always()
env:
SUMMARY_CHECK: YAML
SUMMARY_RESULT: ${{ job.status }}
SUMMARY_FAILED_STEP:
${{ steps.check.conclusion == 'failure' && 'Check or image build' || steps.tools.conclusion == 'failure'
&& 'Tool setup' || steps.source.conclusion == 'failure' && 'Source checkout' || '' }}
shell: bash
run: |
if [ -f .gitea/workflows/release.py ]; then
python3 .gitea/workflows/release.py check-summary
elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
printf '## %s\n\n- Result: **%s**\n- Failed step: %s\n' "$SUMMARY_CHECK" "$SUMMARY_RESULT" "$SUMMARY_FAILED_STEP" >>"$GITHUB_STEP_SUMMARY" || true
fi
dockerfiles:
name: Dockerfiles
runs-on: homelab
timeout-minutes: 15
steps:
- name: Checkout repository
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
id: source
- name: Prepare pinned tools
shell: bash
run: |
set -euo pipefail
tools_dir="$(bash .gitea/workflows/install-ci-tools.sh hadolint)"
echo "$tools_dir" >> "$GITHUB_PATH"
id: tools
- name: Lint Dockerfiles
shell: bash
run: |
set -euo pipefail
mapfile -t dockerfiles < <(
git ls-files ':(glob)**/Dockerfile' ':(glob)**/Dockerfile.*'
)
if [ "${#dockerfiles[@]}" -eq 0 ]; then
echo "No Dockerfiles found."
exit 0
fi
hadolint -c .hadolint.yaml "${dockerfiles[@]}"
id: check
- name: Write the job result
if: always()
env:
SUMMARY_CHECK: Dockerfiles
SUMMARY_RESULT: ${{ job.status }}
SUMMARY_FAILED_STEP:
${{ steps.check.conclusion == 'failure' && 'Check or image build' || steps.tools.conclusion == 'failure'
&& 'Tool setup' || steps.source.conclusion == 'failure' && 'Source checkout' || '' }}
shell: bash
run: |
if [ -f .gitea/workflows/release.py ]; then
python3 .gitea/workflows/release.py check-summary
elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
printf '## %s\n\n- Result: **%s**\n- Failed step: %s\n' "$SUMMARY_CHECK" "$SUMMARY_RESULT" "$SUMMARY_FAILED_STEP" >>"$GITHUB_STEP_SUMMARY" || true
fi
kubernetes:
name: Kubernetes
runs-on: homelab
timeout-minutes: 15
steps:
- name: Checkout repository
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
id: source
- name: Prepare pinned tools
shell: bash
run: |
set -euo pipefail
tools_dir="$(bash .gitea/workflows/install-ci-tools.sh kubeconform)"
echo "$tools_dir" >> "$GITHUB_PATH"
id: tools
- name: Validate Kubernetes manifests against JSON schemas
shell: bash
run: |
set -euo pipefail
mapfile -t manifests < <(
git ls-files ':(glob)**/k8s/**/*.yaml' ':(glob)**/k8s/**/*.yml' \
| grep -Ev '(^|/)(kustomization\.ya?ml|.*\.example\.ya?ml|.*values\.ya?ml|patch-.*\.ya?ml)$'
)
if [ "${#manifests[@]}" -eq 0 ]; then
echo "No Kubernetes manifests found."
exit 0
fi
kubeconform \
-strict \
-ignore-missing-schemas \
-summary \
"${manifests[@]}"
id: check
- name: Write the job result
if: always()
env:
SUMMARY_CHECK: Kubernetes
SUMMARY_RESULT: ${{ job.status }}
SUMMARY_FAILED_STEP:
${{ steps.check.conclusion == 'failure' && 'Check or image build' || steps.tools.conclusion == 'failure'
&& 'Tool setup' || steps.source.conclusion == 'failure' && 'Source checkout' || '' }}
shell: bash
run: |
if [ -f .gitea/workflows/release.py ]; then
python3 .gitea/workflows/release.py check-summary
elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
printf '## %s\n\n- Result: **%s**\n- Failed step: %s\n' "$SUMMARY_CHECK" "$SUMMARY_RESULT" "$SUMMARY_FAILED_STEP" >>"$GITHUB_STEP_SUMMARY" || true
fi
image-plan:
needs: [compose, workflows, shell, formatting, python, yaml, dockerfiles, kubernetes]
if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main'
runs-on: homelab
timeout-minutes: 10
outputs:
matrix: ${{ steps.plan.outputs.matrix }}
steps:
- name: Checkout repository
id: source
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
with:
fetch-depth: 0
- name: Detect build inputs against successful CI
id: plan
env:
GITEA_TOKEN: ${{ github.token }}
run: python3 .gitea/workflows/release.py prepare --output build-plan.json
- name: Store the image plan
id: artifact
uses: actions/upload-artifact@c6a366c94c3e0affe28c06c8df20a878f24da3cf
with:
name: build-plan
path: build-plan.json
if-no-files-found: error
retention-days: 30
- name: Write the plan result
if: always()
env:
SUMMARY_CHECK: Image plan
SUMMARY_RESULT: ${{ job.status }}
SUMMARY_FAILED_STEP: >-
${{ steps.plan.conclusion == 'failure' && 'Build input detection' ||
steps.artifact.conclusion == 'failure' && 'Plan upload' ||
steps.source.conclusion == 'failure' && 'Source checkout' || '' }}
shell: bash
run: |
if [ -f .gitea/workflows/release.py ]; then
python3 .gitea/workflows/release.py check-summary
elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
printf '## Image plan\n\nResult: %s\n' "$SUMMARY_RESULT" >>"$GITHUB_STEP_SUMMARY" || true
fi
images:
name: Image (${{ matrix.name }})
needs: [image-plan]
if: needs.image-plan.result == 'success'
runs-on: homelab
timeout-minutes: 60
strategy:
max-parallel: 1
fail-fast: false
matrix: ${{ fromJSON(needs.image-plan.outputs.matrix || '{"include":[{"name":"inactive"}]}') }}
steps:
- name: Checkout repository
id: source
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
- name: Download the checked image plan
id: inputs
uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a
with:
name: build-plan
- name: Build or reuse this image
id: check
env:
IMAGE_NAME: ${{ matrix.name }}
REGISTRY_USERNAME: ${{ secrets.REGISTRY_USERNAME }}
REGISTRY_PASSWORD: ${{ secrets.REGISTRY_PASSWORD }}
run: python3 .gitea/workflows/release.py image --image "$IMAGE_NAME" --output image.json
- name: Store the image result
id: artifact
uses: actions/upload-artifact@c6a366c94c3e0affe28c06c8df20a878f24da3cf
with:
name: image-${{ matrix.name }}
path: image.json
if-no-files-found: error
retention-days: 30
- name: Write the job result
if: always()
env:
SUMMARY_CHECK: Image (${{ matrix.name }})
SUMMARY_RESULT: ${{ job.status }}
SUMMARY_FAILED_STEP: >-
${{ steps.check.conclusion == 'failure' && 'Build or tag images' ||
steps.artifact.conclusion == 'failure' && 'Artifact upload' ||
steps.inputs.conclusion == 'failure' && 'Artifact download' ||
steps.source.conclusion == 'failure' && 'Source checkout' || '' }}
shell: bash
run: |
if [ -f .gitea/workflows/release.py ]; then
python3 .gitea/workflows/release.py check-summary
elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
printf '## %s\n\n- Result: **%s**\n- Failed step: %s\n' "$SUMMARY_CHECK" "$SUMMARY_RESULT" "$SUMMARY_FAILED_STEP" >>"$GITHUB_STEP_SUMMARY" || true
fi
# Retain the build job name required by the immutable release deployment gate.
build:
needs: [image-plan, images]
runs-on: homelab
timeout-minutes: 15
steps:
- name: Checkout repository
id: source
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
- name: Download all image results
id: inputs
uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a
with:
path: artifacts
- name: Pin SHA tags and write the complete release
id: check
env:
REGISTRY_USERNAME: ${{ secrets.REGISTRY_USERNAME }}
REGISTRY_PASSWORD: ${{ secrets.REGISTRY_PASSWORD }}
run: >-
python3 .gitea/workflows/release.py finalize
--plan artifacts/build-plan/build-plan.json
- name: Store commit release
id: artifact
uses: actions/upload-artifact@c6a366c94c3e0affe28c06c8df20a878f24da3cf
with:
name: release-${{ github.sha }}
path: release.json
if-no-files-found: error
retention-days: 30
- name: Write the job result
if: always()
env:
SUMMARY_CHECK: Image release and SHA tags
SUMMARY_RESULT: ${{ job.status }}
SUMMARY_FAILED_STEP: >-
${{ steps.check.conclusion == 'failure' && 'Build or tag images' ||
steps.artifact.conclusion == 'failure' && 'Artifact upload' ||
steps.inputs.conclusion == 'failure' && 'Artifact download' ||
steps.source.conclusion == 'failure' && 'Source checkout' || '' }}
shell: bash
run: |
if [ -f .gitea/workflows/release.py ]; then
python3 .gitea/workflows/release.py check-summary
elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
printf '## %s\n\n- Result: **%s**\n- Failed step: %s\n' "$SUMMARY_CHECK" "$SUMMARY_RESULT" "$SUMMARY_FAILED_STEP" >>"$GITHUB_STEP_SUMMARY" || true
fi
-45
View File
@@ -1,45 +0,0 @@
#!/usr/bin/env bash
# Shared helpers for validating Compose files. Sourced both by steps in
# .gitea/workflows/ci.yaml and by deploy-lib.sh on the workstation.
#
# Two levels of checking, matching how the repo is structured:
#
# general every committed Compose file, active or not. Pure structure check:
# no ${VAR} interpolation, no .env lookup, no bind-mount path
# resolution. Disabled stacks deliberately have no .env in the repo
# and no values on the CI runner, so a full `config` run would fail on
# their `${VAR:?}` guards for reasons that have nothing to do with the
# change under review.
#
# full active stacks only, with interpolation and env-file resolution, so
# required variables and referenced files are actually resolved. Needs
# the gitignored .env files, so this only runs in the deploy workflow
# on the workstation.
#
# This file is meant to be sourced, not executed.
# All committed Compose files, including the ones deploy never starts.
compose_files() {
git ls-files \
'*compose.yaml' '*compose.yml'
}
# Prints the flags that turn `docker compose config` into the general check.
# Probed rather than hardcoded so an older Compose without --no-env-resolution
# still gets the flags it does support.
compose_safe_flags() {
local help flag
help="$(docker compose config --help 2>/dev/null || true)"
for flag in --no-interpolate --no-env-resolution --no-path-resolution; do
if printf '%s' "$help" | grep -q -- "$flag"; then
printf '%s\n' "$flag"
fi
done
}
# validate_compose_file <file> [extra docker compose config flags...]
validate_compose_file() {
local file="$1"
shift
docker compose -f "$file" config --quiet "$@"
}
-93
View File
@@ -1,93 +0,0 @@
#!/usr/bin/env python3
"""Resolve Compose images without changing project names or local bind paths."""
import json
import os
import re
import subprocess
import sys
from pathlib import Path
def output(*args, **kwargs):
return subprocess.check_output(args, text=True, **kwargs).strip() # noqa: S603
def resolve(reference):
if '@sha256:' in reference:
return reference
descriptor = json.loads(
output('docker', 'buildx', 'imagetools', 'inspect', reference, '--format', '{{json .Manifest}}')
)
digest = descriptor['digest']
if not re.fullmatch(r'sha256:[0-9a-f]{64}', digest):
raise ValueError(f'Invalid registry digest for {reference}')
# Strip tag only from the final path segment (registry ports are preserved).
repository = reference.rsplit('/', 1)
repository[-1] = repository[-1].split(':')[0]
return '/'.join(repository) + '@' + digest
def prepare(source_file):
config_repo = Path(os.environ['CONFIG_REPO'])
source_repo = Path(os.environ['REPO'])
directory = Path(os.environ['RUN_DIR'])
relative = source_file.relative_to(source_repo)
project_dir = config_repo / relative.parent
base = ['docker', 'compose', '--project-directory', str(project_dir), '-f', str(source_file)]
config = json.loads(output(*base, 'config', '--format', 'json', cwd=config_repo))
project = config['name']
previous_file = directory / 'previous.json'
previous = json.loads(previous_file.read_text()) if previous_file.exists() else {}
images_file = directory / 'compose-images.json'
locks = json.loads(images_file.read_text()) if images_file.exists() else previous.get('compose-images', {})
release = json.loads((directory / 'release.json').read_text())
before = json.loads(json.dumps(config))
for service, settings in config['services'].items():
reference = settings.get('image')
if not reference or settings.get('build'):
raise ValueError(f'{project}/{service}: Compose deploy requires a published image')
image_repo = reference.split('@')[0].rsplit('/', 1)
image_repo[-1] = image_repo[-1].split(':')[0]
image_repo = '/'.join(image_repo)
if image_repo in release['images']:
pinned = image_repo + '@' + release['images'][image_repo]
elif os.environ.get('REFRESH_IMAGES') != 'true' and reference in locks:
pinned = locks[reference]
else:
pinned = resolve(reference)
settings['image'] = pinned
locks[reference] = pinned
# Capture what is running, not the current value of its mutable tag.
ids = output(
'docker',
'ps',
'-aq',
'--filter',
f'label=com.docker.compose.project={project}',
'--filter',
f'label=com.docker.compose.service={service}',
).splitlines()
actual = set()
for container in ids:
image_id = output('docker', 'inspect', container, '--format', '{{.Image}}')
digests = json.loads(output('docker', 'image', 'inspect', image_id, '--format', '{{json .RepoDigests}}'))
actual.add(next((d for d in digests or [] if d.split('@')[0] == image_repo), image_id))
if len(actual) > 1:
raise ValueError(f'{project}/{service}: mixed running images, cannot capture one recovery config')
before['services'][service]['image'] = next(iter(actual)) if actual else reference
for name, data in (('compose', config), ('compose-before', before)):
folder = directory / name
folder.mkdir(mode=0o700, exist_ok=True)
destination = folder / f'{relative.parent.name}.json'
destination.write_text(json.dumps(data, indent=2) + '\n')
destination.chmod(0o600)
images_file.write_text(json.dumps(locks, indent=2) + '\n')
print(f'Compose {project}: images pinned; local paths preserved')
print(
f'Recovery: docker compose --project-directory {project_dir} -p {project} -f {directory}/compose-before/{relative.parent.name}.json up -d --pull never'
)
if __name__ == '__main__':
prepare(Path(sys.argv[1]))
-422
View File
@@ -1,422 +0,0 @@
#!/usr/bin/env python3
"""Durable workstation deployment controller. Install with setup-workstation.sh."""
import argparse
import contextlib
import fcntl
import importlib.util
import json
import math
import os
import re
import shutil
import subprocess
import sys
import time
from pathlib import Path
STATE = Path(os.environ.get('HOMELAB_STATE', Path.home() / '.local/state/homelab-deploy'))
CONFIG_REPO = Path(os.environ.get('HOMELAB_REPO', '/srv/homelab'))
RUN_ID = re.compile(r'[0-9]+-[0-9]+')
def command(*args, **kwargs):
return subprocess.check_output(args, text=True, **kwargs).strip() # noqa: S603, S607
def atomic_json(path, data):
temporary = path.with_suffix('.tmp')
temporary.write_text(json.dumps(data, indent=2) + '\n')
temporary.chmod(0o600)
temporary.replace(path)
@contextlib.contextmanager
def lock(name):
STATE.mkdir(mode=0o700, parents=True, exist_ok=True)
with (STATE / name).open('a') as stream:
fcntl.flock(stream, fcntl.LOCK_EX)
yield
def load_module(name, path):
spec = importlib.util.spec_from_file_location(name, path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
def run_directory(run_id):
if not RUN_ID.fullmatch(run_id):
raise ValueError('Run ID must be numeric workflow-id and attempt')
return STATE / 'runs' / run_id
def start(run_id):
payload = sys.stdin.buffer.read(256 * 1024 + 1)
if len(payload) > 256 * 1024:
raise ValueError('Deploy request exceeds 256 KiB')
request = json.loads(payload)
sha = request['release']['sha']
if not re.fullmatch(r'[0-9a-f]{40}', sha) or request['mode'] not in ('changed', 'full', 'plan'):
raise ValueError('Invalid deploy SHA or mode')
if not isinstance(request['refresh_images'], bool):
raise ValueError('refresh_images must be boolean')
directory = run_directory(run_id)
with lock('prepare.lock'):
if (directory / 'request.json').exists():
if json.loads((directory / 'request.json').read_text()) != request:
raise ValueError('Run ID already belongs to a different request')
else:
directory.mkdir(mode=0o700, parents=True, exist_ok=True)
command('git', '-C', str(CONFIG_REPO), 'fetch', '--quiet', 'origin', 'main')
command('git', '-C', str(CONFIG_REPO), 'merge-base', '--is-ancestor', sha, 'origin/main')
if not (directory / 'source').exists():
command('git', '-C', str(CONFIG_REPO), 'worktree', 'add', '--detach', str(directory / 'source'), sha)
if command('git', '-C', str(directory / 'source'), 'rev-parse', 'HEAD') != sha:
raise ValueError('Prepared source does not match deploy SHA')
release_module = load_module('release', directory / 'source/.gitea/workflows/release.py')
release_module.validate_release(request['release'], sha)
atomic_json(directory / 'release.json', request['release'])
atomic_json(directory / 'request.json', request)
if not (directory / 'status.json').exists():
atomic_json(directory / 'status.json', {'state': 'queued', 'stages': {}})
# Starting an existing active or finished ID is idempotent; never re-apply it.
if json.loads((directory / 'status.json').read_text())['state'] == 'queued':
command('systemctl', '--user', 'start', '--no-block', f'homelab-deploy@{run_id}.service')
print(f'Accepted deploy {run_id} ({sha})')
def environment(directory):
request = json.loads((directory / 'request.json').read_text())
return {
**os.environ,
'REPO': str(directory / 'source'),
'CONFIG_REPO': str(CONFIG_REPO),
'RUN_DIR': str(directory),
'DEPLOY_SHA': request['release']['sha'],
'RELEASE_FILE': str(directory / 'release.json'),
'DEPLOY_PLAN': str(directory / 'plan.json'),
'DEPLOY_SNAPSHOT_DIR': str(directory / 'snapshot'),
'REFRESH_IMAGES': str(request['refresh_images']).lower(),
'ROLLOUT_PARALLELISM': '4',
}
def stage(directory, name, budget):
status = json.loads((directory / 'status.json').read_text())
if name in status['stages'] and status['stages'][name].get('result') in ('success', 'failure'):
return status['stages'][name]['result'] == 'success'
started = time.time()
status['stages'][name] = {'result': 'running', 'started': started}
atomic_json(directory / 'status.json', status)
script = directory / 'source/.gitea/workflows/deploy-stage.sh'
with (directory / f'{name}.log').open('a') as log:
# timeout kills the whole stage process group, including children, before recovery.
result = subprocess.run( # noqa: S603, S607
[
shutil.which('timeout') or '/usr/bin/timeout',
'--signal=TERM',
'--kill-after=30s',
str(budget),
'bash',
str(script),
name,
],
env=environment(directory),
stdout=log,
stderr=subprocess.STDOUT,
check=False,
).returncode
status = json.loads((directory / 'status.json').read_text())
status['stages'][name].update(
result='success' if result == 0 else 'failure', exit_code=result, seconds=round(time.time() - started)
)
atomic_json(directory / 'status.json', status)
return result == 0
def make_plan(directory):
source = directory / 'source'
planner = load_module('deploy_plan', source / '.gitea/workflows/deploy-plan.py')
request = json.loads((directory / 'request.json').read_text())
previous = json.loads((STATE / 'last-success.json').read_text()) if (STATE / 'last-success.json').exists() else None
helm = json.loads(command('helm', 'list', '--all', '-A', '-o', 'json'))
plan = planner.make_plan(source, CONFIG_REPO, request['release'], previous, request['mode'], helm)
if request['refresh_images']:
plan['selected']['compose'] = plan['active']['compose']
atomic_json(directory / 'plan.json', plan)
if previous:
atomic_json(directory / 'previous.json', previous)
# Local config is deliberately separate from the immutable Git source.
return plan
def finish_success(directory, plan):
# Repeating finalization after a crash is safe while holding deploy.lock.
plan['run_id'] = directory.name
path = directory / 'compose-images.json'
previous = directory / 'previous.json'
plan['compose-images'] = (
json.loads(path.read_text())
if path.exists()
else json.loads(previous.read_text()).get('compose-images', {})
if previous.exists()
else {}
)
atomic_json(STATE / 'last-success.json', plan)
status = json.loads((directory / 'status.json').read_text())
status['state'] = 'success'
atomic_json(directory / 'status.json', status)
try:
retain_completed(directory)
except (OSError, subprocess.CalledProcessError) as error:
print(f'Retention deferred: {error}', flush=True)
def recover(directory, retry=False):
status = json.loads((directory / 'status.json').read_text())
if status['state'] in ('success', 'planned'):
return
completed = ('doctor', 'validate', 'apply-k8s', 'apply-compose', 'verify-k8s', 'smoke')
if all(status['stages'].get(name, {}).get('result') == 'success' for name in completed):
finish_success(directory, json.loads((directory / 'plan.json').read_text()))
return
if retry:
for name in ('verify-k8s', 'smoke'):
if status['stages'].get(name, {}).get('result') == 'failure':
del status['stages'][name]
atomic_json(directory / 'status.json', status)
snapshot = directory / 'snapshot/current'
if snapshot.exists():
stage(directory, 'verify-k8s', 7200)
stage(directory, 'smoke', 600)
status = json.loads((directory / 'status.json').read_text())
status['state'] = 'failure'
atomic_json(directory / 'status.json', status)
def execute(run_id):
directory = run_directory(run_id)
with lock('deploy.lock'):
status = json.loads((directory / 'status.json').read_text())
if status['state'] != 'queued':
return
# A crashed predecessor must be recovered before another apply begins.
for other in (STATE / 'runs').iterdir():
if (
other != directory
and (other / 'status.json').exists()
and json.loads((other / 'status.json').read_text())['state'] == 'running'
):
raise ValueError(f'Interrupted deploy {other.name}; run recover first')
status['state'] = 'running'
atomic_json(directory / 'status.json', status)
phase = 'plan'
try:
plan = make_plan(directory)
print(
json.dumps({'selected': plan['selected'], 'helm': plan['helm'], 'manual_removals': plan['removed']}),
flush=True,
)
phase = 'doctor'
if not stage(directory, 'doctor', 600):
raise RuntimeError('Preflight failed')
phase = 'validate'
if not stage(directory, 'validate', 1200):
raise RuntimeError('Validation failed')
if json.loads((directory / 'request.json').read_text())['mode'] == 'plan':
status = json.loads((directory / 'status.json').read_text())
status['state'] = 'planned'
atomic_json(directory / 'status.json', status)
return
# Budget includes both rollout checks and rollback waves, plus API overhead.
phase = 'Recovery budget'
count = int(
command(
'bash',
str(directory / 'source/.gitea/workflows/deploy-stage.sh'),
'workload-count',
env=environment(directory),
)
)
verify_budget = max(600, 2 * math.ceil(count / 4) * 300 + 120)
if verify_budget > 7200:
raise ValueError('More than two hours of recovery required; split this deploy')
phase = 'apply-k8s'
k8s_ok = stage(directory, 'apply-k8s', 2700)
phase = 'apply-compose'
compose_ok = stage(directory, 'apply-compose', 1800) if k8s_ok else False
phase = 'verify-k8s'
verify_ok = stage(directory, 'verify-k8s', verify_budget)
phase = 'smoke'
smoke_ok = stage(directory, 'smoke', 600)
if not all((k8s_ok, compose_ok, verify_ok, smoke_ok)):
raise RuntimeError('Deploy failed; inspect stage logs and recovery report')
phase = 'Save the successful baseline'
finish_success(directory, plan)
except Exception as error:
status = json.loads((directory / 'status.json').read_text())
status['failure_stage'] = next(
(name for name, result in status['stages'].items() if result.get('result') == 'failure'), phase
)
atomic_json(directory / 'status.json', status)
with (directory / 'controller.log').open('a') as stream:
stream.write(f'{error}\n')
recover(directory)
raise
def retain_completed(current):
finished = []
for directory in (STATE / 'runs').iterdir():
status_file = directory / 'status.json'
if status_file.exists() and json.loads(status_file.read_text())['state'] in ('success', 'planned'):
finished.append(directory)
for directory in sorted(finished, key=lambda p: p.stat().st_mtime, reverse=True)[20:]:
if directory == current:
continue
command('git', '-C', str(CONFIG_REPO), 'worktree', 'remove', '--force', str(directory / 'source'))
shutil.rmtree(directory)
def follow(run_id, phase):
directory = run_directory(run_id)
groups = {
'apply': ('doctor', 'validate', 'apply-k8s', 'apply-compose'),
'verify': ('verify-k8s',),
'smoke': ('smoke',),
}
names = groups[phase]
offsets = {}
while True:
status = json.loads((directory / 'status.json').read_text())
for name in (*names, 'controller'):
path = directory / f'{name}.log'
if path.exists():
with path.open() as stream:
stream.seek(offsets.get(name, 0))
content = stream.read()
if content:
print(content, end='', flush=True)
offsets[name] = stream.tell()
stages = status['stages']
if all(stages.get(name, {}).get('result') in ('success', 'failure') for name in names):
return all(stages[name]['result'] == 'success' for name in names)
if status['state'] in ('success', 'failure', 'planned'):
return status['state'] in ('success', 'planned')
time.sleep(3)
def summary(run_id):
directory = run_directory(run_id)
request = json.loads((directory / 'request.json').read_text())
release = request['release']
plan_file = directory / 'plan.json'
lines = [
f'## Deploy `{release["sha"]}`',
'',
f'- Mode: `{request["mode"]}`',
f'- Refresh third-party images: `{request["refresh_images"]}`',
]
status = json.loads((directory / 'status.json').read_text())
if status.get('failure_stage'):
lines.append(f'- Failed stage: **{status["failure_stage"]}**')
lines.extend(
[
'',
f'- Observed run state: **{status["state"]}**',
'',
'### Stage results',
'| Stage | Result | Exit code |',
'| --- | --- | --- |',
]
)
for name in ('doctor', 'validate', 'apply-k8s', 'apply-compose', 'verify-k8s', 'smoke'):
stage_result = status['stages'].get(name, {})
lines.append(f'| {name} | {stage_result.get("result", "not started")} | {stage_result.get("exit_code", "—")} |')
lines.extend(['', '### Apply and Helm recovery results'])
events_file = directory / 'apply-events.jsonl'
events = []
if events_file.exists():
for line in events_file.read_text().splitlines():
try:
events.append(json.loads(line))
except json.JSONDecodeError:
lines.append('- An operation record is incomplete. Check the stage log.')
latest = {(event['action'], event['target']): event['result'] for event in events}
lines.extend(f'- `{action}` `{target}`: **{result}**' for (action, target), result in latest.items())
if not latest:
lines.append('- No apply results were recorded.')
lines.append('- A completed apply does not confirm health. See verification and smoke results.')
lines.extend(['', '### Kubernetes recovery'])
pointer = directory / 'snapshot/current'
failed = Path(pointer.read_text().strip()) / 'failed-workloads' if pointer.exists() else None
if failed and failed.exists():
contents = failed.read_text()
counts = dict(re.findall(r'^(ROLLED_BACK|UNRECOVERED)=([0-9]+)$', contents, re.MULTILINE))
if not contents.strip():
lines.append('- No failed workloads were recorded. See the verification result above.')
elif counts:
lines.append(f'- Workloads restored: **{counts.get("ROLLED_BACK", "unknown")}**')
lines.append(f'- Workloads that need manual recovery: **{counts.get("UNRECOVERED", "unknown")}**')
else:
lines.append('- Rollback has no recorded result yet. Check the verification log.')
else:
lines.append('- No workload rollback was recorded. This does not confirm health.')
lines.append('- Compose requires manual recovery. Use the saved command in the apply log.')
if not plan_file.exists():
lines.extend(['', 'Plan was not created. Check the controller log.'])
print('\n'.join(lines))
return
plan = json.loads(plan_file.read_text())
lines.extend(['', '### Selected services'])
count = 0
for kind, services in plan['selected'].items():
for service in services:
lines.append(f'- `{kind}`: `{service}`')
count += 1
if not count:
lines.append('- None')
lines.extend(['', '### Selected Helm releases'])
lines.extend(f'- `{release}`' for release in plan.get('helm', []))
if not plan.get('helm'):
lines.append('- None')
lines.extend(['', '### Images pinned in the checked release'])
lines.extend(f'- `{image}@{digest}`' for image, digest in sorted(release['images'].items()))
lines.extend(['', '### Removed resources requiring manual review'])
lines.extend(f'- `{item}`' for item in plan.get('removed', []))
if not plan.get('removed'):
lines.append('- None')
print('\n'.join(lines))
def main():
os.umask(0o077)
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('action', choices=('start', 'execute', 'recover', 'status', 'follow', 'summary'))
parser.add_argument('run_id')
parser.add_argument('phase', nargs='?', choices=('apply', 'verify', 'smoke'))
parser.add_argument('--retry', action='store_true', help='Retry failed recovery checks; never repeat apply')
args = parser.parse_args()
directory = run_directory(args.run_id)
if args.action == 'start':
start(args.run_id)
elif args.action == 'execute':
execute(args.run_id)
elif args.action == 'recover':
with lock('deploy.lock'):
recover(directory, retry=args.retry)
elif args.action == 'status':
print((directory / 'status.json').read_text())
if (directory / 'plan.json').exists():
plan = json.loads((directory / 'plan.json').read_text())
print(json.dumps({k: plan[k] for k in ('sha', 'selected', 'helm', 'removed')}, indent=2))
elif args.action == 'summary':
summary(args.run_id)
elif not follow(args.run_id, args.phase):
sys.exit(1)
if __name__ == '__main__':
main()
File diff suppressed because it is too large. Load diff
-124
View File
@@ -1,124 +0,0 @@
#!/usr/bin/env python3
"""Calculate selected components against the last fully successful deploy."""
import hashlib
import json
import re
import subprocess
from pathlib import Path
def output(*args, **kwargs):
return subprocess.check_output(args, text=True, **kwargs).strip() # noqa: S603
def tracked(repo):
return output('git', '-C', str(repo), 'ls-files').splitlines()
def helm_releases(repo):
text = (repo / '.gitea/workflows/deploy-lib.sh').read_text()
return [line.split('|') for line in re.findall(r'^ "([^"\n]+\|[^"\n]+)"$', text, re.MULTILINE)]
def inventory(repo):
files = tracked(repo)
k8s = sorted(
{f.split('/k8s/')[0] for f in files if '/k8s/' in f and (repo / f.split('/k8s/')[0] / 'k8s/active').is_file()}
)
compose = sorted(
{
str(Path(f).parent)
for f in files
if Path(f).name in ('compose.yaml', 'compose.yml') and (repo / Path(f).parent / 'active').is_file()
}
)
return {'k8s': k8s, 'compose': compose}
def file_hash(path):
return hashlib.sha256(path.read_bytes()).hexdigest() if path.is_file() else 'missing'
def make_plan(repo, config_repo, release, previous, mode, live_helm):
active = inventory(repo)
all_services = set(active['k8s'] + active['compose'])
helm_inputs = {}
helm_selected = []
for name, chart, namespace, version, values, marker in helm_releases(repo):
if not (repo / marker).is_file():
continue
value_path = repo / values if (repo / values).is_file() else config_repo / values
if not value_path.is_file():
raise ValueError(f'Missing Helm values: {values}')
stamp = hashlib.sha256(f'{chart}|{version}|{file_hash(value_path)}'.encode()).hexdigest()
helm_inputs[name] = stamp
live = next((h for h in live_helm if h['name'] == name and h['namespace'] == namespace), None)
if (
mode == 'full'
or previous is None
or previous.get('helm_inputs', {}).get(name) != stamp
or live is None
or live.get('status') != 'deployed'
or live.get('chart') != f'{chart.split("/")[-1]}-{version}'
):
helm_selected.append(name)
local_inputs = {}
for service in all_services:
candidates = [config_repo / service / '.env']
if service in active['compose']:
candidates.append(config_repo / '.env')
cfg = config_repo / service / 'config'
if cfg.is_dir():
candidates.extend(
p for p in cfg.rglob('*') if p.is_file() and p.suffix in ('.yaml', '.yml', '.json', '.conf')
)
local_inputs[service] = hashlib.sha256(
'\n'.join(f'{p.relative_to(config_repo)}:{file_hash(p)}' for p in sorted(candidates)).encode()
).hexdigest()
if previous is None:
if mode == 'changed':
raise ValueError('No successful baseline; run deploy in full mode first')
changed = set(all_services)
removed = []
else:
paths = output('git', '-C', str(repo), 'diff', '--name-only', previous['sha'], release['sha']).splitlines()
changed = {path.split('/')[0] for path in paths}
if any(path.startswith('.gitea/') for path in paths):
changed |= all_services
changed |= {s for s in all_services if previous.get('local_inputs', {}).get(s) != local_inputs[s]}
for file in tracked(repo):
service = file.split('/')[0]
if service not in all_services or not file.endswith(('.yaml', '.yml')):
continue
text = (repo / file).read_text()
if any(
image in text and previous.get('images', {}).get(image) != digest
for image, digest in release['images'].items()
):
changed.add(service)
removed = sorted(
set(previous.get('active', {}).get('k8s', []) + previous.get('active', {}).get('compose', []))
- all_services
)
removed += [path for path in paths if '/k8s/' in path and not (repo / path).exists()]
if mode == 'full':
changed = set(all_services)
dependencies = json.loads((repo / '.gitea/deploy-dependencies.json').read_text())
while True:
expanded = changed | {dependent for service in changed for dependent in dependencies.get(service, [])}
if expanded == changed:
break
changed = expanded
return {
'version': 1,
'sha': release['sha'],
'images': release['images'],
'active': active,
'selected': {kind: sorted(set(services) & changed) for kind, services in active.items()},
'helm': helm_selected,
'helm_inputs': helm_inputs,
'local_inputs': local_inputs,
'removed': sorted(set(removed)),
'full_smoke': mode == 'full' or 'traefik' in changed,
}
-10
View File
@@ -1,10 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
source "${REPO:?}/.gitea/workflows/deploy-lib.sh"
case "${1:?stage required}" in
workload-count)
select_manifests >/dev/null
selected_workload_refs | sort -u | wc -l
;;
*) run_stage "$1" ;;
esac
-140
View File
@@ -1,140 +0,0 @@
name: deploy
on:
workflow_run:
workflows: [ci]
branches: [main]
types: [completed]
workflow_dispatch:
inputs:
deploy_ref:
description: "Commit already checked by successful main CI (main or SHA)"
default: main
required: true
deploy_mode:
description: "First deploy requires full; plan changes no production resources"
type: choice
options: [changed, full, plan]
default: changed
refresh_images:
description: "Explicitly refresh mutable third-party Compose tags"
type: boolean
default: false
permissions:
contents: read
actions: read
concurrency:
group: deploy-main
cancel-in-progress: false
env:
DEPLOY_HOST: ${{ vars.DEPLOY_HOST || secrets.DEPLOY_HOST }}
DEPLOY_PORT: ${{ vars.DEPLOY_PORT || secrets.DEPLOY_PORT }}
DEPLOY_USER: ${{ vars.DEPLOY_USER || secrets.DEPLOY_USER }}
DEPLOY_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
DEPLOY_KNOWN_HOSTS: ${{ vars.DEPLOY_KNOWN_HOSTS }}
DEPLOY_RUN_ID: ${{ github.run_id }}-${{ github.run_attempt || 1 }}
DEPLOY_MODE: ${{ inputs.deploy_mode || 'changed' }}
REFRESH_IMAGES: ${{ inputs.refresh_images && 'true' || 'false' }}
jobs:
gate:
if: >-
github.ref == 'refs/heads/main' &&
(vars.AUTODEPLOY == 'true' || github.event_name == 'workflow_dispatch') &&
(github.event_name != 'workflow_run' ||
(github.event.workflow_run.conclusion == 'success' && github.event.workflow_run.head_branch == 'main'))
runs-on: homelab
timeout-minutes: 10
outputs:
sha: ${{ steps.release.outputs.sha }}
steps:
- name: Checkout repository
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
with:
fetch-depth: 0
- name: Check successful CI and download the exact commit release
id: release
env:
GITEA_TOKEN: ${{ github.token }}
DEPLOY_REF: ${{ inputs.deploy_ref || 'main' }}
EVENT_SHA: ${{ github.event.workflow_run.head_sha }}
run: python3 .gitea/workflows/release.py gate --ref "$DEPLOY_REF" --event-sha "$EVENT_SHA"
- name: Submit durable deploy to workstation
run: bash .gitea/workflows/ssh-run.sh start
- name: Write the request result
if: always()
env:
REQUEST_RESULT: ${{ job.status }}
CHECKED_SHA: ${{ steps.release.outputs.sha }}
run: |
if [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
printf '## Deploy request\n\n- Result: **%s**\n- Checked commit: %s\n- Mode: %s\n' "$REQUEST_RESULT" "${CHECKED_SHA:-not checked}" "$DEPLOY_MODE" >>"$GITHUB_STEP_SUMMARY"
if [ "$REQUEST_RESULT" != success ]; then
echo 'Open the failed step log. If SSH submission failed, check the remote controller state.' >>"$GITHUB_STEP_SUMMARY"
fi
fi
apply:
needs: [gate]
runs-on: homelab
timeout-minutes: 100
steps:
- name: Checkout checked commit
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
with:
ref: ${{ needs.gate.outputs.sha }}
- name: Follow validation and sequential Kubernetes / Compose apply
run: bash .gitea/workflows/ssh-run.sh apply
- name: Write the deploy result
if: always()
run: |
if [ -f .gitea/workflows/ssh-run.sh ]; then
bash .gitea/workflows/ssh-run.sh summary
elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
echo 'Source checkout failed. The remote deploy state is unknown. Check the job log.' >>"$GITHUB_STEP_SUMMARY"
fi
verify:
needs: [gate, apply]
if: always() && needs.gate.result == 'success'
runs-on: homelab
timeout-minutes: 130
steps:
- name: Checkout checked commit
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
with:
ref: ${{ needs.gate.outputs.sha }}
- name: Follow workload verification and recovery
run: bash .gitea/workflows/ssh-run.sh verify
- name: Write the deploy result
if: always()
run: |
if [ -f .gitea/workflows/ssh-run.sh ]; then
bash .gitea/workflows/ssh-run.sh summary
elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
echo 'Source checkout failed. The remote deploy state is unknown. Check the job log.' >>"$GITHUB_STEP_SUMMARY"
fi
smoke:
needs: [gate, verify]
if: always() && needs.gate.result == 'success'
runs-on: homelab
timeout-minutes: 15
steps:
- name: Checkout checked commit
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
with:
ref: ${{ needs.gate.outputs.sha }}
- name: Follow public route checks
run: bash .gitea/workflows/ssh-run.sh smoke
- name: Write the deploy result
if: always()
run: |
if [ -f .gitea/workflows/ssh-run.sh ]; then
bash .gitea/workflows/ssh-run.sh summary
elif [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
echo 'Source checkout failed. The remote deploy state is unknown. Check the job log.' >>"$GITHUB_STEP_SUMMARY"
fi
-285
View File
@@ -1,285 +0,0 @@
#!/usr/bin/env bash
# Installs the pinned CI tools into "$TOOLS_DIR/bin" and echoes that directory
# on stdout, so callers can do:
#
# export PATH="$(bash .gitea/workflows/install-ci-tools.sh kubeconform shellcheck):$PATH"
#
# Versions come from tool-versions.env next to this script and are kept fresh by
# Renovate. Re-running is cheap: an already-installed tool at the pinned version
# is left alone.
set -euo pipefail
here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck source=tool-versions.env
. "$here/tool-versions.env"
TOOLS_DIR="${TOOLS_DIR:-${XDG_CACHE_HOME:-$HOME/.cache}/homelab-ci}"
BIN_DIR="$TOOLS_DIR/bin"
mkdir -p "$BIN_DIR"
# A runner may accept overlapping workflows even though each workflow is sequential.
exec 9>"$TOOLS_DIR/install.lock"
flock -w 300 9
export UV_TOOL_DIR="$TOOLS_DIR/uv-tools"
export UV_CACHE_DIR="$TOOLS_DIR/uv-cache"
# The just-installed tools must resolve inside this script too: callers only
# prepend BIN_DIR to PATH after the script exits, so a bare `uv` below would
# miss the binary install_uv just placed (exit 127 on a clean runner).
export PATH="$BIN_DIR:$PATH"
arch="$(uname -m)"
# Upstream projects disagree on arch spelling: kubeconform and actionlint use
# Go names (amd64/arm64), shellcheck uses uname names (x86_64/aarch64), node
# uses neither (x64/arm64), and hadolint mixes the two in a single release
# (x86_64 but arm64).
case "$arch" in
x86_64 | amd64)
goarch=amd64
sharch=x86_64
nodearch=x64
hadolintarch=x86_64
;;
aarch64 | arm64)
goarch=arm64
sharch=aarch64
nodearch=arm64
hadolintarch=arm64
;;
*)
echo "install-ci-tools: unsupported architecture: $arch" >&2
exit 1
;;
esac
fetch() {
# fetch <url> <dest>
if command -v curl >/dev/null 2>&1; then
curl -sSLf --connect-timeout 15 --max-time 120 --retry 3 -o "$2" "$1"
elif command -v wget >/dev/null 2>&1; then
wget -q -O "$2" "$1"
else
echo "install-ci-tools: neither curl nor wget is available" >&2
exit 1
fi
}
# resolve <command>
# Absolute path to use for invoking a tool: the copy in BIN_DIR when present,
# otherwise the name for PATH lookup. Every version check and every in-script
# invocation goes through this, so a tool missing from both places reads as
# "not installed" instead of dying with 127 under `set -e`.
resolve() {
if [ -x "$BIN_DIR/$1" ]; then
printf '%s' "$BIN_DIR/$1"
else
printf '%s' "$1"
fi
}
# installed_version <command>
# Prints the version of an already-installed tool, or nothing. Each tool spells
# its version flag differently, hence the case.
installed_version() {
local bin out
bin="$(resolve "$1")"
if ! command -v "$bin" >/dev/null 2>&1; then
return 0
fi
case "$1" in
kubeconform) out="$("$bin" -v 2>/dev/null | head -1 || true)" ;;
*) out="$("$bin" --version 2>/dev/null | head -1 || true)" ;;
esac
printf '%s' "$out"
}
# at_version <command> <expected>
at_version() {
local version expected="${2#v}"
version="$(installed_version "$1")"
if [[ "$version" =~ (^|[^0-9.])v?([0-9]+(\.[0-9]+)+) ]]; then
[ "${BASH_REMATCH[2]}" = "$expected" ]
else
return 1
fi
}
install_kubeconform() {
if at_version kubeconform "v${KUBECONFORM_VERSION}"; then
return 0
fi
local tmp
tmp="$(mktemp -d)"
fetch "https://github.com/yannh/kubeconform/releases/download/v${KUBECONFORM_VERSION}/kubeconform-linux-${goarch}.tar.gz" \
"$tmp/kubeconform.tar.gz"
tar -xzf "$tmp/kubeconform.tar.gz" -C "$tmp" kubeconform
install -m 0755 "$tmp/kubeconform" "$BIN_DIR/kubeconform"
rm -rf "$tmp"
}
install_shellcheck() {
if at_version shellcheck "${SHELLCHECK_VERSION}"; then
return 0
fi
local tmp
tmp="$(mktemp -d)"
fetch "https://github.com/koalaman/shellcheck/releases/download/v${SHELLCHECK_VERSION}/shellcheck-v${SHELLCHECK_VERSION}.linux.${sharch}.tar.xz" \
"$tmp/shellcheck.tar.xz"
tar -xJf "$tmp/shellcheck.tar.xz" -C "$tmp" --strip-components=1 "shellcheck-v${SHELLCHECK_VERSION}/shellcheck"
install -m 0755 "$tmp/shellcheck" "$BIN_DIR/shellcheck"
rm -rf "$tmp"
}
install_jq() {
if at_version jq "${JQ_VERSION}"; then
return 0
fi
fetch "https://github.com/jqlang/jq/releases/download/jq-${JQ_VERSION}/jq-linux-${goarch}" \
"$BIN_DIR/jq"
chmod 0755 "$BIN_DIR/jq"
}
install_uv() {
if at_version uv "${UV_VERSION}"; then
return 0
fi
local tmp
tmp="$(mktemp -d)"
# uv release tags carry no leading v, unlike every other tool installed here.
fetch "https://github.com/astral-sh/uv/releases/download/${UV_VERSION}/uv-${sharch}-unknown-linux-gnu.tar.gz" \
"$tmp/uv.tar.gz"
tar -xzf "$tmp/uv.tar.gz" -C "$tmp" --strip-components=1 "uv-${sharch}-unknown-linux-gnu/uv"
install -m 0755 "$tmp/uv" "$BIN_DIR/uv"
rm -rf "$tmp"
}
install_hadolint() {
if at_version hadolint "${HADOLINT_VERSION}"; then
return 0
fi
# A bare binary, no archive: hadolint ships one file per platform.
fetch "https://github.com/hadolint/hadolint/releases/download/v${HADOLINT_VERSION}/hadolint-linux-${hadolintarch}" \
"$BIN_DIR/hadolint"
chmod 0755 "$BIN_DIR/hadolint"
}
# ruff and yamllint both come from PyPI as wheels, which uv unpacks for us.
install_uv_tool() {
# <package> <pinned version>
if at_version "$1" "$2"; then
return 0
fi
install_uv
UV_TOOL_BIN_DIR="$BIN_DIR" "$BIN_DIR/uv" tool install --force "$1==$2" >/dev/null
}
install_ruff() {
install_uv_tool ruff "${RUFF_VERSION}"
}
install_yamllint() {
install_uv_tool yamllint "${YAMLLINT_VERSION}"
}
install_pip_audit() {
install_uv_tool pip-audit "${PIP_AUDIT_VERSION}"
}
install_prettier() {
install_node
if at_version prettier "${PRETTIER_VERSION}"; then
return 0
fi
# Not a standalone binary: prettier's entry point requires ../package.json
# relative to its own real path, so the package directory has to survive
# next to it. Hence a versioned directory plus a relative symlink, rather
# than copying the one file out as the other installers do.
local dir="$BIN_DIR/prettier-${PRETTIER_VERSION}"
if [ ! -f "$dir/package/package.json" ]; then
rm -rf "$dir"
mkdir -p "$dir"
fetch "https://registry.npmjs.org/prettier/-/prettier-${PRETTIER_VERSION}.tgz" "$dir/prettier.tgz"
tar -xzf "$dir/prettier.tgz" -C "$dir"
rm -f "$dir/prettier.tgz"
# npm strips the exec bit from bin/ on the way into the tarball.
chmod 0755 "$dir/package/bin/prettier.cjs"
fi
# Relative, so the whole tree stays valid if TOOLS_DIR is relocated.
ln -sfn "prettier-${PRETTIER_VERSION}/package/bin/prettier.cjs" "$BIN_DIR/prettier"
}
install_node() {
# npm gets checked by running it, not by looking it up: what matters is that
# it answers, so a stub, a half-removed Arch package or a name that resolves
# to something broken all have to read as "not installed". The runner's npm
# is a symlink into /usr/lib/node_modules/npm, which is exactly the kind of
# thing that disappears between runs.
if at_version node "v${NODE_VERSION}" && [ -n "$(installed_version npm)" ]; then
return 0
fi
# Same shape as prettier above: the tarball's bin/npm and bin/npx are links
# into lib/node_modules, so the whole tree has to survive next to them.
local dir="$BIN_DIR/node-${NODE_VERSION}"
if [ ! -x "$dir/bin/node" ]; then
rm -rf "$dir"
mkdir -p "$dir"
fetch "https://nodejs.org/dist/v${NODE_VERSION}/node-v${NODE_VERSION}-linux-${nodearch}.tar.xz" \
"$dir/node.tar.xz"
tar -xJf "$dir/node.tar.xz" -C "$dir" --strip-components=1 "node-v${NODE_VERSION}-linux-${nodearch}"
rm -f "$dir/node.tar.xz"
fi
# Relative, so the whole tree stays valid if TOOLS_DIR is relocated.
for bin in node npm npx; do
ln -sfn "node-${NODE_VERSION}/bin/${bin}" "$BIN_DIR/${bin}"
done
}
install_actionlint() {
if at_version actionlint "${ACTIONLINT_VERSION}"; then
return 0
fi
local tmp
tmp="$(mktemp -d)"
fetch "https://github.com/rhysd/actionlint/releases/download/v${ACTIONLINT_VERSION}/actionlint_${ACTIONLINT_VERSION}_linux_${goarch}.tar.gz" \
"$tmp/actionlint.tar.gz"
tar -xzf "$tmp/actionlint.tar.gz" -C "$tmp" actionlint
install -m 0755 "$tmp/actionlint" "$BIN_DIR/actionlint"
rm -rf "$tmp"
}
main() {
wanted=("$@")
if [ "${#wanted[@]}" -eq 0 ]; then
wanted=(node jq kubeconform shellcheck actionlint prettier ruff yamllint hadolint)
fi
for tool in "${wanted[@]}"; do
case "$tool" in
kubeconform) install_kubeconform ;;
shellcheck) install_shellcheck ;;
jq) install_jq ;;
actionlint) install_actionlint ;;
prettier) install_prettier ;;
ruff) install_ruff ;;
yamllint) install_yamllint ;;
pip-audit) install_pip_audit ;;
hadolint) install_hadolint ;;
node) install_node ;;
uv) install_uv ;;
*)
echo "install-ci-tools: unknown tool: $tool" >&2
exit 1
;;
esac
done
for old in "$BIN_DIR"/node-* "$BIN_DIR"/prettier-*; do
[ -d "$old" ] || continue
case "$(basename "$old")" in
"node-$NODE_VERSION"|"prettier-$PRETTIER_VERSION") ;;
*) rm -rf "$old" ;;
esac
done
if [ -x "$BIN_DIR/uv" ]; then "$BIN_DIR/uv" cache prune >/dev/null; fi
printf '%s\n' "$BIN_DIR"
}
if [ "${BASH_SOURCE[0]}" = "$0" ]; then main "$@"; fi
-516
View File
@@ -1,516 +0,0 @@
#!/usr/bin/env python3
"""CI release artifacts and the SHA-specific Gitea deployment gate (stdlib only)."""
import argparse
import hashlib
import io
import itertools
import json
import os
import re
import shutil
import subprocess
import sys
import tempfile
import urllib.error
import urllib.parse
import urllib.request
import zipfile
from pathlib import Path
SHA = re.compile(r'[0-9a-f]{40}')
DIGEST = re.compile(r'sha256:[0-9a-f]{64}')
IMAGES = {
'error-pages': ('errorpages', 'errorpages/Dockerfile'),
'forust-homepage': ('homepages', 'homepages/Dockerfile.forust'),
'xdfnx-homepage': ('homepages', 'homepages/Dockerfile.xdfnx'),
}
def command(*args, **kwargs):
"""Arguments are passed directly to the executable, never to a shell."""
return subprocess.check_output(args, text=True, **kwargs).strip() # noqa: S603, S607
def validate_release(data, sha=None):
if data.get('version') != 1 or not SHA.fullmatch(data.get('sha', '')):
raise ValueError('Invalid release version or SHA')
if sha is not None and data['sha'] != sha:
raise ValueError('Release SHA does not match the checked CI commit')
expected = {f'gcr.forust.xyz/forust/{name}' for name in IMAGES}
if set(data.get('images', {})) != expected:
raise ValueError('Release must contain all owned images')
if not all(DIGEST.fullmatch(value) for value in data['images'].values()):
raise ValueError('Release has an invalid image digest')
if set(data.get('inputs', {})) != expected or not all(
re.fullmatch(r'[0-9a-f]{64}', value) for value in data['inputs'].values()
):
raise ValueError('Release has invalid build input fingerprints')
return data
class NoRedirect(urllib.request.HTTPRedirectHandler):
def redirect_request(self, _req, _fp, _code, _msg, _headers, _newurl):
return None
class Gitea:
def __init__(self):
self.origin = os.environ['GITHUB_SERVER_URL'].rstrip('/')
if urllib.parse.urlsplit(self.origin).scheme != 'https':
raise ValueError('Gitea API must use HTTPS')
self.repository = os.environ['GITHUB_REPOSITORY']
if not re.fullmatch(r'[\w.-]+/[\w.-]+', self.repository):
raise ValueError('Invalid Gitea repository')
self.token = os.environ['GITEA_TOKEN']
self.base = f'{self.origin}/api/v1/repos/{self.repository}'
def request(self, url, *, archive=False):
if not url.startswith(self.base + '/'):
raise ValueError('Refusing to send the Actions token to another origin')
req = urllib.request.Request(url, headers={'Authorization': f'token {self.token}'}) # noqa: S310 -- HTTPS origin validated above
opener = urllib.request.build_opener(NoRedirect())
try:
response = opener.open(req, timeout=30) # noqa: S310
except urllib.error.HTTPError as error:
if not archive or error.code not in (301, 302, 303, 307, 308):
raise RuntimeError(f'Gitea API returned HTTP {error.code}') from None
target = urllib.parse.urljoin(url, error.headers['Location'])
if urllib.parse.urlsplit(target).scheme != 'https':
raise ValueError('Artifact redirect must use HTTPS') from None
# Signed storage redirects must never receive the Gitea token.
response = urllib.request.urlopen(target, timeout=30) # noqa: S310
with response:
payload = response.read(8 * 1024 * 1024 + 1)
if len(payload) > 8 * 1024 * 1024:
raise ValueError('Gitea response exceeds 8 MiB')
return payload if archive else json.loads(payload)
def pages(self, path, key, **params):
for page in range(1, 101):
query = urllib.parse.urlencode({**params, 'page': page, 'limit': 50})
data = self.request(f'{self.base}/{path}?{query}')
entries = data[key]
yield from entries
if len(entries) < 50:
return
raise RuntimeError('Gitea pagination limit exceeded')
def successful_runs(self, sha=None):
params = {'branch': 'main', 'status': 'success', 'exclude_pull_requests': 'true'}
if sha:
params['head_sha'] = sha
for run in self.pages('actions/workflows/ci.yaml/runs', 'workflow_runs', **params):
if (
run.get('status') == 'completed'
and run.get('conclusion') == 'success'
and run.get('head_branch') == 'main'
and run.get('event') in ('push', 'workflow_dispatch')
and (run.get('repository') or {}).get('full_name') == self.repository
and (run.get('head_repository') or run.get('repository') or {}).get('full_name') == self.repository
and (sha is None or run.get('head_sha') == sha)
):
yield run
def release(self, run):
sha = run['head_sha']
jobs = list(self.pages(f'actions/runs/{run["id"]}/jobs', 'jobs'))
# A green workflow with a skipped build must not authorize a deploy.
if not any(job.get('name') == 'build' and job.get('conclusion') == 'success' for job in jobs):
raise ValueError('CI build job did not succeed')
artifacts = self.request(f'{self.base}/actions/runs/{run["id"]}/artifacts')['artifacts']
matching = [a for a in artifacts if a['name'] == f'release-{sha}' and not a.get('expired')]
if len(matching) != 1:
raise ValueError('CI release artifact is missing, expired or ambiguous; rerun CI')
blob = self.request(f'{self.base}/actions/artifacts/{matching[0]["id"]}/zip', archive=True)
with zipfile.ZipFile(io.BytesIO(blob)) as archive:
files = [entry for entry in archive.infolist() if not entry.is_dir()]
if len(files) != 1 or files[0].filename != 'release.json' or files[0].file_size > 256 * 1024:
raise ValueError('Unexpected release archive contents')
return validate_release(json.loads(archive.read(files[0])), sha)
def fingerprint(context, dockerfile):
tree = command('git', 'ls-tree', '-r', 'HEAD', '--', context, dockerfile, '.gitea/workflows/release.py')
return hashlib.sha256(tree.encode()).hexdigest()
def gate(output, requested_ref, event_sha):
command('git', 'fetch', '--quiet', 'origin', 'main')
if event_sha:
if not SHA.fullmatch(event_sha):
raise ValueError('Invalid workflow_run SHA')
sha = event_sha
else:
if requested_ref == 'main':
requested_ref = 'origin/main'
sha = command('git', 'rev-parse', '--verify', '--end-of-options', f'{requested_ref}^{{commit}}')
if not SHA.fullmatch(sha):
raise ValueError('Invalid deploy SHA')
command('git', 'merge-base', '--is-ancestor', sha, 'origin/main')
api = Gitea()
runs = list(api.successful_runs(sha))
if not runs:
raise ValueError(f'No successful main CI for {sha}; run CI before deploying')
release = api.release(max(runs, key=lambda run: run['id']))
output.write_text(json.dumps(release, indent=2) + '\n')
if os.environ.get('GITHUB_OUTPUT'):
with Path(os.environ['GITHUB_OUTPUT']).open('a') as stream:
stream.write(f'sha={sha}\n')
print(f'CI gate accepted {sha}')
def prepare_images(output):
sha = command('git', 'rev-parse', 'HEAD')
if sha != os.environ['GITHUB_SHA'] or not SHA.fullmatch(sha):
raise ValueError('Build checkout does not match GITHUB_SHA')
api = Gitea()
previous = None
for run in sorted(itertools.islice(api.successful_runs(), 50), key=lambda item: item['id'], reverse=True):
if str(run['id']) == os.environ.get('GITHUB_RUN_ID'):
continue
try:
previous = api.release(run)
break
except ValueError:
# Expired artifacts only cost a rebuild; mutable tags are never a fallback.
continue
targets = []
for name, (context, dockerfile) in IMAGES.items():
image = f'gcr.forust.xyz/forust/{name}'
inputs = fingerprint(context, dockerfile)
old_digest = (previous or {}).get('images', {}).get(image)
targets.append(
{
'name': name,
'image': image,
'context': context,
'dockerfile': dockerfile,
'inputs': inputs,
'reuse_digest': old_digest if (previous or {}).get('inputs', {}).get(image) == inputs else None,
}
)
output.write_text(json.dumps({'sha': sha, 'targets': targets}, indent=2) + '\n')
if os.environ.get('GITHUB_OUTPUT'):
with Path(os.environ['GITHUB_OUTPUT']).open('a') as stream:
stream.write('matrix=' + json.dumps({'include': targets}, separators=(',', ':')) + '\n')
print(f'Prepared {len(targets)} image jobs; {sum(t["reuse_digest"] is None for t in targets)} require builds')
def checked_plan(path):
data = json.loads(path.read_text())
sha = command('git', 'rev-parse', 'HEAD')
if data.get('sha') != sha or sha != os.environ['GITHUB_SHA'] or not SHA.fullmatch(sha):
raise ValueError('Image plan does not match the checked source commit')
targets = data.get('targets', [])
if sorted(t['name'] for t in targets) != sorted(IMAGES):
raise ValueError('Image plan must contain each owned image once')
for target in targets:
name = target['name']
context, dockerfile = IMAGES[name]
if (target['context'], target['dockerfile'], target['image']) != (
context,
dockerfile,
f'gcr.forust.xyz/forust/{name}',
) or target['inputs'] != fingerprint(context, dockerfile):
raise ValueError('Image plan has invalid build inputs')
if target['reuse_digest'] is not None and not DIGEST.fullmatch(target['reuse_digest']):
raise ValueError('Image plan has an invalid reuse digest')
return data
def build_images(output, report, name, plan):
data = checked_plan(plan)
sha = data['sha']
target = next(t for t in data['targets'] if t['name'] == name)
context, dockerfile = IMAGES[name]
docker_config = tempfile.mkdtemp(prefix='homelab-registry-')
builder_config = Path.home() / '.cache/homelab-ci/buildx'
builder_config.mkdir(parents=True, exist_ok=True)
env = {**os.environ, 'DOCKER_CONFIG': docker_config, 'BUILDX_CONFIG': str(builder_config)}
try:
report['phase'] = 'Registry login'
subprocess.run( # noqa: S603, S607
[
shutil.which('docker') or '/usr/bin/docker',
'login',
'gcr.forust.xyz',
'-u',
os.environ['REGISTRY_USERNAME'],
'--password-stdin',
],
input=os.environ['REGISTRY_PASSWORD'],
text=True,
check=True,
env=env,
)
report['phase'] = 'Prepare the builder'
builder = 'homelab-ci'
versions = dict(
re.findall(r'^([A-Z_]+)="([^"\n]+)"$', Path('.gitea/workflows/tool-versions.env').read_text(), re.MULTILINE)
)
image = versions['BUILDKIT_IMAGE']
signature = builder_config / 'homelab-ci-image'
exists = (
subprocess.run( # noqa: S603
[shutil.which('docker') or '/usr/bin/docker', 'buildx', 'inspect', builder],
capture_output=True,
env=env,
).returncode
== 0
)
if exists and (not signature.exists() or signature.read_text().strip() != image):
command('docker', 'buildx', 'rm', '--keep-state', builder, env=env)
exists = False
if not exists:
command(
'docker',
'buildx',
'create',
'--name',
builder,
'--driver',
'docker-container',
'--driver-opt',
f'image={image}',
'--buildkitd-config',
'.gitea/runner/buildkitd.toml',
env=env,
)
signature.write_text(image + '\n')
release = {'version': 1, 'sha': sha, 'images': {}, 'inputs': {}}
report['images'] = release['images']
report['phase'] = f'Build or reuse {name}'
report['current'] = name
image = f'gcr.forust.xyz/forust/{name}'
inputs = target['inputs']
old_digest = target['reuse_digest']
exists = False
if old_digest:
exists = (
subprocess.run( # noqa: S603, S607
[
shutil.which('docker') or '/usr/bin/docker',
'buildx',
'imagetools',
'inspect',
f'{image}@{old_digest}',
],
capture_output=True,
env=env,
timeout=60,
).returncode
== 0
)
if exists:
print(f'Reuse {name}: inputs unchanged')
digest = old_digest
report['reused'].append(name)
else:
print(f'Build {name}', flush=True)
metadata = Path(docker_config) / 'metadata.json'
command(
'docker',
'buildx',
'build',
'--builder',
builder,
'--platform',
'linux/amd64',
'--provenance=false',
'--cache-from',
f'type=registry,ref={image}:buildcache',
'--cache-to',
f'type=registry,ref={image}:buildcache,mode=max',
'--output',
f'type=image,name={image},push-by-digest=true,name-canonical=true,push=true',
'--metadata-file',
str(metadata),
'--file',
dockerfile,
context,
env=env,
)
digest = json.loads(metadata.read_text())['containerimage.digest']
report['built'].append(name)
release['images'][image] = digest
release['inputs'][image] = inputs
if not DIGEST.fullmatch(digest):
raise ValueError('Image job returned an invalid digest')
output.write_text(json.dumps(release, indent=2) + '\n')
report['current'] = None
report['phase'] = 'Release file saved'
finally:
# Cleanup errors must neither leak credentials nor mask the original build error.
try:
subprocess.run( # noqa: S603
[
shutil.which('docker') or '/usr/bin/docker',
'buildx',
'prune',
'--builder',
'homelab-ci',
'--force',
'--max-used-space',
'1gb',
],
env=env,
timeout=60,
)
except (OSError, subprocess.TimeoutExpired):
print('CI builder cache cleanup deferred', flush=True)
finally:
shutil.rmtree(docker_config)
def write_summary(lines):
path = os.environ.get('GITHUB_STEP_SUMMARY')
if path:
try:
with Path(path).open('a') as stream:
stream.write('\n'.join(lines) + '\n\n')
except OSError:
print('WARNING: cannot write the job summary')
def check_summary():
lines = [
f'## {os.environ["SUMMARY_CHECK"]}',
'',
f'- Commit: `{os.environ.get("GITHUB_SHA", "unknown")}`',
f'- Result: **{os.environ["SUMMARY_RESULT"]}**',
]
if os.environ.get('SUMMARY_FAILED_STEP'):
lines.append(f'- Failed step: {os.environ["SUMMARY_FAILED_STEP"]}')
if os.environ['SUMMARY_RESULT'] != 'success':
lines.append('- Open the failed step log for the error details.')
write_summary(lines)
def build(output, name, plan):
report = {'phase': 'Check the source commit', 'current': None, 'built': [], 'reused': [], 'images': {}}
result = 'failure'
try:
build_images(output, report, name, plan)
result = 'success'
finally:
lines = [
f'## Image release `{os.environ.get("GITHUB_SHA", "unknown")}`',
'',
f'- Result: **{result}**',
f'- Last stage: {report["phase"]}',
]
if result == 'failure':
lines.append('- No release from this build can be deployed. Open the failed step log.')
if report['current']:
lines.append(f'- Image at the failure: `{report["current"]}`')
for title, key in (('Built', 'built'), ('Reused from successful CI', 'reused')):
lines.extend(['', f'### {title}'])
lines.extend(f'- `{name}`' for name in report[key])
if not report[key]:
lines.append('- None')
lines.extend(['', '### Completed image digests'])
lines.extend(f'- `{image}@{digest}`' for image, digest in report['images'].items())
if not report['images']:
lines.append('- None')
write_summary(lines)
def render(stream, destination):
release = validate_release(json.loads(Path(os.environ['RELEASE_FILE']).read_text()), os.environ['DEPLOY_SHA'])
image_line = re.compile(
r"^(\s*(?:-\s*)?image:\s*)(['\"]?)(gcr\.forust\.xyz/forust/[\w.-]+)(?::[\w.-]+|@sha256:[0-9a-f]{64})\2(\s*(?:#.*)?)$"
)
rendered = []
for line in stream:
match = image_line.fullmatch(line.rstrip('\n'))
if match:
prefix, quote, image, tail = match.groups()
if image not in release['images']:
raise ValueError(f'Owned image missing from checked release: {image}')
line = f'{prefix}{quote}{image}@{release["images"][image]}{quote}{tail}\n'
elif re.match(r'\s*(?:-\s*)?image:', line) and 'gcr.forust.xyz/forust/' in line:
raise ValueError('Unsupported owned image syntax; refusing to apply a mutable tag')
rendered.append(line)
destination.writelines(rendered)
def finalize_images(output, fragments, plan):
data = checked_plan(plan)
sha = data['sha']
release = {'version': 1, 'sha': sha, 'images': {}, 'inputs': {}}
for name in IMAGES:
fragment = json.loads((fragments / f'image-{name}' / 'image.json').read_text())
image = f'gcr.forust.xyz/forust/{name}'
if fragment.get('sha') != sha or fragment.get('version') != 1 or set(fragment.get('images', {})) != {image}:
raise ValueError('Image job artifact is missing or belongs to another commit')
target = next(t for t in data['targets'] if t['name'] == name)
if fragment.get('inputs') != {image: target['inputs']}:
raise ValueError('Image artifact does not match the build plan')
release['images'].update(fragment['images'])
release['inputs'].update(fragment['inputs'])
validate_release(release, sha)
# Only a complete set of successful image jobs can publish the release tags.
docker_config = tempfile.mkdtemp(prefix='homelab-registry-')
env = {**os.environ, 'DOCKER_CONFIG': docker_config}
try:
subprocess.run( # noqa: S603, S607
[
shutil.which('docker') or '/usr/bin/docker',
'login',
'gcr.forust.xyz',
'-u',
os.environ['REGISTRY_USERNAME'],
'--password-stdin',
],
input=os.environ['REGISTRY_PASSWORD'],
text=True,
check=True,
env=env,
)
for image, digest in release['images'].items():
command(
'docker',
'buildx',
'imagetools',
'create',
'--prefer-index=false',
'--tag',
f'{image}:sha-{sha}',
f'{image}@{digest}',
env=env,
timeout=90,
)
output.write_text(json.dumps(release, indent=2) + '\n')
finally:
shutil.rmtree(docker_config)
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('action', choices=('prepare', 'image', 'finalize', 'gate', 'render', 'check-summary'))
parser.add_argument('--output', type=Path, default=Path('release.json'))
parser.add_argument('--ref', default='main')
parser.add_argument('--event-sha', default='')
parser.add_argument('--image', choices=IMAGES)
parser.add_argument('--plan', type=Path, default=Path('build-plan.json'))
parser.add_argument('--fragments', type=Path, default=Path('artifacts'))
args = parser.parse_args()
if args.action == 'check-summary':
check_summary()
elif args.action == 'render':
render(sys.stdin, sys.stdout)
elif args.action == 'gate':
gate(args.output, args.ref, args.event_sha)
elif args.action == 'prepare':
prepare_images(args.output)
elif args.action == 'image':
if not args.image:
parser.error('--image is required')
build(args.output, args.image, args.plan)
else:
finalize_images(args.output, args.fragments, args.plan)
if __name__ == '__main__':
main()
-90
View File
@@ -1,90 +0,0 @@
name: renovate-ci
on:
pull_request:
paths:
- "renovate/**"
- ".gitea/workflows/renovate-ci.yaml"
- ".gitea/workflows/sync-renovate-configmap.sh"
- ".gitea/workflows/compose-lint.sh"
- ".gitea/workflows/install-ci-tools.sh"
- ".gitea/workflows/tool-versions.env"
push:
branches:
- main
paths:
- "renovate/**"
- ".gitea/workflows/renovate-ci.yaml"
- ".gitea/workflows/sync-renovate-configmap.sh"
- ".gitea/workflows/compose-lint.sh"
- ".gitea/workflows/install-ci-tools.sh"
- ".gitea/workflows/tool-versions.env"
workflow_dispatch:
permissions:
contents: read
jobs:
validate-renovate:
runs-on: homelab
timeout-minutes: 20
steps:
- name: Checkout repository
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
# renovate/k8s/cronjob.yaml is the single source of truth for the image tag,
# so the same version that runs in the cluster is the one validated here.
- name: Resolve the deployed Renovate image
id: image
shell: bash
run: |
set -euo pipefail
image="$(sed -n 's|.*image:[[:space:]]*\(renovate/renovate:[^[:space:]]*\).*|\1|p' \
renovate/k8s/cronjob.yaml | head -1)"
if [ -z "$image" ]; then
echo "::error::no renovate/renovate image found in renovate/k8s/cronjob.yaml"
exit 1
fi
echo "using $image"
echo "image=$image" >> "$GITHUB_OUTPUT"
- name: Validate Renovate repository config
shell: bash
run: |
set -euo pipefail
docker run --rm \
-v "$PWD/renovate:/opt/renovate:ro" \
-e RENOVATE_CONFIG_FILE=/opt/renovate/renovate.json \
"${{ steps.image.outputs.image }}" \
renovate-config-validator /opt/renovate/renovate.json
# The CronJob cannot read the repository, so renovate/k8s/configmap.yaml
# carries an inlined copy of the config. Fail if it no longer matches.
- name: Check the generated Renovate ConfigMap
shell: bash
run: |
set -euo pipefail
./.gitea/workflows/sync-renovate-configmap.sh --check
- name: Validate Renovate Kubernetes manifests
shell: bash
run: |
set -euo pipefail
tools_dir="$(bash .gitea/workflows/install-ci-tools.sh kubeconform)"
export PATH="$tools_dir:$PATH"
kubeconform \
-strict \
-ignore-missing-schemas \
-summary \
renovate/k8s/namespace.yaml \
renovate/k8s/configmap.yaml \
renovate/k8s/cronjob.yaml
- name: Validate Renovate Compose file
shell: bash
run: |
set -euo pipefail
source .gitea/workflows/compose-lint.sh
mapfile -t safe_flags < <(compose_safe_flags)
validate_compose_file renovate/renovate-compose.yaml \
${safe_flags[@]+"${safe_flags[@]}"}
-92
View File
@@ -1,92 +0,0 @@
name: renovate-run
on:
workflow_dispatch:
inputs:
repositories:
description: "Repositories to scan (comma-separated)"
required: false
default: "forust/homelab"
log_level:
description: "Renovate log level"
required: false
default: "info"
type: choice
options:
- info
- debug
dry_run:
description: "Plan only, do not open or update PRs"
required: false
default: false
type: boolean
# Renovate writes through its own bot PAT, passed in as RENOVATE_TOKEN, so the
# Actions token is only ever used to read the checkout.
permissions:
contents: read
concurrency:
group: renovate-run
cancel-in-progress: false
jobs:
run-renovate:
runs-on: homelab
timeout-minutes: 60
steps:
- name: Checkout repository
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
# renovate/k8s/cronjob.yaml is the single source of truth for the image tag.
# Reading it here means this workflow validates and runs the exact version
# that is deployed, instead of a copy that silently goes stale.
- name: Resolve the deployed Renovate image
id: image
shell: bash
run: |
set -euo pipefail
image="$(sed -n 's|.*image:[[:space:]]*\(renovate/renovate:[^[:space:]]*\).*|\1|p' \
renovate/k8s/cronjob.yaml | head -1)"
if [ -z "$image" ]; then
echo "::error::no renovate/renovate image found in renovate/k8s/cronjob.yaml"
exit 1
fi
echo "using $image"
echo "image=$image" >> "$GITHUB_OUTPUT"
- name: Validate Renovate config
shell: bash
run: |
set -euo pipefail
docker run --rm \
-v "$PWD/renovate/renovate.json:/opt/renovate/renovate.json:ro" \
-e RENOVATE_CONFIG_FILE=/opt/renovate/renovate.json \
"${{ steps.image.outputs.image }}" \
renovate-config-validator
- name: Run Renovate
shell: bash
env:
RENOVATE_TOKEN: ${{ secrets.RENOVATE_TOKEN }}
RENOVATE_GITHUB_COM_TOKEN: ${{ secrets.RENOVATE_GITHUB_COM_TOKEN }}
RENOVATE_REPOSITORIES: ${{ inputs.repositories }}
RENOVATE_DRY_RUN: ${{ inputs.dry_run && 'full' || '' }}
LOG_LEVEL: ${{ inputs.log_level }}
run: |
set -euo pipefail
: "${RENOVATE_TOKEN:?missing RENOVATE_TOKEN secret — add a renovate-bot PAT in repo/org Actions secrets}"
docker run --rm \
-v "$PWD/renovate/renovate.json:/opt/renovate/renovate.json:ro" \
-e RENOVATE_PLATFORM=gitea \
-e RENOVATE_ENDPOINT=https://git.forust.xyz/api/v1 \
-e RENOVATE_TOKEN="$RENOVATE_TOKEN" \
-e RENOVATE_GITHUB_COM_TOKEN="${RENOVATE_GITHUB_COM_TOKEN:-}" \
-e RENOVATE_REPOSITORIES="${RENOVATE_REPOSITORIES:-forust/homelab}" \
-e RENOVATE_DRY_RUN="${RENOVATE_DRY_RUN:-}" \
-e RENOVATE_CONFIG_FILE=/opt/renovate/renovate.json \
-e RENOVATE_BASE_DIR=/tmp/renovate \
-e LOG_LEVEL="${LOG_LEVEL:-info}" \
"${{ steps.image.outputs.image }}"
-13
View File
@@ -1,13 +0,0 @@
# kubectl emits a List for files containing multiple resources.
(if .kind == "List" then .items[] else . end)
| (.metadata.namespace // "default") as $ns
| [
(.. | objects
| (.secretRef? // empty), (.secretKeyRef? // empty), (.secret? // empty)
| select(.optional != true)
| .name // .secretName // empty),
(.. | objects | .imagePullSecrets[]?.name)
]
| unique[]
| select(. != null and . != "")
| "\($ns) \(.)"
-70
View File
@@ -1,70 +0,0 @@
#!/usr/bin/env bash
# The SSH client submits once and follows durable stages on workstation.
set -euo pipefail
: "${DEPLOY_HOST:?missing DEPLOY_HOST}"
: "${DEPLOY_USER:?missing DEPLOY_USER}"
: "${DEPLOY_KEY:?missing DEPLOY_SSH_KEY}"
: "${DEPLOY_KNOWN_HOSTS:?configure pinned DEPLOY_KNOWN_HOSTS}"
: "${DEPLOY_RUN_ID:?missing DEPLOY_RUN_ID}"
[[ "$DEPLOY_USER" =~ ^[A-Za-z_][A-Za-z0-9_.-]*$ ]] || exit 1
[[ "$DEPLOY_HOST" =~ ^[A-Za-z0-9_.:-]+$ ]] || exit 1
[[ "$DEPLOY_RUN_ID" =~ ^[0-9]+-[0-9]+$ ]] || exit 1
[[ "${DEPLOY_PORT:-22}" =~ ^[0-9]+$ ]] || exit 1
key_dir="$(mktemp -d "${RUNNER_TEMP:-/tmp}/homelab-deploy-key.XXXXXXXX")"
trap 'rm -rf "$key_dir"' EXIT
chmod 700 "$key_dir"
printf '%s\n' "$DEPLOY_KEY" >"$key_dir/key"
printf '%s\n' "$DEPLOY_KNOWN_HOSTS" >"$key_dir/known_hosts"
chmod 600 "$key_dir/key" "$key_dir/known_hosts"
ssh_opts=(-i "$key_dir/key" -p "${DEPLOY_PORT:-22}" -o BatchMode=yes -o StrictHostKeyChecking=yes
-o "UserKnownHostsFile=$key_dir/known_hosts" -o ConnectTimeout=15
-o ServerAliveInterval=15 -o ServerAliveCountMax=4)
controller=.local/lib/homelab-deploy/controller.py
case "${1:?start, apply, verify, smoke or summary required}" in
start)
python3 - <<'PY' >"$key_dir/request.json"
import json
import os
from pathlib import Path
release = json.loads(Path('release.json').read_text())
print(json.dumps({'release': release, 'mode': os.environ.get('DEPLOY_MODE', 'changed'),
'refresh_images': os.environ.get('REFRESH_IMAGES', 'false') == 'true'}))
PY
for attempt in 1 2 3; do
rc=0
# shellcheck disable=SC2029 # The run ID and operation are validated local arguments, not remote variables.
ssh "${ssh_opts[@]}" "$DEPLOY_USER@$DEPLOY_HOST" python3 "$controller" start "$DEPLOY_RUN_ID" <"$key_dir/request.json" || rc=$?
[ "$rc" -eq 0 ] && exit 0
[ "$rc" -eq 255 ] || exit "$rc"
sleep 5
done
exit "$rc"
;;
apply|verify|smoke)
result=0
for attempt in 1 2 3; do
rc=0
# shellcheck disable=SC2029 # The run ID and operation are validated local arguments, not remote variables.
ssh "${ssh_opts[@]}" "$DEPLOY_USER@$DEPLOY_HOST" python3 "$controller" follow "$DEPLOY_RUN_ID" "$1" || rc=$?
[ "$rc" -eq 0 ] && break
[ "$rc" -eq 255 ] || { result="$rc"; break; }
echo "SSH disconnected; reconnecting to the existing deploy ($attempt/3)"
if [ "$attempt" -eq 3 ]; then result=255; break; fi
sleep 5
done
exit "$result"
;;
summary)
if [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
rc=0
# shellcheck disable=SC2029 # The run ID is validated above.
ssh "${ssh_opts[@]}" "$DEPLOY_USER@$DEPLOY_HOST" python3 "$controller" summary "$DEPLOY_RUN_ID" >"$key_dir/deploy-summary.md" || rc=$?
if [ "$rc" -eq 0 ]; then
cat "$key_dir/deploy-summary.md" >>"$GITHUB_STEP_SUMMARY" || echo "WARNING: cannot write the deploy summary"
else
echo 'Deploy summary is unavailable. The SSH connection failed or the controller did not respond. Check the job log.' >>"$GITHUB_STEP_SUMMARY" || true
fi
fi
;;
*) echo "Unknown SSH operation: $1" >&2; exit 1 ;;
esac
@@ -1,55 +0,0 @@
#!/usr/bin/env bash
# Regenerates renovate/k8s/configmap.yaml from renovate/renovate.json.
#
# renovate/renovate.json is the single source of truth: the CronJob, the Compose
# file and the renovate-run workflow all mount that exact file. A ConfigMap cannot
# read a file from the repository, so the same bytes are inlined here as a literal
# block. This script keeps the copy honest:
#
# .gitea/workflows/sync-renovate-configmap.sh # rewrite in place
# .gitea/workflows/sync-renovate-configmap.sh --check # fail if out of date
#
# renovate-ci runs the --check form on every PR and push, so a config change that
# forgets to regenerate the ConfigMap cannot be merged.
set -euo pipefail
here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
repo="$(git -C "$here" rev-parse --show-toplevel)"
src="$repo/renovate/renovate.json"
dst="$repo/renovate/k8s/configmap.yaml"
[ -f "$src" ] || {
echo "missing $src" >&2
exit 1
}
render() {
cat <<'HEADER'
# GENERATED FILE - do not edit by hand.
# Source: renovate/renovate.json
# Regenerate: .gitea/workflows/sync-renovate-configmap.sh
# Verify: .gitea/workflows/sync-renovate-configmap.sh --check
apiVersion: v1
kind: ConfigMap
metadata:
name: renovate-config
namespace: renovate
data:
renovate.json: |
HEADER
sed 's/^/ /' "$src"
}
if [ "${1:-}" = "--check" ]; then
if ! diff -u "$dst" <(render) >/dev/null 2>&1; then
echo "ERROR: $dst is out of sync with renovate/renovate.json"
echo "Run: .gitea/workflows/sync-renovate-configmap.sh"
diff -u "$dst" <(render) || true
exit 1
fi
echo "renovate/k8s/configmap.yaml is in sync with renovate/renovate.json"
exit 0
fi
render >"$dst"
echo "wrote $dst"
-39
View File
@@ -1,39 +0,0 @@
# Pinned versions of the CI tools installed by install-ci-tools.sh.
# Renovate keeps these up to date (see customManagers in renovate/renovate.json).
#
# Every version here except NODE_VERSION matches what was already installed on
# the runner, so pinning them changes what CI does not at all. It changes what
# CI does when the runner is rebuilt with something else: today
# install-ci-tools.sh finds the pinned version already on PATH and installs
# nothing, and a runner that drifts gets the pinned one installed over it.
#
# The renovate image version is NOT pinned here: renovate/k8s/cronjob.yaml is the
# single source of truth and the workflows read the tag from it, so there is
# nothing to drift.
ACTIONLINT_VERSION="1.7.7"
SHELLCHECK_VERSION="0.11.0"
KUBECONFORM_VERSION="0.8.0"
PRETTIER_VERSION="3.8.1"
RUFF_VERSION="0.16.8"
YAMLLINT_VERSION="1.38.0"
HADOLINT_VERSION="2.14.0"
# pip-audit reads the advisory database over the network, so a floating version
# would make the same commit report different things on different days. Pin it
# like the rest: the advisories themselves are the moving part, not the tool.
PIP_AUDIT_VERSION="2.10.1"
# uv builds the throwaway venv the pytest job runs in, and unpacks the PyPI
# wheels for ruff, yamllint and pip-audit.
UV_VERSION="0.12.17"
# node runs `npm ci` for the frontend tests and the npm audit, and it is the one
# pin here that does NOT come from the runner: the runner's system node is a
# rolling Arch package (it was node 26 with no npm at all when this was pinned),
# and the panel image is node:22-alpine. Pinned to the image's major on purpose,
# so the tree that gets tested is the tree that gets built. Renovate keeps this
# in step with the Dockerfile's node: tag via the "node runtime" group.
NODE_VERSION="22.23.3"
# Secret-reference regression tests parse rendered Kubernetes objects.
JQ_VERSION="1.8.1"
# BuildKit is the only auxiliary CI container; jobs themselves stay on the host.
BUILDKIT_IMAGE="moby/buildkit:v0.33.1"
+4 -27
View File
@@ -5,12 +5,12 @@ sync.ffs_lock
# Copyparty
*.hist/
# Volumes, configs and data directories
# Volumes and data directories
gitea/gitea-db/
gitea/gitea-data/*
n8n/n8n-data/*
n8n/n8n-node-data/*
adguardhome/conf/*
adguardhome/data/*
dockmon/data/*
portainer/portainer_data/*
metube/MeTube_downloads
@@ -21,10 +21,6 @@ checkmk/checkmk/*
downtify/Downtify_downloads
headscale/config/*
headscale/data/*
# NetBird local hostnames and generated secrets
netbird/.env
netbird/secrets/
searxng/core-config/*
# Steaming services files
streaming/jellyfin/*
@@ -36,22 +32,17 @@ streaming/qbittorrent/*
streaming/prowlarr/*
# Homepage
homepages/forust_files/assets/images/team/*
homepages/forust_files/.well-known/*
# Traefik files
traefik/letsencrypt/acme.json
traefik/dynamic/fileservers.yml
traefik/dynamic/*.local.y*ml.*
traefik/dynamic/*.external.y*ml
traefik/k8s/fileservers.y*ml
traefik/k8s/aliasHeadersStrategy.md
traefik/logs/*
# SSL Certificates
adguardhome/certs/*
traefik/certs/*
certs/
# Monitoring
monitoring/prometheus.yml
@@ -86,17 +77,14 @@ replacements.txt
# Git
.gitattributes
# Gitea/github Runners
.runner
# Misc
.DS_Store
.idea
# Temp files
edu_master/temp/
temp/*
# Local-only tooling scratch space (pinned CI tools, verification scripts)
tmp/
# Environment
.env
@@ -104,14 +92,3 @@ tmp/
.env.forust
.env.*
!*example
# kubernetes
*/k8s/*secret*
!*/k8s/*secret*.example
**/k8s/*secret*
!**/k8s/*secret*.example
# Local-only tweaks, not for upstream
prometheus-stack/k8s/grafana-values.yaml
traefik/k8s/local-tls.yaml
converters/k8s/config.yaml
convertx/k8s/config.yaml
-10
View File
@@ -1,10 +0,0 @@
ignored:
- DL3008
- DL3042
- DL3018
- DL3059
trustedRegistries:
- docker.io
- ghcr.io
- quay.io
- gcr.forust.xyz
-8
View File
@@ -1,8 +0,0 @@
{
"default": true,
"MD013": false,
"MD024": false,
"MD033": false,
"MD041": false,
"MD046": false
}
-7
View File
@@ -1,7 +0,0 @@
bracketSameLine: true
htmlWhitespaceSensitivity: css
printWidth: 120
tabWidth: 2
trailingComma: all
proseWrap: preserve
endOfLine: lf
-22
View File
@@ -1,22 +0,0 @@
extends: default
rules:
comments:
min-spaces-from-content: 1
comments-indentation: false
document-start: disable
line-length: disable
braces:
min-spaces-inside: 0
max-spaces-inside: 1
brackets:
min-spaces-inside: 0
max-spaces-inside: 1
indentation:
spaces: 2
indent-sequences: consistent
truthy:
allowed-values:
- "true"
- "false"
- "on"
-40
View File
@@ -1,40 +0,0 @@
{
"tab_size": 2,
"soft_wrap": "prefer_line",
"preferred_line_length": 120,
"format_on_save": "on",
"languages": {
"YAML": {
"tab_size": 2,
"hard_tabs": false,
"format_on_save": "on",
"formatter": {
"language_server": { "name": "yaml-language-server" },
},
},
"Python": {
"tab_size": 4,
"format_on_save": "on",
"language_servers": ["pyright", "ruff"],
"formatter": {
"language_server": { "name": "ruff" },
},
},
},
"lsp": {
"yaml-language-server": {
"settings": {
"yaml": {
"schemas": {
"kubernetes": ["**/k8s/*.yaml", "**/k8s/*.yml"],
},
"validate": true,
"completion": true,
"format": {
"enable": true,
},
},
},
},
},
}
+12 -10
View File
@@ -1,6 +1,6 @@
services:
adguard:
image: adguard/adguardhome:v0.107.79
image: adguard/adguardhome:latest
container_name: adguardhome
restart: unless-stopped
ports:
@@ -11,29 +11,33 @@ services:
# - "68:68/tcp" # DHCP
# - "3000:3000/tcp"
volumes:
- data:/opt/adguardhome/work
- ./conf:/opt/adguardhome/conf
- ./data/work:/opt/adguardhome/work
- ./data/conf:/opt/adguardhome/conf
- ./certs:/certs:ro
labels:
- "traefik.enable=true"
- "traefik.docker.network=proxy"
- "traefik.http.services.adguard.loadbalancer.server.port=3000"
# Prod Router
- "traefik.http.routers.adguard.rule=Host(`dns.forust.xyz`) || Host(`adguard.forust.xyz`)"
- "traefik.http.routers.adguard.entrypoints=websecure"
- "traefik.http.routers.adguard.tls.certresolver=letsencrypt"
- "traefik.http.routers.adguard.middlewares=security-headers@file"
- "traefik.http.routers.adguard.tls=true"
# Local Router
- "traefik.http.routers.adguard-local.rule=Host(`adguard.workstation.internal`) || Host(`dns.workstation.internal`)"
- "traefik.http.routers.adguard-local.entrypoints=websecure"
- "traefik.http.routers.adguard-local.middlewares=security-headers@file"
- "traefik.http.routers.adguard-local.tls=true"
# Dev Router
- "traefik.http.routers.adguard-dev.rule=Host(`adguard.gigaforust.internal`) || Host(`dns.gigaforust.internal`)"
- "traefik.http.routers.adguard-dev.rule=Host(`adguard.gigaforust.internal`) Host(`dns.gigaforust.internal`)"
- "traefik.http.routers.adguard-dev.entrypoints=websecure"
- "traefik.http.routers.adguard-dev.middlewares=security-headers@file"
- "traefik.http.routers.adguard-dev.tls=true"
# DoH Router
- "traefik.http.routers.dns-over-https.rule=(Host(`dns.forust.xyz`) || Host(`adguard.forust.xyz`)) && PathPrefix(`/dns-query`)"
- "traefik.http.routers.dns-over-https.entrypoints=websecure"
- "traefik.http.routers.dns-over-https.tls.certresolver=letsencrypt"
- "traefik.http.routers.dns.rule=(Host(`dns.forust.xyz`) && PathPrefix(`/dns-query`))"
- "traefik.http.routers.dns.entrypoints=websecure"
- "traefik.http.routers.dns.tls.certresolver=letsencrypt"
# Glance Metadata
- glance.name=adguard
@@ -41,8 +45,6 @@ services:
- glance.description=AdGuard Home is a network-wide software for blocking ads.
networks:
- proxy
volumes:
data:
networks:
proxy:
external: true
-127
View File
@@ -1,127 +0,0 @@
apiVersion: v1
kind: Service
metadata:
name: adguard-lb-service
namespace: adguard
annotations:
metallb.io/loadBalancerIPs: "192.168.80.3"
spec:
type: LoadBalancer
externalTrafficPolicy: Local
selector:
app: adguard
ports:
- name: dns-udp
port: 53
targetPort: 53
protocol: UDP
- name: dns-tcp
port: 53
targetPort: 53
protocol: TCP
- name: dot
port: 853
targetPort: 853
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
name: adguard-service
namespace: adguard
spec:
selector:
app: adguard
ports:
- port: 3000
name: webui
targetPort: 3000
- port: 53
name: dns
targetPort: 53
protocol: UDP
- port: 53
name: dns-tcp
targetPort: 53
protocol: TCP
- port: 853
name: dot
targetPort: 853
---
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
reloader.stakater.com/auto: "true"
name: adguard-deployment
namespace: adguard
spec:
replicas: 1
selector:
matchLabels:
app: adguard
strategy:
type: Recreate
template:
metadata:
labels:
app: adguard
spec:
containers:
- name: adguard
image: adguard/adguardhome:v0.107.79
resources:
limits:
memory: "1.5Gi"
cpu: "300m"
requests:
memory: "512Mi"
cpu: "50m"
ports:
- containerPort: 3000
name: webui
- containerPort: 53
name: dns
- containerPort: 853
name: dot
readinessProbe:
tcpSocket:
port: dns
initialDelaySeconds: 5
periodSeconds: 5
successThreshold: 1
failureThreshold: 3
volumeMounts:
- name: adguard-data
mountPath: /opt/adguardhome/work
subPath: work
- name: adguard-data
mountPath: /opt/adguardhome/conf
subPath: conf
- name: adguard-certs
mountPath: /certs
readOnly: true
volumes:
- name: adguard-data
persistentVolumeClaim:
claimName: adguard-pvc
- name: adguard-certs
secret:
secretName: adguard-certs
items:
- key: tls.crt
path: fullchain.pem
- key: tls.key
path: privkey.pem
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: adguard-pvc
namespace: adguard
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 2Gi
-12
View File
@@ -1,12 +0,0 @@
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: adguard-certs
namespace: adguard
spec:
secretName: adguard-certs
dnsNames:
- dns.forust.xyz
issuerRef:
name: letsencrypt-prod
kind: ClusterIssuer
-43
View File
@@ -1,43 +0,0 @@
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: adguard-prod
namespace: adguard
spec:
entryPoints:
- websecure
routes:
- match: Host(`dns.forust.xyz`)
kind: Rule
services:
- name: adguard-service
port: 3000
- match: (Host(`dns.forust.xyz`)) && PathPrefix(`/dns-query`)
kind: Rule
services:
- name: adguard-service
port: 3000
tls:
secretName: adguard-certs
---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: adguard-local
namespace: adguard
spec:
entryPoints:
- websecure
routes:
- match: Host(`adguard.workstation.internal`) || Host(`dns.workstation.internal`) || Host(`adguard.gigaforust.internal`) || Host(`dns.gigaforust.internal`)
kind: Rule
services:
- name: adguard-service
port: 3000
- match: (Host(`adguard.workstation.internal`) || Host(`dns.workstation.internal`) || Host(`adguard.gigaforust.internal`) || Host(`dns.gigaforust.internal`)) && PathPrefix(`/dns-query`)
kind: Rule
services:
- name: adguard-service
port: 3000
tls:
secretName: internal-wildcard-tls
-15
View File
@@ -1,15 +0,0 @@
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: internal-wildcard-tls
namespace: adguard
spec:
secretName: internal-wildcard-tls
dnsNames:
- "*.workstation.internal"
- "*.gigaforust.internal"
- workstation.internal
- gigaforust.internal
issuerRef:
name: internal-ca
kind: ClusterIssuer
-4
View File
@@ -1,4 +0,0 @@
apiVersion: v1
kind: Namespace
metadata:
name: adguard
-10
View File
@@ -1,10 +0,0 @@
kubectl apply -f k8s/namespace.yaml && \
kubectl create secret tls adguard-certs -n adguard \
--cert=certs/fullchain.pem \
--key=certs/privkey.pem --dry-run=client -o yaml > \
k8s/secrets.yaml
# OR WITH NO FILE CREATION:
kubectl create secret tls adguard-certs -n adguard \
--cert=certs/fullchain.pem --key=certs/privkey.pem \
--save-config
+9 -3
View File
@@ -1,6 +1,6 @@
services:
postgresql:
image: docker.io/library/postgres:15.19-alpine
image: docker.io/library/postgres:15-alpine
restart: unless-stopped
env_file:
- .env
@@ -42,20 +42,26 @@ services:
- ./custom-templates:/templates
labels:
- "traefik.enable=true"
- "traefik.docker.network=proxy"
- "traefik.http.services.authentik-server.loadbalancer.server.port=9000"
# Prod Router
- "traefik.http.routers.authentik-server.rule=Host(`auth.forust.xyz`)"
- "traefik.http.routers.authentik-server.entrypoints=websecure"
- "traefik.http.routers.authentik-server.tls.certresolver=letsencrypt"
- "traefik.http.routers.authentik-server.middlewares=security-headers@file"
- "traefik.http.routers.authentik-server.service=authentik-server"
- "traefik.http.routers.authentik-server.tls=true"
# Local Router
- "traefik.http.routers.authentik-server-local.rule=Host(`auth.workstation.internal`)"
- "traefik.http.routers.authentik-server-local.rule=Host(`auth.workstation.internal`) || Host(`auth-dashboard.internal`)"
- "traefik.http.routers.authentik-server-local.entrypoints=websecure"
- "traefik.http.routers.authentik-server-local.middlewares=security-headers@file"
- "traefik.http.routers.authentik-server-local.service=authentik-server"
- "traefik.http.routers.authentik-server-local.tls=true"
# Dev Router
- "traefik.http.routers.authentik-server-dev.rule=Host(`auth.gigaforust.internal`)"
- "traefik.http.routers.authentik-server-dev.entrypoints=websecure"
- "traefik.http.routers.authentik-server-dev.middlewares=security-headers@file"
- "traefik.http.routers.authentik-server-dev.service=authentik-server"
- "traefik.http.routers.authentik-server-dev.tls=true"
networks:
- proxy
-101
View File
@@ -1,101 +0,0 @@
apiVersion: v1
kind: Service
metadata:
name: authentik-server-service
namespace: authentik
spec:
type: ClusterIP
selector:
app: authentik-server
ports:
- port: 9000
targetPort: 9000
---
apiVersion: v1
kind: Service
metadata:
name: authentik-worker-service
namespace: authentik
spec:
type: ClusterIP
selector:
app: authentik-worker
ports:
- port: 9000
targetPort: 9000
---
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
reloader.stakater.com/auto: "true"
name: authentik-server-deployment
namespace: authentik
spec:
replicas: 1
selector:
matchLabels:
app: authentik-server
strategy:
type: Recreate
template:
metadata:
labels:
app: authentik-server
spec:
containers:
- name: authentik-server
image: ghcr.io/goauthentik/server:2026.8.3
args: ["server"]
envFrom:
- configMapRef:
name: authentik-config
- secretRef:
name: authentik-secrets
ports:
- containerPort: 9000
resources:
requests:
memory: "768Mi"
cpu: "100m"
limits:
memory: "1.5Gi"
cpu: "1000m"
---
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
reloader.stakater.com/auto: "true"
name: authentik-worker-deployment
namespace: authentik
spec:
replicas: 1
selector:
matchLabels:
app: authentik-worker
strategy:
type: Recreate
template:
metadata:
labels:
app: authentik-worker
spec:
containers:
- name: authentik-worker
image: ghcr.io/goauthentik/server:2026.8.3
args: ["worker"]
securityContext:
runAsUser: 0
envFrom:
- configMapRef:
name: authentik-config
- secretRef:
name: authentik-secrets
resources:
requests:
memory: "320Mi"
cpu: "100m"
limits:
memory: "768Mi"
cpu: "700m"
-28
View File
@@ -1,28 +0,0 @@
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: authentik-prod-tls
namespace: authentik
spec:
secretName: authentik-prod-tls
dnsNames:
- auth.forust.xyz
issuerRef:
name: letsencrypt-prod
kind: ClusterIssuer
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: internal-wildcard-tls
namespace: authentik
spec:
secretName: internal-wildcard-tls
dnsNames:
- "*.workstation.internal"
- "*.gigaforust.internal"
- workstation.internal
- gigaforust.internal
issuerRef:
name: internal-ca
kind: ClusterIssuer
-11
View File
@@ -1,11 +0,0 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: authentik-config
namespace: authentik
data:
AUTHENTIK_IMAGE: ghcr.io/goauthentik/server
AUTHENTIK_TAG: "2025.10.2"
AUTHENTIK_POSTGRESQL__HOST: postgres.database.svc.cluster.local
AUTHENTIK_POSTGRESQL__NAME: authentik
AUTHENTIK_ERROR_REPORTING__ENABLED: "true"
-33
View File
@@ -1,33 +0,0 @@
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: authentik-prod
namespace: authentik
spec:
entryPoints:
- websecure
routes:
- match: Host(`auth.forust.xyz`)
kind: Rule
services:
- name: authentik-server-service
port: 9000
tls:
secretName: authentik-prod-tls
---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: authentik-local
namespace: authentik
spec:
entryPoints:
- websecure
routes:
- match: Host(`auth.workstation.internal`) || Host(`auth.gigaforust.internal`)
kind: Rule
services:
- name: authentik-server-service
port: 9000
tls:
secretName: internal-wildcard-tls
-4
View File
@@ -1,4 +0,0 @@
apiVersion: v1
kind: Namespace
metadata:
name: authentik
-11
View File
@@ -1,11 +0,0 @@
apiVersion: v1
kind: Secret
metadata:
name: authentik-secrets
namespace: authentik
type: Opaque
stringData:
AUTHENTIK_SECRET_KEY: ""
AUTHENTIK_POSTGRESQL__PASSWORD: ""
AUTHENTIK_POSTGRESQL__USER: authentik
AUTHENTIK_BOOTSTRAP_PASSWORD: authentik
@@ -1,9 +0,0 @@
crds:
enabled: true
prometheus:
servicemonitor:
enabled: true
interval: 60s
scrapeTimeout: 30s
labels:
release: prometheus-stack
-29
View File
@@ -1,29 +0,0 @@
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
spec:
acme:
email: bobrovod@national.shitposting.agency
server: https://acme-staging-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: letsencrypt-staging-account-key
solvers:
- http01:
ingress:
class: traefik
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
email: bobrovod@national.shitposting.agency
server: https://acme-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: letsencrypt-prod-account-key
solvers:
- http01:
ingress:
class: traefik
@@ -1,30 +0,0 @@
-----BEGIN CERTIFICATE-----
MIIFFjCCAv6gAwIBAgIUetKpTfEDOn2985FFMu6G26itT+wwDQYJKoZIhvcNAQEN
BQAwIzEhMB8GA1UEAxMYaG9tZWxhYiBpbnRlcm5hbCByb290IENBMB4XDTI2MDky
MzEyNDA0N1oXDTM2MDkyMDEyNDA0N1owIzEhMB8GA1UEAxMYaG9tZWxhYiBpbnRl
cm5hbCByb290IENBMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAvmNP
ZCOoD8NtNuYJKVXBlTPjX7D7sJCSK5neH7ZbYV5+lmUlEErY8Mik7j37V5k5NfpF
Ig85pOjP7RckTPz5V6ek3yaN40s4AL053sN5ZPauDVYjalaEHTgj5sEMqlLACQWI
yZmJOZspZykae8dIpQnqCoFpRT4FurJ78v4a0ylnFVLMQn/lyCHedwTjkEdtYWYr
ccJy8vQwqkzs/rWvEH1lDqZhennLOrmcCjfonG7D/pruMn4z+6E28p4+ejkRrI6x
luak3KnpT1XMeHtgU21hiRGaMDBchHMFgAhnY1qosymKenXvfTZItwgjZbwa1hJI
GAiDm+jQDKMjzRZ3rH6Xfc0auUcykNz73PpNu1NGm78nndXwCXcXn1LFKNQJ+r1U
sJiyAmUZmXVn4aM4OMf2F38k7wTYIKg7nRGaUkNeKDlNkjA4HvgWw+jwO1KmdHQ/
mOem1rosDWHRK01wg+Gga9mQCnhNhxglg3t/UeSic6uOaRsvaz4qkzHq8MbCujVz
DpKQjqdikYOAXZOs4KlBLWrS7NaK4NzfSD02pBUErh54ruJfY/bWz9KyXzBD/lQZ
VUTKyvUVB0bkVHEdf1jJmX3H4IZRQSF5JPqOBotW6bJI5fEGNBvj9Zxy4nm2WWGz
yyP3uWsQz8U/Wdx9nXZLHInTZBsvgLYtKUAWA30CAwEAAaNCMEAwDgYDVR0PAQH/
BAQDAgKkMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFEkKm2rxPaK6+O9WD80z
BLC6F9QsMA0GCSqGSIb3DQEBDQUAA4ICAQAnFyHz97Umf5VIu+dKTJid7C73VugJ
TIar/xJBs/4CxP+znBxhJjXygRoyIfzoVGWcB2ZSL//vL78Qlts79K/Imc9a4RFF
wMvCxsRXAEQ4TpeWi3ophPNcs4rhsP+gQKQFtnyKP9519bqpfxp0bTqwOV2o18fn
za7rlQViiEnNV58j7CVoM9+mJvVVfBEX1Km+GyJL9GadzbIQ7FxClVJZefCbft93
zHVk9gDOw8ys1XGSR2OUCyCLinXO6mqS16CmBb2MAKXq/YyH7E0N8iotAPGtfA8V
M/0ddy947rY0xCrtECfWwvGQpJS7NRv/Z9b2jCfXrI5LXmL2nfQRg0y9GE4Vjwr+
WxtGU5jOeFt0jQ+xRzcgG0Op+qK3x55l5LSo2hOcOVYbiHxcHEJFgwNi1ADeBFwb
q/HdysfURSOghqjIpMMAUabBp+DBUg2EUF7pIaUqbdqExFYcr9EYisEMiNsmKmN+
8ZbcOeerFKDQj+t/R0bFXa7UBn2UWsjI8zlR74aa2kLDXwtyz/XlO/FlYm66eBFo
2/eYUSeU+S4ej+wUAs/dvjF7f190/DUQGuwOTlLTahqWDztmhCk7qzbECu56CwKT
E5Ect2P72UleYwdblkVOVd352AmiwEzdOaziIRrPh8uenEknH6JBYPO3mjk7cCg+
GPGcNIBctjdXhg==
-----END CERTIFICATE-----
-33
View File
@@ -1,33 +0,0 @@
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: selfsigned
spec:
selfSigned: {}
---
# Homelab internal root CA (10y). Install the .crt on clients (see below).
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: internal-ca-root
namespace: cert-manager
spec:
isCA: true
commonName: homelab internal root CA
duration: 87600h
renewBefore: 7200h
secretName: internal-ca-root
privateKey:
algorithm: RSA
size: 4096
issuerRef:
name: selfsigned
kind: ClusterIssuer
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: internal-ca
spec:
ca:
secretName: internal-ca-root
-4
View File
@@ -1,4 +0,0 @@
apiVersion: v1
kind: Namespace
metadata:
name: cert-manager
-15
View File
@@ -1,15 +0,0 @@
CLOUDFLARE_API_TOKEN=YOUR_CLOUDFLARE_API_TOKEN
DOMAINS=example.com,dns.example.com,mc.example.com,auth.example.com,ssh.example.com
IP4_DOMAINS=
IP6_DOMAINS=
IP4_PROVIDER=cloudflare.trace
IP6_PROVIDER=none # change if you want to update AAAA
UPDATE_CRON=@every 5m
UPDATE_ON_START=true
DELETE_ON_STOP=false
DELETE_ON_FAILURE=true
TTL=1
PROXIED=!is(dns.example.com) && !is(mc.example.com) && !is(ssh.example.com)
EMOJI=true
UPTIMEKUMA=https://uptime-kuma.example.com/api/push/AsaSDFGFkfklaFALSKffkfFKfkfkfkFK?status=up&msg=OK&ping=
REJECT_CLOUDFLARE_IPS=true
+6 -23
View File
@@ -1,30 +1,13 @@
services:
cloudflare-ddns:
image: timothyjmiller/cloudflare-ddns:2.2.0
image: timothyjmiller/cloudflare-ddns:latest
container_name: cloudflare-ddns
restart: unless-stopped
security_opt:
- no-new-privileges:true
network_mode: "host"
# https://github.com/timothymiller/cloudflare-ddns#-quick-start
network_mode: 'host'
environment:
- CLOUDFLARE_API_TOKEN=${CLOUDFLARE_API_TOKEN:?Cloudflare API token is required}
- DOMAINS=${DOMAINS:-}
- IP4_DOMAINS=${IP4_DOMAINS:-}
- IP6_DOMAINS=${IP6_DOMAINS:-}
- IP4_PROVIDER=${IP4_PROVIDER:-cloudflare.trace}
- IP6_PROVIDER=${IP6_PROVIDER:-none}
- UPDATE_CRON=${UPDATE_CRON:-@every 5m}
- UPDATE_ON_START=${UPDATE_ON_START:-true}
- DELETE_ON_STOP=${DELETE_ON_STOP:-false}
- DELETE_ON_FAILURE=${DELETE_ON_FAILURE:-true}
- TTL=${TTL:-1} # 1=auto
# to proxy only "dns.example.com" and "wfs.example.com" use "!is(dns.domain.com) && !is (wfs.domain.com)"
- PROXIED=${PROXIED:-true}
- EMOJI=${EMOJI:-true}
- UPTIMEKUMA=${UPTIMEKUMA:-}
- HEALTHCHECKS=${HEALTHCHECKS:-}
- REJECT_CLOUDFLARE_IPS=${REJECT_CLOUDFLARE_IPS:-true}
# volumes:
# Prefer using environment variables for configuration, config.json legacy support
# - ./config.json:/config.json
- PUID=1000
- PGID=1000
volumes:
- ./config.json:/config.json
+1 -1
View File
@@ -7,7 +7,7 @@
"api_key": {
"api_key": "api_key_here",
"account_email": "your_email_here"
},
}
"zone_id": "your_zone-id",
"subdomains": [
{ "name": "", "proxied": true },
-1
View File
@@ -1 +0,0 @@
secret.yaml
View File
Whitespace-only changes.
-36
View File
@@ -1,36 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
reloader.stakater.com/auto: "true"
name: cfddns
labels:
app: cfddns
spec:
replicas: 1
selector:
matchLabels:
app: cfddns
strategy:
type: Recreate
template:
metadata:
labels:
app: cfddns
spec:
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
containers:
- name: cloudflare-ddns
image: timothyjmiller/cloudflare-ddns:2.2.0
imagePullPolicy: Always
resources:
requests:
memory: "32Mi"
cpu: "30m"
limits:
memory: "128Mi"
cpu: "50m"
envFrom:
- secretRef:
name: cfddns-secrets
-18
View File
@@ -1,18 +0,0 @@
apiVersion: v1
kind: Secret
metadata:
name: cfddns-secrets
type: Opaque
stringData:
CLOUDFLARE_API_TOKEN: your_token
DOMAINS: "example.com,www.example.com"
IP4_PROVIDER: cloudflare.trace
IP6_PROVIDER: none
UPDATE_CRON: "@every 5m"
UPDATE_ON_START: "true"
DELETE_ON_STOP: "false"
DELETE_ON_FAILURE: "true"
TTL: "1"
PROXIED: "true"
EMOJI: "true"
REJECT_CLOUDFLARE_IPS: "true"
+6 -3
View File
@@ -1,6 +1,6 @@
services:
checkmk:
image: "checkmk/check-mk-raw:2.4.0-2026.09.14"
image: "checkmk/check-mk-raw:2.4.0-latest"
container_name: "checkmk"
restart: unless-stopped
# ports:
@@ -13,21 +13,24 @@ services:
environment:
- CMK_PASSWORD=${CMK_PASSWORD:-password}
- CMK_SITE_ID=cmk
- TZ=${TZ:-Etc/UTC}
labels:
- "traefik.enable=true"
- "traefik.docker.network=proxy"
- "traefik.http.services.checkmk.loadbalancer.server.port=5000"
# Prod Router
- "traefik.http.routers.checkmk.rule=Host(`cmk.forust.xyz`)"
- "traefik.http.routers.checkmk.entrypoints=websecure"
- "traefik.http.routers.checkmk.tls.certresolver=letsencrypt"
- "traefik.http.routers.checkmk.middlewares=security-headers@file"
- "traefik.http.routers.checkmk.tls=true"
# Local Router
- "traefik.http.routers.checkmk-local.rule=Host(`cmk.workstation.internal`)"
- "traefik.http.routers.checkmk-local.entrypoints=websecure"
- "traefik.http.routers.checkmk-local.middlewares=security-headers@file"
- "traefik.http.routers.checkmk-local.tls=true"
# Dev Router
- "traefik.http.routers.checkmk-dev.rule=Host(`cmk.gigaforust.internal`)"
- "traefik.http.routers.checkmk-dev.middlewares=security-headers@file"
- "traefik.http.routers.checkmk-dev.entrypoints=websecure"
- "traefik.http.routers.checkmk-dev.tls=true"
networks:
-28
View File
@@ -1,28 +0,0 @@
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: checkmk-prod-tls
namespace: checkmk
spec:
secretName: checkmk-prod-tls
dnsNames:
- cmk.forust.xyz
issuerRef:
name: letsencrypt-prod
kind: ClusterIssuer
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: internal-wildcard-tls
namespace: checkmk
spec:
secretName: internal-wildcard-tls
dnsNames:
- "*.workstation.internal"
- "*.gigaforust.internal"
- workstation.internal
- gigaforust.internal
issuerRef:
name: internal-ca
kind: ClusterIssuer
-79
View File
@@ -1,79 +0,0 @@
apiVersion: v1
kind: Service
metadata:
name: checkmk-service
namespace: checkmk
spec:
selector:
app: checkmk
ports:
- name: web
port: 5000
targetPort: 5000
- name: agent-receiver
port: 8000
targetPort: 8000
---
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
reloader.stakater.com/auto: "true"
name: checkmk-deployment
namespace: checkmk
spec:
replicas: 1
selector:
matchLabels:
app: checkmk
strategy:
type: Recreate
template:
metadata:
labels:
app: checkmk
spec:
containers:
- name: checkmk
image: checkmk/check-mk-raw:2.4.0-2026.09.14
envFrom:
- secretRef:
name: checkmk-secrets
- configMapRef:
name: checkmk-config
ports:
- name: web
containerPort: 5000
- name: agent-receiver
containerPort: 8000
volumeMounts:
- name: sites
mountPath: /omd/sites
- name: tmp
mountPath: /opt/omd/sites/cmk/tmp
resources:
requests:
memory: "2Gi"
cpu: "600m"
limits:
memory: "5Gi"
cpu: "4"
volumes:
- name: sites
persistentVolumeClaim:
claimName: checkmk-sites-pvc
- name: tmp
emptyDir:
medium: Memory
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: checkmk-sites-pvc
namespace: checkmk
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
-8
View File
@@ -1,8 +0,0 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: checkmk-config
namespace: checkmk
data:
TZ: Europe/Bratislava
CMK_SITE_ID: cmk
-49
View File
@@ -1,49 +0,0 @@
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: checkmk-prod
namespace: checkmk
spec:
entryPoints:
- websecure
routes:
- match: Host(`cmk.forust.xyz`)
kind: Rule
services:
- name: checkmk-service
port: 5000
tls:
secretName: checkmk-prod-tls
---
apiVersion: traefik.io/v1alpha1
kind: IngressRouteTCP
metadata:
name: checkmk-agent-receiver
namespace: checkmk
spec:
entryPoints:
- checkmk-agent
routes:
- match: HostSNI(`*`)
services:
- name: checkmk-service
port: 8000
tls:
passthrough: true
---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: checkmk-local
namespace: checkmk
spec:
entryPoints:
- websecure
routes:
- match: Host(`cmk.workstation.internal`) || Host(`cmk.gigaforust.internal`)
kind: Rule
services:
- name: checkmk-service
port: 5000
tls:
secretName: internal-wildcard-tls
-4
View File
@@ -1,4 +0,0 @@
apiVersion: v1
kind: Namespace
metadata:
name: checkmk
-8
View File
@@ -1,8 +0,0 @@
apiVersion: v1
kind: Secret
metadata:
name: checkmk-secrets
namespace: checkmk
type: Opaque
stringData:
CMK_PASSWORD: "password"
-1
View File
@@ -1 +0,0 @@
secret.yaml
-41
View File
@@ -1,41 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
reloader.stakater.com/auto: "true"
name: cloudflared
labels:
app: cloudflared
spec:
replicas: 1
selector:
matchLabels:
app: cloudflared
strategy:
type: Recreate
template:
metadata:
labels:
app: cloudflared
spec:
containers:
- name: cloudflared
image: cloudflare/cloudflared:2026.10.0
imagePullPolicy: IfNotPresent
args:
- tunnel
- --no-autoupdate
- run
env:
- name: TUNNEL_TOKEN
valueFrom:
secretKeyRef:
name: cloudflared-secrets
key: TUNNEL_TOKEN
resources:
requests:
memory: "128Mi"
cpu: "30m"
limits:
memory: "256Mi"
cpu: "200m"
-7
View File
@@ -1,7 +0,0 @@
apiVersion: v1
kind: Secret
metadata:
name: cloudflared-secrets
type: Opaque
stringData:
TUNNEL_TOKEN: your_tunnel_token_here
-9
View File
@@ -1,9 +0,0 @@
ACCOUNT_REGISTRATION=false
HTTP_ALLOWED=false
ALLOW_UNAUTHENTICAED=false
AUTO_DELETE_EVERY_N_HOURS=24
WEBROOT=/convert
HIDE_HISTORY=false
LANGUAGE=en
UNAUTHED_USER_SHARING=false
MAX_CONVERT_PROCESS=0
-70
View File
@@ -1,70 +0,0 @@
services:
convertx:
container_name: convertx
image: ghcr.io/c4illin/convertx:v0.19.0
restart: unless-stopped
ports:
- "9992:3000"
# https://github.com/C4illin/ConvertX#environment-variables
environment:
- JWT_SECRET=$(JWT_SECRET)
- ACCOUNT_REGISTRATION=$(ACCOUNT_REGISTRATION:-false)
- HTTP_ALLOWED=$(HTTP_ALLOWED:-false)
- ALLOW_UNAUTHENTICATED=$(ALLOW_UNAUTHENTICATED:-false)
- AUTO_DELETE_EVERY_N_HOURS=$(AUTO_DELETE_EVERY_N_HOURS:-24)
- WEBROOT=$(WEBROOT)
- HIDE_HISTORY=$(HIDE_HISTORY:-false)
- LANGUAGE=$(LANGUAGE:-en)
- UNAUTHENTICATED_USER_SHARING=$(UNAUTHENTICATED_USER_SHARING:-false)
- MAX_CONVERT_PROCESS=$(MAX_CONVERT_PROCESS:-0)
labels:
- "traefik.enable=true"
- "traefik.http.services.convertx.loadbalancer.server.port=3000"
# Prod Router
- "traefik.http.routers.convertx.rule=(Host(`forust.xyz`) || Host(`www.forust.xyz`)) && PathPrefix(`/convert`)"
- "traefik.http.routers.convertx.entrypoints=websecure"
- "traefik.http.routers.convertx.priority=50"
- "traefik.http.routers.convertx.tls.certresolver=letsencrypt"
# Local Router
- "traefik.http.routers.convertx-local.rule=Host(`workstation.internal`) && PathPrefix(`/convert`)"
- "traefik.http.routers.convertx-local.entrypoints=websecure"
- "traefik.http.routers.convertx-local.priority=50"
- "traefik.http.routers.convertx-local.tls=true"
# Dev Router
- "traefik.http.routers.convertx-dev.rule=Host(`gigaforust.internal`) && PathPrefix(`/convert`)"
- "traefik.http.routers.convertx-dev.entrypoints=websecure"
- "traefik.http.routers.convertx-dev.priority=50"
- "traefik.http.routers.convertx-dev.tls=true"
networks:
- proxy
volumes:
- data:/app/data
bentopdf:
container_name: bentopdf
image: bentopdf/bentopdf@sha256:4eb4ec8f5030faf87c29a73d3d5a2781f28a597cf440c3ab111eb96aee550871
restart: unless-stopped
labels:
- "traefik.enable=true"
- "traefik.http.services.bentopdf.loadbalancer.server.port=8080"
# Prod router
- "traefik.http.routers.bentopdf.rule=Host(`pdf.forust.xyz`)"
- "traefik.http.routers.bentopdf.entrypoints=websecure"
- "traefik.http.routers.bentopdf.tls.certresolver=letsencrypt"
- "traefik.http.routers.bentopdf.tls=true"
# Local router
- "traefik.http.routers.bentopdf-local.rule=Host(`pdf.workstation.internal`)"
- "traefik.http.routers.bentopdf-local.entrypoints=websecure"
- "traefik.http.routers.bentopdf-local.tls=true"
# Dev router
- "traefik.http.routers.bentopdf-dev.rule=Host(`pdf.gigaforust.internal`)"
- "traefik.http.routers.bentopdf-dev.entrypoints=websecure"
- "traefik.http.routers.bentopdf-dev.tls=true"
networks:
- proxy
networks:
proxy:
external: true
volumes:
data:
View File
Whitespace-only changes.
-43
View File
@@ -1,43 +0,0 @@
kind: Service
apiVersion: v1
metadata:
name: bentopdf-service
namespace: converters
spec:
selector:
app: bentopdf
ports:
- port: 8080
targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: bentopdf-deployment
namespace: converters
spec:
replicas: 2
selector:
matchLabels:
app: bentopdf
template:
metadata:
labels:
app: bentopdf
spec:
containers:
- image: bentopdf/bentopdf@sha256:4eb4ec8f5030faf87c29a73d3d5a2781f28a597cf440c3ab111eb96aee550871
imagePullPolicy: Always
name: bentopdf
ports:
- containerPort: 8080
# p95 4M, max 11M over 7 days. Was 50Mi/700Mi.
resources:
requests:
memory: "32Mi"
cpu: "50m"
ephemeral-storage: "100Mi"
limits:
memory: "128Mi"
cpu: "700m"
ephemeral-storage: "5Gi"
-42
View File
@@ -1,42 +0,0 @@
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: convertx-prod-tls
namespace: converters
spec:
secretName: convertx-prod-tls
dnsNames:
- forust.xyz
- www.forust.xyz
issuerRef:
name: letsencrypt-prod
kind: ClusterIssuer
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: bentopdf-prod-tls
namespace: converters
spec:
secretName: bentopdf-prod-tls
dnsNames:
- pdf.forust.xyz
issuerRef:
name: letsencrypt-prod
kind: ClusterIssuer
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: internal-wildcard-tls
namespace: converters
spec:
secretName: internal-wildcard-tls
dnsNames:
- "*.workstation.internal"
- "*.gigaforust.internal"
- workstation.internal
- gigaforust.internal
issuerRef:
name: internal-ca
kind: ClusterIssuer
-16
View File
@@ -1,16 +0,0 @@
# test manifest with docker and k8s config keys mismatch
apiVersion: v1
kind: ConfigMap
metadata:
name: convertx-config
namespace: converters
data:
ACCOUNT_REGISTRATION: "false"
HTTP_ALLOWED: "false"
ALLOW_UNAUTHENTICAED: "false"
AUTO_DELETE_EVERY_N_HOURS: "24"
WEBROOT: "/convert"
HIDE_HISTORY: "false"
LANGUAGE: "en"
UNAUTHED_USER_SHARING: "false"
MAX_CONVERT_PROCESS: "0"
-68
View File
@@ -1,68 +0,0 @@
apiVersion: v1
kind: Service
metadata:
name: convertx-service
namespace: converters
spec:
selector:
app: convertx
ports:
- port: 3000
targetPort: 3000
---
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
reloader.stakater.com/auto: "true"
name: convertx-deployment
namespace: converters
spec:
replicas: 1
selector:
matchLabels:
app: convertx
strategy:
type: Recreate
template:
metadata:
labels:
app: convertx
spec:
containers:
- image: ghcr.io/c4illin/convertx:v0.19.0
name: convertx
envFrom:
- configMapRef:
name: convertx-config
- secretRef:
name: convertx-secrets
ports:
- containerPort: 3000
volumeMounts:
- mountPath: /data
name: data
# p95 85M, max 136M over 7 days, spikes while converting. Was 250Mi/1.5Gi.
resources:
requests:
memory: "128Mi"
cpu: "100m"
limits:
cpu: "1500m"
memory: "512Mi"
volumes:
- name: data
persistentVolumeClaim:
claimName: convertx-pvc
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: convertx-pvc
namespace: converters
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 2Gi
-70
View File
@@ -1,70 +0,0 @@
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: convertx-prod
namespace: converters
spec:
entryPoints:
- websecure
routes:
- match: (Host(`forust.xyz`) || Host(`www.forust.xyz`)) && PathPrefix(`/convert`)
kind: Rule
priority: 50
services:
- name: convertx-service
port: 3000
tls:
secretName: convertx-prod-tls
---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: convertx-local
namespace: converters
spec:
entryPoints:
- websecure
routes:
- match: (Host(`workstation.internal`) || Host(`gigaforust.internal`)) && PathPrefix(`/convert`)
kind: Rule
priority: 50
services:
- name: convertx-service
port: 3000
tls:
secretName: internal-wildcard-tls
---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: bentopdf-prod
namespace: converters
spec:
entryPoints:
- websecure
routes:
- match: Host(`pdf.forust.xyz`)
kind: Rule
services:
- name: bentopdf-service
port: 8080
tls:
secretName: bentopdf-prod-tls
---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: bentopdf-local
namespace: converters
spec:
entryPoints:
- websecure
routes:
- match: Host(`pdf.workstation.internal`) || Host(`pdf.gigaforust.internal`)
kind: Rule
services:
- name: bentopdf-service
port: 8080
tls:
secretName: internal-wildcard-tls
-4
View File
@@ -1,4 +0,0 @@
apiVersion: v1
kind: Namespace
metadata:
name: converters
-8
View File
@@ -1,8 +0,0 @@
apiVersion: v1
kind: Secret
metadata:
name: convertx-secrets
namespace: converters
type: Opaque
stringData:
jwt-secret: ""
-203
View File
@@ -1,203 +0,0 @@
container_runtime: containerd
agent:
acquisition: []
additionalAcquisition:
- labels:
type: traefik
limit: 1000
query: |
{namespace="traefik"}
source: loki
url: http://loki.prometheus.svc.cluster.local:3100/
wait_for_ready: 30s
env:
- name: COLLECTIONS
value: crowdsecurity/traefik crowdsecurity/base-http-scenarios
- name: DISABLE_COLLECTIONS
value: crowdsecurity/sshd
# Bans on 401/403 bursts hurt more than they protect: with L3 enforcement
# a false positive cuts the IP off everything (SSH included), and past
# incidents show legit automation (deploy runner, mesh peers, registry
# pulls) tripping this probe. Probing/XSS/SQLi/CVE scenarios stay.
- name: DISABLE_SCENARIOS
value: crowdsecurity/http-generic-bf
metrics:
enabled: true
serviceMonitor:
additionalLabels:
release: prometheus-stack
enabled: true
# Static machine identity: agent pods mount pre-created LAPI credentials
# (Secret crowdsec-agent-credentials, key local_api_credentials.yaml)
# at the exact path the agent entrypoint expects. Together with the
# patched register-init (enforced by janitor-cronjob.yaml) the agent
# never calls `cscli lapi register` in steady state, so pod names,
# restarts and reboots can no longer break it.
extraVolumes:
- name: static-creds
secret:
secretName: crowdsec-agent-credentials
items:
- key: local_api_credentials.yaml
path: local_api_credentials.yaml
extraVolumeMounts:
- name: static-creds
mountPath: /tmp_config/local_api_credentials.yaml
subPath: local_api_credentials.yaml
readOnly: true
resources:
limits:
cpu: 200m
memory: 500Mi
requests:
cpu: 50m
memory: 100Mi
config:
parsers:
s02-enrich:
mobile-whitelist.yaml: |
name: forust/mobile-whitelist
description: "Whitelist SWAN/4ka mobile network"
whitelist:
reason: "Mobile IP whitelist"
cidr:
- "84.245.64.0/18"
# CrowdSec's own guidance: CIDR allowlisting belongs at the parser stage.
# A parser whitelist discards the event before it reaches a bucket, so
# these addresses never produce an overflow and never become a decision.
# A postoverflow whitelist is checked only *after* the ban exists, and
# the bouncer answers 403 for as long as it does - which is a window we
# do not want the deploy sitting in.
local-network.yaml: |
name: forust/local-network
description: "Whitelist loopback, private and VPN networks"
whitelist:
reason: "Local network"
cidr:
- "127.0.0.0/8"
- "10.0.0.0/8"
- "172.16.0.0/12"
- "192.168.0.0/16"
# CGNAT range (RFC 6598). The workstation and the k0s node live
# here on WireGuard, and 100.64.0.0/10 is not covered by the
# RFC 1918 blocks above.
- "100.64.0.0/10"
- "169.254.0.0/16"
- "fc00::/7"
- "fe80::/10"
vps-whitelist.yaml: |
name: forust/vps-whitelist
description: "Whitelist static VPS"
whitelist:
reason: "VPS"
ip:
- "193.181.211.79"
postoverflows:
s01-whitelist:
# The one whitelist that has to stay here: resolving a hostname is a
# network call, and the docs put expensive lookups in postoverflows on
# purpose - it runs only when a bucket actually overflows.
# ddns.forust.xyz is the public home address, not a private one, so
# forust/local-network does not cover it.
home-dynamic-ip.yaml: |
name: forust/home-dynamic-ip
description: "Whitelist home dynamic IP"
whitelist:
reason: "Home dynamic IP"
expression:
- evt.Overflow.Alert.Source.IP in LookupHost("ddns.forust.xyz")
# LAPI-only main config override, merged over config.yaml. NOTE: the
# chart's own default for this key is REPLACED, not merged, so its
# auto_registration block is repeated verbatim below - drop it and the
# agent can no longer register itself.
config.yaml.local: |
api:
server:
auto_registration: # Activate if not using TLS for authentication
enabled: true
token: "${REGISTRATION_TOKEN}" # /!\ Do not modify this variable (auto-generated and handled by the chart)
allowed_ranges: # /!\ Make sure to adapt to the pod IP ranges used by your cluster
- "127.0.0.1/32"
- "192.168.0.0/16"
- "10.0.0.0/8"
- "172.16.0.0/12"
# This homelab has no egress to console.crowdsec.cloud: DNS does
# not resolve. The LAPI kept trying anyway ("Signal push: N
# signals to push", "capi metrics: sending" every 10s) and each
# attempt sat on a resolver timeout WHILE HOLDING A WRITE
# TRANSACTION, which is what kept stalling per-request decision
# lookups even with WAL enabled. Nothing to share and nothing to
# pull - turn the Central API off instead of letting it block the
# only database writer we have.
online_client:
sharing: false
pull:
community: false
blocklists: false
disable_usage_metrics_export: true
db_config:
# SQLite without WAL serialises every reader behind the writer's
# rollback journal, and the LAPI writes constantly: the agent pushes
# Traefik alerts read from Loki, the metrics collector counts
# decisions, the bouncer touches "last pull" on every request.
# Symptom: decision lookups taking 10-30s (and a second connection
# that could not even open the database) while the LAPI sat at 28m
# CPU - the process was blocked in fsync, not computing. Every
# bouncer-protected request then blew through the plugin timeout and
# fail-closed with 403, on every site at once.
# The PVC is local-path-retain (hostPath), not a network share, so
# WAL is safe here; the crowdsec docs recommend it for exactly this
# ("allowing more concurrency in SQLite that will improve
# performances in most scenarios").
use_wal: true
# Keeps the alert table bounded. At the 5000/7d default the file
# reached 54MB in 15 days off the Traefik access log alone, and the
# metrics collector counts decisions on a timer; a smaller working
# set means fewer full scans. Crowdsec only prunes - SQLite never
# shrinks the file, so the size stays until a manual VACUUM.
flush:
max_items: 1000
max_age: 24h
lapi:
env:
- name: COLLECTIONS
value: crowdsecurity/traefik crowdsecurity/base-http-scenarios
- name: DISABLE_COLLECTIONS
value: crowdsecurity/linux crowdsecurity/sshd
metrics:
enabled: true
serviceMonitor:
additionalLabels:
release: prometheus-stack
enabled: true
persistentVolume:
config:
enabled: true
size: 100Mi
storageClassName: local-path-retain
data:
enabled: true
size: 1Gi
storageClassName: local-path-retain
# LAPI answers a blocking /v1/decisions lookup for EVERY bouncer-protected
# request (whole Traefik front door), so it is the hot path of the proxy.
# At 400m/500Mi it went CPU-throttled and idle lookups measured 1.3-7.4s,
# which pushed requests into the bouncer's fail-closed 403.
# Single replica on purpose: LAPI is stateful (BoltDB on the `data` PVC,
# credentials on the `config` PVC) - two replicas sharing those RWO
# volumes would corrupt the decision store. Scale up CPU, not replicas.
resources:
limits:
cpu: 1500m
memory: 1Gi
requests:
cpu: 250m
memory: 500Mi
service:
type: ClusterIP
storeLAPICscliCredentialsInSecret: true
-32
View File
@@ -1,32 +0,0 @@
apiVersion: v1
data:
crowdsec-overview.json: "{\n \"__inputs\": [\n {\n \"name\": \"DS_PROMETHEUS\",\n \"label\": \"Prometheus\",\n \"description\": \"\",\n \"type\": \"datasource\",\n \"pluginId\": \"prometheus\",\n \"pluginName\": \"Prometheus\"\n }\n ],\n \"__requires\": [\n {\n \"type\": \"grafana\",\n \"id\": \"grafana\",\n \"name\": \"Grafana\",\n \"version\": \"8.1.2\"\n },\n {\n \"type\": \"panel\",\n \"id\": \"graph\",\n \"name\": \"Graph (old)\",\n \"version\": \"\"\n },\n {\n \"type\": \"datasource\",\n \"id\": \"prometheus\",\n \"name\": \"Prometheus\",\n \"version\": \"1.0.0\"\n },\n {\n \"type\": \"panel\",\n \"id\": \"stat\",\n \"name\": \"Stat\",\n \"version\": \"\"\n },\n {\n \"type\": \"panel\",\n \"id\": \"timeseries\",\n \"name\": \"Time series\",\n \"version\": \"\"\n }\n ],\n \"annotations\": {\n \"list\": [\n {\n \"builtIn\": 1,\n \"datasource\": \"-- Grafana --\",\n \"enable\": true,\n \"hide\": true,\n \"iconColor\": \"rgba(0, 211, 255, 1)\",\n \"name\": \"Annotations & Alerts\",\n \"target\": {\n \"limit\": 100,\n \"matchAny\": false,\n \"tags\": [],\n \"type\": \"dashboard\"\n },\n \"type\": \"dashboard\"\n }\n ]\n },\n \"editable\": true,\n \"gnetId\": null,\n \"graphTooltip\": 0,\n \"id\": null,\n \"links\": [],\n \"panels\": [\n {\n \"collapsed\": false,\n \"datasource\": null,\n \"gridPos\": {\n \"h\": 1,\n \"w\": 24,\n \"x\": 0,\n \"y\": 0\n },\n \"id\": 24,\n \"panels\": [],\n \"title\": \"Summary\",\n \"type\": \"row\"\n },\n {\n \"cacheTimeout\": null,\n \"datasource\": \"${DS_PROMETHEUS}\",\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"thresholds\"\n },\n \"mappings\": [\n {\n \"options\": {\n \"match\": \"null\",\n \"result\": {\n \"text\": \"N/A\"\n }\n },\n \"type\": \"special\"\n }\n ],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"#E02F44\",\n \"value\": null\n },\n {\n \"color\": \"#E02F44\",\n \"value\": 10\n },\n {\n \"color\": \"#299c46\",\n \"value\": 10\n }\n ]\n },\n \"unit\": \"none\"\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 6,\n \"x\": 0,\n \"y\": 1\n },\n \"id\": 2,\n \"interval\": null,\n \"links\": [],\n \"maxDataPoints\": 100,\n \"options\": {\n \"colorMode\": \"background\",\n \"graphMode\": \"none\",\n \"justifyMode\": \"auto\",\n \"orientation\": \"horizontal\",\n \"reduceOptions\": {\n \"calcs\": [\n \"lastNotNull\"\n ],\n \"fields\": \"\",\n \"values\": false\n },\n \"text\": {},\n \"textMode\": \"auto\"\n },\n \"pluginVersion\": \"8.1.2\",\n \"targets\": [\n {\n \"exemplar\": true,\n \"expr\": \"count(cs_info)\",\n \"interval\": \"\",\n \"legendFormat\": \"\",\n \"refId\": \"A\"\n }\n ],\n \"timeFrom\": null,\n \"timeShift\": null,\n \"title\": \"Running Crowdsec\",\n \"transparent\": true,\n \"type\": \"stat\"\n },\n {\n \"aliasColors\": {},\n \"bars\": false,\n \"dashLength\": 10,\n \"dashes\": false,\n \"datasource\": \"${DS_PROMETHEUS}\",\n \"decimals\": 1,\n \"fieldConfig\": {\n \"defaults\": {\n \"links\": []\n },\n \"overrides\": []\n },\n \"fill\": 1,\n \"fillGradient\": 0,\n \"gridPos\": {\n \"h\": 8,\n \"w\": 18,\n \"x\": 6,\n \"y\": 1\n },\n \"hiddenSeries\": false,\n \"id\": 8,\n \"legend\": {\n \"alignAsTable\": true,\n \"avg\": false,\n \"current\": false,\n \"max\": false,\n \"min\": false,\n \"rightSide\": true,\n \"show\": true,\n \"sort\": \"total\",\n \"sortDesc\": true,\n \"total\": true,\n \"values\": true\n },\n \"lines\": true,\n \"linewidth\": 1,\n \"nullPointMode\": \"null\",\n \"options\": {\n \"alertThreshold\": true\n },\n \"percentage\": false,\n \"pluginVersion\": \"8.1.2\",\n \"pointradius\": 2,\n \"points\": false,\n \"renLine truncated
kind: ConfigMap
metadata:
labels:
app.kubernetes.io/managed-by: manual
grafana_dashboard: "1"
name: crowdsec-crowdsec-overview
namespace: prometheus
---
apiVersion: v1
data:
crowdsec-lapi-metrics.json: "{\n \"__inputs\": [\n {\n \"name\": \"DS_PROMETHEUS\",\n \"label\": \"Prometheus\",\n \"description\": \"\",\n \"type\": \"datasource\",\n \"pluginId\": \"prometheus\",\n \"pluginName\": \"Prometheus\"\n }\n ],\n \"__requires\": [\n {\n \"type\": \"panel\",\n \"id\": \"bargauge\",\n \"name\": \"Bar gauge\",\n \"version\": \"\"\n },\n {\n \"type\": \"grafana\",\n \"id\": \"grafana\",\n \"name\": \"Grafana\",\n \"version\": \"8.1.2\"\n },\n {\n \"type\": \"datasource\",\n \"id\": \"prometheus\",\n \"name\": \"Prometheus\",\n \"version\": \"1.0.0\"\n }\n ],\n \"annotations\": {\n \"list\": [\n {\n \"builtIn\": 1,\n \"datasource\": \"-- Grafana --\",\n \"enable\": true,\n \"hide\": true,\n \"iconColor\": \"rgba(0, 211, 255, 1)\",\n \"name\": \"Annotations & Alerts\",\n \"target\": {\n \"limit\": 100,\n \"matchAny\": false,\n \"tags\": [],\n \"type\": \"dashboard\"\n },\n \"type\": \"dashboard\"\n }\n ]\n },\n \"editable\": true,\n \"gnetId\": null,\n \"graphTooltip\": 0,\n \"id\": null,\n \"iteration\": 1655915193937,\n \"links\": [],\n \"panels\": [\n {\n \"collapsed\": false,\n \"datasource\": null,\n \"gridPos\": {\n \"h\": 1,\n \"w\": 24,\n \"x\": 0,\n \"y\": 0\n },\n \"id\": 10,\n \"panels\": [],\n \"title\": \"Agents\",\n \"type\": \"row\"\n },\n {\n \"datasource\": \"${DS_PROMETHEUS}\",\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"thresholds\"\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n }\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 0,\n \"y\": 1\n },\n \"id\": 2,\n \"options\": {\n \"displayMode\": \"gradient\",\n \"orientation\": \"vertical\",\n \"reduceOptions\": {\n \"calcs\": [\n \"lastNotNull\"\n ],\n \"fields\": \"\",\n \"values\": false\n },\n \"showUnfilled\": false,\n \"text\": {}\n },\n \"pluginVersion\": \"8.1.2\",\n \"repeat\": \"query0\",\n \"repeatDirection\": \"h\",\n \"targets\": [\n {\n \"exemplar\": false,\n \"expr\": \"sum(rate(cs_lapi_request_duration_seconds_bucket{endpoint=\\\"/v1/watchers/login\\\", instance=\\\"$lapi\\\"}[$__rate_interval])) by (le)\",\n \"format\": \"heatmap\",\n \"interval\": \"\",\n \"legendFormat\": \"{{le}}\",\n \"refId\": \"A\"\n }\n ],\n \"title\": \"Agents Login\",\n \"type\": \"heatmap\"\n },\n {\n \"datasource\": \"${DS_PROMETHEUS}\",\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"thresholds\"\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n }\n ]\n },\n \"unit\": \"none\"\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 12,\n \"x\": 12,\n \"y\": 1\n },\n \"id\": 6,\n \"options\": {\n \"displayMode\": \"gradient\",\n \"orientation\": \"auto\",\n \"reduceOptions\": {\n \"calcs\": [\n \"lastNotNull\"\n ],\n \"fields\": \"\",\n \"values\": false\n },\n \"showUnfilled\": false,\n \"text\": {}\n },\n \"pluginVersion\": \"8.1.2\",\n \"targets\": [\n {\n \"exemplar\": true,\n \"expr\": \"sum(rate(cs_lapi_request_duration_seconds_bucket{endpoint=\\\"/v1/watchers/login\\\"}[$__rate_interval])) by (le)\",\n \"format\": \"heatmap\",\n \"interval\": \"\",\n \"legendFormat\": \"{{le}}\",\n \"refId\": \"A\"\n }\n ],\n \"title\": \"Heartbeat\",\n \"type\": \"heatmap\"\n },\n {\n \"collapsed\": false,\n \"datasource\": null,\n \"gridPos\": {\n \"h\": 1,\n \"w\": 24,\n \"x\": 0,\n \"y\": 9\n },\n \"id\": 12,\n \"panels\": [],\n \"title\": \"Decisions\",\n \"type\": \"row\"\n },\n {\n \"datasource\": \"${DS_PROMETHEUS}\",\n Line truncated
kind: ConfigMap
metadata:
labels:
app.kubernetes.io/managed-by: manual
grafana_dashboard: "1"
name: crowdsec-crowdsec-lapi-metrics
namespace: prometheus
---
apiVersion: v1
data:
crowdsec-insight.json: "{\n \"__inputs\": [\n {\n \"name\": \"DS_PROMETHEUS\",\n \"label\": \"Prometheus\",\n \"description\": \"\",\n \"type\": \"datasource\",\n \"pluginId\": \"prometheus\",\n \"pluginName\": \"Prometheus\"\n }\n ],\n \"__requires\": [\n {\n \"type\": \"panel\",\n \"id\": \"bargauge\",\n \"name\": \"Bar gauge\",\n \"version\": \"\"\n },\n {\n \"type\": \"panel\",\n \"id\": \"gauge\",\n \"name\": \"Gauge\",\n \"version\": \"\"\n },\n {\n \"type\": \"grafana\",\n \"id\": \"grafana\",\n \"name\": \"Grafana\",\n \"version\": \"8.1.2\"\n },\n {\n \"type\": \"datasource\",\n \"id\": \"prometheus\",\n \"name\": \"Prometheus\",\n \"version\": \"1.0.0\"\n },\n {\n \"type\": \"panel\",\n \"id\": \"stat\",\n \"name\": \"Stat\",\n \"version\": \"\"\n }\n ],\n \"annotations\": {\n \"list\": [\n {\n \"builtIn\": 1,\n \"datasource\": \"-- Grafana --\",\n \"enable\": true,\n \"hide\": true,\n \"iconColor\": \"rgba(0, 211, 255, 1)\",\n \"name\": \"Annotations & Alerts\",\n \"target\": {\n \"limit\": 100,\n \"matchAny\": false,\n \"tags\": [],\n \"type\": \"dashboard\"\n },\n \"type\": \"dashboard\"\n }\n ]\n },\n \"editable\": true,\n \"gnetId\": null,\n \"graphTooltip\": 0,\n \"id\": null,\n \"iteration\": 1655915159751,\n \"links\": [],\n \"panels\": [\n {\n \"collapsed\": true,\n \"datasource\": null,\n \"gridPos\": {\n \"h\": 1,\n \"w\": 24,\n \"x\": 0,\n \"y\": 0\n },\n \"id\": 22,\n \"panels\": [\n {\n \"cacheTimeout\": null,\n \"datasource\": \"${DS_PROMETHEUS}\",\n \"fieldConfig\": {\n \"defaults\": {\n \"color\": {\n \"mode\": \"thresholds\"\n },\n \"mappings\": [\n {\n \"options\": {\n \"match\": \"null\",\n \"result\": {\n \"text\": \"N/A\"\n }\n },\n \"type\": \"special\"\n }\n ],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n },\n \"unit\": \"dateTimeAsIso\"\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 9,\n \"w\": 5,\n \"x\": 2,\n \"y\": 1\n },\n \"id\": 2,\n \"interval\": null,\n \"links\": [],\n \"maxDataPoints\": 100,\n \"options\": {\n \"colorMode\": \"none\",\n \"graphMode\": \"none\",\n \"justifyMode\": \"auto\",\n \"orientation\": \"horizontal\",\n \"reduceOptions\": {\n \"calcs\": [\n \"lastNotNull\"\n ],\n \"fields\": \"\",\n \"values\": false\n },\n \"text\": {},\n \"textMode\": \"auto\"\n },\n \"pluginVersion\": \"8.1.2\",\n \"targets\": [\n {\n \"exemplar\": true,\n \"expr\": \"(process_start_time_seconds{instance=\\\"$instance\\\"})*1000\",\n \"interval\": \"\",\n \"legendFormat\": \"{{instance}}\",\n \"refId\": \"A\"\n }\n ],\n \"timeFrom\": null,\n \"timeShift\": null,\n \"title\": \"Up since\",\n \"type\": \"stat\"\n },\n {\n \"datasource\": \"${DS_PROMETHEUS}\",\n \"fieldConfig\": {\n \"defaults\": {\n \"displayName\": \"\",\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n }\n ]\n },\n \"unit\": \"decbytes\"\n },\n \"overrides\": []\n },\n \"gridPos\": {\n \"h\": 9,\n \"w\": 5,\n \"x\": 7,\n \"y\": 1\n },\n \"id\": 4,\n \"options\": {\n \"orientation\": \"auto\",\n \"reduceOptions\": {\n \"calcs\": [\n \"mean\"\n ],\n \"fields\": \"\",\n \"values\": false\n },\Line truncated
kind: ConfigMap
metadata:
labels:
app.kubernetes.io/managed-by: manual
grafana_dashboard: "1"
name: crowdsec-crowdsec-insight
namespace: prometheus
-202
View File
@@ -1,202 +0,0 @@
# CrowdSec self-healing: static machine identity + enforcement loops.
#
# Problem it fixes: the chart's agent init container runs
# `cscli lapi register --machine "$POD_NAME" ...`
# unconditionally. Credentials live in an emptyDir, the machine row lives
# in LAPI's persistent DB. Any init re-run for an already-known pod name
# (kubelet restart, node reboot) dies with
# 403 Forbidden: user '<pod>' already exist
# and the DaemonSet pod sticks in Init forever. Every DS restart also
# leaves an orphan machine row that is never cleaned.
#
# Design (name-independent):
# * Agent identity is a STATIC machine `crowdsec-agent-workstation`
# whose password lives in Secret `crowdsec-agent-credentials`
# (created once, manually - like all other secrets in this repo).
# The secret is mounted into agent pods at
# /tmp_config/local_api_credentials.yaml (see extraVolumeMounts in
# crowdsec-values.yaml), which is exactly the path the agent's main
# container copies into place at startup.
# * The DS init command is patched (strategic merge, by container name)
# to SKIP registration when that file exists, keeping the legacy
# register path only as fallback. Detection marker in the patched
# command: `[ -s /tmp_config`.
# * This CronJob enforces the desired state hourly, so recovery is
# automatic even after `helm upgrade` reverts the DS patch or the
# LAPI database is wiped:
# 1. patch DS init if it still has the unconditional register
# (no-op otherwise - no restart churn);
# 2. prune machines with no heartbeat for 2h (orphan hygiene);
# 3. ensure the static machine exists, recreating it with the
# Secret password if missing (agent retry loops reconnect
# on their own - same name + same password);
# 4. prune bouncer entries idle for 30d.
#
# It used to also delete LePresidente/http-generic-403-bf decisions hourly.
# That was a workaround for the bouncer failing closed on a slow LAPI and
# 403-ing the deploy runner into a 4h ban. The bouncer now polls decisions
# into a cache and never blocks on an unreachable LAPI, so it cannot
# manufacture those 403s any more, and the scenario only fires against real
# scanners - deleting their decisions hourly was undoing a working ban.
#
# Manual apply (crowdsec/k8s is NOT managed by deploy.yaml):
# kubectl apply -f crowdsec/k8s/janitor-cronjob.yaml
# Force a run:
# kubectl create job -n crowdsec --from=cronjob/crowdsec-janitor janitor-now
#
# Helm upgrades: the janitor's strategic patch puts the DS field under
# the `kubectl-patch` field manager, so a plain `helm upgrade` FAILS
# with an SSA conflict on initContainers[].command. Procedure:
# 1. revert init to chart state (kills the conflict):
# helm template crowdsec crowdsec/crowdsec --version <ver> \
# -n crowdsec -f crowdsec/k8s/crowdsec-values.yaml > /tmp/r.yaml
# python3 -c "import yaml,json; ..." # build revert patch from
# the rendered DaemonSet init command, then
# kubectl patch ds crowdsec-agent -n crowdsec \
# --type strategic -p "\$(cat /tmp/revert_patch.json)"
# 2. helm upgrade --install crowdsec ... (no --force needed)
# 3. janitor-now right away (upgrade reverts init; new pods would
# sit in Init until the next hourly run otherwise).
#
# One-time bootstrap (order matters):
# 1. Create Secret + static machine (see commands in chat).
# 2. Apply this file, trigger janitor-now, wait for agent 1/1.
# 3. One-time orphan cleanup:
# kubectl exec -n crowdsec deploy/crowdsec-lapi -- \
# cscli machines prune --duration 1h --force
# 4. Only then `helm upgrade` crowdsec with the extraVolumes values.
# Upgrade reverts the DS patch; trigger janitor-now right after it
# (otherwise new pods sit in Init until the next hourly run, then
# self-heal anyway).
#
# Password rotation: update the Secret, delete the machine
# (`cscli machines delete crowdsec-agent-workstation`), trigger
# janitor-now (recreates it), then `kubectl rollout restart
# ds/crowdsec-agent -n crowdsec` (agent reads the file at startup only).
apiVersion: v1
kind: ServiceAccount
metadata:
name: crowdsec-janitor
namespace: crowdsec
labels:
app.kubernetes.io/part-of: crowdsec
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: crowdsec-janitor
namespace: crowdsec
labels:
app.kubernetes.io/part-of: crowdsec
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create"]
- apiGroups: ["apps"]
resources: ["daemonsets"]
verbs: ["get", "patch"]
# `kubectl exec deploy/<name>` resolves deploy -> replicaset -> pod,
# which needs read access to these (exec itself is pods/exec above).
- apiGroups: ["apps"]
resources: ["deployments", "replicasets"]
verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: crowdsec-janitor
namespace: crowdsec
labels:
app.kubernetes.io/part-of: crowdsec
subjects:
- kind: ServiceAccount
name: crowdsec-janitor
namespace: crowdsec
roleRef:
kind: Role
name: crowdsec-janitor
apiGroup: rbac.authorization.k8s.io
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: crowdsec-janitor
namespace: crowdsec
labels:
app.kubernetes.io/part-of: crowdsec
spec:
schedule: "17 * * * *"
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 3
jobTemplate:
spec:
activeDeadlineSeconds: 300
template:
metadata:
labels:
app.kubernetes.io/part-of: crowdsec
spec:
serviceAccountName: crowdsec-janitor
restartPolicy: OnFailure
containers:
- name: janitor
# Same image the chart itself uses for registration jobs;
# IfNotPresent so it works while the node is offline
# (layer cached from the chart install).
image: alpine/kubectl:latest
imagePullPolicy: IfNotPresent
env:
- name: AGENT_PASSWORD
valueFrom:
secretKeyRef:
name: crowdsec-agent-credentials
key: password
command:
- /bin/sh
- -c
- |
set -eu
LAPI_EXEC="kubectl exec -n crowdsec deploy/crowdsec-lapi --"
echo "== 1. enforce patched agent init =="
CUR=$(kubectl get ds crowdsec-agent -n crowdsec \
-o jsonpath='{.spec.template.spec.initContainers[0].command[2]}')
case "$CUR" in
*'-s /tmp_config'*)
echo "init already patched"
;;
*)
echo "patching init"
WAIT='until nc "$LAPI_HOST" "$LAPI_PORT" -z'
WAIT="$WAIT; do echo waiting for lapi to start; sleep 5; done"
LINK='ln -s /staging/etc/crowdsec /etc/crowdsec'
REG='cscli lapi register --machine "$USERNAME"'
REG="$REG -u \"\$LAPI_URL\" --token \"\$REGISTRATION_TOKEN\""
CREDS=/tmp_config/local_api_credentials.yaml
CMD="$WAIT; $LINK; [ -s $CREDS ] || {"
CMD="$CMD $REG && cp"
CMD="$CMD /etc/crowdsec/local_api_credentials.yaml $CREDS; }"
ESC=$(printf '%s' "$CMD" | sed 's/"/\\"/g')
PATCH='{"spec":{"template":{"spec":{"initContainers":'
PATCH=$PATCH'[{"name":"wait-for-lapi-and-register",'
PATCH=$PATCH'"command":["sh","-c","'$ESC'"]}]}}}}'
kubectl patch ds crowdsec-agent -n crowdsec \
--type strategic -p "$PATCH"
;;
esac
echo "== 2. prune orphan machines (no heartbeat for 2h) =="
$LAPI_EXEC cscli machines prune --duration 2h --force
echo "== 3. ensure static machine exists =="
if $LAPI_EXEC cscli machines inspect \
crowdsec-agent-workstation >/dev/null 2>&1; then
echo "static machine present"
else
echo "recreating static machine"
$LAPI_EXEC cscli machines add crowdsec-agent-workstation \
--password "$AGENT_PASSWORD" --force
fi
echo "== 4. prune stale bouncers (no pull for 30d) =="
$LAPI_EXEC cscli bouncers prune -d 720h --force
-6
View File
@@ -1,6 +0,0 @@
apiVersion: v1
kind: Namespace
metadata:
name: crowdsec
labels:
app.kubernetes.io/part-of: crowdsec
-34
View File
@@ -1,34 +0,0 @@
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: crowdsec-lapi
namespace: crowdsec
spec:
podSelector:
matchLabels:
k8s-app: crowdsec
type: lapi
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: traefik
podSelector:
matchLabels:
app.kubernetes.io/name: traefik
- podSelector:
matchLabels:
k8s-app: crowdsec
type: agent
ports:
- protocol: TCP
port: 8080
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: prometheus
ports:
- protocol: TCP
port: 6060
+45 -41
View File
@@ -1,46 +1,50 @@
services:
dockmon:
image: darthnorse/dockmon:2.5.0
container_name: dockmon
restart: unless-stopped
# ports:
# - 8000:443
volumes:
- data:/app/data
- /var/run/docker.sock:/var/run/docker.sock
healthcheck:
test: ["CMD", "curl", "-k", "-f", "https://localhost:443/health"]
interval: 30s
timeout: 10s
retries: 3
labels:
- "traefik.enable=true"
- "traefik.http.services.dockmon.loadbalancer.server.port=443"
- "traefik.http.services.dockmon.loadbalancer.server.scheme=https"
- "traefik.http.services.dockmon.loadbalancer.serverstransport=insecureTransport@file"
dockmon:
image: darthnorse/dockmon:latest
container_name: dockmon
restart: unless-stopped
# ports:
# - 8000:443
volumes:
- data:/app/data
- /var/run/docker.sock:/var/run/docker.sock
healthcheck:
test: [ "CMD", "curl", "-k", "-f", "https://localhost:443/health" ]
interval: 30s
timeout: 10s
retries: 3
labels:
- "traefik.enable=true"
- "traefik.docker.network=proxy"
- "traefik.http.services.dockmon.loadbalancer.server.port=443"
- "traefik.http.services.dockmon.loadbalancer.server.scheme=https"
- "traefik.http.services.dockmon.loadbalancer.serverstransport=insecureTransport@file"
# Prod Router
- "traefik.http.routers.dockmon.rule=Host(`dockmon.forust.xyz`)"
- "traefik.http.routers.dockmon.entrypoints=websecure"
- "traefik.http.routers.dockmon.middlewares=security-headers@file"
- "traefik.http.routers.dockmon.tls.certresolver=letsencrypt"
# Local Router
- "traefik.http.routers.dockmon-local.rule=Host(`dockmon.workstation.internal`)"
- "traefik.http.routers.dockmon-local.entrypoints=websecure"
- "traefik.http.routers.dockmon-local.tls=true"
# Dev Router
- "traefik.http.routers.dockmon-dev.rule=Host(`dockmon.gigaforust.internal`)"
- "traefik.http.routers.dockmon-dev.entrypoints=websecure"
- "traefik.http.routers.dockmon-dev.tls=true"
# Prod Router
- "traefik.http.routers.dockmon.rule=Host(`dockmon.forust.xyz`)"
- "traefik.http.routers.dockmon.entrypoints=websecure"
- "traefik.http.routers.dockmon.middlewares=security-chain@file"
- "traefik.http.routers.dockmon.service=dockmon"
- "traefik.http.routers.dockmon.tls=true"
# Local Router
- "traefik.http.routers.dockmon-local.rule=Host(`dockmon.workstation.internal`)"
- "traefik.http.routers.dockmon-local.entrypoints=websecure"
- "traefik.http.routers.dockmon-local.middlewares=security-headers@file"
- "traefik.http.routers.dockmon-local.tls=true"
# Dev Router
- "traefik.http.routers.dockmon-dev.rule=Host(`dockmon.gigaforust.internal`)"
- "traefik.http.routers.dockmon-dev.entrypoints=websecure"
- "traefik.http.routers.dockmon-dev.middlewares=security-chain@file"
- "traefik.http.routers.dockmon-dev.tls=true"
# Glance Metadata
- glance.name=dockmon
- glance.url=https://dockmon.forust.xyz/
- glance.description=Dockmon is a lightweight Docker container monitoring and management tool with a user-friendly web interface.
networks:
- proxy
# Glance Metadata
- glance.name=dockmon
- glance.url=https://dockmon.forust.xyz/
- glance.description=Dockmon is a lightweight Docker container monitoring and management tool with a user-friendly web interface.
networks:
- proxy
volumes:
data:
data:
networks:
proxy:
external: true
proxy:
external: true
-28
View File
@@ -1,28 +0,0 @@
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: dockmon-prod-tls
namespace: dockmon
spec:
secretName: dockmon-prod-tls
dnsNames:
- dockmon.forust.xyz
issuerRef:
name: letsencrypt-prod
kind: ClusterIssuer
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: internal-wildcard-tls
namespace: dockmon
spec:
secretName: internal-wildcard-tls
dnsNames:
- "*.workstation.internal"
- "*.gigaforust.internal"
- workstation.internal
- gigaforust.internal
issuerRef:
name: internal-ca
kind: ClusterIssuer
-68
View File
@@ -1,68 +0,0 @@
apiVersion: v1
kind: Service
metadata:
name: dockmon-service
namespace: dockmon
spec:
clusterIP: None
selector:
app: dockmon
ports:
- port: 443
targetPort: 443
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: dockmon-statefulset
namespace: dockmon
spec:
serviceName: dockmon-service
replicas: 1
selector:
matchLabels:
app: dockmon
template:
metadata:
labels:
app: dockmon
spec:
containers:
- name: dockmon
image: darthnorse/dockmon:2.5.0
ports:
- containerPort: 443
volumeMounts:
- name: data
mountPath: /app/data
- name: docker-sock
mountPath: /var/run/docker.sock
livenessProbe:
httpGet:
path: /health
port: 443
scheme: HTTPS
initialDelaySeconds: 30
periodSeconds: 30
timeoutSeconds: 10
failureThreshold: 3
resources:
requests:
memory: "512Mi"
cpu: "200m"
limits:
memory: "1.5Gi"
cpu: "700m "
volumes:
- name: docker-sock
hostPath:
path: /var/run/docker.sock
type: Socket
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 1Gi
-45
View File
@@ -1,45 +0,0 @@
apiVersion: traefik.io/v1alpha1
kind: ServersTransport
metadata:
name: dockmon-transport
namespace: dockmon
spec:
insecureSkipVerify: true
---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: dockmon-prod
namespace: dockmon
spec:
entryPoints:
- websecure
routes:
- match: Host(`dockmon.forust.xyz`)
kind: Rule
middlewares:
- name: security-headers@file
services:
- name: dockmon-service
port: 443
serversTransport: dockmon-transport
tls:
secretName: dockmon-prod-tls
---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: dockmon-local
namespace: dockmon
spec:
entryPoints:
- websecure
routes:
- match: Host(`dockmon.workstation.internal`) || Host(`dockmon.gigaforust.internal`)
kind: Rule
services:
- name: dockmon-service
port: 443
serversTransport: dockmon-transport
tls:
secretName: internal-wildcard-tls
-4
View File
@@ -1,4 +0,0 @@
apiVersion: v1
kind: Namespace
metadata:
name: dockmon
+16 -14
View File
@@ -1,29 +1,31 @@
services:
downtify:
container_name: downtify
image: ghcr.io/henriquesebastiao/downtify:3.4.0
restart: unless-stopped
image: ghcr.io/henriquesebastiao/downtify:latest
# ports:
# - '7077:8000'
volumes:
- ./Downtify_downloads:/downloads
labels:
- "traefik.enable=true"
- "traefik.http.services.downtify.loadbalancer.server.port=8000"
- traefik.enable=true
- traefik.docker.network=proxy
- traefik.http.services.downtify.loadbalancer.server.port=8000
# Prod Router
- "traefik.http.routers.downtify.rule=Host(`downtify.forust.xyz`)"
- "traefik.http.routers.downtify.entrypoints=websecure"
- "traefik.http.routers.downtify.middlewares=security-chain@file"
- "traefik.http.routers.downtify.tls.certresolver=letsencrypt"
- traefik.http.routers.downtify.rule=Host(`downtify.forust.xyz`)
- traefik.http.routers.downtify.entrypoints=websecure
- traefik.http.routers.downtify.middlewares=security-chain@file
- traefik.http.routers.downtify.tls=true
# Local Router
- "traefik.http.routers.downtify-local.rule=Host(`downtify.workstation.internal`)"
- "traefik.http.routers.downtify-local.entrypoints=websecure"
- "traefik.http.routers.downtify-local.tls=true"
- traefik.http.routers.downtify-local.rule=Host(`downtify.workstation.internal`) || Host(`downtify.internal`)
- traefik.http.routers.downtify-local.entrypoints=websecure
- traefik.http.routers.downtify-local.middlewares=security-headers@file
- traefik.http.routers.downtify-local.tls=true
# Dev Router
- "traefik.http.routers.downtify-dev.rule=Host(`downtify.gigaforust.internal`)"
- "traefik.http.routers.downtify-dev.entrypoints=websecure"
- "traefik.http.routers.downtify-dev.tls=true"
- traefik.http.routers.downtify-dev.rule=Host(`downtify.gigaforust.internal`)
- traefik.http.routers.downtify-dev.entrypoints=websecure
- traefik.http.routers.downtify-dev.middlewares=security-chain@file
- traefik.http.routers.downtify-dev.tls=true
networks:
- proxy
networks:
-60
View File
@@ -1,60 +0,0 @@
apiVersion: v1
kind: Service
metadata:
name: downtify-service
namespace: downtify
spec:
selector:
app: downtify
ports:
- port: 8000
targetPort: 8000
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: downtify-deployment
namespace: downtify
spec:
replicas: 1
selector:
matchLabels:
app: downtify
strategy:
type: Recreate
template:
metadata:
labels:
app: downtify
spec:
containers:
- name: downtify
image: ghcr.io/henriquesebastiao/downtify:3.4.0
ports:
- containerPort: 8000
volumeMounts:
- name: downloads
mountPath: /downloads
resources:
requests:
memory: "128Mi"
cpu: "200m"
limits:
memory: "1Gi"
cpu: "1"
volumes:
- name: downloads
persistentVolumeClaim:
claimName: downtify-downloads-pvc
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: downtify-downloads-pvc
namespace: downtify
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
-35
View File
@@ -1,35 +0,0 @@
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: downtify-prod
namespace: downtify
spec:
entryPoints:
- websecure
routes:
- match: Host(`downtify.forust.xyz`)
kind: Rule
middlewares:
- name: security-chain@file
services:
- name: downtify-service
port: 8000
tls:
secretName: downtify-prod-tls
---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: downtify-local
namespace: downtify
spec:
entryPoints:
- websecure
routes:
- match: Host(`downtify.workstation.internal`) || Host(`downtify.gigaforust.internal`)
kind: Rule
services:
- name: downtify-service
port: 8000
tls:
secretName: internal-wildcard-tls
-4
View File
@@ -1,4 +0,0 @@
apiVersion: v1
kind: Namespace
metadata:
name: downtify
+13
View File
@@ -0,0 +1,13 @@
FROM python:3.9-alpine
WORKDIR /app
# Установка зависимостей
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# Копирование кода
COPY main.py .
COPY .env .
# Запуск бота
CMD ["python", "-u", "main.py"]
+373
View File
@@ -0,0 +1,373 @@
Mozilla Public License Version 2.0
==================================
1. Definitions
--------------
1.1. "Contributor"
means each individual or legal entity that creates, contributes to
the creation of, or owns Covered Software.
1.2. "Contributor Version"
means the combination of the Contributions of others (if any) used
by a Contributor and that particular Contributor's Contribution.
1.3. "Contribution"
means Covered Software of a particular Contributor.
1.4. "Covered Software"
means Source Code Form to which the initial Contributor has attached
the notice in Exhibit A, the Executable Form of such Source Code
Form, and Modifications of such Source Code Form, in each case
including portions thereof.
1.5. "Incompatible With Secondary Licenses"
means
(a) that the initial Contributor has attached the notice described
in Exhibit B to the Covered Software; or
(b) that the Covered Software was made available under the terms of
version 1.1 or earlier of the License, but not also under the
terms of a Secondary License.
1.6. "Executable Form"
means any form of the work other than Source Code Form.
1.7. "Larger Work"
means a work that combines Covered Software with other material, in
a separate file or files, that is not Covered Software.
1.8. "License"
means this document.
1.9. "Licensable"
means having the right to grant, to the maximum extent possible,
whether at the time of the initial grant or subsequently, any and
all of the rights conveyed by this License.
1.10. "Modifications"
means any of the following:
(a) any file in Source Code Form that results from an addition to,
deletion from, or modification of the contents of Covered
Software; or
(b) any new file in Source Code Form that contains any Covered
Software.
1.11. "Patent Claims" of a Contributor
means any patent claim(s), including without limitation, method,
process, and apparatus claims, in any patent Licensable by such
Contributor that would be infringed, but for the grant of the
License, by the making, using, selling, offering for sale, having
made, import, or transfer of either its Contributions or its
Contributor Version.
1.12. "Secondary License"
means either the GNU General Public License, Version 2.0, the GNU
Lesser General Public License, Version 2.1, the GNU Affero General
Public License, Version 3.0, or any later versions of those
licenses.
1.13. "Source Code Form"
means the form of the work preferred for making modifications.
1.14. "You" (or "Your")
means an individual or a legal entity exercising rights under this
License. For legal entities, "You" includes any entity that
controls, is controlled by, or is under common control with You. For
purposes of this definition, "control" means (a) the power, direct
or indirect, to cause the direction or management of such entity,
whether by contract or otherwise, or (b) ownership of more than
fifty percent (50%) of the outstanding shares or beneficial
ownership of such entity.
2. License Grants and Conditions
--------------------------------
2.1. Grants
Each Contributor hereby grants You a world-wide, royalty-free,
non-exclusive license:
(a) under intellectual property rights (other than patent or trademark)
Licensable by such Contributor to use, reproduce, make available,
modify, display, perform, distribute, and otherwise exploit its
Contributions, either on an unmodified basis, with Modifications, or
as part of a Larger Work; and
(b) under Patent Claims of such Contributor to make, use, sell, offer
for sale, have made, import, and otherwise transfer either its
Contributions or its Contributor Version.
2.2. Effective Date
The licenses granted in Section 2.1 with respect to any Contribution
become effective for each Contribution on the date the Contributor first
distributes such Contribution.
2.3. Limitations on Grant Scope
The licenses granted in this Section 2 are the only rights granted under
this License. No additional rights or licenses will be implied from the
distribution or licensing of Covered Software under this License.
Notwithstanding Section 2.1(b) above, no patent license is granted by a
Contributor:
(a) for any code that a Contributor has removed from Covered Software;
or
(b) for infringements caused by: (i) Your and any other third party's
modifications of Covered Software, or (ii) the combination of its
Contributions with other software (except as part of its Contributor
Version); or
(c) under Patent Claims infringed by Covered Software in the absence of
its Contributions.
This License does not grant any rights in the trademarks, service marks,
or logos of any Contributor (except as may be necessary to comply with
the notice requirements in Section 3.4).
2.4. Subsequent Licenses
No Contributor makes additional grants as a result of Your choice to
distribute the Covered Software under a subsequent version of this
License (see Section 10.2) or under the terms of a Secondary License (if
permitted under the terms of Section 3.3).
2.5. Representation
Each Contributor represents that the Contributor believes its
Contributions are its original creation(s) or it has sufficient rights
to grant the rights to its Contributions conveyed by this License.
2.6. Fair Use
This License is not intended to limit any rights You have under
applicable copyright doctrines of fair use, fair dealing, or other
equivalents.
2.7. Conditions
Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted
in Section 2.1.
3. Responsibilities
-------------------
3.1. Distribution of Source Form
All distribution of Covered Software in Source Code Form, including any
Modifications that You create or to which You contribute, must be under
the terms of this License. You must inform recipients that the Source
Code Form of the Covered Software is governed by the terms of this
License, and how they can obtain a copy of this License. You may not
attempt to alter or restrict the recipients' rights in the Source Code
Form.
3.2. Distribution of Executable Form
If You distribute Covered Software in Executable Form then:
(a) such Covered Software must also be made available in Source Code
Form, as described in Section 3.1, and You must inform recipients of
the Executable Form how they can obtain a copy of such Source Code
Form by reasonable means in a timely manner, at a charge no more
than the cost of distribution to the recipient; and
(b) You may distribute such Executable Form under the terms of this
License, or sublicense it under different terms, provided that the
license for the Executable Form does not attempt to limit or alter
the recipients' rights in the Source Code Form under this License.
3.3. Distribution of a Larger Work
You may create and distribute a Larger Work under terms of Your choice,
provided that You also comply with the requirements of this License for
the Covered Software. If the Larger Work is a combination of Covered
Software with a work governed by one or more Secondary Licenses, and the
Covered Software is not Incompatible With Secondary Licenses, this
License permits You to additionally distribute such Covered Software
under the terms of such Secondary License(s), so that the recipient of
the Larger Work may, at their option, further distribute the Covered
Software under the terms of either this License or such Secondary
License(s).
3.4. Notices
You may not remove or alter the substance of any license notices
(including copyright notices, patent notices, disclaimers of warranty,
or limitations of liability) contained within the Source Code Form of
the Covered Software, except that You may alter any license notices to
the extent required to remedy known factual inaccuracies.
3.5. Application of Additional Terms
You may choose to offer, and to charge a fee for, warranty, support,
indemnity or liability obligations to one or more recipients of Covered
Software. However, You may do so only on Your own behalf, and not on
behalf of any Contributor. You must make it absolutely clear that any
such warranty, support, indemnity, or liability obligation is offered by
You alone, and You hereby agree to indemnify every Contributor for any
liability incurred by such Contributor as a result of warranty, support,
indemnity or liability terms You offer. You may include additional
disclaimers of warranty and limitations of liability specific to any
jurisdiction.
4. Inability to Comply Due to Statute or Regulation
---------------------------------------------------
If it is impossible for You to comply with any of the terms of this
License with respect to some or all of the Covered Software due to
statute, judicial order, or regulation then You must: (a) comply with
the terms of this License to the maximum extent possible; and (b)
describe the limitations and the code they affect. Such description must
be placed in a text file included with all distributions of the Covered
Software under this License. Except to the extent prohibited by statute
or regulation, such description must be sufficiently detailed for a
recipient of ordinary skill to be able to understand it.
5. Termination
--------------
5.1. The rights granted under this License will terminate automatically
if You fail to comply with any of its terms. However, if You become
compliant, then the rights granted under this License from a particular
Contributor are reinstated (a) provisionally, unless and until such
Contributor explicitly and finally terminates Your grants, and (b) on an
ongoing basis, if such Contributor fails to notify You of the
non-compliance by some reasonable means prior to 60 days after You have
come back into compliance. Moreover, Your grants from a particular
Contributor are reinstated on an ongoing basis if such Contributor
notifies You of the non-compliance by some reasonable means, this is the
first time You have received notice of non-compliance with this License
from such Contributor, and You become compliant prior to 30 days after
Your receipt of the notice.
5.2. If You initiate litigation against any entity by asserting a patent
infringement claim (excluding declaratory judgment actions,
counter-claims, and cross-claims) alleging that a Contributor Version
directly or indirectly infringes any patent, then the rights granted to
You by any and all Contributors for the Covered Software under Section
2.1 of this License shall terminate.
5.3. In the event of termination under Sections 5.1 or 5.2 above, all
end user license agreements (excluding distributors and resellers) which
have been validly granted by You or Your distributors under this License
prior to termination shall survive termination.
************************************************************************
* *
* 6. Disclaimer of Warranty *
* ------------------------- *
* *
* Covered Software is provided under this License on an "as is" *
* basis, without warranty of any kind, either expressed, implied, or *
* statutory, including, without limitation, warranties that the *
* Covered Software is free of defects, merchantable, fit for a *
* particular purpose or non-infringing. The entire risk as to the *
* quality and performance of the Covered Software is with You. *
* Should any Covered Software prove defective in any respect, You *
* (not any Contributor) assume the cost of any necessary servicing, *
* repair, or correction. This disclaimer of warranty constitutes an *
* essential part of this License. No use of any Covered Software is *
* authorized under this License except under this disclaimer. *
* *
************************************************************************
************************************************************************
* *
* 7. Limitation of Liability *
* -------------------------- *
* *
* Under no circumstances and under no legal theory, whether tort *
* (including negligence), contract, or otherwise, shall any *
* Contributor, or anyone who distributes Covered Software as *
* permitted above, be liable to You for any direct, indirect, *
* special, incidental, or consequential damages of any character *
* including, without limitation, damages for lost profits, loss of *
* goodwill, work stoppage, computer failure or malfunction, or any *
* and all other commercial damages or losses, even if such party *
* shall have been informed of the possibility of such damages. This *
* limitation of liability shall not apply to liability for death or *
* personal injury resulting from such party's negligence to the *
* extent applicable law prohibits such limitation. Some *
* jurisdictions do not allow the exclusion or limitation of *
* incidental or consequential damages, so this exclusion and *
* limitation may not apply to You. *
* *
************************************************************************
8. Litigation
-------------
Any litigation relating to this License may be brought only in the
courts of a jurisdiction where the defendant maintains its principal
place of business and such litigation shall be governed by laws of that
jurisdiction, without reference to its conflict-of-law provisions.
Nothing in this Section shall prevent a party's ability to bring
cross-claims or counter-claims.
9. Miscellaneous
----------------
This License represents the complete agreement concerning the subject
matter hereof. If any provision of this License is held to be
unenforceable, such provision shall be reformed only to the extent
necessary to make it enforceable. Any law or regulation which provides
that the language of a contract shall be construed against the drafter
shall not be used to construe this License against a Contributor.
10. Versions of the License
---------------------------
10.1. New Versions
Mozilla Foundation is the license steward. Except as provided in Section
10.3, no one other than the license steward has the right to modify or
publish new versions of this License. Each version will be given a
distinguishing version number.
10.2. Effect of New Versions
You may distribute the Covered Software under the terms of the version
of the License under which You originally received the Covered Software,
or under the terms of any subsequent version published by the license
steward.
10.3. Modified Versions
If you create software not governed by this License, and you want to
create a new license for such software, you may create and use a
modified version of this License if you rename the license and remove
any references to the name of the license steward (except to note that
such modified license differs from this License).
10.4. Distributing Source Code Form that is Incompatible With Secondary
Licenses
If You choose to distribute Source Code Form that is Incompatible With
Secondary Licenses under the terms of this version of the License, the
notice described in Exhibit B of this License must be attached.
Exhibit A - Source Code Form License Notice
-------------------------------------------
This Source Code Form is subject to the terms of the Mozilla Public
License, v. 2.0. If a copy of the MPL was not distributed with this
file, You can obtain one at https://mozilla.org/MPL/2.0/.
If it is not possible or desirable to put the notice in a particular
file, then You may include the notice in a location (such as a LICENSE
file in a relevant directory) where a recipient would be likely to look
for such a notice.
You may add additional accurate notices of copyright ownership.
Exhibit B - "Incompatible With Secondary Licenses" Notice
---------------------------------------------------------
This Source Code Form is "Incompatible With Secondary Licenses", as
defined by the Mozilla Public License, v. 2.0.
+14
View File
@@ -0,0 +1,14 @@
services:
dtek_notif:
build:
context: .
dockerfile: Dockerfile
restart: unless-stopped
environment:
- TZ=Europe/Kyiv
dns:
- 1.1.1.1
- 8.8.8.8
networks:
- default
+778
View File
@@ -0,0 +1,778 @@
import requests
import json
import asyncio
from bs4 import BeautifulSoup
from datetime import datetime, timedelta
from aiogram import Bot, Dispatcher, types
from aiogram.filters import Command
from aiogram.types import Message, KeyboardButton
from aiogram.utils.keyboard import ReplyKeyboardBuilder
import logging
import os
from dotenv import load_dotenv
from typing import Optional, List, Dict
# Загрузка переменных окружения
load_dotenv()
# Настройки
TELEGRAM_TOKEN = os.getenv("TELEGRAM_TOKEN", "YOUR_TOKEN_HERE")
ALLOWED_CHAT_IDS = list(map(int, os.getenv("ALLOWED_CHAT_IDS", "").split(","))) if os.getenv("ALLOWED_CHAT_IDS") else []
CHECK_INTERVAL = int(os.getenv("CHECK_INTERVAL", "120"))
# Параметры для запроса
VOE_CITY_ID = int(os.getenv("VOE_CITY_ID", "VOE_CITY_ID"))
VOE_STREET_ID = int(os.getenv("VOE_STREET_ID", "VOE_STREET_ID"))
VOE_HOUSE_ID = int(os.getenv("VOE_HOUSE_ID", "VOE_HOUSE_ID"))
# Настройка логирования
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
logger = logging.getLogger(__name__)
# Глобальные переменные
bot = Bot(token=TELEGRAM_TOKEN)
dp = Dispatcher()
last_schedule: Optional[List[Dict]] = None
last_notification_time: Dict[str, datetime] = {}
# ============================================================================
# УТИЛИТЫ
# ============================================================================
def format_time_duration(minutes: int) -> str:
"""Форматирует время из минут в часы и минуты"""
hours = minutes // 60
mins = minutes % 60
if hours == 0:
return f"{mins}м"
elif mins == 0:
return f"{hours}ч"
return f"{hours}ч {mins}м"
def get_day_statistics(day_blocks: List[Dict]) -> Dict[str, int]:
"""Получает статистику по дню"""
total_minutes = 0
confirmed_minutes = 0
possible_minutes = 0
for block in day_blocks:
for half in [block["first_half"], block["second_half"]]:
if half["status"] == "off":
total_minutes += 30
if half["confirmed"]:
confirmed_minutes += 30
else:
possible_minutes += 30
return {
"total": total_minutes,
"confirmed": confirmed_minutes,
"possible": possible_minutes
}
# ============================================================================
# ПАРСИНГ ДАННЫХ
# ============================================================================
def parse_html(html: str) -> List[Dict]:
"""Парсит HTML с графиком отключений (логика от 15.11.2024)"""
soup = BeautifulSoup(html, "html.parser")
cells = soup.select(".disconnection-detailed-table-cell.cell")
schedule = []
current_hour = 0
current_day = 0
for cell in cells:
if 'legend' in cell.get('class', []) or 'head' in cell.get('class', []):
continue
cell_classes = cell.get('class', [])
# ПРоверка статуса отключения на весь час
full_hour_off = 'has_disconnection' in cell_classes and 'full_hour' in cell_classes
no_disconnection_cell = 'no_disconnection' in cell_classes
hour_block = cell.select_one(".hour_block")
if not hour_block:
continue
# Проверка подтверждённости отключения для всего часа
cell_confirmed = None
if 'confirm_1' in cell_classes:
cell_confirmed = True
elif 'confirm_0' in cell_classes:
cell_confirmed = False
# Проверка половин часа
left = hour_block.select_one(".half.left")
right = hour_block.select_one(".half.right")
def parse_half(half, is_full_hour_off: bool) -> Dict:
"""Парсит половину часа"""
if not half:
return {"status": "on", "queue": None, "confirmed": None}
half_classes = half.get('class', [])
# Если вся ячейка full_hour - используем статус ячейки
if is_full_hour_off:
return {"status": "off", "queue": None, "confirmed": cell_confirmed}
# Определяем статус половины
if 'has_disconnection' in half_classes:
status = "off"
elif 'no_disconnection' in half_classes:
status = "on"
else:
status = "on" # По умолчанию считаем включенным
# Если выключено - ищем подробности
queue = None
confirmed = None
if status == "off":
disconnection_div = half.select_one(".disconnection")
if disconnection_div:
# Ищем номер черги в title
if disconnection_div.has_attr("title"):
title = disconnection_div["title"]
if "Номер черги" in title or "Номер черги:" in title:
try:
queue = title.split(":")[-1].strip()
except:
pass
# Определяем подтверждение
disc_classes = disconnection_div.get('class', [])
if 'disconnection_confirm_1' in disc_classes:
confirmed = True
elif 'disconnection_confirm_0' in disc_classes:
confirmed = False
return {"status": status, "queue": queue, "confirmed": confirmed}
first_half_data = parse_half(left, full_hour_off)
second_half_data = parse_half(right, full_hour_off)
schedule.append({
"hour": current_hour,
"day": current_day,
"first_half": first_half_data,
"second_half": second_half_data,
})
current_hour += 1
if current_hour >= 24:
current_hour = 0
current_day += 1
return schedule
def get_voe_html(city_id: int, street_id: int, house_id: int) -> str:
"""Получает HTML с сайта VOE"""
url = "https://www.voe.com.ua/disconnection/detailed?ajax_form=1&_wrapper_format=drupal_ajax"
headers = {
"Content-Type": "application/x-www-form-urlencoded; charset=UTF-8",
"X-Requested-With": "XMLHttpRequest",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
}
data = {
"search_type": 0,
"city_id": city_id,
"street_id": street_id,
"house_id": house_id,
"form_build_id": "form-Irv5aHw1R2FT_Ik2apyHOZ47hTH5xPNH_LQnBrmpSTc",
"form_id": "disconnection_detailed_search_form",
"_triggering_element_name": "search",
"_triggering_element_value": "Показати",
"_drupal_ajax": 1,
}
try:
response = requests.post(url, headers=headers, data=data, timeout=10)
response.raise_for_status()
resp_json = response.json()
insert_html = next(
(item["data"] for item in resp_json if item.get("command") == "insert"),
None
)
if not insert_html:
raise ValueError("HTML не найден в ответе")
return insert_html
except requests.exceptions.RequestException as e:
logger.error(f"Ошибка запроса VOE: {e}")
raise
# ============================================================================
# ФОРМАТИРОВАНИЕ СООБЩЕНИЙ
# ============================================================================
def get_main_keyboard():
"""Создает главную клавиатуру"""
builder = ReplyKeyboardBuilder()
builder.row(
KeyboardButton(text="📊 Графік"),
KeyboardButton(text="🔄 Оновити")
)
builder.row(
KeyboardButton(text="📅 Сьогодні"),
KeyboardButton(text="📅 Завтра")
)
builder.row(KeyboardButton(text="ℹ️ Про бота"))
return builder.as_markup(resize_keyboard=True)
def format_schedule_message(schedule: List[Dict], days_to_show: int = 2) -> str:
"""Форматирует полный график на несколько дней"""
lines = [
"⚡️ <b>Графік відключень світла</b>",
f"🕐 Оновлено: {datetime.now().strftime('%d.%m.%Y %H:%M:%S')}",
"─" * 30,
""
]
start_date = datetime.now()
for day in range(min(days_to_show, 2)):
day_blocks = [b for b in schedule if b["day"] == day]
if not day_blocks:
continue
date_str = (start_date + timedelta(days=day)).strftime('%d.%m.%Y')
day_name = "🌅 <b>Сьогодні</b>" if day == 0 else "🌄 <b>Завтра</b>"
lines.append(f"{day_name} ({date_str})")
# Статистика
stats = get_day_statistics(day_blocks)
if stats["total"] > 0:
lines.append(f"⏱ Всього: <code>{format_time_duration(stats['total'])}</code>")
if stats["confirmed"] > 0:
lines.append(f"🔴 Підтверджено: <code>{format_time_duration(stats['confirmed'])}</code>")
if stats["possible"] > 0:
lines.append(f"🟠 Можливо: <code>{format_time_duration(stats['possible'])}</code>")
else:
lines.append("🟢 <b>Відключень немає!</b>")
lines.append("")
# Детальный список отключений
disconnections = []
current_status = None
start_time = None
current_confirmed = None
current_queue = None
for block in day_blocks:
hour = block["hour"]
for half_idx, half in enumerate([block["first_half"], block["second_half"]]):
time_str = f"{hour:02d}:00" if half_idx == 0 else f"{hour:02d}:30"
if half["status"] == "off":
if current_status != "off":
start_time = time_str
current_confirmed = half["confirmed"]
current_queue = half["queue"]
current_status = "off"
else:
if current_status == "off":
icon = "🔴" if current_confirmed else "🟠"
queue_text = f" (Ч{current_queue})" if current_queue else ""
disconnections.append(f"{icon} <code>{start_time} - {time_str}</code>{queue_text}")
current_status = half["status"]
# Если день закончился на отключении
if current_status == "off":
icon = "🔴" if current_confirmed else "🟠"
queue_text = f" (Ч{current_queue})" if current_queue else ""
next_hour = (day_blocks[-1]["hour"] + 1) % 24
end_time = f"{next_hour:02d}:00"
disconnections.append(f"{icon} <code>{start_time} - {end_time}</code>{queue_text}")
if disconnections:
for idx, disc in enumerate(disconnections, 1):
lines.append(f"{idx}. {disc}")
lines.append("")
lines.append("<i>🔴 = підтверджено • 🟠 = можливо • 🟢 = світло</i>")
return "\n".join(lines)
def format_single_day_schedule(schedule: List[Dict], day: int) -> str:
"""Форматирует график на один день"""
day_blocks = [b for b in schedule if b["day"] == day]
if not day_blocks:
return "❌ Немає даних для цього дня"
start_date = datetime.now()
date_str = (start_date + timedelta(days=day)).strftime('%d.%m.%Y')
day_name = "🟠 <b>Сьогодні</b>" if day == 0 else "🔶 <b>Завтра</b>"
lines = [
f"{day_name} • {date_str}",
""
]
# Статистика
lines.append("<b>📊 Статистика</b>")
stats = get_day_statistics(day_blocks)
if stats["total"] == 0:
lines.append("└ 🟢 <b>Відключень немає!</b>")
else:
total_time = format_time_duration(stats["total"])
lines.append(f"├ ⏱ Всього: <code>{total_time}</code>")
if stats["confirmed"] > 0:
confirmed_time = format_time_duration(stats["confirmed"])
lines.append(f"├ 🔴 Підтверджено: <code>{confirmed_time}</code>")
if stats["possible"] > 0:
possible_time = format_time_duration(stats["possible"])
lines.append(f"└ 🟠 Можливо: <code>{possible_time}</code>")
else:
lines.append(f"└ 🟢 Решта часу світло")
lines.append("")
# Детальный список отключений
disconnections = []
current_status = None
start_time = None
current_confirmed = None
current_queue = None
for block in day_blocks:
hour = block["hour"]
for half_idx, half in enumerate([block["first_half"], block["second_half"]]):
time_str = f"{hour:02d}:00" if half_idx == 0 else f"{hour:02d}:30"
if half["status"] == "off":
if current_status != "off":
start_time = time_str
current_confirmed = half["confirmed"]
current_queue = half["queue"]
current_status = "off"
else:
if current_status == "off":
icon = "🔴" if current_confirmed else "🟠"
queue_text = f" (Ч.{current_queue})" if current_queue else ""
disconnections.append(f"{icon} <code>{start_time} - {time_str}</code>{queue_text}")
current_status = half["status"]
# Если день закончился на отключении
if current_status == "off":
icon = "🔴" if current_confirmed else "🟠"
queue_text = f" (Ч.{current_queue})" if current_queue else ""
next_hour = (day_blocks[-1]["hour"] + 1) % 24
end_time = f"{next_hour:02d}:00"
disconnections.append(f"{icon} <code>{start_time} - {end_time}</code>{queue_text}")
if disconnections:
lines.append("<b>⚡️ Розклад відключень</b>")
for idx, disc in enumerate(disconnections, 1):
lines.append(f"{idx}. {disc}")
lines.append("")
lines.append("<i>🔴 підтверджено • 🟠 можливо • 🟢 світло</i>")
return "\n".join(lines)
def schedules_differ(old_schedule: Optional[List[Dict]], new_schedule: Optional[List[Dict]]) -> bool:
"""Проверяет отличия между графиками"""
if old_schedule is None or new_schedule is None:
return True
if len(old_schedule) != len(new_schedule):
return True
for old, new in zip(old_schedule, new_schedule):
if old["day"] >= 2:
break
if (old["first_half"] != new["first_half"] or
old["second_half"] != new["second_half"]):
return True
return False
# ============================================================================
# УВЕДОМЛЕНИЯ
# ============================================================================
async def send_to_all_users(message_text: str, parse_mode: str = "HTML"):
"""Отправляет сообщение всем пользователям"""
if not ALLOWED_CHAT_IDS:
logger.warning("Нет допущенных ID чатов для отправки уведомлений")
return
for chat_id in ALLOWED_CHAT_IDS:
try:
await bot.send_message(chat_id, message_text, parse_mode=parse_mode)
logger.info(f"✅ Сообщение отправлено пользователю {chat_id}")
except Exception as e:
logger.error(f"❌ Ошибка отправки пользователю {chat_id}: {e}")
await asyncio.sleep(0.5)
async def check_schedule():
"""Проверяет график и отправляет уведомления"""
global last_schedule
try:
logger.info("🔍 Проверка графика...")
html = get_voe_html(VOE_CITY_ID, VOE_STREET_ID, VOE_HOUSE_ID)
new_schedule = parse_html(html)
if schedules_differ(last_schedule, new_schedule):
logger.info("✨ Обнаружены изменения!")
message = format_schedule_message(new_schedule, days_to_show=2)
if last_schedule is not None:
await send_to_all_users(f"🔄 <b>Графік оновлено!</b>\n\n{message}")
last_schedule = new_schedule
else:
logger.info("✓ Графік без змін")
except Exception as e:
logger.error(f"❌ Ошибка при проверке графика: {e}")
async def check_upcoming_disconnections():
"""Проверяет предстоящие события и отправляет предупреждения за 5 минут"""
global last_notification_time
if last_schedule is None:
return
now = datetime.now()
today_blocks = [b for b in last_schedule if b["day"] == 0]
# Создаем список всех переходов (off -> on или on -> off)
transitions = []
prev_status = None
for block in today_blocks:
hour = block["hour"]
for half_idx, half in enumerate([block["first_half"], block["second_half"]]):
minute = 0 if half_idx == 0 else 30
time_str = f"{hour:02d}:{minute:02d}"
current_status = half["status"]
# Если статус изменился - это переход
if prev_status is not None and prev_status != current_status:
transitions.append({
"hour": hour,
"minute": minute,
"time_str": time_str,
"from_status": prev_status,
"to_status": current_status,
"confirmed": half.get("confirmed"),
"queue": half.get("queue")
})
prev_status = current_status
# Проверяем переходы
for transition in transitions:
event_time = now.replace(
hour=transition["hour"],
minute=transition["minute"],
second=0,
microsecond=0
)
time_until = (event_time - now).total_seconds() / 60
notification_key = f"{transition['hour']}:{transition['minute']}_{transition['to_status']}"
# Если за 5 минут до события (±1 минута) и еще не отправляли
if 4 <= time_until <= 6:
# Проверяем, не отправляли ли уже уведомление сегодня
if notification_key in last_notification_time:
last_notif_time = last_notification_time[notification_key]
if last_notif_time.date() == now.date():
continue # Уже отправляли сегодня
# Переход на ОТКЛЮЧЕНИЕ (on -> off)
if transition["from_status"] == "on" and transition["to_status"] == "off":
icon = "🔴" if transition["confirmed"] else "🟠"
status = "підтверджено" if transition["confirmed"] else "можливе"
queue_info = f" (Черга {transition['queue']})" if transition['queue'] else ""
warning = (
f"⚠️ <b>УВАГА! ВІДКЛЮЧЕННЯ</b>\n\n"
f"Через ~5 хвилин\n"
f"Час: <code>{transition['time_str']}</code>\n"
f"Статус: {icon} {status}{queue_info}"
)
await send_to_all_users(warning)
last_notification_time[notification_key] = now
logger.info(f"📢 Відправлено попередження про ВІДКЛЮЧЕННЯ в {transition['time_str']}")
# Переход на ВКЛЮЧЕНИЕ (off -> on)
elif transition["from_status"] == "off" and transition["to_status"] == "on":
warning = (
f"✅ <b>УВАГА! ВКЛЮЧЕННЯ</b>\n\n"
f"Через ~5 хвилин буде світло\n"
f"Час: <code>{transition['time_str']}</code>"
)
await send_to_all_users(warning)
last_notification_time[notification_key] = now
logger.info(f"📢 Відправлено попередження про ВКЛЮЧЕННЯ в {transition['time_str']}")
async def monitoring_loop():
"""Основной цикл мониторинга"""
await check_schedule()
while True:
try:
await asyncio.sleep(CHECK_INTERVAL)
await check_schedule()
await check_upcoming_disconnections()
except Exception as e:
logger.error(f"Ошибка в цикле мониторинга: {e}")
await asyncio.sleep(5)
# ============================================================================
# ОБРАБОТЧИКИ КОМАНД
# ============================================================================
@dp.message(Command("start"))
async def cmd_start(message: Message):
"""Обработчик /start"""
if message.chat.id not in ALLOWED_CHAT_IDS:
await message.answer("❌ У вас немає доступу до цього бота.")
return
await message.answer(
"👋 <b>Ласкаво просимо!</b>\n\n"
"🤖 <b>Бот для моніторингу графіку відключень світла</b>\n\n"
"✨ <b>Можливості:</b>\n"
"• 📊 Перегляд графіку на сьогодні і завтра\n"
"• 🔔 Автоматичні сповіщення за 5 хвилин до подій\n"
"• 🔄 Моніторинг змін графіку\n\n"
"Використовуйте кнопки нижче 👇",
parse_mode="HTML",
reply_markup=get_main_keyboard()
)
@dp.message(lambda msg: msg.text == "ℹ️ Про бота")
async def cmd_info(message: Message):
"""Показывает информацию о боте"""
if message.chat.id not in ALLOWED_CHAT_IDS:
return
await message.answer(
"<b>ℹ️ Про бота</b>\n\n"
"🚀 <b>Версія:</b> 2.2 (Стабільна)\n\n"
"📝 <b>Реліз-ноути:</b>\n"
"├ 15.11.2024: Адаптація під оновлену логіку сайту VOE\n"
"├ Виправлено парсинг half.left та half.right\n"
"├ Покращено визначення підтвердження відключень\n"
"└ Оптимізовано обробку статусу для всієї години\n\n"
"⚡ <b>Функціональність:</b>\n"
"├ Моніторинг графіку 24/7\n"
"├ Сповіщення за 5 хвилин\n"
"├ Детальна статистика дня\n"
"└ Красива візуалізація\n\n"
"🔐 <b>Безпека:</b> Використовуються .env файли\n"
"💾 <b>Джерело:</b> voe.com.ua",
parse_mode="HTML",
reply_markup=get_main_keyboard()
)
@dp.message(Command("schedule"))
async def cmd_schedule(message: Message):
"""Показывает полный график"""
if message.chat.id not in ALLOWED_CHAT_IDS:
await message.answer("❌ У вас немає доступу.")
return
try:
await message.answer("⏳ Завантаження графіку...")
html = get_voe_html(VOE_CITY_ID, VOE_STREET_ID, VOE_HOUSE_ID)
schedule = parse_html(html)
text = format_schedule_message(schedule, days_to_show=2)
await message.answer(text, parse_mode="HTML", reply_markup=get_main_keyboard())
except Exception as e:
await message.answer(
f"❌ <b>Помилка:</b> {str(e)}",
parse_mode="HTML",
reply_markup=get_main_keyboard()
)
@dp.message(Command("today"))
async def cmd_today(message: Message):
"""Показывает график на сегодня"""
if message.chat.id not in ALLOWED_CHAT_IDS:
await message.answer("❌ У вас немає доступу.")
return
try:
await message.answer("⏳ Завантаження графіку сьогодні...")
html = get_voe_html(VOE_CITY_ID, VOE_STREET_ID, VOE_HOUSE_ID)
schedule = parse_html(html)
text = format_single_day_schedule(schedule, 0)
await message.answer(text, parse_mode="HTML", reply_markup=get_main_keyboard())
except Exception as e:
await message.answer(
f"❌ <b>Помилка:</b> {str(e)}",
parse_mode="HTML",
reply_markup=get_main_keyboard()
)
@dp.message(Command("tomorrow"))
async def cmd_tomorrow(message: Message):
"""Показывает график на завтра"""
if message.chat.id not in ALLOWED_CHAT_IDS:
await message.answer("❌ У вас немає доступу.")
return
try:
await message.answer("⏳ Завантаження графіку завтра...")
html = get_voe_html(VOE_CITY_ID, VOE_STREET_ID, VOE_HOUSE_ID)
schedule = parse_html(html)
text = format_single_day_schedule(schedule, 1)
await message.answer(text, parse_mode="HTML", reply_markup=get_main_keyboard())
# await message.answer("❌ Функція тимчасово недоступна. Чекаємо на оновлення сайту", parse_mode="HTML", reply_markup=get_main_keyboard())
except Exception as e:
await message.answer(
f"❌ <b>Помилка:</b> {str(e)}",
parse_mode="HTML",
reply_markup=get_main_keyboard()
)
@dp.message(Command("check"))
async def cmd_check(message: Message):
"""Принудительная проверка графика"""
if message.chat.id not in ALLOWED_CHAT_IDS:
await message.answer("❌ У вас немає доступу.")
return
try:
await message.answer("🔄 <b>Перевіряю графік...</b>", parse_mode="HTML")
html = get_voe_html(VOE_CITY_ID, VOE_STREET_ID, VOE_HOUSE_ID)
new_schedule = parse_html(html)
prefix = "✅ <b>Знайдено зміни!</b>\n\n" if schedules_differ(last_schedule, new_schedule) else "✓ <b>Графік без змін</b>\n\n"
result = prefix + format_schedule_message(new_schedule, days_to_show=2)
await message.answer(result, parse_mode="HTML", reply_markup=get_main_keyboard())
except Exception as e:
await message.answer(
f"❌ <b>Помилка:</b> {str(e)}",
parse_mode="HTML",
reply_markup=get_main_keyboard()
)
@dp.message()
async def handle_text(message: Message):
"""Обработчик текстовых сообщений и кнопок"""
if message.chat.id not in ALLOWED_CHAT_IDS:
return
text = message.text
# Кнопка "Графік"
if text == "📊 Графік":
await cmd_schedule(message)
# Кнопка "Сьогодні"
elif text == "📅 Сьогодні":
await cmd_today(message)
# Кнопка "Завтра"
elif text == "📅 Завтра":
await cmd_tomorrow(message)
# Кнопка "Оновити"
elif text == "🔄 Оновити":
await cmd_check(message)
# Кнопка "Про бота"
elif text == "ℹ️ Про бота":
await cmd_info(message)
# Неизвестная команда
else:
await message.answer(
"❓ <b>Команда не розпізнана</b>\n\n"
"Використовуйте кнопки на клавіатурі або команди:\n"
"/start • /today • /tomorrow • /schedule • /check",
parse_mode="HTML",
reply_markup=get_main_keyboard()
)
# ============================================================================
# ГЛАВНАЯ ФУНКЦИЯ
# ============================================================================
async def main():
"""Главная функция"""
logger.info("=" * 50)
logger.info("ЗАПУСК БОТА V2.2 (stable 2.2, 15.11.2025)")
logger.info("=" * 50)
if not TELEGRAM_TOKEN or TELEGRAM_TOKEN = os.getenv("TELEGRAM_TOKEN", "YOUR_TOKEN_HERE")
logger.error("❌ TELEGRAM_TOKEN не конфігурований! Напишіть токен в .env файл")
return
if not ALLOWED_CHAT_IDS:
logger.error("❌ ALLOWED_CHAT_IDS не конфігуровані! Напишіть ID в .env файл")
return
logger.info(f"📌 Allowed chat ids: {ALLOWED_CHAT_IDS}")
logger.info(f"⏱ Інтервал перевірки: {CHECK_INTERVAL} сек")
logger.info("=" * 50)
# Запускаем мониторинг
monitoring_task = asyncio.create_task(monitoring_loop())
try:
await dp.start_polling(bot)
except KeyboardInterrupt:
logger.info("⏹ Бот зупинений користувачем")
finally:
monitoring_task.cancel()
await bot.session.close()
logger.info("✓ Підключення закрито")
if __name__ == "__main__":
try:
asyncio.run(main())
except KeyboardInterrupt:
logger.info("⏹ Завершено")
+7
View File
@@ -0,0 +1,7 @@
[project]
name = "dtek-notif"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.13"
dependencies = []
Loaded 100 of 487 files, more files were not shown because too many files have changed in this diff. Show more