44 lines
2.3 KiB
Markdown
44 lines
2.3 KiB
Markdown
# VictoriaMetrics
|
|
|
|
The `victoria-operator` Helm release converts Prometheus Operator
|
|
`ServiceMonitor` resources into owned `VMServiceScrape` resources. The
|
|
`VMAgent` selects converted scrapes labeled `release: prometheus-stack` in all
|
|
namespaces and writes them to the existing single-node VictoriaMetrics
|
|
instance. Changes to selected `ServiceMonitor` resources are reconciled
|
|
automatically; there is no copied Prometheus scrape-config blob to regenerate.
|
|
|
|
The agent drops targets for the Prometheus server service to avoid duplicating
|
|
its self-scrape. `scraper: victoria` identifies the samples ingested by this
|
|
VMAgent.
|
|
|
|
The VictoriaMetrics Operator chart and its CRDs are installed before the
|
|
Kubernetes manifests by the normal deploy workflow. On a cluster where the
|
|
operator CRDs are not installed yet, CI skips the server-side dry-run of the
|
|
`VMAgent` resource; the deploy installs the chart before applying that resource.
|
|
|
|
## Application metrics
|
|
|
|
The application ServiceMonitors use a 30s interval and a 10s timeout:
|
|
|
|
- Headscale: the external Service points to the Compose host on port 19090.
|
|
A VMServiceScrape uses EndpointSlice discovery for this manually managed target.
|
|
The Compose configuration must bind metrics to `0.0.0.0:9090`.
|
|
- NetBird: the combined server exports `/metrics` on port 9090. The existing
|
|
`server.metricsPort` setting enables the listener.
|
|
- Gitea: `GITEA__metrics__ENABLED` enables `/metrics` on the HTTP port. The public
|
|
ingress excludes this path. The monitor uses the internal Service directly.
|
|
- Immich: `IMMICH_TELEMETRY_INCLUDE=all` enables API and worker metrics on ports
|
|
8081 and 8082. The monitor scrapes both ports on each server replica.
|
|
|
|
Deploy through the existing CI and deploy workflow. Gitea and Immich reload their
|
|
ConfigMap changes through Reloader. Check the VMAgent targets after deployment
|
|
and query `up{scraper="victoria",namespace=~"netbird|gitea|immich|headscale"}` in
|
|
VictoriaMetrics. All targets should report 1.
|
|
|
|
For rollback, revert the application metrics changes, run CI, and deploy the
|
|
revert. Remove the three application ServiceMonitors and the Headscale VMServiceScrape explicitly: the deployment
|
|
workflow applies manifests and does not prune removed resources.
|
|
|
|
For Headscale rollback, remove its VMServiceScrape and Service label, restore the
|
|
previous Compose metrics bind address, and restart only the Headscale service.
|