fix(server): k8s rollout readiness
ci / lint-prettier (push) Failing after 10s
ci / lint-ruff (push) Failing after 4s
ci / lint-yaml (push) Successful in 5s
ci / lint-dockerfiles (push) Successful in 5s
ci / validate (push) Successful in 5s
ci / lint-audit (push) Failing after 49s
ci / publish (push) Has been skipped
ci / lint-prettier (push) Failing after 10s
ci / lint-ruff (push) Failing after 4s
ci / lint-yaml (push) Successful in 5s
ci / lint-dockerfiles (push) Successful in 5s
ci / validate (push) Successful in 5s
ci / lint-audit (push) Failing after 49s
ci / publish (push) Has been skipped
- TRUSTED_HOSTS env: configurable trusted hostnames for proxy-domain access (default stays strict: localhost/loopback/private IP); k8s manifest sets tg.workstation.internal (L-8 follow-up) - Media allowlist +10: mkv/mk3d/heic/tgs/flv/3gp/ogv/asf/wmv/djvu (live disk has .tgs x44, .mkv x2) - Cache buster: app.js?v=4 -> ?v=5 so browsers pick up the new bundle - +6 tests (64 passing); REVIEW.md updated with live-cluster rollout notes
This commit is contained in:
1 parent
bc2e93353a
commit
2a75537fb9
5 files changed
+126
-2
No files matched your search
@@ -211,4 +211,15 @@
|
|||||||
- **SSRF**: юзер-контролируемого фетча URL нет (только MTProto).
|
- **SSRF**: юзер-контролируемого фетча URL нет (только MTProto).
|
||||||
- **Десериализация**: только JSON, без pickle/yaml.
|
- **Десериализация**: только JSON, без pickle/yaml.
|
||||||
- **Command injection**: нет subprocess/os.system в продакшн-путях.
|
- **Command injection**: нет subprocess/os.system в продакшн-путях.
|
||||||
- **Secrets в image**: `.dockerignore` исключает `data/` и `session/`.
|
- **Secrets в image**: `.dockerignore` исключает `data/` и `session/`.
|
||||||
|
---
|
||||||
|
|
||||||
|
### Follow-up (шестой проход — k8s-rollout audit живого кластера)
|
||||||
|
|
||||||
|
- **TRUSTED_HOSTS** (`webui_server.py`, L-8): `_is_trusted_host` отвергал ЛЮБОЙ hostname → через Traefik-домен `tg.workstation.internal` (`k8s/telegram-scraper.yaml:97`) браузерные POST/DELETE + SSE `/api/jobs/*/events` возвращали 403. Добавлен `_TRUSTED_HOSTS_ENV` (parse `TRUSTED_HOSTS` на импорте, нормализация `.strip().lower().rstrip(".")`); проверка после localhost-set. Дефолт строгий: без env результаты идентичны прежним для ВСЕХ входов (regression guard); с env настроенный hostname (любой case, опциональный trailing dot) проходит, остальные — нет. IPv6-with-port (`[::1]:8080`) — out of scope, не тронут.
|
||||||
|
- **Media allowlist** (`_MEDIA_FILE_EXTENSIONS`): +10 суффиксов с комментарием `# extended coverage (animated stickers, matroska, legacy containers)` — `.mkv .mk3d .heic .tgs .flv .3gp .ogv .asf .wmv .djvu`. Живой диск: `.tgs` x44 / `.mkv` x2 (возвращали 403). `.exe`/`.ts` оставлены 403. `guess_media_kind` для новых суффиксов возвращает `"file"` (fall-through как у `.zip`) — viewer рендерит "Open file" link. `serve_media` уже lowercases suffix, `.MP4`/`.MOV` ок.
|
||||||
|
- **Cache-buster**: `webui/index.html` `app.js?v=4` → `app.js?v=5` (иначе stale JS после deploy).
|
||||||
|
- **k8s-манифест**: `containers[0].env: TRUSTED_HOSTS="tg.workstation.internal"` (после `tty: true`); image/probes/securityContext/resources/PVCs/IngressRoute не тронуты; YAML проверен `yaml.safe_load`.
|
||||||
|
- **Live-cluster факты**: local-path PVCs, state v2 `accounts=[default, forust]`, смешанное владение 1000/root → обязательный `chown -R 1000:1000 /app/data /app/session` ДО первого старта нового пода.
|
||||||
|
|
||||||
|
Тесты: **58 → 64 passed** (+3 TRUSTED_HOSTS env, +3 extended media).
|
||||||
@@ -41,6 +41,9 @@ spec:
|
|||||||
image: gcr.forust.xyz/forust/telegram-scraper:latest
|
image: gcr.forust.xyz/forust/telegram-scraper:latest
|
||||||
stdin: true
|
stdin: true
|
||||||
tty: true
|
tty: true
|
||||||
|
env:
|
||||||
|
- name: TRUSTED_HOSTS
|
||||||
|
value: "tg.workstation.internal"
|
||||||
ports:
|
ports:
|
||||||
- containerPort: 8080
|
- containerPort: 8080
|
||||||
resources:
|
resources:
|
||||||
|
|||||||
@@ -977,6 +977,103 @@ class TestTrustedHost:
|
|||||||
assert h3._check_same_origin() is True
|
assert h3._check_same_origin() is True
|
||||||
|
|
||||||
|
|
||||||
|
class TestTrustedHostsEnv:
|
||||||
|
"""TRUSTED_HOSTS env: proxy-domain fix with strict default.
|
||||||
|
|
||||||
|
_TRUSTED_HOSTS_ENV is read at import, so tests patch the module
|
||||||
|
attribute directly (monkeypatch) instead of mutating os.environ.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def _ws(self):
|
||||||
|
import webui_server as ws_module
|
||||||
|
return ws_module
|
||||||
|
|
||||||
|
def test_default_env_domain_untrusted(self, monkeypatch):
|
||||||
|
ws = self._ws()
|
||||||
|
monkeypatch.setattr(ws, "_TRUSTED_HOSTS_ENV", frozenset())
|
||||||
|
assert ws._is_trusted_host("tg.workstation.internal") is False
|
||||||
|
# regression guard: loopback/private literals still trusted
|
||||||
|
for host in ("localhost", "127.0.0.1", "10.0.0.5", "192.168.1.50"):
|
||||||
|
assert ws._is_trusted_host(host) is True, f"{host!r} should be trusted"
|
||||||
|
|
||||||
|
def test_configured_domain_trusted(self, monkeypatch):
|
||||||
|
ws = self._ws()
|
||||||
|
monkeypatch.setattr(ws, "_TRUSTED_HOSTS_ENV", {"tg.workstation.internal"})
|
||||||
|
assert ws._is_trusted_host("tg.workstation.internal") is True
|
||||||
|
assert ws._is_trusted_host("TG.WORKSTATION.INTERNAL") is True
|
||||||
|
assert ws._is_trusted_host("tg.workstation.internal.") is True
|
||||||
|
assert ws._is_trusted_host("unknown.example") is False
|
||||||
|
|
||||||
|
def test_check_same_origin_with_trusted_domain(self, monkeypatch):
|
||||||
|
ws = self._ws()
|
||||||
|
monkeypatch.setattr(ws, "_TRUSTED_HOSTS_ENV", {"tg.workstation.internal"})
|
||||||
|
h = _make_ws_handler(
|
||||||
|
headers={
|
||||||
|
"Host": "tg.workstation.internal",
|
||||||
|
"Origin": "https://tg.workstation.internal",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
assert h._check_same_origin() is True
|
||||||
|
|
||||||
|
|
||||||
|
class TestExtendedMediaServing:
|
||||||
|
"""Extended media allowlist: .tgs/.mkv/... served, .exe/.ts still 403."""
|
||||||
|
|
||||||
|
def _make_full_media_handler(self, relative, data_dir):
|
||||||
|
import webui_server as ws_module
|
||||||
|
self._ws_orig_data = ws_module.DATA_DIR
|
||||||
|
ws_module.DATA_DIR = data_dir
|
||||||
|
handler = object.__new__(ws_module.TelegramScraperRequestHandler)
|
||||||
|
handler.headers = {}
|
||||||
|
handler.rfile = io.BytesIO()
|
||||||
|
handler.wfile = io.BytesIO()
|
||||||
|
handler.path = "/media/" + relative
|
||||||
|
handler.command = "GET"
|
||||||
|
handler.client_address = ("127.0.0.1", 4321)
|
||||||
|
handler.server = MagicMock()
|
||||||
|
handler.send_error_json = MagicMock()
|
||||||
|
handler.send_response = MagicMock()
|
||||||
|
handler.send_header = MagicMock()
|
||||||
|
handler.end_headers = MagicMock()
|
||||||
|
return handler
|
||||||
|
|
||||||
|
def _serve_and_status(self, relative):
|
||||||
|
import webui_server as ws_module
|
||||||
|
h = self._make_full_media_handler(relative, TEST_DATA)
|
||||||
|
try:
|
||||||
|
ws_module.DATA_DIR = TEST_DATA
|
||||||
|
h.serve_media(relative)
|
||||||
|
finally:
|
||||||
|
ws_module.DATA_DIR = self._ws_orig_data
|
||||||
|
if h.send_error_json.called:
|
||||||
|
return int(h.send_error_json.call_args[0][0])
|
||||||
|
return int(h.send_response.call_args[0][0])
|
||||||
|
|
||||||
|
def test_serves_tgs_and_mkv(self):
|
||||||
|
media_dir = TEST_DATA / "ext-media"
|
||||||
|
media_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
(media_dir / "sticker.tgs").write_bytes(b"\x1f\x8b\x08\x00")
|
||||||
|
(media_dir / "clip.mkv").write_bytes(b"\x1a\x45\xdf\xa3")
|
||||||
|
assert self._serve_and_status("ext-media/sticker.tgs") == 200
|
||||||
|
assert self._serve_and_status("ext-media/clip.mkv") == 200
|
||||||
|
|
||||||
|
def test_guess_media_kind_new_suffixes_fall_through_to_file(self):
|
||||||
|
import webui_server as ws_module
|
||||||
|
for name in (
|
||||||
|
"a.tgs", "a.mkv", "a.mk3d", "a.heic", "a.flv",
|
||||||
|
"a.3gp", "a.ogv", "a.asf", "a.wmv", "a.djvu",
|
||||||
|
):
|
||||||
|
assert ws_module.guess_media_kind(name, None) == "file", name
|
||||||
|
|
||||||
|
def test_exe_and_ts_still_forbidden(self):
|
||||||
|
media_dir = TEST_DATA / "ext-media"
|
||||||
|
media_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
(media_dir / "evil.exe").write_bytes(b"MZ")
|
||||||
|
(media_dir / "stream.ts").write_bytes(b"\x47" * 188)
|
||||||
|
assert self._serve_and_status("ext-media/evil.exe") == 403
|
||||||
|
assert self._serve_and_status("ext-media/stream.ts") == 403
|
||||||
|
|
||||||
|
|
||||||
class TestMediaServingLockdown:
|
class TestMediaServingLockdown:
|
||||||
"""M-1: /media/ must never serve state.json / *.db / *.session."""
|
"""M-1: /media/ must never serve state.json / *.db / *.session."""
|
||||||
|
|
||||||
|
|||||||
+1
-1
@@ -372,6 +372,6 @@
|
|||||||
</div>
|
</div>
|
||||||
</template>
|
</template>
|
||||||
|
|
||||||
<script src="/static/app.js?v=4"></script>
|
<script src="/static/app.js?v=5"></script>
|
||||||
</body>
|
</body>
|
||||||
</html>
|
</html>
|
||||||
@@ -118,15 +118,25 @@ def parse_bool(value: Any, default: bool = False) -> bool:
|
|||||||
# ── L-8: trusted-host check ──────────────────────────────────────────────
|
# ── L-8: trusted-host check ──────────────────────────────────────────────
|
||||||
import ipaddress # noqa: E402
|
import ipaddress # noqa: E402
|
||||||
|
|
||||||
|
# Comma-separated extra trusted hostnames for browser state-mutating
|
||||||
|
# requests when the panel is served through a proxy domain.
|
||||||
|
# Example: TRUSTED_HOSTS="tg.workstation.internal,example.com"
|
||||||
|
_TRUSTED_HOSTS_ENV = frozenset(
|
||||||
|
h.strip().lower().rstrip(".") for h in os.environ.get("TRUSTED_HOSTS", "").split(",") if h.strip()
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def _is_trusted_host(host: str) -> bool:
|
def _is_trusted_host(host: str) -> bool:
|
||||||
"""Return True if *host* (the ``Host`` header value) is a loopback /
|
"""Return True if *host* (the ``Host`` header value) is a loopback /
|
||||||
private address that this local-only deployment should trust."""
|
private address that this local-only deployment should trust."""
|
||||||
hostname = host.split("@")[-1].split(":")[0] # strip auth / port
|
hostname = host.split("@")[-1].split(":")[0] # strip auth / port
|
||||||
|
hostname = hostname.strip().lower().rstrip(".")
|
||||||
if not hostname:
|
if not hostname:
|
||||||
return False
|
return False
|
||||||
if hostname in {"localhost", "127.0.0.1", "::1"}:
|
if hostname in {"localhost", "127.0.0.1", "::1"}:
|
||||||
return True
|
return True
|
||||||
|
if hostname in _TRUSTED_HOSTS_ENV:
|
||||||
|
return True
|
||||||
try:
|
try:
|
||||||
addr = ipaddress.ip_address(hostname)
|
addr = ipaddress.ip_address(hostname)
|
||||||
return addr.is_loopback or addr.is_private or addr.is_link_local
|
return addr.is_loopback or addr.is_private or addr.is_link_local
|
||||||
@@ -157,6 +167,9 @@ _MEDIA_FILE_EXTENSIONS = frozenset({
|
|||||||
# archives / other Telegram document types
|
# archives / other Telegram document types
|
||||||
".zip", ".rar", ".7z", ".apk", ".epub", ".tar", ".gz", ".bz2", ".xz",
|
".zip", ".rar", ".7z", ".apk", ".epub", ".tar", ".gz", ".bz2", ".xz",
|
||||||
".odt", ".ods", ".odp",
|
".odt", ".ods", ".odp",
|
||||||
|
# extended coverage (animated stickers, matroska, legacy containers)
|
||||||
|
".mkv", ".mk3d", ".heic", ".tgs", ".flv", ".3gp", ".ogv", ".asf",
|
||||||
|
".wmv", ".djvu",
|
||||||
})
|
})
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
Reference in new issue
Block a user