S245 Daily Summary — Hetzner self-hosted sandbox ops & gograb-dev infrastructure

Documentation for Sessions

S245 — Daily summary (2026-03-20)

1. Executive summary

This session solidified how GoGrab runs and observes the sandbox orchestrator on Hetzner bare metal (Firecracker): fixed dashboard-summary 404 class failures, clarified that Railway is optional for the orchestrator (not required for real VMs), added operator UX in gograb-dev (/infrastructure/hetzner), a deploy script (local nx build + scp + systemctl), build identity on /health and dashboard JSON, and documentation for long-term operators. A consolidated infrastructure decision was logged in MCP (dev_1774023685822_69gab5i, confidence 0.91).

2. Problem statement & root cause

2.1 Symptom (infra dashboard)

With SANDBOX_API_KEY set on backend-ai-api, the Hetzner/self-hosted row still showed failures like:

GET …/sandboxes/meta/dashboard-summary → 404 Cannot GET …
GET …/sandboxes/dashboard-summary → 404 Sandbox not found
GET …/internal/dashboard-summary → 404 Cannot GET …

2.2 Root cause (high confidence)

Cause	Evidence
Stale orchestrator binary	`Sandbox not found` on `/sandboxes/dashboard-summary` means `dashboard-summary` was handled as `@Get(':id')` (sandbox id), not as a dedicated meta route. `Cannot GET` on `/sandboxes/meta/...` means routes for the meta controller were not registered in the running process.
Wrong `SANDBOX_SERVER_URL`	Same 404 pattern if ai-api talks to a host that is not the orchestrator (wrong app, path, or TLS).
Not “missing API key”	Bad/missing key on the orchestrator yields 401 from `ApiKeyGuard`, not this 404 triad.

2.3 Fix direction

Deploy a current sandbox-orchestrator build that registers SandboxMetaController at sandboxes/meta/dashboard-summary (and related fallbacks), ensure nginx proxies /sandboxes/ (and optionally /internal/) to Node, and align SANDBOX_SERVER_URL with that host.

3. Architecture decisions (recorded)

3.1 Question

Where should sandbox-orchestrator + Firecracker run, how should gograb-dev observe it, and how should operators deploy?

3.2 Alternatives considered

Run orchestrator on Railway like other APIs (full parity).
Hetzner bare metal as source of truth; Railway orchestrator optional (HTTP-only / staging only).
Deploy only via git pull + nx build on the server (full monorepo on host).
Deploy via local nx build + scp main.js + systemd restart (minimal server footprint).

3.3 Chosen approach

Hetzner bare metal is the system of record for Firecracker (KVM/TAP/iptables). Railway does not replace that for production VMs; repo may still contain optional Railway scaffolding for a future HTTP-only service.
Observability: backend-ai-api uses SANDBOX_SERVER_URL + SANDBOX_API_KEY; it probes /health and prefers GET /sandboxes/meta/dashboard-summary (two path segments so it is never bound as :id).
Deploy (current repo script): local nx build sandbox-orchestrator, scp artifact to SANDBOX_DEPLOY_PATH (default /var/lib/sandbox-orchestrator/main.js), ssh systemctl restart SANDBOX_SYSTEMD_UNIT (default sandbox-orchestrator). No monorepo clone required on the server for this path.

3.4 When this decision would be wrong

You require git-based deploys on the host (then the scp script is the wrong primary runbook).
Webpack output layout changes (e.g. not a single main.js) without updating deploy-orchestrator-host.sh.
systemd unit or remote path differs from defaults and env vars are not set.

3.5 Early warning signals

Return of 404 triad on dashboard hints → likely stale binary or URL/nginx regression.
build.gitSha / source: unknown in /health → missing .git on host or no ORCHESTRATOR_GIT_SHA env (see §6).

4. Implementation inventory

4.1 Sandbox orchestrator (`APIs/sandbox-orchestrator/`)

Area	Files / notes
Meta route (dashboard summary)	`src/app/sandbox/sandbox-meta.controller.ts` — `GET sandboxes/meta/dashboard-summary`
Internal alias	`src/app/sandbox/dashboard-summary.controller.ts` — `GET internal/dashboard-summary`
Module registration	`src/app/sandbox/sandbox.module.ts`
Build identity	`src/app/build/orchestrator-build-info.service.ts` — env chain + cached `git rev-parse`; `processStartedAt` = Node boot time
Health	`src/app/health/health.controller.ts` — includes `build` object
Dashboard JSON	`src/app/sandbox/dashboard-summary.service.ts` — includes `build`

4.2 Backend AI API (`APIs/backend-ai-api/`)

Area	Files / notes
Aggregation + probes	`src/app/infrastructure/services/hetzner-infra.service.ts` — health RTT, dashboard fetch order, `normalizeOrchestratorBuild`
Tests	`src/app/infrastructure/__tests__/hetzner-infra.service.spec.ts`

4.3 gograb-dev (`apps/gograb-dev/`)

Area	Files / notes
Page	`src/app/pages/infrastructure/hetzner-servers/hetzner-servers.component.{ts,html,scss}`
Copy snippets	`hetzner-deploy-playbook.ts` — ops cheatsheet + incident snippets
Models	`models/hetzner-dashboard.model.ts` — `orchestratorBuild`, etc.

UI capabilities (quick wins):

Build line — Node, git short SHA (tooltip full), source, process start time (when summary includes valid build).
Prometheus link — {orchestratorOrigin}/metrics with warning about unauthenticated scrape exposure.
Stale snapshot banner — if fetchedAt from ai-api is older than ~150s (heuristic; suppresses if timestamp >1 min in the future).
Incident response section — copy-paste dmesg/journal, disk, nginx/3010, curl meta snippets.

4.4 Deployment & docs

Path	Purpose
`deployment/sandbox-orchestrator/deploy-orchestrator-host.sh`	scp + systemd deploy (see header for env/flags)
`deployment/sandbox-orchestrator/nginx.conf`	Example reverse proxy (health, `/sandboxes/`, `/internal/`, `/metrics`, VNC)
`deployment/sandbox-orchestrator/RAILWAY.md`	Optional Railway notes (not required for Firecracker)
`docs/english/Deployment/hetzner-firecracker-operations.md`	Primary operator runbook (Hetzner-first)
`docs/english/Deployment/sandbox-orchestrator-optional-railway-and-dashboard.md`	Optional Railway + 404 triad troubleshooting
`deployment/cli/lib/deployment-config.js`	Optional `sandbox-orchestrator` Railway target + comment
`package.json`	`yarn sandbox:deploy:host`

5. Environment variables (contract)

5.1 backend-ai-api (Railway or local)

Variable	Role
`SANDBOX_SERVER_URL`	Public origin of orchestrator (e.g. `https://sandbox.gograb.space`)
`SANDBOX_API_KEY`	Same secret the orchestrator expects (`X-API-Key`) for dashboard summary + destroy
`SANDBOX_SERVER_LABEL` / `SANDBOX_SERVER_HARDWARE`	Optional row display
`HETZNER_API_TOKEN`	Optional — merges Hetzner Cloud servers; power actions only for those rows
`SELF_HOSTED_INFRA_JSON`	Optional multi-host extension (see service implementation)

5.2 sandbox-orchestrator (host)

Variable	Role
`SANDBOX_API_KEY`	Required for guarded routes
`ORCHESTRATOR_GIT_SHA` / `GIT_COMMIT` / `RAILWAY_GIT_COMMIT_SHA` / `GITHUB_SHA` / `COMMIT_SHA`	Optional override for `build.gitSha` when `.git` is absent
`KERNEL_PATH`, `BASE_ROOTFS_PATH`	VM prerequisites (defaults in code)

6. Operator runbook (condensed)

6.1 Deploy orchestrator (from dev laptop)

cd /path/to/gspace1
yarn sandbox:deploy:host --dry-run    # optional
yarn sandbox:deploy:host
# Overrides:
#   SANDBOX_ORCHESTRATOR_SSH=root@host SANDBOX_DEPLOY_PATH=/path/main.js SANDBOX_SYSTEMD_UNIT=unit yarn sandbox:deploy:host

Note: SANDBOX_REMOTE_REPO is not used by this script; repo root is derived from the script path.

6.2 Verify

curl -sS "https://sandbox.gograb.space/health" | jq .
curl -sS -H "X-API-Key: $SANDBOX_API_KEY" \
  "https://sandbox.gograb.space/sandboxes/meta/dashboard-summary" | head -c 500

6.3 gograb-dev

URL: /infrastructure/hetzner (e.g. port 6300 locally).
Refresh + auto poll 60s; stale banner uses ~150s threshold on fetchedAt.

6.4 When to redeploy backend-ai-api

After changes to hetzner-infra.service.ts or related controllers/DTOs — use your normal yarn deploy:ai-api (or equivalent).

7. MCP / institutional memory

Field	Value
Decision ID	`dev_1774023685822_69gab5i`
Type	infrastructure
Confidence	0.91
Outcome	Reported success (session setup + alignment with live Hetzner dashboard)

Session enforcement: MCP session 245; after log_decision, sub-session advanced (e.g. 245.2). Git branch may remain session-242-245; preflight can warn vs canonical session-245 — follow team convention for combined branches.

SAGP commits: [S245.<sub>] type: description — stage specific files only.

8. Known gaps & risks (honest)

Gap / risk	Mitigation
scp deploy assumes a single `main.js` and matching systemd `ExecStart`	If you add native deps or multi-file bundles, revisit script and unit.
`/metrics` often unauthenticated	Restrict in nginx (IP allowlist or auth) on public interfaces.
Stale banner is ai-api fetch age, not orchestrator-internal TTL	Do not treat it as SLA; tune constant if refresh cadence changes.
`processStartedAt` ≠ image build time	Use git SHA + deploy logs to prove “what commit is live.”
Optional Railway docs vs scp-first reality	Operators should treat `hetzner-firecracker-operations.md` + `deploy-orchestrator-host.sh` as primary.
`find_similar` (infrastructure) did not return a close prior decision for this thread	Keyword index may miss; this doc compensates.

9. Related documentation (index)

Document	Use when
hetzner-firecracker-operations.md	Day-to-day Hetzner + Firecracker ops
sandbox-orchestrator-optional-railway-and-dashboard.md	Optional Railway + 404 pattern table
RAILWAY.md	Only if deploying orchestrator to Railway
railway-deployment-scripts.md	Smart deploy CLI for APIs

10. Decision log (same day)

Time (UTC)	Type	Summary
~16:21	infrastructure	Hetzner SoR for Firecracker; ai-api observation contract; meta dashboard path; scp+systemd deploy; risks documented (full text in MCP + §3 above).

Total logged decisions this doc covers: 1 (see MCP for canonical fields).

11. Follow-ups (optional)

Align docs/english/Deployment/hetzner-firecracker-operations.md if any scp vs git-on-host wording still references an older script variant.
Confirm systemd unit name on prod matches sandbox-orchestrator or document SANDBOX_SYSTEMD_UNIT in internal wiki.
If session branch must be session-245, reconcile with session-242-245 per SAGP / team rule.