Hetzner + Firecracker — operations

What runs where

Production sandboxes require KVM / TAP / iptables on that host. Do not expect a generic container host to replace it.

Dual-server fleet (AX102-U vs AX42-U)

We run two dedicated hosts (see SANDBOX_SERVERS_JSON + SELF_HOSTED_INFRA_JSON in .env.local / Railway). DNS is not interchangeable — each hostname points at one machine.

Git HTTP vs SANDBOX_SERVERS_JSON (authoritative)

• All git API / clone traffic (/git/*, RemoteGitProviderService, studio universe git) uses PLATFORM_GIT_SERVER_URL + PLATFORM_GIT_API_KEY only. In production that target is AX102-U (ax102-01.gograb.space). User role and VM placement do not change the git host.
• The git-operations string on a fleet entry in SANDBOX_SERVERS_JSON is not consumed by SandboxPlacementRouter for routing (VM placement uses customer-sandbox / executive-sandbox in WORKLOAD_ROLE_MAP only). Treat git-operations as fleet metadata (documentation of which host runs the git HTTP stack)—not a second routing mechanism for git or sandboxes.
• Early warning: If git push or dashboard “git identity” points at the wrong host, check PLATFORM_GIT_*, proxy.conf /sandbox-api upstream, and duplicate env keys in .env (last wins).

Code paths: SandboxPlacementRouter (libs/libs-api/sandbox-orchestration) filters by role + priority + capacity. backend-ai-api ComputerUseLoop passes workloadType from goal.persistentVm. Vitto VmSessionManager passes workloadType: 'executive' for executive full-stack workspace cold boots so placement matches ai-api behavior.

Operators: deploy orchestrator to both hosts when you change sandbox-orchestrator; use yarn sandbox:deploy:all or per-host SANDBOX_ORCHESTRATOR_SSH. Snippets: apps/gograb-dev/.../hetzner-deploy-playbook.ts.

How gograb-dev “talks” to your infrastructure

1. backend-ai-api reads SANDBOX_SERVER_URL (public origin of the primary orchestrator, e.g. https://ax102-01.gograb.space) and optionally SANDBOX_SERVERS_JSON for multi-host aggregation.
2. With SANDBOX_API_KEY set on backend-ai-api (same value as on the orchestrator), the API can call GET /sandboxes/meta/dashboard-summary so this UI shows load, VMs, active sandboxes, Node + git identity (from the orchestrator), and can destroy a sandbox via the orchestrator.

No Hetzner Cloud token is required for that path. Optional HETZNER_API_TOKEN only adds Hetzner Cloud dedicated servers to the same table and enables cloud power actions.

Orchestrator build identity (git / Node)

• GET /health and dashboard summary include a build object: Node version, git SHA (from env or git rev-parse HEAD in process.cwd()), and processStartedAt (when the Node process started — useful to confirm restart, not image build time).
• Env overrides (first match wins): ORCHESTRATOR_GIT_SHA, GIT_COMMIT, RAILWAY_GIT_COMMIT_SHA, GITHUB_SHA, COMMIT_SHA. Use a systemd Environment= or drop-in if the deploy tree has no .git (e.g. rsync without VCS).
• Prometheus: orchestrator /metrics is linked from the expanded server row in gograb-dev. It is typically unauthenticated in nginx.conf — restrict by IP or auth if the host is on a public interface.

UI: stale snapshot banner

If backend-ai-api’s list fetchedAt is older than ~2.5 minutes, gograb-dev shows an amber banner. That measures aggregation time on ai-api, not orchestrator clock skew (clocks >1 min ahead suppress the banner). Tune the interval in code if your refresh cadence changes.

Deploy script (recommended)

From the monorepo:

# On the Hetzner box (already SSH’d in, repo at ~/gspace1 or set SANDBOX_REMOTE_REPO)
yarn sandbox:deploy:host

# From your laptop (SSH key access to root@host; optional SANDBOX_REMOTE_REPO=/root/gspace1)
SANDBOX_ORCHESTRATOR_SSH=root@YOUR_HOST yarn sandbox:deploy:host -- remote
# or
yarn sandbox:deploy:host -- remote root@YOUR_HOST

• --dry-run — print steps only (on your laptop, set SANDBOX_REMOTE_REPO to your local clone path if ~/gspace1 does not exist; on the server, default $HOME/gspace1 is usually enough)
• --no-pull — build without git pull
• --skip-build — only systemctl restart (use with care)
• SANDBOX_SYSTEMD_UNIT — defaults to sandbox-orchestrator — confirm with systemctl status sandbox-orchestrator on the box
• SANDBOX_RESTART_CMD — if you do not use systemd (e.g. pm2 restart …)

Full header comment: deployment/sandbox-orchestrator/deploy-orchestrator-host.sh

Does this fix dashboard-summary 404s?
Yes, when the cause is an old orchestrator binary (missing /sandboxes/meta/dashboard-summary) and you deploy current code to the same origin as SANDBOX_SERVER_URL, then restart.
No, if SANDBOX_SERVER_URL is wrong, nginx does not proxy /sandboxes/ to Node, or the failure is something else.

Day-to-day operator flow

1. Monitor — Open Dedicated infrastructure in gograb-dev; use Refresh or wait for the auto refresh.
2. Rebuild orchestrator after code changes — use yarn sandbox:deploy:host (on host or remote via SSH), or the Operations cheatsheet on that page for manual copy/paste.
3. Verify — From your laptop, curl /health and /sandboxes/meta/dashboard-summary with X-API-Key (snippet on the same page).
4. Nginx — After editing deployment/sandbox-orchestrator/nginx.conf on the server, nginx -t and reload.

If dashboard metrics show 404 hints

See §3 in sandbox-orchestrator optional Railway & dashboard troubleshooting — the 404 pattern section applies to whatever host SANDBOX_SERVER_URL points at (almost always: stale orchestrator binary on Hetzner or wrong URL). That doc’s Railway sections are not part of your Firecracker path.

Related files