Sandbox orchestrator — optional Railway deploy & dashboard-summary troubleshooting

Clarifies that Railway deployment of sandbox-orchestrator is optional; explains when to use it; documents why infra dashboard shows dashboard-summary 404s and how to fix them on bare metal.

Sandbox orchestrator: optional Railway + dashboard errors

1. Two different things (do not conflate them)

Topic	Required?	Where it runs
Real Firecracker sandboxes	Yes, for self-hosted VMs	Bare metal (e.g. Hetzner) — KVM, TAP, iptables
Deploying `sandbox-orchestrator` on Railway	No	Optional; normal Railway containers cannot run Firecracker workloads

SANDBOX_API_KEY on backend-ai-api is only the client credential used when calling your orchestrator. It does not deploy code to the orchestrator host and does not add HTTP routes there.

2. Repo changes for Railway (`deployment/sandbox-orchestrator/*`, deploy CLI)

These exist so that if you ever want the same monorepo deploy pattern as backend-api / backend-ai-api (Dockerfile + railway.toml + yarn deploy:sandbox-orchestrator), you can.

You can ignore all of that when:

Production orchestrator stays only on your dedicated server (e.g. https://sandbox.gograb.space).
You never plan to run a second HTTP-only copy on Railway.

Consider Railway orchestrator only if you want one or more of:

(X) Staging / parity — a managed URL that runs current Nest code for smoke tests (/health, JSON meta), without touching production Hetzner.
(Y) Monitoring split — an extra health endpoint in Railway’s project canvas (know it still won’t prove Firecracker works).
(Z) Future HTTP-only experiments — anything that only needs REST + auth, not VM lifecycle on that host.

If none of X / Y / Z apply, no Railway service is required and no configuration change is required solely for “skipping Railway.”

Note: yarn deploy:all intentionally does not include sandbox-orchestrator; it stays opt-in.

3. Infra dashboard: “dashboard summary did not succeed” + three HTTP hints

backend-ai-api loads self-hosted rows from SANDBOX_SERVER_URL (and optional SELF_HOSTED_INFRA_JSON). For each orchestrator base URL it tries, in order:

GET {base}/sandboxes/meta/dashboard-summary (preferred — two path segments, never mistaken for a sandbox id)
GET {base}/sandboxes/dashboard-summary (legacy / fallback)
GET {base}/internal/dashboard-summary (fallback when nginx only proxies certain prefixes)

All requests send header X-API-Key: <SANDBOX_API_KEY> (or per-entry override from SELF_HOSTED_INFRA_JSON).

3.1 If you see this exact pattern

/sandboxes/meta/dashboard-summary → 404 Cannot GET /sandboxes/meta/dashboard-summary
/sandboxes/dashboard-summary → 404 Sandbox not found
/internal/dashboard-summary → 404 Cannot GET /internal/dashboard-summary

That combination almost always means one of the following.

A) Orchestrator binary is older than the repo (most common)

Current code registers:

SandboxMetaController → /sandboxes/meta/dashboard-summary
DashboardSummaryController → /internal/dashboard-summary

An old process often:

Has no /sandboxes/meta/... or /internal/dashboard-summary → Nest responds with Cannot GET …
Still has @Get(':id') on sandboxes → path /sandboxes/dashboard-summary is treated as id = "dashboard-summary" → Sandbox not found

Fix: On the machine that actually runs the orchestrator (Hetzner), rebuild and restart from the current monorepo (same revision that contains sandbox-meta.controller.ts and dashboard-summary.controller.ts), then verify with curl (see §3.3).

B) `SANDBOX_SERVER_URL` on Railway `backend-ai-api` points at the wrong host

Examples of wrong targets:

Another API (ai-api, vitto-api, etc.) — wrong app, no orchestrator routes
URL with an accidental path suffix so requests hit a non-existent sub-path
HTTP instead of HTTPS when TLS is required, or hostname that does not resolve to the orchestrator

Fix: In Railway (or .env for local ai-api), set SANDBOX_SERVER_URL to the public origin of the orchestrator only, e.g. https://sandbox.gograb.space (no trailing slash required; code strips it).

C) Nginx on the orchestrator box is out of date

If /sandboxes/* works for normal API but /internal/dashboard-summary never reaches Node, confirm deployment/sandbox-orchestrator/nginx.conf includes a location /internal/ block proxying to the Node port, then nginx -t and reload.

If nginx’s default location / returns JSON like {"error":"Not found","service":"sandbox-orchestrator"}, that is nginx, not Nest — update the config.

3.2 API key is not the cause of 404

ApiKeyGuard on the orchestrator rejects bad/missing keys with 401 Unauthorized, not 404.

If you see 404 with the messages above, the request reached something that treated the path as unknown or as a sandbox id — not “wrong key.”

3.3 Verify from your laptop

export BASE=https://sandbox.gograb.space   # your real orchestrator origin
export SANDBOX_API_KEY='...'               # same value as backend-ai-api

curl -sS -o /dev/null -w "%{http_code}\n" "$BASE/health"

curl -sS -H "X-API-Key: $SANDBOX_API_KEY" "$BASE/sandboxes/meta/dashboard-summary" | head -c 400

Expect 200 and JSON with host and orchestrator objects. If you still get the 404 triad here, the problem is on the orchestrator server (build/version or nginx), not Railway SANDBOX_API_KEY alone.

4. Where to look in code (for maintainers)

Piece	Location
Orchestrator meta route	`APIs/sandbox-orchestrator/src/app/sandbox/sandbox-meta.controller.ts`
Internal alias route	`APIs/sandbox-orchestrator/src/app/sandbox/dashboard-summary.controller.ts`
Module registration	`APIs/sandbox-orchestrator/src/app/sandbox/sandbox.module.ts`
ai-api fetch + hints	`APIs/backend-ai-api/src/app/infrastructure/services/hetzner-infra.service.ts`
Nginx (bare metal)	`deployment/sandbox-orchestrator/nginx.conf`
Optional Railway runbook	`deployment/sandbox-orchestrator/RAILWAY.md`

5. Operator UX (gograb-dev)

Primary runbook for Hetzner + Firecracker (your real path): hetzner-firecracker-operations.md.

The Dedicated infrastructure page (/infrastructure/hetzner) includes an Operations cheatsheet with copy buttons for SSH rebuild, nginx, logs, curl verification, and backend-ai-api release when infra code changes. It does not execute remote commands.

6. Related

Railway smart deploy CLI — yarn deploy:sandbox-orchestrator when you choose Railway.