S260: Hetzner Multi-Server Infrastructure Design

Session: 260 | Date: 2026-03-23 | Status: In Progress (Phase 1 complete, Phase 2 building) Author: Claude Code (Opus 4.6) | Reviewer: fguijarro

Executive Summary

GoGrab operates two Hetzner dedicated servers as a unified fleet with role-based workload routing. Both servers run concurrently — AX102-U handles customer VMs and git operations, AX42-U handles executive VMs and serves as overflow. A SandboxPlacementRouter routes workloads to the right server based on type, health, and capacity.

Finalized Architecture (Post-Discussion)

Server IDs: {hardware-class}-{sequence} — roles are metadata, not identity.

┌─────────────────────────────────────────────────────────────────┐
│  ax102-01  (AX102-U, 16C/32T, 128GB, 1.71TB NVMe)            │
│  DNS: ax102-01.gograb.space                                    │
│  Roles: customer-sandbox (primary), git-operations             │
│  Future: build-server, local-inference                         │
│  Capacity: 60 VMs                                              │
├─────────────────────────────────────────────────────────────────┤
│  ax42-01   (AX42-U, 8C/16T, 128GB, 1TB NVMe)                 │
│  DNS: sandbox.gograb.space (legacy, kept as-is)                │
│  Roles: executive-sandbox, customer-sandbox (overflow)          │
│  Future: monitoring, backups, cron                              │
│  Capacity: 30 VMs                                              │
└─────────────────────────────────────────────────────────────────┘

1. Hardware Analysis

Current AX42-U Usage (live metrics)

• CPU load: 0.1 / 0.03 / 0.01 (1m/5m/15m) — essentially idle
• Memory: 1.97 GB used / 128 GB total — 1.5% utilization
• Disk: 2.4% used
• Active VMs: 0 / 30 capacity
• Firecracker: Ready (kernel + rootfs present)
• Git repos: 2 repos (chrome-space-1/conejo with 1868 files, chrome-space-1/tortuga with 1717 files)
• Orchestrator: Node v20.20.0, running since 2026-03-21

Honest Assessment

The AX42-U is dramatically underutilized. At 1.5% memory and near-zero CPU, it's running $124/mo of hardware to host ~3.5MB of git repos and zero VMs. This isn't a criticism of past decisions — the infrastructure was built correctly and is ready to scale. But it means we have significant headroom on both servers.

2. Server Role Strategy

Recommendation: AX102-U = Production Compute, AX42-U = Internal Dev Server (IDS)

After analyzing the workloads, hardware, and growth trajectory, here's the recommended split:

Why NOT move everything to AX102-U

You suggested moving most functionality to AX102-U. I partially agree, but with a nuance:

Move VMs to AX102-U — Yes, absolutely. Customer VMs are the heaviest workload and benefit most from the 7950X3D's performance.

Keep git repos on AX42-U — The git API is I/O-bound (disk reads), not CPU-bound. AX42-U's NVMe is more than adequate. Moving git repos introduces risk (downtime during migration, DNS changes breaking existing integrations) with minimal benefit.

Keep AX42-U as the "stable" server — It's been running for 23+ days with zero issues. It has SSL certs, DNS, firewall, systemd — all configured. Disrupting it for a migration gains nothing.

Counterpoint to consider: If you plan to scale to 10+ servers in the future, a cleaner architecture would have ALL compute on dedicated servers and a separate lightweight control plane. But at 2 servers, pragmatism beats purity.

3. AX102-U Setup Plan

Phase 1: Server Bootstrap (SSH session required)

These commands must be run manually via SSH since we can't pipe the password programmatically:

# 1. SSH in (use password from Hetzner: s2hD?J2EhBpAm5)
ssh [email protected]

# 2. IMMEDIATELY change root password
passwd

# 3. Set hostname
hostnamectl set-hostname gograb-compute

# 4. Update system
apt update && apt upgrade -y

# 5. Install essentials
apt install -y curl wget git ufw fail2ban htop tmux jq \
  build-essential software-properties-common

# 6. Install Node.js 20 (LTS)
curl -fsSL https://deb.nodesource.com/setup_20.x | bash -
apt install -y nodejs
node --version  # Verify v20.x

# 7. Install NGINX + Certbot
apt install -y nginx certbot python3-certbot-nginx

# 8. Configure firewall
ufw default deny incoming
ufw default allow outgoing
ufw allow ssh
ufw allow 80/tcp    # HTTP (certbot + redirect)
ufw allow 443/tcp   # HTTPS
ufw enable

# 9. Set up SSH key auth (from your local machine)
# Run locally: ssh-copy-id [email protected]
# Then disable password auth:
sed -i 's/#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
systemctl restart sshd

# 10. Set timezone
timedatectl set-timezone UTC

Phase 2: Deploy Sandbox Orchestrator

# On AX102-U:

# 1. Create directory structure
mkdir -p /var/lib/sandbox-orchestrator
mkdir -p /data/platforms
mkdir -p /data/templates

# 2. Create .env file
cat > /var/lib/sandbox-orchestrator/.env << 'EOF'
NODE_ENV=production
PORT=3010
SANDBOX_API_KEY=<GENERATE_NEW_KEY_FOR_THIS_SERVER>
PLATFORM_REPOS_ROOT=/data/platforms
PLATFORM_TEMPLATE_PATH=/data/templates/platform-base-template
MAX_CONTAINERS=60
VM_VCPU_COUNT=2
VM_MEM_SIZE_MIB=2048
EOF

# 3. Install systemd service (same file, just copy)
# scp from local: deployment/sandbox-orchestrator/sandbox-orchestrator.service

# 4. Deploy binary
# From local machine:
#   yarn sandbox:deploy:host --host [email protected]

# 5. Install NGINX config (see Phase 3)
# 6. Start service
systemctl daemon-reload
systemctl enable sandbox-orchestrator
systemctl start sandbox-orchestrator

Phase 3: DNS + SSL + NGINX

# 1. Add DNS A record (in Cloudflare or DNS provider):
#    compute.gograb.space → 178.63.165.83
#    AAAA: compute.gograb.space → 2a01:4f8:2210:1340::2

# 2. Copy nginx config (adapted for compute.gograb.space)
# See Section 6 for the full nginx.conf

# 3. Get SSL cert
certbot --nginx -d compute.gograb.space --non-interactive --agree-tos -m [email protected]

# 4. Test
curl https://compute.gograb.space/health

Phase 4: Firecracker Setup (for customer VMs)

# 1. Install Firecracker
ARCH=$(uname -m)
FC_VERSION=1.6.0
curl -fsSL "https://github.com/firecracker-microvm/firecracker/releases/download/v${FC_VERSION}/firecracker-v${FC_VERSION}-${ARCH}.tgz" | tar xz
mv release-v${FC_VERSION}-${ARCH}/firecracker-v${FC_VERSION}-${ARCH} /usr/local/bin/firecracker
chmod +x /usr/local/bin/firecracker

# 2. Set up kernel + rootfs directories
mkdir -p /var/lib/firecracker/kernels
mkdir -p /var/lib/firecracker/images
mkdir -p /var/lib/sandboxes

# 3. Copy kernel + rootfs from AX42-U
scp [email protected]:/var/lib/firecracker/kernels/vmlinux-5.10 /var/lib/firecracker/kernels/
scp [email protected]:/var/lib/firecracker/images/base-desktop.ext4 /var/lib/firecracker/images/

# 4. Set up network bridge
cat > /etc/systemd/network/br-sandbox.netdev << 'EOF'
[NetDev]
Name=br-sandbox
Kind=bridge
EOF

cat > /etc/systemd/network/br-sandbox.network << 'EOF'
[Match]
Name=br-sandbox

[Network]
Address=172.16.0.1/24
EOF

systemctl restart systemd-networkd

# 5. Enable IP forwarding + NAT
echo 'net.ipv4.ip_forward=1' >> /etc/sysctl.conf
sysctl -p
iptables -t nat -A POSTROUTING -s 172.16.0.0/24 -o eth0 -j MASQUERADE
# Make iptables persistent:
apt install -y iptables-persistent
netfilter-persistent save

# 6. Verify
firecracker --version
ls /var/lib/firecracker/kernels/vmlinux-5.10  # should exist
ls /var/lib/firecracker/images/base-desktop.ext4  # should exist
curl -H "X-API-Key: $SANDBOX_API_KEY" https://compute.gograb.space/health
# vmPrerequisites.ready should be true

Phase 5: Copy Platform Template

# On AX102-U: Copy the platform base template from AX42-U
scp -r [email protected]:/data/templates/platform-base-template /data/templates/

# Verify
ls /data/templates/platform-base-template/

4. API Key Strategy

Critical decision: Should both servers share the same API key?

Recommendation: Separate keys per server

Decision: Generate a NEW API key for AX102-U. The backend already supports per-host API keys via SELF_HOSTED_INFRA_JSON.

# Generate new key for AX102-U
node -e "console.log(require('crypto').randomBytes(32).toString('base64url'))"
# Example output: xK7mP9qR2vN5bY8cJ3wF6tA1hD4gL0sE_uI

5. .env.local Changes

New variables to add:

# ── Hetzner Server Fleet (Session 260) ──────────────────────────────────
# Multi-server config: each server has its own URL, API key, label, and role.
# SANDBOX_SERVER_URL remains the PRIMARY server (used by SelfHostedProvider for VM creation).
# SELF_HOSTED_INFRA_JSON adds ADDITIONAL servers to the infrastructure dashboard.
#
# Server fleet:
#   AX42-U  (144.76.69.55)  = IDS (Internal Dev Server)  — sandbox.gograb.space
#   AX102-U (178.63.165.83) = Compute (Customer VMs)      — compute.gograb.space

# Primary sandbox server (used for VM creation by SelfHostedProvider)
# S260: Changed to AX102-U (more powerful, customer-facing)
SANDBOX_SERVER_URL=https://compute.gograb.space
SANDBOX_API_KEY=<AX102-U_API_KEY>
SANDBOX_SERVER_LABEL="Hetzner AX102-U — Compute (Falkenstein, DE)"
SANDBOX_SERVER_HARDWARE="Ryzen 9 7950X3D (16C/32T), 128GB DDR5, 2x1TB NVMe"

# Additional self-hosted servers (shown in infrastructure dashboard)
# Format: JSON array with per-host healthUrl, label, hardware, role, apiKey
SELF_HOSTED_INFRA_JSON=[{"healthUrl":"https://sandbox.gograb.space/health","label":"Hetzner AX42-U — IDS (Falkenstein, DE)","hardware":"Ryzen 7 PRO 8700GE (8C/16T), 128GB DDR5, 2x512GB NVMe","role":"internal-dev-server","apiKey":"W8Wp0ltL8El_qRnXyoUoD0ffuSfZzmbn7Y7SVrx1k8s"}]

What changes vs current:

Why this mapping: SANDBOX_SERVER_URL is used by SelfHostedProvider to CREATE VMs. We want new VMs on the more powerful AX102-U. AX42-U becomes a "secondary" server visible in the dashboard via SELF_HOSTED_INFRA_JSON.

Railway environment variables to add:

On Railway, both backend-ai-api and backend-vitto-api need:

SANDBOX_SERVER_URL=https://compute.gograb.space
SANDBOX_API_KEY=<AX102-U_API_KEY>
SELF_HOSTED_INFRA_JSON=[{"healthUrl":"https://sandbox.gograb.space/health","label":"AX42-U — IDS","hardware":"AX42-U","role":"ids","apiKey":"W8Wp0ltL8El_qRnXyoUoD0ffuSfZzmbn7Y7SVrx1k8s"}]

6. NGINX Configuration for AX102-U

The existing nginx.conf has a critical gap: no /git/ routes. The git API works on AX42-U only because requests go directly to port 3010 (or the route happens to match a catch-all). The nginx config in the repo explicitly returns 404 for unmatched paths.

Fixed nginx.conf (for both servers)

# GoGrab Sandbox Orchestrator — Nginx reverse proxy
# Session 260: Dual-server infrastructure
#
# Server: compute.gograb.space (AX102-U) or sandbox.gograb.space (AX42-U)
# Handles: SSL termination, WebSocket upgrade (VNC), rate limiting, git API proxy

limit_req_zone $binary_remote_addr zone=sandbox_create:10m rate=5r/m;

server {
    listen 80;
    server_name _;  # Replace with actual domain during deployment
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl http2;
    server_name _;  # Replace with actual domain during deployment

    # SSL managed by certbot
    ssl_certificate /etc/letsencrypt/live/$DOMAIN/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/$DOMAIN/privkey.pem;
    include /etc/letsencrypt/options-ssl-nginx.conf;
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

    # Proxy timeouts
    proxy_connect_timeout 10s;
    proxy_send_timeout 120s;   # Increased from 60s for large git commits
    proxy_read_timeout 120s;   # Increased from 60s for large git commits

    # S251: Max body size for platform generation (136+ file commits)
    client_max_body_size 50m;

    # ── Health check (no auth) ──────────────────────────────────────────
    location /health {
        proxy_pass http://127.0.0.1:3010/health;
        proxy_set_header Host $host;
    }

    # ── Metrics (no auth, for Prometheus) ───────────────────────────────
    location /metrics {
        proxy_pass http://127.0.0.1:3010/metrics;
        proxy_set_header Host $host;
    }

    # ── VNC streaming (WebSocket upgrade) ───────────────────────────────
    location /vnc/ {
        proxy_pass http://127.0.0.1:3010/vnc/;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_read_timeout 86400;
        proxy_send_timeout 86400;
    }

    # ── Sandbox creation (rate limited) ─────────────────────────────────
    location = /sandboxes {
        limit_req zone=sandbox_create burst=3 nodelay;
        proxy_pass http://127.0.0.1:3010/sandboxes;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-API-Key $http_x_api_key;
    }

    # ── All other sandbox routes ────────────────────────────────────────
    location /sandboxes/ {
        proxy_pass http://127.0.0.1:3010/sandboxes/;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-API-Key $http_x_api_key;
    }

    # ── Git operations API (S251 addition — WAS MISSING FROM NGINX) ────
    location /git/ {
        proxy_pass http://127.0.0.1:3010/git/;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-API-Key $http_x_api_key;
        # Large body for file commits
        client_max_body_size 50m;
    }

    # ── Internal dashboard summary ──────────────────────────────────────
    location /internal/ {
        proxy_pass http://127.0.0.1:3010/internal/;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-API-Key $http_x_api_key;
    }

    # ── Default: 404 ────────────────────────────────────────────────────
    location / {
        return 404 '{"error":"Not found","service":"sandbox-orchestrator"}';
        add_header Content-Type application/json;
    }
}

Issues found in current nginx.conf:

1. Missing /git/ location — Git API calls only work because the proxy was configured differently on the actual server, or requests bypass NGINX
2. client_max_body_size 10m — Should be 50m per S251 fix. The fix was applied on the running server but the repo config wasn't updated
3. Hardcoded server_name sandbox.gograb.space — Need parameterized for multi-server

7. Git-Operations Library: Multi-Server Support

Current limitation

The git-operations library assumes a single baseUrl (one sandbox-orchestrator). The GIT_OPS_BASE_URL token points to /sandbox-api which proxies to ONE server.

Proposed approach: Server context per request

For Phase 1 (now), we DON'T need multi-server git routing. Both servers run the orchestrator, but git repos stay on AX42-U. The git-operations dashboard talks to /sandbox-api → AX42-U (as today).

For Phase 2 (future), when we want to browse repos across both servers:

// Option A: Multiple provider instances (cleanest)
// Each "server card" in the dashboard injects its own GitOperationsApiService instance
// with a different baseUrl. No library changes needed — just DI configuration.

// Option B: Server-aware API service (more flexible)
// Add optional serverUrl param to each API method:
// listRepos(serverUrl?: string): Observable<GitRepoInfo[]>

Recommendation: Don't change the library now. The infrastructure dashboard (/infrastructure/hetzner) already shows both servers with their metrics. The git-operations page can stay pointed at AX42-U. This avoids unnecessary complexity.

8. Infrastructure Dashboard Integration

The Hetzner dashboard at /infrastructure/hetzner already supports multiple self-hosted servers via SELF_HOSTED_INFRA_JSON. Once we add the AX102-U entry to .env.local and deploy the updated env to Railway's backend-ai-api, both servers will appear automatically in the dashboard.

What will show:

No code changes needed for the dashboard

The HetznerInfraService.buildSelfHostedServerViews() iterates getSelfHostedEntries() which reads both SANDBOX_SERVER_URL (primary) and SELF_HOSTED_INFRA_JSON (additional). Each gets its own synthetic ID, health probe, and dashboard-summary fetch.

9. Deploy Script Enhancement

The current deploy-orchestrator-host.sh defaults to [email protected]. We should make it multi-host aware.

Proposed: Named host targets

# Deploy to IDS (default, backwards compatible)
yarn sandbox:deploy:host

# Deploy to Compute server
yarn sandbox:deploy:host --host [email protected]

# Deploy to both
yarn sandbox:deploy:all

A new deploy-orchestrator-all.sh script that iterates both hosts would be useful once AX102-U is running.

10. Platform Git Server Routing

Current flow (generation pipeline)

Vitto (backend-vitto-api)
  → PLATFORM_GIT_SERVER_URL (https://sandbox.gograb.space)
  → AX42-U sandbox-orchestrator
  → /git/repos/:org/:repo/files (commit generated code)

After S260

No change needed yet. Git repos stay on AX42-U. The generation pipeline continues to commit to sandbox.gograb.space. If we later decide to move git repos to AX102-U, we just change PLATFORM_GIT_SERVER_URL.

Future consideration: Git repo replication

For disaster recovery, we could add a post-commit hook or cron job that git push --mirror repos from AX42-U to AX102-U. This gives us:

• Hot standby for git repos
• Zero-downtime migration capability (swap DNS, clients don't notice)

Not needed now, but the architecture supports it cleanly.

11. Risks and Mitigations

12. Execution Order

Step 1: Local changes (no server access needed)

1. Update .env.local with SELF_HOSTED_INFRA_JSON to show AX42-U as secondary
2. Fix nginx.conf in repo (add /git/, fix client_max_body_size)
3. Verify dashboard shows both servers (AX42-U will show, AX102-U will show as "unreachable" until set up)

Step 2: AX102-U bootstrap (SSH session — manual)

1. SSH in with password
2. Change password, set hostname, install packages
3. Set up SSH key, disable password auth
4. Configure firewall (ufw)

Step 3: Deploy orchestrator to AX102-U

1. Create directories, .env, systemd service
2. Build + deploy binary from local
3. Verify health endpoint (HTTP, no SSL yet)

Step 4: DNS + SSL

1. Add compute.gograb.space DNS record
2. Run certbot
3. Configure NGINX with the fixed config
4. Verify HTTPS health

Step 5: Firecracker setup

1. Install Firecracker binary
2. Copy kernel + rootfs from AX42-U
3. Set up network bridge + NAT
4. Test VM creation

Step 6: Integration verification

1. Update .env.local with real AX102-U API key
2. Verify dashboard shows both servers as "running"
3. Test VM creation on AX102-U from the platform
4. Update Railway env vars

Step 7: Fix AX42-U nginx (while we're at it)

1. SSH to AX42-U
2. Add /git/ location block
3. Update client_max_body_size to 50m
4. nginx -t && systemctl reload nginx

13. Future Considerations (not for this session)

1. Git repo replication — Mirror repos between servers for DR
2. Load-balanced VM creation — Route new VM requests to the server with most capacity
3. Prometheus + Grafana — Centralized monitoring on AX42-U watching both servers
4. Automated deploy pipeline — GitHub Actions → build → deploy to both servers
5. Server fleet config — Move from env vars to a servers.json config file when we hit 3+ servers
6. Firewall between servers — Hetzner vSwitch for private inter-server traffic (same DC campus)

14. Validation Checklist

After all steps complete:

• curl https://compute.gograb.space/health returns status: ok
• curl -H "X-API-Key: $KEY" https://compute.gograb.space/sandboxes/meta/dashboard-summary returns host metrics
• curl https://sandbox.gograb.space/health still works (no regression)
• Dashboard at /infrastructure/hetzner shows both servers as "running"
• VM creation on AX102-U succeeds (create → screenshot → destroy)
• Git operations on AX42-U still work (list repos, browse files)
• Deploy script works: yarn sandbox:deploy:host --host [email protected]

Generated by Claude Code (Session 260.1) — March 23, 2026