Internal Dev Server (IDS) — Architecture Design Spec

Date: 2026-03-20 Session: S244 Status: Draft Scope: Replace per-executive Firecracker microVMs with a shared, persistent, dedicated development server using git worktrees and Linux namespace isolation for internal gspace1 development.

Problem Statement

The current executive workspace VM plan provisions individual 12 GiB / 4 vCPU Firecracker microVMs per executive. This has five compounding problems:

1. Slow cold start. 10s Firecracker boot + 5-10 min yarn install + git clone = executives wait minutes before Vitto can write code.
2. Resource waste. Each VM duplicates the OS (~1 GiB), .git objects (~1.5 GiB), node_modules (~500 MB), and toolchain. Five executives = 5x duplication.
3. Customer blast radius. Executive VMs share the Hetzner AX42-U (128 GiB) with customer-facing sandboxes. 6+ concurrent executives = host OOM, customer VMs affected.
4. Wrong isolation model. Firecracker provides hardware-level isolation for untrusted multi-tenant workloads. Executives are trusted users on the same codebase — they need branch isolation, not kernel isolation.
5. No admission control. MAX_CONTAINERS=30 counts units, not memory. The orchestrator will provision the 11th 12 GiB VM that OOM-kills the host.

Goal: 3-8 second session start (not minutes), 6x less memory per session, zero customer impact, proper branch isolation between executives.

Architecture Overview

Current Flow (Per-Executive Firecracker VM)

Executive sends WhatsApp message to Vitto
  channel-processing extracts platformRole
  tier-resolver selects 'vm' tier
  vm-session-manager.acquire()
    VmApiClient.createSandbox(memSizeMib=12288, vcpuCount=4)
      backend-ai-api proxy
        sandbox-orchestrator Firecracker microVM (cold boot, 10s)
          vm-init.sh: seed entropy, mount fs, start services
            (manual) git clone gspace1, yarn install (5-10 min)
              finally ready

Total time to first productive command: 5-11 minutes.

Proposed Flow (Internal Dev Server)

Executive sends WhatsApp message to Vitto
  channel-processing extracts platformRole
  tier-resolver selects 'ids' tier (new)
  ids-session-manager.acquire()
    IDS Agent REST API: POST /sessions { userId, branch }
      git worktree add /sessions/{id} -b session-{N}  (~1s)
      pnpm install (symlinks from global store)         (~2-5s)
      create namespace container (systemd-nspawn)        (~0.5s)
      return { sessionId, endpoint }
        ready

Total time to first productive command: 3-8 seconds.

Design Principles

1. Separate workloads, separate infrastructure. Internal dev NEVER shares host with customer sandboxes.
2. Share what is identical, isolate what is different. Same repo, same dependencies = shared. Different branches, different processes = isolated.
3. Persistent, not ephemeral. The server is always on, always warm. Sessions are lightweight overlays.
4. Additive, not destructive. Existing Firecracker infrastructure stays intact for customer VMs and standard developer tier. IDS is a new provider alongside it.

Infrastructure

Dedicated Server

Recommendation: Hetzner AX102 (~109 EUR/month, pre-April 2026 pricing)

Why not reuse the existing Hetzner AX42-U?

• Customer sandbox workloads compete for RAM with executive dev
• A rogue nx build OOM could kill customer VMs
• Different SLA: customer sandboxes need high availability, dev needs speed
• Security: internal source code should not coexist with customer workloads

Pre-Installed Toolchain

Baked into the server on first setup (not per-session):

Core Components

1. Canonical Clone (Shared, Read-Only Base)

/workspace/gspace1/              -- always on main, auto-fetched
  .git/objects/                  -- shared object store (all worktrees reference this)
  .pnpm-store/                   -- content-addressable dependency store
  .nx/cache/                     -- shared NX computation cache
  node_modules/                  -- main branch dependencies (pnpm symlinks)

Maintenance cron (every 15 minutes):

cd /workspace/gspace1
git fetch --all --prune
pnpm install --frozen-lockfile  # update store if lockfile changed on main

This ensures new branches/commits are available to all worktrees within 15 minutes of being pushed.

2. Session Worktrees (Per-Executive, Isolated)

/workspace/sessions/
  ids-{userId}-{sessionNumber}/     -- git worktree, own branch
      node_modules/                 -- pnpm symlinks (instant)
      dist/                         -- per-session build output
      [full checkout]

Lifecycle:

1. Create: git worktree add /workspace/sessions/{id} -b session-{N}
2. Install: cd /workspace/sessions/{id} && pnpm install (2-5s, symlinks to global store)
3. Use: Vitto runs commands in the worktree via namespace container
4. Idle timeout: 24 hours of no activity triggers cleanup
5. Cleanup: git worktree remove /workspace/sessions/{id} && git worktree prune

Branch rules:

• Session branches follow SAGP: session-{N} (no collisions by design)
• Git enforces: one branch per worktree (two executives cannot check out the same branch)
• If an executive needs to work on an existing branch: git worktree add ... existing-branch (worktree linked to remote branch, no conflict with other worktrees on different branches)

3. Namespace Containers (Process + Network Isolation)

Each active session runs inside a lightweight Linux namespace container:

Host: Hetzner AX102

  nspawn: session-245             nspawn: session-246
  PID ns: isolated                PID ns: isolated
  Net ns: own stack               Net ns: own stack
  Ports: 3000-3099 (own lo)       Ports: 3000-3099 (own lo)
  cgroup: 8 GiB mem, 4 CPU       cgroup: 8 GiB mem, 4 CPU
  bind: /sessions/s-245/          bind: /sessions/s-246/

Isolation guarantees:

Why systemd-nspawn over Docker?

• No daemon overhead (direct kernel calls)
• Native .nspawn config files for persistent settings
• Built into systemd (already on Ubuntu)
• machinectl for lifecycle management
• Simpler than Docker for this use case (no image builds, no registries)

4. IDS Agent (REST API)

A lightweight NestJS service running on the dedicated server. Mirrors the sandbox-orchestrator's API surface for easy integration.

Endpoints:

Auth: API key (same pattern as sandbox-orchestrator's X-API-Key).

Config (.env):

IDS_API_KEY=<shared secret>
IDS_PORT=3020
WORKSPACE_BASE=/workspace/gspace1
SESSIONS_DIR=/workspace/sessions
SESSION_IDLE_TTL_HOURS=24
SESSION_MAX_MEMORY_MIB=8192        # per-session cgroup limit
SESSION_MAX_CPU_SHARES=4           # per-session CPU allocation
MAX_CONCURRENT_SESSIONS=15         # admission control
TOTAL_MEMORY_BUDGET_MIB=102400     # ~100 GiB budget (128 GiB - OS overhead)
GIT_FETCH_INTERVAL_MINUTES=15

Admission control (the fix for Gap 1 in current plan):

On POST /sessions:
  1. Check MAX_CONCURRENT_SESSIONS (count-based limit)
  2. Check TOTAL_MEMORY_BUDGET_MIB - sum(active session cgroup limits) >= SESSION_MAX_MEMORY_MIB
  3. If either fails: 503 Service Unavailable with reason

5. Command Execution Model

Commands run inside the session's namespace via machinectl shell:

IDS Agent receives: POST /sessions/{id}/exec { command: "nx build backend-vitto-api" }
  Validate command against allowlist (reuse VITTO_COMMAND_ALLOWLIST)
  machinectl shell ids-session-{id} /bin/bash -c "cd /workspace && nx build backend-vitto-api"
  Stream stdout/stderr back
  Return exit code + output

Security:

• Same command allowlist as current monorepo tools (nx, git, yarn, node, npx, etc.)
• Structured API calls (no shell string interpolation)
• cgroup enforces memory/CPU limits
• Namespace prevents lateral movement between sessions

Integration with Existing Codebase

What Changes

What Stays Unchanged

• monorepo-remote.plugin.ts — same tool definitions, same role gating
• tool-registry.service.ts — same role filtering
• tool-security.service.ts — same allowlists
• role-derivation.ts — isExecutiveFullStackWorkspaceVmEligible() reused (or aliased to isIdsEligible())
• channel-processing.service.ts — same role extraction
• agent-loop.service.ts — same tool dispatch
• sub-vitto-orchestrator.service.ts — same tier tool mapping
• sandbox-orchestrator — completely untouched (stays for customer VMs)
• All customer-facing infrastructure — zero impact

The Key Routing Change

In vm-session-manager.service.ts:

async acquire(userId: string, template?: string, opts?: VmAcquireOptions): Promise<VmSession> {
  const useIds = isExecutiveFullStackWorkspaceVmEligible(
    opts?.platformRole,
    opts?.jwtRoles,
  );

  if (useIds) {
    // Route to Internal Dev Server (shared, persistent, fast)
    return this.idsSessionManager.acquire(userId, {
      branch: opts?.branch,
      platformRole: opts?.platformRole,
    });
  }

  // Existing Firecracker path for standard developer VMs (unchanged)
  // ...
}

The VmSession interface stays the same — both Firecracker and IDS return { vmId, userId, status }. Downstream code does not know or care which backend fulfilled the session.

pnpm Migration Strategy

The biggest performance enabler is pnpm with enableGlobalVirtualStore, which turns per-worktree dependency installation from 5-10 minutes to 2-5 seconds.

Why pnpm?

Phased Approach

Phase A (immediate, no pnpm): Pre-built node_modules tarball from main branch. On session create, untar into worktree (~30-60s). Good enough for MVP.

Phase B (after validation): Migrate monorepo to pnpm. Update CI, lock file, package.json packageManager field. Session create drops to 2-5s.

Phase C (optimization): Enable pnpm's enableGlobalVirtualStore. One content-addressable store on disk, all worktrees are pure symlinks. Near-zero marginal disk per worktree.

The IDS architecture works with any phase. pnpm just makes it shine.

Session Lifecycle Details

Acquire (Executive Sends Message)

1. Check activeSessions map for userId
   HIT: return existing session (instant, 0 API calls)
   MISS: continue

2. Check IDS Agent: GET /sessions/user/{userId}
   HIT (idle session, <24h old): restart namespace, return (~1-2s)
   MISS: continue

3. Create new session:
   POST /sessions {
     userId,
     branch: "session-{N}",    // from SAGP protocol
   }

   IDS Agent:
     a. git fetch origin (incremental, ~1-2s)
     b. git worktree add /workspace/sessions/{id} -b session-{N}
     c. cd worktree && pnpm install (or untar node_modules)
     d. Create nspawn container with cgroup limits
     e. Return session handle

4. Store in activeSessions map
5. Return to caller

Execute (Vitto Runs Tools)

monorepo_run_command("nx test my-lib --watch=false")
  sandboxed-tool-proxy detects IDS-backed session
  idsApiClient.exec(sessionId, { command, cwd, timeoutMs })
  IDS Agent: machinectl shell in namespace
  Return stdout/stderr/exitCode

Release (Conversation Ends or Idle)

1. Vitto conversation ends: idsSessionManager.release(sessionId)
2. IDS Agent: POST /sessions/{id}/release
   a. Kill all processes in namespace (dev servers, watchers)
   b. Keep worktree on disk (uncommitted changes preserved)
   c. Mark session as "idle" with timestamp
3. Remove from activeSessions map

After 24h idle:
   a. git stash (preserve any uncommitted work)
   b. git worktree remove /workspace/sessions/{id}
   c. git worktree prune
   d. Remove nspawn config

Cleanup Cron (Every Hour)

# Remove idle sessions older than TTL
for session in /workspace/sessions/ids-*; do
  last_used=$(stat -c %Y "$session")
  age_hours=$(( (now - last_used) / 3600 ))
  if [ $age_hours -gt $SESSION_IDLE_TTL_HOURS ]; then
    git -C /workspace/gspace1 worktree remove "$session" --force
  fi
done
git -C /workspace/gspace1 worktree prune

Observability

IDS Agent /metrics Endpoint

{
  "host": {
    "totalMemoryMib": 131072,
    "usedMemoryMib": 24576,
    "cpuUsagePercent": 35,
    "diskUsedGib": 120,
    "diskTotalGib": 1920
  },
  "sessions": {
    "active": 3,
    "idle": 2,
    "maxConcurrent": 15,
    "memoryBudgetRemainingMib": 77824
  },
  "perSession": [
    {
      "sessionId": "ids-user1-245",
      "userId": "user1",
      "branch": "session-245",
      "status": "active",
      "memoryUsageMib": 3200,
      "cpuPercent": 12,
      "createdAt": "2026-03-20T10:00:00Z",
      "lastCommand": "nx serve backend-vitto-api",
      "lastCommandAt": "2026-03-20T14:30:00Z"
    }
  ]
}

Alerts

Fallback Strategy

If the IDS server is unreachable (maintenance, failure, network):

// In ids-session-manager.service.ts
async acquire(userId: string, opts?: IdsAcquireOptions): Promise<VmSession> {
  try {
    return await this.idsApiClient.createSession(userId, opts);
  } catch (error) {
    this.logger.warn(
      `IDS unreachable, falling back to Firecracker VM: ${error.message}`,
    );
    // Fall back to existing Firecracker path (slower but works)
    return this.vmSessionManager.acquireFirecracker(userId, opts);
  }
}

The Firecracker path stays intact as a fallback. Executives get slower sessions during IDS outages, but are never blocked.

Security Model

Trust Boundary

Executives are trusted internal users with platformRole === 'executive'. They already have:

• monorepo_write_file access (can modify any file in gspace1)
• monorepo_run_command access (allowlisted shell commands)
• Full source code visibility

The IDS server is inside the trust boundary. We are not adding new capabilities — just a faster backend.

Defense in Depth

What IDS Does NOT Protect Against

• A kernel exploit inside a namespace can escape to the host (same as Docker). This is acceptable because executives are trusted.
• An executive with monorepo_run_command could theoretically git push --force main. This is already true today. The allowlist + SAGP branch naming convention mitigates this.

Comparison: Current Plan vs IDS

Implementation Plan

Phase 1: Dedicated Server + Basic IDS Agent (2-3 days)

Deliverables:

• Provision Hetzner AX102
• Install toolchain (Node 20, pnpm, git, build-essential)
• Clone gspace1, run initial pnpm install (or yarn install if pnpm migration deferred)
• Build IDS Agent (NestJS): POST /sessions, DELETE /sessions/:id, POST /sessions/:id/exec, POST /sessions/:id/read-file, POST /sessions/:id/write-file, GET /health
• Basic isolation: separate Linux users per session (no nspawn yet)
• Wire IdsApiClientService in backend-vitto-api
• Add conditional routing in vm-session-manager.service.ts

Phase 2: Namespace Isolation + Cgroup Limits (1 day)

Deliverables:

• Add systemd-nspawn containers per session
• cgroup v2 memory limits (8 GiB default per session)
• Network namespace isolation (each session gets own loopback + port space)
• Memory-aware admission control in IDS Agent

Phase 3: Dependency Optimization (independent timeline)

Deliverables:

• Pre-built node_modules tarball on main branch (Phase A)
• OR: pnpm migration for the monorepo (Phase B)
• NX cache sharing across worktrees

Phase 4: Observability + Hardening (1 day)

Deliverables:

• /metrics endpoint with per-session stats
• Stale worktree cleanup cron
• Alert integration (Slack/ops channel)
• Fallback path: IDS unreachable leads to auto-degrade to Firecracker
• Load testing: 10 concurrent executive sessions

Risks and Mitigations

Open Questions

1. pnpm migration timeline. Should this be done before or after IDS? IDS works with yarn classic (tarball approach), but pnpm unlocks the full speed benefit.
2. Dev server persistence. When an executive asks Vitto to yarn serve:backend-vitto-api, should the dev server survive across conversations? Current design: killed on session release. Alternative: keep running if session is "idle" (not "released").
3. Multiple simultaneous sessions per executive. Current design: one session per userId (matching existing VmSessionManager). Should one executive be able to have two worktrees on different branches simultaneously?
4. IDS Agent deployment. Standalone binary? Docker container on the Hetzner? systemd service? (Recommendation: systemd service, matching sandbox-orchestrator pattern.)
5. NX remote cache. Should we set up NX Cloud or a local remote cache on the IDS server so that builds cached by one executive benefit all others?

Operational visibility (dedicated hosts)

GoGrab Dev → System → Hetzner (/infrastructure/hetzner): primary data path is self-hosted — SANDBOX_SERVER_URL on backend-ai-api (same as computer-use / SelfHostedProvider); the API probes GET {origin}/health on sandbox-orchestrator. No Hetzner Cloud account required. Optional SELF_HOSTED_INFRA_JSON adds more rows (e.g. future IDS host). Optional: HETZNER_API_TOKEN merges servers from the Hetzner Cloud API; reboot / power actions apply only to those cloud rows (bare metal is SSH / host panel).

Appendix A: Compatibility with Existing Uncommitted Changes

The current branch (session-239-244_2) has ~50 uncommitted files from a prior agent implementing the per-executive Firecracker VM plan. Analysis of the diff shows:

No reverts needed. IDS is additive. The existing changes provide the role infrastructure that IDS depends on.

Appendix B: IDS Agent — Minimal Implementation Skeleton

ids-agent/
  src/
    main.ts                      # NestJS bootstrap
    app.module.ts                # Module registration
    sessions/
      sessions.controller.ts   # REST endpoints
      sessions.service.ts      # Worktree + namespace lifecycle
      sessions.types.ts        # DTOs and interfaces
      sessions.guard.ts        # API key validation
    worktree/
      worktree.service.ts      # Git worktree operations
    namespace/
      namespace.service.ts     # systemd-nspawn lifecycle
    health/
      health.controller.ts     # /health, /metrics
      metrics.service.ts       # Per-session resource tracking
    cleanup/
      cleanup.service.ts       # TTL-based worktree cleanup cron
  .env                             # Configuration
  package.json
  tsconfig.json

Estimated: ~500-800 lines of application code. The simplicity is the point.