150 Work Report
Documentation for Sessions
Session 150 — Work Summary (February 27, 2026)
What Was Done
Session 150 built the entire self-hosted sandbox infrastructure — our own replacement for E2B Cloud. This is the system that lets Vitto (our AI agent) control a virtual desktop (click, type, browse the web) to complete tasks for users. Until now, we rented virtual desktops from E2B Cloud. Now we're building our own.
Why This Matters (For Non-Developers)
Before (E2B Cloud):
- We pay E2B for every virtual desktop session
- Sessions are capped at 1 hour (free) or 24 hours (paid)
- We can't install our own Chrome extension inside the virtual desktops
- E2B holds all the data (what users browse, screenshots, etc.)
- Costs scale: $150/month for 10 users, potentially $300+ for 100 users
After (Self-Hosted):
- We own the hardware — a dedicated server at $46/month flat, regardless of users
- Sessions have no time limit — a task can run as long as needed
- We can pre-install our Chrome extension, bookmarks, and custom configs
- All data stays on our server — full privacy and control
- Break-even point: ~10 active users, then it's cheaper than E2B forever
Think of it like renting vs. owning your office space. Renting is convenient when you're small, but once you hit a certain size, buying is always better.
What Was Built — The Non-Technical Version
1. The Virtual Desktop Image (Day 1 of the plan)
We created a "blueprint" for what each virtual desktop looks like. When Vitto needs to browse the web, a new desktop is created from this blueprint in seconds. Each desktop has:
- A lightweight desktop environment (XFCE — like a basic Windows/Mac desktop)
- Google Chrome (pre-configured, no annoying "first run" popups)
- Automation tools (so Vitto can move the mouse, click, type, take screenshots)
- A live-view server (so users can watch what Vitto is doing in real-time)
2. The Orchestrator (Day 3-5 of the plan)
This is the "manager" that handles creating, monitoring, and cleaning up virtual desktops. When a user asks Vitto to "log into Gmail and check my inbox," the orchestrator:
- Creates a fresh virtual desktop (in seconds)
- Gives it an internet connection
- Tells Vitto the desktop's address
- Watches for idle desktops and shuts them down (saves resources)
- Cleans up everything when done
Think of the orchestrator as a hotel front desk — it assigns rooms, tracks guests, and cleans up after checkout.
3. The Connection (Day 5)
The existing Vitto system was already designed to work with a self-hosted server (built in Session 144). Session 150 enhanced this connection with:
- Security: An API key so only our system can talk to the sandbox server
- Reliability: If a request fails, it automatically retries (twice, with increasing wait times)
- Health monitoring: Every 60 seconds, it checks if the sandbox server is still alive
4. Server Infrastructure (Day 1-2 of the plan)
Scripts that set up a brand-new server from scratch:
- Install Firecracker (the VM technology used by AWS Lambda — battle-tested)
- Configure networking so virtual desktops can access the internet
- Set up security (SSL encryption, firewalls, API authentication)
- Automatic cleanup of crashed or orphaned desktops
What Was Built — The Technical Version
Architecture
User Request → backend-ai-api (Railway)
↓
SandboxProviderFactory
↓
SelfHostedProvider (HTTP client)
↓ HTTPS + API Key
Sandbox Orchestrator (NestJS on Hetzner bare metal)
↓
Firecracker microVMs
┌───────┴───────┐
│ VM per session │
│ Ubuntu + XFCE │
│ Chrome + xdotool│
│ x11vnc + SSH │
└──────────────────┘
Files Created (32 total)
NestJS Sandbox Orchestrator (APIs/sandbox-orchestrator/) — 21 files:
| Service | What It Does |
|---|---|
firecracker.service.ts |
Starts/stops Firecracker VM processes, configures them via UNIX API socket (boot-source, drives, machine-config, network), copies rootfs images, generates per-session SSH keys |
network.service.ts |
Manages IP address pool (172.16.0.2-254), creates/destroys TAP devices, attaches to bridge |
vm-executor.service.ts |
Executes xdotool/scrot commands inside VMs via SSH — replicates every E2B SDK method |
sandbox-manager.service.ts |
Orchestrates create→boot→waitForReady→running→destroy lifecycle |
sandbox.controller.ts |
11 REST endpoints matching SelfHostedProvider's HTTP contract exactly |
cleanup.service.ts |
Reaps idle VMs (30min) and expired VMs (60min), detects dead Firecracker processes |
api-key.guard.ts |
X-API-Key header validation with constant-time comparison |
vnc-proxy.service.ts |
WebSocket↔TCP proxy for live desktop streaming (frontend → nginx → proxy → VM x11vnc) |
metrics.service.ts |
Prometheus metrics (active VMs, capacity, create/screenshot durations, IP pool) |
health.controller.ts |
GET /health for SelfHostedProvider connectivity checks |
VM Rootfs Image (deployment/sandbox-orchestrator/) — 7 files:
| File | What It Does |
|---|---|
Dockerfile.rootfs |
Ubuntu 22.04 + XFCE4 + Chrome + xdotool + scrot + x11vnc + SSH + fonts |
vm-init.sh |
PID 1 init process inside VM — parses kernel boot args, starts display→desktop→Chrome→VNC→SSH |
vm-network.sh |
Configures networking from kernel boot arguments (static IP, gateway, DNS) |
build-rootfs.sh |
Docker→ext4 conversion pipeline (build in Docker for reproducibility, export to ext4 for Firecracker) |
Dockerfile.orchestrator |
Node.js Alpine image with SSH client + iproute2 for the orchestrator |
sandbox-orchestrator.service |
systemd unit file for auto-restart |
nginx.conf |
Reverse proxy with SSL, WebSocket upgrade for VNC, rate limiting for sandbox creation |
Server Scripts (tools/scripts/sandbox/) — 3 files:
| Script | What It Does |
|---|---|
provision-server.sh |
Full Hetzner AX41 setup: Firecracker v1.7.0, KVM, Node.js 20, nginx, SSL, firewall |
setup-network.sh |
Bridge network (br-sandbox, 172.16.0.1/24), NAT masquerade, iptables rules, persistence |
deploy-orchestrator.sh |
Build NX app → rsync to server → restart systemd → health check |
Enhanced Existing File — 1 modified:
| File | Change |
|---|---|
self-hosted.provider.ts |
Added API key header, 2-retry exponential backoff, 60s health monitoring, 45s screenshot timeout |
How the E2B SDK Methods Map to Our Implementation
This table shows exactly what E2B Cloud does vs. what our system does for each operation:
| Operation | E2B Cloud (before) | Our Firecracker System (after) |
|---|---|---|
| Create desktop | E2BSandbox.create() → E2B Cloud API |
Firecracker VM boot + Xvfb + Chrome startup |
| Take screenshot | sandbox.screenshot('bytes') → native API |
ssh root@VM "DISPLAY=:99 scrot -o /tmp/screen.png && base64 /tmp/screen.png" |
| Move mouse | sandbox.moveMouse(x, y) |
ssh root@VM "DISPLAY=:99 xdotool mousemove x y" |
| Click | sandbox.leftClick() |
ssh root@VM "DISPLAY=:99 xdotool click 1" |
| Type text | sandbox.write(text) |
ssh root@VM "DISPLAY=:99 xdotool type -- 'text'" |
| Press key | sandbox.press(key) |
ssh root@VM "DISPLAY=:99 xdotool key key" |
| Scroll | sandbox.scroll(dir, n) |
ssh root@VM "DISPLAY=:99 xdotool click --repeat n 4|5" |
| Run command | sandbox.commands.run(cmd) |
ssh root@VM "bash -c 'cmd'" |
| Live view | sandbox.stream.getUrl() |
x11vnc → WebSocket proxy → browser |
| Destroy | sandbox.kill() |
Kill Firecracker process, delete TAP, release IP, remove rootfs |
Key Design Decisions
1. Firecracker Over Docker
Why: The user explicitly chose Firecracker for hardware-level isolation (each VM gets its own kernel). Docker shares a kernel — weaker isolation. Firecracker is what AWS Lambda uses for multi-tenant workloads.
Tradeoff: More complex setup (KVM bare metal required, TAP networking, custom rootfs), but the SandboxProvider interface makes the backend completely agnostic — zero changes to ComputerUseLoopService.
2. SSH for VM Communication (v1)
Why: SSH is universally available, well-understood, and requires no custom agent inside the VM. Just apt install openssh-server in the rootfs and we're done.
Tradeoff: ~50-100ms overhead per command vs E2B's native SDK. Acceptable — the screenshot loop runs at 2-3 seconds regardless. Future optimization: Firecracker vsock (direct hypervisor channel, no networking overhead).
3. Docker-Based Rootfs Building
Why: Building a Firecracker ext4 rootfs from scratch (debootstrap + chroot) is fragile and non-reproducible. Using a Dockerfile gives us the same reproducibility as any container image, then we just export the filesystem to ext4.
4. Per-Session VM Isolation (Session 140 Carried Forward)
Why: Each Computer Use session gets its own isolated VM. Parallel tasks never share a desktop — no click/type conflicts. Session 140 already changed the model from 1-per-user to 1-per-session.
Timeline & Milestones
| When | What | Status |
|---|---|---|
| Session 139 | Computer Use with E2B Cloud — first working version | Done |
| Session 140 | Per-session sandboxes — parallel task execution | Done |
| Session 141 | Self-hosted E2B plan written (original, 420 lines) | Done |
| Session 144 | Provider abstraction — SandboxProvider interface, 3 providers, factory | Done |
| Session 150 | Orchestrator built + deployment scripts + rootfs image + provider enhanced | Done |
| Next | Provision Hetzner AX41, build rootfs, deploy orchestrator, E2E test | Planned |
Next Steps (What Remains)
Immediate (Before Going Live)
- Order Hetzner AX41-NVMe ($46/mo, bare metal with KVM)
- Run provision-server.sh on the new server
- Build rootfs using build-rootfs.sh (test in QEMU first)
- Deploy orchestrator using deploy-orchestrator.sh
- Set env vars in backend-ai-api:
SANDBOX_SERVER_URL+SANDBOX_API_KEY - E2E test: trigger Computer Use from frontend, verify full loop
After Going Live
- Load test: 10+ concurrent VMs, measure memory/CPU
- Wire metrics into Observatory dashboard
- Consider btrfs for instant CoW rootfs copies (currently uses sparse cp)
- Consider Firecracker vsock for lower-latency VM communication
Future
- Custom Chrome extension pre-installed in rootfs
- User session persistence (login once, reuse across tasks)
- Multi-server scaling (add second Hetzner server)
- Firecracker snapshots for instant VM restore
Cost Comparison
| Scenario | E2B Free | E2B Pro | Self-Hosted |
|---|---|---|---|
| 1 user, 10 tasks/day | $0 | $0 | $46/mo |
| 10 users, 50 tasks/day | Limited | $150/mo | $46/mo |
| 50 users, 200 tasks/day | Over limit | ~$250/mo | $46/mo |
| 100 users, 500 tasks/day | Over limit | ~$400/mo | $46-66/mo |
| Session time limit | 1 hour | 24 hours | Unlimited |
| Custom Chrome extension | No | No | Yes |
| Data sovereignty | No | No | Yes |
Break-even: ~10 active users. After that, self-hosted saves money every month while being more capable.
Risk Assessment
| Risk | How Likely | Impact | Mitigation |
|---|---|---|---|
| Chrome OOM in 2GB VM | Medium | VM crashes | Default 2GB, configurable to 3GB. Prometheus metrics track memory. |
| Firecracker kernel incompatibility | Low | VM won't boot | Test rootfs in QEMU first. Pre-built FC kernels support all needed syscalls. |
| SSH latency | Low | Slower operations | Acceptable (50-100ms). Can optimize with ControlMaster persistent connections or vsock. |
| Server down | Medium | All VMs lost | VMs are ephemeral by design. Backend falls back to Playwright. Auto-restart via systemd. |
| Rootfs copy time | Medium | Slower VM creation | Sparse copy (~5s for 4GB). Future: btrfs reflink for instant CoW. |
Files Changed
See the technical section above for complete file list. Total: 32 new files, 1 modified file.
All builds verified:
nx build sandbox-orchestrator— passes cleantsc --noEmiton backend-ai-api — passes clean (SelfHostedProvider changes are type-safe)