📝 Milestone: Computer Use: End-to-end Gmail task completed — what was bui...

Date: 2026-02-24
Time: 18:54:21 UTC
Session: gspace1-139.2
Agent: angular
Confidence: 95%
Status: ✅ Documented

📋 Decision Context

Question:

Computer Use: End-to-end Gmail task completed — what was built and what are the key architectural decisions?

Task: Computer Use full stack implementation

🎯 Decision Made

Chosen:

Full Computer Use stack: E2B Desktop (primary) → Playwright (fallback) → @jitsi/robotjs (desktop). Service profiles, credentials, mid-session reply, Chrome enforcement, dynamic tool versioning.

Options Considered:

• No alternatives documented

💭 Reasoning

Session 139 built Computer Use from broken (sessions never appeared) to fully functional (23-step Gmail login+compose+send). Key decisions: (1) E2B Desktop as primary — each session gets a disposable Firecracker VM, eliminating Playwright page/context leak issues. (2) Dynamic Anthropic tool versioning — model ID determines computer_20251124 vs computer_20250124 + matching beta flag. (3) Service profiles — static registry of 9 services (Gmail, Calendar, GitHub, etc.) with systemPromptHint that helps Claude navigate each service. (4) Mid-session reply — polling-based user message injection for 2FA/clarification. (5) Chrome enforcement — xdg-settings + pkill firefox at sandbox creation + explicit google-chrome launch. (6) @jitsi/robotjs for desktop mode replacing unmaintained cliclick. (7) Session-scoped agentId for Playwright context isolation. (8) Non-blocking executeGoal with completedSessions map for UI polling.

📁 Files Involved

• apis/backend-ai-api/src/app/infrastructure/services/computer-use-loop.service.ts
• apis/backend-ai-api/src/app/infrastructure/services/e2b-desktop.service.ts
• apis/backend-ai-api/src/app/infrastructure/services/browser-automation.service.ts
• apis/backend-ai-api/src/app/infrastructure/services/desktop.service.ts
• apis/backend-ai-api/src/app/infrastructure/services/service-profiles.ts
• apis/backend-ai-api/src/app/infrastructure/controllers/computer-use.controller.ts
• libs/libs-ai/vitto-hub/src/lib/components/computer-use-panel/computer-use-panel.component.ts
• libs/libs-ai/vitto-hub/src/lib/services/vitto-api.service.ts
• libs/libs-api/backend-core/src/lib/config/ai-models.config.ts

🔗 Related

• Decision ID: dev_1771959261175_u92nrai
• Session: 139
• Workspace: gspace1

Auto-generated by MCP Smart Consolidation