Architecture: Session 181: How to give Vitto visual awareness of the user'...

Documentation for Sessions

🏗️ Architecture: Session 181: How to give Vitto visual awareness of the user'...

Date: 2026-03-05
Time: 21:53:33 UTC
Session: gspace1-180.3
Agent: backend
Confidence: 85%
Status: ✅ Documented

📋 Decision Context

Question:

Session 181: How to give Vitto visual awareness of the user's actual browser?

Task: Add user screen capture for Vitto visual awareness

🎯 Decision Made

Chosen:

New capture_user_screen tool: backend sends WebSocket event to frontend via LocalBridgeGateway, frontend VittoScreenCaptureService captures viewport (html2canvas + visible text extraction + page context), returns via tool:result event

Options Considered:

html2canvas via WebSocket (chosen)
getDisplayMedia API (requires user permission popup)
Remove screenshot entirely, rely on pageContext text only

💭 Reasoning

browser_screenshot uses server-side Playwright which has no page loaded (blank screenshots). The LocalBridge WebSocket already existed for tool execution. Extended it with capture_screen event. Frontend dynamically imports html2canvas (optional) and falls back to structured text extraction (headings, buttons, form fields, page context enrichment). The text fallback is often MORE useful to the LLM than a screenshot image.

📁 Files Involved

APIs/backend-vitto-api/src/app/engine/local-bridge.gateway.ts
libs/libs-ai/vitto-core/src/lib/services/vitto-screen-capture.service.ts
APIs/backend-vitto-api/src/app/tools/tool-registry.service.ts

🔗 Related

Decision ID: dev_1772747613186_qhomesm
Session: 180
Workspace: gspace1

Auto-generated by MCP Smart Consolidation