🐛 Bugfix: Why couldn't Vitto see images sent by users in WhatsApp, Tel...

Date: 2026-03-10
Time: 17:58:56 UTC
Session: gspace1-198.4
Agent: backend
Confidence: 97%
Status: ✅ Documented

📋 Decision Context

Question:

Why couldn't Vitto see images sent by users in WhatsApp, Telegram, or web chat?

Task: S198.1: Multimodal vision + Cloudflare R2 storage

🎯 Decision Made

Chosen:

Fixed 5 pipeline stages where images were dropped: (1) normalizeInbound() ignored mediaUrl0/photo fields, (2) VittoTask had no attachments field, (3) channel-processing didn't convert attachments to LLMContentBlock[], (4) agent-loop only accepted string content, (5) all 3 LLM providers JSON.stringify'd content blocks instead of converting to native vision format. The LLM type system (LLMContentBlock with type:'image') already existed — it was just never wired.

Options Considered:

1. Fix providers only (no R2, use Twilio/Telegram URLs directly)
2. Full pipeline fix with R2 storage
3. Add vision as a separate tool call instead of inline content

💭 Reasoning

Root cause was incremental: vision types were designed in Session 98 but never connected because no storage layer existed for media. WhatsApp handler (Session 136) extracted mediaUrl0 into metadata but it never reached the LLM. The critical bug was in all 3 providers — claude.provider.ts line 142, openai.provider.ts line 215, gemini.provider.ts line 173 — all doing JSON.stringify(m.content) when content was LLMContentBlock[], converting structured image blocks into a useless JSON string. Fix: convert to each provider's native format (Anthropic url source, OpenAI image_url, Gemini inlineData/text fallback).

📁 Files Involved

• APIs/backend-vitto-api/src/app/llm/providers/claude.provider.ts
• APIs/backend-vitto-api/src/app/llm/providers/openai.provider.ts
• APIs/backend-vitto-api/src/app/llm/providers/gemini.provider.ts
• APIs/backend-vitto-api/src/app/connectivity/channel-gateway.service.ts
• APIs/backend-vitto-api/src/app/engine/channel-processing.service.ts
• APIs/backend-vitto-api/src/app/engine/agent-loop.service.ts

🔗 Related

• Decision ID: dev_1773165536664_6eel50e
• Session: 198
• Workspace: gspace1

Auto-generated by MCP Smart Consolidation