MULTIMODAL VISION PIPELINE

Ecosistema completo para organización y gestión de contenido con interfaces espaciales

Multimodal Vision Pipeline

Session 198.1 | March 10, 2026

Overview

Vitto can now see and understand images sent by users across all chat channels. When a user sends a photo with a question (e.g., "Explain this" + a math problem image), Vitto analyzes the image and responds with context-aware understanding.

Supported Channels

Channel	Inbound Images	Outbound Images	Status
WhatsApp	Download from Twilio → R2	Via `sendWhatsAppMedia()`	Live
Telegram	Download via Bot API → R2	Via `sendPhoto()` (planned)	Live
Web Panel	Upload via `POST /engine/media/upload` → R2	Display in chat bubble	Live
Group Chats	Inherits from channel	Inherits from channel	Live

Data Flow

┌─────────────┐    ┌──────────────────┐    ┌─────────────┐    ┌──────────────┐
│   Channel    │───>│  Channel Gateway  │───>│  R2 Storage  │───>│  Signed URL  │
│ (WhatsApp/   │    │ processInbound    │    │  upload      │    │  (7-day)     │
│  Telegram/   │    │ Media()           │    │              │    │              │
│  Web Panel)  │    └──────────────────┘    └─────────────┘    └──────┬───────┘
└─────────────┘                                                       │
                                                                      ▼
┌─────────────┐    ┌──────────────────┐    ┌─────────────┐    ┌──────────────┐
│   Claude /   │<───│  LLM Provider     │<───│  Agent Loop  │<───│  Channel     │
│   OpenAI /   │    │ toAnthropicContent│    │ userContent  │    │  Processing  │
│   Gemini     │    │ Blocks()          │    │ merge w/text │    │ buildMulti   │
│              │    │                    │    │              │    │ modalContent │
└──────┬──────┘    └──────────────────┘    └─────────────┘    └──────────────┘
       │
       ▼
  AI sees the image
  and responds with
  visual understanding

Pipeline Stages

Stage 1: Media Extraction (channel-gateway.service.ts)

WhatsApp: Twilio webhook includes NumMedia, MediaUrl0, MediaContentType0. The processInboundMedia() method downloads from Twilio (with Basic auth) and uploads to R2.

Telegram: Webhook includes message.photo[] array (multiple resolutions). Takes largest resolution, calls getFile API to get download URL, uploads to R2.

Web Panel: Frontend uploads via POST /engine/media/upload (multipart form). Backend stores in R2, returns signed URL.

Stage 2: Attachment Threading (vitto.controller.ts → channel-processing.service.ts)

Attachments are added to TaskContext.attachments in the controller and passed through the engine to channel-processing.service.ts.

Stage 3: Content Block Construction (channel-processing.service.ts)

buildMultimodalContent() converts attachments to LLMContentBlock[]:

Image attachments → { type: 'image', imageUrl: r2SignedUrl, mimeType }
Text message → { type: 'text', text: userMessage }
Images placed before text (LLMs process visual context better this way)

Stage 4: Injection Safety (agent-loop.service.ts)

mergeTextWithContentBlocks() replaces the text block with the injection-scanned version. Images bypass text injection scanning (correct — images aren't text-injectable). The security model is preserved.

Stage 5: Provider Conversion (claude/openai/gemini.provider.ts)

Each provider converts LLMContentBlock[] to its native vision format:

Provider	Format	Status
Claude	`{ type: 'image', source: { type: 'url', url } }`	Full support
OpenAI	`{ type: 'image_url', image_url: { url, detail: 'auto' } }`	Full support
Gemini	Text fallback: `[Image attached: mime — url]`	Partial (needs base64 Phase 2)

The Bug That Was Fixed

All 3 LLM providers had the same bug since Session 98:

// BEFORE (broken): JSON.stringify'd the content blocks
content: typeof m.content === 'string' ? m.content : JSON.stringify(m.content)

// AFTER (fixed): Convert to native vision format
content: typeof m.content === 'string'
  ? m.content
  : this.toAnthropicContentBlocks(m.content as LLMContentBlock[])

The LLMContentBlock type already supported type: 'image' — it was designed in Session 98 but never wired. The JSON.stringify turned structured image data into a useless string that the LLM couldn't interpret.

Type Changes

Fields added across all message layers:

// Backend task context
TaskContext.attachments?: Array<{ type: string; url: string; name?: string; r2Key?: string }>;

// Agent loop options
AgentLoopOptions.userContent?: LLMContentBlock[];

// Frontend chat message
ChatMessage.attachments?: Array<{ type: string; url: string; name?: string }>;

// Backend session schema
IntelligentMessageDoc.attachments?: Array<{ type: string; url: string; name?: string; r2Key?: string }>;

// Sync message
SyncMessage.attachments?: Array<{ type: string; url: string; name?: string }>;

Error Handling

R2 unavailable: Media processing catches errors, keeps original channel URLs as fallback
Twilio download fails: 3 retries with exponential backoff in uploadFromUrl()
Telegram getFile fails: Logs warning, message proceeds without image context
Image-only messages: Supported — Telegram handler allows photo-only (no text required)
Large files: 10MB limit enforced via multer on web upload endpoint

Related Files

File	Changes
`APIs/backend-vitto-api/src/app/llm/providers/claude.provider.ts`	`toAnthropicContentBlocks()`
`APIs/backend-vitto-api/src/app/llm/providers/openai.provider.ts`	`toOpenAIContentParts()`
`APIs/backend-vitto-api/src/app/llm/providers/gemini.provider.ts`	`toGeminiParts()`
`APIs/backend-vitto-api/src/app/connectivity/channel-gateway.service.ts`	`processInboundMedia()`, `processTelegramInboundPhoto()`
`APIs/backend-vitto-api/src/app/engine/channel-processing.service.ts`	`buildMultimodalContent()`
`APIs/backend-vitto-api/src/app/engine/agent-loop.service.ts`	`userContent`, `mergeTextWithContentBlocks()`
`APIs/backend-vitto-api/src/app/engine/vitto-engine.service.ts`	Web chat multimodal path
`APIs/backend-vitto-api/src/app/engine/vitto.controller.ts`	`POST /engine/media/upload`, attachments pass-through

Future Work

Gemini vision: Add base64 pre-fetch step for inlineData format
Chat bubble rendering: Display image thumbnails in web panel chat messages
Image-in-response: When LLM generates/references images, attach to outbound channel response
Video/audio support: Extend LLMContentBlock for audio transcription and video frames
Content moderation: Scan uploaded images before storing in R2 (for gograb-space-production)