2026 03 05 Vitto Intelligence Upgrade

Documentation for Plans

Vitto Intelligence Upgrade — Session 181

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Fix three critical bugs (pageContext not sent, web prompt is second-class, screenshots see wrong page) and add intelligence features to make Vitto significantly smarter in the web chat context.

Architecture: The Vitto web chat has two paths: direct /ai-api/chat (fallback, includes contextEnrichment) and /vitto-api/engine/message (primary, full tool loop). The primary path is missing page context, uses a weaker system prompt, and its screenshot tool captures a server-side blank Playwright page instead of the user's actual screen.

Tech Stack: Angular 20+, NestJS, Socket.IO, Playwright, RxJS, Signals

Phase 1: Critical Fixes (Bugs)

Task 1: Wire pageContext into Vitto engine POST

Files:

Problem: sendMessageViaVitto() sends { message, spaceId } but omits pageContext. The backend controller accepts it, the engine uses it, but the frontend never sends it.

Step 1: Add pageContext to the POST body

In sendMessageViaVitto, line 337 currently reads:

{ message: content, spaceId: spaceId || undefined },

Change to:

{
  message: content,
  spaceId: spaceId || undefined,
  pageContext: this.contextService.systemPromptEnrichment() || undefined,
},

Step 2: Verify the serve process shows no TypeScript errors

Run: Check terminal running yarn serve:gograb-ai — should recompile with no errors.

Step 3: Commit

git add libs/libs-ai/vitto-chat/src/lib/services/vitto-chat-engine.service.ts
git commit -m "fix(vitto-chat): wire pageContext into Vitto engine POST body

The sendMessageViaVitto path was sending {message, spaceId} but omitting
pageContext. The backend controller and engine already accept and use it,
but the frontend never sent it. This caused Vitto to operate blind to the
user's current page when using the primary (tool-enabled) pipeline."

Task 2: Upgrade buildWebSystemPrompt to full intelligence parity

Files:

Problem: buildWebSystemPrompt is a stripped-down version that skips: semantic memory recall, learning intelligence, awareness hub, emotional arc, persona, channel session history, and conversation threading. The WhatsApp channel gets ALL of these. The web panel — where the user is actually looking at Vitto — gets none.

Step 1: Upgrade buildWebSystemPrompt signature

Change the method to accept userMessage so it can do semantic recall:

async buildWebSystemPrompt(params: WebPromptParams & { userMessage?: string }): Promise<string> {

Step 2: Add parallel intelligence loading (matching buildChannelSystemPrompt)

Replace the existing parallel fetch block (lines 324-327) with:

const { userId, userName, spaceId, pageContext, userMessage } = params;

const now = new Date();
const hour = now.getHours();
const awarenessContext: ActivationContext = {
  userId,
  sessionKey: `${userId}:${spaceId || 'default'}`,
  channel: 'panel' as ActivationContext['channel'],
  spaceId,
  activeTasks: [],
  pendingApprovals: [],
  timeOfDay: hour < 12 ? 'morning' : hour < 17 ? 'afternoon' : hour < 21 ? 'evening' : 'night',
  dayOfWeek: ['Sunday', 'Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday'][now.getDay()],
  hour,
};

const [userContextResult, recentHistory, semanticHistory, learningIntel, awarenessEvents] = await Promise.all([
  this.userKnowledge.getPromptContextForTask(userId, spaceId, {
    layer: 'platform_builder',
    complexity: 'moderate',
  }).catch(() => ({ context: '', budget: {} as Record<string, number>, debug: {} as Record<string, unknown> })),
  this.mind.memory.recallRecent(userId, 10, spaceId).catch(() => []),
  userMessage && userMessage.length > 10
    ? this.mind.memory.recall(userMessage, {
        userId,
        spaceId,
        limit: 8,
        minSimilarity: 0.65,
      }).catch(() => [])
    : Promise.resolve([]),
  this.aiClient.getLearningIntelligence(userMessage || ''),
  this.awarenessHub
    ? this.awarenessHub.activateForUser(userId, awarenessContext).catch(() => [])
    : Promise.resolve([]),
]);
const userContext = userContextResult.context;

Step 3: Add memory merging and deduplication (same as channel path)

const seenIds = new Set<string>();
const mergedHistory: typeof recentHistory = [];
for (const m of semanticHistory as any[]) {
  if (!seenIds.has(m.memoryId)) {
    seenIds.add(m.memoryId);
    mergedHistory.push(m);
  }
}
for (const m of recentHistory) {
  if (!seenIds.has(m.memoryId)) {
    seenIds.add(m.memoryId);
    mergedHistory.push(m);
  }
}

const CONTEXT_MEMORY_TYPES = ['conversation', 'user_preference', 'correction', 'learning'];
const conversationLines = mergedHistory
  .filter((m: any) => CONTEXT_MEMORY_TYPES.includes(m.type))
  .slice(-12)
  .map((m: any) => {
    if (m.type === 'conversation') return m.summary || '';
    if (m.type === 'user_preference') return `[Known] ${m.summary}`;
    if (m.type === 'correction') return `[Correction] ${m.summary}`;
    if (m.type === 'learning') return `[Insight] ${m.summary}`;
    return `[Learned] ${m.summary}`;
  })
  .filter(Boolean);

Step 4: Rebuild the system prompt with full intelligence sections

Replace the existing return block (lines 335-388) with the same sectioned structure as buildChannelSystemPrompt, but adapted for web:

const dayOfWeek = awarenessContext.dayOfWeek;
const timeStr = now.toLocaleTimeString('en-US', { hour: '2-digit', minute: '2-digit', hour12: true });

return [
  `You are Vitto. You work for ${userName}. You know them. You remember.`,
  `Platform: GoGrab web interface (authenticated).`,
  '',
  userContext ? `WHAT YOU KNOW ABOUT ${userName.toUpperCase()}:\n${userContext}` : '',
  conversationLines.length > 0
    ? `PAST INTERACTIONS:\n${conversationLines.join('\n')}`
    : '',
  this.buildLearningSection(learningIntel),
  this.awarenessHub ? this.awarenessHub.formatForSystemPrompt(awarenessEvents) : '',
  '',
  `TIME: ${dayOfWeek} ${timeStr}.`,
  '',
  'WHO YOU ARE:',
  `You are not a chatbot. You are not an assistant. You are an AI agent named Vitto who genuinely knows ${userName} and works alongside them. You have a persistent memory of every conversation. You have real capabilities — browser control, code execution, API access, file management, module generation. You can actually DO things, not just talk about them.`,
  '',
  'HOW YOU THINK:',
  `1. Before responding, consider: do I already know something relevant about ${userName}?`,
  `2. When ${userName} asks what you can do: SHOW them. Don't list capabilities — demonstrate one.`,
  `3. When ${userName} shares personal info: acknowledge it warmly, remember it, reference it later.`,
  '4. When you don\'t know something: say so honestly. Then offer to find out.',
  '5. When something seems like a continuation: pick up where you left off naturally.',
  '',
  'YOUR TOOLS (use them when appropriate):',
  '',
  'BROWSER & COMPUTER:',
  '- browser_navigate, browser_interact, browser_screenshot: Web automation on the SERVER-SIDE browser',
  '- computer_use: Autonomous goal-oriented browser control',
  '- capture_user_screen: Capture what the USER is currently seeing in their browser (use this to understand their current page)',
  '',
  'FILE SYSTEM (use freely — this is the dev machine):',
  '- read_file, write_file: Read/write files on the server',
  '- run_command: Run shell commands (nx, jest, git, ng, npm, yarn, mkdir, etc.)',
  '',
  'GOGRAB PLATFORM:',
  '- start_module_generation, check_module_generation: Generate GoGrab modules',
  '- http_request: Call GoGrab APIs (api/ai-api/docs-api backends)',
  '- query_platform, query_docs, search_docs, read_doc: Search platform docs/codebase',
  '',
  'KNOWLEDGE & MEMORY:',
  '- recall_memory, recall_vitto_memory: Search past conversations and facts',
  '- search_decisions: Search past MCP decisions and patterns',
  '',
  'CODE EXECUTION:',
  '- execute_code: Run code in E2B Firecracker VMs (JS, TS, Python, Bash, Go, Rust, Ruby, Java)',
  '',
  'DELEGATION:',
  '- create_handoff: Delegate tasks to specialized agents (browser, code)',
  '- task_complete: Signal done (REQUIRED when you\'ve used tools and are finished)',
  '',
  'BEHAVIOR:',
  '1. For greetings and simple chat: respond naturally without tools.',
  '2. When the user asks you to DO something: DO IT immediately using your tools.',
  '3. When asked about account, status, or data: use http_request to CHECK, don\'t guess.',
  '4. When asked to build something: use start_module_generation.',
  '5. If something fails, try a different approach.',
  '6. Reply in the same language as the user.',
  '7. When you\'ve used tools to complete a task, call task_complete with a summary.',
  '',
  'NEVER:',
  '- Start with "Hello! I\'m Vitto" or any self-introduction',
  '- Say "Great question!", "Absolutely!", "I\'d be happy to help!"',
  '- List your capabilities in bullet points',
  '- Start your response with "I"',
  '- Give a generic greeting when you have context about this person',
  '',
  'ALWAYS:',
  `- Reference what you know about ${userName} when relevant`,
  '- Pick up conversations naturally',
  '- Use your tools when they add value',
  '- Call task_complete with a natural summary when you\'ve used tools to finish a task',
  `- Reply in the same language as ${userName}`,
  '',
  pageContext ? `CURRENT PAGE CONTEXT:\n${pageContext}` : '',
  !userContext && conversationLines.length === 0
    ? `This appears to be your first interaction with ${userName}. Be welcoming.`
    : '',
].filter(Boolean).join('\n');

Step 5: Update the caller in vitto-engine.service.ts to pass userMessage

In runConversationalLoop (line 765), change:

const systemPrompt = await this.contextAssembly.buildWebSystemPrompt({ userId, userName, spaceId, pageContext });

To:

const systemPrompt = await this.contextAssembly.buildWebSystemPrompt({ userId, userName, spaceId, pageContext, userMessage: message });

Step 6: Verify via serve process, then commit

git add APIs/backend-vitto-api/src/app/engine/context-assembly.service.ts APIs/backend-vitto-api/src/app/engine/vitto-engine.service.ts
git commit -m "feat(vitto): upgrade web system prompt to full intelligence parity

buildWebSystemPrompt was a stripped-down version missing: semantic memory
recall, learning intelligence, awareness hub activation, memory dedup,
conversation/preference/correction/learning types, and identity rules.
The WhatsApp channel got full intelligence; the web panel got none.

Now both paths use the same intelligence stack: parallel context loading,
semantic recall, learning pipeline patterns, and awareness hub events."

Task 3: Add capture_user_screen tool via WebSocket

Files:

Problem: browser_screenshot uses server-side Playwright, which has no page loaded. Vitto needs to see what the USER is looking at. The LocalBridgeGateway already supports bidirectional tool execution — we extend it for screen capture.

Step 1: Add screen capture request to LocalBridgeGateway

Add after the executePageAction method in local-bridge.gateway.ts:

/**
 * Session 181: Request a screenshot from the user's browser via WebSocket.
 * The frontend captures the current viewport and returns base64 PNG.
 */
async captureUserScreen(
  userId: string,
): Promise<{ success: boolean; screenshot?: string; url?: string; title?: string; error?: string }> {
  const socketId = this.userSockets.get(userId);
  if (!socketId) {
    return { success: false, error: 'No frontend connection available for this user.' };
  }

  const requestId = `capture-${userId}:${Date.now()}-${Math.random().toString(36).slice(2, 7)}`;

  return new Promise((resolve) => {
    const timer = setTimeout(() => {
      this.pending.delete(requestId);
      resolve({ success: false, error: 'Screen capture timed out after 15s' });
    }, 15000);

    this.pending.set(requestId, {
      resolve: (res: BridgeToolResult) => {
        try {
          const data = JSON.parse(res.result);
          resolve({ success: res.success, screenshot: data.screenshot, url: data.url, title: data.title });
        } catch {
          resolve({ success: res.success, error: res.result });
        }
      },
      reject: () =>
        resolve({ success: false, error: 'Screen capture delivery failed' }),
      timer,
    });

    this.server.to(socketId).emit('capture_screen', { requestId });
  });
}

Step 2: Register capture_user_screen tool in ToolRegistryService

Add after the browser_screenshot registration (~line 651):

// Session 181: Capture what the USER is actually seeing (via WebSocket to their browser)
this.register({
  name: 'capture_user_screen',
  description: 'Capture a screenshot of what the user is currently seeing in their browser. This shows the ACTUAL page the user is looking at, unlike browser_screenshot which controls a separate server-side browser. Use this when the user asks "what do you see?" or references their current page.',
  parameters: {
    type: 'object',
    properties: {},
  },
  layers: ['developer', 'platform_builder', 'end_user'],
  execute: async (_input, context?: { userId?: string }) => {
    if (!this.localBridge) {
      return JSON.stringify({ error: 'Local bridge not available' });
    }
    const userId = context?.userId || 'unknown';
    try {
      const result = await this.localBridge.captureUserScreen(userId);
      if (result.success && result.screenshot) {
        return JSON.stringify({
          success: true,
          url: result.url,
          title: result.title,
          screenshotBase64: result.screenshot,
          note: 'This is what the user is currently seeing in their browser.',
        });
      }
      return JSON.stringify({ success: false, error: result.error || 'No screenshot returned' });
    } catch (err: any) {
      return JSON.stringify({ error: `Screen capture failed: ${err.message}` });
    }
  },
});

Step 3: Update ToolHandler interface to accept context

In tool-registry.service.ts, update the execute signature:

execute: (input: Record<string, unknown>, context?: { userId?: string; taskId?: string }) => Promise<string>;

And update executeTool to pass context through.

Step 4: Create frontend VittoScreenCaptureService

Create libs/libs-ai/vitto-core/src/lib/services/vitto-screen-capture.service.ts:

/**
 * VittoScreenCaptureService - Session 181
 * Captures the user's current viewport when requested by Vitto via WebSocket.
 * Uses html2canvas for reliable cross-browser screenshot capture.
 *
 * Fallback: If html2canvas is not available, captures a basic DOM snapshot description.
 */

import { Injectable, inject } from '@angular/core';
import { VITTO_API_URL } from '@libs-foundational/api-config';
import { io, Socket } from 'socket.io-client';

@Injectable({ providedIn: 'root' })
export class VittoScreenCaptureService {
  private readonly vittoApiUrl = inject(VITTO_API_URL, { optional: true });
  private socket: Socket | null = null;

  /**
   * Connect to the local-bridge WebSocket namespace to listen for capture requests.
   * Called once during app initialization when the user is authenticated.
   */
  connect(userId: string): void {
    if (!this.vittoApiUrl || this.socket) return;

    const wsUrl = this.vittoApiUrl.replace('/vitto-api', '');
    this.socket = io(`${wsUrl}/local-bridge`, {
      query: {
        userId,
        bridgeId: `browser-${Date.now()}`,
        capabilities: 'capture_screen,page_action',
      },
      transports: ['websocket'],
      reconnection: true,
      reconnectionDelay: 5000,
    });

    this.socket.on('capture_screen', async (request: { requestId: string }) => {
      try {
        const result = await this.captureViewport();
        this.socket?.emit('tool:result', {
          requestId: request.requestId,
          success: true,
          result: JSON.stringify(result),
          durationMs: 0,
        });
      } catch (err) {
        this.socket?.emit('tool:result', {
          requestId: request.requestId,
          success: false,
          result: JSON.stringify({ error: (err as Error).message }),
          durationMs: 0,
        });
      }
    });

    // Also handle page_action requests (Session 176 — reconnect existing feature)
    this.socket.on('page_action', async (request: { requestId: string; actionName: string; params: Record<string, unknown> }) => {
      // Delegate to VittoPageContextRegistry (injected lazily to avoid circular deps)
      this.socket?.emit('tool:result', {
        requestId: request.requestId,
        success: false,
        result: 'Page action handler not yet connected',
        durationMs: 0,
      });
    });
  }

  disconnect(): void {
    this.socket?.disconnect();
    this.socket = null;
  }

  /**
   * Capture the current viewport as a base64 PNG.
   * Uses canvas-based capture if available, falls back to DOM description.
   */
  private async captureViewport(): Promise<{ screenshot?: string; url: string; title: string; viewportSize: string }> {
    const url = window.location.href;
    const title = document.title;
    const viewportSize = `${window.innerWidth}x${window.innerHeight}`;

    try {
      // Try html2canvas (dynamically imported to keep bundle size small)
      const { default: html2canvas } = await import('html2canvas');
      const canvas = await html2canvas(document.body, {
        useCORS: true,
        allowTaint: true,
        scale: 0.5, // Half resolution to keep payload reasonable
        logging: false,
        width: window.innerWidth,
        height: window.innerHeight,
        windowWidth: window.innerWidth,
        windowHeight: window.innerHeight,
      });
      const screenshot = canvas.toDataURL('image/png', 0.7).split(',')[1]; // base64 only
      return { screenshot, url, title, viewportSize };
    } catch {
      // Fallback: return page metadata without screenshot
      return { url, title, viewportSize };
    }
  }
}

Step 5: Initialize the capture service in the chat engine

In vitto-chat-engine.service.ts, after the constructor or init block where auth is resolved, add:

// Session 181: Connect screen capture WebSocket for Vitto visual awareness
private readonly screenCapture = inject(VittoScreenCaptureService);

// In the auth resolution effect (around line 145):
if (authUser?.id) {
  this.screenCapture.connect(authUser.id);
}

Step 6: Add capture_user_screen to the webSafeTools list

In vitto-engine.service.ts line 767-771, add it:

const webSafeTools = [
  ...this.channelProcessing.getChannelSafeTools(),
  'read_file', 'write_file', 'run_command',
  'start_module_generation', 'check_module_generation',
  'capture_user_screen',
];

Step 7: Commit

git add -A
git commit -m "feat(vitto): add capture_user_screen tool via WebSocket bridge

browser_screenshot uses server-side Playwright (blank page, wrong context).
New capture_user_screen tool requests a screenshot from the user's actual
browser via the LocalBridge WebSocket. Frontend uses html2canvas to capture
the viewport and returns base64 PNG with URL and title metadata.

This gives Vitto actual visual awareness of what the user is looking at."

Phase 2: Intelligence Enhancements

Task 4: Add environment-aware tool selection

Files:

Problem: The webSafeTools list is static. It doesn't adapt based on what the user is doing. If the user is on the module builder page, start_module_generation should be prioritized. If they're on the chat page, memory tools matter more.

Step 1: Add tool prioritization based on pageContext

In context-assembly.service.ts, add a new method:

/**
 * Session 181: Recommend tool priority based on current page context.
 * Returns tools that should be emphasized for this context.
 */
getToolPriorityHint(pageContext?: string): string {
  if (!pageContext) return '';

  const lower = pageContext.toLowerCase();
  const hints: string[] = [];

  if (lower.includes('skill') || lower.includes('builder') || lower.includes('module')) {
    hints.push('TOOL PRIORITY: User is in a builder context. Prefer start_module_generation, query_platform, and read_file for understanding existing code.');
  }
  if (lower.includes('memory') || lower.includes('knowledge')) {
    hints.push('TOOL PRIORITY: User is viewing memory/knowledge. Prefer recall_vitto_memory, recall_memory, and search_decisions to demonstrate memory capabilities.');
  }
  if (lower.includes('observatory') || lower.includes('dashboard') || lower.includes('status')) {
    hints.push('TOOL PRIORITY: User is on a monitoring page. Prefer http_request to fetch live data and capture_user_screen to see what they see.');
  }
  if (lower.includes('chat') || lower.includes('landing')) {
    hints.push('TOOL PRIORITY: User is in general chat mode. Be conversational first. Use tools only when the request clearly requires action.');
  }
  if (lower.includes('form') || lower.includes('empty fields')) {
    hints.push('TOOL PRIORITY: User has a form open. You can help fill it. Check the available page actions.');
  }

  return hints.join('\n');
}

Step 2: Inject tool priority into the system prompt

In buildWebSystemPrompt, add before the CURRENT PAGE CONTEXT section:

pageContext ? this.getToolPriorityHint(pageContext) : '',

Step 3: Commit

git add APIs/backend-vitto-api/src/app/engine/context-assembly.service.ts
git commit -m "feat(vitto): add environment-aware tool prioritization

System prompt now includes tool priority hints based on the user's current
page context. Builder pages emphasize generation tools, memory pages
emphasize recall, dashboards emphasize data fetching."

Task 5: Add proactive context injection (what Vitto already knows)

Files:

Problem: When the user asks "What can you help me with on this page?", Vitto doesn't proactively offer context-specific help. It gives a generic capabilities list because it doesn't have enough signal about what's on the page.

Step 1: Add proactive assistance generation

Add a method to ContextAssemblyService:

/**
 * Session 181: Generate proactive assistance hints based on all available context.
 * This is injected at the END of the system prompt so it's closest to the user message.
 */
buildProactiveHints(params: {
  pageContext?: string;
  userContext?: string;
  conversationLines: string[];
  userName: string;
}): string {
  const hints: string[] = [];

  // If user has context, reference it
  if (params.userContext && params.conversationLines.length > 0) {
    hints.push(`You already know ${params.userName}. Reference what you know naturally.`);
  }

  // If there are past conversations, offer continuity
  if (params.conversationLines.length > 3) {
    hints.push('You have conversation history with this person. If their message could be a follow-up to a previous topic, connect them.');
  }

  // If page context mentions actions, be action-oriented
  if (params.pageContext?.includes('Available actions')) {
    hints.push('Page actions are available. When the user asks for help, OFFER to perform specific actions rather than just explaining what you can do.');
  }

  if (hints.length === 0) return '';
  return `\nPROACTIVE INTELLIGENCE:\n${hints.join('\n')}`;
}

Step 2: Inject into buildWebSystemPrompt (before the final join)

this.buildProactiveHints({ pageContext, userContext, conversationLines, userName }),

Step 3: Commit

git add APIs/backend-vitto-api/src/app/engine/context-assembly.service.ts
git commit -m "feat(vitto): add proactive intelligence hints to web prompt

System prompt now includes proactive hints: reference known user context,
offer conversation continuity, and suggest specific page actions instead
of generic capability lists."

Task 6: Pass userId context through tool execution pipeline

Files:

Problem: The capture_user_screen tool (and future user-scoped tools) needs userId to route the WebSocket request. Currently executeTool only receives the ToolCall — no execution context. The agent loop has the taskId and system prompt but doesn't pass user context to tools.

Step 1: Add execution context to tool pipeline

In agent-loop.service.ts, add userId to AgentLoopOptions:

export interface AgentLoopOptions {
  // ... existing fields
  /** Session 181: User ID for user-scoped tools (screen capture, page actions) */
  userId?: string;
}

Step 2: Thread userId through executeTool calls

In AgentLoopService.run(), when calling this.toolRegistry.executeTool(tc), change to:

this.toolRegistry.executeTool(tc, { userId: options.userId, taskId: options.taskId })

Step 3: Update executeTool to accept and pass context

In tool-registry.service.ts, update the executeTool method to accept a context parameter and pass it to the handler's execute function.

Step 4: Update startConversationalMessage to pass userId

In vitto-engine.service.ts, the agentLoop.run() call (line 773):

const result = await this.agentLoop.run({
  // ... existing
  userId,  // Session 181: Pass userId for user-scoped tools
}, streamSubject);

Step 5: Commit

git add APIs/backend-vitto-api/src/app/tools/tool-registry.service.ts APIs/backend-vitto-api/src/app/engine/agent-loop.service.ts APIs/backend-vitto-api/src/app/engine/vitto-engine.service.ts
git commit -m "feat(vitto): thread userId through tool execution pipeline

Tools like capture_user_screen need userId to route WebSocket requests.
AgentLoopOptions now accepts userId, which is passed through executeTool
to individual tool handlers via an execution context object."

Task 7: Improve browser_screenshot tool description and disambiguation

Files:

Problem: browser_screenshot description is misleading — it says "Take a screenshot of the current browser page" without clarifying it's a SERVER-SIDE browser. Vitto (the LLM) has no way to know the difference.

Step 1: Update browser_screenshot description

name: 'browser_screenshot',
description: 'Take a screenshot of the SERVER-SIDE headless browser (Playwright). This is NOT what the user sees — it is a separate browser controlled by your browser_navigate commands. You MUST call browser_navigate first to load a page. To see what the user is currently looking at, use capture_user_screen instead.',

Step 2: Commit

git add APIs/backend-vitto-api/src/app/tools/tool-registry.service.ts
git commit -m "fix(vitto): disambiguate browser_screenshot vs capture_user_screen

Updated browser_screenshot description to clarify it captures the
server-side Playwright browser, not the user's screen. Points to
capture_user_screen for user viewport capture."

Phase 3: Advanced Intelligence Features

Task 8: Add conversation genome hydration to web prompt

Files:

Problem: The web prompt uses raw recent memory, not the compressed conversation genome. For returning users with long history, this wastes context window on low-signal old memories instead of using the 10-30x compressed genome.

Step 1: Add genome hydration to the parallel fetch in buildWebSystemPrompt

Add to the Promise.all:

this.aiClient.getConversationGenome(/* sessionId needs to be passed */).catch(() => null),

Note: This requires passing sessionId through WebPromptParams. Add it:

export interface WebPromptParams {
  // ... existing
  /** Session 181: Chat session ID for genome hydration */
  sessionId?: string;
}

Step 2: If genome is available, use hydrated context instead of raw memory lines

if (genomeData?.summary) {
  // Use compressed genome instead of raw conversation lines
  parts.push(`CONVERSATION SUMMARY:\n${genomeData.summary}`);
  if (genomeData.topics?.length) {
    parts.push(`Topics discussed: ${genomeData.topics.join(', ')}`);
  }
  if (genomeData.decisions?.length) {
    parts.push(`Decisions made: ${genomeData.decisions.map(d => d.description).join('; ')}`);
  }
} else {
  // Fall back to raw memory lines
  // ... existing conversationLines code
}

Step 3: Commit

git add APIs/backend-vitto-api/src/app/engine/context-assembly.service.ts APIs/backend-vitto-api/src/app/engine/vitto-engine.service.ts
git commit -m "feat(vitto): add conversation genome hydration to web prompt

For returning users with long chat history, the web prompt now uses
compressed conversation genome (10-30x smaller) instead of raw memory
lines. Falls back to raw memory when genome is unavailable."

Task 9: Add smart tool suggestion in system prompt based on user message classification

Files:

Problem: Vitto sometimes uses browser_screenshot when it should use capture_user_screen, or calls recall_memory when the answer is in the page context. The LLM needs clearer guidance about WHICH tool to use WHEN.

Step 1: Add intent-to-tool mapping

/**
 * Session 181: Classify user intent and suggest optimal tool strategy.
 */
classifyIntentForTools(userMessage: string): string {
  const lower = userMessage.toLowerCase();

  if (/what.*see|look.*page|current.*screen|what.*page|this page/i.test(lower)) {
    return 'INTENT: User is asking about their current screen. Use capture_user_screen, NOT browser_screenshot.';
  }
  if (/remember|last time|we talked|our conversation|you said/i.test(lower)) {
    return 'INTENT: User is asking about past conversations. Use recall_vitto_memory first.';
  }
  if (/build|create|generate|make.*module/i.test(lower)) {
    return 'INTENT: User wants to build something. Use start_module_generation.';
  }
  if (/status|health|check|how.*many|metrics/i.test(lower)) {
    return 'INTENT: User wants live data. Use http_request to the appropriate backend.';
  }
  if (/navigate|go to|open|browse/i.test(lower)) {
    return 'INTENT: User wants to navigate. Use browser_navigate (server-side) or describe where to go.';
  }
  return '';
}

Step 2: Inject into buildWebSystemPrompt

Add after the BEHAVIOR section:

userMessage ? this.classifyIntentForTools(userMessage) : '',

Step 3: Commit

git add APIs/backend-vitto-api/src/app/engine/context-assembly.service.ts
git commit -m "feat(vitto): add intent-based tool routing in system prompt

Classifies user intent (screen viewing, memory recall, building, data
checking) and injects tool routing guidance to prevent common mistakes
like using browser_screenshot instead of capture_user_screen."

Summary of Changes

# Type Impact Files
1 Bug fix Vitto now knows what page user is on vitto-chat-engine.service.ts
2 Bug fix Web prompt gets full intelligence stack context-assembly.service.ts, vitto-engine.service.ts
3 Feature Real screenshots of user's browser local-bridge.gateway.ts, tool-registry.service.ts, NEW: vitto-screen-capture.service.ts
4 Feature Context-aware tool prioritization context-assembly.service.ts
5 Feature Proactive assistance hints context-assembly.service.ts
6 Plumbing userId flows through tool pipeline agent-loop.service.ts, tool-registry.service.ts
7 Bug fix Clear tool descriptions prevent misuse tool-registry.service.ts
8 Feature Genome-compressed history in prompts context-assembly.service.ts
9 Feature Intent-based tool routing context-assembly.service.ts

Dependencies

Optimal execution order

Parallel batch 1: Tasks 1, 6, 7 Parallel batch 2: Tasks 2, 3 Parallel batch 3: Tasks 4, 5, 8, 9

Verification

After all tasks:

  1. Run yarn serve:gograb-ai — no TypeScript errors
  2. Run yarn serve:backend-ai — no startup errors
  3. Run yarn serve:backend-vitto — no startup errors
  4. Open Vitto chat, send "What can you help me with on this page?" — should get context-specific answer with page awareness
  5. Open Vitto chat, send "What does my screen look like?" — should trigger capture_user_screen (verify WebSocket connection)