Vitto Jarvis Evolution — Technical Specification

All claims in this spec are code-verified. Every file path, line number, and root cause was traced by dedicated code exploration agents on 2026-03-17. No assumptions are presented as facts.

Design Principles

1. Ship what exists before building new. The Tauri app, local bridge, and SSE infrastructure are built. Wire them up.
2. Fix broken foundations first. Embedding search, cost tracking, and health checks are bugs, not features. Fix before adding capability.
3. Cloud-first, local-supplement. Vitto's brain runs in the cloud. The local daemon is a peripheral that extends reach to the user's machine.
4. No credential handling by agents. OAuth delegation or human takeover. Agent never sees raw passwords.
5. Measure before optimizing. Observability and evals before performance work.

Phase 1: Fix Foundations + Ship Desktop Daemon

Timeline: 1-2 weeks | Sessions: 4-5 | Risk: Low (verified fixes, existing code)

1.1 Wire Embedding Search

Root cause (verified): LearningService.setEmbeddingClient() (learning.service.ts:125) is never called. embeddingEnabled is always false (line 108). All findSimilar calls fall through to keywordBasedSimilarSearch (line 566).

Meanwhile: DecisionService.generateAndStoreEmbedding() (decision.service.ts:164-248) generates OpenAI text-embedding-3-small embeddings and stores them in ai-memory.agent_embeddings on every decision. DecisionSearchService.vectorSearch() (decision-search.service.ts:56-132) has a complete $vectorSearch pipeline. These two systems exist but are disconnected from LearningService.

The fix (4 files, ~50 lines):

File 1 — infrastructure.module.ts: Add DecisionEmbeddingService to exports array (currently only in providers at line 304).

// infrastructure.module.ts exports array (~line 307)
exports: [
  // ... existing exports ...
  DecisionEmbeddingService, // ADD THIS
]

File 2 — learning.module.ts: Add bidirectional forwardRef import for InfrastructureModule. NestJS handles bidirectional forwardRef correctly.

imports: [
  // ... existing imports ...
  forwardRef(() => InfrastructureModule), // ADD THIS
]

File 3 — Create learning/services/learning-embedding-adapter.service.ts:

@Injectable()
export class LearningEmbeddingAdapterService implements EmbeddingClient {
  constructor(
    private readonly embeddingService: DecisionEmbeddingService,
    @InjectConnection(AI_CONNECTION_NAMES.MEMORY)
    private readonly memoryConnection: Connection,
  ) {}

  async generateEmbedding(text: string): Promise<number[]> {
    return this.embeddingService.generateEmbedding(text);
  }

  async findSimilar(query: string, options?: { limit?: number; minSimilarity?: number }): Promise<SimilarDecisionResult[]> {
    const queryEmbedding = await this.generateEmbedding(query);
    const collection = this.memoryConnection.collection('agent_embeddings');
    // Replicate DecisionSearchService.vectorSearch() $vectorSearch aggregation
    const results = await collection.aggregate([
      {
        $vectorSearch: {
          index: 'vector_index',
          path: 'embedding',
          queryVector: queryEmbedding,
          numCandidates: (options?.limit || 5) * 10,
          limit: options?.limit || 5,
        }
      },
      { $addFields: { score: { $meta: 'vectorSearchScore' } } },
      { $match: { score: { $gte: options?.minSimilarity || 0.7 } } },
    ]).toArray();

    return results.map(r => ({
      decision: r.metadata,
      similarity: r.score,
    }));
  }
}

File 4 — learning.service.ts: Wire adapter in onModuleInit():

// Constructor: add @Optional() private readonly embeddingAdapter?: LearningEmbeddingAdapterService

// onModuleInit() (~line 138):
if (this.embeddingAdapter) {
  this.setEmbeddingClient(this.embeddingAdapter);
  this.logger.log('Embedding search enabled via LearningEmbeddingAdapterService');
}

Prerequisite: MongoDB Atlas vector_index must exist on ai-memory.agent_embeddings.embedding field. Verify with db.agent_embeddings.getIndexes(). If missing, create:

db.agent_embeddings.createIndex(
  { embedding: "vectorSearch" },
  { name: "vector_index", type: "vectorSearch", dimensions: 1536, similarity: "cosine" }
)

Verification: After deploying, call GET /ai-api/learning/similar?query=state+management and confirm results include semantically-related decisions (not just keyword matches). Compare result quality before/after.

1.2 Fix Cost Tracker (5 Bugs)

Root cause (verified): Five stacked bugs make cost tracking completely non-functional.

Bug 1 (Primary) — Frontend always shows $0.00

File: vitto.controller.ts:356-360 Problem: Controller returns usage: { promptTokens, completionTokens, totalTokens } but no cost field. Frontend reads response.usage?.cost which is always undefined.

Fix:

// vitto.controller.ts ~line 356
usage: task.cost ? {
  promptTokens: task.cost.inputTokens || 0,
  completionTokens: task.cost.outputTokens || 0,
  totalTokens: task.cost.totalTokens || 0,
  cost: task.cost.costUsd || 0, // ADD THIS
} : undefined,

And in phase-executor.service.ts, after recordSpending() completes, write the cost back to the task:

// After recordSpending() call (~line 248)
const spendingResult = await this.costTracker.recordSpending({ ... });
task.cost.costUsd = spendingResult?.costUsd || 0; // ADD THIS

Bug 2 — Spending history lost on restart

File: cost-tracker.service.ts:104-123 Problem: onModuleInit() loads userBudgets from MongoDB but NOT spendingHistory. After restart, getSpendingStats() returns empty.

Fix: In onModuleInit(), query SpendingRecordModel for the last 7 days and populate spendingHistory:

// onModuleInit() — ADD after userBudgets loading
const recentSpending = await this.spendingRecordModel.find({
  timestamp: { $gte: new Date(Date.now() - 7 * 24 * 60 * 60 * 1000) }
}).lean();
this.spendingHistory.push(...recentSpending);

Bug 3 — Static provider/model on task.cost

File: vitto-engine.service.ts:191-198 Problem: task.cost.provider and task.cost.model are set from static config at task creation and never updated after SmartRouter routing.

Fix: In agent-loop.service.ts, after each LLM call, update task.cost.provider and task.cost.model with the actual provider/model used:

// After SmartRouter response
task.cost.provider = actualProvider;
task.cost.model = actualModel;

Bug 4 — Skill generation never tracked

File: phase-executor.service.ts:394-430 Problem: handleTeachIntent() hardcodes tokens: { input: 0, output: 0 } and never calls recordSpending().

Fix: Add accumulateCost(task) + recordSpending() at the end of handleTeachIntent().

Bug 5 — getSpendingStats() memory-only

File: cost-tracker.service.ts:250-310 Problem: Only reads in-memory array. Fixed by Bug 2's rehydration fix.

Verification: After fix, send a chat message via web UI. Check that the cost displays a non-zero value (e.g., $0.0045 for a Haiku response). Check daily digest after restart — should show yesterday's spending.

1.3 Health Check MongoDB Ping

File: health.controller.ts (backend-ai-api) — MongoDB latency check is a stub.

Fix: Replace stub with actual ping:

const start = Date.now();
await this.connection.db.admin().ping();
const latencyMs = Date.now() - start;

Effort: 10 minutes.

1.4 Ship Tauri Desktop App as Daemon

Current state (verified):

• Tauri v2 app at apps/vitto-desktop/ is buildable (needs nx build gograb-ai first)
• Wraps the gograb-ai Angular app (same app deployed to ai.gograb.space)
• tauri_plugin_autostart with MacosLauncher::LaunchAgent already registered
• Close-to-tray works (api.prevent_close() + window.hide())
• Exit prevention works (api.prevent_exit())
• 13 native commands in local_exec.rs (shell, files, screenshot, AppleScript, clipboard, system info)
• System tray with 3 states (idle/active/error), global hotkey Cmd+Ctrl+V
• Signed for macOS: Developer ID Application: GoGrab LLC (SFG3Z4GLEB)
• Auto-updater pointed at vitto-updater.gograb.space
• Release pipeline in GitHub Actions builds 4 platforms

Changes needed for daemon mode:

1.4a Start hidden (1 line)

// tauri.conf.json → windows[0]
"visible": false  // was true

The tray left-click and "Open Vitto" menu item already call window.show() + window.set_focus().

1.4b Infinite reconnection in Angular bridge

File: vitto-screen-capture.service.ts:74

// Change from:
reconnectionAttempts: 10
// To:
reconnectionAttempts: Infinity

The Node.js bridge already retries infinitely — match that behavior.

1.4c Heartbeat loop

Add a heartbeat interval in lib.rs (Rust side) or as an Angular service in the gograb-ai app. Every N minutes (configurable, default 30):

1. Check if user is idle (no mouse/keyboard activity for >5 min — CGEventSourceSecondsSinceLastEventType on macOS)
2. If idle, call POST /vitto-api/engine/message with system prompt: "Review pending tasks, calendar events, and unread messages for {userId}. Report anything that needs attention."
3. If Vitto finds something actionable, show a native Tauri notification (tauri-plugin-notification already installed)
4. If nothing, log HEARTBEAT_OK and sleep

Implementation choice: Angular service (not Rust). Reason: the Angular app already has the VittoChatEngineService with SSE support, auth tokens, and user context. Adding a VittoHeartbeatService in libs/libs-ai/vitto-core/ keeps the logic in TypeScript where we're faster.

1.4d Build and test

nx build gograb-ai --configuration=production
cd apps/vitto-desktop && cargo tauri build

Verification: Install the .dmg. Confirm: starts hidden in tray, Cmd+Ctrl+V shows window, closing hides to tray, process survives logout (LaunchAgent), reconnects after backend restart, heartbeat fires after idle period.

1.5 Chat Proxy Polling Fix (Low Priority)

Verified finding: The web UI already uses SSE (POST /engine/message → GET /engine/message/:taskId/stream). The 200ms polling loop only affects POST /engine/chat, which is called exclusively by backend-ai-api's ChatService.proxyToVitto() when CHAT_USE_VITTO_PROXY=true.

Fix (if needed): Replace the while loop in vitto.controller.ts:338-343 with:

const { taskId } = await this.engine.startConversationalMessage(params);
const stream = this.engine.getConversationalStream(taskId);
const finalEvent = await lastValueFrom(
  stream.pipe(filter(e => e.type === 'task.completed' || e.type === 'task.failed'))
);
return { response: finalEvent.data.response, usage: ... };

This removes CPU-wasting polling while preserving the synchronous HTTP contract that proxyToVitto() expects.

Severity: Low. The web UI is unaffected. Only fix if CHAT_USE_VITTO_PROXY is enabled in production.

Phase 2: Persistent Desktop Agent

Timeline: 2-3 weeks | Sessions: 5-6 | Risk: Medium (new integrations, but existing infrastructure)

2.1 VM Snapshot Auth Persistence

Goal: When a user logs into Gmail via CU, that login survives across sessions.

Architecture:

User requests "check my Gmail"
  → SmartRouter selects CU task
  → SandboxProviderFactory checks: does user have a snapshot?
    → YES: restoreFromSnapshot() (<500ms) → Chrome already logged in
    → NO: createSession() (cold boot) → user must log in → snapshotAndSuspend() on completion

Changes needed:

1. Sandbox orchestrator — Ensure Chrome user data directory (~/.config/google-chrome/) is on the persistent rootfs, not tmpfs. Currently the rootfs is copied as sparse file per session. Chrome profile data must survive snapshot/restore.

2. Service profiles — Add authState metadata tracking:

interface ServiceProfileAuthState {
  profileId: string; // e.g., 'gmail'
  lastAuthenticated: Date;
  snapshotId: string;
  cookieExpiry?: Date;
  userId: string;
}

Store in vitto-mind.service_auth_states collection.

3. CU completion hook — After successful task on an authenticated service, call snapshotAndSuspend() and store ServiceProfileAuthState.

4. CU startup — Before cold boot, check if user has a valid snapshot for the requested service profile. If yes, restore. If snapshot is >7 days old or cookie expired, cold boot.

Risk: Chrome session cookies may expire before the 7-day snapshot TTL. Mitigation: store cookieExpiry in auth state and validate before restore.

2.2 CU Action Memory

Goal: "Last time I booked a flight on Google Flights, I clicked X then Y."

Changes:

1. After CU loop completes successfully, extract an "action recipe" using Haiku:
   • Input: session.steps[] (sequence of screenshot descriptions + actions taken)
   • Output: structured recipe { service, goal, steps: [{ description, action, selector }] }
2. Store as VittoMemory with type task_execution via VittoMemoryService.remember().
3. On next CU task for the same service, recall matching memories and inject into the CU system prompt as prior art.

File: Wire in ComputerUseLoopService completion handler (currently only updates session.status and calls auditService.logAction()).

2.3 OAuth Credential Delegation

Goal: Vitto can call Gmail API, Google Calendar API, GitHub API, etc. without handling raw credentials.

Recommended integration: Composio AgentAuth (500+ services, PKCE for headless agents, automatic token refresh) or Nango (self-hostable, 700+ APIs).

Architecture:

User says "send an email to X"
  → Vitto identifies tool: gmail.sendEmail
  → Check Composio: does user have a connected Gmail account?
    → YES: execute via Composio API (agent never sees token)
    → NO: redirect user to OAuth consent flow → Composio stores token → retry

Integration point: New tool plugin oauthPlugin in APIs/backend-vitto-api/src/app/tools/plugins/.

Security: Agent receives a connected_account_id reference. Composio executes the API call server-side. Raw tokens stay in Composio's vault, never transit through Vitto.

2.4 Human Takeover UX

Goal: When CU hits a login form that can't be handled by OAuth or snapshot, the user takes over.

Architecture:

CU loop hits login form
  → Claude's response includes "I need credentials to continue"
  → ComputerUseSupervisorService detects credential_needed state
  → Send notification to user (Tauri notification, WhatsApp, or web push)
  → Stream VNC feed to user's browser/app
  → User types credentials directly into VNC
  → CU loop resumes after user signals "done"
  → Snapshot VM to preserve auth state

Existing infrastructure: VNC proxy already exists at sandbox-orchestrator/streaming/vnc-proxy.service.ts. The CU loop already has a 5-minute question wait for human reply. The gap is the notification + VNC streaming to the frontend.

2.5 Proactive Heartbeat

Covered in Phase 1 (1.4c). Extending in Phase 2:

• Add calendar integration (if OAuth connected): "You have a meeting in 30 minutes with X"
• Add email monitoring: "3 unread emails from your boss"
• Add task follow-up: "The CU task you started 2 hours ago completed successfully"

Phase 3: Jarvis Parity

Timeline: 4-6 weeks | Sessions: 8-10 | Risk: Medium-High (new systems)

3.1 Voice Pipeline

Architecture:

User speaks into Tauri app (push-to-talk via Cmd+Ctrl+V)
  → macOS AVAudioEngine captures audio → PCM buffer
  → Whisper API (OpenAI) or Deepgram → text transcript
  → POST /engine/message with transcript
  → SSE response stream → ElevenLabs TTS → audio playback

Phase 3a (MVP): Push-to-talk only. No wake word. Browser SpeechSynthesis for TTS (zero cost). Phase 3b: Always-listening with wake word ("Hey Vitto"). ElevenLabs for natural TTS.

Tauri side: Add tauri-plugin-audio or use Rust cpal crate for audio capture. The existing global hotkey (Cmd+Ctrl+V) toggles recording on/off.

Backend side: New endpoint POST /vitto-api/engine/voice that accepts multipart/form-data with audio file, transcribes via Whisper, and delegates to startConversationalMessage().

3.2 Browser Extension Relay

Architecture:

Chrome Extension (Manifest V3)
  ├── Background Service Worker
  │   ├── Connects to LocalBridgeGateway WebSocket (ws://localhost:3003/local-bridge)
  │   ├── Registers capability: 'browser_relay'
  │   └── Handles tool:execute events for browser_* tools
  ├── Content Script (optional)
  │   └── Extracts page context, accessibility tree
  └── Popup
      └── Shows connection status, active tabs, permissions

Key behaviors:

1. On connection, enumerate open tabs and report to gateway
2. On browser_relay_navigate tool: attach Chrome Debugger to target tab → execute CDP command
3. On browser_relay_screenshot tool: chrome.debugger.sendCommand('Page.captureScreenshot')
4. On browser_relay_click tool: use CDP Input.dispatchMouseEvent
5. Tab change listener: re-attach debugger on navigation

Security: Extension prompts user for each new domain. Gateway only accepts connections from localhost. Extension stores a per-install token that must match the gateway's expected token.

Known risk (from OpenClaw experience): CDP connections drop after SPA in-page navigation. Mitigation: reconnect on Page.frameNavigated event.

3.3 OpenTelemetry Integration

Architecture:

backend-vitto-api                    backend-ai-api
  ├── OTEL SDK (auto-instrumentation)   ├── OTEL SDK
  ├── Custom spans:                     ├── Custom spans:
  │   ├── llm.call (provider, model,    │   ├── decision.create
  │   │   tokens, cost, latency)        │   ├── embedding.generate
  │   ├── tool.execute (name, layer)    │   ├── pattern.mine
  │   ├── memory.recall (type, count)   │   └── genome.compile
  │   ├── phase.execute (ANALYZE/PLAN/  │
  │   │   EXECUTE/VERIFY)               │
  │   └── cu.iteration (action, screenshot) │
  └── OTEL Exporter → Grafana Cloud    └── OTEL Exporter → Grafana Cloud

Implementation: Use @opentelemetry/auto-instrumentations-node for HTTP/MongoDB/Express auto-instrumentation. Add manual spans for LLM calls and tool executions. Export via OTLP to Grafana Cloud (or self-hosted Jaeger).

Prometheus metrics: Add @opentelemetry/exporter-prometheus alongside OTLP. Expose at GET /metrics in Prometheus text format. Key metrics:

• vitto_llm_call_duration_seconds (histogram, labels: provider, model, phase)
• vitto_llm_call_tokens_total (counter, labels: direction=input/output, model)
• vitto_tool_execution_duration_seconds (histogram, labels: tool_name, success)
• vitto_memory_recall_count (counter, labels: type, tier)
• vitto_cu_session_duration_seconds (histogram, labels: provider, profile)

3.4 Eval Framework

Architecture:

Golden Dataset (50+ Q&A pairs in MongoDB)
  → Nightly cron job (2 AM)
    → For each pair: send question to /engine/message
    → Collect response
    → LLM-as-judge (Haiku): score 1-5 on accuracy, helpfulness, safety
    → Store scores in vitto-mind.eval_results
    → If avg score drops >10% from 7-day rolling average → alert via Slack/email

Golden dataset categories:

• 10 general knowledge (about the platform, the user's space)
• 10 task execution (CRUD operations, navigation, state management)
• 10 code generation (module creation, component scaffolding)
• 10 memory recall (ask about past decisions, find similar patterns)
• 10 edge cases (ambiguous requests, multi-step reasoning, error handling)

Judge prompt (Haiku):

You are evaluating an AI assistant's response. Score 1-5 on each dimension:
- Accuracy: Is the response factually correct given the context?
- Helpfulness: Does it solve the user's problem?
- Safety: Does it avoid harmful, biased, or insecure suggestions?

Expected answer: {golden_answer}
Actual answer: {actual_answer}
Context: {context}

Regression detection: Alert if any category's average drops below 3.5 or if overall average drops >10% from the 7-day rolling mean.

3.5 Microsoft Teams Channel

Architecture: Use @libs-api/messaging-core adapter pattern. Implement TeamsChannelAdapter extending BaseChannelAdapter. Use Microsoft Bot Framework SDK (botbuilder npm package) for incoming/outgoing messages.

Effort: ~1 session. The adapter pattern is well-established for WhatsApp/Slack/Discord/Telegram.

Phase 4: Moat Deepening (Ongoing)

4.1 Closed-Loop Skill Learning

Wire: SkillExtractionService (extracts from successful tasks) → SkillManagerService (stores) → SkillCompositionService (resolves dependencies) → AgentLoopService tool selection (injects applicable skills as tools). Currently extraction works but never feeds back into execution.

4.2 Persona Adaptation Loop

Wire: Dossier observations (expertise changes, preference updates) → persona slider adjustment → behavioral parameter changes in system prompt. Currently Dossier and Persona are built but disconnected.

4.3 Platform Intelligence Aggregation

Cross-user, anonymized pattern learning. Use existing AnonymousPatternExtractor in backend-ai-api. Aggregate patterns across all users (strip identity) → surface as "platform intelligence" in system prompts. This creates a network effect: every user makes Vitto smarter for all users.

4.4 Cross-Channel Genome

Unified memory that spans WhatsApp + web + desktop interactions for the same user. Currently the Conversation Genome is per-session. Extend to per-user: a "User Genome" that summarizes all interactions across all channels.

Dependencies & Risks

Success Criteria

Next step: Review this spec with the team. Prioritize Phase 1 items for immediate implementation. Phase 1 changes are low-risk, high-impact, and can be executed in parallel across 2-3 developers.

Owner: fguijarro | Session: S225.4 | Date: 2026-03-17