Vitto Jarvis — Master Specification
Single source of truth for evolving Vitto into a Jarvis-level super agent. Consolidates all prior Jarvis specs, includes codebase-verified audit, 7 capability pillars, dependency-ordered build phases, and programmatic verification gates. Terminal spec — not to be superseded.
Vitto Jarvis — Master Specification
This is the terminal Jarvis spec. It supersedes all 4 prior Jarvis specs, consolidates what's been built (20 features verified), defines the 11 remaining gaps, and includes programmatic verification gates so future sessions can check completion without reading code.
Guiding principles:
- Analyze first — never repeat work. Every section starts with what exists.
- Respect API boundaries — backend-api (user data), backend-ai-api (intelligence), backend-vitto-api (execution).
- Everything wired — no orphaned backend services. If it exists, it must be visible in the chat via plugins.
- Everything testable — verification gates with specific commands/endpoints.
- Document for future agents — handoff docs for deferred work.
Part I: The Jarvis Vision — 7 Capability Pillars
Pillar 1: Cognitive Core (Reasoning Layer)
Gap: Vitto's agent loop is while not done: LLM(history + tools) → execute. Reasoning is implicit in the LLM. There's no persistent reasoning trace, no explicit hypothesis tracking, no post-task analysis of WHY Vitto chose one tool over another.
Solution: A ReasoningService that decorates the existing AgentLoopService — NOT a replacement. Before each LLM call, it captures the current hypothesis. After each tool result, it links the observation back. These flow as metadata ON existing SSE events (not separate events — avoids stream flooding).
Chat visibility: New activity card renderer in the streaming service (see Design Decision 1) shows collapsible reasoning traces inline:
┌─ Reasoning ──────────────────────────────┐
│ Hypothesis: User wants module X updated │
│ ▸ Step 1: read_file(module.ts) → ✓ │
│ Observation: File exists, 240 lines │
│ ▸ Step 2: write_file → ✓ │
│ Confidence: 0.92 │
│ Conclusion: Applied pattern from S234 │
└──────────────────────────────────────────┘
Collapsed by default. Expandable for debugging. Reasoning traces persisted alongside execution traces in MongoDB for eval scoring and future recall.
API ownership: backend-vitto-api.
Pillar 2: Perfect Memory
Delegated to S239. See docs/english/Plans/S239-memory-system-handoff.md.
Covers:
- Cross-channel genome (per-user, not per-session)
- User preference learning (adapt to reactions, not just execution quality)
- Memory anticipation (predict what user needs next)
- Conversation Genome consumption from backend-ai-api
S238 owns the wiring: Once S239 builds these services, S238 ensures context assembly injects them and chat plugins render memory state ("I remember from WhatsApp yesterday...").
Pillar 3: Autonomous Execution (Background Tasks)
Gap: CostGuard caps all tasks at 10-15 minutes. Checkpoints exist (Redis) but only for crash recovery, not for chaining multi-step work across time windows.
Solution: A BackgroundTaskService that decomposes complex tasks into a chain of bounded subtasks. Each subtask runs through the normal AgentLoopService.run() with normal CostGuard limits. The orchestrator chains them via BullMQ job dependencies.
Key design decisions:
- Template-guided decomposition (research→analyze→draft→deliver), NOT free-form LLM decomposition. The LLM fills in specifics within the template. This avoids the known failure mode of LLM-generated task graphs.
- Each subtask is a normal task with its own cost guard — no new risk model.
- Checkpoint data stored in MongoDB (survives process restart), linked by chain ID.
- SmartRouter picks model per subtask (Haiku for search, Sonnet for analysis).
- User can cancel at any subtask boundary.
Chat visibility: Activity card in chat (via ActivityCardHostComponent):
┌─ Background Task: Competitor Analysis ───┐
│ ✓ Step 1/4: Web research (done, $0.12) │
│ ⏳ Step 2/4: Analyzing findings... │
│ ○ Step 3/4: Draft report (pending) │
│ ○ Step 4/4: Format & deliver (pending) │
│ [Cancel] [Pause after current step] │
└──────────────────────────────────────────┘
API ownership: backend-vitto-api. Uses existing TaskQueueService (BullMQ) and AgentLoopService.
Pillar 4: Persistent Presence (Desktop Daemon)
What exists (verified): Tauri v2 app at apps/vitto-desktop/ — v0.3.0, signed for macOS, starts hidden, autostart via LaunchAgent, 13 native commands, global hotkey Cmd+Ctrl+V, auto-updater. See docs/english/Apps/vitto-desktop-app/changelog.md.
4 surgical changes needed (no rewrite):
4a. Smart heartbeat — Replace blind polling in VittoHeartbeatService with idle-aware logic:
- Detect user idle via macOS
CGEventSourceSecondsSinceLastEventType(available via existinglocal_exec.rssystem info commands) - When idle >5 min: check pending tasks, calendar, unread messages
- When active: don't interrupt
- Actionable findings → native Tauri notification (
tauri-plugin-notificationalready installed)
4b. Infinite WebSocket reconnection:
// vitto-screen-capture.service.ts
reconnectionAttempts: Infinity // was 10
4c. Proactive notification → chat bridge — Desktop becomes a first-class channel in ProactiveDispatcherService. Tapping a notification deep-links into chat with context pre-loaded.
4d. Version tracking — Tray "About Vitto" shows current version. /vitto/debug route shows desktop app version when connected via LocalBridge. Every release updates docs/english/Apps/vitto-desktop-app/changelog.md.
Chat visibility: Proactive findings render as summary cards:
┌─ Vitto noticed while you were away ──────┐
│ 3 pending tasks need attention │
│ Meeting with Design team in 45 min │
│ Module auth-guard has 2 failing tests │
│ [Review Tasks] [Dismiss] │
└──────────────────────────────────────────┘
Renders via existing ProactiveToastComponent + new card renderer in the activity card system.
API ownership: Tauri app is frontend. Calls existing POST /engine/message. No new API endpoints.
Pillar 5: External Reach (Sub-Agent Power + OAuth)
What's done: OAuth full cycle, expert sub-agents with generateDefinition() + marketplace(), Teams adapter, parallel sub-Vitto orchestration, context-aware tool filtering.
What's NOT done:
5a. Sub-agent autonomy gating — Sub-agents with elevated tools (run_command, write_file) execute without approval checkpoints. Wire AutonomyService.shouldProceed() before sub-agent execution when elevated tools are present.
Chat visibility: Approval card in the activity card system:
┌─ Sub-Agent Request ──────────────────────┐
│ Agent: "Code Reviewer" (coder role) │
│ Wants: run_command("yarn test") │
│ Autonomy: SUGGEST (needs approval) │
│ [Approve] [Approve All for Session] [Deny]│
└──────────────────────────────────────────┘
5b. OAuth chat wiring — OAuth works on the backend but there's no chat plugin to show connection status or trigger flows inline. When Vitto needs Google access and it's not connected, render a connect button:
┌─ Connect Google ─────────────────────────┐
│ I need access to your Google Calendar. │
│ [Connect Google Account] │
│ (Opens OAuth consent in popup) │
└──────────────────────────────────────────┘
Uses the existing oauth_connect tool which returns an auth URL — the plugin turns it into a clickable action.
VM tier: Deferred. See docs/english/Plans/S-future-vm-agent-tier-handoff.md.
API ownership: backend-vitto-api. Frontend plugins in @libs-ai/vitto-chat.
Pillar 6: Self-Improvement (Closed-Loop Learning)
What's done: Outcome loop closed, SelfLearningPlugin, eval framework (30 golden, nightly cron), learning sync (6AM), anonymous pattern extractor.
What's NOT done:
6a. Closed-loop skill learning:
- Add
confidenceTier(experimental → validated → trusted) to skill schema - After VERIFY phase: record outcome per injected skill via
SkillManagerService.recordOutcome() - Graduate at ≥80% success after 10 uses. Demote at <30% after 5 uses
- Weight retrieval by tier (trusted=1.0, validated=0.7, experimental=0.4)
- Cold-start mitigation: 2-week warm start period where new skills get equal weighting
- Weekly cron mines anti-patterns from failure traces
Chat visibility: When Vitto uses a skill, reasoning plugin shows:
┌─ Skill Used ─────────────────────────────┐
│ "CRUD module pattern" (trusted, 92%) │
│ 18/20 past uses succeeded │
└──────────────────────────────────────────┘
6b. Eval system hardening:
- Add
evalVersionfield (git commit hash) to eval results - Score by task category, not just overall
- Alert threshold: 5% per-category regression (down from 10% overall)
- No multi-judge yet — single judge with better granularity is higher ROI
6c. Persona drift auto-apply:
- Enable auto-apply ONLY for low-risk sliders (explanationDepth, formality)
- Keep manual approval for high-risk sliders (humorLevel, proactivity)
- Max 10% slider change per cycle. 24h per-user cooldown (already implemented)
Chat visibility:
┌─ Vitto Adapted ──────────────────────────┐
│ I noticed you prefer shorter answers. │
│ Adjusted: explanationDepth 7 → 5 │
│ [Undo] [That's right] │
└──────────────────────────────────────────┘
API ownership: backend-vitto-api. Existing services modified, no new services.
Pillar 7: Soul & Identity (Personality + Visual Evolution)
What's done: VittoEvolutionService (maturity scoring, 5 stages), avatar engine (14 moods, gaze tracking), VittoBrainVisualizationComponent (exists but static).
What's NOT done:
7a. Brain visualization wired to live data — Pure frontend. Connect existing SSE conscience stream + maturity score API to the brain page. Show:
- Evolution stage with progress bar
- Live conscience events
- Memory stats, skill portfolio, execution history
7b. Avatar reflects evolution stage — MoodDefinitionService needs a stageOverlay that modifies base mood properties (aura complexity, feature richness) based on current VittoEvolutionService stage.
7c. Evolution milestones in chat — When Vitto crosses a stage boundary, render in the activity card system:
┌─ Vitto Evolved ──────────────────────────┐
│ Stage: Aware → Connected │
│ Trigger: 200th memory stored │
│ Maturity: ████████░░ 62/100 │
└──────────────────────────────────────────┘
Priority: Lowest. Build last. Fun and differentiating but doesn't make Vitto smarter.
API ownership: Frontend (avatar, brain visualization). backend-vitto-api VittoEvolutionService already exists.
Part II: Verified Audit — What's Done vs What's Not
Completed (20 items — codebase-verified 2026-03-19)
| # | Feature | Source Spec | Key File(s) | Verified By |
|---|---|---|---|---|
| 1 | Sub-Vitto engine (parallel orchestration) | S232/S234 | sub-vitto-orchestrator.service.ts (312 lines) |
Promise.allSettled() at line 184, calls agentLoop.run() at line 227 |
| 2 | SubAgent→AgentLoop refactor | S234 | sub-agent.service.ts (1,049 lines) |
agentLoop.run() at line 719 |
| 3 | Context-aware tool filtering | S237 | tool-registry.service.ts |
TOOL_CATEGORIES (lines 40-71), INTENT_TOOL_MAP (lines 74-87), getToolsForContext() (lines 199-260) |
| 4 | Memory priority fix (dossier elevated) | S237 | context-assembly | Dossier elevated to HIGH priority in token budget |
| 5 | Conversation history injection | S237 | agent-loop / vitto-engine | AgentSessionService.getConversationHistory() injected into loop |
| 6 | SSE ticket auth | S232.1-A | sse-ticket.service.ts, sse-ticket.guard.ts, sse-ticket.controller.ts |
JWT 5-min expiry, cross-user protection |
| 7 | AgentProcess redirect to Vitto | S232.1-A | agent-process.controller.ts (ai-api) |
Proxies to vitto-api, logs "[S232.2] PROXYING" |
| 8 | OAuth credential delegation | S232 | oauth.controller.ts, oauth.service.ts (backend-api), oauth.plugin.ts (vitto-api), user-oauth-connection.schema.ts |
Full cycle: consent, AES-256-GCM, proxy, Google+GitHub |
| 9 | Expert sub-agent creation | S232 | sub-agent.service.ts |
generateDefinition() lines 169-240, marketplace() lines 249-282 |
| 10 | Vitto evolution (maturity score) | S232 | vitto-evolution.service.ts (131 lines) |
Normalized scoring, 5 stages, conscience events |
| 11 | Persona adaptation (confidence thresholds) | S232 | dossier-persona-adapter.service.ts (148 lines) |
Confidence 0-1, rate-limited 1/user/24h, auto-apply disabled |
| 12 | Teams channel adapter | S232 | teams-channel.adapter.ts (151 lines) |
RichMessage → Adaptive Cards |
| 13 | MongoDB cleanup | S232 | N/A | channel-user-links and vitto-task-history — zero references in codebase |
| 14 | Embedding search wired | S231.2 | learning-embedding-adapter.service.ts |
Wired in LearningService.onModuleInit() |
| 15 | Cost tracker (5 bug fixes) | S231.2 | cost-tracker.service.ts, vitto.controller.ts |
MongoDB persistence in onModuleInit(), controller returns cost field |
| 16 | Eval framework | S231.2 | eval/eval-runner.service.ts |
30 golden entries, @Cron('0 2 * * *'), 7-day rolling avg |
| 17 | In-app observability | S231.2 | vitto-observability.service.ts (386 lines) |
Metrics aggregation, anomaly detection, slow task tracking |
| 18 | Outcome loop closed | S216/S232.1 | outcome-feedback.service.ts |
sendOutcome() → postExecutionOutcome() → ai-api /learning/execution-outcomes |
| 19 | S216 wiring (9 gaps) | S216 | Multiple files | 8/9 fixed (episodic gate, skill composition, extraction notify, duplicate outcome, channel filter, space brief, rich digest, alert ack, voice note delivery) |
| 20 | Checkpoint/resume | S232.1 | checkpoint.service.ts |
Redis-backed, persists agent loop state before each LLM call |
Remaining (11 items — this spec)
| # | Feature | Pillar | Priority | Sessions Est. | Owner |
|---|---|---|---|---|---|
| 1 | Activity Card System (streaming + renderers) | ALL | HIGH | 2 | S238 |
| 2 | Reasoning layer (ReasoningService) | 1: Cognitive | HIGH | 2 | S238 |
| 3 | Cross-channel genome | 2: Memory | HIGH | ~2 | S239 |
| 4 | User preference learning | 2: Memory | HIGH | ~2 | S239 |
| 5 | Background task chains | 3: Autonomous | HIGH | 3 | S238 |
| 6 | Desktop daemon (4 changes) | 4: Presence | MEDIUM | 2 | S238 |
| 7 | Sub-agent autonomy gating | 5: External | MEDIUM | 1 | S238 |
| 8 | OAuth chat wiring | 5: External | MEDIUM | 1 | S238 |
| 9 | Closed-loop skill learning | 6: Self-Improve | HIGH | 2 | S238 |
| 10 | Eval hardening + persona auto-apply | 6: Self-Improve | MEDIUM | 1 | S238 |
| 11 | Brain visualization wired | 7: Soul | LOW | 1 | S238 |
| S238 scope | ~15 sessions | ||||
| S239 scope (parallel) | ~4 sessions |
Deferred (with handoff docs)
| Feature | Handoff Doc | Reason |
|---|---|---|
| VM agent tier | docs/english/Plans/S-future-vm-agent-tier-handoff.md |
Low ROI — in-process handles 90% of cases |
| Voice pipeline | Deferred by owner | Do it right in a later session |
| Browser extension relay | N/A | VM snapshots solve 80% of same need |
Part III: Build Phases (Dependency-Ordered)
Phase 0: Activity Card System (BLOCKING — All pillars depend on this)
Why first: Every pillar requires chat visibility via inline activity cards. The existing ChatPlugin interface does NOT support SSE-event-to-card dispatch (it supports tabs, slash commands, quick actions, and output components — none map to dynamic SSE card rendering). The existing GenerationPlugin also does NOT do SSE-to-card dispatch. This means Phase 0 must build a new rendering mechanism.
Architecture decision: The correct approach is NOT to extend the ChatPlugin interface. Instead, extend the streaming service and message rendering layer — which already handles tool result rendering inline. The pattern is:
VittoChatStreamingServiceparses SSE events viahandleStreamEvent()(currently a fixedStreamEventtype union:content_delta,phase_start,phase_progress,tool_start,generation_complete,error)- Extend
StreamEventtype union with new event types for activity cards - New events are mapped to activity entries in the message stream (same as how tool results appear inline)
- Activity entries are rendered by a new
ActivityCardHostComponentthat dynamically resolves the correct card component based on event type
What to build:
File 1 — MODIFY: libs/libs-ai/vitto-chat/src/lib/services/vitto-chat-streaming.service.ts
Extend StreamEvent type union:
Existing: 'content_delta' | 'phase_start' | 'phase_progress' | 'tool_start' | 'generation_complete' | 'error'
Add: 'activity_card'
In handleStreamEvent(), add an activity_card case that creates an activity entry in the message stream. The activity entry contains: { cardType: string, data: any, interactive: boolean }.
Backend SSE events that should trigger activity cards:
| Backend SSE Event | cardType | Interactive | Pillar |
|---|---|---|---|
tool.starting (with reasoning metadata) |
reasoning-trace |
No (collapsed) | 1: Cognitive |
background.step_completed, background.chain_status |
background-task |
Yes (cancel/pause) | 3: Autonomous |
conscience.proactive_summary |
proactive-summary |
Yes (review/dismiss) | 4: Presence |
subvitto.approval_required |
sub-agent-approval |
Yes (approve/deny) | 5: External |
tool.called where tool=oauth_connect |
oauth-connect |
Yes (connect button) | 5: External |
conscience.skill_used |
skill-usage |
No | 6: Self-Improve |
conscience.persona_adapted |
persona-adapted |
Yes (undo/confirm) | 6: Self-Improve |
conscience.evolution.visual |
evolution-milestone |
No | 7: Soul |
Note: Memory cards (Pillar 2) deferred to S239. Build 8 card types now, add memory card when S239 delivers.
File 2 — NEW: libs/libs-ai/vitto-chat/src/lib/components/activity-cards/activity-card-host.component.ts
Dynamic component host that resolves card type → component. Uses Angular's NgComponentOutlet or ViewContainerRef.createComponent(). Maps cardType string to component class via a registry map.
File 3 — NEW: libs/libs-ai/vitto-chat/src/lib/components/activity-cards/cards/ (directory with 8 card components)
Each card is a standalone Angular component with OnPush, signal inputs. All follow the same pattern:
- Receives SSE event data as input signal
- Renders a collapsible card (collapsed by default for non-critical, expanded for approval-required)
- Emits actions (approve, dismiss, undo, connect) via output signals
- Actions call backend-vitto-api via
VittoChatEngineService(existing service)
File 4 — MODIFY: libs/libs-ai/vitto-chat/src/lib/components/chat-shell/ (message rendering area)
Register ActivityCardHostComponent as an inline renderer within the message stream, alongside the existing tool result renderers.
Dependencies: None. This is pure frontend. Estimated effort: 2 sessions.
Phase 1: Reasoning Layer (Pillar 1)
Depends on: Phase 0 (VittoActivityPlugin for chat rendering)
What to build:
File 1 — NEW: APIs/backend-vitto-api/src/app/engine/reasoning.service.ts
Purpose: Wraps AgentLoopService with reasoning metadata.
Interface:
- enrichToolCall(toolCall, context) → adds hypothesis field to SSE event
- enrichToolResult(toolResult, hypothesis) → adds observation + confidence
- synthesize(observations[]) → generates conclusion with confidence score
Implementation:
- NOT an LLM call. Uses heuristics to extract hypothesis from the LLM's
response text (regex for "I'll", "Let me", "I think", etc.)
- Confidence = tool success rate for this pattern (from execution traces)
- Persists reasoning trace in execution-traces collection alongside existing data
File 2 — MODIFY: APIs/backend-vitto-api/src/app/engine/agent-loop.service.ts
Inject ReasoningService. Before each tool call emission, call enrichToolCall(). After each tool result, call enrichToolResult(). The existing SSE events (tool.starting, tool.called) gain metadata fields — no new event types needed:
// Existing event:
{ type: 'tool.starting', data: { toolName, args } }
// Becomes:
{ type: 'tool.starting', data: { toolName, args, reasoning: { hypothesis, confidence } } }
File 3 — MODIFY: APIs/backend-vitto-api/src/app/mind/schemas/execution-trace.schema.ts
Add reasoningTrace field (array of { hypothesis: string, observation: string, confidence: number, toolName: string }) to persist reasoning alongside execution data.
File 4 — MODIFY: APIs/backend-vitto-api/src/app/engine/engine.module.ts
Register ReasoningService in providers array.
File 5 — Frontend: ReasoningTraceCardComponent (from Phase 0) renders the enriched events.
Dependencies: Phase 0 for rendering. Estimated effort: 2 sessions.
Phase 2: Background Task Chains (Pillar 3)
Depends on: Phase 1 (reasoning traces help debug chain failures)
What to build:
File 1 — NEW: APIs/backend-vitto-api/src/app/engine/background-task.service.ts
Purpose: Decomposes complex tasks into bounded subtask chains.
Interface:
- decompose(task, template?) → SubtaskChain
- executeChain(chain, userId) → starts BullMQ job chain
- getChainStatus(chainId) → progress across all subtasks
- cancelChain(chainId) → cancels remaining subtasks
- pauseChain(chainId) → stops after current subtask completes
Templates (initial set):
- RESEARCH: search → analyze → synthesize → deliver
- CODE_REVIEW: read → analyze → report → suggest
- GENERATE: plan → scaffold → implement → verify
Implementation:
- Each subtask is a BullMQ job with { chainId, stepIndex, prevOutput }
- Job dependency: step N depends on step N-1
- prevOutput stored in MongoDB (chain-checkpoints collection), NOT BullMQ job data
(BullMQ has payload size limits)
- SmartRouter picks model per step via complexity hint in template
- CostGuard applies per-subtask (existing limits, no changes)
File 2 — MODIFY: APIs/backend-vitto-api/src/app/engine/vitto-engine.service.ts
In processTaskWorkflow(), before delegating to this.phaseExecutor.execute(task, sseSubject), check if RequestClassifierService classifies complexity as 'complex' AND the task description matches a background template. If so, route to BackgroundTaskService.decompose() instead. Note: the classifier uses 'simple' | 'moderate' | 'complex' (not HIGH/CRITICAL).
File 3 — MODIFY: Frontend BackgroundTaskCardComponent (from Phase 0) renders chain progress with cancel/pause actions.
File 4 — NEW: APIs/backend-vitto-api/src/app/engine/schemas/chain-checkpoint.schema.ts
MongoDB schema for chain state: { chainId, stepIndex, output, status, cost, startedAt, completedAt }.
File 5 — MODIFY: APIs/backend-vitto-api/src/app/engine/vitto.controller.ts (or NEW controller)
Add REST endpoints for frontend card actions:
GET /engine/background/:chainId/status → getChainStatus()
POST /engine/background/:chainId/cancel → cancelChain()
POST /engine/background/:chainId/pause → pauseChain()
POST /engine/background/:chainId/resume → resumeChain()
The BackgroundTaskCardComponent calls these when user clicks [Cancel] or [Pause].
File 6 — MODIFY: APIs/backend-vitto-api/src/app/engine/engine.module.ts
Register BackgroundTaskService in providers, register ChainCheckpoint schema via MongooseModule.forFeature().
Dependencies: Phase 0 for rendering. Can start after Phase 0 (does NOT depend on Phase 1 — reasoning traces are nice-to-have for debugging but not functionally required). Estimated effort: 3 sessions.
Phase 3: Self-Improvement (Pillar 6)
Depends on: None (can parallelize with Phase 2)
What to build:
3a. Skill confidence tiers:
File 1 — MODIFY: APIs/backend-vitto-api/src/app/engine/skill-manager.service.ts
Add to schema: confidenceTier (experimental|validated|trusted), outcomeHistory[], successCount, failureCount, rollingSuccessRate, warmStartExpiresAt.
Add/modify methods:
recordOutcome(skillId, outcome, taskId)— increment counters, recalculate rate, graduate/demote. NOTE: The existingrecordOutcome(skillName, success)at line 709 only incrementsusageCountand always receivessuccess=truefromphase-executor.service.ts. Must change signature to accepttaskIdand actual outcome (success/failure/partial). Also update the caller in phase-executor to pass real outcome based on VERIFY phase result, not hardcodedtrue.- Graduation: ≥80% after 10 uses → validated. ≥90% after 25 → trusted. <30% after 5 → demote.
- Warm start: new skills get
warmStartExpiresAt = now + 14 days. During warm start, experimental skills weighted equally with validated.
File 2 — MODIFY: APIs/backend-vitto-api/src/app/engine/phase-executor.service.ts
After VERIFY phase, if task.injectedSkills?.length > 0, call skillManager.recordOutcome() for each.
File 3 — MODIFY: Skill retrieval (in context assembly or skill-retrieval service)
Weight by tier: trusted=1.0, validated=0.7, experimental=0.4 (or 0.7 during warm start). Inject top 3 by weight.
3b. Eval hardening:
File 4 — MODIFY: APIs/backend-vitto-api/src/app/eval/eval-runner.service.ts
Add evalVersion (git commit hash via env var) to each eval result. Score by category (general, task, code, memory, edge). Alert on 5% per-category drop.
3c. Persona auto-apply:
File 5 — MODIFY: APIs/backend-vitto-api/src/app/engine/dossier-persona-adapter.service.ts
Enable auto-apply for explanationDepth and formality only. Keep approval gate for humorLevel, proactivity. Emit persona.adapted SSE event (consumed by PersonaAdaptedCardComponent from Phase 0).
Dependencies: Phase 0 for chat rendering of skill usage and persona adaptation. Estimated effort: 3 sessions (2 for skills, 0.5 for eval, 0.5 for persona).
Phase 4: External Reach (Pillar 5)
Depends on: Phase 0 (approval cards, OAuth cards)
What to build:
4a. Sub-agent autonomy gating:
File 1 — MODIFY: APIs/backend-vitto-api/src/app/engine/sub-agent.service.ts
Before executeTask() runs the agent loop, check if the tool list contains elevated tools (write_file, run_command, etc.). If so, call autonomyService.shouldProceed(). If autonomy level is SUGGEST, emit subvitto.approval_required SSE event and await approval via checkpoint.
File 2 — MODIFY: Frontend SubAgentApprovalCardComponent (from Phase 0) renders approval UI with [Approve], [Approve All for Session], [Deny].
4b. OAuth chat wiring:
File 3 — NEW: libs/libs-ai/vitto-chat/src/lib/plugins/vitto-activity/cards/oauth-connect-card.component.ts
When oauth_connect tool returns an auth URL, the card renders a connect button. On click, opens OAuth consent popup. On completion, refreshes connection status.
File 4 — MODIFY: APIs/backend-vitto-api/src/app/tools/plugins/oauth.plugin.ts
When oauth_api_call fails with "not connected", emit oauth.connect_required SSE event with service name and required scopes. This triggers the frontend card.
Dependencies: Phase 0 for rendering. Estimated effort: 2 sessions (1 for autonomy, 1 for OAuth wiring).
Phase 5: Desktop Daemon (Pillar 4)
Depends on: None (independent track, can parallelize)
What to build:
5a. Smart heartbeat:
NOTE: VittoHeartbeatService (S231.2) already has idle-aware logic with POLL_INTERVAL_MS=10min and IDLE_THRESHOLD_MS=5min using JS mousemove/keydown listeners. What's missing is native macOS idle detection (current JS listeners only work when the app window is focused) and native notification dispatch.
File 1 — MODIFY: libs/libs-ai/vitto-core/src/lib/services/vitto-heartbeat.service.ts
Replace JS-based idle detection with Tauri native command system_info (which already returns system metrics including idle time via CGEventSourceSecondsSinceLastEventType). This makes idle detection work system-wide, not just when the Vitto window is focused.
Add notification dispatch: on actionable result, invoke Tauri's tauri-plugin-notification (already installed). On notification tap, deep-link to chat via vitto://chat?context={taskId}.
5b. Infinite reconnection:
NOTE: vitto-screen-capture.service.ts already has reconnectionAttempts: Infinity (S231.2 fix). Verify this is also true for the LocalBridge WebSocket connection — if not, fix there.
5c. Version tracking:
File 3 — MODIFY: Tray menu in lib.rs — add "About Vitto v{VERSION}" item
File 4 — MODIFY: /vitto/debug route — show desktop version when LocalBridge is connected (version sent in WebSocket handshake)
5d. Changelog rule: Every Tauri release:
- Bump version in
Cargo.toml - Update
docs/english/Apps/vitto-desktop-app/changelog.md - Push to
vitto-updater.gograb.space
Dependencies: None. Estimated effort: 2 sessions.
Phase 6: Soul & Identity (Pillar 7) — Build Last
Depends on: Phase 0 (card rendering)
What to build:
6a. Brain visualization:
File 1 — MODIFY: libs/libs-ai/vitto-hub/src/lib/components/brain/vitto-brain-visualization.component.ts
Wire to live data sources:
VittoEvolutionServiceAPI → maturity score, current stage, progress to next- Conscience SSE stream → live event feed
VittoMemoryServiceAPI → memory count, recall statsSkillManagerServiceAPI → skill portfolio by tier
6b. Avatar stage overlay:
File 2 — MODIFY: libs/libs-anim/vitto-avatar/src/lib/services/vitto-mood.service.ts
Add applyStageOverlay(stage) that modifies base mood properties:
- Nascent: minimal aura, simple features
- Aware: defined features, soft aura
- Connected: warm tones, particle effects
- Attuned: complex features, responsive aura
- Transcendent: full visual richness
6c. Evolution milestones in chat:
Already handled by EvolutionMilestoneCardComponent in Phase 0. Backend already emits conscience.evolution.visual events via VittoEvolutionService (line 99). The streaming service maps this to cardType evolution-milestone.
Dependencies: Phase 0. Estimated effort: 1 session.
Part IV: Verification Matrix
How to check if a feature is done
Every phase has programmatic verification gates. Run these commands to check completion without reading code.
Phase 0: Activity Card System
| Gate | How to Verify | Expected |
|---|---|---|
| Card host exists | ls libs/libs-ai/vitto-chat/src/lib/components/activity-cards/activity-card-host.component.ts |
File exists |
| Card components exist | ls libs/libs-ai/vitto-chat/src/lib/components/activity-cards/cards/ |
8 component directories |
| StreamEvent extended | grep "activity_card" libs/libs-ai/vitto-chat/src/lib/services/vitto-chat-streaming.service.ts |
At least 1 hit |
| Compiles | yarn build gograb-space |
0 errors |
| Renders in chat | Open web chat → trigger any registered SSE event → card appears inline | Visual verification |
Phase 1: Reasoning Layer
| Gate | How to Verify | Expected |
|---|---|---|
| Service exists | ls APIs/backend-vitto-api/src/app/engine/reasoning.service.ts |
File exists |
| Injected in agent loop | grep -n "reasoningService|ReasoningService" APIs/backend-vitto-api/src/app/engine/agent-loop.service.ts |
At least 2 hits |
| SSE events enriched | Send a chat message → inspect SSE stream → tool.starting event has reasoning field |
reasoning.hypothesis present |
| Trace persisted | After task completion → db.getCollection('execution-traces').findOne({}, {sort: {createdAt: -1}}) → check for reasoningTrace field |
Array of hypothesis/observation pairs |
| Chat card renders | Send a message that triggers tool use → reasoning card appears in chat | Collapsible card with hypothesis/observations |
Phase 2: Background Tasks
| Gate | How to Verify | Expected |
|---|---|---|
| Service exists | ls APIs/backend-vitto-api/src/app/engine/background-task.service.ts |
File exists |
| Schema exists | ls APIs/backend-vitto-api/src/app/engine/schemas/chain-checkpoint.schema.ts |
File exists |
| Chain executes | Send complex task via chat → observe chain decomposition → all subtasks complete | Multi-step card shows progress |
| Checkpoint persists | db.getCollection('chain-checkpoints').find({ chainId: '<id>' }) |
Documents per subtask |
| Cancel works | Start chain → click [Cancel] → remaining subtasks don't execute | Chain status = cancelled |
| Cost tracking | Each subtask has independent cost in chat card | Per-step cost displayed |
Phase 3: Self-Improvement
| Gate | How to Verify | Expected |
|---|---|---|
| Skill tiers in schema | grep "confidenceTier" APIs/backend-vitto-api/src/app/engine/skill-manager.service.ts |
Schema field exists |
| Outcome recorded | After task with injected skill → db.getCollection('vitto-skills').findOne({ _id: '<skillId>' }) |
outcomeHistory has entry, counters incremented |
| Graduation | Skill with 10+ successes and ≥80% rate | confidenceTier: 'validated' |
| Eval version | db.getCollection('eval-results').findOne({}, {sort: {createdAt: -1}}) |
evalVersion field present |
| Persona auto-apply | After 10+ interactions → check persona sliders | explanationDepth or formality changed, conscience event emitted |
| Chat cards | Skill usage card and persona adapted card render in chat | Visual verification |
Phase 4: External Reach
| Gate | How to Verify | Expected |
|---|---|---|
| Autonomy gate | Create sub-agent with coder role → execute → approval card appears |
Card with [Approve] [Deny] buttons |
| Approval flow | Click [Approve] → sub-agent executes → result returned | End-to-end success |
| OAuth card | Ask Vitto "check my calendar" without Google connected → connect card appears | Card with [Connect Google Account] button |
| OAuth flow | Click connect → popup → authorize → Vitto retries calendar check | End-to-end success |
Phase 5: Desktop Daemon
| Gate | How to Verify | Expected |
|---|---|---|
| Starts hidden | Install .dmg → app starts → no window visible → tray icon present | Tray icon visible, no window |
| Hotkey | Press Cmd+Ctrl+V |
Window appears |
| Close-to-tray | Close window → tray icon still present → process running | pgrep Vitto returns PID |
| Heartbeat fires | Leave machine idle >5 min → wait → notification appears (if actionable) | Native macOS notification |
| Infinite reconnect | Stop backend → wait 30s → restart backend → app reconnects | WebSocket reconnected (check tray icon state) |
| Version in tray | Right-click tray icon | "About Vitto v0.4.0" visible |
| Survives logout | Logout → login → tray icon present | LaunchAgent auto-restart |
Phase 6: Soul & Identity
| Gate | How to Verify | Expected |
|---|---|---|
| Brain page live data | Navigate to /vitto/brain |
Maturity score, stage, memory stats, skill counts displayed |
| Avatar stage overlay | Check avatar at different maturity stages | Visual difference between nascent and connected |
| Evolution card in chat | Trigger stage boundary (create enough memories) | Milestone card appears |
Part V: Supersession Record
What Each Prior Spec Covered and Current Status
| Spec | Date | Session | Status | Details |
|---|---|---|---|---|
| S216 Wiring Design | 2026-03-13 | S216 | FULLY SUPERSEDED | 9 wiring gaps. 8 fixed, 1 partially fixed (space brief bootstrap returns minimal text — acceptable). All fixes verified in codebase. |
| S225.4 Evolution Spec | 2026-03-17 | S225.4 | FULLY SUPERSEDED | Phase 1 complete (embedding, cost, health, Tauri prep, proxy). Phases 2-4 absorbed into S232 comprehensive spec. Voice deferred. |
| S232.1 Unified Plan | 2026-03-18 | S232.1 | FULLY SUPERSEDED | Track A (security): complete. Track B (sub-agents): complete via S234. Track C (Teams, OAuth): complete. Track D (skill learning, genome): absorbed into this spec Phases 3 & S239. Track E (browser, voice): deferred with handoff docs. |
| S232 Comprehensive Design | 2026-03-18 | S232 | FULLY SUPERSEDED | 8/9 sections complete. VM tier (Section 2) deferred with handoff doc. Remaining work (skill learning, persona auto-apply, brain visualization) absorbed into this spec Phases 3 and 6. |
Delegated Work
| Delegation | Handoff Doc | Status |
|---|---|---|
| Memory system improvements | S239-memory-system-handoff.md | Active — S239 concurrent session |
| VM agent tier | S-future-vm-agent-tier-handoff.md | Deferred — future session |
| Voice pipeline | Not yet written | Deferred by owner — do it right later |
| Desktop changelog | changelog.md | Active — maintained per release |
Design Decisions (This Spec)
| # | Decision | Chosen | Rationale |
|---|---|---|---|
| 1 | Activity card architecture | Extend streaming service + ActivityCardHostComponent with dynamic card registry — NOT the ChatPlugin interface (which lacks SSE event routing) |
ChatPlugin supports tabs/slashCommands/quickActions/outputComponents, NOT SSE-to-card dispatch. Activity cards are inline message elements rendered by the streaming layer, same pattern as tool result renderers. |
| 2 | Reasoning events | Metadata ON existing SSE events, not separate events | Avoids SSE stream flooding. tool.starting gains a reasoning field. StreamEvent type union extended with 'activity_card' for new card types. |
| 3 | Background task decomposition | Template-guided, not free-form LLM | LLM decomposition is unreliable. Templates (research, code_review, generate) provide structure; LLM fills specifics. |
| 4 | VM tier | Deferred with handoff doc | Low ROI — in-process handles 90% of cases. Computer-use already handles sandboxed execution. |
| 5 | Skill cold-start | 2-week warm start period | Prevents experimental skills from being permanently deprioritized due to insufficient usage data. |
| 6 | Persona auto-apply | Low-risk sliders only (explanationDepth, formality) | High-risk sliders (humor, proactivity) require user consent. Matches existing confidence threshold architecture. |
| 7 | Brain visualization | Frontend-only changes | All data sources exist. No new backend work for this pillar. |
| 8 | Desktop daemon scope | 4 surgical changes, no rewrite | Minimize risk for an app with 0 current users. Scale effort with adoption. |
| 9 | Voice pipeline | Deferred entirely | Do it right in a dedicated session. Owner's explicit decision. |
Build Order Summary
Phase 0: Activity Card System ────────────── 2 sessions (BLOCKING)
│
├── Phase 1: Reasoning Layer ──────────── 2 sessions
│
├── Phase 2: Background Tasks ─────────── 3 sessions (PARALLEL with Phase 1)
│
├── Phase 3: Self-Improvement ─────────── 3 sessions (PARALLEL with Phases 1+2)
│
├── Phase 4: External Reach ──────────── 2 sessions (after Phase 0)
│
├── Phase 5: Desktop Daemon ──────────── 2 sessions (INDEPENDENT, any time)
│
└── Phase 6: Soul & Identity ─────────── 1 session (LAST)
Total: ~15 sessions (9 S238 scope + 2 delegated S239 + 4 deferred VM)
Parallelizable: Phases 1+2+3 (all after Phase 0), Phase 5 independent
Critical path: Phase 0 → Phases 1+2+3 in parallel (5 sessions) = 7 sessions
Testing Guide
How to Test Each Pillar End-to-End
Pillar 1: Cognitive Core
- Open web chat at
localhost:6200 - Send: "What modules do we have in this space?"
- Expand the reasoning card that appears below Vitto's response
- Verify: hypothesis, tool calls with observations, conclusion with confidence
- Navigate to
/vitto/introspection→ verify reasoning traces appear in the admin stream
Pillar 2: Memory (S239)
- Send a message on WhatsApp about "project deadline is Friday"
- Open web chat
- Send: "What did we discuss about deadlines?"
- Verify: Vitto recalls the WhatsApp conversation with cross-channel attribution
Pillar 3: Background Tasks
- Send: "Research the top 5 competitors in our market and write a summary report"
- Verify: Background task card appears with 4 steps
- Watch steps complete sequentially with per-step cost
- Click [Pause after current step] → verify chain pauses
- Resume → verify remaining steps complete
- Check MongoDB:
db.getCollection('chain-checkpoints').find({ chainId: '<id>' })
Pillar 4: Desktop Daemon
- Install Vitto.dmg
- Verify: no window on startup, tray icon visible
- Press
Cmd+Ctrl+V→ window appears - Close window → verify tray icon still present
- Leave machine idle 5+ min → verify notification appears if actionable
- Right-click tray → verify "About Vitto v0.4.0" visible
- Kill backend → wait 30s → restart → verify reconnection (tray icon changes state)
Pillar 5: External Reach
- Create a sub-agent with
coderrole at/vitto/agents - In chat: "Ask the code reviewer agent to check the auth module"
- Verify: approval card appears (if autonomy = SUGGEST)
- Click [Approve] → sub-agent executes → result returned
- Ask: "Check my Google Calendar" (without Google connected)
- Verify: OAuth connect card appears
- Click [Connect Google Account] → complete OAuth → Vitto retries
Pillar 6: Self-Improvement
- Chat with Vitto for 10+ messages using short, direct language
- Verify: persona adapted card appears ("Adjusted explanationDepth")
- Click [Undo] → verify slider reverts
- Navigate to
/vitto/skills→ check that skills have confidence tiers - Run eval manually:
POST /eval/run→ check results haveevalVersionfield
Pillar 7: Soul & Identity
- Navigate to
/vitto/brain - Verify: maturity score, current stage, memory stats, skill counts displayed live
- Check avatar: visual properties should reflect current evolution stage
- Trigger stage boundary (e.g., store 200th memory) → evolution card in chat
Owner: fguijarro Session: S238.5 Date: 2026-03-19 Estimated effort: ~15 sessions (S238 scope) + S239 (memory) + future (VM, voice) Prior specs superseded: 4 (S216, S225.4, S232, S232.1) This is the terminal Jarvis spec.