Vitto Jarvis — Unified Implementation Plan
Single source of truth for all remaining Jarvis evolution work. Merges S231.2 completion report, S231.11 updated plan, S231.10 deep analysis, and competitive gap inventory into one actionable spec with exact file changes.
Vitto Jarvis — Unified Implementation Plan
This is the single source of truth for all remaining work to evolve Vitto into a Jarvis-level super agent. It merges findings from:
- S231.10: Deep architecture analysis (Claude Opus 4.6 — 45+ services, 53 collections, 17 tool plugins, 60+ frontend components)
- S231.2: Implementation session (11 of 14 gaps closed, 3 new systems built)
- S231.11: Updated priority plan (7 remaining items tiered)
- S225.4: Original evolution spec (now superseded — Phase 1 complete)
- Competitive analysis: 15+ competitors benchmarked
Current State: What S231.2 Accomplished
All verified in code as of 2026-03-18 (19/19 checks passed).
| # | Gap | Status | Key Files |
|---|---|---|---|
| 1 | Embedding search broken (GAP-01) | FIXED | learning-embedding-adapter.service.ts (NEW) |
| 2 | Cost tracker $0.00 — 5 stacked bugs (GAP-04) | FIXED | cost-tracker.service.ts, phase-executor.service.ts, vitto.controller.ts |
| 3 | Health check stub (GAP-13) | FIXED | unified-health.controller.ts |
| 4 | Tauri daemon mode (GAP-05) | PREPARED | tauri.conf.json, vitto-heartbeat.service.ts (NEW) |
| 5 | Chat proxy polling (GAP-03) | FIXED | vitto.controller.ts |
| 6 | CU action memory (GAP-11) | FIXED | goal-history.schema.ts, computer-use-loop.service.ts |
| 7 | Eval framework (GAP-02) | BUILT | eval/ module (6 files) |
| 8 | In-app observability (GAP-08 alt) | BUILT | vitto-observability.service.ts, perf-alert.service.ts, vitto-traces/ |
| 9 | Persona adaptation | BUILT | dossier-persona-adapter.service.ts (NEW) |
| 10 | Anonymous pattern extractor | ACTIVATED | anonymous-pattern-extractor.service.ts |
| 11 | Skill loop | VERIFIED CLOSED | No changes needed |
Remaining Work: 12 Items in 5 Tracks
Track Overview
| Track | Items | Sessions | Priority |
|---|---|---|---|
| A: Security & Hygiene | 3 | 2 | BLOCKING |
| B: Sub-Agent Power | 2 | 3 | HIGH |
| C: External Reach | 3 | 3 | HIGH |
| D: Intelligence Loops | 2 | 3 | MEDIUM |
| E: Extended Presence | 2 | 5 | LOW (defer) |
| Total | 12 | ~16 |
Track A: Security & Hygiene (BLOCKING — Do First)
A.1 SSE Ticket-Based Authentication
Gap: ~114 @Public() annotations on SSE endpoints in VittoController. EventSource API cannot send Authorization headers. Anyone who knows the URL pattern can listen to any user's task events, conscience stream, or introspection events.
Confidence: Certain | Impact: Critical (data leak in production)
Implementation:
File 1 — NEW: APIs/backend-vitto-api/src/app/engine/sse-ticket.service.ts
@Injectable()
export class SseTicketService {
constructor(private readonly jwtService: JwtService) {}
generateTicket(userId: string): string {
return this.jwtService.sign(
{ sub: userId, purpose: 'sse' },
{ expiresIn: '60s' },
);
}
validateTicket(ticket: string): { userId: string } | null {
try {
const payload = this.jwtService.verify(ticket);
if (payload.purpose !== 'sse') return null;
return { userId: payload.sub };
} catch {
return null;
}
}
}
File 2 — NEW: APIs/backend-vitto-api/src/app/engine/sse-ticket.controller.ts
@Controller('engine/sse')
export class SseTicketController {
constructor(private readonly ticketService: SseTicketService) {}
@Post('ticket')
@UseGuards(JwtAuthGuard) // REQUIRES auth — this is the gate
generateTicket(@Req() req: Request): { ticket: string } {
return { ticket: this.ticketService.generateTicket(req.user.userId) };
}
}
File 3 — MODIFY: APIs/backend-vitto-api/src/app/engine/vitto.controller.ts
- Add
SseTicketServiceto constructor - On each SSE endpoint, replace
@Public()with ticket validation:
@Get('events/task/:taskId')
// REMOVE: @Public()
@Public() // Keep public but validate ticket in handler
@Sse()
taskEvents(@Param('taskId') taskId: string, @Query('ticket') ticket: string) {
const auth = this.sseTicketService.validateTicket(ticket);
if (!auth) throw new UnauthorizedException('Invalid or expired SSE ticket');
// ... existing SSE logic, but filter events by auth.userId
}
File 4 — MODIFY: libs/libs-ai/vitto-chat/src/lib/services/vitto-chat-streaming.service.ts
- Before opening SSE stream, call
POST /engine/sse/ticketto get a ticket - Append
?ticket=xxxto the SSE URL - Note: This service already uses
fetch()(not EventSource), so it already has access to auth headers for the ticket request
Effort: 1 session | Risk: Low (additive, doesn't break existing flow during migration)
Migration strategy: Deploy with dual mode — SSE endpoints accept either valid ticket OR valid JWT cookie. After frontend is updated, remove the JWT cookie path.
A.2 AgentProcessService Hard Redirect
Gap: AgentProcessService on backend-ai-api (port 3001) still runs a full LLM execution engine. The controller has a deprecation notice but nothing prevents callers from using it. This violates the "Vitto is sole executor" rule established in S209.
Confidence: Certain | Impact: High (shadow execution path bypasses cost tracking, trust escalation, quality gates, skill injection, persona overlay, and learning pipeline)
Implementation:
File 1 — MODIFY: APIs/backend-ai-api/src/app/agent-process/controllers/agent-process.controller.ts
- Replace the
execute()method body with an HTTP proxy to vitto-api:
@Post('execute')
async execute(@Body() body: any, @Req() req: Request) {
const vittoUrl = this.configService.get('VITTO_API_URL') || 'http://localhost:3003';
const response = await firstValueFrom(
this.httpService.post(`${vittoUrl}/vitto-api/engine/agent-process/execute`, body, {
headers: { authorization: req.headers.authorization },
timeout: 120_000,
}),
);
return response.data;
}
File 2 — VERIFY: APIs/backend-vitto-api/src/app/engine/agent-process-compat.controller.ts
- Confirm this controller exists and handles
POST /engine/agent-process/execute - It should delegate to
VittoEngineService.processTask()
Effort: 0.5 session | Risk: Low (transparent proxy, callers don't change)
A.3 Ops Hygiene (Quick Fixes)
3a. vitto-audit-trail TTL — Add 365-day TTL index matching security-audit-logs:
// In ComplianceService schema definition or vitto-audit-trail schema
@Prop({ type: Date, default: Date.now, index: true, expires: 365 * 24 * 60 * 60 })
createdAt: Date;
3b. Drop orphaned channel-user-links collection:
// One-time MongoDB operation
db.getCollection('channel-user-links').drop();
3c. VittoController god object — Log as tech debt ticket, don't fix now. Document the intended split:
VittoChatController— conversation endpointsVittoTaskController— task lifecycle endpointsVittoMonitorController— SSE streams, events, statusVittoAdminController— debug, config, admin-only
Effort: 0.5 session
Track B: Sub-Agent Power (HIGH — Unlock Real Delegation)
B.1 Tiered Tool Access for Sub-Agents
Gap: SubAgentService restricts ALL sub-agents to recall_memory, search_decisions, query_docs, task_complete. Sub-agents are read-only spectators that can think but never act.
Confidence: Certain | Impact: Critical (sub-agent system is decorative without this)
Current code: sub-agent.service.ts:419-444 — hardcoded SUB_AGENT_TOOLS constant.
Implementation:
Step 1 — Define role-based tool presets:
File — MODIFY: APIs/backend-vitto-api/src/app/engine/sub-agent.service.ts
const ROLE_TOOL_PRESETS: Record<string, string[]> = {
researcher: [
'recall_memory', 'search_decisions', 'query_docs', 'task_complete',
'web_search', 'fetch_webpage', 'query_platform',
],
coder: [
'recall_memory', 'search_decisions', 'query_docs', 'task_complete',
'read_file', 'write_file', 'search_code', 'run_command',
],
browser: [
'recall_memory', 'search_decisions', 'query_docs', 'task_complete',
'browser_navigate', 'browser_click', 'browser_screenshot',
'web_search', 'fetch_webpage',
],
analyst: [
'recall_memory', 'search_decisions', 'query_docs', 'task_complete',
'http_request', 'query_platform', 'web_search', 'fetch_webpage',
],
reviewer: [
'recall_memory', 'search_decisions', 'query_docs', 'task_complete',
'read_file', 'search_code', 'query_platform',
],
content_creator: [
'recall_memory', 'search_decisions', 'query_docs', 'task_complete',
'web_search', 'fetch_webpage', 'http_request',
],
custom: [], // Uses definition.tools[] directly
};
Step 2 — Use definition's tool list OR role preset:
// In executeSubAgentTool() or executeTask()
private getToolsForAgent(def: SubAgentDefinition): string[] {
if (def.tools?.length > 0) return def.tools; // Custom tool list takes priority
return ROLE_TOOL_PRESETS[def.role] || ROLE_TOOL_PRESETS.researcher; // Fallback to researcher
}
Step 3 — Add sandbox level to definition schema:
File — MODIFY: APIs/backend-vitto-api/src/app/mind/schemas/sub-agent-definition.schema.ts
@Prop({ type: String, enum: ['none', 'process', 'container'], default: 'none' })
sandboxLevel: string;
Step 4 — Enforce sandbox in execution:
// In executeTask(), before running the agent loop:
if (def.sandboxLevel === 'container') {
// Route file/code tools through ContainerSandboxService
} else if (def.sandboxLevel === 'process') {
// Route through ProcessSandboxService
}
// 'none' = direct execution (read-only tools only)
Step 5 — Gate elevated tools behind autonomy:
// Before execution
if (this.hasElevatedTools(toolList)) {
const canProceed = await this.autonomyService.shouldProceed({
action: 'sub_agent_elevated_execution',
agentName: def.name,
tools: toolList,
});
if (!canProceed) {
// Create approval request, wait for human
return { status: 'pending_approval' };
}
}
Step 6 — Update the 4 seeded agents in AgentSeedService:
// Module Gen agent → coder role (read_file, write_file, search_code, run_command)
// Architecture agent → researcher role (web_search, fetch_webpage, query_platform)
// Code Quality agent → reviewer role (read_file, search_code, query_platform)
// Widget agent → coder role
Frontend update:
File — MODIFY: libs/libs-ai/vitto-hub/src/lib/components/sub-agent-manager/sub-agent-manager.component.ts
- Add
sandboxLeveldropdown to the editor tab - Show available tools based on selected role (auto-populate, user can override)
Effort: 2 sessions | Risk: Medium (elevated tools need security review per role)
B.2 Multi-Agent Orchestration (Parallel Sub-Agent Dispatch)
Gap: Vitto can spawn one sub-agent at a time. For complex tasks ("review this PR, write tests, and update docs"), it should be able to dispatch 3 sub-agents in parallel and synthesize results.
Implementation:
File — MODIFY: APIs/backend-vitto-api/src/app/engine/sub-agent.service.ts
async orchestrateParallel(
tasks: Array<{ agentId: string; prompt: string }>,
orchestratorContext: string,
userId: string,
spaceId?: string,
): Promise<OrchestratedResult> {
// 1. Spawn all sub-agents in parallel
const results = await Promise.allSettled(
tasks.map(t => this.executeTask(t.agentId, t.prompt, userId, spaceId)),
);
// 2. Collect outputs
const outputs = results.map((r, i) => ({
agent: tasks[i].agentId,
status: r.status,
result: r.status === 'fulfilled' ? r.value : r.reason?.message,
}));
// 3. Synthesize with a lightweight LLM call
const synthesis = await this.llmProvider.chat([
{ role: 'system', content: `You are synthesizing results from ${tasks.length} sub-agents. Context: ${orchestratorContext}` },
{ role: 'user', content: JSON.stringify(outputs) },
]);
return { outputs, synthesis: synthesis.content };
}
New tool plugin — APIs/backend-vitto-api/src/app/tools/plugins/orchestration.plugin.ts
// Tool: orchestrate_agents
// Description: Dispatch multiple sub-agents in parallel for independent subtasks
// Parameters: tasks[] (agentId + prompt for each), synthesis_prompt
// Vitto's main agent loop can call this tool to fan out work
Effort: 1 session | Risk: Low (builds on existing SubAgentService)
Track C: External Reach (HIGH — Make Vitto Useful Beyond Chat)
C.1 Microsoft Teams Channel Adapter
Gap: No Teams adapter. Enterprise users expect Teams integration.
Confidence: Certain | Impact: High (enterprise unlock)
Implementation follows the existing adapter pattern (4 adapters as templates):
File 1 — NEW: libs/libs-api/messaging-core/src/lib/adapters/teams-channel.adapter.ts
@Injectable()
export class TeamsChannelAdapter extends BaseChannelAdapter {
readonly channelType = 'teams';
// Use @microsoft/botframework-connector for incoming/outgoing
async sendMessage(channel: Channel, message: RichMessage): Promise<void> { ... }
async handleIncoming(activity: Activity): Promise<ChannelMessage> { ... }
async formatForChannel(response: VittoResponse): Promise<TeamsCard> { ... }
}
File 2 — NEW: APIs/backend-vitto-api/src/app/connectivity/teams/teams-webhook.controller.ts
POST /connectivity/teams/webhook— Bot Framework incoming webhook- Delegates to
ChannelProcessingService.handleChannelMessage()
File 3 — MODIFY: libs/libs-api/messaging-core/src/lib/messaging-core.module.ts
- Register
TeamsChannelAdapterin adapter registry
Env vars: TEAMS_APP_ID, TEAMS_APP_PASSWORD (Azure Bot registration)
Package: yarn add @microsoft/botframework-connector botbuilder
Effort: 1 session | Risk: Low (proven adapter pattern, 4 templates exist)
C.2 OAuth Credential Delegation (Plugin Interface)
Gap: Vitto can make HTTP requests but can't authenticate as the user on external services. The VittoConnector schema stores connector metadata, but there's no OAuth flow.
Confidence: Certain | Impact: High (unlocks calendar, email, GitHub, Slack API access)
Phase 1 (this spec): Code-only, no deployment. Write the interfaces and tool plugin. Deploy Nango when VM snapshots work.
File 1 — NEW: APIs/backend-vitto-api/src/app/connectors/oauth/oauth-provider.interface.ts
export interface OAuthProvider {
getAuthUrl(userId: string, service: string, scopes: string[]): Promise<string>;
handleCallback(code: string, state: string): Promise<ConnectedAccount>;
executeApiCall(accountId: string, method: string, url: string, body?: any): Promise<any>;
refreshToken(accountId: string): Promise<void>;
revokeAccess(accountId: string): Promise<void>;
}
export interface ConnectedAccount {
id: string;
userId: string;
service: string; // 'google', 'github', 'slack', 'notion'
scopes: string[];
connectedAt: Date;
expiresAt?: Date;
}
File 2 — NEW: APIs/backend-vitto-api/src/app/connectors/oauth/nango-oauth.service.ts
@Injectable()
export class NangoOAuthService implements OAuthProvider {
constructor(
private readonly httpService: HttpService,
private readonly configService: ConfigService,
) {}
private get nangoUrl() {
return this.configService.get('NANGO_API_URL') || 'http://localhost:3100';
}
async getAuthUrl(userId: string, service: string, scopes: string[]) { ... }
async handleCallback(code: string, state: string) { ... }
async executeApiCall(accountId: string, method: string, url: string, body?: any) { ... }
async refreshToken(accountId: string) { ... }
async revokeAccess(accountId: string) { ... }
}
File 3 — NEW: APIs/backend-vitto-api/src/app/tools/plugins/oauth.plugin.ts
// Tools exposed to Vitto's agent loop:
// - oauth_connect: Initiate OAuth flow for a service (returns auth URL for user)
// - oauth_api_call: Make an authenticated API call via connected account
// - oauth_list_connections: List user's connected services
// - oauth_disconnect: Revoke access to a service
File 4 — NEW: APIs/backend-vitto-api/src/app/connectors/oauth/oauth.controller.ts
// POST /connectors/oauth/connect — initiate flow (returns redirect URL)
// GET /connectors/oauth/callback — handle OAuth callback
// GET /connectors/oauth/connections — list connected accounts
// DELETE /connectors/oauth/:accountId — revoke access
File 5 — MODIFY: APIs/backend-vitto-api/src/app/connectors/schemas/connector.schema.ts
- Add
oauthAccountIdfield linking to Nango-managed account
Effort: 0.5 session (interface + plugin code only) | Risk: None (code-only, no deployment)
C.3 VM Snapshot Auth Persistence
Gap: Every CU session starts with a fresh browser. No saved passwords, no cookies. User must re-login every time.
Confidence: Likely (0.85) | Impact: High (makes CU useful for repeat tasks)
Implementation:
File 1 — NEW: APIs/backend-vitto-api/src/app/mind/schemas/service-auth-state.schema.ts
@Schema({ collection: 'service-auth-states', timestamps: true })
export class ServiceAuthState {
@Prop({ required: true, index: true }) userId: string;
@Prop({ required: true }) profileId: string; // 'gmail', 'github', etc.
@Prop({ required: true }) snapshotId: string;
@Prop() lastAuthenticated: Date;
@Prop() cookieExpiry: Date;
@Prop({ type: Date, expires: 7 * 24 * 60 * 60 }) // 7-day TTL
createdAt: Date;
}
File 2 — MODIFY: APIs/backend-ai-api/src/app/infrastructure/services/computer-use-loop.service.ts
- On task completion for an authenticated service:
- Call sandbox orchestrator
POST /sandbox/snapshotto save VM state - Store
ServiceAuthStatewith snapshotId
- Call sandbox orchestrator
- On task start:
- Check for valid
ServiceAuthStatefor the target service - If found and not expired: restore from snapshot instead of cold boot
- If expired or missing: cold boot, add
--password-store=basicto Chrome flags
- Check for valid
File 3 — MODIFY: sandbox-orchestrator snapshot endpoint
- Ensure Chrome user data directory is on persistent rootfs, not tmpfs
- Add
--password-store=basicto Chrome launch flags (prevents Linux keyring dependency)
Effort: 1.5 sessions | Risk: Medium (Chrome cookie encryption may not survive snapshot)
Mitigation: Feature flag VITTO_CU_AUTH_PERSISTENCE=true. Test with minimal VM first. Store cookieExpiry and validate before restore.
Track D: Intelligence Loops (MEDIUM — Make Vitto Smarter Over Time)
D.1 Closed-Loop Skill Learning
Gap: Skills are extracted from successful tasks and retrieved during context assembly. But there's no outcome tracking per skill invocation — Vitto doesn't know which skills actually help vs. which are noise.
Confidence: Certain | Impact: High (skills accumulate without quality signal)
Implementation:
Step 1 — Extend skill schema:
File — MODIFY: APIs/backend-vitto-api/src/app/engine/skill-manager.service.ts (inline schema)
// Add to VittoSkill schema:
@Prop({ type: String, enum: ['experimental', 'validated', 'trusted'], default: 'experimental' })
confidenceTier: string;
@Prop({ type: [{ outcome: String, taskId: String, timestamp: Date }], default: [] })
outcomeHistory: Array<{ outcome: 'success' | 'failure' | 'partial'; taskId: string; timestamp: Date }>;
@Prop({ type: Number, default: 0 }) successCount: number;
@Prop({ type: Number, default: 0 }) failureCount: number;
@Prop({ type: Number, default: 0.5 }) rollingSuccessRate: number;
Step 2 — Record outcome per skill after VERIFY phase:
File — MODIFY: APIs/backend-vitto-api/src/app/engine/phase-executor.service.ts
// After VERIFY phase, if skills were injected:
if (task.injectedSkills?.length > 0) {
const outcome = verifyResult.passed ? 'success' : 'failure';
for (const skillId of task.injectedSkills) {
await this.skillManager.recordOutcome(skillId, outcome, task.taskId);
}
}
Step 3 — Add recordOutcome() to SkillManagerService:
async recordOutcome(skillId: string, outcome: string, taskId: string) {
const update: any = {
$push: { outcomeHistory: { outcome, taskId, timestamp: new Date() } },
};
if (outcome === 'success') update.$inc = { successCount: 1 };
else update.$inc = { failureCount: 1 };
const skill = await this.skillModel.findByIdAndUpdate(skillId, update, { new: true });
if (!skill) return;
// Recalculate rolling success rate
const total = skill.successCount + skill.failureCount;
const rate = total > 0 ? skill.successCount / total : 0.5;
await this.skillModel.updateOne({ _id: skillId }, { rollingSuccessRate: rate });
// Graduate/demote
if (total >= 10 && rate >= 0.8) await this.updateTier(skillId, 'validated');
if (total >= 25 && rate >= 0.9) await this.updateTier(skillId, 'trusted');
if (total >= 5 && rate < 0.3) await this.updateTier(skillId, 'experimental'); // Demote
}
Step 4 — Weight retrieval by confidence tier:
File — MODIFY: APIs/backend-vitto-api/src/app/engine/skill-retrieval.service.ts
// When retrieving skills for context injection:
const skills = await this.getRelevantSkills(taskDescription);
// Sort by tier: trusted (1.0) > validated (0.7) > experimental (0.4)
const weighted = skills.map(s => ({
...s,
weight: s.confidenceTier === 'trusted' ? 1.0 : s.confidenceTier === 'validated' ? 0.7 : 0.4,
}));
// Only inject top-weighted skills (max 3)
return weighted.sort((a, b) => b.weight - a.weight).slice(0, 3);
Step 5 — Anti-pattern mining (weekly cron):
File — NEW: APIs/backend-vitto-api/src/app/engine/anti-pattern-miner.service.ts
@Injectable()
export class AntiPatternMinerService {
@Cron(CronExpression.EVERY_WEEK) // Sunday 3AM
async mineFailurePatterns() {
// 1. Query execution-traces with outcome='failure' from last 7 days
// 2. Group by task type + failure reason
// 3. Extract common tool sequences that precede failures
// 4. Store as VittoSkill with type='anti_pattern' and negative trigger
// 5. These get injected as "DO NOT do this" in context assembly
}
}
Effort: 2 sessions | Risk: Low (additive, doesn't change existing skill behavior)
D.2 Cross-Channel Conversation Genome
Gap: The Conversation Genome (compact AI-generated summary) is per-session. When a user talks on WhatsApp about project deadlines then opens web chat, Vitto doesn't carry that context.
Confidence: Certain | Impact: High (the "it really knows me" moment)
Implementation:
File 1 — NEW: APIs/backend-vitto-api/src/app/mind/schemas/user-genome.schema.ts
@Schema({ collection: 'user-genomes', timestamps: true })
export class UserGenome {
@Prop({ required: true, unique: true }) userId: string;
@Prop() summary: string; // 500-token max unified summary
@Prop({ type: [{ topic: String, detail: String, channel: String, updatedAt: Date }] })
activeTopics: Array<{ topic: string; detail: string; channel: string; updatedAt: Date }>;
@Prop({ type: [{ fact: String, source: String, learnedAt: Date }] })
recentFacts: Array<{ fact: string; source: string; learnedAt: Date }>;
@Prop() lastCompiled: Date;
@Prop({ type: Number, default: 0 }) interactionCount: number;
}
File 2 — NEW: APIs/backend-vitto-api/src/app/mind/cross-channel-genome.service.ts
@Injectable()
export class CrossChannelGenomeService {
// After every channel message (any channel):
async updateGenome(userId: string, message: string, channel: string) {
// 1. Add to activeTopics (extract topic via lightweight heuristic, no LLM call)
// 2. Increment interactionCount
// 3. If interactionCount % 5 === 0: recompile summary via Haiku
// 4. Evict topics older than 7 days
}
// For context assembly on any channel:
async getGenomeContext(userId: string): Promise<string> {
// Returns formatted cross-channel context:
// "Recent activity across channels: [WhatsApp 2h ago: discussed X] [Web yesterday: reviewed Y]"
}
}
File 3 — MODIFY: APIs/backend-vitto-api/src/app/engine/context-assembly/channel-prompt-builder.service.ts
// In buildSystemPrompt():
const crossChannelContext = await this.crossChannelGenome.getGenomeContext(userId);
// Inject after user brief, before channel-specific history
File 4 — MODIFY: APIs/backend-vitto-api/src/app/engine/channel-processing.service.ts
// After processing each channel message:
await this.crossChannelGenome.updateGenome(userId, message.text, channel.type);
Effort: 2 sessions | Risk: Low (new collection, additive integration)
Track E: Extended Presence (LOW — Defer Unless Prioritized)
E.1 Browser Extension Relay
Gap: No way for Vitto to control the user's existing browser tabs with their existing logins.
Why deferred: VM snapshots (C.3) solve 80% of the same problem. Browser extension is an alternative for users who don't want VMs.
Architecture (for future reference):
- Chrome Extension (Manifest V3) at
apps/vitto-browser-extension/ - Background Service Worker connects to
LocalBridgeGatewayWebSocket - CDP (Chrome DevTools Protocol) for tab control
- Sideload initially, Chrome Web Store later
Effort: 3 sessions | Risk: Medium (CDP drops on SPA navigation)
E.2 Voice Pipeline
Gap: No speech-to-text or text-to-speech capability.
Why deferred: User explicitly deferred. Voice can wait.
Pre-decisions for when ready:
- STT: OpenAI Whisper API ($0.006/min, existing API key)
- TTS: Browser SpeechSynthesis for MVP (free), ElevenLabs if usage validates
- Trigger:
Cmd+Ctrl+Vhotkey as push-to-talk toggle - Backend:
POST /vitto-api/engine/voiceacceptingmultipart/form-data
Effort: 2 sessions
Implementation Order & Dependencies
Week 1-2: Track A (Security)
A.1 SSE Ticket Auth ──────────────────┐
A.2 AgentProcess Redirect ────────────┤
A.3 Ops Hygiene ──────────────────────┘ (no deps)
Week 2-4: Track B (Sub-Agents) + Track C starts
B.1 Tiered Tool Access ──────────────── (no deps)
B.2 Multi-Agent Orchestration ────────── (depends on B.1)
C.1 Teams Channel ────────────────────── (no deps, can parallel with B)
Week 4-6: Track C continues + Track D
C.2 OAuth Plugin Interface ───────────── (no deps, code-only)
C.3 VM Snapshot Persistence ──────────── (no deps)
D.1 Skill Confidence Tiers ──────────── (no deps, can parallel with C)
Week 6-8: Track D continues
D.2 Cross-Channel Genome ────────────── (no deps)
Deferred:
E.1 Browser Extension ────────────────── (after C.3 validated)
E.2 Voice Pipeline ──────────────────── (after user request)
Success Criteria
| Track | Metric | Target |
|---|---|---|
| A.1 | SSE endpoints require valid ticket | 0 public SSE endpoints without ticket validation |
| A.2 | Zero shadow executions on ai-api | AgentProcessService only proxies, never executes |
| B.1 | Sub-agent with coder role can write files |
Successful file write via sub-agent execution |
| B.2 | Parallel dispatch completes 3 sub-tasks | All 3 return results, synthesis is coherent |
| C.1 | Teams message → Vitto response | End-to-end within 5s |
| C.2 | OAuth interfaces compile | No runtime usage yet, but types and plugin register |
| C.3 | CU second run skips login | Gmail task completes without re-authentication |
| D.1 | Skills have confidence tiers | After 10 uses, at least 1 skill graduates to 'validated' |
| D.2 | Cross-channel context appears in prompt | WhatsApp topic visible in web chat's system prompt |
Verification Checklist (Per Track)
Track A Verification
-
POST /engine/sse/ticketreturns a signed JWT (requires auth) - SSE endpoint without
?ticket=returns 401 - SSE endpoint with valid ticket streams events
-
POST /ai-api/agent-process/executeproxies to vitto-api (check logs) -
vitto-audit-trailcollection has TTL index (verify viadb.collection.getIndexes())
Track B Verification
- Create sub-agent with role
coder— tools includeread_file,write_file - Execute coder agent with "read the README.md file" — returns file content
-
orchestrate_agentstool appears in tool registry - Parallel dispatch of 2 agents completes and returns synthesized result
Track C Verification
- Teams webhook receives Bot Framework activity
- Vitto responds in Teams channel
- OAuth interfaces compile (
yarn build backend-vitto-apisucceeds) - VM snapshot creates and restores successfully
- CU task on snapshotted VM skips login flow
Track D Verification
- Skill has
confidenceTier: 'experimental'after creation - After 10 successful uses, tier upgrades to
'validated' - After 5 failures with <30% rate, tier stays/downgrades to
'experimental' -
GET /mind/memory/askincludes cross-channel context - WhatsApp conversation topic appears in web chat system prompt
Architecture Decisions Made
| Decision | Chosen | Why | Confidence |
|---|---|---|---|
| SSE auth mechanism | Ticket-based JWT (60s expiry) | Least disruption, frontend already uses fetch(), spec already written | 0.93 |
| AgentProcess migration | Hard redirect (HTTP proxy) | Zero shadow execution risk, callers don't change | 0.95 |
| Sub-agent tool access | Role-based presets + definition override | Balance between safety and flexibility | 0.90 |
| OAuth provider | Custom interface + Nango (deploy later) | Code-only now, self-hostable later, 700+ APIs | 0.88 |
| Cross-channel genome | Separate collection (user-genomes) | Different access pattern than vitto-memories (single doc per user) | 0.85 |
| Skill confidence | 3-tier (experimental/validated/trusted) | Simple, outcome-driven, no manual curation needed | 0.92 |
| Anti-pattern mining | Weekly cron | Daily too aggressive, monthly too slow | 0.85 |
What This Spec Does NOT Cover
These items are intentionally excluded because they are either already complete, deferred, or belong in separate specs:
- Embedding search — Fixed in S231.2 (verified)
- Cost tracking — Fixed in S231.2 (verified)
- Eval framework — Built in S231.2 (operational)
- Observability — Built in S231.2 (in-app alternative to OTEL)
- Persona adaptation — Built in S231.2 (notification-based)
- Anonymous pattern extractor — Activated in S231.2
- Tauri daemon mode — Prepared in S231.2 (needs cargo build + test)
- Human takeover VNC — Depends on C.3 (VM snapshots). Will be a follow-up spec.
- Platform skill aggregation — Needs 1000+ active users. Too early.
- VWU billing — Business decision, not engineering.
Owner: fguijarro | Session: S232.1 | Date: 2026-03-18 Next review: After Track A completion Estimated total effort: ~16 sessions across 6-8 weeks