S234: Jarvis Evolution — Internal Testing Guide

Audience: QA / internal testing team Date: 2026-03-18 Branch: session-234-235 Prerequisites: All backends running (yarn serve:backend-api, yarn serve:backend-ai, yarn serve:backend-vitto), frontend running (yarn serve:ai)

Quick Reference: Where to Find Things

1. Sub-Vitto Parallel Execution (deploy_agents)

What it does

Vitto can now deploy multiple sub-agents to work on different parts of a task simultaneously. When it does, the chat shows a card group with real-time status.

How to test

Step 1: Open any Vitto chat conversation.

Step 2: Send a message that naturally benefits from parallel work. Examples:

• "I need you to research three things at once: (1) best practices for NestJS authentication, (2) common security vulnerabilities in OAuth, and (3) how other platforms handle token refresh."
• "Search for information about Angular signals, RxJS deprecation plans, and NestJS v11 changes — do these in parallel."
• "Analyze our codebase for: security patterns in auth modules, testing coverage gaps, and code quality issues. Work on all three simultaneously."

Step 3: Watch for the orchestration card in the chat:

┌─────────────────────────────────────┐
│ Deployed 3 agents            2.3s   │
│                                     │
│ ✓ Research NestJS auth       1.2s   │
│ ✓ OAuth vulnerabilities      1.8s   │
│ ⏳ Token refresh patterns           │
│                                     │
│ Synthesis: 3 tasks completed...     │
└─────────────────────────────────────┘

What to verify

• Card appears when Vitto uses the deploy_agents tool
• Each sub-task shows status icon (✓ completed, ✗ failed, ⏳ running, ○ pending)
• Timing information appears per task and total
• Synthesis text appears after all tasks complete
• Failed tasks show error messages (test by asking something impossible in one sub-task)
• Card is collapsible (if the UI supports it)

Edge cases

• Ask Vitto to do something that should NOT be parallelized (simple question). It should answer directly without deploying agents.
• Ask for more than 5 parallel tasks — it should cap at 5.

2. Agent Teamwork — Live Monitor

How to access

Vitto sidebar → Developer section (admin only) → Agent Teamwork Direct URL: /operations/live

What it shows

Real-time dashboard of all active and recent sub-Vitto executions across all users.

How to test

Step 1: Open the Agent Teamwork page in one browser tab.

Step 2: In another tab, trigger sub-Vitto execution (see Section 1).

Step 3: Watch the live monitor update with:

• Active count (blue badge)
• Completed count (green badge)
• Failed count (red badge, if any)
• Individual sub-Vitto cards with:
  • Task ID
  • Tier badge (in-process / vm)
  • Description
  • Current tool being used (while running)
  • Duration (when completed)
  • Result preview or error message

What to verify

• Page loads without errors
• "No sub-Vitto activity detected" shows when nothing is running
• Cards appear when sub-Vittos are spawned
• Status transitions: spawned → running → completed/failed
• Cards are sorted by most recent first
• Color-coded left borders (blue=running, green=completed, red=failed)

3. Connected Services (OAuth)

How to access

Vitto sidebar → Connections → Channels → click "Connected Services" tab

What it shows

Card grid of OAuth-connectable services: Google, GitHub, Slack, Notion. Each shows: service name, description, connection status, connect/disconnect button.

How to test

Note: Google and GitHub are the only providers with backend support. Slack and Notion will show "Connect" buttons but won't work until their provider configs are added.

Prerequisite: Set environment variables:

GOOGLE_OAUTH_CLIENT_ID=<your-google-client-id>
GOOGLE_OAUTH_CLIENT_SECRET=<your-google-client-secret>
GITHUB_OAUTH_CLIENT_ID=<your-github-client-id>
GITHUB_OAUTH_CLIENT_SECRET=<your-github-client-secret>
OAUTH_ENCRYPTION_KEY=<32-bytes-base64>
OAUTH_CALLBACK_URL=http://localhost:3000/api/auth/oauth/callback

To generate a valid encryption key:

node -e "console.log(require('crypto').randomBytes(32).toString('base64'))"

Step 1: Navigate to Channels → Connected Services tab.

Step 2: Click "Connect" on Google.

Step 3: A popup should open with Google's consent screen.

Step 4: Complete the consent flow. The popup should close automatically.

Step 5: The Google card should now show:

• Green border (connected)
• Your Google email
• "Disconnect" button instead of "Connect"

Step 6: Test disconnect: click "Disconnect" → card returns to unconnected state.

What to verify

• Tab bar shows "Channels" and "Connected Services" tabs
• All 4 service cards render (Google, GitHub, Slack, Notion)
• Connect button opens OAuth popup
• Popup closes after consent
• Card updates to show connected state + email
• Disconnect revokes and removes connection
• Connecting state shows "Connecting..." with disabled button
• If popup is closed without completing, button resets

Via Chat (Agent Tools)

You can also test OAuth through Vitto chat:

Step 1: Ask Vitto: "What services do I have connected?"

• Vitto should call oauth_status tool and report connections.

Step 2: Ask Vitto: "What's on my Google Calendar today?"

• If Google is connected, Vitto should call oauth_api_call with the Calendar API endpoint.
• If not connected, Vitto should inform you that Google isn't connected.

What to verify (chat-based)

• oauth_status tool returns connection list
• oauth_api_call successfully proxies to Google API
• Calendar events (or Drive files, etc.) are returned in chat
• Error message if provider not connected

4. Admin OAuth Dashboard

How to access

Vitto sidebar → Connections → Integrations Direct URL: /command/integrations

What it shows

Aggregate OAuth health metrics for administrators.

How to test

Step 1: Navigate to the Integrations page.

Step 2: Verify the dashboard shows:

• Stats grid: Total connections, Active, Expired, Needs Re-auth (4 cards)
• Health bar: Token health percentage with colored progress bar
• By Provider: Breakdown of connections per provider (Google, GitHub, etc.)
• Refresh button in the header

Step 3: Connect a service (Section 3), then refresh this dashboard — counts should update.

What to verify

• Page loads without errors
• Stats cards show correct counts
• Health bar reflects active/total ratio
• Provider breakdown lists all connected providers
• Refresh button reloads data
• Zero connections shows "100% health" (no failures when empty)

5. Expert Sub-Agent Management

How to access

Vitto sidebar → Capabilities → Sub-Agents

What it shows

Three-tab interface for managing specialized AI sub-agents.

How to test

5a. Chat-Based Creation (NEW)

Step 1: In any Vitto chat, ask: "Create a sub-agent that reviews code for security vulnerabilities. It should focus on OWASP Top 10 and use the recall_memory and search_decisions tools."

Step 2: Vitto should use the LLM to generate a complete agent definition and save it.

Step 3: Navigate to Sub-Agents → Registry tab. The new agent should appear with an "ai-generated" tag.

5b. Manual Creation

Step 1: Go to Sub-Agents → click "Create" or switch to Editor tab.

Step 2: Fill in:

• Name: "test-reviewer"
• Description: "Reviews pull requests"
• Role: Select "reviewer" from dropdown
• System Prompt: "You are a code reviewer..."
• Tools: Select tools from available list

Step 3: Save. Agent should appear in Registry.

5c. Quick Test

Step 1: Select an agent in the Registry.

Step 2: Switch to the test panel (if available) or use the Quick Test input.

Step 3: Enter a test prompt: "Review this function for bugs: function add(a, b) { return a - b; }"

Step 4: The agent should execute and return results.

5d. Marketplace (NEW)

Step 1: In the Registry tab, look for a "Publish" action on any agent.

Step 2: Publish an agent to the marketplace.

Step 3: The marketplace shows published agents sorted by quality score.

Step 4: Clone a published agent — it should create a copy in your space with "(clone)" suffix.

What to verify

• Chat-based creation generates a valid agent with correct role, tools, triggers
• "ai-generated" tag appears on chat-created agents
• Manual creation works with all fields
• Quick Test executes and returns results
• Published agents appear in marketplace queries
• Cloning creates an independent copy
• Performance metrics (executions, quality score) display correctly

6. Teams Channel Adapter

How to access

Vitto sidebar → Connections → Channels

Current state

The Teams channel adapter converts rich messages to Microsoft Adaptive Card format. The webhook controller receives incoming messages from Bot Framework.

How to test (limited without Teams app registration)

Without a Teams app:

• Verify the adapter is registered: check server logs on startup for "Registering adapter for channel: teams"
• Send a POST to http://localhost:3003/webhooks/teams/messages with:

{
  "type": "message",
  "id": "test-1",
  "timestamp": "2026-03-18T12:00:00Z",
  "channelId": "msteams",
  "from": { "id": "user-123", "name": "Test User" },
  "conversation": { "id": "conv-123" },
  "recipient": { "id": "bot-456", "name": "Vitto" },
  "text": "Hello Vitto",
  "serviceUrl": "https://smba.trafficmanager.net/teams/"
}

• Expected: 200 OK with { "status": "received" }

With a Teams app (when configured):

• Register a Bot in Azure Bot Service
• Set TEAMS_APP_ID and TEAMS_APP_PASSWORD env vars
• Configure webhook URL: https://your-domain/webhooks/teams/messages
• Send a message to the bot in Teams
• Vitto should respond in the conversation

What to verify

• Webhook endpoint accepts POST requests
• Activity types handled: message, invoke, conversationUpdate
• Server logs show incoming activity details
• Non-message types return { status: "ignored" } or { status: "acknowledged" }

7. Brain Visualization — Evolution Display

How to access

Vitto sidebar → Mind → any brain-related view (Memory, Knowledge Graph, etc.) The brain interactive component appears on brain visualization pages.

What it shows

An evolution indicator showing Vitto's maturity stage and score:

• Nascent (0-20): Curious — just getting started
• Aware (20-40): Attentive — building understanding
• Connected (40-60): Engaged — rich knowledge
• Attuned (60-80): Confident — deep expertise
• Transcendent (80-100): Masterful — full capability

How to test

Step 1: Open the brain visualization page.

Step 2: Look for the evolution indicator at the top:

┌────────────────────────────────────┐
│ Connected — Engaged         47%    │
└────────────────────────────────────┘

Step 3: The score is computed from:

• Memory count (30% weight, max 500 memories)
• Execution count (20% weight, max 200 tasks)
• Dossier depth (30% weight, max 50 facts)
• Skills used (20% weight, max 20 skills)

Step 4: Use Vitto more (execute tasks, build memories) → score should increase over time.

What to verify

• Evolution indicator renders with stage name + score
• Score changes as metrics change
• Brain regions pulse on conscience events (existing behavior)
• No errors in console related to evolution service

Note

The backend endpoint /engine/evolution/:userId/maturity may not exist yet. If it returns a 404, the frontend silently shows no evolution indicator (graceful degradation). The maturity computation logic is in VittoEvolutionService on the backend — it just needs a controller endpoint to expose it.

8. Personality Drift (Automatic)

How it works

After each task, DossierPersonaAdapterService analyzes the user's dossier (expertise, corrections, interaction patterns) and computes a confidence score. If confidence ≥ 0.5, it dispatches a persona suggestion via the conscience stream.

How to test

Step 1: Use Vitto extensively (10+ interactions with technical content).

Step 2: Watch the Conscience view (Vitto sidebar → Mind → Conscience).

Step 3: After enough interactions, a persona suggestion should appear:

• "Based on your recent work, you might prefer the 'Technical' persona. Your expertise suggests you'd benefit from more detailed, technical responses."

Step 4: The suggestion includes metadata:

• confidence: 0.5-1.0 (higher = more certain)
• autoApplyEligible: true if confidence ≥ 0.8 (currently disabled — suggest-only)
• suggestedPersona: name of recommended persona
• currentPersona: current active persona

What to verify

• Suggestions appear after sufficient interaction data
• Confidence score is included in the notification
• Low-confidence suggestions (< 0.5) are NOT surfaced
• 24-hour cooldown prevents spam (max 1 suggestion per user per day)
• Auto-apply is NOT happening (suggest-only mode)
• Conscience view shows personality-category events

9. VM Tier Sub-Vittos

What it does

Sub-Vittos with tier: 'vm' get access to code execution tools: run_command, write_file, read_file, search_code. These run in sandboxed environments.

How to test

Step 1: In chat, ask Vitto something that requires code execution: "Write a simple JavaScript function that calculates factorial, save it to a file, and run the tests."

Step 2: If Vitto deploys sub-agents with VM tier, the orchestration card should show vm tier badges.

Step 3: The sub-Vitto should be able to write files and run commands.

What to verify

• VM-tier sub-Vittos get access to run_command, write_file, read_file
• Sandbox execution produces real output (stdout/stderr)
• Sandbox has timeout protection (30s default)
• Errors are reported cleanly (not raw stack traces)

Prerequisites

• ENABLE_SANDBOX=true in environment
• Either E2B_API_KEY (for Firecracker VMs) or ENABLE_LOCAL_SANDBOX=true (dev only)

10. MongoDB Cleanup Verification

What was cleaned up

• Removed ghost vitto-task-history retention policy from ComplianceService
• The collections channel-user-links and vitto-task-history should be dropped manually in production

How to verify

Step 1: Check the compliance service doesn't reference vitto-task-history:

grep -r "vitto-task-history" APIs/backend-vitto-api/src/

Expected: no results.

Step 2: In MongoDB shell (vitto-mind database):

db.getCollection('channel-user-links').countDocuments()
// Expected: 0 (orphaned, safe to drop)

db.getCollection('vitto-task-history').countDocuments()
// Expected: 0 (no schema ever wrote to this)

Test Environment Checklist

Before starting testing, ensure:

• All 4 backends are running (ports 3000, 3001, 3002, 3003)
• Frontend is running (port 6200 or 6400)
• Redis is running and connected (check backend logs for "Redis connected")
• MongoDB is running with both gograb-api and vitto-mind databases
• Environment variables set:
  • ENABLE_SANDBOX=true (for VM tier testing)
  • OAUTH_ENCRYPTION_KEY (for OAuth testing)
  • GOOGLE_OAUTH_CLIENT_ID / GOOGLE_OAUTH_CLIENT_SECRET (for Google OAuth)
  • GITHUB_OAUTH_CLIENT_ID / GITHUB_OAUTH_CLIENT_SECRET (for GitHub OAuth)

Automated Test Status

All backend tests pass: 1,036 / 1,036 tests across 105 suites.

Run: cd APIs/backend-vitto-api && npx jest --forceExit

Known Limitations (By Design)

1. OAuth auto-apply disabled — Personality drift suggestions are displayed but never auto-applied. This is intentional until real-world validation data is collected.
2. Slack and Notion OAuth — Cards show in UI but Connect will fail (no backend provider config yet). Same pattern as Google/GitHub — just needs config files.
3. Teams full integration — Webhook receives messages but doesn't route to Vitto engine yet. Adapter renders Adaptive Cards. Full Bot Framework registration required for end-to-end.
4. Evolution maturity endpoint — Frontend service calls /engine/evolution/:userId/maturity but the backend controller endpoint may not be exposed yet. The computation logic exists in VittoEvolutionService.
5. Browser tier — Sub-Vittos with tier: 'browser' fall back to in-process. Browser tier requires Computer Use infrastructure which is separate from this work.

Owner: fguijarro Session: S234 Spec: docs/superpowers/specs/2026-03-18-vitto-jarvis-comprehensive-evolution-design.md