Vitto-Mediated Agent Training System — Design Specification

Date: 2026-03-12 Session: S210.3 Status: Draft Supersedes: Direct OpenAI fine-tuning pipeline in TrainingPipelineService

1. Problem Statement

The current agent training system is passive and admin-driven: a human opens the Training Center UI, manually adds examples, clicks "Train", and backend-ai-api validates them. Vitto is completely out of the loop — it doesn't learn from its own successes, doesn't curate its own training data, and can't improve agents through conversation. The pipeline also had an OpenAI fine-tuning dependency that was replaced with few-shot injection (S209), but the architecture still treats training as an external process rather than an intrinsic capability.

Goal: Make Vitto an active participant in training — using conversation, its existing tool access, and its intelligence to observe what works, extract patterns, and improve sub-agents autonomously. All LLM operations flow through Vitto's provider-agnostic LLM layer. The Training Center UI becomes one of two surfaces (alongside chat) that converge on the same Vitto-mediated pipeline.

2. Backend Responsibility Split

backend-vitto-api (port 3003) — Orchestration + Intelligence

Owns: Training decisions, LLM evaluation, conversational teaching, self-learning loops, plugin lifecycle.

New components:

• TrainingPluginRegistry — registers and discovers training plugins
• TrainingOrchestratorService — coordinates plugin execution, routes requests from both UI and chat
• TrainingController — REST endpoints for Training Center UI write operations
• New Vitto tools: train_agent, teach_agent, evaluate_agent, extract_patterns
• SSE training events via existing VittoNarrativeService (conscience + introspection channels)

All LLM calls for training operations use Vitto's existing BaseLLMProvider + ProviderFactory (Claude, OpenAI, Gemini, DeepSeek with automatic failover). No direct provider dependency.

backend-ai-api (port 3001) — Data + Query Layer

Owns: Storage, retrieval, heavy queries, metrics aggregation.

Keeps existing:

• TrainingDataService — CRUD for training examples
• PlatformAgentService — CRUD for agent records
• Pattern mining cron, vector search, learning pyramid queries
• Dashboard read endpoints (stats, agent performance, pattern library)

New endpoints:

• GET /ai-api/platform/training/validation-candidates/:agentId — returns unvalidated examples (quality >= 5, validated = false) for Vitto to evaluate
• POST /ai-api/platform/training/validate-batch — bulk update validation status on examples

Deprecated (orchestration moves to Vitto):

• TrainingPipelineService.startFineTuning() — replaced by Vitto's training plugin. The existing POST /ai-api/platform/agents/:agentId/train route should redirect to POST /vitto-api/training/train/:agentId or return a 410 Gone with migration instructions. Both paths must NOT run simultaneously.
• POST /ai-api/platform/agents/test — replaced by POST /vitto-api/training/evaluate

Follow-on cleanup: The hardcoded qualityScore: isSuccess ? 0.8 : 0.3 in VittoEngineService.postExecutionOutcome() (line ~443) should be replaced with the SelfLearningPlugin's actual LLM-scored quality once Phase 2 is live.

The Routing Rule

3. Training Plugin Architecture

Plugin Interface

interface TrainingPlugin {
  id: string;
  name: string;

  // Tools this plugin adds to Vitto's tool registry
  tools?: ToolHandler[];

  // Hooks into task lifecycle
  onTaskCompleted?: (task: VittoTask, outcome: TaskOutcome) => Promise<void>;
  onDecisionLogged?: (decision: McpDecision) => Promise<void>;
  onSkillExtracted?: (skill: ExtractedSkill) => Promise<void>;

  // REST endpoints for Training Center UI
  endpoints?: TrainingEndpoint[];

  // Plugin lifecycle
  onInit?: () => Promise<void>;
  onDestroy?: () => Promise<void>;
}

Plugin Registry

Uses direct invocation (not an event bus). VittoEngineService injects TrainingPluginRegistry and calls notifyTaskCompleted() directly — matching how SkillExtractionService is already called at task completion. This avoids introducing a pub/sub dependency that doesn't exist in the current architecture.

@Injectable()
class TrainingPluginRegistry implements OnModuleInit {
  private plugins = new Map<string, TrainingPlugin>();

  constructor(
    private toolRegistry: ToolRegistryService,
  ) {}

  register(plugin: TrainingPlugin): void {
    plugin.tools?.forEach(t => this.toolRegistry.register(t));
    this.plugins.set(plugin.id, plugin);
  }

  // Called directly by VittoEngineService after task completion
  // (same pattern as SkillExtractionService.extractIfEligible)
  async notifyTaskCompleted(task: VittoTask, outcome: TaskOutcome): Promise<void> {
    for (const plugin of this.plugins.values()) {
      if (plugin.onTaskCompleted) {
        await plugin.onTaskCompleted(task, outcome).catch(err =>
          this.logger.warn(`Plugin ${plugin.id} onTaskCompleted failed: ${err.message}`)
        );
      }
    }
  }

  // Called by SkillExtractionService after createSkill() succeeds
  // (requires adding a ~2-line callback in extractIfEligible)
  async notifySkillExtracted(skill: ExtractedSkill): Promise<void> {
    for (const plugin of this.plugins.values()) {
      if (plugin.onSkillExtracted) {
        await plugin.onSkillExtracted(skill).catch(err =>
          this.logger.warn(`Plugin ${plugin.id} onSkillExtracted failed: ${err.message}`)
        );
      }
    }
  }

  // Called by MemoryPersistenceService after learnFromMcpDecision() succeeds
  // (requires adding a ~2-line callback in that method)
  async notifyDecisionLogged(decision: McpDecision): Promise<void> {
    for (const plugin of this.plugins.values()) {
      if (plugin.onDecisionLogged) {
        await plugin.onDecisionLogged(decision).catch(err =>
          this.logger.warn(`Plugin ${plugin.id} onDecisionLogged failed: ${err.message}`)
        );
      }
    }
  }

  getPlugin(id: string): TrainingPlugin | undefined {
    return this.plugins.get(id);
  }

  getAllPlugins(): TrainingPlugin[] {
    return Array.from(this.plugins.values());
  }
}

Hook wiring points (3 callsites to add, ~2 lines each):

1. VittoEngineService.processTaskWorkflow() — after this.skillExtraction.extractIfEligible(task) (line ~483), add this.trainingRegistry.notifyTaskCompleted(task, outcome).catch(() => {})
2. SkillExtractionService.extractIfEligible() — after this.skillManager.createSkill() succeeds, add callback to trainingRegistry.notifySkillExtracted(skill)
3. MemoryPersistenceService.learnFromMcpDecision() — after successful processing, add callback to trainingRegistry.notifyDecisionLogged(decision)

Core Plugins

3.1 ConversationalTeachPlugin

Purpose: Parse natural language teaching instructions into structured training examples or knowledge docs for the target agent.

Tool: teach_agent

• Input: { agentId?: string, instruction: string }
• If agentId is omitted, Vitto determines the best target agent from the instruction content using SubAgentService.findBestAgent()
• Uses LLM to classify instruction as either a training example (input/output pair) or a knowledge document (domain fact)
• Persists via BackendAiClientService.submitTrainingExample() or SubAgentService.addKnowledgeDoc()
• Returns: { targetAgent, type, summary, corpusSize }

Endpoint: POST /vitto-api/training/teach

• Same operation as the tool, but callable from Training Center UI
• Body: { agentId?: string, instruction: string, type?: 'example' | 'knowledge' }

Chat integration: The existing /teach slash command and "Teach Vitto" quick action in the Teach plugin trigger Vitto's message processing. The PhaseExecutorService already intercepts teach intents in the ANALYZE phase. ConversationalTeachPlugin provides the teach_agent tool that Vitto selects during execution.

3.2 SelfLearningPlugin

Purpose: Automatically extract training data from successful task executions.

Tools: None (autonomous, triggered by lifecycle hooks only)

Hook — onTaskCompleted:

1. Filter: Only process tasks with status === 'completed' and cost.durationMs > 0
2. Extract the effective prompt (user's original message) and the task's final output
3. Use a cheap LLM model (getForPhase('VERIFY') — Haiku/GPT-4o-mini) to score quality on a 1-10 rubric:
   • Correctness (does it solve the stated problem?)
   • Completeness (are all requirements addressed?)
   • Convention adherence (follows platform patterns?)
   • Code quality (if code output: compilable, typed, testable?)
4. If score >= 7: auto-submit as training example via BackendAiClientService
5. If score 5-6.9: add to human review queue (stored on ai-api with reviewStatus: 'pending')
6. If score < 5: discard, log to introspection narrative
7. Tag all self-learned examples: source: 'vitto-self-learning'

Hook — onDecisionLogged:

• When MCP logs a decision with confidence >= 0.9 AND a successful outcome is later reported, extract the decision as a knowledge doc for the relevant agent type.

Endpoint: GET /vitto-api/training/self-learning/stats

• Returns: { totalLearned, todayCount, reviewQueueSize, rejectedToday, topAgentLearner }

3.3 AgentEvaluationPlugin

Purpose: Test a sub-agent with a real task and score the output using LLM-as-judge.

Tool: evaluate_agent

• Input: { agentId: string, prompt: string, expectedBehavior?: string }
• Isolation: Does NOT call SubAgentService.executeTask() directly (that would create a circular module dependency). Instead, submits a bounded task to Vitto's TaskQueueService with type: 'training-eval'. The SelfLearningPlugin's onTaskCompleted hook explicitly filters out task.type === 'training-eval' to prevent cascading (evaluation → self-learn → evaluation → ...).
• The evaluation task runs through the normal 4-phase pipeline with a reduced cost budget (max 5 iterations, 25K tokens)
• After completion, a separate LLM call judges the output quality (0-10 scale)
• Records test result on ai-api via BackendAiClientService (not PlatformAgentService directly — avoids circular import)
• Returns: { agentId, quality, output, compilable, errors, duration }

Endpoint: POST /vitto-api/training/evaluate

• Same operation via REST for Training Center UI
• Replaces the stubbed POST /ai-api/platform/agents/test

3.4 PatternExtractionPlugin

Purpose: Analyze recent successful tasks and extract reusable patterns that improve all agents.

Tool: extract_patterns

• Input: { agentId?: string, taskCount?: number }
• Fetches recent successful task outcomes from ai-api
• Uses LLM to cluster and generalize common success patterns
• Submits new patterns to ai-api's pattern library

Hook — onSkillExtracted:

• When SkillExtractionService produces a skill, determine which sub-agents should learn from it
• Cross-reference skill domain tags with agent specializations
• Add relevant skills as knowledge docs to matching agents

Endpoint: POST /vitto-api/training/patterns/extract

• Replaces POST /ai-api/platform/patterns/extract (same operation, now LLM-mediated)

4. Frontend Integration

Training Center UI (McpService Changes)

The existing McpService in @libs-ai/ai-agents-training-center is modified to route write operations through Vitto:

Reads stay on ai-api (fast, no LLM overhead):

• GET /ai-api/platform/agents — agent list + stats
• GET /ai-api/platform/training/examples — browse examples
• GET /ai-api/platform/patterns — pattern library
• GET /ai-api/platform/stats — system stats

Writes move to vitto-api (LLM-mediated):

• POST /vitto-api/training/teach — submit training example (was POST /ai-api/platform/training/examples)
• POST /vitto-api/training/train/{agentId} — trigger training (was POST /ai-api/platform/agents/{agentId}/train)
• POST /vitto-api/training/evaluate — test agent (was POST /ai-api/platform/agents/test)
• POST /vitto-api/training/patterns/extract — extract patterns (was POST /ai-api/platform/patterns/extract)

Impact: Add VITTO_API_URL injection (already imported). Route 4 write methods to Vitto. Read methods unchanged. ~20 lines of code change in McpService.

Chat Integration (Teach Plugin Enhancement)

The existing createTeachPlugin() in @libs-ai/vitto-chat already provides:

• /teach slash command
• "Teach Vitto" quick action

Enhancement: Add parameterized slash command /teach {agent} {instruction} (e.g., /teach angular-agent always use OnPush). The message is processed by Vitto's engine, which uses ConversationalTeachPlugin's teach_agent tool. Response shows what was learned.

No new frontend plugin needed — the existing Teach plugin surface is sufficient. Intelligence moves to backend.

SSE Training Events

Training progress flows through existing VittoNarrativeService:

5. Self-Learning Safety

Quality Gate

Every self-learned example goes through two-step validation:

1. Automated LLM scoring — Vitto evaluates task output against original prompt on 1-10 rubric (correctness, completeness, convention adherence, code quality). Uses cheap model via getForPhase('VERIFY').
2. Threshold gate:
   • Score >= 7: auto-accepted
   • Score 5-6.9: human review queue
   • Score < 5: discarded

Rate Limiting

Rate limit storage: Counters stored in Redis (already available via RedisService for BullMQ). In-memory counters reset on deployment and are insufficient for cost control. Keys: training:rate:{agentId}:hour, training:rate:{agentId}:day with TTL-based expiry.

Anti-Pattern Detection

• Duplicate detection: Semantic similarity check against existing corpus via ai-api vector search. Skip if similarity > 0.92.
• Regression guard: If agent success rate drops after recent self-learned examples, flag those examples for review and pause self-learning for that agent.
• Circular learning detection: Track source field on every example. If Vitto generates output → trains on it → generates similar output, detect the cycle and break it. Max 30% of any agent's corpus can be self-learned.
• Corpus diversity: No single topic/pattern can exceed 20% of an agent's total examples.

Human Override

• Training Center UI shows "Review Queue" tab for borderline examples (score 5-6.9)
• Any admin can approve/reject via REST
• Global toggle: TRAINING_SELF_LEARNING_ENABLED=true/false
• Per-agent toggle via agent definition: selfLearningEnabled: boolean
• All self-learned examples tagged source: 'vitto-self-learning' for auditability

Cost Budget

Self-learning LLM calls tracked under training category in existing CostGuard. Default: $5/day. When exhausted, self-learning pauses — Vitto keeps working, just stops learning. Resets daily.

6. New Backend Endpoints

backend-vitto-api (Training Controller)

backend-ai-api (New Data Endpoints)

7. New Vitto Tools

8. Data Flow Diagrams

Conversational Teaching Flow

User → Vitto Chat: "/teach angular-agent always use OnPush"
  → PhaseExecutorService: detects teach intent
  → AgentLoopService: selects teach_agent tool
  → ConversationalTeachPlugin.teach_agent():
    → LLM classifies as knowledge doc
    → BackendAiClientService.submitTrainingExample() → ai-api stores
    → SubAgentService.addKnowledgeDoc() → vitto-mind stores
  → Response: "Added to Angular Agent's knowledge: 'Always use OnPush...'"
  → VittoNarrativeService.emitConscience('conscience.training.taught')

Self-Learning Flow

VittoEngineService: task completed successfully
  → Emits 'task.completed' event
  → SelfLearningPlugin.onTaskCompleted():
    → Extract prompt + output from task
    → Rate limit check (10/hour, 50/day/agent)
    → LLM scoring via ProviderFactory.getForPhase('VERIFY')
    → Quality >= 7: BackendAiClientService.submitTrainingExample()
    → Quality 5-6.9: Store with reviewStatus: 'pending'
    → Quality < 5: Discard
    → VittoNarrativeService.emitIntrospection('introspection.self-learning')

Training Center UI Flow

User clicks "Train" on angular-agent
  → POST /vitto-api/training/train/angular-agent
  → TrainingOrchestratorService:
    → BackendAiClientService.getValidationCandidates('angular-agent')
    → For each candidate: LLM validates quality
    → BackendAiClientService.validateBatch(validatedIds)
    → BackendAiClientService.updateAgentStatus('angular-agent', 'trained')
    → VittoNarrativeService.emitConscience('conscience.training.completed')
  → Response: { success, validatedCount, totalCount, avgQuality }

9. File Inventory

New Files (backend-vitto-api)

Modified Files (backend-vitto-api)

Modified Files (backend-ai-api)

Modified Files (Frontend)

10. Implementation Phases

Phase 1 — Foundation (Gateway + Plugin Infra)

• TrainingPlugin interface + TrainingPluginRegistry
• TrainingModule + TrainingController
• ConversationalTeachPlugin (simplest plugin, proves the architecture)
• Frontend McpService routing changes
• 2 new ai-api convenience endpoints
• Deliverable: Teaching via both chat and UI works through Vitto

Phase 2 — Self-Learning Loop

• SelfLearningPlugin with task completion hooks
• Quality scoring via LLM (cheap model)
• Rate limiting + duplicate detection
• Narrative SSE events for training activity
• Review queue (score 5-6.9)
• Deliverable: Vitto learns from successful task executions

Phase 3 — Agent Evaluation

• AgentEvaluationPlugin (replaces stubbed test endpoint)
• LLM-as-judge pattern for output scoring
• Success rate tracking wired to outcome pipeline
• Deliverable: "Test Agent" button produces real results

Phase 4 — Pattern Extraction + Cross-Teaching

• PatternExtractionPlugin
• Cross-agent knowledge transfer: when a pattern is extracted, the plugin iterates matching agent definitions (by tag/specialization overlap) and calls SubAgentService.addKnowledgeDoc() on each. This loop IS the cross-teaching mechanism — no separate triggerCrossTraining() method needed.
• Regression guard implementation
• Corpus diversity enforcement
• Deliverable: Agents learn from each other's successes

11. Success Criteria

1. All training write operations route through Vitto (zero direct ai-api writes from UI)
2. Self-learning produces >= 5 valid training examples per day from organic task execution
3. Agent success rates improve measurably within 2 weeks of self-learning activation
4. No circular learning detected (< 30% self-learned examples per agent)
5. Training cost stays within $5/day budget
6. "Test Agent" returns real quality scores (not hardcoded 0.85)
7. Chat /teach command successfully structures instructions into agent training data

12. Risks and Mitigations

13. Relationship to Vitto Intelligence v2 Spec

This design implements Subsystem 1: Skills Closed Loop from the Vitto Intelligence v2 spec and advances it from 70% to ~95% completion by:

• Adding the missing runtime skill extraction → training example pipeline (SelfLearningPlugin)
• Replacing the passive admin-driven training with Vitto-mediated orchestration
• Establishing the plugin architecture that Subsystems 2-7 can build on

It also partially implements Subsystem 7: Execution Consolidation by routing all training LLM calls through Vitto's unified provider layer.