🏗️ MCP Architecture Deep Dive

Technical implementation details of the most advanced AI learning system ever built.

🌐 Session 16 Updates (December 2025)

See: Multi-Platform Integration for complete details on cross-platform learning.

🚀 Session 10 Updates (December 2025)

🎯 MCP Init Dashboard

The MCP now includes an intelligent startup dashboard (mcp_init) for each task:

Features

• System Health Check: Verifies backend, MCP, and database connectivity
• Quick Stats: Decisions, embeddings, coverage %, golden count
• Mode Selection: Quick Task vs Multi-Agent Orchestration
• Auto-Select: Defaults to recommended mode after 5 seconds
• Task Complexity Detection: Auto-suggests orchestration for complex tasks
• Recent Context: Shows last 3 decisions
• Pending Outcomes: Reminds about decisions needing feedback

Modes

UX Features

• 5-second auto-select: Proceeds with default if no response
• Double ENTER: Quick confirm for default selection
• Smart suggestion: Complex tasks auto-suggest orchestration

See DATA-ARCHITECTURE-CONSOLIDATION.md for full details.

🎯 System Architecture Overview

For high-level system overview, see MCP Index. For implementation action plan, see Next Steps & Improvements.

Core Design Principles

Architecture Diagram

┌─────────────────────────────────────────────────────────────────────────────────┐
│                            MCP SYSTEM v4.0                                     │
├─────────────────────────────────────────────────────────────────────────────────┤
│  ┌─────────────────────────────────────────────────────────────────────────┐   │
│  │                    THREE-TIER MEMORY SYSTEM                             │   │
│  │  ┌─────────────────────────────────────────────────────────────────┐   │   │
│  │  │  HOT: Current session (always in memory)                      │   │   │
│  │  │  • Frequently accessed decisions                                │   │   │
│  │  │  • Access count >= threshold                                    │   │   │
│  │  │  • Always available, no loading required                       │   │   │
│  │  └─────────────────────────────────────────────────────────────────┘   │   │
│  │                                                                         │   │
│  │  ┌─────────────────────────────────────────────────────────────────┐   │   │
│  │  │  WARM: Recent sessions (loaded on demand)                     │   │   │
│  │  │  • Last 7 days by default                                      │   │   │
│  │  │  • Successful decisions from older sessions                   │   │   │
│  │  │  • Cached with TTL for performance                            │   │   │
│  │  └─────────────────────────────────────────────────────────────────┘   │   │
│  │                                                                         │   │
│  │  ┌─────────────────────────────────────────────────────────────────┐   │   │
│  │  │  COLD: Historical (MongoDB/docs/data only)                     │   │   │
│  │  │  • Rarely accessed historical data                             │   │   │
│  │  │  • Full archival storage                                       │   │   │
│  │  │  • Vector search for semantic retrieval                        │   │   │
│  │  └─────────────────────────────────────────────────────────────────┘   │   │
│  └─────────────────────────────────────────────────────────────────────────┘   │
│                                                                                 │
│  ┌─────────────────────────────────────────────────────────────────────────┐   │
│  │                    AGENT TRAINING SYSTEM                                │   │
│  │  ┌─────────────────────────────────────────────────────────────────┐   │   │
│  │  │  • Per-agent success/failure tracking                         │   │   │
│  │  │  • Confidence calibration algorithms                          │   │   │
│  │  │  • Pattern extraction from successful decisions               │   │   │
│  │  │  • Anti-pattern learning from failures                        │   │   │
│  │  └─────────────────────────────────────────────────────────────────┘   │   │
│  └─────────────────────────────────────────────────────────────────────────┘   │
│                                                                                 │
│  ┌─────────────────────────────────────────────────────────────────────────┐   │
│  │                    MCP TOOLS & API                                     │   │
│  │  ┌─────────────────────────────────────────────────────────────────┐   │   │
│  │  │  9 Available Tools (Multi-Platform):                           │   │   │
│  │  │  • preflight: Intelligent context (~100 tokens)               │   │   │
│  │  │  • log_decision: Decision logging with metadata               │   │   │
│  │  │  • find_similar: Semantic search                                │   │   │
│  │  │  • report_outcome: Success/failure feedback                    │   │   │
│  │  │  • get_conventions: Learned patterns                           │   │   │
│  │  │  • add_convention: Add new patterns                            │   │   │
│  │  │  • get_stats: System analytics + platform breakdown           │   │   │
│  │  │  • get_session_info: Session tracking                          │   │   │
│  │  │  • get_platform_info: Platform capabilities (NEW)             │   │   │
│  │  └─────────────────────────────────────────────────────────────────┘   │   │
│  └─────────────────────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────────────────┘

🏗️ 3-Tier Decision Pipeline (NEW in Session 10)

┌─────────────────────────────────────────────────────────────────┐
│                    3-TIER DECISION PIPELINE                      │
└─────────────────────────────────────────────────────────────────┘

   TIER 1: RAW STORAGE (ai-agents.decisions)
   ┌──────────────────────────────────────┐
   │  All MCP decisions                   │  ← MCP log_decision() writes here
   │  - 9,957+ documents                  │
   │  - Fast writes, no filtering         │
   └────────────────────┬─────────────────┘
                        │ AUTO-EMBED on create ✅
                        ▼
   TIER 2: SEARCHABLE (ai-memory.agent_embeddings)
   ┌──────────────────────────────────────┐
   │  Decisions + vector embeddings       │  ← Enables semantic search
   │  - 6,004+ documents (60%+)           │
   │  - 1536-dimension vectors            │
   │  - MongoDB Atlas Vector Search       │
   └────────────────────┬─────────────────┘
                        │ SMART PROMOTE ✅
                        ▼
   TIER 3: CURATED (ai-agents.golden_decisions)
   ┌──────────────────────────────────────┐
   │  BEST decisions only                 │  ← Quality-filtered knowledge
   │  - High confidence (>= 0.9)          │
   │  - Successful outcomes               │
   │  - Frequently referenced (3+)        │
   │  - Quality score 0-100               │
   └──────────────────────────────────────┘

Smart Promotion Criteria

Decision Flow

User Action → MCP log_decision()
                 ↓
            1. Add to memoryStore (prepend for newest-first)
            2. Increment sessionID (auto-sync to all files)
            3. POST to backend-ai-api/infrastructure/decisions
                 ↓
            4. Backend saves to ai-agents.decisions
            5. Backend auto-generates embedding
            6. Backend triggers adaptive learning checks
                 ↓
            7. MCP saves to .cursor/ai-memory/current-session.json

🧠 Memory System Architecture

Tier Determination Logic

export function determineDecisionTier(decision: DevelopmentDecision): MemoryTier {
  const now = Date.now();
  const decisionTime = new Date(decision.timestamp).getTime();
  const daysSince = (now - decisionTime) / (1000 * 60 * 60 * 24);

  // Current session = always hot
  const currentSession = getCurrentSessionNumber();
  if (decision.session?.sessionNumber === currentSession) {
    return 'hot';
  }

  // High access count = hot
  if ((decision.accessCount || 0) >= TIER_THRESHOLDS.HOT_ACCESS_COUNT) {
    return 'hot';
  }

  // Recent = warm
  if (daysSince < TIER_THRESHOLDS.WARM_AFTER_DAYS) {
    return 'warm';
  }

  // Old but successful = warm
  if (decision.outcome?.wasSuccessful && daysSince < TIER_THRESHOLDS.COLD_AFTER_DAYS) {
    return 'warm';
  }

  // Everything else = cold
  return 'cold';
}

Tier Thresholds Configuration

export const TIER_THRESHOLDS = {
  HOT_ACCESS_COUNT: 10,      // Access count threshold for HOT tier
  WARM_AFTER_DAYS: 7,        // Days after which decisions become WARM
  COLD_AFTER_DAYS: 30,       // Days after which decisions become COLD
} as const;

Working Memory Building

export function buildWorkingMemory(sessionNumber?: number): DecisionSummary[] {
  const targetSession = sessionNumber || getCurrentSessionNumber();
  const sessionDecisions = getDecisionsBySession(targetSession);

  // Get last N decisions from session
  const recentDecisions = sessionDecisions.slice(-MAX_WORKING_MEMORY);

  // Convert to summaries (ultra-compressed for token efficiency)
  const summaries = createSummaries(recentDecisions);

  // Update working memory state
  workingMemory.currentSession = targetSession;
  workingMemory.summaries = summaries;

  return summaries;
}

🤖 Agent Training System

Agent Knowledge Structure

interface AgentKnowledge {
  agentId: string;
  specialization: string[];        // Top keywords for this agent
  successfulPatterns: LearnedPattern[];
  antiPatterns: string[];          // Things to avoid
  confidenceScore: number;         // 0-1 calibration score
  decisionCount: number;
  successRate: number;
}

Pattern Extraction Algorithm

function extractPatterns(decisions: DevelopmentDecision[]): LearnedPattern[] {
  if (decisions.length < 2) return [];

  // Group by decision type
  const byType = new Map<string, DevelopmentDecision[]>();
  for (const d of decisions) {
    const existing = byType.get(d.type) || [];
    existing.push(d);
    byType.set(d.type, existing);
  }

  const patterns: LearnedPattern[] = [];

  for (const [type, typeDecisions] of byType) {
    if (typeDecisions.length < 2) continue;

    // Find common themes in reasoning
    const reasonings = typeDecisions.map(d => d.reasoning).filter(Boolean);
    const commonWords = findCommonWords(reasonings);

    if (commonWords.length > 0) {
      patterns.push({
        id: `pattern-${type}-${Date.now()}`,
        name: `${type.charAt(0).toUpperCase() + type.slice(1)} Pattern`,
        description: `Common approach for ${type} decisions`,
        whenToUse: commonWords.slice(0, 5).join(', '),
        implementation: typeDecisions[0].chosen,
        successRate: 1.0,
        usageCount: typeDecisions.length,
        lastUsed: new Date(),
      });
    }
  }

  return patterns;
}

Confidence Calibration

function calculateConfidenceCalibration(decisions: DevelopmentDecision[]): number {
  const withOutcomes = decisions.filter(d =>
    d.outcome?.wasSuccessful !== undefined &&
    d.confidence !== undefined
  );

  if (withOutcomes.length < 3) return 0.5; // Default

  // Compare predicted confidence vs actual success rate
  let totalError = 0;

  for (const d of withOutcomes) {
    const predicted = d.confidence;
    const actual = d.outcome?.wasSuccessful ? 1 : 0;
    totalError += Math.abs(predicted - actual);
  }

  const meanError = totalError / withOutcomes.length;

  // Convert error to score (lower error = higher score)
  return Math.max(0, 1 - meanError);
}

🛠️ MCP Tools Architecture

Tool Interface Design

interface MCPTool {
  name: string;
  description: string;
  inputSchema: {
    type: 'object';
    properties: Record<string, any>;
    required: string[];
  };
}

Preflight Tool (Most Critical)

Purpose: Provides intelligent context before any task Token Usage: ~100 tokens (vs ~2000 in naive approaches) Cache TTL: 5 minutes Loop Protection: Max 20 calls per minute

export async function handlePreflight(params: PreflightParams): Promise<PreflightResult> {
  // 1. Loop detection
  const loopCheck = checkForLoop();
  if (loopCheck.blocked) {
    return getMinimalResponse();
  }

  // 2. Check cache (5-minute TTL)
  const cacheKey = `v4_${sessionNum}_${lite ? 'lite' : 'full'}`;
  const cached = preflightCache.get(cacheKey);

  // 3. Build ultra-compressed context
  const workingMemory = buildWorkingMemorySummaries(sessionNum, lite);
  const historicalHints = getHistoricalHints(sessionNum, lite);
  const conventions = getMatchingConventions(task, lite);

  return {
    task: truncate(params.task, 40),
    session: currentSession,
    workingMemory,
    historicalHints,
    conventions,
    guidance,
    tokenBudget: { used: estimateTokens(result), max: maxTokens },
    _note: 'Use find_similar() for detailed context.',
  };
}

Decision Logging Schema

interface LogDecisionParams {
  type: 'architecture' | 'implementation' | 'refactor' | 'bugfix' | 'pattern' | 'convention';
  question: string;           // What decision was being made?
  options?: string[];         // What alternatives were considered?
  chosen: string;             // What was chosen?
  reasoning: string;          // Why was this chosen? (MOST IMPORTANT)
  confidence: number;         // Confidence in this decision (0-1)
  files?: string[];           // Relevant files involved
  task?: string;              // Current task or user request
  projectId?: string;         // For external agents
  taskGroup?: string;         // For external agents
}

Outcome Reporting (Critical Gap)

interface ReportOutcomeParams {
  decisionId: string;
  wasSuccessful: boolean;
  feedback?: string;
  rating?: number;  // 1-5 scale
}

📊 Data Architecture

Decision Document Schema

interface DevelopmentDecision {
  id: string;
  timestamp: string;

  // Decision metadata
  type: string;
  question: string;
  options?: string[];
  chosen: string;
  reasoning: string;
  confidence: number;

  // Context
  files?: string[];
  task?: string;
  session?: SessionMetadata;

  // Agent tracking (platform-agnostic)
  agentId: string;  // e.g., "architecture-agent" (legacy "cursor-*" auto-normalized)

  // Learning data
  outcome?: {
    wasSuccessful: boolean;
    feedback?: string;
    rating?: number;
    reportedAt: string;
  };

  // Memory management
  tier: MemoryTier;
  accessCount: number;
  lastAccessed: string;

  // Embeddings for semantic search
  embedding?: number[];
}

interface SessionMetadata {
  type: PlatformType;  // 'cursor' | 'claude-code' | 'claude-desktop' | 'api' | 'external'
  sessionNumber?: number;
  sessionID?: string;
  projectId?: string;
  taskGroup?: string;
  platform?: PlatformMetadata;  // NEW: Extended platform information
}

interface PlatformMetadata {
  type: PlatformType;
  version?: string;      // Platform version
  model?: string;        // AI model (e.g., "claude-opus-4-5-20251101")
  environment?: string;  // Environment type (e.g., "vscode", "cursor", "terminal")
  clientId?: string;     // Unique client session ID
}

Database Architecture

MongoDB Cluster
├── ai-agents (TIER 2 - Platform)
│   ├── agent_tasks         # Task history
│   ├── agent_decisions     # Platform-wide decisions
│   ├── agent_embeddings    # Vector search
│   ├── agent_patterns      # Learned patterns
│   └── agent_calibrations  # Confidence calibration
│
├── ai-memory (Shared)
│   ├── decisions          # All development decisions
│   ├── sessions           # Session tracking
│   └── embeddings         # Vector embeddings
│
└── tenant_{tenantId}_ai (TIER 3 - Tenant)
    ├── user_profiles      # Per-user preferences
    ├── user_interactions  # Per-user history
    ├── user_embeddings    # Per-user vectors
    └── user_patterns      # Learned user patterns

🔄 Data Flow Architecture

Decision Logging Flow

1. Developer/AI makes decision
2. log_decision() called with metadata
3. Decision stored in memory store
4. Decision assigned to appropriate tier
5. Decision synced to MongoDB
6. Decision embedded for semantic search
7. Agent knowledge updated
8. Patterns extracted if successful

Preflight Context Flow

1. preflight() called with task description
2. Check loop detection (max 20/minute)
3. Check cache (5-minute TTL)
4. Build working memory (current session summaries)
5. Get historical hints (successful from other sessions)
6. Retrieve matching conventions
7. Generate guidance
8. Return compressed context (~100 tokens)

Outcome Learning Flow

1. Task completed
2. report_outcome() called with success/failure
3. Decision updated with outcome
4. Agent knowledge recalibrated
5. Success rate updated
6. Confidence calibration improved
7. Patterns reinforced or anti-patterns learned

🚨 Critical Issues & Fixes

Issue 1: Outcome Reporting Gap (CRITICAL)

Problem: 0 outcomes reported despite 51 decisions logged Impact: No learning feedback loop - agents can't improve Solution:

// Auto-prompt for outcomes after decisions
async function promptForOutcome(decisionId: string): Promise<void> {
  const outcome = await vscode.window.showQuickPick([
    { label: '✅ Successful', value: { wasSuccessful: true } },
    { label: '❌ Failed', value: { wasSuccessful: false } },
  ]);

  if (outcome) {
    await report_outcome({
      decisionId,
      wasSuccessful: outcome.value.wasSuccessful,
      feedback: await getFeedback(),
    });
  }
}

Issue 2: Session Tracking Broken

Problem: WORKSPACE_ROOT detection fails in compiled dist/ Impact: Session continuity lost Solution:

// Dynamic workspace root detection
export function getWorkspaceRoot(): string {
  // Try multiple detection methods
  const methods = [
    () => process.env.WORKSPACE_ROOT,
    () => path.resolve(__dirname, '..', '..', '..', '..'),
    () => path.resolve(process.cwd(), '..', '..'),
    () => findPackageJsonRoot(),
  ];

  for (const method of methods) {
    try {
      const root = method();
      if (root && fs.existsSync(path.join(root, 'nx.json'))) {
        return root;
      }
    } catch (error) {
      continue;
    }
  }

  throw new Error('Could not determine workspace root');
}

Issue 3: Success Rate Stuck at 50%

Problem: All agents show 50% success rate (default) Impact: No real performance differentiation Solution: Implement outcome validation workflow

📈 Performance Optimizations

Token Efficiency

Cache Strategy

const CACHE_CONFIG = {
  PREFLIGHT_TTL: 5 * 60 * 1000,      // 5 minutes
  WORKING_MEMORY_TTL: 10 * 60 * 1000, // 10 minutes
  HISTORICAL_HINTS_TTL: 30 * 60 * 1000, // 30 minutes
  CONVENTIONS_TTL: 60 * 60 * 1000,     // 1 hour
} as const;

Memory Management

const MEMORY_LIMITS = {
  MAX_WORKING_MEMORY: 10,      // Recent decisions in working memory
  MAX_HISTORICAL_HINTS: 5,     // Hints from other sessions
  MAX_CONVENTIONS: 7,          // Active conventions
  MAX_CACHE_SIZE: 100,         // Cache entries
} as const;

🔧 Implementation Details

Server Architecture

// MCP Server v4.0 - Modular Architecture
class MCPServer {
  private readonly tools: MCPTool[] = [
    new PreflightTool(),
    new DecisionLogger(),
    new SimilarityFinder(),
    new OutcomeReporter(),
    new ConventionManager(),
    new StatisticsProvider(),
    new SessionTracker(),
  ];

  async initialize(): Promise<void> {
    await this.initializeStorage();
    await this.loadVitalData();
    await this.startAgentTraining();
    await this.setupMemoryTiering();
  }

  async handleToolCall(toolName: string, params: any): Promise<any> {
    const tool = this.tools.find(t => t.name === toolName);
    if (!tool) throw new Error(`Unknown tool: ${toolName}`);

    return await tool.execute(params);
  }
}

Storage Integration

class MCPStorage {
  private memoryStore = new MemoryStore();
  private fileStorage = new FileStorage();
  private docsIntegration = new DocsDataIntegration();

  async storeDecision(decision: DevelopmentDecision): Promise<void> {
    // Store in memory for immediate access
    this.memoryStore.add(decision);

    // Store in files for persistence
    await this.fileStorage.save(decision);

    // Store in docs/data for session tracking
    await this.docsIntegration.save(decision);

    // Store in MongoDB for long-term analysis
    await this.mongoStorage.save(decision);
  }
}

🎯 Competitive Advantages

Technical Superiority

Scalability

• Horizontal: Multi-tenant architecture with isolated data
• Vertical: Memory tiering prevents context bloat
• Performance: Cached responses and background processing
• Reliability: Outcome validation and error recovery

🔮 Future Enhancements

Phase 5: Advanced Intelligence

1. Self-Improvement Automation

   • Agents that analyze their own decision patterns
   • Automatic confidence threshold adjustment
   • Pattern effectiveness tracking

2. Cross-Tier Intelligence

   • Platform patterns become tenant defaults
   • Development learnings improve user experience
   • Federated learning across tenants

3. Predictive Capabilities

   • Predict likely success of decisions
   • Suggest optimal agents for tasks
   • Early warning for likely-to-fail approaches

4. Advanced Analytics

   • Real-time success rate monitoring
   • Pattern effectiveness visualization
   • Agent performance optimization

This architecture represents the most sophisticated AI learning system ever implemented. The MCP system fundamentally changes how AI agents learn and improve over time.

Multi-Platform Architecture: All platforms (Cursor, Claude Code, Claude Desktop, API) share the same decision database, agent knowledge, and learning algorithms for truly unified intelligence across all AI tools.

Last Updated: December 28, 2025 | Session: 16.8