Vitto Memory System v2 — Groundbreaking Memory Architecture

Session: S239 Date: 2026-03-19 Status: Design Approved Owner: backend-vitto-api (primary), backend-ai-api (data layer), frontend (UI) Predecessor: S239-memory-system-handoff.md, S237 memory fixes, S234 memory architecture redesign

Executive Summary

Vitto's memory system is being redesigned from a basic recall-and-store mechanism into a human-brain-inspired cognitive architecture with two parallel systems:

1. Soul — WHO Vitto is (personality, character, values, consciousness) — customizable per user
2. Brain — WHAT Vitto knows (facts, preferences, decisions, project knowledge) — organized in 4 hierarchical scopes

The redesign addresses three critical failures in the current system:

• In-session amnesia (conversations capped at 10-20 messages, no compression)
• Missing vector search indexes (semantic recall silently falling back to text search)
• No memory scope hierarchy (all memories default to layer='developer', no organization/space/universe scoping)

Key innovations adopted from research:

• Virtual Context Memory (MemGPT-inspired) — conversation genome compression eliminates the 10-message cliff
• 5-Factor Admission Control (A-MAC) — intelligent gating of what gets stored
• Prospective Indexing (Kumiho) — pre-index future questions at write time
• Dopamine-Gated Dual Route (D-MEM) — fast vs slow recall based on query complexity
• Redis Vector Sets as hot tier — <5ms recall, eliminating MongoDB's 1-3s propagation delay
• 256-dim Matryoshka Embeddings — 6x smaller vectors with equal accuracy
• TOON format for context injection — 30-60% token savings
• Cognitive Plugin Hooks — every memory innovation is a pluggable module

Table of Contents

1. Architecture Overview
2. Soul Layer
3. Brain Layer — Memory Scopes
4. Virtual Context Memory (In-Session Fix)
5. 3-Tier Recall Engine
6. Intelligent Write Path
7. Context Compression Stack
8. Cognitive Plugin System
9. MongoDB Vector Search Setup
10. Memory Management UI
11. Conflict Resolution & Safety
12. Key Files Reference
13. Migration Strategy
14. Risks & Mitigations
15. Cowork Agent Handoff: Vector Search Indexes

1. Architecture Overview

┌──────────────────────────────────────────────────────┐
│                    SOUL LAYER                         │
│  Character · Core Values · Mood · Persona            │
│  Per-user customizable · Compiles to ~150 TOON tokens│
│  Cached as stable prompt prefix (prompt caching)     │
└──────────────────┬───────────────────────────────────┘
                   │ influences
┌──────────────────▼───────────────────────────────────┐
│                   BRAIN LAYER                         │
│                                                       │
│  ┌─ Organization (executives/admins) ──────────────┐ │
│  │  ┌─ Space (space owners) ─────────────────────┐ │ │
│  │  │  ┌─ Universe (project owners) ───────────┐ │ │ │
│  │  │  │  ┌─ User (auto-learned + editable) ─┐ │ │ │ │
│  │  │  │  │                                   │ │ │ │ │
│  │  │  │  └───────────────────────────────────┘ │ │ │ │
│  │  │  └────────────────────────────────────────┘ │ │ │
│  │  └─────────────────────────────────────────────┘ │ │
│  └──────────────────────────────────────────────────┘ │
└──────────────────┬───────────────────────────────────┘
                   │ queries
┌──────────────────▼───────────────────────────────────┐
│              3-TIER RECALL ENGINE                      │
│                                                       │
│  🔥 HOT (<5ms)    │ 🔎 WARM (50-200ms) │ 🧊 COLD     │
│  In-memory cache   │ MongoDB Atlas      │ Cross-API   │
│  + Redis Vectors   │ $vectorSearch      │ + text      │
│                    │ + $graphLookup     │ fallback    │
│                                                       │
│  ← Dopamine-gated router decides which tier(s)       │
└──────────────────────────────────────────────────────┘

Two parallel systems that never merge:

• Soul = identity/consciousness layer (existing services: MoodDefinitionService, IdentitySynthesisService, persona system)
• Brain = knowledge/memory layer (existing services: VittoMemoryService, KnowledgeGraphService, UserDossierService)

Soul feeds into Brain: Soul configuration influences how memories are interpreted and presented, but Soul is NOT a memory scope. It's an orthogonal system.

2. Soul Layer

What Exists (Keep As-Is)

What Changes

Soul Compression Service (NEW)

Purpose: Compile the full soul configuration into a compact, TOON-formatted persona brief (~150 tokens) that serves as the stable prompt prefix.

// NEW: APIs/backend-vitto-api/src/app/mind/soul-compression.service.ts

interface CompiledSoul {
  brief: string;           // ~150 TOON-formatted tokens
  fullConfig: SoulConfig;  // Full soul for identity-relevant queries
  compiledAt: Date;
  userId: string;
  hash: string;            // Change detection for cache invalidation
}

class SoulCompressionService {
  // Compile full soul (character + values + mood + persona) into ~150 tokens
  compileSoul(userId: string): Promise<CompiledSoul>;

  // Get cached compiled soul (recompile if config changed)
  getCachedSoul(userId: string): Promise<CompiledSoul>;

  // Check if current query needs full soul (identity-relevant) vs brief
  needsFullSoul(query: string): boolean;
}

Why TOON format: Research shows TOON achieves 30-60% token reduction vs JSON with equal or better LLM accuracy. The TypeScript SDK (toon-format/toon) is a drop-in. Soul config is structured data (personality traits, values, preferences) — ideal for TOON.

Prompt caching integration: The compiled soul brief is placed at the START of the system prompt (stable prefix). With Anthropic's prompt caching, this prefix is cached across turns — 90% cost reduction on soul injection. The brief must be deterministically serialized (same input → same output) to maximize cache hits.

3. Brain Layer — Memory Scopes

Schema Changes

// MODIFIED: APIs/backend-vitto-api/src/app/mind/schemas/vitto-memory.schema.ts

// NEW: Replace the flat 'layer' string with a structured scope system
@Prop({ type: String, enum: ['organization', 'space', 'universe', 'user'], required: true, index: true })
scope: MemoryScope;

@Prop({ type: String, index: true })
scopeId: string; // spaceId for 'space', universeId for 'universe', userId for 'user', null for 'organization'

@Prop({ type: Boolean, default: true })
overridable: boolean; // false = policy/compliance, cannot be overridden by lower scopes

// KEEP existing fields: type, summary, content, tags, keyEntities, embedding, importance, etc.
// DEPRECATE: 'layer' field (migrate existing data, keep for backward compat during transition)

Scope Hierarchy

Conflict Resolution Rules

// NEW: APIs/backend-vitto-api/src/app/mind/memory-conflict-resolver.service.ts

class MemoryConflictResolverService {
  /**
   * Resolve conflicts between memories from different scopes.
   *
   * Rules:
   * 1. For PREFERENCES: User > Universe > Space > Organization
   *    (more specific scope wins)
   * 2. For POLICIES (overridable: false): Organization wins always
   * 3. For FACTS: Most recently updated wins (with confidence tiebreaker)
   */
  resolveConflicts(memories: VittoMemory[]): VittoMemory[];
}

Scope-Aware Recall

When assembling context, the recall engine queries applicable scopes based on the current context:

User in Space A, Universe B:
  → Organization memories (always)
  → Space A memories
  → Universe B memories
  → User memories

Simple greeting (low complexity):
  → User memories only (via dopamine-gated fast route)

Cross-project question:
  → Organization + all relevant Universe memories

The dopamine-gated router (Section 5) decides which scopes to query based on query complexity, avoiding the fan-out problem of always querying all 4 scopes.

Updated RecallOptions Interface

// MODIFIED: APIs/backend-vitto-api/src/app/mind/memory/vitto-memory.types.ts

interface RecallOptions {
  userId: string;
  spaceId?: string;
  universeId?: string;          // NEW: for universe-scoped recall
  type?: VittoMemoryType;
  userType?: string;
  scope?: MemoryScope | MemoryScope[];  // NEW: filter by scope(s)
  scopeId?: string;             // NEW: filter by specific scope entity
  minImportance?: number;
  limit?: number;
  // DEPRECATED: layer (kept for backward compat, mapped to scope internally)
  layer?: string;
}

interface RecallResult {
  memoryId: string;
  summary: string;
  content?: string;
  type: VittoMemoryType;
  scope: MemoryScope;           // NEW
  scopeId: string;              // NEW
  importance: number;
  similarity: number;
  source: 'hot_cache' | 'redis' | 'vector_search' | 'text_search' | 'cold_recall';
  tags: string[];
  keyEntities: string[];
  createdAt: Date;
}

4. Virtual Context Memory

This is the fix for in-session amnesia. Replaces the current 10-20 message hard cap with an intelligent compression system.

Current Problem (Verified)

Solution: 4-Layer Context Stack

// NEW: APIs/backend-vitto-api/src/app/mind/virtual-context/

// Layer 1: Pinned Context (always present)
// - Soul brief (~150 tokens)
// - User brief (~200 tokens)
// - Session genome (~300-500 tokens) ← NEW

// Layer 2: Active Window (recent messages, verbatim)
// - Last N messages where N is dynamically calculated based on:
//   - Available token budget (total - pinned - genome - recalled)
//   - Message length (short messages = higher N)
//   - Typical range: 8-30 messages

// Layer 3: Conversation Genome (compressed history)
// - Topics, decisions, facts, action items from evicted messages
// - ~1K tokens covers ~10K of raw conversation
// - Incrementally updated via heuristics (no LLM call per turn)
// - Full recompile via Haiku every 10 evictions

// Layer 4: On-Demand Recall (triggered by reference)
// - When user references old topic ("earlier you said...")
// - Semantic search on FULL session message history
// - Redis Vector Sets for <5ms retrieval
// - Inject specific messages as supplementary context

ConversationGenomeService (NEW — Backend-vitto-api)

Handoff Gap D deviation: The S239 handoff recommended consuming genomes from backend-ai-api via BackendAiClientService.getSessionGenome(). However, the virtual context system requires real-time, per-turn incremental genome updates tightly coupled to the agent loop's eviction cycle. The ai-api genome is post-hoc (compiled after conversation ends). This service handles the live session case; the ai-api genome remains the source for historical sessions. A future UserGenomeService (see Cross-Channel Genome below) will aggregate both sources into a per-user document.

// NEW: APIs/backend-vitto-api/src/app/mind/virtual-context/conversation-genome.service.ts

interface ConversationGenome {
  sessionId: string;
  userId: string;
  topics: GenomeTopic[];        // Major topics discussed
  decisions: GenomeDecision[];  // Decisions made
  facts: GenomeFact[];          // Facts established
  actionItems: string[];        // Pending actions
  summary: string;              // 1-2 sentence session summary
  messageRange: {               // Which messages this covers
    from: number;               // First compressed message index
    to: number;                 // Last compressed message index
  };
  tokenCount: number;           // Current genome size in tokens
  lastUpdated: Date;
  version: number;              // Increment on each update
}

interface GenomeTopic {
  name: string;
  firstMentioned: number;       // Message index
  lastMentioned: number;
  status: 'active' | 'resolved' | 'abandoned';
  keyPoints: string[];          // Max 3 per topic
}

class ConversationGenomeService {
  // Incrementally update genome when messages are evicted from active window
  // Uses heuristics (keyword extraction, topic detection) — NO LLM call
  incrementalUpdate(genome: ConversationGenome, evictedMessages: Message[]): ConversationGenome;

  // Full recompile via cheap LLM (Haiku) — called every 10 evictions
  // or when genome token count exceeds 800 tokens
  fullRecompile(sessionId: string, allMessages: Message[]): Promise<ConversationGenome>;

  // Hydrate genome into prompt-ready text (~300-500 tokens)
  hydrate(genome: ConversationGenome): string;

  // Search genome for topic relevance (used by on-demand recall trigger)
  topicRelevance(query: string, genome: ConversationGenome): number;
}

Dynamic Window Sizing

// NEW: APIs/backend-vitto-api/src/app/mind/virtual-context/window-manager.service.ts

class WindowManagerService {
  private readonly MAX_CONTEXT_TOKENS = 80_000; // From context-window.service.ts

  /**
   * Calculate how many recent messages to include verbatim.
   *
   * Token budget:
   *   Total available = MAX_CONTEXT_TOKENS
   *   - Soul brief: ~150 tokens
   *   - User brief: ~200 tokens
   *   - System prompt: ~500 tokens
   *   - Genome: ~300-500 tokens
   *   - Recalled memories: ~500 tokens
   *   - Skills/tools: ~1000 tokens
   *   - LLM response buffer: ~4000 tokens
   *   ───────────────────────
   *   Available for messages: ~73,000 tokens
   *
   * Each message averages ~200-400 tokens.
   * Practical window: 30-60 messages in active window.
   */
  calculateWindowSize(
    contextBudget: number,
    pinnedTokens: number,
    genomeTokens: number,
    recalledTokens: number,
  ): number;

  /**
   * Manage eviction: when new message arrives and window is full,
   * evict oldest messages and feed them to ConversationGenomeService.
   */
  evictAndCompress(
    messages: Message[],
    windowSize: number,
    genome: ConversationGenome,
  ): { activeMessages: Message[]; updatedGenome: ConversationGenome };
}

On-Demand Recall Trigger

// NEW: APIs/backend-vitto-api/src/app/mind/virtual-context/on-demand-recall.service.ts

class OnDemandRecallService {
  /**
   * Detect when user references past conversation context.
   *
   * Triggers:
   * - Explicit: "earlier you said", "remember when", "what did I say about"
   * - Implicit: User mentions a topic from genome that's not in active window
   * - Semantic: High similarity between query and evicted messages (via Redis)
   *
   * When triggered:
   * 1. Search full session history (Redis Vector Sets, <5ms)
   * 2. Retrieve top 3-5 relevant messages
   * 3. Inject as [RECALLED CONTEXT] block before active window
   * 4. LLM sees: pinned + genome + recalled + active window
   */
  detectRecallNeed(
    query: string,
    genome: ConversationGenome,
    activeWindow: Message[],
  ): Promise<RecallDecision>;

  recallFromSession(
    sessionId: string,
    query: string,
    limit?: number,
  ): Promise<Message[]>;
}

Files to Modify

5. 3-Tier Recall Engine

Tier Architecture

Query arrives
  → DopamineRouter.classify(query)
     │
     ├─ FAST ROUTE (routine queries, greetings, simple follow-ups)
     │   → Hot cache check (0ms)
     │   → Redis Vector Sets (1-5ms)
     │   → Return top results
     │   → Scope: User only
     │
     ├─ STANDARD ROUTE (typical questions, task requests)
     │   → Hot cache + Redis (1-5ms)
     │   → MongoDB $vectorSearch (50-200ms)
     │   → Scope: User + Universe + Space
     │   → Return merged results
     │
     └─ DEEP ROUTE (complex multi-hop, cross-project, analytical)
         → Hot cache + Redis (1-5ms)
         → MongoDB $vectorSearch + $graphLookup (100-300ms)
         → Cross-API cold recall (200-500ms)
         → Scope: All 4 scopes
         → Prospective index matching
         → Return comprehensive results

DopamineRouterService (NEW)

// NEW: APIs/backend-vitto-api/src/app/mind/recall/dopamine-router.service.ts

type RecallRoute = 'fast' | 'standard' | 'deep';

interface RouteDecision {
  route: RecallRoute;
  scopes: MemoryScope[];
  maxResults: number;
  includeGraphTraversal: boolean;
  includeProspectiveMatching: boolean;
  confidence: number;
}

class DopamineRouterService {
  /**
   * Classify query complexity in <10ms using heuristics (NO LLM call).
   *
   * Signals:
   * - Message length (<20 chars → likely fast)
   * - Question words (what/why/how → standard; "compare", "analyze" → deep)
   * - Entity count (multiple entities → deep)
   * - Conversation position (first message → standard; reply to tool output → fast)
   * - Topic novelty vs genome (new topic → standard; known topic → fast)
   */
  classifyRoute(query: string, context: RouteContext): RouteDecision;
}

Redis Vector Sets Integration (NEW)

Infrastructure prerequisite (Phase 3): Redis is NOT currently a dependency of backend-vitto-api. Before Phase 3 implementation:

1. Add ioredis (v5+) to backend-vitto-api dependencies
2. Provision Redis 8.0+ (Vector Sets require 8.0+). Options: Redis Cloud, Railway addon, or self-hosted
3. Add REDIS_URL environment variable to .env.local and deployment configs
4. Graceful degradation: RedisVectorService.isAvailable() returns false if Redis is not connected. The recall pipeline skips Redis tier and proceeds directly to MongoDB $vectorSearch. This means Phases 1-2 work without Redis — the in-memory hot cache + MongoDB vector search provide the recall path. Redis is a SPEED enhancement, not a functional requirement.

// NEW: APIs/backend-vitto-api/src/app/mind/recall/redis-vector.service.ts

class RedisVectorService {
  /**
   * Write-through pattern:
   * 1. Memory saved to MongoDB (source of truth)
   * 2. MongoDB Change Stream triggers
   * 3. Embedding written to Redis VSET
   *
   * Read path:
   * 1. VSIM command with query embedding
   * 2. Returns memoryIds + similarity scores
   * 3. Full documents fetched from MongoDB by ID (or hot cache)
   */

  // Add embedding to Redis Vector Set
  addVector(memoryId: string, embedding: number[], metadata: VectorMetadata): Promise<void>;

  // Similarity search (VSIM command)
  search(queryEmbedding: number[], options: VectorSearchOptions): Promise<VectorResult[]>;

  // Remove vector (on memory deletion)
  removeVector(memoryId: string): Promise<void>;

  // Health check
  isAvailable(): boolean;
}

interface VectorSearchOptions {
  limit: number;
  minSimilarity?: number;
  filter?: {
    userId?: string;
    scope?: MemoryScope;
    scopeId?: string;
  };
}

Enhanced VittoMemoryRecallService

// MODIFIED: APIs/backend-vitto-api/src/app/mind/memory/vitto-memory-recall.service.ts

// Add to existing recall pipeline:
class VittoMemoryRecallService {
  // NEW: Route-aware recall
  async recall(query: string, options: RecallOptions): Promise<RecallResult[]> {
    const route = this.dopamineRouter.classifyRoute(query, options.context);

    // Tier 1: Always check hot cache + Redis
    const hotResults = await this.hotCache.check(query, options);
    const redisResults = await this.redisVector.search(queryEmbedding, {
      limit: route.maxResults,
      filter: { userId: options.userId, scope: { $in: route.scopes } },
    });

    if (route.route === 'fast' && (hotResults.length + redisResults.length) >= route.maxResults) {
      return this.mergeAndRank(hotResults, redisResults);
    }

    // Tier 2: MongoDB $vectorSearch (+ optional $graphLookup)
    const mongoResults = await this.vectorSearch(query, options, route);

    if (route.route === 'standard') {
      return this.mergeAndRank(hotResults, redisResults, mongoResults);
    }

    // Tier 3: Deep route — cross-API + prospective matching
    const coldResults = await this.coldRecall(query, options);
    const prospectiveResults = route.includeProspectiveMatching
      ? await this.prospectiveMatch(query, options)
      : [];

    return this.mergeAndRank(hotResults, redisResults, mongoResults, coldResults, prospectiveResults);
  }
}

6. Intelligent Write Path

5-Factor Admission Control (NEW)

// NEW: APIs/backend-vitto-api/src/app/mind/admission/memory-admission.service.ts

interface AdmissionScore {
  futureUtility: number;      // 0-1: Will this be useful later?
  factualConfidence: number;  // 0-1: Is this reliable information?
  semanticNovelty: number;    // 0-1: Do we already know this? (inverse of existing similarity)
  temporalRecency: number;    // 0-1: Is this still relevant?
  contentTypePrior: number;   // 0-1: Inherent value of this memory type
  composite: number;          // Weighted combination
  decision: 'admit' | 'quarantine' | 'reject';
}

class MemoryAdmissionService {
  private readonly WEIGHTS = {
    futureUtility: 0.25,
    factualConfidence: 0.20,
    semanticNovelty: 0.25,
    temporalRecency: 0.10,
    contentTypePrior: 0.20,
  };

  private readonly TYPE_PRIORS: Record<VittoMemoryType, number> = {
    correction: 0.95,          // User corrections are gold
    user_preference: 0.90,     // Explicit preferences are high value
    decision: 0.80,
    learning: 0.70,
    task_execution: 0.65,
    space_context: 0.60,
    documentation: 0.55,
    conversation: 0.40,        // Most conversations are low-signal
    code_review: 0.50,
    visual_capture: 0.45,
  };

  private readonly THRESHOLDS = {
    admit: 0.50,               // Score >= 0.50: store in active memory
    quarantine: 0.25,          // 0.25 <= score < 0.50: quarantine tier
    reject: 0.25,              // Score < 0.25: reject (truly noise)
  };

  /**
   * Score a potential memory using 5 factors.
   * All factors computed via heuristics (NO LLM call).
   *
   * Future Utility: Based on type prior + entity richness + tags
   * Factual Confidence: Based on source (user-stated vs inferred) + outcome
   * Semantic Novelty: 1 - max_similarity(existing_memories) via Redis VSIM
   * Temporal Recency: Always 1.0 at write time (decays later)
   * Content Type Prior: Static weight per memory type
   */
  async scoreAdmission(memory: MemoryCandidate): Promise<AdmissionScore>;
}

Safety net: Quarantined memories (0.25-0.50 score) are stored in the same vitto-memories collection with status: 'quarantined' (new field, default: 'active'). This avoids cross-collection joins during the monthly promotion cron. They're excluded from active recall but searchable by admins. Monthly cron auto-promotes quarantined memories if their topic resurfaces in active conversations (semantic similarity > 0.8 with any new admitted memory).

Prospective Indexing (NEW)

// NEW: APIs/backend-vitto-api/src/app/mind/indexing/prospective-indexer.service.ts

class ProspectiveIndexerService {
  /**
   * After a memory is admitted, generate 3-5 hypothetical future questions
   * this memory could answer. Store as supplementary embeddings.
   *
   * Runs ASYNC (fire-and-forget) using cheap LLM (Haiku).
   *
   * Example:
   *   Memory: "User prefers dark mode and uses keyboard shortcuts heavily"
   *   Prospective questions:
   *     - "What UI preferences does the user have?"
   *     - "Should I suggest keyboard shortcuts for this feature?"
   *     - "What accessibility considerations matter for this user?"
   *
   * These are stored as additional embeddings pointing to the same memoryId.
   * At recall time, query matches against BOTH the original embedding AND
   * prospective question embeddings. Score = max(original, prospective).
   */
  async generateProspectiveQuestions(memory: VittoMemory): Promise<string[]>;

  async indexProspectiveQuestions(
    memoryId: string,
    questions: string[],
  ): Promise<void>;
}

LLM routing: Uses the existing smart LLM router to select the cheapest model capable of generating good questions. For most memories, Haiku is sufficient. Only complex architectural decisions escalate to Sonnet.

Quality safeguard: Prospective questions are SUPPLEMENTARY embeddings, not replacements. Bad questions can't hurt recall — they can only help. The original memory embedding is always searched too.

Enhanced Write Pipeline

Memory candidate arrives
  → onMemoryAdmission plugin hook
  → MemoryAdmissionService.scoreAdmission()
     ├─ admit (≥0.50): Continue to storage
     ├─ quarantine (0.25-0.50): Store in quarantine collection
     └─ reject (<0.25): Discard (log for observability)
  → VittoMemoryStoreService.remember() (existing Mem0 dedup)
  → PARALLEL (fire-and-forget):
     ├─ VittoMemoryEmbeddingService.generateAndStoreEmbedding() (256-dim Matryoshka)
     ├─ RedisVectorService.addVector() (via Change Stream)
     ├─ KnowledgeGraphService.extractAndStoreFromMemory()
     ├─ ProspectiveIndexerService.generateProspectiveQuestions() (async, cheap LLM)
     └─ onMemoryConsolidation plugin hook

7. Context Compression Stack

Layer 1: TOON Format for Structured Data

All structured context injections (memory lists, tool results, user briefs) are serialized in TOON format instead of JSON.

// NEW: APIs/backend-vitto-api/src/app/engine/context-assembly/toon-serializer.service.ts

import { serialize } from 'toon-format/toon';

class ToonSerializerService {
  /**
   * Serialize structured data to TOON format.
   * 30-60% token reduction vs JSON with +2-7% accuracy improvement.
   *
   * Used for:
   * - Memory recall results injected into prompts
   * - User dossier structured data
   * - Tool call results
   * - Knowledge graph entity lists
   */
  serializeForPrompt(data: unknown): string;
}

Layer 2: Tiered Conversation Summarization

Handled by ConversationGenomeService (Section 4). Three tiers:

• Tier 1 (0-active window): Verbatim messages in active window
• Tier 2 (evicted, <24h): Genome-compressed summaries (~10x reduction)
• Tier 3 (>24h): Memory-extracted facts only (via VittoMemoryService.remember())

Layer 3: Complexity-Aware Token Budget

// MODIFIED: APIs/backend-vitto-api/src/app/engine/context-assembly/context-assembly.service.ts

interface TokenBudget {
  total: number;              // Max context window (e.g., 80K for Sonnet)
  soul: number;               // ~150 tokens (TOON-compressed, cached)
  userBrief: number;          // ~200 tokens (cached)
  genome: number;             // ~300-500 tokens (dynamic)
  recalledMemories: number;   // 0-1000 tokens (route-dependent)
  skills: number;             // 0-1000 tokens (if tools needed)
  activeWindow: number;       // REMAINDER goes to messages
  responseBuffer: number;     // ~4000 tokens reserved for LLM output
}

// Budget varies by query complexity (from DopamineRouter):
// FAST route:    soul(150) + brief(200) + window(REST) + buffer(4000)
// STANDARD route: soul(150) + brief(200) + genome(400) + memories(500) + window(REST) + buffer(4000)
// DEEP route:    soul(150) + brief(200) + genome(500) + memories(1000) + skills(1000) + window(REST) + buffer(4000)

Layer 4: Prompt Caching Optimization

Context assembly must produce a deterministic, append-only prompt structure to maximize cache hits:

[STABLE PREFIX — cached across turns, ~90% cost savings]
  Soul brief (TOON, ~150 tokens)
  System instructions (~500 tokens)
  User brief (TOON, ~200 tokens)

[SEMI-STABLE — cached within session]
  Conversation genome (~300-500 tokens, changes every ~10 turns)
  Active skills/tools definition (~1000 tokens, changes when tools change)

[DYNAMIC — per-turn]
  Recalled memories (scope-filtered, TOON-formatted)
  On-demand recalled messages (if triggered)
  Active conversation window (recent messages)

The key insight from Manus: never modify earlier context sections. Only append. A single token change in the stable prefix invalidates the entire cache. The soul brief MUST be deterministically serialized.

8. Cognitive Plugin System

Plugin Interface

// NEW: APIs/backend-vitto-api/src/app/mind/plugins/cognitive-plugin.interface.ts

interface CognitivePlugin {
  name: string;
  version: string;
  enabled: boolean;

  // Lifecycle hooks (all optional, async)
  onBeforeContextAssembly?(context: AssemblyContext): Promise<AssemblyContext>;
  onBeforeResponse?(context: ResponseContext): Promise<ResponseContext>;
  onAfterResponse?(response: ResponseResult): Promise<void>;
  onMemoryAdmission?(candidate: MemoryCandidate): Promise<AdmissionOverride | null>;
  onMemoryConsolidation?(memories: VittoMemory[]): Promise<ConsolidationResult>;
  onSessionStart?(session: SessionContext): Promise<void>;
  onSessionEnd?(session: SessionContext): Promise<void>;
}

// Override types
interface AdmissionOverride {
  forceAdmit?: boolean;     // Skip admission scoring
  forceReject?: boolean;    // Skip storage
  modifiedCandidate?: MemoryCandidate; // Modify before storing
}

interface ConsolidationResult {
  merged?: VittoMemory[];   // Memories to merge
  promoted?: string[];      // Memory IDs to promote importance
  demoted?: string[];       // Memory IDs to demote
}

Plugin Registry

// NEW: APIs/backend-vitto-api/src/app/mind/plugins/cognitive-plugin-registry.service.ts

class CognitivePluginRegistryService {
  private plugins: Map<string, CognitivePlugin> = new Map();

  register(plugin: CognitivePlugin): void;
  unregister(name: string): void;

  // Execute hooks in registration order, with timeout protection (500ms per plugin)
  async executeHook<T>(hookName: keyof CognitivePlugin, ...args: unknown[]): Promise<T>;

  // List registered plugins with status
  getStatus(): PluginStatus[];
}

Built-in Plugins

Each memory innovation is implemented as a cognitive plugin:

Plugin Configuration

Plugins are configured via environment variables and can be toggled without code changes:

VITTO_PLUGINS_ADMISSION_CONTROL=true
VITTO_PLUGINS_PROSPECTIVE_INDEX=true
VITTO_PLUGINS_DOPAMINE_ROUTER=true
VITTO_PLUGINS_USER_PREFERENCE_LEARNING=true
VITTO_PLUGINS_EMOTIONAL_CONTEXT=true
VITTO_PLUGINS_MEMORY_VERSIONING=true

9. MongoDB Vector Search Setup

Current Status (Verified 2026-03-19)

Embedding Dimension Migration

Current: 1536-dim (text-embedding-3-small, default) Target: 256-dim (text-embedding-3-small with dimensions: 256 parameter)

OpenAI's text-embedding-3-small supports Matryoshka representation — truncating to 256 dimensions retains quality while reducing storage 6x and speeding up similarity computation.

Migration path:

1. Create new indexes with 256 dimensions
2. New memories get 256-dim embeddings
3. Backfill existing memories in batches (re-embed with dimensions: 256)
4. Drop old 1536-dim indexes after backfill complete

Index Definitions

See Section 15 (Cowork Agent Handoff) for the exact JSON definitions to create in MongoDB Atlas.

10. Memory Management UI

Approach: Enhanced Vitto Hub + Export to Docs

Primary editor: /vitto/memory in Vitto hub (existing memory-manager.component.ts) Read-only observatory: Export to docs-angular as markdown snapshots

Vitto Hub Enhancements

Export to Docs-Angular

// NEW: APIs/backend-vitto-api/src/app/mind/export/memory-docs-export.service.ts

class MemoryDocsExportService {
  /**
   * Export Vitto's memory as organized markdown files.
   * Pushed to docs-angular backend via POST /sync/bulk.
   *
   * Structure:
   *   vitto-memory/
   *     organization/
   *       policies.md
   *       business-rules.md
   *     spaces/
   *       {spaceName}/
   *         context.md
   *         team-dynamics.md
   *     universes/
   *       {universeName}/
   *         project-knowledge.md
   *         decisions.md
   *     users/
   *       {userName}/
   *         preferences.md
   *         learning-history.md
   *
   * Triggered: Manual by executive, or weekly cron
   * Database: gograb-docs (existing docs-database cluster)
   * Project: 'vitto-memory' (new project in doc-projects collection)
   */
  async exportToDocsSmart(options?: ExportOptions): Promise<ExportResult>;
}

11. Conflict Resolution & Safety

Memory Versioning

Every memory edit creates a version entry:

// Added to vitto-memory.schema.ts

@Prop({ type: [{ summary: String, tags: [String], editedBy: String, editedAt: Date }], default: [] })
history: MemoryVersion[];

Lightweight — just stores the previous summary + tags before overwriting. No separate collection.

Cascade Policies

Multi-Language Support

Memories are stored with a language tag. When a memory is stored in Spanish, the embedding still captures semantic meaning cross-lingually (OpenAI embeddings handle this). For critical organization memories, consider storing bilingual versions (both en + es) with the same scopeId and a language field.

Quarantine Review

Monthly cron job reviews quarantined memories:

1. Check if topic has resurfaced (semantic similarity > 0.8 with any recently admitted memory)
2. If yes, auto-promote to admitted
3. If no and older than 90 days, permanently delete
4. Executives can manually browse/promote/reject via Vitto hub

12. Key Files Reference

New Files to Create

APIs/backend-vitto-api/src/app/mind/
  ├── soul-compression.service.ts              # Soul → 150-token TOON brief
  ├── virtual-context/
  │   ├── conversation-genome.service.ts       # Session compression
  │   ├── window-manager.service.ts            # Dynamic message window
  │   └── on-demand-recall.service.ts          # Reference-triggered recall
  ├── recall/
  │   ├── dopamine-router.service.ts           # Fast/standard/deep routing
  │   └── redis-vector.service.ts              # Redis Vector Sets integration
  ├── admission/
  │   ├── memory-admission.service.ts          # 5-factor admission gate
  │   └── quarantine.service.ts                # Quarantine management
  ├── indexing/
  │   └── prospective-indexer.service.ts       # Future-question generation
  ├── plugins/
  │   ├── cognitive-plugin.interface.ts        # Plugin interface definition
  │   ├── cognitive-plugin-registry.service.ts # Plugin registry
  │   └── built-in/                            # Built-in plugin implementations
  │       ├── admission-control.plugin.ts
  │       ├── prospective-index.plugin.ts
  │       ├── dopamine-router.plugin.ts
  │       ├── user-preference-learning.plugin.ts
  │       ├── emotional-context.plugin.ts
  │       └── memory-versioning.plugin.ts
  ├── conflict/
  │   └── memory-conflict-resolver.service.ts  # Scope precedence resolution
  └── export/
      └── memory-docs-export.service.ts        # Export to docs-angular

Existing Files to Modify

APIs/backend-vitto-api/src/app/
  ├── mind/
  │   ├── schemas/vitto-memory.schema.ts       # Add scope, scopeId, overridable, history fields
  │   ├── vitto-memory.service.ts              # Integrate plugins, admission, routing
  │   ├── memory/
  │   │   ├── vitto-memory-recall.service.ts   # Add Redis tier, dopamine routing
  │   │   ├── vitto-memory-store.service.ts    # Add admission control, prospective indexing
  │   │   ├── vitto-memory-embedding.service.ts # Switch to 256-dim Matryoshka
  │   │   └── vitto-memory-hot-cache.service.ts # Integrate with Redis as Tier 0
  │   ├── user-brief.service.ts                # TOON serialization
  │   └── identity-synthesis.service.ts        # TOON serialization
  ├── engine/
  │   ├── context-assembly/
  │   │   ├── context-assembly.service.ts      # Add genome, TOON, dynamic budget
  │   │   └── channel-prompt-builder.service.ts # Remove hardcoded 10/20 limits
  │   ├── channel-processing.service.ts        # Remove hardcoded 10 limit
  │   └── agent-session.service.ts             # Integrate with window manager
  └── strategy/
      └── context-window.service.ts            # Integrate with window manager

libs/libs-ai/vitto-hub/src/lib/components/
  └── memory-manager/
      ├── memory-manager.component.ts          # Add scope filter, quarantine view
      └── memory-manager.component.html        # Scope badges, version history

13. Migration Strategy

Phase 1: Critical Fixes (Week 1)

1. Create all missing MongoDB vector search indexes (Cowork agent handoff, Section 15)
2. Fix in-session amnesia:
   • Implement ConversationGenomeService (heuristic-only v1, no LLM)
   • Implement WindowManagerService (replace hardcoded 10/20 limits)
   • Wire into context-assembly.service.ts
3. Add scope and scopeId fields to schema (default: scope='user', scopeId=userId)

Phase 2: Intelligence Layer (Week 2)

4. Implement admission control (5-factor scoring, quarantine)
5. Implement dopamine router (heuristic-based fast/standard/deep)
6. Switch embeddings to 256-dim Matryoshka (update embedding service + backfill)
7. Add TOON serialization for context injection
8. Implement memory conflict resolver (scope precedence rules)

Phase 3: Speed Layer (Week 3)

9. Redis Vector Sets integration (write-through via Change Streams)
10. Implement soul compression (TOON-formatted persona brief)
11. Prompt caching optimization (stable prefix, append-only structure)
12. Implement on-demand recall (reference detection + Redis search)

Phase 4: Innovation Layer (Week 4)

13. Prospective indexing (async future-question generation)
14. Cognitive plugin system (interface, registry, built-in plugins)
15. ConversationGenomeService v2 (Haiku-powered full recompile)
16. Memory versioning (edit history on all updates)

Phase 5: UI & Management (Week 5)

17. Vitto hub enhancements (scope filter, quarantine view, genome viewer)
18. Export to docs-angular (markdown snapshot generation)
19. User preference learning plugin (detect corrections from conversation)

Phase 6: Cross-Channel Genome (Week 6)

20. UserGenomeService — Aggregates per-session genomes into a single per-user document (user-genomes collection in vitto-mind). Addresses S239 Handoff Gap A.
21. Cross-channel awareness — When user switches from WhatsApp to web chat, the user genome provides continuity context. Injected into context assembly AFTER user brief, BEFORE channel-specific history.

Data Migration

Existing 952+ memories need migration:

1. Add scope: 'user', scopeId: <userId> to all existing memories (simple bulk update)
2. No data loss — this is purely additive

Dual-Index Embedding Dimension Transition (Phase 2, Step 6)

The migration from 1536-dim to 256-dim embeddings requires careful coordination:

1. Phase 1 creates 1536-dim indexes — all existing embeddings are 1536-dim, so indexes must match
2. Phase 2 Step 6a: Update VittoMemoryEmbeddingService to generate 256-dim embeddings (set dimensions: 256 in OpenAI API call). New memories get 256-dim stored in embedding_256 field (separate from existing embedding)
3. Phase 2 Step 6b: Create parallel 256-dim vector search indexes (see Section 15 post-migration indexes)
4. Phase 2 Step 6c: Run backfill batch job — re-embed all existing memories at 256-dim, writing to embedding_256 field. Cost: ~$0.02 for 1000 memories.
5. Phase 2 Step 6d: During transition, VittoMemoryRecallService queries BOTH indexes:
   • 1536-dim index for old memories without embedding_256
   • 256-dim index for new/backfilled memories
   • Merge results by memoryId, take highest similarity score
6. Phase 2 Step 6e (after backfill complete): Flip VITTO_EMBEDDING_DIMENSIONS=256 env var. Recall service switches to 256-dim index only. The embedding field is no longer written (only embedding_256).
7. Cleanup (deferred): Rename embedding_256 → embedding, drop old 1536-dim indexes, update index definitions. This can be a future maintenance task.

Key config:

# During transition:
VITTO_EMBEDDING_DIMENSIONS=1536     # Old default
VITTO_EMBEDDING_USE_DUAL_INDEX=true # Query both indexes

# After backfill complete:
VITTO_EMBEDDING_DIMENSIONS=256      # New default
VITTO_EMBEDDING_USE_DUAL_INDEX=false # Single index

14. Risks & Mitigations

15. Cowork Agent Handoff: Vector Search Indexes

Instructions for Cowork Agent

Create the following vector search indexes in MongoDB Atlas. Each index must be created via the Atlas UI (Database → Search → Create Search Index → JSON Editor) or Atlas CLI.

Index 1: agent_embeddings (CRITICAL)

Cluster: gograb-ai (or wherever ai-mcp-dev database lives) Database: ai-mcp-dev Collection: agent_embeddings

{
  "name": "agent_embeddings",
  "type": "vectorSearch",
  "definition": {
    "fields": [
      {
        "type": "vector",
        "path": "embedding",
        "numDimensions": 1536,
        "similarity": "cosine"
      },
      { "type": "filter", "path": "qualityRating" },
      { "type": "filter", "path": "metadata.taskType" },
      { "type": "filter", "path": "metadata.agentType" },
      { "type": "filter", "path": "metadata.decisionType" },
      { "type": "filter", "path": "outcome.wasSuccessful" },
      { "type": "filter", "path": "tenantId" }
    ]
  }
}

Index 2: vitto_memories_vector (CRITICAL)

Cluster: gograb-vitto (or wherever vitto-mind database lives) Database: vitto-mind Collection: vitto-memories

{
  "name": "vitto_memories_vector",
  "type": "vectorSearch",
  "definition": {
    "fields": [
      {
        "type": "vector",
        "path": "embedding",
        "numDimensions": 1536,
        "similarity": "cosine"
      },
      { "type": "filter", "path": "userId" },
      { "type": "filter", "path": "spaceId" },
      { "type": "filter", "path": "type" },
      { "type": "filter", "path": "layer" },
      { "type": "filter", "path": "scope" },
      { "type": "filter", "path": "scopeId" }
    ]
  }
}

Note: Include scope and scopeId filter fields for the new scope system. Include layer for backward compatibility during migration.

Index 3: episodes_vector (IMPORTANT)

Cluster: gograb-ai Database: ai-mcp-dev Collection: episodes

{
  "name": "episodes_vector",
  "type": "vectorSearch",
  "definition": {
    "fields": [
      {
        "type": "vector",
        "path": "embedding",
        "numDimensions": 1536,
        "similarity": "cosine"
      },
      { "type": "filter", "path": "agentId" },
      { "type": "filter", "path": "outcome.success" }
    ]
  }
}

Index 4: chat_sessions_vector (FUTURE)

Cluster: gograb-ai Database: ai-mcp-dev Collection: chat_sessions

{
  "name": "chat_sessions_vector",
  "type": "vectorSearch",
  "definition": {
    "fields": [
      {
        "type": "vector",
        "path": "messages.embedding",
        "numDimensions": 1536,
        "similarity": "cosine"
      },
      { "type": "filter", "path": "userId" }
    ]
  }
}

Note: Nested array embedding. MongoDB Atlas supports this but performance may vary at scale. Monitor after creation.

Post-Migration Indexes (256-dim)

After the embedding dimension migration (Phase 2), create parallel 256-dim indexes:

{
  "name": "vitto_memories_vector_256",
  "type": "vectorSearch",
  "definition": {
    "fields": [
      {
        "type": "vector",
        "path": "embedding_256",
        "numDimensions": 256,
        "similarity": "cosine"
      },
      { "type": "filter", "path": "userId" },
      { "type": "filter", "path": "scope" },
      { "type": "filter", "path": "scopeId" },
      { "type": "filter", "path": "type" }
    ]
  }
}

The 256-dim embeddings will be stored in a separate field (embedding_256) during migration. Once all documents are backfilled, the recall service switches to the new index and the old 1536-dim field + index can be dropped.

Verification Steps

After creating each index:

1. Wait 2-5 minutes for initial index build
2. Check status in Atlas UI → Database → Search → index should show "Active"
3. Test with a sample query:

   db.collection.aggregate([
     {
       $vectorSearch: {
         index: "index_name",
         path: "embedding",
         queryVector: new Array(1536).fill(0.1),
         numCandidates: 100,
         limit: 5
       }
     },
     { $project: { _id: 1, score: { $meta: "vectorSearchScore" } } }
   ])
   

4. Verify results return with scores > 0

Priority Order

1. vitto_memories_vector — Unblocks Vitto's memory recall (currently falling back to text search)
2. agent_embeddings — Unblocks decision intelligence RAG
3. episodes_vector — Enables episodic memory for super-intelligence
4. chat_sessions_vector — Enables per-message semantic search (can wait for implementation)

Research Sources

Brain-Inspired Memory

• D-MEM: Dopamine-Gated Agentic Memory — arXiv:2603.14597
• Neural Brain Framework — arXiv:2505.07634

Memory Consolidation & Admission

• A-MAC: Adaptive Memory Admission Control — arXiv:2603.04549
• SSGM: Stability and Safety-Governed Memory — arXiv:2603.11768
• Neural Paging (Semantic Belady Optimality) — arXiv:2603.02228

Anticipatory Memory

• Kumiho: Prospective Indexing — arXiv:2603.17244
• SpecAgent: Speculative Context — arXiv:2510.17925

Graph-Enhanced Memory

• Graphiti/Zep Temporal Knowledge Graph — arXiv:2501.13956
• Microsoft GraphRAG — microsoft.github.io/graphrag

Context Compression

• LLMLingua (20x compression) — microsoft.com/research
• ACON Agent Context Optimization — arXiv:2510.00615
• TOON Format (30-60% savings) — toonformat.dev
• Context Rot study — research.trychroma.com/context-rot

Multi-Agent Memory

• CrewAI Unified Memory — docs.crewai.com/concepts/memory
• Letta (MemGPT v2) — docs.letta.com/memory
• LangGraph Memory — langchain-ai.github.io/langgraph/concepts/memory

Vector Search

• MongoDB Atlas Vector Search benchmarks — mongodb.com/docs
• Redis Vector Sets — redis.io/blog/announcing-vector-sets
• QVCache — arXiv:2602.02057
• Matryoshka Embeddings — openai.com/index/new-embedding-models

Context Engineering

• Anthropic: Effective Context Engineering — anthropic.com/engineering
• Manus: Context Engineering Lessons — manus.im/blog
• Prompt Caching — platform.claude.com/docs

End of spec. This document is the single source of truth for Vitto Memory System v2. Next step: Implementation plan via writing-plans skill.