Architecture: How should Vitto's memory store handle duplicate and stale m...

Documentation for Sessions

🏗️ Architecture: How should Vitto's memory store handle duplicate and stale m...

Date: 2026-03-19
Time: 11:32:11 UTC
Session: gspace1-234.3
Agent: angular
Confidence: 95%
Status: ✅ Documented

📋 Decision Context

Question:

How should Vitto's memory store handle duplicate and stale memories that cause repetition and bloat?

Task: S234: Memory consolidation for Jarvis-level agent

🎯 Decision Made

Chosen:

Mem0-style consolidation in VittoMemoryStoreService.remember(): before creating a new memory, compare against existing same-type/user memories using Jaccard word-overlap similarity. >0.85 = NOOP (skip), >0.5 with 20% more content = UPDATE (enrich), else ADD. No LLM call — uses instant heuristic.

Options Considered:

  1. Jaccard word-overlap consolidation (chosen — instant, no LLM, no embedding query)
  2. LLM-based classification like Mem0 (too expensive per memory write — adds 200ms+)
  3. Embedding vector similarity pre-check (better precision but adds embedding lookup latency)
  4. Time-based dedup window only (too simple — misses semantic duplicates across sessions)
  5. Graph-based entity merge (Mem0g pattern — future enhancement, requires KnowledgeGraphService wiring)

💭 Reasoning

Root cause: VittoMemoryStoreService.remember() always created a new MongoDB document regardless of whether identical or near-identical information already existed. Over time, this caused: (1) memory bloat — thousands of near-duplicate entries per user, (2) recall noise — vector search returned 5 variations of the same fact, (3) repetitive behavior — Vitto echoed the same information multiple times because it retrieved duplicates. The Mem0 research paper (arxiv 2504.19413) demonstrated a 26% accuracy boost with a two-phase Extract→Consolidate pipeline. Their approach uses LLM classification (ADD/UPDATE/DELETE/NOOP) against vector DB matches. We implemented a lighter version using Jaccard coefficient on normalized text — this is instant (no LLM call, no embedding query) and sufficient for deduplication. The tradeoff is lower precision than vector similarity for semantic duplicates ('user likes Angular' vs 'user prefers Angular framework'), but the 0.5 threshold is conservative enough to catch obvious duplicates without false positives. Future improvement: use embedding similarity from VittoMemoryEmbeddingService for the consolidation step (currently avoided to keep remember() latency low).

📁 Files Involved

🔗 Related

Auto-generated by MCP Smart Consolidation