MongoDB Data Architecture Analysis

Date: 2026-02-21 | Session: 133.4 | Author: Claude Code (Opus 4.6)

This document provides a comprehensive analysis of the GoGrab platform's MongoDB data architecture across all three backends, the MCP server, and the Vitto intelligence system. It maps every database, collection, and data pipeline — identifying what's working, what's broken, and what needs fixing.

Table of Contents

1. Cluster & Database Inventory
2. Three Intelligence Systems
3. Empty Collection Classification
4. Root Cause: The Outcome Detection Lynchpin
5. Dead Code Inventory
6. User-Type Segregation Analysis
7. Data Flow Diagrams
8. Frontend Route -> Data Source Mapping
9. Performance & Architecture Assessment
10. Prioritized Recommendations

1. Cluster & Database Inventory

1.1 Three Atlas Clusters

The platform uses 3 separate MongoDB Atlas clusters sharing the same credentials but hosting different workloads:

Config files:

• .env.local (lines 47-62) — all connection strings
• libs/libs-api/backend-core/src/lib/config/database-names.config.ts — database name constants
• libs/libs-api/backend-core/src/lib/config/mongodb.factory.ts — connection factory with pool sizes

1.2 gograbspace Cluster (~238 MB)

backend-ai-api opens 5 simultaneous connections to this cluster, each targeting a different database:

ai-agents (31 MB, 26 collections)

Primary database for AI agent orchestration.

Decisions breakdown by type:

• bugfix: 6,233 (52.3%)
• architecture: 4,818 (40.4%)
• implementation: 717 (6.0%)
• refactor: 57, convention: 28, pattern: 27, milestone: 19, fix: 12, documentation: 11, testing: 1, system-optimization: 1

Decisions with outcomes: 441 / 11,924 (3.7%) — the rest have NO outcome data.

ai-memory (189 MB, 2 collections)

Largest database. Separated for vector search performance optimization.

Performance justification: At 189 MB (6x larger than ai-agents), keeping vector embeddings in a separate database with dedicated connection pool (maxPoolSize: 20) is architecturally correct. Vector search queries are computationally expensive and would compete with regular CRUD on ai-agents.

ai-mcp-dev (5 MB, 18 collections)

MCP development session management, learning pyramid, and agent coordination.

Intelligence Pyramid (live data):

    ┌──────────────┐
    │  CORE RULES  │  10 rules (confidence 0.95-1.0)
    │  (ai-mcp-dev)│  6 Angular, 3 Build, 1 Backend
    └──────┬───────┘
           │ promoted from
    ┌──────┴───────┐
    │   PATTERNS   │  563 learned patterns
    │  (ai-mcp-dev)│  479 general, 84 architecture
    └──────┬───────┘
           │ clustered from
    ┌──────┴───────┐
    │  DECISIONS   │  11,924 decisions
    │  (ai-agents) │  52% bugfix, 40% architecture
    └──────────────┘

ai-archive (0.08 MB, 4 collections)

Cold storage — never activated. All collections empty.

• archive_policies, archived_decisions, archived_embeddings, archive_jobs

ai-backups (5.8 MB, 3 collections)

Pre-v11 migration backup.

Needs: Retention policy. Data is 2+ months old.

ai-context (1 MB, 5 collections)

AI generation context and usage tracking.

Other gograbspace databases

SpaceAIDatabaseService IS working — it has successfully created per-space agent databases (space-agents_default, space-agents_external) with the correct schema.

1.3 vitto Cluster (8.6 MB)

Single database: vitto-mind

vitto-mind (8.6 MB, 35 collections)

Vitto memories breakdown:

Memory layer distribution: developer: 209 (99.5%), platform_builder: 1 (0.5%), customer: 0

Embedding coverage: 208/210 memories have 1536-dim vector embeddings (text-embedding-3-small)

1.4 MCP File-Based Storage (Active)

The MCP server at MCP/mcp-gograb-ai/ uses a dual-write strategy: write to local JSON files first, then sync to MongoDB via REST API (backend-ai-api).

2. Three Intelligence Systems

The platform has three independent intelligence systems that partially cross-pollinate:

System 1: MCP Intelligence (Developer-Time)

What it is: Real-time learning system for Cursor/Claude Code agents during development sessions.

Data stores:

• Local: .cursor/ai-memory/hot-memory.json (21 decisions), conventions.json (6 conventions), outcome-tracker.json (113 validated outcomes)
• Remote: ai-agents.decisions (11,924), ai-mcp-dev.learned_patterns (563), ai-mcp-dev.core_rules (10)

Data flow:

Developer uses Cursor/Claude Code
    ↓
LLM calls MCP tool: log_decision()
    ↓
MCP Server (mcp-gograb-ai)
    ├──→ [1] In-memory store (prepend to decisions array)
    ├──→ [2] Local JSON file (.cursor/ai-memory/hot-memory.json)
    └──→ [3] REST API → backend-ai-api → ai-agents.decisions
         (works when backend running, fails silently otherwise)

Status: Working end-to-end. 11,924 decisions successfully synced to remote MongoDB.

System 2: AI Infrastructure Intelligence (Backend)

What it is: Server-side learning system with pattern mining, calibration, episodic memory, and causal reasoning.

Data stores:

• ai-agents: decisions, golden_decisions, generated-modules, chat_sessions, platform_agents
• ai-mcp-dev: learned_patterns, core_rules, agents, sessions, handoffs, simulations, trigger_states
• ai-memory: agent_embeddings (8,151), golden_embeddings (200)

Status: Partially working. Decisions are stored and patterns are extracted to the learning pyramid. But agent_patterns mining, calibration, and episodic memory are blocked by missing outcome detection.

System 3: Vitto Mind (Orchestrator)

What it is: Vitto's operational intelligence — memory, strategy, performance tracking, security.

Data stores:

• vitto-mind: vitto-memories (210), routing-decisions (157), execution-traces (26), proactive-alerts (2,031), personas (5)

Cross-pollination:

• 112 of 210 Vitto memories have sourceDecisionId — they were auto-created from MCP decisions
• Vitto's OutcomeFeedbackService sends execution outcomes to backend-ai-api's learning pipeline
• Backend-ai-api's BackendAiClientService queries vitto-mind for strategy context

Status: Growing but several pipelines broken (user-patterns, agent-performance, strategy-memories, skills).

Overlap Matrix

3. Empty Collection Classification

Broken Pipelines (should have data, code exists but doesn't work)

Not Yet Activated (code exists, prerequisites not met)

Correctly Empty (feature not built or manual activation)

Dead Code (should be deleted)

4. Root Cause: The Outcome Detection Lynchpin

The single biggest systemic issue is that decision outcome detection is not wired up. This blocks 5+ downstream features.

The Problem

decision.service.ts (line 97):
  this.eventEmitter.emit('decision.created', { ... });
                    ↓
          NO @OnEvent('decision.created') handler exists
                    ↓
          Outcomes never populated
                    ↓
    ┌───────────────┼───────────────────┐
    ↓               ↓                   ↓
agent_patterns    agent_calibrations   episodes
queries:          queries:             needs:
outcome.was       outcome.was          outcome data
Successful=true   Successful=true      in schema
→ 0 results       → 0 results         → empty

Current State

• 11,924 decisions in ai-agents.decisions
• Only 441 (3.7%) have any outcome data — these came from the MCP's report_outcome tool, which manually reports outcomes
• The backend's OutcomeDetectorService was designed to auto-detect outcomes from git commits, build results, and test passes — but its event handler was never connected

Impact

Without outcomes, these features are dead:

1. Pattern Mining (cron every 6h) — queries outcome.wasSuccessful=true, gets 0 results → 0 patterns
2. Confidence Calibration — queries outcomes to build calibration curves → 0 calibration data
3. Episodic Memory — episodes require outcome data → method signature mismatch compounds the issue
4. Training Pipeline — training jobs need qualityRating >= 7 from outcomes → all 6 jobs fail
5. Agent Outcome Correlations — no outcomes to correlate

Fix

Create an OutcomeDetectionListener that subscribes to decision.created events and auto-populates outcomes based on:

1. Confidence-based heuristic (confidence >= 0.8 → likely successful)
2. MCP outcome-tracker data sync (113 validated outcomes exist in local files)
3. Later: git commit analysis, build results integration

File: apis/backend-ai-api/src/app/infrastructure/services/decision.service.ts (line 97) New file: apis/backend-ai-api/src/app/learning/listeners/outcome-detection.listener.ts

5. Dead Code Inventory

5.1 Duplicate Schemas in backend-api

The following schemas exist in BOTH apis/backend-api/src/app/ai-agents/schemas/ and apis/backend-ai-api/src/app/agents/schemas/:

The backend-ai-api agents.module.ts explicitly documents: "Migrated from: apis/backend-api/src/app/ai-agents/" (line 4).

The entire apis/backend-api/src/app/ai-agents/ module is still registered in apis/backend-api/src/app/app.module.ts (line 134) but should be removed.

Action: Delete the 6 duplicate schema files and remove the module registration.

5.2 Misattributed Schema (NOT Dead)

apis/backend-vitto-api/src/app/mind/schemas/user-notification-prefs.schema.ts is NOT registered in mind.module.ts but IS registered in proactive.module.ts (line 30) and actively used by proactive-monitor.service.ts for user notification preferences.

Action: None needed. Schema is active, just registered in a different module.

5.3 Empty Collections to Drop

• ai-agents.test-modules — 0 docs, migration artifact
• ai-agents.agent_embeddings — 0 docs, embeddings live in ai-memory.agent_embeddings

Action: Drop collections from remote MongoDB.

5.4 Stale Data

6. User-Type Segregation Analysis

6.1 Current State

Vitto memories support 4 user types via the layer field, but almost all data is developer-scoped:

6.2 Schema Support

The schemas have excellent multi-tenant fields, but no pipeline populates them:

vitto-memories schema:

• layer: developer / platform_builder / customer / system
• userId: Per-user isolation
• spaceId: Per-space isolation
• shareLevel: private / team / organization
• teamId: Team-level sharing (Phase 5.3 planned)

ai-agents schemas:

• tenantId: Multi-tenant isolation (on agent_patterns, learned_patterns, agent_tasks)
• spaceId: Space-scoped data (on handoffs, space_agent_tasks)
• userId: User-scoped data (on chat_sessions, tenant_user_profiles)

SpaceAIDatabaseService (backend-ai-api) can dynamically create per-space databases (space-agents_{spaceSlug}) and has successfully created 2 space databases.

6.3 What's Missing

1. Builder Intelligence Pipeline: When a platform builder configures a space, no pipeline captures their decisions as builder-layer memories. The space_builder_preferences collection has 1 doc, but no continuous learning.

2. Customer Intelligence Pipeline: When end-users interact with built spaces (via Vitto customer-facing agent), no pipeline captures their patterns. The customer-sessions collection exists but the customer chat feature is not yet implemented.

3. System Intelligence Pipeline: The SuperIntelligence module (episodes, causal graphs, reflections) is designed for system self-learning but has no data pipeline feeding it.

4. Data Routing Policy: There's no middleware or service that decides which layer a new Vitto memory should be assigned to. Currently, the OutcomeFeedbackService defaults all memories to layer: 'developer'.

6.4 Recommendations

1. Add a MemoryLayerRouter service that determines the layer based on context:

   • MCP decision source → developer
   • Space configuration context → platform_builder
   • Customer chat context → customer
   • System health/monitoring → system

2. Wire the SpaceAIDatabaseService to create platform_builder memories when builders make space changes

3. Plan customer intelligence for Phase 2 when customer-facing Vitto chat is implemented

7. Data Flow Diagrams

7.1 Decision Lifecycle

Developer in Cursor/Claude Code
         │
         ▼
  MCP: log_decision()
         │
    ┌────┴────┐
    ▼         ▼
  LOCAL     REMOTE (REST API)
  Files     POST /ai-api/infrastructure/decisions
    │              │
    ▼              ▼
  hot-memory    ai-agents.decisions (11,924)
  .json (21)         │
                     ├──→ EventEmitter: 'decision.created'
                     │         │
                     │         ▼
                     │    OutcomeDetectionListener (MISSING)
                     │         │
                     │    ┌────┴────────────────┐
                     │    ▼                     ▼
                     │  agent_patterns      agent_calibrations
                     │  (0 docs)            (0 docs)
                     │
                     ├──→ EmbeddingService
                     │         │
                     │         ▼
                     │    ai-memory.agent_embeddings (8,151)
                     │
                     └──→ PatternMiningJob (cron 6h)
                               │
                               ▼
                          ai-mcp-dev.learned_patterns (563)
                               │
                               ▼ (promoted)
                          ai-mcp-dev.core_rules (10)

7.2 Vitto Memory Creation

Interaction Source
  ├── MCP Decision Sync (112 memories)
  │     OutcomeFeedbackService → createMemoryFromDecision()
  │
  ├── Chat Conversation (68 memories)
  │     VittoEngineService → after response → createMemoryFromConversation()
  │
  ├── Task Execution (26 memories)
  │     PhaseExecutor → VERIFY phase → mind.recordExecution()
  │
  └── System Events (3 memories)
        ProactiveMonitor → system alerts → createMemory()
         │
         ▼
   VittoMemoryService
    ├── Generate embedding (OpenAI text-embedding-3-small, 1536 dims)
    ├── Compute importance (outcome 0.3 + type 0.3 + recency 0.2 + access 0.2)
    └── Store in vitto-mind.vitto-memories (210 docs)
         │
         ▼
   Hot Cache (in-memory Map, 30-min TTL)
    ├── Tier 1: In-memory recall (fastest)
    ├── Tier 2: MongoDB Vector Search ($vectorSearch, cosine similarity)
    └── Tier 3: Platform API fallback (backend-ai-api queries)

7.3 Cross-Backend Intelligence Bridge

┌──────────────────────────┐      ┌──────────────────────────┐
│   backend-ai-api (3001)  │      │  backend-vitto-api (3003)│
│                          │      │                          │
│  ai-agents.decisions ────┼──→───┼→ vitto-memories          │
│  (11,924 docs)           │  via │  (112 with sourceDecision│
│                          │  Out-│   Id linking back)       │
│  ai-mcp-dev.learned_     │  come│                          │
│  patterns (563)     ←────┼──←───┼← execution-traces (26)  │
│                          │  Feed│  → sends outcomes to     │
│  ai-memory.agent_        │  back│    learning pyramid      │
│  embeddings (8,151)      │      │                          │
│                          │      │  strategy-memories (1)   │
│  platform_agents (4)     │      │  routing-decisions (157) │
└──────────┬───────────────┘      └──────────┬───────────────┘
           │                                  │
           └──────────┬───────────────────────┘
                      │
                      ▼
              VittoApiService (frontend)
              injects BOTH:
              - AI_API_URL (port 3001)
              - VITTO_API_URL (port 3003)

8. Frontend Route -> Data Source Mapping

9. Performance & Architecture Assessment

9.1 Multi-Database Justification

9.2 Connection Pool Analysis

backend-ai-api currently opens 5 connections to the gograbspace cluster:

Recommendation: Merge ai-archive into the ai-agents connection (keep as collection prefix like archived_*). This reduces to 4 connections. Do NOT merge ai-memory — its 189 MB of vector data justifies isolation.

9.3 Index Coverage

Well-indexed collections:

• decisions: sessionNumber+timestamp, agentId+type+confidence, globalSessionId
• agent_embeddings: Vector search index (manual Atlas creation required)
• handoffs: workspace+status, toAgentType+status
• vitto-memories: userId+createdAt, spaceId+createdAt, type+importance, vector search index

Missing indexes to add:

• decisions: Add index on outcome.wasSuccessful + createdAt (for pattern mining queries)
• proactive-alerts: Add index on resolved + createdAt (for cleanup queries)

10. Prioritized Recommendations

P0: Critical (Blocking Core Features)

P1: High (Dead Code & Data Quality)

P2: Medium (Pipeline Activation)

P3: Low (Future Improvements)

11. Additional Findings & Recommendations (Session 133.4 Deep Dive)

11.1 Index Bloat on ai-agents.decisions

The decisions collection has a critical index-to-data ratio problem:

• Data size: 4.93 MB (11,924 documents, avg 2,095 bytes each)
• Index size: 23.22 MB (30 indexes!)
• Ratio: 4.7:1 — indexes are nearly 5x the data itself

Worst offenders:

Impact: Every decision insert updates 30 indexes. At scale (100K+ decisions), this will severely impact write performance.

Recommendation: Audit and drop unused indexes. Target: reduce from 30 to ~12 core indexes:

• Keep: _id, sessionNumber+timestamp, agentId+type+confidence, outcome.wasSuccessful+confidence, globalSessionId+timestamp, createdAt
• Evaluate: text index (13.6 MB) — if vector search (ai-memory.agent_embeddings) handles semantic search, the text index is redundant
• Drop: low-cardinality indexes (workspace_1, platform_1), duplicate compound indexes

11.2 Agent Registration Spam

The ai-mcp-dev.agents collection has 2,844 unique agent registrations, but many are duplicate reconnections:

Distribution by platform:

• cowork: 2,398 (84%) — heavily over-represented
• vitto: 333 (12%)
• vscode: 64, claude-code: 22, external: 15, test: 9, cursor: 2

Recommendation:

1. Use upsert on agent registration instead of create — key on agentId
2. Add a TTL index: auto-delete agents not seen in 30 days
3. Add a periodic cleanup cron to merge duplicate registrations
4. Expected collection size after dedup: ~50-100 unique agents (vs 2,844)

11.3 Session Lifecycle — 434 "Active" Sessions Never Closed

63 sessions have been "active" for over 30 days. The session lifecycle (active → completed) is not being triggered.

Root cause: The MCP's setActiveSession() creates sessions but completeSession() is rarely called. Sessions stay "active" indefinitely.

Recommendation:

1. Add a session cleanup cron that marks sessions as completed after 7 days of inactivity (no new decisions)
2. Or: auto-complete when a new session starts for the same workspace (only 1 active per workspace)

11.4 Handoff Timing Concerns

27.7% of handoffs expire — they're never picked up. The 75-hour max suggests some handoffs sit unclaimed for days.

Recommendation:

1. Set default expiresAt to 4 hours instead of unlimited
2. Add escalation: if not acknowledged in 30 minutes, re-route to a different agent type
3. Add a dashboard metric for handoff SLA (target: 80% acknowledged within 15 minutes)

11.5 ai-backups Retention

ai-backups contains 5.8 MB of pre-v11 migration data:

• decisions_pre_v11: 11,585 docs
• sessions_pre_v11: 137 docs
• golden_decisions_pre_v11: 200 docs

This data is 2+ months old and was created as a safety net during the v11 schema migration.

Recommendation: Either drop the database (migration is well past) or set a reminder to review in 30 days. This data is not referenced by any service.

11.6 ai_gateway Orphaned Database (Local Only)

The local MongoDB has an ai_gateway database (0.64 MB) with 5 collections containing workflow experiment data (conversations, messages, usage records). No backend currently references this database.

Recommendation: Drop the database from local MongoDB. If the data has historical value, export it first.

11.7 Local vs Remote MongoDB Divergence

Local and remote MongoDB are completely divergent:

• Local: 7 databases, mostly empty (only ai_gateway and vitto-mind.codebase-index have data)
• Remote: 15+ databases with rich data

This means local development never queries real intelligence data. The MCP tools fall back to JSON files, and frontend dashboards show empty states locally.

Recommendation:

1. Document the expected local dev experience (file-based only, no MongoDB intelligence)
2. Or: add a MONGODB_CONNECTION_STRING override for dev that points to a read-only replica of remote
3. Or: create a seed script that populates local MongoDB with sample data for development

11.8 Security: Shared Credentials Across Clusters

All 3 Atlas clusters use the same username/password (paco:31grabThis!@). While acceptable for development, this should be addressed before production:

Recommendation:

1. Create separate MongoDB users per backend (e.g., backend-ai-user, vitto-user)
2. Use least-privilege: each user only accesses its own databases
3. Rotate credentials and use environment-specific secrets
4. Consider MongoDB Atlas RBAC roles for fine-grained access control

11.9 Proactive Alert Investigation

The 2,031 alerts (1,725 critical) deserve deeper investigation:

# Run this to see which checks are generating the most alerts:
mongosh "mongodb+srv://..." --eval "
  db['proactive-alerts'].aggregate([
    { \$group: { _id: { check: '\$checkName', severity: '\$severity' }, count: { \$sum: 1 } } },
    { \$sort: { count: -1 } }
  ])
"

The auto-resolution fix added in this session will help going forward, but the existing 2,031 unresolved alerts should be bulk-resolved or investigated.

11.10 MongoDB Atlas Vector Search Indexes

Both ai-memory.agent_embeddings and vitto-mind.vitto-memories use vector embeddings but require manually created Atlas Vector Search indexes. If these indexes don't exist, semantic search silently falls back to text search or returns no results.

Verification needed:

1. Check Atlas UI for agent_embeddings_vector index on ai-memory cluster
2. Check Atlas UI for vitto_memories_vector index on vitto cluster
3. If missing, create them per the schema comments in agent-embedding.schema.ts and vitto-memory.schema.ts

Appendix: Key File References