MongoDB New Architecture Design
Target architecture for GoGrab Space MongoDB infrastructure with AI optimization
MongoDB New Architecture Design
Purpose: Define the complete target architecture for GoGrab Space MongoDB infrastructure.
📋 Architecture Overview
System Diagram
┌─────────────────────────────────────────────────────────────────────────────┐
│ GOGRAB SPACE │
│ │
│ ┌──────────────────────────────────────────────────────────────────────┐ │
│ │ FRONTEND LAYER │ │
│ │ │ │
│ │ gograb-space (:5200) │ space-admin (:5300) │ gograb-admin │ │
│ │ │ │ (:4300) │ │
│ └────────────────┬────────┴────────────────────────┴───────────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────────────────────────────────────────────────────────┐ │
│ │ API GATEWAY LAYER │ │
│ │ (Future: Kong/Nginx) │ │
│ └────────────────┬─────────────────────────────┬───────────────────────┘ │
│ │ │ │
│ ┌──────────┴──────────┐ ┌──────────┴──────────┐ │
│ ▼ │ ▼ │ │
│ ┌──────────────────┐ │ ┌──────────────────┐ │ │
│ │ backend-api │ │ │ backend-ai-api │ │ │
│ │ :3000 │ │ │ :3001 │ │ │
│ │ │ │ │ │ │ │
│ │ Core Platform │ │ │ AI Platform │ │ │
│ │ • spaces │ │ │ • agents │ │ │
│ │ • widgets │ │ │ • intelligence │ │ │
│ │ • themes │ │ │ • learning │ │ │
│ │ • users │ │ │ • memory │ │ │
│ │ • ecommerce │ │ │ • generation │ │ │
│ │ • singularity │ │ │ • assistant │ │ │
│ └────────┬─────────┘ │ └────────┬─────────┘ │ │
│ │ │ │ │ │
│ └──────────────────┼──────────┴──────────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────────────────────────────────────────────────────────┐ │
│ │ MONGODB ATLAS CLUSTER │ │
│ │ (M30 Production) │ │
│ │ │ │
│ │ ┌───────────────────────────────────────────────────────────────┐ │ │
│ │ │ DATA NODES │ │ │
│ │ │ Primary ────────── Secondary ────────── Secondary │ │ │
│ │ └───────────────────────────────────────────────────────────────┘ │ │
│ │ │ │
│ │ ┌───────────────────────────────────────────────────────────────┐ │ │
│ │ │ SEARCH NODES (Optional) │ │ │
│ │ │ S30 Node 1 ────────────────────────── S30 Node 2 │ │ │
│ │ │ (Vector Search Isolation - Add when needed) │ │ │
│ │ └───────────────────────────────────────────────────────────────┘ │ │
│ │ │ │
│ │ ┌─────────┐ ┌─────────┐ ┌─────────────┐ ┌─────────────────────┐ │ │
│ │ │gograb- │ │gograb- │ │gograb-ai- │ │gograb-ai- │ │ │
│ │ │api │ │agents │ │memory │ │tenants │ │ │
│ │ └─────────┘ └─────────┘ └─────────────┘ └─────────────────────┘ │ │
│ └──────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
🗄️ Database Design
Database: gograb-api (Core Platform)
Purpose: Core platform data for spaces, widgets, themes, etc.
Used by: backend-api
// Collections
{
"spaces": {
// Space definitions
indexes: ["tenantId", "status", "createdAt"]
},
"widgets": {
// Widget catalog
indexes: ["spaceId", "type", "status"]
},
"modules": {
// Module catalog
indexes: ["spaceId", "type", "status"]
},
"themes": {
// Theme definitions
indexes: ["tenantId", "isDefault"]
},
"universes": {
// Universe configurations
indexes: ["spaceId", "status"]
}
}
Database: ai-agents (AI Orchestration)
Purpose: Agent runtime operations and session management
Used by: backend-ai-api/agents
// Collection: sessions
{
"_id": ObjectId,
"sessionId": "session_abc123",
"tenantId": "tenant_xyz",
"agentId": "cursor-angular-agent",
"status": "active", // active, paused, completed
"startedAt": ISODate,
"lastActivityAt": ISODate,
"metadata": {
"type": "cursor",
"sessionNumber": 7,
"sessionID": "7.7"
}
}
// Indexes: tenantId, status, lastActivityAt
// Collection: decisions
{
"_id": ObjectId,
"decisionId": "dev_1765513398060_yghb99h",
"sessionId": "session_abc123",
"agentId": "cursor-angular-agent",
"type": "implementation", // architecture, implementation, refactor, bugfix
"question": "How to structure the component?",
"chosen": "Signal-based state management",
"confidence": 0.95,
"timestamp": ISODate,
"outcome": {
"wasSuccessful": true,
"feedback": "Works perfectly"
}
}
// Indexes: sessionId, agentId, timestamp, type
// Collection: tasks
{
"_id": ObjectId,
"taskId": "task_123",
"userId": "user_abc",
"tenantId": "tenant_xyz",
"goal": "Create new component",
"status": "in_progress",
"steps": [...],
"createdAt": ISODate,
"updatedAt": ISODate
}
// Indexes: userId, tenantId, status, createdAt
// Collection: agent_state
{
"_id": ObjectId,
"agentId": "cursor-angular-agent", // unique
"status": "idle",
"currentTaskId": null,
"isolatedMemory": [...],
"lastUpdated": ISODate
}
// Indexes: agentId (unique)
Database: ai-memory (AI Learning)
Purpose: Vector embeddings, patterns, and AI memory
Used by: backend-ai-api/memory, backend-ai-api/intelligence
⚠️ Vector-heavy - Primary candidate for Search Nodes
// Collection: embeddings
{
"_id": ObjectId,
"content": "Original text that was embedded",
"contentType": "decision", // decision, pattern, document
"embedding": [...], // 1536-dim vector (float32 or int8)
"metadata": {
"agentId": "cursor-angular-agent",
"taskType": "implementation",
"qualityRating": 4.5,
"wasSuccessful": true,
"tenantId": "tenant_xyz"
},
"createdAt": ISODate
}
// Vector Index: embedding (1536 dims, cosine similarity)
// Regular Indexes: contentType, metadata.agentId, metadata.tenantId
// Collection: patterns
{
"_id": ObjectId,
"patternId": "pattern_123",
"name": "Signal-based State Pattern",
"description": "Use Angular Signals for component state",
"whenToUse": "When managing local component state",
"implementation": "...",
"embedding": [...], // Vector for similarity search
"successRate": 0.95,
"usageCount": 42,
"lastUsed": ISODate
}
// Vector Index: embedding
// Regular Indexes: successRate, usageCount
// Collection: context_cache
{
"_id": ObjectId,
"key": "preflight:user123:angular",
"context": {...},
"createdAt": ISODate,
"expiresAt": ISODate // TTL index
}
// Indexes: key (unique), expiresAt (TTL)
// Collection: calibrations
{
"_id": ObjectId,
"modelId": "claude-3-opus",
"calibrationData": {...},
"performance": {
"accuracy": 0.92,
"latency": 150
},
"updatedAt": ISODate
}
// Indexes: modelId, updatedAt
Database: gograb-ai-tenants (Personalization)
Purpose: Per-tenant AI customization and user profiles
Used by: backend-ai-api/tenants
// Collection: user_profiles
{
"_id": ObjectId,
"tenantId": "tenant_xyz",
"userId": "user_abc",
"preferences": {
"preferredLanguage": "typescript",
"codeStyle": "functional",
"verbosity": "concise"
},
"learningHistory": [...],
"createdAt": ISODate,
"updatedAt": ISODate
}
// Indexes: tenantId+userId (compound unique)
// Collection: user_interactions
{
"_id": ObjectId,
"tenantId": "tenant_xyz",
"userId": "user_abc",
"interactionType": "module_generation",
"request": "Create a user management module",
"response": {...},
"wasHelpful": true,
"embedding": [...], // For similarity search
"timestamp": ISODate
}
// Vector Index: embedding
// Regular Indexes: tenantId+userId, interactionType, timestamp
// Collection: preferences
{
"_id": ObjectId,
"tenantId": "tenant_xyz",
"userId": "user_abc",
"aiEnabled": true,
"autoSuggestions": true,
"privacyLevel": "standard",
"updatedAt": ISODate
}
// Indexes: tenantId+userId
🔍 Vector Search Configuration
Current Vector Indexes (Flex Cluster)
| Database | Collection | Index Name | Fields | Status |
|---|---|---|---|---|
| ai-agents | agent_embeddings | vector_index | embedding, taskType, tenantId, etc. | READY |
| ai-agents | agent_patterns | pattern_vector_index | embedding, status, etc. | READY |
| ai-agents | tenant_user_interactions | tenant_interaction_index | embedding, tenantId, userId, etc. | READY |
Target Vector Indexes (After Migration)
| Database | Collection | Index Name | Dimensions | Similarity |
|---|---|---|---|---|
| ai-memory | embeddings | embedding_vector_idx | 1536 | cosine |
| ai-memory | patterns | pattern_vector_idx | 1536 | cosine |
| gograb-ai-tenants | user_interactions | interaction_vector_idx | 1536 | cosine |
Vector Index Definition Example
// Create vector index via Atlas UI or API
{
"name": "embedding_vector_idx",
"type": "vectorSearch",
"definition": {
"fields": [
{
"type": "vector",
"path": "embedding",
"numDimensions": 1536,
"similarity": "cosine"
},
{
"type": "filter",
"path": "metadata.tenantId"
},
{
"type": "filter",
"path": "metadata.agentId"
},
{
"type": "filter",
"path": "contentType"
}
]
}
}
🚀 Scaling Architecture
Phase 1: Development (Current)
┌─────────────────────────────────────┐
│ FLEX CLUSTER (~$35/mo) │
│ │
│ Single shared infrastructure │
│ 10 vector indexes max │
│ Variable performance │
└─────────────────────────────────────┘
Phase 2: Production Launch
┌─────────────────────────────────────┐
│ M30 CLUSTER (~$320/mo) │
│ │
│ ┌───────────────────────────────┐ │
│ │ 3-Node Replica Set │ │
│ │ 8GB RAM, 2 vCPU, 40GB │ │
│ └───────────────────────────────┘ │
│ │
│ Dedicated resources │
│ Unlimited vector indexes │
│ Consistent performance │
│ Auto-scaling enabled │
└─────────────────────────────────────┘
Phase 3: Growth (When Needed)
┌─────────────────────────────────────────────────────────────┐
│ M30 + SEARCH NODES (~$670/mo) │
│ │
│ ┌───────────────────────────────┐ ┌──────────────────┐ │
│ │ Data Nodes (M30) │ │ Search Nodes │ │
│ │ Primary + 2 Secondary │ │ S30 × 2 │ │
│ │ Regular CRUD operations │ │ Vector search │ │
│ └───────────────────────────────┘ └──────────────────┘ │
│ │
│ Complete workload isolation │
│ Vector search on dedicated hardware │
└─────────────────────────────────────────────────────────────┘
Phase 4: Scale (Future)
┌─────────────────────────────────────────────────────────────┐
│ M50 SHARDED + SEARCH NODES (~$1,500+/mo) │
│ │
│ ┌────────────┐ ┌────────────┐ ┌────────────┐ │
│ │ Shard 1 │ │ Shard 2 │ │ Shard 3 │ │
│ │ 3 nodes │ │ 3 nodes │ │ 3 nodes │ │
│ └────────────┘ └────────────┘ └────────────┘ │
│ │ │ │ │
│ └────────────┼────────────┘ │
│ │ │
│ ┌─────────┴─────────┐ │
│ │ mongos Router │ │
│ └───────────────────┘ │
│ │
│ ┌────────────────────────────────────────────────────┐ │
│ │ Search Nodes (S50 × 4) │ │
│ └────────────────────────────────────────────────────┘ │
│ │
│ Horizontal scaling │
│ Terabyte-scale data │
│ Millions of embeddings │
└─────────────────────────────────────────────────────────────┘
🔐 Security Architecture
Network Security
┌─────────────────────────────────────────────────────────────┐
│ SECURITY LAYERS │
│ │
│ ┌───────────────────────────────────────────────────────┐ │
│ │ IP WHITELIST │ │
│ │ • Production servers only │ │
│ │ • VPC peering for cloud environments │ │
│ └───────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌───────────────────────────────────────────────────────┐ │
│ │ TLS ENCRYPTION │ │
│ │ • In-transit encryption (TLS 1.2+) │ │
│ │ • At-rest encryption (AES-256) │ │
│ └───────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌───────────────────────────────────────────────────────┐ │
│ │ DATABASE USER ROLES │ │
│ │ • backend-api-user: gograb-api (readWrite) │ │
│ │ • backend-ai-user: ai-agents, gograb-ai-* │ │
│ └───────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
Database User Configuration
// User: backend-api-user
{
roles: [
{ role: "readWrite", db: "gograb-api" },
{ role: "readWrite", db: "gograb-core" }
]
}
// User: backend-ai-user
{
roles: [
{ role: "readWrite", db: "ai-agents" },
{ role: "readWrite", db: "ai-memory" },
{ role: "readWrite", db: "gograb-ai-tenants" },
{ role: "read", db: "gograb-api" } // Read core data for context
]
}
📊 Monitoring & Alerts
Key Metrics to Monitor
| Metric | Warning | Critical | Action |
|---|---|---|---|
| CPU Utilization | > 70% | > 85% | Scale tier |
| Memory Utilization | > 75% | > 90% | Scale tier |
| Disk Usage | > 70% | > 85% | Increase storage |
| Connections | > 2000 | > 2500 | Scale tier |
| Query Latency (p95) | > 100ms | > 500ms | Optimize/scale |
| Vector Search Latency | > 100ms | > 200ms | Add Search Nodes |
Recommended Alerts
// Alert: High Memory Usage
{
metric: "SYSTEM_MEMORY_USED",
threshold: 80,
operator: "GREATER_THAN",
notification: ["email", "slack"]
}
// Alert: Slow Vector Search
{
metric: "VECTOR_SEARCH_LATENCY_P95",
threshold: 100, // ms
operator: "GREATER_THAN",
notification: ["email", "slack"]
}
🔗 Related Documentation
- Research Analysis - Research and decision rationale
- Architectural Changes - What's changing and why
- Backend AI API Plan - Implementation details
- MongoDB Best Practices - General MongoDB guidelines