MongoDB Architectural Changes

Detailed breakdown of architectural changes from current to target state

MongoDB Architectural Changes

Purpose: Document all architectural changes being made to the MongoDB infrastructure for GoGrab Space.

📋 Change Summary

Change	From	To	Reason
Cluster Tier	Flex	M30 (at launch)	Production requirements
API Architecture	Single backend-api	Separate backend-ai-api	Workload isolation
Database Structure	1 AI database	4 specialized databases	Clear separation of concerns
Vector Search	Basic	Search Nodes (when needed)	Performance isolation

🏗️ Current Architecture (Before)

Single Backend API

┌─────────────────────────────────────────────────────────────────────────┐
│                        backend-api (:3000)                              │
│                                                                         │
│  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────────────┐ │
│  │ Core Features   │  │ AI Features     │  │ Platform Intelligence   │ │
│  │                 │  │                 │  │                         │ │
│  │ • spaces        │  │ • ai-agents     │  │ • platform-intel        │ │
│  │ • widgets       │  │ • ai-learning   │  │ • pattern mining        │ │
│  │ • themes        │  │ • mcp           │  │ • calibration           │ │
│  │ • users         │  │ • generation    │  │                         │ │
│  │ • ecommerce     │  │ • assistant     │  │                         │ │
│  └─────────────────┘  └─────────────────┘  └─────────────────────────┘ │
│                                                                         │
│  PROBLEMS:                                                              │
│  • AI workloads compete with core platform                              │
│  • Can't scale AI independently                                         │
│  • AI crash affects entire platform                                     │
│  • Deployment coupled                                                   │
└─────────────────────────────────────────────────────────────────────────┘

Current MongoDB Structure

MongoDB Atlas (Flex Cluster)
│
├── gograb-api                # Core platform
│   ├── spaces
│   ├── widgets
│   └── ...
│
├── ai-agents             # AI (mixed concerns)
│   ├── agent_calibrations    # 0 docs
│   ├── agent_decisions       # 0 docs  
│   ├── agent_embeddings      # 25 docs ← Vector
│   ├── agent_metrics         # 0 docs
│   ├── agent_outcome_correlations # 0 docs
│   ├── agent_patterns        # 0 docs ← Vector
│   ├── agent_tasks           # 0 docs
│   ├── tenant_user_interactions # 0 docs ← Vector
│   └── tenant_user_profiles  # 0 docs
│
└── gograb-analytics          # Analytics (isolated)

Problems with Current Structure:

❌ All AI concerns mixed in one database
❌ Vector-heavy collections mixed with operational data
❌ No clear separation between agent orchestration and learning
❌ Tenant personalization mixed with platform AI

🎯 Target Architecture (After)

Separate Backend APIs

┌─────────────────────────────────────────────────────────────────────────────┐
│                              FRONTEND                                       │
│                           (gograb-space)                                    │
└────────────────────────────────┬────────────────────────────────────────────┘
                                 │
              ┌──────────────────┴──────────────────┐
              │                                      │
              ▼                                      ▼
┌─────────────────────────────┐    ┌─────────────────────────────────────────┐
│     backend-api (:3000)     │    │         backend-ai-api (:3001)          │
│                             │    │                                         │
│  ┌───────────────────────┐  │    │  ┌───────────────────────────────────┐  │
│  │    Core Features      │  │    │  │          AI Features              │  │
│  │                       │  │    │  │                                   │  │
│  │  • spaces             │  │    │  │  • agents (MCP, orchestration)    │  │
│  │  • widgets            │  │    │  │  • intelligence (platform AI)     │  │
│  │  • themes             │  │    │  │  • learning (self-learning)       │  │
│  │  • users              │  │    │  │  • memory (vectors, embeddings)   │  │
│  │  • ecommerce          │  │    │  │  • generation (module/widget)     │  │
│  │  • singularity        │  │    │  │  • assistant (Space AI)           │  │
│  │  • realtime           │  │    │  │  • mcp (MCP endpoints)            │  │
│  └───────────────────────┘  │    │  └───────────────────────────────────┘  │
│                             │    │                                         │
│  Uses: gograb-api           │    │  Uses: ai-agents                    │
│        gograb-core          │    │        ai-memory                 │
│                             │    │        gograb-ai-tenants                │
└─────────────────────────────┘    └─────────────────────────────────────────┘
              │                                      │
              └──────────────────┬───────────────────┘
                                 │
                                 ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                     MongoDB Atlas (Same Cluster)                            │
│                                                                             │
│  ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────────┐   │
│  │ gograb-api   │ │ai-agents │ │gograb-ai-mem │ │gograb-ai-tenants │   │
│  └──────────────┘ └──────────────┘ └──────────────┘ └──────────────────┘   │
└─────────────────────────────────────────────────────────────────────────────┘

Benefits:

✅ AI scales independently from core platform
✅ AI crash doesn't affect core platform
✅ Independent deployment cycles
✅ Clear separation of concerns
✅ Optimized resources per workload

🗄️ Database Structure Changes

From (Current)

ai-agents (MIXED CONCERNS)
├── agent_calibrations        ─┐
├── agent_decisions           ─┤─ Operational data
├── agent_tasks               ─┤
├── agent_metrics             ─┘
│
├── agent_embeddings          ─┐
├── agent_patterns            ─┤─ Vector-heavy learning
├── agent_outcome_correlations─┘
│
├── tenant_user_interactions  ─┐
└── tenant_user_profiles      ─┘─ Tenant personalization

To (Target)

┌─────────────────────────────────────────────────────────────────────────┐
│                    RESTRUCTURED DATABASE LAYOUT                         │
│                                                                         │
│  ┌─────────────────────────────────────────────────────────────────┐   │
│  │  ai-agents (ORCHESTRATION)                                  │   │
│  │  Purpose: Agent runtime operations                              │   │
│  │                                                                 │   │
│  │  ├── sessions        # Agent session management                 │   │
│  │  │   └── Indexes: tenantId, createdAt, status                  │   │
│  │  ├── decisions       # Decision log (lightweight)               │   │
│  │  │   └── Indexes: sessionId, agentId, timestamp                │   │
│  │  ├── tasks           # Task tracking                            │   │
│  │  │   └── Indexes: userId, status, createdAt                    │   │
│  │  └── agent_state     # Current agent states                     │   │
│  │      └── Indexes: agentId (unique)                             │   │
│  └─────────────────────────────────────────────────────────────────┘   │
│                                                                         │
│  ┌─────────────────────────────────────────────────────────────────┐   │
│  │  ai-memory (LEARNING) ← Vector-heavy                     │   │
│  │  Purpose: AI learning and memory                                │   │
│  │                                                                 │   │
│  │  ├── embeddings      # Vector embeddings ⬅️ MAIN VECTOR INDEX   │   │
│  │  │   └── Vector Index: embedding (1536 dims)                   │   │
│  │  ├── patterns        # Learned patterns                         │   │
│  │  │   └── Indexes: type, successRate                            │   │
│  │  ├── context_cache   # Pre-computed context (TTL)               │   │
│  │  │   └── Indexes: key, expiresAt (TTL)                         │   │
│  │  └── calibrations    # Model calibration data                   │   │
│  │      └── Indexes: modelId, updatedAt                           │   │
│  └─────────────────────────────────────────────────────────────────┘   │
│                                                                         │
│  ┌─────────────────────────────────────────────────────────────────┐   │
│  │  gograb-ai-tenants (PERSONALIZATION)                            │   │
│  │  Purpose: Per-tenant AI customization                           │   │
│  │                                                                 │   │
│  │  ├── user_profiles   # Per-user AI profiles                     │   │
│  │  │   └── Indexes: tenantId+userId (compound unique)            │   │
│  │  ├── user_interactions # Interaction history                    │   │
│  │  │   └── Indexes: tenantId+userId, timestamp                   │   │
│  │  └── preferences     # AI preferences                           │   │
│  │      └── Indexes: tenantId+userId                              │   │
│  └─────────────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────────┘

Why This Separation?

Database	Purpose	Characteristics
ai-agents	Runtime orchestration	High write, moderate read, operational
ai-memory	Learning & vectors	Vector-heavy, memory-intensive, searchable
gograb-ai-tenants	Personalization	Per-tenant isolation, GDPR-friendly

🔧 Service Migration Map

Services Moving to backend-ai-api

Service	Current Location	New Location
`AgentTaskService`	`backend-api/ai-agents`	`backend-ai-api/agents`
`AgentSessionService`	`backend-api/ai-agents`	`backend-ai-api/agents`
`AgentContextService`	`backend-api/ai-agents`	`backend-ai-api/agents`
`AgentOrchestratorService`	`backend-api/ai-agents`	`backend-ai-api/agents`
`AgentLearningBridgeService`	`backend-api/ai-agents`	`backend-ai-api/learning`
`AgentVectorMemoryService`	`backend-api/ai-agents`	`backend-ai-api/memory`
`AgentPatternMiningService`	`backend-api/ai-agents`	`backend-ai-api/intelligence`
`ModuleGenerationAgentService`	`backend-api/ai-agents`	`backend-ai-api/generation`
`WidgetGenerationAgentService`	`backend-api/ai-agents`	`backend-ai-api/generation`
`SpaceAssistantService`	`backend-api/ai-agents`	`backend-ai-api/assistant`
`TenantAIService`	`backend-api/ai-agents`	`backend-ai-api/tenants`
`UnifiedAgentStorageService`	`backend-api/ai-agents`	`backend-ai-api/storage`
`PlatformIntelligenceService`	`backend-api/platform-intelligence`	`backend-ai-api/intelligence`
`AiLearningService`	`backend-api/ai-learning`	`backend-ai-api/learning`
`AIService`	`backend-api/ai`	`backend-ai-api/core`

Services Staying in backend-api

Service	Reason
`SpacesService`	Core platform
`WidgetsService`	Core platform
`ThemesService`	Core platform
`UsersService`	Core platform
`AuthService`	Shared (library)
`SingularityService`	Core platform
`EcommerceService`	Core platform

🔐 Shared Authentication

Both APIs will share authentication via a common library:

// libs/libs-api/auth-core/src/lib/auth.module.ts
@Module({
  imports: [
    PassportModule,
    JwtModule.registerAsync({
      imports: [ConfigModule],
      useFactory: (config: ConfigService) => ({
        secret: config.get('JWT_SECRET'),
        signOptions: { expiresIn: '7d' },
      }),
      inject: [ConfigService],
    }),
  ],
  providers: [JwtStrategy, AuthGuard],
  exports: [JwtStrategy, AuthGuard],
})
export class AuthCoreModule {}

// Usage in backend-ai-api
@Module({
  imports: [
    AuthCoreModule, // Same auth as backend-api
    // ... AI modules
  ],
})
export class AppModule {}

🌐 Frontend Changes

Environment Configuration

// environments/environment.ts
export const environment = {
  production: false,
  apiUrl: 'http://localhost:3000',
  aiApiUrl: 'http://localhost:3001',  // NEW
};

// environments/environment.prod.ts
export const environment = {
  production: true,
  apiUrl: 'https://api.gograb.space',
  aiApiUrl: 'https://ai-api.gograb.space',  // NEW
};

Service Updates

// Before
@Injectable()
export class AIService {
  constructor(private http: HttpClient) {}
  
  chat(message: string) {
    return this.http.post('/api/ai/chat', { message });
  }
}

// After
@Injectable()
export class AIService {
  private aiApiUrl = inject(AI_API_URL); // New injection token
  
  chat(message: string) {
    return this.http.post(`${this.aiApiUrl}/api/assistant/chat`, { message });
  }
}

📊 Migration Timeline

Phase	Duration	Changes
Phase 1	Day 1	Create `backend-ai-api` application
Phase 2	Days 2-3	Migrate services, create shared auth
Phase 3	Day 4	Database restructure, migrate data
Phase 4	Day 5	Frontend updates, testing
Phase 5	Launch	Upgrade Flex → M30

⚠️ Breaking Changes

API Endpoint Changes

Old Endpoint	New Endpoint	Notes
`localhost:3000/api/ai-agents/*`	`localhost:3001/api/agents/*`	New API
`localhost:3000/api/platform-intelligence/*`	`localhost:3001/api/intelligence/*`	New API
`localhost:3000/api/ai-learning/*`	`localhost:3001/api/learning/*`	New API
`localhost:3000/api/ai/*`	`localhost:3001/api/assistant/*`	New API

Database Connection Changes

// Old (in backend-api)
MongooseModule.forRootAsync({
  connectionName: 'ai-agents',
  useFactory: () => ({
    uri: process.env.MONGODB_CONNECTION_STRING,
    dbName: 'ai-agents',
  }),
});

// New (in backend-ai-api) - Multiple connections
MongooseModule.forRootAsync({
  connectionName: 'agents',
  useFactory: () => ({
    dbName: 'ai-agents',  // Orchestration
  }),
});

MongooseModule.forRootAsync({
  connectionName: 'memory',
  useFactory: () => ({
    dbName: 'ai-memory',  // Vector-heavy
  }),
});

MongooseModule.forRootAsync({
  connectionName: 'tenants',
  useFactory: () => ({
    dbName: 'gograb-ai-tenants',  // Personalization
  }),
});

🔗 Related Documentation

Research Analysis - Why these decisions were made
New Architecture - Complete target architecture
Backend AI API Plan - Implementation details