Module Generation Architecture — Complete Knowledge Base

Last updated: 2026-02-23 (Session 138.8) Source: Testing sessions S137–S138.8, MongoDB Atlas analysis, codebase tracing

This document captures everything learned about the platform's module generation pipeline, data persistence, and JSON-to-Angular rendering system. It should be treated as the definitive reference for anyone working on the generation flow.

Table of Contents

1. Pipeline Overview
2. The 7 Phases in Detail
3. SSE Streaming & Frontend Communication
4. Database Persistence Architecture
5. JSON-to-Angular: How Modules Become UI
6. Document Schemas (Full Reference)
7. Key IDs & Mappings
8. File Locations (Codebase Map)
9. Configuration & Defaults
10. Known Bugs & Their Root Causes
11. Performance Data
12. Process Persistence (Session Recovery)
13. Deploy to Universe Flow
14. Testing Checklist

1. Pipeline Overview

Pipeline B (backend-vitto-api/module-generation/) is the ACTIVE generation pipeline. It is triggered from the Vibe Universe Builder UI at:

http://localhost:6200/space/{spaceId}/universe/{universeId}/vibe

The pipeline runs on backend-vitto-api (port 3003) and communicates with the frontend via Server-Sent Events (SSE).

Architecture Diagram

┌──────────────────────┐     SSE Stream      ┌──────────────────────┐
│  Frontend (Angular)  │ ◄──────────────────  │  backend-vitto-api   │
│  Port 6200           │                      │  Port 3003           │
│                      │  GET /start?params   │                      │
│  vibe-universe-      │ ──────────────────►  │  module-generation/  │
│  generator.component │                      │  controller.ts       │
│                      │                      │                      │
│  workflow-state.     │                      │  module-gen-phase.   │
│  service.ts          │                      │  executor.ts         │
└──────────┬───────────┘                      └──────────┬───────────┘
           │                                             │
           │ POST /spaces/:id/modules                    │ POST /ai-api/modules
           ▼                                             ▼
┌──────────────────────┐                      ┌──────────────────────┐
│  backend-api         │                      │  backend-ai-api      │
│  Port 3000           │                      │  Port 3001           │
│                      │                      │                      │
│  space_modules       │                      │  generated-modules   │
│  (GoGrabSpace DB)    │                      │  (ai-agents DB)      │
└──────────────────────┘                      └──────────────────────┘

Entrypoints

2. The 7 Phases in Detail

Each phase is implemented as a separate class in apis/backend-vitto-api/src/app/module-generation/phases/.

Phase 1: ENHANCE

• Purpose: Takes the raw user prompt (e.g., "Fitness Studio") and enriches it with context, requirements, and technical details.
• Input: Raw prompt string + template selection
• Output: enhancedPrompt (enriched text string)
• Key bug fixed: Bug 1 — enhancedPrompt.substring is not a function (was returning an object instead of string)
• Duration: ~15-30s

Phase 2: ANALYZE

• Purpose: Analyzes the enhanced prompt to determine platform type, target audience, key features, and technical requirements.
• Input: Enhanced prompt
• Output: Analysis JSON with platform profile
• Key bug fixed: Bug 3 — JSON truncation at position 14669 (maxTokens raised to 8192, added parseLlmJson fallback)
• Duration: ~30-60s

Phase 3: DECOMPOSE

• Purpose: Breaks the platform into individual modules. Determines module names, descriptions, and relationships.
• Input: Analysis results
• Output: { modules: ModuleSpec[] } — array of module specifications
• Output example: For "Fitness Studio" → member-management, trainer-management, class-scheduling, workout-tracking, studio-dashboard
• Duration: ~30-60s

Phase 4: PLAN

• Purpose: Creates an execution plan for each module — what to generate, in what order, with dependencies.
• Input: Module specifications from DECOMPOSE
• Output: executionPlan[] — ordered array of generation tasks
• Key bug fixed: Bug 2 — executionPlan is not iterable (was receiving an object wrapper)
• Duration: ~20-40s

Phase 5: EXECUTE (Most Complex)

• Purpose: Generates the actual module content — actions, entities, embeddables, and optionally code strings.
• Input: Execution plan + module specs
• Output: Array of GeneratedModule objects
• Key implementation details:
  • Has TWO paths controlled by outputMode:
    • Path A (Visual): Generates GeneratedModule JSON via implement-visual.prompt — actions, embeddables, entities
    • Path B (Code): Generates .ts/.html/.scss code strings via implement.prompt — NEVER PERSISTED
    • BOTH mode (default): Runs both paths sequentially (~180s/module)
  • Uses normalizeForMatch() for case/format-insensitive module name matching between DECOMPOSE and PLAN outputs
  • Each module has 90s timeout via withTimeout() (Promise.race)
  • Emits per-module SSE progress: "Generating module: trainer-management (2/5)"
• Key bugs fixed: Bug 4 (name mismatch), Bug 6 (timeout), Bug 9 (per-module progress)
• Duration: ~5-8 minutes for 5 modules

Phase 6: INTEGRATE

• Purpose: Validates and connects the generated modules — ensures cross-module references work, navigation is consistent.
• Input: Generated modules
• Output: Integrated module set with navigation structure
• Duration: ~15-30s

Phase 7: DEPLOY

• Purpose: Persists the generated modules to the database.
• Input: Integrated module set + spaceId + universeId
• Output: Persisted modules
• Key implementation:
  • Calls persistGeneratedModules() in backend-ai-client.service.ts
  • Posts to POST {baseUrl}/modules on backend-ai-api (port 3001)
  • Only fires when spaceId AND universeId are provided
  • Potential Bug 12: May not execute for every run (evidence: latest test didn't update timestamps)
• Duration: ~5-15s

3. SSE Streaming & Frontend Communication

SSE Event Types

The pipeline emits ModuleGenSSEEvent objects through Server-Sent Events:

interface ModuleGenSSEEvent {
  type: 'phase-started' | 'phase-completed' | 'progress' | 'module-gen.completed' | 'error';
  phase?: string;           // 'enhance' | 'analyze' | ... | 'deploy'
  progress?: number;        // 0-100
  message?: string;         // Human-readable status
  data?: any;               // Phase result data
  sessionId?: string;       // e.g. "modgen-9e2521fb"
  manifestJSON?: string;    // Only on completion — massive JSON blob
}

Frontend Handling

1. WorkflowStateService.startWorkflow() creates an EventSource to /module-generation/start
2. Each SSE event updates signals: currentPhase, progress, moduleProgress
3. On module-gen.completed, handleWorkflowComplete() fires — this is where Bug 10 occurs (6 synchronous blocking operations)

Reconnection

• Backend stores last 50 events in a ReplaySubject per session
• GET /stream/:sessionId reconnects and replays buffered events
• Sessions are in-memory (Map<string, ModuleGenSession>) — lost on server restart

4. Database Persistence Architecture

Three Collections, Two Clusters

┌─────────────────────────────────────────────────────────────┐
│                    GoGrabSpace Cluster                       │
│                                                             │
│  ┌──────────────────────┐  ┌──────────────────────────────┐ │
│  │ ai-agents database   │  │ space_chrome-space-1_* DB    │ │
│  │                      │  │                              │ │
│  │ generated-modules    │  │ space_modules (1,615 docs)   │ │
│  │ (173 docs)           │  │ - Lightweight registry       │ │
│  │ - Full schema        │  │ - No actions/embeddables     │ │
│  │ - actions[]          │  │ - Written by frontend        │ │
│  │ - embeddables[]      │  │   via backend-api            │ │
│  │ - entities[]         │  │                              │ │
│  │ - Written by backend │  │                              │ │
│  │   via backend-ai-api │  │                              │ │
│  └──────────────────────┘  └──────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────┐
│                      Vitto Cluster                           │
│                                                             │
│  ┌──────────────────────────────────────────────────────┐   │
│  │ vitto-mind database                                   │   │
│  │                                                       │   │
│  │ generation_checkpoints (13 docs)                      │   │
│  │ - Pipeline progress tracking                          │   │
│  │ - Phase results for resume                            │   │
│  │ - Error messages on failure                           │   │
│  │ - TTL: 7 days for incomplete sessions                 │   │
│  │ - Written by backend-vitto-api                        │   │
│  └──────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────┘

Write Responsibility Matrix

Upsert Strategy

• generated-modules: findOneAndUpdate({ moduleId, spaceId }, { $set: {...} }, { upsert: true })
  • Unique index on { moduleId: 1, spaceId: 1 }
  • Re-running a generation for the same template updates existing docs (not duplicates)
• space_modules: Simple create() with _id = "mod-page-module-{slug}-{timestamp}"
  • Can create duplicates across runs (timestamp makes _id unique)

5. JSON-to-Angular: How Modules Become UI

The Core Insight

Generated modules NEVER become static Angular component files. The .ts/.html/.scss code strings produced by the EXECUTE phase (Path B) are ephemeral — they exist only in the SSE payload and the manifestJSON blob. They are never compiled, written to disk, or executed as Angular code.

Instead, the platform uses a schema-driven dynamic rendering approach:

The Rendering Pipeline

1. AI generates structured JSON definitions:
   - actions[] → CRUD operations with typed inputs/outputs
   - embeddables[] → UI component specs with inputs/outputs
   - entities[] → Data model definitions

2. These are stored in `generated-modules` collection

3. At runtime, Angular reads these definitions via API

4. Generic rendering components interpret the JSON:
   - EmbeddableRenderer reads embeddable.category ("list", "widget", "form", "detail")
   - ActionFormRenderer reads action.inputs[] to build dynamic forms
   - EntityRenderer reads entity fields for validation

5. Result: Fully functional CRUD UI from pure JSON — no compilation needed

Embeddable Categories

Each module generates 4 embeddables:

Action Categories

Each module generates 5 CRUD actions:

Input Type System

Actions define typed inputs that drive form rendering:

{ "name": "startTime", "type": "datetime", "required": true, "label": "Start Time" }
{ "name": "capacity", "type": "number", "required": false, "label": "Capacity" }
{ "name": "description", "type": "string", "required": false, "label": "Description" }
{ "name": "items", "type": "array", "label": "Class Items" }
{ "name": "item", "type": "object", "label": "Class data" }
{ "name": "loading", "type": "boolean", "label": "Loading state" }

Supported types: string, number, boolean, date, datetime, array, object

Why Code Strings Are Generated But Not Used

The outputMode: BOTH setting runs Path B after Path A, generating .ts, .html, .scss code per module. This:

• Doubles generation time (~180s vs ~90s per module)
• Produces code that goes into manifestJSON as a stringified blob
• Is likely intended for a future "export to standalone Angular project" feature
• Has zero impact on the current runtime rendering

Recommendation: Default to VISUAL_ONLY until the export feature is built.

6. Document Schemas (Full Reference)

generation_checkpoints

interface GenerationCheckpoint {
  _id: ObjectId;
  sessionId: string;                    // "modgen-{8-char-uuid}"
  request: {
    prompt: string;
    templateId?: string;
    spaceId: string;
    universeId: string;
    outputMode: 'visual' | 'code' | 'both';
  };
  status: 'in-progress' | 'completed' | 'failed';
  completedPhases: string[];            // ["enhance", "analyze", ...]
  currentPhase: string | null;          // null when completed
  tokensUsed: Record<string, number>;   // Per-phase token counts
  durationMs: number;                   // Total pipeline duration
  spaceId: string;
  universeId: string;                   // Internal long-form ID
  phaseResults: {
    completedAt?: Date;
    // Per-phase result data for resume
    enhance?: { enhancedPrompt: string };
    analyze?: { analysis: object };
    decompose?: { modules: ModuleSpec[] };
    plan?: { executionPlan: PlanItem[] };
    execute?: { generatedModules: GeneratedModule[] };
    integrate?: { integratedModules: object };
    deploy?: { deployedModules: object };
  };
  error?: string;                       // Error message when status = "failed"
  createdAt: Date;
  updatedAt: Date;
}

generated-modules

interface GeneratedModuleDoc {
  _id: ObjectId;
  moduleId: string;                     // "fitness-studio-class-management"
  spaceId: string;                      // "69250a03857479fe44376823"
  __v: number;

  // Core identity
  name: string;                         // "Class Management"
  slug: string;                         // "fitness-studio-class-management"
  description: string;                  // "Manage class data and operations"
  category: string;                     // "business" | "productivity" | "other"
  status: 'draft' | 'published';
  version: string;                      // "1.0.0"

  // Provenance
  createdBy: string;                    // "vibe-universe"
  universeId: string;                   // "universe_176859..."
  workflowId: string;                   // "workflow-177186..."

  // THE SCHEMA DATA — this is what drives the UI
  actions: Action[];                    // 5 CRUD actions (see above)
  embeddables: Embeddable[];            // 4 UI components (list, card, form, detail)
  entities: Entity[];                   // 1 data model definition

  // Extras
  manifestJSON: string;                 // Massive stringified JSON (includes code strings)
  metadata: object;                     // AI generation metadata
  routes: any[];                        // Empty for now
  tags: string[];                       // ["fitness", "management"]

  createdAt: Date;
  updatedAt: Date;
}

space_modules

interface SpaceModuleDoc {
  _id: string;                          // "mod-page-module-{slug}-{timestamp}"
  spaceId: string;
  name: string;
  slug: string;
  type: 'page-module';
  description: string;
  icon: string | null;
  extendsGlobal: string | null;
  isCustom: boolean;                    // true for generated modules
  moduleId: string;                     // Same as generated-modules.moduleId
  pageId: string | null;
  widgetId: string | null;
  route: string | null;
  position: string | null;
  defaultConfig: {
    layout: object | null;
    styles: object | null;
    ai: object | null;
    flows: object | null;
    flowTriggers: object | null;
    flowParams: object | null;
  };
  actions: null;                        // NOT populated (unlike generated-modules)
  scope: 'universe';
  universeId: string;
  isActive: boolean;
  order: number;
  createdAt: Date;
  updatedAt: Date;
  createdBy: string | null;
}

7. Key IDs & Mappings

The Universe ID Problem

The Angular route uses a human-readable slug, but the database stores an internal long-form ID:

Always use the internal ID when querying MongoDB directly.

Space ID

Session/Process ID Format

8. File Locations (Codebase Map)

Backend — Pipeline B (backend-vitto-api)

apis/backend-vitto-api/src/app/module-generation/
├── module-generation.controller.ts        # SSE endpoints: /start, /stream/:id, /status/:id, /resume/:id
├── module-generation.module.ts
├── module-gen-phase.executor.ts           # Central orchestrator for all 7 phases
├── session/
│   └── module-gen-session.manager.ts      # In-memory Map<string, Session> — lost on restart
├── checkpoint/
│   ├── checkpoint.schema.ts               # MongoDB schema for generation_checkpoints
│   └── checkpoint.service.ts              # Save/load checkpoints
├── phases/
│   ├── enhance.phase.ts
│   ├── analyze.phase.ts
│   ├── decompose.phase.ts
│   ├── plan.phase.ts
│   ├── execute.phase.ts                   # normalizeForMatch(), withTimeout(), emitProgress()
│   ├── integrate.phase.ts
│   └── deploy.phase.ts
└── prompts/
    ├── implement-visual.prompt.ts         # Path A: generates actions/embeddables/entities JSON
    └── implement.prompt.ts                # Path B: generates .ts/.html/.scss code strings

Backend — Module Persistence (backend-ai-api)

apis/backend-ai-api/src/app/workflow/
├── generated-modules.controller.ts        # POST /ai-api/modules (upsert)
├── generated-modules.service.ts
└── schemas/
    └── generated-module.schema.ts         # MongoDB schema with indexes

Backend — Space Modules (backend-api)

apis/backend-api/src/app/spaces/
├── space-modules.controller.ts            # POST /api/spaces/:spaceId/modules
└── space-modules.service.ts               # create() method

Backend — Vitto-to-AI Bridge

apis/backend-vitto-api/src/app/connectivity/
└── backend-ai-client.service.ts           # persistGeneratedModules() (lines 523-570)

Frontend — Vibe Universe Builder

libs/libs-structural/vibe-universe/src/lib/
├── components/
│   └── vibe-universe-generator/
│       └── vibe-universe-generator.component.ts   # Main component (2000+ lines)
│           # MAX_MODULES = 20
│           # autoSyncModulesToCanvas()
│           # deployToUniverse() — 8-step deploy flow
│           # reconnectToGeneration(processId)
│           # updateUrlWithProcessId()
└── vibe-universe.routes.ts                        # process/:processId child route

Frontend — Workflow State

libs/libs-ai/gograb-ai-lib/src/lib/services/
└── workflow-state.service.ts                      # startWorkflow(), resumeWorkflow(), handleWorkflowComplete()

Frontend — Module API

libs/libs-foundational/core-angular/src/lib/services/
└── backend-module-api.service.ts                  # createSpaceModule() (line 649)

9. Configuration & Defaults

Port Configuration

API URL Tokens (Angular DI)

10. Known Bugs & Their Root Causes

Bug 10 Detail — The 6 Blocking Operations

When module-gen.completed SSE fires, these happen synchronously:

1. JSON.parse(manifestJSON) — 500-2000ms (manifest is massive)
2. platformSummary computed signal cascade — 300-800ms
3. contextMode/contextSuggestions computed — 200-500ms
4. JSON.stringify(state) + sessionStorage.setItem — 300-500ms
5. buildGeneratedUniverseTree() — 200-500ms
6. broadcastNodeCreate loop — 100-300ms

Total: 1.6-4.6 seconds of main-thread blocking → Chrome "Page Unresponsive"

11. Performance Data

Generation Times (from checkpoint data)

Average: ~10.8 minutes for 5 modules with outputMode: BOTH.

With VISUAL_ONLY, expect ~5.5 minutes (roughly half).

Collection Sizes

12. Process Persistence (Session Recovery)

What Exists (Backend)

• GET /stream/:sessionId — Reconnects to active SSE stream (ReplaySubject buffer=50)
• POST /resume/:sessionId — Resumes from last checkpoint phase
• GET /status/:sessionId — Returns current session state
• findResumableSessions() — Finds incomplete sessions in MongoDB
• Sessions stored in-memory Map — lost on server restart

What Exists (Frontend)

• process/:processId child route added to vibe-universe routes
• reconnectToGeneration(processId) method added to component
• updateUrlWithProcessId() updates URL on first SSE event
• resumeWorkflow(sessionId) exists in WorkflowStateService

What's Broken

• Bug 11: Refreshing /vibe/process/modgen-xxx redirects to /landing because the space context isn't resolved when the route guard fires on cold load
• The process/:processId route data includes resumeMode: true but the guard doesn't check for it

13. Deploy to Universe Flow

The frontend's deployToUniverse() method runs an 8-step flow after generation completes:

Step 1: Extract modules from generation result
Step 2: Map modules to SpaceModule format
Step 3: For each module:
         POST /api/spaces/:spaceId/modules
         (creates space_modules document)
Step 4: Build navigation tree
Step 5: Update universe metadata
Step 6: Sync modules to canvas (replaceTree + hydrateFromModules)
Step 7: Save state to sessionStorage
Step 8: Broadcast node creation events

Critical dependency: This entire flow runs on the frontend after the completion SSE event. If the frontend freezes (Bug 10), none of this executes.

14. Testing Checklist

When testing the generation pipeline, verify:

• Template picker loads and templates are clickable
• Clicking a template starts generation (no crash)
• URL updates to /vibe/process/modgen-xxx
• All 7 phases complete (check progress bar)
• Per-module progress shows during EXECUTE: "Generating module: X (n/N)"
• No "Page Unresponsive" dialog at completion
• Modules appear in canvas/tree after generation
• Check generation_checkpoints in MongoDB for session record
• Check generated-modules for module definitions with correct universeId
• Check space_modules for lightweight module entries
• Page refresh on /vibe/process/modgen-xxx reconnects (not redirect to /landing)
• Console free of "Module limit exceeded" warnings
• No duplicate module entries in either collection