Closing the Gap: AI Generation Pipeline

Executive Summary

Current State (Session 71): editor2's PhaseExecutorService is 100% wired. All P0 tasks completed:

• ✅ claude-agent-mcp ported to editor2
• ✅ 7 prompt files ported (~55K total chars)
• ✅ PhaseExecutorService.executePhase() wired to MCPBridgeService
• ✅ BuildValidatorService fixed to run in actual workspace
• ✅ Dockerfile updated with Claude Code CLI and claude-agent-mcp build
• ⏳ Pending: Deploy to Railway and test end-to-end

Previous State (Session 70): Orchestration existed but executePhase() was a stub with no LLM calls.

Remaining: Deploy to Railway, test complete generation flow with a real module.

Session 71 Implementation Summary

Completed Tasks

New Files Created

apps/claude-agent-mcp/
├── project.json
├── tsconfig.app.json
├── tsconfig.json
└── src/
    ├── main.ts
    ├── server.ts
    ├── types/index.ts
    ├── tools/index.ts
    └── services/
        ├── index.ts
        ├── claude-runner.ts
        ├── session-manager.ts
        └── output-parser.ts

apis/backend-ai-api/src/app/workflow/
├── prompts/phases/
│   ├── analyze.prompt.md (4.7k)
│   ├── plan.prompt.md (1.3k)
│   ├── validate.prompt.md (1.4k)
│   ├── implement.prompt.md (11k) ← KEY FILE
│   ├── integrate.prompt.md (2.5k)
│   ├── verify.prompt.md (4.6k)
│   └── test-e2e.prompt.md (1.1k)
└── services/
    └── mcp-bridge.service.ts (NEW)

Modified Files

apis/backend-ai-api/src/app/workflow/
├── phase-executor/
│   ├── phase-executor.service.ts ← Now calls MCPBridgeService
│   ├── prompt-loader.service.ts ← Loads external .md files
│   ├── build-validator.service.ts ← Fixed workspace path
│   └── phase.types.ts ← Added PhaseResultMetadata
└── workflow.module.ts ← Added MCPBridgeService

deployment/
├── Dockerfile.railway.backend-ai ← Added Claude Code CLI + claude-agent-mcp
└── deploy-backend-ai.sh ← Updated env var docs

MCP Systems: Complete Reference

Three Distinct MCP Systems

There are three separate MCP servers across the two monorepos. Understanding their distinct purposes is critical.

mcp-gograb-ai (editor2)

Purpose: AI learning and decision management system.

40+ tools including:

• preflight - Get context before task
• log_decision - Record architecture/implementation decisions
• find_similar - Search past decisions
• get_conventions - Get codebase conventions
• predict_outcome - ML-based prediction
• web_search, docs_search - External research
• code_review, test_gen - Analysis tools
• self_healing - Error diagnosis
• Session management, graph operations

NOT for: Code generation, file editing, bash execution.

Deployment: Runs locally via STDIO. Also has HTTP API at /ai-api/mcp/ on Railway.

claude-agent-mcp (editor2 - ported from gograb-modules)

Purpose: Wrapper for Claude Code CLI to enable server-side agentic code generation.

3 tools:

• claude_execute - Run new task with prompt, workingDir, maxTurns, timeout
• claude_continue - Continue existing session
• claude_status - Get CLI availability, active sessions

Key capability: Spawns claude --print --dangerously-skip-permissions with prompt via STDIN.

Why it exists: Claude Code CLI provides file tools (Read/Edit/Write), bash execution, and multi-turn agentic loop that raw API cannot replicate without significant engineering.

Deployment: Runs on Railway backend-ai-api as subprocess via STDIO MCP.

Location: apps/claude-agent-mcp/

gograb-mcp-server (gograb-modules)

Purpose: Project-specific tools for UI configuration generation.

Tools:

• get_widget_catalog - Available UI widgets
• get_widget_details - Widget configuration options
• get_data_sources - Data sources for bindings
• validate_config - Validate page/widget configuration
• suggest_layout - Layout suggestions

Resources: Widget catalog, data sources, JSON schemas, documentation.

Deployment: Runs locally via STDIO. Used by Claude Code during generation.

MCP Interaction Flow (Updated for editor2)

┌─────────────────────────────────────────────────────────────────────┐
│                         GENERATION REQUEST                          │
│                    (from space-builder/user)                         │
└─────────────────────────────────────────────────────────────────────┘
                                  │
                                  ▼
┌─────────────────────────────────────────────────────────────────────┐
│                    backend-ai-api (Railway)                          │
│  ┌─────────────────────────────────────────────────────────────┐    │
│  │  PhaseExecutorService                                        │    │
│  │  - Distributed locking (Redis)                               │    │
│  │  - SSE streaming                                             │    │
│  │  - Progress tracking                                         │    │
│  │  - PromptLoaderService (loads external .md files)           │    │
│  └─────────────────────────────────────────────────────────────┘    │
│                                  │                                   │
│                    ┌─────────────┼─────────────┐                    │
│                    ▼             ▼             ▼                    │
│  ┌──────────────────┐  ┌──────────────────┐  ┌──────────────────┐  │
│  │ mcp-gograb-ai    │  │ MCPBridgeService │  │ GitSyncService   │  │
│  │ (decision log)   │  │        ↓         │  │ (push to GitLab) │  │
│  └──────────────────┘  │ claude-agent-mcp │  └──────────────────┘  │
│                        │   (subprocess)   │                         │
│                        │        ↓         │                         │
│                        │ Claude Code CLI  │                         │
│                        └──────────────────┘                         │
│                                  │                                   │
│                                  ▼                                   │
│  ┌─────────────────────────────────────────────────────────────┐    │
│  │  Temp Workspace on Server                                    │    │
│  │  /tmp/module-gen/{moduleId}/                                 │    │
│  │  - Cloned from user's GitLab repo                           │    │
│  │  - Claude Code edits files HERE                             │    │
│  │  - BuildValidatorService validates HERE                     │    │
│  │  - Pushed back to GitLab when complete                      │    │
│  └─────────────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────────────┘
                                  │
                                  ▼
┌─────────────────────────────────────────────────────────────────────┐
│                    GitLab (gograbspace group)                        │
│  - User's platform repo                                              │
│  - module-gen/* branch created                                       │
│  - CI/CD pipeline triggered                                          │
│  - Deployed to Cloudflare Pages                                      │
└─────────────────────────────────────────────────────────────────────┘

Production Architecture: Server-Side Generation

The Core Question

"How can we accomplish code generation for space-builders (users) in production? We can't run Claude Code locally on their machines."

The Answer: Server-Side Generation

Code generation runs on YOUR servers, not on user machines.

The workflow:

1. User (space-builder) requests module via UI/API
       ↓
2. backend-ai-api receives request
       ↓
3. Creates temp workspace on Railway server:
   /tmp/module-gen/{tenantId}/{moduleId}/
       ↓
4. Clones user's GitLab repo to temp workspace
   (or scaffolds new module structure)
       ↓
5. MCPBridgeService spawns claude-agent-mcp (STDIO subprocess)
       ↓
6. claude-agent-mcp spawns Claude Code CLI:
   claude --print --dangerously-skip-permissions --max-turns 30
       ↓
7. Claude Code runs with FULL capabilities:
   - Reads existing code patterns (Glob, Grep, Read)
   - Creates/edits files (Write, Edit)
   - Runs validation (Bash: nx build)
   - Iterates until task complete
       ↓
8. BuildValidatorService validates in workspace
       ↓
9. Generated files committed to module-gen/* branch
       ↓
10. Pushed to user's GitLab repo
       ↓
11. GitLab CI/CD triggered
       ↓
12. Deployed to Cloudflare Pages
       ↓
13. Temp workspace cleaned up

Why This Works

1. Claude Code CLI runs on server - Not on user's machine
2. Workspace is temporary - Created per-request, cleaned up after
3. User's code stays in GitLab - Never stored permanently on generation server
4. Isolated per-tenant - Each user's generation runs in isolated workspace
5. Distributed locking - Prevents concurrent generations for same module

Railway Deployment Configuration (Session 71)

Dockerfile.railway.backend-ai now includes:

# Install Claude Code CLI globally
RUN npm install -g @anthropic-ai/claude-code

# Build claude-agent-mcp
RUN npx nx build claude-agent-mcp --skip-nx-cache

# Environment for Claude agent
ENV CLAUDE_AGENT_MCP_PATH=/app/dist/apps/claude-agent-mcp/main.js
ENV CLAUDE_AGENT_MAX_TURNS=30
ENV CLAUDE_AGENT_TIMEOUT=600000
ENV CLAUDE_AGENT_SKIP_PERMISSIONS=true

Required Railway Environment Variables:

# Existing
NODE_ENV=production
JWT_SECRET=<same as backend-api>
MONGODB_CONNECTION_STRING=<your MongoDB URI>
ALLOWED_ORIGINS=https://gograb.space,http://localhost:5200

# NEW for code generation
ANTHROPIC_API_KEY=<REQUIRED - enables Claude Code CLI>
CLAUDE_AGENT_MCP_PATH=/app/dist/apps/claude-agent-mcp/main.js
WORKSPACE_ROOT=/app

Memory Requirements: 2GB minimum, 4GB recommended.

Monorepo Comparison (Updated Session 71)

Architecture Overview

Execution Flow Comparison

gograb-modules (working):

ai-gateway → PhaseExecutor → MCPBridgeService.executeToolCall('claude_execute')
    → claude-agent-mcp (STDIO) → spawn('claude', ['--print', '--dangerously-skip-permissions'])
    → Claude Code runs with Read/Edit/Write/Bash tools → multi-turn iteration → output parsed

editor2 (Session 71 - NOW WORKING):

backend-ai-api → PhaseExecutorService.executePhase()
    → PromptLoaderService.getPrompt() (loads external .md files)
    → MCPBridgeService.executeTask()
    → claude-agent-mcp (STDIO subprocess)
    → spawn('claude', ['--print', '--dangerously-skip-permissions'])
    → Claude Code runs with full capabilities
    → OutputParser extracts files created/modified
    → BuildValidatorService.validate() (in actual workspace)

Technical Debt (Session 71 Status)

1. BuildValidatorService runs in temp directory ✅ FIXED - Now runs in actual workspace
2. PromptLoaderService has embedded trivial prompts ✅ FIXED - Loads external 55K prompts
3. No MCP client for Claude execution ✅ FIXED - MCPBridgeService created

AI Generation Pipeline

Why Claude Code CLI, Not Raw API

Key insight: editor2 now runs Claude as a FULL AGENT per phase. Claude iterates and self-corrects. This is why generated code quality will be high.

claude-agent-mcp Structure (Ported to editor2)

Location: editor2/apps/claude-agent-mcp/ (~40KB)

apps/claude-agent-mcp/src/
├── main.ts (1.2k) - Entry point
├── server.ts (5.4k) - MCP server setup
├── services/
│   ├── claude-runner.ts (16k) - CLI execution, timeout, streaming
│   │   - Spawns: claude --print --dangerously-skip-permissions
│   │   - Passes prompt via STDIN (avoids ENAMETOOLONG)
│   │   - Handles timeout, noTimeout, maxTurns, noMaxTurns
│   ├── session-manager.ts (6.4k) - Session tracking per moduleId
│   └── output-parser.ts (7.1k) - Parse files created/modified from output
├── tools/index.ts (11k)
│   ├── claude_execute - Run new task
│   ├── claude_continue - Continue session
│   └── claude_status - Check availability
└── types/index.ts

Known Failure Modes

Prompt Quality (Session 71 - Ported)

Total: ~27K chars of detailed, battle-tested prompts.

Forward Roadmap

Short Term - COMPLETED (Session 71)

Short Term - REMAINING

Mid Term (1-2 months)

Long Term (3-6 months)

What to Avoid

• Don't rewrite orchestration (it's complete)
• Don't build custom file tools yet (use Claude Code's)
• Don't merge mcp-gograb-ai and claude-agent-mcp (different concerns)
• Don't use fresh-opt branch (use fresh with 7 phases)
• Don't run generation on user machines (server-side only)

Success Criteria (Session 71 Update)

Open Questions - ANSWERED (Session 71)

1. Naming: Rename gograb-mcp-server to gograb-ui-catalog-mcp?

Answer: Low priority. This is in gograb-modules, not editor2. The current name is acceptable for now. Consider renaming in a future gograb-modules session if confusion arises.

2. Naming: Rename claude-agent-mcp to claude-code-executor-mcp?

Answer: Not recommended. The name claude-agent-mcp is already established in gograb-modules and now ported to editor2. Renaming would create divergence between the two monorepos. The key differentiator from mcp-gograb-ai is clear from the purpose (execution vs decision tracking).

3. Worker architecture: Single backend-ai-api or separate generation workers?

Answer: Start with single backend-ai-api (current implementation). Consider workers later when:

• More than 5-10 concurrent generations needed
• Generation times consistently exceed 10 minutes
• Need better resource isolation

Current approach: backend-ai-api spawns claude-agent-mcp as subprocess. Simple, works, no additional infrastructure needed.

Future option: Add BullMQ + Redis queue when scaling is required. The current architecture supports this - PhaseExecutorService can be refactored to enqueue jobs instead of direct execution.

4. Railway resources: How much memory/CPU for Claude Code execution?

Answer:

• Minimum: 2GB RAM
• Recommended: 4GB RAM
• CPU: 1-2 cores sufficient (IO-bound, not CPU-bound)

Claude Code CLI is the main memory consumer. Typical usage:

• Claude Code process: 500MB-1.5GB
• backend-ai-api: 200-400MB
• claude-agent-mcp: 50-100MB
• System overhead: 200-300MB

Recommendation: Start with 2GB, monitor, upgrade to 4GB if OOM errors occur.

5. Timeout handling: How to handle 20+ minute generations gracefully?

Answer: Multi-layer timeout handling:

1. Phase-level timeouts (already implemented in calculatePhaseTimeout()):

   • analyze: 2 minutes
   • plan: 3 minutes
   • validate: 1 minute
   • implement: 15 minutes
   • integrate: 5 minutes
   • verify: 3 minutes
   • test-e2e: 5 minutes

2. Claude agent timeout: CLAUDE_AGENT_TIMEOUT=600000 (10 minutes default, can be overridden per phase)

3. HTTP request timeout: Client should use long polling or SSE (not single request). Already have:

   • SSE streaming via EventEmitter2
   • Progress events emitted per phase

4. Job ID pattern (for very long generations):

   POST /ai-api/generate/module → returns { jobId: "..." }
   GET /ai-api/generate/status/{jobId} → returns progress
   SSE /ai-api/generate/stream/{jobId} → real-time updates
   

5. Graceful degradation: If a phase times out:

   • PhaseExecutorService catches the error
   • Emits phase_error event
   • Cleans up artifacts
   • Releases distributed lock
   • Returns partial result with error details

Appendix: MCP Quick Reference

mcp-gograb-ai Tools (editor2)

// Decision tracking
preflight({ task: "..." })
log_decision({ type, question, chosen, reasoning, confidence })
find_similar({ query: "..." })

// Conventions
get_conventions({ category: "..." })
add_convention({ category, rule, examples, source })

// Analysis
code_review({ action: "review", files: [...] })
test_gen({ action: "generate", component: "..." })
self_healing({ action: "diagnose", error: "..." })

claude-agent-mcp Tools (editor2)

// Execute new generation
claude_execute({
  prompt: "Full prompt text",
  moduleId: "unique-id",
  workingDir: "/path/to/workspace",
  maxTurns: 30,
  timeout: 600000,
  noTimeout: false,
  verbose: true
})

// Continue session
claude_continue({
  sessionId: "session-123",
  prompt: "Additional instructions"
})

// Check status
claude_status({
  moduleId: "optional-filter"
})

Branch State

• editor2: editor2-session-71
• gograb-modules: fresh ← Reference for prompts (7 phases)
• gograb-modules: fresh-opt ← DO NOT USE (4 consolidated phases)

Deployment Commands

Git-Based Deployment (Recommended)

The existing git-based deployment process works unchanged with all Session 71 additions.

How it works:

1. Railway watches the deploy/ai-api branch
2. When pushed, Railway reads railway.toml at repo root
3. railway.toml points to deployment/Dockerfile.railway.backend-ai
4. The Dockerfile builds both backend-ai-api AND claude-agent-mcp
5. All new features automatically included

# Deploy backend-ai-api with new Claude agent features
git checkout deploy/ai-api
git merge main
git push origin deploy/ai-api
git checkout main

# Verify deployment
curl -s https://ai-api.gograb.space/ai-api/health | jq

Railway Watch Paths Recommendation

Update watch paths in Railway dashboard to include claude-agent-mcp:

1. Go to: https://railway.com/project/56e862b3-755b-463b-921c-fc38fa628de7
2. Navigate to service Settings → Build
3. Add watch path: apps/claude-agent-mcp/**

Current watch paths:

• apis/backend-ai-api/**
• libs/libs-api/**
• package.json, yarn.lock

Add: apps/claude-agent-mcp/**

Build and Verify Locally

# Build claude-agent-mcp first
npx nx build claude-agent-mcp

# Build backend-ai-api
npx nx build backend-ai-api

# Verify builds exist
ls -la dist/apps/claude-agent-mcp/main.js
ls -la dist/apis/backend-ai-api/main.js

Alternative: CLI Deployment (Deprecated)

# Direct Railway CLI deployment (if git-based fails)
./deployment/deploy-backend-ai.sh

# Check Railway dashboard
# https://railway.com/project/56e862b3-755b-463b-921c-fc38fa628de7

Session 70: Initial handoff document Session 71: Implementation complete, Dockerfile updated, answers documented Next action: Deploy to Railway, test E2E generation flow