50.8 Universe Deployment Testing Guide

Documentation for Sessions

Universe Deployment Testing Guide - Session 50.8

Executive Summary

This document provides a complete step-by-step guide for testing and deploying a new universe through the GoGrab Space platform. It covers the entire flow from universe creation to live deployment on Cloudflare Pages.

Architecture Overview

┌─────────────────────────────────────────────────────────────────────────────┐
│                         DEPLOYMENT PIPELINE                                  │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│  1. VIBE/VISUAL BUILDER                                                      │
│     └── User designs universe (modules, layout, branding)                    │
│                                                                              │
│  2. CODE GENERATION (backend-ai-api:3001)                                    │
│     ├── PlatformCodeGeneratorService → Angular app shell                     │
│     ├── ModuleCodeGeneratorService → NX module libraries                     │
│     └── PlatformDeploymentBridgeService → Orchestrates everything            │
│                                                                              │
│  3. CODE STORAGE (backend-api:3000 → GitLab)                                 │
│     ├── PlatformCodeService → High-level orchestration                       │
│     ├── GitLabProviderService → GitLab API v4                                │
│     └── Creates: organization/{tenant-slug}/{universe-slug}                  │
│                                                                              │
│  4. CI/CD PIPELINE (GitLab)                                                  │
│     ├── .gitlab-ci.yml (auto-generated per project)                          │
│     ├── build: npm ci && npm run build                                       │
│     ├── deploy: wrangler pages deploy dist/browser                           │
│     └── notify: POST /webhooks/gitlab/pipeline (callback)                    │
│                                                                              │
│  5. HOSTING (Cloudflare Pages)                                               │
│     ├── CloudflarePagesService → Project management                          │
│     ├── Production: https://{universe-slug}.pages.dev                        │
│     ├── Custom domain: https://{universe-slug}.gograb.space                  │
│     └── Preview: https://{branch}.{universe-slug}.pages.dev                  │
│                                                                              │
│  6. STATUS TRACKING (MongoDB)                                                │
│     └── BuildPipelineService → Deployment records                            │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘

Prerequisites Checklist

Required Environment Variables

backend-api/.env.local:

# MongoDB
MONGODB_CONNECTION_STRING=mongodb+srv://...
MONGODB_DATABASE=gograb-api

# GitLab
GITLAB_URL=https://gitlab.com
GITLAB_API_TOKEN=glpat-xxxx                    # Personal Access Token with api scope
GITLAB_WEBHOOK_SECRET=your-webhook-secret      # For webhook validation

# Cloudflare
CLOUDFLARE_ACCOUNT_ID=your-account-id
CLOUDFLARE_API_TOKEN=your-api-token            # API Token with Pages:Edit permissions

# Security
JWT_SECRET=your-jwt-secret
SYSTEM_TOKEN=your-system-token                 # For internal service auth

# URLs
API_BASE_URL=https://backend-api-production.railway.app
DOMAIN_SUFFIX=gograb.space

backend-ai-api/.env.local:

# Backend API Communication
BACKEND_API_URL=http://localhost:3000          # Or production URL
BACKEND_API_INTERNAL_TOKEN=your-internal-token # Must match backend-api SYSTEM_TOKEN

# MongoDB (AI database)
MONGODB_AI_CONNECTION_STRING=mongodb+srv://...
MONGODB_AI_DATABASE=gograb-ai

GitLab Project CI/CD Variables (per project):

CLOUDFLARE_ACCOUNT_ID=xxx
CLOUDFLARE_API_TOKEN=xxx
GOGRAB_WEBHOOK_SECRET=xxx
GOGRAB_UNIVERSE_ID=the-universe-id

Required Services Running

Service	Port	Status Check
backend-api	3000	`curl http://localhost:3000/health`
backend-ai-api	3001	`curl http://localhost:3001/health`
MongoDB	27017	Connection established
GitLab	-	API accessible
Cloudflare	-	API accessible

Step-by-Step Testing Guide

Phase 1: Local Service Verification

Step 1.1: Start Backend Services

# Terminal 1: Start backend-api
cd /Users/fguijarro/developer/repos/editor2
nx serve backend-api

# Terminal 2: Start backend-ai-api
nx serve backend-ai-api

Step 1.2: Verify Health Endpoints

# Check backend-api
curl http://localhost:3000/health
# Expected: {"status":"ok"}

# Check backend-ai-api
curl http://localhost:3001/health
# Expected: {"status":"ok"}

Step 1.3: Verify GitLab Connection

# Test GitLab API connection
curl -X GET "http://localhost:3000/platforms/test-connection" \
  -H "x-tenant-id: test-tenant"
# Expected: {"connected":true,"gitlabUrl":"https://gitlab.com"}

Step 1.4: Verify Cloudflare Connection

# This is checked on service init - look for log:
# "Cloudflare Pages service initialized"
# If not configured, you'll see:
# "Cloudflare credentials not configured. Pages deployment will not work."

Phase 2: Universe Creation Test

Step 2.1: Create Test Tenant

# Create a tenant organization in MongoDB (if not exists)
# This should create a GitLab group when platforms are initialized

curl -X POST "http://localhost:3000/tenants" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Test Tenant",
    "slug": "test-tenant"
  }'

Step 2.2: Initialize Universe

# Initialize a new universe (creates MongoDB collections)
curl -X POST "http://localhost:3000/api/universes/initialize" \
  -H "Content-Type: application/json" \
  -H "x-tenant-id: test-tenant" \
  -d '{
    "id": "test-universe-001",
    "name": "Test Universe",
    "slug": "test-universe"
  }'

Phase 3: Code Generation Test

Step 3.1: Test Platform Code Generator Directly

# Generate platform code without deployment (for testing)
curl -X POST "http://localhost:3001/api/workflow/generate-platform" \
  -H "Content-Type: application/json" \
  -H "x-tenant-id: test-tenant" \
  -d '{
    "universeId": "test-universe-001",
    "universeSlug": "test-universe",
    "universeName": "Test Universe",
    "tenantId": "test-tenant",
    "tenantSlug": "test-tenant",
    "modules": [],
    "options": {
      "preview": true
    }
  }'

# Expected: Returns generated files array without pushing to GitLab

Step 3.2: Verify Generated Files

Check the response includes these critical files:

project.json - NX project configuration
package.json - Dependencies
angular.json - Angular configuration
tsconfig.json - TypeScript configuration
src/main.ts - Bootstrap
src/app/app.component.ts - Root component
src/app/app.routes.ts - Routing
src/app/app.config.ts - App configuration
src/environments/environment.ts - Environment config
src/index.html - Entry HTML
src/styles.scss - Global styles

Phase 4: GitLab Integration Test

Step 4.1: Initialize Platform Repository

# Create GitLab group and repository
curl -X POST "http://localhost:3000/platforms/initialize" \
  -H "Content-Type: application/json" \
  -H "x-tenant-id: test-tenant" \
  -H "Authorization: Bearer $SYSTEM_TOKEN" \
  -d '{
    "universeId": "test-universe-001",
    "universeSlug": "test-universe",
    "universeName": "Test Universe",
    "description": "Testing platform deployment"
  }'

# Expected Response:
# {
#   "id": 12345,
#   "name": "test-universe",
#   "fullName": "test-tenant/test-universe",
#   "cloneUrl": "https://gitlab.com/test-tenant/test-universe.git",
#   "sshUrl": "[email protected]:test-tenant/test-universe.git",
#   "defaultBranch": "main",
#   "private": true
# }

Step 4.2: Verify Repository Created

Go to GitLab: https://gitlab.com/test-tenant/test-universe
Check initial files:
- README.md
- .gitlab-ci.yml
- .gitignore

Step 4.3: Push Generated Code

# Push generated code to repository
curl -X POST "http://localhost:3000/platforms/code" \
  -H "Content-Type: application/json" \
  -H "x-tenant-id: test-tenant" \
  -H "x-user-id: test-user" \
  -H "Authorization: Bearer $SYSTEM_TOKEN" \
  -d '{
    "universeId": "test-universe-001",
    "files": [
      {
        "path": "src/main.ts",
        "content": "import { bootstrapApplication } from \"@angular/platform-browser\"; ..."
      }
    ],
    "message": "[Platform] Initial deployment"
  }'

# Expected Response:
# {
#   "commitSha": "abc123...",
#   "branch": "main",
#   "filesCount": 15,
#   "commitUrl": "https://gitlab.com/test-tenant/test-universe/-/commit/abc123"
# }

Step 4.4: Verify Commit in GitLab

Go to: https://gitlab.com/test-tenant/test-universe/-/commits/main
Verify commit message and files

Phase 5: CI/CD Pipeline Test

Step 5.1: Verify Pipeline Triggered

After pushing code, GitLab CI should automatically trigger.

Go to: https://gitlab.com/test-tenant/test-universe/-/pipelines
Check pipeline status:
- build stage: Installing deps and building Angular
- deploy stage: Deploying to Cloudflare Pages
- notify stage: Calling webhook

Step 5.2: Monitor Pipeline Logs

Click on pipeline to see:

Build logs (npm ci, npm run build)
Deploy logs (wrangler pages deploy)
Notify logs (curl to webhook)

Step 5.3: Configure GitLab CI Variables

If pipeline fails due to missing variables:

Go to: Settings → CI/CD → Variables
Add:
- CLOUDFLARE_ACCOUNT_ID (protected, masked)
- CLOUDFLARE_API_TOKEN (protected, masked)
- GOGRAB_WEBHOOK_SECRET (protected, masked)
- GOGRAB_UNIVERSE_ID (protected)

Phase 6: Webhook Callback Test

Step 6.1: Simulate Webhook (for testing without GitLab)

# Simulate successful deployment callback
curl -X POST "http://localhost:3000/webhooks/gitlab/pipeline" \
  -H "Content-Type: application/json" \
  -H "X-Deployment-Secret: $GITLAB_WEBHOOK_SECRET" \
  -d '{
    "universeId": "test-universe-001",
    "status": "ready",
    "environment": "production",
    "url": "https://test-universe.pages.dev",
    "version": "v1.0.0",
    "commitSha": "abc123",
    "branch": "main",
    "pipelineId": "12345",
    "jobId": "67890"
  }'

# Expected: 200 OK with deployment record updated

Step 6.2: Verify Deployment Status Updated

# Check deployment status
curl -X GET "http://localhost:3000/api/universes/test-universe-001/deployment-status" \
  -H "x-tenant-id: test-tenant" \
  -H "x-user-id: test-user"

# Expected Response:
# {
#   "universeId": "test-universe-001",
#   "status": "published",
#   "deploymentUrl": "https://test-universe.pages.dev",
#   "lastPublished": "2024-01-17T..."
# }

Phase 7: Cloudflare Pages Verification

Step 7.1: Check Cloudflare Pages Project

Go to: https://dash.cloudflare.com/{account-id}/pages
Find project: test-universe
Verify:
- Production deployment exists
- URL is accessible
- Environment variables are set

Step 7.2: Test Live Deployment

# Test production URL
curl -I "https://test-universe.pages.dev"
# Expected: HTTP/2 200

# Test custom domain (if configured)
curl -I "https://test-universe.gograb.space"
# Expected: HTTP/2 200

Phase 8: Full End-to-End Test

Step 8.1: Complete Deployment Flow

# Full deployment via backend-ai-api Platform Orchestration Controller
# Endpoint: POST /workflow/platform/:spaceId/universes/:universeId/deploy-platform
curl -X POST "http://localhost:3001/workflow/platform/test-tenant/universes/e2e-test-universe/deploy-platform" \
  -H "Content-Type: application/json" \
  -d '{
    "tenantSlug": "test-tenant",
    "userId": "test-user",
    "universeSlug": "e2e-test",
    "universeName": "E2E Test Universe",
    "description": "End-to-end deployment test",
    "branding": {
      "primaryColor": "#3b82f6",
      "secondaryColor": "#1e40af"
    },
    "layout": {
      "template": "split-sidebar",
      "sidebarPosition": "left"
    },
    "options": {
      "autoDeploy": true,
      "version": "v1.0.0",
      "changelog": "Initial deployment"
    }
  }'

# Expected Response:
# {
#   "success": true,
#   "universeId": "e2e-test-universe",
#   "generation": {
#     "platformFiles": 15,
#     "moduleFiles": 0,
#     "totalLines": 1500
#   },
#   "push": {
#     "commitSha": "abc123",
#     "branch": "main",
#     "commitUrl": "..."
#   },
#   "version": "v1.0.0",
#   "deployment": {
#     "id": "deploy_xxx",
#     "status": "pending",
#     "url": "https://e2e-test.pages.dev"
#   }
# }

Step 8.2: Wait and Verify

Wait 2-5 minutes for pipeline to complete
Check deployment status (should transition: pending → cloning → building → uploading → ready)
Access live URL: https://e2e-test.pages.dev

Critical Gaps Identified

1. Draft Branch Cleanup (Minor)

Issue: Preview environments (draft branches) don't have automatic cleanup mechanism.

Current State:

Drafts created → Preview deployed to Cloudflare
When draft is discarded → Branch deleted from GitLab
Preview deployment NOT automatically cleaned up from Cloudflare (orphaned)

Fix: Add Cloudflare Pages deployment deletion when drafts are discarded.

2. DeploymentService vs BuildPipelineService (Technical Debt)

Issue: Two deployment services exist with different purposes:

Service	Location	Purpose	Storage
`DeploymentService`	`deployment/deployment.service.ts`	Legacy local bundler	In-memory Map
`BuildPipelineService`	`build-pipeline/build-pipeline.service.ts`	GitLab/Cloudflare pipeline	MongoDB

Recommendation: Deprecate DeploymentService or migrate its storage to MongoDB for consistency.

3. Generated Package Dependencies

Issue: Generated package.json may be missing dependencies for @libs-renderable components.

Verify: After code generation, ensure these dependencies exist:

@angular/cdk (if using CDK components)
All @libs-* paths resolved correctly

What's Working Well

1. Deployment Endpoint EXISTS

Location: POST /workflow/platform/:spaceId/universes/:universeId/deploy-platform

Controller: platform-orchestration.controller.ts

Features:

Full deployment flow (generate → push → deploy)
Preview code generation (without deployment)
Module deployment
Platform validation
State aggregation

2. Health Endpoints EXIST

Both services have working /health endpoints:

GET http://localhost:3000/health → backend-api
GET http://localhost:3001/health → backend-ai-api

3. Complete Pipeline Infrastructure

GitLabProviderService - Full GitLab API v4 implementation (1100+ lines)
CloudflarePagesService - Complete Cloudflare Pages integration
BuildPipelineService - Full deployment lifecycle management
WebhookController - GitLab callback handling
Auto-generated .gitlab-ci.yml - Created per-project with Cloudflare deployment

Verification Checklist

Pre-Deployment

Backend-api running on port 3000
Backend-ai-api running on port 3001
MongoDB connected
GitLab API token valid
Cloudflare API token valid
BACKEND_API_INTERNAL_TOKEN matches between services

Code Generation

Platform files generated (15+ files)
Module files generated (if modules selected)
TypeScript compiles without errors
Angular configuration valid

GitLab

Organization/group exists
Repository created
.gitlab-ci.yml present
Initial commit pushed
CI/CD variables configured

Pipeline

Pipeline triggered on push
Build stage succeeds
Deploy stage succeeds
Notify stage succeeds

Cloudflare

Pages project created
Production deployment live
Custom domain configured (if applicable)
Environment variables set

Final

Live URL accessible
Deployment status "ready" in MongoDB
No console errors in deployed app

Troubleshooting

Pipeline Fails at Build Stage

Symptoms: npm ci or npm run build fails

Causes:

Missing dependencies in generated package.json
TypeScript errors in generated code
Missing @libs imports

Fix:

Check build logs for specific error
Verify package.json includes all dependencies
Ensure TypeScript paths configured correctly

Pipeline Fails at Deploy Stage

Symptoms: wrangler pages deploy fails

Causes:

Missing Cloudflare credentials
Invalid project name
Build output directory wrong

Fix:

Verify CI/CD variables: CLOUDFLARE_ACCOUNT_ID, CLOUDFLARE_API_TOKEN
Check project name is DNS-safe (lowercase, alphanumeric, hyphens)
Verify dist/browser exists after build

Webhook Not Received

Symptoms: Deployment stays in "building" state forever

Causes:

Webhook secret mismatch
Backend-api not accessible from GitLab
Network/firewall blocking

Fix:

Verify X-Deployment-Secret header matches GITLAB_WEBHOOK_SECRET
Ensure backend-api has public URL
Check GitLab's webhook delivery logs

Deployment URL Returns 404

Symptoms: Pages URL returns 404 Not Found

Causes:

Build output not in expected location
Cloudflare Pages routing misconfigured
SPA fallback not configured

Fix:

Add _redirects file: /* /index.html 200
Or configure in wrangler.toml

Next Steps for Refinement

Implement Draft Cleanup - Remove Cloudflare preview deployments when drafts are discarded
Add Integration Tests - Automated E2E tests for the complete deployment flow
Add Deployment Notifications - Slack/email/webhook notifications on success/failure
Create Admin Dashboard - UI to monitor deployments across tenants
Deprecate Legacy DeploymentService - Remove or migrate the in-memory deployment service
Add Rollback Testing - Verify rollback to previous versions works correctly

Key Endpoints Reference

Action	Endpoint	Service
Health Check	`GET /health`	backend-api, backend-ai-api
Initialize Platform	`POST /platforms/initialize`	backend-api
Push Code	`POST /platforms/code`	backend-api
Create Version	`POST /platforms/versions`	backend-api
Full Deployment	`POST /workflow/platform/:spaceId/universes/:universeId/deploy-platform`	backend-ai-api
Preview Code	`GET /workflow/platform/:spaceId/universes/:universeId/preview-code`	backend-ai-api
Webhook Callback	`POST /webhooks/gitlab/pipeline`	backend-api
Deployment Status	`GET /api/universes/:id/deployment-status`	backend-api

Session 50.8 Notes

Document created: 2025-01-17
Validates complete deployment pipeline architecture
Infrastructure is ~95% complete
Minor gaps: draft cleanup, legacy service deprecation
Provides comprehensive testing guide for manual validation