GoGrab Platform Infrastructure Enhancement Plan

Date: March 9, 2026 (Updated: March 9, 2026 — post S191 agent changes) Prepared by: Claude (Architecture Review) Assigned to: Internal Dev Agents Scope: 7 Phases | 26 Tasks | ~4 weeks Risk Level: Medium (phased rollout mitigates) APIs Affected: backend-api, backend-ai-api, backend-vitto-api, mcp-gograb-ai

Table of Contents

1. Executive Summary
2. Current Architecture
3. Key Findings
4. Revised Positions
5. Phase 1: Circuit Breakers and Resilience
6. Phase 2: Inter-Service Retry and Caching
7. Phase 3: AI Module Migration Completion
8. Phase 4: BullMQ Event-Driven Alerting
9. Phase 5: Cloudflare Edge Layer
10. Phase 6: MCP Gateway Enhancement
11. Phase 7: Cleanup and Operational Hardening
12. Execution Timeline
13. Verification Protocol

Executive Summary

This is the complete handoff plan for enhancing the GoGrab platform infrastructure. It covers all three backend APIs (backend-api:3000, backend-ai-api:3001, backend-vitto-api:3003), the MCP server (mcp-gograb-ai), Cloudflare edge infrastructure, and the Railway deployment pipeline.

The plan is organized into 7 phases, ordered by impact and dependency. Each phase can be executed independently by a dev agent session, with clear acceptance criteria and rollback procedures.

Current Architecture

Inter-Service Dependency Map

Frontend (Angular)
  ├── → backend-api (3000)       [REST, WebSocket, Yjs]
  ├── → backend-ai-api (3001)    [REST, SSE, WebSocket, Native WS]
  └── → backend-vitto-api (3003) [REST, SSE]

backend-api (3000)
  ├── → Stripe, SendGrid, Twilio, Cloudflare, GitLab
  └── ← backend-vitto-api (resolve-channel)

backend-ai-api (3001)
  ├── → backend-api (3000)         [PlatformApiClient: code storage, CRUD]
  ├── → backend-vitto-api (3003)   [VittoClient: task delegation]
  ├── → OpenAI, Anthropic, Google  [LLM providers]
  └── ← MCP server (tool calls via HTTP + WebSocket)

backend-vitto-api (3003)
  ├── → backend-api (3000)         [user resolution, space queries]
  ├── → backend-ai-api (3001)      [LLM routing, intelligence queries]
  ├── → Twilio, WhatsApp, Telegram, Slack [channels]
  ├── → E2B [sandboxed execution]
  └── ← External webhooks (WhatsApp, Telegram inbound)

Critical circular dependency: backend-ai-api delegates tasks TO Vitto, Vitto queries intelligence FROM backend-ai-api. If either is slow, the other cascades.

Key Findings

• Circuit breakers exist in Vitto (per-LLM-provider) but NOT in backend-ai-api or backend-api. No protection on Stripe, SendGrid, Twilio, Cloudflare, or GitLab calls.

• Inter-service HTTP calls have inconsistent retry patterns. Most calls have zero retries and will silently fail. Only MCP tool execution has 2 retries with backoff. Channel identity resolution (WhatsApp user lookup) has zero retries — a failed call means a dropped message.

• AI module migration is partially complete. Dual cost tracking exists: ai_usage in gograb-api + llm_usage in ai-agents + spending_records in vitto-mind. S191 PROGRESS: Model config and pricing are now centralized in AI_MODEL_CAPABILITIES (@libs-api/backend-core), consumed by all 3 APIs and frontend. But storage targets are still split across databases.

• MCP gateway pattern is already well-designed. 98 tools consolidated into 15 gateways with action-based routing, backward compatibility aliases, and internal chaining. Enhancement should focus on capability tiers, not restructuring.

• Database separation is architecturally correct. 3 clusters (gograbspace, gograb-docs, vitto) with purpose-specific databases within each. Multi-tenant space_* and space-agents_* pattern is sound.

• Proactive monitoring uses 30-min polling. BullMQ already emits events that could drive real-time alerting without new infrastructure.

Revised Positions

Database Architecture: Keep Separated

Initial assessment suggested consolidating 9 databases to 4. After deeper analysis, this was wrong. The current separation is correct because:

• 3 separate MongoDB clusters (gograbspace, gograb-docs, vitto) provide hard failure isolation. If one cluster goes down, the others keep working.
• ai-memory is separate from ai-agents for performance isolation. Vector search is computationally heavy and would degrade decision queries if co-located.
• ai-archive has lower connection pool (5 max vs 20) because cold storage has a fundamentally different access pattern. Merging it would waste pool slots.
• ai-mcp-dev isolates MCP learning data from agent orchestration. Written by different services at different rates.
• gograb-analytics separates write-heavy analytics from the main platform database. User journey tracking should never block a space CRUD operation.
• Multi-tenant space_ databases* (backend-api) and space-agents_ databases* (backend-ai-api) are correctly separated: platform data vs AI data with different access patterns and ownership.

One consolidation opportunity: ai-context (used by backend-api AI module) should merge into ai-agents once the AI module migration (Phase 3) completes. This is the only database with unclear ownership.

MCP Server: Enhance Gateway, Don't Split

Initial assessment suggested splitting into 3-4 separate MCP servers. This was wrong because:

• The gateway pattern already solves the problem. 98 tools are consolidated into 15 gateways with action-based routing, backward compatibility aliases, and internal chaining.
• Single connection point is a feature, not a limitation. Cursor, Claude Code, Cowork, and Claude Desktop all connect through one MCP server. Splitting would require each client to configure multiple servers.
• The real improvement is capability tiers. Add per-client tool filtering so Cursor sees code tools, Cowork sees automation tools, and Claude Code sees everything. The gateway router already supports this via platform detection.

Phase 1: Circuit Breakers and Resilience

Priority: Critical Estimated effort: 2-3 sessions Risk: Low Dependencies: None Branch: session-{N} (follow SAGP v1.2)

Problem Statement

Backend-vitto-api has a CircuitBreakerService (security/resilience/circuit-breaker.service.ts) but it only covers LLM providers via SmartRouterService. Backend-ai-api and backend-api have zero circuit breaker protection on any external calls. If OpenAI goes down, backend-ai-api accumulates hanging HTTP connections until it runs out of memory or hits the 30-second global timeout on every request.

Tasks

1. Extract CircuitBreakerService into shared library. Move from APIs/backend-vitto-api/src/app/security/resilience/circuit-breaker.service.ts to libs/libs-api/backend-core/src/lib/resilience/circuit-breaker.service.ts. Keep existing config: 5 failures to open, 60s reset, 2 half-open test attempts.

2. Extract DeadLetterQueueService similarly. Current location: APIs/backend-vitto-api/src/app/security/resilience/dead-letter-queue.service.ts. Move to shared lib.

3. Add circuit breakers to backend-ai-api LLM calls. Wrap each provider call in @libs-api/llm-integration with the shared circuit breaker. On Claude failure, auto-route to OpenAI. NOTE (S191): Vitto's SmartRouterService.sendWithFailover() already wraps both primary and fallback calls with this.circuitBreaker.execute(). The ProviderFactoryService.wrapWithResilience() creates ResilientLLMProvider instances with the circuit breaker injected. Replicate this pattern at the shared library level so backend-ai-api gets the same protection.

4. Add circuit breakers to backend-api external calls: Stripe (billing module), SendGrid (email module), Twilio (SMS/WhatsApp), Cloudflare Pages API (deployment module), GitLab/Gitea API (git module).

5. Implement provider health endpoint. Expose circuit breaker states at GET /api/health/providers, GET /ai-api/health/providers, GET /vitto-api/health/providers. Return JSON with per-provider state (CLOSED/OPEN/HALF_OPEN), failure count, last failure time.

Key Files to Modify

• libs/libs-api/backend-core/src/lib/ — new resilience/ directory
• libs/libs-api/llm-integration/src/lib/services/ — wrap LLM calls
• APIs/backend-api/src/app/ — wrap external calls in billing, email, deployment modules
• APIs/backend-ai-api/src/app/ — wrap VittoClient, PlatformApiClient calls

Acceptance Criteria

• All LLM provider calls across all 3 APIs go through circuit breakers
• External service calls (Stripe, SendGrid, Twilio, Cloudflare, GitLab) have circuit breakers
• /health/providers endpoint returns JSON with per-provider circuit state
• When Claude API returns 5 consecutive 500s, next call auto-routes to OpenAI within 100ms
• Existing tests pass. New unit tests for CircuitBreakerService in shared lib

Rollback

Circuit breakers default to CLOSED (pass-through). If issues arise, set failureThreshold to 999999 via environment variable to effectively disable without code changes.

Phase 2: Inter-Service Retry and Caching

Priority: High Estimated effort: 2 sessions Risk: Low Dependencies: Phase 1 (shared resilience lib)

Problem Statement

Most inter-service HTTP calls have zero retries. If backend-api is mid-deploy when Vitto tries to resolve a WhatsApp user identity, the call fails silently and the message is lost.

Current State

Tasks

1. Add retry to BackendApiClientService (Vitto). Channel identity resolution MUST retry 3 times with 1s/2s/4s backoff. A failed identity resolution = dropped WhatsApp message.

2. Add retry to BackendAiClientService (Vitto). Orchestration events and execution outcomes: 2 retries. Learning context: keep at 0 retries (graceful degradation acceptable).

3. Add retry to VittoClientService (backend-ai-api). Task orchestration: 2 retries with 2s/4s backoff.

4. Add retry to PlatformApiClientService (backend-ai-api). Platform CRUD queries: 2 retries.

5. Add Redis caching for channel identity resolution in Vitto. Cache key: channel:identity:{channelType}:{channelId}. TTL: 5 minutes. Eliminates backend-api dependency for repeat messages from same WhatsApp user.

6. Add Redis caching for user preferences in Vitto. Cache key: user:prefs:{userId}. TTL: 10 minutes.

Implementation Pattern

Create a shared retryWithBackoff() utility in @libs-api/backend-core that wraps RxJS firstValueFrom calls. Pattern: retry N times with exponential backoff, integrate with circuit breaker state (skip retries if circuit is OPEN), log each retry attempt at WARN level.

Acceptance Criteria

• Channel identity resolution survives a 10-second backend-api restart without dropping messages
• Redis cache hit rate for identity resolution >80% in production (measure after 1 week)
• No change in happy-path latency (retries only trigger on failure)
• Circuit breaker integration: retries stop immediately if circuit is OPEN

Phase 3: AI Module Migration Completion

Priority: High Estimated effort: 2-3 sessions (reduced from 3-4 — S191 completed pricing centralization) Risk: Medium Dependencies: None (can run in parallel with Phases 1-2)

What S191 Already Completed

The coding agent in Session 191 made significant progress on the DRY foundation needed for this phase:

• AI_MODEL_CAPABILITIES in @libs-api/backend-core is now the single source of truth for model pricing, context windows, and feature flags. All 3 APIs and frontend consume it.
• AI_MODELS_CONFIG constants are now used for model ID references (no more hardcoded strings scattered across services).
• @libs-foundational/api-config re-exports the config for frontend Angular libs, maintaining the dependency boundary (frontend libs don't import @libs-api).
• Vitto's CostTrackerService and SmartRouterService now build pricing tables from AI_MODEL_CAPABILITIES via buildPricingTable() and buildModelTiers().
• LlmCostTrackerService (backend-ai-api) derives MODEL_PRICING from AI_MODEL_CAPABILITIES.
• Frontend vitto-chat-cost.service.ts and vitto-chat-model.service.ts both derive their data from the central config.

What Still Needs to Be Done

The pricing config is unified, but the storage targets are still split:

• LlmCostTrackerService (backend-ai-api) writes to llm_usage in ai-agents database
• CostTrackerService (Vitto) writes to spending_records in vitto-mind database
• AIModule (backend-api) writes to ai_usage in gograb-api database

Current Dual Modules

Tasks

1. Audit all frontend calls to /api/ai/ endpoints.* Search Angular apps (gograb-space, gograb-admin, gograb-dev, gograb-ai) for API_BASE_URL + '/ai/' patterns. Document every active call.

2. Create thin proxy controller in backend-api that forwards AI requests to backend-ai-api. Pattern: @Controller('ai') with HttpService forwarding. Maintains backward compatibility while centralizing logic.

3. Unify cost tracking storage. Create a shared LLMUsageTrackingService in @libs-api/backend-core that writes to a single llm_usage collection. Both LlmCostTrackerService (backend-ai-api) and CostTrackerService (Vitto) should use this shared service. The pricing calculation already uses AI_MODEL_CAPABILITIES in both — only the write target differs. Vitto's CostTrackerService has richer features (per-user budgets, context window recommendations) that should be preserved; the shared service handles only the write-path unification.

4. Migrate historical data. Write a migration script that copies ai_usage records from gograb-api and spending_records from vitto-mind into the canonical llm_usage collection in ai-agents. Run in dry-run mode first.

5. Remove AiAgentsModule from backend-api after verifying no direct dependencies. Keep proxy controller.

6. Remove AiLearningModule from backend-api after same verification.

7. Merge ai-context database into ai-agents. Once backend-api AI modules are proxied, update AI_CONNECTION_NAMES.CONTEXT references to point to ai-agents.

Risk Mitigation

• Never remove an endpoint without a proxy replacement. Frontend apps may be cached.
• Run cost migration script in dry-run mode first. Validate record counts match.
• Deploy proxy controller first, verify in production, then remove original modules in a separate deploy.
• Keep ai-context database alive for 2 weeks after migration as rollback safety net.

Phase 4: BullMQ Event-Driven Alerting

Priority: High Estimated effort: 1-2 sessions Risk: Low Dependencies: None (can run in parallel with Phase 1)

Problem Statement

Vitto's proactive monitoring runs on 30-minute heartbeats. Between heartbeats, failed tasks, stalled jobs, and unresponsive tools go undetected. BullMQ already emits completed, failed, and stalled events but these are only used for logging.

Current BullMQ Config

• Queue: vitto-tasks, 3 concurrent workers, 6 rate-limited per 60s
• Job retries: 3 attempts with 5s exponential backoff
• Dead letter queue: 4 retries with 1m/5m/15m/60m schedule
• QueueEvents listener exists but only logs

Tasks

1. Create EventAlertService in backend-vitto-api/src/app/engine/. Subscribe to BullMQ QueueEvents for failed, stalled, error events. On failure of critical tasks (WhatsApp message, security audit, proactive check), trigger immediate notification.

2. Add Slack/WhatsApp notification channel for critical failures. Use existing connectivity module's ChannelGatewayService to send alerts to a configured admin channel.

3. Create stall detection. If a job has been active for >5 minutes without progress events, emit stall warning.

4. Reduce heartbeat interval from 30 minutes to 5 minutes in the proactive module. 30 minutes is too coarse for real-time messaging.

5. Add metrics endpoint: GET /vitto-api/metrics/queue returning current queue depth, active jobs, failed count (last hour), average processing time. Enables external monitoring.

Acceptance Criteria

• A failed WhatsApp task triggers admin notification within 30 seconds (not 30 minutes)
• Stalled jobs detected within 5 minutes
• /vitto-api/metrics/queue returns real-time queue health
• No increase in Redis load (event subscription is lightweight)

Phase 5: Cloudflare Edge Layer

Priority: Medium Estimated effort: 3-4 sessions Risk: Medium Dependencies: Phase 4 (metrics endpoint)

Overview

Deploy two new Cloudflare Workers in front of Vitto. The existing vitto-updater Worker proves the pattern works and the deployment infrastructure (wrangler, deploy scripts) is already in place at deployment/cloudflare/.

Worker 1: vitto-gateway (Webhook Preprocessor)

Purpose: Sit between WhatsApp/Telegram/Slack webhooks and backend-vitto-api. Acknowledge webhooks instantly at the edge (preventing provider retries), deduplicate, and forward to Railway asynchronously.

Endpoints:

• POST /webhook/whatsapp — Verify Twilio HMAC at edge, acknowledge 200, forward to Railway
• POST /webhook/telegram — Verify Telegram bot token, acknowledge 200, forward
• POST /webhook/slack — Verify Slack signature, acknowledge 200, forward
• GET /health — Health check with Railway backend status

Tasks:

1. Create deployment/cloudflare/vitto-gateway-worker/ with wrangler.toml. Domain: gateway.gograb.space.

2. Implement Twilio HMAC verification using Web Crypto API. Store TWILIO_AUTH_TOKEN as Worker secret.

3. Implement message deduplication using Cloudflare KV. Key: sha256(messageId + timestamp). TTL: 5 minutes.

4. Forward verified payloads to vitto-api.gograb.space/vitto-api/connectivity/webhook/* via fetch(). Use waitUntil() for async forwarding after acknowledgment.

5. Update Twilio webhook URL from vitto-api.gograb.space to gateway.gograb.space/webhook/whatsapp. Update Telegram bot webhook similarly.

6. Add Cloudflare DNS CNAME for vitto-gateway subdomain. Create deploy script following vitto-updater pattern.

Rollback: Revert webhook URLs back to direct vitto-api.gograb.space endpoints. Backend webhook handlers remain unchanged.

Worker 2: vitto-pulse (Edge Health Monitor)

Purpose: Run lightweight health checks from the edge every 1 minute. Only wake up Railway backend when something needs attention.

Tasks:

1. Create deployment/cloudflare/vitto-pulse-worker/ with Cron Trigger at 1-minute intervals.

2. Check /vitto-api/health and /vitto-api/metrics/queue (from Phase 4) every minute. Store results in KV with 5-minute TTL.

3. If health check fails 3 consecutive times, send alert to admin Slack/WhatsApp via direct Slack webhook (no backend dependency for alerting about backend failure).

4. If queue depth exceeds threshold (configurable, default: 50 pending jobs), trigger alert.

5. Expose GET /status endpoint returning last-known health of all 4 APIs. Single pane of glass for platform health.

Rollback: Disable Cron Trigger in Cloudflare dashboard. Worker is read-only, makes no changes to backend state.

Phase 6: MCP Gateway Enhancement

Priority: Medium Estimated effort: 2 sessions Risk: Low Dependencies: None (can run in parallel with Phase 5)

Problem Statement

All 15 gateways (98 tools) are exposed to every client equally. Cursor users see browser automation tools they'll never use. Cowork users see git protocol tools that are irrelevant. This wastes context window tokens.

Current State

The MCP server already detects platform via env vars (CLAUDE_CODE_AGENT, CURSOR_AGENT, etc.). The gateway-router.ts routes through a centralized handler. The infrastructure to filter per-platform exists but isn't used for tool filtering.

Tasks

1. Define tool capability tiers in discover-tools.ts. Map each gateway to tiers: core (always), code (Cursor, Claude Code), automation (Cowork, Desktop), intelligence (all), infrastructure (Claude Code only).

2. Add platform-to-tier mapping in gateway-router.ts. Filter GATEWAY_TOOLS in ListToolsRequest based on detected platform.

3. Add capability action to session gateway that returns available tools for current platform.

4. Add dynamic tool loading based on project context. If workspace has package.json, expose code tools. If it has deployment/, expose infrastructure tools.

Tier Mapping

Acceptance Criteria

• Cursor tool list is 30-40% smaller than Claude Code tool list
• Platform detection is automatic (no user configuration)
• Legacy tool aliases still work regardless of tier filtering
• session { action: 'capability' } returns available tools for current platform

Phase 7: Cleanup and Operational Hardening

Priority: Low Estimated effort: 2 sessions Risk: Low Dependencies: Phase 3 (AI migration)

Tasks

1. Remove function-bun service from GoGrab Railway project. Accidentally created during domain migration session.

2. Remove custom domain vitto-api.gograb.space from the WRONG Railway project (standalone Vitto API, project 8d9b4864-25f4-4c4e-b479-f5e49cea4001). Correct domain is on the GoGrab project.

3. Standardize health check endpoints. All 4 APIs return same JSON schema:

   {
     "status": "ok",
     "uptime": 12345,
     "version": "1.0.0",
     "connections": { "mongodb": "connected", "redis": "connected" },
     "circuitBreakers": { "anthropic": "CLOSED", "openai": "CLOSED" }
   }
   

4. Add GITHUB_TOKEN rotation reminder. The vitto-updater Worker PAT expires May 27, 2026. Add proactive check that alerts 30 days before expiry.

5. Consolidate environment variable naming. Standardize to SYSTEM_TOKEN everywhere. Keep BACKEND_API_INTERNAL_TOKEN as deprecated alias.

6. Document the complete inter-service dependency map in docs/english/Infrastructure/. Include timeout values, retry counts, circuit breaker config, and failure modes for every inter-service call.

Execution Timeline

Parallel Execution

Phases 1 and 4 can run in parallel (different codebases, no overlap). Phases 5 and 6 can run in parallel (Cloudflare Workers vs MCP server). This compresses 7 sequential phases to ~4 weeks.

Verification Protocol

After ALL phases complete, run this checklist:

1. Health checks: curl all 4 API /health endpoints. Verify standardized JSON format with circuit breaker states.

2. Circuit breaker test: Set one LLM provider API key to invalid. Verify calls auto-route to fallback within 5 failed attempts.

3. Retry test: Deploy backend-api while sending a WhatsApp message to Vitto. Verify message processes after backend-api comes back up.

4. Cache test: Send two WhatsApp messages from same number within 5 minutes. Verify second resolves identity from Redis cache (no HTTP call to backend-api).

5. Edge test: Send webhook to vitto-gateway Worker. Verify instant 200 acknowledgment at edge and delayed processing at Railway.

6. Monitoring test: Kill a BullMQ job mid-execution. Verify admin receives alert within 5 minutes (not 30).

7. Cost tracking: Make one LLM call from each API. Verify all appear in single llm_usage collection. Verify pricing matches AI_MODEL_CAPABILITIES values (S191 centralized config).

8. MCP test: Connect from Cursor and Claude Code. Verify Cursor sees fewer tools than Claude Code.

Git Protocol Reminder

All implementation work MUST follow SAGP v1.2:

• Branch: session-{N}
• Commits: [S{N}.{sub}] type: description
• Specific file staging (never git add -A)
• Sub-session tracking via commit tags
• Run preflight() at session start
• Run log_decision() and outcome() after implementation