HANDOFF: Session 100 — Production Issues & Warnings for Coding Agents

Handoff ID: SESSION-100-PROD-ISSUES From: Cowork Agent (Browser/Infrastructure — Paco's Cowork session) To: Cursor / Coding Agents Priority: HIGH Created: 2026-02-09 Session: 100 Status: READY FOR PICKUP Type: Bug Fixes & Code Quality Confidence: 0.95

Context

During Session 100, the Cowork agent performed infrastructure work on Railway deployments:

• Fixed Cloudflare DNS CNAME for vitto-api.gograb.space (was only in cPanel, needed in Cloudflare)
• Fixed ai-api.gograb.space 502 Bad Gateway (PORT env var was 3003 instead of 3001)
• Verified all 4 health endpoints now return 200 OK

While fixing these infrastructure issues, a thorough log analysis of both backend-ai-api and backend-vitto-api Railway deploy logs revealed 15 issues that need code-level fixes. These range from critical runtime failures to minor schema warnings.

Current Health Status (all verified 2026-02-09 ~20:45 UTC)

CRITICAL Issues (Immediate Action Required)

Issue 1: MCP Circuit Breaker Permanently Open (backend-ai-api)

Service: MCPIntegrationService Severity: 🔴 CRITICAL Frequency: Every ~2 minutes, continuously since startup

WARN [MCPIntegrationService] MCP circuit breaker opened after 3 failures
WARN [MCPIntegrationService] MCP logDecision failed: Error

Impact: The MCP integration service cannot connect to its target and the circuit breaker stays permanently open. logDecision calls also fail periodically (~every 15 min). This means AI API decision logging, learning, and MCP tool integrations are completely non-functional in production.

Root Cause Investigation: The MCP server the AI API is trying to reach likely isn't available in the Railway environment. Check what URL/host the MCPIntegrationService is configured to connect to — it may be pointing to localhost or a local MCP server that doesn't exist in Railway.

Files to investigate:

• apis/backend-ai-api/src/**/mcp-integration.service.ts
• apis/backend-ai-api/src/**/mcp*.ts
• Any MCP configuration in environment variables or config files

Fix options:

1. Add proper MCP server URL to Railway environment variables
2. Make MCP integration gracefully optional (disable circuit breaker polling when no MCP server configured)
3. Increase circuit breaker log level to prevent log spam (currently WARNs every 2 min)

Issue 2: Vitto HandoffWsClient Permanently Disconnected ✅ RESOLVED (backend-vitto-api)

Service: HandoffWsClient Severity: 🔴 CRITICAL → ✅ RESOLVED (redeployed in Session 100)

WARN [HandoffWsClient] Backend AI API unreachable at wss://ai-api.gograb.space (Unexpected server response: 502) — will retry silently
ERROR [HandoffWsClient] Max reconnect attempts (50) reached — giving up

Impact: The Vitto API's WebSocket client to ai-api.gograb.space exhausted all 50 reconnect attempts and permanently gave up. This happened because ai-api was returning 502 at the time Vitto booted. Now that ai-api is fixed, the WebSocket will NOT auto-recover — it needs a restart.

Immediate Fix:

• Redeploy Vitto API on Railway to restart the WebSocket client (now that ai-api returns 200)

Code Fix (prevent recurrence):

• Add periodic reconnection even after max attempts (e.g., retry every 5 minutes forever)
• Or remove the max reconnect limit entirely for critical connections

Files to investigate:

• apis/backend-vitto-api/src/**/handoff-ws-client*.ts
• WebSocket reconnection logic

HIGH Priority Issues

Issue 3: Playwright Not Installed in Docker (backend-ai-api)

Service: BrowserAutomationService Severity: 🟠 HIGH

WARN [BrowserAutomationService] [BrowserAutomation] Failed to initialize Playwright:
browserType.launch: Executable doesn't exist at
/root/.cache/ms-playwright/chromium_headless_shell-1208/chrome-headless-shell-linux64/chrome-headless-shell

╔═════════════════════════════════════════════════════════════════════════╗
║ Please run the following command to download new browsers:             ║
║   npx playwright install                                              ║
╚═════════════════════════════════════════════════════════════════════════╝

Impact: Browser automation is completely disabled in production.

Fix: Add Playwright browser installation to the Dockerfile:

# In deployment/Dockerfile.railway.backend-ai (after yarn install)
RUN npx playwright install chromium --with-deps

Files to modify:

• deployment/Dockerfile.railway.backend-ai (or equivalent backend-ai-api Dockerfile)

Issue 4: Phase Prompts Not Found — 0/7 Loaded (backend-ai-api)

Service: PromptLoaderService Severity: 🟠 HIGH

WARN [PromptLoaderService] Failed to load phase prompt for analyze: ENOENT: no such file or directory, open '/app/dist/apps/prompts/phases/analyze.prompt.md'
WARN [PromptLoaderService] Failed to load phase prompt for plan: ...
WARN [PromptLoaderService] Failed to load phase prompt for validate: ...
WARN [PromptLoaderService] Failed to load phase prompt for implement: ...
WARN [PromptLoaderService] Failed to load phase prompt for integrate: ...
WARN [PromptLoaderService] Failed to load phase prompt for verify: ...
WARN [PromptLoaderService] Failed to load phase prompt for test-e2e: ...
WARN [PromptLoaderService] Only 0/7 phase prompts loaded. Using embedded fallbacks.

Impact: All 7 phase prompts are missing. The service falls back to embedded defaults, which may be less effective or outdated.

Root Cause: The NX build doesn't copy .prompt.md assets to the dist/ directory. The prompts exist in the source but aren't included in the build output.

Fix options:

1. Add asset copy rules to apis/backend-ai-api/project.json (NX config) to include prompts/ directory
2. Or add a COPY step in the Dockerfile to copy prompts into the dist folder
3. Or configure the PromptLoaderService to look at source paths, not dist paths

Files to investigate:

• apis/backend-ai-api/project.json — NX build assets config
• apis/backend-ai-api/src/**/prompt-loader.service.ts
• apis/backend-ai-api/src/prompts/phases/ — source prompt files

Issue 5: SYSTEM_TOKEN Not Configured (backend-ai-api)

Service: PlatformOrchestrationService Severity: 🟠 HIGH

WARN [PlatformOrchestrationService] ⚠️ SYSTEM_TOKEN not configured — internal API calls may fail with 401

Impact: Internal API calls between services may fail with 401 Unauthorized.

Fix: Add SYSTEM_TOKEN environment variable to the backend-ai-api Railway service. Generate a secure token and share it across services that need to communicate.

Action:

1. Generate a secure token (e.g., openssl rand -hex 32)
2. Add SYSTEM_TOKEN=<token> to Railway env vars for backend-ai-api
3. Add same token to any services that make internal API calls to ai-api

MEDIUM Priority Issues

Issue 6: Legacy Route Wildcard Syntax (backend-ai-api)

Service: LegacyRouteConverter Severity: 🟡 MEDIUM

WARN [LegacyRouteConverter] Unsupported route path: "/ai-api/.well-known/oauth-protected-resource/*".
In previous versions, the symbols ?, *, and + were used to denote optional or repeating path parameters.
The latest version of "path-to-regexp" now requires the use of named parameters.

Fix: Update the route from /* to /*path:

// Current (deprecated):
'/ai-api/.well-known/oauth-protected-resource/*'

// Required (path-to-regexp v8):
'/ai-api/.well-known/oauth-protected-resource/*path'

Files to investigate:

• apis/backend-ai-api/src/**/routes*.ts or controller decorators using this path

Issue 7: LLM Usage Tracking Disabled (backend-ai-api)

Service: LLMUsageTrackingService Severity: 🟡 MEDIUM

WARN [LLMUsageTrackingService] ⚠️ LLM Usage tracking disabled (AIUsage model not provided)

Impact: LLM API usage (tokens, costs) is not being tracked in production.

Fix: Ensure the AIUsage Mongoose model is registered and provided to the LLMUsageTrackingService module.

Files to investigate:

• apis/backend-ai-api/src/**/llm-usage-tracking.service.ts
• Module imports/providers where AIUsage model should be registered

Issue 8: Mongoose Duplicate Schema Indexes (backend-ai-api + backend-vitto-api)

Severity: 🟡 MEDIUM Affects: Both ai-api and vitto-api

[MONGOOSE] Warning: Duplicate schema index on {"expiresAt":1} found.
[MONGOOSE] Warning: Duplicate schema index on {"userId":1} found.
[MONGOOSE] Warning: Duplicate schema index on {"globalSessionId":1} found.
[MONGOOSE] Warning: Duplicate schema index on {"agentId":1} found.
[MONGOOSE] Warning: Duplicate schema index on {"gitBranch":1} found.
[MONGOOSE] Warning: Duplicate schema index on {"docUrl":1} found.
[MONGOOSE] Warning: Duplicate schema index on {"lastHeartbeat":1} found.
[MONGOOSE] Warning: Duplicate schema index on {"createdAt":1} found.

Impact: Duplicate indexes waste MongoDB resources and slow writes. Each field has both index: true on the schema field AND a separate schema.index() call.

Fix: For each schema, remove either the index: true from the field definition or the schema.index() call. Keep whichever is more appropriate (compound indexes need schema.index(); simple field indexes can use index: true).

Files to investigate:

• libs/**/schemas/*.schema.ts (shared library schemas)
• Search for index: true combined with .index({ on the same fields

Issue 9: Backend-api Unreachable from AI-API (backend-ai-api)

Service: PlatformApiClientService Severity: 🟡 MEDIUM

WARN [PlatformApiClientService] Backend-api not available at http://localhost:3000. Online platform features disabled.

Impact: AI API can't communicate with the main backend-api. Platform features that depend on this connection are disabled.

Root Cause: In Railway, services run in separate containers — localhost:3000 only works in local dev. In production, services need to use Railway's private networking or public URLs.

Fix: Configure BACKEND_API_URL env var in Railway to point to the correct internal/external URL for backend-api (e.g., https://api.gograb.space or Railway's private networking address).

LOW Priority Issues

Issue 10: Mongoose Reserved Schema Pathname (backend-ai-api)

Severity: 🟢 LOW

[MONGOOSE] Warning: `errors` is a reserved schema pathname and may break some functionality.

Fix: Either rename the errors field or add suppressReservedKeysWarning: true to the schema options.

Issue 11: Mongoose Custom _id Index (backend-ai-api)

Severity: 🟢 LOW

[MONGOOSE] Warning: Cannot specify a custom index on `_id` for model name "SessionCounter"

Fix: Remove the custom _id index from the SessionCounter schema — MongoDB already indexes _id automatically.

Issue 12: Docker Image Size 3.8 GB (backend-ai-api)

Severity: 🟢 LOW

Impact: Slow deploys, high storage costs, longer cold starts.

Fix: Optimize the Dockerfile with multi-stage build:

1. Build stage: compile with full node_modules
2. Production stage: only copy dist/ and production dependencies
3. Use .dockerignore to exclude unnecessary files

Issue 13: Skipped railway.json Files (build warning)

Severity: 🟢 LOW

skipping 'railway.json' at 'apis/backend-ai-api/railway.json' as it is not rooted at a valid path
skipping 'railway.json' at 'apis/backend-docs-api/railway.json' as it is not rooted at a valid path

Fix: Either remove these unused railway.json files or move them to the root if they're intended to be used. The project currently uses railway.toml at the root.

Issue 14: Remove Temporary DEPLOY_TRIGGER Variable (backend-ai-api)

Severity: 🟢 LOW

A temporary DEPLOY_TRIGGER="1" environment variable was added to Railway's backend-ai-api service to force a redeployment after the PORT fix. It's harmless but should be cleaned up.

Action: Remove DEPLOY_TRIGGER from Railway backend-ai-api environment variables.

Summary: Action Priority Matrix

Infrastructure Reference

Railway Projects

Port Convention

Cloudflare DNS (gograb.space)

All CNAME records proxied through Cloudflare (Memphis PoP):

What Was Completed in Session 100

1. ✅ Added Cloudflare CNAME: vitto → bye1iet8.up.railway.app (Proxied)
2. ✅ Fixed ai-api PORT: 3003 → 3001 in Railway env vars
3. ✅ Triggered successful redeployment of backend-ai-api
4. ✅ Verified all 4 health endpoints return 200 OK
5. ✅ Created this handoff with 14 issues for coding agents
6. ✅ Fixed Vitto API deployment failure — root cause was wrong Dockerfile path in railway.toml on deploy/vitto-api branch (deployment/Dockerfile.railway.backend-vitto → deployment/backend-vitto-api/Dockerfile.railway). The fix was already committed in 077acde4f but Railway hadn't picked it up. Added DEPLOY_TRIGGER=1 env var to force redeployment.
7. ✅ Vitto API successfully redeployed — deployment e468bde0 is now ACTIVE, health endpoint returns 200 OK, commit 077acde4f
8. ✅ HandoffWsClient reconnected — no WebSocket errors in new deployment (previously had 50 failed reconnection attempts due to ai-api 502)

New Issue Found During Session 100 (continued)

Issue 15: Vitto ProactiveMonitor — Backend Services Unreachable (backend-vitto-api)

Service: ProactiveMonitor Severity: 🟡 MEDIUM

WARN [ProactiveMonitor] [api-health] Alert: Services unreachable: backend-api, backend-ai-api, backend-docs-api

Impact: Vitto's health monitoring reports that backend-api, backend-ai-api, and backend-docs-api are unreachable. This suggests the BACKEND_API_URL, BACKEND_AI_API_URL, and BACKEND_DOCS_API_URL environment variables may be pointing to incorrect internal URLs.

Fix: Verify and update these environment variables in Railway for the Vitto API service:

• BACKEND_API_URL → should be https://api.gograb.space (or Railway private networking)
• BACKEND_AI_API_URL → should be https://ai-api.gograb.space (or Railway private networking)
• BACKEND_DOCS_API_URL → should be https://docs-api.gograb.space (or Railway private networking)

Issue 16: Remove DEPLOY_TRIGGER env var (backend-vitto-api)

Severity: 🟢 LOW

A temporary DEPLOY_TRIGGER="1" environment variable was added to Railway's backend-vitto-api service to force a redeployment after the railway.toml fix. It's harmless but should be cleaned up.

Action: Remove DEPLOY_TRIGGER from Railway backend-vitto-api environment variables.