290.25 Architecture How Should We Fix Vitto Prod Dev Staging Separation

Documentation for Sessions

Session 290.2 — Vitto Production/Development/Staging Separation

Date: 2026-03-31 Type: Infrastructure + Architecture Status: Complete Branch: session-290-299 Commits: 8 (2 docs + 6 implementation)


What Was the Problem?

Users on the live GoGrab platform were seeing the error message "Connection to the Vitto engine was lost" when trying to use Vitto (the AI assistant) to generate things like a personal expense tracking system. The platform appeared to be working — all servers showed "healthy" — but Vitto couldn't actually complete any tasks.

Why Was This Happening?

The problem had three layers:

  1. The backend server was lying about its health. When Railway (our cloud hosting) asked "are you healthy?", the Vitto server always said "yes" — even when it was missing critical configuration (like database credentials or connections to other services). It would accept user requests but then fail silently when trying to process them.

  2. Vitto was treated as a second-class citizen. The platform has 4 backend services (Main API, AI API, Docs API, and Vitto API). The first three had a clean, centralized configuration system. Vitto was added later and used a separate, fragile path for figuring out its server address. This meant changes to the configuration system could break Vitto without anyone noticing.

  3. Configuration was fragmented across apps. The platform has 6 frontend applications. Each one had a slightly different way of storing server addresses — some used the centralized system, some had URLs typed directly into code, one was missing URLs entirely. This made it easy for things to fall out of sync.


What Did We Do?

Timeline

Step What Why
1. Investigation Checked all 3 production servers — all healthy and running Ruled out server downtime
2. Root Cause Analysis Traced the error from frontend chat UI through SSE streaming to backend task execution Found 7 distinct issues across backend and frontend
3. Design Wrote a spec covering all issues with 3 approach options User approved "Full Prod/Dev Unification" (Approach B)
4. Staging Integration Incorporated Railway staging environment (production vs staging vs local) Aligned with existing staging environment spec
5. App Audit Checked all 6 frontend apps for environment configuration gaps Found 4 different patterns, 3 apps with issues
6. Implementation 8 files changed across 6 commits All TypeScript compilations pass
7. Documentation Logged 3 decisions, 3 conventions, 5 lessons learned into intelligence system Prevents recurrence

Changes Made (in Plain Language)

1. Backend Now Crashes Loud Instead of Failing Silent

Before: Server starts, says "I'm healthy!", then can't actually do anything because it's missing database credentials. After: Server checks for 4 critical settings on startup. If any are missing, it immediately stops with a clear error message telling you exactly what's missing. No more ghost servers.

2. Vitto Promoted to First-Class Citizen

Before: The platform's configuration system knew about 3 of 4 backend services. Vitto used a separate, fragile path. After: Vitto is now configured through the exact same centralized system as the other 3 services. All 4 APIs are treated identically.

3. Staging URLs Updated

Before: Staging server addresses pointed to old, generic Railway URLs (like backend-api-staging.up.railway.app). After: Staging addresses point to our custom branded domains (staging-api.gograb.space, staging-vitto.gograb.space, etc.).

4. All Frontend Apps Aligned

Before: 6 apps, 4 different patterns:

After: All apps that need Vitto now use the centralized configuration system. Missing addresses added. Hardcoded strings replaced.

5. Stale Port Numbers Fixed

Before: Development fallback ports referenced an old workspace (5200, 5100, 5400). After: Updated to current workspace ports (6200, 6100, 6400). Cosmetic but prevents confusion.


Files Changed

File What Changed
environment-presets.ts (api-config lib) Added Vitto to config interface + all 5 environment presets; fixed 3 stale staging URLs
api-tokens.ts (api-config lib) Added Vitto to composite API configuration token
main.ts (backend-vitto-api) Added startup env validation; fixed 3 stale CORS port fallbacks
environment.shared.ts (gograb-space) Added staging URL keys to all 4 URL constants
environment.railway.ts (gograb-space) Added missing Docs API import and URL field
environment.prod.ts (gograb-ai) Replaced 4 hardcoded URL strings with centralized preset references
environment.ts (space-admin) Full rewrite to use centralized presets (was legacy)
environment.prod.ts (space-admin) Full rewrite to use centralized presets (was legacy)

Three-Environment Model (After Changes)

Local Development Staging (Railway) Production (Railway)
Main API localhost:3000 staging-api.gograb.space api.gograb.space
AI API localhost:3001 staging-ai-api.gograb.space ai-api.gograb.space
Docs API localhost:3002 staging-docs-api.gograb.space docs-api.gograb.space
Vitto API localhost:3003 staging-vitto.gograb.space vitto-api.gograb.space
Detection window.location = localhost NODE_ENV=staging NODE_ENV=production
Safety Forgiving (localhost defaults) Fail-fast (crash if missing) Fail-fast (crash if missing)

Verification Results

Check Result
api-config library build Pass
backend-vitto-api build Pass
gograb-space TypeScript Pass (0 errors)
gograb-ai TypeScript Pass (0 errors)
gograb-admin TypeScript Pass (0 errors)
gograb-dev TypeScript Pass (0 errors)
space-admin TypeScript Pass (0 errors)

Note: Some apps have a pre-existing memoryLimit NX schema warning in their project.json — this is unrelated to our changes and existed before this session.


What This Does NOT Include

These items are covered by other specs or sessions:


Decisions Logged to Intelligence System

Decision Type Confidence ID
Fail-fast env validation in main.ts Infrastructure 0.95 (calibrated: 0.905) dev_1774983921306_89fj8hi
Promote vittoApiUrl to first-class in EnvironmentConfig Architecture 0.95 dev_1774983932233_0bkg358
Unify 5 frontend apps on centralized presets Refactor 0.93 dev_1774983943210_n42ch9a

Conventions Added

  1. New API services must be added to all 5 config layers (EnvironmentConfig, presets, ApiConfiguration, API_CONFIG, app environments)
  2. Railway backends must have fail-fast env var validation before NestFactory.create
  3. Staging URLs use custom domains (staging-*.gograb.space), never .up.railway.app

Lessons Learned (Captured in Intelligence System)

  1. New API services added incrementally drift from centralized patterns — add to all config layers simultaneously
  2. Railway health checks passing does NOT mean the service is functional — fail-fast validation is cheap and effective
  3. Staging URLs rot quickly — update URL constants in the same PR as the spec that defines them
  4. Frontend app environment configs fragment silently — periodic audits prevent drift
  5. CORS fallback ports are cosmetic but stale values cause debugging confusion

Future Steps

  1. Deploy to Railway production — push to deploy/vitto-api branch to activate the fail-fast validation
  2. Set up Railway staging environment — follow the staging environment design spec to create the staging Railway environment, set env vars, configure DNS
  3. Create environment.staging.ts files — once staging is live, add staging build configurations to each frontend app (Phase 5 of staging spec)
  4. Fix pre-existing memoryLimit NX schema issue — remove memoryLimit from project.json build targets across affected apps
  5. Audit other backend APIs — apply the same fail-fast env var validation pattern to backend-api, backend-ai-api, and backend-docs-api

Related Documents