S283: Staging Environment — Session Summary

Session: 283 Date: March 28, 2026 Branch: session-280-285 Status: Complete (first deployment pending)

What This Session Achieved

We added a staging environment to the GoGrab platform so that developers can test changes without affecting production data. Before this session, every database operation — whether from local development, testing, or production servers — went to the same MongoDB databases. Now, production and staging data are fully isolated.

Timeline

Phase 1: Problem Identification & Design

What happened: We analyzed the current infrastructure — 4 backend APIs (backend-api, backend-ai-api, backend-docs-api, backend-vitto-api) deployed on Railway, using 3 MongoDB Atlas Flex clusters in production. We identified that there was no way to test with isolated data.

What we decided:

• Use a single MongoDB Atlas Flex cluster for all staging databases (instead of 3 separate clusters like production)
• All database names get a -staging suffix when running in staging mode
• A collision guard prevents two services from accidentally using the same database name

Why this matters: Without staging, any test data created during development pollutes the production database. A bad migration script or test could corrupt real user data.

Phase 2: Database Safety (Code Changes)

What happened: We made changes to ensure database names can be dynamically suffixed based on the environment, and that no two services can accidentally use the same database name.

Change 1: Database Name Isolation (DB_ENV_SUFFIX)

• File: database-names.config.ts
• What it does: Reads a DB_ENV environment variable. If set to staging, all database names get -staging appended. If not set (production), nothing changes.
• Example: gograb-api becomes gograb-api-staging, ai-agents becomes ai-agents-staging
• Why it's safe: Production servers don't set DB_ENV, so their database names are unchanged. Even if someone accidentally points a staging server at a production MongoDB cluster, the -staging suffix prevents any data collision.

Change 2: Database Name Collision Guard

• File: database-names.config.ts
• What it does: When any backend starts, it checks that all registered database names are unique. If two services register the same name (e.g., both ai-memory and a new vitto database called ai-memory), the server crashes immediately with a clear error message.
• Why it matters: On a shared cluster (like staging where all databases live on one cluster), a name collision means two services silently read/write each other's data. The guard catches this before any database connection opens.

Change 3: Vitto MongoDB Connection Fix

• File: backend-vitto-api/app.module.ts
• What it does: Fixed a fragile pattern where the Vitto API derived its second database connection by doing a text replacement on the connection string (.replace('/vitto-mind', '/vitto-users')). This would silently fail if the connection string didn't contain that exact text.
• New approach: Uses the shared database factory with explicit database names, matching how all other backends work.
• Bonus: The Vitto API now gets production-grade settings (write concern, compression, retry logic) that it was silently missing before.

Change 4: Docs Database Name Centralization

• Files: backend-docs-api/constants.ts, backend-docs-api/app.module.ts
• What it does: The docs API had its database name (docs-database) hardcoded locally instead of using the centralized config. We moved it to the shared database-names.config.ts so the collision guard covers it and the DB_ENV_SUFFIX applies to it.

Change 5: Dynamic Space Database Names

• Files: space-database.utils.ts, space-database-resolver.service.ts
• What it does: Per-tenant databases (space_chrome-space-1_abc123, space-agents_chrome-space-1_abc123) also get the staging suffix when DB_ENV is set.

Phase 3: Testing & Verification

• 13 tests written covering suffix behavior, collision guard, and dynamic space names
• All 4 backends build successfully with zero errors
• Zero production impact verified — without DB_ENV set, all names are identical to before

Phase 4: Deploy CLI Update

What happened: The deployment CLI (yarn deploy) was hardcoded for production only. We added an --env flag so developers can deploy to staging.

• Usage: yarn deploy --env staging --apis api,ai-api,docs-api,vitto-api
• Default: yarn deploy --apis api still deploys to production (backward compatible)
• Deploy branches: Created deploy/staging-api, deploy/staging-ai-api, deploy/staging-docs-api, deploy/staging-vitto-api and pushed to GitHub

Phase 5: MongoDB Atlas Staging Cluster

What happened: Created a new MongoDB Atlas Flex cluster called GoGrab-Staging.

• Tier: Flex ($8-30/month, usage-based)
• Provider/Region: GCP / Iowa (us-central1) — same as production clusters
• MongoDB Version: 8.0.20 — same as production
• Why Flex (not M10): After analyzing actual production metrics, we found all 3 production clusters are well within Flex limits (peak: 113 connections out of 500, 10.7 ops/sec out of 500, 321 MB storage out of 5 GB). M10 would cost $57/month for benefits we can't measure at current scale.

Phase 6: Local Development Staging Support

What happened: Made it easy for developers on any platform (macOS, Linux, Windows) to switch between production and staging databases locally.

• New file: .env.staging.example — template showing what credentials to fill in
• Setup: Developer copies to .env.staging and fills in the staging MongoDB credentials
• Usage: yarn serve:backend-api --staging or yarn serve:backend-api:staging
• How it works: The dev-serve.js script (which already handles all serve commands) loads .env.staging and injects the values before starting the backend. No new dependencies needed.

Phase 7: Railway Staging Environment (In Progress)

What's happening: A browser agent is setting up the Railway staging environment — creating the staging environment in the Railway dashboard, configuring custom domains (staging-api.gograb.space, etc.), setting environment variables, and configuring DNS.

Files Changed

Git Commits

Railway Staging Environment

Status: Configured and ready Environment ID: d688c378-a4fe-455d-af45-671dc0bcf78a

Key environment variables set in Railway staging:

• DB_ENV=staging — suffixes all database names
• MONGODB_CONNECTION_STRING → GoGrab-Staging cluster
• VITTO_MONGODB_URI → GoGrab-Staging cluster
• All inter-service URLs → staging-*.gograb.space
• All API keys/tokens inherited from production
• DNS CNAME records configured in Cloudflare (proxied)

How to Use (For Developers)

First-Time Setup

# Copy the staging env template and fill in the real credentials
cp .env.staging.example .env.staging
# Edit .env.staging with the GoGrab-Staging MongoDB credentials

Day-to-Day Usage

# Normal development — uses production databases (unchanged)
yarn serve:backend-api

# Staging — uses staging databases (all data isolated)
yarn serve:backend-api --staging
# OR
yarn serve:backend-api:staging

# All backends on staging at once
yarn serve:all-backends:staging

Deploying to Staging (after Railway setup is complete)

# Deploy to staging
yarn deploy --env staging --apis api,ai-api,docs-api,vitto-api

# Deploy to production (default, unchanged)
yarn deploy --apis api,ai-api,docs-api,vitto-api

Costs

What's Remaining

Infrastructure Verification Checklist

Before the first deployment, verify in Railway staging dashboard:

• MONGODB_CONNECTION_STRING_DOCS is set to mongodb+srv://[email protected]/ (NOT the production gograb-docs cluster)
• Redis plugin exists in staging environment and REDIS_URL is auto-populated
• DB_ENV=staging is set on all 4 services
• All 4 services have correct deploy branches (deploy/staging-*)

Key Decisions & Rationale

Why one staging cluster instead of three?

Production uses 3 separate clusters for performance isolation under load. Staging doesn't need that — it has minimal traffic. One Flex cluster keeps costs low ($8-30/mo vs $171/mo for three M10s) and simplifies management.

Why Flex instead of M10?

After analyzing real production metrics: peak 10.7 ops/s (2% of Flex's 500 cap), 113 connections (23% of Flex's 500 limit), 321 MB storage (6% of 5 GB). M10 would add $27-49/mo for zero measurable benefit at current scale.

Why suffix database names instead of just using separate clusters?

Defense in depth. Even if someone accidentally points a staging connection string at a production cluster, the -staging suffix means the databases are different. This prevents the worst-case scenario: silent data corruption in production.

Why modify dev-serve.js instead of adding cross-env or modifying app.module.ts?

dev-serve.js already owns the serve lifecycle, handles cross-platform spawning, and controls process.env. Adding --staging there required zero new dependencies and zero changes to production backend code. Works on macOS, Linux, and Windows without any platform-specific syntax.

Architecture Overview

PRODUCTION                              STAGING
──────────                              ───────

Railway (production env)                Railway (staging env)
├── backend-api                         ├── backend-api
│   → api.gograb.space                  │   → staging-api.gograb.space
├── backend-ai-api                      ├── backend-ai-api
│   → ai-api.gograb.space              │   → staging-ai-api.gograb.space
├── backend-docs-api                    ├── backend-docs-api
│   → docs-api.gograb.space            │   → staging-docs-api.gograb.space
├── backend-vitto-api                   ├── backend-vitto-api
│   → vitto.gograb.space               │   → staging-vitto.gograb.space
└── Redis                              └── Redis

DB_ENV=(not set)                        DB_ENV=staging
    ↓                                       ↓
Database names:                         Database names:
  gograb-api                              gograb-api-staging
  ai-agents                               ai-agents-staging
  vitto-mind                               vitto-mind-staging
  docs-database                            docs-database-staging

MongoDB Atlas:                          MongoDB Atlas:
  GoGrabSpace cluster                     GoGrab-Staging cluster
  GoGrab-Docs cluster                     (single Flex cluster
  Vitto cluster                            for all databases)


LOCAL DEVELOPMENT
─────────────────

yarn serve:backend-api                  yarn serve:backend-api --staging
    ↓                                       ↓
.env.local (production)                 .env.staging (overrides)
DB_ENV=(not set)                        DB_ENV=staging
    ↓                                       ↓
Uses production databases               Uses staging databases

MCP Decision Trail

All architectural decisions for this session are logged in the MCP intelligence system for future reference:

Conventions added: 3 (database name centralization, MongoDB factory usage, staging switching) Lessons reflected: 6 (including "verify real metrics before recommending upgrades") Outcomes reported: 5/6 successful (deploy CLI rated 4/5 — first deployment still pending)

Related Documents