V7 Operational Runbook

Documentation for Sessions

Session Tracking v7.0 - Operational Runbook

Quick Reference

This runbook provides step-by-step procedures for common operational tasks and incident response.

For emergencies, see: Critical Incidents For deployment, see: SESSION-V7-DEPLOYMENT-GUIDE.md For monitoring, see: SESSION-V7-MONITORING-GUIDE.md

Daily Operations
Common Issues
Critical Incidents
Maintenance Tasks
Emergency Contacts

Daily Operations

Morning Health Check (5 minutes)

Run this checklist every morning:

# 1. Check system health
node --import tsx tools/scripts/database/session-sync-v7.ts --stats

# Expected output:
# ✅ MongoDB: Connected
# ✅ Circuit Breaker: Closed
# ✅ Cache: Fresh (< 5 min ago)
# ✅ Total Sessions: X
# ✅ Active Sessions: 1-3

# 2. Check for errors
node --import tsx tools/scripts/database/session-cleanup-v7.ts --dry-run

# Expected output:
# ✅ No issues found - workspace is clean!

# 3. Verify MongoDB indexes
node --import tsx tools/scripts/database/verify-mongodb-indexes.ts

# Expected output:
# ✅ All indexes present
# ✅ Query performance < 100ms

Red Flags (investigate immediately):

❌ Circuit breaker OPEN
❌ Multiple active sessions (> 3 per workspace)
❌ MongoDB connection failed
❌ JSON cache stale (> 10 minutes)
❌ Missing indexes

Weekly Tasks (30 minutes)

Every Monday:

# 1. Run performance tests
npm test -- performance.spec.ts --testTimeout=120000

# 2. Review metrics from past week
# - Query latency trends
# - Error rate trends
# - Circuit breaker events
# - Version conflict rates

# 3. Check for stale sessions (> 30 days inactive)
node --import tsx tools/scripts/database/session-cleanup-v7.ts --dry-run --deactivate-stale --stale-days=30

# 4. Backup MongoDB (if not automated)
mongodump --uri=$MONGO_URI_AGENTS --out=backups/sessions-$(date +%Y%m%d)

Monthly Tasks (2 hours)

First of each month:

# 1. Review and archive old sessions (> 90 days)
# (Manual review required - don't auto-delete)

# 2. Update performance baselines
# Re-run performance tests and update monitoring thresholds

# 3. Review circuit breaker metrics
# - How often did it trigger?
# - Were there any stuck-open incidents?
# - Update thresholds if needed

# 4. Index optimization review
node --import tsx tools/scripts/database/verify-mongodb-indexes.ts --analyze-queries

# Look for:
# - Unused indexes (can be removed)
# - Slow queries (need new indexes)
# - COLLSCAN warnings (fix immediately)

Common Issues

Issue 1: Multiple Active Sessions

Symptoms:

node tools/scripts/database/session-cleanup-v7.ts --dry-run

# Output:
# ⚠️  Found 5 active sessions

Cause: Race condition during session creation, or manual edits to MongoDB

Resolution (5 minutes):

# Step 1: Identify which sessions are active
node --import tsx tools/scripts/database/session-cleanup-v7.ts --workspace=editor2 --dry-run

# Output will show:
#   → Keeping: editor2-28 (most recent)
#   → Deactivating: editor2-27
#   → Deactivating: editor2-26
#   ...

# Step 2: Fix (keeps most recent, deactivates others)
node --import tsx tools/scripts/database/session-cleanup-v7.ts --workspace=editor2

# Step 3: Verify fix
node --import tsx tools/scripts/database/session-cleanup-v7.ts --workspace=editor2 --dry-run

# Expected:
# ✅ No multiple active sessions

Prevention:

Use optimistic locking correctly
Don't manually edit MongoDB
Use atomic operations for status changes

Issue 2: Circuit Breaker Open

Symptoms:

Health check shows circuit breaker OPEN
Logs show "Circuit breaker is open" warnings
System using JSON cache fallback

Cause: MongoDB unavailable or repeatedly failing

Resolution (10 minutes):

# Step 1: Check MongoDB connectivity
mongosh $MONGO_URI_AGENTS --eval "db.runCommand({ ping: 1 })"

# If FAILS:
#   → Check if MongoDB process running: systemctl status mongod
#   → Check network connectivity: ping <mongodb-host>
#   → Check MongoDB logs: journalctl -u mongod -n 100

# If SUCCEEDS:
#   → MongoDB is back up
#   → Circuit breaker will auto-recover in 30 seconds
#   → Monitor health check for state transition to CLOSED

# Step 2: Force MongoDB connection test
node --import tsx -e "
  const { MongoClient } = require('mongodb');
  const client = new MongoClient(process.env.MONGO_URI_AGENTS);
  client.connect()
    .then(() => console.log('✅ Connected'))
    .catch(err => console.error('❌ Failed:', err.message))
    .finally(() => client.close());
"

# Step 3: Wait for auto-recovery
# Circuit breaker resets after 30 seconds of successful operations
# Monitor: watch -n 5 'node tools/scripts/database/session-sync-v7.ts --stats'

# Step 4: If stuck open after 5 minutes
# → Escalate to engineering team
# → System will continue working in degraded mode (JSON cache)

Prevention:

Monitor MongoDB health proactively
Set up MongoDB replication for high availability
Tune circuit breaker thresholds if too sensitive

Issue 3: JSON Cache Stale

Symptoms:

node tools/scripts/database/session-sync-v7.ts --stats

# Output:
# ⚠️  Cache is stale (last synced 15 minutes ago)

Cause: Sync service failed, high update frequency, or manual MongoDB edits

Resolution (2 minutes):

# Step 1: Force regenerate cache
node --import tsx tools/scripts/database/session-sync-v7.ts --force

# Step 2: Verify cache is fresh
node --import tsx tools/scripts/database/session-sync-v7.ts --stats

# Expected:
# ✅ Cache is fresh (< 1 min ago)

# Step 3: Check for errors in sync service
# (check application logs for "sync failed" errors)

# Step 4: If cache repeatedly goes stale
# → Check event-driven sync is enabled
# → Check for high update frequency (this is normal)
# → Verify sync service is running

When stale cache is OK:

During high update frequency (lots of session changes)
Event-driven sync debounces for 30 seconds
Cache will be regenerated automatically

Issue 4: High Version Conflicts

Symptoms:

Logs show:
❌ ConflictException: Version conflict. Expected: 5, current: 6

Cause: Multiple processes updating same session concurrently (optimistic locking working as designed)

Resolution:

If < 5% of operations:

✅ This is EXPECTED behavior
Optimistic locking is preventing data corruption
Retry logic should handle these gracefully
No action needed

If > 10% of operations:

# Step 1: Identify which session(s) have high conflict rate
# (query error logs for "Version conflict" grouped by sessionID)

# Step 2: Investigate concurrent access pattern
# - Why are multiple processes updating same session?
# - Can updates be batched or serialized?

# Step 3: Implement retry with exponential backoff
# (should already be in place via RxJS retry logic)

# Step 4: If specific session is hotspot
# → Consider application-level locking
# → Or redesign to reduce contention

Issue 5: Slow Queries (> 100ms)

Symptoms:

Monitoring shows p95 latency > 100ms
Application feels sluggish

Resolution (15 minutes):

# Step 1: Analyze query performance
node --import tsx tools/scripts/database/verify-mongodb-indexes.ts --analyze-queries

# Look for:
# - COLLSCAN (collection scans) → NO INDEXES USED
# - totalDocsExamined >> nReturned → INEFFICIENT INDEX
# - executionTimeMillis > 100 → SLOW QUERY

# Step 2: Check if indexes exist
node --import tsx tools/scripts/database/verify-mongodb-indexes.ts

# If indexes missing:
node --import tsx tools/scripts/database/verify-mongodb-indexes.ts --create-indexes

# Step 3: Check MongoDB server load
mongosh $MONGO_URI_AGENTS --eval "db.serverStatus()"

# Look for:
# - High CPU usage
# - Memory pressure
# - Disk I/O bottleneck

# Step 4: Run performance tests to establish baseline
npm test -- performance.spec.ts

# Step 5: If query still slow after indexes
# → Check query structure in application code
# → Consider adding new indexes
# → Consider MongoDB scaling (vertical or horizontal)

Issue 6: Corrupted JSON Cache

Symptoms:

❌ SyntaxError: Unexpected token } in JSON at position 1234

Cause: Mid-write interruption (should be impossible with atomic writes, but can happen with external edits)

Resolution (3 minutes):

# Step 1: Delete corrupted cache files
rm sessions/active.json
rm -rf sessions/sessions-history/

# Step 2: Regenerate from MongoDB (source of truth)
node --import tsx tools/scripts/database/session-sync-v7.ts --force

# Step 3: Verify cache integrity
cat sessions/active.json | jq .
cat sessions/sessions-history/editor2/history.json | jq . | head

# Step 4: Set cache files to read-only (prevent manual edits)
chmod 444 sessions/active.json
find sessions/sessions-history -name "*.json" -exec chmod 444 {} \;

# Expected:
# ✅ JSON files valid
# ✅ Files are read-only

Prevention:

NEVER manually edit JSON cache files
Files should be read-only (chmod 444)
Use atomic file writer (already implemented)

Critical Incidents

Incident 1: MongoDB Database Lost/Corrupted

Impact: CRITICAL - All session data lost from MongoDB

Immediate Actions (10 minutes):

# 1. STOP all services writing to MongoDB
# (prevent further data corruption)

# 2. Assess damage
mongosh $MONGO_URI_AGENTS --eval "db.sessions.countDocuments({})"

# If 0 or error:
# → MongoDB data is lost

# 3. Check if JSON cache is available
ls -lh sessions/active.json sessions/sessions-history/

# 4. RESTORE from most recent backup
mongorestore --uri=$MONGO_URI_AGENTS --drop backups/sessions-YYYYMMDD/

# 5. If no backup, RE-MIGRATE from JSON files (v6.0 data)
node --import tsx tools/scripts/database/session-migration-v7.ts --workspace=editor2

# 6. Verify restoration
mongosh $MONGO_URI_AGENTS --eval "db.sessions.countDocuments({})"

# 7. Resume services
# 8. Post-incident: Investigate root cause, improve backup strategy

Recovery Time: 10-30 minutes (depending on backup size)

Incident 2: System Completely Unavailable

Impact: CRITICAL - Neither MongoDB nor JSON cache available

Immediate Actions:

# 1. Check MongoDB
mongosh $MONGO_URI_AGENTS --eval "db.runCommand({ ping: 1 })"

# 2. Check JSON cache
ls sessions/active.json sessions/sessions-history/editor2/history.json

# 3. If MongoDB UP but cache missing
→ Regenerate cache: node tools/scripts/database/session-sync-v7.ts --force

# 4. If MongoDB DOWN but cache exists
→ System will auto-fallback to cache (degraded mode)
→ Fix MongoDB, circuit breaker will recover

# 5. If BOTH down
→ RESTORE from backup (both MongoDB dump and JSON files)
→ Follow "MongoDB Database Lost" procedure above

# 6. If backups missing (worst case)
→ Create new session manually
→ Post-incident: Implement automated backups immediately

Incident 3: Data Inconsistency (MongoDB ≠ JSON Cache)

Impact: HIGH - Data out of sync between source of truth and cache

Diagnosis (5 minutes):

# 1. Check MongoDB data
mongosh $MONGO_URI_AGENTS --eval "
  db.sessions.findOne({ workspace: 'editor2', status: 'active' })
"

# 2. Check JSON cache
cat sessions/active.json | jq '.active.editor2'

# 3. Compare sessionIDs, sessionNumbers, status

Resolution (10 minutes):

Option A: MongoDB is correct (most likely)

# Regenerate JSON cache from MongoDB
node --import tsx tools/scripts/database/session-sync-v7.ts --force

# Verify sync
node --import tsx tools/scripts/database/session-sync-v7.ts --stats

Option B: JSON cache is correct (rare - only if recent MongoDB failure)

# This requires manual intervention
# 1. Compare data sources to determine which is accurate
# 2. Manually update MongoDB to match cache (if cache is source of truth)

mongosh $MONGO_URI_AGENTS

db.sessions.updateOne(
  { sessionID: 'editor2-XX', workspace: 'editor2' },
  { $set: { status: 'active', lastUpdatedAt: new Date() } }
)

# 3. Regenerate cache to confirm sync
node --import tsx tools/scripts/database/session-sync-v7.ts --force

Post-Incident:

Investigate how data got out of sync
Check for manual MongoDB edits
Verify sync service is running correctly

Maintenance Tasks

Task 1: Manual Session Creation

When: Creating a session outside of normal application flow

# Use MongoDB shell to create session with proper schema
mongosh $MONGO_URI_AGENTS

use agents

db.sessions.insertOne({
  sessionID: 'editor2-99',
  workspace: 'editor2',
  sessionNumber: 99,
  subSessionNumber: 1,
  title: 'Manual Session',
  platform: 'cursor',
  status: 'inactive',
  startedAt: new Date(),
  lastUpdatedAt: new Date(),
  decisionsLogged: 0,
  filesCreated: 0,
  filesModified: 0,
  linesAdded: 0,
  version: 1,
  cacheVersion: 1,
  schemaVersion: '7.0'
})

# Regenerate JSON cache
exit
node --import tsx tools/scripts/database/session-sync-v7.ts --force

Task 2: Deactivate All Active Sessions

When: Workspace transition, or fixing corrupt state

# 1. Find all active sessions
node --import tsx tools/scripts/database/session-cleanup-v7.ts --workspace=editor2 --dry-run

# 2. Deactivate (keeps most recent)
node --import tsx tools/scripts/database/session-cleanup-v7.ts --workspace=editor2

# 3. Or manually deactivate ALL (including most recent)
mongosh $MONGO_URI_AGENTS

use agents

db.sessions.updateMany(
  { workspace: 'editor2', status: 'active' },
  {
    $set: { status: 'inactive', lastUpdatedAt: new Date() },
    $inc: { version: 1, cacheVersion: 1 }
  }
)

# 4. Regenerate cache
exit
node --import tsx tools/scripts/database/session-sync-v7.ts --force

Task 3: Archive Old Sessions

When: Cleanup sessions older than 90 days

# 1. Identify old sessions (DRY RUN)
mongosh $MONGO_URI_AGENTS --eval "
  db.sessions.find({
    workspace: 'editor2',
    lastUpdatedAt: { \$lt: new Date(Date.now() - 90 * 24 * 60 * 60 * 1000) }
  }).count()
"

# 2. Export to archive
mongoexport --uri=$MONGO_URI_AGENTS \
  --collection=sessions \
  --query='{"workspace":"editor2","lastUpdatedAt":{"$lt":{"$date":"2024-10-01T00:00:00Z"}}}' \
  --out=archives/sessions-editor2-archive-$(date +%Y%m%d).json

# 3. Optionally delete (CAREFUL!)
# Only delete after confirming archive is valid
# mongosh $MONGO_URI_AGENTS --eval "..."

# 4. Regenerate cache
node --import tsx tools/scripts/database/session-sync-v7.ts --force

Emergency Contacts

On-Call Escalation:

P0 (Critical): System completely down, data loss
- Response time: 15 minutes
- Page on-call engineer immediately
P1 (High): Degraded service, circuit breaker stuck open
- Response time: 1 hour
- Slack alert + email
P2 (Medium): Performance degradation, high error rate
- Response time: 4 hours
- Email notification

Key Personnel:

Session Service Owner: [Name/Slack/Email]
MongoDB Admin: [Name/Slack/Email]
On-Call Engineer: [Rotation/Pagerduty]

Useful Links:

Monitoring Dashboard: [URL]
Log Aggregation: [URL]
Runbook Repository: [URL]
Incident Management: [URL]

Appendix: Quick Command Reference

# Health checks
node --import tsx tools/scripts/database/session-sync-v7.ts --stats
node --import tsx tools/scripts/database/session-cleanup-v7.ts --dry-run
node --import tsx tools/scripts/database/verify-mongodb-indexes.ts

# Fixes
node --import tsx tools/scripts/database/session-cleanup-v7.ts --workspace=editor2
node --import tsx tools/scripts/database/session-sync-v7.ts --force
node --import tsx tools/scripts/database/verify-mongodb-indexes.ts --create-indexes

# MongoDB operations
mongosh $MONGO_URI_AGENTS --eval "db.sessions.countDocuments({})"
mongosh $MONGO_URI_AGENTS --eval "db.runCommand({ ping: 1 })"

# Backups
mongodump --uri=$MONGO_URI_AGENTS --out=backups/sessions-$(date +%Y%m%d)
mongorestore --uri=$MONGO_URI_AGENTS --drop backups/sessions-YYYYMMDD/

# Tests
npm test -- performance.spec.ts --testTimeout=120000
npm test -- libs/libs-api/session-service/tests/

Last Updated: 2025-01-07 Version: 1.0 Status: Production-Ready

Session Tracking v7.0 - Operational Runbook

Quick Reference

Table of Contents

Daily Operations

Morning Health Check (5 minutes)

Weekly Tasks (30 minutes)

Monthly Tasks (2 hours)

Common Issues

Issue 1: Multiple Active Sessions

Issue 2: Circuit Breaker Open

Issue 3: JSON Cache Stale

Issue 4: High Version Conflicts

Issue 5: Slow Queries (> 100ms)

Issue 6: Corrupted JSON Cache

Critical Incidents

Incident 1: MongoDB Database Lost/Corrupted

Incident 2: System Completely Unavailable

Incident 3: Data Inconsistency (MongoDB ≠ JSON Cache)

Maintenance Tasks

Task 1: Manual Session Creation

Task 2: Deactivate All Active Sessions

Task 3: Archive Old Sessions

Emergency Contacts

Appendix: Quick Command Reference