Session Tracking v7.0 - Production Monitoring Guide

Overview

This guide covers monitoring, observability, and alerting for the MongoDB-first session tracking system (v7.0).

Table of Contents

1. Key Metrics
2. Health Monitoring
3. Circuit Breaker Monitoring
4. Performance Monitoring
5. Error Tracking
6. Log Aggregation
7. Alerting Rules
8. Dashboards
9. Troubleshooting Playbook

Key Metrics

Application Metrics

Circuit Breaker Metrics

MongoDB Metrics

Health Monitoring

Health Check Endpoint

The system exposes a /health/sessions endpoint (or equivalent health check method) that returns:

{
  "status": "healthy" | "degraded" | "unhealthy",
  "timestamp": "2025-01-07T12:00:00Z",
  "components": {
    "mongodb": {
      "status": "healthy",
      "latency": 15,
      "message": "Connected"
    },
    "circuitBreaker": {
      "status": "closed",
      "failureRate": 0.001,
      "message": "All operations normal"
    },
    "jsonCache": {
      "status": "healthy",
      "lastSync": "2025-01-07T11:55:00Z",
      "staleness": 300,
      "message": "Cache fresh"
    }
  },
  "metrics": {
    "queryLatencyP95": 45,
    "cacheHitRate": 0.85,
    "activeCircuitBreakers": 0
  }
}

Health States

• Healthy: All systems operational

  • MongoDB connected and responsive (< 100ms)
  • Circuit breaker closed
  • JSON cache fresh (< 5 minutes)
  • Error rate < 0.1%

• Degraded: System operational with reduced capacity

  • Circuit breaker open (fallback to cache)
  • JSON cache stale (> 10 minutes)
  • Error rate 0.1% - 1%
  • Query latency elevated (100-500ms)

• Unhealthy: Critical issues

  • MongoDB unavailable
  • Circuit breaker stuck open (> 5 minutes)
  • JSON cache unavailable or corrupted
  • Error rate > 1%
  • Query latency > 500ms

Health Check Script

#!/bin/bash
# Session health check - can be used with monitoring tools

node --import tsx tools/scripts/database/session-health-check.ts

# Returns:
# - Exit code 0: Healthy
# - Exit code 1: Degraded
# - Exit code 2: Unhealthy

Circuit Breaker Monitoring

Circuit Breaker States

Monitor the circuit breaker state transitions:

// Example: Expose circuit breaker metrics
const circuitBreakerMetrics = sessionHealthService.getCircuitBreakerMetrics();

console.log({
  state: circuitBreakerMetrics.state, // 'closed' | 'open' | 'half-open'
  failureCount: circuitBreakerMetrics.failureCount,
  successCount: circuitBreakerMetrics.successCount,
  lastFailureTime: circuitBreakerMetrics.lastFailureTime,
  openedAt: circuitBreakerMetrics.openedAt,
});

Alerts

Critical: Circuit breaker opens

Alert: Circuit Breaker OPEN
Severity: WARNING
Condition: circuitbreaker.state == 1
Action: System auto-fallback to JSON cache. Investigate MongoDB connection.

Critical: Circuit breaker stuck open

Alert: Circuit Breaker Stuck Open
Severity: CRITICAL
Condition: circuitbreaker.open.duration > 300 seconds
Action: Manual intervention required. Check MongoDB connectivity and health.

Performance Monitoring

Query Performance

Track MongoDB query execution times:

# Analyze slow queries
node --import tsx tools/scripts/database/verify-mongodb-indexes.ts --analyze-queries

# Expected output:
# Query: Find session by ID
#   Execution time: 12ms
#   Index used: sessionID_workspace_unique
#   Documents examined: 1
#   Index efficiency: 100%

Performance Baselines

Establish baselines after deployment:

Performance Testing

Run weekly performance tests:

# Run performance test suite
npm test -- performance.spec.ts

# Expected results:
# ✓ Query by ID < 100ms with 1000 sessions
# ✓ Query active sessions < 100ms
# ✓ Handle 50 concurrent updates
# ✓ Generate cache for 500 sessions < 5s
# ✓ Throughput: 100 ops/sec

Error Tracking

Error Categories

1. MongoDB Errors

• Connection failures
• Transaction rollbacks
• Query timeouts
• Index violations (duplicates)

2. Application Errors

• Version conflicts (optimistic locking)
• Validation failures
• Circuit breaker open (degraded mode)

3. Cache Errors

• JSON parse errors (corrupted cache)
• File write failures
• Sync failures

Error Logging Format

{
  "timestamp": "2025-01-07T12:00:00Z",
  "level": "error",
  "category": "mongodb",
  "operation": "createSession",
  "error": {
    "name": "MongoError",
    "message": "E11000 duplicate key error",
    "code": 11000
  },
  "context": {
    "sessionID": "editor2-1",
    "workspace": "editor2",
    "attemptedOperation": "create"
  },
  "stack": "..."
}

Error Rate Monitoring

// Track error rates by category
const errorMetrics = {
  'mongodb.errors': 5,        // Last hour
  'validation.errors': 2,     // Last hour
  'circuitbreaker.errors': 0, // Last hour
  'cache.errors': 1,          // Last hour
};

const totalOps = 10000; // Last hour
const errorRate = (Object.values(errorMetrics).reduce((a, b) => a + b) / totalOps) * 100;

console.log(`Error rate: ${errorRate.toFixed(2)}%`); // Should be < 0.1%

Log Aggregation

Structured Logging

Use structured logs for easy querying:

logger.info('Session created', {
  sessionID: 'editor2-1',
  workspace: 'editor2',
  platform: 'cursor',
  latency: 45, // ms
  operation: 'createSession',
});

logger.warn('Circuit breaker triggered', {
  operation: 'findById',
  fallback: 'json-cache',
  reason: 'MongoDB timeout',
});

logger.error('Session update failed', {
  sessionID: 'editor2-1',
  workspace: 'editor2',
  error: 'Version conflict',
  expectedVersion: 5,
  actualVersion: 6,
});

Log Queries (Example: Elasticsearch/Splunk)

# Find all circuit breaker events in last hour
level:warn AND operation:circuitbreaker AND @timestamp:[now-1h TO now]

# Find slow queries (> 100ms)
level:info AND operation:findById AND latency:>100

# Find version conflicts
level:error AND error:"Version conflict"

# Find MongoDB connection failures
level:error AND category:mongodb AND error:*connection*

Log Retention

• Application logs: 30 days
• Error logs: 90 days
• Audit logs: 1 year
• Performance metrics: 6 months (aggregated)

Alerting Rules

Critical Alerts (Page on-call)

# MongoDB Unavailable
- alert: MongoDBDown
  expr: mongodb.connection.active == 0
  for: 1m
  severity: critical
  annotations:
    summary: "MongoDB connection lost"
    description: "Circuit breaker will trigger. System degraded."

# Circuit Breaker Stuck Open
- alert: CircuitBreakerStuckOpen
  expr: circuitbreaker.state == 1 AND circuitbreaker.open.duration > 300
  for: 5m
  severity: critical
  annotations:
    summary: "Circuit breaker open for > 5 minutes"
    description: "Manual intervention required. Check MongoDB."

# High Error Rate
- alert: HighErrorRate
  expr: rate(session.errors[5m]) > 0.01
  for: 5m
  severity: critical
  annotations:
    summary: "Error rate > 1%"
    description: "Investigate error logs immediately."

Warning Alerts (Slack/Email)

# Slow Queries
- alert: SlowQueries
  expr: histogram_quantile(0.95, session.query.latency) > 100
  for: 10m
  severity: warning
  annotations:
    summary: "p95 query latency > 100ms"
    description: "Check MongoDB indexes and query performance."

# Low Cache Hit Rate
- alert: LowCacheHitRate
  expr: session.cache.hits / (session.cache.hits + session.cache.misses) < 0.5
  for: 15m
  severity: warning
  annotations:
    summary: "Cache hit rate < 50%"
    description: "Cache may be stale or invalidation strategy needs review."

# Frequent Version Conflicts
- alert: FrequentVersionConflicts
  expr: rate(session.version.conflicts[5m]) > 0.05
  for: 10m
  severity: warning
  annotations:
    summary: "Version conflicts > 5%"
    description: "High contention on session updates."

Informational Alerts (Log only)

# Circuit Breaker Triggered
- alert: CircuitBreakerTriggered
  expr: circuitbreaker.state changes to 1
  severity: info
  annotations:
    summary: "Circuit breaker opened"
    description: "System degraded to cache-only mode."

# Cache Sync Slow
- alert: CacheSyncSlow
  expr: session.sync.duration > 5
  for: 5m
  severity: info
  annotations:
    summary: "JSON cache sync > 5 seconds"
    description: "Large dataset or slow disk I/O."

Dashboards

Primary Dashboard: Session Service Overview

Panels:

1. Health Status (Single Stat)

   • Current health state (Healthy/Degraded/Unhealthy)
   • Color-coded: Green/Yellow/Red

2. Query Latency (Graph)

   • p50, p95, p99 latency over time
   • Target line at 100ms

3. Circuit Breaker State (State Timeline)

   • Closed (green) / Half-open (yellow) / Open (red)
   • Show state transitions

4. Cache Hit Rate (Gauge)

   • Current cache hit percentage
   • Target: > 80%

5. Error Rate (Graph)

   • Errors per minute by category
   • Target: < 0.1%

6. MongoDB Query Performance (Graph)

   • Index usage percentage
   • Slow query count (> 100ms)

7. Operation Throughput (Graph)

   • Operations per second
   • Breakdown by operation type (create, update, read)

8. Version Conflicts (Graph)

   • Optimistic locking conflicts over time
   • Should be rare (< 1%)

Secondary Dashboard: MongoDB Deep Dive

Panels:

1. Connection Pool (Graph)

   • Active connections
   • Available connections

2. Transaction Performance (Histogram)

   • Transaction duration distribution
   • Success vs rollback rate

3. Index Efficiency (Table)

   • Index name, usage count, avg latency
   • Highlight unused indexes

4. Slow Queries (Table)

   • Query, execution time, docs examined
   • Sort by slowest first

Troubleshooting Playbook

Scenario 1: High Query Latency

Symptoms:

• p95 latency > 100ms
• Slow MongoDB queries

Investigation:

# 1. Check MongoDB indexes
node --import tsx tools/scripts/database/verify-mongodb-indexes.ts --analyze-queries

# 2. Check for COLLSCAN (full collection scans)
# Look for queries NOT using indexes

# 3. Check MongoDB server load
mongosh $MONGO_URI_AGENTS --eval "db.serverStatus()"

# 4. Run performance tests
npm test -- performance.spec.ts

Resolution:

• If indexes missing: Run verify-mongodb-indexes.ts --create-indexes
• If indexes not being used: Check query structure in application code
• If MongoDB overloaded: Scale MongoDB (vertical or horizontal)

Scenario 2: Circuit Breaker Open

Symptoms:

• Circuit breaker state = OPEN
• Fallback to JSON cache
• MongoDB connectivity warnings

Investigation:

# 1. Check MongoDB connectivity
mongosh $MONGO_URI_AGENTS --eval "db.runCommand({ ping: 1 })"

# 2. Check circuit breaker status
# (via health endpoint or logs)

# 3. Check recent errors
# (via log aggregation platform)

Resolution:

• If MongoDB down: Restart MongoDB, verify network connectivity
• If transient failures: Wait for auto-recovery (30 seconds)
• If persistent: Manual circuit breaker reset may be needed after fixing MongoDB

Scenario 3: Low Cache Hit Rate

Symptoms:

• Cache hit rate < 50%
• Frequent JSON cache regeneration
• Increased MongoDB query load

Investigation:

# 1. Check cache freshness
ls -lh sessions/active.json sessions/sessions-history/editor2/history.json

# 2. Check sync statistics
node --import tsx tools/scripts/database/session-sync-v7.ts --stats

# 3. Check for high update frequency
# (via MongoDB update logs)

Resolution:

• If cache stale: Run session-sync-v7.ts --force
• If high update frequency: This is expected; cache will regenerate
• If cache corruption: Delete JSON files, regenerate from MongoDB

Scenario 4: Frequent Version Conflicts

Symptoms:

• Version conflicts > 5% of operations
• ConflictException errors in logs
• High contention on specific sessions

Investigation:

# 1. Check which sessions have conflicts
# (query error logs for "Version conflict")

# 2. Identify concurrent access patterns
# (check application logs for concurrent updates)

Resolution:

• Expected behavior: Optimistic locking working correctly
• If > 10% conflicts: Consider retry logic with exponential backoff
• If specific session hotspot: Investigate why multiple processes updating same session

Monitoring Tool Integration

Prometheus + Grafana

// Example: Expose metrics for Prometheus
import { register, Counter, Histogram, Gauge } from 'prom-client';

const sessionQueryLatency = new Histogram({
  name: 'session_query_latency_ms',
  help: 'Session query latency in milliseconds',
  buckets: [10, 25, 50, 100, 250, 500, 1000],
});

const circuitBreakerState = new Gauge({
  name: 'circuitbreaker_state',
  help: 'Circuit breaker state (0=closed, 1=open, 2=half-open)',
});

// In your application:
const start = Date.now();
const result = await sessionService.findSessionById(sessionID, workspace);
sessionQueryLatency.observe(Date.now() - start);

Datadog

// Example: Send metrics to Datadog
import { StatsD } from 'hot-shots';

const dogstatsd = new StatsD({ host: 'localhost', port: 8125 });

dogstatsd.histogram('session.query.latency', latency, ['operation:findById']);
dogstatsd.increment('session.version.conflicts', 1, ['workspace:editor2']);
dogstatsd.gauge('circuitbreaker.state', state === 'open' ? 1 : 0);

Success Criteria

After 1 week in production, verify:

1. ✅ Query latency: p95 < 100ms
2. ✅ Cache hit rate: > 80%
3. ✅ Error rate: < 0.1%
4. ✅ Circuit breaker uptime: > 99% (closed state)
5. ✅ Zero data loss: All sessions accounted for
6. ✅ Version conflict rate: < 1%
7. ✅ JSON sync latency: < 5 seconds
8. ✅ MongoDB index efficiency: > 95%

Additional Resources

• SESSION-V7-DEPLOYMENT-GUIDE.md - Deployment procedures
• SESSION-TRACKING-V7-QUICKSTART.md - Usage guide
• verify-mongodb-indexes.ts - Index verification tool
• performance.spec.ts - Performance tests

Last Updated: 2025-01-07 Version: 1.0 Status: Production-Ready