Architecture: How should API Dev Tools handle record persistence and histo...

Documentation for Sessions

🏗️ Architecture: How should API Dev Tools handle record persistence and histo...

Date: 2026-03-17
Time: 18:18:28 UTC
Session: gspace1-229.9
Agent: storage
Confidence: 93%
Status: ✅ Documented

📋 Decision Context

Question:

How should API Dev Tools handle record persistence and historical data? Should we persist activity records to MongoDB, use flash-persistence (IndexedDB), or keep the in-memory ring buffer?

Task: Not specified

🎯 Decision Made

Chosen:

3-tier storage architecture: (1) HOT — in-memory O(1) ring buffer (current, keep as-is) for real-time dashboard, last ~2000 records. (2) WARM — MongoDB Time-Series collection with 7-day TTL for historical queries, fire-and-forget async writes batched 50 records/1s flush. (3) COLD — future S3/R2 daily export. Phase 1 (now): make buffer size configurable from UI. Phase 2 (future): MongoDB warm tier with batched writes + /dev/activity/history endpoint. Phase 3 (future): frontend history mode with date range picker.

Options Considered:

  1. flash-persistence/IndexedDB (frontend-only, wrong for time-series logs)
  2. Replace ring buffer with MongoDB (adds latency to every request)
  3. Redis ring buffer (expensive, network hops, still in-memory)
  4. MongoDB Time-Series as warm tier + keep ring buffer as hot tier (chosen)
  5. Elasticsearch/Loki (overkill for dev tool, operational burden)

💭 Reasoning

flash-persistence (autoFlashState/IndexedDB) is WRONG for this — it's frontend-only, has ~50MB soft limit, no cross-device access, no aggregation queries, and the 5s polling means incomplete data. The in-memory ring buffer is CORRECT for real-time monitoring (zero latency, zero I/O, zero ops). MongoDB Time-Series collections (5.0+) are purpose-built for this: columnar compression (10-20x), automatic bucketing, native TTL, aggregation pipelines. Key design: persistence must be OPTIONAL (@Optional() inject) — ring buffer always works even without MongoDB. Write to MongoDB ASYNC (background queue) — never add latency to the request hot path. At 100 req/s across 4 APIs: 360K writes/hour, ~500MB/day after compression, auto-cleaned by 7-day TTL. Redis rejected: just a more expensive ring buffer with network hops. Don't replace the ring buffer with MongoDB — they serve different time horizons.

📁 Files Involved

🔗 Related

Auto-generated by MCP Smart Consolidation