🏗️ Architecture: How to build observability dashboard: unified health aggrega...

Date: 2026-02-26
Time: 16:53:54 UTC
Session: gspace1-144.8
Agent: angular
Confidence: 92%
Status: ✅ Documented

📋 Decision Context

Question:

How to build observability dashboard: unified health aggregation, LLM cost tracking, and Prometheus metrics for the GoGrab platform?

Task: Add observability dashboard (Gap 11)

🎯 Decision Made

Chosen:

In-memory ObservatoryService with 30s health cache TTL, in-memory LlmCostTrackerService with 24h rolling window and daily totals Map, ObservatoryController with JSON dashboard + Prometheus text format endpoints. Frontend ObservatoryDashboardComponent polls every 30s via ObservatoryDataService.

Options Considered:

1. MongoDB-backed persistent tracking with aggregation pipeline — rejected: adds schema overhead, write amplification on every LLM call, not needed for real-time dashboard
2. In-memory 24h rolling window with daily totals map — chosen: zero DB overhead, fast reads, sufficient for dashboard use case, data resets on restart (acceptable for v1)
3. External Prometheus counter with pushgateway — rejected: adds infrastructure dependency, overkill for current scale, but Prometheus text format endpoint is provided for future scraping

💭 Reasoning

ROOT CAUSE: No unified view of system health existed. Each backend had its own /health endpoint but nothing aggregated them. LLM token spend was completely untracked — no visibility into API costs. SOLUTION: ObservatoryService polls all 4 backend health endpoints with native fetch() and 5s timeout. 30s cache TTL prevents hammering. LlmCostTrackerService uses MODEL_PRICING map (per-1M-token USD rates for each Claude model) to estimate costs from response.usage.input_tokens/output_tokens. Records injected into ComputerUseLoopService after each anthropic.beta.messages.create() call. Prometheus text format (/infrastructure/observatory/metrics) enables future Grafana integration without code changes. Frontend tab system extends existing observatory page — 'Agent Monitor' tab preserves existing AgentObservatoryComponent, 'System Dashboard' tab adds new ObservatoryDashboardComponent. WHAT WOULD MAKE THIS WRONG: If process restarts are frequent (data loss), or if multi-instance deployment needs shared metrics (in-memory won't share across pods). EARLY WARNING: If dailyTotals Map grows unbounded (>365 entries = 1 year), add pruning. If Prometheus scrape interval < 30s, health cache may return stale data.

📁 Files Involved

• APIs/backend-ai-api/src/app/infrastructure/services/observatory.service.ts
• APIs/backend-ai-api/src/app/infrastructure/services/llm-cost-tracker.service.ts
• APIs/backend-ai-api/src/app/infrastructure/controllers/observatory.controller.ts
• libs/libs-ai/gograb-ai-lib/src/lib/components/observatory-dashboard/observatory-dashboard.component.ts
• libs/libs-ai/gograb-ai-lib/src/lib/components/observatory-dashboard/observatory-dashboard.component.html
• libs/libs-ai/gograb-ai-lib/src/lib/components/observatory-dashboard/observatory-dashboard.component.scss
• libs/libs-ai/gograb-ai-lib/src/lib/services/observatory-data.service.ts
• apps/gograb-ai/src/app/pages/observatory/observatory-page.component.ts

🔗 Related

• Decision ID: dev_1772124834761_a3mf9lu
• Session: 144
• Workspace: gspace1

Auto-generated by MCP Smart Consolidation