Bugfix: How should escalation alerts be fixed to prevent WhatsApp sp...

Documentation for Sessions

🐛 Bugfix: How should escalation alerts be fixed to prevent WhatsApp sp...

Date: 2026-03-13
Time: 14:16:43 UTC
Session: gspace1-211.3
Agent: backend
Confidence: 95%
Status: ✅ Documented

📋 Decision Context

Question:

How should escalation alerts be fixed to prevent WhatsApp spam (3+ duplicate messages per outage every 30 minutes)?

Task: S211.1: Fix escalation alert spam on WhatsApp

🎯 Decision Made

Chosen:

Three-part fix: (1) Group unresolved alerts by checkName before dispatching — one consolidated message per root cause with count. (2) Add in-memory escalationCache (Map of checkName→timestamp) to prevent re-escalation within 6 hours. (3) Route through ProactiveDispatcherService.dispatch() instead of calling channelGateway.sendResponse() directly — inherits dedup guard, frequency cap, and quiet hours for free.

Options Considered:

  1. Option A (chosen): Group by checkName + in-memory dedup cache + route through dispatcher
  2. Option B: Add 'escalatedAt' field to alert MongoDB schema, query with $exists filter — rejected: requires schema migration, overkill for ephemeral escalation state
  3. Option C: Just increase the cron interval from 30min to 6h — rejected: masks the bug, doesn't fix grouping or dispatcher bypass

💭 Reasoning

Root cause: escalateUnresolvedCriticalAlerts() iterated over every unresolved alert DOCUMENT and sent a separate WhatsApp message for each. For a single outage like backend-docs-api, the health check creates a new alert doc every cycle, so 3+ docs exist for the same root cause. The method also had no memory of past escalations — every 30-min cron cycle re-sent for the same alerts. Finally, it called channelGateway.sendResponse() directly, bypassing ALL dispatcher safety guards (dedup, frequency cap, quiet hours). The fix addresses all three root causes: grouping eliminates per-document spam, the cache eliminates per-cycle spam, and dispatcher routing inherits all safety guards. Alternative considered: adding an 'escalatedAt' field to the alert schema in MongoDB — rejected because in-memory cache is simpler, sufficient (escalation is ephemeral runtime behavior), and avoids a schema migration.

📁 Files Involved

🔗 Related

Auto-generated by MCP Smart Consolidation