🏗️ Architecture: How should LLM failover be implemented across all consumers ...

Date: 2026-02-25
Time: 15:19:22 UTC
Session: gspace1-138.12
Agent: backend
Confidence: 95%
Status: ✅ Documented

📋 Decision Context

Question:

How should LLM failover be implemented across all consumers (module-generation, engine, future services)?

Task: LLM resilience: automatic failover at infrastructure level

🎯 Decision Made

Chosen:

ResilientLLMProvider pattern: a transparent wrapper that extends BaseLLMProvider, wrapping a primary provider with ordered fallback chain + circuit breaker. ProviderFactoryService.getDefault() and getForPhase() return ResilientLLMProvider. Zero changes to any consumer — all 13+ call sites get failover automatically.

Options Considered:

1. Wire each consumer to SmartRouter individually (rejected — fragile, 9+ files, doesn't protect future code)
2. ResilientLLMProvider transparent wrapper (chosen — zero consumer changes)
3. NestJS interceptor on all LLM calls (rejected — too magic, hard to debug)
4. Move retry+failover into BaseLLMProvider itself (rejected — providers shouldn't know about other providers)

💭 Reasoning

User feedback: wiring each consumer individually to SmartRouter is fragile and doesn't scale. The ResilientLLMProvider makes failover an infrastructure concern — any current or future code calling providerFactory.getDefault().sendMessage() automatically gets failover without knowing about it. Also added OVERLOADED error code (LLMErrorCode) for 529 status with fast-fail (1 retry then yield to failover) reducing time-to-failover from 75s to ~10s. Frontend error messages now categorize errors (overloaded, rate limit, auth, timeout) instead of generic 'Connection lost'.

📁 Files Involved

• apis/backend-vitto-api/src/app/llm/resilient.provider.ts
• apis/backend-vitto-api/src/app/llm/provider-factory.service.ts
• apis/backend-vitto-api/src/app/llm/errors/llm-provider.error.ts
• apis/backend-vitto-api/src/app/llm/base.provider.ts
• libs/libs-ai/gograb-ai-lib/src/lib/services/workflow-state.service.ts

🔗 Related

• Decision ID: dev_1772032762082_g3w8y4z
• Session: 138
• Workspace: gspace1

Auto-generated by MCP Smart Consolidation