Flash Persistence + Conversation Genome Architecture

Date: 2026-03-03 | Sessions: 166.3, 172, 173 | Author: Claude Code (Opus 4.6)

This document describes the unified chat persistence system, the Conversation Genome innovation for LLM context compression, and the reusable patterns that can be applied to any feature needing fast local + durable backend sync.

Session 172 update: The sync pattern was fundamentally revised. autoFlashState is now used as a cache layer for MongoDB with a hydration gate — the old two-signal pattern (_backendData signal) is deprecated. See Section 6.

Table of Contents

1. Problem Statement
2. Architecture Overview
3. Database Layout
4. The Five Layers
5. Conversation Genome
6. Reusable Pattern: Flash-to-Backend Sync
7. Step-by-Step: Apply This Pattern to Any Feature
8. Tiered Storage Lifecycle
9. Key Files Reference
10. Anti-Patterns to Avoid

1. Problem Statement

Three compounding gaps existed before Session 166.3:

2. Architecture Overview

┌──────────────────────────────────────────────────────────────┐
│  FRONTEND (Angular)                                          │
│                                                              │
│  Any Feature State (chat, preferences, drafts, etc.)         │
│  autoFlashState → IndexedDB (instant, < 1ms)                 │
│           │                                                  │
│           ▼                                                  │
│  ChatBackendSyncService (@libs-ai/chat-sync)                 │
│  ├─ Fire-and-forget POST per mutation                        │
│  ├─ Persistent retry queue (survives page refresh)           │
│  ├─ Reconciliation on page load                              │
│  └─ NetworkSyncStateService (offline-aware)                  │
└──────────────────────┬───────────────────────────────────────┘
                       │
                       ▼
┌──────────────────────────────────────────────────────────────┐
│  BACKEND (NestJS — backend-ai-api, port 3001)                │
│                                                              │
│  SessionsController                                          │
│  ├─ POST /:id/messages (append with auto-create)             │
│  ├─ POST /:id/reconcile (bidirectional diff)                 │
│  └─ GET /my/active, /my/history, /:id                        │
│                                                              │
│  ConversationGenomeService (context compression)             │
│  ChatEmbeddingService (vector search)                        │
│  ConversationCompressorService (tiered compression, @Cron)   │
│  SessionArchivalService (lifecycle management, @Cron)        │
└──────────────────────┬───────────────────────────────────────┘
                       │
                       ▼
┌──────────────────────────────────────────────────────────────┐
│  MongoDB Atlas: gograbspace.dgqij2t.mongodb.net              │
│  Database: ai-agents                                         │
│  Collection: chat_sessions                                   │
│  ├─ messages[] with optional embedding[] per message         │
│  ├─ genome{} (Conversation Genome, structured representation)│
│  ├─ status: 'active' | 'archived' | 'cold'                  │
│  └─ source: 'panel' | 'vitto' | 'embedded' | 'desktop'      │
└──────────────────────────────────────────────────────────────┘

3. Database Layout

Cluster & Database

Why ai-agents?

ai-agents is the AI operational data hub (29 collections, 256 indexes). Chat sessions are AI operational data. Alternatives were rejected:

• Dedicated database — overkill, adds connection overhead
• vitto-mind — different cluster entirely, would need cross-cluster queries
• Split across ai-agents/ai-archive — same data type shouldn't span databases

Schema Fields (Session 166.3 additions)

Indexes

{ userId: 1, updatedAt: -1 }    — User session queries
{ status: 1 }                    — Archival lifecycle queries
{ context: 1, updatedAt: -1 }   — Panel context queries
{ createdAt: 1 }                 — Cleanup/TTL queries

4. The Five Layers

Layer 1: Instant Local (IndexedDB via autoFlashState)

What: User sees their action immediately. State saved to IndexedDB in < 1ms via autoFlashState.

How it works:

1. Component calls addMessage(msg) on persistence service
2. autoFlashState signal updates → UI refreshes (0ms)
3. IndexedDB write happens in background (debounced 300ms)
4. Cross-tab sync via BroadcastChannel (instant)

Key files:

• libs/libs-foundational/storage-system/src/flash/helpers/auto-flash-state.ts
• libs/libs-foundational/storage-system/src/flash/services/flash-state-hydrator.service.ts

Layer 2: Durable Backend (MongoDB via ChatBackendSyncService)

What: Every mutation also syncs to MongoDB (fire-and-forget). If backend is down, operations queue in IndexedDB and drain on reconnect.

How it works:

1. After IndexedDB save, ChatBackendSyncService.appendMessages() fires
2. POST /sessions/{id}/messages to backend-ai-api
3. On failure → PendingSyncOp pushed to retryQueue (IndexedDB-persisted)
4. NetworkSyncStateService.onBackendAvailable$ → drainRetryQueue()
5. Max 5 retries, exponential backoff (1s → 16s)

Key files:

• libs/libs-ai/chat-sync/src/lib/services/chat-backend-sync.service.ts
• libs/libs-ai/chat-sync/src/lib/models/chat-sync.models.ts

Layer 3: Reconciliation (on page load)

What: On every page load, compare local IndexedDB state vs backend MongoDB. Fix any drift in both directions.

How it works (Session 172):

1. Wait for autoFlashState.hydrated → cached data loaded from IndexedDB
2. Load backend session history → atomically merge INTO autoFlashState
3. POST /sessions/{id}/reconcile with { localMessageIds, localUpdatedAt }
4. Backend returns: { missingOnBackend[], missingLocally[] }
5. Missing locally → merge into autoFlashState via state.update()
6. Missing on backend → fire appendMessages() to re-sync

Why bidirectional: Fire-and-forget can fail silently. Local may have messages backend missed. Backend may have messages from another device.

Layer 4: Conversation Genome (LLM context compression)

What: Instead of sending 50 raw messages to the LLM, send a structured genome (~500 tokens) that encodes the same information 10x more efficiently.

See Section 5 for full details.

Layer 5: Tiered Storage Lifecycle

What: Sessions automatically move through hot → warm → cold tiers based on activity and age.

See Section 8 for full details.

5. Conversation Genome

What It Is

A ConversationGenome is a compact, structured, vectorized representation of an entire conversation:

interface ConversationGenome {
  version: string;
  sessionId: string;
  compiledAt: number;
  messageRange: { first: number; last: number };

  // Structured facts (~200-400 tokens)
  topics: string[];
  decisions: DecisionPoint[];     // { what, why, status }
  facts: FactEntry[];             // { fact, confidence, source }
  userIntent: string;
  openQuestions: string[];
  artifacts: ArtifactRef[];       // { type, path, description }

  // Semantic vector (1536-dim)
  genomeVector: number[];

  // Rolling summary (~200-500 tokens)
  summary: string;

  // Compression metadata
  totalMessages: number;
  compressionRatio: number;
  recentWindowStart: number;
}

How Context Assembly Changes

Before (wasteful):

System prompt → [50 raw messages, ~10K tokens] → user message → LLM
Total: ~12K tokens

After (Genome-powered):

System prompt
  → CONVERSATION GENOME (structured facts, ~400 tokens)
  → SEMANTIC RECALL (vector-retrieved relevant old snippets, ~300 tokens)
  → RECENT WINDOW (last 3-5 messages verbatim, ~500 tokens)
  → user message
Total: ~2.5K tokens (5-8x reduction)

Compilation Pipeline

Integration Points

• APIs/backend-vitto-api/src/app/engine/context-assembly.service.ts — buildChannelSystemPrompt() uses genome when available
• APIs/backend-ai-api/src/app/chat/chat.service.ts — processDirectChat() uses genome for leaner context

6. Reusable Pattern: Flash-to-Backend Sync

Session 172: This section was rewritten. The old two-signal pattern (_backendData signal) caused a flash-on-reload bug and is deprecated. The correct approach is autoFlashState as cache layer with a hydration gate.

This is the core pattern that can be applied to ANY feature needing instant local UX + durable backend storage.

The Pattern (Session 172)

┌──────────────────────────────────────────────────────────────┐
│  autoFlashState (IndexedDB cache for MongoDB)                │
│  ├─ .hydrated signal gates backend loading                   │
│  ├─ Backend data written INTO autoFlashState (atomic merge)  │
│  ├─ IndexedDB auto-updated on every state change             │
│  └─ Next page load: IndexedDB has backend-validated data     │
└──────────┬───────────────────────────────────────────────────┘
           │
           ▼ (write path: local → backend)
┌──────────────────────────────────────────────────────────────┐
│  BackendSync Service                                         │
│  ├─ Fire-and-forget HTTP to backend on each mutation         │
│  ├─ Persistent retry queue (IndexedDB via autoFlashState)    │
│  └─ NetworkSyncState-aware (queues when offline)             │
└──────────────────────────────────────────────────────────────┘

The Hydration Gate Pattern (critical)

// autoFlashState returns AutoFlashSignal<T> with .hydrated: Signal<boolean>
private readonly state = autoFlashState<MyState>('my-key', defaultState, config);

constructor() {
  effect(() => {
    // Gate 1: Wait for IndexedDB cache to hydrate
    if (!this.state.hydrated()) return;
    // Gate 2: Wait for user context
    const userId = this._userId();
    if (!userId) return;
    // Gate 3: Only load once
    if (this.backendLoadDone) return;
    this.backendLoadDone = true;
    this.loadFromBackend();
  });
}

private loadFromBackend(): void {
  this.http.get<BackendData>(url).subscribe(data => {
    // Write backend data INTO autoFlashState — atomic merge, no flash
    this.state.update(s => ({
      ...s,
      items: this.mergeBackendData(s.items, data),
    }));
  });
}

Why this works: state.hydrated() ensures IndexedDB data is loaded before backend HTTP fires. The merge is a single state.update() — atomic, no remove-then-add, no flash. autoFlashState auto-persists the merged data to IndexedDB. Next page load: IndexedDB has backend-validated data.

Why the old two-signal pattern was wrong: It stored backend data in a separate _backendData signal and merged via computed(). This caused stale IndexedDB data to appear for ~500ms on reload, then disappear when backend data arrived and cleanup ran. The hydration gate eliminates this race entirely.

The Retry Queue Pattern

// Persistent queue — survives page refresh
private readonly retryQueue = autoFlashState<PendingOp[]>('my-retry-queue', []);

// On failure → enqueue
onFailure(op) {
  this.retryQueue.update(q => [...q, { ...op, retryCount: 0, queuedAt: Date.now() }]);
}

// On reconnect → drain
networkState.onBackendAvailable$.subscribe(() => this.drainQueue());

// Dedup by unique ID to prevent double-sends

The Reconciliation Pattern

// On page load (AFTER hydration gate):
// 1. autoFlashState hydrates from IndexedDB (cached data shown instantly)
// 2. Backend history loaded → merged INTO autoFlashState (atomic, no flash)
// 3. POST /reconcile with local message IDs
// 4. Backend returns { missingOnBackend[], missingLocally[] }
// 5. Missing locally → merge INTO autoFlashState via state.update()
// 6. Missing on backend → fire appendMessages() to re-sync

7. Step-by-Step: Apply This Pattern to Any Feature

Session 172: Updated to use hydration gate + single-signal cache pattern.

Example: Applying to User Preferences Sync

Say you want user preferences (theme, language, layout) to survive device changes.

Step 1: Local State with autoFlashState

// autoFlashState returns AutoFlashSignal<T> — has .hydrated: Signal<boolean>
private readonly state = autoFlashState<UserPrefsState>('user-prefs', DEFAULT_STATE, {
  database: 'user-data',
  collection: 'preferences',
  priority: 'critical',
  debounceMs: 300,
});

Step 2: Gate Backend Loading on Hydration

private backendLoadDone = false;

constructor() {
  effect(() => {
    // Wait for IndexedDB cache to hydrate before loading backend
    if (!this.state.hydrated()) return;
    const userId = this._userId();
    if (!userId) return;
    if (this.backendLoadDone) return;
    this.backendLoadDone = true;
    this.loadFromBackend();
  });
}

private loadFromBackend(): void {
  this.http.get<UserPrefs>(`${this.apiUrl}/users/me/preferences`).pipe(
    catchError(() => of(null))
  ).subscribe(backend => {
    if (!backend) return;
    // Write backend data INTO autoFlashState — single signal, no flash
    this.state.update(s => ({ ...s, prefs: backend }));
  });
}

Step 3: Create a Sync Service (or reuse ChatBackendSyncService pattern)

@Injectable({ providedIn: 'root' })
export class PrefsBackendSyncService {
  private readonly retryQueue = autoFlashState<PendingOp[]>('prefs-sync-queue', []);
  private readonly networkState = inject(NetworkSyncStateService);

  syncPrefs(prefs: UserPrefs): void {
    if (!this.networkState.canMakeApiCalls()) {
      this.enqueue(prefs);
      return;
    }
    this.http.put(`${this.apiUrl}/users/me/preferences`, prefs)
      .pipe(catchError(() => { this.enqueue(prefs); return of(null); }))
      .subscribe();
  }
}

Step 4: Backend Endpoint

@Put('me/preferences')
async updatePreferences(@Body() prefs, @CurrentUser() user) {
  await this.prefsService.upsert(user.userId, prefs);
  return { success: true };
}

Checklist for Any New Feature Using This Pattern

• autoFlashState for local IndexedDB cache (single signal — no separate _backend* signal)
• Gate backend loading on state.hydrated() (prevents race condition)
• Write backend data INTO autoFlashState via state.update() (atomic merge, no flash)
• Fire-and-forget sync on every mutation
• Retry queue persisted in IndexedDB (not just in-memory)
• NetworkSyncStateService check before HTTP calls
• Backend load on init (JWT auth, no localStorage userId)
• Reconciliation on page load (compare IDs, fix both directions)
• ReplaySubject(1) for late subscribers to backend load events

8. Tiered Storage Lifecycle

HOT (IndexedDB + MongoDB active)
├─ Active conversations (< 7 days since last message)
├─ Recent conversations (< 100 messages)
├─ Genome for ALL conversations (always hot)
│
│  @Cron('0 2 * * *') — Daily 2 AM
│  Criteria: >100 messages AND >7 days inactive
│
WARM (MongoDB archived, IndexedDB cleaned)
├─ Full messages preserved in MongoDB
├─ Removed from IndexedDB (saves browser space)
├─ Genome stays hot (instant search/preview)
│
│  @Cron('0 4 * * 0') — Weekly Sunday 4 AM
│  Criteria: archived for 90+ days
│
COLD (MongoDB cold, messages archived)
├─ All messages marked archived
├─ Genome still searchable
├─ Retrievable on demand (1-3 seconds)

Critical rule: Genomes stay hot forever. This means semantic search, conversation list, and previews work across ALL conversations without fetching message arrays.

Smart retrieval when user opens archived session:

1. Genome header shows instantly (0ms)
2. Full messages stream from MongoDB (1-3s)
3. User sends new message → session promoted back to active

9. Key Files Reference

Frontend (Angular)

Backend (NestJS)

Configuration

10. Anti-Patterns to Avoid

Session 172: First row updated. The old two-signal pattern IS the anti-pattern now.

This architecture was designed in Session 166.3 and revised in Session 172. The Session 172 hydration gate + cache-layer pattern is the canonical approach for any feature needing fast local + durable backend sync.