MongoDB Integration - Overview & Architecture

Last Updated: October 10, 2025
Version: 1.0.0
Status: Production Ready

Table of Contents

1. Introduction
2. Architecture Overview
3. Core Components
4. Data Flow
5. Design Decisions
6. Performance Characteristics
7. Use Cases

Introduction

The MongoDB Integration extends our storage-system with intelligent cloud synchronization capabilities while preserving the exceptional performance of local operations. This strategic enhancement adds optional cloud sync, real-time features, and collaborative capabilities without compromising the sub-millisecond local cache performance that makes our storage system world-class.

Key Features

• Offline-First Architecture: Local operations remain instant (0.3ms target)
• Flexible Sync Strategies: Choose between offline-first, online-first, or realtime per collection
• Intelligent Queuing: Priority-based sync queue with smart batching
• Real-Time Capabilities: Optional MongoDB Change Streams for live updates
• Conflict Resolution: Automatic or manual conflict handling
• Zero Configuration: Works out of the box with sensible defaults

Philosophy

Our MongoDB integration follows these core principles:

1. Performance First: Local operations never wait for the network
2. Flexibility: Configure sync behavior per collection or even per key
3. Resilience: Works perfectly offline with automatic queue management
4. Intelligence: Smart batching, priority-based sync, and automatic retry
5. Simplicity: Easy to use, hard to misuse

Architecture Overview

The MongoDB integration consists of five major components working together to provide a seamless cloud sync experience:

┌─────────────────────────────────────────────────────────────────┐
│                         Application Layer                        │
│  (Components using autoFlashState, FlashStateService, etc.)    │
└────────────────┬────────────────────────────────────────────────┘
                 │
                 ▼
┌─────────────────────────────────────────────────────────────────┐
│                    Storage System (GgStorage)                    │
│              ┌──────────────────────────────────────┐           │
│              │    FlashStateHydratorService         │           │
│              │  (Pre-loading, instant access)        │           │
│              └──────────────┬───────────────────────┘           │
│                             │                                    │
│              ┌──────────────▼───────────────────────┐           │
│              │      Storage Registry                 │           │
│              │  (OPFS, IndexedDB, localStorage)     │           │
│              └──────────────┬───────────────────────┘           │
└─────────────────────────────┼────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│                    MongoDB Sync Layer                            │
│                                                                  │
│  ┌──────────────────────────────────────────────────────────┐  │
│  │            Intelligent Sync Engine                        │  │
│  │  • Priority-based queue management                        │  │
│  │  • Smart batching by collection                           │  │
│  │  • Exponential backoff retry                              │  │
│  │  • Online/offline detection                               │  │
│  │  • Event notifications                                    │  │
│  └─────────┬────────────────────────────────┬────────────────┘  │
│            │                                 │                   │
│            ▼                                 ▼                   │
│  ┌──────────────────────┐        ┌──────────────────────────┐  │
│  │  MongoDB Sync Adapter│        │ MongoDB Realtime Service │  │
│  │  • Connection pool   │        │  • Change Streams        │  │
│  │  • Sync strategies   │        │  • Auto-reconnect        │  │
│  │  • Conflict resolve  │        │  • Event filtering       │  │
│  │  • Queue persistence │        │  • Bi-directional sync   │  │
│  └──────────┬───────────┘        └──────────┬───────────────┘  │
│             │                                │                   │
│             │         ┌──────────────────────▼────────────────┐ │
│             │         │   Sync Configuration Service          │ │
│             │         │  • Per-collection strategies          │ │
│             │         │  • Configuration presets              │ │
│             │         │  • Runtime strategy switching         │ │
│             │         └───────────────────────────────────────┘ │
└─────────────┼─────────────────────────────────────────────────────┘
              │
              ▼
┌─────────────────────────────────────────────────────────────────┐
│                      MongoDB Atlas                               │
│                    (Cloud Database)                              │
│  • MongoDB 8.0.3                                                │
│  • AWS US-EAST-1                                                │
│  • Change Streams enabled                                       │
└─────────────────────────────────────────────────────────────────┘

Component Relationships

1. Application Layer uses Flash Persistence (autoFlashState) for state management
2. Flash Persistence uses GgStorage for local persistence
3. GgStorage optionally hooks into the MongoDB Sync Layer
4. Intelligent Sync Engine coordinates all sync operations
5. MongoDB Sync Adapter handles the actual database operations
6. Real-Time Service manages Change Streams (optional)
7. Configuration Service provides centralized strategy management

Core Components

1. MongoDB Sync Adapter

The sync adapter is responsible for direct interaction with MongoDB Atlas.

Responsibilities:

• Maintain connection pool with auto-recovery
• Execute CRUD operations (create, update, delete)
• Implement sync strategies (offline-first, online-first, realtime)
• Handle conflict resolution
• Persist offline operation queue

Key Features:

• Connection Pooling: Efficient connection reuse with automatic recovery
• Flexible Schemas: No rigid models, store any JavaScript object
• Sync Strategies: Three strategies to choose from per collection
• Conflict Resolution: Three strategies (last-write-wins, merge, manual)
• Queue Persistence: Operations survive app restarts

Configuration:

const adapter = new MongoDBSyncAdapter({
  connectionString: 'mongodb+srv://...',
  database: 'my-database',
  defaultStrategy: 'offline-first',
  conflictResolution: 'last-write-wins',
  storage: ggStorage
});

2. Intelligent Sync Engine

The sync engine orchestrates all sync operations with intelligence and efficiency.

Responsibilities:

• Manage priority-based sync queue
• Batch operations for network efficiency
• Retry failed operations with exponential backoff
• Track sync status per operation
• Detect online/offline status
• Emit sync events for monitoring

Priority Levels:

• Critical: Syncs in 1-5 seconds (e.g., payment confirmations)
• High: Syncs in 10-30 seconds (e.g., user profile updates)
• Normal: Syncs in 1-5 minutes (e.g., preferences)
• Low: Syncs in 10-30 minutes (e.g., analytics)

Smart Batching:

• Groups operations by collection
• Respects batch size limits (default: 10 operations)
• Uses time windows for efficiency (default: 100ms)
• Processes batches in parallel when possible

Exponential Backoff:

• Automatic retry on failure
• Delay increases exponentially: 1s → 2s → 4s → 8s
• Maximum retry attempts (default: 3)
• Per-operation retry configuration

3. MongoDB Real-Time Service

The real-time service provides optional Change Streams integration for live updates.

Responsibilities:

• Subscribe to MongoDB Change Streams per collection
• Filter change events by operation type or document fields
• Automatically reconnect on connection loss
• Update local storage with remote changes
• Provide bi-directional sync

Change Stream Operations:

• Insert: New document created remotely
• Update: Document modified remotely
• Delete: Document removed remotely
• Replace: Document fully replaced remotely

Event Filtering:

realtimeService.enable('messages', {
  filter: {
    operationType: ['insert', 'update'],
    'fullDocument.status': 'published'
  }
});

Auto-Update Local Storage: When enabled, remote changes automatically update local storage:

• Inserts → Local storage updated
• Updates → Intelligently merged with local data
• Deletes → Removed from local storage
• Conflicts → Resolved based on strategy

4. Sync Configuration Service

The configuration service provides centralized management of sync strategies.

Responsibilities:

• Store per-collection sync configurations
• Provide configuration presets for common scenarios
• Allow runtime strategy switching
• Validate configurations
• Import/export configurations

Configuration Presets:

E-commerce:

{
  cart: 'offline-first',      // Instant cart updates
  inventory: 'online-first',  // Real-time inventory
  orders: 'critical-sync',    // Immediate order sync
  analytics: 'low-priority'   // Background analytics
}

Real-Time Messaging:

{
  messages: 'realtime',       // Live message sync
  drafts: 'offline-first',    // Local-first drafts
  read_receipts: 'realtime',  // Instant read status
  media: 'high-priority'      // Priority media uploads
}

Content Management:

{
  content: 'offline-first',   // Edit offline
  media: 'online-first',      // Cloud media storage
  drafts: 'offline-first',    // Local drafts
  published: 'realtime'       // Live published content
}

5. Angular Providers

The providers make integration into Angular applications seamless.

Full Feature Setup:

import { provideMongoDBSync } from '@storage-system/sync';

bootstrapApplication(AppComponent, {
  providers: [
    provideMongoDBSync({
      connectionString: environment.mongoDBUrl,
      database: 'my-app-db',
      autoConnect: true,
      enableRealtime: ['messages', 'notifications']
    })
  ]
});

Minimal Setup:

import { provideMongoDBSyncMinimal } from '@storage-system/sync';

bootstrapApplication(AppComponent, {
  providers: [
    provideMongoDBSyncMinimal({
      connectionString: environment.mongoDBUrl,
      database: 'my-app-db'
    })
  ]
});

Data Flow

Offline-First Flow (Default)

┌──────────────────────────────────────────────────────────────┐
│  1. User Action (e.g., create task)                         │
└────────────┬─────────────────────────────────────────────────┘
             │
             ▼
┌──────────────────────────────────────────────────────────────┐
│  2. Local Storage Write (OPFS/IndexedDB) - 0.3ms            │
└────────────┬─────────────────────────────────────────────────┘
             │
             ▼
┌──────────────────────────────────────────────────────────────┐
│  3. UI Updated Instantly                                     │
└────────────┬─────────────────────────────────────────────────┘
             │
             ▼
┌──────────────────────────────────────────────────────────────┐
│  4. Operation Queued for Background Sync                     │
└────────────┬─────────────────────────────────────────────────┘
             │
             ▼
┌──────────────────────────────────────────────────────────────┐
│  5. Intelligent Sync Engine Processes Queue                  │
│     • Sorts by priority                                      │
│     • Batches by collection                                  │
│     • Waits for optimal time window                          │
└────────────┬─────────────────────────────────────────────────┘
             │
             ▼
┌──────────────────────────────────────────────────────────────┐
│  6. MongoDB Sync Adapter Executes Batch                      │
│     • Connection pool                                        │
│     • Error handling                                         │
│     • Conflict resolution                                    │
└────────────┬─────────────────────────────────────────────────┘
             │
             ▼
┌──────────────────────────────────────────────────────────────┐
│  7. MongoDB Atlas Updated                                    │
└────────────┬─────────────────────────────────────────────────┘
             │
             ▼
┌──────────────────────────────────────────────────────────────┐
│  8. Success/Failure Event Emitted                            │
│     • UI can show sync status                                │
│     • Retry on failure (exponential backoff)                 │
└──────────────────────────────────────────────────────────────┘

Performance: User sees instant feedback in ~0.3ms, cloud sync happens in background.

Real-Time Flow (Change Streams)

┌──────────────────────────────────────────────────────────────┐
│  1. Remote Change in MongoDB Atlas                           │
│     (Another user, another device, another tab)              │
└────────────┬─────────────────────────────────────────────────┘
             │
             ▼
┌──────────────────────────────────────────────────────────────┐
│  2. MongoDB Change Stream Emits Event                        │
│     • operationType (insert, update, delete, replace)        │
│     • fullDocument (complete document or null)               │
│     • updateDescription (changed fields)                     │
└────────────┬─────────────────────────────────────────────────┘
             │
             ▼
┌──────────────────────────────────────────────────────────────┐
│  3. Real-Time Service Receives Event                         │
│     • Filters by configuration                               │
│     • Checks if collection enabled                           │
└────────────┬─────────────────────────────────────────────────┘
             │
             ▼
┌──────────────────────────────────────────────────────────────┐
│  4. Local Storage Updated (if enabled)                       │
│     • Insert → Create local                                  │
│     • Update → Merge with local                              │
│     • Delete → Remove local                                  │
│     • Replace → Replace local                                │
└────────────┬─────────────────────────────────────────────────┘
             │
             ▼
┌──────────────────────────────────────────────────────────────┐
│  5. Flash Persistence Signals Updated                        │
│     • Reactive UI updates automatically                      │
│     • Computed signals recalculate                           │
└────────────┬─────────────────────────────────────────────────┘
             │
             ▼
┌──────────────────────────────────────────────────────────────┐
│  6. UI Reflects Remote Change                                │
│     • Seamless multi-tab sync                                │
│     • Collaborative features enabled                         │
└──────────────────────────────────────────────────────────────┘

Performance: Real-time updates typically arrive in 100-500ms depending on network.

Design Decisions

1. Why Offline-First?

Decision: Make offline-first the default strategy.

Rationale:

• Users expect instant feedback
• Network is unreliable (mobile, airplane mode, etc.)
• Performance is paramount
• Cloud sync is an enhancement, not a requirement

Benefits:

• ✅ 0.3ms local operations (unchanged)
• ✅ Full app functionality offline
• ✅ Automatic queue management
• ✅ No UX degradation on slow networks

2. Why Flexible Schemas?

Decision: Support any JavaScript object without rigid models.

Rationale:

• MongoDB's strength is schema flexibility
• Developer freedom to evolve data structures
• No migration headaches
• Natural JSON storage

Benefits:

• ✅ Store any object structure
• ✅ No schema migrations needed
• ✅ Natural developer experience
• ✅ Future-proof

3. Why Priority-Based Sync?

Decision: Different sync intervals based on data priority.

Rationale:

• Critical data needs immediate sync
• Low-priority data can batch efficiently
• Network efficiency optimization
• Respects user bandwidth

Benefits:

• ✅ Critical data: 1-5s sync
• ✅ High priority: 10-30s sync
• ✅ Normal: 1-5m sync
• ✅ Low: 10-30m sync

4. Why Multiple Conflict Resolution Strategies?

Decision: Offer last-write-wins, merge, and manual resolution.

Rationale:

• Different data needs different strategies
• Simple data: last-write-wins
• Complex objects: intelligent merging
• Critical data: manual resolution

Benefits:

• ✅ Automatic when safe
• ✅ Intelligent when possible
• ✅ Manual when necessary
• ✅ Configurable per collection

5. Why Optional Real-Time?

Decision: Make Change Streams opt-in per collection.

Rationale:

• Not all data needs real-time
• Bandwidth considerations
• Battery life on mobile
• Performance optimization

Benefits:

• ✅ Real-time when beneficial
• ✅ Offline-first by default
• ✅ Configurable per collection
• ✅ Intelligent defaults

Performance Characteristics

Local Operations

Conclusion: Local operations remain instant regardless of sync configuration.

Sync Operations

Note: Actual sync time depends on network conditions and queue size.

Queue Processing

Conclusion: Queue processing is efficient even with large backlogs.

Real-Time Updates

Conclusion: Real-time updates are near-instantaneous.

Use Cases

1. Offline-First Mobile App

Scenario: Todo/task management app used on mobile with intermittent connectivity.

Strategy:

{
  tasks: 'offline-first',
  tags: 'offline-first',
  analytics: 'low-priority'
}

Benefits:

• Full functionality offline
• Instant task creation/updates
• Automatic sync when online
• No data loss

2. Real-Time Collaboration

Scenario: Collaborative document editing with multiple users.

Strategy:

{
  documents: 'realtime',
  cursors: 'realtime',
  drafts: 'offline-first'
}

Benefits:

• Live multi-user editing
• Instant cursor positions
• Local drafts (fast)
• Conflict resolution

3. E-Commerce Application

Scenario: Online store with cart, inventory, and orders.

Strategy:

{
  cart: 'offline-first',
  inventory: 'online-first',
  orders: 'critical-sync',
  reviews: 'normal'
}

Benefits:

• Instant cart updates
• Real inventory availability
• Immediate order processing
• Efficient review sync

4. Content Management System

Scenario: Blog/CMS with drafts and published content.

Strategy:

{
  drafts: 'offline-first',
  media: 'online-first',
  published: 'realtime',
  analytics: 'low-priority'
}

Benefits:

• Edit drafts offline
• Cloud media storage
• Live published updates
• Background analytics

5. Analytics Dashboard

Scenario: Real-time analytics with high data volume.

Strategy:

{
  live_metrics: 'realtime',
  historical: 'online-first',
  events: 'low-priority'
}

Benefits:

• Live metric updates
• Efficient historical queries
• Background event logging
• Optimized bandwidth

Next Steps

Now that you understand the architecture and core concepts, continue with:

1. Getting Started Guide - Set up MongoDB sync in your app
2. Configuration Reference - All available options
3. Advanced Topics - Custom strategies, performance tuning
4. API Reference - Complete API documentation

Questions?

If you have questions or need help, check:

• Troubleshooting Guide - Common issues and solutions
• FAQ - Frequently asked questions
• Demo Components - Live examples at /storage/mongodb/

Next: Getting Started Guide →