⚡ Real-Time Sync & Storage System

Status: ✅ Production Ready

The GoGrab Space storage system is a state-of-the-art, offline-first, real-time synchronization engine that seamlessly bridges the gap between local browser storage (IndexedDB/OPFS), collaborative editing (Yjs), and persistent cloud storage (MongoDB).

🏗️ Architecture Overview

The system is built on three pillars:

1. Flash Persistence (Local): Instant saves to GgStorage (IndexedDB/OPFS).
2. Real-Time Collaboration (Peer-to-Peer/Server): Yjs CRDTs over WebSocket for sub-second updates across clients.
3. Cloud Sync (Background): HttpMongoDBSyncAdapter for reliable, batched synchronization with MongoDB.

🔄 Data Flow Diagram

graph TD
    subgraph "Browser Client A"
        UI[UI Components] <-->|Signals/Effects| FP[Flash Persistence]
        FP <-->|Read/Write| Gg[GgStorage (IndexedDB)]
        Gg <-->|Events| RTS[RealtimeSyncService]
        RTS <-->|Updates| Yjs[Yjs Doc]
        Gg <-->|Batched Sync| HTTP[HttpMongoDBSyncAdapter]
    end

    subgraph "Browser Client B"
        UI2[UI Components] <--> FP2[Flash Persistence]
        FP2 <--> Gg2[GgStorage]
        Gg2 <-->|Events| RTS2[RealtimeSyncService]
        RTS2 <-->|Updates| Yjs2[Yjs Doc]
    end

    subgraph "Backend Infrastructure"
        WS[WebSocket Gateway] <-->|Sync Protocol| Yjs
        WS <-->|Sync Protocol| Yjs2
        HTTP <-->|REST API| API[Backend API]
        API <-->|Persist| DB[(MongoDB)]
    end

    Yjs <-->|WebSocket| WS
    Yjs2 <-->|WebSocket| WS

🧩 Key Components

1. RealtimeSyncService (@libs-foundational/core-angular)

The brain of the collaboration system.

• Role: Manages WebSocket connections, Yjs documents, and presence.
• Key Feature: Bidirectional Storage Sync.
  • syncStorageToYjs(): Pushes local IndexedDB data to Yjs room on connect.
  • syncRemoteChangesToStorage(): Listens for Yjs updates and writes them to local IndexedDB.
  • broadcastStorageChange(): Manually triggers a broadcast when local data changes.

2. GgStorage (@storage-system)

The unified storage layer.

• Role: Abstraction over IndexedDB, OPFS, and localStorage.
• Key Feature: Event System.
  • Emits events on put, delete, clear.
  • Handles Cross-Tab Sync via BroadcastChannel.
  • Handles Same-Tab Sync via local event listeners.

3. HttpMongoDBSyncAdapter (@storage-system)

The bridge to the cloud.

• Role: Syncs local collections to MongoDB via HTTP.
• Key Feature: Browser-Compatible.
  • Does NOT use MongoDB driver in browser.
  • Uses generic REST endpoints (/api/storage/sync/:collection).
  • Supports batching, retries, and conflict resolution.

4. Yjs Server (backend-api)

The real-time relay.

• Role: WebSocket server implementing Yjs sync protocol.
• Key Feature: Custom Implementation.
  • Implements Sync, Awareness, and Update protocols.
  • Handles room isolation (universe:ID).
  • Broadcasts updates to all connected clients in the room.

🗄️ Database Strategy

The system is designed to handle Generic, Restricted, and User-Specific databases/collections seamlessly.

1. Configuration

Collections are configured in app.config.ts via provideHttpMongoDBSync.

provideHttpMongoDBSync({
  apiUrl: environment.backendApiUrl,
  collections: [
    // 🌌 Core Platform Data (Restricted by Space/Universe ID)
    { name: 'universes', syncInterval: 5000 },
    { name: 'spaces', syncInterval: 10000 },

    // 🌍 Generic/Public Data (Accessible by all users)
    { name: 'public_assets', syncInterval: 30000 },
    { name: 'marketplace_items', syncInterval: 30000 },

    // 👤 User-Specific Data (Restricted by User ID)
    { name: 'user_settings', syncInterval: 5000 },
    { name: 'user_preferences', syncInterval: 10000 }
  ],
  autoInitialize: true,
  autoSync: true
})

2. Access Control (Backend)

The StorageController in the backend handles the generic sync endpoints:

• POST /api/storage/sync/:collection
• PUT /api/storage/sync/:collection/:documentId

Security Model:

• Authentication: All requests require a valid JWT.
• Ownership: Documents are automatically tagged with userId.
• Validation: (Future) Middleware can intercept specific collection names to enforce stricter rules (e.g., checking Space membership).

🚀 How It Works (The "Magic")

Scenario: User Adds a Section to a Universe

1. User Action: User clicks "Add Section" in GridCanvasComponent.
2. Local Save: this.storage.put(layoutKey, newLayout) is called.
3. Flash Update:
   • GgStorage emits local event → UI updates instantly.
   • GgStorage posts to BroadcastChannel → Other tabs update instantly.
4. Real-Time Broadcast:
   • RealtimeSyncService.broadcastStorageChange() is called.
   • Updates Yjs sharedMap.
   • Yjs sends update via WebSocket to Backend.
5. Remote Sync:
   • Backend relays update to other clients.
   • Other Client receives Yjs update.
   • syncRemoteChangesToStorage() writes to their local IndexedDB.
   • Their GgStorage emits event → Their UI updates instantly.
6. Cloud Persistence:
   • HttpMongoDBSyncAdapter (in background) picks up the change.
   • Sends PUT /api/storage/sync/universes/:id to Backend.
   • Backend saves to MongoDB.

✅ Verification

To verify the system is working:

1. Open two different browsers (e.g., Chrome and Safari).
2. Navigate to the same Universe URL.
3. Open the Real-Time Debug Panel (bottom right).
4. Ensure both show:
   • WebSocket: ✅ Connected
   • Yjs Sync: ✅ Synced
5. Make a change in one browser.
6. Observe the change appear instantly in the other browser.

🔮 Future Enhancements

• Presence Awareness: Show "User X is editing this section".
• Field-Level Locking: Prevent two users from editing the exact same text field simultaneously.
• Version History: Use MongoDB to store historical versions of layouts.