📦 Modular Storage

Module-specific storage isolation for platform builders with custom data strategies per module.

🎯 What is Modular Storage?

Modular Storage provides isolated storage namespaces for each module in your application, allowing:

• Isolated Data - Each module's data is completely separate
• Custom Strategies - Per-module sync, caching, and conflict resolution
• Collection-Level Control - Fine-grained strategy per data collection
• Zero Interference - Modules can't accidentally access each other's data

⚡ Quick Start

Basic Usage

import { Component, inject } from '@angular/core';
import { ModularStorageService, ModuleStorageConfig } from '@gograb-space/storage-system';

@Component({
  selector: 'app-my-module',
  standalone: true
})
export class MyModuleComponent {
  private readonly modularStorage = inject(ModularStorageService);
  
  // Define your module's storage configuration
  private readonly config: ModuleStorageConfig = {
    moduleId: 'my-module',
    storage: {
      primary: 'offline-first',
      collections: {
        'users': {
          sync: 'immediate',
          cache: 'aggressive',
          conflict: 'local-wins'
        },
        'settings': {
          sync: 'lazy',
          cache: 'minimal',
          ttl: 3600000 // 1 hour
        }
      }
    }
  };
  
  async ngOnInit() {
    // Get module-specific storage instance
    const storage = this.modularStorage.getModuleStorage(this.config);
    
    // Save data to module's collection
    await storage.set('users', 'user-123', {
      name: 'John Doe',
      email: '[email protected]'
    });
    
    // Read data from module's collection
    const user = await storage.get('users', 'user-123');
    console.log('User:', user);
  }
}

🏗️ Architecture

Namespace Isolation

Global Storage (GgStorage)
    ↓
ModularStorageService
    ├─ module-space-map:users:user-123
    ├─ module-space-map:galaxies:galaxy-1
    ├─ module-flow-composer:flows:flow-456
    ├─ module-flow-composer:nodes:node-789
    └─ module-launch-station:analytics:data-1

Each module gets a unique namespace: module-{moduleId}:{collection}:{key}

📖 Core Concepts

1. Module Storage Config

Define how your module's data should be handled:

interface ModuleStorageConfig {
  moduleId: string;                        // Unique module identifier
  storage: {
    primary: 'offline-first'               // Module-level strategy
           | 'online-first' 
           | 'hybrid';
    collections: Record<string,            // Per-collection strategies
                        CollectionStrategy>;
  };
  realtime?: {                             // Optional real-time sync
    enabled: boolean;
    channels?: string[];
  };
}

2. Collection Strategy

Fine-grained control per data collection:

interface CollectionStrategy {
  sync: 'immediate'      // Sync immediately
      | 'eager'          // Sync ASAP (within 100ms)
      | 'lazy'           // Sync when convenient (5s delay)
      | 'on-demand'      // Only sync on explicit request
      | 'local-only'     // Never sync to cloud
      | 'real-time';     // Real-time bidirectional sync
      
  cache?: 'aggressive'   // Cache everything, long TTL
        | 'minimal'      // Cache only frequently accessed
        | 'windowed'     // Cache recent items only
        | 'spatial';     // Cache based on location/context
        
  ttl?: number;          // Cache time-to-live (milliseconds)
  
  index?: 'standard'     // Standard B-tree index
        | 'geospatial'   // Geo-spatial queries
        | 'text';        // Full-text search
        
  storage?: 'chunked'    // Store in chunks (large data)
          | 'compressed' // Compress before storage
          | 'standard';  // Standard storage
          
  conflict?: 'merge'     // Merge changes intelligently
           | 'local-wins'// Local changes win
           | 'remote-wins'; // Remote changes win
           
  queue?: 'guaranteed'   // Guaranteed delivery (retry forever)
        | 'best-effort'; // Best-effort (retry limited)
}

🚀 Usage Examples

Example 1: Simple Module Storage

// Space Map Module
const spaceMapConfig: ModuleStorageConfig = {
  moduleId: 'space-map',
  storage: {
    primary: 'offline-first',
    collections: {
      'universes': {
        sync: 'immediate',
        cache: 'aggressive'
      },
      'galaxies': {
        sync: 'eager',
        cache: 'aggressive'
      },
      'pages': {
        sync: 'lazy',
        cache: 'windowed',
        ttl: 600000 // 10 minutes
      }
    }
  }
};

// Get module storage
const storage = modularStorage.getModuleStorage(spaceMapConfig);

// Use it
await storage.set('universes', 'uni-1', { name: 'My Universe' });
const universe = await storage.get('universes', 'uni-1');
await storage.delete('universes', 'uni-1');

Example 2: Multi-Module Application

// Flow Composer Module
const flowConfig: ModuleStorageConfig = {
  moduleId: 'flow-composer',
  storage: {
    primary: 'hybrid',
    collections: {
      'flows': {
        sync: 'immediate',
        cache: 'aggressive',
        conflict: 'merge'
      },
      'nodes': {
        sync: 'eager',
        cache: 'aggressive',
        storage: 'compressed'
      },
      'connections': {
        sync: 'lazy',
        cache: 'minimal'
      }
    }
  },
  realtime: {
    enabled: true,
    channels: ['flows', 'nodes']
  }
};

// Launch Station Module
const launchConfig: ModuleStorageConfig = {
  moduleId: 'launch-station',
  storage: {
    primary: 'online-first',
    collections: {
      'analytics': {
        sync: 'immediate',
        cache: 'minimal',
        ttl: 300000 // 5 minutes
      },
      'logs': {
        sync: 'on-demand',
        cache: 'minimal',
        queue: 'best-effort'
      }
    }
  }
};

// Both modules work independently
const flowStorage = modularStorage.getModuleStorage(flowConfig);
const launchStorage = modularStorage.getModuleStorage(launchConfig);

// No data interference!
await flowStorage.set('flows', 'flow-1', { /* ... */ });
await launchStorage.set('analytics', 'event-1', { /* ... */ });

Example 3: Advanced Strategies

// E-commerce Module with complex data strategies
const ecommerceConfig: ModuleStorageConfig = {
  moduleId: 'ecommerce',
  storage: {
    primary: 'offline-first',
    collections: {
      // Critical: Products - immediate sync, aggressive cache
      'products': {
        sync: 'immediate',
        cache: 'aggressive',
        index: 'text',      // Full-text search
        ttl: 3600000,       // 1 hour
        conflict: 'remote-wins'
      },
      
      // Important: Cart - eager sync, aggressive cache
      'cart': {
        sync: 'eager',
        cache: 'aggressive',
        conflict: 'merge',
        queue: 'guaranteed' // Never lose cart data
      },
      
      // Normal: Wishlist - lazy sync, windowed cache
      'wishlist': {
        sync: 'lazy',
        cache: 'windowed',
        ttl: 86400000,      // 24 hours
        conflict: 'local-wins'
      },
      
      // Low priority: Reviews - on-demand sync
      'reviews': {
        sync: 'on-demand',
        cache: 'minimal',
        queue: 'best-effort'
      },
      
      // Never sync: Draft orders (local-only)
      'drafts': {
        sync: 'local-only',
        cache: 'minimal',
        ttl: 604800000      // 7 days
      },
      
      // Real-time: Live inventory
      'inventory': {
        sync: 'real-time',
        cache: 'aggressive',
        conflict: 'remote-wins'
      }
    }
  },
  realtime: {
    enabled: true,
    channels: ['inventory', 'cart']
  }
};

🔧 API Reference

ModularStorageService

Main service for module storage management.

Methods

getModuleStorage(config: ModuleStorageConfig): ModuleStorage

Get storage instance for a module. Creates isolated storage with module-specific configuration.

const storage = modularStorage.getModuleStorage({
  moduleId: 'my-module',
  storage: {
    primary: 'offline-first',
    collections: { /* ... */ }
  }
});

hasModuleStorage(moduleId: string): boolean

Check if module has a storage instance.

if (modularStorage.hasModuleStorage('my-module')) {
  console.log('Module storage exists');
}

getRegisteredModules(): string[]

Get all registered module IDs.

const modules = modularStorage.getRegisteredModules();
console.log('Registered modules:', modules);
// ['space-map', 'flow-composer', 'launch-station']

clearModuleStorage(moduleId: string): Promise<void>

Clear all storage for a specific module.

await modularStorage.clearModuleStorage('my-module');

getStats(): { moduleId: string; itemCount: number }[]

Get storage statistics for all modules.

const stats = modularStorage.getStats();
console.log('Storage stats:', stats);
// [{ moduleId: 'space-map', itemCount: 42 }, ...]

ModuleStorage

Module-specific storage instance.

Methods

get<T>(collection: string, key: string): Promise<T | null>

Get data from module's collection.

const user = await storage.get('users', 'user-123');

set<T>(collection: string, key: string, value: T): Promise<void>

Set data in module's collection.

await storage.set('users', 'user-123', {
  name: 'John Doe',
  email: '[email protected]'
});

delete(collection: string, key: string): Promise<void>

Delete data from module's collection.

await storage.delete('users', 'user-123');

query<T>(collection: string): Promise<T[]>

Query collection (returns all items in collection).

const allUsers = await storage.query('users');

Note: Currently returns empty array. Full implementation coming soon.

clearCollection(collection: string): Promise<void>

Clear entire collection.

await storage.clearCollection('users');

Note: Currently logs warning. Full implementation coming soon.

💡 Best Practices

1. Choose the Right Sync Strategy

// ✅ GOOD: Immediate sync for critical data
'payments': {
  sync: 'immediate',
  cache: 'aggressive',
  queue: 'guaranteed'
}

// ✅ GOOD: Lazy sync for non-critical data
'analytics': {
  sync: 'lazy',
  cache: 'minimal',
  queue: 'best-effort'
}

// ✅ GOOD: Local-only for temporary data
'drafts': {
  sync: 'local-only',
  cache: 'minimal'
}

// ❌ BAD: Immediate sync for everything (unnecessary)
'logs': {
  sync: 'immediate', // Overkill for logs
  cache: 'aggressive'
}

2. Use Collection-Level Strategies

// ✅ GOOD: Different strategies per collection
collections: {
  'products': {
    sync: 'immediate',
    cache: 'aggressive',
    index: 'text'
  },
  'logs': {
    sync: 'on-demand',
    cache: 'minimal'
  }
}

// ❌ BAD: Same strategy for everything
collections: {
  'products': { sync: 'immediate' },
  'logs': { sync: 'immediate' } // Logs don't need immediate sync
}

3. Set Appropriate TTLs

// ✅ GOOD: TTL based on data freshness needs
'products': {
  cache: 'aggressive',
  ttl: 3600000 // 1 hour - products change occasionally
},
'prices': {
  cache: 'aggressive',
  ttl: 300000 // 5 minutes - prices change frequently
},
'static-content': {
  cache: 'aggressive',
  ttl: 86400000 // 24 hours - rarely changes
}

// ❌ BAD: No TTL or inappropriate TTL
'prices': {
  cache: 'aggressive',
  ttl: 86400000 // BAD: Prices might be stale for 24 hours!
}

4. Use Conflict Resolution Wisely

// ✅ GOOD: Local-wins for user preferences
'settings': {
  conflict: 'local-wins' // User's local settings take priority
}

// ✅ GOOD: Remote-wins for inventory
'inventory': {
  conflict: 'remote-wins' // Server has authoritative inventory count
}

// ✅ GOOD: Merge for collaborative data
'documents': {
  conflict: 'merge' // Intelligent merge of concurrent edits
}

🎯 Use Cases

Use Case 1: Multi-Module Platform Builder

Perfect for applications with multiple independent modules:

• Space Map - User's workspace and universes
• Flow Composer - Visual programming flows
• Launch Station - Deployment and analytics

Each module has isolated storage with custom strategies.

Use Case 2: E-commerce Application

Different data types require different strategies:

• Products - Immediate sync, aggressive cache, text index
• Cart - Eager sync, guaranteed delivery, merge conflicts
• Wishlist - Lazy sync, windowed cache
• Inventory - Real-time sync, remote-wins

Use Case 3: Collaborative Editor

Complex collaboration requirements:

• Documents - Real-time sync, merge conflicts, compressed storage
• Comments - Eager sync, aggressive cache
• User Presence - Real-time sync, minimal cache
• Drafts - Local-only, no sync

⚠️ Current Limitations

Implemented

• ✅ Module namespace isolation
• ✅ Collection-level strategy configuration
• ✅ Basic CRUD operations (get/set/delete)
• ✅ TTL support
• ✅ Module registry

Planned (Q1 2025)

• 🚧 query() - Full collection querying
• 🚧 clearCollection() - Collection clearing
• 🚧 Advanced caching strategies (windowed, spatial)
• 🚧 Real-time sync integration
• 🚧 Conflict resolution implementation
• 🚧 Storage statistics per module

🔗 Related Documentation

• Storage System Overview - Overall system features
• Unified Storage Plan - Current implementation status
• Storage Enhancements - Future roadmap (Modular Storage is Priority 1)

🚀 Next Steps

1. Define your module's config - Decide on sync strategies per collection
2. Get module storage instance - Use ModularStorageService.getModuleStorage()
3. Use collection-specific storage - Call get/set/delete with collection names
4. Monitor performance - Check cache hit rates and sync patterns
5. Iterate and optimize - Adjust strategies based on real usage

Ready to use Modular Storage? Start with the Quick Start above! ⚡