👻 Phantom Universe ID Bug

🐛 The Problem

Symptom: Backend logs show "Universe not found" errors when trying to load universes that exist in space.universeIds but not in MongoDB.

[SpaceDatabaseService] ❌ Universe not found: universe_1763577075691

Root Cause: Universe IDs are being added to space.universeIds array BEFORE the actual universe data is saved to MongoDB.

🔍 How This Happens

Scenario 1: Create Universe Flow

// Browser 1: User creates universe
createUniverse() {
  // Step 1: Save universe locally (instant!)
  await this.syncEngine.write(`universe_${id}`, universe, 'universe');
  
  // Step 2: Update space.universeIds locally (instant!)
  space.universeIds.push(universe.id);
  await this.syncEngine.write(`space_${ownerId}`, space, 'space');
  
  // Step 3: Try to sync to backend (5 seconds delay or offline)
  if (this._isOnline()) {
    await this.spaceApi.createUniverse(spaceId, universe); // ❌ Might fail or delay
  } else {
    await this.queueSyncOperation('create_universe', ...); // ❌ Queued for later
  }
}

Problem: Steps 1-2 happen instantly (< 10ms), but Step 3 is delayed (5+ seconds) or might fail.

Result:

• space.universeIds = ["universe_xxx"] ✅ (in MongoDB)
• Universe data: ❌ (not in MongoDB yet!)

Scenario 2: Change Stream Amplification

// Browser 1: Creates universe, updates space.universeIds
// → Change Stream detects space update
// → Broadcasts to Browser 2

// Browser 2: Receives Change Stream event
realtimeSync.onSpaceUpdate((space) => {
  // space.universeIds = ["universe_xxx"] ← NEW UNIVERSE ID!
  this._currentSpace.set(space);
  
  if (idsChanged) {
    this.debouncedLoadUniverses(300); // ← Triggers in 300ms
  }
});

// 300ms later: loadUniverses() runs
loadUniverses() {
  for (const universeId of space.universeIds) {
    // Try to load universe_xxx
    let universe = await this.syncEngine.read(`universe_${universeId}`);
    // ❌ Not in local storage (Browser 2 just got the space update)
    
    // Try to fetch from backend
    const remoteUniverse = await this.spaceApi.getUniverse(spaceId, universeId);
    // ❌ 404 Not Found (backend might not have it yet!)
  }
}

Why This Fails:

1. Browser 1 creates universe locally (instant)
2. Browser 1 updates space.universeIds locally (instant)
3. Browser 1 syncs space to MongoDB (~5 seconds)
4. Browser 1 syncs universe to MongoDB (~5 seconds, might be queued)
5. Change Stream detects space update (< 1 second after MongoDB write)
6. Browser 2 receives space with new universeIds (< 1 second)
7. Browser 2 tries to load universe (300ms debounce)
8. Total Time: Browser 2 tries to fetch at ~6 seconds, but universe might not be in MongoDB until 10+ seconds!

🎯 Solution Strategy

Option 1: Atomic Universe Creation (Backend-First) ⭐ RECOMMENDED

Change the creation flow to be backend-first for critical operations:

async createUniverse(data: { name: string; description?: string }): Promise<UniverseWithDeployment> {
  const space = this._currentSpace();
  if (!space) throw new Error('No space initialized');

  // Create universe object
  const universe: UniverseWithDeployment = {
    id: `universe_${Date.now()}`,
    name: data.name,
    slug: this.generateSlug(data.name),
    spaceId: space.id,
    // ... other fields
  };

  // ✅ OPTION 1: Save to backend FIRST (ensures atomicity)
  if (this._isOnline()) {
    try {
      // Backend handles BOTH operations atomically:
      // 1. Saves universe data to space_{id}/universes
      // 2. Updates space.universeIds array
      const created = await firstValueFrom(
        this.spaceApi.createUniverse(space.id, universe)
      );
      
      // Then update local storage (for offline access)
      await this.syncEngine.write(`universe_${universe.id}`, created, 'universe');
      space.universeIds.push(universe.id);
      await this.syncEngine.write(`space_${space.ownerId}`, space, 'space');
      
      this.logger.info('[SpaceService] ✅ Universe created via backend (atomic)');
      
      // Update UI
      this._currentSpace.set(space);
      this._universes.set([...this._universes(), universe]);
      
      return created;
      
    } catch (error) {
      this.logger.error('[SpaceService] ❌ Failed to create universe via backend:', error);
      throw error; // Don't create locally if backend fails
    }
  } else {
    // Offline: Use queue system (existing logic)
    await this.syncEngine.write(`universe_${universe.id}`, universe, 'universe');
    space.universeIds.push(universe.id);
    await this.syncEngine.write(`space_${space.ownerId}`, space, 'space');
    await this.queueSyncOperation('create_universe', space.id, { universe });
    
    this._currentSpace.set(space);
    this._universes.set([...this._universes(), universe]);
    
    return universe;
  }
}

Benefits:

• ✅ Atomic: Universe data and space.universeIds updated together
• ✅ Consistent: Other browsers always see complete data
• ✅ No phantom IDs: IDs only appear after data exists
• ✅ Still fast: Backend write is ~200ms (acceptable UX)

Tradeoffs:

• ⚠️ Requires online connection for creation
• ⚠️ Slightly slower than local-first (200ms vs 5ms)

Option 2: Delayed Space Update (Local-First with Delay)

Keep local-first but delay the space.universeIds update until backend confirms:

async createUniverse(data): Promise<UniverseWithDeployment> {
  const space = this._currentSpace();
  const universe = { id: `universe_${Date.now()}`, ... };

  // Step 1: Save universe locally (instant for offline access)
  await this.syncEngine.write(`universe_${universe.id}`, universe, 'universe');
  
  // Step 2: Update UI immediately (optimistic)
  this._universes.set([...this._universes(), universe]);
  
  // Step 3: Sync to backend BEFORE updating space.universeIds
  if (this._isOnline()) {
    try {
      await firstValueFrom(this.spaceApi.createUniverse(space.id, universe));
      this.logger.info('[SpaceService] ✅ Universe synced to backend');
      
      // Step 4: ONLY NOW update space.universeIds
      space.universeIds.push(universe.id);
      await this.syncEngine.write(`space_${space.ownerId}`, space, 'space');
      this._currentSpace.set(space);
      
    } catch (error) {
      this.logger.error('[SpaceService] ❌ Failed to sync universe, NOT updating space.universeIds');
      // Universe exists locally but not in space.universeIds
      // Will be orphaned but won't cause phantom ID errors
    }
  } else {
    // Offline: Queue the ENTIRE operation (universe + space update)
    await this.queueSyncOperation('create_universe_with_space_update', space.id, {
      universe,
      spaceUpdate: { universeIds: [...space.universeIds, universe.id] }
    });
  }
  
  return universe;
}

Benefits:

• ✅ No phantom IDs: space.universeIds only updated after backend confirms
• ✅ Fast UI: Users see universe immediately (optimistic)
• ✅ Local-first: Universe data stored locally for offline access

Tradeoffs:

• ⚠️ Orphan universes: If backend fails, universe exists locally but not in space.universeIds
• ⚠️ Complexity: Need to handle orphan cleanup

Option 3: Backend Endpoint Enhancement (Best Long-Term)

Add a new backend endpoint that handles the entire flow atomically:

// Backend: spaces.controller.ts
@Post(':spaceId/universes/atomic')
async createUniverseAtomic(
  @Param('spaceId') spaceId: string,
  @GetUserId() userId: string,
  @Body() universeData: any,
) {
  // START TRANSACTION (MongoDB supports multi-document transactions)
  const session = await this.connection.startSession();
  session.startTransaction();
  
  try {
    // Step 1: Save universe data to space_{id}/universes
    await this.spaceDatabaseService.saveUniverseData(
      spaceId,
      universeData.id,
      universeData,
      { session } // ← Use transaction
    );
    
    // Step 2: Update space.universeIds array
    const space = await this.spaceModel.findById(spaceId).session(session);
    if (!space.universeIds.includes(universeData.id)) {
      space.universeIds.push(universeData.id);
      await space.save({ session });
    }
    
    // COMMIT TRANSACTION (both operations succeed or both fail)
    await session.commitTransaction();
    
    return universeData;
    
  } catch (error) {
    // ROLLBACK on error
    await session.abortTransaction();
    throw error;
  } finally {
    session.endSession();
  }
}

Benefits:

• ✅ True atomicity: MongoDB transactions ensure consistency
• ✅ No phantom IDs: Both operations succeed together or fail together
• ✅ Best reliability: Industry standard approach
• ✅ Scalable: Handles concurrent operations correctly

Tradeoffs:

• ⚠️ Requires MongoDB 4.0+: Transactions need replica set
• ⚠️ Backend work: Need to implement transaction logic

📊 Comparison Matrix

🎯 Recommended Implementation: Option 1 (Backend-First for Mutations)

Why Option 1?

1. Simple: Minimal code changes
2. Reliable: Backend ensures consistency
3. Fast enough: 200ms is imperceptible to users
4. Production-ready: No complex transaction logic needed
5. Debug-friendly: Clear error messages when backend fails

Implementation Plan

Phase 1: Update createUniverse() (30 min)

// libs/libs-foundational/core-angular/src/lib/services/space.service.ts

async createUniverse(data: { name: string; description?: string }): Promise<UniverseWithDeployment> {
  const space = this._currentSpace();
  if (!space) throw new Error('No space initialized');

  const universe: UniverseWithDeployment = {
    id: `universe_${Date.now()}`,
    name: data.name,
    description: data.description || '',
    slug: this.generateSlug(data.name),
    deploymentStatus: 'draft',
    galaxyIds: [],
    metadata: {
      createdAt: new Date().toISOString(),
      updatedAt: new Date().toISOString(),
      createdBy: space.ownerId
    }
  };

  (universe as any).spaceId = space.id;

  // 🔥 NEW APPROACH: Backend-first for consistency
  if (this._isOnline()) {
    try {
      this.logger.info('[SpaceService] 🌐 Creating universe via backend (atomic)...');
      
      // Backend handles BOTH:
      // 1. Save universe data
      // 2. Update space.universeIds
      const created = await firstValueFrom(
        this.spaceApi.createUniverse(space.id, universe)
      );
      
      // Cache locally for offline access
      await this.syncEngine.write(`universe_${universe.id}`, created, 'universe');
      
      // Update local space copy
      space.universeIds = this.cleanUniverseIds([...space.universeIds, universe.id]);
      space.updatedAt = new Date().toISOString();
      await this.syncEngine.write(`space_${space.ownerId}`, space, 'space');
      await this.cacheSpaceSlugVariant(space);
      
      // Update signals
      this._currentSpace.set(space);
      this._universes.set([...this._universes(), created]);
      
      this.logger.info('[SpaceService] ✅ Universe created via backend (atomic)', {
        universeId: created.id,
        spaceId: space.id
      });
      
      return created;
      
    } catch (error) {
      this.logger.error('[SpaceService] ❌ Failed to create universe via backend:', error);
      throw error; // Fail fast - don't create phantom IDs
    }
  } else {
    // Offline: Use existing queue system
    await this.syncEngine.write(`universe_${universe.id}`, universe, 'universe');
    space.universeIds = this.cleanUniverseIds([...space.universeIds, universe.id]);
    space.updatedAt = new Date().toISOString();
    await this.syncEngine.write(`space_${space.ownerId}`, space, 'space');
    await this.cacheSpaceSlugVariant(space);
    
    this._currentSpace.set(space);
    this._universes.set([...this._universes(), universe]);
    
    await this.queueSyncOperation('create_universe', space.id, { universe });
    
    this.logger.info('[SpaceService] ⚡ Universe created offline (queued for sync)');
    return universe;
  }
}

Phase 2: Add Safety Check in loadUniverses() (15 min)

async loadUniverses(): Promise<void> {
  // ... existing code ...
  
  for (const universeId of space.universeIds || []) {
    let universe = await this.syncEngine.read<UniverseWithDeployment>(`universe_${universeId}`);
    
    if (!universe && this._isOnline()) {
      try {
        // 🔥 ADD RETRY LOGIC with exponential backoff
        const maxRetries = 3;
        let retryDelay = 1000; // Start with 1 second
        
        for (let attempt = 1; attempt <= maxRetries; attempt++) {
          try {
            this.logger.info(`[SpaceService] 🔄 Attempt ${attempt}/${maxRetries} to fetch universe:`, universeId);
            
            const remoteUniverse = await firstValueFrom(
              this.spaceApi.getUniverse(space.id, universeId)
            );
            
            if (remoteUniverse) {
              (remoteUniverse as any).spaceId = space.id;
              const normalized = this.normalizeUniverse(universeId, remoteUniverse as UniverseWithDeployment, space.id);
              
              if (normalized) {
                await this.storage.put(`universe_${normalized.id}`, normalized);
                universe = normalized;
                this.logger.info('[SpaceService] ✅ Universe fetched from backend');
                break; // Success!
              }
            }
          } catch (fetchError) {
            if (attempt < maxRetries) {
              this.logger.warn(`[SpaceService] ⏳ Retry ${attempt} failed, waiting ${retryDelay}ms...`);
              await new Promise(resolve => setTimeout(resolve, retryDelay));
              retryDelay *= 2; // Exponential backoff
            } else {
              this.logger.error(`[SpaceService] ❌ All ${maxRetries} attempts failed for universe:`, universeId);
            }
          }
        }
      } catch (error) {
        this.logger.error('[SpaceService] ❌ Failed to fetch universe after retries:', universeId);
      }
    }
    
    // 🔥 CRITICAL: Only add to list if universe exists
    if (universe) {
      // ... existing validation ...
      validUniverses.push(universe);
    } else {
      // 🚨 PHANTOM ID DETECTED: Universe ID in array but no data exists
      this.logger.error('[SpaceService] 👻 PHANTOM UNIVERSE ID detected (exists in space.universeIds but no data found):', {
        universeId,
        spaceId: space.id,
        spaceSlug: space.spaceSlug
      });
      
      invalidUniverseIds.push(universeId);
      
      // TODO: Optionally clean up phantom IDs from space.universeIds
      // (requires backend sync to remove)
    }
  }
  
  // ... rest of method ...
}

Phase 3: Add Cleanup Job (Optional, Future Enhancement)

/**
 * 🧹 Clean up phantom universe IDs (IDs in space.universeIds with no data)
 * Run this periodically or on-demand
 */
async cleanupPhantomUniverseIds(): Promise<void> {
  const space = this._currentSpace();
  if (!space) return;
  
  const validIds: string[] = [];
  const phantomIds: string[] = [];
  
  for (const universeId of space.universeIds) {
    // Check if universe data exists in backend
    try {
      const exists = await firstValueFrom(
        this.spaceApi.getUniverse(space.id, universeId)
      );
      if (exists) {
        validIds.push(universeId);
      } else {
        phantomIds.push(universeId);
      }
    } catch {
      phantomIds.push(universeId);
    }
  }
  
  if (phantomIds.length > 0) {
    this.logger.warn('[SpaceService] 🧹 Cleaning up phantom universe IDs:', phantomIds);
    
    // Update space with only valid IDs
    space.universeIds = validIds;
    space.updatedAt = new Date().toISOString();
    
    await this.syncEngine.write(`space_${space.ownerId}`, space, 'space');
    
    if (this._isOnline()) {
      await firstValueFrom(
        this.spaceApi.updateSpace(space.id, { universeIds: validIds })
      );
    }
    
    this._currentSpace.set(space);
    await this.loadUniverses();
  }
}

✅ Success Criteria

After implementing Option 1, we should see:

1. No more "Universe not found" errors in backend logs
2. Consistent data across all browsers immediately
3. Fast creation: Universe creation completes in ~200ms when online
4. Reliable: If backend fails, creation fails (no partial state)
5. Clean logs: Clear success/failure messages

🎓 Key Takeaways

1. Atomicity Matters: Multi-step operations need to be atomic
2. Backend-First for Mutations: Critical operations should hit backend first
3. Local-First for Reads: Keep reads fast with local cache
4. Change Streams are Fast: They propagate before background sync completes
5. Always Validate: Never trust IDs without data

Created: November 19, 2025 - 1:50 PM
Session: 11.25.3.1
Status: Ready to implement
Priority: High (data consistency issue)