Search Manifest Format Fix

Fixed search index building to work with new manifest structure format

🔧 Search Manifest Format Fix

Date: October 29, 2025
Time: Afternoon
Status: ✅ COMPLETE
Purpose: Fix search index building to work with actual manifest structure

📋 Problem

The search index was empty (🔍 Search index length: 0) because:

setupSearchStream() not running on init: The effect only triggered on language changes, not initial load
Wrong manifest format: Code expected manifest.public and manifest.projects, but actual manifest has manifest.structure array
Hardcoded language: loadFileContent was hardcoded to 'en' instead of using actual language

✅ What Was Fixed

1. Initialize Search Index on Service Start

Before:

private setupSearchStream(): void {
  // Only ran when language changed via effect()
  effect(() => {
    const currentLang = this.documentationService.currentLanguage();
    // ... build index
  });
}

After:

private setupSearchStream(): void {
  // 🎯 IMMEDIATE: Build index on service initialization
  const initialLang = this.documentationService.currentLanguage();
  void (async () => {
    const manifest = await this.structureSyncService.loadManifest(false, initialLang);
    if (manifest) {
      await this.buildSearchIndexFromManifest(manifest, initialLang);
      console.log(`Index built with ${this.searchIndex.length} items`);
    }
  })();

  // 🔄 REACTIVE: Also rebuild when language changes
  effect(() => {
    const currentLang = this.documentationService.currentLanguage();
    // ... rebuild index
  });
}

2. Handle New Manifest Format

Actual Manifest Structure:

{
  "version": "1.0.0",
  "generated": "2025-10-29T15:41:17.768Z",
  "structure": [
    {
      "type": "directory",
      "name": "english",
      "language": "en",
      "isLanguageRoot": true,
      "items": [
        {
          "type": "directory",
          "name": "GoGrab-Space",
          "items": [
            {
              "type": "file",
              "name": "index.md",
              "path": "english/GoGrab-Space/index.md",
              "title": "GoGrab Space",
              "hasContent": true
            }
          ]
        }
      ]
    }
  ]
}

Added New Methods:

private async buildIndexFromStructureArray(structure: any[], language: string): Promise<void> {
  for (const item of structure) {
    if (item.type === 'directory' && item.isLanguageRoot && item.language === language) {
      // Process all items in this language directory
      if (item.items && Array.isArray(item.items)) {
        await this.indexStructureItems(item.items, language);
      }
    }
  }
}

private async indexStructureItems(items: any[], language: string, maxFiles: number = 100): Promise<void> {
  for (const item of items) {
    if (item.type === 'directory' && item.items) {
      // Recursively index directory items
      await this.indexStructureItems(item.items, language, maxFiles - filesIndexed);
    } else if (item.type === 'file' && item.hasContent) {
      // Index this file
      const content = await this.loadFileContent(item.path, language);
      if (content) {
        const projectName = this.extractProjectNameFromPath(item.path);
        this.searchIndex.push({
          path: `/${item.path.replace(/\.(md|mdc)$/, '')}`,
          title: item.title || item.cleanName || item.name,
          content: this.stripMarkdown(content),
          headings: this.extractHeadings(content),
          section: projectName,
          language: item.language || language,
          visibility: item.path.includes('/private/') ? 'private' : 'public',
          projectName,
          tags: item.frontMatter?.tags || [],
          searchablePhrases: item.frontMatter?.searchablePhrases || []
        });
      }
    }
  }
}

3. Extract Project Name from Path

private extractProjectNameFromPath(path: string): string {
  // Remove language prefix (english/ or spanish/)
  const withoutLang = path.replace(/^(english|spanish)\//, '');
  
  // Get first segment as project name
  const segments = withoutLang.split('/');
  return segments[0] || 'public';
}

Examples:

english/GoGrab-Space/apps/index.md → GoGrab-Space
english/public/development/index.md → public
spanish/GoGrab-Pulse/guides/index.md → GoGrab-Pulse

4. Fixed loadFileContent Language Parameter

Before:

private async loadFileContent(manifestPath: string): Promise<string> {
  const content = await this.markdownService.loadContent(manifestPath, 'en').toPromise(); // Hardcoded!
}

After:

private async loadFileContent(manifestPath: string, language: string = 'en'): Promise<string> {
  const content = await this.markdownService.loadContent(manifestPath, language).toPromise();
}

🔍 Console Output (Expected)

When you refresh the app, you should now see:

🔍 SearchService: Initializing...
🔍 setupSearchStream: Setting up search index building...
🔍 setupSearchStream: Building initial index for language: en
🔍 buildSearchIndexFromManifest: Starting...
🔍 buildSearchIndexFromManifest: Manifest structure: {hasStructure: true, hasPublic: false, hasProjects: false, manifestKeys: ['version', 'generated', 'basePath', 'structure']}
🔍 buildSearchIndexFromManifest: Using new structure format
🔍 buildIndexFromStructureArray: Processing 2 top-level items
🔍 buildIndexFromStructureArray: Found language root: english (en)
🔍 indexStructureItems: Indexed 10 files so far...
🔍 indexStructureItems: Indexed 20 files so far...
...
🔍 buildIndexFromStructureArray: Indexed 100 total items
🔍 buildSearchIndexFromManifest: Complete! Index now has 100 items
🔍 setupSearchStream: Initial index built with 100 items

Then when you search:

🔍 CONTENT SEARCH STARTING
🔍 Search index length: 100  ← NOT ZERO!
🔍 Search index items: [{title: "GoGrab Space", project: "GoGrab-Space", ...}, ...]

🧪 Testing

Test 1: Search Index Builds on Load

Open browser console
Refresh the page
Look for "setupSearchStream: Initial index built with X items"
X should be > 0 (probably 100, the limit)

Test 2: Search Works

Press ⌘K to open search
Type "space" or any term
Should see actual results from indexed files

Test 3: Project Scoping Works

Navigate to GoGrab-Space project
Search for "widgets"
Should see results from GoGrab-Space (and public if configured)
Should NOT see results from GoGrab-Pulse

Test 4: Global Search Toggle Works

Click the "Local/Global" toggle
Search again
Should now see results from multiple projects

📊 Summary

What Changed:

✅ setupSearchStream() now builds index immediately on init
✅ Added support for new manifest structure array format
✅ Added buildIndexFromStructureArray() method
✅ Added indexStructureItems() recursive method
✅ Added extractProjectNameFromPath() helper
✅ Fixed loadFileContent to use actual language parameter
✅ Comprehensive console logging for debugging

Files Changed:

search.service.ts - Updated index building logic

Result:

Search index now builds from actual manifest files
Default limit: 100 files (configurable)
Works with new manifest structure format
Respects language selection
Proper project name extraction

Last Updated: October 29, 2025
Status: ✅ COMPLETE - TEST NOW

🎯 Next Steps

Test the search - Refresh and try searching
Check console - Verify index builds successfully
Adjust maxFiles - If needed, increase from 100 to index more files
Monitor performance - Watch for slow indexing

The search should now work exactly as configured! 🚀