Nx Cache Corruption

Documentation for Developers

Nx Cache Corruption: Prevention & Recovery Guide

🚨 Critical Issue Overview

Problem: Recurring Nx cache corruption causing development server failures and library deletion

Impact:

Development servers fail to start with ENOENT errors
dist/ux-portal/ and other library builds get deleted
Significant time lost debugging the same issue repeatedly
Developer productivity severely impacted

Root Cause: Nx cache system becomes corrupted, leaving references to non-existent terminal output files

🔍 Issue Analysis

Error Pattern

Error: ENOENT: no such file or directory, open '/Users/.../editor/.nx/cache/terminalOutputs/[RANDOM_ID]'
    at writeFileSync (node:fs:2426:20)
    at ForkedProcessTaskRunner.writeTerminalOutput

Why `ux-portal` Gets Deleted

Library Build Process: ux-portal is configured to output to dist/ux-portal/ via ng-package.json
Build Dependency Chain: When cache corruption occurs, builds fail mid-process
Cleanup Behavior: Failed builds can trigger cleanup of output directories
Non-Critical Status: Since ux-portal isn't critical for main apps, it gets cleaned first

Corruption Triggers

Concurrent Builds: Multiple build processes accessing cache simultaneously
Interrupted Processes: Killing builds mid-execution leaves cache in inconsistent state
Storage Issues: Disk space or permission problems affecting cache writes
Node Version Changes: Different Node versions can invalidate cache entries

🛡️ Prevention Strategies

1. Automated Cache Health Monitoring

Create a cache health check script:

#!/bin/bash
# scripts/check-nx-cache-health.sh

CACHE_DIR=".nx/cache"
HEALTH_FILE="$CACHE_DIR/.health-check"

echo "🔍 Checking Nx cache health..."

# Check if cache directory exists and is writable
if [ ! -d "$CACHE_DIR" ]; then
    echo "❌ Nx cache directory missing"
    exit 1
fi

if [ ! -w "$CACHE_DIR" ]; then
    echo "❌ Nx cache directory not writable"
    exit 1
fi

# Check for orphaned terminal output references
ORPHANED_COUNT=$(find "$CACHE_DIR/terminalOutputs" -name "*.tmp" 2>/dev/null | wc -l)
if [ "$ORPHANED_COUNT" -gt 10 ]; then
    echo "⚠️ Found $ORPHANED_COUNT orphaned terminal outputs - cache may be corrupted"
    exit 1
fi

# Write health check timestamp
echo "$(date)" > "$HEALTH_FILE"
echo "✅ Nx cache is healthy"

2. Pre-Development Setup Script

#!/bin/bash
# scripts/pre-dev-setup.sh

echo "🚀 Pre-development environment setup..."

# 1. Check Nx cache health
./scripts/check-nx-cache-health.sh
if [ $? -ne 0 ]; then
    echo "🔧 Cache issues detected, performing reset..."
    yarn nx reset
fi

# 2. Verify critical library builds exist
CRITICAL_LIBS=("core-types" "core-models" "theme-system" "ux-portal")
NEEDS_BUILD=false

for lib in "${CRITICAL_LIBS[@]}"; do
    if [ ! -d "dist/$lib" ]; then
        echo "📦 Missing build for $lib"
        NEEDS_BUILD=true
    fi
done

# 3. Build missing libraries if needed
if [ "$NEEDS_BUILD" = true ]; then
    echo "🔨 Building missing libraries..."
    yarn build:libs
fi

# 4. Check port availability
PORTS=(5100 5200 5300 9100)
for port in "${PORTS[@]}"; do
    if lsof -i :$port > /dev/null; then
        echo "⚠️ Port $port is already in use"
    fi
done

echo "✅ Environment ready for development"

3. Enhanced Package Scripts

Add to package.json:

{
  "scripts": {
    "//HEALTH": "=== HEALTH & PREVENTION ===",
    "health:check": "./scripts/check-nx-cache-health.sh",
    "health:fix": "yarn nx reset && yarn build:libs",
    "dev:safe": "./scripts/pre-dev-setup.sh && yarn unified:dev",
    
    "//ERROR RECOVERY": "=== ERROR RECOVERY ===",
    "recover:cache": "yarn nx reset && yarn install && yarn build:libs",
    "recover:full": "rm -rf node_modules dist .nx && yarn install && yarn build:libs"
  }
}

🔧 Automated Prevention System

1. Pre-commit Hook

#!/bin/bash
# .git/hooks/pre-commit

# Check for cache corruption before commits
./scripts/check-nx-cache-health.sh
if [ $? -ne 0 ]; then
    echo "❌ Cache corruption detected - run 'yarn health:fix' before committing"
    exit 1
fi

2. Development Server Wrapper

Create scripts/safe-dev-server.sh:

#!/bin/bash

echo "🛡️ Starting safe development server..."

# Function to cleanup on exit
cleanup() {
    echo "🧹 Cleaning up development processes..."
    pkill -f "nx serve" 2>/dev/null
    pkill -f "run-dev-server" 2>/dev/null
}

# Set trap for cleanup
trap cleanup EXIT INT TERM

# Pre-flight checks
./scripts/pre-dev-setup.sh
if [ $? -ne 0 ]; then
    echo "❌ Pre-flight checks failed"
    exit 1
fi

# Start development server with monitoring
echo "🚀 Starting unified development server..."
yarn unified:dev &
DEV_PID=$!

# Monitor for cache corruption
while kill -0 $DEV_PID 2>/dev/null; do
    sleep 30
    ./scripts/check-nx-cache-health.sh >/dev/null
    if [ $? -ne 0 ]; then
        echo "🚨 Cache corruption detected during development!"
        echo "🔧 Attempting automatic recovery..."
        kill $DEV_PID
        yarn nx reset
        yarn unified:dev &
        DEV_PID=$!
    fi
done

wait $DEV_PID

📊 Detection & Monitoring

1. Cache Corruption Detector

Create tools/cache-monitor.js:

#!/usr/bin/env node
const fs = require('fs');
const path = require('path');
const { execSync } = require('child_process');

class CacheMonitor {
  constructor() {
    this.cacheDir = '.nx/cache';
    this.terminalOutputsDir = path.join(this.cacheDir, 'terminalOutputs');
  }

  checkHealth() {
    const issues = [];

    // Check if cache directory exists
    if (!fs.existsSync(this.cacheDir)) {
      issues.push('Cache directory missing');
      return { healthy: false, issues };
    }

    // Check for orphaned terminal outputs
    try {
      const terminalFiles = fs.readdirSync(this.terminalOutputsDir);
      const orphanedFiles = terminalFiles.filter(file => 
        file.endsWith('.tmp') || file.includes('undefined')
      );

      if (orphanedFiles.length > 10) {
        issues.push(`${orphanedFiles.length} orphaned terminal outputs found`);
      }
    } catch (error) {
      issues.push('Cannot read terminal outputs directory');
    }

    // Check for corrupted cache entries
    try {
      const cacheEntries = fs.readdirSync(this.cacheDir);
      const suspiciousEntries = cacheEntries.filter(entry => 
        entry.includes('undefined') || entry.length < 5
      );

      if (suspiciousEntries.length > 0) {
        issues.push(`${suspiciousEntries.length} suspicious cache entries found`);
      }
    } catch (error) {
      issues.push('Cannot read cache directory');
    }

    return {
      healthy: issues.length === 0,
      issues
    };
  }

  autoFix() {
    console.log('🔧 Attempting automatic cache fix...');
    
    try {
      // Reset Nx cache
      execSync('yarn nx reset', { stdio: 'inherit' });
      
      // Rebuild critical libraries
      const criticalLibs = ['core-types', 'core', 'theme-system', 'ux-portal'];
      for (const lib of criticalLibs) {
        if (!fs.existsSync(`dist/${lib}`)) {
          console.log(`📦 Rebuilding ${lib}...`);
          execSync(`yarn nx build ${lib}`, { stdio: 'inherit' });
        }
      }
      
      console.log('✅ Cache fix completed');
      return true;
    } catch (error) {
      console.error('❌ Cache fix failed:', error.message);
      return false;
    }
  }
}

// CLI usage
if (require.main === module) {
  const monitor = new CacheMonitor();
  const health = monitor.checkHealth();
  
  if (health.healthy) {
    console.log('✅ Cache is healthy');
    process.exit(0);
  } else {
    console.log('❌ Cache issues detected:');
    health.issues.forEach(issue => console.log(`  - ${issue}`));
    
    if (process.argv.includes('--fix')) {
      monitor.autoFix();
    } else {
      console.log('\nRun with --fix to attempt automatic repair');
      process.exit(1);
    }
  }
}

module.exports = CacheMonitor;

🎯 Library Protection Strategy

Why `ux-portal` Gets Deleted

Build Output Configuration:

// libs/ux-portal/ng-package.json
{
  "dest": "../../dist/ux-portal"
}

Build Failure Cleanup: When builds fail, cleanup processes can remove output directories
Non-Critical Dependency: Since ux-portal isn't imported by main apps, it's often cleaned first

Protection Measures

Library Build Verification:

#!/bin/bash
# scripts/verify-critical-libs.sh

CRITICAL_LIBS=("core-types" "core-models" "theme-system" "ux-portal" "widgets")

for lib in "${CRITICAL_LIBS[@]}"; do
    if [ ! -d "dist/$lib" ]; then
        echo "❌ Critical library missing: $lib"
        echo "🔨 Rebuilding $lib..."
        yarn nx build $lib
        
        if [ $? -ne 0 ]; then
            echo "💥 Failed to rebuild $lib - manual intervention required"
            exit 1
        fi
    fi
done

Protected Build Process:

#!/bin/bash
# scripts/protected-build.sh

LIB_NAME=$1
BACKUP_DIR="dist/.backup"

# Create backup if library exists
if [ -d "dist/$LIB_NAME" ]; then
    mkdir -p "$BACKUP_DIR"
    cp -r "dist/$LIB_NAME" "$BACKUP_DIR/$LIB_NAME-$(date +%s)"
fi

# Attempt build
yarn nx build $LIB_NAME

# Restore from backup if build failed and no output exists
if [ $? -ne 0 ] && [ ! -d "dist/$LIB_NAME" ] && [ -d "$BACKUP_DIR" ]; then
    LATEST_BACKUP=$(ls -t "$BACKUP_DIR" | grep "$LIB_NAME" | head -1)
    if [ -n "$LATEST_BACKUP" ]; then
        echo "🔄 Restoring $LIB_NAME from backup: $LATEST_BACKUP"
        cp -r "$BACKUP_DIR/$LATEST_BACKUP" "dist/$LIB_NAME"
    fi
fi

🚀 Implementation Plan

Phase 1: Immediate Prevention (Complete Today)

Add health check scripts:

mkdir -p scripts
# Create check-nx-cache-health.sh
# Create pre-dev-setup.sh
chmod +x scripts/*.sh

Update package.json:
```
# Add health and recovery scripts
```
Test prevention system:
```
yarn health:check
yarn dev:safe
```

Phase 2: Automated Monitoring (This Week)

Implement cache monitor
Add pre-commit hooks
Create protected build scripts

Phase 3: Advanced Prevention (Next Week)

Continuous monitoring during development
Automatic recovery mechanisms
Metrics and alerting

📋 Daily Workflow Changes

Before Starting Development:

# Old way (problematic)
yarn unified:dev

# New way (safe)
yarn dev:safe

When Issues Occur:

# Quick fix
yarn health:fix

# Full recovery (if needed)
yarn recover:full

Regular Maintenance:

# Weekly cache health check
yarn health:check

# Monthly full cleanup
yarn recover:cache

🔍 Troubleshooting Guide

Error: "ENOENT: no such file or directory"

Solution: yarn health:fix

Error: "ux-portal missing from dist"

Solution: yarn nx build ux-portal

Error: "Multiple build processes"

Solution:

pkill -f "nx serve"
yarn dev:safe

Error: "Port already in use"

Solution:

lsof -i :5100 -i :5200 -i :5300 -i :9100
kill -9 [PID]

🎯 Success Metrics

Zero cache corruption incidents in development
Automatic recovery from 90%+ of issues
<30 seconds to detect and fix problems
100% library build availability during development

This comprehensive prevention system will eliminate the recurring cache corruption issues and protect critical library builds like ux-portal.

Nx Cache Corruption: Prevention & Recovery Guide

🚨 Critical Issue Overview

🔍 Issue Analysis

Error Pattern

Why ux-portal Gets Deleted

Corruption Triggers

🛡️ Prevention Strategies

1. Automated Cache Health Monitoring

2. Pre-Development Setup Script

3. Enhanced Package Scripts

🔧 Automated Prevention System

1. Pre-commit Hook

2. Development Server Wrapper

📊 Detection & Monitoring

1. Cache Corruption Detector

🎯 Library Protection Strategy

Why ux-portal Gets Deleted

Protection Measures

🚀 Implementation Plan

Phase 1: Immediate Prevention (Complete Today)

Phase 2: Automated Monitoring (This Week)

Phase 3: Advanced Prevention (Next Week)

📋 Daily Workflow Changes

Before Starting Development:

When Issues Occur:

Regular Maintenance:

🔍 Troubleshooting Guide

Error: "ENOENT: no such file or directory"

Error: "ux-portal missing from dist"

Error: "Multiple build processes"

Error: "Port already in use"

🎯 Success Metrics

Why `ux-portal` Gets Deleted

Why `ux-portal` Gets Deleted