Cache Monitoring and Health Tools

🔍 Overview

These tools help monitor, diagnose, and fix NX cache issues that can cause build failures and performance problems in our monorepo.

📁 Available Tools

1. cache-monitor.js - Advanced Cache Monitor

Purpose: Advanced detection and automatic fixing of cache corruption.

Usage:

# Check cache health
node tools/scripts/cache-monitor.js

# Attempt automatic repair
node tools/scripts/cache-monitor.js --fix

# Generate detailed report
node tools/scripts/cache-monitor.js --report

# Check critical libraries
node tools/scripts/cache-monitor.js --libs

Features:

• Detects orphaned terminal outputs
• Identifies corrupted cache entries
• Finds stale lock files
• Monitors disk space usage
• Automatic repair capabilities
• Critical library status checking

Health Checks:

• Cache directory existence and permissions
• Orphaned terminal output files
• Files with 'undefined' in names (corruption indicator)
• Stale lock files (older than 24 hours)
• Cache size monitoring (warns if > 1GB)
• Critical library build status

2. check-nx-cache-health.sh - Cache Health Check

Purpose: Shell script for quick cache health verification.

Usage:

# Check cache health
./tools/scripts/check-nx-cache-health.sh

# Use in CI/CD pipelines
if ./tools/scripts/check-nx-cache-health.sh; then
  echo "Cache is healthy"
else
  echo "Cache issues detected"
fi

What it checks:

• Cache directory existence and writability
• Orphaned terminal output files
• Files with 'undefined' in names
• Available disk space
• Stale lock files
• Writes health check timestamp

🛠️ Cache Management Commands

Quick Health Check

# Basic health check
node tools/scripts/cache-monitor.js

# Shell script version
./tools/scripts/check-nx-cache-health.sh

Automatic Repair

# Fix cache issues automatically
node tools/scripts/cache-monitor.js --fix

Detailed Analysis

# Generate comprehensive report
node tools/scripts/cache-monitor.js --report

# Check critical libraries
node tools/scripts/cache-monitor.js --libs

Manual Cache Reset

# Reset NX cache completely
yarn nx reset

# Or with npm
npx nx reset

🔧 Cache Issues and Solutions

Common Cache Problems

1. Orphaned Terminal Outputs

   • Symptom: Build hangs or fails with terminal output errors
   • Solution: Run node tools/scripts/cache-monitor.js --fix

2. Corrupted Cache Entries

   • Symptom: "Cannot destructure property 'pos'" errors
   • Solution: Clean cache and rebuild

3. Stale Lock Files

   • Symptom: Build processes hang or fail to start
   • Solution: Automatic cleanup via --fix option

4. Low Disk Space

   • Symptom: Cache operations fail
   • Solution: Clean cache and free up disk space

5. Permission Issues

   • Symptom: Cache directory not writable
   • Solution: Fix directory permissions

Cache Corruption Indicators

• Files with 'undefined' in names
• Orphaned .tmp files
• Lock files older than 24 hours
• Cache size exceeding 1GB
• Terminal output files without corresponding processes

📊 Monitoring and Reporting

Health Report Format

{
  "timestamp": "2025-01-15T10:30:00.000Z",
  "healthy": true,
  "issues": [],
  "cacheSize": {
    "bytes": 52428800,
    "mb": 50,
    "gb": 0.05
  },
  "criticalLibraries": [
    {
      "name": "core-types",
      "built": true,
      "path": "dist/core-types"
    }
  ]
}

Critical Libraries Monitoring

The tools monitor these critical libraries:

• core-types - Base types and interfaces
• core - Core functionality
• theme-system - Theme system
• ux-portal - UX portal components
• widgets - Widget system

🚀 Quick Start Examples

Daily Health Check

# Quick health check
node tools/scripts/cache-monitor.js

# If issues found, auto-fix
node tools/scripts/cache-monitor.js --fix

Pre-Build Cache Check

# Check cache before building
if node tools/scripts/cache-monitor.js; then
  echo "Cache healthy, proceeding with build"
  node tools/scripts/build-scripts/build-all.js
else
  echo "Cache issues detected, fixing..."
  node tools/scripts/cache-monitor.js --fix
  node tools/scripts/build-scripts/build-all.js
fi

CI/CD Integration

# In CI pipeline
./tools/scripts/check-nx-cache-health.sh || {
  echo "Cache health check failed"
  exit 1
}

Troubleshooting Build Issues

# When builds fail, check cache first
node tools/scripts/cache-monitor.js --report

# Fix any issues
node tools/scripts/cache-monitor.js --fix

# Check critical libraries
node tools/scripts/cache-monitor.js --libs

# Rebuild if needed
node tools/scripts/build-scripts/build-libs.js core-types

🔄 Maintenance Schedule

Daily

• Run basic health check: node tools/scripts/cache-monitor.js

Weekly

• Generate detailed report: node tools/scripts/cache-monitor.js --report
• Check critical libraries: node tools/scripts/cache-monitor.js --libs

Monthly

• Full cache reset if issues persist: yarn nx reset
• Review cache size and clean if needed

Before Major Builds

• Always check cache health first
• Fix any issues before building
• Monitor critical library status

🐛 Troubleshooting

Cache Monitor Not Working

# Check if script is executable
chmod +x tools/scripts/cache-monitor.js

# Run with Node directly
node tools/scripts/cache-monitor.js --help

Permission Issues

# Fix cache directory permissions
chmod -R 755 .nx/cache

# Or reset completely
rm -rf .nx/cache
yarn nx reset

Disk Space Issues

# Check disk space
df -h

# Clean cache if needed
rm -rf .nx/cache
yarn nx reset

Build Hangs

# Check for stale locks
node tools/scripts/cache-monitor.js --report

# Clean locks
rm -rf .nx/cache/locks/*

# Reset cache
yarn nx reset

📈 Performance Optimization

Cache Size Management

• Monitor cache size regularly
• Clean cache when it exceeds 1GB
• Use --skip-nx-cache for one-off builds

Disk Space Monitoring

• Ensure at least 1GB free space
• Clean old build artifacts
• Monitor cache growth over time

Lock File Management

• Automatic cleanup of stale locks
• Monitor lock file age
• Prevent concurrent build conflicts

🔧 Configuration

Environment Variables

# Use alternative cache directory
NX_CACHE_DIRECTORY=/tmp/nx-cache

# Disable cache
NX_SKIP_CACHE=true

Custom Cache Locations

# Use custom cache directory
NX_CACHE_DIRECTORY=/custom/cache/path node tools/scripts/cache-monitor.js

📝 Best Practices

1. Regular Health Checks: Run cache monitor daily
2. Pre-Build Verification: Always check cache before major builds
3. Automatic Repair: Use --fix option for common issues
4. Monitor Critical Libraries: Ensure core libraries are built
5. Disk Space Management: Keep adequate free space
6. CI/CD Integration: Include cache health checks in pipelines

🤝 Contributing

To improve cache monitoring tools:

1. Add new health check criteria
2. Improve error detection algorithms
3. Add more detailed reporting
4. Enhance automatic repair capabilities
5. Add performance metrics

📄 License

These tools are part of the project and follow the same license terms.