Generation Pipeline Optimization — Speed, Cost, Interactivity

Session: 257 | Date: 2026-03-23 | Status: Design (revised after spec review)

Problem Statement

Platform generation is slow (15 min for 5 modules), expensive ($2.50 per generation), and feels unresponsive — users watch repetitive phase messages with no visual feedback until everything completes.

Root causes:

1. Default output mode is VISUAL — generates JSON blueprints but no code files. The existing progressive preview code (S246) is dead because SSE events never carry files.
2. The implement.prompt.md system prompt (~10K tokens) is re-sent identically with every EXECUTE call — no prompt caching.
3. Visual path runs in the critical path — when set to BOTH, visual JSON and code generate as parallel LLM calls per module, doubling API calls.
4. All phases run strictly sequentially — PLAN must finish ALL modules before EXECUTE starts any.

What's already working (verified):

• Per-phase model routing is fully wired (S212). All 6 phases call getForPhase() and use AI_MODELS_CONFIG.moduleGeneration routing: ENHANCE/INTEGRATE → Haiku, ANALYZE/DECOMPOSE/EXECUTE → Sonnet, PLAN → Opus.
• Progressive preview code exists (S246) in studio-generation-listener.service.ts:275-304. It writes files to VFS immediately on module-gen.module-generated events — but is dead because VISUAL mode produces no files.

Changes (4 items)

A. CODE-First Output, Visual Deferred to Background Haiku

What: Change the default outputMode from VISUAL to CODE. After all code modules complete, fire background Haiku calls to generate visual JSON metadata (non-blocking, fire-and-forget with error logging).

Where:

• module-generation.controller.ts — Change default on all three VISUAL-defaulting endpoints: POST generate (line 102), GET start (line 191), POST generate-single (line 230). The POST fullstack endpoint already defaults to CODE.
• execute.phase.ts — After the main CODE generation loop completes, enqueue deferred visual generation. Implementation: Promise.allSettled() with individual try/catch per module, logged but not awaited.
• module-generation.types.ts — No new enum needed; just change the default. The existing CODE value is sufficient.
• Frontend audit: Grep all frontend callers for explicit outputMode: 'visual' — any hardcoded overrides must be updated.

Flow change:

BEFORE: Per module → [Visual LLM (Sonnet)] + [Code LLM (Sonnet)] in parallel
AFTER:  Per module → [Code LLM (Sonnet)] only in critical path
        After all complete → [Visual LLM (Haiku)] per module, fire-and-forget background

Impact:

• Removes visual from critical path — halves EXECUTE LLM calls (5 instead of 10)
• Activates existing progressive preview code (S246) — SSE events now carry files
• Visual uses Haiku ($1/$5) instead of Sonnet ($3/$15) — saves ~$0.10/generation
• Saves ~3-5 minutes wall time (no parallel visual calls blocking progress)

Risk: Low. Visual JSON arrives late for Studio workbench metadata panel. Acceptable — users watch the live preview, not the metadata panel.

B. Prompt Caching on EXECUTE System Prompts

What: Enable Anthropic prompt caching on the implement.prompt.md system prompt, which is identical for every EXECUTE call (only the user message changes per module).

Where:

• APIs/backend-vitto-api/src/app/llm/providers/claude.provider.ts — Refactor required: The system parameter is currently passed as a plain string (line 50: system: options.systemPrompt). Anthropic's cache_control requires content block format:

  // Before:
  system: options.systemPrompt
  // After:
  system: options.cacheSystemPrompt
    ? [{ type: 'text', text: options.systemPrompt, cache_control: { type: 'ephemeral' } }]
    : options.systemPrompt
  

• APIs/backend-vitto-api/src/app/llm/llm.types.ts — Add cacheSystemPrompt?: boolean to SendOptions.
• APIs/backend-vitto-api/src/app/module-generation/phases/execute.phase.ts — Pass cacheSystemPrompt: true in SendOptions for EXECUTE code generation calls.

How Anthropic prompt caching works:

• First call: full price, system prompt cached for 5 minutes (requires ≥1,024 tokens — our ~10K prompt qualifies)
• Subsequent calls within 5 min: 90% discount on cached prefix input tokens
• For 5 modules over ~10 minutes: modules 2-5 get cached pricing on the ~10K system prompt

Cost savings calculation:

• 5 modules × 10K system prompt = 50K total input tokens from system prompt
• Modules 2-5 cached: 4 × 10K × $3.00/M × 0.90 = $0.108 savings per generation

Risk: Low. Cache misses just fall back to normal pricing. The provider refactoring is localized to claude.provider.ts.

C. Pipeline Overlap: Stream Modules from PLAN → EXECUTE

What: Currently PLAN collects ALL modules into an execution plan, then EXECUTE starts. Instead, PLAN emits each module's plan as it resolves, and EXECUTE starts generating code immediately.

Where:

• plan.phase.ts — Return an AsyncIterable<ModuleGenPlanItem> (using existing ModuleGenPlanItem type from the executionPlan array) alongside the full ModuleGenPlanResult.
• execute.phase.ts — Accept streamed plan items, start processing as they arrive. Respects PARALLEL_MODULE_GEN_BATCH limit and dependency ordering.
• module-gen-phase.executor.ts — Orchestrate the streaming handoff. The executor is 900+ lines with checkpointing, cost tracking, and SSE emission — changes must preserve all of these.

Flow change:

BEFORE: PLAN (all modules) ──────────→ EXECUTE starts first module
AFTER:  PLAN(M1) → EXECUTE(M1) starts immediately
        PLAN(M2) → EXECUTE(M2) starts (parallel with M1, if independent)
        PLAN(M3) → queued until batch slot opens

Constraints:

• Can only stream modules whose dependencies are already planned (PLAN must respect dependency graph order)
• INTEGRATE phase still needs all modules — it runs after both PLAN and EXECUTE complete
• Checkpoints must be redesigned for partial completion states
• The completedModules array in execute.phase.ts (used for inter-module context) relies on sequential processing — streaming may cause early modules to miss context from later-planned dependencies

Impact: First module starts generating ~30-60s earlier. More noticeable for platforms with 5+ modules.

Risk: HIGH. This is the most complex change. The executor orchestration, checkpointing, and inter-module context all need redesign. Should be done last, possibly in a separate session.

D. Activate Progressive Preview (Already Built, Currently Dead)

What: Change A (CODE-first output) automatically activates the existing S246 progressive preview code. SSE events will now carry files, and handleModuleGenerated() will process them instead of returning early.

Additional work needed:

• bundler-filesystem.service.ts — synthesizeAppShell() must handle partial module sets:
  • Build routes from available modules only
  • Add a catch-all placeholder route for pending modules
  • Re-synthesize the shell each time a new module arrives
• Recompilation trigger: Verify how the VFS file writes from AiFileWriterService.receiveGeneratedFiles() trigger rspack recompilation. The BundlerService.compile() must be called explicitly after each module's files are written — there is no automatic signal chain. The component that holds the preview (likely PlatformPreviewPanelComponent) needs to listen for VFS changes and trigger recompile.

Impact: User sees the first module rendering in preview ~3 minutes after generation starts, instead of waiting 15+ minutes. Each subsequent module appears incrementally.

Risk: Low for basic functionality. App shell partial module handling is new but straightforward.

Revised Cost Model

5-module CRUD platform generation:

The primary value is timing and UX, not cost reduction.

Timing model:

Implementation Order

1. A. CODE-first output — Highest impact, unblocks D. Change defaults on 3 controller endpoints + frontend audit.
2. D. Progressive preview — Depends on A. Partial app shell synthesis + recompile trigger verification.
3. B. Prompt caching — Independent. Requires claude.provider.ts refactoring (string → content block format).
4. C. Pipeline overlap — Highest complexity, do last (possibly separate session). Requires executor/checkpoint redesign.

Files to Modify

E. Increase Parallel Batch Size (PARALLEL_MODULE_GEN_BATCH=5)

What: Change the parallel module generation batch size from 3 to 5. For a 5-module platform, this means all modules generate in a single batch instead of two sequential batches — cutting EXECUTE wall time by ~40%.

Where: .env.local — env var PARALLEL_MODULE_GEN_BATCH=5

Impact: For 5-module platforms, drops from 2 sequential batches to 1. Wall time improvement: ~4-6 minutes faster.

Risk: Higher concurrent API load. Anthropic rate limits may throttle at 5 parallel calls. Monitor for 429 errors. Can be tuned down to 4 if needed.

F. JSON Re-Prompt on Parse Failure

What: When extractCodeFiles() fails all 3 parsing strategies (parseLlmJson, brace-depth, individual object extraction), instead of returning empty, re-prompt the LLM with a strict JSON structure instruction. Adopted from gmodules1's explicit retry pattern.

Where: APIs/backend-vitto-api/src/app/module-generation/phases/execute.phase.ts — end of extractCodeFiles() method.

Flow:

LLM response → parseLlmJson → FAIL
            → brace-depth  → FAIL
            → individual objects → FAIL
            → RE-PROMPT: "Your response was not valid JSON. Return ONLY: { files: [...] }"
            → Parse retry response → SUCCESS or give up

Impact: Eliminates a class of generation failures where the LLM produces valid code wrapped in invalid JSON (markdown fences, explanatory text, truncated braces).

Risk: Low. Adds one extra LLM call only on failure (currently ~5% of generations). Cost is negligible.

G. CONFIRM Phase — User Gate Before EXECUTE

What: After PLAN completes, emit an SSE event with the execution plan and pause. The user sees: what modules will be generated, estimated cost, dependencies. They can approve, modify, or cancel before spending money on code generation.

Where:

• module-gen-phase.executor.ts — After PLAN, before EXECUTE, emit module-gen.confirm-plan SSE event and wait for confirmation.
• Frontend — Show a confirmation dialog/panel with the plan details and Approve/Cancel buttons.
• Controller — Accept a confirmation endpoint POST /module-generation/:sessionId/confirm.

Flow:

ENHANCE → ANALYZE → DECOMPOSE → PLAN
  → Emit SSE: module-gen.confirm-plan { modules, estimatedCost, dependencies }
  → PAUSE (wait for user confirmation or 5-min auto-approve timeout)
  → User approves → EXECUTE → INTEGRATE
  → User cancels → Emit module-gen.cancelled, clean up session

Impact: Saves ~$2 when user catches a bad plan early. Builds trust — users feel in control. Reduces wasted generations.

Risk: Medium. Adds complexity to session state management (paused state, confirmation endpoint, timeout handling). Frontend needs a new UI component.

What's NOT Changing

• Per-phase model routing — Already fully wired (S212), verified in all 6 phase files
• EXECUTE maxTokens stays at 32,768 — No artificial limits
• Validate/Fix loop stays at 2 attempts — Cheap insurance
• implement.prompt.md content unchanged — Auto-correction handles drift
• SSE event format unchanged — Frontend already handles module-gen.module-generated with files

Removed from Spec

Pipeline overlap (PLAN→EXECUTE streaming) — Removed after analysis. PLAN is a single LLM call that produces all modules at once — there's nothing to stream. Overlapping saves ~30s on a 12-minute pipeline (3% improvement) for HIGH complexity (executor/checkpoint redesign). Not worth it.

Future Investigation

Structured output / tool_use for code generation — Instead of prompting the LLM to return JSON and parsing with fallback strategies, use Claude's tool_use with a strict JSON schema that enforces { files: [{ path, content, language }] }. This would eliminate parsing failures entirely. Needs investigation to verify Claude handles 32K+ token tool responses.