dev-intel-v2/docs/prd-v3.md

Product Requirements Document: Dev Intel V3

1. Problem Statement

Dev Intel V2 successfully generates documentation from our Foxtrot monorepo, achieving a 93% agent eval and 78% human eval score. However, the pipeline relies on ~2000 lines of custom JavaScript. Much of this custom code duplicates the functionality of well-established Open Source Software (OSS). We need to simplify the architecture, reduce the maintenance burden, and embrace community-standard tools without sacrificing output quality. Our "ratchet loop" is functionally just a "Ralph Wiggum" loop, and we should embrace a simplified, brute-force bash loop with clear objective completion criteria rather than complex custom code.

2. Architecture

The V3 architecture adopts a hybrid approach: "OSS for the heavy lifting, custom code for the magic."

OSS Replacements

• Terraform Documentation: terraform-docs (Replaces extract-terraform.js)
• Helm Chart Documentation: helm-docs (Replaces extract-helm.js & sysdoc.js chart section)
• Evaluation Harness: promptfoo (Replaces eval-agent.js, eval-human.js, eval.js)
• Documentation Serving: mkdocs-material (Replaces custom doc serving)
• Ratchet Loop: Simple Ralph Wiggum bash loop (Replaces ratchet.js)

Retained Custom Components (The Value Add)

• Graph Builder (graph.js + extract.js): Tree-sitter extraction to build a unified knowledge graph across 13 repositories.
• Subsystem Aggregator (subsystem.js): Grouping files into logical subsystems and detecting cross-cutting concerns.
• Cross-Chart Interaction Analysis: Analyzing shared secrets, ports, and service references across Helm charts (which helm-docs cannot do natively).
• LLM Prose Enrichment (prose.js): Feeding the dependency matrix and anomaly flags into Claude to generate "why" explanations.
• Glue Layer: Minimal orchestration connecting OSS tools and custom analysis into unified output.

3. Requirements

• LLM Engine: Use http://192.168.86.11:8000/v1 with the claude-haiku-4.5 model.
• Scale: Must handle the Foxtrot monorepo (13 subdirectories, 17K+ files).
• Footprint constraint: The pipeline should be composed of ~500 lines of custom Node.js code plus config files.
• Speed constraint: Must run end-to-end in under 10 minutes (excluding LLM execution wait times).
• Cost constraint: Target cost is $1.00 per release.
• Code Implementation: Replace the existing terraform and per-chart Helm doc generation with the CLI tools (terraform-docs and helm-docs).
• Docs Website: Implement an mkdocs.yml configuration to serve the output as a searchable site.
• Evaluation Implementation: Configure promptfoo via YAML to act as the objective judge.

4. Ralph Wiggum Loop Spec

The previous ratchet.js implementation will be replaced by a bash script. This runs an AI agent in a simple, well-known ratchet pattern: loop until objective completion criteria are met.

Execution Flow:

1. Generate: Run the Dev Intel V3 pipeline.
2. Evaluate: Run promptfoo eval against the pipeline's output.
3. Diagnose: Check the promptfoo score against the required threshold.
4. Condition:
   • If Score >= Threshold: Success, exit the loop.
   • If Score < Threshold: Re-feed the previous output and failure context (the evaluation feedback) back into the generator prompt for context.
5. Repeat: Continue up to N iterations until criteria are met.

5. Success Metrics

• Quality Parity or Better: Agent eval score >= 93%, Human eval score >= 78%.
• Simplicity: Custom codebase shrinks from ~2000 lines to ~500 lines.
• Performance: Execution overhead is under 10 minutes.
• Efficiency: Pipeline inference costs remain <= $1 per release.

6. Migration Plan

To safely deprecate V2 while maintaining documentation pipelines:

1. Remove Custom Extractors: Delete extract-terraform.js, extract-helm.js, and the Helm-specific logic inside sysdoc.js.
2. Remove Custom Evaluators: Delete eval-agent.js, eval-human.js, and eval.js.
3. Remove Custom Ratchet: Delete ratchet.js.
4. Integrate CLI Binaries: Install and wire up terraform-docs and helm-docs.
5. Add Configs: Write promptfoo.yaml for evaluations and mkdocs.yml for serving docs.
6. Implement Bash Script: Write the Ralph Wiggum loop.
7. Re-wire Glue Code: Connect the outputs from the OSS tools into the preserved prose.js module.