dev-intel-v2/specs/bmad-agnostic-refactor-spec.md

BMad Spec: Dev-Intel V2 Repo-Agnostic Refactor

1. Problem Statement

The Dev-Intel V2 pipeline currently possesses a fatal flaw: it is severely overfit to the "Foxtrot" infrastructure monorepo. While the AST parsing (extract.js) and graph construction (graph.js, subsystem.js) are generic (~40% of the codebase), the deep extraction, synthesis, and evaluation layers (~60%) are entirely bespoke to Foxtrot's specific tech stack and naming conventions.

What breaks when pointing at a non-Foxtrot repo:

• Extraction (extract-deep.js, extract-patterns.js): Hardcodes regexes for vpc_cidr, product_id, ou_id, EKS addon block formats, AWS/GCP region names, and specific state services (elasticsearch, redis, cassandra). A non-infra repo (e.g., a frontend React app or a Java microservice) yields zero deep insights. LAYER_PATTERNS are hardcoded to app, compute, network.
• Synthesis (prose.js): The synthesizeReferencePages function hardcodes prompts expecting CIDR allocations, VPCs, and Jenkins jobs, and hardcodes the output files (network-architecture.md, operations.md, configuration.md, dependencies.md).
• Evaluation (eval-questions.js): Ground-truth questions are explicitly hardcoded to ask about mdm-app, cassandra, jenkins, vault-secret. Running the eval against any other repo results in a 0% score because the questions are invalid for that repo.

2. Architecture

The refactored pipeline shifts from a static, rule-based extraction/generation model to a dynamic, LLM-guided schema discovery model.

Pipeline Flow:

1. Generic Extraction (extract.js, extract-helm.js): Stays largely the same. Extracts ASTs, dependencies, and resources.
2. Semantic Profiling (repo-profiler.js - NEW): Before deep extraction, an LLM analyzes the graph and root configuration files (e.g., package.json, Chart.yaml, go.mod) to determine the repository's "Archetype" (e.g., Infrastructure, Frontend SPA, Backend Microservices, Data Pipeline).
3. Dynamic Deep Extraction (extract-dynamic.js - REPLACES extract-deep/patterns.js): Based on the archetype, generic heuristics and LLM prompts scan for archetype-specific configuration surfaces, state boundaries, and network contracts.
4. Adaptive Synthesis (prose.js): synthesizeReferencePages dynamically determines which reference pages to generate. It asks the LLM: "Given these extracted facts and this repo archetype, what are the 3-5 most critical reference topics?" It then generates those pages (e.g., ui-components.md for a frontend, instead of network-architecture.md).
5. Generative Evaluation (eval-generator.js - REPLACES eval-questions.js): The question bank is no longer hardcoded. An LLM agent generates valid, repo-specific Q&A pairs by reading the generated AST graph and code snippets, establishing a dynamic ground truth for the agent-browsing benchmark.

Module Boundaries:

• Extractor Layer: Purely deterministic AST/YAML/HCL parsing. No repo-specific logic.
• Context/Profile Layer: LLM-driven determination of what the repo is and what matters.
• Synthesis Layer: Transforms context into Divio-structured Markdown dynamically.
• Eval Layer: Independent subsystem that generates tests from the raw graph, then tests the agent against the synthesized docs.

3. Acceptance Criteria

1. No Hardcoded Values: Zero occurrences of Foxtrot-specific strings (vpc_cidr, elasticsearch, mdm-app, AWS regions) in pipeline source code.
2. Dynamic Outputs: sysdoc.js successfully generates a different set of reference markdown files depending on the repo (e.g., must not generate network-architecture.md for a pure frontend repo).
3. Repo-Agnostic Eval: Running eval-generator.js against an arbitrary open-source repo (e.g., expressjs/express or a generic Helm chart) produces \ge 20 valid, specific ground-truth questions.
4. Threshold Met: The pipeline runs on Foxtrot and achieves \ge 77% on the generated eval, AND runs on a test non-Foxtrot repo (e.g., BCE or AnyCloud) and achieves \ge 70% on its respective generated eval.
5. Resilience: Pipeline does not crash or throw unhandled exceptions when encountering unknown languages or missing configuration files.

4. Test-First Plan

Before changing the implementation, the following tests must be established:

1. Repo-Agnostic Eval Question Generation (Unit/Integration)

   • Test: Run eval-generator.js (to be written) against a mock "Microservice" repo graph and a mock "Infra" repo graph.
   • Assert: Verify that generated questions do not reference Foxtrot artifacts, and that the answers are strictly derived from the provided graph.

2. Synthesis Quality Tests (Unit)

   • Test: Pass a mock context (e.g., a React frontend archetype) to synthesizeReferencePages.
   • Assert: Verify the LLM determines appropriate page titles (e.g., components.md, state-management.md) and does not output infra-specific pages.

3. Pipeline Integration Tests (E2E)

   • Test: Execute wiggum-v2.sh against a tiny, non-Foxtrot fixture repository (e.g., a simple Node.js Express API).
   • Assert: Docs are generated without errors. The generated index maps to valid, generated reference files.

5. Implementation Plan

Step 1: Overhaul Evaluation (The Yardstick)

• Delete hardcoded questions in eval-questions.js.
• Write eval-generator.js that uses callLLM to generate ground truth questions from GraphStore and discoverCharts.
• Manually verify the generated questions for Foxtrot are high quality.

Step 2: Abstract Deep Extraction

• Deprecate extract-deep.js and extract-patterns.js.
• Create repo-profiler.js to establish the Repo Archetype.
• Create extract-dynamic.js that uses LLM prompts to extract state services, config surfaces, and architectural patterns generically based on the Archetype.

Step 3: Dynamic Synthesis

• Modify prose.js -> synthesizeReferencePages.
• Implement a two-pass LLM prompt:
  1. "What 4 reference pages should be created for this repo?" -> Returns JSON array of { title, filename, focus }.
  2. For each page, generate the markdown content using the extracted context.
• Update sysdoc.js to dynamically write these files instead of hardcoding the filenames.

Step 4: Script Cleanup

• Update wiggum-v2.sh to trigger eval-generator.js before running the agent benchmark.
• Remove any remaining bespoke scripts.

Step 5: Run & Tune

• Run the full loop on Foxtrot. Tune prompts until the score > 77%.
• Run the full loop on a secondary repo. Tune prompts until the score > 70%.

6. Risk Assessment

• LLM Quality Variance: Relying on the LLM to dynamically determine reference pages and extract facts increases token usage and latency. Mitigation: Use strong models (Sonnet/Opus) for schema/page definition, use Haiku for bulk prose generation. Implement heavy JSON-schema enforcement for extraction.
• Extraction Gaps for Non-Infra Repos: The current AST extractor may not capture enough semantic meaning for frontend/backend repos compared to Helm/TF, leading to thin docs. Mitigation: Ensure extract.js captures standard imports and package dependencies correctly to give the LLM enough context.
• Eval Score Regression: Foxtrot scores might drop because the eval questions are generated dynamically and might be harder or more ambiguous than the hardcoded ones. Mitigation: The eval-generator.js must instruct the LLM to generate highly specific, "exact match" or "list" type questions to prevent subjective scoring failures.