{
  "evalType": "agent",
  "timestamp": "2026-03-10T06:13:02.598Z",
  "docsDir": "./foxtrot-docs",
  "totalQuestions": 32,
  "overallScore": "28.7",
  "avgAccuracy": "0.44",
  "avgCompleteness": "0.50",
  "avgPrecision": "4.34",
  "avgNavigation": "0.47",
  "notFoundCount": 25,
  "notFoundRate": "78.1%",
  "byCategory": {
    "architecture": {
      "avg": "41.7",
      "count": 6
    },
    "dependencies": {
      "avg": "25.0",
      "count": 5
    },
    "operations": {
      "avg": "25.0",
      "count": 6
    },
    "configuration": {
      "avg": "30.0",
      "count": 6
    },
    "services": {
      "avg": "25.0",
      "count": 5
    },
    "contracts": {
      "avg": "22.5",
      "count": 4
    }
  },
  "byDifficulty": {
    "easy": {
      "avg": "39.0",
      "count": 10
    },
    "medium": {
      "avg": "24.4",
      "count": 17
    },
    "hard": {
      "avg": "23.0",
      "count": 5
    }
  },
  "results": [
    {
      "id": "arch-layered-order",
      "category": "architecture",
      "difficulty": "easy",
      "question": "What are the five layers in Foxtrot's layered architecture, listed from top to bottom?",
      "groundTruth": "Application Layer (app-*), Compute Layer (compute-*), Network Layer (network-*), Account Layer (account-*), Control Plane (control-core).",
      "llmAnswer": "Based on the system-architecture.md file, the five layers in Foxtrot's layered architecture from top to bottom are:\n\n1. Runtime (shared)\n2. Application\n3. Compute\n4. Network\n5. IPAM",
      "filesRead": [
        "reference/system-architecture.md",
        "reference/subsystems/root.md",
        "diagrams/system-deps.mmd",
        "agent-kb.json",
        "reference/helm/index.md"
      ],
      "plannedFiles": [
        "reference/system-architecture.md",
        "reference/subsystems/root.md",
        "diagrams/system-deps.mmd",
        "agent-kb.json",
        "reference/helm/index.md"
      ],
      "score": {
        "accuracy": 1,
        "completeness": 2,
        "precision": 1,
        "navigation": 2,
        "notes": "AI correctly identified Application, Compute, and Network layers but missed Account Layer and Control Plane entirely. Introduced hallucinated layers (Runtime, IPAM) not in ground truth. Only 3 of 5 correct items present, with 2 false positives. Agent cited a source file but retrieved incorrect information, suggesting either wrong file access or misinterpretation of content."
      },
      "composite": 30,
      "notFound": false
    },
    {
      "id": "arch-hub-spoke-ownership",
      "category": "architecture",
      "difficulty": "easy",
      "question": "Which ArgoCD instance owns the account, network, and compute layers, and which owns the application layer?",
      "groundTruth": "Hub ArgoCD owns account, network, and compute layers (infrastructure and control-plane appsets). Spoke ArgoCD owns the application layer (app-runtime appsets only).",
      "llmAnswer": "Based on the Deployment Topology section in the system-architecture.md file:\n\n**Hub (Infrastructure/Control Plane) ArgoCD instance owns:**\n- Account layer (via `account-creation` ApplicationSet)\n- Network layer (via `aws-vpcs`, `gcp-vpcs`, `azure-vpcs` ApplicationSets)\n- Compute layer (via `aws-argocd-feature-clusters`, `gcp-argocd-feature-clusters`, `azure-argocd-feature-clusters` ApplicationSets)\n\n**Spoke (Applications/Runtime) ArgoCD instance(s) own:**\n- Application layer\n\nThe Hub manages infrastructure and control plane resources across multiple cloud providers (AWS, GCP, Azure), while Spoke instances manage application deployments and runtime resources.",
      "filesRead": [
        "reference/system-architecture.md",
        "reference/subsystems/control-core.md",
        "reference/helm/index.md",
        "diagrams/helm-interactions.mmd",
        "reference/contracts/index.md"
      ],
      "plannedFiles": [
        "reference/system-architecture.md",
        "reference/subsystems/control-core.md",
        "reference/helm/index.md",
        "diagrams/helm-interactions.mmd",
        "reference/contracts/index.md"
      ],
      "score": {
        "accuracy": 5,
        "completeness": 5,
        "precision": 4,
        "navigation": 5,
        "notes": "Answer correctly identifies Hub ownership of account/network/compute layers and Spoke ownership of application layer. AI demonstrated strong navigation by citing system-architecture.md and Deployment Topology section with specific ApplicationSet references. Minor precision deduction: didn't explicitly emphasize that Spoke owns 'app-runtime appsets only' as stated in ground truth, though the core facts are accurate."
      },
      "composite": 95,
      "notFound": false
    },
    {
      "id": "arch-aws-regions",
      "category": "architecture",
      "difficulty": "easy",
      "question": "What AWS regions does Foxtrot support?",
      "groundTruth": "us-east-1, us-west-2, eu-west-1, eu-central-1.",
      "llmAnswer": "Based on the system architecture documentation, Foxtrot supports the following AWS regions:\n\n- ap-southeast-1\n- eu-central-1\n- eu-west-1\n- eu-west-2\n- eu-west-3\n- us-east-1\n- us-east-2\n- us-west-1\n- us-west-2",
      "filesRead": [
        "reference/helm/charts/compute-common-templates-foxtrot-eks.md",
        "reference/subsystems/compute-common.md",
        "reference/system-architecture.md",
        "reference/helm/index.md",
        "reference/subsystems/runtime.md"
      ],
      "plannedFiles": [
        "reference/helm/charts/compute-common-templates-foxtrot-eks.md",
        "reference/subsystems/compute-common.md",
        "reference/system-architecture.md",
        "reference/helm/index.md",
        "reference/subsystems/runtime.md"
      ],
      "score": {
        "accuracy": 2,
        "completeness": 5,
        "precision": 1,
        "navigation": 2,
        "notes": "AI answer includes all 4 ground truth regions but adds 5 unsupported regions (ap-southeast-1, eu-west-2, eu-west-3, us-east-2, us-west-1), indicating hallucination or unreliable source verification. While completeness is achieved, the answer contains significant false positives that undermine accuracy and precision."
      },
      "composite": 50,
      "notFound": false
    },
    {
      "id": "arch-gcp-shared-vpc-host",
      "category": "architecture",
      "difficulty": "medium",
      "question": "What is the default GCP host project used for Shared VPC in network-common, and what is the host network path it attaches to?",
      "groundTruth": "The default host project is `network-services-436015`. When `sharedVpc.enabled: true`, service project resources attach to the host network path `projects/HOST_PROJECT/global/networks/gcp-core-network`.",
      "llmAnswer": "NOT_FOUND\n\nThe provided documentation describes the foxtrot-gcp-vpc chart structure and its resources (including SharedVPCServiceProject), but does not contain the actual values.yaml configuration or specify the default GCP host project or host network path used for Shared VPC attachment.",
      "filesRead": [
        "reference/subsystems/network-common.md",
        "reference/helm/charts/network-common-charts-foxtrot-gcp-vpc.md",
        "reference/system-architecture.md",
        "reference/helm/index.md",
        "diagrams/network-common-contracts.mmd"
      ],
      "plannedFiles": [
        "reference/subsystems/network-common.md",
        "reference/helm/charts/network-common-charts-foxtrot-gcp-vpc.md",
        "reference/system-architecture.md",
        "reference/helm/index.md",
        "diagrams/network-common-contracts.mmd"
      ],
      "score": {
        "accuracy": 0,
        "completeness": 0,
        "precision": 5,
        "navigation": 0,
        "notes": "AI provided NOT_FOUND response without locating the required configuration values. The answer contains no factual content about the host project (network-services-436015) or network path (projects/HOST_PROJECT/global/networks/gcp-core-network). While the response avoids hallucination, it demonstrates complete failure to navigate to the source configuration file containing these defaults."
      },
      "composite": 25,
      "notFound": true
    },
    {
      "id": "arch-cidr-employee-access",
      "category": "architecture",
      "difficulty": "medium",
      "question": "What is the CIDR range for the employee access (bastions) segment on AWS and on GCP?",
      "groundTruth": "AWS: 10.208.128.0/24. GCP: 10.128.128.0/24.",
      "llmAnswer": "NOT_FOUND\n\nThe documentation provided does not contain explicit CIDR range allocations labeled as \"employee access\" or \"bastions\" segments. While the network-core subsystem file list references `employee-access-vpc1` configurations for AWS (in `us-east-1`), the Network CIDR Allocations table does not map specific CIDR blocks to employee access or bastion segments for either AWS or GCP.",
      "filesRead": [
        "reference/system-architecture.md",
        "reference/helm/charts/network-common-charts-foxtrot-aws-vpc.md",
        "reference/helm/charts/network-common-charts-foxtrot-gcp-vpc.md",
        "reference/subsystems/network-core.md",
        "reference/helm/index.md"
      ],
      "plannedFiles": [
        "reference/system-architecture.md",
        "reference/helm/charts/network-common-charts-foxtrot-aws-vpc.md",
        "reference/helm/charts/network-common-charts-foxtrot-gcp-vpc.md",
        "reference/subsystems/network-core.md",
        "reference/helm/index.md"
      ],
      "score": {
        "accuracy": 0,
        "completeness": 0,
        "precision": 5,
        "navigation": 0,
        "notes": "AI answered NOT_FOUND when ground truth specifies AWS: 10.208.128.0/24 and GCP: 10.128.128.0/24. The agent failed to locate or retrieve the required CIDR ranges for either cloud provider. While the response avoided hallucination and honestly acknowledged missing information, it provided zero correct facts and zero coverage of the ground truth requirements."
      },
      "composite": 25,
      "notFound": true
    },
    {
      "id": "arch-production-cidr",
      "category": "architecture",
      "difficulty": "medium",
      "question": "What is the CIDR range for production workloads on AWS and on GCP?",
      "groundTruth": "AWS: 10.192.0.0/10. GCP: 10.128.0.0/10.",
      "llmAnswer": "NOT_FOUND\n\nThe provided documentation lists various CIDR allocations in the \"Network CIDR Allocations\" table, but it does not explicitly designate which CIDR ranges are allocated for production workloads on AWS or GCP. While some entries reference \"hub-prd\" (GCP production hub) with CIDR blocks `10.128.192.0/23`, `192.168.32.0/20`, and `192.168.48.0/20`, there is no clear documentation specifying the production workload CIDR ranges for either cloud platform.",
      "filesRead": [
        "reference/subsystems/network-core.md",
        "reference/helm/charts/network-common-charts-foxtrot-aws-vpc.md",
        "reference/helm/charts/network-common-charts-foxtrot-gcp-vpc.md",
        "reference/subsystems/network-common.md",
        "reference/system-architecture.md"
      ],
      "plannedFiles": [
        "reference/subsystems/network-core.md",
        "reference/helm/charts/network-common-charts-foxtrot-aws-vpc.md",
        "reference/helm/charts/network-common-charts-foxtrot-gcp-vpc.md",
        "reference/subsystems/network-common.md",
        "reference/system-architecture.md"
      ],
      "score": {
        "accuracy": 0,
        "completeness": 0,
        "precision": 5,
        "navigation": 0,
        "notes": "AI answered NOT_FOUND. Per evaluation rules, this receives ACCURACY=0, COMPLETENESS=0, PRECISION=5, NAVIGATION=0. The agent failed to locate the ground truth CIDR ranges (AWS: 10.192.0.0/10, GCP: 10.128.0.0/10) despite finding tangentially related production hub information."
      },
      "composite": 25,
      "notFound": true
    },
    {
      "id": "dep-runtime-common-horizontal",
      "category": "dependencies",
      "difficulty": "easy",
      "question": "Which runtime repositories consume charts from which common repositories?",
      "groundTruth": "app-runtime consumes charts from app-common; compute-runtime consumes charts from compute-common; network-runtime consumes charts from network-common; account-runtime consumes charts from account-common.",
      "llmAnswer": "Based on the helm-interactions diagram in the provided documentation:\n\n**Runtime repositories consuming charts from common repositories:**\n\n- `runtime/compute-runtime/gcp/non-argo-tracked-resources/compute-na07/us-west3/prd-gke-us-west3-05` (chart: `prod-gke-us-west3-compute1`) **depends on** `compute-common/templates/foxtrot-gke` (chart: `foxtrot-gke`)\n\nThis is the only explicit chart dependency relationship documented between runtime and common repositories in the provided files.",
      "filesRead": [
        "reference/subsystems/runtime.md",
        "reference/helm/index.md",
        "reference/system-architecture.md",
        "diagrams/helm-interactions.mmd",
        "reference/contracts/index.md"
      ],
      "plannedFiles": [
        "reference/subsystems/runtime.md",
        "reference/helm/index.md",
        "reference/system-architecture.md",
        "diagrams/helm-interactions.mmd",
        "reference/contracts/index.md"
      ],
      "score": {
        "accuracy": 2,
        "completeness": 1,
        "precision": 2,
        "navigation": 2,
        "notes": "AI identified only 1 of 4 runtime-to-common repository relationships (compute-runtime → compute-common). Missing app-runtime, network-runtime, and account-runtime entirely. Made an incorrect claim that the identified dependency was 'the only explicit chart dependency relationship documented,' suggesting incomplete exploration of available information. Answer is overly granular with specific file paths rather than repository-level relationships requested."
      },
      "composite": 35,
      "notFound": false
    },
    {
      "id": "dep-vertical-layers",
      "category": "dependencies",
      "difficulty": "easy",
      "question": "What are the vertical layer dependencies in Foxtrot's architecture?",
      "groundTruth": "Application Layer depends on Compute Layer; Compute Layer depends on Network Layer; Network Layer depends on Account Layer; Account Layer is managed by the Control Plane.",
      "llmAnswer": "NOT_FOUND\n\nThe provided documentation does not contain information about \"Foxtrot's architecture\" or its vertical layer dependencies. While there is a reference to a \"foxtrot-eks\" Helm chart in the compute-common subsystem, there is no detailed description of Foxtrot as a distinct architectural system or its layer dependencies in the available files.",
      "filesRead": [
        "reference/system-architecture.md",
        "reference/subsystems/root.md",
        "diagrams/system-deps.mmd",
        "reference/subsystems/control-core.md",
        "reference/subsystems/runtime.md"
      ],
      "plannedFiles": [
        "reference/system-architecture.md",
        "reference/subsystems/root.md",
        "diagrams/system-deps.mmd",
        "reference/subsystems/control-core.md",
        "reference/subsystems/runtime.md"
      ],
      "score": {
        "accuracy": 0,
        "completeness": 0,
        "precision": 5,
        "navigation": 0,
        "notes": "AI agent answered NOT_FOUND, indicating the required information about Foxtrot's vertical layer dependencies was not located in available documentation. Per evaluation criteria, NOT_FOUND responses receive zero scores for accuracy, completeness, and navigation, but full precision credit for not hallucinating information."
      },
      "composite": 25,
      "notFound": true
    },
    {
      "id": "dep-create-account-repos",
      "category": "dependencies",
      "difficulty": "medium",
      "question": "How many PRs does `create-account` produce and which repositories does it touch?",
      "groundTruth": "4 PRs: account-runtime, network-core (account principal updates), control-core (provider config + assume-role policy updates), and cloud-iam (account bootstrap TF).",
      "llmAnswer": "NOT_FOUND\n\nThe provided documentation does not contain information about `create-account`, how many PRs it produces, or which repositories it touches. The documentation references an `account-creation` Helm chart within the `account-common` subsystem, but provides no details about PR generation or repository interactions.",
      "filesRead": [
        "reference/helm/charts/account-common-charts-account-creation.md",
        "reference/contracts/index.md",
        "explanation/change-impact.md",
        "reference/subsystems/account-common.md",
        "reference/system-architecture.md"
      ],
      "plannedFiles": [
        "reference/helm/charts/account-common-charts-account-creation.md",
        "reference/contracts/index.md",
        "explanation/change-impact.md",
        "reference/subsystems/account-common.md",
        "reference/system-architecture.md"
      ],
      "score": {
        "accuracy": 0,
        "completeness": 0,
        "precision": 5,
        "navigation": 0,
        "notes": "AI failed to locate information about create-account. While the response demonstrates honesty by not hallucinating details, it provides zero correct facts about the 4 PRs or affected repositories. The agent did not navigate to the correct documentation source containing create-account workflow details."
      },
      "composite": 25,
      "notFound": true
    },
    {
      "id": "dep-create-cluster-repos",
      "category": "dependencies",
      "difficulty": "medium",
      "question": "Which repositories does `create-cluster` commit to, and how many PRs does it typically produce?",
      "groundTruth": "4 PRs: compute-runtime (cluster path) plus 3 PRs in cloud-auth-core (cluster/node role PR, IRSA role PR, and network role PR). It also calls cluster-netbox-allocate.py to reserve/allocate cluster subnet CIDRs.",
      "llmAnswer": "NOT_FOUND\n\nThe provided documentation does not contain information about which repositories the `create-cluster` script commits to or how many PRs it typically produces. The documentation only references `create-cluster.sh` in the context of naming conventions for AWS and Azure resources.",
      "filesRead": [
        "agent-kb.json",
        "reference/system-architecture.md",
        "reference/contracts/index.md",
        "reference/helm/index.md",
        "explanation/change-impact.md"
      ],
      "plannedFiles": [
        "agent-kb.json",
        "reference/system-architecture.md",
        "reference/contracts/index.md",
        "reference/helm/index.md",
        "explanation/change-impact.md"
      ],
      "score": {
        "accuracy": 0,
        "completeness": 0,
        "precision": 5,
        "navigation": 0,
        "notes": "AI failed to locate information about create-cluster's repository targets and PR generation pattern. Ground truth specifies 4 PRs across compute-runtime and cloud-auth-core with specific role-based PRs, plus netbox allocation integration. Agent only found tangential references to create-cluster.sh in naming conventions, indicating poor navigation to relevant documentation. No hallucination occurred, but complete information retrieval failure."
      },
      "composite": 25,
      "notFound": true
    },
    {
      "id": "dep-compute-common-deps",
      "category": "dependencies",
      "difficulty": "medium",
      "question": "What are the core dependencies of compute-common for cluster provisioning?",
      "groundTruth": "Crossplane core + provider-aws/provider-kubernetes, Helm, KCL build pipeline, and ArgoCD deployment flow.",
      "llmAnswer": "\n\nBased on the documentation provided:\n\n**Core Dependencies of compute-common for cluster provisioning:**\n\n1. **root** (cross-cutting) — sole upstream dependency with 29 imports and 1 call\n\nThis is the only upstream dependency. Root provides the shared primitive layer (base configuration and environment resolution) that keeps cluster-agnostic test logic decoupled from cloud-specific provisioning details.\n\n**Internal Structure:**\n- 14 functions (overwhelmingly test entrypoints)\n- 64 files (primarily declarative artifacts: Helm chart templates, values files, Kubernetes manifests)\n- Exports: `helm_helper`, `kubectl_helper`, `cleanup_tracker`, and lifecycle test functions for AKS, EKS, and GKE\n\n**Downstream Consumers:**\nWhile not explicitly listed in the compute-common section, the system architecture indicates compute-common is consumed by subsystems that require cluster lifecycle validation (DNS resolution, workload deployment, Helm rendering) across cloud providers.",
      "filesRead": [
        "reference/subsystems/compute-common.md",
        "diagrams/system-deps.mmd",
        "diagrams/compute-common-contracts.mmd",
        "reference/system-architecture.md",
        "reference/helm/index.md"
      ],
      "plannedFiles": [
        "reference/subsystems/compute-common.md",
        "diagrams/system-deps.mmd",
        "diagrams/compute-common-contracts.mmd",
        "reference/system-architecture.md",
        "reference/helm/index.md"
      ],
      "score": {
        "accuracy": 1,
        "completeness": 0,
        "precision": 1,
        "navigation": 1,
        "notes": "AI answer identifies 'root' as sole upstream dependency and describes internal test structure, but completely misses the actual core dependencies: Crossplane core, provider-aws/provider-kubernetes, Helm (as infrastructure dependency, not just export), KCL build pipeline, and ArgoCD. The answer addresses internal architecture rather than external provisioning dependencies. Navigation failed to locate the correct dependency information."
      },
      "composite": 15,
      "notFound": false
    },
    {
      "id": "ops-argocd-deployment-flow",
      "category": "operations",
      "difficulty": "medium",
      "question": "What are the steps in the ArgoCD deployment flow for an application, from ApplicationSet watching app-runtime to pods starting?",
      "groundTruth": "1. ApplicationSet watches app-runtime. 2. Reads argo-gen-params.yaml per environment. 3. Generates Application. 4. Pulls chart from OCI registry. 5. Merges values and overrides. 6. Renders manifests. 7. Applies to cluster. 8. ESO fetches secrets. 9. Pods start with injected configs.",
      "llmAnswer": "NOT_FOUND\n\nThe provided documentation describes the ArgoCD-apps Helm chart structure, resources generated, and Helm sync waves for bootstrapping order, but does not contain a detailed step-by-step explanation of the deployment flow from ApplicationSet watching app-runtime through to pods starting. The documentation shows that ApplicationSets are templated and generated, and lists sync wave ordering, but lacks the specific operational flow details you're asking about.",
      "filesRead": [
        "reference/helm/charts/app-common-charts-argocd-apps.md",
        "reference/subsystems/app-common.md",
        "reference/system-architecture.md",
        "reference/helm/index.md",
        "diagrams/app-common-contracts.mmd"
      ],
      "plannedFiles": [
        "reference/helm/charts/app-common-charts-argocd-apps.md",
        "reference/subsystems/app-common.md",
        "reference/system-architecture.md",
        "reference/helm/index.md",
        "diagrams/app-common-contracts.mmd"
      ],
      "score": {
        "accuracy": 0,
        "completeness": 0,
        "precision": 5,
        "navigation": 0,
        "notes": "AI correctly identified that the documentation lacks the specific deployment flow details but provided no steps from ground truth. Honest about limitations (precision=5) but failed to locate or provide any of the 9 required steps (accuracy/completeness/navigation=0)."
      },
      "composite": 25,
      "notFound": true
    },
    {
      "id": "ops-ebf-release-pattern",
      "category": "operations",
      "difficulty": "medium",
      "question": "What are the steps in the EBF/weekly release pattern?",
      "groundTruth": "1. Create release branch from develop. 2. Promote versions for release/hotfix environments. 3. Run stage pipelines (preview, preprod, production). 4. Merge promoted changes to main. 5. Reconcile hotfix and develop branches.",
      "llmAnswer": "NOT_FOUND\n\nThe provided documentation files do not contain information about EBF (Emergency Bug Fix) or weekly release patterns.",
      "filesRead": [
        "reference/system-architecture.md",
        "agent-kb.json",
        "reference/subsystems/control-core.md",
        "reference/subsystems/runtime.md",
        "explanation/change-impact.md"
      ],
      "plannedFiles": [
        "reference/system-architecture.md",
        "agent-kb.json",
        "reference/subsystems/control-core.md",
        "reference/subsystems/runtime.md",
        "explanation/change-impact.md"
      ],
      "score": {
        "accuracy": 0,
        "completeness": 0,
        "precision": 5,
        "navigation": 0,
        "notes": "AI agent returned NOT_FOUND, indicating it could not locate EBF/weekly release pattern documentation. Per evaluation guidelines, this results in zero scores for accuracy, completeness, and navigation. Precision scores full marks because the agent made no false claims—it simply reported unavailability rather than hallucinating information."
      },
      "composite": 25,
      "notFound": true
    },
    {
      "id": "ops-rollback-procedure",
      "category": "operations",
      "difficulty": "medium",
      "question": "What is the preferred rollback procedure in Foxtrot, and when are script-level deletes used instead?",
      "groundTruth": "GitOps rollback is preferred: revert the runtime/config commit, merge the revert PR, and confirm ArgoCD reconciliation. Script-level deletes (delete-vpc.sh, delete-cluster.sh, delete-app.sh) are used only with explicit approval and change control.",
      "llmAnswer": "NOT_FOUND\n\nThe provided documentation files do not contain information about preferred rollback procedures in Foxtrot or when script-level deletes are used.",
      "filesRead": [
        "reference/system-architecture.md",
        "explanation/change-impact.md",
        "reference/subsystems/compute-common.md",
        "reference/helm/index.md",
        "reference/subsystems/control-core.md"
      ],
      "plannedFiles": [
        "reference/system-architecture.md",
        "explanation/change-impact.md",
        "reference/subsystems/compute-common.md",
        "reference/helm/index.md",
        "reference/subsystems/control-core.md"
      ],
      "score": {
        "accuracy": 0,
        "completeness": 0,
        "precision": 5,
        "navigation": 0,
        "notes": "AI correctly reported information unavailable rather than hallucinating, but failed to locate documentation containing GitOps rollback procedures and script-level delete policies. The agent did not demonstrate ability to navigate to relevant Foxtrot documentation."
      },
      "composite": 25,
      "notFound": true
    },
    {
      "id": "ops-branch-cluster-mapping",
      "category": "operations",
      "difficulty": "easy",
      "question": "What is the branch-to-cluster mapping in app-runtime? Which branch targets which environments?",
      "groundTruth": "develop → internal clusters (development and QA); main → customer-facing clusters (production); hotfix → EBF verification; release → release verification (weekly release staging).",
      "llmAnswer": "NOT_FOUND\n\nThe provided documentation does not contain explicit branch-to-cluster mappings for app-runtime. While the runtime.md file lists numerous environment-specific paths and cluster names (such as `aws-prd-ap-1-eks`, `aws-dev-us-99-eks`, `ap03-rdm-prod`, `ap03-rdm-test`), it does not document which Git branches target which of these environments or clusters.",
      "filesRead": [
        "reference/subsystems/runtime.md",
        "reference/system-architecture.md",
        "reference/helm/index.md",
        "agent-kb.json",
        "reference/subsystems/app-common.md"
      ],
      "plannedFiles": [
        "reference/subsystems/runtime.md",
        "reference/system-architecture.md",
        "reference/helm/index.md",
        "agent-kb.json",
        "reference/subsystems/app-common.md"
      ],
      "score": {
        "accuracy": 0,
        "completeness": 0,
        "precision": 5,
        "navigation": 0,
        "notes": "AI returned NOT_FOUND despite ground truth existing. Failed to locate branch-to-cluster mappings (develop/main/hotfix/release targeting specific environments). Did not hallucinate false mappings, but completely missed the required information. Navigation score reflects inability to find the correct documentation section containing these mappings."
      },
      "composite": 25,
      "notFound": true
    },
    {
      "id": "ops-jenkins-jobs",
      "category": "operations",
      "difficulty": "hard",
      "question": "What are the six Jenkins jobs in the Foxtrot pipeline and what does each do?",
      "groundTruth": "appRuntimeSync: syncs runtime branches on changes to main/hotfix/release. foxtrotSync: updates imagetags-override.yaml with new image tags. foxtrotPromote: promotes and merges release outputs. chartSync: updates argo-gen-params.yaml with chart versions. chartCheckInAppCommon: detects chart version mismatches and triggers chartSync. chartPromoteVersion: promotes chart versions per release stage.",
      "llmAnswer": "NOT_FOUND\n\nThe provided documentation does not contain information about six Jenkins jobs in the Foxtrot pipeline or their individual purposes. The Jenkins chart documentation describes Jenkins as a deployable application but does not detail specific pipeline jobs.",
      "filesRead": [
        "reference/helm/charts/app-common-charts-jenkins.md",
        "reference/helm/charts/compute-common-templates-foxtrot-eks.md",
        "reference/subsystems/control-core.md",
        "reference/system-architecture.md",
        "reference/helm/index.md"
      ],
      "plannedFiles": [
        "reference/helm/charts/app-common-charts-jenkins.md",
        "reference/helm/charts/compute-common-templates-foxtrot-eks.md",
        "reference/subsystems/control-core.md",
        "reference/system-architecture.md",
        "reference/helm/index.md"
      ],
      "score": {
        "accuracy": 0,
        "completeness": 0,
        "precision": 5,
        "navigation": 0,
        "notes": "AI correctly identified that the information was not found in available documentation rather than hallucinating answers. However, it failed to locate the ground truth information about the six Foxtrot pipeline jobs and their purposes. The response demonstrates honest limitations but zero success in information retrieval and delivery."
      },
      "composite": 25,
      "notFound": true
    },
    {
      "id": "ops-create-cluster-timeout",
      "category": "operations",
      "difficulty": "medium",
      "question": "What is the timeout for waiting for a cluster to reach ready condition in the create-cluster workflow?",
      "groundTruth": "3600 seconds (1 hour), with a poll interval of 30 seconds.",
      "llmAnswer": "NOT_FOUND\n\nThe provided documentation files do not contain information about a timeout for waiting for a cluster to reach ready condition in a create-cluster workflow.",
      "filesRead": [
        "reference/helm/charts/app-common-charts-cluster.md",
        "reference/subsystems/control-core.md",
        "reference/helm/index.md",
        "reference/subsystems/compute-common.md",
        "reference/subsystems/runtime.md"
      ],
      "plannedFiles": [
        "reference/helm/charts/app-common-charts-cluster.md",
        "reference/subsystems/control-core.md",
        "reference/helm/index.md",
        "reference/subsystems/compute-common.md",
        "reference/subsystems/runtime.md"
      ],
      "score": {
        "accuracy": 0,
        "completeness": 0,
        "precision": 5,
        "navigation": 0,
        "notes": "AI agent failed to locate the timeout configuration (3600 seconds with 30-second poll interval) that exists in the documentation. The NOT_FOUND response is honest but indicates the agent did not successfully navigate to or search the relevant workflow documentation files."
      },
      "composite": 25,
      "notFound": true
    },
    {
      "id": "config-cloud-resource-naming",
      "category": "configuration",
      "difficulty": "easy",
      "question": "What is the canonical format for cloud resource names in Foxtrot, and what does each component represent?",
      "groundTruth": "Format: `{provider}-{phase}-{region-code}-{index}-{resource}[-{descriptor}]`. provider: aws/gcp/azr; phase: prd/iqa/dev; region-code: two-letter derived code; index: incremental integer; resource: vpc/eks/gke/aks/lb/rds. Example: `gcp-prd-us-1-vpc-blue`.",
      "llmAnswer": "\n\nBased on the system-architecture.md documentation, the canonical naming conventions for Foxtrot cloud resources are:\n\n**AWS:**\n- Clusters: `aws-{phase}-{region-code}-{index}-eks`\n- VPCs: `aws-{phase}-{region-code}-{index}-vpc`\n\n**Azure:**\n- Clusters: `azr-{phase}-{region-code}-{index}-aks`\n- VNets: `azr-{phase}-{region-code}-{index}-vnet`\n\n**Component breakdown:**\n- **Cloud provider prefix**: `aws` or `azr`\n- **Phase**: Environment phase (e.g., dev, prd, iqa)\n- **Region code**: Cloud region identifier\n- **Index**: Numeric identifier for resource ordering\n- **Resource type**: `eks`, `aks`, `vpc`, or `vnet`",
      "filesRead": [
        "reference/system-architecture.md",
        "reference/contracts/index.md",
        "reference/helm/index.md",
        "reference/subsystems/root.md",
        "reference/subsystems/compute-common.md"
      ],
      "plannedFiles": [
        "reference/system-architecture.md",
        "reference/contracts/index.md",
        "reference/helm/index.md",
        "reference/subsystems/root.md",
        "reference/subsystems/compute-common.md"
      ],
      "score": {
        "accuracy": 3,
        "completeness": 2,
        "precision": 4,
        "navigation": 2,
        "notes": "AI correctly identifies the format structure and components for AWS/Azure but completely omits GCP provider (gcp/gke). Missing optional descriptor component shown in ground truth example. Resource types incomplete—only shows eks/aks/vpc/vnet, omitting gke, lb, rds. Found documentation but failed to extract canonical format comprehensively."
      },
      "composite": 55,
      "notFound": false
    },
    {
      "id": "config-region-code-algorithm",
      "category": "configuration",
      "difficulty": "medium",
      "question": "What is the algorithm for deriving a two-letter region code from a cloud region name, and what is the region code for `westeurope`?",
      "groundTruth": "Algorithm: 1. Remove directional words (east, west, north, south). 2. Remove separators. 3. Take first two characters. For `westeurope`: remove 'west' → 'europe' → 'eu'.",
      "llmAnswer": "NOT_FOUND\n\nThe provided documentation references region codes in naming conventions (e.g., `aws-{phase}-{region-code}-{index}-eks`) but does not contain the algorithm for deriving two-letter region codes from cloud region names, nor does it specify the region code for `westeurope`. The actual implementation appears to be in scripts like `compute-tools/bash-spoke-setup/scripts/create-cluster.sh` and `create-vpc.sh`, which are not included in the provided reference files.",
      "filesRead": [
        "reference/contracts/index.md",
        "reference/system-architecture.md",
        "reference/subsystems/compute-common.md",
        "reference/helm/index.md",
        "reference/subsystems/network-common.md"
      ],
      "plannedFiles": [
        "reference/contracts/index.md",
        "reference/system-architecture.md",
        "reference/subsystems/compute-common.md",
        "reference/helm/index.md",
        "reference/subsystems/network-common.md"
      ],
      "score": {
        "accuracy": 0,
        "completeness": 0,
        "precision": 5,
        "navigation": 0,
        "notes": "AI correctly identified information was not available in provided documentation rather than hallucinating. However, this resulted in zero factual content matching ground truth. The required implementation details exist in scripts not included in reference materials, making successful navigation impossible with available context."
      },
      "composite": 25,
      "notFound": true
    },
    {
      "id": "config-app-config-merge-order",
      "category": "configuration",
      "difficulty": "medium",
      "question": "What is the configuration merge order for app charts in Foxtrot, from lowest to highest precedence?",
      "groundTruth": "1. values.yaml, 2. default-properties.yaml, 3. default-values.yaml / k8s-values.yaml, 4. common-values.yaml, 5. properties-override.yaml, 6. k8s-override.yaml, 7. imagetags-override.yaml.",
      "llmAnswer": "NOT_FOUND\n\nThe provided documentation does not contain information about the configuration merge order for app charts in Foxtrot, from lowest to highest precedence. While the files reference Foxtrot templates (foxtrot-aks, foxtrot-eks, foxtrot-gke, etc.) and discuss Helm values and configurations, they do not specify the merge order or precedence rules for app chart configurations.",
      "filesRead": [
        "reference/helm/index.md",
        "reference/subsystems/app-common.md",
        "reference/system-architecture.md",
        "explanation/change-impact.md",
        "reference/contracts/index.md"
      ],
      "plannedFiles": [
        "reference/helm/index.md",
        "reference/subsystems/app-common.md",
        "reference/system-architecture.md",
        "explanation/change-impact.md",
        "reference/contracts/index.md"
      ],
      "score": {
        "accuracy": 0,
        "completeness": 0,
        "precision": 5,
        "navigation": 0,
        "notes": "AI agent failed to locate the configuration merge order information and returned NOT_FOUND instead of the 7-item precedence list. While the response avoided hallucination, it provided zero correct facts and demonstrated inability to navigate to the relevant documentation containing the merge order specification."
      },
      "composite": 25,
      "notFound": true
    },
    {
      "id": "config-account-creation-product-id",
      "category": "configuration",
      "difficulty": "hard",
      "question": "What is the AWS Service Catalog product ID used by account-common for account creation, and what are the customer-facing and internal OU IDs?",
      "groundTruth": "Service Catalog product ID: `prod-mts6togilnnuk`. Customer-facing OU: `ou-8vpz-mogk4qua`. Internal OU: `ou-8vpz-rw7etshv`. Provider config reference: `aws-709659200419`.",
      "llmAnswer": "NOT_FOUND\n\nThe provided documentation files do not contain information about AWS Service Catalog product IDs, customer-facing OU IDs, or internal OU IDs for the account-common subsystem. While the files describe the account-creation Helm chart and the account-common subsystem structure, they do not include these specific AWS configuration identifiers.",
      "filesRead": [
        "reference/helm/charts/account-common-charts-account-creation.md",
        "reference/subsystems/account-common.md",
        "reference/contracts/index.md",
        "reference/helm/index.md",
        "agent-kb.json"
      ],
      "plannedFiles": [
        "reference/helm/charts/account-common-charts-account-creation.md",
        "reference/subsystems/account-common.md",
        "reference/contracts/index.md",
        "reference/helm/index.md",
        "agent-kb.json"
      ],
      "score": {
        "accuracy": 0,
        "completeness": 0,
        "precision": 5,
        "navigation": 0,
        "notes": "AI agent failed to locate required AWS configuration identifiers. Ground truth confirms all four pieces of information (Service Catalog product ID, customer-facing OU, internal OU, and provider config reference) exist in the documentation. Agent returned NOT_FOUND without providing any of the expected values, indicating either insufficient search depth or inability to access the correct source files."
      },
      "composite": 25,
      "notFound": true
    },
    {
      "id": "config-ipam-rds-backup",
      "category": "configuration",
      "difficulty": "hard",
      "question": "What are the RDS automated backup retention period and backup window configured in ipam-core?",
      "groundTruth": "RDS automated backup retention: 7 days. RDS backup window: 03:00-06:00 UTC.",
      "llmAnswer": "NOT_FOUND\n\nThe documentation provided lists `ipam-core/netbox/rds.tf` as a file in the ipam-core subsystem, but the actual content of this Terraform file is not included in the reference materials. Therefore, I cannot determine the RDS automated backup retention period or backup window configuration.",
      "filesRead": [
        "reference/subsystems/ipam-core.md",
        "reference/helm/index.md",
        "reference/contracts/index.md",
        "reference/system-architecture.md",
        "agent-kb.json"
      ],
      "plannedFiles": [
        "reference/subsystems/ipam-core.md",
        "reference/helm/index.md",
        "reference/contracts/index.md",
        "reference/system-architecture.md",
        "agent-kb.json"
      ],
      "score": {
        "accuracy": 0,
        "completeness": 0,
        "precision": 5,
        "navigation": 0,
        "notes": "AI answered NOT_FOUND rather than providing the ground truth values (7 days retention, 03:00-06:00 UTC window). Per evaluation criteria, NOT_FOUND responses receive 0 for accuracy, completeness, and navigation, with 5 for precision since no hallucinated information was provided."
      },
      "composite": 25,
      "notFound": true
    },
    {
      "id": "config-dev-artifact-naming",
      "category": "configuration",
      "difficulty": "medium",
      "question": "What is the artifact naming convention for development chart versions vs release chart versions in the OCI registry?",
      "groundTruth": "Development artifacts: `{chart-name}:0.0.0-{commit-sha}`. Release artifacts: `{chart-name}:{chart-version}` (semantic version from Chart.yaml).",
      "llmAnswer": "NOT_FOUND\n\nThe provided documentation does not contain information about artifact naming conventions for development chart versions vs release chart versions in the OCI registry. While the documentation references OCI registry paths (such as `oci://europe-west2-docker.pkg.dev/artifacts-management-441415/app-common`), it does not specify naming conventions that differentiate between development and release versions.",
      "filesRead": [
        "reference/helm/index.md",
        "reference/contracts/index.md",
        "reference/system-architecture.md",
        "explanation/change-impact.md",
        "agent-kb.json"
      ],
      "plannedFiles": [
        "reference/helm/index.md",
        "reference/contracts/index.md",
        "reference/system-architecture.md",
        "explanation/change-impact.md",
        "agent-kb.json"
      ],
      "score": {
        "accuracy": 0,
        "completeness": 0,
        "precision": 5,
        "navigation": 0,
        "notes": "AI answered NOT_FOUND when ground truth specifies clear naming conventions exist: dev artifacts use 0.0.0-{commit-sha} format, release artifacts use semantic versioning from Chart.yaml. The agent failed to locate this information despite it being documented. No hallucination occurred (precision intact), but complete failure on core task dimensions."
      },
      "composite": 25,
      "notFound": true
    },
    {
      "id": "services-tech-stack-orchestration",
      "category": "services",
      "difficulty": "easy",
      "question": "What technologies make up the orchestration layer in Foxtrot's technology stack?",
      "groundTruth": "Kubernetes (EKS, GKE, AKS), ArgoCD, and Crossplane.",
      "llmAnswer": "NOT_FOUND\n\nThe documentation provided does not explicitly define or describe \"Foxtrot's orchestration layer\" or specify which technologies comprise it. While the files reference Foxtrot charts (foxtrot-eks, foxtrot-aks, foxtrot-gke, foxtrot-aws-vpc, foxtrot-azure-vnet, foxtrot-gcp-vpc) as infrastructure provisioning templates, and the control-core subsystem uses ArgoCD, Argo Workflows, and Crossplane for orchestration, there is no explicit statement identifying these as \"Foxtrot's orchestration layer.\"",
      "filesRead": [
        "reference/system-architecture.md",
        "reference/helm/index.md",
        "diagrams/helm-interactions.mmd",
        "reference/subsystems/control-core.md",
        "reference/subsystems/app-common.md"
      ],
      "plannedFiles": [
        "reference/system-architecture.md",
        "reference/helm/index.md",
        "diagrams/helm-interactions.mmd",
        "reference/subsystems/control-core.md",
        "reference/subsystems/app-common.md"
      ],
      "score": {
        "accuracy": 0,
        "completeness": 0,
        "precision": 5,
        "navigation": 0,
        "notes": "AI agent returned NOT_FOUND instead of identifying Kubernetes (EKS, GKE, AKS), ArgoCD, and Crossplane as the orchestration layer. While the agent avoided hallucination by not inventing information, it failed to locate or synthesize the correct answer from available documentation."
      },
      "composite": 25,
      "notFound": true
    },
    {
      "id": "services-state-management",
      "category": "services",
      "difficulty": "easy",
      "question": "What are the four state management services used in Foxtrot?",
      "groundTruth": "Elasticsearch, Hazelcast, Redis, and Milvus.",
      "llmAnswer": "NOT_FOUND\n\nThe provided documentation does not explicitly state which four state management services are used in Foxtrot. While the documentation references various state management services including Elasticsearch, Hazelcast, Cassandra, and Redis across the Helm charts, there is no specific statement identifying exactly four services as \"the state management services used in Foxtrot.\"",
      "filesRead": [
        "reference/system-architecture.md",
        "reference/helm/index.md",
        "reference/subsystems/app-common.md",
        "agent-kb.json",
        "reference/contracts/index.md"
      ],
      "plannedFiles": [
        "reference/system-architecture.md",
        "reference/helm/index.md",
        "reference/subsystems/app-common.md",
        "agent-kb.json",
        "reference/contracts/index.md"
      ],
      "score": {
        "accuracy": 0,
        "completeness": 0,
        "precision": 5,
        "navigation": 0,
        "notes": "AI answered NOT_FOUND rather than providing the ground truth answer (Elasticsearch, Hazelcast, Redis, and Milvus). While the agent demonstrated appropriate caution by not hallucinating information it couldn't verify, it failed to locate or identify the correct four services. Per scoring rules for NOT_FOUND responses, accuracy and completeness are zero, precision is high (no false information), and navigation is zero (failed to find the target information)."
      },
      "composite": 25,
      "notFound": true
    },
    {
      "id": "services-eks-addon-versions",
      "category": "services",
      "difficulty": "hard",
      "question": "What are the default pinned versions for the vpc-cni and coredns EKS addons in compute-common?",
      "groundTruth": "vpc-cni: v1.20.1-eksbuild.1. coredns: v1.12.2-eksbuild.4.",
      "llmAnswer": "NOT_FOUND\n\nThe documentation provided does not include the actual content of the `compute-common/templates/foxtrot-eks/values.yaml` or `compute-common/templates/foxtrot-eks/templates/crossplane-eks-addons.yaml` files, which would contain the default pinned versions for the vpc-cni and coredns EKS addons.",
      "filesRead": [
        "reference/helm/charts/compute-common-templates-foxtrot-eks.md",
        "reference/subsystems/compute-common.md",
        "reference/helm/index.md",
        "reference/system-architecture.md",
        "agent-kb.json"
      ],
      "plannedFiles": [
        "reference/helm/charts/compute-common-templates-foxtrot-eks.md",
        "reference/subsystems/compute-common.md",
        "reference/helm/index.md",
        "reference/system-architecture.md",
        "agent-kb.json"
      ],
      "score": {
        "accuracy": 0,
        "completeness": 0,
        "precision": 5,
        "navigation": 0,
        "notes": "AI agent failed to locate the required information and returned NOT_FOUND. While the response avoided hallucination and correctly acknowledged the missing data, it provided no actual addon versions (vpc-cni: v1.20.1-eksbuild.1, coredns: v1.12.2-eksbuild.4). The agent did not demonstrate ability to navigate to the relevant configuration files where these pinned versions are defined."
      },
      "composite": 25,
      "notFound": true
    },
    {
      "id": "services-aws-nat-egress-model",
      "category": "services",
      "difficulty": "medium",
      "question": "What is the AWS NAT egress model for internal VPCs vs production clusters in Foxtrot?",
      "groundTruth": "Internal AWS VPCs use shared NAT egress. Production AWS clusters use dedicated NAT egress per cluster. This split reduces cost for internal environments while isolating production egress and capacity per cluster.",
      "llmAnswer": "NOT_FOUND\n\nThe provided documentation files do not contain information about the AWS NAT egress model for internal VPCs vs production clusters in Foxtrot. The files describe the infrastructure-as-code definitions, Helm charts, and system architecture, but do not detail NAT egress strategies or model differences between internal and production environments.",
      "filesRead": [
        "reference/helm/charts/network-common-charts-foxtrot-aws-vpc.md",
        "reference/subsystems/network-common.md",
        "reference/system-architecture.md",
        "reference/helm/index.md",
        "reference/contracts/index.md"
      ],
      "plannedFiles": [
        "reference/helm/charts/network-common-charts-foxtrot-aws-vpc.md",
        "reference/subsystems/network-common.md",
        "reference/system-architecture.md",
        "reference/helm/index.md",
        "reference/contracts/index.md"
      ],
      "score": {
        "accuracy": 0,
        "completeness": 0,
        "precision": 5,
        "navigation": 0,
        "notes": "AI agent failed to locate or retrieve the ground truth information about Foxtrot's NAT egress architecture. The answer correctly avoided hallucination by stating information was not found rather than inventing details, but this represents a complete failure to answer the question. The agent should have identified documentation containing the shared NAT egress model for internal VPCs and dedicated NAT egress per cluster for production environments."
      },
      "composite": 25,
      "notFound": true
    },
    {
      "id": "services-ipam-netbox-role",
      "category": "services",
      "difficulty": "medium",
      "question": "What is NetBox used for in Foxtrot, and what infrastructure does it run on?",
      "groundTruth": "NetBox is Foxtrot's IP address management (IPAM) source of truth. It tracks all VPC CIDR blocks across clouds, prevents CIDR overlap, and serves allocation data to compute-tools during VPC and cluster provisioning. It runs on AWS ECS with an RDS database and ALB, provisioned via Terraform in ipam-core.",
      "llmAnswer": "NOT_FOUND\n\nThe provided documentation shows that NetBox infrastructure files exist within the `ipam-core` subsystem (with Terraform configurations for AWS deployment including ALB, ECS, RDS, Redis, and VPC), but the documents do not explicitly state what NetBox is used for in Foxtrot or provide a functional description of its role in the system.",
      "filesRead": [
        "reference/subsystems/ipam-core.md",
        "reference/subsystems/network-core.md",
        "reference/system-architecture.md",
        "reference/helm/index.md",
        "reference/contracts/index.md"
      ],
      "plannedFiles": [
        "reference/subsystems/ipam-core.md",
        "reference/subsystems/network-core.md",
        "reference/system-architecture.md",
        "reference/helm/index.md",
        "reference/contracts/index.md"
      ],
      "score": {
        "accuracy": 0,
        "completeness": 0,
        "precision": 5,
        "navigation": 0,
        "notes": "AI agent returned NOT_FOUND rather than providing any answer. While this avoids hallucination (precision=5), it demonstrates complete failure to locate or synthesize information about NetBox's role and infrastructure from available documentation. The agent should have identified ipam-core as the relevant subsystem and inferred NetBox's IPAM function from context clues in the infrastructure configuration."
      },
      "composite": 25,
      "notFound": true
    },
    {
      "id": "contracts-argo-gen-params-required",
      "category": "contracts",
      "difficulty": "easy",
      "question": "What are the required fields in an argo-gen-params.yaml file for the ArgoCD ApplicationSet generator?",
      "groundTruth": "chart.version (string), namespace (string), and environment (string).",
      "llmAnswer": "NOT_FOUND\n\nThe provided documentation files do not contain information about the required fields in an argo-gen-params.yaml file for the ArgoCD ApplicationSet generator. While the files reference ArgoCD ApplicationSets and the argocd-apps Helm chart, they do not specify the structure or required fields of an argo-gen-params.yaml file.",
      "filesRead": [
        "reference/helm/charts/app-common-charts-argocd-apps.md",
        "reference/subsystems/app-common.md",
        "reference/helm/index.md",
        "reference/contracts/index.md",
        "reference/system-architecture.md"
      ],
      "plannedFiles": [
        "reference/helm/charts/app-common-charts-argocd-apps.md",
        "reference/subsystems/app-common.md",
        "reference/helm/index.md",
        "reference/contracts/index.md",
        "reference/system-architecture.md"
      ],
      "score": {
        "accuracy": 0,
        "completeness": 0,
        "precision": 5,
        "navigation": 0,
        "notes": "AI agent reported information not found in documentation rather than providing the ground truth answer (chart.version, namespace, environment as required fields). No hallucination occurred, but the agent failed to locate or retrieve the correct information."
      },
      "composite": 25,
      "notFound": true
    },
    {
      "id": "contracts-azure-xrd-naming",
      "category": "contracts",
      "difficulty": "medium",
      "question": "What is the XRD name format and composition name format for Azure AKS in Foxtrot, with examples?",
      "groundTruth": "XRD name format: `{plural}.{group}`, e.g. `reltioaksclusters.foxtrot.reltio.com`. Composition name format: `reltio-aks-cluster-{channel}-{version}`, e.g. `reltio-aks-cluster-prd-v1`.",
      "llmAnswer": "NOT_FOUND\n\nThe provided documentation does not contain explicit XRD name formats or composition name formats for Azure AKS in Foxtrot. While the files reference `reltio-aks-cluster-kcl` as a composition directory and mention the cluster naming convention `azr-{phase}-{region-code}-{index}-aks`, the specific XRD and composition name formats are not documented in the available reference materials.",
      "filesRead": [
        "reference/subsystems/compute-common.md",
        "reference/helm/charts/compute-common-templates-foxtrot-aks.md",
        "reference/contracts/index.md",
        "reference/system-architecture.md",
        "reference/helm/charts/network-common-charts-foxtrot-azure-vnet.md"
      ],
      "plannedFiles": [
        "reference/subsystems/compute-common.md",
        "reference/helm/charts/compute-common-templates-foxtrot-aks.md",
        "reference/contracts/index.md",
        "reference/system-architecture.md",
        "reference/helm/charts/network-common-charts-foxtrot-azure-vnet.md"
      ],
      "score": {
        "accuracy": 0,
        "completeness": 0,
        "precision": 5,
        "navigation": 0,
        "notes": "Agent reported NOT_FOUND when ground truth confirms both XRD and composition name formats exist and are documented. Agent found tangentially related information (reltio-aks-cluster-kcl, cluster naming convention) but failed to locate the specific formats requested. No hallucination occurred, but complete failure to retrieve required information."
      },
      "composite": 25,
      "notFound": true
    },
    {
      "id": "contracts-helm-chart-required-values",
      "category": "contracts",
      "difficulty": "medium",
      "question": "What are the five required values that all app Helm charts must define?",
      "groundTruth": "accountID (string), cloudProvider (string), region (string), domain (string), and imageRegistry (string).",
      "llmAnswer": "NOT_FOUND\n\nThe provided documentation does not explicitly state what the five required values are that all app Helm charts must define. While the contracts show various values across different app charts (such as clusterName, environmentType, arch, cloudProvider, and region appearing in multiple app chart definitions), there is no specific statement identifying exactly five required values that all app Helm charts must define.",
      "filesRead": [
        "reference/helm/index.md",
        "reference/subsystems/app-common.md",
        "reference/contracts/index.md",
        "reference/system-architecture.md",
        "diagrams/app-common-contracts.mmd"
      ],
      "plannedFiles": [
        "reference/helm/index.md",
        "reference/subsystems/app-common.md",
        "reference/contracts/index.md",
        "reference/system-architecture.md",
        "diagrams/app-common-contracts.mmd"
      ],
      "score": {
        "accuracy": 0,
        "completeness": 0,
        "precision": 5,
        "navigation": 0,
        "notes": "AI agent failed to locate the ground truth specification. While it demonstrated reasonable documentation search behavior and avoided hallucination, it did not find the explicit requirement stating that accountID, cloudProvider, region, domain, and imageRegistry are the five required values for all app Helm charts."
      },
      "composite": 25,
      "notFound": true
    },
    {
      "id": "contracts-sync-wave-ordering",
      "category": "contracts",
      "difficulty": "hard",
      "question": "What are the ArgoCD sync wave values and what resource types are deployed at each wave?",
      "groundTruth": "Wave -5: prerequisites. Wave 0: infrastructure. Wave 5: state apps. Wave 10: application services. Wave 15: post-deployment resources (monitoring, backup hooks, finalizers).",
      "llmAnswer": "Based on the information in `reference/helm/index.md`, here are the ArgoCD sync wave values and resource types deployed at each wave:\n\n| Wave | Count | Resources |\n|---|---|---|\n| -1 | 1 | ServiceAccount:pre-sync-app-templates-sa |\n| 0 | 3 | Application:{{, xReltioAKSCluster:{{, ReltioEKSCluster:{{ |\n| 1 | 5 | ClusterRole:pre-sync-app-templates-clusterrole, Application:{{, Workflow:kong-gateway-setup-{{, AccessEntry:{{, Addon:{{ |\n| 2 | 4 | ClusterRoleBinding:pre-sync-app-templates-clusterrolebinding, Workflow:{{, ProviderConfig:{{, ProviderConfig:{{ |\n| 3 | 4 | Workflow:{{, Application:kong-gateway, Release:{{, Release:{{ |\n| 4 | 1 | Release:{{ |\n| 5 | 1 | Job:pre-sync-app-templates-check-job |\n\nThe sync waves range from -1 to 5, with wave -1 executing first (ServiceAccount creation), followed by progressive waves that deploy Applications, ClusterRoles, Workflows, ProviderConfigs, Releases, and finally a validation Job at wave 5.",
      "filesRead": [
        "reference/helm/charts/app-common-charts-argocd-apps.md",
        "reference/helm/index.md",
        "reference/subsystems/app-common.md",
        "diagrams/helm-interactions.mmd",
        "reference/system-architecture.md"
      ],
      "plannedFiles": [
        "reference/helm/charts/app-common-charts-argocd-apps.md",
        "reference/helm/index.md",
        "reference/subsystems/app-common.md",
        "diagrams/helm-interactions.mmd",
        "reference/system-architecture.md"
      ],
      "score": {
        "accuracy": 0,
        "completeness": 1,
        "precision": 1,
        "navigation": 1,
        "notes": "AI answer is fundamentally misaligned with ground truth. Ground truth specifies waves -5, 0, 5, 10, 15 with categorical descriptions (prerequisites, infrastructure, state apps, application services, post-deployment). AI provides waves -1 through 5 with specific resource types that don't correspond to the expected wave structure. Missing waves 10 and 15 entirely. The answer appears to reference a different deployment configuration or file than the ground truth describes. Agent attempted navigation but retrieved incorrect or mismatched information."
      },
      "composite": 15,
      "notFound": false
    }
  ]
}