744 lines
46 KiB
JSON
744 lines
46 KiB
JSON
|
{
|
||
|
|
"evalType": "human",
|
||
|
|
"timestamp": "2026-03-10T04:32:17.678Z",
|
||
|
|
"docsDir": "./foxtrot-docs",
|
||
|
|
"totalQuestions": 28,
|
||
|
|
"overallScore": "79.3",
|
||
|
|
"avgClarity": "4.07",
|
||
|
|
"avgActionability": "3.61",
|
||
|
|
"avgCompleteness": "3.96",
|
||
|
|
"avgFindability": "4.21",
|
||
|
|
"notFoundCount": 1,
|
||
|
|
"notFoundRate": "3.6%",
|
||
|
|
"byCategory": {
|
||
|
|
"structural": {
|
||
|
|
"avg": "82.5",
|
||
|
|
"count": 4
|
||
|
|
},
|
||
|
|
"dependency": {
|
||
|
|
"avg": "59.0",
|
||
|
|
"count": 5
|
||
|
|
},
|
||
|
|
"contract": {
|
||
|
|
"avg": "90.0",
|
||
|
|
"count": 1
|
||
|
|
},
|
||
|
|
"configuration": {
|
||
|
|
"avg": "68.0",
|
||
|
|
"count": 5
|
||
|
|
},
|
||
|
|
"interaction": {
|
||
|
|
"avg": "92.5",
|
||
|
|
"count": 2
|
||
|
|
},
|
||
|
|
"resource": {
|
||
|
|
"avg": "96.3",
|
||
|
|
"count": 4
|
||
|
|
},
|
||
|
|
"cross-subsystem": {
|
||
|
|
"avg": "92.5",
|
||
|
|
"count": 2
|
||
|
|
},
|
||
|
|
"architectural": {
|
||
|
|
"avg": "97.5",
|
||
|
|
"count": 2
|
||
|
|
},
|
||
|
|
"scenario": {
|
||
|
|
"avg": "71.7",
|
||
|
|
"count": 3
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"byDifficulty": {
|
||
|
|
"easy": {
|
||
|
|
"avg": "78.5",
|
||
|
|
"count": 10
|
||
|
|
},
|
||
|
|
"medium": {
|
||
|
|
"avg": "75.8",
|
||
|
|
"count": 13
|
||
|
|
},
|
||
|
|
"hard": {
|
||
|
|
"avg": "90.0",
|
||
|
|
"count": 5
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"results": [
|
||
|
|
{
|
||
|
|
"id": "struct-001",
|
||
|
|
"category": "structural",
|
||
|
|
"difficulty": "easy",
|
||
|
|
"question": "How many Helm charts are defined across the Foxtrot monorepo?",
|
||
|
|
"groundTruth": "76",
|
||
|
|
"humanAnswer": "76",
|
||
|
|
"docsUsed": [
|
||
|
|
"reference/system-architecture.md",
|
||
|
|
"reference/helm/index.md"
|
||
|
|
],
|
||
|
|
"score": {
|
||
|
|
"clarity": 5,
|
||
|
|
"actionability": 2,
|
||
|
|
"completeness": 3,
|
||
|
|
"findability": 5,
|
||
|
|
"notes": "Reader extracted correct answer from well-organized reference docs. Helm index and system architecture are intuitive locations. However, knowing the chart count alone provides minimal actionability—engineers need chart names, purposes, and dependencies to act. Docs appear to contain the metric but likely lack operational context."
|
||
|
|
},
|
||
|
|
"composite": 75,
|
||
|
|
"notFound": false
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"id": "struct-002",
|
||
|
|
"category": "structural",
|
||
|
|
"difficulty": "medium",
|
||
|
|
"question": "Which 5 Helm charts produce the most Kubernetes resources? List them with their resource counts.",
|
||
|
|
"groundTruth": "cluster (app-common/apps-provider/addons/charts/common/cluster): 55\ncluster (app-common/charts/cluster): 32\njenkins (app-common/charts/jenkins): 29\nkube-state-metrics (app-common/charts/otel-collector/charts/kube-state-metrics): 27\nfoxtrot-eks (compute-common/templates/foxtrot-eks): 24",
|
||
|
|
"humanAnswer": "1. cluster (`app-common/apps-provider/addons/charts/common/cluster`) - 55 resources\n2. cluster (`app-common/charts/cluster`) - 32 resources\n3. jenkins (`app-common/charts/jenkins`) - 29 resources\n4. kube-state-metrics (`app-common/charts/otel-collector/charts/kube-state-metrics`) - 27 resources\n5. foxtrot-eks (`compute-common/templates/foxtrot-eks`) - 24 resources",
|
||
|
|
"docsUsed": [
|
||
|
|
"reference/system-architecture.md",
|
||
|
|
"reference/helm/index.md"
|
||
|
|
],
|
||
|
|
"score": {
|
||
|
|
"clarity": 4,
|
||
|
|
"actionability": 3,
|
||
|
|
"completeness": 5,
|
||
|
|
"findability": 4,
|
||
|
|
"notes": "Reader extracted exact answer with all 5 charts and resource counts correct. Docs clearly presented the data in a structured way. Findability is strong (helm/index.md is logical), though 'index.md' could be ambiguous. Actionability limited because resource counts alone don't prescribe actions—useful for analysis but not execution."
|
||
|
|
},
|
||
|
|
"composite": 80,
|
||
|
|
"notFound": false
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"id": "struct-003",
|
||
|
|
"category": "structural",
|
||
|
|
"difficulty": "easy",
|
||
|
|
"question": "How many subsystems does the Foxtrot codebase contain?",
|
||
|
|
"groundTruth": "12",
|
||
|
|
"humanAnswer": "12\n\nThe documentation explicitly states in the System Architecture summary: \"**Subsystems:** 12\"\n\nThese are: account-common, app-common, app-tools, compute-common, compute-tools, control-core, ipam-core, ipam-tools, network-common, network-core, runtime, and root.",
|
||
|
|
"docsUsed": [
|
||
|
|
"reference/system-architecture.md",
|
||
|
|
"reference/subsystems/account-common.md",
|
||
|
|
"reference/subsystems/app-tools.md",
|
||
|
|
"reference/subsystems/compute-common.md",
|
||
|
|
"reference/subsystems/compute-tools.md",
|
||
|
|
"reference/subsystems/control-core.md",
|
||
|
|
"reference/subsystems/ipam-core.md",
|
||
|
|
"reference/subsystems/ipam-tools.md",
|
||
|
|
"reference/subsystems/network-common.md",
|
||
|
|
"reference/subsystems/root.md"
|
||
|
|
],
|
||
|
|
"score": {
|
||
|
|
"clarity": 5,
|
||
|
|
"actionability": 5,
|
||
|
|
"completeness": 5,
|
||
|
|
"findability": 5,
|
||
|
|
"notes": "Documentation explicitly states subsystem count with complete enumeration. System Architecture summary provides immediate answer. Logical file structure (reference/system-architecture.md for overview, reference/subsystems/* for details) enables natural discovery path. Reader obtained correct answer efficiently."
|
||
|
|
},
|
||
|
|
"composite": 100,
|
||
|
|
"notFound": false
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"id": "struct-004",
|
||
|
|
"category": "structural",
|
||
|
|
"difficulty": "easy",
|
||
|
|
"question": "Which subsystem contains the most files, and how many?",
|
||
|
|
"groundTruth": "runtime: 16309 files",
|
||
|
|
"humanAnswer": "**runtime** contains the most files with **16,309** files.",
|
||
|
|
"docsUsed": [
|
||
|
|
"reference/system-architecture.md",
|
||
|
|
"reference/subsystems/account-common.md",
|
||
|
|
"reference/subsystems/app-tools.md",
|
||
|
|
"reference/subsystems/compute-common.md",
|
||
|
|
"reference/subsystems/compute-tools.md",
|
||
|
|
"reference/subsystems/control-core.md",
|
||
|
|
"reference/subsystems/ipam-core.md",
|
||
|
|
"reference/subsystems/ipam-tools.md",
|
||
|
|
"reference/subsystems/network-common.md",
|
||
|
|
"reference/subsystems/root.md"
|
||
|
|
],
|
||
|
|
"score": {
|
||
|
|
"clarity": 4,
|
||
|
|
"actionability": 2,
|
||
|
|
"completeness": 5,
|
||
|
|
"findability": 4,
|
||
|
|
"notes": "Reader extracted correct answer with proper formatting, indicating clear presentation. Information was complete and located in expected reference documentation. Actionability is limited since this is a factual query rather than prescriptive guidance—useful for understanding system composition but not directly executable."
|
||
|
|
},
|
||
|
|
"composite": 75,
|
||
|
|
"notFound": false
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"id": "dep-external-dns-001",
|
||
|
|
"category": "dependency",
|
||
|
|
"difficulty": "medium",
|
||
|
|
"question": "What are the dependencies of the \"external-dns\" chart (at app-common/apps-provider/addons/charts/common/external-dns)?",
|
||
|
|
"groundTruth": "external-dns (1.17.0) [condition: external-dns.enabled]",
|
||
|
|
"humanAnswer": "Looking at the Helm Charts table in `reference/helm/index.md`, the external-dns chart at `app-common/apps-provider/addons/charts/common/external-dns` has the following dependency:\n\n- `external-dns`",
|
||
|
|
"docsUsed": [
|
||
|
|
"reference/system-architecture.md",
|
||
|
|
"reference/helm/index.md",
|
||
|
|
"reference/subsystems/account-common.md",
|
||
|
|
"reference/subsystems/app-tools.md",
|
||
|
|
"reference/subsystems/compute-common.md",
|
||
|
|
"reference/subsystems/compute-tools.md",
|
||
|
|
"reference/subsystems/ipam-core.md"
|
||
|
|
],
|
||
|
|
"score": {
|
||
|
|
"clarity": 3,
|
||
|
|
"actionability": 2,
|
||
|
|
"completeness": 2,
|
||
|
|
"findability": 4,
|
||
|
|
"notes": "Reader located the correct source document (reference/helm/index.md) and identified the dependency name, but extracted incomplete information. Ground truth includes version (1.17.0) and condition (external-dns.enabled), neither of which appear in the reader's answer. Documentation either lacks these details or presented them unclearly enough that the reader missed them. Missing version constraint and conditional logic prevents full actionability."
|
||
|
|
},
|
||
|
|
"composite": 55,
|
||
|
|
"notFound": false
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"id": "dep-ingress-nginx-001",
|
||
|
|
"category": "dependency",
|
||
|
|
"difficulty": "medium",
|
||
|
|
"question": "What are the dependencies of the \"ingress-nginx\" chart (at app-common/apps-provider/addons/charts/common/ingress-nginx)?",
|
||
|
|
"groundTruth": "ingress-nginx (4.12.0) [condition: ingress-nginx.enabled]",
|
||
|
|
"humanAnswer": "Looking at the \"All Charts\" table in the Helm Charts documentation, the ingress-nginx chart at `app-common/apps-provider/addons/charts/common/ingress-nginx` has the following dependencies:\n\n- `ingress-nginx`",
|
||
|
|
"docsUsed": [
|
||
|
|
"reference/system-architecture.md",
|
||
|
|
"reference/helm/index.md",
|
||
|
|
"reference/subsystems/account-common.md",
|
||
|
|
"reference/subsystems/app-tools.md",
|
||
|
|
"reference/subsystems/compute-common.md",
|
||
|
|
"reference/subsystems/compute-tools.md",
|
||
|
|
"reference/subsystems/ipam-core.md"
|
||
|
|
],
|
||
|
|
"score": {
|
||
|
|
"clarity": 2,
|
||
|
|
"actionability": 2,
|
||
|
|
"completeness": 2,
|
||
|
|
"findability": 3,
|
||
|
|
"notes": "Reader identified ingress-nginx as a dependency but omitted critical details: version (4.12.0) and condition (ingress-nginx.enabled). The documentation either failed to surface these details prominently or the reader didn't fully extract available information. Missing version and condition significantly reduces actionability for engineers needing to configure or troubleshoot this dependency."
|
||
|
|
},
|
||
|
|
"composite": 45,
|
||
|
|
"notFound": false
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"id": "dep-kube-state-metrics-001",
|
||
|
|
"category": "dependency",
|
||
|
|
"difficulty": "medium",
|
||
|
|
"question": "What are the dependencies of the \"kube-state-metrics\" chart (at app-common/apps-provider/addons/charts/common/kube-state-metrics)?",
|
||
|
|
"groundTruth": "kube-state-metrics (5.26.0) [condition: kube-state-metrics.enabled]",
|
||
|
|
"humanAnswer": "Looking at the Helm charts table in `reference/helm/index.md`, the kube-state-metrics chart at `app-common/apps-provider/addons/charts/common/kube-state-metrics` has the following dependencies:\n\n- `kube-state-metrics`",
|
||
|
|
"docsUsed": [
|
||
|
|
"reference/system-architecture.md",
|
||
|
|
"reference/helm/index.md",
|
||
|
|
"reference/subsystems/account-common.md",
|
||
|
|
"reference/subsystems/app-tools.md",
|
||
|
|
"reference/subsystems/compute-common.md",
|
||
|
|
"reference/subsystems/compute-tools.md",
|
||
|
|
"reference/subsystems/ipam-core.md"
|
||
|
|
],
|
||
|
|
"score": {
|
||
|
|
"clarity": 3,
|
||
|
|
"actionability": 2,
|
||
|
|
"completeness": 1,
|
||
|
|
"findability": 2,
|
||
|
|
"notes": "Reader identified the dependency name but missed critical details: version (5.26.0) and condition constraint (kube-state-metrics.enabled). The answer is incomplete for practical use. Documentation either lacks these details in the consulted location or they're scattered across files, making the information insufficient for an engineer to properly configure the dependency."
|
||
|
|
},
|
||
|
|
"composite": 40,
|
||
|
|
"notFound": false
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"id": "dep-open-telemetry-001",
|
||
|
|
"category": "dependency",
|
||
|
|
"difficulty": "medium",
|
||
|
|
"question": "What are the dependencies of the \"open-telemetry\" chart (at app-common/apps-provider/addons/charts/common/otel-collector)?",
|
||
|
|
"groundTruth": "opentelemetry-operator (0.26.4) [condition: opentelemetry-operator.enabled]\nk8s-metrics (0.1.0) [condition: k8s-metrics.enabled]\nagent (0.1.1) [condition: agent.enabled]\ncommon (0.1.0) [condition: common.enabled]\ngw-metrics (0.1.0) [condition: gw-metrics.enabled]\ngw-traces (0.1.0) [condition: gw-traces.enabled]\nprom-scraper (0.1.0) [condition: prom-scraper.enabled]",
|
||
|
|
"humanAnswer": "The dependencies of the \"open-telemetry\" chart at `app-common/apps-provider/addons/charts/common/otel-collector` are:\n\n- **opentelemetry-operator** (0.26.4) *if opentelemetry-operator.enabled*\n- **k8s-metrics** (0.1.0) *if k8s-metrics.enabled*\n- **agent** (0.1.1) *if agent.enabled*\n- **common** (0.1.0) *if common.enabled*\n- **gw-metrics** (0.1.0) *if gw-metrics.enabled*\n- **gw-traces** (0.1.0) *if gw-traces.enabled*\n- **prom-scraper** (0.1.0) *if prom-scraper.enabled*\n\nAll dependencies are conditional based on their respective enabled flags in the values configuration.",
|
||
|
|
"docsUsed": [
|
||
|
|
"reference/system-architecture.md",
|
||
|
|
"reference/helm/index.md",
|
||
|
|
"reference/helm/charts/app-common-apps-provider-addons-charts-common-otel-collector-charts-agent.md",
|
||
|
|
"reference/helm/charts/app-common-apps-provider-addons-charts-common-otel-collector-charts-common.md",
|
||
|
|
"reference/helm/charts/app-common-apps-provider-addons-charts-common-otel-collector-charts-gw-metrics.md",
|
||
|
|
"reference/helm/charts/app-common-apps-provider-addons-charts-common-otel-collector-charts-gw-traces.md",
|
||
|
|
"reference/helm/charts/app-common-apps-provider-addons-charts-common-otel-collector-charts-k8s-metrics.md",
|
||
|
|
"reference/helm/charts/app-common-apps-provider-addons-charts-common-otel-collector-charts-prom-scraper.md",
|
||
|
|
"reference/helm/charts/app-common-apps-provider-addons-charts-common-otel-collector.md",
|
||
|
|
"reference/helm/charts/app-common-charts-otel-collector-charts-agent.md",
|
||
|
|
"reference/helm/charts/app-common-charts-otel-collector-charts-common.md",
|
||
|
|
"reference/helm/charts/app-common-charts-otel-collector-charts-gw-metrics.md",
|
||
|
|
"reference/helm/charts/app-common-charts-otel-collector-charts-gw-traces.md",
|
||
|
|
"reference/helm/charts/app-common-charts-otel-collector-charts-k8s-metrics.md",
|
||
|
|
"reference/helm/charts/app-common-charts-otel-collector-charts-kube-state-metrics.md",
|
||
|
|
"reference/helm/charts/app-common-charts-otel-collector-charts-prom-scraper.md",
|
||
|
|
"reference/helm/charts/app-common-charts-otel-collector.md"
|
||
|
|
],
|
||
|
|
"score": {
|
||
|
|
"clarity": 5,
|
||
|
|
"actionability": 5,
|
||
|
|
"completeness": 5,
|
||
|
|
"findability": 5,
|
||
|
|
"notes": "Reader extracted all 7 dependencies with exact versions and conditions matching ground truth. Main chart documentation file consulted is the natural location for dependency information. Answer is well-structured and immediately actionable for configuration."
|
||
|
|
},
|
||
|
|
"composite": 100,
|
||
|
|
"notFound": false
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"id": "dep-reltio-prometheus-001",
|
||
|
|
"category": "dependency",
|
||
|
|
"difficulty": "medium",
|
||
|
|
"question": "What are the dependencies of the \"reltio-prometheus\" chart (at app-common/apps-provider/addons/charts/common/prometheus)?",
|
||
|
|
"groundTruth": "kube-prometheus-stack (69.4.1) [condition: kube-prometheus-stack.enabled]\nprometheus-adapter (4.11.0) [condition: prometheus-adapter.enabled]",
|
||
|
|
"humanAnswer": "\n\nLooking at the \"All Charts\" table in `reference/helm/index.md`, the reltio-prometheus chart (at `app-common/apps-provider/addons/charts/common/prometheus`) has the following dependencies:\n\n- `kube-prometheus-stack`\n- `prometheus-adapter`",
|
||
|
|
"docsUsed": [
|
||
|
|
"reference/system-architecture.md",
|
||
|
|
"reference/helm/index.md",
|
||
|
|
"reference/subsystems/account-common.md",
|
||
|
|
"reference/subsystems/app-tools.md",
|
||
|
|
"reference/subsystems/compute-common.md",
|
||
|
|
"reference/subsystems/compute-tools.md",
|
||
|
|
"reference/subsystems/ipam-core.md"
|
||
|
|
],
|
||
|
|
"score": {
|
||
|
|
"clarity": 3,
|
||
|
|
"actionability": 2,
|
||
|
|
"completeness": 2,
|
||
|
|
"findability": 4,
|
||
|
|
"notes": "Reader correctly identified dependency names from the Helm index table, but missed critical details: version numbers (69.4.1, 4.11.0) and conditional flags (kube-prometheus-stack.enabled, prometheus-adapter.enabled). The table format apparently didn't make these details sufficiently prominent. An engineer using this information would lack version pinning and conditional logic needed for proper deployment configuration."
|
||
|
|
},
|
||
|
|
"composite": 55,
|
||
|
|
"notFound": false
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"id": "contract-001",
|
||
|
|
"category": "contract",
|
||
|
|
"difficulty": "medium",
|
||
|
|
"question": "Which secrets or ConfigMaps are shared across multiple Helm charts? List each with the charts that use it.",
|
||
|
|
"groundTruth": "logdna-agent-key: cluster\nlogdna-agent-tags: cluster\nvault-secret: ai-app, cleanse-app, pms-app, rdm-app, shared-app\nvault-secret-rdm-tasks: ai-app, cleanse-app, pms-app, rdm-app, shared-app",
|
||
|
|
"humanAnswer": "Based on the \"Shared Secrets & ConfigMaps\" section in the documentation:\n\n- **logdna-agent-key** — used by: cluster, cluster\n- **logdna-agent-tags** — used by: cluster, cluster\n- **vault-secret** — used by: ai-app, cleanse-app, pms-app, rdm-app, shared-app\n- **vault-secret-rdm-tasks** — used by: ai-app, cleanse-app, pms-app, rdm-app, shared-app\n\nThe vault-secret and vault-secret-rdm-tasks are the most widely shared, appearing across five different application charts. The logdna-agent-key and logdna-agent-tags appear to be used by the cluster charts (there are two cluster charts in the system based on the interaction diagram).",
|
||
|
|
"docsUsed": [
|
||
|
|
"reference/system-architecture.md",
|
||
|
|
"reference/helm/index.md"
|
||
|
|
],
|
||
|
|
"score": {
|
||
|
|
"clarity": 4,
|
||
|
|
"actionability": 4,
|
||
|
|
"completeness": 5,
|
||
|
|
"findability": 5,
|
||
|
|
"notes": "Reader extracted all four secrets and correct chart mappings. Minor issue: 'cluster, cluster' duplication suggests ambiguity in how single-chart usage was presented. The vault-secret items correctly identified as most widely shared. Documentation location and section naming are ideal for this query type."
|
||
|
|
},
|
||
|
|
"composite": 90,
|
||
|
|
"notFound": false
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"id": "config-mdm-app-domain",
|
||
|
|
"category": "configuration",
|
||
|
|
"difficulty": "easy",
|
||
|
|
"question": "What is the default value of \"domain\" in the mdm-app chart?",
|
||
|
|
"groundTruth": "cloud.reltio.com",
|
||
|
|
"humanAnswer": "`cloud.reltio.com`",
|
||
|
|
"docsUsed": [
|
||
|
|
"reference/system-architecture.md",
|
||
|
|
"reference/helm/index.md",
|
||
|
|
"reference/helm/charts/app-common-charts-mdm-app.md"
|
||
|
|
],
|
||
|
|
"score": {
|
||
|
|
"clarity": 5,
|
||
|
|
"actionability": 5,
|
||
|
|
"completeness": 5,
|
||
|
|
"findability": 5,
|
||
|
|
"notes": "Reader extracted exact answer from appropriate Helm chart documentation. Default value was clearly documented and readily discoverable in the mdm-app chart reference file."
|
||
|
|
},
|
||
|
|
"composite": 100,
|
||
|
|
"notFound": false
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"id": "config-ai-app-domain",
|
||
|
|
"category": "configuration",
|
||
|
|
"difficulty": "easy",
|
||
|
|
"question": "What is the default value of \"domain\" in the ai-app chart?",
|
||
|
|
"groundTruth": "cloud.reltio.com",
|
||
|
|
"humanAnswer": "`cloud.reltio.com`",
|
||
|
|
"docsUsed": [
|
||
|
|
"reference/system-architecture.md",
|
||
|
|
"reference/helm/index.md",
|
||
|
|
"reference/helm/charts/app-common-charts-ai-app.md"
|
||
|
|
],
|
||
|
|
"score": {
|
||
|
|
"clarity": 5,
|
||
|
|
"actionability": 5,
|
||
|
|
"completeness": 5,
|
||
|
|
"findability": 5,
|
||
|
|
"notes": "Reader successfully located the correct default value. The ai-app chart reference file is the logical location for Helm chart defaults, and the information was presented clearly enough for accurate extraction."
|
||
|
|
},
|
||
|
|
"composite": 100,
|
||
|
|
"notFound": false
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"id": "config-elasticsearch-domain",
|
||
|
|
"category": "configuration",
|
||
|
|
"difficulty": "easy",
|
||
|
|
"question": "What is the default value of \"domain\" in the elasticsearch chart?",
|
||
|
|
"groundTruth": "etalon.reltio.com",
|
||
|
|
"humanAnswer": "NOT_FOUND\n\nThe elasticsearch chart documentation lists `domain` as a configuration key with type `unknown` and indicates a default exists, but the actual default value is not specified in the provided documentation.",
|
||
|
|
"docsUsed": [
|
||
|
|
"reference/system-architecture.md",
|
||
|
|
"reference/helm/index.md",
|
||
|
|
"reference/helm/charts/app-common-apps-provider-addons-charts-state-cluster-elasticsearch.md",
|
||
|
|
"reference/helm/charts/app-common-charts-elasticsearch.md"
|
||
|
|
],
|
||
|
|
"score": {
|
||
|
|
"clarity": 0,
|
||
|
|
"actionability": 0,
|
||
|
|
"completeness": 0,
|
||
|
|
"findability": 0,
|
||
|
|
"notes": "Documentation indicates a default value exists but fails to specify it. Reader consulted appropriate helm chart references but could not extract the actual default (etalon.reltio.com). This represents a critical gap: the docs signal information presence without delivering it, leaving engineers unable to configure or understand the system."
|
||
|
|
},
|
||
|
|
"composite": 0,
|
||
|
|
"notFound": true
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"id": "config-hazelcast-domain",
|
||
|
|
"category": "configuration",
|
||
|
|
"difficulty": "easy",
|
||
|
|
"question": "What is the default value of \"domain\" in the hazelcast chart?",
|
||
|
|
"groundTruth": "etalon.reltio.com",
|
||
|
|
"humanAnswer": "internal.reltio.com",
|
||
|
|
"docsUsed": [
|
||
|
|
"reference/system-architecture.md",
|
||
|
|
"reference/helm/index.md",
|
||
|
|
"reference/helm/charts/app-common-apps-provider-addons-charts-state-cluster-hazelcast.md",
|
||
|
|
"reference/helm/charts/app-common-charts-hazelcast.md"
|
||
|
|
],
|
||
|
|
"score": {
|
||
|
|
"clarity": 2,
|
||
|
|
"actionability": 1,
|
||
|
|
"completeness": 2,
|
||
|
|
"findability": 3,
|
||
|
|
"notes": "Reader located relevant Hazelcast chart documentation but extracted incorrect default value. Docs likely present multiple domain values without clear distinction of which is the default, or default is de-emphasized relative to other configuration options. Reader could act on the information but would implement wrong configuration. Correct docs were found; incorrect value was extracted from them."
|
||
|
|
},
|
||
|
|
"composite": 40,
|
||
|
|
"notFound": false
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"id": "config-redis-domain",
|
||
|
|
"category": "configuration",
|
||
|
|
"difficulty": "easy",
|
||
|
|
"question": "What is the default value of \"domain\" in the redis chart?",
|
||
|
|
"groundTruth": "internal.reltio.com",
|
||
|
|
"humanAnswer": "internal.reltio.com",
|
||
|
|
"docsUsed": [
|
||
|
|
"reference/system-architecture.md",
|
||
|
|
"reference/helm/index.md",
|
||
|
|
"reference/helm/charts/app-common-charts-redis.md"
|
||
|
|
],
|
||
|
|
"score": {
|
||
|
|
"clarity": 5,
|
||
|
|
"actionability": 5,
|
||
|
|
"completeness": 5,
|
||
|
|
"findability": 5,
|
||
|
|
"notes": "Reader located correct default value. Documentation path (app-common-charts-redis.md) is intuitive for redis chart configuration. Information was sufficiently clear and complete for accurate extraction and use."
|
||
|
|
},
|
||
|
|
"composite": 100,
|
||
|
|
"notFound": false
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"id": "interaction-001",
|
||
|
|
"category": "interaction",
|
||
|
|
"difficulty": "medium",
|
||
|
|
"question": "Which Helm charts reference Kubernetes services from other components? List each chart and the service it calls.",
|
||
|
|
"groundTruth": "hazelcast (app-common/apps-provider/addons/charts/state-apps/hz) → kubernetes\nhazelcast (app-common/charts/hazelcast) → kubernetes\ngw-metrics (app-common/charts/otel-collector/charts/gw-metrics) → argo-rollouts-metrics\nredis (app-common/charts/redis) → headless",
|
||
|
|
"humanAnswer": "**Service-to-Service References:**\n\n| Chart | Service Called |\n|---|---|\n| hazelcast | `kubernetes` |\n| gw-metrics | `argo-rollouts-metrics` |\n| redis | `headless` |\n\nNote: The hazelcast entry appears twice in the source documentation, likely representing the two hazelcast chart variants (`app-common/apps-provider/addons/charts/state-apps/hz` and `app-common/charts/hazelcast`), both calling the same `kubernetes` service.",
|
||
|
|
"docsUsed": [
|
||
|
|
"reference/system-architecture.md",
|
||
|
|
"reference/helm/index.md"
|
||
|
|
],
|
||
|
|
"score": {
|
||
|
|
"clarity": 4,
|
||
|
|
"actionability": 4,
|
||
|
|
"completeness": 5,
|
||
|
|
"findability": 4,
|
||
|
|
"notes": "Reader extracted all required service references correctly and presented them clearly. The table format aids comprehension. Completeness is strong—all three charts and their services are captured, including the hazelcast variant distinction. Actionability is slightly reduced because full chart paths (e.g., `app-common/apps-provider/addons/charts/state-apps/hz`) aren't included in the answer, which an engineer might need to locate and modify these charts. Findability is solid; reference/helm/index.md and system-architecture.md are logical locations for this information."
|
||
|
|
},
|
||
|
|
"composite": 85,
|
||
|
|
"notFound": false
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"id": "interaction-002",
|
||
|
|
"category": "interaction",
|
||
|
|
"difficulty": "hard",
|
||
|
|
"question": "Which network ports are used by multiple Helm charts? List each port and the charts that expose it.",
|
||
|
|
"groundTruth": "Port 443: cluster, twistlock-defender, eck-operator, twistlock\nPort 8889: gw-metrics, gw-traces, kube-state-metrics\nPort 9200: eck-operator, elasticsearch\nPort 13133: agent, gw-metrics, gw-traces, k8s-metrics, prom-scraper, elasticsearch",
|
||
|
|
"humanAnswer": "**Port Allocation (shared)**\n\n- **443**: cluster, twistlock-defender, eck-operator, twistlock\n- **8889**: gw-metrics, gw-traces, kube-state-metrics\n- **9200**: eck-operator, elasticsearch\n- **13133**: agent, gw-metrics, gw-traces, k8s-metrics, prom-scraper, elasticsearch",
|
||
|
|
"docsUsed": [
|
||
|
|
"reference/system-architecture.md",
|
||
|
|
"reference/helm/index.md"
|
||
|
|
],
|
||
|
|
"score": {
|
||
|
|
"clarity": 5,
|
||
|
|
"actionability": 5,
|
||
|
|
"completeness": 5,
|
||
|
|
"findability": 5,
|
||
|
|
"notes": "Reader extracted exact answer matching ground truth. Information presented in scannable format with clear port-to-chart mapping. Docs consulted (system-architecture, helm/index) are logical locations for port allocation data. Engineer can immediately act on this for deployment planning and network configuration."
|
||
|
|
},
|
||
|
|
"composite": 100,
|
||
|
|
"notFound": false
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"id": "resource-001",
|
||
|
|
"category": "resource",
|
||
|
|
"difficulty": "medium",
|
||
|
|
"question": "What are the 10 most common Kubernetes resource types generated across all Foxtrot Helm charts?",
|
||
|
|
"groundTruth": "ClusterRole: 51\nServiceAccount: 49\nSecret: 43\nExternalSecret: 38\nService: 36\nRole: 30\nClusterRoleBinding: 29\nDeployment: 29\nConfigMap: 25\nRoleBinding: 25",
|
||
|
|
"humanAnswer": "Based on the \"Kubernetes Resource Types (across all charts)\" table in the system architecture documentation, the 10 most common Kubernetes resource types are:\n\n1. ClusterRole — 51\n2. ServiceAccount — 49\n3. Secret — 43\n4. ExternalSecret — 38\n5. Service — 36\n6. Role — 30\n7. ClusterRoleBinding — 29\n8. Deployment — 29\n9. ConfigMap — 25\n10. RoleBinding — 25",
|
||
|
|
"docsUsed": [
|
||
|
|
"reference/system-architecture.md",
|
||
|
|
"reference/helm/index.md"
|
||
|
|
],
|
||
|
|
"score": {
|
||
|
|
"clarity": 5,
|
||
|
|
"actionability": 3,
|
||
|
|
"completeness": 5,
|
||
|
|
"findability": 4,
|
||
|
|
"notes": "Reader extracted exact ground truth from a dedicated table. Information is presented clearly with counts. Docs are in expected locations (system-architecture, helm reference). Actionability limited because data is informational rather than prescriptive—useful for understanding chart composition but lacks guidance on what to do with these metrics."
|
||
|
|
},
|
||
|
|
"composite": 85,
|
||
|
|
"notFound": false
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"id": "resource-mdm-app-001",
|
||
|
|
"category": "resource",
|
||
|
|
"difficulty": "medium",
|
||
|
|
"question": "What Kubernetes resource types does the mdm-app chart generate? List each type and count.",
|
||
|
|
"groundTruth": "Secret: 5\nJob: 4\nDeployment: 2\nPersistentVolumeClaim: 2\nService: 2\nExternalSecret: 2\nHorizontalPodAutoscaler: 1\nRollout: 1\nPersistentVolume: 1\nPodDisruptionBudget: 1\nServiceAccount: 1",
|
||
|
|
"humanAnswer": "Based on the mdm-app chart documentation, here are the Kubernetes resource types generated:\n\n| Resource Type | Count |\n|---|---|\n| Deployment | 2 |\n| Job | 4 |\n| Secret | 5 |\n| Service | 2 |\n| PersistentVolumeClaim | 2 |\n| ExternalSecret | 2 |\n| HorizontalPodAutoscaler | 1 |\n| Rollout | 1 |\n| PersistentVolume | 1 |\n| PodDisruptionBudget | 1 |\n| ServiceAccount | 1 |\n\n**Total: 22 resources**",
|
||
|
|
"docsUsed": [
|
||
|
|
"reference/system-architecture.md",
|
||
|
|
"reference/helm/index.md",
|
||
|
|
"reference/helm/charts/app-common-charts-mdm-app.md"
|
||
|
|
],
|
||
|
|
"score": {
|
||
|
|
"clarity": 5,
|
||
|
|
"actionability": 5,
|
||
|
|
"completeness": 5,
|
||
|
|
"findability": 5,
|
||
|
|
"notes": "Reader extracted all 11 resource types with exact counts matching ground truth. Table format presents information immediately and unambiguously. Documentation path (app-common-charts-mdm-app.md) is the natural location for chart resource specifications. Information is directly actionable for infrastructure planning and validation."
|
||
|
|
},
|
||
|
|
"composite": 100,
|
||
|
|
"notFound": false
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"id": "resource-cassandra-001",
|
||
|
|
"category": "resource",
|
||
|
|
"difficulty": "medium",
|
||
|
|
"question": "What Kubernetes resource types does the cassandra chart generate? List each type and count.",
|
||
|
|
"groundTruth": "Service: 2\nStatefulSet: 1\nExternalSecret: 1\nClusterSecretStore: 1\nConfigMap: 1\nSecret: 1\nServiceAccount: 1\nRole: 1\nRoleBinding: 1\nClusterRole: 1\nClusterRoleBinding: 1\nCronJob: 1\nJob: 1",
|
||
|
|
"humanAnswer": "Based on the cassandra chart documentation, here are the Kubernetes resource types generated:\n\n| Resource Type | Count |\n|---|---|\n| Service | 2 |\n| StatefulSet | 1 |\n| ExternalSecret | 1 |\n| ClusterSecretStore | 1 |\n| ConfigMap | 1 |\n| Secret | 1 |\n| ServiceAccount | 1 |\n| Role | 1 |\n| RoleBinding | 1 |\n| ClusterRole | 1 |\n| ClusterRoleBinding | 1 |\n| CronJob | 1 |\n| Job | 1 |\n\n**Total: 14 resources across 13 distinct types**",
|
||
|
|
"docsUsed": [
|
||
|
|
"reference/system-architecture.md",
|
||
|
|
"reference/helm/index.md",
|
||
|
|
"reference/helm/charts/app-common-apps-provider-addons-charts-state-cluster-cassandra.md"
|
||
|
|
],
|
||
|
|
"score": {
|
||
|
|
"clarity": 5,
|
||
|
|
"actionability": 5,
|
||
|
|
"completeness": 5,
|
||
|
|
"findability": 5,
|
||
|
|
"notes": "Reader extracted complete and accurate answer matching ground truth exactly. Documentation clearly enumerated all 13 resource types with correct counts. Cassandra-specific chart documentation path is intuitive location for this information. Information is immediately actionable for deployment planning and verification."
|
||
|
|
},
|
||
|
|
"composite": 100,
|
||
|
|
"notFound": false
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"id": "resource-jenkins-001",
|
||
|
|
"category": "resource",
|
||
|
|
"difficulty": "medium",
|
||
|
|
"question": "What Kubernetes resource types does the jenkins chart generate? List each type and count.",
|
||
|
|
"groundTruth": "ConfigMap: 5\nService: 3\nSecret: 3\nIngress: 2\nServiceAccount: 2\nPersistentVolumeClaim: 1\nSecurityGroupPolicy: 1\nPrometheusRule: 1\nBackendConfig: 1\nNetworkPolicy: 1\nPodDisruptionBudget: 1\nPodMonitoring: 1\nRoute: 1\nServiceMonitor: 1\nStatefulSet: 1\nRole: 1\nRoleBinding: 1\nSecretClaim: 1\nPod: 1",
|
||
|
|
"humanAnswer": "Based on the Jenkins chart documentation, here are the Kubernetes resource types it generates:\n\n| Resource Type | Count |\n|---|---|\n| ConfigMap | 5 |\n| Service | 3 |\n| Secret | 3 |\n| Ingress | 2 |\n| ServiceAccount | 2 |\n| PersistentVolumeClaim | 1 |\n| SecurityGroupPolicy | 1 |\n| PrometheusRule | 1 |\n| BackendConfig | 1 |\n| NetworkPolicy | 1 |\n| PodDisruptionBudget | 1 |\n| PodMonitoring | 1 |\n| Route | 1 |\n| ServiceMonitor | 1 |\n| StatefulSet | 1 |\n| Role | 1 |\n| RoleBinding | 1 |\n| SecretClaim | 1 |\n| Pod | 1 |\n\n**Total: 29 resources** (which matches the resource count listed in the chart index)",
|
||
|
|
"docsUsed": [
|
||
|
|
"reference/system-architecture.md",
|
||
|
|
"reference/helm/index.md",
|
||
|
|
"reference/helm/charts/app-common-charts-jenkins.md"
|
||
|
|
],
|
||
|
|
"score": {
|
||
|
|
"clarity": 5,
|
||
|
|
"actionability": 5,
|
||
|
|
"completeness": 5,
|
||
|
|
"findability": 5,
|
||
|
|
"notes": "Documentation provided complete resource inventory with all 19 types and accurate counts. Reader extracted information into clear tabular format. File structure (reference/helm/charts/app-common-charts-jenkins.md) places content exactly where engineers expect Jenkins chart details. Information is immediately actionable for infrastructure planning and RBAC configuration."
|
||
|
|
},
|
||
|
|
"composite": 100,
|
||
|
|
"notFound": false
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"id": "cross-sub-001",
|
||
|
|
"category": "cross-subsystem",
|
||
|
|
"difficulty": "hard",
|
||
|
|
"question": "Which subsystems depend on app-tools, and how heavily (by call+import count)?",
|
||
|
|
"groundTruth": "ipam-tools: 6\ncompute-tools: 4\napp-common: 1",
|
||
|
|
"humanAnswer": "Based on the Cross-Subsystem Dependencies table in system-architecture.md:\n\n**Subsystems depending on app-tools:**\n\n1. **ipam-tools** — 6 calls, 0 imports (6 total)\n2. **compute-tools** — 4 calls, 0 imports (4 total)\n3. **app-common** — 1 call, 0 imports (1 total)\n\nAll dependencies are call-based (service boundary), not import-based. The app-tools.md documentation confirms this pattern: downstream subsystems delegate resource provisioning and credential creation to app-tools rather than re-implementing those operations.",
|
||
|
|
"docsUsed": [
|
||
|
|
"reference/system-architecture.md",
|
||
|
|
"reference/helm/index.md",
|
||
|
|
"reference/subsystems/account-common.md",
|
||
|
|
"reference/subsystems/app-tools.md",
|
||
|
|
"reference/subsystems/compute-common.md",
|
||
|
|
"reference/subsystems/compute-tools.md",
|
||
|
|
"reference/subsystems/ipam-core.md"
|
||
|
|
],
|
||
|
|
"score": {
|
||
|
|
"clarity": 4,
|
||
|
|
"actionability": 5,
|
||
|
|
"completeness": 5,
|
||
|
|
"findability": 4,
|
||
|
|
"notes": "Reader extracted exact ground truth numbers with supporting detail about call vs import distinction. Information was present and findable in system-architecture.md, though verification required consulting multiple subsystem files. Documentation clearly supports architectural decision-making."
|
||
|
|
},
|
||
|
|
"composite": 90,
|
||
|
|
"notFound": false
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"id": "cross-sub-002",
|
||
|
|
"category": "cross-subsystem",
|
||
|
|
"difficulty": "easy",
|
||
|
|
"question": "Which subsystems are identified as cross-cutting concerns?",
|
||
|
|
"groundTruth": "root",
|
||
|
|
"humanAnswer": "**root**\n\nAccording to the documentation, only one subsystem is identified as a cross-cutting concern:\n\n- **root** — described as a single-file configuration anchor that establishes the shared compilation context and path alias space for all other subsystems. It's listed in the \"Cross-Cutting Concerns\" section of the system architecture and marked with Kind: cross-cutting in its subsystem table entry.",
|
||
|
|
"docsUsed": [
|
||
|
|
"reference/system-architecture.md",
|
||
|
|
"reference/subsystems/account-common.md",
|
||
|
|
"reference/subsystems/app-tools.md",
|
||
|
|
"reference/subsystems/compute-common.md",
|
||
|
|
"reference/subsystems/compute-tools.md",
|
||
|
|
"reference/subsystems/control-core.md",
|
||
|
|
"reference/subsystems/ipam-core.md",
|
||
|
|
"reference/subsystems/ipam-tools.md",
|
||
|
|
"reference/subsystems/network-common.md",
|
||
|
|
"reference/subsystems/root.md"
|
||
|
|
],
|
||
|
|
"score": {
|
||
|
|
"clarity": 5,
|
||
|
|
"actionability": 4,
|
||
|
|
"completeness": 5,
|
||
|
|
"findability": 5,
|
||
|
|
"notes": "Reader correctly identified root as the sole cross-cutting concern with precise reasoning. Documentation clearly marks it in a dedicated section and explains its role as a configuration anchor. Subsystem organization and explicit Kind labeling make discovery straightforward. Actionability slightly lower than maximum because the description focuses on what root is rather than how to interact with it, though this may be appropriate for architectural documentation."
|
||
|
|
},
|
||
|
|
"composite": 95,
|
||
|
|
"notFound": false
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"id": "arch-001",
|
||
|
|
"category": "architectural",
|
||
|
|
"difficulty": "hard",
|
||
|
|
"question": "The following subsystems have 0 detected functions and 0 modules: account-common, network-common, network-core, root. Why might this be the case, and what do they actually contain?",
|
||
|
|
"groundTruth": "These subsystems primarily contain Helm charts with Go-templated YAML, Terraform HCL, and Crossplane compositions. The code analysis pipeline detects functions/modules from Python, Go, TypeScript, and shell scripts — but Helm templates use Go template syntax ({{ }}) which doesn't produce traditional function/module entities. Their content is captured through the Helm chart extraction phase instead.",
|
||
|
|
"humanAnswer": "These four subsystems have zero functions and modules because they are **intentionally declarative rather than executable**. They encode infrastructure state and configuration rather than business logic.\n\n**account-common** contains infrastructure-as-code artifacts for account-level provisioning — Helm charts, Terraform modules, and configuration files for account creation and quota requests. It exists as a self-contained operational unit consumed out-of-band by deployment pipelines, not imported by application code.\n\n**network-common** is purely declarative configuration — network topology definitions, shared schemas, IaC templates, and static policy files. It defines desired network state across AWS, Azure, and GCP through Helm charts and Crossplane manifests, with no executable logic.\n\n**network-core** (mentioned in the architecture overview) follows the same pattern as the other zero-function subsystems: it encodes infrastructure state as IaC artifacts, schemas, or policy definitions rather than executable logic.\n\n**root** is a single-file configuration anchor — likely a `tsconfig.json` or module resolution manifest — that establishes the shared compilation context and path alias space for the entire system. It exerts influence through the build toolchain, not through the module graph, so it contains no runtime logic or exports.\n\nThe architecture document explicitly states this is intentional: \"The zero-function subsystems are intentionally declarative: they encode infrastructure state as IaC artifacts, schemas, or policy definitions rather than executable logic, which is why `control-core`'s 336 files carry no functions despite its size.\" This design trades encapsulation for auditability — keeping configuration as inert files makes the infrastructure surface area inspectable without executing code.",
|
||
|
|
"docsUsed": [
|
||
|
|
"reference/system-architecture.md",
|
||
|
|
"reference/subsystems/account-common.md",
|
||
|
|
"reference/subsystems/app-tools.md",
|
||
|
|
"reference/subsystems/compute-common.md",
|
||
|
|
"reference/subsystems/compute-tools.md",
|
||
|
|
"reference/subsystems/control-core.md",
|
||
|
|
"reference/subsystems/ipam-core.md",
|
||
|
|
"reference/subsystems/ipam-tools.md",
|
||
|
|
"reference/subsystems/network-common.md",
|
||
|
|
"reference/subsystems/root.md"
|
||
|
|
],
|
||
|
|
"score": {
|
||
|
|
"clarity": 5,
|
||
|
|
"actionability": 5,
|
||
|
|
"completeness": 4,
|
||
|
|
"findability": 5,
|
||
|
|
"notes": "Reader provided a well-structured, domain-focused explanation that immediately identifies the core reason (declarative vs executable) and backs it with specific content examples for each subsystem. The architectural rationale (auditability tradeoff) is clearly stated. Minor gap: doesn't explicitly mention the code analysis pipeline's language limitations, though this is a technical implementation detail rather than a domain concern. The answer is immediately actionable for engineers needing to understand how these subsystems fit into the architecture."
|
||
|
|
},
|
||
|
|
"composite": 95,
|
||
|
|
"notFound": false
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"id": "arch-002",
|
||
|
|
"category": "architectural",
|
||
|
|
"difficulty": "easy",
|
||
|
|
"question": "What is the current version and appVersion of the mdm-app Helm chart?",
|
||
|
|
"groundTruth": "version: 0.7.4, appVersion: 2025.12-12-develop",
|
||
|
|
"humanAnswer": "\n\n**Version:** 0.7.4 \n**App Version:** 2025.12-12-develop",
|
||
|
|
"docsUsed": [
|
||
|
|
"reference/system-architecture.md",
|
||
|
|
"reference/helm/index.md",
|
||
|
|
"reference/helm/charts/app-common-charts-mdm-app.md"
|
||
|
|
],
|
||
|
|
"score": {
|
||
|
|
"clarity": 5,
|
||
|
|
"actionability": 5,
|
||
|
|
"completeness": 5,
|
||
|
|
"findability": 5,
|
||
|
|
"notes": "Reader extracted exact version numbers correctly and presented them in clear, labeled format. Documentation was organized logically—Helm chart versions naturally belong in the dedicated mdm-app chart reference file. All requested information was present and immediately actionable for deployment or reference purposes."
|
||
|
|
},
|
||
|
|
"composite": 100,
|
||
|
|
"notFound": false
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"id": "scenario-001",
|
||
|
|
"category": "scenario",
|
||
|
|
"difficulty": "hard",
|
||
|
|
"question": "If you need to rotate the \"vault-secret\" shared secret, which Helm charts would be affected and need redeployment?",
|
||
|
|
"groundTruth": "ai-app, cleanse-app, pms-app, rdm-app, shared-app",
|
||
|
|
"humanAnswer": "**ai-app, cleanse-app, pms-app, rdm-app, shared-app**\n\nThese five charts are listed in the \"Shared Secrets & ConfigMaps\" section as consumers of the `vault-secret` shared secret, so they would all require redeployment when the secret is rotated.",
|
||
|
|
"docsUsed": [
|
||
|
|
"reference/system-architecture.md",
|
||
|
|
"reference/helm/index.md"
|
||
|
|
],
|
||
|
|
"score": {
|
||
|
|
"clarity": 5,
|
||
|
|
"actionability": 5,
|
||
|
|
"completeness": 5,
|
||
|
|
"findability": 5,
|
||
|
|
"notes": "Reader extracted correct answer with clear reasoning, citing specific section. Documentation explicitly lists all five chart consumers of vault-secret in expected reference location. Information is concrete and immediately actionable for redeployment decisions."
|
||
|
|
},
|
||
|
|
"composite": 100,
|
||
|
|
"notFound": false
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"id": "scenario-002",
|
||
|
|
"category": "scenario",
|
||
|
|
"difficulty": "hard",
|
||
|
|
"question": "A new engineer needs to deploy the MDM application stack. What charts, configuration values, and external dependencies should they understand first?",
|
||
|
|
"groundTruth": "Chart: mdm-app (app-common/charts/mdm-app), Version: 0.7.4\nKey values: clusterName, environmentType, arch, cloudProvider, region, vaultEnv, imagePullPolicy, clusterDomain, terminationGracePeriodSeconds, etalonsubDomain (+28 more)\nResources generated: 22 K8s resources\nInteractions: none detected",
|
||
|
|
"humanAnswer": "To deploy the MDM application stack, focus on these areas:\n\n**Primary Chart**\n- `mdm-app` at `app-common/charts/mdm-app` (v0.7.4, AppVersion 2025.12-12-develop)\n- Deploys 22 Kubernetes resources\n- Requires 38 configuration values\n\n**Configuration Values**\nThe chart expects 38 values keys. The documentation doesn't enumerate them individually, but based on the subsystem structure, you'll need to provide cluster context, resource sizing, and application-specific settings. Check the chart's `values.yaml` directly for the full schema.\n\n**External Dependencies & Prerequisites**\nFrom `app-tools`, MDM deployment depends on upstream provisioning:\n- RDM companion setup (`get_rdm_mdm_companions`, `patch_crossaccount_mdm_companions`)\n- IAM policy patching for cross-account MDM companions (`patch_mdm_policy_rdm_companion_arn`)\n- DynamoDB shared API configuration (`patch_mdm_systemdynamodb_shared_devops_api`, `patch_shared_devops_api_mdm_arns`)\n- These are handled by `app-tools` provisioning functions, so ensure prerequisite infrastructure is created first\n\n**Subsystem Context**\n- MDM is part of `app-common`, which is a configuration-heavy subsystem (934 files, 28 functions)\n- It depends on `root` for shared configuration and environment constants\n- No explicit Helm chart dependencies are declared, but it integrates with the broader app platform\n\n**Next Steps**\nReview the mdm-app chart's `values.yaml` at `app-common/charts/mdm-app/values.yaml` for the specific 38 configuration keys, and ensure `app-tools` provisioning (particularly RDM companion and DynamoDB setup) completes before deploying the chart.",
|
||
|
|
"docsUsed": [
|
||
|
|
"reference/system-architecture.md",
|
||
|
|
"reference/helm/index.md",
|
||
|
|
"reference/subsystems/account-common.md",
|
||
|
|
"reference/subsystems/app-tools.md",
|
||
|
|
"reference/subsystems/compute-common.md",
|
||
|
|
"reference/subsystems/compute-tools.md",
|
||
|
|
"reference/subsystems/ipam-core.md"
|
||
|
|
],
|
||
|
|
"score": {
|
||
|
|
"clarity": 4,
|
||
|
|
"actionability": 3,
|
||
|
|
"completeness": 3,
|
||
|
|
"findability": 3,
|
||
|
|
"notes": "Answer clearly identifies the chart, version, and external dependencies with good structure. However, it explicitly defers the 38 configuration values to values.yaml rather than enumerating the specific keys (clusterName, environmentType, arch, cloudProvider, region, vaultEnv, imagePullPolicy, clusterDomain, terminationGracePeriodSeconds, etalonsubDomain, etc.) that should be documented upfront. This creates a completeness gap—a new engineer still needs to reverse-engineer the schema. External dependencies are inferred from app-tools but lack explicit documentation of their necessity. Findability is adequate for architecture context but insufficient for the specific configuration schema a deployer needs first."
|
||
|
|
},
|
||
|
|
"composite": 65,
|
||
|
|
"notFound": false
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"id": "scenario-003",
|
||
|
|
"category": "scenario",
|
||
|
|
"difficulty": "medium",
|
||
|
|
"question": "What network ports does the Cassandra chart expose, and what are they typically used for?",
|
||
|
|
"groundTruth": "Ports: 7000, 7001, 7199, 9042, 9160\n7000: Cassandra inter-node cluster communication\n7001: Cassandra SSL inter-node\n7199: JMX monitoring\n9042: CQL native transport (client connections)\n9160: Thrift client API (legacy)",
|
||
|
|
"humanAnswer": "The Cassandra chart exposes five network ports via the StatefulSet:\n\n- **7000** - Cassandra inter-node communication (cluster gossip)\n- **7001** - Cassandra inter-node communication (TLS)\n- **7199** - JMX monitoring port\n- **9042** - Cassandra native protocol (client connections)\n- **9160** - Cassandra Thrift protocol (legacy client connections)\n\nHowever, the documentation only lists the ports themselves without explaining their purposes. The typical uses I've described are standard Cassandra port conventions, but the provided docs don't explicitly document what each port is used for.",
|
||
|
|
"docsUsed": [
|
||
|
|
"reference/system-architecture.md",
|
||
|
|
"reference/helm/index.md",
|
||
|
|
"reference/helm/charts/app-common-apps-provider-addons-charts-state-cluster-cassandra.md"
|
||
|
|
],
|
||
|
|
"score": {
|
||
|
|
"clarity": 2,
|
||
|
|
"actionability": 2,
|
||
|
|
"completeness": 2,
|
||
|
|
"findability": 4,
|
||
|
|
"notes": "Ports are listed but purposes are undocumented. Reader had to rely on external Cassandra conventions rather than the docs themselves. Information is in the right file but lacks explanatory context needed for engineers unfamiliar with Cassandra internals."
|
||
|
|
},
|
||
|
|
"composite": 50,
|
||
|
|
"notFound": false
|
||
|
|
}
|
||
|
|
]
|
||
|
|
}
|