dd0c/products/05-aws-cost-anomaly/test-architecture/review.md

dd0c/cost — Test Architecture Review

Reviewer: TDD Consultant (Manual Review) Date: March 1, 2026 Verdict: 🔴 NEEDS SIGNIFICANT WORK — This is the weakest test architecture of all 6 products.

---

1. Coverage Analysis

This document is 232 lines. For comparison, P1 (route) is 2,241 lines and P6 (run) is 1,762 lines. The coverage gaps are massive.

Epic	Coverage Status	Notes
Epic 1: CloudTrail Ingestion	⚠️ Partial	Section 3.1 has 5 test cases for the normalizer. Missing: SQS FIFO deduplication tests, DLQ retry behavior, EventBridge cross-account rule tests, S3 raw event archival. Story 1.2 (SQS + DLQ) has zero dedicated tests.
Epic 2: Anomaly Detection	✅ Decent	Section 3.2 covers Z-score, novelty, cold-start. But missing: composite score weighting tests, edge cases (zero stddev, negative costs, NaN handling), baseline maturity transition tests.
Epic 3: Zombie Hunter	❌ Missing	Zero test cases. The daily scan for idle/stopped resources that are still costing money has no tests at all.
Epic 4: Notification & Remediation	⚠️ Thin	Section 4.2 has 3 integration tests for cross-account actions. Missing: Slack Block Kit formatting tests, daily digest aggregation, snooze/dismiss logic, interactive payload signature validation.
Epic 5: Onboarding & PLG	❌ Missing	Zero test cases. CloudFormation template generation, Stripe billing, free tier enforcement — none tested.
Epic 6: Dashboard API	❌ Missing	Zero test cases. REST API endpoints, tenant isolation, query performance — nothing.
Epic 7: Dashboard UI	❌ Missing	Zero test cases.
Epic 8: Infrastructure (CDK)	❌ Missing	Zero test cases. No CDK snapshot tests, no infrastructure drift detection (ironic).
Epic 9: Multi-Account Management	❌ Missing	Zero test cases. Account linking, bulk scanning, cross-account permissions — nothing.
Epic 10: Transparent Factory	🔴 Skeletal	Section 8 has exactly 3 test cases total across 2 of 5 tenets. Elastic Schema, Cognitive Durability, and Semantic Observability have zero tests.

Bottom line: 5 of 10 epics have zero test coverage in this document. This is a skeleton, not a test architecture.

---

2. TDD Workflow Critique

The philosophy in Section 1 is sound — "test the math first" is correct for an anomaly detection product. But the execution is incomplete:

• The "strict TDD" list correctly identifies scoring and governance as test-first. Good.
• The "integration tests lead" for CloudTrail ingestion is acceptable.
• Missing: No guidance on testing the Welford algorithm implementation. This is a numerical algorithm with known floating-point edge cases (catastrophic cancellation with large values). The test architecture should mandate property-based testing (e.g., fast-check) for the baseline calculator, not just 3 example-based tests.
• Missing: No guidance on testing the 14-day auto-promotion state machine. This is a time-dependent state transition that needs fake clock testing.

---

3. Test Pyramid Balance

The 70/20/10 ratio is stated but not justified. For dd0c/cost:

• Unit tests should be higher (80%) — the anomaly scoring engine is pure math. It should have exhaustive property-based tests, not just 50 example tests.
• Integration tests (15%) — DynamoDB Single-Table patterns, EventBridge→SQS→Lambda pipeline, cross-account STS.
• E2E (5%) — two journeys is fine for V1, but they need to be more detailed.

The current Section 6 (Performance) has exactly 2 test cases. For a product that processes CloudTrail events at scale, this is dangerously thin.

---

4. Anti-Patterns

1. Section 3.3 — Welford Algorithm: Only 3 tests for a numerical algorithm. This is the "happy path only" anti-pattern. Missing: what happens when stddev is 0 (division by zero in Z-score)? What happens with a single data point? What happens with extremely large values (float overflow)?

2. Section 4.1 — DynamoDB Transaction Test: "writes CostEvent and updates Baseline in single transaction" — this tests the happy path. Where's the test for transaction failure? DynamoDB transactions can fail due to conflicts, and the system must handle partial writes.

3. Section 5 — E2E Journeys: Journey 2 tests "Stop Instance" remediation but doesn't test what happens when the customer's IAM role has been revoked between alert and remediation click. This is a real-world race condition.

4. No negative tests anywhere. What happens when CloudTrail sends malformed JSON? What happens when the pricing table doesn't have the instance type? (Section 3.1 mentions "fallback pricing" but there's only 1 test for it.)

---

5. Transparent Factory Gaps

Section 8 is the biggest problem. It has 3 test cases across 2 tenets. Here's what's missing:

Atomic Flagging (1 test → needs ~10)

• Missing: flag default state (off), flag TTL enforcement, flag owner metadata, local evaluation (no network calls), CI block on expired flags, multiple concurrent flags.
• The single circuit breaker test uses ">10 alerts/hour" but Epic 10.1 specifies ">3x baseline" — inconsistency.

Elastic Schema (0 tests → needs ~8)

• Zero tests. Need: migration lint (no DROP/RENAME/TYPE), additive-only DynamoDB attribute changes, V1 code ignoring V2 attributes, sunset date enforcement, dual-write during migration.

Cognitive Durability (0 tests → needs ~5)

• Zero tests. Need: decision log schema validation, CI enforcement for scoring PRs, cyclomatic complexity gate, decision log presence check.

Semantic Observability (0 tests → needs ~8)

• Zero tests. Need: OTEL span emission on every anomaly scoring decision, span attributes (cost.anomaly_score, cost.z_score, cost.baseline_days), PII protection (account ID hashing), fast-path span attributes.

Configurable Autonomy (2 tests → needs ~8)

• The 14-day auto-promotion tests are good but incomplete. Missing: panic mode activation (<1s), panic mode stops all alerting, per-account governance override, policy decision logging, governance drift monitoring.

---

6. Performance Test Gaps

Section 6 has 2 tests. For a real-time cost monitoring product, this is inadequate:

• Missing: Burst ingestion (what happens when 1000 CloudTrail events arrive in 1 second during an auto-scaling event?)
• Missing: Baseline calculation performance with 90 days of historical data per account
• Missing: Anomaly scoring latency under concurrent multi-account evaluation
• Missing: DynamoDB hot partition detection (all events for one account hitting the same partition key)
• Missing: SQS FIFO throughput limits (300 msg/s per message group — what happens when a large account exceeds this?)
• Missing: Lambda cold start impact on end-to-end latency

---

7. Missing Test Scenarios

Security

• CloudTrail event forgery: What if someone sends fake CloudTrail events to the EventBridge bus? HMAC/signature validation?
• Slack interactive payload signature: Slack sends a signing secret with interactive payloads. No test validates this.
• Cross-account IAM role revocation: Customer revokes the dd0c role between alert and remediation click.
• Remediation authorization: Who can click "Terminate"? No RBAC tests for remediation actions.

Data Integrity

• CloudTrail event deduplication: CloudTrail can send duplicate events. SQS FIFO dedup is mentioned in Epic 1.2 but has zero tests.
• Baseline corruption recovery: What if a DynamoDB write partially fails and corrupts the running mean/stddev? No recovery tests.
• Pricing table staleness: Static pricing tables will become stale. No test validates that the system handles unknown instance types gracefully beyond the single "fallback pricing" test.
• Cost calculation precision: Floating-point arithmetic on money. No tests for rounding behavior or currency precision.

Operational

• DLQ overflow: What happens when the DLQ fills up? No backpressure tests.
• Multi-tenant isolation: No tests ensuring one tenant's anomalies don't leak to another tenant's Slack channel.
• Account onboarding race condition: What if CloudTrail events arrive before the account is fully onboarded?

---

8. Top 5 Recommendations (Prioritized)

1. Expand to cover all 10 epics. 5 epics have zero tests. At minimum, add unit test stubs for Zombie Hunter (Epic 3), Onboarding (Epic 5), and Dashboard API (Epic 6). These are customer-facing features.

2. Rewrite Section 8 (Transparent Factory) from scratch. 3 tests across 2 tenets is unacceptable. Every tenet needs 5-10 tests. The Elastic Schema and Semantic Observability sections are completely empty.

3. Add property-based testing for the anomaly math. The Welford algorithm, Z-score calculation, and composite scoring are numerical — they need fast-check or equivalent, not just example-based tests. Test edge cases: zero stddev, single data point, NaN, Infinity, negative costs.

4. Add security tests. Slack payload signature validation, CloudTrail event authenticity, cross-account IAM revocation handling, remediation RBAC. This product executes StopInstances and DeleteDBInstance — security testing is non-negotiable.

5. Expand performance section to 10+ tests. Burst ingestion, baseline calculation at scale, DynamoDB hot partitions, SQS FIFO throughput limits, Lambda cold starts. The current 2 tests give zero confidence in production readiness.

---

This document needs a complete rewrite before it can guide TDD implementation. The scoring engine tests are a good start, but everything else is a placeholder.