# AWP Quality and CI Planning Design **Date:** 2026-08-19 **Status:** Working product-design baseline; approved direction, not yet implementation-grade spec. **Purpose:** Define how AWP plans testing and CI without blanket AI-generated tests, while supporting Simple/Expert Planning participation and enforcing evidence-driven verification. ## 1. Product position AWP owns the **quality strategy and CI semantics** for a Plan. It does not equate quality with adding more tests or running more commands. The planner must answer: ```text What can fail? What user/business behavior matters most? What evidence is required before merge/release/deploy? What is the smallest reliable test/check that can falsify this change? What testing is unnecessary, duplicated, or too expensive for the evidence it gives? How will failures be diagnosed and acted on? ``` Default: **risk-, behavior-, and evidence-driven verification**, not a fixed test-count target, universal coverage percentage, or generic unit/integration/E2E checklist. ## 2. Industry-grounded baseline AWP should recommend these principles unless Project requirements justify a different policy: 1. Use small/fast tests where they provide useful isolation. 2. Verify real integration boundaries directly. 3. Use E2E selectively for **Critical User Journeys (CUJs)** and outcomes smaller tests cannot prove. 4. Prefer externally observable behavior over implementation-detail assertions. 5. Select non-functional testing from actual product risk. 6. Optimize speed, maintainability, utilization, reliability and fidelity instead of maximizing one test type. 7. Treat code coverage as diagnostic evidence, not correctness proof. 8. Feed escaped defects back into the cheapest reliable earlier catcher. 9. Put each check at the earliest reliable enforcement point. 10. Trust green only when expected-vs-executed completeness and failure behavior are themselves proven. Do not encode fixed ratios such as 70/20/10 as universal policy. ## 3. User-journey-first functional testing For user-facing products, identify CUJs during Experience Definition and carry them into Quality Planning. Example: ```text Goal: publish a translated article sign in open article request translation review result publish verify public localized page ``` A journey test proves meaningful behavior. Smoke checks such as "button exists", "route returns 200", or "modal opens" do not substitute for feature correctness. Smoke tests remain narrow operability checks: ```text application boots deployment responds critical dependency reachable basic auth path operational ``` ## 4. Verification discipline Do not confuse exhaustive activity with verification. ```text CHANGE | v What can actually detect a mistake in this change? | +-- local logic -------> focused unit/regression test +-- types/contracts ---> targeted typecheck +-- formatting/static -> targeted lint/static rule +-- integration -------> focused integration/contract test +-- user flow ---------> focused E2E/browser journey | v Evidence sufficient? | | yes no | | STOP broaden carefully ``` Rules: - Test changed behavior, not the whole application by default. - Prefer the narrowest meaningful verification first. - Start with targeted static checks and affected-package tests. - Bug fixes should have regression evidence when practical. - New behavior should have focused evidence for the behavior introduced. - Prefer tests that would fail before the change and pass after it. - Do not repeatedly run broad suites/builds/smoke tests as a substitute for reasoning. - Do not run every available test merely because it exists. - Stop re-testing a property once sufficient valid evidence exists unless a concrete input/risk changed. - Do not add redundant tests merely to increase test count/coverage. - Do not change production code to satisfy irrelevant tests. - Do not weaken/delete/skip meaningful failing tests merely to get green. Broaden only for recorded reasons such as: ```text targeted failure exposed wider problem shared infrastructure/public contract changed wide dependency impact required completion gate final pre-merge qualification release/deployment qualification ``` ## 5. Test-level selection Choose the smallest boundary with sufficient fidelity: ```text pure business rule -> unit/regression module + real dependency -> integration API/schema/provider compatibility -> contract/integration critical user outcome -> E2E journey security control -> targeted security/standard-derived check performance requirement -> smallest realistic performance/load check failure/recovery -> fault-injection/recovery ``` Do not duplicate the same assertion across layers without a reason. ## 6. Quality planning state ```text QualityStrategy quality goals CriticalUserJourneys QualityRisks QualityRequirements verification-selection rules test-level recommendations non-functional requirements accepted gaps/rationale CIPlan CIInvariants Checks required/optional/advisory classification trigger/stage affected-scope rules escalation/broadening rules environment/data requirements evidence/receipt requirements failure policy runtime/resource budget ``` `QualityStrategy` says what confidence is required and why. `CIPlan` says how it is continuously enforced. Repository tests/workflows/provider check IDs are implementation/evidence, not planning truth. ## 7. CI stages Put each check at the earliest reliable enforcement point: ```text write/commit focused cheap checks pre-merge affected correctness + explicit completion gate post-merge integration/delivery preparation pre-release release-specific qualification pre-deploy artifact/environment checks post-deploy health/synthetic/CUJ where justified scheduled expensive compatibility/security/load maintenance suites ``` A check that is not on the path to the action it protects enforces nothing. ## 8. Simple and Expert Planning behavior Quality/CI is always planned. Participation mode changes whether the user is interviewed about it. ### Simple AWP resolves QualityStrategy and CIPlan from: ```text ProjectPlanningDefaults / ProjectDeliveryProfile repository evidence CUJs architecture/security risk industry-grounded defaults runtime/cost constraints ``` The plan becomes visible in compact technical rows, for example: ```text Quality CUJ-first + affected verification project default CI GitHub Actions + ARC inherited ``` The user is **not** asked `Use this plan?` on every Plan merely because a technical strategy exists. A compact informational update is sufficient: ```text Technical plan updated Quality and CI were resolved from Project defaults. No action required. [Review] ``` User involvement is promoted only when Quality/CI creates an OwnerRequired/PolicyRequired consequence, such as: ```text release-risk tolerance customer/browser compatibility promise regulatory/security commitment CI spend/latency budget beyond policy manual acceptance requirement ``` ### Expert The integrated recommendation becomes a conversational review: ```text Quality & CI recommendation Functional focused regressions for changed behavior integration tests for real boundaries E2E for critical user journeys smoke only for boot/deploy health Verification policy narrowest meaningful check first affected-scope CI during iteration full repository gate once at final pre-merge qualification CI GitHub Actions + ARC ephemeral runners required pre-merge invariants heavier scheduled suites only when justified [ Use recommended plan ] Review or customize ``` `Review or customize` opens only the relevant section. Expert remains recommendation-first, not a blank test-strategy questionnaire. ## 9. Recommendation provenance Material Quality/CI recommendations carry: ```text confidence basis material consequences what would change the recommendation ``` A low-confidence technical choice remains visible. In Simple mode it does not automatically become a chat question unless its consequence crosses into user-owned/policy territory. ## 10. First Expert Plan -> Project defaults A first Expert Plan may establish reusable Project quality/CI defaults. Default-eligible examples: ```text journey-test emphasis verification philosophy CI provider/runner model completion-gate policy pre-merge latency budget coverage policies manual/exploratory expectations required non-functional standards full-gate frequency ``` AWP explicitly proposes `Use as project defaults`. Plan-specific test requirements are excluded. A future Plan exception must not silently rewrite Project defaults. ## 11. Full Quality/CI interview topics Only ask what the Plan/repository does not already answer: ```text critical user/business journeys risk/release criticality supported platforms/browser/device promises API/backward-compatibility promises data/migration risk security/compliance accessibility target performance budgets fault tolerance/recovery external dependency behavior test runtime/cost budget acceptable pre-merge latency manual/exploratory expectations ``` ## 12. CI plan quality rules Carry forward these provider-independent rules: ```text pre-merge protects merge; post-merge is not the safety net prove every new gate can fail before trusting green measure from machine-readable evidence verify expected vs executed work so missing shards/tests cannot fake green preserve required receipts/evidence split long suites into dispatchable units before inventing schedulers choose concurrency from real resource/timing constraints flakiness is a defect in the signal place each rule at its cheapest reliable enforcement tier valid evidence prevents redundant recomputation ``` GitHub Actions/ARC are mechanics, not AWP's domain model. ## 13. Enforcement, not agent advice The methodology is enforced by AWP: ```text ChangeSet -> ChangeClassifier -> affected packages/contracts/CUJs -> QualityStrategy -> VerificationPlanner -> VerificationPlan -> trusted execution / accepted CI -> EvidenceLedger -> sufficient | broaden | blocked | complete ``` Working concepts: ```text VerificationPlan VerificationStep VerificationEvidence VerificationEscalation CompletionGate ``` Rules: - Agents can request verification; they do not declare lifecycle verification complete. - Broad/redundant checks require an admitted machine-readable reason. - Smoke cannot satisfy regression/behavior requirements unless operability-only. - A full-suite pass cannot replace a required focused regression. - Evidence is invalidated only when relevant inputs change. - Unaffected evidence is reused across Task/Review/Completion/Merge/Release/Deployment gates. - Final completion gates are deliberate lifecycle checkpoints, not per-edit rituals. - AWP records why each test ran and what property it proved. ## 14. Evidence sufficiency and STOP Every VerificationStep names the property it is meant to falsify. ```text Property invalid auth session cannot mutate Project Evidence focused authz regression targeted typecheck Result pass; no shared auth contract changed Decision property established STOP ``` Testing more is not inherently safer. Extra verification needs a reason and consumes explicit runtime/CI budget. ## 15. Coverage policy Track separately: ```text code coverage changed-code coverage behavior/CUJ coverage feature/requirement coverage risk/control coverage platform/browser/environment coverage ``` No universal 100% target; no generated tests solely to raise a percentage. ## 16. Flakiness and trust Define retry, flaky classification, owner, quarantine and expiry. A quarantined/flaky test normally cannot satisfy a required invariant. Retries may diagnose nondeterminism; they must not silently convert flakiness into proof. ## 17. Test data/environment planning Plan enough before suites are designed: ```text hermetic vs external dependencies fixtures/reference data per-worker isolation production-like schema/database needs third-party stubs/contracts secret handling cleanup/reset PII/privacy browser/device/platform matrix ``` Bad test-data architecture is part of Quality Planning, not a testing afterthought. ## 18. Conditional non-functional testing Select from risk: ```text security accessibility performance/load fault tolerance/recovery migration/rollback backup/restore privacy localization compatibility upgrade/backward compatibility usability/manual exploratory evaluation ``` Do not run all categories for every project. ## 19. Affected testing example ```text CUJ-03 Publish translated article Docs-only ChangeSet -> CUJ unaffected -> do not run Translation-local domain change -> focused domain regression -> browser journey only if boundary/journey contract affected Shared auth/session change -> authenticated CUJ impact broadens -> broader integration/E2E justified ``` Optimization may remove irrelevant work; it may not weaken required invariants. ## 20. Feedback loop after release ```text escaped defect / incident -> classify missed behavior/risk -> identify cheapest reliable earlier catcher -> propose QualityStrategy/CIPlan revision -> normal review/Project-default decision if broadly reusable ``` The answer is not automatically another E2E test. ## 21. Planning workspace placement ```text DEFINE DESIGN SPECIFY DELIVER Technical delivery summary Quality CI Release/CD Deployment Execution LAUNCH ``` Simple mode collapses Quality/CI into readable technical summaries while keeping attention states visible. Expert mode expands the same state into detailed review. ## 22. Open specification questions ```text Q1 CriticalUserJourney identity vs typed artifact. Q2 Exact QualityStrategy / CIPlan / VerificationPlan relationships. Q3 Recommendation scoring across risk/fidelity/runtime/maintainability/cost. Q4 Project/global quality preference/default schema. Q5 Existing repository audit/mapping into QualityStrategy. Q6 Versioned non-functional standards/catalogs. Q7 Accepted quality-gap/risk UX. Q8 Evidence cache/invalidation identity. Q9 Admission response for explicitly requested broad redundant gates. Q10 Simple-mode escalation when a technical quality choice creates business/policy consequences. ```