# Evidence-backed delivery pilot scorecard - **Pilot window:** 3 completed trials minimum, 5 maximum - **Current progress:** 5/5 trials completed - **Adoption state:** Pilot complete; maintainer chose adopt with revisions on 2026-09-03 - **Source baseline:** [initial pilot report](SOFTWARE_FACTORY_PILOT_2026-09-02.md) - **Decision record:** [Agent Delivery Harness v1 adoption](AGENT_DELIVERY_HARNESS_V1_ADOPTION_2026-09-03.md) ## Measurement rules Record only observable GitHub state or a value explicitly supplied by the author or reviewer. Never infer time, effort, reviewer intent, or evidence usefulness from commit timestamps, comment timestamps, an AI score, or the absence of comments. | Measure | Operational definition | | ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Intake completeness | PR states its purpose, at least one testable acceptance criterion, affected boundaries, verification, and explicit exclusions or `none` before human review begins. | | Reviewer usage | A human answers `yes`, `partly`, or `no` to “Did the evidence help you understand or verify this change?”, with an optional reason. Record whether this was independent review or sole-maintainer self-review; never present self-review as independent evidence. Bot activity does not count. | | Clarification comments | Count human reviewer comments that request missing scope, acceptance, safety, verification, or evidence information. Exclude bot comments, approvals, and implementation findings unrelated to handoff clarity. | | Maintenance effort | Author-reported minutes spent preparing, sanitising, checking, and handing off pilot-specific evidence. Record `not measured` when no timer was used. | | Privacy incidents | Count confirmed disclosures separately from scanner false positives and prevented findings. Never copy sensitive content into this scorecard. | | First-pass verification | Record whether the first complete relevant check set passed. Separate branch regressions, unchanged baseline failures, environment limits, and false positives. | Take one timestamped snapshot when the PR first requests human review and one when the trial closes. A draft with no human response cannot produce a reviewer-usage result, even when it has zero clarification comments. ## Trial register | Trial | PR | Change type | Intake complete | Evidence applicable/generated | Human evidence response | Clarification comments | Maintenance minutes | Confirmed privacy incidents | First-pass verification | Trial state | | ----: | -------------------------------------------------------- | --------------------------- | --------------- | ----------------------------- | ------------------------------------ | ---------------------: | ------------------- | --------------------------: | --------------------------------------------------------------------------------- | ----------------------- | | 1 | [#333](https://github.com/rikster/stratofusion/pull/333) | Repository workflow/tooling | Yes | Yes / local-only package | Partly / sole-maintainer self-review | N/A: self-review | Not measured | 0 | No: an initial redaction defect and a later CASA advisory failure were remediated | Completed; human-merged | | 2 | [#334](https://github.com/rikster/stratofusion/pull/334) | Documentation | Yes | Yes / deterministic output | Yes / sole-maintainer self-review | N/A: self-review | 7 | 0 | Yes: local and hosted checks passed | Completed; human-merged | | 3 | [#336](https://github.com/rikster/stratofusion/pull/336) | Documentation | Yes | Yes / deterministic output | Yes / sole-maintainer self-review | N/A: self-review | 2 | 0 | Yes: local and hosted checks passed | Completed; human-merged | | 4 | [#338](https://github.com/rikster/stratofusion/pull/338) | UI refactor | Yes | Yes / deterministic tests | Not reported / sole maintainer | N/A: self-review | 2 | 0 | No: a test-only case-sensitive query was corrected; hosted checks passed | Completed; human-merged | | 5 | [#340](https://github.com/rikster/stratofusion/pull/340) | UI refactor | Yes | Yes / deterministic tests | Yes / sole-maintainer self-review | N/A: self-review | 2 | 0 | No: a test-only nested-text query was corrected; hosted checks passed | Completed; human-merged | ## Trial 1 evidence Draft snapshot taken on 2026-09-02, followed by a ready-for-review snapshot at `2026-09-02T11:22:38Z` and a close snapshot at `2026-09-02T12:29:54Z`: - The PR body includes purpose, acceptance criteria, boundaries, changed behaviour, exact verification, local evidence instructions, privacy review, rollback, limitations, the hypothesis, and a no-merge/no-deploy statement. - The ignored smoke package contains two inspected 1440x900 images, `manifest.json`, and `report.md`; the helper uploaded nothing. - Local focused tests finished with 18/18 passing after the helper suite exposed and fixed one metadata-redaction defect. - GitHub CI run `33620172786` passed lint/typecheck, both application-test shards, rclone worker checks, the production build, and anonymous security E2E. - CI run `33624471878` passed lint/typecheck, both application-test shards, rclone worker checks, the production build, and anonymous security E2E. CASA and the aggregate check failed before follow-up remediation because the production graph contained two high and one low advisories. - Follow-up remediation pins `browserslist` 4.28.8 and `postcss-selector-parser` 6.1.4 through PNPM overrides. A fresh production audit reported zero advisories, and local CASA, the focused 18-test set, lint/typecheck, and the production build passed. CI run `33626898879` then passed every repository job, including CASA and aggregate `build-and-test`, on remediation commit `459255cc`. - The sole maintainer reported dismissing GitGuardian incident `36837173` as a false positive. A fresh GitGuardian scan then passed on head `0acd16c3`; no confirmed privacy disclosure occurred. - CI run `33628767467` passed every repository job on the closing head: lint/typecheck, both application-test shards, rclone worker checks, the production build, CASA, anonymous security E2E, and aggregate `build-and-test`. - The sole maintainer recorded the evidence as `partly` helpful: structured acceptance criteria, safety boundaries, and exact checks helped verification, while fixture screenshots added little value for this nonvisual change and duplicated report/scorecard maintenance created avoidable CI churn. This is self-review, not independent reviewer evidence. - Clarification-comment measurement is `N/A` because the author and reviewer are the same person. It cannot support a reviewer-comprehension claim. - Author maintenance time was not timed and must remain `not measured`; no estimate was substituted for a timed value. - The author changed the PR from draft to ready for review. This opens the human measurement window; it does not constitute approval or establish that the PR is mergeable. - At that snapshot, head `0a5f6c33` used CI run `33623875572`: lint/typecheck, both test shards, the production build, and worker checks passed; anonymous security E2E remained in progress; CASA retained the documented dependency-baseline failure. - Trial 1 closed without an agent merge or deployment. The sole maintainer subsequently merged PR #333 at `2026-09-02T12:46:10Z`, preserving the pilot's human-authority boundary. ## Trial 2 evidence Trial 2 was explicitly continued by the sole maintainer. Draft PR #334 was opened at `2026-09-02T12:57:26Z` from isolated branch `cdx-gqc007v/pilot-trial-2-streaming-docs`, based on `origin/dev` commit `c817e667`: - Issue #243 is a documentation-only task. Its acceptance criteria require a current SSE/NDJSON decision guide, a source-verified endpoint inventory, lifecycle and fallback behavior, explicit architecture/security boundaries, removal of unsupported claims, and no runtime or dependency change. - Before review, the PR body records purpose, linked issue, testable acceptance criteria, affected and excluded boundaries, exact verification, nonvisual evidence, privacy review, rollback, limitations, and the no-merge/no-deploy handoff. - Screenshots are not applicable because no visual behavior changed. A deterministic PowerShell inventory found all four documented SSE producers and four documented NDJSON producers with their expected transport markers. - `pnpm test src/lib/docs.test.ts src/app/dev/docs/page.test.tsx` passed 3/3 tests; targeted Prettier, `pnpm check`, and `git diff --check` passed. - The changed documentation contains public route names, repository-relative source paths, synthetic examples, and public reference links only. No confirmed privacy incident occurred. - CI run `33632870763` was the first complete hosted check set and passed lint/typecheck, both application-test shards, rclone worker checks, the production build, CASA, anonymous security E2E, and aggregate `build-and-test` on initial documentation head `9c74b7ae`. GitGuardian also passed. - Pilot-specific evidence and handoff maintenance was timed from `2026-09-02T12:56:46Z` through `2026-09-02T13:03:34Z` and recorded as 7 whole minutes. - The ready-for-review snapshot was taken at `2026-09-02T13:04:22Z`. Closing CI run `33633530730` passed every repository job on head `cdaaf38f`, including CASA, GitGuardian, anonymous security E2E, and aggregate `build-and-test`. - At the close snapshot on `2026-09-02T20:27:29Z`, the sole maintainer answered `yes` to whether the evidence helped them understand or verify the change. No reason was supplied. This is a sole-maintainer self-review and does not establish independent reviewer comprehension. - GitHub showed no review objects or comments. Clarification-comment measurement is `N/A` because the author and reviewer are the same person, not zero inferred from silence. - Trial 2 closed without an agent merge or deployment. The sole maintainer merged PR #334 at `2026-09-02T20:27:36Z`, preserving the pilot's human-authority boundary. - Follow-up PR #335 carries the closeout record because PR #334 was human-merged before the final scorecard commits reached its branch. Its first CASA run then detected newly published baseline advisories in unchanged dependency paths: two `qs` findings affected the web and worker graphs, and one `@xmldom/xmldom` finding affected the web graph. The follow-up pins `qs` 6.16.0 and `@xmldom/xmldom` 0.8.15 through package-root overrides without weakening CASA. Both production audits and local CASA passed; 25 focused DOCX extraction tests, all 239 worker tests, lint/typecheck, the production build, and `git diff --check` also passed. This security remediation is separate follow-up work and is not included in Trial 2's 7 pilot-maintenance minutes. ## Trial 3 evidence Trial 3 was explicitly continued by the sole maintainer. Draft PR #336 was opened at `2026-09-02T21:03:17Z` from isolated branch `cdx-gqc007v/pilot-trial-3-provider-architecture`, based on `origin/dev` commit `dd2166b7`: - Issue #251 is a documentation-only task. Its acceptance criteria require a source-verified explanation of the full runtime service contract, narrower application port, default registry path, provider facades and adapters, canonical models, multi-account identity boundary, provider differences, and control-plane/data-plane split. - Before review, the PR body records purpose, linked issue, testable acceptance criteria, affected and excluded boundaries, exact verification, nonvisual evidence, privacy review, rollback, limitations, and the no-merge/no-deploy handoff. - Screenshots are not applicable because no visual behavior changed. A deterministic PowerShell inventory found all 14 referenced contract, service, registry, provider-facade, and use-case paths; a separate Markdown inventory found that every local link in the changed docs resolves. - The focused verification set passed 9/9 test files and 117 tests, with one existing intentional skip. It covers documentation routing, service capability config, the provider facades/base adapters, and the generic adapter registry. Targeted Prettier, `pnpm check`, and `git diff --check` passed. - The changed documentation contains public repository paths, synthetic examples, and public provider names only. No confirmed privacy incident occurred. - Pilot-specific evidence and handoff maintenance was timed from `2026-09-02T21:02:55Z` through `2026-09-02T21:04:06Z` and recorded as 2 whole minutes. - CI run `33683046123` was the first complete hosted check set and passed lint/typecheck, both application-test shards, rclone worker checks, the production build, CASA, anonymous security E2E, and aggregate `build-and-test` on head `f7c39a64`. GitGuardian also passed. - The ready-for-review snapshot was taken at `2026-09-02T21:12:23Z`. At that snapshot, human evidence response and clarification-comment measurement were pending. - Closing CI run `33683867499` passed every repository job on ready head `97237a31`, including CASA, GitGuardian, anonymous security E2E, and aggregate `build-and-test`. - At the close snapshot on `2026-09-02T22:05:44Z`, the sole maintainer answered `yes` to whether the evidence helped them understand or verify the change. No reason was supplied. This is a sole-maintainer self-review and does not establish independent reviewer comprehension. - GitHub showed no review objects or comments. Clarification-comment measurement is `N/A` because the author and reviewer are the same person, not zero inferred from silence. - Trial 3 closed without an agent merge or deployment. PR #336 remained open and assigned to the sole maintainer for the human merge decision. - The sole maintainer subsequently merged PR #336 at `2026-09-02T22:13:10Z`, preserving the pilot's human-authority boundary. ## Trial 4 evidence Trial 4 was explicitly continued by the sole maintainer. Draft PR #338 was opened at `2026-09-02T22:25:51Z` from isolated branch `cdx-gqc007v/pilot-trial-4-dashboard-usage`, based on `origin/dev` commit `3d705aa0`: - Issue #273 is broad, so this trial is limited to one review finding: `ActionToolbar.tsx` duplicated the same eight selection actions across content and search variants and exceeded the 500-line repository limit. The PR does not claim that every React component was reviewed. - The extracted `ToolbarSelectionActions` remains render-only. Its inputs are derived selection booleans and callbacks; it adds no fetching, provider branching, quota policy, path construction, or data-plane behavior. - A deterministic PowerShell inventory measured `ActionToolbar.tsx` at 774 lines on the base and 275 lines after extraction. The new shared implementation measures 302 lines, keeping both changed implementation files below 500 lines. - The focused verification set passed 3/3 test files and 27 tests. It covers both toolbar variants plus empty, writable, shared-location, callback, backup, and disabled-tooltip behavior. Targeted ESLint, Prettier, `pnpm check`, and `git diff --check` passed. - The first run of the new shared-component characterization test used lowercase exact accessible-name queries and failed against the existing title-case labels. The assertions were corrected to be case-insensitive; this was a test-only defect, but first-pass verification remains `no` under the pilot definition. - The React best-practices review found no async, hydration, hook, or bundle-boundary concern in the extraction. The component uses direct imports, cheap render-time derivation, and no new state or effects. - Screenshots are not applicable because the refactor intentionally makes no visual change. Evidence uses synthetic component state only and contains no private issue screenshot, provider account details, credentials, browser state, customer filenames, environment values, or production logs. No confirmed privacy incident occurred. - Pilot-specific evidence and handoff maintenance was timed from `2026-09-02T22:25:51Z` through `2026-09-02T22:26:54Z` and recorded as 2 whole minutes. - CI run `33690467757` passed every repository job on head `84f33d7b`, including lint/typecheck, both application-test shards, rclone worker checks, the production build, CASA, anonymous security E2E, and aggregate `build-and-test`. GitGuardian also passed. - The ready-for-review snapshot was taken at `2026-09-02T22:33:50Z`. At that snapshot, human evidence response and clarification-comment measurement were pending. - Closing CI run `33691040087` passed every repository job on ready head `74cec09d`, including CASA, GitGuardian, anonymous security E2E, and aggregate `build-and-test`. - GitHub showed no review objects or comments. Clarification-comment measurement is `N/A` because the author and reviewer are the same person, not zero inferred from silence. - The sole maintainer reported the merge but did not answer the usefulness question, so reviewer usage is recorded as `not reported` rather than inferred from the merge. - Trial 4 closed without an agent merge or deployment. The sole maintainer merged PR #338 at `2026-09-02T22:52:23Z` as commit `92f0c23f`, preserving the pilot's human-authority boundary. ## Trial 5 evidence Trial 5 was explicitly continued by the sole maintainer. Draft PR #340 was opened at `2026-09-02T23:05:18Z` from isolated branch `cdx-gqc007v/pilot-trial-5-survey-review`, based on `origin/dev` commit `92f0c23f`: - Issue #273 is broad, so this trial is limited to one review finding: the 543-line `ServiceRequestSurvey.tsx` embedded a self-contained submitted-results visualization. The PR does not claim that every React component was reviewed. - The extracted `ServiceRequestResults` is render-only. It receives normalized results, selected service names, and an edit callback; it adds no fetching, authentication, provider branching, quota policy, path construction, or data-plane behavior. - A deterministic PowerShell inventory measured `ServiceRequestSurvey.tsx` at 543 lines on the base and 452 lines after extraction. The extracted component measured 96 lines and its focused test measured 60 lines in the working tree, keeping changed implementation files below 500 lines. - The focused verification set passed 2/2 test files and 23 tests. It covers the existing survey flows plus response totals, percentages, zero-vote filtering, selected-service markers, progress-bar accessibility attributes, and edit callback forwarding. Targeted ESLint and `pnpm check` passed. - The first run of the new characterization test queried a text node split by nested markup. The assertion was corrected to verify its parent text content; this was a test-only defect, but first-pass verification remains `no` under the pilot definition. - The React best-practices review found no new async, hydration, hook, state, effect, or bundle-boundary concern. The extracted component uses direct imports and renders only its typed inputs. - The tracked survey container retains its pre-existing CRLF convention. A Windows-aware `git -c core.whitespace=cr-at-eol diff HEAD^ HEAD --check` passed; the two new files use LF. - Screenshots are not applicable because the refactor intentionally makes no visual change. Evidence uses synthetic service names only and contains no provider account details, credentials, browser state, customer filenames, environment values, production logs, or private screenshots. No confirmed privacy incident occurred. - Pilot-specific evidence and handoff maintenance was timed from `2026-09-02T23:04:49Z` through `2026-09-02T23:06:13Z` and recorded as 2 whole minutes. - CI run `33693565375` passed every repository job on head `aaaa5daf`, including lint/typecheck, both application-test shards, rclone worker checks, the production build, CASA, anonymous security E2E, and aggregate `build-and-test`. GitGuardian also passed. - The ready-for-review snapshot was taken at `2026-09-02T23:14:14Z`. At that snapshot, human evidence response and clarification-comment measurement were pending. - Closing CI run `33694210761` passed every repository job on ready head `d6bf2384`, including CASA, GitGuardian, anonymous security E2E, and aggregate `build-and-test`. - At the close snapshot on `2026-09-02T23:23:54Z`, the sole maintainer answered `yes` to whether the evidence helped them understand or verify the change. No reason was supplied. This is a sole-maintainer self-review and does not establish independent reviewer comprehension. - GitHub showed no review objects or comments. Clarification-comment measurement is `N/A` because the author and reviewer are the same person, not zero inferred from silence. - Trial 5 closed without an agent merge or deployment. The sole maintainer merged PR #340 at `2026-09-02T23:23:31Z` as commit `1f1eaf52`, preserving the pilot's human-authority boundary. ## Entry procedure for trials 2–5 1. Confirm the task explicitly opts into the pilot and is suitable for private, non-sensitive evidence. 2. Reserve the next unused row; never backfill a non-opted-in PR. 3. Add `Pilot scorecard: trial N/5` to the PR's limitations section. 4. Before human review, record intake completeness, evidence applicability, first check results, and author-timed maintenance minutes. 5. Ask one human: `Did the evidence help you understand or verify this change: yes, partly, or no? Why?` Prefer a reviewer other than the author. In a single-maintainer repository, record the author's answer as sole-maintainer self-review and do not treat it as independent evidence. 6. At close, count qualifying clarification comments, separate real privacy incidents from prevented findings/false positives, link check runs, and mark the trial completed or withdrawn. 7. Do not put screenshots, credentials, customer data, filenames, identifiers, logs, environment values, or private reviewer details in this repository file. ## Decision gate Evaluate after three completed trials; extend to five only if the result remains ambiguous. Do not claim adoption or time savings before the gate. Continue or adopt with revisions only when: - all completed trials have complete intake before review; - at least two evidence-applicable trials receive a human usefulness response, labelled as independent review or sole-maintainer self-review; - there are zero confirmed privacy disclosures; - every branch-introduced verification failure is resolved before the trial closes; - maintenance minutes are reported for at least two trials; and - no participant reports that the pilot weakens architecture, CI, CASA, or human approval authority. Sole-maintainer self-review can measure whether the evidence helps the maintainer's verification workflow, but it cannot support claims about independent reviewer comprehension. Any adoption recommendation must preserve that distinction. Stop and recommend removal if evidence requires sensitive auth/provider state, a confirmed disclosure occurs, the workflow repeatedly produces unusable evidence, or maintenance cannot be justified by reviewer feedback. Inconclusive data means continue only up to the five-trial cap, then choose adopt with revisions or reject and remove. ## Five-trial outcome The decision gate supported **adopt with revisions**. The maintainer made that decision on 2026-09-03; the linked v1 adoption record defines the revised operating contract: - All five trials completed intake before review and used evidence suited to the change. - Four evidence-applicable trials received a usefulness response: three `yes` and one `partly`; Trial 4 received no response. Every response was sole-maintainer self-review, so the pilot provides no independent-reviewer comprehension evidence. - Four trials reported maintenance time, totalling 13 minutes; Trial 1 remained `not measured` rather than estimated. - No confirmed privacy disclosure occurred in any trial. - Two trials passed their first complete relevant check set. The other three exposed test, redaction, or dependency-gate findings that were resolved before close; no trial weakened CI, CASA, or human approval authority. Recommended operating revisions: 1. Keep the workflow opt-in and use one concise evidence block in the PR body; do not create a separate evidence package unless the change genuinely needs one. 2. Prefer deterministic tests and source inventories. Capture screenshots only for intentional visual changes, and keep all evidence free of credentials, account data, customer filenames, logs, environment values, and private browser state. 3. Record interim measurements in the PR body, then make one consolidated scorecard update at close to reduce duplicate maintenance and avoid evidence-only CI churn. 4. Preserve CI, CASA, security scanners, and the human merge decision as authoritative. Never let the evidence workflow merge or deploy automatically. 5. Continue labelling sole-maintainer responses as self-review. Re-evaluate reviewer comprehension if an independent contributor becomes available.