Record only observable GitHub state or a value explicitly supplied by the author or reviewer. Never infer time, effort, reviewer intent, or evidence usefulness from commit timestamps, comment timestamps, an AI score, or the absence of comments.
Measure
Operational definition
Intake completeness
PR states its purpose, at least one testable acceptance criterion, affected boundaries, verification, and explicit exclusions or none before human review begins.
Reviewer usage
A human answers yes, partly, or no to “Did the evidence help you understand or verify this change?”, with an optional reason. Record whether this was independent review or sole-maintainer self-review; never present self-review as independent evidence. Bot activity does not count.
Clarification comments
Count human reviewer comments that request missing scope, acceptance, safety, verification, or evidence information. Exclude bot comments, approvals, and implementation findings unrelated to handoff clarity.
Maintenance effort
Author-reported minutes spent preparing, sanitising, checking, and handing off pilot-specific evidence. Record not measured when no timer was used.
Privacy incidents
Count confirmed disclosures separately from scanner false positives and prevented findings. Never copy sensitive content into this scorecard.
First-pass verification
Record whether the first complete relevant check set passed. Separate branch regressions, unchanged baseline failures, environment limits, and false positives.
Take one timestamped snapshot when the PR first requests human review and one when the trial closes. A draft with no human response cannot produce a reviewer-usage result, even when it has zero clarification comments.
No: a test-only nested-text query was corrected; hosted checks passed
Completed; human-merged
Trial 1 evidence
Draft snapshot taken on 2026-09-02, followed by a ready-for-review snapshot at 2026-09-02T11:22:38Z and a close snapshot at 2026-09-02T12:29:54Z:
The PR body includes purpose, acceptance criteria, boundaries, changed behaviour, exact verification, local evidence instructions, privacy review, rollback, limitations, the hypothesis, and a no-merge/no-deploy statement.
The ignored smoke package contains two inspected 1440x900 images, manifest.json, and report.md; the helper uploaded nothing.
Local focused tests finished with 18/18 passing after the helper suite exposed and fixed one metadata-redaction defect.
GitHub CI run 33620172786 passed lint/typecheck, both application-test shards, rclone worker checks, the production build, and anonymous security E2E.
CI run 33624471878 passed lint/typecheck, both application-test shards, rclone worker checks, the production build, and anonymous security E2E. CASA and the aggregate check failed before follow-up remediation because the production graph contained two high and one low advisories.
Follow-up remediation pins browserslist 4.28.8 and postcss-selector-parser 6.1.4 through PNPM overrides. A fresh production audit reported zero advisories, and local CASA, the focused 18-test set, lint/typecheck, and the production build passed. CI run 33626898879 then passed every repository job, including CASA and aggregate build-and-test, on remediation commit 459255cc.
The sole maintainer reported dismissing GitGuardian incident 36837173 as a false positive. A fresh GitGuardian scan then passed on head 0acd16c3; no confirmed privacy disclosure occurred.
CI run 33628767467 passed every repository job on the closing head: lint/typecheck, both application-test shards, rclone worker checks, the production build, CASA, anonymous security E2E, and aggregate build-and-test.
The sole maintainer recorded the evidence as partly helpful: structured acceptance criteria, safety boundaries, and exact checks helped verification, while fixture screenshots added little value for this nonvisual change and duplicated report/scorecard maintenance created avoidable CI churn. This is self-review, not independent reviewer evidence.
Clarification-comment measurement is N/A because the author and reviewer are the same person. It cannot support a reviewer-comprehension claim.
Author maintenance time was not timed and must remain not measured; no estimate was substituted for a timed value.
The author changed the PR from draft to ready for review. This opens the human measurement window; it does not constitute approval or establish that the PR is mergeable.
At that snapshot, head 0a5f6c33 used CI run 33623875572: lint/typecheck, both test shards, the production build, and worker checks passed; anonymous security E2E remained in progress; CASA retained the documented dependency-baseline failure.
Trial 1 closed without an agent merge or deployment. The sole maintainer subsequently merged PR #333 at 2026-09-02T12:46:10Z, preserving the pilot's human-authority boundary.
Trial 2 evidence
Trial 2 was explicitly continued by the sole maintainer. Draft PR #334 was opened at 2026-09-02T12:57:26Z from isolated branch cdx-gqc007v/pilot-trial-2-streaming-docs, based on origin/dev commit c817e667:
Issue #243 is a documentation-only task. Its acceptance criteria require a current SSE/NDJSON decision guide, a source-verified endpoint inventory, lifecycle and fallback behavior, explicit architecture/security boundaries, removal of unsupported claims, and no runtime or dependency change.
Before review, the PR body records purpose, linked issue, testable acceptance criteria, affected and excluded boundaries, exact verification, nonvisual evidence, privacy review, rollback, limitations, and the no-merge/no-deploy handoff.
Screenshots are not applicable because no visual behavior changed. A deterministic PowerShell inventory found all four documented SSE producers and four documented NDJSON producers with their expected transport markers.
pnpm test src/lib/docs.test.ts src/app/dev/docs/page.test.tsx passed 3/3 tests; targeted Prettier, pnpm check, and git diff --check passed.
The changed documentation contains public route names, repository-relative source paths, synthetic examples, and public reference links only. No confirmed privacy incident occurred.
CI run 33632870763 was the first complete hosted check set and passed lint/typecheck, both application-test shards, rclone worker checks, the production build, CASA, anonymous security E2E, and aggregate build-and-test on initial documentation head 9c74b7ae. GitGuardian also passed.
Pilot-specific evidence and handoff maintenance was timed from 2026-09-02T12:56:46Z through 2026-09-02T13:03:34Z and recorded as 7 whole minutes.
The ready-for-review snapshot was taken at 2026-09-02T13:04:22Z. Closing CI run 33633530730 passed every repository job on head cdaaf38f, including CASA, GitGuardian, anonymous security E2E, and aggregate build-and-test.
At the close snapshot on 2026-09-02T20:27:29Z, the sole maintainer answered yes to whether the evidence helped them understand or verify the change. No reason was supplied. This is a sole-maintainer self-review and does not establish independent reviewer comprehension.
GitHub showed no review objects or comments. Clarification-comment measurement is N/A because the author and reviewer are the same person, not zero inferred from silence.
Trial 2 closed without an agent merge or deployment. The sole maintainer merged PR #334 at 2026-09-02T20:27:36Z, preserving the pilot's human-authority boundary.
Follow-up PR #335 carries the closeout record because PR #334 was human-merged before the final scorecard commits reached its branch. Its first CASA run then detected newly published baseline advisories in unchanged dependency paths: two qs findings affected the web and worker graphs, and one @xmldom/xmldom finding affected the web graph. The follow-up pins qs 6.16.0 and @xmldom/xmldom 0.8.15 through package-root overrides without weakening CASA. Both production audits and local CASA passed; 25 focused DOCX extraction tests, all 239 worker tests, lint/typecheck, the production build, and git diff --check also passed. This security remediation is separate follow-up work and is not included in Trial 2's 7 pilot-maintenance minutes.
Trial 3 evidence
Trial 3 was explicitly continued by the sole maintainer. Draft PR #336 was opened at 2026-09-02T21:03:17Z from isolated branch cdx-gqc007v/pilot-trial-3-provider-architecture, based on origin/dev commit dd2166b7:
Issue #251 is a documentation-only task. Its acceptance criteria require a source-verified explanation of the full runtime service contract, narrower application port, default registry path, provider facades and adapters, canonical models, multi-account identity boundary, provider differences, and control-plane/data-plane split.
Before review, the PR body records purpose, linked issue, testable acceptance criteria, affected and excluded boundaries, exact verification, nonvisual evidence, privacy review, rollback, limitations, and the no-merge/no-deploy handoff.
Screenshots are not applicable because no visual behavior changed. A deterministic PowerShell inventory found all 14 referenced contract, service, registry, provider-facade, and use-case paths; a separate Markdown inventory found that every local link in the changed docs resolves.
The focused verification set passed 9/9 test files and 117 tests, with one existing intentional skip. It covers documentation routing, service capability config, the provider facades/base adapters, and the generic adapter registry. Targeted Prettier, pnpm check, and git diff --check passed.
The changed documentation contains public repository paths, synthetic examples, and public provider names only. No confirmed privacy incident occurred.
Pilot-specific evidence and handoff maintenance was timed from 2026-09-02T21:02:55Z through 2026-09-02T21:04:06Z and recorded as 2 whole minutes.
CI run 33683046123 was the first complete hosted check set and passed lint/typecheck, both application-test shards, rclone worker checks, the production build, CASA, anonymous security E2E, and aggregate build-and-test on head f7c39a64. GitGuardian also passed.
The ready-for-review snapshot was taken at 2026-09-02T21:12:23Z. At that snapshot, human evidence response and clarification-comment measurement were pending.
Closing CI run 33683867499 passed every repository job on ready head 97237a31, including CASA, GitGuardian, anonymous security E2E, and aggregate build-and-test.
At the close snapshot on 2026-09-02T22:05:44Z, the sole maintainer answered yes to whether the evidence helped them understand or verify the change. No reason was supplied. This is a sole-maintainer self-review and does not establish independent reviewer comprehension.
GitHub showed no review objects or comments. Clarification-comment measurement is N/A because the author and reviewer are the same person, not zero inferred from silence.
Trial 3 closed without an agent merge or deployment. PR #336 remained open and assigned to the sole maintainer for the human merge decision.
The sole maintainer subsequently merged PR #336 at 2026-09-02T22:13:10Z, preserving the pilot's human-authority boundary.
Trial 4 evidence
Trial 4 was explicitly continued by the sole maintainer. Draft PR #338 was opened at 2026-09-02T22:25:51Z from isolated branch cdx-gqc007v/pilot-trial-4-dashboard-usage, based on origin/dev commit 3d705aa0:
Issue #273 is broad, so this trial is limited to one review finding: ActionToolbar.tsx duplicated the same eight selection actions across content and search variants and exceeded the 500-line repository limit. The PR does not claim that every React component was reviewed.
The extracted ToolbarSelectionActions remains render-only. Its inputs are derived selection booleans and callbacks; it adds no fetching, provider branching, quota policy, path construction, or data-plane behavior.
A deterministic PowerShell inventory measured ActionToolbar.tsx at 774 lines on the base and 275 lines after extraction. The new shared implementation measures 302 lines, keeping both changed implementation files below 500 lines.
The focused verification set passed 3/3 test files and 27 tests. It covers both toolbar variants plus empty, writable, shared-location, callback, backup, and disabled-tooltip behavior. Targeted ESLint, Prettier, pnpm check, and git diff --check passed.
The first run of the new shared-component characterization test used lowercase exact accessible-name queries and failed against the existing title-case labels. The assertions were corrected to be case-insensitive; this was a test-only defect, but first-pass verification remains no under the pilot definition.
The React best-practices review found no async, hydration, hook, or bundle-boundary concern in the extraction. The component uses direct imports, cheap render-time derivation, and no new state or effects.
Screenshots are not applicable because the refactor intentionally makes no visual change. Evidence uses synthetic component state only and contains no private issue screenshot, provider account details, credentials, browser state, customer filenames, environment values, or production logs. No confirmed privacy incident occurred.
Pilot-specific evidence and handoff maintenance was timed from 2026-09-02T22:25:51Z through 2026-09-02T22:26:54Z and recorded as 2 whole minutes.
CI run 33690467757 passed every repository job on head 84f33d7b, including lint/typecheck, both application-test shards, rclone worker checks, the production build, CASA, anonymous security E2E, and aggregate build-and-test. GitGuardian also passed.
The ready-for-review snapshot was taken at 2026-09-02T22:33:50Z. At that snapshot, human evidence response and clarification-comment measurement were pending.
Closing CI run 33691040087 passed every repository job on ready head 74cec09d, including CASA, GitGuardian, anonymous security E2E, and aggregate build-and-test.
GitHub showed no review objects or comments. Clarification-comment measurement is N/A because the author and reviewer are the same person, not zero inferred from silence.
The sole maintainer reported the merge but did not answer the usefulness question, so reviewer usage is recorded as not reported rather than inferred from the merge.
Trial 4 closed without an agent merge or deployment. The sole maintainer merged PR #338 at 2026-09-02T22:52:23Z as commit 92f0c23f, preserving the pilot's human-authority boundary.
Trial 5 evidence
Trial 5 was explicitly continued by the sole maintainer. Draft PR #340 was opened at 2026-09-02T23:05:18Z from isolated branch cdx-gqc007v/pilot-trial-5-survey-review, based on origin/dev commit 92f0c23f:
Issue #273 is broad, so this trial is limited to one review finding: the 543-line ServiceRequestSurvey.tsx embedded a self-contained submitted-results visualization. The PR does not claim that every React component was reviewed.
The extracted ServiceRequestResults is render-only. It receives normalized results, selected service names, and an edit callback; it adds no fetching, authentication, provider branching, quota policy, path construction, or data-plane behavior.
A deterministic PowerShell inventory measured ServiceRequestSurvey.tsx at 543 lines on the base and 452 lines after extraction. The extracted component measured 96 lines and its focused test measured 60 lines in the working tree, keeping changed implementation files below 500 lines.
The focused verification set passed 2/2 test files and 23 tests. It covers the existing survey flows plus response totals, percentages, zero-vote filtering, selected-service markers, progress-bar accessibility attributes, and edit callback forwarding. Targeted ESLint and pnpm check passed.
The first run of the new characterization test queried a text node split by nested markup. The assertion was corrected to verify its parent text content; this was a test-only defect, but first-pass verification remains no under the pilot definition.
The React best-practices review found no new async, hydration, hook, state, effect, or bundle-boundary concern. The extracted component uses direct imports and renders only its typed inputs.
The tracked survey container retains its pre-existing CRLF convention. A Windows-aware git -c core.whitespace=cr-at-eol diff HEAD^ HEAD --check passed; the two new files use LF.
Screenshots are not applicable because the refactor intentionally makes no visual change. Evidence uses synthetic service names only and contains no provider account details, credentials, browser state, customer filenames, environment values, production logs, or private screenshots. No confirmed privacy incident occurred.
Pilot-specific evidence and handoff maintenance was timed from 2026-09-02T23:04:49Z through 2026-09-02T23:06:13Z and recorded as 2 whole minutes.
CI run 33693565375 passed every repository job on head aaaa5daf, including lint/typecheck, both application-test shards, rclone worker checks, the production build, CASA, anonymous security E2E, and aggregate build-and-test. GitGuardian also passed.
The ready-for-review snapshot was taken at 2026-09-02T23:14:14Z. At that snapshot, human evidence response and clarification-comment measurement were pending.
Closing CI run 33694210761 passed every repository job on ready head d6bf2384, including CASA, GitGuardian, anonymous security E2E, and aggregate build-and-test.
At the close snapshot on 2026-09-02T23:23:54Z, the sole maintainer answered yes to whether the evidence helped them understand or verify the change. No reason was supplied. This is a sole-maintainer self-review and does not establish independent reviewer comprehension.
GitHub showed no review objects or comments. Clarification-comment measurement is N/A because the author and reviewer are the same person, not zero inferred from silence.
Trial 5 closed without an agent merge or deployment. The sole maintainer merged PR #340 at 2026-09-02T23:23:31Z as commit 1f1eaf52, preserving the pilot's human-authority boundary.
Entry procedure for trials 2–5
Confirm the task explicitly opts into the pilot and is suitable for private, non-sensitive evidence.
Reserve the next unused row; never backfill a non-opted-in PR.
Add Pilot scorecard: trial N/5 to the PR's limitations section.
Before human review, record intake completeness, evidence applicability, first check results, and author-timed maintenance minutes.
Ask one human: Did the evidence help you understand or verify this change: yes, partly, or no? Why? Prefer a reviewer other than the author. In a single-maintainer repository, record the author's answer as sole-maintainer self-review and do not treat it as independent evidence.
At close, count qualifying clarification comments, separate real privacy incidents from prevented findings/false positives, link check runs, and mark the trial completed or withdrawn.
Do not put screenshots, credentials, customer data, filenames, identifiers, logs, environment values, or private reviewer details in this repository file.
Decision gate
Evaluate after three completed trials; extend to five only if the result remains ambiguous. Do not claim adoption or time savings before the gate.
Continue or adopt with revisions only when:
all completed trials have complete intake before review;
at least two evidence-applicable trials receive a human usefulness response, labelled as independent review or sole-maintainer self-review;
there are zero confirmed privacy disclosures;
every branch-introduced verification failure is resolved before the trial closes;
maintenance minutes are reported for at least two trials; and
no participant reports that the pilot weakens architecture, CI, CASA, or human approval authority.
Sole-maintainer self-review can measure whether the evidence helps the maintainer's verification workflow, but it cannot support claims about independent reviewer comprehension. Any adoption recommendation must preserve that distinction.
Stop and recommend removal if evidence requires sensitive auth/provider state, a confirmed disclosure occurs, the workflow repeatedly produces unusable evidence, or maintenance cannot be justified by reviewer feedback. Inconclusive data means continue only up to the five-trial cap, then choose adopt with revisions or reject and remove.
Five-trial outcome
The decision gate supported adopt with revisions. The maintainer made that
decision on 2026-09-03; the linked v1 adoption record defines the revised
operating contract:
All five trials completed intake before review and used evidence suited to the change.
Four evidence-applicable trials received a usefulness response: three yes and one partly; Trial 4 received no response. Every response was sole-maintainer self-review, so the pilot provides no independent-reviewer comprehension evidence.
Four trials reported maintenance time, totalling 13 minutes; Trial 1 remained not measured rather than estimated.
No confirmed privacy disclosure occurred in any trial.
Two trials passed their first complete relevant check set. The other three exposed test, redaction, or dependency-gate findings that were resolved before close; no trial weakened CI, CASA, or human approval authority.
Recommended operating revisions:
Keep the workflow opt-in and use one concise evidence block in the PR body; do not create a separate evidence package unless the change genuinely needs one.
Prefer deterministic tests and source inventories. Capture screenshots only for intentional visual changes, and keep all evidence free of credentials, account data, customer filenames, logs, environment values, and private browser state.
Record interim measurements in the PR body, then make one consolidated scorecard update at close to reduce duplicate maintenance and avoid evidence-only CI churn.
Preserve CI, CASA, security scanners, and the human merge decision as authoritative. Never let the evidence workflow merge or deploy automatically.
Continue labelling sole-maintainer responses as self-review. Re-evaluate reviewer comprehension if an independent contributor becomes available.