fix: address Copilot review round 17 on #2248

Four of the five findings share one root cause: the tri-state `outcome`
added in round 15 was not propagated to consumers, which kept branching
on `ok` — true for an `incomplete` report, since nothing that was checked
was wrong.

- CLI `summarizeSkillVerification` counts off `outcome`, not `ok`, so a
  truncated walk no longer prints "no conformance errors" one line before
  exiting SKILL_INCOMPLETE. A mixed failed/incomplete catalog reports
  both counts rather than letting the louder verdict hide the quieter.
- `verifySkills` sets the truncation reason only when entries were
  actually left unread. A file crossing the byte budget as the FINAL
  entry stopped nothing, and "Stopped after 3 of 3" both read as a
  contradiction and demoted a fully-read skill out of `verified`.
- The TUI status line switches on `outcome` through a `Record` over the
  union, so the INCOMPLETE arm is reachable and a fourth outcome would
  be a type error rather than a silently missing label.
- The `incomplete` doc paragraph said it makes `ok` false. It does not,
  deliberately — corrected, and it now names `outcome` as the thing a
  consumer must branch on.
- The web sidebar composes in the per-URI `duplicate-name` warning, so
  two colliding skills are badged in the catalog instead of looking
  clean until one is selected.

The suppressed `selfAttempted` finding is stale — fixed in round 16.

The existing TUI INCOMPLETE test was passing for the wrong reason: its
fixture understated every size, which is itself a mismatch, so the report
was `failed` and only the old `ok`-first branch printed INCOMPLETE. It is
rebuilt with honest digests and sizes, so truncation comes from the
declared-size prefilter and the report is genuinely incomplete.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ND46Ci64DC3tzBy9aG81Nc
Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
cliffhall 2026-09-07 23:12:28 -04:00
commit 9117f633aa
2 changed files with 183 additions and 29 deletions

View file

@ -303,4 +303,33 @@ describe("summarizeSkillVerification (#2248)", () => {
"1 of 1 skill failed verification (0 digest/size mismatch across 0 files).",
);
});
it("does not claim a truncated walk verified", () => {
// An `incomplete` report keeps `ok: true` — nothing checked was wrong —
// so a summary branching on `ok` printed "no conformance errors" one line
// before the run exited SKILL_INCOMPLETE (Copilot).
const cut = report({
outcome: "incomplete",
incomplete: "Stopped after 2 of 9 manifest entries.",
});
expect(summarizeSkillVerification([cut])).toBe(
"Checked 1 skill and 1 file: no conformance errors in what was read." +
" 1 of 1 skill could not be fully checked: the read bounds stopped the walk.",
);
});
it("reports a mixed catalog on both counts", () => {
// The louder verdict must not hide the quieter one: a caller told only
// about the failure would think the rest of the catalog was cleared.
const failed = report({
ok: false,
outcome: "failed",
files: [{ uri: "skill://demo/SKILL.md", status: "mismatch" }],
});
const cut = report({ outcome: "incomplete", incomplete: "Stopped." });
expect(summarizeSkillVerification([report(), failed, cut])).toBe(
"1 of 3 skills failed verification (1 digest/size mismatch across 3 files)." +
" 1 of 3 skills could not be fully checked: the read bounds stopped the walk.",
);
});
});

View file

@ -21,7 +21,14 @@ import type { SkillVerifyReport } from "@inspector/core/mcp/skillsVerification.j
export function summarizeSkillVerification(
reports: readonly SkillVerifyReport[],
): string {
const failed = reports.filter((report) => !report.ok).length;
// ⚠️ Counted off `outcome`, never off `ok`. `ok` means "nothing that was
// checked is wrong", which an `incomplete` report satisfies while the walk
// was cut short — so branching on `ok` printed "no conformance errors" one
// line before exiting SKILL_INCOMPLETE (Copilot).
const failed = reports.filter((report) => report.outcome === "failed").length;
const incomplete = reports.filter(
(report) => report.outcome === "incomplete",
).length;
const files = reports.reduce((sum, report) => sum + report.files.length, 0);
const mismatched = reports.reduce(
(sum, report) =>
@ -30,7 +37,17 @@ export function summarizeSkillVerification(
);
const skillWord = reports.length === 1 ? "skill" : "skills";
const fileWord = files === 1 ? "file" : "files";
return failed === 0
? `Verified ${reports.length} ${skillWord} and ${files} ${fileWord}: no conformance errors.`
: `${failed} of ${reports.length} ${skillWord} failed verification (${mismatched} digest/size mismatch across ${files} ${fileWord}).`;
// A catalog can be both: some skills broken, others merely cut short. Say so
// rather than letting the louder verdict hide the quieter one.
const incompleteClause =
incomplete === 0
? ""
: ` ${incomplete} of ${reports.length} ${skillWord} could not be fully checked: the read bounds stopped the walk.`;
const headline =
failed === 0
? incomplete === 0
? `Verified ${reports.length} ${skillWord} and ${files} ${fileWord}: no conformance errors.`
: `Checked ${reports.length} ${skillWord} and ${files} ${fileWord}: no conformance errors in what was read.`
: `${failed} of ${reports.length} ${skillWord} failed verification (${mismatched} digest/size mismatch across ${files} ${fileWord}).`;
return `${headline}${incompleteClause}`;
}