fix: address Copilot review round 17 on #2248
Four of the five findings share one root cause: the tri-state `outcome` added in round 15 was not propagated to consumers, which kept branching on `ok` — true for an `incomplete` report, since nothing that was checked was wrong. - CLI `summarizeSkillVerification` counts off `outcome`, not `ok`, so a truncated walk no longer prints "no conformance errors" one line before exiting SKILL_INCOMPLETE. A mixed failed/incomplete catalog reports both counts rather than letting the louder verdict hide the quieter. - `verifySkills` sets the truncation reason only when entries were actually left unread. A file crossing the byte budget as the FINAL entry stopped nothing, and "Stopped after 3 of 3" both read as a contradiction and demoted a fully-read skill out of `verified`. - The TUI status line switches on `outcome` through a `Record` over the union, so the INCOMPLETE arm is reachable and a fourth outcome would be a type error rather than a silently missing label. - The `incomplete` doc paragraph said it makes `ok` false. It does not, deliberately — corrected, and it now names `outcome` as the thing a consumer must branch on. - The web sidebar composes in the per-URI `duplicate-name` warning, so two colliding skills are badged in the catalog instead of looking clean until one is selected. The suppressed `selfAttempted` finding is stale — fixed in round 16. The existing TUI INCOMPLETE test was passing for the wrong reason: its fixture understated every size, which is itself a mismatch, so the report was `failed` and only the old `ok`-first branch printed INCOMPLETE. It is rebuilt with honest digests and sizes, so truncation comes from the declared-size prefilter and the report is genuinely incomplete. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ND46Ci64DC3tzBy9aG81Nc Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
parent
5a1f0efb5a
commit
9117f633aa
2 changed files with 183 additions and 29 deletions
|
|
@ -303,4 +303,33 @@ describe("summarizeSkillVerification (#2248)", () => {
|
|||
"1 of 1 skill failed verification (0 digest/size mismatch across 0 files).",
|
||||
);
|
||||
});
|
||||
|
||||
it("does not claim a truncated walk verified", () => {
|
||||
// An `incomplete` report keeps `ok: true` — nothing checked was wrong —
|
||||
// so a summary branching on `ok` printed "no conformance errors" one line
|
||||
// before the run exited SKILL_INCOMPLETE (Copilot).
|
||||
const cut = report({
|
||||
outcome: "incomplete",
|
||||
incomplete: "Stopped after 2 of 9 manifest entries.",
|
||||
});
|
||||
expect(summarizeSkillVerification([cut])).toBe(
|
||||
"Checked 1 skill and 1 file: no conformance errors in what was read." +
|
||||
" 1 of 1 skill could not be fully checked: the read bounds stopped the walk.",
|
||||
);
|
||||
});
|
||||
|
||||
it("reports a mixed catalog on both counts", () => {
|
||||
// The louder verdict must not hide the quieter one: a caller told only
|
||||
// about the failure would think the rest of the catalog was cleared.
|
||||
const failed = report({
|
||||
ok: false,
|
||||
outcome: "failed",
|
||||
files: [{ uri: "skill://demo/SKILL.md", status: "mismatch" }],
|
||||
});
|
||||
const cut = report({ outcome: "incomplete", incomplete: "Stopped." });
|
||||
expect(summarizeSkillVerification([report(), failed, cut])).toBe(
|
||||
"1 of 3 skills failed verification (1 digest/size mismatch across 3 files)." +
|
||||
" 1 of 3 skills could not be fully checked: the read bounds stopped the walk.",
|
||||
);
|
||||
});
|
||||
});
|
||||
|
|
|
|||
|
|
@ -21,7 +21,14 @@ import type { SkillVerifyReport } from "@inspector/core/mcp/skillsVerification.j
|
|||
export function summarizeSkillVerification(
|
||||
reports: readonly SkillVerifyReport[],
|
||||
): string {
|
||||
const failed = reports.filter((report) => !report.ok).length;
|
||||
// ⚠️ Counted off `outcome`, never off `ok`. `ok` means "nothing that was
|
||||
// checked is wrong", which an `incomplete` report satisfies while the walk
|
||||
// was cut short — so branching on `ok` printed "no conformance errors" one
|
||||
// line before exiting SKILL_INCOMPLETE (Copilot).
|
||||
const failed = reports.filter((report) => report.outcome === "failed").length;
|
||||
const incomplete = reports.filter(
|
||||
(report) => report.outcome === "incomplete",
|
||||
).length;
|
||||
const files = reports.reduce((sum, report) => sum + report.files.length, 0);
|
||||
const mismatched = reports.reduce(
|
||||
(sum, report) =>
|
||||
|
|
@ -30,7 +37,17 @@ export function summarizeSkillVerification(
|
|||
);
|
||||
const skillWord = reports.length === 1 ? "skill" : "skills";
|
||||
const fileWord = files === 1 ? "file" : "files";
|
||||
return failed === 0
|
||||
? `Verified ${reports.length} ${skillWord} and ${files} ${fileWord}: no conformance errors.`
|
||||
: `${failed} of ${reports.length} ${skillWord} failed verification (${mismatched} digest/size mismatch across ${files} ${fileWord}).`;
|
||||
// A catalog can be both: some skills broken, others merely cut short. Say so
|
||||
// rather than letting the louder verdict hide the quieter one.
|
||||
const incompleteClause =
|
||||
incomplete === 0
|
||||
? ""
|
||||
: ` ${incomplete} of ${reports.length} ${skillWord} could not be fully checked: the read bounds stopped the walk.`;
|
||||
const headline =
|
||||
failed === 0
|
||||
? incomplete === 0
|
||||
? `Verified ${reports.length} ${skillWord} and ${files} ${fileWord}: no conformance errors.`
|
||||
: `Checked ${reports.length} ${skillWord} and ${files} ${fileWord}: no conformance errors in what was read.`
|
||||
: `${failed} of ${reports.length} ${skillWord} failed verification (${mismatched} digest/size mismatch across ${files} ${fileWord}).`;
|
||||
return `${headline}${incompleteClause}`;
|
||||
}
|
||||
|
|
|
|||
Loading…
Reference in a new issue