docs: accept several skill names in skills:eval and re-aim overlapping cases

Adds a multi-name filter to `npm run skills:eval` so a set of related
skills can be iterated together, and makes an unknown name a hard error
rather than an empty run — a typo would otherwise report a green 0/0
that reads exactly like a clean pass of the skill that was meant.

Re-aims two `local-dev` cases and one `test-servers` case that partially
overlapped `AGENTS.md`. Partial overlap oscillates across runs rather
than sitting at zero, so it reads as eval noise instead of a defect in
the case; `docs/skill-authoring.md` now records that failure mode and
the observed scores.

Widens the `test-servers` description to the "Use when …" shape and
points the `testing` skill at it, since anything needing a real server
to run against is a procedure `testing` does not carry. That growth
moves the skill listing from 2,921 to 3,144 of 4,000 characters, so the
figure recorded in AGENTS.md is updated to match.

Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
cliffhall 2026-09-01 00:18:43 -04:00
commit a344225e23

View file

@ -322,6 +322,42 @@ test("focused mode narrows the cases but not the repo's own skill set", () => {
rmSync(root, { recursive: true, force: true });
});
test("focused mode accepts several skill names", () => {
const root = skillsFixture();
const focused = collectCases(["alpha", "beta"], root);
assert.deepEqual(focused.cases.map((c) => c.prompt).sort(), [
"a+",
"a-",
"b+",
"b-",
]);
// Narrowing to every skill is still not the same as no filter — `ours` is
// unchanged either way, which is what keeps negative scoring honest.
assert.deepEqual([...focused.ours].sort(), ["alpha", "beta"]);
// A single name keeps working, so the one-argument form is unaffected.
assert.deepEqual(
collectCases("beta", root).cases.map((c) => c.prompt),
["b+", "b-"],
);
rmSync(root, { recursive: true, force: true });
});
test("an unknown skill name is an error, not an empty run", () => {
// A typo would otherwise enqueue zero cases and report a green 0/0, which
// reads exactly like a clean pass of the skill that was meant.
const root = skillsFixture();
assert.throws(
() => collectCases(["alpha", "aplha"], root),
/no model-invoked skill named `aplha`/,
);
// A name-only skill is not model-invoked, so asking for it is the same error.
assert.throws(() => collectCases(["gamma"], root), /no model-invoked skill/);
rmSync(root, { recursive: true, force: true });
});
test("a name-only skill is not part of the repo's model-invoked set", () => {
const root = skillsFixture();
const { ours } = collectCases(undefined, root);