docs: state the gap the gate cannot cover

`verify:skills` checks that a skill is well-formed and that its cases exist;
only `skills:eval` observes whether it actually fires, and that cannot be
gated. Say so in the script header and in AGENTS.md rather than leaving the
split to be inferred.

Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
cliffhall 2026-08-29 00:53:49 -04:00
commit 4520e6aa2c
2 changed files with 18 additions and 3 deletions

View file

@ -180,6 +180,12 @@ that from happening:
("how does the `@inspector/core` alias resolve?") invites a `Read`, which is ("how does the `@inspector/core` alias resolve?") invites a `Read`, which is
a *better* answer than a skill. Good cases are "how do I / where does this go" a *better* answer than a skill. Good cases are "how do I / where does this go"
questions whose answer is a procedure. questions whose answer is a procedure.
⚠️ **The gate cannot catch a description that never matches.** `verify:skills`
checks that a skill is well-formed and that its cases exist; only
`skills:eval` observes whether it actually fires, and that cannot be gated —
it spends metered model calls, its result is a hit rate rather than a verdict,
and it goes red on a rate limit. So the eval is a tuning tool you run
deliberately, and a case below threshold is a signal, not a build break.
4. **Re-check the listing budget when adding a skill.** Claude Code loads a 4. **Re-check the listing budget when adding a skill.** Claude Code loads a
listing of every skill's name and description into context, truncates it when it listing of every skill's name and description into context, truncates it when it
overflows, and drops the least-invoked entries **first** — which are exactly the overflows, and drops the least-invoked entries **first** — which are exactly the

View file

@ -11,9 +11,18 @@
// regression, and it is the failure nobody notices by hand. `verify:skills` // regression, and it is the failure nobody notices by hand. `verify:skills`
// requires both. // requires both.
// //
// NOT part of `validate`, `local:gate`, or CI: it needs the `claude` CLI and // NOT part of `validate`, `local:gate`, or CI, and that is a decision rather
// real model calls, so it is metered and non-deterministic. Run it when adding // than an omission. A trigger eval cannot be a gate: it spends metered model
// a skill or editing a model-invoked skill's description. // calls on every push, the measurement IS a hit rate over samples so it is
// non-deterministic by construction, and it goes red on a rate limit or an
// expired token — failures unrelated to the diff, whose first consequence is
// that people stop trusting the gate. Run it when adding a skill or editing a
// model-invoked skill's description.
//
// The cost of that split, stated plainly: `verify:skills` cannot detect a
// well-formed skill whose description simply never matches anything. Nothing in
// the gate can. Closing that would need a deterministic trigger oracle, which
// does not exist.
// //
// Usage: // Usage:
// npm run skills:eval # every model-invoked skill's cases // npm run skills:eval # every model-invoked skill's cases