docs: state the gap the gate cannot cover
`verify:skills` checks that a skill is well-formed and that its cases exist; only `skills:eval` observes whether it actually fires, and that cannot be gated. Say so in the script header and in AGENTS.md rather than leaving the split to be inferred. Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
parent
a51e62dfa1
commit
4520e6aa2c
2 changed files with 18 additions and 3 deletions
|
|
@ -180,6 +180,12 @@ that from happening:
|
||||||
("how does the `@inspector/core` alias resolve?") invites a `Read`, which is
|
("how does the `@inspector/core` alias resolve?") invites a `Read`, which is
|
||||||
a *better* answer than a skill. Good cases are "how do I / where does this go"
|
a *better* answer than a skill. Good cases are "how do I / where does this go"
|
||||||
questions whose answer is a procedure.
|
questions whose answer is a procedure.
|
||||||
|
⚠️ **The gate cannot catch a description that never matches.** `verify:skills`
|
||||||
|
checks that a skill is well-formed and that its cases exist; only
|
||||||
|
`skills:eval` observes whether it actually fires, and that cannot be gated —
|
||||||
|
it spends metered model calls, its result is a hit rate rather than a verdict,
|
||||||
|
and it goes red on a rate limit. So the eval is a tuning tool you run
|
||||||
|
deliberately, and a case below threshold is a signal, not a build break.
|
||||||
4. **Re-check the listing budget when adding a skill.** Claude Code loads a
|
4. **Re-check the listing budget when adding a skill.** Claude Code loads a
|
||||||
listing of every skill's name and description into context, truncates it when it
|
listing of every skill's name and description into context, truncates it when it
|
||||||
overflows, and drops the least-invoked entries **first** — which are exactly the
|
overflows, and drops the least-invoked entries **first** — which are exactly the
|
||||||
|
|
|
||||||
|
|
@ -11,9 +11,18 @@
|
||||||
// regression, and it is the failure nobody notices by hand. `verify:skills`
|
// regression, and it is the failure nobody notices by hand. `verify:skills`
|
||||||
// requires both.
|
// requires both.
|
||||||
//
|
//
|
||||||
// NOT part of `validate`, `local:gate`, or CI: it needs the `claude` CLI and
|
// NOT part of `validate`, `local:gate`, or CI, and that is a decision rather
|
||||||
// real model calls, so it is metered and non-deterministic. Run it when adding
|
// than an omission. A trigger eval cannot be a gate: it spends metered model
|
||||||
// a skill or editing a model-invoked skill's description.
|
// calls on every push, the measurement IS a hit rate over samples so it is
|
||||||
|
// non-deterministic by construction, and it goes red on a rate limit or an
|
||||||
|
// expired token — failures unrelated to the diff, whose first consequence is
|
||||||
|
// that people stop trusting the gate. Run it when adding a skill or editing a
|
||||||
|
// model-invoked skill's description.
|
||||||
|
//
|
||||||
|
// The cost of that split, stated plainly: `verify:skills` cannot detect a
|
||||||
|
// well-formed skill whose description simply never matches anything. Nothing in
|
||||||
|
// the gate can. Closing that would need a deterministic trigger oracle, which
|
||||||
|
// does not exist.
|
||||||
//
|
//
|
||||||
// Usage:
|
// Usage:
|
||||||
// npm run skills:eval # every model-invoked skill's cases
|
// npm run skills:eval # every model-invoked skill's cases
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue