From 4520e6aa2c042575e8f24d06978df4681c83cc07 Mon Sep 17 00:00:00 2001 From: cliffhall Date: Sat, 29 Aug 2026 00:53:49 -0400 Subject: [PATCH] docs: state the gap the gate cannot cover `verify:skills` checks that a skill is well-formed and that its cases exist; only `skills:eval` observes whether it actually fires, and that cannot be gated. Say so in the script header and in AGENTS.md rather than leaving the split to be inferred. Signed-off-by: cliffhall --- AGENTS.md | 6 ++++++ scripts/skill-eval.mjs | 15 ++++++++++++--- 2 files changed, 18 insertions(+), 3 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index fa477277..3461d9a2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -180,6 +180,12 @@ that from happening: ("how does the `@inspector/core` alias resolve?") invites a `Read`, which is a *better* answer than a skill. Good cases are "how do I / where does this go" questions whose answer is a procedure. + ⚠️ **The gate cannot catch a description that never matches.** `verify:skills` + checks that a skill is well-formed and that its cases exist; only + `skills:eval` observes whether it actually fires, and that cannot be gated — + it spends metered model calls, its result is a hit rate rather than a verdict, + and it goes red on a rate limit. So the eval is a tuning tool you run + deliberately, and a case below threshold is a signal, not a build break. 4. **Re-check the listing budget when adding a skill.** Claude Code loads a listing of every skill's name and description into context, truncates it when it overflows, and drops the least-invoked entries **first** — which are exactly the diff --git a/scripts/skill-eval.mjs b/scripts/skill-eval.mjs index 454bb368..971e8587 100755 --- a/scripts/skill-eval.mjs +++ b/scripts/skill-eval.mjs @@ -11,9 +11,18 @@ // regression, and it is the failure nobody notices by hand. `verify:skills` // requires both. // -// NOT part of `validate`, `local:gate`, or CI: it needs the `claude` CLI and -// real model calls, so it is metered and non-deterministic. Run it when adding -// a skill or editing a model-invoked skill's description. +// NOT part of `validate`, `local:gate`, or CI, and that is a decision rather +// than an omission. A trigger eval cannot be a gate: it spends metered model +// calls on every push, the measurement IS a hit rate over samples so it is +// non-deterministic by construction, and it goes red on a rate limit or an +// expired token — failures unrelated to the diff, whose first consequence is +// that people stop trusting the gate. Run it when adding a skill or editing a +// model-invoked skill's description. +// +// The cost of that split, stated plainly: `verify:skills` cannot detect a +// well-formed skill whose description simply never matches anything. Nothing in +// the gate can. Closing that would need a deterministic trigger oracle, which +// does not exist. // // Usage: // npm run skills:eval # every model-invoked skill's cases