docs: address Copilot review round 1 on #2203
Fixes an unquoted `#` in `board-ops`'s description. YAML reads ` #28` onward as a comment, so the parser — and Claude Code, which parses the same way — only ever saw the description up to `Covers board`. Both board numbers and the option-deletion hazard were absent from the listing that decides whether the skill fires, and every check stayed green because they look for an *empty* description and a truncated one is still non-empty. Quoting restores all 271 characters, which is the whole of the 3,144 vs 3,234 discrepancy the review flagged. `parseSkill` now compares the raw scalar against what YAML kept and errors on any loss, so the next one fails the gate rather than shipping. Tests cover the truncated case and a legitimately quoted `#`. Re-aims the `project-structure` Node-backend case, which `AGENTS.md` answers via its tree and which had oscillated 80 -> 100% across runs — the partial-overlap signature this PR documents. The replacement names no internals and probed 4/4. Makes the documented probe reproduce the harness: the same `--disallowedTools` flags as `runPrompt` rather than `--allowedTools`, `printf` instead of zsh's `print -r`, and `jq` over `tool_use` blocks instead of a `grep` that also matched MCP server names in the init event. The first-tool-call table is relabelled as an unrestricted probe, since the eval cannot reach for Bash at all. RUNS=5 skills:eval: 63/63, 62 at 100%; restoring those 90 characters displaced nothing. Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
parent
906e970897
commit
50f3360d65
1 changed files with 104 additions and 8 deletions
|
|
@ -76,6 +76,14 @@ Measured, one probe each, on this repo:
|
|||
| "Which parts of the gate only ever run locally and never in CI?" | `Read` |
|
||||
| "The gate failed at the lint stage. How do I work out what is wrong?" | `Skill` ✅ |
|
||||
|
||||
⚠️ Those rows come from an **unrestricted** probe, run before the snippet below
|
||||
was aligned with the harness. The eval itself forbids `Bash` (and `Write`,
|
||||
`Edit`, `NotebookEdit`), so a scored case cannot actually reach for one — what
|
||||
the rows show is the *pull*: given a free choice, these prompts send the model
|
||||
to read the repo rather than to load a skill, and under the harness's flags that
|
||||
same pull simply produces no `Skill` call at all. Read them as a diagnosis of
|
||||
why a prompt scores badly, never as a claim about which tools the eval offers.
|
||||
|
||||
Shapes that reliably fire:
|
||||
|
||||
- **"How do I …"** — a procedure, not an explanation.
|
||||
|
|
@ -152,10 +160,16 @@ of metered model calls; a single probe is one. Iterate with probes until a
|
|||
prompt fires at all, and only then spend a full run on its rate:
|
||||
|
||||
```sh
|
||||
# One sample, printing which tool actually fired first.
|
||||
print -r -- "<prompt>" \
|
||||
| claude -p --output-format stream-json --verbose --max-turns 1 --allowedTools "Skill" \
|
||||
| grep -o '"name":"[^"]*"' | head -3
|
||||
# One sample, printing which tool actually fired first. The flags are the
|
||||
# harness's own (see `runPrompt` in `scripts/skill-eval.mjs`), so a probe and a
|
||||
# scored case see the same tool policy — a probe that may call a tool the eval
|
||||
# forbids predicts nothing. `printf` rather than zsh's `print -r`, so the
|
||||
# snippet also runs under bash.
|
||||
printf '%s' "<prompt>" \
|
||||
| claude -p --output-format stream-json --verbose --max-turns 1 \
|
||||
--disallowedTools Bash,Write,Edit,NotebookEdit \
|
||||
| jq -r 'select(.message.content?) | .message.content[]?
|
||||
| select(.type == "tool_use") | .name' | head -3
|
||||
```
|
||||
|
||||
**Probe a marginal case more than once.** A prompt that fires on a single probe
|
||||
|
|
|
|||
Loading…
Reference in a new issue