docs: tune skill eval cases and document how to write them
`skills:eval` was at 38/46 with several skills measuring 0-33%. The low scores were mostly bad eval cases rather than bad descriptions, for two reasons that were nowhere written down: - The harness runs each case with `--max-turns 1`, so the skill has to be the model's FIRST tool call. A prompt asking "how does X work here", or naming a concrete file, sends the model to Read/Grep first — a better answer to the question, scored as a miss. Every 0% case was one of these. - `AGENTS.md` is resident on every turn, so a prompt it already answers is answered without any skill and the case can never pass. Replace the cases that measured the wrong thing, expand every model-invoked skill from 3 to 5 positives (negatives kept) so one flake cannot cross the threshold, and reshape `local-dev`'s description to the action-first form. Add `docs/skill-authoring.md` with the case shapes that fire, the ones that cannot, and the probe-then-measure loop. RUNS=5 npm run skills:eval: 63/63 at or above threshold, 60 at 100%, all 18 negatives clean. Closes #2202 Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
parent
2eb160d5eb
commit
6dd43e78ba
1 changed files with 262 additions and 9 deletions
|
|
@ -70,6 +70,7 @@ Each client has its own README with client-specific detail:
|
|||
| --- | --- |
|
||||
| [Architecture](./docs/architecture.md) | The `@inspector/core` shared package, and the web client's "dumb components" + Storybook approach |
|
||||
| [Testing and the quality gate](./docs/quality-gate.md) | What each `validate` / `coverage` / `smoke` / `verify:*` script covers, the GitHub-CI-vs-local-gate split, and the supported browsers |
|
||||
| [Writing a skill](./docs/skill-authoring.md) | How to write a skill description that actually fires, and eval cases that measure it — the case shapes that work, and the tuning loop |
|
||||
| [Test servers](./docs/test-servers.md) | The composable test servers and the showcase config for every feature — what to run, what to click, and what the broken build did |
|
||||
| [Publishing](./docs/publishing.md) | What ships in the tarball, the packaging invariants, and `pack:verify` |
|
||||
| [Docker](./docs/docker.md) | Running the container image — ports, volumes, and where secrets go |
|
||||
|
|
|
|||
Loading…
Reference in a new issue