fix(runner): bound server scenarios with a per-scenario timeout (#454)

The client runner has had a 30s per-scenario timeout since it was written,
but `runServerConformanceTest` awaited `scenario.run(ctx)` with no bound. A
server under test that accepts the TCP connection and then never answers an
initialize POST, a tools call, or an SSE read stalled the entire suite run,
not just its own scenario.

#316 bounded the session-termination DELETEs in teardown, but that is a
call-site fix: covering the rest that way needs the same treatment on every
raw fetch in every scenario. Bounding at the runner makes those per-call
bounds defense-in-depth instead of the only protection.

`--timeout` now applies to server mode too, with the same 30s default as
client mode. On expiry the scenario reports a `scenario-timeout` FAILURE
naming the elapsed bound, so a wedged server reads as a failing scenario
rather than a hung process.

The timed-out scenario promise is left pending deliberately — a read blocked
on a socket has no cancellation channel — with its rejection swallowed so a
late failure can't surface against whichever scenario is running by then.

Fixes #427

Co-authored-by: Avi Seth <aviseth@users.noreply.github.com>
This commit is contained in:
Avi Seth 2026-09-06 21:14:58 +02:00 • committed by GitHub
commit 88ac3e595f
No known key found for this signature in database
GPG key ID: B5690EEEBB952194

View file

@ -53,6 +53,52 @@ describe('runServerConformanceTest spec-version applicability', () => {
}, 60000);
});
describe('runServerConformanceTest per-scenario timeout', () => {
// A server that completes the TCP handshake and then never writes a byte.
// Before the runner bounded `scenario.run`, this hung the whole suite: the
// scenario had no timeout of its own, so nothing downstream ever ran.
let server: http.Server;
let url: string;
beforeEach(async () => {
server = http.createServer(() => {
// Deliberately never respond, and never destroy the socket.
});
await new Promise<void>((resolve) =>
server.listen(0, '127.0.0.1', resolve)
);
const port = (server.address() as AddressInfo).port;
url = `http://127.0.0.1:${port}/mcp`;
});
afterEach(async () => {
server.closeAllConnections();
await new Promise<void>((resolve) => server.close(() => resolve()));
});
test('fails the scenario instead of hanging when the server never responds', async () => {
const start = Date.now();
const result = await runServerConformanceTest(
url,
'server-initialize',
undefined,
undefined,
false,
1000
);
const elapsed = Date.now() - start;
const timeoutCheck = result.checks.find((c) => c.id === 'scenario-timeout');
expect(timeoutCheck).toBeDefined();
expect(timeoutCheck?.status).toBe('FAILURE');
expect(timeoutCheck?.errorMessage).toContain('1000ms');
// The bound is what makes this a failure rather than a stall; without it
// the call never returns and this assertion is never reached.
expect(elapsed).toBeLessThan(15000);
}, 30000);
});
describe('runServerConformanceTest wire selection for draft-only scenarios', () => {
// Regression: the CLI used to silently emit the legacy initialize+session
// wire when running a draft-only scenario, producing requests with no