ruby-sdk/test/mcp/tool_test.rb
Koichi ITO 9d4dcbf1a4 Speed Up Tool::Schema Validation by 5x to 100x
## Motivation and Context

`MCP::Tool::Schema` previously used the pure-Ruby `json-schema` gem for
both construction-time metaschema validation and runtime argument /
result validation. For deep schemas this cost ~32-100ms per
construction (see issue #364).

`json_schemer` (https://rubygems.org/gems/json_schemer, v2.5.0) is
an actively maintained alternative that is much faster on this workload.
Switching reduces cold-start warming pressure in horizontally scaled
multi-tenant deployments (the scenario raised in #364) and brings
runtime data validation down to sub-millisecond per call once
the per-instance schemer is memoized.

The implementation preserves the legacy behavior of the `json-schema`
based code. Metaschema is pinned to draft-04 (`meta_schema:` option);
the `$schema` dialect URI is still emitted in `to_h` as 2020-12
but runtime validation continues to use draft-04, exactly as before.
Moving the runtime validator itself to 2020-12 is out of scope for
this PR and warrants its own discussion. `format` keywords are not enforced
(`format: false` option), matching what `json-schema`'s draft-04 path did.
Malformed schemas (e.g. an invalid `pattern` regex) continue to
surface as `ArgumentError, "Invalid JSON Schema: ..."`;
the `RegexpError` `json_schemer` raises during construction is wrapped.
`Symbol` values in argument data are coerced to strings inside
the internal `stringify` helper so they validate against `type: "string"`
the same way they did before.

The public `Schema#schema` accessor is dropped. It was a refactor
artifact from #198 (the consolidation of `InputSchema`'s
`properties` / `required` readers into a single hash) with no callers
in `lib/`; tests that needed the merged hash now read it through
`to_h`. The accessor's only remaining role would have been to expose
a mutation path that could desynchronize the memoized `@schemer`,
which this change removes.

Runtime dependency delta: drops `json-schema` (and `addressable`),
adds `json_schemer`, `hana`, `regexp_parser`, and `simpleidn`. Each added
gem is a single-purpose library tied to JSON Schema spec compliance.

Closes #364.

## How Has This Been Tested?

Side-by-side benchmark on representative schemas (simple, with `$ref` /
`additionalProperties`, nested, depth 20, depth 40):
construction-time metaschema validation is 4.7x to ~100x faster;
runtime data validation with the memoized schemer is sub-millisecond
across all sizes.

Four existing tests had their stub targets or message assertions updated
to track the new validator while preserving the original test intent.
The "unexpected errors bubble up" tests in `input_schema_test.rb`
and `output_schema_test.rb` now stub `JSONSchemer::Schema#validate`
instead of `JSON::Validator.fully_validate`. The cache tests in
`schema_test.rb` now stub `JSONSchemer::Schema#validate_schema`.
The two "detailed error message" tests in `tool_test.rb` switched from
asserting `json-schema`'s exact wording to format-agnostic substrings
(`"properties/count/minimum"`, `"number"`). The "required arguments are
converted to strings" test in `input_schema_test.rb` now reads the result
through `to_h[:required]` instead of the removed `schema` accessor.

Regression tests added for the legacy-behavior preservations above:
`format` keyword is not enforced, invalid `pattern` raises `ArgumentError`
(not `RegexpError`), and `Symbol` values validate against `type: "string"`.

## Breaking Changes

`Schema#schema` is no longer a public method. The accessor had no callers
outside the test suite, but anyone reading the internal representation
directly (rather than through `to_h`) will need to switch.
`Schema` / `InputSchema` / `OutputSchema` otherwise keep
their constructor signatures, `to_h` output, `ValidationError` /
`ArgumentError` raise points, and the `$schema` dialect emission.
The validator error message wording inside `ArgumentError` and
`ValidationError` now comes from `json_schemer` and still includes
the JSON pointer path and a description of the mismatch.
2026-06-01 21:47:30 +09:00

552 lines
18 KiB
Ruby

# frozen_string_literal: true
require "test_helper"
module MCP
class ToolTest < ActiveSupport::TestCase
class TestTool < Tool
tool_name "test_tool"
description "a test tool for testing"
icons [Icon.new(mime_type: "image/png", sizes: ["48x48", "96x96"], src: "https://example.com", theme: "light")]
input_schema({ properties: { message: { type: "string" } }, required: ["message"] })
annotations(
destructive_hint: false,
idempotent_hint: true,
open_world_hint: false,
read_only_hint: true,
title: "Test Tool",
)
meta(
foo: "bar",
)
class << self
def call(message:, server_context: nil)
Tool::Response.new([{ type: "text", content: "OK" }])
end
end
end
test "#to_h returns a hash including name, description, icons, and inputSchema" do
expected = {
name: "mock_tool",
title: "Mock Tool",
description: "a mock tool for testing",
icons: [{ mimeType: "image/png", sizes: ["48x48", "96x96"], src: "https://example.com", theme: "light" }],
inputSchema: { "$schema": "https://json-schema.org/draft/2020-12/schema", type: "object" },
}
tool = Tool.define(
name: "mock_tool",
title: "Mock Tool",
description: "a mock tool for testing",
icons: [Icon.new(mime_type: "image/png", sizes: ["48x48", "96x96"], src: "https://example.com", theme: "light")],
)
assert_equal(expected, tool.to_h)
end
test "#to_h does not have `:title` key when title is omitted" do
tool = Tool.define(
name: "mock_tool",
description: "a mock tool for testing",
)
refute tool.to_h.key?(:title)
end
test "#to_h does not have `:icons` key when icons is empty" do
tool = Tool.define(
name: "tool_without_icons",
description: "a tool without icons",
)
refute tool.to_h.key?(:icons)
end
test "#to_h does not have `:icons` key when icons is nil" do
tool = Tool.define(
name: "tool_without_icons",
description: "a tool without icons",
icons: nil,
)
refute tool.to_h.key?(:icons)
end
test "#to_h includes annotations when present" do
tool = TestTool
expected_annotations = {
destructiveHint: false,
idempotentHint: true,
openWorldHint: false,
readOnlyHint: true,
title: "Test Tool",
}
assert_equal expected_annotations, tool.to_h[:annotations]
end
test "#to_h includes meta when present" do
tool = TestTool
expected_meta = {
foo: "bar",
}
assert_equal expected_meta, tool.to_h[:_meta]
end
test "#call invokes the tool block and returns the response" do
tool = TestTool
response = tool.call(message: "test")
assert_equal [{ type: "text", content: "OK" }], response.content
refute response.error?
end
test "allows declarative definition of tools as classes" do
class MockTool < Tool
tool_name "my_mock_tool"
description "a mock tool for testing"
input_schema({ properties: { message: { type: "string" } }, required: ["message"] })
end
tool = MockTool
assert_equal "my_mock_tool", tool.name_value
assert_equal "a mock tool for testing", tool.description
assert_equal(
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
type: "object",
properties: { message: { type: "string" } },
required: ["message"],
},
tool.input_schema.to_h,
)
end
test "defaults to class name as tool name" do
class DefaultNameTool < Tool
end
tool = DefaultNameTool
assert_equal "default_name_tool", tool.tool_name
end
test "input schema defaults to an empty hash" do
class NoInputSchemaTool < Tool; end
tool = NoInputSchemaTool
expected = { "$schema": "https://json-schema.org/draft/2020-12/schema", type: "object" }
assert_equal expected, tool.input_schema.to_h
end
test "accepts input schema as an InputSchema object" do
class InputSchemaTool < Tool
input_schema InputSchema.new(properties: { message: { type: "string" } }, required: ["message"])
end
tool = InputSchemaTool
expected = {
"$schema": "https://json-schema.org/draft/2020-12/schema",
type: "object",
properties: { message: { type: "string" } },
required: ["message"],
}
assert_equal expected, tool.input_schema.to_h
end
test "raises detailed error message for invalid schema" do
error = assert_raises(ArgumentError) do
Class.new(MCP::Tool) do
input_schema(
properties: {
count: { type: "integer", minimum: "not a number" },
},
required: ["count"],
)
end
end
assert_includes error.message, "Invalid JSON Schema"
assert_includes error.message, "properties/count/minimum"
assert_includes error.message, "number"
end
test ".define allows definition of simple tools with a block" do
tool = Tool.define(
name: "mock_tool",
title: "Mock Tool",
description: "a mock tool for testing",
) do
Tool::Response.new([{ type: "text", content: "OK" }])
end
assert_equal "mock_tool", tool.name_value
assert_equal "a mock tool for testing", tool.description
assert_equal Tool::InputSchema.new, tool.input_schema
end
test ".define allows definition of tools with annotations" do
tool = Tool.define(
name: "mock_tool",
title: "Mock Tool",
description: "a mock tool for testing",
annotations: {
read_only_hint: true,
title: "Mock Tool",
},
) do
Tool::Response.new([{ type: "text", content: "OK" }])
end
assert_equal "mock_tool", tool.name_value
assert_equal "Mock Tool", tool.title
assert_equal "a mock tool for testing", tool.description
assert_equal tool.input_schema, Tool::InputSchema.new
assert_equal({ destructiveHint: true, idempotentHint: false, openWorldHint: true, readOnlyHint: true, title: "Mock Tool" }, tool.annotations_value.to_h)
end
test ".define allows definition of tools with meta" do
tool = Tool.define(
name: "mock_tool",
title: "Mock Tool",
description: "a mock tool for testing",
meta: { foo: "bar" },
) do |_|
Tool::Response.new([{ type: "text", content: "OK" }])
end
assert_equal "mock_tool", tool.name_value
assert_equal "Mock Tool", tool.title
assert_equal "a mock tool for testing", tool.description
assert_equal tool.input_schema, Tool::InputSchema.new
assert_equal({ foo: "bar" }, tool.meta_value)
end
test "Tool class method annotations can be set and retrieved" do
class AnnotationsTestTool < Tool
tool_name "annotations_test"
annotations(
read_only_hint: true,
title: "Annotations Test",
)
end
tool = AnnotationsTestTool
assert_instance_of Tool::Annotations, tool.annotations_value
assert_equal "Annotations Test", tool.annotations_value.title
assert tool.annotations_value.read_only_hint
end
test "Tool class method annotations can be updated" do
class UpdatableAnnotationsTool < Tool
tool_name "updatable_annotations"
end
tool = UpdatableAnnotationsTool
tool.annotations(title: "Initial")
assert_equal "Initial", tool.annotations_value.title
tool.annotations(title: "Updated")
assert_equal "Updated", tool.annotations_value.title
end
test "Tool class method meta can be set and retrieved" do
class MetaTestTool < Tool
tool_name "meta_test"
meta(foo: "bar")
end
tool = MetaTestTool
assert_instance_of Hash, tool.meta_value
assert_equal "bar", tool.meta_value[:foo]
end
test "Tool class method meta can be updated" do
class UpdatableMetaTool < Tool
tool_name "updatable_meta"
end
tool = UpdatableMetaTool
tool.meta(foo: "baz")
assert_equal({ foo: "baz" }, tool.meta_value)
tool.meta(foo: "qux")
assert_equal({ foo: "qux" }, tool.meta_value)
end
test "#call with Sorbet typed tools invokes the tool block and returns the response" do
skip "sorbet-static-and-runtime requires Ruby 3.0+." if RUBY_VERSION < "3.0"
class TypedTestTool < Tool
tool_name "test_tool"
description "a test tool for testing"
input_schema({ properties: { message: { type: "string" } }, required: ["message"] })
class << self
extend T::Sig
sig { params(message: String, server_context: T.nilable(T.untyped)).returns(Tool::Response) }
def call(message:, server_context: nil)
Tool::Response.new([{ type: "text", content: "OK" }])
end
end
end
tool = TypedTestTool
response = tool.call(message: "test")
assert_equal [{ type: "text", content: "OK" }], response.content
refute response.error?
end
class TestToolWithoutServerContext < Tool
tool_name "test_tool_without_server_context"
description "a test tool for testing without server context"
input_schema({ properties: { message: { type: "string" } }, required: ["message"] })
class << self
def call(message:)
Tool::Response.new([{ type: "text", content: "OK" }])
end
end
end
class TestToolWithoutRequired < Tool
tool_name "test_tool_without_required"
description "a test tool for testing without required server context"
class << self
def call(message, server_context: nil)
Tool::Response.new([{ type: "text", content: "OK" }])
end
end
end
test "tool call without server context" do
tool = TestToolWithoutServerContext
response = tool.call(message: "test")
assert_equal [{ type: "text", content: "OK" }], response.content
end
test "tool call with server context and without required" do
tool = TestToolWithoutRequired
response = tool.call("test", server_context: { foo: "bar" })
assert_equal [{ type: "text", content: "OK" }], response.content
end
test "input_schema accepts $ref in schema" do
schema_with_ref = {
properties: {
foo: { "$ref" => "#/definitions/bar" },
},
required: ["foo"],
definitions: {
bar: { type: "string" },
},
}
tool_class = Class.new(MCP::Tool) do
input_schema schema_with_ref
end
assert_equal "#/definitions/bar", tool_class.input_schema.to_h[:properties][:foo][:$ref]
end
test "#to_h includes outputSchema when present" do
expected = {
name: "mock_tool",
title: "Mock Tool",
description: "a mock tool for testing",
icons: [{ mimeType: "image/png", sizes: ["48x48", "96x96"], src: "https://example.com", theme: "light" }],
inputSchema: { "$schema": "https://json-schema.org/draft/2020-12/schema", type: "object" },
outputSchema: {
"$schema": "https://json-schema.org/draft/2020-12/schema",
type: "object",
properties: { result: { type: "string" } },
required: ["result"],
},
}
tool = Tool.define(
name: "mock_tool",
title: "Mock Tool",
description: "a mock tool for testing",
icons: [Icon.new(mime_type: "image/png", sizes: ["48x48", "96x96"], src: "https://example.com", theme: "light")],
output_schema: { properties: { result: { type: "string" } }, required: ["result"] },
)
assert_equal expected, tool.to_h
end
test "#to_h does not include outputSchema when not set" do
tool = Tool.define(
name: "mock_tool",
description: "a mock tool for testing",
)
refute tool.to_h.key?(:outputSchema)
end
test "output_schema defaults to nil" do
class NoOutputSchemaTool < Tool; end
tool = NoOutputSchemaTool
assert_nil tool.output_schema
end
test "accepts output_schema as a hash" do
class HashOutputSchemaTool < Tool
output_schema({ properties: { result: { type: "string" } }, required: ["result"] })
end
tool = HashOutputSchemaTool
expected = {
"$schema": "https://json-schema.org/draft/2020-12/schema",
type: "object",
properties: { result: { type: "string" } },
required: ["result"],
}
assert_equal expected, tool.output_schema.to_h
end
test "accepts output_schema as an OutputSchema object" do
class OutputSchemaObjectTool < Tool
output_schema Tool::OutputSchema.new(properties: { result: { type: "string" } }, required: ["result"])
end
tool = OutputSchemaObjectTool
expected = {
"$schema": "https://json-schema.org/draft/2020-12/schema",
type: "object",
properties: { result: { type: "string" } },
required: ["result"],
}
assert_equal expected, tool.output_schema.to_h
end
test "output_schema raises detailed error message for invalid schema" do
error = assert_raises(ArgumentError) do
Class.new(MCP::Tool) do
output_schema(
properties: {
count: { type: "integer", minimum: "not a number" },
},
required: ["count"],
)
end
end
assert_includes error.message, "Invalid JSON Schema"
assert_includes error.message, "properties/count/minimum"
assert_includes error.message, "number"
end
test "output_schema accepts $ref in schema" do
schema_with_ref = {
properties: {
foo: { "$ref" => "#/definitions/bar" },
},
required: ["foo"],
definitions: {
bar: { type: "string" },
},
}
tool_class = Class.new(MCP::Tool) do
output_schema schema_with_ref
end
assert_equal "#/definitions/bar", tool_class.output_schema.to_h[:properties][:foo][:$ref]
end
test ".define allows definition of tools with output_schema" do
tool = Tool.define(
name: "mock_tool",
title: "Mock Tool",
description: "a mock tool for testing",
output_schema: { properties: { result: { type: "string" } }, required: ["result"] },
) do
Tool::Response.new([{ type: "text", content: "OK" }])
end
assert_equal "mock_tool", tool.name_value
assert_equal "a mock tool for testing", tool.description
assert_instance_of Tool::OutputSchema, tool.output_schema
expected_output_schema = {
"$schema": "https://json-schema.org/draft/2020-12/schema",
type: "object",
properties: { result: { type: "string" } },
required: ["result"],
}
assert_equal expected_output_schema, tool.output_schema.to_h
end
class TestToolWithOutputSchema < Tool
tool_name "test_tool_with_output"
description "a test tool with output schema"
input_schema({ properties: { message: { type: "string" } }, required: ["message"] })
output_schema({ properties: { result: { type: "string" }, success: { type: "boolean" } }, required: ["result", "success"] })
class << self
def call(message:, server_context: nil)
Tool::Response.new([{ type: "text", content: "OK" }])
end
end
end
test "declarative definition of tools with output schema" do
tool = TestToolWithOutputSchema
assert_equal "test_tool_with_output", tool.name_value
assert_equal "a test tool with output schema", tool.description
expected_input = {
"$schema": "https://json-schema.org/draft/2020-12/schema",
type: "object",
properties: { message: { type: "string" } },
required: ["message"],
}
assert_equal expected_input, tool.input_schema.to_h
expected_output = {
"$schema": "https://json-schema.org/draft/2020-12/schema",
type: "object",
properties: { result: { type: "string" }, success: { type: "boolean" } },
required: ["result", "success"],
}
assert_equal expected_output, tool.output_schema.to_h
end
test "accepts valid tool names" do
assert Tool.define(name: "getUser")
assert Tool.define(name: "DATA_EXPORT_v2")
assert Tool.define(name: "admin.tools.list")
assert Tool.define(name: "a" * 128)
end
test "raises an error when tool name is empty in class definition" do
error = assert_raises(ArgumentError) do
class EmptyTitleNameTool < Tool
tool_name ""
end
end
assert_equal("Tool names should be between 1 and 128 characters in length (inclusive).", error.message)
end
test "allows nil tool name in class definition" do
assert_nothing_raised do
class EmptyTitleNameTool < Tool
tool_name nil
end
end
end
test "raises an error when tool name is empty" do
error = assert_raises(ArgumentError) { Tool.define(name: "") }
assert_equal("Tool names should be between 1 and 128 characters in length (inclusive).", error.message)
end
test "raises an error when tool name exceeds 128 characters" do
error = assert_raises(ArgumentError) { Tool.define(name: "a" * 129) }
assert_equal("Tool names should be between 1 and 128 characters in length (inclusive).", error.message)
end
test "raises an error when tool name includes invalid characters (e.g., spaces)" do
error = assert_raises(ArgumentError) { Tool.define(name: "foo bar") }
assert_equal(<<~MESSAGE, error.message)
Tool names only allowed characters: uppercase and lowercase ASCII letters (A-Z, a-z), digits (0-9), underscore (_), hyphen (-), and dot (.).
MESSAGE
end
end
end