registry/internal/api/router/router.go
Radoslav Dimitrov 4e147072f0
metrics: stop counting client-cancelled requests as server errors (#1255)
## Summary

- Remap recorded status to `499` (NGINX-style "client closed") when
`ctx.Context().Err()` is `context.Canceled` after the handler returns
- Skip the `http_errors_total` increment for `499`s so the availability
metric reflects server-visible errors only

## Why

Under scraper-driven load, clients with short timeouts paginate
`/v0.1/servers`, give up while PG is busy, and close the TCP connection.
The handler's DB iteration returns `context canceled`, the handler
converts that to `huma.Error500InternalServerError`, and tries to write
a response to the now-closed socket. NGINX records `499` (client closed)
and never delivers anything to the client — but the registry's
middleware records `status_code=500` and bumps
`mcp_registry_http_errors_total`.

That's what's been firing the `Availability dropped below 95%` alert
during the daily 17:00 UTC bursts. Recent prod numbers from one pod
(~19h uptime):

```
mcp_registry_http_errors_total{path="/v0.1/servers", status_code="500"} 7396
mcp_registry_http_errors_total{path="/v0.1/servers/{serverName}/versions", status_code="500"} 283
mcp_registry_http_errors_total{path="/v0/servers", status_code="500"} 352
```

…against zero 5xx in NGINX ingress logs over the same window. The alert
was correct *given the data it had*; the data was misclassifying client
cancellations as server errors.

## What this does

In `internal/api/router/router.go` (the metric middleware): after
`next(ctx)`, check `ctx.Context().Err()`. If it's `context.Canceled`,
override `statusCode` to `499` and skip the error counter. Requests
counter and duration histogram still record so the cancellation rate
stays visible.

`context.DeadlineExceeded` is intentionally **not** remapped — that
would indicate a server-side timeout if we ever add per-request
deadlines, and should still count as a server error.

## What this does *not* do

- Doesn't reduce the underlying load (ServiceNow + notion catalog
scrapers driving the daily peak — separate work, see the `updated_since`
outreach we're doing).
- Doesn't change handler behaviour — handlers still call
`huma.Error500…` on `context.Canceled`. We could short-circuit earlier
(return without writing) to save the work, but that's a bigger refactor
and the metric fix unblocks the alert immediately.

## Companion change (no code, just a Grafana annotation tweak)

The `Availability dropped below 95%` alert (UID `bexrc60etvhmoa`)
currently has the annotation:

> No. of 5xx are increased from acceptable limit i.e. 5% of total
throughput

The query is on `mcp_registry_http_errors_total`, not 5xx specifically.
After this change the count is meaningful again, but the wording is
still misleading. Suggest updating to:

> Internal error rate exceeded 5% of total throughput

(no PromQL change required.)

## Test plan

- [x] `go test ./internal/api/router/...` — new test passes
- [x] `golangci-lint run ./internal/...` — clean
- [ ] After merge, watch a daily 17:00 UTC burst —
`mcp_registry_http_errors_total{status_code="500",
path="/v0.1/servers"}` should stay flat while
`mcp_registry_http_requests_total{status_code="499",
path="/v0.1/servers"}` grows. Availability alert should not fire on
cancellation-only bursts.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 05:58:05 -04:00

251 lines
8.5 KiB
Go

// Package router contains API routing logic
package router
import (
"context"
"encoding/json"
"errors"
"fmt"
"net/http"
"strings"
"time"
"github.com/danielgtaylor/huma/v2"
"github.com/danielgtaylor/huma/v2/adapters/humago"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/metric"
v0 "github.com/modelcontextprotocol/registry/internal/api/handlers/v0"
"github.com/modelcontextprotocol/registry/internal/config"
"github.com/modelcontextprotocol/registry/internal/service"
"github.com/modelcontextprotocol/registry/internal/telemetry"
)
// statusClientClosed mirrors NGINX's non-standard 499 — used for requests
// where the client disconnected before we finished. Distinguishing these from
// real server errors keeps the availability metric meaningful under bursts of
// scraper traffic that time out and reconnect.
const statusClientClosed = 499
// Middleware configuration options
type middlewareConfig struct {
skipPaths map[string]bool
}
type MiddlewareOption func(*middlewareConfig)
// getRoutePath extracts the route pattern from the context
func getRoutePath(ctx huma.Context) string {
// Try to get the operation from context
if op := ctx.Operation().Path; op != "" {
return ctx.Operation().Path
}
// Fallback to URL path (less ideal for metrics as it includes path parameters)
return ctx.URL().Path
}
func MetricTelemetryMiddleware(metrics *telemetry.Metrics, options ...MiddlewareOption) func(huma.Context, func(huma.Context)) {
config := &middlewareConfig{
skipPaths: make(map[string]bool),
}
for _, opt := range options {
opt(config)
}
return func(ctx huma.Context, next func(huma.Context)) {
path := ctx.URL().Path
// Skip instrumentation for specified paths
// extract the last part of the path to match against skipPaths
pathParts := strings.Split(path, "/")
pathToMatch := "/" + pathParts[len(pathParts)-1]
if config.skipPaths[pathToMatch] || config.skipPaths[path] {
next(ctx)
return
}
start := time.Now()
method := ctx.Method()
routePath := getRoutePath(ctx)
next(ctx)
duration := time.Since(start).Seconds()
statusCode := ctx.Status()
// If the client disconnected before the handler finished, the handler
// likely converted the resulting context.Canceled into a huma 5xx and
// tried to write a response to a closed socket. NGINX records that
// case as a 499 (client closed). Without this remap we count it as a
// server error: a single ServiceNow-style burst that times out a
// few thousand list-servers requests inflates http_errors_total even
// though no client ever saw a 5xx, and the availability alert fires
// on what is effectively just slow responses.
//
// Only context.Canceled is remapped — context.DeadlineExceeded would
// indicate a server-side timeout we set ourselves and should still
// count as a server error if/when we add per-request deadlines.
if reqErr := ctx.Context().Err(); reqErr != nil && errors.Is(reqErr, context.Canceled) {
statusCode = statusClientClosed
}
// Combine common and custom attributes
attrs := []attribute.KeyValue{
attribute.String("method", method),
attribute.String("path", routePath),
attribute.Int("status_code", statusCode),
}
// Record metrics
metrics.Requests.Add(ctx.Context(), 1, metric.WithAttributes(attrs...))
// Skip the error counter for client-closed requests so the availability
// metric reflects server-visible errors only.
if statusCode >= 400 && statusCode != statusClientClosed {
metrics.ErrorCount.Add(ctx.Context(), 1, metric.WithAttributes(attrs...))
}
metrics.RequestDuration.Record(ctx.Context(), duration, metric.WithAttributes(attrs...))
}
}
// WithSkipPaths allows skipping instrumentation for specific paths
func WithSkipPaths(paths ...string) MiddlewareOption {
return func(c *middlewareConfig) {
for _, path := range paths {
c.skipPaths[path] = true
}
}
}
// handle404 returns a helpful 404 error with suggestions for common mistakes
func handle404(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/problem+json")
w.Header().Set("X-Content-Type-Options", "nosniff")
w.WriteHeader(http.StatusNotFound)
path := r.URL.Path
detail := "Endpoint not found. See /docs for the API documentation."
// Provide suggestions for common API endpoint mistakes
if !strings.HasPrefix(path, "/v0/") && !strings.HasPrefix(path, "/v0.1/") {
detail = fmt.Sprintf(
"Endpoint not found. Did you mean '%s' or '%s'? See /docs for the API documentation.",
"/v0.1"+path,
"/v0"+path,
)
}
errorBody := map[string]interface{}{
"title": "Not Found",
"status": 404,
"detail": detail,
}
// Use JSON marshal to ensure consistent formatting
jsonData, err := json.Marshal(errorBody)
if err != nil {
http.Error(w, "Internal Server Error", http.StatusInternalServerError)
return
}
_, err = w.Write(jsonData)
if err != nil {
http.Error(w, "Internal Server Error", http.StatusInternalServerError)
}
}
// NewHumaAPI creates a new Huma API with all routes registered
func NewHumaAPI(cfg *config.Config, registry service.RegistryService, mux *http.ServeMux, metrics *telemetry.Metrics, versionInfo *v0.VersionBody) huma.API {
// Create Huma API configuration
humaConfig := huma.DefaultConfig("Official MCP Registry", "1.0.0")
humaConfig.Info.Description = "A community driven registry service for Model Context Protocol (MCP) servers.\n\n[GitHub repository](https://github.com/modelcontextprotocol/registry) | [Documentation](https://github.com/modelcontextprotocol/registry/tree/main/docs)"
// Disable $schema property in responses: https://github.com/danielgtaylor/huma/issues/230
humaConfig.CreateHooks = []func(huma.Config) huma.Config{}
// Create a new API using humago adapter for standard library
api := humago.New(mux, humaConfig)
// Add OpenAPI tag metadata with descriptions
api.OpenAPI().Tags = []*huma.Tag{
{
Name: "servers",
Description: "Operations for discovering and retrieving MCP servers",
},
{
Name: "publish",
Description: "Operations for publishing MCP servers to the registry",
},
{
Name: "auth",
Description: "Authentication operations for obtaining tokens to publish servers",
},
{
Name: "admin",
Description: "Administrative operations for managing servers (requires elevated permissions)",
},
{
Name: "health",
Description: "Health check endpoint for monitoring service availability",
},
{
Name: "ping",
Description: "Simple ping endpoint for testing connectivity",
},
{
Name: "version",
Description: "Version information endpoint for retrieving build and version details",
},
}
// Add metrics middleware with options
api.UseMiddleware(MetricTelemetryMiddleware(metrics,
WithSkipPaths("/health", "/metrics", "/ping", "/docs"),
))
// Register routes for all API versions
RegisterV0Routes(api, cfg, registry, metrics, versionInfo)
RegisterV0_1Routes(api, cfg, registry, metrics, versionInfo)
// Add /metrics for Prometheus metrics using promhttp
mux.Handle("/metrics", metrics.PrometheusHandler())
// Add UI and 404 handler for all other routes
mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/" {
// Serve UI at root. The page renders publisher-controlled content
// (server names, descriptions, repository URLs) — server-side
// validation plus a JS escape function are the primary XSS
// defences; these headers are defence-in-depth in case any of
// those slip.
w.Header().Set("Content-Type", "text/html; charset=utf-8")
w.Header().Set("X-Content-Type-Options", "nosniff")
w.Header().Set("X-Frame-Options", "DENY")
// connect-src is unrestricted because the UI exposes a base-URL
// selector (prod / staging / custom) that issues cross-origin
// XHRs to whichever target the operator picks. Constraining
// connect-src would silently break that affordance. The other
// directives still meaningfully limit the page's attack surface.
w.Header().Set("Content-Security-Policy",
"default-src 'self'; "+
"script-src 'self' 'unsafe-inline' https://cdn.tailwindcss.com; "+
"style-src 'self' 'unsafe-inline' https://cdn.tailwindcss.com; "+
"img-src 'self' data:; "+
"connect-src *; "+
"frame-ancestors 'none'; "+
"base-uri 'self'; "+
"form-action 'self'")
_, err := w.Write([]byte(v0.GetUIHTML()))
if err != nil {
http.Error(w, "Failed to write response", http.StatusInternalServerError)
}
return
}
// Handle 404 for all non-matched routes
handle404(w, r)
})
return api
}