example-remote-client/docs/inference_provider_interface.md

InferenceProvider Interface Design

This document outlines the design for the InferenceProvider interface and its implementations, which abstracts LLM inference capabilities for the MCP client.

Goals

1. Provider Agnostic: Support multiple inference providers (OpenRouter, Anthropic, OpenAI, Google, etc.)
2. Tool Calling: Native support for function/tool calling with MCP tools
3. Authentication: Flexible auth patterns (API keys, OAuth, etc.)
4. Model Selection: Dynamic model listing and selection
5. Error Handling: Consistent error handling across providers

Core Interface

interface InferenceProvider {
  // Provider identification
  readonly name: string;
  readonly id: string;

  // Authentication state
  readonly isAuthenticated: boolean;
  readonly authError?: string;

  // Available models
  readonly models: Model[];
  readonly selectedModel?: Model;

  // Core inference method
  generateResponse(request: InferenceRequest): Promise<InferenceResponse>;

  // Model management
  loadModels(): Promise<Model[]>;
  selectModel(modelId: string): void;

  // Authentication
  authenticate(config: AuthConfig): Promise<void>;
  logout(): void;

  // Provider-specific capabilities
  getCapabilities(): ProviderCapabilities;
}

Supporting Types

Model

interface Model {
  id: string;
  name: string;
  description?: string;
  contextLength: number;
  inputCost?: number;  // per token
  outputCost?: number; // per token
  provider: string;
  capabilities: ModelCapabilities;
}

interface ModelCapabilities {
  supportsTools: boolean;
  supportsVision: boolean;
  maxTokens: number;
}

Inference Request/Response

interface InferenceRequest {
  messages: ChatMessage[];
  tools?: Tool[]; // toolChoice defaults to 'auto' when tools provided
  maxTokens?: number;
  temperature?: number;
  stopSequences?: string[];
}

interface InferenceResponse {
  message: ChatMessage;
  usage: TokenUsage;
  stopReason: 'stop' | 'max_tokens' | 'tool_calls' | 'error';
  error?: string;
}

interface ChatMessage {
  role: 'user' | 'assistant' | 'system' | 'tool';
  content: string | ContentBlock[];
  toolCalls?: ToolCall[];
  toolCallId?: string; // for tool response messages
}

interface ToolCall {
  id: string;
  type: 'function';
  function: {
    name: string;
    arguments: string; // JSON string
  };
}

Authentication

interface AuthConfig {
  type: 'api_key' | 'oauth';
  apiKey?: string;
  oauthConfig?: {
    clientId?: string;
    redirectUri?: string;
    scopes?: string[];
  };
}

interface ProviderCapabilities {
  authMethods: ('api_key' | 'oauth')[];
  supportsModelListing: boolean;
  supportsToolCalling: boolean;
  requiresAuth: boolean;
}

Provider Implementations

OpenRouter Providers

For simplicity, we implement two separate providers using composition:

OpenRouterApiProvider:

• API key authentication only
• Simpler auth flow and error handling
• Direct API access

OpenRouterOAuthProvider:

• OAuth PKCE authentication only
• Browser-based auth popup flow
• Token refresh handling

Shared Configuration:

interface OpenRouterBaseConfig {
  baseUrl?: string; // defaults to https://openrouter.ai/api/v1
  defaultModel?: string;
  httpReferrer?: string;
  appName?: string;
}

interface OpenRouterApiConfig extends OpenRouterBaseConfig {
  apiKey: string;
}

interface OpenRouterOAuthConfig extends OpenRouterBaseConfig {
  clientId?: string; // for custom OAuth apps
  redirectUri?: string;
}

Shared Logic: Both providers use composition with shared utilities:

• OpenRouterClient - HTTP client and request formatting
• OpenRouterModelParser - Model list parsing and capabilities
• OpenRouterToolFormatter - Tool call formatting

Future Providers

AnthropicProvider:

• OAuth integration with Anthropic Console
• Claude model family support
• Message batching optimization

OpenAIProvider:

• OpenAI API key or OAuth
• GPT model family support
• Function calling optimization

GoogleProvider:

• Google Cloud OAuth
• Gemini model family support
• Tool calling with Google AI format

React Integration

Provider Context

interface InferenceContextValue {
  provider: InferenceProvider | null;
  isLoading: boolean;
  error: string | null;
  
  // Actions
  setProvider: (provider: InferenceProvider) => void;
  clearProvider: () => void;
  generateResponse: (request: InferenceRequest) => Promise<InferenceResponse>;
  selectModel: (modelId: string) => void;
  loadModels: () => Promise<Model[]>;

  // Convenience getters
  models: Model[];
  selectedModel: Model | undefined;
  isAuthenticated: boolean;
}

const InferenceContext = createContext<InferenceContextValue | null>(null);

Hook Usage

const useInference = () => {
  const context = useContext(InferenceContext);
  if (!context) {
    throw new Error('useInference must be used within InferenceProvider');
  }
  return context;
};

Error Handling

Error Types

interface InferenceError {
  type: 'auth' | 'network' | 'rate_limit' | 'invalid_request' | 'provider_error';
  message: string;
  details?: any;
  retryable: boolean;
}

Error Scenarios

• Authentication failures: Invalid API key, expired tokens
• Rate limiting: Provider-specific rate limits
• Network errors: Connection issues, timeouts
• Validation errors: Invalid tool schemas, malformed requests
• Provider errors: Service unavailable, model not found

Implementation Strategy

Phase 1: Basic OpenRouter Support ✅

1. Implement core InferenceProvider interface
2. Create OpenRouterApiProvider with API key auth
3. Basic model listing and selection (filtered to tool-capable models only)
4. Simple tool calling support (toolChoice=auto)
5. React context with proper state management for UI reactivity

Phase 2: OAuth and Enhanced Features ✅

1. Create OpenRouterOAuthProvider with PKCE auth
2. Implement error recovery and retries
3. Add usage tracking and cost estimation
4. Improve tool call validation

Current Status

• ✅ Two OpenRouter providers (API key & OAuth) implemented
• ✅ Provider-level model filtering (only tool-capable models)
• ✅ React context with proper UI state synchronization
• ✅ Interactive test UI with tool calling demonstration
• ✅ OAuth callback routing for future MCP server integration

Phase 3: Multi-Provider Support

1. Add AnthropicProvider
2. Add OpenAIProvider
3. Provider switching UI
4. Unified configuration management

Testing Strategy

Unit Tests

• Mock provider implementations for testing
• Tool call serialization/deserialization
• Error handling scenarios
• Authentication flows

Integration Tests

• Real provider API integration (with test keys)
• End-to-end tool calling workflows
• Provider switching scenarios
• Authentication persistence

Security Considerations

1. API Key Storage: Secure storage in browser (encrypted localStorage)
2. OAuth Flow: Secure popup-based OAuth with PKCE
3. Token Refresh: Automatic token refresh handling
4. Rate Limiting: Client-side rate limiting to prevent abuse
5. Input Validation: Strict validation of tool calls and responses

Key Design Decisions

1. Provider-Level Model Filtering

All providers filter models to only return those that support tool calling. This simplifies consumers and prevents tool-calling errors:

// In OpenRouterClient.fetchModels()
return response.data
  .filter(model => model.supported_parameters?.includes('tools'))
  .map(this.parseModel);

Benefits:

• No tool capability checking needed in UI components
• Eliminates "model doesn't support tools" errors
• Cleaner separation of concerns

2. React State Management for UI Reactivity

The React context tracks model selection in React state to ensure UI updates when models are selected:

const [selectedModelId, setSelectedModelId] = useState<string | undefined>();

// In selectModel callback
provider.selectModel(modelId);        // Update provider internal state
setSelectedModelId(modelId);          // Update React state → triggers re-render

// In context value - compute selectedModel from React-tracked ID
selectedModel: selectedModelId ? provider?.models.find(m => m.id === selectedModelId) : undefined

Why this pattern:

• Provider internal state changes don't trigger React re-renders automatically
• React needs to track state changes to update the UI
• Keeps provider logic decoupled from React specifics

3. Composition Over Inheritance

Two separate OpenRouter providers (API & OAuth) use shared utilities rather than complex inheritance:

• OpenRouterClient - HTTP client and request formatting
• Shared model parsing and tool formatting logic
• Clean separation of authentication concerns

Benefits:

• Simpler code paths (no branching auth logic)
• Easier testing and maintenance
• Clear separation of API vs OAuth concerns

Future Enhancements

1. Caching: Response caching for repeated requests
2. Load Balancing: Multi-provider load balancing
3. Analytics: Usage analytics and monitoring
4. Custom Models: Support for custom/fine-tuned models
5. Advanced Tool Choice: Support for specific tool selection (beyond auto)
6. Streaming Support: Real-time response streaming (if needed later)