OpenRouter Function Calling Test

This script tests OpenRouter models for native OpenAI function calling support and provides utilities to list models and their endpoints.

Setup

1. Install uv (if not already installed):

curl -LsSf https://astral.sh/uv/install.sh | sh

2. Install dependencies (automatically creates venv with Python 3.12):

uv sync --python 3.12

3. Set your OpenRouter API key:

export OPENROUTER_API_KEY="your_api_key_here"

Usage

List all available models

uv run python openrouter_test.py --list-models

List all endpoints for all models

uv run python openrouter_test.py --list-endpoints

Test function calling for a specific model

uv run python openrouter_test.py --test-function-calling --model "anthropic/claude-3-sonnet"

Test function calling with specific provider

uv run python openrouter_test.py --test-function-calling --model "anthropic/claude-3-sonnet" --provider "anthropic"

Test structured output for a specific model

uv run python openrouter_test.py --test-structured-output --model "openai/gpt-4o"

Test structured output with specific provider

uv run python openrouter_test.py --test-structured-output --model "openai/gpt-4o" --provider "openai"

List endpoints with required parameters

uv run python openrouter_test.py --list-endpoints --required-parameters "tools,response_format,max_tokens"

Test function calling for all models

uv run python openrouter_test.py --test-all-models

Test with limits and save results

uv run python openrouter_test.py --test-all-models --limit 10 --save results.json

Test with custom concurrency and timeout

uv run python openrouter_test.py --test-all-models --concurrency 16 --timeout 30 --save results.json

What it tests

The script tests two key OpenAI API features:

Function Calling

1. Defines a get_weather function with JSON schema
2. Sends a weather-related query to the model
3. Checks if the model responds with proper function calls
4. Records success/failure and any errors

Structured Outputs

1. Tests response_format with type: "json_schema" and strict: true
2. Defines a weather response schema with required fields
3. Verifies the model returns valid JSON matching the schema

Provider Routing

• Optionally enforces specific providers using the provider parameter
• Useful for testing provider-specific implementations

Performance Features

Parallel Testing

• Concurrent model testing: Test up to 8 models simultaneously (configurable via --concurrency)
• Sequential endpoint testing: Within each model, endpoints are tested sequentially for clear progress tracking
• Automatic timeout handling: Configurable request timeouts prevent hanging on unresponsive endpoints
• Incremental saving: Results are saved after each completed model to prevent data loss

Smart Parameter Filtering

• Capability-aware testing: Only tests function calling when tools parameter is supported
• Endpoint validation: Only tests structured output when response_format parameter is supported
• Efficient skipping: Avoids unnecessary API calls to unsupported endpoints

Output

• ✅ Model supports function calling and used it correctly
• ⚠️ Model responded but didn't use function calling
• ❌ Model failed with an error

Results can be saved to JSON files for further analysis.