I set up an automated benchmark that sends a request to 15 LLM models every hour and publishes all raw data. In the first snapshot, the median time to first token (TTFT) ranged from 321ms (Gemini 2.5 Flash Lite) to 4,226ms (Grok 4.3), a factor of about 13.
Setup
Every hour, the benchmark sends the prompt "Hi" with a limit of 20 output tokens to each model through OpenRouter. The short prompt and output limit keep generation time small, so the numbers mainly describe how long it takes until a response starts. The probes run with llmprobe.
The models are the three most used models per provider in OpenRouter's weekly popularity rankings:
- OpenAI: GPT-5.4, GPT-5.5, GPT-OSS-120B
- Anthropic: Claude Sonnet 4.6, Claude Opus 4.6, Claude Opus 4.7
- Google: Gemini 2.5 Flash, Gemini 2.5 Flash Lite, Gemini 2.0 Flash
- DeepSeek: DeepSeek v3.2, DeepSeek v4 Flash, DeepSeek v4 Pro
- xAI: Grok 4 Fast, Grok 4.1 Fast, Grok 4.3
Scope
When I wrote this post, the benchmark had been running for a little over 30 hours. The table is based on at most 17 hourly probes per model. For GPT-OSS-120B and DeepSeek v4 Pro there were only 2 and 3 samples, so their values are not meaningful.
All requests went through OpenRouter from one deployment location. The numbers include OpenRouter's proxy overhead and depend on how OpenRouter routes requests at a given time. They describe this setup in this period and are not a general ranking of the providers.
Results
| Model | Median TTFT | Median Throughput | Median Latency | Samples |
|---|---|---|---|---|
| google/gemini-2.5-flash-lite | 321ms | 191.9 tok/s | 395ms | 17 |
| google/gemini-2.0-flash-001 | 405ms | 203.2 tok/s | 468ms | 17 |
| google/gemini-2.5-flash | 412ms | 235.6 tok/s | 464ms | 17 |
| openai/gpt-5.4 | 912ms | 44.8 tok/s | 1,147ms | 17 |
| openai/gpt-5.5 | 1,158ms | 44.0 tok/s | 1,501ms | 17 |
| openai/gpt-oss-120b | 1,491ms | 1,977.0 tok/s | 1,584ms | 2 |
| anthropic/claude-opus-4.6 | 1,709ms | 70.4 tok/s | 1,939ms | 17 |
| deepseek/deepseek-v3.2 | 1,734ms | 24.7 tok/s | 2,372ms | 17 |
| anthropic/claude-sonnet-4.6 | 2,120ms | 44.3 tok/s | 2,842ms | 17 |
| anthropic/claude-opus-4.7 | 2,494ms | 94.9 tok/s | 2,599ms | 17 |
| x-ai/grok-4.1-fast | 2,545ms | 2,985.4 tok/s | 2,593ms | 17 |
| deepseek/deepseek-v4-flash | 3,122ms | 251.6 tok/s | 3,560ms | 16 |
| deepseek/deepseek-v4-pro | 3,411ms | 108.1 tok/s | 3,816ms | 3 |
| x-ai/grok-4-fast | 3,618ms | 1,338.7 tok/s | 3,682ms | 17 |
| x-ai/grok-4.3 | 4,226ms | 1,114.8 tok/s | 4,328ms | 17 |
TTFT by provider
All three Gemini models had a median TTFT below 500ms. Gemini 2.5 Flash Lite had the lowest value of all 15 models (321ms).
The three Anthropic models had medians between 1,709ms (Claude Opus 4.6) and 2,494ms (Claude Opus 4.7). Claude Sonnet 4.6 was at 2,120ms.
The five highest medians belonged to the xAI models and to DeepSeek v4 Flash and v4 Pro. Grok 4.3 had the highest median, 4,226ms. DeepSeek v3.2 was lower at 1,734ms.
Throughput
The Grok models had a high median TTFT and the highest measured throughput, between 1,114.8 and 2,985.4 tok/s. DeepSeek v3.2 had the lowest throughput, 24.7 tok/s.
Throughput is computed from the tokens received after the first token and the time between the first token and the end of the stream. With an output limit of 20 tokens this interval is short. Values above 1,000 tok/s may therefore reflect tokens that arrive in large chunks rather than generation speed. This data does not distinguish between the two.
Total latency and timeouts
Median total latency ranged from 395ms to 4,328ms across the 15 models. A fixed timeout that suits one model can therefore be exceeded regularly after a switch to another model, including another model from the same provider.
Dashboard and raw data
The benchmark runs continuously. The dashboard with charts is at bench.jonathanwrede.de.
All raw data is published as JSONL at github.com/Jwrede/llm-bench-data. The model list is updated daily from OpenRouter's popularity rankings.
The probes use llmprobe, an open-source Go CLI that measures TTFT, latency and throughput with plain HTTP and its own SSE parser. llmprobe can also run as an MCP server, so Claude Code and other MCP hosts can call its probes.