Latency of six LLM APIs over seven days, measured with llmprobe

llmprobe is an open-source Go CLI I wrote. It sends streaming requests to LLM endpoints and measures time to first token (TTFT), total latency and generation throughput. It uses plain HTTP and its own SSE parser instead of provider SDKs.

I ran it for seven days against six models through OpenRouter.

Setup

Every 60 seconds, llmprobe sent the prompt "Hello" with a limit of 20 output tokens to each model. The short prompt and output limit keep generation time small, so the numbers mainly describe how long it takes until a response starts.

Models: GPT-4o-mini, Claude 3.5 Haiku, Gemini 2.0 Flash, Llama 3.3 70B, DeepSeek Chat, Mistral Small.

Total probes: 60,480 (about 10,080 per model).

All requests went through OpenRouter. The numbers therefore include OpenRouter's proxy overhead and describe this route during this week. They are not measurements of the providers' native APIs.

Results

Median and p95 TTFT

Gemini 2.0 Flash had the lowest median TTFT (556ms). Its p95 was 2,313ms, about 4 times the median. For GPT-4o-mini and Claude 3.5 Haiku the p95 was less than 1.7 times the median.

Mistral Small had a median TTFT of 2,735ms and a p95 of 10,852ms, so about 5% of its requests took more than 10 seconds until the first token arrived. Its hourly median stayed above that of the other models for nearly the whole week (see the timeline below).

TTFT and generation throughput

Mistral Small had the highest TTFT and also the highest measured throughput, 191.6 tok/s. Gemini 2.0 Flash followed with 136.8 tok/s. DeepSeek Chat had a high TTFT (1,068ms) and the lowest throughput (26.0 tok/s).

Throughput is computed from the tokens received after the first token and the time between the first token and the end of the stream. With an output limit of 20 tokens this interval is short, so the throughput values are a rough measure.

TTFT distribution

GPT-4o-mini and Claude 3.5 Haiku had the narrowest TTFT distributions. For both, the p95 was below 2 seconds, so a 2-second TTFT timeout would have been exceeded by fewer than 5% of their requests. Gemini 2.0 Flash and Llama 3.3 70B had longer tails. Mistral Small had the widest distribution.

TTFT over the seven days

The timeline shows the hourly median TTFT per model.

• Mistral Small fluctuated strongly throughout the week, with peaks close to 5,000ms.
• DeepSeek Chat had several windows of higher latency. The data does not show the cause.
• GPT-4o-mini and Claude 3.5 Haiku varied little over the week.

Using llmprobe as a CI check

llmprobe can run in a deploy pipeline and fail the job when an endpoint does not meet its configured thresholds.

# .github/workflows/deploy.yml
- name: Check LLM providers
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
  run: |
    go install github.com/Jwrede/llmprobe@latest
    llmprobe probe --fail-on degraded

With --fail-on degraded, llmprobe exits with code 1 if a request fails or an endpoint exceeds one of its thresholds, and the workflow step fails.

Thresholds are set per model in probes.yml:

providers:
  - name: openai
    api_key: ${OPENAI_API_KEY}
    models:
      - name: gpt-4o-mini
        thresholds:
          max_ttft: 2s
          max_latency: 5s
          min_tokens_per_sec: 50

The following example TTFT thresholds lie above the p95 values from this week. They are based on this route and period and should be checked against one's own measurements.

Supported providers

llmprobe supports OpenAI, Anthropic, Google, Azure, AWS Bedrock and any OpenAI-compatible endpoint (Groq, Together, Fireworks, DeepSeek, Mistral, OpenRouter, Ollama, vLLM). Each provider client is a small HTTP wrapper with SSE parsing. The Bedrock client implements SigV4 signing and parsing of the AWS binary event stream without an SDK.

go install github.com/Jwrede/llmprobe@latest
llmprobe watch --tui --interval 60s

GitHub: github.com/Jwrede/llmprobe

Follow-up: hourly benchmark

After this post I set up a continuous benchmark based on llmprobe. It probes 15 models from OpenAI, Anthropic, Google, DeepSeek and xAI every hour. The model list is updated daily from OpenRouter's weekly popularity rankings.

The dashboard is at bench.jonathanwrede.de and the raw data is published as JSONL at github.com/Jwrede/llm-bench-data.

Infrastructure code: github.com/Jwrede/llm-bench