Jonathan Wrede

What I worked on in llm-d

My merged pull requests to the llm-d project: mostly CI hardening, tests and observability, plus one flow-control configuration option and a few small bug fixes.

llm-d is an open-source project for running LLM inference on Kubernetes, maintained by engineers from Red Hat, IBM and Google. Its router sits between clients and model servers such as vLLM and decides which backend handles each request, based on signals such as load and cached prefixes. A flow-control layer in the router queues requests by priority.

Between May and July 2026 I had pull requests merged in several llm-d repositories. I do not operate LLM serving myself. Most of these pull requests are CI, test, observability and hardening work; the rest are one configuration feature, a few bug fixes and documentation. This post lists what each of them changed.

CI and supply-chain hardening

The router project has an RFC with a checklist of CI/CD improvements. Several of my pull requests implement items from it.

  • llm-d-router #960 pins a mutable GitHub Action reference to a commit SHA, adds least-privilege permissions blocks to nine workflows, sets a 30-minute timeout on the lint-and-test job (a hung kind cluster could otherwise hold a runner for up to six hours), adds concurrency groups so that new pushes cancel stale runs, replaces a five-minute automerge cron with event triggers and a slower fallback cron, uploads Trivy results to the GitHub Security tab as SARIF, and adds govulncheck for known vulnerabilities in Go dependencies.
  • llm-d-router #1424 makes the job fail when test coverage drops, adds a dependency review workflow for pull requests, and adds a workflow that builds the CI builder image on changes and pushes it on merge.
  • llm-d-router #1579 pins :latest image tags in the example GPU manifests to fixed vLLM and SGLang versions and makes the presubmit check for :latest tags blocking.
  • llm-d-router #1572 passes a Hugging Face token to the end-to-end tests, which were failing intermittently on Hub rate limits (HTTP 429).
  • llm-d-kv-cache #623 applies the same first checklist to the KV cache repository: SARIF upload, a pinned action, a Go module integrity check, concurrency groups, job timeouts, and a missing ZeroMQ package in the examples workflow.

Flow-control configuration

llm-d-router #1099 is the only feature-sized pull request. The flow-control layer assigns each request a numeric priority, and negative priority marks traffic that may be shed. When a request arrives at a priority level without a configured band, the router creates a band from a single default template. Positive and negative priorities therefore received the same queue budget, and it was not possible to configure, for example, 1 GB of queue budget for normal traffic and none for sheddable traffic.

The pull request adds a second template, DefaultNegativePriorityBand, which is used only when a band is created for a priority below zero. If it is not set, the router uses the existing default, so existing configurations behave as before. The change itself is a few lines in the provisioning logic. Most of the diff is the user-facing API config, the internal config with validation and deep copy, and tests for construction, API translation, clone isolation and dynamic provisioning for positive, negative and fallback cases.

Metrics

llm-d-router #1429 exports three Prometheus metrics for the multimodal encoder cache data producer: a hit ratio histogram, a query counter and a hit counter. They follow the metrics that the prefix cache plugin already exported.

Bug fixes

  • llm-d-router #1426: requests without a model field were rejected before generic model rewrite rules could fill in a model name. The rewrite now runs first.
  • llm-d-router #1580: the concurrency detector reported full saturation when no endpoint had metadata yet, for example right after pods started or for a passive pod in an HA setup. It now counts the capacity of those endpoints and reports zero load for them.
  • llm-d-router #1695: with no InferencePool created and modelServers.matchLabels missing, the standalone Helm chart rendered an empty endpoint selector, and the router failed at startup. The chart now stops at render time with an error message.
  • llm-d-inference-sim #530: all HTTP endpoints of the inference simulator now return errors in the OpenAI JSON error format, which OpenAI SDK clients can parse.
  • llm-d-kv-cache #589: removes a 16 MB minimum staging buffer size that padded every KV cache file written to disk.

Smaller changes

  • llm-d-inference-payload-processor #151 adds GetModels(predicate) to the datastore interface, so callers get all matching models in one call under one read lock.
  • Documentation: logging conventions in the router's AGENTS.md (#1571), a typo in llm-d-inference-sim (#529), and outdated kgateway references in llm-d-benchmark (#1437).