The problem
Grafana Mimir stores metrics as Prometheus TSDB blocks in object storage. It has no API to delete series from those blocks. Once a label value with an email address, a token or a customer name has been written, it stays until retention removes the whole block. Issue #4968, "Support deleting or rewriting series in long-term storage", has been open since May 2023 with 77 reactions. The use cases in the thread are sensitive label values, recording rules that recorded wrong values, and wrongly backfilled data. An older issue about a deletion API was closed in favour of #4968, and a maintainer wrote in another one that "Mimir currently doesn't support delete series endpoint".
In #4968 a maintainer explains that when they needed this, they used the bucket rewrite tool from Thanos, which only works with direct access to the blocks. Mimir ships tools to copy, list, mark and undelete blocks, but none that rewrites series. I did not find a third-party tool for it.
What the tool does
mimir-rewrite is a Go command that works directly on the bucket (a directory or S3). plan downloads only the index of each block of a tenant and counts what a change would touch. apply rewrites the affected blocks. Three kinds of change can be combined: deleting series that match a selector, for all time or inside a time range; Prometheus relabel_configs; and replacing label values by a salted SHA-256 prefix, which keeps the series but removes the value.
Deletions use the Prometheus TSDB library directly: they are written as tombstones, and Prometheus' compactor writes a new block that copies the chunks and drops what the tombstones cover. Relabeling re-encodes every sample under its new labels. Before anything is uploaded, the tool computes from the original block how many samples each output series must have and compares that with the new block.
The Mimir-specific part is the swap. The old block first gets a no-compact mark, so the compactor does not start a job on it during the rewrite. The new block is uploaded with its own no-compact mark and meta.json last, then the old block is marked for deletion in both places Mimir uses (block directory and markers/), then both no-compact marks are removed. Mimir's compactor ignores blocks with a deletion mark immediately, so it never sees old and new block together and cannot merge deleted data back. The new meta.json keeps Mimir's external labels; if labels changed, __compactor_shard_id__ is dropped because the series may no longer hash to that shard.
Example
From the end-to-end test against Mimir 3.2.4, after uploading a block with three users through Mimir's block upload API:
$ mimir-rewrite apply -bucket /data/blocks -tenant e2e-upload -delete '{user="bob@example.com"}' -yes
BLOCK FROM TO SERIES DELETE REMOVED RELABELED DROPPED ACTION
01M6ZEBBJ620SVTW3HXYQPZ017 2026-11-06T06:00Z 2026-11-06T07:59Z 4 2 2 0 0 rewrite
block 01M6ZEBBJ620SVTW3HXYQPZ017: samples 1560 -> 960, series 4 -> 2, verified
block 01M6ZEBBJ620SVTW3HXYQPZ017: uploaded replacement 01M6ZEBFP1MKPHZNQ2TBZPY0TR
block 01M6ZEBBJ620SVTW3HXYQPZ017: marked for deletionHow it is tested
Unit tests build blocks with float, native histogram and float histogram series and cover deletions, relabeling, hashing, collisions and the order of bucket operations. The end-to-end test starts Mimir in monolithic mode. One case uploads a block through the upload API, deletes a user, hashes the remaining user values and waits until the compactor has deleted the replaced blocks. The other writes data with remote write, lets the ingester flush it, and deletes 30 minutes of one series. Every step is checked through Mimir's query API, and the test fails if any query fails during the swaps. CI runs this against Mimir 3.2.4 with a filesystem bucket and with S3 (SeaweedFS), and against Mimir 2.17.0.
The test found one thing worth knowing. With delays where the store-gateway dropped a marked block before the queriers reloaded the bucket index, queries failed with a consistency-check error for about 35 seconds. Mimir's defaults keep the order the other way round, and the test now uses short delays in that order.
Limits
The tool needs write access to the bucket, so it is for operators, not a per-tenant API like the one the issue asks for. It supports directories and S3, not GCS or Azure. Data still held by ingesters is not touched. Relabeling keeps one block's series in memory. Changes become visible only after Mimir's deletion delays, up to about an hour with defaults. Not tested: split-and-merge compaction with several shards, out-of-order blocks, and a compactor job that is already running when the tool starts.
The code is at github.com/Jwrede/mimir-rewrite.