The problem
Elasticsearch cannot tell a client what changed. Issue #1242, "Changes API", has been open since August 2011 and has 330 reactions and 173 comments. People want it for cache invalidation, syncing other databases and disaster-recovery copies.
What exists today does not cover the general case. Cross-cluster replication reads the shard history, but only into another Elasticsearch cluster and only with a Platinum or Enterprise license. Polling on a timestamp field needs every writer to set the field, can miss documents whose timestamp lags the poll window, and never sees deletes. The plugins linked in the issue were last changed in 2016 and 2021 and cannot be installed on hosted clusters.
How it works
Since version 6.0, every write to a shard gets a sequence number, and each document stores the number of the operation that last wrote it. es-changefeed keeps, per primary shard, the highest sequence number it has delivered. Each round it reads the shard's global checkpoint from the index stats, refreshes the index if something changed, and searches that shard for documents with a sequence number between its checkpoint and the global checkpoint, sorted by sequence number.
The global checkpoint is what makes this correct. Indexing threads finish out of order, so a document with sequence number 7 can become searchable before one with 6. A program that polls for "greater than the highest number seen" would skip 6. Everything up to the global checkpoint has been processed on all in-sync copies, so after a refresh none of it is missing. Operations above the global checkpoint can also be rolled back after a primary fails, and the feed never reads them.
Deletes leave nothing that search can find. With -deletes, the feed records which documents it delivered, lists the ids that still exist every few minutes, and emits a delete for each delivered document that is gone. The scan runs between the shard reads of an index. A re-creation that is not yet visible therefore has a higher sequence number than anything delivered, and its upsert arrives after the delete.
Output goes to stdout, a file, an HTTP endpoint, or _bulk into another cluster. Patterns, aliases and data streams are followed across rollovers. Output from the CLI test against Elasticsearch 9.5.5, after one update and one delete:
{"op":"upsert","index":"escf-cli","index_uuid":"BZfHMdwmRv-iskXKFEMGgA","shard":1,"id":"1","seq_no":2,"primary_term":1,"version":2,"source":{"title":"edited"}}
{"op":"delete","index":"escf-cli","index_uuid":"BZfHMdwmRv-iskXKFEMGgA","shard":1,"id":"2"}How it is tested
Unit tests run against an in-memory fake cluster that models what the algorithm depends on: sequence numbers assigned before an operation finishes, local and global checkpoints, and visibility that only changes on refresh. A randomized test mixes out-of-order operations, checkpoint lag, routing, deletes and restarts, and checks that replaying the events reproduces the source.
CI then runs the same flows against real clusters: Elasticsearch 9.5.5, 8.19.23 and 7.17.28, OpenSearch 3.9.0 and 2.19.7, and Elasticsearch with TLS and a password. Four writers index, update and delete for eight seconds while the feed mirrors into a second index, and afterwards both must hold the same documents. One job does this on a three-node cluster with replicas and kills a node in the middle.
All jobs pass. The mirrors held between 1573 and 2029 documents and matched the source exactly in every job, including the one with the killed node. The first runs against real clusters found two errors that the fake had not shown. Documents were tracked by _id alone, but with custom routing the same _id can exist on two shards as two documents, so they are now tracked per shard. And when the test stopped the feed while a sink request was in flight, the request still reached the target. The feed treated it as failed, so when the document was deleted before the retry, no delete was ever sent. The ids are now recorded before the write. Reading the CI output turned up a third, smaller error: the first document in a shard has sequence number 0, and the JSON encoder dropped the field.
Limits
Each event is the current state of a document, so if a document changes twice between two rounds, only the second version is emitted. Delete detection keeps every delivered id in memory (about 110 bytes per id in my measurement) and lists all ids of an index per scan. I have not run it against Elastic Cloud or Amazon OpenSearch Service, on indices with more than a few thousand documents, or with a failover that actually rolls back operations.
The code is at github.com/Jwrede/es-changefeed.