The problem
yq parses YAML into a node tree and writes
the tree back out. The parser does not keep blank lines or the original
indentation, so yq -i reformats the whole file. For hand-maintained files
such as Kubernetes manifests or Helm values, a one-line change produces a
large diff.
The requests have been open for years:
- mikefarah/yq#515, "yq write strips completely blank lines from the output", has 156 reactions and 41 comments and has been open since August 2020.
- mikefarah/yq#465, "Preserve formatting with in place writing", has 116 reactions.
- mikefarah/yq#1359 has 37 reactions and mikefarah/yq#462 has 26.
The maintainer traced the cause to the YAML library
(go-yaml/yaml#627), where a fix
is in progress but not released. The most-upvoted workaround in #515 (61
reactions) pipes the output of yq through diff | patch. Later comments in
the thread report that it drops comments or merges list items in some
cases. Format-preserving
editors such as yamlpath exist, but
they use their own query syntax, so existing yq expressions have to be
rewritten.
How it works
yq-keep keeps yq as the engine and
accepts any expression that works with yq -i. For each file it runs
yq '.' FILE to get yq's rendering of the unchanged file and
yq EXPRESSION FILE to get the rendering after the edit. Both outputs share
yq's formatting, so the line diff between them contains only the edit. The
lines of the first rendering are aligned with the original file, ignoring
blank lines and whitespace differences, and each hunk of the diff is applied
at the aligned position in the original. Inserted lines are re-indented to
match their nearest sibling in the original.
The result is parsed again with yq and compared with the output of
yq EXPRESSION FILE as JSON data. If the two differ, the file is left
unchanged and yq-keep exits with code 2. --diff prints the change
without writing it.
This is the output for a small kustomization file with a blank line between sections, a four-space indented sequence and three spaces before a comment:
$ yq-keep --diff '.resources += ["rg.yaml"] | .images[0].newTag = "1.3.0"' kustomization.yaml
--- a/kustomization.yaml
+++ b/kustomization.yaml
@@ -4,7 +4,8 @@
resources:
- deployment.yaml # main
- service.yaml
+- rg.yaml
images:
- name: app
- newTag: "1.2.3"
+ newTag: "1.3.0"The same edit with yq -i changes 11 lines: it also removes both blank
lines, re-indents the sequences under resources and images, and collapses
the spaces before # main.
Testing
The unit tests run the real yq binary. They cover the blank-line case from
#515, appending to a list, comments and spacing on untouched lines, the
indentation style of inserted lines, deleting a block, multi-document files,
CRLF line endings and a missing final newline. A parametrized test applies
six expressions, among them sort_keys(..), a key rename with
with_entries, and edits on anchors and merge keys, to a file with block
scalars and flow sequences, and checks that the result holds the same data
as yq's own output. CI runs the suite
on Python 3.10 to 3.13 with yq v4.54.1.
yq-keep was also run against 361 YAML files (Kubernetes manifests, CI
workflows, Compose and pre-commit files) with five edits each. 1,621 of 1,636 edits were written. Of the 15 refusals,
11 were files where yq's own round trip already changes the data. For the
1,088 edits other than sort_keys that changed a file, yq-keep changed 2,819
lines in total and yq -i changed 26,634.
Limitations
- A changed line is taken from yq's output and gets yq's spacing, for example one space before its trailing comment.
- Edits that reorder or rewrite most of a file (
sort_keys(..), renaming every key) produce correct data, but the blank lines stay at their old positions. - Where yq loses a comment in a changed region, yq-keep loses it too.
- Only YAML input is supported. yq runs four times per file, so yq-keep is
slower than
yq -ion very large files.
The code and installation instructions are at github.com/Jwrede/yq-keep.