Jonathan Wrede

gitignore-excludes: .gitignore rules for restic and borg

A small tool that asks Git which paths are ignored in every working tree below a directory and writes them as an exclude file for restic or borg.

The problem

restic and borg have no option to read .gitignore files. Whoever backs up a directory of source code either backs up every node_modules, virtualenv and build directory with it or maintains exclude patterns by hand.

The request is old:

  • restic/restic#1514, "Use .gitignore (or other VCS) for excluding files", has 70 reactions and 27 comments and has been open since December 2017. In the thread a restic contributor voted for "an easy way to use a suitable external tool" instead of building it into restic.
  • restic/restic#4227, the pull request implementing it, has 20 reactions and has been open since March 2023 without a review.
  • borgbackup/borg#641 has 24 reactions; borg#937, "exclude files specified in .gitignore", was closed without a feature.

The workarounds posted there pipe rg --files or fd into restic backup --files-from-verbatim. They match ignore files with a different implementation than Git's, and every listed file becomes a snapshot path, so the path list changes whenever a file is added or removed (restic#2246 describes the resulting problem with parent snapshot detection). Another approach writes CACHEDIR.TAG files into ignored directories, which modifies the working trees and cannot exclude single files.

How it works

gitignore-excludes walks the given directories and runs the following command in every directory that contains .git:

git ls-files -z --others --ignored --exclude-standard --directory

The result therefore follows Git's own rules, including nested .gitignore files, negation (!keep.log), .git/info/exclude and core.excludesFile. Tracked files are never excluded, even when a pattern matches them. A directory that is ignored as a whole becomes one line, and the walk does not descend into it. Each path is written as an absolute, escaped pattern that matches only that path. For restic, *, ?, [ and \ are escaped and $ becomes $$, because restic expands environment variables in exclude files. For borg, paths become pp: literal path prefixes, and names that end in whitespace use fm: with escaped characters. --keep GLOB keeps ignored entries whose name matches, for example .env*; an ignored directory that contains a kept file is then excluded child by child.

Example from the README:

$ gitignore-excludes ~/code > /tmp/code-excludes.txt
gitignore-excludes: 11 working trees, 27 ignored paths
$ restic backup ~/code --exclude-file /tmp/code-excludes.txt

$ head -4 /tmp/code-excludes.txt
# generated by gitignore-excludes; regenerate before each backup
/home/me/code/api/.venv
/home/me/code/api/src/api/__pycache__
/home/me/code/web/node_modules

On a folder with 11 Git repositories, the restic backup went from 4253 files (86.6 MiB) to 604 files (1.1 MiB). The tool needs Python 3.10 or later and git, and has no other dependencies.

Testing

The tests build a fixture tree with two repositories, one nested inside the other. It contains nested .gitignore files, a negated pattern, a tracked file that matches an ignore pattern, ignored directories, a directory outside any repository, and file names with *, ?, [, \, $ and leading or trailing spaces. The unit tests check that the scan returns exactly the expected set of paths, that a repository inside an ignored directory is not scanned separately, how --keep splits an ignored directory, and the escaping for both formats.

The end-to-end tests back up the fixture with the real restic and borg binaries using the generated exclude file and compare the snapshot contents with the files Git does not ignore. The restic test also runs with --keep '.env*' and sets HOME to a dummy value to confirm that a $HOME in a file name is not expanded. CI runs the suite on Python 3.10 to 3.13 with restic 0.19.1 and borg 1.4.5.

Limitations

  • Ignored files are often secrets such as .env files and keystores, which may exist nowhere else. The README recommends running with -f plain once, reading the list and adding --keep patterns.
  • The list contains concrete paths and has to be regenerated before every backup.
  • The tool and the backup command must get the same root paths. Symlinks are not followed.
  • Only Git working trees are handled. A repository inside an ignored directory is excluded with it.
  • File names containing a newline cannot be written exactly; the newline becomes ?.
  • Tested on Linux only.

The code and installation instructions are at github.com/Jwrede/gitignore-excludes.