Jonathan Wrede

compose-inline: Compose configs and secrets on a remote Docker host

A wrapper around docker compose that passes local config and secret files by value, so that projects using them start on a remote Docker host.

The problem

Docker Compose implements configs.*.file and secrets.*.file as bind mounts, and a bind mount is resolved on the Docker host. When Compose talks to a remote host (DOCKER_HOST=ssh://..., a docker context or tcp://...), the file has to exist at the same absolute path on that machine. Otherwise the container does not start:

Error response from daemon: invalid mount config for type "bind":
bind source path does not exist: /home/me/project/secrets/db_password.txt

The issue is documented:

  • docker/compose#11867, "Support remote DOCKER_HOST by copying configs and secrets files instead of bind mounting them", has been open since May 2024 with 45 reactions and 50 comments. The maintainer agrees with the direction but wants an opt-in syntax, because copying changes existing behaviour. Five pull requests that implemented it were closed without merge. One commenter asked for a way to avoid having to "manually pre-process the configuration files in order to copy them into the content property".
  • Earlier reports of the same failure are #7857, #11141 and #7110.
  • Bind-mounted configs and secrets ignore uid, gid and mode (#9648, 30 reactions).

The workarounds in the thread are copying the files to the Docker host and using absolute paths there, keeping a separate override file per host, or baking the files into an image.

How it works

Compose already has a code path that works remotely: configs with content: and configs or secrets with environment: are written into the container through the Docker API before it starts. compose-inline, a Python tool without dependencies, moves files onto that path. It reads the resolved model with docker compose config --format json. Every config whose file exists on the local machine becomes content:, with $ escaped because Compose interpolates the model again. Secrets, and configs that are not valid UTF-8, become environment: entries whose values exist only in the environment of the child process, so secret values are never written to disk. With --bind-files, single-file bind mounts inside the project directory are converted to configs with the file's permission bits as mode.

Compose does not hash config contents into the service, so an edited file would not recreate the container on the next up. compose-inline therefore adds a label with a hash of the inlined contents to each affected service. It writes the rewritten model to a private temporary file and runs the requested docker compose command with it.

This is the README example, a project with one config, one secret and a bind-mounted nginx.conf, run against a host that does not have the project files (paths shortened in the README):

$ export DOCKER_HOST=ssh://deploy@server
$ docker compose up -d
 Container app-web-1 Error response from daemon: invalid mount config for type "bind": bind source path does not exist: /home/me/app/secrets/db_password.txt

$ compose-inline --bind-files --verbose up -d --wait
compose-inline: inlined config site from /home/me/app/site.conf (87 bytes)
compose-inline: inlined secret db_password from /home/me/app/secrets/db_password.txt (22 bytes)
compose-inline: inlined bind mount /home/me/app/nginx.conf:/etc/nginx/nginx.conf of service web (57 bytes)
 Container app-web-1 Created
 Container app-web-1 Started
 Container app-web-1 Healthy

Testing

18 unit tests cover the rewrite and the argument handling: escaped dollar signs, secrets and non-UTF-8 configs through the environment, rejected NUL bytes and oversized secrets, missing files and directories that stay bind mounts, --bind-files, and a hash label that changes with the content.

The end-to-end test runs against a docker:27-dind container reached over TCP, which cannot see the files of the checkout. It checks that plain docker compose fails with "bind source path does not exist", that compose-inline delivers the config, the bind-mounted file and the secret with modes 600 and 640, that up recreates the container only after a file changed, and that a config path that exists only on the Docker host stays a bind mount. CI runs the unit tests with Python 3.9, 3.12 and 3.13 and the end-to-end test with Compose v2.38.2 and v2.24.7. The older version parses a mode string as a decimal number, so the tool writes modes as integers.

Limitations

  • Files are copied once, when the container is created. A local edit reaches the container only at the next up.
  • Compose refuses to copy into services with read_only: true.
  • Copied files are owned by root unless the config or secret sets uid and gid. Configs default to mode 0444.
  • Directories and paths that do not exist locally stay bind mounts. --bind-files converts only files inside the project directory.
  • Files with NUL bytes are rejected. Secrets go through the environment, which Linux limits to 128 KiB per value.
  • The com.docker.compose.project.config_files label of the containers points at the deleted temporary file.
  • It is tested with Compose v5.0.0, v2.38.2 and v2.24.7 against Docker 27.5.1. It is not tested with Podman.

The code and installation instructions are at github.com/Jwrede/compose-inline.