Jonathan Wrede

sandbox-shell: a shell in the sandbox of a systemd service

A small tool that reads a unit file and its drop-ins and starts a shell or command with the same user, environment and sandbox settings, also when the service is not running.

The problem

Modern systemd services often run in a sandbox: DynamicUser=yes gives them a temporary user, ProtectSystem=strict makes the file system read-only, ReadWritePaths= opens a few directories again, and there are capability, system call and namespace settings on top. When such a service fails, the question is usually "what does the world look like from inside?". Which user am I, can I write this directory, does this path exist?

The usual answers are nsenter into the running process, which does not help when the service fails at start, or systemd-run with one -p option per line of the unit file and its drop-ins, copied by hand.

Others have asked for a better way:

  • systemd/systemd#28163, "Get a shell in the environment/sandbox of an existing service", open since June 2023, 37 reactions. systemd 258 added systemd-analyze unit-shell, which enters a running unit. The latest comments in the thread ask for the same thing for a unit that crashed, as its dynamic user instead of root.
  • systemd/systemd#9907, "Allow passing a .service file to systemd-run", open since August 2018, 20 reactions.

How it works

sandbox-shell reads the unit with systemctl cat, parses it the way systemd does (comments, line continuations, empty assignments that reset a setting), and keeps the [Service] settings that describe the execution environment. It drops the ones that describe the lifecycle: ExecStart= and the other Exec*= lines, Type=, Restart=, timeouts, RuntimeMaxSec=, the standard streams. It expands specifiers such as %N with the name of the original unit, so StateDirectory=%N still points to the service's directory. Then it starts the command with systemd-run --wait --collect --pty.

Two details matter for not disturbing the real service. With DynamicUser=yes and no User=, systemd derives the user name from the unit name. The transient unit has a different name, so it would get a different UID, and systemd would chown the service's state directory to it. The tool therefore sets User= to the name systemd would derive, including the SipHash-based name for unit names that are not valid user names. Units with the same dynamic user name share the UID. And a RuntimeDirectory= gets RuntimeDirectoryPreserve=yes, so it is not deleted when the shell exits.

Before the real run, the tool starts /bin/true once with the same settings. If systemd rejects a setting for transient units, it is left out and named. If the sandbox cannot be set up at all, the tool prints the exit status and the journal lines that explain it:

$ sudo sandbox-shell demo-app.service
sandbox-shell: the sandbox of demo-app.service cannot be set up: the test process exited with status 226/NAMESPACE before it ran.
  Started demo-app-sandbox-574315-preflight.service - sandbox-shell preflight.
  demo-app-sandbox-574315-preflight.service: Failed to set up mount namespacing: /srv/demo-app/uploads: No such file or directory
  demo-app-sandbox-574315-preflight.service: Main process exited, code=exited, status=226/NAMESPACE
  demo-app-sandbox-574315-preflight.service: Failed with result 'exit-code'.
demo-app.service fails to start for the same reason. Use --drop KEY or -p KEY=VALUE to try a fix, or --no-preflight to start anyway.

After creating the directory, the same service, still not running:

$ sudo sandbox-shell demo-app.service -- sh -c 'id; pwd; echo "$APP_ENV"; touch /srv/demo-app/uploads/x && echo uploads writable; touch /etc/x'
sandbox-shell: running sh -c 'id; pwd; echo "$APP_ENV"; touch /srv/demo-app/uploads/x && echo uploads writable; touch /etc/x' as demo-app with 7 settings of demo-app.service
  not carried over: ExecStart
  added: User=demo-app (the dynamic user of demo-app.service)
uid=61953(demo-app) gid=61953(demo-app) groups=61953(demo-app)
/var/lib/private/demo-app
production
touch: cannot touch '/srv/demo-app/uploads/x': Permission denied
touch: cannot touch '/etc/x': Read-only file system

The next problem is visible right away: the uploads directory belongs to root, and the dynamic user cannot write to it.

Testing

27 unit tests cover the parser, the choice of settings, the specifiers and the dynamic user name; the SipHash implementation is checked against the reference test vectors. Eight integration tests run against the real system manager. They install throwaway units under /run/systemd/system and check that the command runs with the UID of the running service's main process (also for a unit name with dots, which needs the hashed user name), sees the environment from a drop-in, has a read-only /usr and a writable state directory, and that the running service and its runtime directory are still there afterwards. Further tests cover a failed service, a template instance, a sandbox failure with status 226, an unknown setting and the exit status. CI runs them on the GitHub runners with systemd 255 and 249.

Limitations

  • It starts a new transient unit and does not join the namespaces of a running service. With PrivateTmp=yes the shell gets its own /tmp. For the running process, systemd-analyze unit-shell or nsenter is the right tool.
  • Socket activation and ExecStartPre= steps are not reproduced.
  • Credentials from LoadCredential= end up in the transient unit's own credentials directory.
  • It has only been tested with systemd 255 and 249.

The code is at github.com/Jwrede/sandbox-shell.