borgmatic-collective/borgmatic
borgmatic-collective/borgmatic
12
63
Fork 16
Code Issues 49 Pull Requests 2 Actions Packages Releases 134 Activity

borgmatic run as a system service should not use /run/user/$UID #934

New Issue
Closed
opened 2024-11-13 08:33:12 +00:00 by SimonPilkington Β· 10 comments
SimonPilkington commented 2024-11-13 08:33:12 +00:00
Copy Link

Why this could be a problem: /run/user/$UID is a tmpfs created by pam_systemd, meaning, if the user does not have any running sessions, it will not exist. Therefore, for example, it isn't possible to make this directory writable if using ProtectSystem=strict.

Suggested solution

Add RuntimeDirectory=borgmatic to the system service unit. This will make systemd create /run/borgmatic, make it writable (even with ProtectSystem=strict), and set RUNTIME_DIRECTORY to that path.
In borgmatic, check if the RUNTIME_DIRECTORYenvironment variable is set, and if yes use that directory.

Note: systemd will wipe any RuntimeDirectories when the unit finishes running, but I don't think this is a problem. See also the docs.

Why this could be a problem: `/run/user/$UID` is a tmpfs created by [pam_systemd](https://www.freedesktop.org/software/systemd/man/latest/pam_systemd.html), meaning, if the user does not have any running sessions, it will not exist. Therefore, for example, it isn't possible to make this directory writable if using `ProtectSystem=strict`. # Suggested solution Add `RuntimeDirectory=borgmatic` to the system service unit. This will make systemd create `/run/borgmatic`, make it writable (even with `ProtectSystem=strict`), and set `RUNTIME_DIRECTORY` to that path. In borgmatic, check if the `RUNTIME_DIRECTORY`environment variable is set, and if yes use that directory. Note: systemd will wipe any RuntimeDirectories when the unit finishes running, but I don't think this is a problem. See also the [docs](https://www.freedesktop.org/software/systemd/man/latest/systemd.exec.html#RuntimeDirectory=).
thomastc commented 2024-11-13 10:45:29 +00:00
Copy Link

I ran into the same issue. It seems to have been introduced in borgmatic 1.9.0 through #562. The documentation for the new user_runtime_directory configuration setting describes the behaviour:

# Path for storing temporary runtime data like streaming database
# dumps and bootstrap metadata. borgmatic automatically creates and
# uses a "borgmatic" subdirectory here. Defaults to $XDG_RUNTIME_DIR
# or or $TMPDIR or $TEMP or /run/user/$UID.
# user_runtime_directory: /run/user/1001/borgmatic

When running from a systemd unit, none of these variables are set, so borgmatic falls back to /run/user/$UID. I think this is the actual bug: if XDG_RUNTIME_DIR does not point at this directory explicitly, borgmatic should not be trying to use or create it. Instead, borgmatic should fall back to /tmp, which is acceptable in POSIX. However, if $XDG_RUNTIME_DIR is set, there should be no harm in using it.

@SimonPilkington's idea of adding $RUNTIME_DIRECTORY to support systemd's RuntimeDirectory= is a good one. I suspect most borgmatic users will be using systemd nowadays, and if not, it does little harm.

So I think the order should be: $XDG_RUNTIME_DIR, $RUNTIME_DIRECTORY, $TMPDIR, $TEMP, /tmp.

Workaround: tell systemd to create a runtime directory, then point TMPDIR at it:

[Service]
RuntimeDirectory=borgmatic # Creates /run/borgmatic and cleans it up afterwards.
ExecStart=/bin/sh -c 'TMPDIR=$RUNTIME_DIRECTORY borgmatic'

You can also just set Environment="TMPDIR=/tmp" in the systemd unit, or user_runtime_directory: /tmp in the borgmatic config, but then systemd won't clean up the directory automatically.

I ran into the same issue. It seems to have been introduced in borgmatic 1.9.0 through #562. The [documentation](https://torsion.org/borgmatic/docs/reference/configuration/) for the new `user_runtime_directory` configuration setting describes the behaviour: ```yaml # Path for storing temporary runtime data like streaming database # dumps and bootstrap metadata. borgmatic automatically creates and # uses a "borgmatic" subdirectory here. Defaults to $XDG_RUNTIME_DIR # or or $TMPDIR or $TEMP or /run/user/$UID. # user_runtime_directory: /run/user/1001/borgmatic ``` When running from a systemd unit, none of these variables are set, so borgmatic falls back to `/run/user/$UID`. **I think this is the actual bug: if `XDG_RUNTIME_DIR` does not point at this directory explicitly, borgmatic should not be trying to use or create it.** Instead, borgmatic should fall back to `/tmp`, which is [acceptable](https://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xbd_chap10.html) in POSIX. However, if `$XDG_RUNTIME_DIR` _is_ set, there should be no harm in using it. @SimonPilkington's idea of adding `$RUNTIME_DIRECTORY` to support systemd's `RuntimeDirectory=` is a good one. I suspect most borgmatic users will be using systemd nowadays, and if not, it does little harm. So I think the order should be: `$XDG_RUNTIME_DIR`, `$RUNTIME_DIRECTORY`, `$TMPDIR`, `$TEMP`, `/tmp`. --- **Workaround**: tell systemd to create a runtime directory, then point `TMPDIR` at it: ``` [Service] RuntimeDirectory=borgmatic # Creates /run/borgmatic and cleans it up afterwards. ExecStart=/bin/sh -c 'TMPDIR=$RUNTIME_DIRECTORY borgmatic' ``` You can also just set `Environment="TMPDIR=/tmp"` in the systemd unit, or `user_runtime_directory: /tmp` in the [borgmatic config](https://torsion.org/borgmatic/docs/reference/configuration/), but then systemd won't clean up the directory automatically.
witten commented 2024-11-13 20:00:24 +00:00
Owner
Copy Link

Use cases

I'm going to co-opt this ticket to take into account both the use cases outlined here and the use cases from #928. So unless I'm missing any, here they are.

Running borgmatic as:

  • A Linux user with a user session. XDG_RUNTIME_DIR is set and /run/user/$UID exists (specific to that user).
  • A Linux user without a session, e.g. a systemd service with ProtectSystem=strict. XDG_RUNTIME_DIR isn't set and /run/user/$UID doesn't exist. But RUNTIME_DIRECTORY could be set if RuntimeDirectory is configured in the systemd service. So if RuntimeDircetory=borgmatic, then RUNTIME_DIRECTORY will be /run/borgmatic.
  • A macOS user. TMPDIR is set to an existent temporary directory specific to that user, something like /var/folders/g7/7du81ti_b7mm84n184fn3k910000lg/T/.
  • A macOS root user, e.g. a global daemon. TMPDIR isn't set, but running getconf DARWIN_USER_TEMP_DIR may get a directory like /var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/. But there's also maybe convention to using a path like ~/Library/Caches/TemporaryItems/$appname/$version?
  • An OpenBSD user. TMPDIR may be set (but to a global directory like /tmp)?
  • A Cygwin user. TEMP may be set?

Some additional considerations:

  • Global temp directories like /tmp are not safe to write to without using something like mktemp or Python's tempfile. User-specific temp directories should be safe to write to with or without using that.
  • borgmatic currently appends /borgmatic to all user runtime directories. The rationale is that, this way, runtime files like database dumps end up stored in the Borg archive at a consistent path, /borgmatic. (borgmatic uses the Borg 1.4+ path stripping feature so only that /borgmatic/... portion of the path gets stored. That is, unless you're using Borg 1.2 or below.)

Potential solutions

There is a Python platformdirs module that handles a lot of this, but it doesn't necessarily handle all of it and isn't customizable if needed. Additionally, it would introduce new Python dependencies to borgmatic.

So here's what I propose, riffing on @thomastc's ideas above. When constructing its runtime directory, borgmatic could probe for and use each of these in turn until finding one that is set:

  • borgmatic's user_runtime_directory option.
  • XDG_RUNTIME_DIR
  • RUNTIME_DIRECTORY
  • TMPDIR + random string (via Python's tempfile)
  • TEMP + random string
  • "/tmp" + random string

There could be the additional optimization that if the path already ends in /borgmatic (e.g. if RUNTIME_DIRECTORY is already /run/borgmatic), then don't tack another /borgmatic onto the end of it.

I think this handles all of the use cases:

  • A Linux user with a user session. Will use XDG_RUNTIME_DIR.
  • A Linux user without a session. Will use RUNTIME_DIRECTORY (assuming RuntimeDirectory is configured in the systemd service).
  • A macOS user. Will use TMPDIR + random string. The random string isn't strictly necessary for this use case, but see OpenBSD below.
  • A macOS root user, e.g. a global daemon. Will fall back to /tmp + random string. And I guess /tmp is a link to /private/tmp/?
  • An OpenBSD user. Will use TMPDIR + random string or /tmp + random string. The random string is needed in the first case because TMPDIR can just be the global /tmp.
  • A Cygwin user. Will use TEMP + random string.

Again, /borgmatic implicitly gets tacked onto all of these. And with this particular approach, everything is done without having to perform platform probing or run external executables or probe for paths on the filesystems. (Although I guess doing the random string thing with tempfile does hit the filesystem.)

Phew.

Okay, what am I missing?

### Use cases I'm going to co-opt this ticket to take into account both the use cases outlined here *and* the use cases from #928. So unless I'm missing any, here they are. Running borgmatic as: * A Linux user with a user session. `XDG_RUNTIME_DIR` is set and `/run/user/$UID` exists (specific to that user). * A Linux user without a session, e.g. a systemd service with `ProtectSystem=strict`. `XDG_RUNTIME_DIR` isn't set and `/run/user/$UID` doesn't exist. But `RUNTIME_DIRECTORY` could be set if `RuntimeDirectory` is configured in the systemd service. So if `RuntimeDircetory=borgmatic`, then `RUNTIME_DIRECTORY` will be `/run/borgmatic`. * A macOS user. `TMPDIR` is set to an existent temporary directory specific to that user, something like `/var/folders/g7/7du81ti_b7mm84n184fn3k910000lg/T/`. * A macOS root user, e.g. a global daemon. `TMPDIR` isn't set, but running `getconf DARWIN_USER_TEMP_DIR` may get a directory like `/var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/`. But there's also maybe convention to using a path like `~/Library/Caches/TemporaryItems/$appname/$version`? * An OpenBSD user. `TMPDIR` *may* be set (but to a global directory like `/tmp`)? * A Cygwin user. `TEMP` may be set? Some additional considerations: * Global temp directories like `/tmp` are not safe to write to without using something like `mktemp` or Python's `tempfile`. User-specific temp directories should be safe to write to with or without using that. * borgmatic currently appends `/borgmatic` to all user runtime directories. The rationale is that, this way, runtime files like database dumps end up stored in the Borg archive at a consistent path, `/borgmatic`. (borgmatic uses the Borg 1.4+ path stripping feature so only that `/borgmatic/...` portion of the path gets stored. That is, unless you're using Borg 1.2 or below.) ### Potential solutions There is a Python [platformdirs](https://github.com/tox-dev/platformdirs) module that handles a lot of this, but it doesn't necessarily handle all of it and isn't customizable if needed. Additionally, it would introduce new Python dependencies to borgmatic. So here's what I propose, riffing on @thomastc's ideas above. When constructing its runtime directory, borgmatic could probe for and use each of these in turn until finding one that is set: * borgmatic's `user_runtime_directory` option. * `XDG_RUNTIME_DIR` * `RUNTIME_DIRECTORY` * `TMPDIR` + random string (via Python's `tempfile`) * `TEMP` + random string * `"/tmp"` + random string There could be the additional optimization that if the path already ends in `/borgmatic` (e.g. if `RUNTIME_DIRECTORY` is already `/run/borgmatic`), then don't tack another `/borgmatic` onto the end of it. I *think* this handles all of the use cases: * A Linux user with a user session. Will use `XDG_RUNTIME_DIR`. * A Linux user without a session. Will use `RUNTIME_DIRECTORY` (assuming `RuntimeDirectory` is configured in the systemd service). * A macOS user. Will use `TMPDIR` + random string. The random string isn't strictly necessary for this use case, but see OpenBSD below. * A macOS root user, e.g. a global daemon. Will fall back to `/tmp` + random string. And I guess `/tmp` is a link to `/private/tmp/`? * An OpenBSD user. Will use `TMPDIR` + random string or `/tmp` + random string. The random string is needed in the first case because `TMPDIR` can just be the global `/tmp`. * A Cygwin user. Will use `TEMP` + random string. Again, `/borgmatic` implicitly gets tacked onto all of these. And with this particular approach, everything is done without having to perform platform probing or run external executables or probe for paths on the filesystems. (Although I guess doing the random string thing with `tempfile` does hit the filesystem.) Phew. Okay, what am I missing?
sthen commented 2024-11-14 10:12:42 +00:00
Copy Link

I think that covers everything.

I think that covers everything.
πŸ‘ 1
SimonPilkington commented 2024-11-14 10:19:57 +00:00
Author
Copy Link

I only speak for the Linux platform but XDG_RUNTIME_DIR for users and RUNTIME_DIRECTORY for system unless overridden by borgmatic config LGTM. Falling back to /tmp if nothing is set also LGTM. If borgmatic is creating a subdirectory it should have mode 0700 to be XDG-compliant.

However, now that I've seen mention of database dumps (which I don't use) I'm wondering if XDG_RUNTIME_DIR is actually suitable.

The spec says:

Applications should use this directory for communication and synchronization purposes and should not place larger files in it, since it might reside in runtime memory and cannot necessarily be swapped out to disk.

And a database dump seems like it could potentially get pretty large. (For reference on my system XDG_RUNTIME_DIR is a ~6GiB tmpfs.)

There could be the additional optimization that if the path already ends in /borgmatic (e.g. if RUNTIME_DIRECTORY is already /run/borgmatic), then don't tack another /borgmatic onto the end of it.

I believe an endswith('/borgmatic') check should be okay for this. A directory called borgmatic is probably intended to be used by borgmatic.

I only speak for the Linux platform but `XDG_RUNTIME_DIR` for users and `RUNTIME_DIRECTORY` for system unless overridden by borgmatic config LGTM. Falling back to `/tmp` if nothing is set also LGTM. If borgmatic is creating a subdirectory it should have mode `0700` to be XDG-compliant. However, now that I've seen mention of database dumps (which I don't use) I'm wondering if `XDG_RUNTIME_DIR` is actually suitable. The spec says: ``` Applications should use this directory for communication and synchronization purposes and should not place larger files in it, since it might reside in runtime memory and cannot necessarily be swapped out to disk. ``` And a database dump seems like it could potentially get pretty large. (For reference on my system `XDG_RUNTIME_DIR` is a ~6GiB tmpfs.) >There could be the additional optimization that if the path already ends in /borgmatic (e.g. if RUNTIME_DIRECTORY is already /run/borgmatic), then don't tack another /borgmatic onto the end of it. I believe an `endswith('/borgmatic')` check should be okay for this. A directory called borgmatic is probably intended to be used by borgmatic.
witten commented 2024-11-14 15:20:41 +00:00
Owner
Copy Link

If borgmatic is creating a subdirectory it should have mode 0700 to be XDG-compliant.

Understood. It's certainly the case for anything created by tempfile.

And a database dump seems like it could potentially get pretty large. (For reference on my system XDG_RUNTIME_DIR is a ~6GiB tmpfs.)

Good point. However in the common case, borgmatic doesn't put database dumps there; it puts named pipes there for streaming database dumps directly to Borg. And on this particular page about XDG, the description of XDG_RUNTIME_DIR says it's for (among other things) "non-essential, user-specific data files such as sockets, named pipes, etc."

The one exception I can think of is for "directory" format databases, which by their nature can't be streamed directly to Borg and have to be dumped to disk. But most users don't use that format.

> If borgmatic is creating a subdirectory it should have mode 0700 to be XDG-compliant. Understood. It's certainly the case for anything created by `tempfile`. > And a database dump seems like it could potentially get pretty large. (For reference on my system XDG_RUNTIME_DIR is a ~6GiB tmpfs.) Good point. However in the common case, borgmatic doesn't put database dumps there; it puts named pipes there for _streaming_ database dumps directly to Borg. And on [this particular page about XDG](https://wiki.archlinux.org/title/XDG_Base_Directory), the description of `XDG_RUNTIME_DIR` says it's for (among other things) "non-essential, user-specific data files such as sockets, named pipes, etc." The one exception I can think of is for "directory" format databases, which by their nature can't be streamed directly to Borg and have to be dumped to disk. But most users don't use that format.
SimonPilkington commented 2024-11-14 15:32:17 +00:00
Author
Copy Link

Good point. However in the common case, borgmatic doesn't put database dumps there; it puts named pipes there for streaming database dumps directly to Borg. And on this particular page about XDG, the description of XDG_RUNTIME_DIR says it's for (among other things) "non-essential, user-specific data files such as sockets, named pipes, etc."

The one exception I can think of is for "directory" format databases, which by their nature can't be streamed directly to Borg and have to be dumped to disk. But most users don't use that format.

Oh I see, that's an implementation detail I didn't know. Pardon my nonsense then, pipes absolutely belong in XDG_RUNTIME_DIR.

I suppose there is only the caveat of an increased chance of eating an ENOSPC with the directory format compared to dumping to $HOME/.borgmatic. If you wish to mitigate this and be XDG-compliant I believe XDG_CACHE_HOME would be the correct choice.

> Good point. However in the common case, borgmatic doesn't put database dumps there; it puts named pipes there for _streaming_ database dumps directly to Borg. And on [this particular page about XDG](https://wiki.archlinux.org/title/XDG_Base_Directory), the description of `XDG_RUNTIME_DIR` says it's for (among other things) "non-essential, user-specific data files such as sockets, named pipes, etc." > > The one exception I can think of is for "directory" format databases, which by their nature can't be streamed directly to Borg and have to be dumped to disk. But most users don't use that format. Oh I see, that's an implementation detail I didn't know. Pardon my nonsense then, pipes absolutely belong in `XDG_RUNTIME_DIR`. I suppose there is only the caveat of an increased chance of eating an `ENOSPC` with the directory format compared to dumping to `$HOME/.borgmatic`. If you wish to mitigate this and be XDG-compliant I believe `XDG_CACHE_HOME` would be the correct choice.
witten commented 2024-11-14 15:41:57 +00:00
Owner
Copy Link

Thanks, I'll consider that as part of this. But it may not be worth writing to different places for different dump formats given the additional complexity. And a user can always set their own "runtime" directory via the borgmatic configuration file option if they do run out of space.

Thanks, I'll consider that as part of this. But it may not be worth writing to different places for different dump formats given the additional complexity. And a user can always set their own "runtime" directory via the borgmatic configuration file option if they do run out of space.
πŸ‘ 1
witten referenced this issue from a commit 2024-11-16 15:26:51 +00:00
Fix outdated comment (#934).
witten commented 2024-11-17 19:58:45 +00:00
Owner
Copy Link

Implemented in main, and will part of the next release. The updated documentation should be online here shortly: https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#runtime-directory

Thanks to everyone for weighing in on this one!

Implemented in main, and will part of the next release. The updated documentation should be online here shortly: https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#runtime-directory Thanks to everyone for weighing in on this one!
πŸŽ‰ 1 πŸš€ 1
thomastc commented 2024-11-18 09:15:34 +00:00
Copy Link

Thank you for the amazing response time on this!

Thank you for the amazing response time on this!
πŸ‘ 1
witten commented 2024-11-19 04:45:54 +00:00
Owner
Copy Link

Released in borgmatic 1.9.2!

Released in borgmatic 1.9.2!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
blocked
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
No Label
Milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
4 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#934
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?