manifest.json missing with certain exclude patterns #1122

New Issue

lingfish · 2025-07-31T04:42:49Z

lingfish commented

2025-07-31 04:42:49 +00:00

What I'm trying to do and why

Trying to use config bootstrap and discovered a backup is missing the manifest stuff.

Steps to reproduce

Use a fairly stock config, with this exclude config:

source_directories:
    - /

patterns:
    - R /
    - '- home/*/.cache'
    - '- home/*/Downloads'
    - '- **/CacheStorage'
    - '+ home/'
    - '! re:^(dev|proc|run|sys|tmp)'
    - '+ etc/'
    - '- */lib/'
    - '- var/lib'
    - '- **/cache/**'
    - '+ var/'
    - '- **'

Actual behavior

borg(matic) chose a runtime dir under /tmp, and upon listing the archive, manifest.json is missing.

Expected behavior

Even though /tmp is excluded, I thought borgmatic would still include it's manifest.

Other notes / implementation ideas

No response

borgmatic version

2.0.7

borgmatic installation method

pipx

Borg version

borg 1.4.1

Python version

Python 3.9.5

Database version (if applicable)

No response

Operating system and version

NAME="Ubuntu" VERSION="20.04.6 LTS (Focal Fossa)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 20.04.6 LTS" VERSION_ID="20.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=focal UBUNTU_CODENAME=focal

### What I'm trying to do and why Trying to use `config bootstrap` and discovered a backup is missing the manifest stuff. ### Steps to reproduce Use a fairly stock config, with this exclude config: ``` source_directories: - / patterns: - R / - '- home/*/.cache' - '- home/*/Downloads' - '- **/CacheStorage' - '+ home/' - '! re:^(dev|proc|run|sys|tmp)' - '+ etc/' - '- */lib/' - '- var/lib' - '- **/cache/**' - '+ var/' - '- **' ``` ### Actual behavior borg(matic) chose a runtime dir under `/tmp`, and upon listing the archive, `manifest.json` is missing. ### Expected behavior Even though `/tmp` is excluded, I thought `borgmatic` would still include it's manifest. ### Other notes / implementation ideas _No response_ ### borgmatic version 2.0.7 ### borgmatic installation method pipx ### Borg version borg 1.4.1 ### Python version Python 3.9.5 ### Database version (if applicable) _No response_ ### Operating system and version NAME="Ubuntu" VERSION="20.04.6 LTS (Focal Fossa)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 20.04.6 LTS" VERSION_ID="20.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=focal UBUNTU_CODENAME=focal

witten commented

2025-08-09 04:27:42 +00:00

Thanks for filing this! I'll see if I can reproduce it and let you know. (I'm not sure at this point though if the solution is just documenting this interaction better.)

witten commented

2025-08-09 04:48:19 +00:00

This does repro for me locally when I exclude /tmp and the runtime directory is in there. Unfortunately there's not a good way for borgmatic to override the exclude, because ! in a pattern indicates to Borg not to recurse into that directory (even when there's an explicit + include for another path inside of it).

There is actually some existing borgmatic logic that errors if you inadvertently exclude the runtime directory as is the case here, but it's only triggered for a different use case (when databases are enabled). That may need to change.

This does repro for me locally when I exclude `/tmp` and the runtime directory is in there. Unfortunately there's not a good way for borgmatic to override the exclude, because `!` in a pattern indicates to Borg not to recurse into that directory (even when there's an explicit `+` include for another path inside of it). There is actually some existing borgmatic logic that errors if you inadvertently exclude the runtime directory as is the case here, but it's only triggered for a different use case (when databases are enabled). That may need to change.

lingfish commented

2025-08-11 07:02:58 +00:00

Hi @witten and thanks for looking into it.

So what is the solution (or I guess more accurately, the workaround)? Referring to here, we have what I see the following issues:

Number	Location	Issue
1	The `user_runtime_directory`	No sane default set?
2	`XDG_RUNTIME_DIR`	Almost always in a transient directory, not likely something sane to backup
4...	Various iterations of `/tmp`	See original issue ;)

I guess, overall, this feature assumes backing up "temporary" directories.

(Oh, also, the doco around runtime is currently in a section/page about database backups as you'd of course know... though originally for them, I guess it's scope grew and it belongs elsewhere?)

Hi @witten and thanks for looking into it. So what is the solution (or I guess more accurately, the workaround)? Referring to [here](https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#runtime-directory), we have what I see the following issues: | Number | Location | Issue | | --- | --- | --- | | 1 | The `user_runtime_directory` | No sane default set? | | 2 | `XDG_RUNTIME_DIR` | Almost always in a transient directory, not likely something sane to backup | | 4... | Various iterations of `/tmp` | See original issue ;) | I guess, overall, this feature assumes backing up "temporary" directories. (Oh, also, the doco around runtime is currently in a section/page about database backups as you'd of course know... though originally for them, I guess it's scope grew and it belongs elsewhere?)

witten commented

2025-09-12 16:23:23 +00:00

So what is the solution (or I guess more accurately, the workaround)?

The workaround is for the user to avoid excluding any path containing the runtime directory! 😄 As for a "solution," my plan is to look into whether borgmatic can be made smart enough to warn the user (or outright error) when it detects that the runtime directory has been excluded. No promises though.

1 The user_runtime_directory No sane default set?

I'm not sure what you mean here. The items numbered 2 through 6 in the linked documentation are the default if user_runtime_directory isn't set. Or are you saying those defaults aren't sane?

2 XDG_RUNTIME_DIR Almost always in a transient directory, not likely something sane to backup

The piece that you may be missing here is that borgmatic rewrites the borgmatic runtime directory on its way into the Borg archive (assuming you're using Borg version 1.4+, which it looks like you are). So for instance if XDG_RUNTIME_DIR is /run/user/1000 and borgmatic therefore uses /run/user/1000/borgmatic for its runtime directory, that path won't show up in the Borg archive as /run/user/1000/borgmatic. Instead, it'll show up as a fixed /borgmatic within the archive due to the magic of path rewriting.

4... Various iterations of /tmp See original issue ;)

Same thing here. Any borgmatic runtime directory in /tmp will get rewritten on its way into the archive—assuming it doesn't get inadvertently excluded as per the original issue.

(Oh, also, the doco around runtime is currently in a section/page about database backups as you'd of course know... though originally for them, I guess it's scope grew and it belongs elsewhere?)

Totally. In general, the documentation could benefit from some refactoring. See #942.

> So what is the solution (or I guess more accurately, the workaround)? The workaround is for the user to avoid excluding any path containing the runtime directory! 😄 As for a "solution," my plan is to look into whether borgmatic can be made smart enough to warn the user (or outright error) when it detects that the runtime directory has been excluded. No promises though. > 1 The `user_runtime_directory` No sane default set? I'm not sure what you mean here. The items numbered 2 through 6 in the [linked documentation](https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#runtime-directory) *are* the default if `user_runtime_directory` isn't set. Or are you saying those defaults aren't sane? > 2 `XDG_RUNTIME_DIR` Almost always in a transient directory, not likely something sane to backup The piece that you may be missing here is that borgmatic rewrites the borgmatic runtime directory on its way into the Borg archive (assuming you're using Borg version 1.4+, which it looks like you are). So for instance if `XDG_RUNTIME_DIR` is `/run/user/1000` and borgmatic therefore uses `/run/user/1000/borgmatic` for its runtime directory, that path *won't* show up in the Borg archive as `/run/user/1000/borgmatic`. Instead, it'll show up as a fixed `/borgmatic` within the archive due to the magic of path rewriting. > 4... Various iterations of `/tmp` See original issue ;) Same thing here. Any borgmatic runtime directory in `/tmp` will get rewritten on its way into the archive—assuming it doesn't get inadvertently excluded as per the original issue. > (Oh, also, the doco around runtime is currently in a section/page about database backups as you'd of course know... though originally for them, I guess it's scope grew and it belongs elsewhere?) Totally. In general, the documentation could benefit from some refactoring. See #942.