LVM and runtime dir exclusion

lingfish commented

2025-12-18 01:47:04 +00:00

Contributor

What I'm trying to do and why

In the aftermath of #1122, I now am finding that LVM snap based backups "silently"ish soft fail when runtime dir is inadvertently excluded.

Steps to reproduce

This config:

lvm:

repositories:
[ ... ]

patterns:
    - R /
    - '! dev'
    - '! cifs'
    - '! net'
    - '! media'
    - '! mnt'
    - '! net'
    - '! proc'
    #- '! run'
    - '! sys'
    - '! tmp'
    - '! var/tmp'
    - '! home/user/.cache'
    - '! home/user/Resolve/media'
    - '! scratch'

Running via cron
verbosity shows /tmp for runtime:
Using runtime directory /tmp/borgmatic-4ams1w6c/borgmatic
/tmp is obviously excluded as above

Actual behavior

Snaps etc are created, but the resulting archive only has:

qnap-elara: Listing archive elara-2025-12-18T11:59:02.357296
-rw------- root   root       1375 Thu, 2025-12-18 11:33:07 etc/borgmatic.d/elara.yaml
drwxr-xr-x root   root          0 Thu, 2025-12-18 11:59:01 borgmatic/bootstrap
-rw-r--r-- root   root         80 Thu, 2025-12-18 11:59:01 borgmatic/bootstrap/manifest.json

This obviously happens because the LVM snap is mounted under /tmp/borgs_runtime_dir.

Expected behavior

I had kinda expected #1122 to error out; the context of that bug of course was manifest, but ... yeah.

Other notes / implementation ideas

Though I get why a sane fallback to /tmp for runtime, I also don't want to backup /tmp, so this'll make for a fascinating series of pattern includes/excludes 😉

borgmatic version

2.0.12

borgmatic installation method

pipx

Borg version

borg 1.4.3

Python version

Python 3.13.5

Database version (if applicable)

No response

Operating system and version

PRETTY_NAME="Debian GNU/Linux 13 (trixie)" NAME="Debian GNU/Linux" VERSION_ID="13" VERSION="13 (trixie)" VERSION_CODENAME=trixie DEBIAN_VERSION_FULL=13.2 ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/"

### What I'm trying to do and why In the aftermath of #1122, I now am finding that LVM snap based backups "silently"ish soft fail when runtime dir is inadvertently excluded. ### Steps to reproduce This config: ``` lvm: repositories: [ ... ] patterns: - R / - '! dev' - '! cifs' - '! net' - '! media' - '! mnt' - '! net' - '! proc' #- '! run' - '! sys' - '! tmp' - '! var/tmp' - '! home/user/.cache' - '! home/user/Resolve/media' - '! scratch' ``` - Running via `cron` - verbosity shows `/tmp` for runtime: `Using runtime directory /tmp/borgmatic-4ams1w6c/borgmatic` - `/tmp` is obviously excluded as above ### Actual behavior Snaps etc are created, but the resulting archive only has: ``` qnap-elara: Listing archive elara-2025-12-18T11:59:02.357296 -rw------- root root 1375 Thu, 2025-12-18 11:33:07 etc/borgmatic.d/elara.yaml drwxr-xr-x root root 0 Thu, 2025-12-18 11:59:01 borgmatic/bootstrap -rw-r--r-- root root 80 Thu, 2025-12-18 11:59:01 borgmatic/bootstrap/manifest.json ``` This obviously happens because the LVM snap is mounted under `/tmp/borgs_runtime_dir`. ### Expected behavior I had kinda expected #1122 to error out; the context of that bug of course was manifest, but ... yeah. ### Other notes / implementation ideas Though I get why a sane fallback to `/tmp` for runtime, I also don't want to backup `/tmp`, so this'll make for a fascinating series of pattern includes/excludes 😉 ### borgmatic version 2.0.12 ### borgmatic installation method pipx ### Borg version borg 1.4.3 ### Python version Python 3.13.5 ### Database version (if applicable) _No response_ ### Operating system and version ```PRETTY_NAME="Debian GNU/Linux 13 (trixie)" NAME="Debian GNU/Linux" VERSION_ID="13" VERSION="13 (trixie)" VERSION_CODENAME=trixie DEBIAN_VERSION_FULL=13.2 ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/"```

witten commented

2025-12-18 03:21:41 +00:00

Owner

A few thoughts:

Are you sure that the reason the archive is missing LVM snapshots is because /tmp is excluded? I see that you have '! tmp' in the excludes, but I would not expect that to match anything unless: 1. borgmatic was run from the / directory or 2. the working_directory was set to /.

I know that you have R / as a root pattern, but my understanding is that that subsequent exclude patterns aren't automatically relative to the root. You'd have to specify an absolute path like ! /tmp (or using working_directory or whatever).

So if these excludes are invalid, then why doesn't all of / end up in the archive? That I don't know. Can I see the portion of the verbose borgmatic output that prints out the processed patterns that borgmatic is sending to Borg?

A few thoughts: Are you sure that the reason the archive is missing LVM snapshots is because `/tmp` is excluded? I see that you have `'! tmp'` in the excludes, but I would not expect that to match anything unless: 1. borgmatic was run from the `/` directory or 2. the `working_directory` was set to `/`. I know that you have `R /` as a root pattern, but [my understanding](https://torsion.org/borgmatic/reference/configuration/patterns-and-excludes/#patterns) is that that subsequent exclude patterns *aren't* automatically relative to the root. You'd have to specify an absolute path like `! /tmp` (or using `working_directory` or whatever). So if these excludes are invalid, then why doesn't all of `/` end up in the archive? That I don't know. Can I see the portion of the verbose borgmatic output that prints out the [processed patterns](https://torsion.org/borgmatic/reference/configuration/patterns-and-excludes/#debugging) that borgmatic is sending to Borg?

lingfish commented

2025-12-18 04:35:46 +00:00

Author

Contributor

A few thoughts:

Are you sure that the reason the archive is missing LVM snapshots is because /tmp is excluded? I see that you have '! tmp' in the excludes, but I would not expect that to match anything unless: 1. borgmatic was run from the / directory or 2. the working_directory was set to /.

I'm sure. I actually saw borg output the exclusion using the progress/list option. I've lost it in my scrollback, but there was definitely a:

x /run/0/....

... that covered the whole snap mount.

Re the leading slashes -- I tried both, and was using leading up until this issue (which was all brought about/broken by me upgrading to Debian 13, and I still don't know why this broke my backups overall).

I ended up with removing the leaders because of this example in the borg doco:

# Define the recursion root
R /
# Exclude all iso files in any directory
- **/*.iso
# Explicitly include all inside etc and root
+ etc/**
+ root/**
# Exclude a specific directory under each user's home directories
- home/*/.cache
# Explicitly include everything in /home
+ home/**
# Explicitly exclude some directories without recursing into them
! re:^(dev|proc|run|sys|tmp)
# Exclude all other files and directories
# that are not specifically included earlier.
- **

I know that you have R / as a root pattern, but my understanding is that that subsequent exclude patterns aren't automatically relative to the root. You'd have to specify an absolute path like ! /tmp (or using working_directory or whatever).

Yeah, see above. It worked with them just fine before upgrading. The same behaviour is exhibited either way. I also tried:

Comparing Python versions (only went from 3.11 to 13)
Downgrading both borg and borgmatic to the versions I was using before the trixie upgrade

So if these excludes are invalid, then why doesn't all of / end up in the archive? That I don't know. Can I see the portion of the verbose borgmatic output that prints out the processed patterns that borgmatic is sending to Borg?

Sure, but it wasn't helpful to me:

R /etc/borgmatic.d/elara.yaml
+ /etc/borgmatic.d/elara.yaml
R /run/user/0/./borgmatic/bootstrap
+ /run/user/0/./borgmatic/bootstrap
R /run/user/0/borgmatic/lvm_snapshots/3751af5bd81011e6d21d/./
! /dev
! /run/user/0/borgmatic/lvm_snapshots/3751af5bd81011e6d21d/./cifs
! /run/user/0/borgmatic/lvm_snapshots/3751af5bd81011e6d21d/./net
! /run/user/0/borgmatic/lvm_snapshots/3751af5bd81011e6d21d/./media
! /run/user/0/borgmatic/lvm_snapshots/3751af5bd81011e6d21d/./mnt
! /proc
! /run
! /sys
! /tmp
! /run/user/0/borgmatic/lvm_snapshots/3751af5bd81011e6d21d/./var/tmp
! /run/user/0/borgmatic/lvm_snapshots/3751af5bd81011e6d21d/./home/jason/.cache
! /run/user/0/borgmatic/lvm_snapshots/3751af5bd81011e6d21d/./home/jason/Resolve/media
+ /run/user/0/borgmatic/lvm_snapshots/3751af5bd81011e6d21d/./
! /scratch

Debugging was further confused due to me fault-finding using a shell, vs it normally running via cron, which of course changed the runtime dir from /run/user/0 to /tmp respectively. The paste above was from an interactive.

The only thing that revealed exclusion was using -v2 -n create -c /etc/borgmatic.d/myconfig.yaml --list. Whilst interactive, commenting out the /run line fixed it.

And now, commenting out the /tmp line and scheduling a cron run fixed it too, confirming my suspicions.

Something weird with the dot slash hack? I dunno.

> A few thoughts: > > Are you sure that the reason the archive is missing LVM snapshots is because `/tmp` is excluded? I see that you have `'! tmp'` in the excludes, but I would not expect that to match anything unless: 1. borgmatic was run from the `/` directory or 2. the `working_directory` was set to `/`. I'm sure. I actually saw borg output the exclusion using the progress/list option. I've lost it in my scrollback, but there was definitely a: ``` x /run/0/.... ``` ... that covered the whole snap mount. Re the leading slashes -- I tried both, and was using leading up until this issue (which was all brought about/broken by me upgrading to Debian 13, and I still don't know why this broke my backups overall). I ended up with removing the leaders because of this example in the [borg doco](https://borgbackup.readthedocs.io/en/stable/usage/help.html#borg-help-patterns): ``` # Define the recursion root R / # Exclude all iso files in any directory - **/*.iso # Explicitly include all inside etc and root + etc/** + root/** # Exclude a specific directory under each user's home directories - home/*/.cache # Explicitly include everything in /home + home/** # Explicitly exclude some directories without recursing into them ! re:^(dev|proc|run|sys|tmp) # Exclude all other files and directories # that are not specifically included earlier. - ** ``` > I know that you have `R /` as a root pattern, but [my understanding](https://torsion.org/borgmatic/reference/configuration/patterns-and-excludes/#patterns) is that that subsequent exclude patterns *aren't* automatically relative to the root. You'd have to specify an absolute path like `! /tmp` (or using `working_directory` or whatever). Yeah, see above. It worked with them just fine before upgrading. The same behaviour is exhibited either way. I also tried: - Comparing Python versions (only went from 3.11 to 13) - Downgrading both `borg` and `borgmatic` to the versions I was using before the trixie upgrade > So if these excludes are invalid, then why doesn't all of `/` end up in the archive? That I don't know. Can I see the portion of the verbose borgmatic output that prints out the [processed patterns](https://torsion.org/borgmatic/reference/configuration/patterns-and-excludes/#debugging) that borgmatic is sending to Borg? Sure, but it wasn't helpful to me: ``` R /etc/borgmatic.d/elara.yaml + /etc/borgmatic.d/elara.yaml R /run/user/0/./borgmatic/bootstrap + /run/user/0/./borgmatic/bootstrap R /run/user/0/borgmatic/lvm_snapshots/3751af5bd81011e6d21d/./ ! /dev ! /run/user/0/borgmatic/lvm_snapshots/3751af5bd81011e6d21d/./cifs ! /run/user/0/borgmatic/lvm_snapshots/3751af5bd81011e6d21d/./net ! /run/user/0/borgmatic/lvm_snapshots/3751af5bd81011e6d21d/./media ! /run/user/0/borgmatic/lvm_snapshots/3751af5bd81011e6d21d/./mnt ! /proc ! /run ! /sys ! /tmp ! /run/user/0/borgmatic/lvm_snapshots/3751af5bd81011e6d21d/./var/tmp ! /run/user/0/borgmatic/lvm_snapshots/3751af5bd81011e6d21d/./home/jason/.cache ! /run/user/0/borgmatic/lvm_snapshots/3751af5bd81011e6d21d/./home/jason/Resolve/media + /run/user/0/borgmatic/lvm_snapshots/3751af5bd81011e6d21d/./ ! /scratch ``` Debugging was further confused due to me fault-finding using a shell, vs it normally running via `cron`, which of course changed the runtime dir from `/run/user/0` to `/tmp` respectively. The paste above was from an interactive. The only thing that revealed exclusion was using `-v2 -n create -c /etc/borgmatic.d/myconfig.yaml --list`. Whilst interactive, commenting out the `/run` line fixed it. And now, commenting out the `/tmp` line and scheduling a cron run fixed it too, confirming my suspicions. Something weird with the dot slash hack? I dunno.

witten commented

2025-12-18 18:46:20 +00:00

Owner

Thanks for details and the explanation about the leading slashes. I guess because Borg doesn't store leading slashes as part of archive paths, it may not need them in patterns either.

Re the leading slashes -- I tried both, and was using leading up until this issue (which was all brought about/broken by me upgrading to Debian 13, and I still don't know why this broke my backups overall).

Fun fact: Debian 13 changed /tmp to use tmpfs, which earlier Debian versions didn't. (/run is also tmpfs, but that may have already been the case.) But I'm assuming you don't have one_file_system set in your config file, so the Debian change should hopefully not impact anything here. Maybe though the Debian upgrade changed some of the inputs to the runtime directory path determination.

In any case, there are four checks gating the error you were expecting about the runtime directory. All have to evaluate as true in order to produce that error:

The runtime directory is a parent directory of any (root) pattern passed to Borg, meaning we expect that Borg is planning to backup the runtime directory.
But of the paths that Borg actually reports it plans to backup, none of them are in the borgmatic runtime directory.
This isn't a dry run.
And the runtime directory exists on disk.

Looking at your computed patterns, check number 1 looks to be satisfied. And I'm guessing checks 3 and 4 are true as well. So that leaves check 2. Given that the bootstrap directory is getting included in the archive, that leads me to believe that at least some portion of the runtime directory was in the paths that Borg planned to backup. And that's consistent with the computed patterns, in which the bootstrap directory gets inserted before any user excludes.

So, code-wise, that explains why you're not receiving the error you expect. The borgmatic runtime directory is only getting partially excluded from the Borg archive, and therefore the error doesn't trigger.

I'll need to think about what if anything I can do about this.

Thanks for details and the explanation about the leading slashes. I guess because Borg doesn't store leading slashes as part of archive paths, it may not need them in patterns either. > Re the leading slashes -- I tried both, and was using leading up until this issue (which was all brought about/broken by me upgrading to Debian 13, and I still don't know why this broke my backups overall). Fun fact: [Debian 13 changed `/tmp` to use `tmpfs`](https://lwn.net/Articles/975565/), which earlier Debian versions didn't. (`/run` is also `tmpfs`, but that may have already been the case.) But I'm assuming you don't have `one_file_system` set in your config file, so the Debian change should hopefully not impact anything here. Maybe though the Debian upgrade changed some of the inputs to the [runtime directory path determination](https://torsion.org/borgmatic/reference/configuration/runtime-directory/). In any case, there are four checks gating the error you were expecting about the runtime directory. All have to evaluate as true in order to produce that error: 1. The runtime directory is a parent directory of any (root) pattern passed to Borg, meaning we expect that Borg is planning to backup the runtime directory. 2. But of the paths that Borg actually reports it plans to backup, none of them are in the borgmatic runtime directory. 3. This isn't a dry run. 4. And the runtime directory exists on disk. Looking at your computed patterns, check number 1 looks to be satisfied. And I'm guessing checks 3 and 4 are true as well. So that leaves check 2. Given that the bootstrap directory *is* getting included in the archive, that leads me to believe that at least some portion of the runtime directory was in the paths that Borg planned to backup. And that's consistent with the computed patterns, in which the bootstrap directory gets inserted before any user excludes. So, code-wise, that explains why you're not receiving the error you expect. The borgmatic runtime directory is only getting partially excluded from the Borg archive, and therefore the error doesn't trigger. I'll need to think about what if anything I can do about this.

👍 1

witten added the

bug

label

2025-12-18 22:08:10 +00:00

witten referenced this issue from a commit

2025-12-19 03:05:07 +00:00

Fix that errors and exits when the borgmatic runtime directory is partially excluded by configured excludes (#1203).

witten commented

2025-12-19 03:08:10 +00:00

Owner

Okay, this is fixed in main and will be part of the next release. Previously, you only got the runtime directory error if all patterns inside the runtime directory got excluded. Now, you'll get the error if any patterns inside the runtime directory get excluded.

One other note though on the topic of leading slash paths in your patterns. I do recommend using leading slashes there, because otherwise borgmatic can't tell if a pattern is intended to represent an absolute or relative path. And due to that, borgmatic won't rewrite any exclude patterns such that they apply to rewritten LVM snapshots paths—meaning that some of your excludes won't get applied.

Anyway, thanks for bringing this to my attention!

Okay, this is fixed in main and will be part of the next release. Previously, you only got the runtime directory error if _all_ patterns inside the runtime directory got excluded. Now, you'll get the error if _any_ patterns inside the runtime directory get excluded. One other note though on the topic of leading slash paths in your patterns. I do recommend using leading slashes there, because otherwise borgmatic can't tell if a pattern is intended to represent an absolute or relative path. And due to that, borgmatic won't rewrite any exclude patterns such that they apply to rewritten LVM snapshots paths—meaning that some of your excludes won't get applied. Anyway, thanks for bringing this to my attention!

❤️ 1

witten closed this issue

2025-12-19 03:08:10 +00:00

lingfish commented

2025-12-19 03:13:00 +00:00

Author

Contributor

Noted re leading slashes!

Dan, always a pleasure to work with you. I really appreciate your politeness, the time you take with issues, and the responsiveness. As silly as this sounds, I enjoy logging issues with you 😉.

borgmatic is awesome, and you are a great example to the open source community.

Noted re leading slashes! Dan, always a pleasure to work with you. I really appreciate your politeness, the time you take with issues, and the responsiveness. As silly as this sounds, I enjoy logging issues with you 😉. `borgmatic` is awesome, and you are a great example to the open source community.

witten commented

2025-12-19 03:19:19 +00:00

Owner

Thank you so much for the kind words. I really appreciate it! 😊 Keep on filing your extensively detailed tickets whenever you come across problems.

witten commented

2025-12-21 01:27:35 +00:00

Owner

Released in borgmatic 2.0.13!

❤️ 1

witten referenced this issue

2025-12-23 18:36:14 +00:00

Snapshots cause incorrect error "runtime directory /run/borgmatic overlaps with the configured excludes" #1211

witten referenced this issue

2025-12-24 01:05:34 +00:00

Snapshots cause incorrect error "runtime directory /run/borgmatic overlaps with the configured excludes" #1211

witten commented

2025-12-27 07:00:31 +00:00

Owner

FYI the approach here caused an issue with a different use case, so the precise check needed some updates. I also downgraded the error to a warning. See #1211 for more information.

Rows
Columns

LVM and runtime dir exclusion #1203

What I'm trying to do and why

Steps to reproduce

Actual behavior

Expected behavior

Other notes / implementation ideas

borgmatic version

borgmatic installation method

Borg version

Python version

Database version (if applicable)

Operating system and version