(Exclude) patterns having no effect at all (Borgmatic 2.0.10)

sm8ps commented

2025-10-25 10:19:55 +00:00

What I'm trying to do and why

I want to exclude parts of a Btrfs-volume from the backup and my trouble-shooting efforts have lead me back to base zero where I am trying to exclude everything.

Steps to reproduce

In the config file, I have source_directories, repositories and

exclude_caches: true
exclude_if_present: - .nobackup
keep_exclude_tags: true
patterns:
  - '- **'
  - '- **/*'
  - '- **/.*'

Actual behavior

Each and every file and folder of the volume is backed up.
I have run Borgmatic with the flags -v2 --list and attached the output (reduced by the bulk of content files). Previous runs that got interrupted left stale snapshots that are deleted first. However, there is no difference in behavior even after a cleanly terminated run.

Expected behavior

No file or folder should be backed up.

Other notes / implementation ideas

Originally I wanted to split the backup of a LXC-container into two parts and have the data folders backed-up separately.

borgmatic version

2.0.10

borgmatic installation method

pipx

Borg version

borg 1.4.1

Python version

Python 3.12.3

Database version (if applicable)

No response

Operating system and version

Ubuntu 24.04.3 LTS

### What I'm trying to do and why I want to exclude parts of a Btrfs-volume from the backup and my trouble-shooting efforts have lead me back to base zero where I am trying to _exclude everything_. ### Steps to reproduce In the config file, I have source_directories, repositories and ``` exclude_caches: true exclude_if_present: - .nobackup keep_exclude_tags: true patterns: - '- **' - '- **/*' - '- **/.*' ``` ### Actual behavior Each and every file and folder of the volume is backed up. I have run Borgmatic with the flags `-v2 --list` and attached the output (reduced by the bulk of content files). Previous runs that got interrupted left stale snapshots that are deleted first. However, there is no difference in behavior even after a cleanly terminated run. ### Expected behavior No file or folder should be backed up. ### Other notes / implementation ideas Originally I wanted to split the backup of a LXC-container into two parts and have the data folders backed-up separately. ### borgmatic version 2.0.10 ### borgmatic installation method pipx ### Borg version borg 1.4.1 ### Python version Python 3.12.3 ### Database version (if applicable) _No response_ ### Operating system and version Ubuntu 24.04.3 LTS

Borgmatic.1.log

8.1 KiB

witten commented

2025-10-25 17:22:27 +00:00

Owner

I have a pretty good idea what's going on here. To address #1150 and specifically a user's global exclude inadvertently excluding the borgmatic runtime directory and Btrfs snapshot paths (among other things), borgmatic now more aggressively "punches through" such excludes by doing an explicit include of the Btrfs snapshot path first.

The problem with that strategy, as you've apparently discovered, is that it also overrides any intentional user excludes of portions of the snapshot path.

So can I get an example of the non-global partial snapshot path exclude you were doing initially (redacted as necessary) + corresponding debug logs? At this point, I have no idea what a solution might be that doesn't break the inadvertent global exclude case, but I can noodle on it and see what I can come up with.

I have a pretty good idea what's going on here. To address #1150 and specifically a user's global exclude inadvertently excluding the borgmatic runtime directory and Btrfs snapshot paths (among other things), borgmatic now more aggressively "punches through" such excludes by doing an explicit include of the Btrfs snapshot path *first*. The problem with that strategy, as you've apparently discovered, is that it also overrides any intentional user excludes of portions of the snapshot path. So can I get an example of the non-global partial snapshot path exclude you were doing initially (redacted as necessary) + corresponding debug logs? At this point, I have no idea what a solution might be that doesn't break the inadvertent global exclude case, but I can noodle on it and see what I can come up with.

witten referenced this issue

2025-10-25 17:23:48 +00:00

[Borgmatic 2.0.8] The runtime directory overlaps with the configured excludes or patterns with excludes. #1150

sm8ps commented

2025-10-25 17:34:58 +00:00

Author

Many thanks, @witten, for your quick reply which makes perfect sense. Would you be OK with me sending you the config and (abbreviated) log files via private message? It is probably best to see it all unredacted but I do not want to make them publicly available.

witten commented

2025-10-25 17:52:30 +00:00

Owner

Yup, that's totally fine. You can email me or if you prefer message me (witten) on IRC.

FYI I do have a repro here, but I want to make sure it's a representative repro of your use case.

Yup, that's totally fine. You can [email me](https://torsion.org/borgmatic/#other) or if you prefer message me (witten) [on IRC](https://torsion.org/borgmatic/#chat). FYI I do have a repro here, but I want to make sure it's a representative repro of your use case.

witten referenced this issue from a commit

2025-10-26 21:13:09 +00:00

Fix for a regression in the ZFS, LVM, and Btrfs hooks in which partial excludes of snapshot paths were ignored (#1169).

witten commented

2025-10-26 21:15:59 +00:00

Owner

Okay, I've pushed a fix to main, which will be part of the next release. The solution I went with was making sure that any snapshot include patterns that borgmatic inserts get inserted into the user's patterns after any partial exclude patterns for the snapshot path (but still before any global exclude patterns).

Thanks for bringing this to my attention!

Okay, I've pushed a fix to main, which will be part of the next release. The solution I went with was making sure that any snapshot include patterns that borgmatic inserts get inserted into the user's patterns *after* any partial exclude patterns for the snapshot path (but still before any global exclude patterns). Thanks for bringing this to my attention!

witten

2025-10-26 21:15:59 +00:00

closed this issue
added the
bug
label

sm8ps commented

2025-10-27 07:25:03 +00:00

Author

Great, thanks a lot for taking care of this issue! I am awaiting the upcoming release to test out the fix.

Not worth a ticket of its own because it is mostly a wish: could you please include (: some explanation about the role of the recursion root for patterns in the configuration documentation? AFAICT that is the only place where R appears and it is unclear (to me at least) what its role is even after reading through the output of "borg help patterns" and related documentation by Borg.

Many thanks for providing this valuable application!

Great, thanks a lot for taking care of this issue! I am awaiting the upcoming release to test out the fix. Not worth a ticket of its own because it is mostly a wish: could you please include (: some explanation about the role of the recursion root for patterns in the [configuration documentation](https://torsion.org/borgmatic/reference/configuration/)? AFAICT that is the only place where `R` appears and it is unclear (to me at least) what its role is even after reading through the output of "borg help patterns" and related documentation by Borg. Many thanks for providing this valuable application!

witten commented

2025-10-27 21:37:28 +00:00

Owner

I took a stab at that here:

            Any paths matching these patterns are included/excluded from
            backups. Recursion root patterns ("R ...") are effectively the same
            as "source_directories"; they tell Borg which paths to backup
            (modulo any excludes). Globs are expanded. (Tildes are not.) See
            the output of "borg help patterns" for more details. Quote any value
            if it contains leading punctuation, so it parses correctly.

Let me know if you have any feedback on it.

I took a stab at that here: ``` Any paths matching these patterns are included/excluded from backups. Recursion root patterns ("R ...") are effectively the same as "source_directories"; they tell Borg which paths to backup (modulo any excludes). Globs are expanded. (Tildes are not.) See the output of "borg help patterns" for more details. Quote any value if it contains leading punctuation, so it parses correctly. ``` Let me know if you have any feedback on it.

sm8ps commented

2025-10-28 16:14:35 +00:00

Author

(This is a copy of my direct response via e-mail to the notification. Sorry for that out-of-band communication! Maybe the link below can be helpful to others searching for further explanations.)

Aaaha, now I got it! Thanks for explaining. The take-away for me is that I do not have to worry about it with Borgmatic because it is already set, right?

Would it make sense to add this statement? In the current state with the example following below it looks like one would need to set the recursion root anyways.

The real problem was that the output of borg help patterns was not very clear to me. With a bit of research, however, one finds an instructional explanation by Thomas Waldmann. So in combination this should suffice for getting anybody started well.

Thanks for your consideration.

(This is a copy of my direct response via e-mail to the notification. Sorry for that out-of-band communication! Maybe the link below can be helpful to others searching for further explanations.) Aaaha, now I got it! Thanks for explaining. The take-away for me is that I do not have to worry about it with Borgmatic because it is already set, right? Would it make sense to add this statement? In the current state with the example following below it looks like one would need to set the recursion root anyways. The real problem was that the output of `borg help patterns` was not very clear to me. With a bit of research, however, one finds an [instructional explanation by Thomas Waldmann](https://github.com/borgbackup/borg/discussions/7660#discussioncomment-6232962). So in combination this should suffice for getting anybody started well. Thanks for your consideration.

witten commented

2025-10-28 18:31:27 +00:00

Owner

(This is a copy of my direct response via e-mail to the notification. Sorry for that out-of-band communication! Maybe the link below can be helpful to others searching for further explanations.)

FYI I don't think incoming email processing is currently enabled here. Maybe it should be...

Aaaha, now I got it! Thanks for explaining. The take-away for me is that I do not have to worry about it with Borgmatic because it is already set, right?

Depends. Do you have the "roots" set in source_directories? If so, then yes, they are already set. But if you omit them from source_directories, then they'll need to be listed in patterns as recursion roots.

Would it make sense to add this statement? In the current state with the example following below it looks like one would need to set the recursion root anyways.

Which statement? The bit about the recursion roots already being set? I'm happy to clarify the docs.. I just want to clarify what exactly I'd be clarifying. 😄

The real problem was that the output of borg help patterns was not very clear to me. With a bit of research, however, one finds an instructional explanation by Thomas Waldmann. So in combination this should suffice for getting anybody started well.

Yeah, that explanation is helpful. Maybe borgmatic could benefit from its own reference documentation page on patterns.

> (This is a copy of my direct response via e-mail to the notification. Sorry for that out-of-band communication! Maybe the link below can be helpful to others searching for further explanations.) FYI I don't think [incoming email processing](https://docs.gitea.com/usage/incoming-email) is currently enabled here. Maybe it should be... > Aaaha, now I got it! Thanks for explaining. The take-away for me is that I do not have to worry about it with Borgmatic because it is already set, right? Depends. Do you have the "roots" set in `source_directories`? If so, then yes, they are already set. But if you omit them from `source_directories`, then they'll need to be listed in `patterns` as recursion roots. > Would it make sense to add this statement? In the current state with the example following below it looks like one would need to set the recursion root anyways. Which statement? The bit about the recursion roots already being set? I'm happy to clarify the docs.. I just want to clarify what exactly I'd be clarifying. 😄 > The real problem was that the output of `borg help patterns` was not very clear to me. With a bit of research, however, one finds an instructional explanation by Thomas Waldmann. So in combination this should suffice for getting anybody started well. Yeah, that explanation is helpful. Maybe borgmatic could benefit from its own reference documentation page on patterns.

sm8ps commented

2025-10-28 21:07:32 +00:00

Author

Aaaha, now I got it! Thanks for explaining. The take-away for me is that I do not have to worry about it with Borgmatic because it is already set, right?

Depends. Do you have the "roots" set in source_directories? If so, then yes, they are already set. But if you omit them from source_directories, then they'll need to be listed in patterns as recursion roots.

Would it make sense to add this statement? In the current state with the example following below it looks like one would need to set the recursion root anyways.

Which statement? The bit about the recursion roots already being set? I'm happy to clarify the docs.. I just want to clarify what exactly I'd be clarifying. 😄

Err, I was speaking of the above statement that the recursion root does not need to be set explicitly -- which turns out to be false as you say. 😕

Together with the explanation by Thomas Waldmann I believe I am starting to understand how things should work. I shall wait for the next release and tinker with the exclude patterns. Once I get my head around all these things, I will be able to formulate more clearly what would have helped me now. 😀

> > Aaaha, now I got it! Thanks for explaining. The take-away for me is that I do not have to worry about it with Borgmatic because it is already set, right? > > Depends. Do you have the "roots" set in `source_directories`? If so, then yes, they are already set. But if you omit them from `source_directories`, then they'll need to be listed in `patterns` as recursion roots. > > > Would it make sense to add this statement? In the current state with the example following below it looks like one would need to set the recursion root anyways. > > Which statement? The bit about the recursion roots already being set? I'm happy to clarify the docs.. I just want to clarify what exactly I'd be clarifying. 😄 Err, I was speaking of the above statement that the recursion root does not need to be set explicitly -- which turns out to be false as you say. 😕 Together with the explanation by Thomas Waldmann I believe I am starting to understand how things should work. I shall wait for the next release and tinker with the exclude patterns. Once I get my head around all these things, I will be able to formulate more clearly what would have helped me now. 😀

witten commented

2025-10-28 21:11:51 +00:00

Owner

Okay, please feel free to file subsequent tickets for documentation or other improvements.. The only way I find out about things that are unclear are when I hear about them from users like you!

Rows
Columns

(Exclude) patterns having no effect at all (Borgmatic 2.0.10) #1169

What I'm trying to do and why

Steps to reproduce

Actual behavior

Expected behavior

Other notes / implementation ideas

borgmatic version

borgmatic installation method

Borg version

Python version

Database version (if applicable)

Operating system and version