Exclusions not preventing recursion into directories

maxhamon commented

2025-01-27 17:19:56 +00:00

Contributor

What I'm trying to do and why

I’m encountering an issue (or a misunderstanding) while trying to back up a folder in my home directory with some exclusions. My setup includes a directory that contains many subfolders and files, which I’ve added to my exclusions. However, it seems that Borgmatic is recursing into the directory, and the logs indicate that all files within are being tagged as excluded. I believe this is unnecessary, as the entire folder is meant to be excluded entirely.

This behavior is problematic because it results in unnecessary traversal of files and folders that I intend to exclude completely. I’d like to confirm whether this is expected behavior or a misconfiguration on my end.

Steps to reproduce

Here some testing configurations i made :

source_directories:
  - '/home/vagrant/test'
exclude_patterns:
  - '/home/*/donotbackup1'
  - '/home/*/donotbackup2'

repositories:
  - path: 'ssh://xxxxxxxxxx/./repo'
encryption_passphrase: xxxx

archive_name_format: '{fqdn}-{now}'
skip_actions:
  - prune
  - check

➜ tree -a /home/vagrant/test 
/home/vagrant/test
├── backup1
│   ├── file1
│   └── file2
├── backup2
│   ├── file1
│   └── file2
├── donotbackup1
│   ├── file1
│   └── file2
└── donotbackup2
    ├── file1
    └── file2

Actual behavior

Processing files ...
x /home/vagrant/test/donotbackup1/file1
x /home/vagrant/test/donotbackup1/file2
x /home/vagrant/test/donotbackup2/file1
x /home/vagrant/test/donotbackup2/file2
A /run/user/900/borgmatic/bootstrap/manifest.json

Attached full log

Expected behavior

I don't want each file to be read; I want the parent folder to be excluded.

Processing files ...
x /home/vagrant/test/donotbackup1
x /home/vagrant/test/donotbackup2
A /run/user/900/borgmatic/bootstrap/manifest.json

Other notes / implementation ideas

No response

borgmatic version

1.9.8

borgmatic installation method

pip install

Borg version

1.4.0

Python version

3.11.2

Database version (if applicable)

No response

Operating system and version

Debian 12

### What I'm trying to do and why I’m encountering an issue (or a misunderstanding) while trying to back up a folder in my home directory with some exclusions. My setup includes a directory that contains many subfolders and files, which I’ve added to my exclusions. However, it seems that Borgmatic is recursing into the directory, and the logs indicate that all files within are being tagged as excluded. I believe this is unnecessary, as the entire folder is meant to be excluded entirely. This behavior is problematic because it results in unnecessary traversal of files and folders that I intend to exclude completely. I’d like to confirm whether this is expected behavior or a misconfiguration on my end. ### Steps to reproduce Here some testing configurations i made : ```yaml source_directories: - '/home/vagrant/test' exclude_patterns: - '/home/*/donotbackup1' - '/home/*/donotbackup2' repositories: - path: 'ssh://xxxxxxxxxx/./repo' encryption_passphrase: xxxx archive_name_format: '{fqdn}-{now}' skip_actions: - prune - check ``` ```bash ➜ tree -a /home/vagrant/test /home/vagrant/test ├── backup1 │ ├── file1 │ └── file2 ├── backup2 │ ├── file1 │ └── file2 ├── donotbackup1 │ ├── file1 │ └── file2 └── donotbackup2 ├── file1 └── file2 ``` ### Actual behavior ``` Processing files ... x /home/vagrant/test/donotbackup1/file1 x /home/vagrant/test/donotbackup1/file2 x /home/vagrant/test/donotbackup2/file1 x /home/vagrant/test/donotbackup2/file2 A /run/user/900/borgmatic/bootstrap/manifest.json ``` Attached full log ### Expected behavior I don't want each file to be read; I want the parent folder to be excluded. ``` Processing files ... x /home/vagrant/test/donotbackup1 x /home/vagrant/test/donotbackup2 A /run/user/900/borgmatic/bootstrap/manifest.json ``` ### Other notes / implementation ideas _No response_ ### borgmatic version 1.9.8 ### borgmatic installation method pip install ### Borg version 1.4.0 ### Python version 3.11.2 ### Database version (if applicable) _No response_ ### Operating system and version Debian 12

borgmatic_20250127.log

10 KiB

maxhamon changed title from ~~Exclusions not preventing eecursion into directories~~ to Exclusions not preventing recursion into directories

2025-01-27 17:24:40 +00:00

witten commented

2025-01-27 17:39:45 +00:00

Owner

Thanks for filing this. I believe this is actually a regression since borgmatic 1.9.6, but fortunately one that should be easy to fix. In borgmatic 1.9.6, exclude patterns were changed under the hood to be treated as generic Borg patterns. This unification makes it much easier for borgmatic to manipulate paths internally, which is needed for things like database and filesystem hooks.

However, apparently the change meant that exclude patterns started recursing into subdirectories as you're seeing—which is the usual behavior for generic Borg patterns because it allows the user to include subdirectories within excluded parent directories. A fix for this would be for borgmatic to treat configured excluded patterns as "no recurse" subdirectories, which is a supported pattern type with Borg. I'll look into implementing that now!

Thanks for filing this. I believe this is actually a regression since borgmatic 1.9.6, but fortunately one that should be easy to fix. In borgmatic 1.9.6, exclude patterns were changed under the hood to be treated as [generic Borg patterns](https://borgbackup.readthedocs.io/en/stable/usage/help.html#borg-help-patterns). This unification makes it much easier for borgmatic to manipulate paths internally, which is needed for things like database and filesystem hooks. However, apparently the change meant that exclude patterns started recursing into subdirectories as you're seeing—which is the usual behavior for generic Borg patterns because it allows the user to *include* subdirectories within excluded parent directories. A fix for this would be for borgmatic to treat configured excluded patterns as "no recurse" subdirectories, which is a supported pattern type with Borg. I'll look into implementing that now!

maxhamon commented

2025-01-27 17:51:30 +00:00

Author

Contributor

Yes everything works fine with 1.9.5, i'll wait for the fix before upgrade then. Thanks.

👍 1

witten added the

bug

label

2025-01-27 17:57:24 +00:00

witten referenced this issue from a commit

2025-01-27 18:07:40 +00:00