Excludes are ignored since 1.9.6 #971

Closed
opened 2025-01-19 17:43:57 +00:00 by sandr01d · 9 comments

What I'm trying to do and why

I'm using a list of excludes outside the borgmatic config with the exclude_from key.

Steps to reproduce

I've created a simplified environment for reproduction. The file structure in the source directory to be backed up looks like this:

source
├── excluded
│   └── file2.txt
└── included
    └── file1.txt

My config.yaml looks like this

source_directories:
    - /home/user/borgmatic-bug/source/

repositories:
    - path: /home/user/borgmatic-bug/target/
      label: local

exclude_from:
    - /home/user/borgmatic-bug/excludes

compression: none

And the exclude file at /home/user/borgmatic-bug/excludes looks like this:

/home/user/borgmatic-bug/source/excluded/

Actual behavior

When creating an archive with borgmatic 1.9.5, everything works as expected: file1.txt in the included directory is included in the backup, file2.txt in the excluded directory isn't. With borgmatic 1.9.6, both files are included in the backup.

Expected behavior

The same behavior as with borgmatic 1.9.5.

Other notes / implementation ideas

No response

borgmatic version

1.9.6

borgmatic installation method

Arch Linux Package

Borg version

borg 1.4.0

Python version

Python 3.13.1

Database version (if applicable)

No response

Operating system and version

Arch Linux

### What I'm trying to do and why I'm using a list of excludes outside the borgmatic config with the `exclude_from` key. ### Steps to reproduce I've created a simplified environment for reproduction. The file structure in the source directory to be backed up looks like this: ``` source ├── excluded │   └── file2.txt └── included └── file1.txt ``` My config.yaml looks like this ```yaml source_directories: - /home/user/borgmatic-bug/source/ repositories: - path: /home/user/borgmatic-bug/target/ label: local exclude_from: - /home/user/borgmatic-bug/excludes compression: none ``` And the exclude file at /home/user/borgmatic-bug/excludes looks like this: ``` /home/user/borgmatic-bug/source/excluded/ ``` ### Actual behavior When creating an archive with borgmatic 1.9.5, everything works as expected: file1.txt in the included directory is included in the backup, file2.txt in the excluded directory isn't. With borgmatic 1.9.6, both files are included in the backup. ### Expected behavior The same behavior as with borgmatic 1.9.5. ### Other notes / implementation ideas _No response_ ### borgmatic version 1.9.6 ### borgmatic installation method Arch Linux Package ### Borg version borg 1.4.0 ### Python version Python 3.13.1 ### Database version (if applicable) _No response_ ### Operating system and version Arch Linux
Owner

Can I see the full output of your borgmatic create run with --verbosity 2 --files added? Feel free to redact as necessary. I'm most interested in output starting with Writing patterns to ... and the following lines all the way up to and including the logged Borg command. The files listing introduced by --files would be handy too. Thanks.

Can I see the full output of your `borgmatic create` run with `--verbosity 2 --files` added? Feel free to redact as necessary. I'm most interested in output starting with `Writing patterns to ...` and the following lines all the way up to and including the logged Borg command. The files listing introduced by `--files` would be handy too. Thanks.
Author

Thanks for the quick reply @witten!

Can I see the full output of your borgmatic create run with --verbosity 2 --files added?

Sure. Here is the output when creating a backup from the reproduction environment I've mentioned above. Unfortunately I can not create the same from my production environment, as that would try to upload ~1TB of data to borgbase, which would take several hours and exceed my quota.

This is the command I executed:

BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=y borgmatic create --verbosity 2 --files -c ./config.yaml

And here is the output:

/home/user/borgmatic-bug/config.yaml: No commands to run for pre-everything hook
BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_EXIT_CODES=*** borg --version --debug --show-rc
/home/user/borgmatic-bug/config.yaml: Borg 1.4.0
local: Running actions for repository
/home/user/borgmatic-bug/config.yaml: No commands to run for pre-actions hook
/home/user/borgmatic-bug/config.yaml: No commands to run for pre-backup hook
local: Creating archive
local: Using runtime directory /run/user/1000/borgmatic
/home/user/borgmatic-bug/target/: Calling bootstrap hook function remove_data_source_dumps
/home/user/borgmatic-bug/target/: Looking for bootstrap manifest files to remove in /run/user/1000/borgmatic/bootstrap
/home/user/borgmatic-bug/target/: Calling btrfs hook function remove_data_source_dumps
/home/user/borgmatic-bug/target/: Calling lvm hook function remove_data_source_dumps
/home/user/borgmatic-bug/target/: Calling mariadb hook function remove_data_source_dumps
/home/user/borgmatic-bug/target/: Removing MariaDB data source dumps
/home/user/borgmatic-bug/target/: Calling mongodb hook function remove_data_source_dumps
/home/user/borgmatic-bug/target/: Removing MongoDB data source dumps
/home/user/borgmatic-bug/target/: Calling mysql hook function remove_data_source_dumps
/home/user/borgmatic-bug/target/: Removing MySQL data source dumps
/home/user/borgmatic-bug/target/: Calling postgresql hook function remove_data_source_dumps
/home/user/borgmatic-bug/target/: Removing PostgreSQL data source dumps
/home/user/borgmatic-bug/target/: Calling sqlite hook function remove_data_source_dumps
/home/user/borgmatic-bug/target/: Removing SQLite data source dumps
/home/user/borgmatic-bug/target/: Calling zfs hook function remove_data_source_dumps
/home/user/borgmatic-bug/target/: Calling bootstrap hook function dump_data_sources
/home/user/borgmatic-bug/target/: Writing patterns to /run/user/1000/borgmatic/tmpzbzcq2om:
R /home/user/borgmatic-bug/source/
R /home/user/borgmatic-bug/config.yaml
R /run/user/1000/./borgmatic/bootstrap
BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_EXIT_CODES=*** borg create --patterns-from /run/user/1000/borgmatic/tmpzbzcq2om --compression none --list --filter AMEx- --debug --show-rc /home/user/borgmatic-bug/target/::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}
using builtin fallback logging configuration
33 self tests completed in 0.05 seconds
Verified integrity of /home/user/borgmatic-bug/target/index.1
TAM-verified manifest
security: read previous location '/home/user/borgmatic-bug/target'
security: read manifest timestamp '2025-01-19T18:02:32.291484'
security: determined newest manifest timestamp as 2025-01-19T18:02:32.291484
security: repository checks ok, allowing access
Creating archive at "/home/user/borgmatic-bug/target/::hostname-2025-01-19T19:04:23.066190"
Verified integrity of /home/user/.cache/borg/ce2c64bf392d726daba76cd70babf5bc111c45074257398923ed4d54deff054a/chunks
Reading files cache ...
security: read previous location '/home/user/borgmatic-bug/target'
security: read manifest timestamp '2025-01-19T18:02:32.291484'
security: determined newest manifest timestamp as 2025-01-19T18:02:32.291484
security: repository checks ok, allowing access
Processing files ...
A /home/user/borgmatic-bug/source/included/file1.txt
A /home/user/borgmatic-bug/source/excluded/file2.txt
A /home/user/borgmatic-bug/config.yaml
Cleaned up 0 uncommitted segment files (== everything after segment 1).
Verified integrity of /home/user/borgmatic-bug/target/hints.1
A /run/user/1000/borgmatic/bootstrap/manifest.json
check_free_space: few segments, not requiring a full free segment
check_free_space: calculated working space for compact as 24 bytes
check_free_space: required bytes 169232, free bytes 127013179392
security: saving state for ce2c64bf392d726daba76cd70babf5bc111c45074257398923ed4d54deff054a to /home/user/.config/borg/security/ce2c64bf392d726daba76cd70babf5bc111c45074257398923ed4d54deff054a
security: current location   /home/user/borgmatic-bug/target
security: key type           2
security: manifest timestamp 2025-01-19T18:04:23.130488
terminating with success status, rc 0
/home/user/borgmatic-bug/config.yaml: Calling bootstrap hook function remove_data_source_dumps
/home/user/borgmatic-bug/config.yaml: Looking for bootstrap manifest files to remove in /run/user/1000/borgmatic/bootstrap
/home/user/borgmatic-bug/config.yaml: Removing bootstrap manifest at /run/user/1000/borgmatic/bootstrap/manifest.json
/home/user/borgmatic-bug/config.yaml: Calling btrfs hook function remove_data_source_dumps
/home/user/borgmatic-bug/config.yaml: Calling lvm hook function remove_data_source_dumps
/home/user/borgmatic-bug/config.yaml: Calling mariadb hook function remove_data_source_dumps
/home/user/borgmatic-bug/config.yaml: Removing MariaDB data source dumps
/home/user/borgmatic-bug/config.yaml: Calling mongodb hook function remove_data_source_dumps
/home/user/borgmatic-bug/config.yaml: Removing MongoDB data source dumps
/home/user/borgmatic-bug/config.yaml: Calling mysql hook function remove_data_source_dumps
/home/user/borgmatic-bug/config.yaml: Removing MySQL data source dumps
/home/user/borgmatic-bug/config.yaml: Calling postgresql hook function remove_data_source_dumps
/home/user/borgmatic-bug/config.yaml: Removing PostgreSQL data source dumps
/home/user/borgmatic-bug/config.yaml: Calling sqlite hook function remove_data_source_dumps
/home/user/borgmatic-bug/config.yaml: Removing SQLite data source dumps
/home/user/borgmatic-bug/config.yaml: Calling zfs hook function remove_data_source_dumps
/home/user/borgmatic-bug/config.yaml: No commands to run for post-backup hook
/home/user/borgmatic-bug/config.yaml: No commands to run for post-actions hook
/home/user/borgmatic-bug/config.yaml: No commands to run for post-everything hook

summary:
/home/user/borgmatic-bug/config.yaml: Loading configuration file
/home/user/borgmatic-bug/config.yaml: Successfully ran configuration file
Thanks for the quick reply @witten! > Can I see the full output of your borgmatic create run with --verbosity 2 --files added? Sure. Here is the output when creating a backup from the reproduction environment I've mentioned above. Unfortunately I can not create the same from my production environment, as that would try to upload ~1TB of data to borgbase, which would take several hours and exceed my quota. This is the command I executed: `BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=y borgmatic create --verbosity 2 --files -c ./config.yaml` And here is the output: ``` /home/user/borgmatic-bug/config.yaml: No commands to run for pre-everything hook BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_EXIT_CODES=*** borg --version --debug --show-rc /home/user/borgmatic-bug/config.yaml: Borg 1.4.0 local: Running actions for repository /home/user/borgmatic-bug/config.yaml: No commands to run for pre-actions hook /home/user/borgmatic-bug/config.yaml: No commands to run for pre-backup hook local: Creating archive local: Using runtime directory /run/user/1000/borgmatic /home/user/borgmatic-bug/target/: Calling bootstrap hook function remove_data_source_dumps /home/user/borgmatic-bug/target/: Looking for bootstrap manifest files to remove in /run/user/1000/borgmatic/bootstrap /home/user/borgmatic-bug/target/: Calling btrfs hook function remove_data_source_dumps /home/user/borgmatic-bug/target/: Calling lvm hook function remove_data_source_dumps /home/user/borgmatic-bug/target/: Calling mariadb hook function remove_data_source_dumps /home/user/borgmatic-bug/target/: Removing MariaDB data source dumps /home/user/borgmatic-bug/target/: Calling mongodb hook function remove_data_source_dumps /home/user/borgmatic-bug/target/: Removing MongoDB data source dumps /home/user/borgmatic-bug/target/: Calling mysql hook function remove_data_source_dumps /home/user/borgmatic-bug/target/: Removing MySQL data source dumps /home/user/borgmatic-bug/target/: Calling postgresql hook function remove_data_source_dumps /home/user/borgmatic-bug/target/: Removing PostgreSQL data source dumps /home/user/borgmatic-bug/target/: Calling sqlite hook function remove_data_source_dumps /home/user/borgmatic-bug/target/: Removing SQLite data source dumps /home/user/borgmatic-bug/target/: Calling zfs hook function remove_data_source_dumps /home/user/borgmatic-bug/target/: Calling bootstrap hook function dump_data_sources /home/user/borgmatic-bug/target/: Writing patterns to /run/user/1000/borgmatic/tmpzbzcq2om: R /home/user/borgmatic-bug/source/ R /home/user/borgmatic-bug/config.yaml R /run/user/1000/./borgmatic/bootstrap BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_EXIT_CODES=*** borg create --patterns-from /run/user/1000/borgmatic/tmpzbzcq2om --compression none --list --filter AMEx- --debug --show-rc /home/user/borgmatic-bug/target/::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} using builtin fallback logging configuration 33 self tests completed in 0.05 seconds Verified integrity of /home/user/borgmatic-bug/target/index.1 TAM-verified manifest security: read previous location '/home/user/borgmatic-bug/target' security: read manifest timestamp '2025-01-19T18:02:32.291484' security: determined newest manifest timestamp as 2025-01-19T18:02:32.291484 security: repository checks ok, allowing access Creating archive at "/home/user/borgmatic-bug/target/::hostname-2025-01-19T19:04:23.066190" Verified integrity of /home/user/.cache/borg/ce2c64bf392d726daba76cd70babf5bc111c45074257398923ed4d54deff054a/chunks Reading files cache ... security: read previous location '/home/user/borgmatic-bug/target' security: read manifest timestamp '2025-01-19T18:02:32.291484' security: determined newest manifest timestamp as 2025-01-19T18:02:32.291484 security: repository checks ok, allowing access Processing files ... A /home/user/borgmatic-bug/source/included/file1.txt A /home/user/borgmatic-bug/source/excluded/file2.txt A /home/user/borgmatic-bug/config.yaml Cleaned up 0 uncommitted segment files (== everything after segment 1). Verified integrity of /home/user/borgmatic-bug/target/hints.1 A /run/user/1000/borgmatic/bootstrap/manifest.json check_free_space: few segments, not requiring a full free segment check_free_space: calculated working space for compact as 24 bytes check_free_space: required bytes 169232, free bytes 127013179392 security: saving state for ce2c64bf392d726daba76cd70babf5bc111c45074257398923ed4d54deff054a to /home/user/.config/borg/security/ce2c64bf392d726daba76cd70babf5bc111c45074257398923ed4d54deff054a security: current location /home/user/borgmatic-bug/target security: key type 2 security: manifest timestamp 2025-01-19T18:04:23.130488 terminating with success status, rc 0 /home/user/borgmatic-bug/config.yaml: Calling bootstrap hook function remove_data_source_dumps /home/user/borgmatic-bug/config.yaml: Looking for bootstrap manifest files to remove in /run/user/1000/borgmatic/bootstrap /home/user/borgmatic-bug/config.yaml: Removing bootstrap manifest at /run/user/1000/borgmatic/bootstrap/manifest.json /home/user/borgmatic-bug/config.yaml: Calling btrfs hook function remove_data_source_dumps /home/user/borgmatic-bug/config.yaml: Calling lvm hook function remove_data_source_dumps /home/user/borgmatic-bug/config.yaml: Calling mariadb hook function remove_data_source_dumps /home/user/borgmatic-bug/config.yaml: Removing MariaDB data source dumps /home/user/borgmatic-bug/config.yaml: Calling mongodb hook function remove_data_source_dumps /home/user/borgmatic-bug/config.yaml: Removing MongoDB data source dumps /home/user/borgmatic-bug/config.yaml: Calling mysql hook function remove_data_source_dumps /home/user/borgmatic-bug/config.yaml: Removing MySQL data source dumps /home/user/borgmatic-bug/config.yaml: Calling postgresql hook function remove_data_source_dumps /home/user/borgmatic-bug/config.yaml: Removing PostgreSQL data source dumps /home/user/borgmatic-bug/config.yaml: Calling sqlite hook function remove_data_source_dumps /home/user/borgmatic-bug/config.yaml: Removing SQLite data source dumps /home/user/borgmatic-bug/config.yaml: Calling zfs hook function remove_data_source_dumps /home/user/borgmatic-bug/config.yaml: No commands to run for post-backup hook /home/user/borgmatic-bug/config.yaml: No commands to run for post-actions hook /home/user/borgmatic-bug/config.yaml: No commands to run for post-everything hook summary: /home/user/borgmatic-bug/config.yaml: Loading configuration file /home/user/borgmatic-bug/config.yaml: Successfully ran configuration file ```
Owner

Really odd. It does appear that your excludes are not making their way into the patterns passed to Borg. Can I get a copy of your repro config file as well, so I can try to get a repro here as well? Feel free to either attach the file or paste its contents, whatever's easier. Thank you!

Really odd. It does appear that your excludes are not making their way into the patterns passed to Borg. Can I get a copy of your repro config file as well, so I can try to get a repro here as well? Feel free to either attach the file or paste its contents, whatever's easier. Thank you!
Owner

Ugh, I think I found the problem. I'm embarassed to say it's a typo in both the code and the tests.. It's looking for excludes_from rather than exclude_from, which is what the option is actually called in config. Therefore the excludes aren't getting consumed at all. Unfortunately you won't be able to rename your config option, as only exclude_from is valid in config.

(I guess I don't need to see your config after all!)

As a temporary workaround, you could inline your excludes and use exclude_patterns instead of exclude_from. Or you could just use borgmatic 1.9.5 until this is fixed. I might do a quick release on this though since it's such an egregious regression.

Ugh, I think I found the problem. I'm embarassed to say it's a typo in both the code and the tests.. It's looking for `excludes_from` rather than `exclude_from`, which is what the option is actually called in config. Therefore the excludes aren't getting consumed at all. Unfortunately you won't be able to rename your config option, as only `exclude_from` is valid in config. (I guess I don't need to see your config after all!) As a temporary workaround, you could inline your excludes and use `exclude_patterns` instead of `exclude_from`. Or you could just use borgmatic 1.9.5 until this is fixed. I might do a quick release on this though since it's such an egregious regression.
witten added the bug label 2025-01-19 18:22:21 +00:00
Author

Sure, here is the config file

source_directories:
    - /home/user/borgmatic-bug/source/

repositories:
    - path: /home/user/borgmatic-bug/target/
      label: local

exclude_from:
    - /home/user/borgmatic-bug/excludes

compression: none
Sure, here is the config file ```yaml source_directories: - /home/user/borgmatic-bug/source/ repositories: - path: /home/user/borgmatic-bug/target/ label: local exclude_from: - /home/user/borgmatic-bug/excludes compression: none ```
Author

Ugh, I think I found the problem. I'm embarassed to say it's a typo in both the code and the tests.. It's looking for excludes_from rather than exclude_from, which is what the option is actually called in config. Therefore the excludes aren't getting consumed at all. Unfortunately you won't be able to rename your config option, as only exclude_from is valid in config.

(I guess I don't need to see your config after all!)

As a temporary workaround, you could inline your excludes and use exclude_patterns instead of exclude_from. Or you could just use borgmatic 1.9.5 until this is fixed. I might do a quick release on this though since it's such an egregious regression.

Well, glad to hear you found it 🙂 I'll stick with borgmatic 1.9.5 for now, thank you!

> Ugh, I think I found the problem. I'm embarassed to say it's a typo in both the code and the tests.. It's looking for `excludes_from` rather than `exclude_from`, which is what the option is actually called in config. Therefore the excludes aren't getting consumed at all. Unfortunately you won't be able to rename your config option, as only `exclude_from` is valid in config. > > (I guess I don't need to see your config after all!) > > As a temporary workaround, you could inline your excludes and use `exclude_patterns` instead of `exclude_from`. Or you could just use borgmatic 1.9.5 until this is fixed. I might do a quick release on this though since it's such an egregious regression. Well, glad to hear you found it 🙂 I'll stick with borgmatic 1.9.5 for now, thank you!
Owner

Oh, I didn't realize that was the entire config. Thanks! In any case, I think I have a fix in hand.

Oh, I didn't realize that was the entire config. Thanks! In any case, I think I have a fix in hand.
Owner

This is fixed in main and will be part of the next release. Thanks!

This is fixed in main and will be part of the next release. Thanks!
Owner

Released in borgmatic 1.9.7!

Released in borgmatic 1.9.7!
Sign in to join this conversation.
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#971