borgmatic-collective/borgmatic
9
73
Fork 51
Code Issues 52 Pull Requests 3 Actions 1 Packages Releases 158 Activity

exclude_patterns in config file also seem to be ignored since 1.9.6 #972

New Issue
Closed
opened 2025-01-20 22:28:39 +00:00 by iljur · 9 comments
iljur commented 2025-01-20 22:28:39 +00:00
Copy Link

What I'm trying to do and why

I have quite common exclude_patterns in my config file configured.

Since 1.9.6 borgmatic seems to look at least in the /proc/ directory trying to process some things there and throwing errors exiting with warninig status rc1. This seems similar to #970 and #971 but not the same so maybe not fixed yet.

1.9.5 works without warnings.

Steps to reproduce

source_directories:
- /

exclude_patterns:
- /dev/
- /proc/
- /run/
- /sys/
- /tmp/

...

Actual behavior

...
Writing patterns to /tmp/borgmatic-h6_n3gib/borgmatic/tmp1ijmm4do:
R /
- fm:/dev/
- fm:/proc/
- fm:/run/
- fm:/sys/
- fm:/tmp/

...
Processing files ...
/proc/3213786/task/3213786/fd/9: stat: [Errno 2] No such file or directory: '9'
/proc/3213786/task/3213786/fdinfo/9: stat: [Errno 2] No such file or directory: '9'
/proc/3213786/fd/7: stat: [Errno 2] No such file or directory: '7'
/proc/3213786/fdinfo/7: stat: [Errno 2] No such file or directory: '7'
/proc/3213831: stat: [Errno 2] No such file or directory: '3213831'
/proc/3213904: stat: [Errno 2] No such file or directory: '3213904'
terminating with warning status, rc 1
...

Expected behavior

The same behavior as with borgmatic 1.9.5.

Other notes / implementation ideas

No response

borgmatic version

1.9.6

borgmatic installation method

pipx

Borg version

No response

Python version

No response

Database version (if applicable)

No response

Operating system and version

No response

### What I'm trying to do and why I have quite common exclude_patterns in my config file configured. Since 1.9.6 borgmatic seems to look at least in the /proc/ directory trying to process some things there and throwing errors exiting with warninig status rc1. This seems similar to #970 and #971 but not the same so maybe not fixed yet. 1.9.5 works without warnings. ### Steps to reproduce source_directories: - / exclude_patterns: - /dev/ - /proc/ - /run/ - /sys/ - /tmp/ ... ### Actual behavior ``` ... Writing patterns to /tmp/borgmatic-h6_n3gib/borgmatic/tmp1ijmm4do: R / - fm:/dev/ - fm:/proc/ - fm:/run/ - fm:/sys/ - fm:/tmp/ ... Processing files ... /proc/3213786/task/3213786/fd/9: stat: [Errno 2] No such file or directory: '9' /proc/3213786/task/3213786/fdinfo/9: stat: [Errno 2] No such file or directory: '9' /proc/3213786/fd/7: stat: [Errno 2] No such file or directory: '7' /proc/3213786/fdinfo/7: stat: [Errno 2] No such file or directory: '7' /proc/3213831: stat: [Errno 2] No such file or directory: '3213831' /proc/3213904: stat: [Errno 2] No such file or directory: '3213904' terminating with warning status, rc 1 ... ``` ### Expected behavior The same behavior as with borgmatic 1.9.5. ### Other notes / implementation ideas _No response_ ### borgmatic version 1.9.6 ### borgmatic installation method pipx ### Borg version _No response_ ### Python version _No response_ ### Database version (if applicable) _No response_ ### Operating system and version _No response_
witten commented 2025-01-20 22:51:20 +00:00
Owner
Copy Link

Super odd! I can't seem to reproduce this here, even with the same source_directories and exclude_patterns. In borgmatic's verbose output, does the Borg command that's run (logged right after those patterns) include --patterns-from along with that /tmp patterns file path? Is there anything else in your configuration that might be relevant here? working_directory, exclude_from, patterns, patterns_from, etc? It might help to see your whole config file (redacted as needed).

Super odd! I can't seem to reproduce this here, even with the same `source_directories` and `exclude_patterns`. In borgmatic's verbose output, does the Borg command that's run (logged right after those patterns) include `--patterns-from` along with that `/tmp` patterns file path? Is there anything else in your configuration that might be relevant here? `working_directory`, `exclude_from`, `patterns`, `patterns_from`, etc? It might help to see your whole config file (redacted as needed).
witten commented 2025-01-20 22:54:38 +00:00
Owner
Copy Link

Another thing you can do is include --files in your borgmatic create command to see what specific source files are getting archived. Additionally, if you don't want to do a whole backup, you can include --dry-run to simulate one. (/proc paths should be preceded by x in the --files output to indicate they're excluded.)

Another thing you can do is include `--files` in your `borgmatic create` command to see what specific source files are getting archived. Additionally, if you don't want to do a whole backup, you can include `--dry-run` to simulate one. (`/proc` paths should be preceded by `x ` in the `--files` output to indicate they're excluded.)
iljur commented 2025-01-21 10:31:39 +00:00
Author
Copy Link

Thanks for the quick reply!

My setup has actually 2 different ssh repos as targets. A common config file (including the exclude patterns) is imported into each repo config file on execution as described in the reference. Works just fine with 1.9.5. I attached the config files.

In borgmatic's verbose output, does the Borg command that's run (logged right after those patterns) include --patterns-from along with that /tmp patterns file path?

Yes, it does:

BORG_PASSPHRASE=*** BORG_RSH=*** BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_EXIT_CODES=*** borg create --patterns-from /tmp/borgmatic-66joah3l/borgmatic/tmpihus5d0w --exclude-caches --compression auto,zstd,6 --lock-wait 300 --list --filter AMEx- --dry-run --stats --show-rc --debug --show-rc

Example --files output:
...
x /proc/452770/fdinfo/0
x /proc/452770/fdinfo/1
x /proc/452770/fdinfo/2
x /proc/452770/fdinfo/3
x /proc/452770/fdinfo/4
x /proc/452770/fdinfo/5
x /proc/452770/fdinfo/6
x /proc/452770/fdinfo/7
/proc/452770/fdinfo/7: stat: [Errno 2] No such file or directory: '7'
x /proc/452770/ns
...

Thanks for the quick reply! My setup has actually 2 different ssh repos as targets. A common config file (including the exclude patterns) is imported into each repo config file on execution as described in the reference. Works just fine with 1.9.5. I attached the config files. > In borgmatic's verbose output, does the Borg command that's run (logged right after those patterns) include --patterns-from along with that /tmp patterns file path? Yes, it does: > BORG_PASSPHRASE=*** BORG_RSH=*** BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_EXIT_CODES=*** borg create --patterns-from /tmp/borgmatic-66joah3l/borgmatic/tmpihus5d0w --exclude-caches --compression auto,zstd,6 --lock-wait 300 --list --filter AMEx- --dry-run --stats --show-rc --debug --show-rc Example --files output: ... x /proc/452770/fdinfo/0 x /proc/452770/fdinfo/1 x /proc/452770/fdinfo/2 x /proc/452770/fdinfo/3 x /proc/452770/fdinfo/4 x /proc/452770/fdinfo/5 x /proc/452770/fdinfo/6 x /proc/452770/fdinfo/7 /proc/452770/fdinfo/7: stat: [Errno 2] No such file or directory: '7' x /proc/452770/ns ...
storagebox.yaml.txt
408 B
common.yaml.txt
1.3 KiB
DS420.yaml.txt
316 B
maxhamon commented 2025-01-21 16:55:31 +00:00
Copy Link

Same issue here, except i'm using exclude_from with files containing my exclusions, but they seem to be ignored.
If you need more input i can provide you some too

Not related to this issue

~~Same issue here, except i'm using `exclude_from` with files containing my exclusions, but they seem to be ignored. If you need more input i can provide you some too~~ Not related to this issue
witten commented 2025-01-21 17:48:15 +00:00
Owner
Copy Link

@maxhamon I believe the exclude_from issue is separate and fixed in main. See #971.

@iljur Thanks for the additional info. That --files output suggests that Borg is correctly excluding the /proc files! ("x" means excluded.) So that would suggest the excludes are working correctly except, as you suggest, something appears to be "trying to process some things" in the excluded /proc as part of the create action. The fact that those "No such file or directory" errors seem to be interspersed directly within the Borg output makes me think they're coming from Borg. I'll look into this and see if I can repro it with your additional details and/or determine what's going on.

@maxhamon I believe the `exclude_from` issue is separate and fixed in main. See #971. @iljur Thanks for the additional info. That `--files` output suggests that Borg is correctly excluding the `/proc` files! (["x" means excluded.](https://borgbackup.readthedocs.io/en/stable/usage/create.html#item-flags)) So that would suggest the excludes are working correctly except, as you suggest, something appears to be "trying to process some things" in the excluded `/proc` as part of the `create` action. The fact that those "No such file or directory" errors seem to be interspersed directly within the Borg output makes me think they're coming from Borg. I'll look into this and see if I can repro it with your additional details and/or determine what's going on.
maxhamon commented 2025-01-21 17:57:23 +00:00
Copy Link

Sorry for confusion guys, thanks for your great works
!

Sorry for confusion guys, thanks for your great works !
👍 1
witten commented 2025-01-21 18:40:06 +00:00
Owner
Copy Link

I'm not sure why this didn't happen for you with 1.9.5, but this Borg ticket seems to suggest that it's a known behavior of Borg. The dev's recommendation is, for /proc and /run in particular, to use a "no recurse" exclude pattern (!) instead of a standard exclude (-). Maybe try that for /proc and see if the errors/warnings go away?

EDIT: Also be aware of this other Borg issue; you might need to remove the trailing slash on the pattern. In fact, it might be a good idea to removing the trailing slash regardless.

I'm not sure why this didn't happen for you with 1.9.5, but [this Borg ticket](https://github.com/borgbackup/borg/issues/8279) seems to suggest that it's a known behavior of Borg. The dev's recommendation is, for `/proc` and `/run` in particular, to use a "no recurse" exclude pattern (`!`) instead of a standard exclude (`-`). Maybe try that for `/proc` and see if the errors/warnings go away? EDIT: Also be aware of [this other Borg issue](https://github.com/borgbackup/borg/issues/7220); you might need to remove the trailing slash on the pattern. In fact, it might be a good idea to removing the trailing slash regardless.
iljur commented 2025-01-23 20:34:24 +00:00
Author
Copy Link

My guess is that there is somehow a link between this issue and the new temporary pattern file that is generated by borgmatic 1.9.6 although it looks formaly alright. It really seems to be a borg issue though and not borgmatic related.

But anyway I solved my issue/warnings by switching to the following pattern file inspired by @dukelsky in the mentioned borg issue.

+ re:^(dev|proc|run|sys|tmp|media|mnt)$
! re:^(dev|proc|run|sys|tmp|media|mnt)
- var/run/
- var/cache/
- var/lib/docker/
- var/log/journal/
- srv/nc_data/Team/files/IntraNet/06_IT-Kram/
- srv/nc_data/gk/files/Uploads/
- opt/docker/vaultwarden/psql/
- **/tmp/
- lost+found/

Like in the cited borg issue I want to include the directories, but not their contents. So the trailing slashes should be there... With the regex pattern also /proc is handled like expected with borgmatic 1.9.6.

But thanks anyway for the quick reply and your outstanding work on Borgmatic!

My guess is that there is somehow a link between this issue and the new temporary pattern file that is generated by borgmatic 1.9.6 although it looks formaly alright. It really seems to be a borg issue though and not borgmatic related. But anyway I solved my issue/warnings by switching to the following pattern file inspired by @dukelsky in the mentioned [borg issue](https://github.com/borgbackup/borg/issues/8279). ``` + re:^(dev|proc|run|sys|tmp|media|mnt)$ ! re:^(dev|proc|run|sys|tmp|media|mnt) - var/run/ - var/cache/ - var/lib/docker/ - var/log/journal/ - srv/nc_data/Team/files/IntraNet/06_IT-Kram/ - srv/nc_data/gk/files/Uploads/ - opt/docker/vaultwarden/psql/ - **/tmp/ - lost+found/ ``` Like in the cited borg issue I want to include the directories, but not their contents. So the trailing slashes should be there... With the regex pattern also /proc is handled like expected with borgmatic 1.9.6. But thanks anyway for the quick reply and your outstanding work on Borgmatic!
witten commented 2025-01-23 21:44:47 +00:00
Owner
Copy Link

I'm glad to hear you have a work-around for now.. Hopefully there are no more surprises stemming from the 1.9.6 pattern changes! And thanks for using borgmatic and filing tickets!

I'm glad to hear you have a work-around for now.. Hopefully there are no more surprises stemming from the 1.9.6 pattern changes! And thanks for using borgmatic and filing tickets!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
blocked
breaking
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
No Label
Milestone
No items
No Milestone
Assignees
Clear assignees
witten
No Assignees
3 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#972
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?