borgmatic runtime directory error when running spot check with a database hook enabled #965

Closed
opened 2025-01-12 06:27:40 +00:00 by witten · 5 comments
Owner

What I'm trying to do and why

I'm reproing an error mentioned in the comments on #947.

Steps to reproduce

  • Enable a database hook. (This triggers the collect special files code path.)
  • Configure one root pattern. Although this also appears to repro with a source directory instead.
  • Configure spot checks.

Run a spot check, e.g.:

borgmatic -c test.yaml -v 2 check --only spot --force

Actual behavior

...
test: Error running actions for repository
The runtime directory /tmp/borgmatic-5m9_kc54/borgmatic overlaps with the configured excludes or patterns. Please remove it from excludes/patterns or change the runtime directory.
...

Expected behavior

Spot checks run without that error.

Other notes / implementation ideas

When I ran a spot check, what happens is that the spot check calls borgmatic.borg.create.make_base_create_command() so it can collect the paths from the filesystem to spot check. make_base_create_command() calls collect_special_file_paths(), which among other things, checks to see if the borgmatic runtime directory is effectively excluded from the files borg create consumes from the filesystem. If it is, then it errors—to prevent potential Borg hangs when borgmatic writes to a database dump named pipe in the runtime directory and Borg never reads from it because the runtime directory isn't included in the patterns/source_directories given to Borg!

The problem is that when going through the spot check code path, the borgmatic runtime directory never gets injected into the patterns or source directories.. That only happens during the create code path when any given database hook is asked to dump its databases and therefore appends a path containing the borgmatic runtime directory to source_directories.

I'm not sure what the solution is here to this unfortunate confluence of factors. I could remove the check and error message, but that would open us back up to potential Borg hangs. Maybe the cleanest thing would be to only run this code path during create rather than check? After all, there's no risk of these particular hangs during a check by itself.

borgmatic version

1.9.5 or main

borgmatic installation method

pipx

Borg version

1.4.0

Python version

3.12.7

Database version (if applicable)

14.3

Operating system and version

Manjaro stable

### What I'm trying to do and why I'm reproing an error mentioned in the comments on #947. ### Steps to reproduce * Enable a database hook. (This triggers the collect special files code path.) * Configure one root pattern. Although this also appears to repro with a source directory instead. * Configure spot checks. Run a spot check, e.g.: ```bash borgmatic -c test.yaml -v 2 check --only spot --force ``` ### Actual behavior ``` ... test: Error running actions for repository The runtime directory /tmp/borgmatic-5m9_kc54/borgmatic overlaps with the configured excludes or patterns. Please remove it from excludes/patterns or change the runtime directory. ... ``` ### Expected behavior Spot checks run without that error. ### Other notes / implementation ideas When I ran a spot check, what happens is that the spot check calls `borgmatic.borg.create.make_base_create_command()` so it can collect the paths from the filesystem to spot check. `make_base_create_command()` calls `collect_special_file_paths()`, which among other things, checks to see if the borgmatic runtime directory is effectively excluded from the files `borg create` consumes from the filesystem. If it is, then it errors—to prevent potential Borg hangs when borgmatic writes to a database dump named pipe in the runtime directory and Borg never reads from it because the runtime directory isn't included in the `patterns`/`source_directories` given to Borg! The problem is that when going through the spot check code path, the borgmatic runtime directory never gets injected into the patterns or source directories.. That only happens during the `create` code path when any given database hook is asked to dump its databases and therefore appends a path containing the borgmatic runtime directory to `source_directories`. I'm not sure what the solution is here to this unfortunate confluence of factors. I could remove the check and error message, but that would open us back up to potential Borg hangs. Maybe the cleanest thing would be to only run this code path during `create` rather than `check`? After all, there's no risk of these particular hangs during a `check` by itself. ### borgmatic version 1.9.5 or main ### borgmatic installation method pipx ### Borg version 1.4.0 ### Python version 3.12.7 ### Database version (if applicable) 14.3 ### Operating system and version Manjaro stable
witten added the
bug
label 2025-01-12 06:27:46 +00:00

Yes that's exactly my configuration, database backup plus spot check.

Yes that's exactly my configuration, database backup plus spot check.
Author
Owner

Great. Now that I have a repro, I'll see what I can do about the error. Thanks for your patience.

Great. Now that I have a repro, I'll see what I can do about the error. Thanks for your patience.
Author
Owner

Also, you can temporarily disable the spot check if you like so as not to block your borgmatic use until this is fixed!

Also, you can temporarily disable the spot check if you like so as not to block your borgmatic use until this is fixed!
Author
Owner

Okay, this should be fixed now in main and will be part of the next release.

Okay, this should be fixed now in main and will be part of the next release.
Author
Owner

Released in borgmatic 1.9.6!

Released in borgmatic 1.9.6!
Sign in to join this conversation.
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#965
No description provided.