Database backup runtime directory error #999

New Issue

Closed

opened 2025-02-17 19:35:14 +00:00 by sean-wallace · 8 comments

sean-wallace commented 2025-02-17 19:35:14 +00:00

Copy Link

What I'm trying to do and why

I can't seem to get any combination of runtime_directory and database backups to work since this change that went out in 1.9.5.

It always triggers the ValueError below pretty much whatever I do.

raise ValueError(
    f'The runtime directory {os.path.normpath(borgmatic_runtime_directory)} overlaps with the configured excludes. Please remove it from excludes or change the runtime directory.'
)

Any ideas?

Steps to reproduce

No response

Actual behavior

No response

Expected behavior

No response

Other notes / implementation ideas

No response

borgmatic version

1.9.5-1.9.10

borgmatic installation method

pip install, homebrew

Borg version

1.4.0

Python version

3.12.9

Database version (if applicable)

No response

Operating system and version

No response

### What I'm trying to do and why I can't seem to get any combination of runtime_directory and database backups to work since [this change](https://projects.torsion.org/borgmatic-collective/borgmatic/commit/267af5b372cd28025d2db286ca4cd8cb9bf51b71) that went out in 1.9.5. It always triggers the ValueError below pretty much whatever I do. ``` raise ValueError( f'The runtime directory {os.path.normpath(borgmatic_runtime_directory)} overlaps with the configured excludes. Please remove it from excludes or change the runtime directory.' ) ``` Any ideas? ### Steps to reproduce _No response_ ### Actual behavior _No response_ ### Expected behavior _No response_ ### Other notes / implementation ideas _No response_ ### borgmatic version 1.9.5-1.9.10 ### borgmatic installation method pip install, homebrew ### Borg version 1.4.0 ### Python version 3.12.9 ### Database version (if applicable) _No response_ ### Operating system and version _No response_

witten commented 2025-02-17 19:46:57 +00:00

Owner

Copy Link

Can I get a look at your configuration and excludes to help diagnose this? Also, when you run borgmatic with --verbosity 2, what is the runtime directory displayed? It should show up on line that says: "Using runtime directory ..."

Thanks!

Can I get a look at your configuration and excludes to help diagnose this? Also, when you run borgmatic with `--verbosity 2`, what is the runtime directory displayed? It should show up on line that says: "Using runtime directory ..." Thanks!

sean-wallace commented 2025-02-17 19:59:30 +00:00

Author

Copy Link

Sure, slightly truncated config:

source_directories:
    - ~/Desktop
    - ~/Documents
    - ~/Downloads
    - ~/projects

repositories:
    - path: /Volumes/SW Backup/borg
      label: swbackup

exclude_patterns:
    - '*.pyc'

exclude_if_present:
    - .nobackup

keep_exclude_tags: true

extra_borg_options:
    create: --list --filter=AME --stats

    prune: --list --stats

postgresql_databases:
    - name: all

And the runtime directory line is showing as Using runtime directory /var/folders/h3/hh7xwqyj3bxdjdlkpnqlyl940000gp/T/borgmatic-499_rck4/borgmatic

Sure, slightly truncated config: ``` source_directories: - ~/Desktop - ~/Documents - ~/Downloads - ~/projects repositories: - path: /Volumes/SW Backup/borg label: swbackup exclude_patterns: - '*.pyc' exclude_if_present: - .nobackup keep_exclude_tags: true extra_borg_options: create: --list --filter=AME --stats prune: --list --stats postgresql_databases: - name: all ``` And the runtime directory line is showing as `Using runtime directory /var/folders/h3/hh7xwqyj3bxdjdlkpnqlyl940000gp/T/borgmatic-499_rck4/borgmatic`

sean-wallace commented 2025-02-17 20:04:31 +00:00

Author

Copy Link

@witten Is there a chance that the if not skip_paths condition should be if skip_paths? I will admit I don't fully understand the point of what it's doing, but it seems like it might be backwards to me.

@witten Is there a chance that the `if not skip_paths` condition should be `if skip_paths`? I will admit I don't fully understand the point of what it's doing, but it seems like it might be backwards to me.

witten commented 2025-02-17 21:57:24 +00:00

Owner

Copy Link

And the runtime directory line is showing as Using runtime directory /var/folders/h3/hh7xwqyj3bxdjdlkpnqlyl940000gp/T/borgmatic-499_rck4/borgmatic

Interesting, yeah, I don't see anything in your config that would obviously overlap the runtime directory. One thing you can do is look at the output of a borgmatic create run with --files (which you might not actually need since you've already got --list --filter=AME in your extra_borg_options). That should display exactly what's being excluded (via the x prefix). To temporarily get past the error so you can see that, you can add --dry-run which should suppress the ValueError. (See below!)

@witten Is there a chance that the if not skip_paths condition should be if skip_paths? I will admit I don't fully understand the point of what it's doing, but it seems like it might be backwards to me.

That code is pretty tricky, and maybe needs better commenting and/or variable naming. What's going on is that paths is the list of all file paths to backup as determined by a Borg create dry run—file paths that have not been otherwise excluded. Then skip_paths is the subset of those files paths contained by the borgmatic runtime directory. If there are none (if not skip_paths:), that means somewhere along the way, the borgmatic runtime directory (which gets auto-included by borgmatic) got excluded by the user (e.g. via exclude_patterns or similar), and therefore we should error.

Phew.

So. Seeing your excludes would help determine what if anything is causing the runtime directory to get excluded. It's also possible, of course, that something is going wrong in the logic described above.

...

Hold up! I just tried your extra_borg_options: locally, specifically the create, and even more explicitly the --filter=AME and got a repro of your error! I think what's going on is that the --filter=AME is getting passed to the internal borgmatic borg create --dry-run in that function (collect_special_file_paths()) and causing the dry-run output to list zero file paths—which triggers the error above because it thinks the borgmatic runtime directory has been excluded.

Temporarily, I think you can remove --filter AME and that should let you bypass the error. But in terms of fixing this for real, I think I need to make this function ignore or override any user --filter options so as not to inadvertently trigger the error.

> And the runtime directory line is showing as Using runtime directory /var/folders/h3/hh7xwqyj3bxdjdlkpnqlyl940000gp/T/borgmatic-499_rck4/borgmatic ~~Interesting, yeah, I don't see anything in your config that would obviously overlap the runtime directory. One thing you can do is look at the output of a `borgmatic create` run with `--files` (which you might not actually need since you've already got `--list --filter=AME` in your `extra_borg_options`). That should display exactly what's being excluded (via the `x ` prefix). To temporarily get past the error so you can see that, you can add `--dry-run` which should suppress the ValueError.~~ (See below!) > @witten Is there a chance that the if not skip_paths condition should be if skip_paths? I will admit I don't fully understand the point of what it's doing, but it seems like it might be backwards to me. That code is pretty tricky, and maybe needs better commenting and/or variable naming. What's going on is that `paths` is the list of all file paths to backup as determined by a Borg `create` dry run—file paths that have not been otherwise excluded. Then `skip_paths` is the subset of those files paths contained by the borgmatic runtime directory. If there are none (`if not skip_paths:`), that means somewhere along the way, the borgmatic runtime directory (which gets auto-included by borgmatic) got excluded by the user (e.g. via `exclude_patterns` or similar), and therefore we should error. Phew. ~~So. Seeing your excludes would help determine what if anything is causing the runtime directory to get excluded. It's also possible, of course, that something is going wrong in the logic described above.~~ ... Hold up! I just tried your `extra_borg_options:` locally, specifically the `create`, and even more explicitly the `--filter=AME` and got a repro of your error! I think what's going on is that the `--filter=AME` is getting passed to the internal borgmatic `borg create --dry-run` in that function (`collect_special_file_paths()`) and causing the dry-run output to list zero file paths—which triggers the error above because it thinks the borgmatic runtime directory has been excluded. Temporarily, I think you can remove `--filter AME` and that should let you bypass the error. But in terms of fixing this for real, I think I need to make this function ignore or override any user `--filter` options so as not to inadvertently trigger the error.

witten added the bug label 2025-02-17 21:57:31 +00:00

sean-wallace commented 2025-02-17 22:11:08 +00:00

Author

Copy Link

Yep, removing the --filter=AME works! I also removed --list so it didn't take forever to run. Thanks very much for figuring it out!

Yep, removing the `--filter=AME` works! I also removed `--list` so it didn't take forever to run. Thanks very much for figuring it out!

witten referenced this issue from a commit 2025-02-17 22:13:24 +00:00

Improve clarity of comments and variable names of runtime directory exclude detection logic (#999).

witten commented 2025-02-17 22:14:39 +00:00

Owner

Copy Link

Thanks for your detailed repro info! That definitely helped. Okay, I'll work on coming up with a fix so you can add back --filter=AME if you want.

Thanks for your detailed repro info! That definitely helped. Okay, I'll work on coming up with a fix so you can add back `--filter=AME` if you want.

👍 1

sean-wallace referenced a pull request that will close this issue 2025-02-17 22:31:05 +00:00

Add to argument exclusions for dry-run #1000

witten referenced this issue from a commit 2025-02-22 22:26:39 +00:00

Fix a runtime directory error from a conflict between "extra_borg_options" and special file detection (#999).

witten commented 2025-02-22 22:27:57 +00:00

Owner

Copy Link

I went ahead and implemented a fix for this. It turned out that argparse was probably overkill, and instead I managed to iterate through the arguments such that I could filter out both --filter and its value. Thanks again for your help here!

The fix will be part of the next release.

I went ahead and implemented a fix for this. It turned out that `argparse` was probably overkill, and instead I managed to iterate through the arguments such that I could filter out both `--filter` and its value. Thanks again for your help here! The fix will be part of the next release.

witten closed this issue 2025-02-22 22:27:58 +00:00

witten referenced this issue from a commit 2025-02-22 22:32:59 +00:00

Fix broken tests (#999).

witten commented 2025-02-22 22:43:13 +00:00

Owner

Copy Link

Released in borgmatic 1.9.11!

Released in borgmatic 1.9.11!

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

command-hook-steps

2.1.3

2.1.2

2.1.1

2.1.0

2.0.13

2.0.12

2.0.11

2.0.10

2.0.9

2.0.8

2.0.7

2.0.6

2.0.5

2.0.4

2.0.3

2.0.2

2.0.1

2.0.0

1.9.14

1.9.13

1.9.12

1.9.11

1.9.10

1.9.9

1.9.8

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.14

1.8.13

1.8.12

1.8.11

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.24

1.5.23

1.5.22

1.5.21

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.14

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.7.dev0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.22

1.4.21

1.4.20

1.4.19

1.4.18

1.4.17

1.4.16

1.4.15

1.4.14

1.4.13

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.26

1.3.25

1.3.24

1.3.23

1.3.22

1.3.21

1.3.20

1.3.19

1.3.18

1.3.17

1.3.16

1.3.15

1.3.14

1.3.13

1.3.12

1.3.11

1.3.10

1.3.9

1.3.8

1.3.7

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.18

1.2.17

1.2.16

1.2.15

1.2.14

1.2.13

1.2.12

1.2.11

1.2.10

1.2.9

1.2.8

1.2.7

1.2.6

1.2.5

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.15

1.1.14

1.1.13

1.1.12

1.1.11

1.1.10

1.1.9

1.1.7

1.1.6

1.1.5

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.3

1.0.2

1.0.1

1.0.0

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

0.0.2

Labels

Clear labels

blocked

breaking

bug

data loss

design finalized

good first issue

new feature area

question / support

security

waiting for response

No Label bug

Milestone

No items

No Milestone

Assignees

Clear assignees

witten

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#999