borgmatic-collective/borgmatic
borgmatic-collective
/
borgmatic
15
57
Fork 40
Code Issues 54 Pull Requests 1 Actions Packages Releases 121 Activity

Allow 'borgmatic check' to run orphaned objects check without separate configuration file #779

New Issue
Closed
opened 2023-10-29 10:15:54 +00:00 by jens-arnold · 8 comments
jens-arnold commented 2023-10-29 10:15:54 +00:00
Copy Link

What I'd like to do and why

With borgmatic < version 1.8 it has been possible to check all archives in a repository, which makes "borg" check for orphaned objects. In order to do this, one would have specified prefix: "" in config section consistency:. With the new, flat configuration file format (and the prefix option being deprecated in borg) this is no longer possible.

Looking at the documentation, there is the new command line option --match-archives. Unfortunately this option is not supported for borgmatic check, but only for actions "info", "list", "rlist" and "transfer".

Next idea has been to use --override to override archive_name_format. Unfortunately this doesn't work either:

# borgmatic check --config XXXXX.yaml --override archive_name_format="" --only archives --force -v 1
XXXXX.yaml: No valid configuration files found

summary:
XXXXX.yaml: Error parsing configuration file
An error occurred while parsing a configuration file at XXXXX.yaml:
At 'archive_name_format': None is not of type 'string'
...

Next try...

# borgmatic check --config XXXXX.yaml --override archive_name_format="*" --only archives --force -v 1
XXXXX.yaml: No valid configuration files found

summary:
XXXXX.yaml: Error parsing configuration file
Invalid override 'archive_name_format=*': did not find expected alphabetic or numeric character
...

And another seemingly clever idea:

# borgmatic check --config XXXXX.yaml --override archive_name_format="{now}" --only archives --force -v 1
XXXXX.yaml: No valid configuration files found

summary:
XXXXX.yaml: Error parsing configuration file
An error occurred while parsing a configuration file at XXXXX.yaml:
At 'archive_name_format': {'now': None} is not of type 'string'

I did not test whether it works using an extra configuration file (it probably does, but it kinda misses the point of using borgmatic instead of naked borg).

Afaics there are two possible solutions to this problem:

  1. Make command line option --match_archives work for borgmatic check. Combined with the already implemented #734 --match_archives "*" should allow orphaned objects check
  2. Implement an additional options (e.g. --all-archives) which makes this behavior explicit.

Other notes / implementation ideas

No response

### What I'd like to do and why With borgmatic < version 1.8 it has been possible to check all archives in a repository, which makes "borg" check for orphaned objects. In order to do this, one would have specified `prefix: ""` in config section `consistency:`. With the new, flat configuration file format (and the prefix option being deprecated in borg) this is no longer possible. Looking at the documentation, there is the new command line option `--match-archives`. Unfortunately this option is not supported for `borgmatic check`, but only for actions "info", "list", "rlist" and "transfer". Next idea has been to use `--override` to override `archive_name_format`. Unfortunately this doesn't work either: ``` # borgmatic check --config XXXXX.yaml --override archive_name_format="" --only archives --force -v 1 XXXXX.yaml: No valid configuration files found summary: XXXXX.yaml: Error parsing configuration file An error occurred while parsing a configuration file at XXXXX.yaml: At 'archive_name_format': None is not of type 'string' ... ``` Next try... ``` # borgmatic check --config XXXXX.yaml --override archive_name_format="*" --only archives --force -v 1 XXXXX.yaml: No valid configuration files found summary: XXXXX.yaml: Error parsing configuration file Invalid override 'archive_name_format=*': did not find expected alphabetic or numeric character ... ``` And another seemingly clever idea: ``` # borgmatic check --config XXXXX.yaml --override archive_name_format="{now}" --only archives --force -v 1 XXXXX.yaml: No valid configuration files found summary: XXXXX.yaml: Error parsing configuration file An error occurred while parsing a configuration file at XXXXX.yaml: At 'archive_name_format': {'now': None} is not of type 'string' ``` I did not test whether it works using an extra configuration file (it probably does, but it kinda misses the point of using borgmatic instead of naked borg). Afaics there are two possible solutions to this problem: 1. Make command line option `--match_archives` work for `borgmatic check`. Combined with the already implemented #734 `--match_archives "*"` should allow orphaned objects check 2. Implement an additional options (e.g. `--all-archives`) which makes this behavior explicit. ### Other notes / implementation ideas _No response_
witten commented 2023-10-29 16:00:14 +00:00
Owner
Copy Link

Thanks for filing this! There's no good reason why borgmatic check shouldn't have a --match-archives flag like some of the other actions do. Seems like an easy addition.

As for your various --override attempts, I think what's going on is that the value is (intentionally) getting parsed as JSON.. which unfortunately mean it's not easy to specify a Borg pattern there. So I'll also look into whether there's any way around that without losing the JSON functionality.

Thanks for filing this! There's no good reason why `borgmatic check` shouldn't have a `--match-archives` flag like some of the other actions do. Seems like an easy addition. As for your various `--override` attempts, I think what's going on is that the value is (intentionally) getting parsed as JSON.. which unfortunately mean it's not easy to specify a Borg pattern there. So I'll also look into whether there's any way around that without losing the JSON functionality.
witten commented 2023-10-29 23:24:47 +00:00
Owner
Copy Link

I added a --match-archives flag to the check action in main, and it'll be part of the next release!

(Side note: Your ISP is currently bouncing notification emails from this server, so you may not be notified of this by email.)

I added a `--match-archives` flag to the `check` action in main, and it'll be part of the next release! (Side note: Your ISP is currently bouncing notification emails from this server, so you may not be notified of this by email.)
witten commented 2023-10-30 02:04:08 +00:00
Owner
Copy Link

And I've fixed the override issue in main as well. So now if an --override is for an option with a string type, borgmatic won't try to parse those values as JSON/YAML and will just leave it as a plain string.

And I've fixed the override issue in main as well. So now if an `--override` is for an option with a string type, borgmatic won't try to parse those values as JSON/YAML and will just leave it as a plain string.
witten commented 2023-10-31 16:10:21 +00:00
Owner
Copy Link

Done! (See comments above. You may not have received some comment notifications due to your ISP blocking them.)

Done! (See comments above. You may not have received some comment notifications due to your ISP blocking them.)
witten commented 2023-11-20 05:23:14 +00:00
Owner
Copy Link

Just released with borgmatic 1.8.5!

Just released with borgmatic 1.8.5!
witten commented 2024-04-03 01:26:12 +00:00
Owner
Copy Link

From amiconn on IRC:

Now that borgmatic >= 1.8.5 is finally available in Debian sid, I was able to test the new --match-archives option for "borgmatic check" as well as the fix for ---override and string parameters

Unfortunately neither --match-archives "*" nor --match archives "" enable the orphan check. The former matches all archives, but still causes borg to state "Orphaned objects check skipped (needs all archives checked)."

The latter works as if wasn't specified at all, i.e. it only matches archives as specified in the config file.

However, using --override it is possible to make borg check for orphans. --override archive_name_format="" doesn't do the trick, but both --override archive_name_format="*" and --override archive_name_format="{now}" do work.

From amiconn on IRC: > Now that borgmatic >= 1.8.5 is finally available in Debian sid, I was able to test the new --match-archives option for "borgmatic check" as well as the fix for ---override and string parameters > > Unfortunately neither --match-archives "*" nor --match archives "" enable the orphan check. The former matches all archives, but still causes borg to state "Orphaned objects check skipped (needs all archives checked)." > > The latter works as if wasn't specified at all, i.e. it only matches archives as specified in the config file. > > However, using --override it is possible to make borg check for orphans. --override archive_name_format="" doesn't do the trick, but both --override archive_name_format="*" and --override archive_name_format="{now}" do work.
witten reopened this issue 2024-04-03 01:26:15 +00:00
witten commented 2024-04-16 17:39:56 +00:00
Owner
Copy Link

Re-fixed in main. It looks like Borg decides not to run the orphaned objects check when it receives --match-archives *, so I worked around that by not passing through --match-archives at all when the value is * or similar.

Re-fixed in main. It looks like Borg decides not to run the orphaned objects check when it receives `--match-archives *`, so I worked around that by not passing through `--match-archives` at all when the value is `*` or similar.
witten commented 2024-04-16 17:44:58 +00:00
Owner
Copy Link

Released in 1.8.10!

Released in 1.8.10!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
main
spot-check
1.8.10
1.8.9
1.8.8
1.8.7
1.8.6
1.8.5
1.8.4
1.8.3
1.8.2
1.8.1
1.8.0
1.7.15
1.7.14
1.7.13
1.7.12
1.7.11
1.7.10
1.7.9
1.7.8
1.7.7
1.7.6
1.7.5
1.7.4
1.7.3
1.7.2
1.7.1
1.7.0
1.6.6
1.6.5
1.6.4
1.6.3
1.6.2
1.6.1
1.6.0
1.5.24
1.5.23
1.5.22
1.5.21
1.5.20
1.5.19
1.5.18
1.5.17
1.5.16
1.5.15
1.5.14
1.5.13
1.5.12
1.5.11
1.5.10
1.5.9
1.5.8
1.5.7
1.5.7.dev0
1.5.6
1.5.5
1.5.4
1.5.3
1.5.2
1.5.1
1.5.0
1.4.22
1.4.21
1.4.20
1.4.19
1.4.18
1.4.17
1.4.16
1.4.15
1.4.14
1.4.13
1.4.12
1.4.11
1.4.10
1.4.9
1.4.8
1.4.7
1.4.6
1.4.5
1.4.4
1.4.3
1.4.2
1.4.1
1.4.0
1.3.26
1.3.25
1.3.24
1.3.23
1.3.22
1.3.21
1.3.20
1.3.19
1.3.18
1.3.17
1.3.16
1.3.15
1.3.14
1.3.13
1.3.12
1.3.11
1.3.10
1.3.9
1.3.8
1.3.7
1.3.6
1.3.5
1.3.4
1.3.3
1.3.2
1.3.1
1.3.0
1.2.18
1.2.17
1.2.16
1.2.15
1.2.14
1.2.13
1.2.12
1.2.11
1.2.10
1.2.9
1.2.8
1.2.7
1.2.6
1.2.5
1.2.4
1.2.3
1.2.2
1.2.1
1.2.0
1.1.15
1.1.14
1.1.13
1.1.12
1.1.11
1.1.10
1.1.9
1.1.7
1.1.6
1.1.5
1.1.4
1.1.3
1.1.2
1.1.1
1.1.0
1.0.3
1.0.2
1.0.1
1.0.0
0.1.8
0.1.7
0.1.6
0.1.5
0.1.4
0.1.3
0.1.2
0.1.1
0.1.0
0.0.7
0.0.6
0.0.5
0.0.4
0.0.3
0.0.2
Labels
Clear labels   
bug

   
data loss

   
design finalized

   
good first issue

   
new feature area

   
question / support

   
security

   
waiting for response
No Label
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#779
No description provided.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?