Allow 'borgmatic check' to run orphaned objects check without separate configuration file #779

New Issue

Closed

opened 2023-10-29 10:15:54 +00:00 by jens-arnold · 8 comments

jens-arnold commented 2023-10-29 10:15:54 +00:00

Copy Link

What I'd like to do and why

With borgmatic < version 1.8 it has been possible to check all archives in a repository, which makes "borg" check for orphaned objects. In order to do this, one would have specified prefix: "" in config section consistency:. With the new, flat configuration file format (and the prefix option being deprecated in borg) this is no longer possible.

Looking at the documentation, there is the new command line option --match-archives. Unfortunately this option is not supported for borgmatic check, but only for actions "info", "list", "rlist" and "transfer".

Next idea has been to use --override to override archive_name_format. Unfortunately this doesn't work either:

# borgmatic check --config XXXXX.yaml --override archive_name_format="" --only archives --force -v 1
XXXXX.yaml: No valid configuration files found

summary:
XXXXX.yaml: Error parsing configuration file
An error occurred while parsing a configuration file at XXXXX.yaml:
At 'archive_name_format': None is not of type 'string'
...

Next try...

# borgmatic check --config XXXXX.yaml --override archive_name_format="*" --only archives --force -v 1
XXXXX.yaml: No valid configuration files found

summary:
XXXXX.yaml: Error parsing configuration file
Invalid override 'archive_name_format=*': did not find expected alphabetic or numeric character
...

And another seemingly clever idea:

# borgmatic check --config XXXXX.yaml --override archive_name_format="{now}" --only archives --force -v 1
XXXXX.yaml: No valid configuration files found

summary:
XXXXX.yaml: Error parsing configuration file
An error occurred while parsing a configuration file at XXXXX.yaml:
At 'archive_name_format': {'now': None} is not of type 'string'

I did not test whether it works using an extra configuration file (it probably does, but it kinda misses the point of using borgmatic instead of naked borg).

Afaics there are two possible solutions to this problem:

1. Make command line option --match_archives work for borgmatic check. Combined with the already implemented #734 --match_archives "*" should allow orphaned objects check
2. Implement an additional options (e.g. --all-archives) which makes this behavior explicit.

Other notes / implementation ideas

No response

### What I'd like to do and why With borgmatic < version 1.8 it has been possible to check all archives in a repository, which makes "borg" check for orphaned objects. In order to do this, one would have specified `prefix: ""` in config section `consistency:`. With the new, flat configuration file format (and the prefix option being deprecated in borg) this is no longer possible. Looking at the documentation, there is the new command line option `--match-archives`. Unfortunately this option is not supported for `borgmatic check`, but only for actions "info", "list", "rlist" and "transfer". Next idea has been to use `--override` to override `archive_name_format`. Unfortunately this doesn't work either: ``` # borgmatic check --config XXXXX.yaml --override archive_name_format="" --only archives --force -v 1 XXXXX.yaml: No valid configuration files found summary: XXXXX.yaml: Error parsing configuration file An error occurred while parsing a configuration file at XXXXX.yaml: At 'archive_name_format': None is not of type 'string' ... ``` Next try... ``` # borgmatic check --config XXXXX.yaml --override archive_name_format="*" --only archives --force -v 1 XXXXX.yaml: No valid configuration files found summary: XXXXX.yaml: Error parsing configuration file Invalid override 'archive_name_format=*': did not find expected alphabetic or numeric character ... ``` And another seemingly clever idea: ``` # borgmatic check --config XXXXX.yaml --override archive_name_format="{now}" --only archives --force -v 1 XXXXX.yaml: No valid configuration files found summary: XXXXX.yaml: Error parsing configuration file An error occurred while parsing a configuration file at XXXXX.yaml: At 'archive_name_format': {'now': None} is not of type 'string' ``` I did not test whether it works using an extra configuration file (it probably does, but it kinda misses the point of using borgmatic instead of naked borg). Afaics there are two possible solutions to this problem: 1. Make command line option `--match_archives` work for `borgmatic check`. Combined with the already implemented #734 `--match_archives "*"` should allow orphaned objects check 2. Implement an additional options (e.g. `--all-archives`) which makes this behavior explicit. ### Other notes / implementation ideas _No response_

witten commented 2023-10-29 16:00:14 +00:00

Owner

Copy Link

Thanks for filing this! There's no good reason why borgmatic check shouldn't have a --match-archives flag like some of the other actions do. Seems like an easy addition.

As for your various --override attempts, I think what's going on is that the value is (intentionally) getting parsed as JSON.. which unfortunately mean it's not easy to specify a Borg pattern there. So I'll also look into whether there's any way around that without losing the JSON functionality.

Thanks for filing this! There's no good reason why `borgmatic check` shouldn't have a `--match-archives` flag like some of the other actions do. Seems like an easy addition. As for your various `--override` attempts, I think what's going on is that the value is (intentionally) getting parsed as JSON.. which unfortunately mean it's not easy to specify a Borg pattern there. So I'll also look into whether there's any way around that without losing the JSON functionality.

witten referenced this issue from a commit 2023-10-29 23:23:09 +00:00

Add a "--match-archives" flag to the "check" action (#779).

witten commented 2023-10-29 23:24:47 +00:00

Owner

Copy Link

I added a --match-archives flag to the check action in main, and it'll be part of the next release!

(Side note: Your ISP is currently bouncing notification emails from this server, so you may not be notified of this by email.)

I added a `--match-archives` flag to the `check` action in main, and it'll be part of the next release! (Side note: Your ISP is currently bouncing notification emails from this server, so you may not be notified of this by email.)

witten referenced this issue from a commit 2023-10-29 23:25:56 +00:00

Correct changelog addition (#779).

witten referenced this issue from a commit 2023-10-30 02:02:48 +00:00

Only parse "--override" values as complex data types when they're for options of those types (#779).

witten commented 2023-10-30 02:04:08 +00:00

Owner

Copy Link

And I've fixed the override issue in main as well. So now if an --override is for an option with a string type, borgmatic won't try to parse those values as JSON/YAML and will just leave it as a plain string.

And I've fixed the override issue in main as well. So now if an `--override` is for an option with a string type, borgmatic won't try to parse those values as JSON/YAML and will just leave it as a plain string.

witten commented 2023-10-31 16:10:21 +00:00

Owner

Copy Link

Done! (See comments above. You may not have received some comment notifications due to your ISP blocking them.)

Done! (See comments above. You may not have received some comment notifications due to your ISP blocking them.)

witten closed this issue 2023-10-31 16:10:21 +00:00

witten commented 2023-11-20 05:23:14 +00:00

Owner

Copy Link

Just released with borgmatic 1.8.5!

Just released with borgmatic 1.8.5!

witten commented 2024-04-03 01:26:12 +00:00

Owner

Copy Link

From amiconn on IRC:

Now that borgmatic >= 1.8.5 is finally available in Debian sid, I was able to test the new --match-archives option for "borgmatic check" as well as the fix for ---override and string parameters

Unfortunately neither --match-archives "*" nor --match archives "" enable the orphan check. The former matches all archives, but still causes borg to state "Orphaned objects check skipped (needs all archives checked)."

The latter works as if wasn't specified at all, i.e. it only matches archives as specified in the config file.

However, using --override it is possible to make borg check for orphans. --override archive_name_format="" doesn't do the trick, but both --override archive_name_format="*" and --override archive_name_format="{now}" do work.

From amiconn on IRC: > Now that borgmatic >= 1.8.5 is finally available in Debian sid, I was able to test the new --match-archives option for "borgmatic check" as well as the fix for ---override and string parameters > > Unfortunately neither --match-archives "*" nor --match archives "" enable the orphan check. The former matches all archives, but still causes borg to state "Orphaned objects check skipped (needs all archives checked)." > > The latter works as if wasn't specified at all, i.e. it only matches archives as specified in the config file. > > However, using --override it is possible to make borg check for orphans. --override archive_name_format="" doesn't do the trick, but both --override archive_name_format="*" and --override archive_name_format="{now}" do work.

witten reopened this issue 2024-04-03 01:26:15 +00:00

witten referenced this issue from a commit 2024-04-16 17:39:47 +00:00

When "--match-archives *" is used with "check" action, don't skip Borg's orphaned objects check (#779).

witten commented 2024-04-16 17:39:56 +00:00

Owner

Copy Link

Re-fixed in main. It looks like Borg decides not to run the orphaned objects check when it receives --match-archives *, so I worked around that by not passing through --match-archives at all when the value is * or similar.

Re-fixed in main. It looks like Borg decides not to run the orphaned objects check when it receives `--match-archives *`, so I worked around that by not passing through `--match-archives` at all when the value is `*` or similar.

witten closed this issue 2024-04-16 17:39:56 +00:00

witten commented 2024-04-16 17:44:58 +00:00

Owner

Copy Link

Released in 1.8.10!

Released in 1.8.10!

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

1.8.14

1.8.13

1.8.12

1.8.11

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.24

1.5.23

1.5.22

1.5.21

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.14

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.7.dev0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.22

1.4.21

1.4.20

1.4.19

1.4.18

1.4.17

1.4.16

1.4.15

1.4.14

1.4.13

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.26

1.3.25

1.3.24

1.3.23

1.3.22

1.3.21

1.3.20

1.3.19

1.3.18

1.3.17

1.3.16

1.3.15

1.3.14

1.3.13

1.3.12

1.3.11

1.3.10

1.3.9

1.3.8

1.3.7

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.18

1.2.17

1.2.16

1.2.15

1.2.14

1.2.13

1.2.12

1.2.11

1.2.10

1.2.9

1.2.8

1.2.7

1.2.6

1.2.5

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.15

1.1.14

1.1.13

1.1.12

1.1.11

1.1.10

1.1.9

1.1.7

1.1.6

1.1.5

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.3

1.0.2

1.0.1

1.0.0

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

0.0.2

Labels

Clear labels   

bug

  

data loss

  

design finalized

  

good first issue

  

new feature area

  

question / support

  

security

  

waiting for response

No Label

bug

data loss

design finalized

good first issue

new feature area

question / support

security

waiting for response

Milestone

Clear milestone

No items

No Milestone

Assignees

Clear assignees

No Assignees

2 Participants

Notifications

Subscribe

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#779

No description provided.