Give the ability to supply additional Borg options #427

New Issue

Alexander-Shukaev · 2021-06-26T15:11:56Z

Alexander-Shukaev commented

2021-06-26 15:11:56 +00:00

What I'm trying to do and why

Borg has additional options like --bypass-lock. This option is dangerous to use in general, i.e. using it to allow multiple producers to write to repository is a big bang. Don't do this. However, for read-only operations (e.g. info, list, mount, etc.), even during archive creation, it's totally fine (see also https://github.com/borgbackup/borg/issues/5261).

The above is just an example, there are other useful options as well. But let's continue with the above one for the sake of example.

I need a way to invoke both info and list exactly in that order for each repository using --bypass-lock option as well. Currently, there are multiple issues at play here.

First of all, if we use regular borgmatic ... info ... list ..., then

I see that the order is hard-coded in borgmatic, that is list will always be called before info, and hence, appear in the output before. I think this should be changed to follow the order on the command line as it matters (in the current case for presentation purposes, but in other cases might matter logically). As a side question here, when invoked as borg ... info ... list ..., I see that Borg processes these in one go. Is there any particular reason why Borgmatic splits those into two separate invocations? I'm asking because this also affects the order and is another way to fix the described issue.
There is no way to supply --bypass-lock. To support that, there should either be a way to supply those common Borg options for any borgmatic command in general and/or at the very least info and list should be added into the extra_borg_options dictionary (though the issue will still facilitate itself for any other/future commands not in the dictionary).
The order in which info and list appear per repository is interleaved. This is good and expected as it's important to present information aggregated per repository. The only issue is still from the first point about list coming before info.

Secondly, due to two issues described above, I tried to resort to the new borgmatic ... borg ... command, here is my feedback on it:

Without looking at the code, it looks like Borgmatic tries hard to parse each borg ... invocation thoroughly. As a result, it's not possible to run either borgmatic ... borg info ... list ... or borgmatic ... borg info ... borg list .... This immediately leads to impossibility to display info and list per repository interleaved. As invoking them separately will aggregate first info for all repositories and then list for all repositories. The only good part is that --bypass-lock option is accepted though only after info or list, which is semantically not the right place for such common Borg options. I think parsing can be improved by introducing special delimiters which would allow both options in front of Borg commands and multiple Borg commands to appear in one invocation, e.g. borgmatic ... borg [borgmatic borg options] -- [common borg options] info ... --- [borgmatic borg options] -- [common borg options] list ... ---- borg [borgmatic borg options] -- [common borg options] create ... --- [borgmatic borg options] -- [common borg options] prune .... So -- ends Borgmatic options, regular convention, --- separates multiple Borg commands within one borg invocation, ---- return back to Borgmatic, so it can either run Borgmatic command(s) or run new Borg command(s) again.

Environment

borgmatic version: 1.5.15

Borg version: 1.1.16

Python version: 3.9.1

#### What I'm trying to do and why Borg has additional options like `--bypass-lock`. This option is dangerous to use in general, i.e. using it to allow multiple producers to write to repository is a big bang. Don't do this. However, for read-only operations (e.g. `info`, `list`, `mount`, etc.), even during archive creation, it's totally fine (see also https://github.com/borgbackup/borg/issues/5261). The above is just an example, there are other useful options as well. But let's continue with the above one for the sake of example. I need a way to invoke both `info` and `list` exactly in that order for each repository using `--bypass-lock` option as well. Currently, there are multiple issues at play here. First of all, if we use regular `borgmatic ... info ... list ...`, then 1. I see that the order is hard-coded in `borgmatic`, that is `list` will always be called before `info`, and hence, appear in the output before. I think this should be changed to follow the order on the command line as it matters (in the current case for presentation purposes, but in other cases might matter logically). As a side question here, when invoked as `borg ... info ... list ...`, I see that Borg processes these in one go. Is there any particular reason why Borgmatic splits those into two separate invocations? I'm asking because this also affects the order and is another way to fix the described issue. 2. There is no way to supply `--bypass-lock`. To support that, there should either be a way to supply those common Borg options for any `borgmatic` command in general and/or at the very least `info` and `list` should be added into the `extra_borg_options` dictionary (though the issue will still facilitate itself for any other/future commands not in the dictionary). 3. The order in which `info` and `list` appear per repository is interleaved. This is good and expected as it's important to present information aggregated per repository. The only issue is still from the first point about `list` coming before `info`. Secondly, due to two issues described above, I tried to resort to the new `borgmatic ... borg ...` command, here is my feedback on it: - Without looking at the code, it looks like Borgmatic tries hard to parse each `borg ...` invocation thoroughly. As a result, it's not possible to run either `borgmatic ... borg info ... list ...` or `borgmatic ... borg info ... borg list ...`. This immediately leads to impossibility to display `info` and `list` per repository interleaved. As invoking them separately will aggregate first `info` for all repositories and then `list` for all repositories. The only good part is that `--bypass-lock` option is accepted though only after `info` or `list`, which is semantically not the right place for such common Borg options. I think parsing can be improved by introducing special delimiters which would allow both options in front of Borg commands and multiple Borg commands to appear in one invocation, e.g. `borgmatic ... borg [borgmatic borg options] -- [common borg options] info ... --- [borgmatic borg options] -- [common borg options] list ... ---- borg [borgmatic borg options] -- [common borg options] create ... --- [borgmatic borg options] -- [common borg options] prune ...`. So `--` ends Borgmatic options, regular convention, `---` separates multiple Borg commands within one `borg` invocation, `----` return back to Borgmatic, so it can either run Borgmatic command(s) or run new Borg command(s) again. #### Environment **borgmatic version:** `1.5.15` **Borg version:** `1.1.16` **Python version:** `3.9.1`

Alexander-Shukaev referenced this issue

2021-12-31 12:36:37 +00:00

before_everything / after_everything hook does not apply for "borgmatic info/list" #487

witten commented

2022-04-24 03:39:56 +00:00

For point 2 above, there are existing options that almost support that:

storage:
    extra_borg_options:
        init: --extra-option
        prune: --extra-option
        ...

However that doesn't yet support list or info. However it wouldn't be difficult to add that support!

Also, borgmatic doesn't parse each borg action thoroughly. However it does at minimum need to parse out that Borg command. That's because borgmatic needs to pass the borgmatic --repository and --archive to Borg in the right spot in the constructed command-line—or else Borg blows up.

For point 2 above, there are existing options that *almost* support that: ```yaml storage: extra_borg_options: init: --extra-option prune: --extra-option ... ``` However that doesn't yet support `list` or `info`. However it wouldn't be difficult to add that support! Also, borgmatic doesn't parse each `borg` action *thoroughly*. However it does at minimum need to parse out that Borg command. That's because borgmatic needs to pass the borgmatic `--repository` and `--archive` to Borg in the right spot in the constructed command-line—or else Borg blows up.

witten added the

good first issue

label 2023-02-04 21:20:36 +00:00

witten referenced this issue

2023-03-23 21:13:07 +00:00

Allow restoring to other database server #326

Sign in to join this conversation.