Give the ability to supply additional Borg options

Alexander-Shukaev commented

2021-06-26 15:11:56 +00:00

What I'm trying to do and why

Borg has additional options like --bypass-lock. This option is dangerous to use in general, i.e. using it to allow multiple producers to write to repository is a big bang. Don't do this. However, for read-only operations (e.g. info, list, mount, etc.), even during archive creation, it's totally fine (see also https://github.com/borgbackup/borg/issues/5261).

The above is just an example, there are other useful options as well. But let's continue with the above one for the sake of example.

I need a way to invoke both info and list exactly in that order for each repository using --bypass-lock option as well. Currently, there are multiple issues at play here.

First of all, if we use regular borgmatic ... info ... list ..., then

I see that the order is hard-coded in borgmatic, that is list will always be called before info, and hence, appear in the output before. I think this should be changed to follow the order on the command line as it matters (in the current case for presentation purposes, but in other cases might matter logically). As a side question here, when invoked as borg ... info ... list ..., I see that Borg processes these in one go. Is there any particular reason why Borgmatic splits those into two separate invocations? I'm asking because this also affects the order and is another way to fix the described issue.
There is no way to supply --bypass-lock. To support that, there should either be a way to supply those common Borg options for any borgmatic command in general and/or at the very least info and list should be added into the extra_borg_options dictionary (though the issue will still facilitate itself for any other/future commands not in the dictionary).
The order in which info and list appear per repository is interleaved. This is good and expected as it's important to present information aggregated per repository. The only issue is still from the first point about list coming before info.

Secondly, due to two issues described above, I tried to resort to the new borgmatic ... borg ... command, here is my feedback on it:

Without looking at the code, it looks like Borgmatic tries hard to parse each borg ... invocation thoroughly. As a result, it's not possible to run either borgmatic ... borg info ... list ... or borgmatic ... borg info ... borg list .... This immediately leads to impossibility to display info and list per repository interleaved. As invoking them separately will aggregate first info for all repositories and then list for all repositories. The only good part is that --bypass-lock option is accepted though only after info or list, which is semantically not the right place for such common Borg options. I think parsing can be improved by introducing special delimiters which would allow both options in front of Borg commands and multiple Borg commands to appear in one invocation, e.g. borgmatic ... borg [borgmatic borg options] -- [common borg options] info ... --- [borgmatic borg options] -- [common borg options] list ... ---- borg [borgmatic borg options] -- [common borg options] create ... --- [borgmatic borg options] -- [common borg options] prune .... So -- ends Borgmatic options, regular convention, --- separates multiple Borg commands within one borg invocation, ---- return back to Borgmatic, so it can either run Borgmatic command(s) or run new Borg command(s) again.

Environment

borgmatic version: 1.5.15

Borg version: 1.1.16

Python version: 3.9.1

#### What I'm trying to do and why Borg has additional options like `--bypass-lock`. This option is dangerous to use in general, i.e. using it to allow multiple producers to write to repository is a big bang. Don't do this. However, for read-only operations (e.g. `info`, `list`, `mount`, etc.), even during archive creation, it's totally fine (see also https://github.com/borgbackup/borg/issues/5261). The above is just an example, there are other useful options as well. But let's continue with the above one for the sake of example. I need a way to invoke both `info` and `list` exactly in that order for each repository using `--bypass-lock` option as well. Currently, there are multiple issues at play here. First of all, if we use regular `borgmatic ... info ... list ...`, then 1. I see that the order is hard-coded in `borgmatic`, that is `list` will always be called before `info`, and hence, appear in the output before. I think this should be changed to follow the order on the command line as it matters (in the current case for presentation purposes, but in other cases might matter logically). As a side question here, when invoked as `borg ... info ... list ...`, I see that Borg processes these in one go. Is there any particular reason why Borgmatic splits those into two separate invocations? I'm asking because this also affects the order and is another way to fix the described issue. 2. There is no way to supply `--bypass-lock`. To support that, there should either be a way to supply those common Borg options for any `borgmatic` command in general and/or at the very least `info` and `list` should be added into the `extra_borg_options` dictionary (though the issue will still facilitate itself for any other/future commands not in the dictionary). 3. The order in which `info` and `list` appear per repository is interleaved. This is good and expected as it's important to present information aggregated per repository. The only issue is still from the first point about `list` coming before `info`. Secondly, due to two issues described above, I tried to resort to the new `borgmatic ... borg ...` command, here is my feedback on it: - Without looking at the code, it looks like Borgmatic tries hard to parse each `borg ...` invocation thoroughly. As a result, it's not possible to run either `borgmatic ... borg info ... list ...` or `borgmatic ... borg info ... borg list ...`. This immediately leads to impossibility to display `info` and `list` per repository interleaved. As invoking them separately will aggregate first `info` for all repositories and then `list` for all repositories. The only good part is that `--bypass-lock` option is accepted though only after `info` or `list`, which is semantically not the right place for such common Borg options. I think parsing can be improved by introducing special delimiters which would allow both options in front of Borg commands and multiple Borg commands to appear in one invocation, e.g. `borgmatic ... borg [borgmatic borg options] -- [common borg options] info ... --- [borgmatic borg options] -- [common borg options] list ... ---- borg [borgmatic borg options] -- [common borg options] create ... --- [borgmatic borg options] -- [common borg options] prune ...`. So `--` ends Borgmatic options, regular convention, `---` separates multiple Borg commands within one `borg` invocation, `----` return back to Borgmatic, so it can either run Borgmatic command(s) or run new Borg command(s) again. #### Environment **borgmatic version:** `1.5.15` **Borg version:** `1.1.16` **Python version:** `3.9.1`

Alexander-Shukaev referenced this issue

2021-12-31 12:36:37 +00:00

before_everything / after_everything hook does not apply for "borgmatic info/list" #487

witten commented

2022-04-24 03:39:56 +00:00

Owner

For point 2 above, there are existing options that almost support that:

storage:
    extra_borg_options:
        init: --extra-option
        prune: --extra-option
        ...

However that doesn't yet support list or info. However it wouldn't be difficult to add that support!

Also, borgmatic doesn't parse each borg action thoroughly. However it does at minimum need to parse out that Borg command. That's because borgmatic needs to pass the borgmatic --repository and --archive to Borg in the right spot in the constructed command-line—or else Borg blows up.

For point 2 above, there are existing options that *almost* support that: ```yaml storage: extra_borg_options: init: --extra-option prune: --extra-option ... ``` However that doesn't yet support `list` or `info`. However it wouldn't be difficult to add that support! Also, borgmatic doesn't parse each `borg` action *thoroughly*. However it does at minimum need to parse out that Borg command. That's because borgmatic needs to pass the borgmatic `--repository` and `--archive` to Borg in the right spot in the constructed command-line—or else Borg blows up.

witten added the

good first issue

label

2023-02-04 21:20:36 +00:00

witten referenced this issue

2023-03-23 21:13:07 +00:00

Allow restoring to other database server #326

witten commented

2024-10-25 01:15:30 +00:00

Owner

I see that the order is hard-coded in borgmatic, that is list will always be called before info, and hence, appear in the output before. I think this should be changed to follow the order on the command line as it matters (in the current case for presentation purposes, but in other cases might matter logically).

Since borgmatic 1.7.9, the order of actions given on the command-line is the order they run in.

> 1. I see that the order is hard-coded in borgmatic, that is list will always be called before info, and hence, appear in the output before. I think this should be changed to follow the order on the command line as it matters (in the current case for presentation purposes, but in other cases might matter logically). Since borgmatic 1.7.9, the order of actions given on the command-line is the order they run in.

witten commented

2024-11-20 06:38:11 +00:00

Owner

Looks like --bypass-lock is gone in Borg 2.

Looks like `--bypass-lock` is gone in [Borg 2](https://borgbackup.readthedocs.io/en/2.0.0b14/changes.html).

michael referenced this issue

2025-03-09 17:28:17 +00:00

Add support for extra Borg options in info and list commands #1022

witten referenced this issue

2025-03-28 18:53:05 +00:00

Access borg repositories in read only mode #388

TheProgrammingArchive commented

2025-03-31 03:05:47 +00:00

Hi, I would like to try and work on this issue!!

witten commented

2025-03-31 18:37:04 +00:00

Owner

Great! I think this should just be a question of expanding the existing extra_borg_options to support additional Borg subcommands, but please let me know if you have any questions along the way. And I'd be happy to brainstorm the approach if you want to do that ahead of a PR.

Great! I *think* this should just be a question of expanding the existing `extra_borg_options` to support additional Borg subcommands, but please let me know if you have any questions along the way. And I'd be happy to brainstorm the approach if you want to do that ahead of a PR.

witten referenced this issue

2025-04-03 20:02:27 +00:00

Recreate not removing excluded files when using lvm snapshots #1053

HanslettTheDev commented

2025-04-08 14:35:20 +00:00

Hello there
What are the most essential subcommands that need to be added? Or I'm free to decide which can be ported

Hello there What are the most essential subcommands that need to be added? Or I'm free to decide which can be ported

HanslettTheDev commented

2025-04-08 14:56:18 +00:00

Since this is 4 years old and while checking the PR that linked this issue, it seems the most essential borg commands have been ported\

witten commented

2025-04-08 16:52:34 +00:00

Owner

What are the most essential subcommands that need to be added? Or I'm free to decide which can be ported

My off-the-cuff answer is "all of them." But more realistically, I think you are correct that determining which subcommands should be made available through this configuration is up to the implementer. It may be the case that some just don't make sense to wrap.

As of right now, these are the only Borg subcommands wrapped by borgmatic's extra_borg_options:

init (which incidentally should probably be renamed to repo-create at some point)
create
prune
compact
check

> What are the most essential subcommands that need to be added? Or I'm free to decide which can be ported My off-the-cuff answer is "all of them." But more realistically, I think you are correct that determining which subcommands should be made available through this configuration is up to the implementer. It may be the case that some just don't make sense to wrap. As of right now, these are the only Borg subcommands wrapped by borgmatic's `extra_borg_options`: * `init` (which incidentally should probably be renamed to `repo-create` at some point) * `create` * `prune` * `compact` * `check`

HanslettTheDev commented

2025-04-08 17:35:56 +00:00

Alright
Got it
i will open up a pull request on the subject

Alright Got it i will open up a pull request on the subject

👍 1

HanslettTheDev commented

2025-04-08 17:41:57 +00:00

I left message a message in the IRC chat as well regarding my proposal

witten commented

2025-04-08 17:51:48 +00:00

Owner

I left message a message in the IRC chat as well regarding my proposal

I'm not seeing it there, so you might want to include it on this ticket. Thanks.

> I left message a message in the IRC chat as well regarding my proposal I'm not seeing it there, so you might want to include it on this ticket. Thanks.

HanslettTheDev commented

2025-04-08 17:53:17 +00:00

It was just a link to my proposal and review
But time is late
But here is the link

It was just a link to my proposal and review But time is late But here is the [link]([url](https://docs.google.com/document/d/1J_vSu0il7ElfxUMtNtMSrxkLde3_W8h8sPfQnPjDuxc/edit?tab=t.0))

HanslettTheDev commented

2025-04-08 18:00:50 +00:00

Link to the docs

[Link to the docs](https://docs.google.com/document/d/1J_vSu0il7ElfxUMtNtMSrxkLde3_W8h8sPfQnPjDuxc/edit?tab=t.0)

HanslettTheDev commented

2025-04-08 18:02:09 +00:00

I couldn't complete it in time. but I hope I can proof my ability to work before the results are out
I'll like to be assigned this issue
And proceed working on it

I couldn't complete it in time. but I hope I can proof my ability to work before the results are out I'll like to be assigned this issue And proceed working on it

HanslettTheDev commented

2025-04-08 18:04:48 +00:00

Sorry, @witten, that comment I left was via Element

Sorry, @witten, that comment I left was via [Element](https://matrix.to/#/!RaDrIDERRFKzJVvRFg:matrix.org/$SjxrXl91Xx9RY3yZD1b_LUIYRmoh2K3wc4xshzyXIng?via=gnome.org&via=matrix.org&via=envs.net)

witten commented

2025-04-09 05:56:24 +00:00

Owner

Oh no worries! We received your proposal on the main GSoC website as well.

But in regards to this ticket, pull requests are always welcome! Thank you.

Oh no worries! We received your proposal on the main GSoC website as well. But in regards to this ticket, pull requests are always welcome! Thank you.

HanslettTheDev commented

2025-04-19 14:16:40 +00:00

Taking a look at the borgmatic folder structure, I'm a bit confused precisely where the extra sub commands from borg are added. Currently I notice two folders [Borg, Actions] share the same file and coding structure
So do this folders describe the different operations carried out on borg like the actions folder contains borg actions while the commands folder contains the sub commands for the borg actions
If that's the design, wouldn't it be better to rename the commands folder to sub_commands to be more clear?

![image.png](/attachments/53b0203c-a04c-47b7-bb50-93ec21c5c956) Taking a look at the borgmatic folder structure, I'm a bit confused precisely where the extra sub commands from borg are added. Currently I notice two folders [`Borg`, `Actions`] share the same file and coding structure So do this folders describe the different operations carried out on borg like the `actions` folder contains `borg` actions while the `commands` folder contains the `sub commands` for the borg actions If that's the design, wouldn't it be better to rename the `commands` folder to `sub_commands` to be more clear?

image.png

231 KiB

witten commented

2025-04-19 16:10:47 +00:00

Owner

Please check out the source code reference, which hopefully answers many of these questions. In borgmatic, "sub-commands" are called "actions." So the actions directory contains borgmatic-specific logic for its own actions, while the borg directory contains Borg-specific logic for them (calling out to Borg sub-commands). commands, in contrast, contains code for top-level commands like borgmatic including all the argument parsing it does for both actions and global flags.

Please check out the [source code reference](https://torsion.org/borgmatic/docs/reference/source-code/), which hopefully answers many of these questions. In borgmatic, "sub-commands" are called "actions." So the `actions` directory contains borgmatic-specific logic for its own actions, while the `borg` directory contains Borg-specific logic for them (calling out to Borg sub-commands). `commands`, in contrast, contains code for top-level commands like `borgmatic` including all the argument parsing it does for both actions and global flags.

HanslettTheDev commented

2025-04-21 21:14:52 +00:00

oh sorry my bad for not checking the docs properly
I'd open a draft PR on this to proceed
Thanks for the reply

oh sorry my bad for not checking the docs properly I'd open a draft PR on this to proceed Thanks for the reply

witten commented

2025-04-23 16:39:31 +00:00

Owner

No worries!

witten referenced this issue

2025-05-29 16:33:18 +00:00

Extend extra_borg_options with other commands #1103

witten referenced this issue

2025-10-22 19:21:12 +00:00

Add --extra-borg-options.list option #1164

witten referenced this issue from a commit

2025-10-24 04:14:23 +00:00

Expand the "borg_extra_options" option to support every Borg sub-command that borgmatic uses (#427).

witten commented

2025-10-24 04:17:20 +00:00

Owner

Well, I thought somebody would implement this in the last four plus years. Turns out, in the end, that somebody was me. 😄

This feature is implemented in main and will be part of the next release. Every Borg sub-command that borgmatic uses now has an option under borg_extra_options. In cases where a sub-command is hyphenated, the corresponding option name has an underscore instead.

Also, as part of this change, the init option under borg_extra_options is deprecated in favor of repo_create. Both still work though.

Well, I thought _somebody_ would implement this in the last four plus years. Turns out, in the end, that somebody was me. 😄 This feature is implemented in main and will be part of the next release. Every Borg sub-command that borgmatic uses now has an option under `borg_extra_options`. In cases where a sub-command is hyphenated, the corresponding option name has an underscore instead. Also, as part of this change, the `init` option under `borg_extra_options` is deprecated in favor of `repo_create`. Both still work though.

witten closed this issue

2025-10-24 04:17:20 +00:00

witten referenced this issue

2025-10-24 04:19:23 +00:00

Add --extra-borg-options.list option #1164

witten commented

2025-10-24 04:51:35 +00:00

Owner

Released in borgmatic 2.0.10!