Set borgmatic command-line flag defaults in config file #303

New Issue

samuelharmer · 2020-04-28T14:42:53Z

samuelharmer commented

2020-04-28 14:42:53 +00:00

What I'm trying to do and why

Specify --encryption type in config.yaml.

Steps to reproduce (if a bug)

[...]
storage:
    extra_borg_options:
        init: --make-parent-dirs --encryption keyfile-blake2
[...]

Actual behavior (if a bug)

$ borgmatic --verbosity 2 --config config.yaml init
usage: borgmatic init -e ENCRYPTION_MODE [--append-only]
                      [--storage-quota STORAGE_QUOTA] [-h]
borgmatic init: error: the following arguments are required: -e/--encryption
Error parsing arguments: <long_path>/borgmatic --verbosity 2 --config <long_path>/config.yaml init

Need some help? https://torsion.org/borgmatic/#issues

Expected behavior (if a bug)

The borg init command would be passed the --encryption option from extra_borg_options.

Other notes / implementation ideas

It would be really helpful if all borgmatic command-line options (excluding --config obviously) could be specified via config.yaml.

As borgmatic with no arguments is meant to 'just work' this would allow other borg commands to similarly 'just work' and I could capture my output preferences as well. As it is, I need to use a script to wrap borgmatic.

To go with this, it would also be useful if --config could be read from the environment (e.g. ${BORGMATIC_CONFIG}).

Environment

borgmatic version: 1.5.2

$ pipdeptree -p borgmatic
borgmatic==1.5.2
  - colorama [required: >=0.4.1,<0.5, installed: 0.4.3]
  - pykwalify [required: >=1.6.0,<14.06, installed: 1.7.0]
    - docopt [required: >=0.6.2, installed: 0.6.2]
    - python-dateutil [required: >=2.4.2, installed: 2.8.1]
      - six [required: >=1.5, installed: 1.14.0]
    - PyYAML [required: >=3.11, installed: 5.3.1]
  - requests [required: Any, installed: 2.23.0]
    - certifi [required: >=2017.4.17, installed: 2020.4.5.1]
    - chardet [required: >=3.0.2,<4, installed: 3.0.4]
    - idna [required: >=2.5,<3, installed: 2.9]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.9]
  - ruamel.yaml [required: >0.15.0,<0.17.0, installed: 0.16.10]
    - ruamel.yaml.clib [required: >=0.1.2, installed: 0.2.0]
  - setuptools [required: Any, installed: 40.8.0]

borgmatic installation method: pip

Borg version: 1.1.9

Python version: 3.7.3

operating system and version: Debian GNU/Linux 10 (buster)

#### What I'm trying to do and why Specify `--encryption` type in `config.yaml`. #### Steps to reproduce (if a bug) ```yaml [...] storage: extra_borg_options: init: --make-parent-dirs --encryption keyfile-blake2 [...] ``` #### Actual behavior (if a bug) ``` $ borgmatic --verbosity 2 --config config.yaml init usage: borgmatic init -e ENCRYPTION_MODE [--append-only] [--storage-quota STORAGE_QUOTA] [-h] borgmatic init: error: the following arguments are required: -e/--encryption Error parsing arguments: <long_path>/borgmatic --verbosity 2 --config <long_path>/config.yaml init Need some help? https://torsion.org/borgmatic/#issues ``` #### Expected behavior (if a bug) The borg init command would be passed the `--encryption` option from `extra_borg_options`. #### Other notes / implementation ideas It would be really helpful if all borgmatic command-line options (excluding `--config` obviously) could be specified via `config.yaml`. As `borgmatic` with no arguments is meant to 'just work' this would allow other `borg` commands to similarly 'just work' and I could capture my output preferences as well. As it is, I need to use a script to wrap borgmatic. To go with this, it would also be useful if `--config` could be read from the environment (e.g. `${BORGMATIC_CONFIG}`). #### Environment **borgmatic version:** 1.5.2 ``` $ pipdeptree -p borgmatic borgmatic==1.5.2 - colorama [required: >=0.4.1,<0.5, installed: 0.4.3] - pykwalify [required: >=1.6.0,<14.06, installed: 1.7.0] - docopt [required: >=0.6.2, installed: 0.6.2] - python-dateutil [required: >=2.4.2, installed: 2.8.1] - six [required: >=1.5, installed: 1.14.0] - PyYAML [required: >=3.11, installed: 5.3.1] - requests [required: Any, installed: 2.23.0] - certifi [required: >=2017.4.17, installed: 2020.4.5.1] - chardet [required: >=3.0.2,<4, installed: 3.0.4] - idna [required: >=2.5,<3, installed: 2.9] - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.9] - ruamel.yaml [required: >0.15.0,<0.17.0, installed: 0.16.10] - ruamel.yaml.clib [required: >=0.1.2, installed: 0.2.0] - setuptools [required: Any, installed: 40.8.0] ``` **borgmatic installation method:** pip **Borg version:** 1.1.9 **Python version:** 3.7.3 **operating system and version:** Debian GNU/Linux 10 (buster)

👍 1

samuelharmer changed title from ~~extra_borg_args not parsed early enough to permit --encryption type for init command to be provided in config~~ to extra_borg_args not parsed so encryption type cannot be provided in config

2020-04-28 14:59:47 +00:00

samuelharmer commented

2020-04-28 15:02:14 +00:00

Sorry, just remembered that borgmatic doesn't look at extra_borg_options at all. However the problem is roughly the same. I.e. Borgmatic configuration file can't handle all the configuration options needed by Borg.

Sorry, just remembered that borgmatic doesn't look at `extra_borg_options` at all. However the problem is roughly the same. I.e. Borgmatic configuration file can't handle all the configuration options needed by Borg.

witten commented

2020-04-28 16:28:40 +00:00

Neat idea! Could you say some more about your use case? Is it that you're running borgmatic init manually on occasion, and you don't want to have to type/remember all the necessary command-line arguments every time? Or are you running it in an automated fashion, and it's a pain to pass in the arguments?

And is it the encryption mode flag in particular? Or are there others that you find yourself wanting to have defaults?

Also, for the BORGMATIC_CONFIG environment variable idea, what's your use case? Are you invoking borgmatic from somewhere (e.g. a Docker container) where passing in environment variables is more convenient than command-line flags? Thanks.

Neat idea! Could you say some more about your use case? Is it that you're running `borgmatic init` manually on occasion, and you don't want to have to type/remember all the necessary command-line arguments every time? Or are you running it in an automated fashion, and it's a pain to pass in the arguments? And is it the encryption mode flag in particular? Or are there others that you find yourself wanting to have defaults? Also, for the `BORGMATIC_CONFIG` environment variable idea, what's your use case? Are you invoking borgmatic from somewhere (e.g. a Docker container) where passing in environment variables is more convenient than command-line flags? Thanks.

samuelharmer commented

2020-04-29 07:39:41 +00:00

Is it that you're running borgmatic init manually on occasion, and you don't want to have to type/remember all the necessary command-line arguments every time?

Yes, that's what led me to this. More generally it's to achieve DRYness which would help with configuration management. With the ruamel.yaml extension to include other files I'm aiming for a 'single generic file with all my settings'. Then some simple deployment for the machine-specific settings (i.e. hooks and repos). For me, things like verbosity and encryption type are not machine-specific therefore should live in the config file.

And is it the encryption mode flag in particular? Or are there others that you find yourself wanting to have defaults?

I feel it should be all options. What's irritating is that argparse for Borgmatic is erroring (because I guess that option has been set to required) but certainly in the case of encryption, I would expect it to just pass the arguments through to Borg (adding in extra arguments like the repo of course, but otherwise just combining). The way it behaves at the moment, it could work but argparse's configuration stops it (I am guessing).

Also, for the BORGMATIC_CONFIG environment variable idea, what's your use case? Are you invoking borgmatic from somewhere (e.g. a Docker container) where passing in environment variables is more convenient than command-line flags? Thanks.

Yep. Pretty much. But generally, being able to alter the configuration prior to invocation is the motivation. Could also be during development/testing from an IDE configuration.

> Is it that you're running `borgmatic init` manually on occasion, and you don't want to have to type/remember all the necessary command-line arguments every time? Yes, that's what led me to this. More generally it's to achieve [DRYness](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself) which would help with configuration management. With the `ruamel.yaml` extension to include other files I'm aiming for a 'single generic file with all my settings'. Then some simple deployment for the machine-specific settings (i.e. hooks and repos). For me, things like verbosity and encryption type are not machine-specific therefore should live in the config file. > And is it the encryption mode flag in particular? Or are there others that you find yourself wanting to have defaults? I feel it should be all options. What's irritating is that argparse for Borgmatic is erroring (because I guess that option has been set to required) but certainly in the case of encryption, I would expect it to just pass the arguments through to Borg (adding in extra arguments like the repo of course, but otherwise just combining). The way it behaves at the moment, it *could* work but argparse's configuration stops it (I am guessing). > Also, for the `BORGMATIC_CONFIG` environment variable idea, what's your use case? Are you invoking borgmatic from somewhere (e.g. a Docker container) where passing in environment variables is more convenient than command-line flags? Thanks. Yep. Pretty much. But generally, being able to alter the configuration prior to invocation is the motivation. Could also be during development/testing from an IDE configuration.

witten commented

2020-04-29 17:10:39 +00:00

For me, things like verbosity and encryption type are not machine-specific therefore should live in the config file.

Fair enough. I think I'll re-title the ticket in terms of this ask.

I feel it should be all options.

Could be most. But there are probably flags like --repository or --archive that only make sense at the command-line.

What's irritating is that argparse for Borgmatic is erroring (because I guess that option has been set to required) but certainly in the case of encryption, I would expect it to just pass the arguments through to Borg (adding in extra arguments like the repo of course, but otherwise just combining). The way it behaves at the moment, it could work but argparse's configuration stops it (I am guessing).

I think I'm missing something here. Borg requires the --encryption flag as well. So what would it accomplish for borgmatic not to require it?

Yep. Pretty much. But generally, being able to alter the configuration prior to invocation is the motivation. Could also be during development/testing from an IDE configuration.

And to be clear, you're thinking that BORGMATIC_CONFIG would be a path to config file? Or the actual config YAML contents?

> For me, things like verbosity and encryption type are not machine-specific therefore should live in the config file. Fair enough. I think I'll re-title the ticket in terms of this ask. > I feel it should be all options. Could be most. But there are probably flags like `--repository` or `--archive` that only make sense at the command-line. > What's irritating is that argparse for Borgmatic is erroring (because I guess that option has been set to required) but certainly in the case of encryption, I would expect it to just pass the arguments through to Borg (adding in extra arguments like the repo of course, but otherwise just combining). The way it behaves at the moment, it could work but argparse's configuration stops it (I am guessing). I think I'm missing something here. Borg requires the `--encryption` flag as well. So what would it accomplish for borgmatic not to require it? > Yep. Pretty much. But generally, being able to alter the configuration prior to invocation is the motivation. Could also be during development/testing from an IDE configuration. And to be clear, you're thinking that `BORGMATIC_CONFIG` would be a path to config file? Or the actual config YAML contents?

witten changed title from ~~extra_borg_args not parsed so encryption type cannot be provided in config~~ to Set borgmatic command-line flag defaults in config file

2020-04-29 17:11:15 +00:00

samuelharmer commented

2020-04-29 17:53:53 +00:00

Thanks. That's a better title.

Could be most. But there are probably flags like --repository or --archive that only make sense at the command-line.

Absolutely. I should have worded that better. I suppose what I meant was "all backup related options". As you point out, restore related options generally don't make sense in a configuration file. Although maybe there is a use-case for extracting a random sample and comparing the contents but that might be better served using consistency.checks: extract or could probably be done with hooks -- either way its certainly beyond the scope of this idea.

Borg requires the --encryption flag as well

It does, but if the argparse configuration was more permissive, I might have been able to pass it through by in storage.extra_borg_options.init. Not important though if (backup related) options can be included in config.

And to be clear, you're thinking that BORGMATIC_CONFIG would be a path to config file? Or the actual config YAML contents?

Just the path to the config file.

Thanks. That's a better title. > Could be most. But there are probably flags like --repository or --archive that only make sense at the command-line. Absolutely. I should have worded that better. I suppose what I meant was "all **backup** related options". As you point out, **restore** related options generally don't make sense in a configuration file. Although maybe there is a use-case for extracting a random sample and comparing the contents but that might be better served using `consistency.checks: extract` or could probably be done with hooks -- either way its certainly beyond the scope of this idea. > Borg requires the --encryption flag as well It does, but if the `argparse` configuration was more permissive, I might have been able to pass it through by in `storage.extra_borg_options.init`. Not important though if (backup related) options can be included in config. > And to be clear, you're thinking that BORGMATIC_CONFIG would be a path to config file? Or the actual config YAML contents? Just the path to the config file.

👍 1

ams_tschoening commented

2021-07-25 10:00:16 +00:00

Neat idea! Could you say some more about your use case? Is it that you're running borgmatic init manually on occasion, and you don't want to have to type/remember all the necessary command-line arguments every time?[...]

That's exactly my reason as well: I like the idea of borgmatic providing a config combining most/all aspects of the backup, interacting with the repo etc. very much. That includes how to create the repo in the first step, even if it would never be required latter again. Though, it would be "forever" documented in the config file, which is part of some GIT or SVN repo very likely, that e.g. encryption mode keyfile was used and at the same time make it easier to create a repo without needing to provide command line arguments manually.

Besides that, init might not be a one-shot task especially for new users as well. When I started with borg and borgmatic and tested things, I recreated some repos for various different reasons. Of course there's the shell history most likely making this easy as well, but that history is not forever and command simply become shorter when aspects like encryption mode are in the config file already.

> Neat idea! Could you say some more about your use case? Is it that you're running borgmatic init manually on occasion, and you don't want to have to type/remember all the necessary command-line arguments every time?[...] That's exactly my reason as well: I like the idea of borgmatic providing a config combining most/all aspects of the backup, interacting with the repo etc. very much. That includes how to create the repo in the first step, even if it would never be required latter again. Though, it would be "forever" documented in the config file, which is part of some GIT or SVN repo very likely, that e.g. encryption mode `keyfile` was used and at the same time make it easier to create a repo without needing to provide command line arguments manually. Besides that, `init` might not be a one-shot task especially for new users as well. When I started with borg and borgmatic and tested things, I recreated some repos for various different reasons. Of course there's the shell history most likely making this easy as well, but that history is not forever and command simply become shorter when aspects like encryption mode are in the config file already.

witten commented

2023-03-29 18:41:45 +00:00

I'm scoping this ticket down a bit for a couple of reasons: 1. I think it'll be more likely to get more traction that way, and 2. there are practical limitations to what borgmatic command-line flags can be in the configuration file. For instance, it'd be tough to put logging and verbosity flags in configuration, because logging and verbosity necessarily take effect before configuration is even loaded. (Although see #793.) And as discussed above, there are some flags like --repository or --repair that would typically only ever be used on the command-line.

So here's my proposal on existing command-line flags to add configuration options for. The idea is that if such an option is set in the configuration file, it would take effect unless overridden on the command-line.

rcreate (a.k.a. init)
- --encryption
- --append-only
- --storage-quota
- --make-parent-dirs
transfer
- --progress
create
- --progress
- --stats
- --list
prune
- --stats
- --list
compact
- --progress
- --threshold (could also apply to when borgmatic runs compact automatically)
check
- --progress
extract
- --progress
export-tar
- --list
~~rlist and list~~ (cut to reduce scope for now; can add in a subsequent ticket if needed)
- ~~--short~~
- ~~--format~~

This is just a proposal. Feel free to add/subtract supported flags as appropriate. For instance, an argument could be made for supporting --prefix or --match-archives. Same or --first and --last. But it might make sense to start small and expand the supported flags from there.

Where to put the new options in the configuration file is left as exercise to the implementer. And in some cases where the same flag exists across multiple actions, there could possibly be a single option that controls them.

I'm scoping this ticket down a bit for a couple of reasons: 1. I think it'll be more likely to get more traction that way, and 2. there are practical limitations to what borgmatic command-line flags can be in the configuration file. For instance, it'd be tough to put logging and verbosity flags in configuration, because logging and verbosity necessarily take effect before configuration is even loaded. (Although see [#793](https://projects.torsion.org/borgmatic-collective/borgmatic/issues/793).) And as discussed above, there are some flags like `--repository` or `--repair` that would typically only ever be used on the command-line. So here's my proposal on existing command-line flags to add configuration options for. The idea is that if such an option is set in the configuration file, it would take effect unless overridden on the command-line. * `rcreate` (a.k.a. `init`) * `--encryption` * `--append-only` * `--storage-quota` * `--make-parent-dirs` * `transfer` * `--progress` * `create` * `--progress` * `--stats` * `--list` * `prune` * `--stats` * `--list` * `compact` * `--progress` * `--threshold` (could also apply to when borgmatic runs compact automatically) * `check` * `--progress` * `extract` * `--progress` * `export-tar` * `--list` * ~~`rlist` and `list`~~ (cut to reduce scope for now; can add in a subsequent ticket if needed) * ~~`--short`~~ * ~~`--format`~~ This is just a proposal. Feel free to add/subtract supported flags as appropriate. For instance, an argument could be made for supporting `--prefix` or `--match-archives`. Same or [`--first` and `--last`](https://projects.torsion.org/borgmatic-collective/borgmatic/issues/305). But it might make sense to start small and expand the supported flags from there. Where to put the new options in the configuration file is left as exercise to the implementer. And in some cases where the same flag exists across multiple actions, there could possibly be a single option that controls them.

witten added the good first issue label 2023-03-29 18:41:52 +00:00

scy commented

2024-03-09 14:43:10 +00:00

I'm new to borgmatic and was confused about this, too.

One of the main reasons why I found borgmatic so attractive is that I can write things in a config file and get rid of having to create and maintain my own wrapper scripts where I call Borg with lines and lines of options and flags. A command like borgmatic create --verbosity 1 --list --progress --stats is kind of the opposite of the elegance that I aim to achieve, and almost forces me to write a wrapper script again, but this time around borgmatic.

So, in addition to the per-command arguments listed above, I'd like to suggest global arguments, too:

--verbosity (-v), together with --syslog-verbosity, --log-file-verbosity and --monitoring-verbosity.

YAML could actually provide a way to set all of these at once, or at least in a less verbose fashion:

verbosity: 1  # set --verbosity and --*-verbosity all to 1

verbosity:
  display: 2  # set --verbosity 2
  syslog: 0   # set --syslog-verbosity 0
  # all that are not explicitly mentioned stay at their defaults

--log-file

I'm new to borgmatic and was confused about this, too. One of the main reasons why I found borgmatic so attractive is that I can write things in a config file and get rid of having to create and maintain my own wrapper scripts where I call Borg with lines and lines of options and flags. A command like `borgmatic create --verbosity 1 --list --progress --stats` is kind of the opposite of the elegance that I aim to achieve, and almost forces me to write a wrapper script again, but this time around borgmatic. So, in addition to the per-command arguments listed above, I'd like to suggest global arguments, too: * `--verbosity` (`-v`), together with `--syslog-verbosity`, `--log-file-verbosity` and `--monitoring-verbosity`. * YAML could actually provide a way to set all of these at once, or at least in a less verbose fashion: ```yaml verbosity: 1 # set --verbosity and --*-verbosity all to 1 ``` ```yaml verbosity: display: 2 # set --verbosity 2 syslog: 0 # set --syslog-verbosity 0 # all that are not explicitly mentioned stay at their defaults ``` * `--log-file`

witten referenced this issue

2024-03-09 16:53:21 +00:00

Configuration of logging in YAML config file #793

witten commented

2024-03-09 16:57:06 +00:00

Thanks for weighing in. Part of the reason verbosity is only on the command-line now is that verbosity settings apply before configuration is even loaded! However see #793 for some ideas on how both logging and verbosity setup can potentially be stored in a borgmatic configuration file despite that current limitation.