Use of the borg option "--max-duration" in borgmatic is not behaving as expected. #814

New Issue

tehNiemer · 2024-01-18T13:28:42Z

tehNiemer commented

2024-01-18 13:28:42 +00:00

What I'm trying to do and why

Hopefully this is a simple case of user error. I'm trying to pass the borg option of --max-duration into borgmatic with extra_borg_options , but it's not working as I would have hoped.

Steps to reproduce

Run

borgmatic check --override checks.name=repository --override extra_borg_options.check="--repository-only --max-duration=3600" --only repository

I've tried multiple variants of the command, with and without single and double quotations.

Actual behavior

This error is generated.

Traceback (most recent call last):
  File "/root/.local/bin/borgmatic", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/root/.local/pipx/venvs/borgmatic/lib/python3.11/site-packages/borgmatic/commands/borgmatic.py", line 859, in main
    configs, parse_logs = load_configurations(
                          ^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/pipx/venvs/borgmatic/lib/python3.11/site-packages/borgmatic/commands/borgmatic.py", line 528, in load_configurations
    configs[config_filename], parse_logs = validate.parse_configuration(
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/pipx/venvs/borgmatic/lib/python3.11/site-packages/borgmatic/config/validate.py", line 112, in parse_configuration
    override.apply_overrides(config, schema, overrides)
  File "/root/.local/pipx/venvs/borgmatic/lib/python3.11/site-packages/borgmatic/config/override.py", line 129, in apply_overrides
    set_values(config, keys, value)
  File "/root/.local/pipx/venvs/borgmatic/lib/python3.11/site-packages/borgmatic/config/override.py", line 22, in set_values
    set_values(config[first_key], keys[1:], value)
  File "/root/.local/pipx/venvs/borgmatic/lib/python3.11/site-packages/borgmatic/config/override.py", line 16, in set_values
    config[first_key] = value
    ~~~~~~^^^^^^^^^^^
TypeError: list indices must be integers or slices, not str

Expected behavior

A repository check runs for 3600 seconds.

Other notes / implementation ideas

The borg version of this command runs fine.
borg check myrepository --repository-only --max-duration=3600

Config file

source_directories:
    - /home
    - /etc
    - /root
    - /opt
    - /var/spool/cron
    - /var/log/syslog*

repositories:
    - path: /my/path
      label: mylabel

exclude_patterns:
    - '/root/.config/borg/security/*/nonce'

exclude_caches: true

exclude_if_present:
    - .nobackup

encryption_passphrase: "some super secret passphrase"

retries: 3

retry_wait: 300

archive_name_format: '{hostname}-{now:%Y-%m-%dT%H:%M:%S}'

match_archives: sh:{hostname}-*

relocated_repo_access_is_ok: true

keep_daily: 7
keep_weekly: 5
keep_monthly: 2
keep_yearly: 0

checks:
    - name: archives
      frequency: 2 weeks
    - name: data
      frequency: 1 month

before_backup:
    - echo Starting a backup. $(date +"%T %Z")

before_prune:
    - echo Starting pruning. $(date +"%T %Z")

before_compact:
    - echo Starting compaction. $(date +"%T %Z")

before_check:
    - echo Starting checks. $(date +"%T %Z")
    
after_backup:
    - echo Finished a backup. $(date +"%T %Z")

after_compact:
    - echo Finished compaction. $(date +"%T %Z")

after_prune:
    - echo Finished pruning. $(date +"%T %Z")

after_check:
    - echo Finished checks. $(date +"%T %Z")

borgmatic version

1.8.6

borgmatic installation method

pip install

Borg version

1.2.4

Python version

3.11.2

Database version (if applicable)

No response

Operating system and version

Debian GNU/Linux 12 (bookworm)

### What I'm trying to do and why Hopefully this is a simple case of user error. I'm trying to pass the borg option of `--max-duration` into borgmatic with `extra_borg_options` , but it's not working as I would have hoped. ### Steps to reproduce Run `borgmatic check --override checks.name=repository --override extra_borg_options.check="--repository-only --max-duration=3600" --only repository` I've tried multiple variants of the command, with and without single and double quotations. ### Actual behavior This error is generated. ``` Traceback (most recent call last): File "/root/.local/bin/borgmatic", line 8, in <module> sys.exit(main()) ^^^^^^ File "/root/.local/pipx/venvs/borgmatic/lib/python3.11/site-packages/borgmatic/commands/borgmatic.py", line 859, in main configs, parse_logs = load_configurations( ^^^^^^^^^^^^^^^^^^^^ File "/root/.local/pipx/venvs/borgmatic/lib/python3.11/site-packages/borgmatic/commands/borgmatic.py", line 528, in load_configurations configs[config_filename], parse_logs = validate.parse_configuration( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/.local/pipx/venvs/borgmatic/lib/python3.11/site-packages/borgmatic/config/validate.py", line 112, in parse_configuration override.apply_overrides(config, schema, overrides) File "/root/.local/pipx/venvs/borgmatic/lib/python3.11/site-packages/borgmatic/config/override.py", line 129, in apply_overrides set_values(config, keys, value) File "/root/.local/pipx/venvs/borgmatic/lib/python3.11/site-packages/borgmatic/config/override.py", line 22, in set_values set_values(config[first_key], keys[1:], value) File "/root/.local/pipx/venvs/borgmatic/lib/python3.11/site-packages/borgmatic/config/override.py", line 16, in set_values config[first_key] = value ~~~~~~^^^^^^^^^^^ TypeError: list indices must be integers or slices, not str ``` ### Expected behavior A repository check runs for 3600 seconds. ### Other notes / implementation ideas The borg version of this command runs fine. `borg check myrepository --repository-only --max-duration=3600` Config file ``` source_directories: - /home - /etc - /root - /opt - /var/spool/cron - /var/log/syslog* repositories: - path: /my/path label: mylabel exclude_patterns: - '/root/.config/borg/security/*/nonce' exclude_caches: true exclude_if_present: - .nobackup encryption_passphrase: "some super secret passphrase" retries: 3 retry_wait: 300 archive_name_format: '{hostname}-{now:%Y-%m-%dT%H:%M:%S}' match_archives: sh:{hostname}-* relocated_repo_access_is_ok: true keep_daily: 7 keep_weekly: 5 keep_monthly: 2 keep_yearly: 0 checks: - name: archives frequency: 2 weeks - name: data frequency: 1 month before_backup: - echo Starting a backup. $(date +"%T %Z") before_prune: - echo Starting pruning. $(date +"%T %Z") before_compact: - echo Starting compaction. $(date +"%T %Z") before_check: - echo Starting checks. $(date +"%T %Z") after_backup: - echo Finished a backup. $(date +"%T %Z") after_compact: - echo Finished compaction. $(date +"%T %Z") after_prune: - echo Finished pruning. $(date +"%T %Z") after_check: - echo Finished checks. $(date +"%T %Z") ``` ### borgmatic version 1.8.6 ### borgmatic installation method pip install ### Borg version 1.2.4 ### Python version 3.11.2 ### Database version (if applicable) _No response_ ### Operating system and version Debian GNU/Linux 12 (bookworm)

witten commented

2024-01-18 18:11:14 +00:00

Thanks for providing the detailed bug report! It looks like there are at least two things wrong here, one with the command you're running and one with borgmatic itself:

When overriding an option that's a list (and in this case a list of mappings), you need to use the appropriate syntax on the command-line to match the configuration schema. So instead of --override checks.name=repository try --override checks=[{name: repository}]. Or even better, don't use --override at all and just use the borgmatic check --only flag, e.g. --only repository
borgmatic should never error with a traceback on "bad" user input. Instead, it should produce a (hopefully) more useful error message. So I'll see about fixing that.

Please let me know if one of the above solutions works for you! As far as I can tell, the problem you're seeing has nothing to do with your use of --max-duration. I'll note that you shouldn't need --repository-only because borgmatic will pass that to Borg implicitly if you go with one one of the solutions described above.

Also, what do you think of borgmatic supporting --max-duration natively, either as a configuration option or a command-line flag or both?

Thanks for providing the detailed bug report! It looks like there are at least two things wrong here, one with the command you're running and one with borgmatic itself: 1. When overriding an option that's a list (and in this case a list of mappings), you need to use the appropriate syntax on the command-line to match the configuration schema. So instead of `--override checks.name=repository` try `--override checks=[{name: repository}]`. Or even better, don't use `--override` at all and just use the `borgmatic check --only` flag, e.g. `--only repository` 2. borgmatic should never error with a traceback on "bad" user input. Instead, it should produce a (hopefully) more useful error message. So I'll see about fixing that. Please let me know if one of the above solutions works for you! As far as I can tell, the problem you're seeing has nothing to do with your use of `--max-duration`. I'll note that you shouldn't need `--repository-only` because borgmatic will pass that to Borg implicitly if you go with one one of the solutions described above. Also, what do you think of borgmatic supporting `--max-duration` natively, either as a configuration option or a command-line flag or both?

witten added the

bug

label 2024-01-18 18:16:44 +00:00

witten referenced this issue from a commit

2024-01-18 18:39:54 +00:00

Fix a traceback when providing an invalid "--override" value for a list option (#814).

tehNiemer commented

2024-01-19 12:54:29 +00:00

Thanks! This seems to do what I want borgmatic check --only repository --override extra_borg_options.check="--max-duration=3600". I appreciate your help.

The command-line reference documentation indicates that overrides should be invoked by --override OPTION.SUBOPTION=VALUE which is why I formatted it the way I did. Not sure if the documentation needs to be updated or I interpreted it incorrectly, I'm not a power user by any means.

I also included the --override call because I don't have a repository check in my config file, so I thought I would have to include that in the command line. What I now gather from your comment is that the command line options will add functionality not included in the config file.

I think supporting --max-duration natively in both/either the config and command line won't hurt anything, I personally would like to be able to run any borg command from borgmatic natively, but that's probably a tall order. The only potential hiccup I see with --max-duration is that it can only be run on repositories, so if there were additional checks in the configuration, they would have to be called separately from repository checks within borgmatic.

Thanks! This seems to do what I want `borgmatic check --only repository --override extra_borg_options.check="--max-duration=3600"`. I appreciate your help. The command-line reference documentation indicates that overrides should be invoked by `--override OPTION.SUBOPTION=VALUE` which is why I formatted it the way I did. Not sure if the documentation needs to be updated or I interpreted it incorrectly, I'm not a power user by any means. I also included the `--override` call because I don't have a repository check in my config file, so I thought I would have to include that in the command line. What I now gather from your comment is that the command line options will add functionality not included in the config file. I think supporting `--max-duration` natively in both/either the config and command line won't hurt anything, I personally would like to be able to run any borg command from borgmatic natively, but that's probably a tall order. The only potential hiccup I see with `--max-duration` is that it can only be run on repositories, so if there were additional checks in the configuration, they would have to be called separately from repository checks within borgmatic.

tehNiemer closed this issue

2024-01-19 12:54:43 +00:00

witten referenced this issue

2024-01-19 19:46:20 +00:00

borg check --max-duration support #817

witten commented

2024-01-19 19:49:12 +00:00

Thanks, that makes sense! I've filed #817 for the native --max-duration feature. Feel free to follow along there.

The command-line reference documentation indicates that overrides should be invoked by --override OPTION.SUBOPTION=VALUE which is why I formatted it the way I did. Not sure if the documentation needs to be updated or I interpreted it incorrectly, I'm not a power user by any means.

Yeah, the command-line help necessarily isn't as extensive as the online docs: https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/#configuration-overrides

But maybe the command-line help can be expanded a bit.

Thanks, that makes sense! I've filed #817 for the native `--max-duration` feature. Feel free to follow along there. > The command-line reference documentation indicates that overrides should be invoked by --override OPTION.SUBOPTION=VALUE which is why I formatted it the way I did. Not sure if the documentation needs to be updated or I interpreted it incorrectly, I'm not a power user by any means. Yeah, the command-line help necessarily isn't as extensive as the online docs: https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/#configuration-overrides But maybe the command-line help can be expanded a bit.

witten referenced this issue from a commit

2024-01-19 19:55:52 +00:00

Clarify "--override" command-line help (#814)

witten commented

2024-01-21 22:57:44 +00:00

Released in borgmatic 1.8.7!