Invalid override value when using command-line override to specify a remote repository location #528

Closed
opened 3 months ago by noresistence · 4 comments

What I'm trying to do and why

I try to use override option: --override location.repositories=[ssh://borg@server:1234/~/backup_repository].

We have two external storage repositories configured, for redundancy. But as soon as one of these is not available or fails for some reason, borgmatic would fail (instead of continuing with the next repository before failing).

What we actually want is separate failures for separate repositories, and as far as I understand the documentation of borgmatic, this behaviour is not supported out of the box. For this reason, we call borgmatic for each remote repository separately.

Steps to reproduce (if a bug)

generate-borgmatic-config --destination /etc/borgmatic.d/myconfig.yaml
borgmatic prune --override location.repositories=[ssh://borg@server:1234/~/backup_repository]

Actual behavior (if a bug)

Output:

/etc/borgmatic/config.yaml /etc/borgmatic.d /root/.config/borgmatic/config.yaml /root/.config/borgmatic.d: No valid configuration files found

summary:
/etc/borgmatic.d/myconfig.yaml: Error parsing configuration file
Invalid override value: while scanning a plain scalar
  in "<file>", line 1, column 2
found unexpected ':'
  in "<file>", line 1, column 5

[... repeat for all config files in /etc/borgmatic.d ...]

/etc/borgmatic/config.yaml /etc/borgmatic.d /root/.config/borgmatic/config.yaml /root/.config/borgmatic.d: No valid configuration files found

Expected behavior (if a bug)

It should be possible to use the --override.location.repositories option with a list of values that contain a colon.

Furthermore, the error message in "<file>" was confusing, as my actual configs also uses includes to merge settings from a common configuration file (and i initially thought the culprit would be an override that happens during config merge). I spent a lot of time trying to figure out how these could have a syntax error while validate-borgmatic-config does not complain. The new message since borgmatic 1.5.22 helps already, but the message in "<command-line override>" would have been even more helpful to distinguish from config merge.

Other notes / implementation ideas

This did work in the past with borgmatic 1.5.18, but after a fresh install with the same version, it suddenly started to fail although I did not change the scripts.

For this reason, I suspect a problem with YAML parsing that starts to appear with some recent (well, more recent than the two-year-old setup that worked) version of a YAML parsing library or other dependencies.

Environment

borgmatic version: 1.6.0 (as well as 1.5.18 and 1.5.24)

borgmatic installation method: custom docker image

Borg version: 1.1.11

Python version: 3.8.10

pip freeze:

appdirs==1.4.4
attrs==21.4.0
borgbackup==1.1.11
borgmatic==1.6.0
CacheControl==0.12.6
certifi==2020.4.5.1
chardet==3.0.4
colorama==0.4.3
contextlib2==0.6.0
distlib==0.3.0
distro==1.5.0
html5lib==1.0.1
idna==2.9
importlib-resources==5.7.1
jsonschema==4.5.1
lockfile==0.12.2
msgpack==1.0.0
ordered-set==4.0.1
packaging==20.4
pep517==0.8.2
progress==1.5
pyparsing==2.4.7
pyrsistent==0.18.1
pytoml==0.1.21
pyzmq==18.1.1
requests==2.23.0
retrying==1.3.3
ruamel.yaml==0.16.10
ruamel.yaml.clib==0.2.0
six==1.15.0
toml==0.10.1
urllib3==1.25.9
webencodings==0.5.1
zipp==3.8.0

operating system and version: Docker image based on Alpine Linux

#### What I'm trying to do and why I try to use override option: `--override location.repositories=[ssh://borg@server:1234/~/backup_repository]`. We have two external storage repositories configured, for redundancy. But as soon as one of these is not available or fails for some reason, borgmatic would fail (instead of continuing with the next repository before failing). What we actually want is separate failures for separate repositories, and as far as I understand the documentation of borgmatic, this behaviour is not supported out of the box. For this reason, we call borgmatic for each remote repository separately. #### Steps to reproduce (if a bug) ``` generate-borgmatic-config --destination /etc/borgmatic.d/myconfig.yaml borgmatic prune --override location.repositories=[ssh://borg@server:1234/~/backup_repository] ``` #### Actual behavior (if a bug) Output: ``` /etc/borgmatic/config.yaml /etc/borgmatic.d /root/.config/borgmatic/config.yaml /root/.config/borgmatic.d: No valid configuration files found summary: /etc/borgmatic.d/myconfig.yaml: Error parsing configuration file Invalid override value: while scanning a plain scalar in "<file>", line 1, column 2 found unexpected ':' in "<file>", line 1, column 5 [... repeat for all config files in /etc/borgmatic.d ...] /etc/borgmatic/config.yaml /etc/borgmatic.d /root/.config/borgmatic/config.yaml /root/.config/borgmatic.d: No valid configuration files found ``` #### Expected behavior (if a bug) It should be possible to use the `--override.location.repositories` option with a list of values that contain a colon. Furthermore, the error message `in "<file>"` was confusing, as my actual configs also uses includes to merge settings from a common configuration file (and i initially thought the culprit would be an override that happens during config merge). I spent a lot of time trying to figure out how these could have a syntax error while `validate-borgmatic-config` does not complain. The new message since borgmatic 1.5.22 helps already, but the message `in "<command-line override>"` would have been even more helpful to distinguish from config merge. #### Other notes / implementation ideas This did work in the past with borgmatic 1.5.18, but after a fresh install with the same version, it suddenly started to fail although I did not change the scripts. For this reason, I suspect a problem with YAML parsing that starts to appear with some recent (well, more recent than the two-year-old setup that worked) version of a YAML parsing library or other dependencies. #### Environment **borgmatic version:** 1.6.0 (as well as 1.5.18 and 1.5.24) **borgmatic installation method:** custom docker image **Borg version:** 1.1.11 **Python version:** 3.8.10 **pip freeze:** ``` appdirs==1.4.4 attrs==21.4.0 borgbackup==1.1.11 borgmatic==1.6.0 CacheControl==0.12.6 certifi==2020.4.5.1 chardet==3.0.4 colorama==0.4.3 contextlib2==0.6.0 distlib==0.3.0 distro==1.5.0 html5lib==1.0.1 idna==2.9 importlib-resources==5.7.1 jsonschema==4.5.1 lockfile==0.12.2 msgpack==1.0.0 ordered-set==4.0.1 packaging==20.4 pep517==0.8.2 progress==1.5 pyparsing==2.4.7 pyrsistent==0.18.1 pytoml==0.1.21 pyzmq==18.1.1 requests==2.23.0 retrying==1.3.3 ruamel.yaml==0.16.10 ruamel.yaml.clib==0.2.0 six==1.15.0 toml==0.10.1 urllib3==1.25.9 webencodings==0.5.1 zipp==3.8.0 ``` **operating system and version:** Docker image based on Alpine Linux

i guess ruamel.yaml expects the remote server repository string to be enclosed in quotes. This is not obvious from the examples in the documentation.

python3
>>> import ruamel.yaml
>>> override = "[ssh://borg@server:1234/~/abc]"
>>> import io
>>> ruamel.yaml.YAML(typ='safe').load(io.StringIO(override))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.8/site-packages/ruamel/yaml/main.py", line 343, in load
    return constructor.get_single_data()
  File "/usr/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 111, in get_single_data
    node = self.composer.get_single_node()
  File "_ruamel_yaml.pyx", line 703, in _ruamel_yaml.CParser.get_single_node
  File "_ruamel_yaml.pyx", line 904, in _ruamel_yaml.CParser._parse_next_event
ruamel.yaml.scanner.ScannerError: while scanning a plain scalar
  in "<file>", line 1, column 2
found unexpected ':'
  in "<file>", line 1, column 5
>>> ruamel.yaml.YAML(typ='safe').load(io.StringIO("[]"))
[]
>>> ruamel.yaml.YAML(typ='safe').load(io.StringIO("[a,bc,d]"))
['a', 'bc', 'd']
>>> ruamel.yaml.YAML(typ='safe').load(io.StringIO("['ssh://abc']"))
['ssh://abc']
>>> ruamel.yaml.YAML(typ='safe').load(io.StringIO('["ssh://abc"]'))
['ssh://abc']
>>> ruamel.yaml.YAML(typ='safe').load(io.StringIO('["ssh://borg@server:1234"]'))['ssh://borg@server:1234']

But using a double-quoted or single-quoted string in the command does not help (i did try that):

borgmatic prune --override location.repositories=["ssh://borg@server:1234/~/
backup_repository"] --verbosity 2
# fails again
borgmatic prune --override location.repositories=['ssh://borg@server:1234/~/
backup_repository'] --verbosity 2
# and again
i guess `ruamel.yaml` expects the remote server repository string to be enclosed in quotes. This is not obvious from the examples in the documentation. ``` python3 >>> import ruamel.yaml >>> override = "[ssh://borg@server:1234/~/abc]" >>> import io >>> ruamel.yaml.YAML(typ='safe').load(io.StringIO(override)) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3.8/site-packages/ruamel/yaml/main.py", line 343, in load return constructor.get_single_data() File "/usr/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 111, in get_single_data node = self.composer.get_single_node() File "_ruamel_yaml.pyx", line 703, in _ruamel_yaml.CParser.get_single_node File "_ruamel_yaml.pyx", line 904, in _ruamel_yaml.CParser._parse_next_event ruamel.yaml.scanner.ScannerError: while scanning a plain scalar in "<file>", line 1, column 2 found unexpected ':' in "<file>", line 1, column 5 >>> ruamel.yaml.YAML(typ='safe').load(io.StringIO("[]")) [] >>> ruamel.yaml.YAML(typ='safe').load(io.StringIO("[a,bc,d]")) ['a', 'bc', 'd'] >>> ruamel.yaml.YAML(typ='safe').load(io.StringIO("['ssh://abc']")) ['ssh://abc'] >>> ruamel.yaml.YAML(typ='safe').load(io.StringIO('["ssh://abc"]')) ['ssh://abc'] >>> ruamel.yaml.YAML(typ='safe').load(io.StringIO('["ssh://borg@server:1234"]'))['ssh://borg@server:1234'] ``` But using a double-quoted or single-quoted string in the command does not help (i did try that): ``` borgmatic prune --override location.repositories=["ssh://borg@server:1234/~/ backup_repository"] --verbosity 2 # fails again borgmatic prune --override location.repositories=['ssh://borg@server:1234/~/ backup_repository'] --verbosity 2 # and again ```

Workaround: Enclose both the repository url and the whole array in quotes:

borgmatic prune --override location.repositories="['ssh://borg@server:1234/~
/backup_repository']" --verbosity 2

This works. (Quoting either repository url or whole array, but not the repository url string, does not work)

It would be helpful to update the examples in https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/#configuration-overrides accordingly :)

Workaround: Enclose _both_ the repository url _and_ the whole array in quotes: ``` borgmatic prune --override location.repositories="['ssh://borg@server:1234/~ /backup_repository']" --verbosity 2 ``` This works. (Quoting _either_ repository url _or_ whole array, but not the repository url string, does not work) It would be helpful to update the examples in https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/#configuration-overrides accordingly :)
Owner

Thank you for detailed ticket and all the diagnosing! Super helpful. I've improved the error message and also updated the documentation as you suggest (update will be live momentarily). Here's an example of the new error message:

# borgmatic -c test.yaml -v 1 create --override location.repositories=[ssh://borg@server:1234/~/backup_repository]
test.yaml: No valid configuration files found

summary:
test.yaml: Error parsing configuration file
Invalid override 'location.repositories=[ssh://borg@server:1234/~/backup_repository]': found unexpected ':'
test.yaml: No valid configuration files found

Need some help? https://torsion.org/borgmatic/#issues

This is currently in the master and will be part of the next release.

Thank you for detailed ticket and all the diagnosing! Super helpful. I've improved the error message and also updated the documentation as you suggest (update will be live momentarily). Here's an example of the new error message: ```bash # borgmatic -c test.yaml -v 1 create --override location.repositories=[ssh://borg@server:1234/~/backup_repository] test.yaml: No valid configuration files found summary: test.yaml: Error parsing configuration file Invalid override 'location.repositories=[ssh://borg@server:1234/~/backup_repository]': found unexpected ':' test.yaml: No valid configuration files found Need some help? https://torsion.org/borgmatic/#issues ``` This is currently in the master and will be part of the next release.
witten closed this issue 3 months ago
Owner

Released in borgmatic 1.6.1!

Released in borgmatic 1.6.1!
Sign in to join this conversation.
No Milestone
No Assignees
2 Participants
Notifications
Due Date

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#528
Loading…
There is no content yet.