Invalid override value when using command-line override to specify a remote repository location #528

New Issue

Closed

opened 2022-05-06 11:01:54 +00:00 by noresistence · 4 comments

noresistence commented 2022-05-06 11:01:54 +00:00

Copy Link

What I'm trying to do and why

I try to use override option: --override location.repositories=[ssh://borg@server:1234/~/backup_repository].

We have two external storage repositories configured, for redundancy. But as soon as one of these is not available or fails for some reason, borgmatic would fail (instead of continuing with the next repository before failing).

What we actually want is separate failures for separate repositories, and as far as I understand the documentation of borgmatic, this behaviour is not supported out of the box. For this reason, we call borgmatic for each remote repository separately.

Steps to reproduce (if a bug)

generate-borgmatic-config --destination /etc/borgmatic.d/myconfig.yaml
borgmatic prune --override location.repositories=[ssh://borg@server:1234/~/backup_repository]

Actual behavior (if a bug)

Output:

/etc/borgmatic/config.yaml /etc/borgmatic.d /root/.config/borgmatic/config.yaml /root/.config/borgmatic.d: No valid configuration files found

summary:
/etc/borgmatic.d/myconfig.yaml: Error parsing configuration file
Invalid override value: while scanning a plain scalar
  in "<file>", line 1, column 2
found unexpected ':'
  in "<file>", line 1, column 5

[... repeat for all config files in /etc/borgmatic.d ...]

/etc/borgmatic/config.yaml /etc/borgmatic.d /root/.config/borgmatic/config.yaml /root/.config/borgmatic.d: No valid configuration files found

Expected behavior (if a bug)

It should be possible to use the --override.location.repositories option with a list of values that contain a colon.

Furthermore, the error message in "<file>" was confusing, as my actual configs also uses includes to merge settings from a common configuration file (and i initially thought the culprit would be an override that happens during config merge). I spent a lot of time trying to figure out how these could have a syntax error while validate-borgmatic-config does not complain. The new message since borgmatic 1.5.22 helps already, but the message in "<command-line override>" would have been even more helpful to distinguish from config merge.

Other notes / implementation ideas

This did work in the past with borgmatic 1.5.18, but after a fresh install with the same version, it suddenly started to fail although I did not change the scripts.

For this reason, I suspect a problem with YAML parsing that starts to appear with some recent (well, more recent than the two-year-old setup that worked) version of a YAML parsing library or other dependencies.

Environment

borgmatic version: 1.6.0 (as well as 1.5.18 and 1.5.24)

borgmatic installation method: custom docker image

Borg version: 1.1.11

Python version: 3.8.10

pip freeze:

appdirs==1.4.4
attrs==21.4.0
borgbackup==1.1.11
borgmatic==1.6.0
CacheControl==0.12.6
certifi==2020.4.5.1
chardet==3.0.4
colorama==0.4.3
contextlib2==0.6.0
distlib==0.3.0
distro==1.5.0
html5lib==1.0.1
idna==2.9
importlib-resources==5.7.1
jsonschema==4.5.1
lockfile==0.12.2
msgpack==1.0.0
ordered-set==4.0.1
packaging==20.4
pep517==0.8.2
progress==1.5
pyparsing==2.4.7
pyrsistent==0.18.1
pytoml==0.1.21
pyzmq==18.1.1
requests==2.23.0
retrying==1.3.3
ruamel.yaml==0.16.10
ruamel.yaml.clib==0.2.0
six==1.15.0
toml==0.10.1
urllib3==1.25.9
webencodings==0.5.1
zipp==3.8.0

operating system and version: Docker image based on Alpine Linux

#### What I'm trying to do and why I try to use override option: `--override location.repositories=[ssh://borg@server:1234/~/backup_repository]`. We have two external storage repositories configured, for redundancy. But as soon as one of these is not available or fails for some reason, borgmatic would fail (instead of continuing with the next repository before failing). What we actually want is separate failures for separate repositories, and as far as I understand the documentation of borgmatic, this behaviour is not supported out of the box. For this reason, we call borgmatic for each remote repository separately. #### Steps to reproduce (if a bug) ``` generate-borgmatic-config --destination /etc/borgmatic.d/myconfig.yaml borgmatic prune --override location.repositories=[ssh://borg@server:1234/~/backup_repository] ``` #### Actual behavior (if a bug) Output: ``` /etc/borgmatic/config.yaml /etc/borgmatic.d /root/.config/borgmatic/config.yaml /root/.config/borgmatic.d: No valid configuration files found summary: /etc/borgmatic.d/myconfig.yaml: Error parsing configuration file Invalid override value: while scanning a plain scalar in "<file>", line 1, column 2 found unexpected ':' in "<file>", line 1, column 5 [... repeat for all config files in /etc/borgmatic.d ...] /etc/borgmatic/config.yaml /etc/borgmatic.d /root/.config/borgmatic/config.yaml /root/.config/borgmatic.d: No valid configuration files found ``` #### Expected behavior (if a bug) It should be possible to use the `--override.location.repositories` option with a list of values that contain a colon. Furthermore, the error message `in "<file>"` was confusing, as my actual configs also uses includes to merge settings from a common configuration file (and i initially thought the culprit would be an override that happens during config merge). I spent a lot of time trying to figure out how these could have a syntax error while `validate-borgmatic-config` does not complain. The new message since borgmatic 1.5.22 helps already, but the message `in "<command-line override>"` would have been even more helpful to distinguish from config merge. #### Other notes / implementation ideas This did work in the past with borgmatic 1.5.18, but after a fresh install with the same version, it suddenly started to fail although I did not change the scripts. For this reason, I suspect a problem with YAML parsing that starts to appear with some recent (well, more recent than the two-year-old setup that worked) version of a YAML parsing library or other dependencies. #### Environment **borgmatic version:** 1.6.0 (as well as 1.5.18 and 1.5.24) **borgmatic installation method:** custom docker image **Borg version:** 1.1.11 **Python version:** 3.8.10 **pip freeze:** ``` appdirs==1.4.4 attrs==21.4.0 borgbackup==1.1.11 borgmatic==1.6.0 CacheControl==0.12.6 certifi==2020.4.5.1 chardet==3.0.4 colorama==0.4.3 contextlib2==0.6.0 distlib==0.3.0 distro==1.5.0 html5lib==1.0.1 idna==2.9 importlib-resources==5.7.1 jsonschema==4.5.1 lockfile==0.12.2 msgpack==1.0.0 ordered-set==4.0.1 packaging==20.4 pep517==0.8.2 progress==1.5 pyparsing==2.4.7 pyrsistent==0.18.1 pytoml==0.1.21 pyzmq==18.1.1 requests==2.23.0 retrying==1.3.3 ruamel.yaml==0.16.10 ruamel.yaml.clib==0.2.0 six==1.15.0 toml==0.10.1 urllib3==1.25.9 webencodings==0.5.1 zipp==3.8.0 ``` **operating system and version:** Docker image based on Alpine Linux

noresistence commented 2022-05-06 11:24:34 +00:00

Author

Copy Link

i guess ruamel.yaml expects the remote server repository string to be enclosed in quotes. This is not obvious from the examples in the documentation.

python3
>>> import ruamel.yaml
>>> override = "[ssh://borg@server:1234/~/abc]"
>>> import io
>>> ruamel.yaml.YAML(typ='safe').load(io.StringIO(override))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.8/site-packages/ruamel/yaml/main.py", line 343, in load
    return constructor.get_single_data()
  File "/usr/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 111, in get_single_data
    node = self.composer.get_single_node()
  File "_ruamel_yaml.pyx", line 703, in _ruamel_yaml.CParser.get_single_node
  File "_ruamel_yaml.pyx", line 904, in _ruamel_yaml.CParser._parse_next_event
ruamel.yaml.scanner.ScannerError: while scanning a plain scalar
  in "<file>", line 1, column 2
found unexpected ':'
  in "<file>", line 1, column 5
>>> ruamel.yaml.YAML(typ='safe').load(io.StringIO("[]"))
[]
>>> ruamel.yaml.YAML(typ='safe').load(io.StringIO("[a,bc,d]"))
['a', 'bc', 'd']
>>> ruamel.yaml.YAML(typ='safe').load(io.StringIO("['ssh://abc']"))
['ssh://abc']
>>> ruamel.yaml.YAML(typ='safe').load(io.StringIO('["ssh://abc"]'))
['ssh://abc']
>>> ruamel.yaml.YAML(typ='safe').load(io.StringIO('["ssh://borg@server:1234"]'))['ssh://borg@server:1234']

But using a double-quoted or single-quoted string in the command does not help (i did try that):

borgmatic prune --override location.repositories=["ssh://borg@server:1234/~/
backup_repository"] --verbosity 2
# fails again
borgmatic prune --override location.repositories=['ssh://borg@server:1234/~/
backup_repository'] --verbosity 2
# and again

i guess `ruamel.yaml` expects the remote server repository string to be enclosed in quotes. This is not obvious from the examples in the documentation. ``` python3 >>> import ruamel.yaml >>> override = "[ssh://borg@server:1234/~/abc]" >>> import io >>> ruamel.yaml.YAML(typ='safe').load(io.StringIO(override)) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3.8/site-packages/ruamel/yaml/main.py", line 343, in load return constructor.get_single_data() File "/usr/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 111, in get_single_data node = self.composer.get_single_node() File "_ruamel_yaml.pyx", line 703, in _ruamel_yaml.CParser.get_single_node File "_ruamel_yaml.pyx", line 904, in _ruamel_yaml.CParser._parse_next_event ruamel.yaml.scanner.ScannerError: while scanning a plain scalar in "<file>", line 1, column 2 found unexpected ':' in "<file>", line 1, column 5 >>> ruamel.yaml.YAML(typ='safe').load(io.StringIO("[]")) [] >>> ruamel.yaml.YAML(typ='safe').load(io.StringIO("[a,bc,d]")) ['a', 'bc', 'd'] >>> ruamel.yaml.YAML(typ='safe').load(io.StringIO("['ssh://abc']")) ['ssh://abc'] >>> ruamel.yaml.YAML(typ='safe').load(io.StringIO('["ssh://abc"]')) ['ssh://abc'] >>> ruamel.yaml.YAML(typ='safe').load(io.StringIO('["ssh://borg@server:1234"]'))['ssh://borg@server:1234'] ``` But using a double-quoted or single-quoted string in the command does not help (i did try that): ``` borgmatic prune --override location.repositories=["ssh://borg@server:1234/~/ backup_repository"] --verbosity 2 # fails again borgmatic prune --override location.repositories=['ssh://borg@server:1234/~/ backup_repository'] --verbosity 2 # and again ```

noresistence commented 2022-05-06 11:30:08 +00:00

Author

Copy Link

Workaround: Enclose both the repository url and the whole array in quotes:

borgmatic prune --override location.repositories="['ssh://borg@server:1234/~
/backup_repository']" --verbosity 2

This works. (Quoting either repository url or whole array, but not the repository url string, does not work)

It would be helpful to update the examples in https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/#configuration-overrides accordingly :)

Workaround: Enclose _both_ the repository url _and_ the whole array in quotes: ``` borgmatic prune --override location.repositories="['ssh://borg@server:1234/~ /backup_repository']" --verbosity 2 ``` This works. (Quoting _either_ repository url _or_ whole array, but not the repository url string, does not work) It would be helpful to update the examples in https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/#configuration-overrides accordingly :)

witten referenced this issue from a commit 2022-05-20 20:39:13 +00:00

Improve the error message when a configuration override contains an invalid value (#528).

witten commented 2022-05-20 20:40:32 +00:00

Owner

Copy Link

Thank you for detailed ticket and all the diagnosing! Super helpful. I've improved the error message and also updated the documentation as you suggest (update will be live momentarily). Here's an example of the new error message:

# borgmatic -c test.yaml -v 1 create --override location.repositories=[ssh://borg@server:1234/~/backup_repository]
test.yaml: No valid configuration files found

summary:
test.yaml: Error parsing configuration file
Invalid override 'location.repositories=[ssh://borg@server:1234/~/backup_repository]': found unexpected ':'
test.yaml: No valid configuration files found

Need some help? https://torsion.org/borgmatic/#issues

This is currently in the master and will be part of the next release.

Thank you for detailed ticket and all the diagnosing! Super helpful. I've improved the error message and also updated the documentation as you suggest (update will be live momentarily). Here's an example of the new error message: ```bash # borgmatic -c test.yaml -v 1 create --override location.repositories=[ssh://borg@server:1234/~/backup_repository] test.yaml: No valid configuration files found summary: test.yaml: Error parsing configuration file Invalid override 'location.repositories=[ssh://borg@server:1234/~/backup_repository]': found unexpected ':' test.yaml: No valid configuration files found Need some help? https://torsion.org/borgmatic/#issues ``` This is currently in the master and will be part of the next release.

witten closed this issue 2022-05-20 20:40:32 +00:00

witten commented 2022-05-24 23:26:28 +00:00

Owner

Copy Link

Released in borgmatic 1.6.1!

Released in borgmatic 1.6.1!

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

spot-check

1.8.11

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.24

1.5.23

1.5.22

1.5.21

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.14

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.7.dev0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.22

1.4.21

1.4.20

1.4.19

1.4.18

1.4.17

1.4.16

1.4.15

1.4.14

1.4.13

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.26

1.3.25

1.3.24

1.3.23

1.3.22

1.3.21

1.3.20

1.3.19

1.3.18

1.3.17

1.3.16

1.3.15

1.3.14

1.3.13

1.3.12

1.3.11

1.3.10

1.3.9

1.3.8

1.3.7

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.18

1.2.17

1.2.16

1.2.15

1.2.14

1.2.13

1.2.12

1.2.11

1.2.10

1.2.9

1.2.8

1.2.7

1.2.6

1.2.5

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.15

1.1.14

1.1.13

1.1.12

1.1.11

1.1.10

1.1.9

1.1.7

1.1.6

1.1.5

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.3

1.0.2

1.0.1

1.0.0

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

0.0.2

Labels

Clear labels   

bug

  

data loss

  

design finalized

  

good first issue

  

new feature area

  

question / support

  

security

  

waiting for response

No Label

bug

data loss

design finalized

good first issue

new feature area

question / support

security

waiting for response

Milestone

Clear milestone

No items

No Milestone

Assignees

Clear assignees

No Assignees

2 Participants

Notifications

Subscribe

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#528

No description provided.