borgmatic-collective/borgmatic
borgmatic-collective/borgmatic
Watch
9
74
Fork
You've already forked borgmatic
 57
Code Issues 71 Pull requests 3 Releases 162 Packages 1 Activity Actions 1

Merging multiple include files leads to validation error #380

New issue
Closed
opened 2020-12-08 12:32:38 +00:00 by jefferyto · 4 comments
jefferyto commented 2020-12-08 12:32:38 +00:00
Contributor
Copy link

What I'm trying to do and why

My config files are split into multiple includes (as a consequence of working around #281). I'm also using includes to isolate passphrases from config files (so that only the passphrase include files need to be restricted to root access only). So I am trying to merge two include files under storage.

Steps to reproduce (if a bug)

  1. Set up a config file with something like:

    storage:
    <<: !include /etc/borgmatic.d/common/storage.yaml
    <<: !include /etc/borgamtic.d/passphrases/secret.yaml

  2. Run validate-borgmatic-config.

Actual behavior (if a bug)

validate-borgmatic-config returns an error:

Traceback (most recent call last):
  File "/usr/bin/validate-borgmatic-config", line 11, in <module>
    load_entry_point('borgmatic==1.5.1', 'console_scripts', 'validate-borgmatic-config')()
  File "/usr/lib/python3/dist-packages/borgmatic/commands/validate_config.py", line 45, in main
    validate.parse_configuration(config_filename, validate.schema_filename())
  File "/usr/lib/python3/dist-packages/borgmatic/config/validate.py", line 101, in parse_configuration
    config = load.load_configuration(config_filename)
  File "/usr/lib/python3/dist-packages/borgmatic/config/load.py", line 20, in load_configuration
    return yaml.load(open(filename))
  File "/usr/lib/python3/dist-packages/ruamel/yaml/main.py", line 348, in load
    return constructor.get_single_data()
  File "/usr/lib/python3/dist-packages/ruamel/yaml/constructor.py", line 113, in get_single_data
    return self.construct_document(node)
  File "/usr/lib/python3/dist-packages/ruamel/yaml/constructor.py", line 123, in construct_document
    for _dummy in generator:
  File "/usr/lib/python3/dist-packages/ruamel/yaml/constructor.py", line 723, in construct_yaml_map
    value = self.construct_mapping(node)
  File "/usr/lib/python3/dist-packages/ruamel/yaml/constructor.py", line 439, in construct_mapping
    self.flatten_mapping(node)
  File "/usr/lib/python3/dist-packages/borgmatic/config/load.py", line 59, in flatten_mapping
    super(Include_constructor, self).flatten_mapping(node)
  File "/usr/lib/python3/dist-packages/ruamel/yaml/constructor.py", line 396, in flatten_mapping
    raise DuplicateKeyError(*args)
ruamel.yaml.constructor.DuplicateKeyError: while constructing a mapping
  in "/etc/borgmatic.d/example.yaml", line 8, column 5
found duplicate key "<<"
  in "/etc/borgmatic.d/example.yaml", line 9, column 5

To suppress this check see:
   http://yaml.readthedocs.io/en/latest/api.html#duplicate-keys

Duplicate keys will become an error in future releases, and are errors
by default when using the new API.

Expected behavior (if a bug)

I can merge multiple include files.

Other notes / implementation ideas

I've tried multiple plain !include includes, and mixing one !include and one <<: !include, but none of these validate either.

Environment

borgmatic version: 1.5.1

borgmatic installation method: Ubuntu package

Borg version: 1.1.14

Python version: 3.8.6

Database version (if applicable): N/A

operating system and version: Ubuntu 20.10

#### What I'm trying to do and why My config files are split into multiple includes (as a consequence of working around #281). I'm also using includes to isolate passphrases from config files (so that only the passphrase include files need to be restricted to root access only). So I am trying to merge two include files under `storage`. #### Steps to reproduce (if a bug) 1. Set up a config file with something like: ```yaml storage: <<: !include /etc/borgmatic.d/common/storage.yaml <<: !include /etc/borgamtic.d/passphrases/secret.yaml ``` 2. Run `validate-borgmatic-config`. #### Actual behavior (if a bug) `validate-borgmatic-config` returns an error: ``` Traceback (most recent call last): File "/usr/bin/validate-borgmatic-config", line 11, in <module> load_entry_point('borgmatic==1.5.1', 'console_scripts', 'validate-borgmatic-config')() File "/usr/lib/python3/dist-packages/borgmatic/commands/validate_config.py", line 45, in main validate.parse_configuration(config_filename, validate.schema_filename()) File "/usr/lib/python3/dist-packages/borgmatic/config/validate.py", line 101, in parse_configuration config = load.load_configuration(config_filename) File "/usr/lib/python3/dist-packages/borgmatic/config/load.py", line 20, in load_configuration return yaml.load(open(filename)) File "/usr/lib/python3/dist-packages/ruamel/yaml/main.py", line 348, in load return constructor.get_single_data() File "/usr/lib/python3/dist-packages/ruamel/yaml/constructor.py", line 113, in get_single_data return self.construct_document(node) File "/usr/lib/python3/dist-packages/ruamel/yaml/constructor.py", line 123, in construct_document for _dummy in generator: File "/usr/lib/python3/dist-packages/ruamel/yaml/constructor.py", line 723, in construct_yaml_map value = self.construct_mapping(node) File "/usr/lib/python3/dist-packages/ruamel/yaml/constructor.py", line 439, in construct_mapping self.flatten_mapping(node) File "/usr/lib/python3/dist-packages/borgmatic/config/load.py", line 59, in flatten_mapping super(Include_constructor, self).flatten_mapping(node) File "/usr/lib/python3/dist-packages/ruamel/yaml/constructor.py", line 396, in flatten_mapping raise DuplicateKeyError(*args) ruamel.yaml.constructor.DuplicateKeyError: while constructing a mapping in "/etc/borgmatic.d/example.yaml", line 8, column 5 found duplicate key "<<" in "/etc/borgmatic.d/example.yaml", line 9, column 5 To suppress this check see: http://yaml.readthedocs.io/en/latest/api.html#duplicate-keys Duplicate keys will become an error in future releases, and are errors by default when using the new API. ``` #### Expected behavior (if a bug) I can merge multiple include files. #### Other notes / implementation ideas I've tried multiple plain `!include` includes, and mixing one `!include` and one `<<: !include`, but none of these validate either. #### Environment **borgmatic version:** 1.5.1 **borgmatic installation method:** Ubuntu package **Borg version:** 1.1.14 **Python version:** 3.8.6 **Database version (if applicable):** N/A **operating system and version:** Ubuntu 20.10
witten commented 2022-04-26 04:30:04 +00:00
Owner
Copy link

Sorry for the lengthy delay here. I think this behavior is a limitation of the YAML parsing library borgmatic uses—and possibly of YAML itself.

However, with #381 (deep merging) now implemented in master, it's possible that you'll be able to get rid of all of your per-section includes and move those to a common include for the whole file. Example:

<<: !include /etc/borgmatic.d/common/common.yaml

storage:
    <<: !include /etc/borgamtic.d/passphrases/secret.yaml

Does that seem like it'd work for your use case?

Sorry for the lengthy delay here. I think this behavior is a limitation of the YAML parsing library borgmatic uses—and possibly of YAML itself. However, with #381 (deep merging) now implemented in master, it's possible that you'll be able to get rid of all of your per-section includes and move those to a common include for the whole file. Example: ``` <<: !include /etc/borgmatic.d/common/common.yaml storage: <<: !include /etc/borgamtic.d/passphrases/secret.yaml ``` Does that seem like it'd work for your use case?
witten commented 2022-05-23 23:17:21 +00:00
Owner
Copy link

Closing for now given the work-around, but please feel free to re-open if you have additional follow-up. Thanks!

Closing for now given the work-around, but please feel free to re-open if you have additional follow-up. Thanks!
jefferyto commented 2022-05-26 06:02:32 +00:00
Author
Contributor
Copy link

Apologies for the late reply. I haven't tried the new deep merging includes yet so I'm not sure how it interacts with other includes (my setup is more complicated than the given example).

I don't mind keeping this issue closed, though as an end user I still find it unintuitive that only one include is allowed per section.

Apologies for the late reply. I haven't tried the new deep merging includes yet so I'm not sure how it interacts with other includes (my setup is more complicated than the given example). I don't mind keeping this issue closed, though as an end user I still find it unintuitive that only one include is allowed per section.
witten commented 2022-05-26 06:05:52 +00:00
Owner
Copy link

I agree.. I'll try to document this limitation better! And feel free to reopen this ticket if the deep merging doesn't end up working for your use case.

I agree.. I'll try to document this limitation better! And feel free to reopen this ticket if the deep merging doesn't end up working for your use case.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
parallel-borgmatic
command-hook-steps
2.1.7
2.1.6
2.1.5
2.1.4
2.1.3
2.1.2
2.1.1
2.1.0
2.0.13
2.0.12
2.0.11
2.0.10
2.0.9
2.0.8
2.0.7
2.0.6
2.0.5
2.0.4
2.0.3
2.0.2
2.0.1
2.0.0
1.9.14
1.9.13
1.9.12
1.9.11
1.9.10
1.9.9
1.9.8
1.9.7
1.9.6
1.9.5
1.9.4
1.9.3
1.9.2
1.9.1
1.9.0
1.8.14
1.8.13
1.8.12
1.8.11
1.8.10
1.8.9
1.8.8
1.8.7
1.8.6
1.8.5
1.8.4
1.8.3
1.8.2
1.8.1
1.8.0
1.7.15
1.7.14
1.7.13
1.7.12
1.7.11
1.7.10
1.7.9
1.7.8
1.7.7
1.7.6
1.7.5
1.7.4
1.7.3
1.7.2
1.7.1
1.7.0
1.6.6
1.6.5
1.6.4
1.6.3
1.6.2
1.6.1
1.6.0
1.5.24
1.5.23
1.5.22
1.5.21
1.5.20
1.5.19
1.5.18
1.5.17
1.5.16
1.5.15
1.5.14
1.5.13
1.5.12
1.5.11
1.5.10
1.5.9
1.5.8
1.5.7
1.5.7.dev0
1.5.6
1.5.5
1.5.4
1.5.3
1.5.2
1.5.1
1.5.0
1.4.22
1.4.21
1.4.20
1.4.19
1.4.18
1.4.17
1.4.16
1.4.15
1.4.14
1.4.13
1.4.12
1.4.11
1.4.10
1.4.9
1.4.8
1.4.7
1.4.6
1.4.5
1.4.4
1.4.3
1.4.2
1.4.1
1.4.0
1.3.26
1.3.25
1.3.24
1.3.23
1.3.22
1.3.21
1.3.20
1.3.19
1.3.18
1.3.17
1.3.16
1.3.15
1.3.14
1.3.13
1.3.12
1.3.11
1.3.10
1.3.9
1.3.8
1.3.7
1.3.6
1.3.5
1.3.4
1.3.3
1.3.2
1.3.1
1.3.0
1.2.18
1.2.17
1.2.16
1.2.15
1.2.14
1.2.13
1.2.12
1.2.11
1.2.10
1.2.9
1.2.8
1.2.7
1.2.6
1.2.5
1.2.4
1.2.3
1.2.2
1.2.1
1.2.0
1.1.15
1.1.14
1.1.13
1.1.12
1.1.11
1.1.10
1.1.9
1.1.7
1.1.6
1.1.5
1.1.4
1.1.3
1.1.2
1.1.1
1.1.0
1.0.3
1.0.2
1.0.1
1.0.0
0.1.8
0.1.7
0.1.6
0.1.5
0.1.4
0.1.3
0.1.2
0.1.1
0.1.0
0.0.7
0.0.6
0.0.5
0.0.4
0.0.3
0.0.2
Labels
Clear labels   
blocked

   
breaking

   
bug

   
data loss

   
design finalized

   
good first issue

   
new feature area

   
question / support

   
security

   
waiting for response
No labels
blocked
breaking
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
borgmatic-collective/borgmatic#380
No description provided.