configuration override not working for "required" fields #607

What I'm trying to do and why

I try to use the include merging of configuration files described here:
https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/#include-merging

If I try to include a "required" element without overriding it, the parser gives an error:

For example, the location.repositories is described only in an included configuration. When issuing sudo borgmatic create the following error appears:

config.yaml: No valid configuration files found

summary:
config.yaml: Error parsing configuration file
An error occurred while parsing a configuration file at config.yaml:
At 'location': 'repositories' is a required property
config.yaml: No valid configuration files found

With "required" element I mean any field in the configuration file that is descibed with (required) here: https://torsion.org/borgmatic/docs/reference/configuration/
e.g. location.repository ; or for prune command any element of retention.

Steps to reproduce

Create two files:

config.yaml:

<<: !include include.yaml
location:
    repositories:
        - <path/to/test/repo>
    source_directories:
        - <path/to/test/data>

include.yaml:

storage:
    archive_name_format: 'include-success-{now}'

retention:
    prefix: 'include'

and create a repo:

$ sudo borgmatic init --encryption none --config config.yaml
$ sudo borgmatic create --verbosity 1 --config config.yaml

Result: archive named "include-success..." created. Therefore the include works fine

Now change the files:
config.yaml:

<<: !include include.yaml
location:
#    repositories:
#        - <path/to/test/repo>
    source_directories:
        - <path/to/test/data>

include.yaml:

location:
    repositories:
        - <path/to/test/repo>
storage:
    archive_name_format: 'include-success-{now}'

retention:
    prefix: 'include'

and execute the borgmatic create again
Result: the error included in the top of this post.

Actual behavior

The whole --verbosity 2 output is:

nsuring legacy configuration is upgraded
config.yaml: No valid configuration files found

summary:
config.yaml: Error parsing configuration file
An error occurred while parsing a configuration file at config.yaml:
At 'location': 'repositories' is a required property
config.yaml: No valid configuration files found

Need some help? https://torsion.org/borgmatic/#issues

Expected behavior

A repository created like in the first step

Other notes / implementation ideas

If I (additionally) override it at in the command, it work's again:
sudo borgmatic create --override location.repositories='[/path/to/test/repo]'

Environment

borgmatic version: 1.5.24

borgmatic installation method: openSUSE package manger

Borg version: 1.2.2

Python version: 3.10.7

operating system and version:
Operating System: openSUSE Tumbleweed 20221026
Kernel Version: 6.0.3-1-default (64-bit)

#### What I'm trying to do and why I try to use the include merging of configuration files described here: https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/#include-merging If I try to include a "required" element without overriding it, the parser gives an error: For example, the `location.repositories` is described only in an included configuration. When issuing `sudo borgmatic create` the following error appears: ``` config.yaml: No valid configuration files found summary: config.yaml: Error parsing configuration file An error occurred while parsing a configuration file at config.yaml: At 'location': 'repositories' is a required property config.yaml: No valid configuration files found ``` With "required" element I mean any field in the configuration file that is descibed with `(required)` here: https://torsion.org/borgmatic/docs/reference/configuration/ e.g. `location.repository` ; or for prune command any element of `retention`. #### Steps to reproduce Create two files: config.yaml: ``` yaml <<: !include include.yaml location: repositories: - <path/to/test/repo> source_directories: - <path/to/test/data> ``` include.yaml: ``` yaml storage: archive_name_format: 'include-success-{now}' retention: prefix: 'include' ``` and create a repo: ``` $ sudo borgmatic init --encryption none --config config.yaml $ sudo borgmatic create --verbosity 1 --config config.yaml ``` Result: archive named "include-success..." created. Therefore the include works fine Now change the files: config.yaml: ``` yaml <<: !include include.yaml location: # repositories: # - <path/to/test/repo> source_directories: - <path/to/test/data> ``` include.yaml: ``` yaml location: repositories: - <path/to/test/repo> storage: archive_name_format: 'include-success-{now}' retention: prefix: 'include' ``` and execute the `borgmatic create` again Result: the error included in the top of this post. #### Actual behavior The whole `--verbosity 2` output is: ``` nsuring legacy configuration is upgraded config.yaml: No valid configuration files found summary: config.yaml: Error parsing configuration file An error occurred while parsing a configuration file at config.yaml: At 'location': 'repositories' is a required property config.yaml: No valid configuration files found Need some help? https://torsion.org/borgmatic/#issues ``` #### Expected behavior A repository created like in the first step #### Other notes / implementation ideas If I (additionally) override it at in the command, it work's again: `sudo borgmatic create --override location.repositories='[/path/to/test/repo]'` #### Environment **borgmatic version:** 1.5.24 **borgmatic installation method:** openSUSE package manger **Borg version:** 1.2.2 **Python version:** 3.10.7 **operating system and version:** Operating System: openSUSE Tumbleweed 20221026 Kernel Version: 6.0.3-1-default (64-bit)

Thanks for the detailed ticket! I think the problem is that the version of borgmatic you're using is too old to support "deep merging". Without this feature, you can't merge one location: section that contains source_directories with another that contains repositories. In the older version of borgmatic you're using, if you try to do that, then only one of the location: sections "wins"—the one that's not included.

So as I see it, here are your options:

Upgrade borgmatic! At minimum version 1.6.0 to get the deep merging feature.
Or, put your repositories and source_directories (and all other location: options) all in one configuration file, whether included or not.

Thanks for the detailed ticket! I think the problem is that the version of borgmatic you're using is too old to support ["deep merging"](https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/#deep-merge). Without this feature, you can't merge one `location:` section that contains `source_directories` with another that contains `repositories`. In the older version of borgmatic you're using, if you try to do that, then only one of the `location:` sections "wins"—the one that's not included. So as I see it, here are your options: 1. Upgrade borgmatic! At minimum version 1.6.0 to get the deep merging feature. 2. Or, put your `repositories` and `source_directories` (and all other `location:` options) all in one configuration file, whether included or not.

Thank you for the quick response.

I saw the version note about the deep merge in the section below.

But the section "Include Merging" has already an example about the "YAML merge of included configurations" and an override example. Only the next section "Deep merge" has the version note. I assumed this "Deep merge" is regarding: "Colliding list values are appended together."

Nevertheless, if it is really fixed in v 1.6 I'm glad anyways :)

Do you have any clue when this version will come to openSUSE software repository? The openSUSE tumbleweed is usually only a few weeks behind the releases. Having an 8 month old (v1.5.2) software version looks strange.

Best wishes
Simon

Thank you for the quick response. I saw the version note about the deep merge in the section below. But the section "Include Merging" has already an example about the "YAML merge of included configurations" and an override example. Only the next section "Deep merge" has the version note. I assumed this "Deep merge" is regarding: "Colliding list values are appended together." Nevertheless, if it is really fixed in v 1.6 I'm glad anyways :) Do you have any clue when this version will come to openSUSE software repository? The openSUSE tumbleweed is usually only a few weeks behind the releases. Having an 8 month old (v1.5.2) software version looks strange. Best wishes Simon

You are of course correct that the "Include Merging" section shows an example that only works with deep merge and borgmatic 1.6.0+—without any indication of that fact! I'll update the example accordingly. I appreciate you pointing this out.

I don't unfortunately have insight into the openSUSE packaging timeline for borgmatic. That entirely a third-party process. I'd recommend contacting the packager or trying one of the other borgmatic installation methods. (I realize that's less convenient than a distro-specific package.)

You are of course correct that the "Include Merging" section shows an example that only works with deep merge and borgmatic 1.6.0+—without any indication of that fact! I'll update the example accordingly. I appreciate you pointing this out. I don't unfortunately have insight into the openSUSE packaging timeline for borgmatic. That entirely a third-party process. I'd recommend [contacting the packager](https://build.opensuse.org/users/dimstar_suse) or trying [one of the other borgmatic installation methods](https://torsion.org/borgmatic/docs/how-to/set-up-backups/#installation). (I realize that's less convenient than a distro-specific package.)

Thank you, I have now installed it via pip3 and everything works fine now :)

Labels Milestones