Institution doesn't appear to work in include filenames #674

New Issue

Closed

opened 2023-04-10 21:59:53 +00:00 by alsoeric · 2 comments

alsoeric commented 2023-04-10 21:59:53 +00:00

Copy Link

What I'm trying to do and why

Trying to work around the deep versus shallow merge issue with a include pathname hack

Steps to reproduce (if a bug)

ceate three configuration files. Topmost with fields that are deep merged. Template common with include path with the constant substitution field, and the configuration file with fields that are imported via the calculated pathname in the template common. See below for examples.

Topmost config

constants:
  incl_name: home

<<: !include /etc/borgmatic/pw-common.yaml
location:
  atime: true
  ...

master template fragment

location:
    <<: !include {incl_name}/location.yaml
    

home/location.yaml

source_directories:
  - /home
  - /etc
repositories:
  - path: 7909@usw-s007.rsync.net:pw-home
    label: backupserver

Actual behavior (if a bug)

(borg-env) root@lake:~# validate-borgmatic-config -c home.yaml
home.yaml: Error parsing configuration file
An error occurred while parsing a configuration file at home.yaml:
while parsing a block mapping
  in "<unicode string>", line 17, column 5
did not find expected key
  in "<unicode string>", line 17, column 29
    

Expected behavior (if a bug)

If the constant substitution occurred in the filename then the directory specified in the top level configuration file would be imported into the core template fragment effectively replacing that part of the configuration file and simulating a shallow merge.

While this example is is a hack around the lack of a shallow merge, I believe it would also be useful in general to calculate pathnames for different fig file fragments.

Other notes / implementation ideas

Environment

borgmatic version: [1.7.11]

Use sudo borgmatic --version or sudo pip show borgmatic | grep ^Version

borgmatic installation method: [pip3 into virtual environment]

Borg version: [1.2.4]

Use sudo borg --version

Python version: [3.10.6]

Use python3 --version

Database version (if applicable): [version here]

Use psql --version or mysql --version on client and server.

operating system and version: [Ubuntu 22.04]

#### What I'm trying to do and why Trying to work around the deep versus shallow merge issue with a include pathname hack #### Steps to reproduce (if a bug) ceate three configuration files. Topmost with fields that are deep merged. Template common with include path with the constant substitution field, and the configuration file with fields that are imported via the calculated pathname in the template common. See below for examples. Topmost config ``` constants: incl_name: home <<: !include /etc/borgmatic/pw-common.yaml location: atime: true ... ``` master template fragment ``` location: <<: !include {incl_name}/location.yaml ``` home/location.yaml ``` source_directories: - /home - /etc repositories: - path: 7909@usw-s007.rsync.net:pw-home label: backupserver ``` #### Actual behavior (if a bug) ``` (borg-env) root@lake:~# validate-borgmatic-config -c home.yaml home.yaml: Error parsing configuration file An error occurred while parsing a configuration file at home.yaml: while parsing a block mapping in "<unicode string>", line 17, column 5 did not find expected key in "<unicode string>", line 17, column 29 ``` #### Expected behavior (if a bug) If the constant substitution occurred in the filename then the directory specified in the top level configuration file would be imported into the core template fragment effectively replacing that part of the configuration file and simulating a shallow merge. While this example is is a hack around the lack of a shallow merge, I believe it would also be useful in general to calculate pathnames for different fig file fragments. #### Other notes / implementation ideas #### Environment **borgmatic version:** [1.7.11] Use `sudo borgmatic --version` or `sudo pip show borgmatic | grep ^Version` **borgmatic installation method:** [pip3 into virtual environment] **Borg version:** [1.2.4] Use `sudo borg --version` **Python version:** [3.10.6] Use `python3 --version` **Database version (if applicable):** [version here] Use `psql --version` or `mysql --version` on client and server. **operating system and version:** [Ubuntu 22.04]

witten commented 2023-04-10 22:22:00 +00:00

Owner

Copy Link

Thanks for filing this. Constants not working across includes is a known/documented limitation that was put in place for simplicity of implementation and use, although I see why you'd want that limitation lifted for your use case.

If this were to be implemented, I think the way to do it would be to accumulate constants as each successive file is included (potentially replacing constants if they've already been defined at a higher include level), and doing the interpolation on each file as it's included.

I'm not sure though whether this is a "better" idea than just implementing #672, since this ticket (#674) is a work-around for not having that feature.

Thanks for filing this. Constants not working across includes is a [known/documented limitation](https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/#constant-interpolation) that was put in place for simplicity of implementation and use, although I see why you'd want that limitation lifted for your use case. If this were to be implemented, I think the way to do it would be to accumulate constants as each successive file is included (potentially replacing constants if they've already been defined at a higher include level), and doing the interpolation on each file as it's included. I'm not sure though whether this is a "better" idea than just implementing #672, since this ticket (#674) is a work-around for not having that feature.

alsoeric commented 2023-04-10 23:51:45 +00:00

Author

Copy Link

Implementing #672 is a better idea. ok to close this

Implementing https://projects.torsion.org/borgmatic-collective/borgmatic/issues/672 is a better idea. ok to close this

👍 1

alsoeric closed this issue 2023-04-10 23:51:50 +00:00

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

spot-check

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.24

1.5.23

1.5.22

1.5.21

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.14

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.7.dev0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.22

1.4.21

1.4.20

1.4.19

1.4.18

1.4.17

1.4.16

1.4.15

1.4.14

1.4.13

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.26

1.3.25

1.3.24

1.3.23

1.3.22

1.3.21

1.3.20

1.3.19

1.3.18

1.3.17

1.3.16

1.3.15

1.3.14

1.3.13

1.3.12

1.3.11

1.3.10

1.3.9

1.3.8

1.3.7

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.18

1.2.17

1.2.16

1.2.15

1.2.14

1.2.13

1.2.12

1.2.11

1.2.10

1.2.9

1.2.8

1.2.7

1.2.6

1.2.5

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.15

1.1.14

1.1.13

1.1.12

1.1.11

1.1.10

1.1.9

1.1.7

1.1.6

1.1.5

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.3

1.0.2

1.0.1

1.0.0

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

0.0.2

Labels

Clear labels   

bug

  

data loss

  

design finalized

  

good first issue

  

new feature area

  

question / support

  

security

  

waiting for response

No Label

bug

data loss

design finalized

good first issue

new feature area

question / support

security

waiting for response

Milestone

Clear milestone

No items

No Milestone

Assignees

Clear assignees

No Assignees

2 Participants

Notifications

Subscribe

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#674

No description provided.