YAML !include with relative path #532

Closed
opened 2022-05-13 19:06:29 +00:00 by league · 4 comments

What I'm trying to do and why

I'd like to be able to use the YAML include file and merging features with relative path names, something like this:

# [...]
retention:
  keep_within: 10d
<<: !include ./no-checks.yaml

It seems that any non-absolute filename is taken as relative to the current working directory, rather than relative to the file in which it sits.

I guess I want this because I'm keeping configs for several machines on other machines, so that it's easy to maintain and restore repos on behalf of other hosts. Essentially I'm doing this by lifting local-machine configs into borgmatic.d and leaving the other-machine configs in a subdir.

Also to further complicate matters, I'm using NixOS and installing the files into /etc as symlinks to the nix store. I'd like it to work either from the installed default location (/etc/borgmatic.d) or from a draft "dotfiles" location (borgmatic -c ~/dotfiles/sys/borgmatic/HOST.yaml).

Steps to reproduce (if a bug)

% sudo borgmatic -c sys/borgmatic/host1.yaml info -v2

where host1.yaml is like:

location:
  source_directories:
    - /Users/league/p
  repositories:
    - foo@foo.repo.borgbase.com:repo
storage:
  ssh_command: ssh -i ~root/.ssh/id_protected
  encryption_passcommand: cat /run/agenix/BLAHBLAH
retention:
  keep_within: 10d
<<: !include ./no-checks.yaml

and no-checks.yaml is:

consistency:
  checks:
    - disabled

Actual behavior (if a bug)

Ensuring legacy configuration is upgraded
sys/borgmatic/host1.yaml: No valid configuration files found

summary:
sys/borgmatic/host1.yaml: Error parsing configuration file
[Errno 2] No such file or directory: './no-checks.yaml'
sys/borgmatic/host1.yaml: No valid configuration files found

Expected behavior (if a bug)

[...]
summary:
sys/borgmatic/host1.yaml: Successfully ran configuration file

Which is what happens now when I change the include to a path relative to CWD:

<<: !include sys/borgmatic/no-checks.yaml

Other notes / implementation ideas

I can definitely see this is a niche use case. But if it's relatively easy to say what the CWD should be when loading ./something.yaml that would be sweet. All the examples I'm finding in documentation or the issues forum use absolute paths.

(If this becomes a WONTFIX, a line in the docs saying "must use absolute paths" would be appropriate.)

Thankss for work on borgmatic... at first I was writing, then generating my own hacky borg scripts, but this will end up being cleaner and more reliable!

Environment

borgmatic version: 1.5.18

borgmatic installation method: NixOS, nixpkgs

Borg version: 1.1.17

Python version: 3.9.12

Database version (if applicable): [version here]

operating system and version: NixOS 21.11, MacOS

#### What I'm trying to do and why I'd like to be able to use the YAML include file and merging features with relative path names, something like this: ```yaml # [...] retention: keep_within: 10d <<: !include ./no-checks.yaml ``` It seems that any non-absolute filename is taken as relative to the current working directory, rather than relative to the file in which it sits. I guess I want this because I'm keeping configs for several machines on other machines, so that it's easy to maintain and restore repos on behalf of other hosts. Essentially I'm doing this by lifting local-machine configs into `borgmatic.d` and leaving the other-machine configs in a subdir. Also to further complicate matters, I'm using NixOS and installing the files into /etc as symlinks to the nix store. I'd like it to work either from the installed default location (`/etc/borgmatic.d`) or from a draft "dotfiles" location (`borgmatic -c ~/dotfiles/sys/borgmatic/HOST.yaml`). #### Steps to reproduce (if a bug) ``` % sudo borgmatic -c sys/borgmatic/host1.yaml info -v2 ``` where `host1.yaml` is like: ```yaml location: source_directories: - /Users/league/p repositories: - foo@foo.repo.borgbase.com:repo storage: ssh_command: ssh -i ~root/.ssh/id_protected encryption_passcommand: cat /run/agenix/BLAHBLAH retention: keep_within: 10d <<: !include ./no-checks.yaml ``` and `no-checks.yaml` is: ```yaml consistency: checks: - disabled ``` #### Actual behavior (if a bug) ``` Ensuring legacy configuration is upgraded sys/borgmatic/host1.yaml: No valid configuration files found summary: sys/borgmatic/host1.yaml: Error parsing configuration file [Errno 2] No such file or directory: './no-checks.yaml' sys/borgmatic/host1.yaml: No valid configuration files found ``` #### Expected behavior (if a bug) ``` [...] summary: sys/borgmatic/host1.yaml: Successfully ran configuration file ``` Which is what happens now when I change the include to a path relative to CWD: ```yaml <<: !include sys/borgmatic/no-checks.yaml ``` #### Other notes / implementation ideas I can definitely see this is a niche use case. But if it's relatively easy to say what the CWD should be when loading `./something.yaml` that would be sweet. All the examples I'm finding in documentation or the issues forum use absolute paths. (If this becomes a WONTFIX, a line in the docs saying "must use absolute paths" would be appropriate.) Thankss for work on borgmatic... at first I was writing, then generating my own hacky borg scripts, but this will end up being cleaner and more reliable! #### Environment **borgmatic version:** 1.5.18 **borgmatic installation method:** NixOS, nixpkgs **Borg version:** 1.1.17 **Python version:** 3.9.12 **Database version (if applicable):** [version here] **operating system and version:** NixOS 21.11, MacOS
Owner

First off, I appreciate the detailed ticket write-up. I think a feature like this could work in borgmatic. However, if borgmatic simply changed includes from being relative-from-working-directory to relative-from-config, that would be a breaking change for anyone relying on relative includes today. There have been breaking changes in the past (see version 1.6.0 for an example). But one way to do this without breaking anything would change the behavior to probe: So when doing a relative include, first probe for relative-from-working-directory. If not found, try relative-from-config next. Or I suppose the other way around could work too.

Please let me know your thoughts!

First off, I appreciate the detailed ticket write-up. I think a feature like this could work in borgmatic. However, if borgmatic simply changed includes from being relative-from-working-directory to relative-from-config, that would be a breaking change for anyone relying on relative includes today. There *have* been breaking changes in the past (see [version 1.6.0](https://projects.torsion.org/borgmatic-collective/borgmatic/releases/tag/1.6.0) for an example). But one way to do this without breaking anything would change the behavior to probe: So when doing a relative include, first probe for relative-from-working-directory. If not found, try relative-from-config next. Or I suppose the other way around could work too. Please let me know your thoughts!
Author

I think a probe and fallback should be fine. Or even a list of directories conveyed in a PATH-style environment variable would be usable. Might need to be a little cautious when it comes to includes-within-includes, or symlinks.

I wouldn't want to overengineer it... after all we're just trying to unify a few fragments of config to keep settings consistent. In other words, don't go nuts to highlight some obscure corners... it's not all that much harder to use some preprocessor to generate configs.

If one wants to argue for one behavior over the other, with "principle of least surprise", I'd say most of those preprocessors (cpp, m4, makefile include) are relative to the file in which the include is found. It does get confusing when the same file might be reached through different symlinks. I don't know offhand if those tools ‘realpath’ it, or what. Anyway, thanks for considering.

I think a probe and fallback should be fine. Or even a list of directories conveyed in a PATH-style environment variable would be usable. Might need to be a little cautious when it comes to includes-within-includes, or symlinks. I wouldn't want to overengineer it... after all we're just trying to unify a few fragments of config to keep settings consistent. In other words, don't go nuts to highlight some obscure corners... it's not **all** that much harder to use some preprocessor to generate configs. If one wants to argue for one behavior over the other, with "principle of least surprise", I'd say most of those preprocessors (cpp, m4, makefile include) are relative to the file in which the include is found. It does get confusing when the same file might be reached through different symlinks. I don't know offhand if those tools ‘realpath’ it, or what. Anyway, thanks for considering.
Owner

This is implemented in master now!

It does a probe and fallback of: 1. the current working directory, and 2. the directory containing the file doing the including. Not otherwise configurable, for now. It does do the realpath equivalent on the second path.

Feel free to give it a shot or wait for the next release. And thanks for suggesting this!

This is implemented in master now! It does a probe and fallback of: 1. the current working directory, and 2. the directory containing the file doing the including. Not otherwise configurable, for now. It does do the `realpath` equivalent on the second path. Feel free to give it a shot or wait for the next release. And thanks for suggesting this!
Owner

Released in borgmatic 1.6.1!

Released in borgmatic 1.6.1!
Sign in to join this conversation.
No Milestone
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#532
No description provided.