YAML !include with relative path
What I'm trying to do and why
I'd like to be able to use the YAML include file and merging features with relative path names, something like this:
# [...] retention: keep_within: 10d <<: !include ./no-checks.yaml
It seems that any non-absolute filename is taken as relative to the current working directory, rather than relative to the file in which it sits.
I guess I want this because I'm keeping configs for several machines on other machines, so that it's easy to maintain and restore repos on behalf of other hosts. Essentially I'm doing this by lifting local-machine configs into
borgmatic.d and leaving the other-machine configs in a subdir.
Also to further complicate matters, I'm using NixOS and installing the files into /etc as symlinks to the nix store. I'd like it to work either from the installed default location (
/etc/borgmatic.d) or from a draft "dotfiles" location (
borgmatic -c ~/dotfiles/sys/borgmatic/HOST.yaml).
Steps to reproduce (if a bug)
% sudo borgmatic -c sys/borgmatic/host1.yaml info -v2
host1.yaml is like:
location: source_directories: - /Users/league/p repositories: - firstname.lastname@example.org:repo storage: ssh_command: ssh -i ~root/.ssh/id_protected encryption_passcommand: cat /run/agenix/BLAHBLAH retention: keep_within: 10d <<: !include ./no-checks.yaml
consistency: checks: - disabled
Actual behavior (if a bug)
Ensuring legacy configuration is upgraded sys/borgmatic/host1.yaml: No valid configuration files found summary: sys/borgmatic/host1.yaml: Error parsing configuration file [Errno 2] No such file or directory: './no-checks.yaml' sys/borgmatic/host1.yaml: No valid configuration files found
Expected behavior (if a bug)
[...] summary: sys/borgmatic/host1.yaml: Successfully ran configuration file
Which is what happens now when I change the include to a path relative to CWD:
<<: !include sys/borgmatic/no-checks.yaml
Other notes / implementation ideas
I can definitely see this is a niche use case. But if it's relatively easy to say what the CWD should be when loading
./something.yaml that would be sweet. All the examples I'm finding in documentation or the issues forum use absolute paths.
(If this becomes a WONTFIX, a line in the docs saying "must use absolute paths" would be appropriate.)
Thankss for work on borgmatic... at first I was writing, then generating my own hacky borg scripts, but this will end up being cleaner and more reliable!
borgmatic version: 1.5.18
borgmatic installation method: NixOS, nixpkgs
Borg version: 1.1.17
Python version: 3.9.12
Database version (if applicable): [version here]
operating system and version: NixOS 21.11, MacOS
First off, I appreciate the detailed ticket write-up. I think a feature like this could work in borgmatic. However, if borgmatic simply changed includes from being relative-from-working-directory to relative-from-config, that would be a breaking change for anyone relying on relative includes today. There have been breaking changes in the past (see version 1.6.0 for an example). But one way to do this without breaking anything would change the behavior to probe: So when doing a relative include, first probe for relative-from-working-directory. If not found, try relative-from-config next. Or I suppose the other way around could work too.
Please let me know your thoughts!
I think a probe and fallback should be fine. Or even a list of directories conveyed in a PATH-style environment variable would be usable. Might need to be a little cautious when it comes to includes-within-includes, or symlinks.
I wouldn't want to overengineer it... after all we're just trying to unify a few fragments of config to keep settings consistent. In other words, don't go nuts to highlight some obscure corners... it's not all that much harder to use some preprocessor to generate configs.
If one wants to argue for one behavior over the other, with "principle of least surprise", I'd say most of those preprocessors (cpp, m4, makefile include) are relative to the file in which the include is found. It does get confusing when the same file might be reached through different symlinks. I don't know offhand if those tools ‘realpath’ it, or what. Anyway, thanks for considering.
This is implemented in master now!
It does a probe and fallback of: 1. the current working directory, and 2. the directory containing the file doing the including. Not otherwise configurable, for now. It does do the
realpath equivalent on the second path.
Feel free to give it a shot or wait for the next release. And thanks for suggesting this!
Deleting a branch is permanent. It CANNOT be undone. Continue?