Per-repository options in one config file? #73
Labels
No Label
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
No Milestone
No Assignees
4 Participants
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: borgmatic-collective/borgmatic#73
Loading…
Reference in New Issue
No description provided.
Delete Branch "%!s(<nil>)"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Use case:
I have a
/etc/borgmatic/config.yaml
to backup my server to two differentrepositories
:While I would like to run the default
consistency
checks on repository 1, on repository 2 it simply is not feasible since the consistency checks alone take about 20 hours each day.All other options should be the same.
Question: Is there (or will be) a way to configure differing per-repository settings for individual options in a borgmatic config file while sharing all other options?
A current solution would be to just have several config files with one repository each. But then I have to edit multiple files every time I want to change one of the common options (like
source_directories
) and take extra care to keep them in sync.I have the same use case, one of my repository is much slower. That's not the first time I wish I could use borgmatic and its config the same way I use docker-compose and its config file.
Brainstorming solutions here:
We could change the consistency checks section from looking like this..
.. to also support something like this:
I honestly don't know how feasible it would be to support both formats, as that would mean
checks
has to support two different data types (lists and mappings).Does something like that sounds like it would solve your use cases?
what about:
Yeah, that might be a more natural way of expressing it. It'd also be more of a shake-up of the existing config file format though. Here's one challenge in regards to that (other than backwards compatibility): I bet that in some cases people want retention config shared across repositories (and thus not repeated), and in other cases they want retention config to differ by repository. Same goes for consistency config, etc. So for any given type of config, sometimes you'd want it at the global scope, and sometimes you'd want it nested within a single repository.
More brainstorming to maybe support that sort of thing: File includes. So you could have each repository defined in a separate config file, but then if there was common config you wanted to pull in without repetition, you could include it from another file. That adds complexity to both parsing and validation, but it could potentially address this class of use cases.
keep in mind that this is a config file, not a source code file. From an ops point of vue, I rather have a config file which has repetition but where one can just read it and get it. I tend to hate config files for which I have to refer to the documentation to understand things.
And for the repetition struggle, copy-and-paste is a great tool ;)
I don't disagree with that when it comes to my own needs, and am a big believer in infrastructure as config, not code. But it sounds like one of flerrer's main concerns was about repetition and editing the same thing in multiple places (multiple files). So in brainstorming solutions, I'm trying to come up with a way to balance the competing needs: Minimize repetition on multiple repositories, support either sharing or not sharing config across multiple repositories, and of course avoid devolving into a full-on programming language. :)
maybe using yaml pointers is the way to go?
Yup, a YAML feature like that could certainly address the sharing or non-sharing use cases. Assuming the proposed config format was in place so that you could "point" from the retention or consistency config for one repository to that of another repository.
this is actually native yaml feature, no need to code anything to support it ;)
Check the following in one of your actual config:
Right, a pointer like that already works for that simple case. But going back to the original ask, it's basically: "I want to have one config file with multiple different repositories, sharing almost all of their config, but with different consistency checks for each repository." Unless I'm lacking in YAML imagination, that will require at least some config/code changes to support, perhaps in addition to using YAML pointers as you demonstrate.
Quick requirements question for both of you: Do you want want to run n consistency checks on one repository, and zero consistency checks on the other? Or is it instead n consistency checks on one repository, and m consistency checks on the other?
On my current setup it is indeed
n:0
.(The more flexible configuration variant would of course be be
n:m
. However, I don't think I currently would need that)Okay, then what I'm thinking is the adding support for the following to the consistency section of the config file:
The idea is that if the
check_repositories
option is supplied, then consistency checks would only run on those listed repositories. Otherwise, if not supplied, checks would run on all repositories (the current behavior).Let me know if that sounds like it would work for you. It's a much less far-reaching / flexible proposal than some of the options discussed above, but I think it does satisfy the immediate need. And it'd be pretty easy to implement.
Well, I went ahead and implemented this. Just released as part of borgmatic 1.2.8. Give it a shot and let me know how it works out for you!
I don't know if I have borgmatic 1.2.8 there is no --version switch :P
You can run the following to get the version:
pip3 show borgmatic
. If a--version
flag would be convenient though, feel free to file a ticket for it!FYI, I just added a
borgmatic --version
flag as well.ok I have 1.2.12
So what is there to test? (if you already wrote how to, link the message)
See the comment above on how to configure consistency checks just for a subset of your repositories. And here's the comment on this feature from the newly generated config:
Ah ok, well I guess it doesn't require any testing lol and I run consistency checks on all my repos (2)
I put together a proposal for borgmatic configuration includes and reuse. Check it out here: #148. And please feel free to comment with your feedback / ideas.
errnesto referenced this issue2019-12-05 01:26:26 +00:00
errnesto referenced this issue2019-12-05 01:27:15 +00:00
errnesto referenced this issue2019-12-05 01:28:02 +00:00
errnesto referenced this issue2019-12-05 01:28:45 +00:00
errnesto referenced this issue2019-12-05 01:29:04 +00:00
FYI, in case anyone's still interested.. Some discussion on this general topic of configuration varying per-repository: witten/borgmatic#281