borgmatic-collective/borgmatic
borgmatic-collective
/
borgmatic
15
57
Fork 40
Code Issues 54 Pull Requests 1 Actions Packages Releases 121 Activity

Per-repository options in one config file? #73

New Issue
Closed
opened 2018-06-28 06:56:02 +00:00 by flesser · 22 comments
flesser commented 2018-06-28 06:56:02 +00:00
Copy Link

Use case:
I have a /etc/borgmatic/config.yaml to backup my server to two different repositories:

  1. one on another server machine
  2. one on a "poor man's NAS" (Raspberry + USB hard drive)

While I would like to run the default consistency checks on repository 1, on repository 2 it simply is not feasible since the consistency checks alone take about 20 hours each day.
All other options should be the same.

Question: Is there (or will be) a way to configure differing per-repository settings for individual options in a borgmatic config file while sharing all other options?

A current solution would be to just have several config files with one repository each. But then I have to edit multiple files every time I want to change one of the common options (like source_directories) and take extra care to keep them in sync.

Use case: I have a `/etc/borgmatic/config.yaml` to backup my server to two different `repositories`: 1. one on another server machine 2. one on a "poor man's NAS" (Raspberry + USB hard drive) While I would like to run the default `consistency` checks on repository 1, on repository 2 it simply is not feasible since the consistency checks alone take about 20 hours each day. All other options should be the same. Question: Is there (or will be) a way to configure differing per-repository settings for individual options in a borgmatic config file while sharing all other options? A current solution would be to just have several config files with one repository each. But then I have to edit multiple files every time I want to change one of the common options (like `source_directories`) and take extra care to keep them in sync.
thomasleveil commented 2018-06-28 08:57:03 +00:00
Contributor
Copy Link

I have the same use case, one of my repository is much slower. That's not the first time I wish I could use borgmatic and its config the same way I use docker-compose and its config file.

I have the same use case, one of my repository is much slower. That's not the first time I wish I could use borgmatic and its config the same way I use docker-compose and its config file.
witten commented 2018-07-01 22:06:18 +00:00
Owner
Copy Link

Brainstorming solutions here:

We could change the consistency checks section from looking like this..

    checks:
        - repository
        - archives

.. to also support something like this:

    checks:
        "user@backupserver:repository.borg":
          - repository
          - archives
        "other@server:2ndrepository.borg":
          - repository

I honestly don't know how feasible it would be to support both formats, as that would mean checks has to support two different data types (lists and mappings).

Does something like that sounds like it would solve your use cases?

Brainstorming solutions here: We could change the consistency checks section from looking like this.. ``` checks: - repository - archives ``` .. to also support something like this: ``` checks: "user@backupserver:repository.borg": - repository - archives "other@server:2ndrepository.borg": - repository ``` I honestly don't know how feasible it would be to support both formats, as that would mean `checks` has to support two different data types (lists and mappings). Does something like that sounds like it would solve your use cases?
thomasleveil commented 2018-07-01 22:33:22 +00:00
Contributor
Copy Link

what about:

# List of source directories to backup. Globs are expanded.
source_directories:
    - /home
    - /etc
    - /var/log/syslog*


# Any paths matching these patterns are excluded from backups.
exclude_patterns:
    - /home/*/.cache

# Paths to local or remote repositories.
repositories:
  whatever_name:
    url: user@backupserver:sourcehostname.borg
    retention:
        # Retention policy for how many backups to keep in each category.
        keep_daily: 7
        keep_weekly: 4
        keep_monthly: 6

    consistency:
        # List of consistency checks to run: "repository", "archives", or both.
        checks:
            - repository

  my_local_repo:
    url: /var/backups/borg/my_project
    retention:
        # Retention policy for how many backups to keep in each category.
        keep_daily: 7
        keep_weekly: 4
        keep_monthly: 6

    consistency:
        # List of consistency checks to run: "repository", "archives", or both.
        checks:
            - repository
            - archives
what about: ```yaml # List of source directories to backup. Globs are expanded. source_directories: - /home - /etc - /var/log/syslog* # Any paths matching these patterns are excluded from backups. exclude_patterns: - /home/*/.cache # Paths to local or remote repositories. repositories: whatever_name: url: user@backupserver:sourcehostname.borg retention: # Retention policy for how many backups to keep in each category. keep_daily: 7 keep_weekly: 4 keep_monthly: 6 consistency: # List of consistency checks to run: "repository", "archives", or both. checks: - repository my_local_repo: url: /var/backups/borg/my_project retention: # Retention policy for how many backups to keep in each category. keep_daily: 7 keep_weekly: 4 keep_monthly: 6 consistency: # List of consistency checks to run: "repository", "archives", or both. checks: - repository - archives ```
witten commented 2018-07-01 22:44:03 +00:00
Owner
Copy Link

Yeah, that might be a more natural way of expressing it. It'd also be more of a shake-up of the existing config file format though. Here's one challenge in regards to that (other than backwards compatibility): I bet that in some cases people want retention config shared across repositories (and thus not repeated), and in other cases they want retention config to differ by repository. Same goes for consistency config, etc. So for any given type of config, sometimes you'd want it at the global scope, and sometimes you'd want it nested within a single repository.

More brainstorming to maybe support that sort of thing: File includes. So you could have each repository defined in a separate config file, but then if there was common config you wanted to pull in without repetition, you could include it from another file. That adds complexity to both parsing and validation, but it could potentially address this class of use cases.

Yeah, that might be a more natural way of expressing it. It'd also be more of a shake-up of the existing config file format though. Here's one challenge in regards to that (other than backwards compatibility): I bet that in some cases people want retention config shared across repositories (and thus not repeated), and in other cases they want retention config to differ by repository. Same goes for consistency config, etc. So for any given type of config, sometimes you'd want it at the global scope, and sometimes you'd want it nested within a single repository. More brainstorming to maybe support that sort of thing: File includes. So you could have each repository defined in a separate config file, but then if there was common config you wanted to pull in without repetition, you could include it from another file. That adds complexity to both parsing and validation, but it could potentially address this class of use cases.
thomasleveil commented 2018-07-01 22:55:29 +00:00
Contributor
Copy Link

keep in mind that this is a config file, not a source code file. From an ops point of vue, I rather have a config file which has repetition but where one can just read it and get it. I tend to hate config files for which I have to refer to the documentation to understand things.

And for the repetition struggle, copy-and-paste is a great tool ;)

keep in mind that this is a config file, not a source code file. From an ops point of vue, I rather have a config file which has repetition but where one can just read it and get it. I tend to hate config files for which I have to refer to the documentation to understand things. And for the repetition struggle, copy-and-paste is a great tool ;)
witten commented 2018-07-01 23:02:27 +00:00
Owner
Copy Link

I don't disagree with that when it comes to my own needs, and am a big believer in infrastructure as config, not code. But it sounds like one of flerrer's main concerns was about repetition and editing the same thing in multiple places (multiple files). So in brainstorming solutions, I'm trying to come up with a way to balance the competing needs: Minimize repetition on multiple repositories, support either sharing or not sharing config across multiple repositories, and of course avoid devolving into a full-on programming language. :)

I don't disagree with that when it comes to my own needs, and am a big believer in infrastructure as config, not code. But it sounds like one of flerrer's main concerns was about repetition and editing the same thing in multiple places (multiple files). So in brainstorming solutions, I'm trying to come up with a way to balance the competing needs: Minimize repetition on multiple repositories, support either sharing *or* not sharing config across multiple repositories, and of course avoid devolving into a full-on programming language. :)
thomasleveil commented 2018-07-01 23:06:10 +00:00
Contributor
Copy Link

maybe using yaml pointers is the way to go?

# List of source directories to backup. Globs are expanded.
source_directories:
    - /home
    - /etc
    - /var/log/syslog*


# Any paths matching these patterns are excluded from backups.
exclude_patterns:
    - /home/*/.cache

# Paths to local or remote repositories.
repositories:
  whatever_name:
    url: user@backupserver:sourcehostname.borg
    retention: &common_retention
        # Retention policy for how many backups to keep in each category.
        keep_daily: 7
        keep_weekly: 4
        keep_monthly: 6

    consistency:
        # List of consistency checks to run: "repository", "archives", or both.
        checks:
            - repository

  my_local_repo:
    url: /var/backups/borg/my_project
    retention: *common_retention

    consistency:
        # List of consistency checks to run: "repository", "archives", or both.
        checks:
            - repository
            - archives
maybe using [yaml pointers](https://coderwall.com/p/wqgm7g/using-pointers-to-merge-elements-in-yaml-file) is the way to go? ```yaml # List of source directories to backup. Globs are expanded. source_directories: - /home - /etc - /var/log/syslog* # Any paths matching these patterns are excluded from backups. exclude_patterns: - /home/*/.cache # Paths to local or remote repositories. repositories: whatever_name: url: user@backupserver:sourcehostname.borg retention: &common_retention # Retention policy for how many backups to keep in each category. keep_daily: 7 keep_weekly: 4 keep_monthly: 6 consistency: # List of consistency checks to run: "repository", "archives", or both. checks: - repository my_local_repo: url: /var/backups/borg/my_project retention: *common_retention consistency: # List of consistency checks to run: "repository", "archives", or both. checks: - repository - archives ```
witten commented 2018-07-01 23:13:38 +00:00
Owner
Copy Link

Yup, a YAML feature like that could certainly address the sharing or non-sharing use cases. Assuming the proposed config format was in place so that you could "point" from the retention or consistency config for one repository to that of another repository.

Yup, a YAML feature like that could certainly address the sharing or non-sharing use cases. Assuming the proposed config format was in place so that you could "point" from the retention or consistency config for one repository to that of another repository.
thomasleveil commented 2018-07-01 23:18:01 +00:00
Contributor
Copy Link

this is actually native yaml feature, no need to code anything to support it ;)

Check the following in one of your actual config:


hooks:
  before_backup: &test
    - echo "I'am the before_backup hook"

  after_backup: *test
this is actually native yaml feature, no need to code anything to support it ;) Check the following in one of your actual config: ```yaml hooks: before_backup: &test - echo "I'am the before_backup hook" after_backup: *test ```
witten commented 2018-07-01 23:23:09 +00:00
Owner
Copy Link

Right, a pointer like that already works for that simple case. But going back to the original ask, it's basically: "I want to have one config file with multiple different repositories, sharing almost all of their config, but with different consistency checks for each repository." Unless I'm lacking in YAML imagination, that will require at least some config/code changes to support, perhaps in addition to using YAML pointers as you demonstrate.

Right, a pointer like that already works for that simple case. But going back to the original ask, it's basically: "I want to have one config file with multiple different repositories, sharing almost all of their config, but with different consistency checks for each repository." Unless I'm lacking in YAML imagination, that will require at least some config/code changes to support, perhaps in addition to using YAML pointers as you demonstrate.
witten commented 2018-10-08 05:16:15 +00:00
Owner
Copy Link

Quick requirements question for both of you: Do you want want to run n consistency checks on one repository, and zero consistency checks on the other? Or is it instead n consistency checks on one repository, and m consistency checks on the other?

Quick requirements question for both of you: Do you want want to run n consistency checks on one repository, and zero consistency checks on the other? Or is it instead n consistency checks on one repository, and m consistency checks on the other?
flesser commented 2018-10-08 08:05:34 +00:00
Author
Copy Link

On my current setup it is indeed n:0.

(The more flexible configuration variant would of course be be n:m. However, I don't think I currently would need that)

On my current setup it is indeed `n:0`. (The more flexible configuration variant would of course be be `n:m`. However, I don't think I currently would need that)
witten commented 2018-10-13 22:08:10 +00:00
Owner
Copy Link

Okay, then what I'm thinking is the adding support for the following to the consistency section of the config file:

consistency:
  check_repositories:
    - user@backupserver:sourcehostname.borg
  ...

The idea is that if the check_repositories option is supplied, then consistency checks would only run on those listed repositories. Otherwise, if not supplied, checks would run on all repositories (the current behavior).

Let me know if that sounds like it would work for you. It's a much less far-reaching / flexible proposal than some of the options discussed above, but I think it does satisfy the immediate need. And it'd be pretty easy to implement.

Okay, then what I'm thinking is the adding support for the following to the consistency section of the config file: ```yaml consistency: check_repositories: - user@backupserver:sourcehostname.borg ... ``` The idea is that if the `check_repositories` option is supplied, then consistency checks would *only* run on those listed repositories. Otherwise, if not supplied, checks would run on all repositories (the current behavior). Let me know if that sounds like it would work for you. It's a much less far-reaching / flexible proposal than some of the options discussed above, but I think it does satisfy the immediate need. And it'd be pretty easy to implement.
witten commented 2018-10-14 19:16:48 +00:00
Owner
Copy Link

Well, I went ahead and implemented this. Just released as part of borgmatic 1.2.8. Give it a shot and let me know how it works out for you!

Well, I went ahead and implemented this. Just released as part of borgmatic 1.2.8. Give it a shot and let me know how it works out for you!
witten added the
design finalized
 label 2018-10-14 19:16:58 +00:00
patricko commented 2018-12-25 11:09:13 +00:00
Copy Link

I don't know if I have borgmatic 1.2.8 there is no --version switch :P

I don't know if I have borgmatic 1.2.8 there is no --version switch :P
witten commented 2018-12-25 17:50:15 +00:00
Owner
Copy Link

You can run the following to get the version: pip3 show borgmatic. If a --version flag would be convenient though, feel free to file a ticket for it!

You can run the following to get the version: `pip3 show borgmatic`. If a `--version` flag would be convenient though, feel free to file a ticket for it!
witten commented 2018-12-26 05:06:50 +00:00
Owner
Copy Link

FYI, I just added a borgmatic --version flag as well.

FYI, I just added a `borgmatic --version` flag as well.
patricko commented 2018-12-29 15:02:14 +00:00
Copy Link

ok I have 1.2.12

So what is there to test? (if you already wrote how to, link the message)

ok I have 1.2.12 So what is there to test? (if you already wrote how to, link the message)
witten commented 2018-12-29 21:21:48 +00:00
Owner
Copy Link

See the comment above on how to configure consistency checks just for a subset of your repositories. And here's the comment on this feature from the newly generated config:

# Paths to a subset of the repositories in the location section on which to run
# consistency checks. Handy in case some of your repositories are very large, and
# so running consistency checks on them would take too long. Defaults to running
# consistency checks on all repositories configured in the location section.
See the [comment above](https://projects.torsion.org/witten/borgmatic/issues/73#issuecomment-909) on how to configure consistency checks just for a subset of your repositories. And here's the comment on this feature from the newly generated config: ```raw # Paths to a subset of the repositories in the location section on which to run # consistency checks. Handy in case some of your repositories are very large, and # so running consistency checks on them would take too long. Defaults to running # consistency checks on all repositories configured in the location section. ```
patricko commented 2018-12-31 19:12:49 +00:00
Copy Link

Ah ok, well I guess it doesn't require any testing lol and I run consistency checks on all my repos (2)

Ah ok, well I guess it doesn't require any testing lol and I run consistency checks on all my repos (2)
witten commented 2019-03-06 18:41:52 +00:00
Owner
Copy Link

I put together a proposal for borgmatic configuration includes and reuse. Check it out here: #148. And please feel free to comment with your feedback / ideas.

I put together a proposal for borgmatic configuration includes and reuse. Check it out here: #148. And please feel free to comment with your feedback / ideas.
witten commented 2020-01-25 23:15:56 +00:00
Owner
Copy Link

FYI, in case anyone's still interested.. Some discussion on this general topic of configuration varying per-repository: witten/borgmatic#281

FYI, in case anyone's still interested.. Some discussion on this general topic of configuration varying per-repository: https://projects.torsion.org/witten/borgmatic/issues/281
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
main
spot-check
1.8.10
1.8.9
1.8.8
1.8.7
1.8.6
1.8.5
1.8.4
1.8.3
1.8.2
1.8.1
1.8.0
1.7.15
1.7.14
1.7.13
1.7.12
1.7.11
1.7.10
1.7.9
1.7.8
1.7.7
1.7.6
1.7.5
1.7.4
1.7.3
1.7.2
1.7.1
1.7.0
1.6.6
1.6.5
1.6.4
1.6.3
1.6.2
1.6.1
1.6.0
1.5.24
1.5.23
1.5.22
1.5.21
1.5.20
1.5.19
1.5.18
1.5.17
1.5.16
1.5.15
1.5.14
1.5.13
1.5.12
1.5.11
1.5.10
1.5.9
1.5.8
1.5.7
1.5.7.dev0
1.5.6
1.5.5
1.5.4
1.5.3
1.5.2
1.5.1
1.5.0
1.4.22
1.4.21
1.4.20
1.4.19
1.4.18
1.4.17
1.4.16
1.4.15
1.4.14
1.4.13
1.4.12
1.4.11
1.4.10
1.4.9
1.4.8
1.4.7
1.4.6
1.4.5
1.4.4
1.4.3
1.4.2
1.4.1
1.4.0
1.3.26
1.3.25
1.3.24
1.3.23
1.3.22
1.3.21
1.3.20
1.3.19
1.3.18
1.3.17
1.3.16
1.3.15
1.3.14
1.3.13
1.3.12
1.3.11
1.3.10
1.3.9
1.3.8
1.3.7
1.3.6
1.3.5
1.3.4
1.3.3
1.3.2
1.3.1
1.3.0
1.2.18
1.2.17
1.2.16
1.2.15
1.2.14
1.2.13
1.2.12
1.2.11
1.2.10
1.2.9
1.2.8
1.2.7
1.2.6
1.2.5
1.2.4
1.2.3
1.2.2
1.2.1
1.2.0
1.1.15
1.1.14
1.1.13
1.1.12
1.1.11
1.1.10
1.1.9
1.1.7
1.1.6
1.1.5
1.1.4
1.1.3
1.1.2
1.1.1
1.1.0
1.0.3
1.0.2
1.0.1
1.0.0
0.1.8
0.1.7
0.1.6
0.1.5
0.1.4
0.1.3
0.1.2
0.1.1
0.1.0
0.0.7
0.0.6
0.0.5
0.0.4
0.0.3
0.0.2
Labels
Clear labels   
bug

   
data loss

   
design finalized

   
good first issue

   
new feature area

   
question / support

   
security

   
waiting for response
No Label
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
4 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#73
No description provided.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?