borgmatic-collective/borgmatic
borgmatic-collective
/
borgmatic
15
56
Fork 39
Code Issues 52 Pull Requests 1 Actions Packages Releases 121 Activity

Support custom variables in borgmatic config files #472

New Issue
Closed
opened 2021-11-27 15:12:23 +00:00 by tikeyknax · 6 comments
tikeyknax commented 2021-11-27 15:12:23 +00:00
Copy Link

What I'm trying to do and why

I have several config files for borgmatic set up that all backup to the same repository. The reason to have them all back up to the same repository is the awesome deduplication feature of borg. The reason to use several config files instead of one is to e.g. use different retention policies.

Imagine e.g. the following config files:

Config pictures
location:
    source_directories:
        - /home/my_user/pictures

    <<: !include /etc/borgmatic.d/common/common_location.yaml

storage:
    archive_name_format: "{hostname}-pictures-{now:%Y-%m-%dT%H:%M:%S}"

    <<: !include /etc/borgmatic.d/common/common_storage.yaml


retention:
    <<: !include /etc/borgmatic.d/common/retention_1year.yaml

    prefix: "{hostname}-pictures-"

consistency:
    prefix: "{hostname}-pictures-"

hooks:
    on_error:
        - echo "[borgmatic - pictures] ERROR Error during prune/create/check." | mail -s "[borgmatic - pictures] ERROR" -r alert@mydomain.com alert@mydomain.com

    before_everything:
        - echo "[borgmatic - pictures] Starting actions."

    after_everything:
        - echo "[borgmatic - pictures] Finished all actions."
Config some_folder
location:
    source_directories:
        - /home/my_user/some_folder

    <<: !include /etc/borgmatic.d/common/common_location.yaml

storage:
    archive_name_format: "{hostname}-somefolder-{now:%Y-%m-%dT%H:%M:%S}"

    <<: !include /etc/borgmatic.d/common/common_storage.yaml


retention:
    <<: !include /etc/borgmatic.d/common/retention_short.yaml

    prefix: "{hostname}-somefolder-"

consistency:
    prefix: "{hostname}-somefolder-"

hooks:
    on_error:
        - echo "[borgmatic - somefolder] ERROR Error during prune/create/check." | mail -s "[borgmatic - somefolder] ERROR" -r alert@mydomain.com alert@mydomain.com

    before_everything:
        - echo "[borgmatic - somefolder] Starting actions."

    after_everything:
        - echo "[borgmatic - somefolder] Finished all actions."

Now, for only 2 configs this is still fairly easy but having more configs, it would be really nice if one could even shorten these configs. So if borgmatic would support (some) custom variables, the 2 config files could be simplified even further:

Config pictures
variables:
    app_name: pictures
    
location:
    source_directories:
        - /home/my_user/pictures

    <<: !include /etc/borgmatic.d/common/common_location.yaml

storage:
    archive_name_format: "{hostname}-{app_name}-{now:%Y-%m-%dT%H:%M:%S}"

    <<: !include /etc/borgmatic.d/common/common_storage.yaml


retention:
    <<: !include /etc/borgmatic.d/common/retention_1year.yaml

    prefix: "{hostname}-{app_name}-"

consistency:
    prefix: "{hostname}-{app_name}-"

hooks:
    on_error:
        - echo "[borgmatic - {app_name}] ERROR Error during prune/create/check." | mail -s "[borgmatic - pictures] ERROR" -r alert@mydomain.com alert@mydomain.com

    before_everything:
        - echo "[borgmatic - {app_name}] Starting actions."

    after_everything:
        - echo "[borgmatic - {app_name}] Finished all actions."
Config some_folder
variables:
    app_name: some_folder
    
location:
    source_directories:
        - /home/my_user/some_folder

    <<: !include /etc/borgmatic.d/common/common_location.yaml

storage:
    archive_name_format: "{hostname}-{app_name}-{now:%Y-%m-%dT%H:%M:%S}"

    <<: !include /etc/borgmatic.d/common/common_storage.yaml


retention:
    <<: !include /etc/borgmatic.d/common/retention_short.yaml

    prefix: "{hostname}-{app_name}-"

consistency:
    prefix: "{hostname}-{app_name}-"

hooks:
    on_error:
        - echo "[borgmatic - {app_name}] ERROR Error during prune/create/check." | mail -s "[borgmatic - somefolder] ERROR" -r alert@mydomain.com alert@mydomain.com

    before_everything:
        - echo "[borgmatic - {app_name}] Starting actions."

    after_everything:
        - echo "[borgmatic - {app_name}] Finished all actions."

Now, the archive_name_format, the prefixes and the hooks are exactly the same in both files and can thus be moved into the include files and one would end up with something like:

Config pictures
variables:
    app_name: pictures
    
location:
    source_directories:
        - /home/my_user/pictures

    <<: !include /etc/borgmatic.d/common/common_location.yaml

storage:
    <<: !include /etc/borgmatic.d/common/common_storage.yaml


retention:
    <<: !include /etc/borgmatic.d/common/retention_short.yaml

consistency:
    <<: !include /etc/borgmatic.d/common/common_consistency.yaml

hooks:
    <<: !include /etc/borgmatic.d/common/common_hooks.yaml
Config some_folder
variables:
    app_name: some_folder
    
location:
    source_directories:
        - /home/my_user/some_folder

    <<: !include /etc/borgmatic.d/common/common_location.yaml

storage:
    <<: !include /etc/borgmatic.d/common/common_storage.yaml


retention:
    <<: !include /etc/borgmatic.d/common/retention_short.yaml

consistency:
    <<: !include /etc/borgmatic.d/common/common_consistency.yaml

hooks:
    <<: !include /etc/borgmatic.d/common/common_hooks.yaml
#### What I'm trying to do and why I have several config files for borgmatic set up that all backup to the same repository. The reason to have them all back up to the same repository is the awesome deduplication feature of borg. The reason to use several config files instead of one is to e.g. use different retention policies. Imagine e.g. the following config files: ##### Config pictures ``` location: source_directories: - /home/my_user/pictures <<: !include /etc/borgmatic.d/common/common_location.yaml storage: archive_name_format: "{hostname}-pictures-{now:%Y-%m-%dT%H:%M:%S}" <<: !include /etc/borgmatic.d/common/common_storage.yaml retention: <<: !include /etc/borgmatic.d/common/retention_1year.yaml prefix: "{hostname}-pictures-" consistency: prefix: "{hostname}-pictures-" hooks: on_error: - echo "[borgmatic - pictures] ERROR Error during prune/create/check." | mail -s "[borgmatic - pictures] ERROR" -r alert@mydomain.com alert@mydomain.com before_everything: - echo "[borgmatic - pictures] Starting actions." after_everything: - echo "[borgmatic - pictures] Finished all actions." ``` ##### Config some_folder ``` location: source_directories: - /home/my_user/some_folder <<: !include /etc/borgmatic.d/common/common_location.yaml storage: archive_name_format: "{hostname}-somefolder-{now:%Y-%m-%dT%H:%M:%S}" <<: !include /etc/borgmatic.d/common/common_storage.yaml retention: <<: !include /etc/borgmatic.d/common/retention_short.yaml prefix: "{hostname}-somefolder-" consistency: prefix: "{hostname}-somefolder-" hooks: on_error: - echo "[borgmatic - somefolder] ERROR Error during prune/create/check." | mail -s "[borgmatic - somefolder] ERROR" -r alert@mydomain.com alert@mydomain.com before_everything: - echo "[borgmatic - somefolder] Starting actions." after_everything: - echo "[borgmatic - somefolder] Finished all actions." ``` Now, for only 2 configs this is still fairly easy but having more configs, it would be really nice if one could even shorten these configs. So if borgmatic would support (some) custom variables, the 2 config files could be simplified even further: ##### Config pictures ``` variables: app_name: pictures location: source_directories: - /home/my_user/pictures <<: !include /etc/borgmatic.d/common/common_location.yaml storage: archive_name_format: "{hostname}-{app_name}-{now:%Y-%m-%dT%H:%M:%S}" <<: !include /etc/borgmatic.d/common/common_storage.yaml retention: <<: !include /etc/borgmatic.d/common/retention_1year.yaml prefix: "{hostname}-{app_name}-" consistency: prefix: "{hostname}-{app_name}-" hooks: on_error: - echo "[borgmatic - {app_name}] ERROR Error during prune/create/check." | mail -s "[borgmatic - pictures] ERROR" -r alert@mydomain.com alert@mydomain.com before_everything: - echo "[borgmatic - {app_name}] Starting actions." after_everything: - echo "[borgmatic - {app_name}] Finished all actions." ``` ##### Config some_folder ``` variables: app_name: some_folder location: source_directories: - /home/my_user/some_folder <<: !include /etc/borgmatic.d/common/common_location.yaml storage: archive_name_format: "{hostname}-{app_name}-{now:%Y-%m-%dT%H:%M:%S}" <<: !include /etc/borgmatic.d/common/common_storage.yaml retention: <<: !include /etc/borgmatic.d/common/retention_short.yaml prefix: "{hostname}-{app_name}-" consistency: prefix: "{hostname}-{app_name}-" hooks: on_error: - echo "[borgmatic - {app_name}] ERROR Error during prune/create/check." | mail -s "[borgmatic - somefolder] ERROR" -r alert@mydomain.com alert@mydomain.com before_everything: - echo "[borgmatic - {app_name}] Starting actions." after_everything: - echo "[borgmatic - {app_name}] Finished all actions." ``` Now, the archive_name_format, the prefixes and the hooks are exactly the same in both files and can thus be moved into the include files and one would end up with something like: ##### Config pictures ``` variables: app_name: pictures location: source_directories: - /home/my_user/pictures <<: !include /etc/borgmatic.d/common/common_location.yaml storage: <<: !include /etc/borgmatic.d/common/common_storage.yaml retention: <<: !include /etc/borgmatic.d/common/retention_short.yaml consistency: <<: !include /etc/borgmatic.d/common/common_consistency.yaml hooks: <<: !include /etc/borgmatic.d/common/common_hooks.yaml ``` ##### Config some_folder ``` variables: app_name: some_folder location: source_directories: - /home/my_user/some_folder <<: !include /etc/borgmatic.d/common/common_location.yaml storage: <<: !include /etc/borgmatic.d/common/common_storage.yaml retention: <<: !include /etc/borgmatic.d/common/retention_short.yaml consistency: <<: !include /etc/borgmatic.d/common/common_consistency.yaml hooks: <<: !include /etc/borgmatic.d/common/common_hooks.yaml ```
tikeyknax commented 2021-12-11 21:13:26 +00:00
Author
Copy Link

I figured out that {configuration_filename} is not only supported in the on error hook but also in the other hooks. So a way to at least unify the hooks for all the backups is:

after_backup:
- echo "[backup] Finished backup in `echo {configuration_filename} | awk -F\/ '{print $NF}'` on `date +'%a %d.%m.%Y - %H:%M:%S'`"

I have not yet checked if this would also work for the prefixes.

I figured out that `{configuration_filename}` is not only supported in the on error hook but also in the other hooks. So a way to at least unify the hooks for all the backups is: ``` after_backup: - echo "[backup] Finished backup in `echo {configuration_filename} | awk -F\/ '{print $NF}'` on `date +'%a %d.%m.%Y - %H:%M:%S'`" ``` I have not yet checked if this would also work for the prefixes.
witten commented 2022-04-26 04:44:04 +00:00
Owner
Copy Link

Sorry for the delay here. This is a pretty interesting idea! I'll have to think about how this would work given that the interpolation would (presumably) have to work for all values and not just within hooks like the existing borgmatic variable interpolation.

Anyway, I wanted to mention that with #381 now implemented in master, you will be able to simplify configuration like the following:

location:
    source_directories:
        - /home/my_user/some_folder

    <<: !include /etc/borgmatic.d/common/common_location.yaml

storage:
    <<: !include /etc/borgmatic.d/common/common_storage.yaml


retention:
    <<: !include /etc/borgmatic.d/common/retention_short.yaml

consistency:
    <<: !include /etc/borgmatic.d/common/common_consistency.yaml

hooks:
    <<: !include /etc/borgmatic.d/common/common_hooks.yaml

... to something more like this:

<<: !include /etc/borgmatic.d/common/common.yaml

location:
    source_directories:
        - /home/my_user/some_folder

retention:
    <<: !include /etc/borgmatic.d/common/retention_short.yaml

That wouldn't support the custom variables described in this ticket (yet), but it would at minimum cut down on some of the boilerplate repeated in each configuration file.

Sorry for the delay here. This is a pretty interesting idea! I'll have to think about how this would work given that the interpolation would (presumably) have to work for all values and not just within hooks like the existing borgmatic variable interpolation. Anyway, I wanted to mention that with #381 now implemented in master, you will be able to simplify configuration like the following: ```yaml location: source_directories: - /home/my_user/some_folder <<: !include /etc/borgmatic.d/common/common_location.yaml storage: <<: !include /etc/borgmatic.d/common/common_storage.yaml retention: <<: !include /etc/borgmatic.d/common/retention_short.yaml consistency: <<: !include /etc/borgmatic.d/common/common_consistency.yaml hooks: <<: !include /etc/borgmatic.d/common/common_hooks.yaml ``` ... to something more like this: ```yaml <<: !include /etc/borgmatic.d/common/common.yaml location: source_directories: - /home/my_user/some_folder retention: <<: !include /etc/borgmatic.d/common/retention_short.yaml ``` That wouldn't support the custom variables described in this ticket (yet), but it would at minimum cut down on some of the boilerplate repeated in each configuration file.
tikeyknax commented 2022-05-07 08:11:49 +00:00
Author
Copy Link

Thanks for the hint and for implementing #381. This is indeed already a very big step. If that possibility existed when I was writing my config files, I wouldn't have opened this ticket. :)

Thanks for the hint and for implementing #381. This is indeed already a very big step. If that possibility existed when I was writing my config files, I wouldn't have opened this ticket. :)
witten commented 2022-05-07 16:30:54 +00:00
Owner
Copy Link

Glad to hear it might be useful. Now that #381 is released, do you still have the need for custom variables? Either way, I just wanted to confirm. Thanks!

Glad to hear it might be useful. Now that #381 is released, do you still have the need for custom variables? Either way, I just wanted to confirm. Thanks!
tikeyknax commented 2022-05-07 18:24:47 +00:00
Author
Copy Link

Thanks for getting back to me. I think there would be an advantage fo also having the variables but to be honest it is now rather small. So I'm totally happy with what you have done.

Thanks for getting back to me. I think there would be an advantage fo also having the variables but to be honest it is now rather small. So I'm totally happy with what you have done.
witten commented 2022-05-08 16:09:16 +00:00
Owner
Copy Link

Okay, I'll close this for now. But if the need for the feature increases or anyone else has the same use case, please feel free to reopen!

Okay, I'll close this for now. But if the need for the feature increases or anyone else has the same use case, please feel free to reopen!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
main
spot-check
1.8.10
1.8.9
1.8.8
1.8.7
1.8.6
1.8.5
1.8.4
1.8.3
1.8.2
1.8.1
1.8.0
1.7.15
1.7.14
1.7.13
1.7.12
1.7.11
1.7.10
1.7.9
1.7.8
1.7.7
1.7.6
1.7.5
1.7.4
1.7.3
1.7.2
1.7.1
1.7.0
1.6.6
1.6.5
1.6.4
1.6.3
1.6.2
1.6.1
1.6.0
1.5.24
1.5.23
1.5.22
1.5.21
1.5.20
1.5.19
1.5.18
1.5.17
1.5.16
1.5.15
1.5.14
1.5.13
1.5.12
1.5.11
1.5.10
1.5.9
1.5.8
1.5.7
1.5.7.dev0
1.5.6
1.5.5
1.5.4
1.5.3
1.5.2
1.5.1
1.5.0
1.4.22
1.4.21
1.4.20
1.4.19
1.4.18
1.4.17
1.4.16
1.4.15
1.4.14
1.4.13
1.4.12
1.4.11
1.4.10
1.4.9
1.4.8
1.4.7
1.4.6
1.4.5
1.4.4
1.4.3
1.4.2
1.4.1
1.4.0
1.3.26
1.3.25
1.3.24
1.3.23
1.3.22
1.3.21
1.3.20
1.3.19
1.3.18
1.3.17
1.3.16
1.3.15
1.3.14
1.3.13
1.3.12
1.3.11
1.3.10
1.3.9
1.3.8
1.3.7
1.3.6
1.3.5
1.3.4
1.3.3
1.3.2
1.3.1
1.3.0
1.2.18
1.2.17
1.2.16
1.2.15
1.2.14
1.2.13
1.2.12
1.2.11
1.2.10
1.2.9
1.2.8
1.2.7
1.2.6
1.2.5
1.2.4
1.2.3
1.2.2
1.2.1
1.2.0
1.1.15
1.1.14
1.1.13
1.1.12
1.1.11
1.1.10
1.1.9
1.1.7
1.1.6
1.1.5
1.1.4
1.1.3
1.1.2
1.1.1
1.1.0
1.0.3
1.0.2
1.0.1
1.0.0
0.1.8
0.1.7
0.1.6
0.1.5
0.1.4
0.1.3
0.1.2
0.1.1
0.1.0
0.0.7
0.0.6
0.0.5
0.0.4
0.0.3
0.0.2
Labels
Clear labels   
bug

   
data loss

   
design finalized

   
good first issue

   
new feature area

   
question / support

   
security

   
waiting for response
No Label
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#472
No description provided.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?