borgmatic-collective/borgmatic
borgmatic-collective/borgmatic
14
58
Fork 42
Code Issues 62 Pull Requests 2 Actions Packages Releases 122 Activity

define custom variable in configuration file #612

New Issue
Closed
opened 2022-11-21 10:43:36 +00:00 by jvies · 16 comments
jvies commented 2022-11-21 10:43:36 +00:00
Copy Link

What I'm trying to do and why

I'd like to deploy a per application (containerized web apps) configuration with a factorization of my configuration.

In my current configuration, I use the same borg repo for all my apps, but with a specific prefix for each one.

Steps to reproduce (if a bug)

Actual behavior (if a bug)

I need to specify my prefix in several sections (storage, retention, consistency, and ntfy).

Expected behavior (if a bug)

I'd like to define a variable "prefix" that I can use in all my prefix entries. And I'd like this variable to be expendanded after the configuration files merge.

Other notes / implementation ideas

Environment

borgmatic version: 1.7.4

borgmatic installation method: Debian package (from testing)

Borg version: 1.2.2

Python version: 3.9.2

Database version (if applicable): NA

operating system and version: Debian 11

#### What I'm trying to do and why I'd like to deploy a per application (containerized web apps) configuration with a factorization of my configuration. In my current configuration, I use the same borg repo for all my apps, but with a specific prefix for each one. #### Steps to reproduce (if a bug) #### Actual behavior (if a bug) I need to specify my prefix in several sections (storage, retention, consistency, and ntfy). #### Expected behavior (if a bug) I'd like to define a variable "prefix" that I can use in all my prefix entries. And I'd like this variable to be expendanded after the configuration files merge. #### Other notes / implementation ideas #### Environment **borgmatic version:** 1.7.4 **borgmatic installation method:** Debian package (from testing) **Borg version:** 1.2.2 **Python version:** 3.9.2 **Database version (if applicable):** NA **operating system and version:** Debian 11
witten commented 2022-11-21 17:01:30 +00:00
Owner
Copy Link

Have you seen the environment variable interpolation feature? Does that work for your use case? Or do you actually need to lay down the variable values within the configuration before runtime for some reason?

Have you seen the [environment variable interpolation](https://torsion.org/borgmatic/docs/how-to/provide-your-passwords/#environment-variable-interpolation) feature? Does that work for your use case? Or do you actually need to lay down the variable values within the configuration before runtime for some reason?
jvies commented 2022-11-23 13:08:43 +00:00
Author
Copy Link

Hi,
My borgmatic jobs are triggered by the systemd timer. Since a few releases, the debian package deploys the borgmatic.service and borgmatic.timer. Since then, I replaced my custom cron jobs by the usage of this timer and associated service.
I do not see how to use the env variable interpollation in my case, as all my per application configurations uses the same environment (the one provided by the systemd service).

I think such variable mechanism would help configurations like this:

location:
    source_directories:
        - /some/directory
    repositories:
        - ssh://someuser@somehost/./somerepo
        - ssh://someotheruser@someotherhost/./someotherepo
    numeric_owner: true
storage:
    encryption_passcommand: "cat /root/.borg_passphrase"
    archive_name_format: 'metrics-{now}'
retention:
    keep_daily: 7
    keep_weekly: 4
    keep_monthly: 12
    prefix: metrics-
consistency:
    check_repositories:
        - ssh://someuser@somehost/./somerepo
    check_last: 3
    prefix: metrics-

I do check on only one repo (which might not be a good idea... I wille change this), but more important I use a custom prefix, and thus need to copy/paste this prefix in several places.

Hi, My borgmatic jobs are triggered by the systemd timer. Since a few releases, the debian package deploys the borgmatic.service and borgmatic.timer. Since then, I replaced my custom cron jobs by the usage of this timer and associated service. I do not see how to use the env variable interpollation in my case, as all my per application configurations uses the same environment (the one provided by the systemd service). I think such variable mechanism would help configurations like this: ``` location: source_directories: - /some/directory repositories: - ssh://someuser@somehost/./somerepo - ssh://someotheruser@someotherhost/./someotherepo numeric_owner: true storage: encryption_passcommand: "cat /root/.borg_passphrase" archive_name_format: 'metrics-{now}' retention: keep_daily: 7 keep_weekly: 4 keep_monthly: 12 prefix: metrics- consistency: check_repositories: - ssh://someuser@somehost/./somerepo check_last: 3 prefix: metrics- ``` I do check on only one repo (which might not be a good idea... I wille change this), but more important I use a custom prefix, and thus need to copy/paste this prefix in several places.
witten commented 2022-11-23 17:18:50 +00:00
Owner
Copy Link

Got it. So you're looking for something like the following?

constants:
    my_prefix: metrics-
location:
    source_directories:
        - /some/directory
    repositories:
        - ssh://someuser@somehost/./somerepo
        - ssh://someotheruser@someotherhost/./someotherepo
    numeric_owner: true
storage:
    encryption_passcommand: "cat /root/.borg_passphrase"
    archive_name_format: '{my_prefix}{now}'
retention:
    keep_daily: 7
    keep_weekly: 4
    keep_monthly: 12
    prefix: {my_prefix}
consistency:
    check_repositories:
        - ssh://someuser@somehost/./somerepo
    check_last: 3
    prefix: {my_prefix}

Is that the sort of thing you had in mind?

Got it. So you're looking for something like the following? ``` constants: my_prefix: metrics- location: source_directories: - /some/directory repositories: - ssh://someuser@somehost/./somerepo - ssh://someotheruser@someotherhost/./someotherepo numeric_owner: true storage: encryption_passcommand: "cat /root/.borg_passphrase" archive_name_format: '{my_prefix}{now}' retention: keep_daily: 7 keep_weekly: 4 keep_monthly: 12 prefix: {my_prefix} consistency: check_repositories: - ssh://someuser@somehost/./somerepo check_last: 3 prefix: {my_prefix} ``` Is that the sort of thing you had in mind?
jvies commented 2022-11-26 07:25:29 +00:00
Author
Copy Link

Yes something like that.
Maybe "variables" instead of "constants" ?

And it would be great if these can be handled in "deepe merge". So we can define a default value in a template file, and then override it in each per-application file.

Yes something like that. Maybe "variables" instead of "constants" ? And it would be great if these can be handled in "deepe merge". So we can define a default value in a template file, and then override it in each per-application file.
witten commented 2022-11-26 17:18:01 +00:00
Owner
Copy Link

That makes sense!

That makes sense!
witten added the
design finalized
 label 2022-11-26 17:18:10 +00:00
diivi commented 2023-03-23 04:47:34 +00:00
Collaborator
Copy Link

Nice idea! I'd like to add this functionality to Borgmatic. @witten could you help me understand where the default ({now} etc) interpolation from config takes place, maybe that'll help.

Nice idea! I'd like to add this functionality to Borgmatic. @witten could you help me understand where the default ({now} etc) interpolation from config takes place, maybe that'll help.
witten commented 2023-03-23 05:01:57 +00:00
Owner
Copy Link

There are actually two levels of interpolation going on, and unfortunately I don't think either of them will work directly for this use case (although maybe some of the code could be reused/repurposed):

There are actually two levels of interpolation going on, and unfortunately I don't think either of them will work directly for this use case (although maybe some of the code could be reused/repurposed): * [borgmatic command hook variable interpolation](https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/#variable-interpolation): The borgmatic command hooks support a couple of fixed variables, useful for passing borgmatic information into scripts. The code for this is at `borgmatic/hooks/command.py:interpolate_context(). * [Borg archive name placeholders](https://borgbackup.readthedocs.io/en/stable/usage/create.html#description): This includes placeholders like `{now}`, used in generating archive names.
👀 1
diivi commented 2023-03-23 06:23:07 +00:00
Collaborator
Copy Link

Edit: Skip this and see the solution below.

Right, I was able to get this working with a "trick", at the end of normalize.py, and at the beginning of borgmatic.py::run_configuration(), I added this:

    config_str = str(config)
    if 'constants' in config:
        # replace every occurrence of {key} in the config with the value of key
        # in the constants section, convert config to string to do the replacement
        # and then convert it back to a dict
        for key, value in config['constants'].items():
            config_str = config_str.replace(f'{{{key}}}', value)

    config = eval(config_str)

However, this works for the config you mentioned above only when you add quotes to string parameters, like so:

constants:
  x: /home/divyansh/Desktop/os
location:
    # List of source directories to backup. Globs and tildes are
    # expanded. Do not backslash spaces in path names.
    source_directories:
        - '{x}'

Otherwise yaml interprets it as a dict. Now we can get this to work for strings by adding quotes, but what about integer params etc.?

Two options:

  1. Somehow tell yaml to interpret this field as not a dict?
  2. Give every parameter between quotes, interpret the type of the variable (int/bool/str?) and replace the string by this value internally, and continue execution.

Let me know if you have a better option too.

Edit: Skip this and see the solution below. Right, I was able to get this working with a "trick", at the end of `normalize.py`, and at the beginning of `borgmatic.py::run_configuration()`, I added this: ```python config_str = str(config) if 'constants' in config: # replace every occurrence of {key} in the config with the value of key # in the constants section, convert config to string to do the replacement # and then convert it back to a dict for key, value in config['constants'].items(): config_str = config_str.replace(f'{{{key}}}', value) config = eval(config_str) ``` However, this works for the config you mentioned [above](https://projects.torsion.org/borgmatic-collective/borgmatic/issues/612#issuecomment-5188) only when you add quotes to string parameters, like so: ``` constants: x: /home/divyansh/Desktop/os location: # List of source directories to backup. Globs and tildes are # expanded. Do not backslash spaces in path names. source_directories: - '{x}' ``` Otherwise yaml interprets it as a dict. Now we can get this to work for strings by adding quotes, but what about integer params etc.? Two options: 1. Somehow tell yaml to interpret this field as not a dict? 2. Give every parameter between quotes, interpret the type of the variable (`int`/`bool`/`str`?) and replace the string by this value internally, and continue execution. Let me know if you have a better option too.
diivi commented 2023-03-23 08:20:23 +00:00
Collaborator
Copy Link

Simpler solution that supports all types too:

load.py::load_configuration() (I think this is where Borgmatic first checks the config file too).

with open(filename) as f:
    file_contents = f.read()
    config = yaml.load(file_contents)
    if 'constants' in config:
        for key, value in config['constants'].items():
            file_contents = file_contents.replace(f'{{{key}}}', str(value))
    return yaml.load(file_contents)

I can move this to another function, and this seems to be a cleaner and straightforward approach if I am not missing anything.

Simpler solution that supports all types too: `load.py::load_configuration()` (I think this is where Borgmatic first checks the config file too). ```python with open(filename) as f: file_contents = f.read() config = yaml.load(file_contents) if 'constants' in config: for key, value in config['constants'].items(): file_contents = file_contents.replace(f'{{{key}}}', str(value)) return yaml.load(file_contents) ``` I can move this to another function, and this seems to be a cleaner and straightforward approach if I am not missing anything.
witten commented 2023-03-23 18:16:53 +00:00
Owner
Copy Link

Your most recent approach seems totally reasonable to me. A couple of thoughts:

  • The only optimization I would suggest is that if constants is not found in the configuration, then don't reload the configuration a second time.
  • I wonder if you need to actually remove constants from the loaded configuration after you consume them?
  • To support more complex data types as values in constants, you probably need to YAML-encode the value when you replace the constant name—not just call str(). For instance, let's say a constant value is an entire YAML dict or list like:
constants:
    foo:
        bar: baz

location:
    my_option: {foo}

See render_configuration() for one example of dumping YAML to a string.

Thanks for digging into this!

Your most recent approach seems totally reasonable to me. A couple of thoughts: * The only optimization I would suggest is that if `constants` is _not_ found in the configuration, then don't reload the configuration a second time. * I wonder if you need to actually *remove* `constants` from the loaded configuration after you consume them? * To support more complex data types as values in `constants`, you probably need to YAML-encode the `value` when you replace the constant name—not just call `str()`. For instance, let's say a constant value is an entire YAML dict or list like: ```yaml constants: foo: bar: baz location: my_option: {foo} ``` See `render_configuration()` for one example of dumping YAML to a string. Thanks for digging into this!
👍 1
witten commented 2023-03-25 06:48:30 +00:00
Owner
Copy Link

This is implemented in master now (thanks to @diivi!) and will be part of the next release. We did end up going with constants: though. You can find the documentation here once it goes live in the next few minutes: https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/#constant-interpolation

This is implemented in master now (thanks to @diivi!) and will be part of the next release. We did end up going with `constants:` though. You can find the documentation here once it goes live in the next few minutes: https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/#constant-interpolation
witten commented 2023-03-25 06:48:45 +00:00
Owner
Copy Link

And thank you for suggesting this feature!

And thank you for suggesting this feature!
witten commented 2023-03-27 23:22:15 +00:00
Owner
Copy Link

Released in borgmatic 1.7.10!

Released in borgmatic 1.7.10!
davidtorosyan commented 2023-07-08 05:03:06 +00:00
Copy Link

I know this is closed, but I had a question about it - let me know if I should open a new issue instead.

The original ask was:

I'd like to define a variable "prefix" that I can use in all my prefix entries. And I'd like this variable to be expendanded after the configuration files merge.
...
So we can define a default value in a template file, and then override it in each per-application file.

However, the docs for the implementation say:

Constants don't work across includes or separate configuration files though

Am I understanding right that this doesn't actually work for the original use case?

I'm asking because I'm trying to do this:

common.yaml:

storage:
    archive_name_format: '{app}-{now}'

someApp.yaml:

constants:
  app: someApp

<<: !include /etc/borgmatic/common.yaml

But that fails with:

CRITICAL Invalid placeholder "app" in string: ssh://someUser@someServer/some/path/borgmatic::{app}-{now}
I know this is closed, but I had a question about it - let me know if I should open a new issue instead. --- The original ask was: > I'd like to define a variable "prefix" that I can use in all my prefix entries. And I'd like this variable to be expendanded after the configuration files merge. > ... > So we can define a default value in a template file, and then override it in each per-application file. However, the docs for the implementation say: > Constants don't work across includes or separate configuration files though Am I understanding right that this doesn't actually work for the original use case? --- I'm asking because I'm trying to do this: common.yaml: ``` storage: archive_name_format: '{app}-{now}' ``` someApp.yaml: ``` constants: app: someApp <<: !include /etc/borgmatic/common.yaml ``` But that fails with: ``` CRITICAL Invalid placeholder "app" in string: ssh://someUser@someServer/some/path/borgmatic::{app}-{now} ```
witten commented 2023-07-08 16:39:50 +00:00
Owner
Copy Link

I think you're correct that "after configuration files merge" portion isn't implemented, but if you look at the example above, that use case is addressed. Namely, if you've got multiple prefix options in a single configuration file, you can use a user-defined constant in them. (Perhaps ironically, those particular prefix options are now deprecated.)

As for your example use case, I agree that would be a pretty useful enhancement. Would you mind filing a separate ticket for it?

I think you're correct that "after configuration files merge" portion isn't implemented, but if you look at [the example above](https://projects.torsion.org/borgmatic-collective/borgmatic/issues/612#issuecomment-5187), that use case is addressed. Namely, if you've got multiple `prefix` options in a single configuration file, you can use a user-defined constant in them. (Perhaps ironically, those particular `prefix` options are now deprecated.) As for your example use case, I agree that would be a pretty useful enhancement. Would you mind filing a separate ticket for it?
davidtorosyan commented 2023-07-08 17:15:22 +00:00
Copy Link

Got it, thanks! Will file a separate ticket.

Got it, thanks! Will file a separate ticket.
👍 1
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels   
bug

   
data loss

   
design finalized

   
good first issue

   
new feature area

   
question / support

   
security

   
waiting for response
No Label
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
4 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#612
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?