Configuration-specific healthchecks hook but other common hooks in a single !include file #579

New Issue

DrBrynzo · 2022-09-03T07:16:34Z

DrBrynzo commented

2022-09-03 07:16:34 +00:00

What I'm trying to do and why

I would like to be able to specify a common set of options from a "!include" file yet also have a specific per-configuration healthchecks URL.

Steps to reproduce (if a bug)

I know the !include here is wrong, but this is the general idea:

<other configuration above...>
hooks:
    healthchecks: https://hc-ping.com/<healthcheck identifier>
    !include /path/to/includes/standard-hook-actions.yaml

Actual behavior (if a bug)

my_config.yaml: No valid configuration files found

summary:
my_config.yaml: Error parsing configuration file
An error occurred while parsing a configuration file at my_config.yaml:
while scanning a simple key
  in "my_config.yaml", line 36, column 5
could not find expected ':'
  in "my_config.yaml", line 37, column 1
my_config.yaml: No valid configuration files found

Need some help? https://torsion.org/borgmatic/#issues

Expected behavior (if a bug)

Other notes / implementation ideas

It seems this is a limitation of the python parser from some Google searching, but I'm wondering if there is some other way to achieve this. I'd like to have a full suite of standard hooks (before_backup, after_backup, etc.) using existing variables like "{configuration_filename}" and some custom scripts (Slack webhooks, for example) for echoing some verbose output to stdout and performing some custom hooks using curl and such without having to update a bunch (28) of borgmatic configuration files if something changes.

I have a specialized backup scripting environment and borgmatic is called inline within a larger job along with several other functions performed during maintenance. Everything to stdout is captured and parsed and displayed elsewhere so even "echo" statements are helpful for monitoring job progress.

Environment

borgmatic version: 1.7.1

borgmatic installation method: pip

Borg version: borg 1.2.2

Python version: Python 3.10.6

operating system and version: MacOS 12.5.1 and Debian 11.4

#### What I'm trying to do and why I would like to be able to specify a common set of options from a "!include" file yet also have a specific per-configuration healthchecks URL. #### Steps to reproduce (if a bug) I know the !include here is wrong, but this is the general idea: ``` <other configuration above...> hooks: healthchecks: https://hc-ping.com/<healthcheck identifier> !include /path/to/includes/standard-hook-actions.yaml ``` #### Actual behavior (if a bug) ``` my_config.yaml: No valid configuration files found summary: my_config.yaml: Error parsing configuration file An error occurred while parsing a configuration file at my_config.yaml: while scanning a simple key in "my_config.yaml", line 36, column 5 could not find expected ':' in "my_config.yaml", line 37, column 1 my_config.yaml: No valid configuration files found Need some help? https://torsion.org/borgmatic/#issues ``` #### Expected behavior (if a bug) #### Other notes / implementation ideas It seems this is a limitation of the python parser from some Google searching, but I'm wondering if there is some other way to achieve this. I'd like to have a full suite of standard hooks (before_backup, after_backup, etc.) using existing variables like "{configuration_filename}" and some custom scripts (Slack webhooks, for example) for echoing some verbose output to stdout and performing some custom hooks using curl and such without having to update a bunch (28) of borgmatic configuration files if something changes. I have a specialized backup scripting environment and borgmatic is called inline within a larger job along with several other functions performed during maintenance. Everything to stdout is captured and parsed and displayed elsewhere so even "echo" statements are helpful for monitoring job progress. #### Environment **borgmatic version:** 1.7.1 **borgmatic installation method:** pip **Borg version:** borg 1.2.2 **Python version:** Python 3.10.6 **operating system and version:** MacOS 12.5.1 and Debian 11.4

xLogiiCx commented

2022-10-04 09:44:23 +00:00

Try it like this:
config.yaml:

# Shell commands, scripts, or integrations to execute at various
# points during a borgmatic run. IMPORTANT: All provided commands and
# scripts are executed with user permissions of borgmatic. Do not
# forget to set secure permissions on this configuration file (chmod
# 0600) as well as on any script called from a hook (chmod 0700) to
# prevent potential shell injection or privilege escalation.
hooks:
    <<: !include /etc/borgmatic/common_hooks.yaml
    healthchecks:
        #ping_url: https://hc-ping.com/xxxxxx-xxxx-xxxx-xxxx-xxxxxx

common_hooks.yaml

    healthchecks:
        ping_url: https://hc-ping.com/xxxxxx-xxxx-xxxx-xxxx-xxxxxx
        send_logs: true
    cronitor:
        ping_url: https://cronitor.link/p/xxxxxxx/xxxxx

Try it like this: config.yaml: ```bash # Shell commands, scripts, or integrations to execute at various # points during a borgmatic run. IMPORTANT: All provided commands and # scripts are executed with user permissions of borgmatic. Do not # forget to set secure permissions on this configuration file (chmod # 0600) as well as on any script called from a hook (chmod 0700) to # prevent potential shell injection or privilege escalation. hooks: <<: !include /etc/borgmatic/common_hooks.yaml healthchecks: #ping_url: https://hc-ping.com/xxxxxx-xxxx-xxxx-xxxx-xxxxxx ``` common_hooks.yaml ``` healthchecks: ping_url: https://hc-ping.com/xxxxxx-xxxx-xxxx-xxxx-xxxxxx send_logs: true cronitor: ping_url: https://cronitor.link/p/xxxxxxx/xxxxx ```

witten commented

2022-10-04 16:16:40 +00:00

Thanks for commenting.. Yes, that looks like it would work! The key detail is that the plain !include style of include is only for pulling in individual values. If, instead, you want to reuse configuration keys and values, you'll need the <<: !include style merge include.

I bet the following would also work if you want an even more targeted include:

config.yaml:

hooks:
    healthchecks:
        <<: !include /etc/borgmatic/common_healthchecks.yaml

common_healthchecks.yaml:

ping_url: https://hc-ping.com/xxxxxx-xxxx-xxxx-xxxx-xxxxxx
send_logs: true

Thanks for commenting.. Yes, that looks like it would work! The key detail is that the plain `!include` style of include is only for pulling in individual values. If, instead, you want to reuse configuration keys *and* values, you'll need the `<<: !include` style merge include. I bet the following would also work if you want an even more targeted include: config.yaml: ``` hooks: healthchecks: <<: !include /etc/borgmatic/common_healthchecks.yaml ``` common_healthchecks.yaml: ``` ping_url: https://hc-ping.com/xxxxxx-xxxx-xxxx-xxxx-xxxxxx send_logs: true ```

witten commented

2022-10-04 16:16:56 +00:00

I'll close this for now, but please feel free to post any follow-up questions here.

witten closed this issue

2022-10-04 16:16:56 +00:00

witten added the

question / support

label 2022-10-04 16:17:11 +00:00

DrBrynzo commented

2022-10-07 01:25:43 +00:00

I'm not sure the two examples address what I was asking unless I'm missing something. All they appear to do is move the problem to a common external file instead of the main configuration file.

Putting "ping_url" in the example "common_healthchecks.yaml" still means I would have one unified hc-ping.com URL called by every backup configuration.

The goal would be to have a common set of actions for the standard hooks (before_backup, after_prune, etc.) but maintain separate hc-ping.com URLs for the "healthchecks" in each backup configuration file.

I'm not sure the two examples address what I was asking unless I'm missing something. All they appear to do is move the problem to a common external file instead of the main configuration file. Putting "ping_url" in the example "common_healthchecks.yaml" still means I would have one unified hc-ping.com URL called by every backup configuration. The goal would be to have a common set of actions for the standard hooks (before_backup, after_prune, etc.) but maintain separate hc-ping.com URLs for the "healthchecks" in each backup configuration file.

witten commented

2022-10-07 18:05:43 +00:00

Oh, my bad.. I misunderstood. If you want a different Healthchecks ping URL in each of several configuration files and a common set of other hooks shared between them, you can do something like this:

standard-hook-actions.yaml:

before_backup: ...
after_prune: ...

individual_backup_configuration.yaml:

<other configuration above...>

hooks:
    healthchecks: https://hc-ping.com/<healthcheck identifier>
    <<: !include /path/to/includes/standard-hook-actions.yaml

Does something like that work for your use case?

Oh, my bad.. I misunderstood. If you want a different Healthchecks ping URL in each of several configuration files and a common set of other hooks shared between them, you can do something like this: standard-hook-actions.yaml: ``` before_backup: ... after_prune: ... ``` individual_backup_configuration.yaml: ``` <other configuration above...> hooks: healthchecks: https://hc-ping.com/<healthcheck identifier> <<: !include /path/to/includes/standard-hook-actions.yaml ``` Does something like that work for your use case?

witten reopened this issue

2022-10-07 18:05:47 +00:00

DrBrynzo commented

2022-10-15 00:22:06 +00:00

It looks like it will at face value. I'll check it out early next week. I've been migrating several repos to new hardware this week and imposed a configuration freeze until that was complete today.

witten commented

2022-10-15 00:59:10 +00:00

Great, I'll leave this open for now until you can confirm one way or the other. I hope it works out!

DrBrynzo commented

2022-10-26 01:13:33 +00:00

Just tried this on a simple test configuration with some simple "echo" outputs and it seemed to work. I think this is good to go but I'll migrate it to a couple production configs and make sure. You can probably call this closed.

Thanks for sticking with it. Much appreciated.

Just tried this on a simple test configuration with some simple "echo" outputs and it seemed to work. I think this is good to go but I'll migrate it to a couple production configs and make sure. You can probably call this closed. Thanks for sticking with it. Much appreciated.

👍 1

DrBrynzo closed this issue

2022-10-26 01:13:34 +00:00

Sign in to join this conversation.