Additional environment variables for hooks #436

New Issue

ams_tschoening · 2021-07-25T10:49:19Z

ams_tschoening commented

2021-07-25 10:49:19 +00:00

What I'm trying to do and why

I've already had a look at the monitoring hooks but prefer an approach with which simply mails are send after a successful or erroneous backup using my own local mail server. Those mails should contain the host or type of backup, when things have started and ended and minor other things like free storage on the backup destination. The latter is not of borgmatic's business and easy to get using some simple script.

Though, other aspects of the the details that I would see could be somewhat easily provided by borgmatic and it does so for some hooks already. E.g. on_error not only provides an optional error argument to the executed script, but as well things like configuration_filename, which would be useful to have for all hooks in my opinion.

Additionally, especially things like when the backup started, when did it end, etc. would be far easier to forward into the hooks e.g. using environment variables instead of being maintained by the hooks on their own. An implementation of before_backup would need to work with some file e.g. storing when the backup started or would require that borgmatic logs into a file and one would need to parse that file or something like that. borgmatic could instead provide those pieces of information for each hook in a far easier way.

BORGMATIC_HOOK_CONFIG_PATH=/path/to/current_config.yaml
BORGMATIC_HOOK_BEFORE_BACKUP_START=20210725T102700.000Z
BORGMATIC_HOOK_BEFORE_BACKUP_END=[...]
BORGMATIC_HOOK_AFTER_BACKUP_START=[...]
BORGMATIC_HOOK_AFTER_BACKUP_END=[...]

In my case, the purpose of the backup is coded into the path to the config file already, which distinguishes backups for hosts, for virtual machines, for Postgres databases etc. already. I would easily be able to remove the common prefix of all those config files in some script and have something useful to use in mail subjects, which my mail client can easily filter again.

C:.
│   pve.yaml
│
├───hosts
│       org.example.host1.yaml
│       org.example.host2.yaml
│
├───mysql
│   ├───dumps
│   └───files
├───postgres
│   ├───dumps
│   ├───files_full
│   │       org.example.host1.yaml
│   │
│   └───files_incr
└───vbox
        org.example.host3.yaml
        org.example.host4.yaml

Having timestamps of each hook's start and end in a known format would allow to calculate runtimes as of interest. With timestamps per hook, one could provide multiple individual runtimes for backup, check, prune etc., all the useful stuff borgmatics executes by default already. It's additionally somewhat easy to provide those runtimes in individual lines in a mail which can easily be understood by some people without re-inventing fully fledged monitoring software. Some of those generated mails would go directly to non-technical customers and should therefore be as easy to understand as possible.

Thank's for considering!

Environment

borgmatic version: 1.5.15
borgmatic installation method: PIP, system wide
Borg version: 1.1.16
Python version: 3.8.10
operating system and version: Ubuntu 20.04

#### What I'm trying to do and why I've already had a look at the monitoring hooks but prefer an approach with which simply mails are send after a successful or erroneous backup using my own local mail server. Those mails should contain the host or type of backup, when things have started and ended and minor other things like free storage on the backup destination. The latter is not of borgmatic's business and easy to get using some simple script. Though, other aspects of the the details that I would see could be somewhat easily provided by borgmatic and it does so for some hooks already. E.g. `on_error` not only provides an optional `error` argument to the executed script, but as well things like `configuration_filename`, which would be useful to have for all hooks in my opinion. Additionally, especially things like when the backup started, when did it end, etc. would be far easier to forward into the hooks e.g. using environment variables instead of being maintained by the hooks on their own. An implementation of `before_backup` would need to work with some file e.g. storing when the backup started or would require that borgmatic logs into a file and one would need to parse that file or something like that. borgmatic could instead provide those pieces of information for each hook in a far easier way. ```bash BORGMATIC_HOOK_CONFIG_PATH=/path/to/current_config.yaml BORGMATIC_HOOK_BEFORE_BACKUP_START=20210725T102700.000Z BORGMATIC_HOOK_BEFORE_BACKUP_END=[...] BORGMATIC_HOOK_AFTER_BACKUP_START=[...] BORGMATIC_HOOK_AFTER_BACKUP_END=[...] ``` In my case, the purpose of the backup is coded into the path to the config file already, which distinguishes backups for hosts, for virtual machines, for Postgres databases etc. already. I would easily be able to remove the common prefix of all those config files in some script and have something useful to use in mail subjects, which my mail client can easily filter again. ``` C:. │ pve.yaml │ ├───hosts │ org.example.host1.yaml │ org.example.host2.yaml │ ├───mysql │ ├───dumps │ └───files ├───postgres │ ├───dumps │ ├───files_full │ │ org.example.host1.yaml │ │ │ └───files_incr └───vbox org.example.host3.yaml org.example.host4.yaml ``` Having timestamps of each hook's start and end in a known format would allow to calculate runtimes as of interest. With timestamps per hook, one could provide multiple individual runtimes for backup, check, prune etc., all the useful stuff borgmatics executes by default already. It's additionally somewhat easy to provide those runtimes in individual lines in a mail which can easily be understood by some people without re-inventing fully fledged monitoring software. Some of those generated mails would go directly to non-technical customers and should therefore be as easy to understand as possible. **Thank's for considering!** #### Environment **borgmatic version:** 1.5.15 **borgmatic installation method:** PIP, system wide **Borg version:** 1.1.16 **Python version:** 3.8.10 **operating system and version:** Ubuntu 20.04

ams_tschoening changed title from ~~Environment variables for the lifecycle of hooks~~ to Additional environment variables for hooks

2021-07-25 10:57:15 +00:00

ams_tschoening commented

2021-07-27 13:47:54 +00:00

Especially the error values provided by borgmatic can be passed to shell scripts using environment variables using the following approach. What I find useful about that is that one doesn't need to remember positions of arguments for individual values, doesn't need to forward everything at various places etc., but can instead focus on custom arguments for custom scripts.

    # List of one or more shell commands or scripts to execute
    # when an exception occurs during a "prune", "create", or
    # "check" action or an associated before/after hook.
    on_error:
        - |
          export BMC_ERR_VAR_CONFIG_PATH="{configuration_filename}"
          export BMC_ERR_VAR_REPO="{repository}"
          export BMC_ERR_VAR_ERROR_MSG="{error}"
          export BMC_ERR_VAR_ERROR_OUT="{output}"

          /some/script.sh some_arg

BMC aka BorgMatiC

Especially the error values provided by borgmatic can be passed to shell scripts using environment variables using the following approach. What I find useful about that is that one doesn't need to remember positions of arguments for individual values, doesn't need to forward everything at various places etc., but can instead focus on custom arguments for custom scripts. ```yaml # List of one or more shell commands or scripts to execute # when an exception occurs during a "prune", "create", or # "check" action or an associated before/after hook. on_error: - | export BMC_ERR_VAR_CONFIG_PATH="{configuration_filename}" export BMC_ERR_VAR_REPO="{repository}" export BMC_ERR_VAR_ERROR_MSG="{error}" export BMC_ERR_VAR_ERROR_OUT="{output}" /some/script.sh some_arg ``` BMC aka BorgMatiC

witten commented

2023-03-27 19:27:42 +00:00

I apologize for the lengthy delay here. Perhaps since this was originally filed, borgmatic has added support for passing values to command hooks via interpolated variables. So my recommendation would be to use that mechanism instead of environment variables and extend it with additional fields. But if that's insufficient, you might consider consuming data from a feature like borgmatic writing data to a JSON file for monitoring purposes.

So assuming that a feature like one of these is of still value to you, which data points would be useful to you?

I apologize for the lengthy delay here. Perhaps since this was originally filed, borgmatic has added support for passing values to command hooks via [interpolated variables](https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/#variable-interpolation). So my recommendation would be to use that mechanism instead of environment variables and extend it with additional fields. But if that's insufficient, you might consider consuming data from a feature like [borgmatic writing data to a JSON file](https://projects.torsion.org/borgmatic-collective/borgmatic/issues/617) for monitoring purposes. So assuming that a feature like one of these is of still value to you, which data points would be useful to you?

witten commented

2023-06-28 18:43:53 +00:00

I'm closing this one for now due to inactivity, but I'd be happy to open it up again if you have further thoughts. Thank you!

witten closed this issue

2023-06-28 18:43:54 +00:00

Sign in to join this conversation.