Provide separate hook to check repository availability #456

New Issue

andreasferber · 2021-09-29T00:08:56Z

andreasferber commented

2021-09-29 00:08:56 +00:00

For certain use cases it would be good to have a separate hook to check repository availability. This hook should be called once for each of the configured repositories (with the {repository} placeholder available, or alternatively provide the repository that should be checked as an environment variable). If the hook returns a soft failure (exit code 75) the repository should be skipped for this particular borgmatic run (with appropriate log/stdout messages to inform the user about it). When none of the repositories are available then the job should report a failure.

My particular use case is that I use multiple removable USB disks for my backups, with each disk containing separate borg repositories. The disks are rotated regularly and the ones not in use stored off site. At the moment what I have to do is have a separate configuration file for each disk, with repository availability checks in the before_{backup,check,prune} hooks, because those hooks are per config file and not per repository.

For one this is quite a hassle for administration because the files have to be kept in sync when doing configuration changes (this can be somewhat mitigated by using includes, but I do other stuff in the before_* hooks as well and that can't be included because of the way includes are merged). Also I'm backing up multiple machines to those disks using remote repositories, so the config duplication has to be done on every single one of those machines as well.

And second it doesn't play nice with doing other stuff in the before_* hooks (like creating LVM snapshots in my case for example) in case more than one of the disks happens to be available at the same time for some reason. With multiple repositories in a single configuration the sequence would be "before_backup"->"write archive to repo1"->"write archive to repo2"->"after_backup", but with the separate files it's now "before_backup"->"write archive to repo1"->"after_backup"->"before_backup"->"write archive to repo2"->"after_backup".

And one final thing is that this new hook could also be used by the list command to provide nicer output (warning) messages for the unavailable repositories.

For certain use cases it would be good to have a separate hook to check repository availability. This hook should be called once for each of the configured repositories (with the {repository} placeholder available, or alternatively provide the repository that should be checked as an environment variable). If the hook returns a soft failure (exit code 75) the repository should be skipped for this particular borgmatic run (with appropriate log/stdout messages to inform the user about it). When none of the repositories are available then the job should report a failure. My particular use case is that I use multiple removable USB disks for my backups, with each disk containing separate borg repositories. The disks are rotated regularly and the ones not in use stored off site. At the moment what I have to do is have a separate configuration file for each disk, with repository availability checks in the before_{backup,check,prune} hooks, because those hooks are per config file and not per repository. For one this is quite a hassle for administration because the files have to be kept in sync when doing configuration changes (this can be somewhat mitigated by using includes, but I do other stuff in the before_* hooks as well and that can't be included because of the way includes are merged). Also I'm backing up multiple machines to those disks using remote repositories, so the config duplication has to be done on every single one of those machines as well. And second it doesn't play nice with doing other stuff in the before_* hooks (like creating LVM snapshots in my case for example) in case more than one of the disks happens to be available at the same time for some reason. With multiple repositories in a single configuration the sequence would be "before_backup"->"write archive to repo1"->"write archive to repo2"->"after_backup", but with the separate files it's now "before_backup"->"write archive to repo1"->"after_backup"->"before_backup"->"write archive to repo2"->"after_backup". And one final thing is that this new hook could also be used by the list command to provide nicer output (warning) messages for the unavailable repositories.

andreasferber closed this issue

2021-09-29 02:08:21 +00:00

andreasferber reopened this issue

2021-09-29 02:08:29 +00:00

noresistence commented

2021-11-28 10:49:48 +00:00

I want to support this Issue. My use case is similar:

There is a pre-backup script that should be executed only once, not once for each repository
The repositories may or may not be available (in our case, they are stored in decentral locations and accessed via SSH. It does happen surprisingly often that during the backup, one internet provider interrupts the connection to one of the repositories, or that the connection to one of them cannot be established)
If one repository is not available, or fails during the backup, this should not cancel the remaining backup on the remaining repositories; instead, it should produce a warning
If all repositories fail, the backup fails

It would be great to be able to specify the on_failure behaviour for each repository:

ignore failures
trigger warnings (but continue with other repositories and fail if no repository succeeded)
trigger error and disrupt backup (default behaviour)

I want to support this Issue. My use case is similar: - There is a pre-backup script that should be executed only once, not once for each repository - The repositories may or may not be available (in our case, they are stored in decentral locations and accessed via SSH. It does happen surprisingly often that during the backup, one internet provider interrupts the connection to one of the repositories, or that the connection to one of them cannot be established) - If one repository is not available, or fails during the backup, this should not cancel the remaining backup on the remaining repositories; instead, it should produce a warning - If all repositories fail, the backup fails It would be great to be able to specify the *on_failure* behaviour for each repository: - ignore failures - trigger warnings (but continue with other repositories and fail if no repository succeeded) - trigger error and disrupt backup (default behaviour)

noresistence commented

2021-11-28 11:12:25 +00:00

maybe even allow a certain amount of retries and a wait period in case of connection failures

witten commented

2022-04-28 20:58:33 +00:00

Sorry for the lengthy delay here. The good news: I believe this is implemented now—as long as you don't mind doing these hooks per-action. As of borgmatic 1.6.0, all before_* and after_* hooks are per-repository instead of per configuration file. And you can pass in the {repository} variable to your hook commands if you like.

This means you can declare a before_backup hook and it'll run for a particular repository. So you can check for repository available there if you like.

Please let me know if this works for your use cases!

Also note that there is some discussion on a separate ticket about potential new hooks that would run for all borgmatic actions rather than just one.

this can be somewhat mitigated by using includes, but I do other stuff in the before_* hooks as well and that can't be included because of the way includes are merged

This is also fixed! In borgmatic 1.6.0, merged includes are deep merged, meaning you can include common configuration options across sections with a single include.

@noresistence, while your ask is certainly related to this issue, and it sounds like a nice thing to have, I consider it a separate enhancement. So would you mind filing a separate ticket for it?

Sorry for the lengthy delay here. The good news: I believe this is implemented now—as long as you don't mind doing these hooks per-action. As of borgmatic 1.6.0, all `before_*` and `after_*` hooks are per-repository instead of per configuration file. And you can pass in the `{repository}` variable to your hook commands if you like. This means you can declare a `before_backup` hook and it'll run for a particular repository. So you can check for repository available there if you like. Please let me know if this works for your use cases! Also note that there is some [discussion on a separate ticket](https://projects.torsion.org/borgmatic-collective/borgmatic/issues/463#issuecomment-4338) about potential new hooks that would run for *all* borgmatic actions rather than just one. > this can be somewhat mitigated by using includes, but I do other stuff in the before_* hooks as well and that can't be included because of the way includes are merged This is also fixed! In borgmatic 1.6.0, merged includes are *deep merged*, meaning you can include common configuration options across sections with a single include. @noresistence, while your ask is certainly related to this issue, and it sounds like a nice thing to have, I consider it a separate enhancement. So would you mind filing a separate ticket for it?

witten added the

waiting for response

label 2022-04-28 20:58:46 +00:00

noresistence commented

2022-05-06 13:47:13 +00:00

The changes in 1.6.0 did simplify the usage of common config, nice work there :)

Please let me know if this works for your use cases!

Using before_backup to check for availability of a repository would work indeed, if the problem was only the availability. But in other cases, like backups that are interrupted mid-way due to unstable connections, an availability check would not help to solve the issue that in a redundant system (multiple repositories) a failing component (repository) should not break the whole system (stop any attempt to backup to the other repositories).

while your ask is certainly related to this issue, and it sounds like a nice thing to have, I consider it a separate enhancement. So would you mind filing a separate ticket for it?

You're right, I'll go ahead and file a ticket for this.

The changes in 1.6.0 did simplify the usage of common config, nice work there :) > Please let me know if this works for your use cases! Using `before_backup` to check for availability of a repository would work indeed, if the problem was only the availability. But in other cases, like backups that are interrupted mid-way due to unstable connections, an availability check would not help to solve the issue that in a redundant system (multiple repositories) a failing component (repository) should not break the whole system (stop any attempt to backup to the other repositories). > while your ask is certainly related to this issue, and it sounds like a nice thing to have, I consider it a separate enhancement. So would you mind filing a separate ticket for it? You're right, I'll go ahead and file a ticket for this.

noresistence commented

2022-05-06 14:46:08 +00:00

nevermind, with the repository queue and retries that have been introduced (180018fd81) borgmatic now actually behaves much more in the resilient way that i had wished for; this change of behavior was not really reflected in the changelog (or i did overlook it). But it's a nice surprise! I believe that with this, additional per-repository on_failure specifications unneccassary are not necessary anymore.

So i won't create a new ticket, but congratulate and thank for the great work :)

nevermind, with the repository queue and retries that have been introduced (https://github.com/borgmatic-collective/borgmatic/commit/180018fd81b4cb15bd0324bcf457d09c483d5751) borgmatic now actually behaves much more in the resilient way that i had wished for; this change of behavior was not really reflected in the changelog (or i did overlook it). But it's a nice surprise! I believe that with this, additional per-repository `on_failure` specifications unneccassary are not necessary anymore. So i won't create a new ticket, but congratulate and thank for the great work :)

witten commented

2022-05-20 18:57:21 +00:00

Awesome, glad to hear that use case is covered!

witten closed this issue

2022-05-20 18:57:22 +00:00

witten removed the

waiting for response

label 2022-05-20 18:57:42 +00:00

Sign in to join this conversation.