Separate before/after backup run and individual archive hooks #192

Closed
opened 2019-06-16 17:49:28 +00:00 by elho · 5 comments

The current before_backup/after_backup hooks are only sufficient in a single configuration setup.

When doing backups to multiple archives using /etc/borgmatic.d, there is demand for hooks to run once before all of these individual backups and once after (e.g. creating/removing LVM/filesystem snapshots of the entire system), plus some to run before and after each archive (e.g. dumping a database that goes to one archive only).

The current `before_backup`/`after_backup` hooks are only sufficient in a single configuration setup. When doing backups to multiple archives using `/etc/borgmatic.d`, there is demand for hooks to run once before all of these individual backups and once after (e.g. creating/removing LVM/filesystem snapshots of the entire system), plus some to run before and after each archive (e.g. dumping a database that goes to one archive only).
Owner

Thank you for filing this.. This is something I've wondered about with more complex configurations. I assume when you say "archive" here you mean a Borg "repository"? If so, can you say more about the case when you'd want to dump a database that goes to one repository only? And I'm assuming this is all within one config file that has multiple repositories listed?

Current hooks run per-configuration file, at the time that configuration file is executed. So for instance, if you have a directory full of configuration files in /etc/borgmatic.d, hooks in each configuration file will run before/after all the repositories in that config file do backups.

So if I'm understanding your ask, it sounds like you'd like some hooks to be defined and run per-repository rather than just per-configuration file. (E.g. for a configuration file that has multiple repositories.) Any thoughts on how do define them syntactically? One idea I have is to call out any repositories that the hook applies to when defining them. Example:

hooks:
    before_repository_backup:
        repositories:
          - user@backupserver:example.borg
          - user@otherserver:example.borg
        commands:
          - dump-a-database /to/file.sql
    ...

Would it be at all useful to be able to declare this for multiple repositories as per the example above? Or would it only ever be one repository at a time?

Or am I completely misunderstanding your ask?

Thank you for filing this.. This is something I've wondered about with more complex configurations. I assume when you say "archive" here you mean a Borg "repository"? If so, can you say more about the case when you'd want to dump a database that goes to one repository only? And I'm assuming this is all within one config file that has multiple repositories listed? Current hooks run per-configuration file, at the time that configuration file is executed. So for instance, if you have a directory full of configuration files in `/etc/borgmatic.d`, hooks in each configuration file will run before/after *all* the repositories in that config file do backups. So if I'm understanding your ask, it sounds like you'd like some hooks to be defined and run *per-repository* rather than just per-configuration file. (E.g. for a configuration file that has multiple repositories.) Any thoughts on how do define them syntactically? One idea I have is to call out any repositories that the hook applies to when defining them. Example: ``` hooks: before_repository_backup: repositories: - user@backupserver:example.borg - user@otherserver:example.borg commands: - dump-a-database /to/file.sql ... ``` Would it be at all useful to be able to declare this for multiple repositories as per the example above? Or would it only ever be one repository at a time? Or am I completely misunderstanding your ask?
Author

No, by "archive" I do mean borg archive, its the exact opposite situation. 😉
I (currently, due to still testing and refining the setup) only have one repository, but different source directories going into different archives there to allow different retention policies for them, each setup in a separate file in /etc/borgmatic.d, YAML including defaults and repository config from /etc/borgmatic. (In other words, its like the "per-application backups" scenario in the documentation.)

And thus the current hooks would be run whenever one of the config files (for one of the archives) is processed, but I do need tasks to be done before the first and after the last one only.
The workaround is quite easy, I do have a wrapper script around borgmatic that invokes the scripts to run before and after borgmatic that works for me, but having this integrated would be nicer, of course.

No, by "archive" I do mean borg archive, its the exact opposite situation. :wink: I (currently, due to still testing and refining the setup) only have one repository, but different source directories going into different archives there to allow different retention policies for them, each setup in a separate file in `/etc/borgmatic.d`, YAML including defaults and repository config from `/etc/borgmatic`. (In other words, its like the "per-application backups" scenario in the documentation.) And thus the current hooks would be run whenever one of the config files (for one of the archives) is processed, but I do need tasks to be done before the first and after the last one only. The workaround is quite easy, I do have a wrapper script around borgmatic that invokes the scripts to run before and after borgmatic that works for me, but having this integrated would be nicer, of course.
Owner

Gotcha! Thanks for clarifying. Okay, so I could see something like this potentially working:

hooks:
    before_all_backups:
        - set-stuff-up-globally
    after_all_backups:
        - clean-stuff-up-globally
    ...

And then you could define those in one or more of your existing borgmatic configuration files. They'd get all gathered up from the various configuration files and run together before/after all backups.

Does that sound like it'd work for you?

Gotcha! Thanks for clarifying. Okay, so I could see something like this potentially working: ```yaml hooks: before_all_backups: - set-stuff-up-globally after_all_backups: - clean-stuff-up-globally ... ``` And then you could define those in one or more of your existing borgmatic configuration files. They'd get all gathered up from the various configuration files and run together before/after all backups. Does that sound like it'd work for you?
Author

Yeah, sounds good.

Yeah, sounds good.
witten added the
design finalized
label 2019-06-17 20:45:53 +00:00
Owner

Implemented and released as part of borgmatic 1.3.21. As soon as the docs deploy, you can read about the new hooks here: https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/

They're called before_everything and after_everything. Let me know how they work out for you!

Implemented and released as part of borgmatic 1.3.21. As soon as the docs deploy, you can read about the new hooks here: https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/ They're called `before_everything` and `after_everything`. Let me know how they work out for you!
Sign in to join this conversation.
No Milestone
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#192
No description provided.