#192 Separate before/after backup run and individual archive hooks

Open
opened 3 months ago by elho · 4 comments
elho commented 3 months ago

The current before_backup/after_backup hooks are only sufficient in a single configuration setup.

When doing backups to multiple archives using /etc/borgmatic.d, there is demand for hooks to run once before all of these individual backups and once after (e.g. creating/removing LVM/filesystem snapshots of the entire system), plus some to run before and after each archive (e.g. dumping a database that goes to one archive only).

The current `before_backup`/`after_backup` hooks are only sufficient in a single configuration setup. When doing backups to multiple archives using `/etc/borgmatic.d`, there is demand for hooks to run once before all of these individual backups and once after (e.g. creating/removing LVM/filesystem snapshots of the entire system), plus some to run before and after each archive (e.g. dumping a database that goes to one archive only).
witten commented 3 months ago
Owner

Thank you for filing this.. This is something I’ve wondered about with more complex configurations. I assume when you say “archive” here you mean a Borg “repository”? If so, can you say more about the case when you’d want to dump a database that goes to one repository only? And I’m assuming this is all within one config file that has multiple repositories listed?

Current hooks run per-configuration file, at the time that configuration file is executed. So for instance, if you have a directory full of configuration files in /etc/borgmatic.d, hooks in each configuration file will run before/after all the repositories in that config file do backups.

So if I’m understanding your ask, it sounds like you’d like some hooks to be defined and run per-repository rather than just per-configuration file. (E.g. for a configuration file that has multiple repositories.) Any thoughts on how do define them syntactically? One idea I have is to call out any repositories that the hook applies to when defining them. Example:

hooks:
    before_repository_backup:
        repositories:
          - user@backupserver:example.borg
          - user@otherserver:example.borg
        commands:
          - dump-a-database /to/file.sql
    ...

Would it be at all useful to be able to declare this for multiple repositories as per the example above? Or would it only ever be one repository at a time?

Or am I completely misunderstanding your ask?

Thank you for filing this.. This is something I've wondered about with more complex configurations. I assume when you say "archive" here you mean a Borg "repository"? If so, can you say more about the case when you'd want to dump a database that goes to one repository only? And I'm assuming this is all within one config file that has multiple repositories listed? Current hooks run per-configuration file, at the time that configuration file is executed. So for instance, if you have a directory full of configuration files in `/etc/borgmatic.d`, hooks in each configuration file will run before/after *all* the repositories in that config file do backups. So if I'm understanding your ask, it sounds like you'd like some hooks to be defined and run *per-repository* rather than just per-configuration file. (E.g. for a configuration file that has multiple repositories.) Any thoughts on how do define them syntactically? One idea I have is to call out any repositories that the hook applies to when defining them. Example: ``` hooks: before_repository_backup: repositories: - user@backupserver:example.borg - user@otherserver:example.borg commands: - dump-a-database /to/file.sql ... ``` Would it be at all useful to be able to declare this for multiple repositories as per the example above? Or would it only ever be one repository at a time? Or am I completely misunderstanding your ask?
elho commented 3 months ago
Poster

No, by “archive” I do mean borg archive, its the exact opposite situation. :wink: I (currently, due to still testing and refining the setup) only have one repository, but different source directories going into different archives there to allow different retention policies for them, each setup in a separate file in /etc/borgmatic.d, YAML including defaults and repository config from /etc/borgmatic. (In other words, its like the “per-application backups” scenario in the documentation.)

And thus the current hooks would be run whenever one of the config files (for one of the archives) is processed, but I do need tasks to be done before the first and after the last one only. The workaround is quite easy, I do have a wrapper script around borgmatic that invokes the scripts to run before and after borgmatic that works for me, but having this integrated would be nicer, of course.

No, by "archive" I do mean borg archive, its the exact opposite situation. :wink: I (currently, due to still testing and refining the setup) only have one repository, but different source directories going into different archives there to allow different retention policies for them, each setup in a separate file in `/etc/borgmatic.d`, YAML including defaults and repository config from `/etc/borgmatic`. (In other words, its like the "per-application backups" scenario in the documentation.) And thus the current hooks would be run whenever one of the config files (for one of the archives) is processed, but I do need tasks to be done before the first and after the last one only. The workaround is quite easy, I do have a wrapper script around borgmatic that invokes the scripts to run before and after borgmatic that works for me, but having this integrated would be nicer, of course.
witten commented 3 months ago
Owner

Gotcha! Thanks for clarifying. Okay, so I could see something like this potentially working:

hooks:
    before_all_backups:
        - set-stuff-up-globally
    after_all_backups:
        - clean-stuff-up-globally
    ...

And then you could define those in one or more of your existing borgmatic configuration files. They’d get all gathered up from the various configuration files and run together before/after all backups.

Does that sound like it’d work for you?

Gotcha! Thanks for clarifying. Okay, so I could see something like this potentially working: ```yaml hooks: before_all_backups: - set-stuff-up-globally after_all_backups: - clean-stuff-up-globally ... ``` And then you could define those in one or more of your existing borgmatic configuration files. They'd get all gathered up from the various configuration files and run together before/after all backups. Does that sound like it'd work for you?
elho commented 3 months ago
Poster

Yeah, sounds good.

Yeah, sounds good.
witten added the
design finalized
label 3 months ago
Sign in to join this conversation.
No Milestone
No Assignees
2 Participants
Due Date

No due date set.

Dependencies

This issue currently doesn't have any dependencies.

Loading…
Cancel
Save
There is no content yet.