borgmatic-collective/borgmatic
borgmatic-collective/borgmatic
Watch
9
74
Fork
You've already forked borgmatic
 57
Code Issues 71 Pull requests 3 Releases 162 Packages 1 Activity Actions 1

Run before_backup once (with multiple repositories) #790

New issue
Closed
opened 2023-11-14 23:22:07 +00:00 by shadow7412 ยท 23 comments
shadow7412 commented 2023-11-14 23:22:07 +00:00
Copy link

What I'd like to do and why

I have a local and remote borg repo.

For a couple of applications I am backing up, there is a command that I need to run. This exports all of the databases/artefacts/etc to a specific directory. This is not a small operation (~10-25 minutes depending on the application). Getting around this is not possible (or at least not desirable, as it would complicate the restore process).

Using before_backup does work, but I only need this to run once rather than multiplying the time my backup takes to prepare by the amount of repos I have defined.

Current behaviour:

  • before_backups
  • backup to repo1
  • after_backups
  • before_backups
  • backup to repo2
  • after_backups

Desired behaviour:

  • before_backups
  • backup to repo1
  • backup to repo2
  • after_backups

Other notes / implementation ideas

Perhaps we could consider introducing a before_backup_once, to avoid breaking backwards compatibility for those who rely on this behaviour.

This would run once, just before running the backup command associated with the first repository in the list. If borg handles the before_backup config, perhaps the contents of before_backup_all could be prepended to before_backup for the first repo or something.

(Perhaps a matching after_backup_once is appropriate, not that I need it)

It should be noted that I would still expect multiple configurations to have each of their relevant before_backup_once handled individually.

### What I'd like to do and why I have a local and remote borg repo. For a couple of applications I am backing up, there is a command that I need to run. This exports all of the databases/artefacts/etc to a specific directory. This is not a small operation (~10-25 minutes depending on the application). Getting around this is not possible (or at least not desirable, as it would complicate the restore process). Using before_backup does work, but I only need this to run once rather than multiplying the time my backup takes to prepare by the amount of repos I have defined. #### Current behaviour: * before_backups * backup to repo1 * after_backups * before_backups * backup to repo2 * after_backups #### Desired behaviour: * before_backups * backup to repo1 * backup to repo2 * after_backups ### Other notes / implementation ideas Perhaps we could consider introducing a `before_backup_once`, to avoid breaking backwards compatibility for those who rely on this behaviour. This would run once, just before running the backup command associated with the first repository in the list. If borg handles the `before_backup` config, perhaps the contents of `before_backup_all` could be prepended to `before_backup` for the first repo or something. (Perhaps a matching `after_backup_once` is appropriate, not that I need it) It should be noted that I would still expect multiple configurations to have each of their relevant `before_backup_once` handled individually.
๐Ÿ‘ 1
shadow7412 commented 2023-11-14 23:26:14 +00:00
Author
Copy link

Actually, re-reading the documentation... is this what before_everything is supposed to do? I only just noticed the caveat of it only running when create is among the operations.

Actually, re-reading the documentation... is this what `before_everything` is supposed to do? I only just noticed the caveat of it only running when `create` is among the operations.
shadow7412 commented 2023-11-14 23:26:58 +00:00
Author
Copy link

These are collected from all configuration files and then run once after all of them (after any action).

Does this mean that if any of the pre-commands fail, the entire backup doesn't run?

> These are collected from all configuration files and then run once after all of them (after any action). Does this mean that if any of the pre-commands fail, the entire backup doesn't run?
witten commented 2023-11-14 23:40:27 +00:00
Owner
Copy link

Yup, before_everything is the closest to what you're looking for. And it does run only when there's an implicit or explicit create action.

These are collected from all configuration files and then run once after all of them (after any action).

Does this mean that if any of the pre-commands fail, the entire backup doesn't run?

That's correct!

Yup, `before_everything` is the closest to what you're looking for. And it does run only when there's an implicit or explicit `create` action. > > These are collected from all configuration files and then run once after all of them (after any action). > > Does this mean that if any of the pre-commands fail, the entire backup doesn't run? That's correct!
๐Ÿ‘ 1
shadow7412 commented 2023-11-14 23:42:06 +00:00
Author
Copy link

Hmm. It probably would be nicer to have that single config file fail than all of them...

Hmm. It probably would be nicer to have that single config file fail than all of them...
witten commented 2023-11-14 23:59:13 +00:00
Owner
Copy link

before_backup actually used to work exactly as you've described above.. Prior to borgmatic 1.6.0 and #473, command hooks ran per-configuration rather than per-repository. The change to be per-repository was made though to better support running timing-sensitive tasks like pausing containers, thereby keeping each hook temporally closer to the action(s) it's wrapping.

So short of downgrading borgmatic (which I don't recommend), your best bet for now might be to:

  1. Use before_everything and just deal with any failures that affect all configuration files.
  2. Or use before_everything but invoke borgmatic separately for each configuration file that you want to succeed/fail independently. (See the --config flag.)
  3. Or use before_backup and deal with the repetition. (In fact, the built-in borgmatic database dumping feature also repeats dumping per-repository because it streams dumps directly to Borg without hitting disk.)

Hope this helps some.

`before_backup` actually used to work exactly as you've described above.. Prior to borgmatic 1.6.0 and #473, command hooks ran per-configuration rather than per-repository. The change to be per-repository was made though to better support running timing-sensitive tasks like pausing containers, thereby keeping each hook temporally _closer_ to the action(s) it's wrapping. So short of downgrading borgmatic (which I don't recommend), your best bet for now might be to: 1. Use `before_everything` and just deal with any failures that affect all configuration files. 2. Or use `before_everything` but invoke borgmatic separately for each configuration file that you want to succeed/fail independently. (See the `--config` flag.) 3. Or use `before_backup` and deal with the repetition. (In fact, the built-in borgmatic database dumping feature also repeats dumping per-repository because it streams dumps directly to Borg without hitting disk.) Hope this helps some.
๐Ÿ‘€ 1
shadow7412 commented 2023-11-15 00:39:53 +00:00
Author
Copy link

Taking a service down temporarily is a pretty reasonable usecase - so I understand the change.

But do you see value in implementing the old behaviour but as a different option?

For now, I think I'll have to take the before_everything route... but it would be nice to not have all backups fail because of a specific config's prerun.

Actually if we're talking optimisation, in my situation there's nothing really preventing the idea of running the differently config before steps in parallel. I daresay that'd be a complexity jump that doesn't attract huge interest though...

Taking a service down temporarily is a pretty reasonable usecase - so I understand the change. But do you see value in implementing the old behaviour but as a different option? For now, I think I'll have to take the before_everything route... but it would be nice to not have all backups fail because of a specific config's prerun. Actually if we're talking optimisation, in my situation there's nothing really preventing the idea of running the differently config `before` steps in parallel. I daresay that'd be a complexity jump that doesn't attract huge interest though...
witten commented 2023-12-21 16:50:10 +00:00
Owner
Copy link

I see the value in the old behavior, but I'm hesitant to build directly atop the existing command hook syntax to support it. That's because that approach would end up encoding so much detail into the option name itself, which gets kind of awkward IMO especially as more use cases are added. A more flexible (if less backwards-compatible) approach would be to come up with a new schema for command hooks that better supports these kinds of use cases. Here's a made-up example:

commands:
    # Before the prune action for each repository, run the command. Same as before_prune.
    - before: action
      when:
        - prune
      run:
        - echo test

    # Before all the actions for each repository, run the command. Same as before_actions.
    - before: repository
      run:
        - echo test

    # Before all the actions for all the repositories in the current configuration file, run the command...
    # but only when there is a create or check action. No analog today. Should satisfy your use case.
    - before: configuration
      when:
        - create
        - check
      run:
        - echo test

    # Before all configuration files, run the command... but only when there is a create or prune action.
    # Similar to before_everything but with the ability to chose the actions.
    - before: everything
      when:
        - create
        - prune
      run:
        - echo test

The big downside is verbosity and complexity. The upside is this is way more flexible for a variety of use cases including yours. I'm not 100% sold that the trade-off is worth it, but I thought it's worth continuing the discussion. (This is not the first time this kind of approach has been discussed.)

Actually if we're talking optimisation, in my situation there's nothing really preventing the idea of running the differently config before steps in parallel. I daresay that'd be a complexity jump that doesn't attract huge interest though...

Yeah, parallelism has come up in other contexts as well, but thus far nobody has worked on it.

I see the value in the old behavior, but I'm hesitant to build directly atop the existing command hook syntax to support it. That's because that approach would end up encoding so much detail into the option name itself, which gets kind of awkward IMO especially as more use cases are added. A more flexible (if less backwards-compatible) approach would be to come up with a new schema for command hooks that better supports these kinds of use cases. Here's a made-up example: ```yaml commands: # Before the prune action for each repository, run the command. Same as before_prune. - before: action when: - prune run: - echo test # Before all the actions for each repository, run the command. Same as before_actions. - before: repository run: - echo test # Before all the actions for all the repositories in the current configuration file, run the command... # but only when there is a create or check action. No analog today. Should satisfy your use case. - before: configuration when: - create - check run: - echo test # Before all configuration files, run the command... but only when there is a create or prune action. # Similar to before_everything but with the ability to chose the actions. - before: everything when: - create - prune run: - echo test ``` The big downside is verbosity and complexity. The upside is this is way more flexible for a variety of use cases including yours. I'm not 100% sold that the trade-off is worth it, but I thought it's worth continuing the discussion. (This is not the first time this kind of approach has been discussed.) > Actually if we're talking optimisation, in my situation there's nothing really preventing the idea of running the differently config before steps in parallel. I daresay that'd be a complexity jump that doesn't attract huge interest though... Yeah, parallelism has come up in other contexts as well, but thus far nobody has worked on it.
๐Ÿ‘ 2
akostadinov commented 2025-01-12 17:39:44 +00:00
Copy link

My issue is not the same but I think it is very related, and because here you speak about refactoring hooks, I dare to add my use case to be considered. What I'm trying to achieve is

  1. stopping services
  2. xfs_freeze -f
  3. taking LVM snapshots
  4. xfs_freeze -u
  5. starting services
  6. actual backup
  7. cleaning-up LVM snapshots

And most sense it makes to have the 1,2,3,5 as hooks valid for only the current configuration file.

Currently the only way I see to implement is by custom before_backup hook. Because I see no way to add the services start/stop logic to the LVM snapshot logic. This means writing my own script to make the snapshots/mounts and control the services instead of using the built-in LVM feature.

wrt running borgmatic separately for each configuration file, this makes using a systemd service not really viable. Would be nice if this mode of operation is well supported.

My issue is not the same but I think it is very related, and because here you speak about refactoring hooks, I dare to add my use case to be considered. What I'm trying to achieve is 1. stopping services 2. xfs_freeze -f 3. taking LVM snapshots 4. xfs_freeze -u 5. starting services 6. actual backup 7. cleaning-up LVM snapshots And most sense it makes to have the 1,2,3,5 as hooks valid for only the current configuration file. Currently the only way I see to implement is by custom `before_backup` hook. Because I see no way to add the services start/stop logic to the LVM snapshot logic. This means writing my own script to make the snapshots/mounts and control the services instead of using the built-in LVM feature. wrt running borgmatic separately for each configuration file, this makes using a systemd service not really viable. Would be nice if this mode of operation is well supported.
witten commented 2025-01-12 19:35:15 +00:00
Owner
Copy link

Interesting. What's your use case for stopping and starting services around the LVM snapshot? Just so you get a consistent picture of the filesystem without the services modifying it? And the snapshot itself isn't sufficient to accomplish that?

Anyway, if there were hooks where you could stop and start services for this use case, and it was using the proposed hook scheme above, it might look something like this:

    - before: dump_data_sources
      run:
        - systemctl stop services or whatever
    - after: dump_data_sources
      run:
        - systemctl start services or whatever

This is of note because it's effectively a hook that triggers before and after another hook is triggered! Not sure how I feel about that, just conceptually. But one downside I can see to this approach is that it would trigger before/after all data source hooksโ€”including database dumpsโ€”which might not be what you're hoping for in this use case.

EDIT: Here's a variant of above that may solve this triggering after all data source hooks:

    - before: dump_data_sources
      hook: lvm
      run:
        - systemctl stop services or whatever
    - after: dump_data_sources
      hook: lvm
      run:
        - systemctl start services or whatever

The idea is that this would be restricted to being triggered before/after only the LVM hook dumping data sourcesโ€”and not after any other hooks do.

Interesting. What's your use case for stopping and starting services around the LVM snapshot? Just so you get a consistent picture of the filesystem without the services modifying it? And the snapshot itself isn't sufficient to accomplish that? Anyway, if there *were* hooks where you could stop and start services for this use case, and it was using the proposed hook scheme above, it might look something like this: ```yaml - before: dump_data_sources run: - systemctl stop services or whatever - after: dump_data_sources run: - systemctl start services or whatever ``` This is of note because it's effectively a hook that triggers before and after another hook is triggered! Not sure how I feel about that, just conceptually. But one downside I can see to this approach is that it would trigger before/after *all* data source hooksโ€”including database dumpsโ€”which might not be what you're hoping for in this use case. EDIT: Here's a variant of above that may solve this triggering after all data source hooks: ```yaml - before: dump_data_sources hook: lvm run: - systemctl stop services or whatever - after: dump_data_sources hook: lvm run: - systemctl start services or whatever ``` The idea is that this would be restricted to being triggered before/after *only* the LVM hook dumping data sourcesโ€”and not after any other hooks do.
akostadinov commented 2025-01-13 11:07:33 +00:00
Copy link

Just so you get a consistent picture of the filesystem without the services modifying it?

correct

And the snapshot itself isn't sufficient to accomplish that?

There are possible inconsistencies that I prefer not to deal with. I'm running atm immich and seafile and some other services. They are reasonably resilient to abrupt interruption with the possibility to leave stale objects or objects that are in the database but not on disk. Depending on whether file was added/removed but the database operation was not fully committed.

Since I don't want to know these services on the very low-level I prefer to err more on the safe side and not have to deal low-level inconsistencies.

Also I use large HDDs. But it turned out database performance was (no surprise) horrible. So I moved the DBs to smaller SSDs.

Now they are two different filesystems that cannot be frozen at exactly the same moment. Also not sure how mysql and postgres like abrupt interruptions. I assume they are also reasonably resilient but don't want to make this a normal course of operation.

I know borgmatic has a database dump function. It seems easier for me to just save the files though. And don't think too much how to restore, resource usage, etc. Minimizes my downtime. Also I don't have hooks around database dumps, which could allow stopping only the services, make snapshot, dump database, restart services. Would be some more downtime.

Probably I'm overly-concerned. But this is the simplest way for me to start and see later where can I further optimize.

wrt hooks, I really don't know. For me it is reasonable to have before/after_snapshot hooks. Like before and after all snapshotting, not around individual snapshots. Snapshotting is not a hook from user point of view. Just part of borgmatic steps. Not sure how it is implemented. And I don't see a big reason not to have many hooks around each execution state.

Thank you for the consideration! At the moment my implementation is reasonable. I have only one repository in my services configuration file. Will add another configuration file with another repository for OS backup but I understood they will work independently. So no rush on my end. Just wanted to provide feedback about possible new hooks.

> Just so you get a consistent picture of the filesystem without the services modifying it? correct > And the snapshot itself isn't sufficient to accomplish that? There are possible inconsistencies that I prefer not to deal with. I'm running atm immich and seafile and some other services. They are reasonably resilient to abrupt interruption with the possibility to leave stale objects or objects that are in the database but not on disk. Depending on whether file was added/removed but the database operation was not fully committed. Since I don't want to know these services on the very low-level I prefer to err more on the safe side and not have to deal low-level inconsistencies. Also I use large HDDs. But it turned out database performance was (no surprise) horrible. So I moved the DBs to smaller SSDs. Now they are two different filesystems that cannot be frozen at exactly the same moment. Also not sure how mysql and postgres like abrupt interruptions. I assume they are also reasonably resilient but don't want to make this a normal course of operation. I know borgmatic has a database dump function. It seems easier for me to just save the files though. And don't think too much how to restore, resource usage, etc. Minimizes my downtime. Also I don't have hooks around database dumps, which could allow stopping only the services, make snapshot, dump database, restart services. Would be some more downtime. Probably I'm overly-concerned. But this is the simplest way for me to start and see later where can I further optimize. wrt hooks, I really don't know. For me it is reasonable to have `before/after_snapshot` hooks. Like before and after all snapshotting, not around individual snapshots. Snapshotting is not a hook from user point of view. Just part of borgmatic steps. Not sure how it is implemented. And I don't see a big reason not to have many hooks around each execution state. Thank you for the consideration! At the moment my implementation is reasonable. I have only one repository in my services configuration file. Will add another configuration file with another repository for OS backup but I understood they will work independently. So no rush on my end. Just wanted to provide feedback about possible new hooks.
akostadinov commented 2025-01-13 19:55:29 +00:00
Copy link

As advised, I'm pasting here my precise hooks that cover my use case pretty well. So that it can be considered for future hooks enablement in borgmatic.

before_backup:
    - echo Starting apps backup.
    - "systemctl --user -M myuser@.host stop immich-server.service immich-machine-learning.service seafile.service mariadb.service immich-pgsql.service"
    - xfs_freeze -f /media/dbstore/
    - lvcreate --snapshot --size 200GB --name containers_data_borgmatic /dev/mapper/fedora_myhost-containers_data ; exitcode=$? ; xfs_freeze -u /media/dbstore/ ; exit $exitcode
    - xfs_freeze -f /media/datastore/
    - lvcreate --snapshot --size 47GB --name luks_borgmatic /dev/DataVG/luks_lv ; exitcode=$? ; xfs_freeze -u /media/datastore/ ; exit $exitcode
    - "systemctl --user -M myuser@.host start immich-server.service immich-machine-learning.service seafile.service mariadb.service immich-pgsql.service"
    - cryptsetup open --key-file /etc/luks-key /dev/DataVG/luks_borgmatic datastore_borgmatic
    - mount -o nouuid /dev/mapper/fedora_myhost-containers_data_borgmatic /media/borg/dbstore
    - mount -o nouuid /dev/mapper/datastore_borgmatic /media/borg/datastore

after_backup:
    - echo Finished apps backup and clean-up.
    - umount /media/borg/dbstore
    - umount /media/borg/datastore
    - cryptsetup close /dev/mapper/datastore_borgmatic
    - lvremove -y /dev/mapper/fedora_myhost-containers_data_borgmatic
    - lvremove -y /dev/mapper/DataVG-luks_borgmatic
As [advised](https://github.com/borgmatic-collective/borgmatic/pull/87#issuecomment-2587721645), I'm pasting here my precise hooks that cover my use case pretty well. So that it can be considered for future hooks enablement in borgmatic. ```yml before_backup: - echo Starting apps backup. - "systemctl --user -M myuser@.host stop immich-server.service immich-machine-learning.service seafile.service mariadb.service immich-pgsql.service" - xfs_freeze -f /media/dbstore/ - lvcreate --snapshot --size 200GB --name containers_data_borgmatic /dev/mapper/fedora_myhost-containers_data ; exitcode=$? ; xfs_freeze -u /media/dbstore/ ; exit $exitcode - xfs_freeze -f /media/datastore/ - lvcreate --snapshot --size 47GB --name luks_borgmatic /dev/DataVG/luks_lv ; exitcode=$? ; xfs_freeze -u /media/datastore/ ; exit $exitcode - "systemctl --user -M myuser@.host start immich-server.service immich-machine-learning.service seafile.service mariadb.service immich-pgsql.service" - cryptsetup open --key-file /etc/luks-key /dev/DataVG/luks_borgmatic datastore_borgmatic - mount -o nouuid /dev/mapper/fedora_myhost-containers_data_borgmatic /media/borg/dbstore - mount -o nouuid /dev/mapper/datastore_borgmatic /media/borg/datastore ``` ```yml after_backup: - echo Finished apps backup and clean-up. - umount /media/borg/dbstore - umount /media/borg/datastore - cryptsetup close /dev/mapper/datastore_borgmatic - lvremove -y /dev/mapper/fedora_myhost-containers_data_borgmatic - lvremove -y /dev/mapper/DataVG-luks_borgmatic ```
๐Ÿ‘ 1
witten commented 2025-01-14 22:11:04 +00:00
Owner
Copy link

Thanks for adding your thoughts about starting/stopping services around filesystem snapshotting. It's really helpful to get that context from your point of view to better understand the ask! I have a feeling the current hooks approach might be due for an overhaul at some point, and this is certainly another important data point towards that.

Thanks for adding your thoughts about starting/stopping services around filesystem snapshotting. It's really helpful to get that context from your point of view to better understand the ask! I have a feeling the current hooks approach might be due for an overhaul at some point, and this is certainly another important data point towards that.
witten commented 2025-03-11 21:29:06 +00:00
Owner
Copy link

This is implemented in main and will be part of the next release! The documentation will be online here shortly: https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/

The TL;DR is that borgmatic command hooks are being replaced with a new hook configuration format that should hopefully be more flexible and address both of the above use cases. Example:

commands:
    - before: configuration
      when: [create]
      run:
          - export-all-of-the-databases-artifacts-etc.sh
    - before: dump_data_sources
      hooks: [lvm]
      run:
          - systemctl stop services or whatever
    - after: dump_data_sources
      hooks: [lvm]
      run:
          - systemctl start services or whatever
This is implemented in main and will be part of the next release! The documentation will be online here shortly: https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/ The TL;DR is that borgmatic command hooks are being replaced with a new hook configuration format that should hopefully be more flexible and address both of the above use cases. Example: ```yaml commands: - before: configuration when: [create] run: - export-all-of-the-databases-artifacts-etc.sh - before: dump_data_sources hooks: [lvm] run: - systemctl stop services or whatever - after: dump_data_sources hooks: [lvm] run: - systemctl start services or whatever ```
๐ŸŽ‰ 1
shadow7412 commented 2025-03-11 22:12:21 +00:00
Author
Copy link

Awesome! I look forward to giving it a whirl. Thanks very much.

Awesome! I look forward to giving it a whirl. Thanks very much.
๐Ÿ‘ 1
akostadinov commented 2025-03-12 09:59:24 +00:00
Copy link

So xfsfreeze and cryptsetup will possibly be implementable in this way? Sorry it was not clear from the example and it is fine for me to wait until the documentation is ready.

So `xfsfreeze` and `cryptsetup` will possibly be implementable in this way? Sorry it was not clear from the example and it is fine for me to wait until the documentation is ready.
witten commented 2025-03-12 16:49:26 +00:00
Owner
Copy link

Yes, something like this:

commands:
    - before: dump_data_sources
      hooks: [lvm]
      run:
          - systemctl stop services
          - xfs_freeze ...
          - cryptsetup open ...
          - mount ...
    - after: dump_data_sources
      hooks: [lvm]
      run:
          - umount ...
          - cryptsetup close ...
          - systemctl start services

This is assumes the built-in LVM snapshotting feature is being used.

Yes, something like this: ```yaml commands: - before: dump_data_sources hooks: [lvm] run: - systemctl stop services - xfs_freeze ... - cryptsetup open ... - mount ... - after: dump_data_sources hooks: [lvm] run: - umount ... - cryptsetup close ... - systemctl start services ``` This is assumes the [built-in LVM snapshotting feature](https://torsion.org/borgmatic/docs/how-to/snapshot-your-filesystems/#lvm) is being used.
โค๏ธ 1
akostadinov commented 2025-03-16 17:46:31 +00:00
Copy link

Thanks a lot! Would be nice to have these hooks documented. I make some sense out of the examples but I feel like guessing. Not urgent because I'm doing other setup before looking into upgrading borgmatic.

Thanks a lot! Would be nice to have these hooks documented. I make some sense out of the examples but I feel like guessing. Not urgent because I'm doing other setup before looking into upgrading borgmatic.
witten commented 2025-03-16 17:54:36 +00:00
Owner
Copy link

Have you seen the linked documentation here? https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/

I'd be happy to improve the docs if they're not clear/detailed enough.

Have you seen the linked documentation here? https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/ I'd be happy to improve the docs if they're not clear/detailed enough.
akostadinov commented 2025-03-16 18:54:03 +00:00
Copy link

My bad! I was searching the docs for hook and didn't spot preparation and cleanup steps.

It makes a lot more sense now.

It looks much more robust. Two questions come to mind about the dump_data_sources hook.

  1. If selected multiple other hooks (e.g. hooks: [zfs,btrfs] will the hook execute for both of these separately or once before/after both?
  2. lvm hook is not listed in documentation. It will perform the snapshotting before backup and will remove it after backup. Seems like the before hook will run before lvm snapshot and the after hook will run after snapshot, correct? This is good. I see nothing though to run before/after snapshot being removed. Maybe such commands (e.g. cryptsetup close) could go in after: action ๐Ÿค”

Or if after dump_data_sources runs only after snapshot is removed, then all services will be down for the whole backup.

Or maybe I miss something?

My bad! I was searching the docs for `hook` and didn't spot preparation and cleanup steps. It makes a lot more sense now. It looks much more robust. Two questions come to mind about the `dump_data_sources` hook. 1. If selected multiple other hooks (e.g. `hooks: [zfs,btrfs]` will the hook execute for both of these separately or once before/after both? 2. `lvm` hook is not listed in documentation. It will perform the snapshotting before backup and will remove it after backup. Seems like the `before` hook will run before lvm snapshot and the `after` hook will run after snapshot, correct? This is good. I see nothing though to run before/after snapshot being removed. Maybe such commands (e.g. `cryptsetup close`) could go in `after: action` ๐Ÿค” Or if `after` `dump_data_sources` runs only after snapshot is removed, then all services will be down for the whole backup. Or maybe I miss something?
witten commented 2025-03-17 17:27:34 +00:00
Owner
Copy link

My bad! I was searching the docs for hook and didn't spot preparation and cleanup steps.

Yeah, I probably need to reorganize the docs to make finding stuff easier. Search would probably help too.

If selected multiple other hooks (e.g. hooks: [zfs,btrfs] will the hook execute for both of these separately or once before/after both?

It will execute once before/after the ZFS snapshotting and again before/after the Btrfs snapshotting. I'll clarify in the docs.

lvm hook is not listed in documentation. It will perform the snapshotting before backup and will remove it after backup. Seems like the before hook will run before lvm snapshot and the after hook will run after snapshot, correct? This is good.

That's correct. The LVM hook is documented here: https://torsion.org/borgmatic/docs/how-to/snapshot-your-filesystems/#lvm

I see nothing though to run before/after snapshot being removed. Maybe such commands (e.g. cryptsetup close) could go in after: action ๐Ÿค”

Yeah, I haven't implemented a hook for snapshot removal yet. And in fact it might not work like you're expecting, because snapshot removal actually happens twice during any given borgmatic run (assuming a single data source hook). Would running cryptsetup close work in an after: dump_data_sources hook, or is there is a reason that's too early?

> My bad! I was searching the docs for hook and didn't spot preparation and cleanup steps. Yeah, I probably need to [reorganize the docs](https://projects.torsion.org/borgmatic-collective/borgmatic/issues/942) to make finding stuff easier. Search would probably help too. > If selected multiple other hooks (e.g. hooks: [zfs,btrfs] will the hook execute for both of these separately or once before/after both? It will execute once before/after the ZFS snapshotting and again before/after the Btrfs snapshotting. I'll clarify in the docs. > lvm hook is not listed in documentation. It will perform the snapshotting before backup and will remove it after backup. Seems like the before hook will run before lvm snapshot and the after hook will run after snapshot, correct? This is good. That's correct. The LVM hook is documented here: https://torsion.org/borgmatic/docs/how-to/snapshot-your-filesystems/#lvm > I see nothing though to run before/after snapshot being removed. Maybe such commands (e.g. cryptsetup close) could go in after: action ๐Ÿค” Yeah, I haven't implemented a hook for snapshot removal yet. And in fact it might not work like you're expecting, because snapshot removal actually happens twice during any given borgmatic run (assuming a single data source hook). Would running `cryptsetup close` work in an `after: dump_data_sources` hook, or is there is a reason that's too early?
akostadinov commented 2025-03-19 17:21:54 +00:00
Copy link

The cryptsetup close should run between umount and lvm snapshot removal (this is also after the actual backup has taken place). Otherwise source will be mounted and cryptsetup close will fail.

But then cryptsetup open should run between lvm snapshot create and mount. Which is probably somewhere within dump_data_sources and not sure it is supportable. That perhaps make it even more complicated.

I'm not too attached to having this functionality. Currently what I have as before/after backup hooks is working pretty well. I see another important detail though. It needs to be known which hooks will execute in case of an error at the different stages.

I'm currently on borgmatic 1.9 and it seems like after_backup does not run in case backup failed. Which leaves the system in an inconsistent state and in my case it causes a second backup attempt to also fail in the before_backup stage, and that in turn leaves my services down.

Perhaps "after" hooks should always run in case the "before" hook has run (whether such was defined by the user or not). That would probably enable the ability to restore system state in case of a problem.

The `cryptsetup close` should run between `umount` and lvm snapshot removal (this is also **after** the actual backup has taken place). Otherwise source will be mounted and `cryptsetup close` will fail. But then `cryptsetup open` should run between lvm snapshot create and `mount`. Which is probably somewhere within `dump_data_sources` and not sure it is supportable. That perhaps make it even more complicated. I'm not too attached to having this functionality. Currently what I have as before/after backup hooks is working pretty well. I see another important detail though. It needs to be known which hooks will execute in case of an error at the different stages. I'm currently on borgmatic 1.9 and it seems like `after_backup` does not run in case backup failed. Which leaves the system in an inconsistent state and in my case it causes a second backup attempt to also fail in the `before_backup` stage, and that in turn leaves my services down. Perhaps "after" hooks should always run in case the "before" hook has run (whether such was defined by the user or not). That would probably enable the ability to restore system state in case of a problem.
witten commented 2025-03-20 17:47:57 +00:00
Owner
Copy link

That's unfortunate that the new dump_data_sources hook won't work for your use case after all! I'll probably just remove it since I'm not sure there's currently another use case for it. And you are correct that after_backup indeed does not run in the case of a failed backup, but the good news is that the forthcoming after command hooks described in this ticket do run even when an error occurs.

That's unfortunate that the new `dump_data_sources` hook won't work for your use case after all! I'll probably just remove it since I'm not sure there's currently another use case for it. And you are correct that `after_backup` indeed does not run in the case of a failed backup, but the good news is that the forthcoming `after` command hooks described in this ticket _do_ run even when an error occurs.
witten commented 2025-04-06 15:45:33 +00:00
Owner
Copy link

Released in borgmatic 2.0.0!

Released in borgmatic 2.0.0!
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
parallel-borgmatic
command-hook-steps
2.1.7
2.1.6
2.1.5
2.1.4
2.1.3
2.1.2
2.1.1
2.1.0
2.0.13
2.0.12
2.0.11
2.0.10
2.0.9
2.0.8
2.0.7
2.0.6
2.0.5
2.0.4
2.0.3
2.0.2
2.0.1
2.0.0
1.9.14
1.9.13
1.9.12
1.9.11
1.9.10
1.9.9
1.9.8
1.9.7
1.9.6
1.9.5
1.9.4
1.9.3
1.9.2
1.9.1
1.9.0
1.8.14
1.8.13
1.8.12
1.8.11
1.8.10
1.8.9
1.8.8
1.8.7
1.8.6
1.8.5
1.8.4
1.8.3
1.8.2
1.8.1
1.8.0
1.7.15
1.7.14
1.7.13
1.7.12
1.7.11
1.7.10
1.7.9
1.7.8
1.7.7
1.7.6
1.7.5
1.7.4
1.7.3
1.7.2
1.7.1
1.7.0
1.6.6
1.6.5
1.6.4
1.6.3
1.6.2
1.6.1
1.6.0
1.5.24
1.5.23
1.5.22
1.5.21
1.5.20
1.5.19
1.5.18
1.5.17
1.5.16
1.5.15
1.5.14
1.5.13
1.5.12
1.5.11
1.5.10
1.5.9
1.5.8
1.5.7
1.5.7.dev0
1.5.6
1.5.5
1.5.4
1.5.3
1.5.2
1.5.1
1.5.0
1.4.22
1.4.21
1.4.20
1.4.19
1.4.18
1.4.17
1.4.16
1.4.15
1.4.14
1.4.13
1.4.12
1.4.11
1.4.10
1.4.9
1.4.8
1.4.7
1.4.6
1.4.5
1.4.4
1.4.3
1.4.2
1.4.1
1.4.0
1.3.26
1.3.25
1.3.24
1.3.23
1.3.22
1.3.21
1.3.20
1.3.19
1.3.18
1.3.17
1.3.16
1.3.15
1.3.14
1.3.13
1.3.12
1.3.11
1.3.10
1.3.9
1.3.8
1.3.7
1.3.6
1.3.5
1.3.4
1.3.3
1.3.2
1.3.1
1.3.0
1.2.18
1.2.17
1.2.16
1.2.15
1.2.14
1.2.13
1.2.12
1.2.11
1.2.10
1.2.9
1.2.8
1.2.7
1.2.6
1.2.5
1.2.4
1.2.3
1.2.2
1.2.1
1.2.0
1.1.15
1.1.14
1.1.13
1.1.12
1.1.11
1.1.10
1.1.9
1.1.7
1.1.6
1.1.5
1.1.4
1.1.3
1.1.2
1.1.1
1.1.0
1.0.3
1.0.2
1.0.1
1.0.0
0.1.8
0.1.7
0.1.6
0.1.5
0.1.4
0.1.3
0.1.2
0.1.1
0.1.0
0.0.7
0.0.6
0.0.5
0.0.4
0.0.3
0.0.2
Labels
Clear labels   
blocked

   
breaking

   
bug

   
data loss

   
design finalized

   
good first issue

   
new feature area

   
question / support

   
security

   
waiting for response
No labels
blocked
breaking
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
borgmatic-collective/borgmatic#790
No description provided.