Shouldn't before_everything hook also work for list command? #463

New Issue

kriansa · 2021-11-08T22:23:45Z

kriansa commented

2021-11-08 22:23:45 +00:00

What I'm trying to do and why

I currently have my backups sitting at a NAS and I connect to them by mounting a SMB mount onto a /mnt/Backup directory. That works automatically for backups and restore by using hooks.before_everything hook, however for the list command it won't work as it won't execute the hook commands beforehand. The same thing goes for the after_everything hook.

Are there any design reasons why this is not working?

Steps to reproduce (if a bug)

Have a simple backup setup with borgmatic and add this hook to it:

hooks:
  before_everything:
    - echo "Hey!! I ran!"

Run borgmatic list

Actual behavior (if a bug)

You won't see a Hey!! I ran! message

Expected behavior (if a bug)

You should probably see it?

Other notes / implementation ideas

Environment

borgmatic version: 1.5.20

borgmatic installation method: Arch Package

Borg version: 1.1.17

Python version: 3.9.7

operating system and version: Arch Linux

#### What I'm trying to do and why I currently have my backups sitting at a NAS and I connect to them by mounting a SMB mount onto a `/mnt/Backup` directory. That works automatically for backups and restore by using `hooks.before_everything` hook, however for the `list` command it won't work as it won't execute the hook commands beforehand. The same thing goes for the `after_everything` hook. Are there any design reasons why this is not working? #### Steps to reproduce (if a bug) 1. Have a simple backup setup with borgmatic and add this hook to it: ```yaml hooks: before_everything: - echo "Hey!! I ran!" ``` 2. Run `borgmatic list` #### Actual behavior (if a bug) You won't see a `Hey!! I ran!` message #### Expected behavior (if a bug) You should _probably_ see it? #### Other notes / implementation ideas #### Environment **borgmatic version:** 1.5.20 **borgmatic installation method:** Arch Package **Borg version:** 1.1.17 **Python version:** 3.9.7 **operating system and version:** Arch Linux

witten commented

2021-11-08 22:44:18 +00:00

Thanks for the ticket! This limitation is by design (and documented as such), but I'd be open to changing the design if it's not working for use cases like this one.

The current idea is that before_everything is for any setup needed before running a create action to make a backup—run before all other actions across all configuration files. However, as you've discovered, there's often common setup needed for both create and list, like mounting the path where the repository resides.

The challenge here is that I believe some folks are using before_everything for running commands that they truly only want on create. For instance, in the original ticket where before_everything was implemented, the use case is for creating a global LVM snapshot before any backups run. Presumably, for that use case, you wouldn't want to create an LVM snapshot before running list.

So one option I could see is changing the before_everything hook anyway to run on all actions, and those users would just have to somehow deal with the change. Maybe an unnecessary LVM snapshot isn't a big deal, especially if it gets torn down after?

Another option would be to add a new hook for your case—something that gets run for any repository setup, regardless of action.

Thanks for the ticket! This limitation is by design (and [documented as such](https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/)), but I'd be open to changing the design if it's not working for use cases like this one. The current idea is that `before_everything` is for any setup needed before running a `create` action to make a backup—run before all other actions across all configuration files. However, as you've discovered, there's often common setup needed for both `create` and `list`, like mounting the path where the repository resides. The challenge here is that I believe some folks are using `before_everything` for running commands that they truly only want on `create`. For instance, in the [original ticket](https://projects.torsion.org/borgmatic-collective/borgmatic/issues/192) where `before_everything` was implemented, the use case is for creating a global LVM snapshot before any backups run. Presumably, for that use case, you wouldn't want to create an LVM snapshot before running `list`. So one option I could see is changing the `before_everything` hook anyway to run on all actions, and those users would just have to somehow deal with the change. Maybe an unnecessary LVM snapshot isn't a big deal, especially if it gets torn down after? Another option would be to add a new hook for your case—something that gets run for any repository setup, regardless of action.

witten commented

2021-11-08 22:53:02 +00:00

One other more out there idea: If you're just issuing a mount command for the SMB-hosted Borg repository, I could see making a general-purpose borgmatic "mount" hook that lets you specify a path to mount (and mount options) for particular repositories. And then borgmatic would take care of mounting and unmounting it without having to give explicit commands for that. Part of the difficulty there though would be making a clear distinction from the existing borg mount and borgmatic mount actions (which mount archives in repositories, not Unix paths).

One other more out there idea: If you're just issuing a mount command for the SMB-hosted Borg repository, I could see making a general-purpose borgmatic "mount" hook that lets you specify a path to mount (and mount options) for particular repositories. And then borgmatic would take care of mounting and unmounting it without having to give explicit commands for that. Part of the difficulty there though would be making a clear distinction from the existing `borg mount` and `borgmatic mount` actions (which mount archives in repositories, not Unix paths).

kriansa commented

2021-11-09 01:49:34 +00:00

@witten Thank you for your quick response!

I saw the original ticket and it does make sense to have separate hook configurations for before-all-backups and before-all-commands sort of thing.

If I understand correctly - and I apologize if I haven't - I think that the chosen naming convention does not reflect the real behavior of the feature (before_everything means something wider than something that would be executed prior to the creation of every backup setup). However, I don't like breaking changes that could easily hurt existing users, so I would be very careful about renaming it, and doing it in phases (deprecation then removal). But I also understand that the effort to do that might not worth it.

As an alternative, I would suggest a separate feature that could be used as a "pre-command" hook, like that:

hooks:
  # This hook will be executed prior to every `borgmatic` subcommand, except `init`
  before_commands:
    - echo "Hey!!"

And perhaps, if you like, we could even transform the whole hook functionality into a more generic feature so we could have several hooks that would be applied only to specified commands, thus making hooks very flexible. Something like that:

hooks:
  - commands: [check]
    precmd: 
      - echo "Starting check..."
    postcmd:
      - echo "Check finished!"
      - curl -XPOST "https://my-monitoring.com/check"

(This is just an idea, not an actual proposal, as it is much more complex :P)

One other more out there idea: If you're just issuing a mount command for the SMB-hosted Borg repository, I could see making a general-purpose borgmatic "mount" hook that lets you specify a path to mount (and mount options) for particular repositories. And then borgmatic would take care of mounting and unmounting it without having to give explicit commands for that. Part of the difficulty there though would be making a clear distinction from the existing borg mount and borgmatic mount actions (which mount archives in repositories, not Unix paths).

I like the idea because it suggests, by design, a good practice for "linking" a backup repository to the local machine. As you mentioned, however, I don't like the ambiguity that the word mount can have in this context, because it can mean something very different on borg. Although borgmatic is a different software, they are very close in the same ecosystem and users could easily get confused about the two operations. Names are indeed hard, but what do you think about borgmatic mount-repo?

@witten Thank you for your quick response! I saw the original ticket and it does make sense to have separate hook configurations for `before-all-backups` and `before-all-commands` sort of thing. If I understand correctly - and I apologize if I haven't - I think that the chosen naming convention does not reflect the real behavior of the feature (`before_everything` means something wider than something that would be executed prior to the creation of every backup setup). However, I don't like breaking changes that could easily hurt existing users, so I would be very careful about renaming it, and doing it in phases (deprecation then removal). But I also understand that the effort to do that might not worth it. As an alternative, I would suggest a separate feature that could be used as a "pre-command" hook, like that: ```yaml hooks: # This hook will be executed prior to every `borgmatic` subcommand, except `init` before_commands: - echo "Hey!!" ``` And perhaps, if you like, we could even transform the whole hook functionality into a more generic feature so we could have several hooks that would be applied only to specified commands, thus making hooks very flexible. Something like that: ```yaml hooks: - commands: [check] precmd: - echo "Starting check..." postcmd: - echo "Check finished!" - curl -XPOST "https://my-monitoring.com/check" ``` (This is just an idea, not an actual proposal, as it is much more complex :P) > One other more out there idea: If you're just issuing a mount command for the SMB-hosted Borg repository, I could see making a general-purpose borgmatic "mount" hook that lets you specify a path to mount (and mount options) for particular repositories. And then borgmatic would take care of mounting and unmounting it without having to give explicit commands for that. Part of the difficulty there though would be making a clear distinction from the existing borg mount and borgmatic mount actions (which mount archives in repositories, not Unix paths). I like the idea because it suggests, by design, a good practice for "linking" a backup repository to the local machine. As you mentioned, however, I don't like the ambiguity that the word `mount` can have in this context, because it can mean something very different on borg. Although borgmatic is a different software, they are very close in the same ecosystem and users could easily get confused about the two operations. Names are indeed hard, but what do you think about `borgmatic mount-repo`?

witten commented

2021-11-15 18:23:35 +00:00

Your suggestions about the various "before" hooks make sense to me. And, yeah, the challenge is making it flexible enough for the various use cases without making either the user's configuration or the underlying implementation too cumbersome. (Or too different from the existing before_backup/before_list hooks.)

As for the mount hook, I wasn't thinking so much of a new borgmatic command-line action, but more of a configuration file hook that automatically gets applied before and after doing anything with a repository. For instance, maybe it could look something like this:

hooks:
    repository_mounts:
        - repository: /mnt/Backup
          type: cifs
          device: //server_name/share_name
          mount_options: username=DOMAIN\Administrator
          mount_flags: --read-only

Which would cause borgmatic to mount the repository before use and unmount it afterwards. Of course this would only work if you're using the mount command to mount your SMB share and umount to unmount it...

Your suggestions about the various "before" hooks make sense to me. And, yeah, the challenge is making it flexible enough for the various use cases without making either the user's configuration or the underlying implementation too cumbersome. (Or too different from the existing `before_backup`/`before_list` hooks.) As for the mount hook, I wasn't thinking so much of a new borgmatic command-line action, but more of a configuration file hook that automatically gets applied before and after doing anything with a repository. For instance, maybe it could look something like this: ```yaml hooks: repository_mounts: - repository: /mnt/Backup type: cifs device: //server_name/share_name mount_options: username=DOMAIN\Administrator mount_flags: --read-only ``` Which would cause borgmatic to mount the repository before use and unmount it afterwards. Of course this would only work if you're using the `mount` command to mount your SMB share and `umount` to unmount it...

❤️ 1

kriansa commented

2021-11-15 23:53:23 +00:00

As for the mount hook, I wasn't thinking so much of a new borgmatic command-line action, but more of a configuration file hook that automatically gets applied before and after doing anything with a repository. For instance, maybe it could look something like this:

Oh I see! It makes sense, yeah. However I think that this deviates a little bit from the "hook" concept, wouldn't that belong somewhere else in the YAML, such as in the repository definition? Maybe something like that:

repositories:
  - /my/backup/folder
  - path: /my/other/backup/folder/automatically/mounted
    type: cifs
    device: //server_name/share_name
    mount_options: username=DOMAIN\Administrator
    mount_flags: --read-only
hooks:
  # ...

(we can make this backwards compatible)

What do you think?

> As for the mount hook, I wasn't thinking so much of a new borgmatic command-line action, but more of a configuration file hook that automatically gets applied before and after doing anything with a repository. For instance, maybe it could look something like this: Oh I see! It makes sense, yeah. However I think that this deviates a little bit from the "hook" concept, wouldn't that belong somewhere else in the YAML, such as in the `repository` definition? Maybe something like that: ```yaml repositories: - /my/backup/folder - path: /my/other/backup/folder/automatically/mounted type: cifs device: //server_name/share_name mount_options: username=DOMAIN\Administrator mount_flags: --read-only hooks: # ... ``` (we can make this backwards compatible) What do you think?

witten commented

2021-11-16 00:03:59 +00:00

Yeah, that makes sense in the repository configuration. Good idea! Might take some work to support multiple "types" for the repositories schema, but that's hopefully just an implementation detail.

Yeah, that makes sense in the repository configuration. Good idea! Might take some work to support multiple "types" for the `repositories` schema, but that's hopefully just an implementation detail.

witten referenced this issue

2022-04-22 05:19:36 +00:00

before_everything / after_everything hook does not apply for "borgmatic info/list" #487

kriansa commented

2022-04-27 19:19:27 +00:00

I know it's been some time since we discussed, but as I re-read this issue I just wanted to clarify what we've talked and summarize things a little bit.

Your suggestions about the various "before" hooks make sense to me. And, yeah, the challenge is making it flexible enough for the various use cases without making either the user's configuration or the underlying implementation too cumbersome. (Or too different from the existing before_backup/before_list hooks.)

Is it right to assume that you liked/approved the suggestion of the addition of a new hooks entry on the YAML so that we could have multiple entries for commands? I.e.:

hooks:
  - commands: [check]
    precmd: 
      - echo "Starting check..."
    postcmd:
      - echo "Check finished!"
      - curl -XPOST "https://my-monitoring.com/check"

Yeah, that makes sense in the repository configuration. Good idea!

Amazing, so we're on the same page as far as a new way of defining repositories as described below:

repositories:
  # simple declaration - backwards-compatible with existing syntax
  - /my/backup/folder
  # extended declaration - new way
  - path: /my/other/backup/folder/automatically/mounted
    type: cifs
    device: //server_name/share_name
    mount_options: username=DOMAIN\Administrator
    mount_flags: --read-only

Might take some work to support multiple "types" for the repositories schema, but that's hopefully just an implementation detail.

Yes, that makes sense. What do you think of having a starting point so we can build upon later, by adding, for now, only a single adapter named shellcmd? That way we wouldn't need to care too much about how to implement the underlying structure of many other adapters at first, and yet we make it very reusable. I.e.:

repositories:
  - path: /mnt/MyBackup
    type: shellcmd
    mountcmd:
      - test -d /mnt/MyBackup || mkdir /mnt/MyBackup
      - mountpoint /mnt/MyBackup || mount nfs.myserver.com:/mybkps /mnt/MyBackup
    unmountcmd:
      - umount /mnt/MyBackup
      - rmdir /mnt/MyBackup

That way we open the possibilities right out the gate and only later we can start thinking about implementing adapters such as cifs, nfs, s3, etc, and they could be much easily implemented (IMHO).

Lastly, if I understand everything correctly, I think the new repository declaration suggestion should have a higher priority than the hooks one, since I believe it is more useful and more semantically accurate for that particular purpose. (In fact it could even deprecates the whole hooks feature but that might be another topic.)

I know it's been some time since we discussed, but as I re-read this issue I just wanted to clarify what we've talked and summarize things a little bit. > Your suggestions about the various "before" hooks make sense to me. And, yeah, the challenge is making it flexible enough for the various use cases without making either the user's configuration or the underlying implementation too cumbersome. (Or too different from the existing before_backup/before_list hooks.) Is it right to assume that you liked/approved the suggestion of the addition of a new `hooks` entry on the YAML so that we could have multiple entries for commands? I.e.: ```yaml hooks: - commands: [check] precmd: - echo "Starting check..." postcmd: - echo "Check finished!" - curl -XPOST "https://my-monitoring.com/check" ``` > Yeah, that makes sense in the repository configuration. Good idea! Amazing, so we're on the same page as far as a new way of defining repositories as described below: ```yaml repositories: # simple declaration - backwards-compatible with existing syntax - /my/backup/folder # extended declaration - new way - path: /my/other/backup/folder/automatically/mounted type: cifs device: //server_name/share_name mount_options: username=DOMAIN\Administrator mount_flags: --read-only ``` > Might take some work to support multiple "types" for the `repositories` schema, but that's hopefully just an implementation detail. Yes, that makes sense. What do you think of having a starting point so we can build upon later, by adding, for now, only a single adapter named `shellcmd`? That way we wouldn't need to care too much about how to implement the underlying structure of many other adapters at first, and yet we make it very reusable. I.e.: ```yaml repositories: - path: /mnt/MyBackup type: shellcmd mountcmd: - test -d /mnt/MyBackup || mkdir /mnt/MyBackup - mountpoint /mnt/MyBackup || mount nfs.myserver.com:/mybkps /mnt/MyBackup unmountcmd: - umount /mnt/MyBackup - rmdir /mnt/MyBackup ``` That way we open the possibilities right out the gate and only later we can start thinking about implementing adapters such as `cifs`, `nfs`, `s3`, etc, and they could be much easily implemented (IMHO). Lastly, if I understand everything correctly, I think the new `repository` declaration suggestion should have a higher priority than the `hooks` one, since I believe it is more useful and more semantically accurate for that particular purpose. _(In fact it could even deprecates the whole `hooks` feature but that might be another topic.)_

witten commented

2022-04-28 02:55:52 +00:00

Is it right to assume that you liked/approved the suggestion of the addition of a new hooks entry on the YAML so that we could have multiple entries for commands?

I'd certainly be open to something like this if the use cases support it! Sounds like we might have another solution for your mounting/unmounting use case though. Specifically, your idea about mount/unmount commands on a per-repository basis. I like the princicle of starting small and then expanding from there.

However, I'm now thinking through some of the implications of adding per-repository options as in your example. First of all, there will be the natural feature creep of: Well, why can't I also set the remote_path per-repository? E.g., I have a config file with two repositories and I want all the options in common—except for remote_path. Then, someone else asks: Well what about encryption_passhprase or keep_hourly or check_last. Surely those could be snuck in as per-repository options as well. And before you know it, you've gotten a whole borgmatic configuration file stuffed into the repositories: list—potentially multiple times. (And this isn't all theoretical, either. Folks have already asked for a bunch of this.)

Hopefully you'd agree that that could get pretty unwieldy. So I've got an alternate proposal based on some recent enhancements in borgmatic 1.6.0 (just released). Consider something like this:

common.yaml:

# All of your common options across repositories go in this common.yaml file.

location:
    source_directories:
        - /etc
        ...
storage:
    ...
retention:
    ...
consistency:
    ...
hooks:
    ...

repository1.yaml:

# This repository uses all the common configuration and then layers on repo-specific
# mount/unmount commands.

<<: !include common.yaml

location:
    repositories:
        - /mnt/MyBackup

hooks:
    # Proposed new hooks analogous to your "mountcmd"/"unmountcmd".
    before_repository:
      - test -d /mnt/MyBackup || mkdir /mnt/MyBackup
      - mountpoint /mnt/MyBackup || mount nfs.myserver.com:/mybkps /mnt/MyBackup
    after_repository:
      - umount /mnt/MyBackup
      - rmdir /mnt/MyBackup

repository2.yaml:

# This repository just uses all the common configuration and then layers in its own
# repo-specific options.

<<: !include common.yaml

location:
    repositories:
        - k8pDxu32@k8pDxu32.repo.borgbase.com:repo
        
retention:
    keep_hourly: 24

This approach is perhaps more verbose or less convenient than what you suggested, but my thinking is that it would support arbitrary borgmatic options per-repository or shared across repositories—including mounting and unmounting.

Thoughts? Please feel free to shoot holes in this proposal.

> Is it right to assume that you liked/approved the suggestion of the addition of a new hooks entry on the YAML so that we could have multiple entries for commands? I'd certainly be open to something like this if the use cases support it! Sounds like we might have another solution for your mounting/unmounting use case though. Specifically, your idea about mount/unmount commands on a per-repository basis. I like the princicle of starting small and then expanding from there. However, I'm now thinking through some of the implications of adding per-repository options as in your example. First of all, there will be the natural feature creep of: Well, why can't I also set the `remote_path` per-repository? E.g., I have a config file with two repositories and I want all the options in common—except for `remote_path`. Then, someone else asks: Well what about `encryption_passhprase` or `keep_hourly` or `check_last`. Surely those could be snuck in as per-repository options as well. And before you know it, you've gotten a whole borgmatic configuration file stuffed into the `repositories:` list—potentially multiple times. (And this isn't all theoretical, either. Folks have already asked for a bunch of this.) Hopefully you'd agree that that could get pretty unwieldy. So I've got an alternate proposal based on some recent enhancements in borgmatic 1.6.0 (just released). Consider something like this: common.yaml: ``` # All of your common options across repositories go in this common.yaml file. location: source_directories: - /etc ... storage: ... retention: ... consistency: ... hooks: ... ``` repository1.yaml: ``` # This repository uses all the common configuration and then layers on repo-specific # mount/unmount commands. <<: !include common.yaml location: repositories: - /mnt/MyBackup hooks: # Proposed new hooks analogous to your "mountcmd"/"unmountcmd". before_repository: - test -d /mnt/MyBackup || mkdir /mnt/MyBackup - mountpoint /mnt/MyBackup || mount nfs.myserver.com:/mybkps /mnt/MyBackup after_repository: - umount /mnt/MyBackup - rmdir /mnt/MyBackup ``` repository2.yaml: ``` # This repository just uses all the common configuration and then layers in its own # repo-specific options. <<: !include common.yaml location: repositories: - k8pDxu32@k8pDxu32.repo.borgbase.com:repo retention: keep_hourly: 24 ``` This approach is perhaps more verbose or less convenient than what you suggested, but my thinking is that it would support arbitrary borgmatic options per-repository *or* shared across repositories—including mounting and unmounting. Thoughts? Please feel free to shoot holes in this proposal.

witten referenced this issue

2022-04-28 20:58:33 +00:00

Provide separate hook to check repository availability #456

kriansa commented

2022-05-05 04:20:05 +00:00

You're absolutely right. Having experienced people asking for that set of functionality in the past is definitely a tell that maybe the most reusability we could get if we were to redesign the schema, would be to make the config repository-centric rather than backup-centric. But I understand that would be a disrupting change and hardly backwards compatible.

I don't find holes in your suggestion and I really like it -- using YAML's built-in power to work around that is awesome. IMHO though, I find it a little difficult to document that and make it clear that this modular config is meant to make configs more reusable and repository-centric. Maybe this is the case for more advanced users? I know that people using Borg aren't beginners (or are they?), but I'm not a maintainer and I have no idea of your pain in that sense.

The only thing I didn't pick up so far is how that is also going to help with the original issue, which is to be able to reuse hooks when running list.

You're absolutely right. Having experienced people asking for that set of functionality in the past is definitely a tell that maybe the most reusability we could get if we were to redesign the schema, would be to make the config repository-centric rather than backup-centric. But I understand that would be a disrupting change and hardly backwards compatible. I don't find holes in your suggestion and I really like it -- using YAML's built-in power to work around that is awesome. IMHO though, I find it a little difficult to document that and make it clear that this modular config is meant to make configs more reusable and repository-centric. Maybe this is the case for more advanced users? I know that people using Borg aren't beginners (or are they?), but I'm not a maintainer and I have no idea of your pain in that sense. The only thing I didn't pick up so far is how that is also going to help with the original issue, which is to be able to reuse `hooks` when running `list`.

witten commented

2022-05-05 04:46:44 +00:00

... would be to make the config repository-centric rather than backup-centric.

It's an interesting idea, and I'd be open to looking at proposals on that front. I think though that while some folks look at their configuration as repository-centric, others look at it by other axes (application-centric, source-centric, etc.). So I'm not sure a one-size-fits-all approach would work here.

IMHO though, I find it a little difficult to document that and make it clear that this modular config is meant to make configs more reusable and repository-centric. Maybe this is the case for more advanced users? I know that people using Borg aren't beginners (or are they?), but I'm not a maintainer and I have no idea of your pain in that sense.

I agree with this concern. Borg and borgmatic users, in my experience, include novice users and expert users and everyone in between. I think the best we could do for the YAML include approach would be to document it well and points folks at those docs. Right now, the include functionality is documented. I think what'd be needed on top of that is perhaps a separate how-to guide on using per-repository configuration like the example above, as that's a pretty common use case.

The only thing I didn't pick up so far is how that is also going to help with the original issue, which is to be able to reuse hooks when running list.

The idea with the proposed before_repository and after_repository hooks is that they'd run for any borgmatic action related to repositories configured in that configuration file. And that'd include list. So yes, you could declare common (or not common!) before/after repository hooks and they'd get run for any borgmatic list action or any other action, once per repository.

Let me know your thoughts!

> ... would be to make the config repository-centric rather than backup-centric. It's an interesting idea, and I'd be open to looking at proposals on that front. I think though that while some folks look at their configuration as repository-centric, others look at it by other axes (application-centric, source-centric, etc.). So I'm not sure a one-size-fits-all approach would work here. > IMHO though, I find it a little difficult to document that and make it clear that this modular config is meant to make configs more reusable and repository-centric. Maybe this is the case for more advanced users? I know that people using Borg aren't beginners (or are they?), but I'm not a maintainer and I have no idea of your pain in that sense. I agree with this concern. Borg and borgmatic users, in my experience, include novice users and expert users and everyone in between. I think the best we could do for the YAML include approach would be to document it well and points folks at those docs. Right now, the [include functionality](https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/#include-merging) is documented. I think what'd be needed on top of that is perhaps a separate how-to guide on using per-repository configuration like the example above, as that's a pretty common use case. > The only thing I didn't pick up so far is how that is also going to help with the original issue, which is to be able to reuse hooks when running list. The idea with the proposed `before_repository` and `after_repository` hooks is that they'd run for *any* borgmatic action related to repositories configured in that configuration file. And that'd include `list`. So yes, you could declare common (or not common!) `before`/`after` repository hooks and they'd get run for any `borgmatic list` action or any other action, once per repository. Let me know your thoughts!

kriansa commented

2022-05-05 20:43:32 +00:00

I think though that while some folks look at their configuration as repository-centric, others look at it by other axes (application-centric, source-centric, etc.). So I'm not sure a one-size-fits-all approach would work here.

Yes, I agree that a redesign at this point is overkill, and if not carefully crafted, it might just piss off many users just to benefit a few. The proposal so far is very flexible and meets all requirements you've been asked for, so I wouldn't touch that right now.

I think what'd be needed on top of that is perhaps a separate how-to guide on using per-repository configuration like the example above, as that's a pretty common use case.

Maybe right now that would be the best we could do to address that use-case. The docs are already very good at that (pointing out specific use-cases rather than just being an html version of --help), so I suspect this should be fine - at least for now.

The idea with the proposed before_repository and after_repository hooks is that they'd run for any borgmatic action related to repositories configured in that configuration file. And that'd include list. So yes, you could declare common (or not common!) before/after repository hooks and they'd get run for any borgmatic list action or any other action, once per repository.

Got it! It makes sense now and I really like it. TBH, I don't find the naming *_repository super intuitive, but I lack the creativity to suggest something else 😂

> I think though that while some folks look at their configuration as repository-centric, others look at it by other axes (application-centric, source-centric, etc.). So I'm not sure a one-size-fits-all approach would work here. Yes, I agree that a redesign at this point is overkill, and if not carefully crafted, it might just piss off many users just to benefit a few. The proposal so far is very flexible and meets all requirements you've been asked for, so I wouldn't touch that right now. > I think what'd be needed on top of that is perhaps a separate how-to guide on using per-repository configuration like the example above, as that's a pretty common use case. Maybe right now that would be the best we could do to address that use-case. The docs are already very good at that (_pointing out specific use-cases rather than just being an html version of `--help`_), so I suspect this should be fine - at least for now. > The idea with the proposed before_repository and after_repository hooks is that they'd run for any borgmatic action related to repositories configured in that configuration file. And that'd include list. So yes, you could declare common (or not common!) before/after repository hooks and they'd get run for any borgmatic list action or any other action, once per repository. Got it! It makes sense now and I really like it. TBH, I don't find the naming `*_repository` super intuitive, but I lack the creativity to suggest something else 😂

witten commented

2022-05-06 02:37:18 +00:00

Maybe right now that would be the best we could do to address that use-case. The docs are already very good at that (pointing out specific use-cases rather than just being an html version of --help), so I suspect this should be fine - at least for now.

Sounds good!

Got it! It makes sense now and I really like it. TBH, I don't find the naming *_repository super intuitive, but I lack the creativity to suggest something else 😂

I appreciate the feedback. (It's much easier to change it at the design stage rather than once implemented and in use.) What do you think of before_action/after_action instead?

> Maybe right now that would be the best we could do to address that use-case. The docs are already very good at that (pointing out specific use-cases rather than just being an html version of --help), so I suspect this should be fine - at least for now. Sounds good! > Got it! It makes sense now and I really like it. TBH, I don't find the naming *_repository super intuitive, but I lack the creativity to suggest something else 😂 I appreciate the feedback. (It's much easier to change it at the design stage rather than once implemented and in use.) What do you think of `before_action`/`after_action` instead?

witten referenced this issue from a commit

2022-08-22 04:51:21 +00:00

Add "before_actions" and "after_actions" command hooks that run before/after all the actions for each repository, update docs to cover per-repository configurations (#463).

witten commented

2022-08-22 04:51:56 +00:00

I took a stab at this in master. The changes will be part of the next release. Here's what's changed:

Add "before_actions" and "after_actions" command hooks that run before/after all the actions for each repository.
Update documentation to cover per-repository configurations: https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/ (Will be live shortly.)

I didn't deprecate the before_everything/after_everything hooks for now, as I don't yet have a good alterative. But I'd be happy to revisit that on another ticket.

Thanks for the discussion and brainstorming!

I took a stab at this in master. The changes will be part of the next release. Here's what's changed: * Add "before_actions" and "after_actions" command hooks that run before/after all the actions for each repository. * Update documentation to cover per-repository configurations: https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/ (Will be live shortly.) I didn't deprecate the `before_everything`/`after_everything` hooks for now, as I don't yet have a good alterative. But I'd be happy to revisit that on another ticket. Thanks for the discussion and brainstorming!

witten closed this issue

2022-08-22 04:51:56 +00:00

witten commented

2022-08-22 04:59:22 +00:00

Just released in borgmatic 1.7.0!

kriansa commented

2022-08-23 02:45:21 +00:00

Hi @witten ! Thanks for that! I really appreciate it.

I apologize for not replying the last inquiry, but I'm fine with the chosen naming.

Thanks for being accepting of new ideas and working on them. I'll be changing my configs in the next days and I'll report back my findings (and I'll finally get rid of my wrapper script 😂 )

Hi @witten ! Thanks for that! I really appreciate it. I apologize for not replying the last inquiry, but I'm fine with the chosen naming. Thanks for being accepting of new ideas and working on them. I'll be changing my configs in the next days and I'll report back my findings (and I'll finally get rid of my [wrapper script](https://github.com/kriansa/dotfiles/blob/1315b956372a0260daf4b25b0ac561707ee84506/modules/borgmatic/bin/borgmatic) 😂 )

witten commented

2022-08-23 05:45:55 +00:00

Awesome! I'm interested to know how it works out.

kriansa commented

2022-09-01 03:40:52 +00:00

@witten I just wanted to let you know that everything worked fine! I've updated my settings and so far it's working like a charm!

Thanks for the great work, the docs are also fantastic!

@witten I just wanted to let you know that everything worked fine! I've updated my settings and so far it's working like a charm! Thanks for the great work, the docs are also fantastic!

witten commented

2022-09-01 03:42:55 +00:00

Glad to hear it! Thank you for reporting back.

C-Duv referenced this issue

2023-03-22 12:57:24 +00:00

Hooks to mount/umount remote (e.g. NFS) folder containing the Borg repository? #657

witten referenced this issue

2023-03-22 17:18:08 +00:00

Hooks to mount/umount remote (e.g. NFS) folder containing the Borg repository? #657

C-Duv referenced this issue

2023-03-22 18:04:41 +00:00

Hooks to mount/umount remote (e.g. NFS) folder containing the Borg repository? #657