Hooks to mount/umount remote (e.g. NFS) folder containing the Borg repository? #657

New Issue

C-Duv · 2023-03-22T12:57:24Z

C-Duv commented

2023-03-22 12:57:24 +00:00

What I'm trying to do and why

I am looking for a way to NFS-mount a remote folder containing the Borg repository prior to any Borg command being run by borgmatic.

The before_actions and after_actions hooks are great. I can test+mount in before and unmount in after:

before_actions:
    - test -d "/mnt/network_folder" || mkdir --parents "/mnt/network_folder"
    - mountpoint "/mnt/network_folder" || mount -t nfs "host.example.com:/export/backups" "/mnt/network_folder"
after_actions:
    - mountpoint "/mnt/network_folder" && umount "/mnt/network_folder"

It works, except for the borgmatic mount command where the after_actions tries to unmount the remote folder borgmatic just accessed to open the Borg repository for borgfs.

Issue #463 had interesting ideas with per-repository mountcmd/unmountcmd options that could have been the solution to my issue.

The solutions I can think of:

Use AutoFS or systemd automount system to delegate the mount/unmount job outside borgmatic. But that adds additional dependency :(
Perform more checks during after_actions: check any borgfs mount before umounting.

Other notes / implementation ideas

If I opt for performing more checks during after_actions, having access via an environment variable to the complete executed borgmatic CLI command (e.g. BORGMATIC_COMMAND containing borgmatic mount --mount-point /mnt/test) or at least to the borgmatic action (eg. BORGMATIC_ACTION containing mount) in the hooks' commands would be great.

Environment

borgmatic version: 1.7.4

borgmatic installation method: Debian package

Borg version: 1.1.16

Python version: 3.9.2

operating system and version: Debian 11 (Bullseye)

#### What I'm trying to do and why I am looking for a way to *NFS*-mount a remote folder containing the Borg repository prior to any Borg command being run by borgmatic. The `before_actions` and `after_actions` hooks are great. I can test+mount in *before* and unmount in *after*: ```yaml before_actions: - test -d "/mnt/network_folder" || mkdir --parents "/mnt/network_folder" - mountpoint "/mnt/network_folder" || mount -t nfs "host.example.com:/export/backups" "/mnt/network_folder" after_actions: - mountpoint "/mnt/network_folder" && umount "/mnt/network_folder" ``` It works, except for the `borgmatic mount` command where the `after_actions` tries to unmount the remote folder borgmatic just accessed to open the Borg repository for borgfs. [Issue #463](https://projects.torsion.org/borgmatic-collective/borgmatic/issues/463#issuecomment-4333) had interesting ideas with per-repository `mountcmd`/`unmountcmd` options that could have been the solution to my issue. The solutions I can think of: * Use AutoFS or systemd automount system to delegate the mount/unmount job outside borgmatic. But that adds additional dependency :( * Perform more checks during `after_actions`: check any borgfs mount before umounting. #### Other notes / implementation ideas If I opt for performing more checks during `after_actions`, having access via an environment variable to the complete executed borgmatic CLI command (e.g. `BORGMATIC_COMMAND` containing `borgmatic mount --mount-point /mnt/test`) or at least to the borgmatic action (eg. `BORGMATIC_ACTION` containing `mount`) in the hooks' commands would be great. #### Environment **borgmatic version:** 1.7.4 **borgmatic installation method:** Debian package **Borg version:** 1.1.16 **Python version:** 3.9.2 **operating system and version:** Debian 11 (Bullseye)

witten commented

2023-03-22 17:18:08 +00:00

Thanks for taking the time to file this! It's helpful to see exactly the shell commands you're running from hooks.

It works, except for the borgmatic mount command where the after_actions tries to unmount the remote folder borgmatic just accessed to open the Borg repository for borgfs.

Got it. Out of curiosity, is the after_actions unmount blowing up because the repository is still in use by the borg mount? Or does it successfully unmount the repository and therefore the Borg-mounted directory stops working?

Issue #463 had interesting ideas with per-repository mountcmd/unmountcmd options that could have been the solution to my issue.

I'm not sure that actually solves your issue, because if mountcmd and unmountcmd for instance are run before/after all the actions for a given repository, that's identical to the existing before_actions and after_actions. So presumably you'd have the exact same probably with mount.. Although I suppose borgmatic could be smart enough not to run the unmountcmd after any mount action.

Use AutoFS or systemd automount system to delegate the mount/unmount job outside borgmatic. But that adds additional dependency :(

Yeah, that seems like it'd be up to the user as to whether they'd want to set that up. I can see why you'd instead want everything contained in borgmatic's config file.

Perform more checks during after_actions: check any borgfs mount before umounting.

If I opt for performing more checks during after_actions, having access via an environment variable to the complete executed borgmatic CLI command (e.g. BORGMATIC_COMMAND containing borgmatic mount --mount-point /mnt/test) or at least to the borgmatic action (eg. BORGMATIC_ACTION containing mount) in the hooks' commands would be great.

The main trick here is that there can be multiple actions at once. There is a single borgmatic command-line, of course, but it can contain multiple actions and their flags, and I'm not sure I'd expect a shell command to parse that. BORGMATIC_ACTION is interesting, and it'd be easier to parse for a shell command, although it'd of course have to be BORGMATIC_ACTIONS to accommodate multiple. And instead of an environment variable, the convention is to use variable interpolation for passing borgmatic information to hook commands.

Okay, so next steps. My inclination is to solve this requirement within borgmatic (or at least in borgmatic with some user configuration) rather than requiring ever more complex shell scripting. So at the risk of blowing up scope ... Imagine if all the before_*/after_* hooks were replaced with something like this (inspired by #463 comments among others):

hooks:
    commands:
        - exclude_actions: [mount]
          before:
              - test -d "/mnt/network_folder" || mkdir --parents "/mnt/network_folder"
              - mountpoint "/mnt/network_folder" || mount -t nfs "host.example.com:/export/backups" "/mnt/network_folder"
          after:
              - mountpoint "/mnt/network_folder" && umount "/mnt/network_folder"

This sort of structure could also replace any of the other command hooks. For instance, this might be a replacement for before_check:

hooks:
    commands:
        - include_actions: [check]
          before:
              - echo "About to run a check"

Or even combining hooks:

hooks:
    commands:
        - include_actions: [prune, compact]
          before:
              - echo "Pruning and/or compacting"

(One thing to work out for this last example is if it runs twice, once before prune and once before compact—or if it just runs once before all actions if any of them are compact or prune.)

Thanks for taking the time to file this! It's helpful to see exactly the shell commands you're running from hooks. > It works, except for the borgmatic mount command where the after_actions tries to unmount the remote folder borgmatic just accessed to open the Borg repository for borgfs. Got it. Out of curiosity, is the `after_actions` unmount blowing up because the repository is still in use by the `borg mount`? Or does it successfully unmount the repository and therefore the Borg-mounted directory stops working? > Issue #463 had interesting ideas with per-repository mountcmd/unmountcmd options that could have been the solution to my issue. I'm not sure that actually solves your issue, because if `mountcmd` and `unmountcmd` for instance are run before/after all the actions for a given repository, that's identical to the existing `before_actions` and `after_actions`. So presumably you'd have the exact same probably with `mount`.. Although I suppose borgmatic could be smart enough not to run the `unmountcmd` after any `mount` action. > Use AutoFS or systemd automount system to delegate the mount/unmount job outside borgmatic. But that adds additional dependency :( Yeah, that seems like it'd be up to the user as to whether they'd want to set that up. I can see why you'd instead want everything contained in borgmatic's config file. > Perform more checks during after_actions: check any borgfs mount before umounting. > If I opt for performing more checks during after_actions, having access via an environment variable to the complete executed borgmatic CLI command (e.g. BORGMATIC_COMMAND containing borgmatic mount --mount-point /mnt/test) or at least to the borgmatic action (eg. BORGMATIC_ACTION containing mount) in the hooks' commands would be great. The main trick here is that there can be multiple actions at once. There _is_ a single borgmatic command-line, of course, but it can contain multiple actions and their flags, and I'm not sure I'd expect a shell command to parse that. `BORGMATIC_ACTION` is interesting, and it'd be easier to parse for a shell command, although it'd of course have to be `BORGMATIC_ACTIONS` to accommodate multiple. And instead of an environment variable, the convention is to use [variable interpolation](https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/#variable-interpolation) for passing borgmatic information to hook commands. Okay, so next steps. My inclination is to solve this requirement within borgmatic (or at least in borgmatic with some user configuration) rather than requiring ever more complex shell scripting. So at the risk of blowing up scope ... Imagine if all the `before_*`/`after_*` hooks were replaced with something like this (inspired by #463 comments among others): ```yaml hooks: commands: - exclude_actions: [mount] before: - test -d "/mnt/network_folder" || mkdir --parents "/mnt/network_folder" - mountpoint "/mnt/network_folder" || mount -t nfs "host.example.com:/export/backups" "/mnt/network_folder" after: - mountpoint "/mnt/network_folder" && umount "/mnt/network_folder" ``` This sort of structure could also replace any of the other command hooks. For instance, this might be a replacement for `before_check`: ```yaml hooks: commands: - include_actions: [check] before: - echo "About to run a check" ``` Or even combining hooks: ```yaml hooks: commands: - include_actions: [prune, compact] before: - echo "Pruning and/or compacting" ``` (One thing to work out for this last example is if it runs twice, once before `prune` and once before `compact`—or if it just runs once before all actions if any of them are `compact` or `prune`.)

C-Duv commented

2023-03-22 17:52:13 +00:00

My attempt to check for a borgfs mount before trying to unmount my remote folder:

hooks:
    before_actions:
        - test -d "/mnt/network_folder" || mkdir --parents "/mnt/network_folder"
        - mountpoint "/mnt/network_folder" > /dev/null || mount -t nfs "host.example.com:/export/backups" "/mnt/network_folder"

    after_actions:
        - 'findmnt --type fuse --source borgfs || { mountpoint "/mnt/network_folder" && umount "/mnt/network_folder" ; } && { echo "I am after_actions: There is still a borgfs mount." ; }'

It should have worked but did not: When I run borgmatic umount, when the after_actions hook executes there is still a borgfs mount (according to findmnt --type fuse --source borgfs) :-/

It looks like borgmatic calls borg umount (which effectively unmounts the Borg repository) after executing the after_actions hooks.

In the end, I've found an ugly way of "scheduling" an unmount after borgmatic ends with:

hooks:
    after_actions:
        - "nohup /bin/sh -c 'sleep 5s ; findmnt --type fuse --source borgfs || { mountpoint \"/mnt/network_folder\" && umount \"/mnt/network_folder\" ; }' > /dev/null 2>&1 &"

Spoiler: it works ;)

My attempt to check for a borgfs mount before trying to unmount my remote folder: ```YAML hooks: before_actions: - test -d "/mnt/network_folder" || mkdir --parents "/mnt/network_folder" - mountpoint "/mnt/network_folder" > /dev/null || mount -t nfs "host.example.com:/export/backups" "/mnt/network_folder" after_actions: - 'findmnt --type fuse --source borgfs || { mountpoint "/mnt/network_folder" && umount "/mnt/network_folder" ; } && { echo "I am after_actions: There is still a borgfs mount." ; }' ``` It *should* have worked but did not: When I run `borgmatic umount`, when the `after_actions` hook executes there is still a *borgfs* mount (according to `findmnt --type fuse --source borgfs`) :-/ It looks like borgmatic calls `borg umount` (which effectively unmounts the Borg repository) after executing the `after_actions` hooks. In the end, I've found an **ugly** way of "scheduling" an unmount after borgmatic ends with: ```YAML hooks: after_actions: - "nohup /bin/sh -c 'sleep 5s ; findmnt --type fuse --source borgfs || { mountpoint \"/mnt/network_folder\" && umount \"/mnt/network_folder\" ; }' > /dev/null 2>&1 &" ``` Spoiler: it works ;)

C-Duv commented

2023-03-22 18:04:41 +00:00

(Saw your reply after I finished my tries and error)

Thanks for taking the time to file this! It's helpful to see exactly the shell commands you're running from hooks.

It works, except for the borgmatic mount command where the after_actions tries to unmount the remote folder borgmatic just accessed to open the Borg repository for borgfs.

Got it. Out of curiosity, is the after_actions unmount blowing up because the repository is still in use by the borg mount? Or does it successfully unmount the repository and therefore the Borg-mounted directory stops working?

Originally It was failing for borgmatic mount because the repository was intentionally still in use. I said it to highlight the limit of my first idea (where I thought I could wrap borgmatic actions with simple mount and umount).

I'm not sure that actually solves your issue, […]. Although I suppose borgmatic could be smart enough not to run the unmountcmd after any mount action.

I was indeed counting on that: if I tell borgmatic how to mount/unmount the path of a repository, I trust him to do the mounting and unmounting when needed.

Perform more checks during after_actions: check any borgfs mount before umounting.

If I opt for performing more checks during after_actions, having access via an environment variable to the complete executed borgmatic CLI command (e.g. BORGMATIC_COMMAND containing borgmatic mount --mount-point /mnt/test) or at least to the borgmatic action (eg. BORGMATIC_ACTION containing mount) in the hooks' commands would be great.

The main trick here is that there can be multiple actions at once. There is a single borgmatic command-line, of course, but it can contain multiple actions and their flags, and I'm not sure I'd expect a shell command to parse that. BORGMATIC_ACTION is interesting, and it'd be easier to parse for a shell command, although it'd of course have to be BORGMATIC_ACTIONS to accommodate multiple. And instead of an environment variable, the convention is to use variable interpolation for passing borgmatic information to hook commands.

I must admit I overlooked the documentation and failed to notice the variable interpolation which could do just fine. I guess we could add a action_name (or similar) variable?

Okay, so next steps. My inclination is to solve this requirement within borgmatic (or at least in borgmatic with some user configuration) rather than requiring ever more complex shell scripting. So at the risk of blowing up scope ... Imagine if all the before_*/after_* hooks were replaced with something like this (inspired by #463 comments among others):

Theses a great solutions! But I guess they require a lot of work.

I have no idea on how many people are having similar issues.

(Saw your reply after I finished my tries and error) > Thanks for taking the time to file this! It's helpful to see exactly the shell commands you're running from hooks. > > > It works, except for the borgmatic mount command where the after_actions tries to unmount the remote folder borgmatic just accessed to open the Borg repository for borgfs. > > Got it. Out of curiosity, is the `after_actions` unmount blowing up because the repository is still in use by the `borg mount`? Or does it successfully unmount the repository and therefore the Borg-mounted directory stops working? Originally It was failing for `borgmatic mount` because the repository was intentionally still in use. I said it to highlight the limit of my first idea (where I thought I could wrap borgmatic actions with simple `mount` and `umount`). > I'm not sure that actually solves your issue, […]. Although **I suppose borgmatic could be smart enough** not to run the `unmountcmd` after any `mount` action. I was indeed counting on that: if I tell borgmatic how to mount/unmount the path of a repository, I trust him to do the mounting and unmounting when needed. > > > Perform more checks during after_actions: check any borgfs mount before umounting. > > > If I opt for performing more checks during after_actions, having access via an environment variable to the complete executed borgmatic CLI command (e.g. BORGMATIC_COMMAND containing borgmatic mount --mount-point /mnt/test) or at least to the borgmatic action (eg. BORGMATIC_ACTION containing mount) in the hooks' commands would be great. > > The main trick here is that there can be multiple actions at once. There _is_ a single borgmatic command-line, of course, but it can contain multiple actions and their flags, and I'm not sure I'd expect a shell command to parse that. `BORGMATIC_ACTION` is interesting, and it'd be easier to parse for a shell command, although it'd of course have to be `BORGMATIC_ACTIONS` to accommodate multiple. And instead of an environment variable, the convention is to use [variable interpolation](https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/#variable-interpolation) for passing borgmatic information to hook commands. I must admit I overlooked the documentation and failed to notice the [variable interpolation](https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/#variable-interpolation) which could do just fine. I guess we could add a `action_name` (or similar) variable? > Okay, so next steps. My inclination is to solve this requirement within borgmatic (or at least in borgmatic with some user configuration) rather than requiring ever more complex shell scripting. So at the risk of blowing up scope ... Imagine if all the `before_*`/`after_*` hooks were replaced with something like this (inspired by #463 comments among others): Theses a great solutions! But I guess they require a lot of work. I have no idea on how many people are having similar issues.

witten commented

2023-03-23 18:33:22 +00:00

One reason I'm a little resistant to the mount/unmount command options is they'd only solve a subset of use cases and therefore could be confusing. For instance, there are use cases like yours around mounting repository directories. But there are also legitimate use cases around mounting source directories! (Or even, I suppose, mounting directories containing PostgreSQL client certificates.) I guess introducing options for each would be a possibility.

Or.. just adding an action_names interpolated variable (say, a comma-separated list of actions) could be a place to start. And I think it would be a fairly surgical change, unlike the giant hook refactor I was proposing above. (Although ultimately that may be a good idea.)

Thoughts?

One reason I'm a little resistant to the mount/unmount command options is they'd only solve a subset of use cases and therefore could be confusing. For instance, there are use cases like yours around mounting repository directories. But there are also legitimate use cases around mounting source directories! (Or even, I suppose, mounting directories containing PostgreSQL client certificates.) I guess introducing options for each would be a possibility. Or.. just adding an `action_names` interpolated variable (say, a comma-separated list of actions) could be a place to start. And I think it would be a fairly surgical change, unlike the giant hook refactor I was proposing above. (Although ultimately that may be a good idea.) Thoughts?

C-Duv commented

2023-03-24 01:04:08 +00:00

For a start the interpolated variable is fine start, it gives the user all the card to shape it's hooks commands for it's own use case without adding complexity to borgmatic.

Something like this?:

diff --git a/borgmatic/commands/borgmatic.py b/borgmatic/commands/borgmatic.py
index fbea260..62ae5f7 100644
--- a/borgmatic/commands/borgmatic.py
+++ b/borgmatic/commands/borgmatic.py
@@ -270,8 +270,12 @@ def run_actions(
         'repository': repository_path,
         # Deprecated: For backwards compatibility with borgmatic < 1.6.0.
         'repositories': ','.join(location['repositories']),
+        'action_names': [],
     }

+    for (action_name, _) in arguments.items():
+        hook_context['action_names'].append(action_name)
+
     command.execute_hook(
         hooks.get('before_actions'),
         hooks.get('umask'),
diff --git a/borgmatic/hooks/command.py b/borgmatic/hooks/command.py
index 756f877..8496243 100644
--- a/borgmatic/hooks/command.py
+++ b/borgmatic/hooks/command.py
@@ -16,7 +16,10 @@ def interpolate_context(config_filename, hook_description, command, context):
     names/values, interpolate the values by "{name}" into the command and return the result.
     '''
     for name, value in context.items():
-        command = command.replace('{%s}' % name, str(value))
+        if isinstance(value, list):
+            command = command.replace('{%s}' % name, ",".join(value))
+        else:
+            command = command.replace('{%s}' % name, str(value))

     for unsupported_variable in re.findall(r'{\w+}', command):
         logger.warning(

For a start the interpolated variable is fine start, it gives the user all the card to shape it's hooks commands for it's own use case without adding complexity to borgmatic. Something like this?: ```diff diff --git a/borgmatic/commands/borgmatic.py b/borgmatic/commands/borgmatic.py index fbea260..62ae5f7 100644 --- a/borgmatic/commands/borgmatic.py +++ b/borgmatic/commands/borgmatic.py @@ -270,8 +270,12 @@ def run_actions( 'repository': repository_path, # Deprecated: For backwards compatibility with borgmatic < 1.6.0. 'repositories': ','.join(location['repositories']), + 'action_names': [], } + for (action_name, _) in arguments.items(): + hook_context['action_names'].append(action_name) + command.execute_hook( hooks.get('before_actions'), hooks.get('umask'), diff --git a/borgmatic/hooks/command.py b/borgmatic/hooks/command.py index 756f877..8496243 100644 --- a/borgmatic/hooks/command.py +++ b/borgmatic/hooks/command.py @@ -16,7 +16,10 @@ def interpolate_context(config_filename, hook_description, command, context): names/values, interpolate the values by "{name}" into the command and return the result. ''' for name, value in context.items(): - command = command.replace('{%s}' % name, str(value)) + if isinstance(value, list): + command = command.replace('{%s}' % name, ",".join(value)) + else: + command = command.replace('{%s}' % name, str(value)) for unsupported_variable in re.findall(r'{\w+}', command): logger.warning( ```

witten commented

2023-03-24 22:24:08 +00:00

That totally works, but I think it could be simplified even further! Here's an example:

diff --git a/borgmatic/commands/borgmatic.py b/borgmatic/commands/borgmatic.py
index 73bde94..d15de05 100644
--- a/borgmatic/commands/borgmatic.py
+++ b/borgmatic/commands/borgmatic.py
@@ -264,6 +264,7 @@ def run_actions(
     dry_run_label = ' (dry run; not making any changes)' if global_arguments.dry_run else ''
     hook_context = {
         'repository': repository_path,
+        'actions': ','.join(key for key in arguments.keys() if key != 'global'),
         # Deprecated: For backwards compatibility with borgmatic < 1.6.0.
         'repositories': ','.join(location['repositories']),
     }

That totally works, but I think it could be simplified even further! Here's an example: ```diff diff --git a/borgmatic/commands/borgmatic.py b/borgmatic/commands/borgmatic.py index 73bde94..d15de05 100644 --- a/borgmatic/commands/borgmatic.py +++ b/borgmatic/commands/borgmatic.py @@ -264,6 +264,7 @@ def run_actions( dry_run_label = ' (dry run; not making any changes)' if global_arguments.dry_run else '' hook_context = { 'repository': repository_path, + 'actions': ','.join(key for key in arguments.keys() if key != 'global'), # Deprecated: For backwards compatibility with borgmatic < 1.6.0. 'repositories': ','.join(location['repositories']), } ```

C-Duv commented

2023-03-25 11:07:19 +00:00

True 👍

I wanted to keep the action_names as a list as long as possible ;) (and my Python skills failed me for the key for key in arguments.keys() part too :P)

True 👍 I wanted to keep the `action_names` as a list as long as possible ;) (and my Python skills failed me for the `key for key in arguments.keys()` part too :P)

witten commented

2023-03-25 20:59:04 +00:00

Gotcha, that makes sense! Let me know if you want to take a stab at a PR (ideally with tests) or if you'd prefer me to take care of this. I'm totally fine either way.