Best way to detect unavailable devices. #284

Closed
opened 2020-01-20 17:27:48 +00:00 by codingforfun · 8 comments

What I'm trying to do and why

I have some backup configs where source folders or target repos are located on external devices.

I like to just call borgmatic to execute borgmatic for all configs in ~/.config/borgmatic.d/.

Currently I use hooks to check if devices are available and only run those configurations where necessary mount points are available.

hooks:
    before_backup:
        - echo "Checking devices are mounted."
        - findmnt -rno TARGET -M /media/<user>/source_mount > /dev/null
        - findmnt -rno TARGET -M /media/<user>/target_mount > /dev/null

I would like to know if there is maybe a better way to do this.

Maybe it would be an interesting idea to build this directly into borgmatic.

Are there some special return values to signal expected error and skip a config without error messages (for the device not mounted case)

Environment

borgmatic version: 1.4.21

borgmatic installation method: pip

Borg version: 1.1.10

Python version: 3.5.2

#### What I'm trying to do and why I have some backup configs where source folders or target repos are located on external devices. I like to just call `borgmatic` to execute borgmatic for all configs in `~/.config/borgmatic.d/`. Currently I use hooks to check if devices are available and only run those configurations where necessary mount points are available. ``` hooks: before_backup: - echo "Checking devices are mounted." - findmnt -rno TARGET -M /media/<user>/source_mount > /dev/null - findmnt -rno TARGET -M /media/<user>/target_mount > /dev/null ``` I would like to know if there is maybe a better way to do this. Maybe it would be an interesting idea to build this directly into borgmatic. Are there some special return values to signal expected error and skip a config without error messages (for the _device not mounted_ case) #### Environment **borgmatic version:** 1.4.21 **borgmatic installation method:** pip **Borg version:** 1.1.10 **Python version:** 3.5.2
Owner

Seems reasonable.. Could you say a little more about the use case though? Is the idea that you run borgmatic regularly, say to do cloud backups.. But then you want it to opportunistically backup to external devices as well, devices that may or may not be plugged in / mounted during any given borgmatic run? And you want borgmatic not to error if the external devices are absent? Something like that?

Currently I use hooks to check if devices are available and only run those configurations where necessary mount points are available.

Out of curiosity, how do you make a borgmatic hook influence whether certain borgmatic configurations get run or not?

I would like to know if there is maybe a better way to do this.

Off the top of my head, not right now, but I think I need to understand your use case a little better.

Are there some special return values to signal expected error and skip a config without error messages (for the device not mounted case)

Not currently, no. Although that could certainly be one way to support this sort of thing. (There is some code that treats a Borg-specific warning as either a borgmatic warning or error in particular situations, but that's it.)

Do you get a specific shell exit code when the source directory or repository is not available / when the external device is absent?

Seems reasonable.. Could you say a little more about the use case though? Is the idea that you run borgmatic regularly, say to do cloud backups.. But then you want it to opportunistically backup to external devices as well, devices that may or may not be plugged in / mounted during any given borgmatic run? And you want borgmatic not to error if the external devices are absent? Something like that? > Currently I use hooks to check if devices are available and only run those configurations where necessary mount points are available. Out of curiosity, how do you make a borgmatic hook influence whether certain borgmatic configurations get run or not? > I would like to know if there is maybe a better way to do this. Off the top of my head, not right now, but I think I need to understand your use case a little better. > Are there some special return values to signal expected error and skip a config without error messages (for the device not mounted case) Not currently, no. Although that could certainly be one way to support this sort of thing. (There is some code that treats a Borg-specific warning as either a borgmatic warning or error in particular situations, but that's it.) Do you get a specific shell exit code when the source directory or repository is not available / when the external device is absent?
Author

Could you say a little more about the use case though?

My use case is not server backup but backup of personal used laptop.
My goal is to have a press a single button backup solution (once configured), so I don’t need to think about what goes where later anymore.

I backup to 2 external USB disks as I have to much data for the cloud (it would be too expensive).

One of these disks is stored at home and one is stored at work and I swap them regularly. They are set up with independent borg repos, so they are not binary identical copies, but as I swap them regularly I have pretty much the same recent data on both for redundancy in case of burglar, fire at one of the places etc.

One backup is for my entire home (with exceptions using .nobackup etc.).

As my laptop disk isn’t big enough for my complete collection of music, raw image files and movies I store data like that on another USB drive I call Archive.
I use this USB drive as a single location of organization of the data, but of course I don’t want to have this data only on a single location for backup, so I also backup this disk using borgmatic if it is mounted.

So what I would like to have ideally is just running borgmatic, trying all the configs in e.g. ~/.config/borgmatic.d/*, checking if they can get applied (i.e. checking if source and destination exist) and run all configs that can get applied.

There are some other details that might be interesting in this scenario: e.g. borgmatic list could output a short heading line displaying the path of the current repo it list instead of displaying everything in a long list (same for info etc.)

Out of curiosity, how do you make a borgmatic hook influence whether certain borgmatic configurations get run or not?

I just checked what borgmatic is doing if the hook returns a non zero return code. Currently it errors out and stops execution for the currently applied config. I thought that’s expected behaviour.

Do you get a specific shell exit code when the source directory or repository is not available / when the external device is absent?

Yes, but the exact return code can be set by the user in the hook script. So it doesn’t matter what my current command returns. I could return everything borgmatic needs. E.g. for the command in my previous post to return 123:

( findmnt -rno TARGET -M /media/<user>/target_mount > /dev/null || exit 123 )
> Could you say a little more about the use case though? My use case is not server backup but backup of personal used laptop. My goal is to have a _press a single button_ backup solution (once configured), so I don’t need to think about what goes where later anymore. I backup to 2 external USB disks as I have to much data for the cloud (it would be too expensive). One of these disks is stored at home and one is stored at work and I swap them regularly. They are set up with independent borg repos, so they are not binary identical copies, but as I swap them regularly I have pretty much the same recent data on both for redundancy in case of burglar, fire at one of the places etc. One backup is for my entire home (with exceptions using `.nobackup` etc.). As my laptop disk isn’t big enough for my complete collection of music, raw image files and movies I store data like that on another USB drive I call _Archive_. I use this USB drive as a single location of organization of the data, but of course I don’t want to have this data only on a single location for backup, so I also backup this disk using borgmatic if it is mounted. So what I would like to have ideally is just running borgmatic, trying all the configs in e.g. `~/.config/borgmatic.d/*`, checking if they can get applied (i.e. checking if source and destination exist) and run all configs that can get applied. There are some other details that might be interesting in this scenario: e.g. `borgmatic list` could output a short heading line displaying the path of the current repo it list instead of displaying everything in a long list (same for `info` etc.) > Out of curiosity, how do you make a borgmatic hook influence whether certain borgmatic configurations get run or not? I just checked what borgmatic is doing if the hook returns a non zero return code. Currently it errors out and stops execution for the currently applied config. I thought that’s expected behaviour. > Do you get a specific shell exit code when the source directory or repository is not available / when the external device is absent? Yes, but the exact return code can be set by the user in the hook script. So it doesn’t matter what my current command returns. I could return everything borgmatic needs. E.g. for the command in my previous post to return 123: ``` ( findmnt -rno TARGET -M /media/<user>/target_mount > /dev/null || exit 123 ) ```
Owner

Thanks for the background. That certainly helps me understand the use case.

I just checked what borgmatic is doing if the hook returns a non zero return code. Currently it errors out and stops execution for the currently applied config. I thought that’s expected behaviour.

Ah, yes. But that (currently) results in errors for those configurations that can't be run, e.g. due to a missing mount point. Ideally, what would borgmatic's behavior be for such configurations? Warnings? Silently succeeding?

FYI, the behavior with the current code is:

  1. If a source directory is missing, Borg only gives a warning, which borgmatic does not error on.
  2. If a source directory is empty, Borg + borgmatic happily back it up.
  3. If a repository is missing or empty, Borg + borgmatic error.

Anyway, here is a riff on your idea of a special return value.. There could be a new hook called should_backup or something. It can run one or more commands, and if any of them return a non-zero exit status, the backup gets skipped without error. Example based on your example:

hooks:
  should_backup:
    - findmnt -rno TARGET -M /media/<user>/source_mount > /dev/null
    - findmnt -rno TARGET -M /media/<user>/target_mount > /dev/null

Let me know your thoughts. That wouldn't be as built-in as borgmatic itself probing individual source directories or repositories, but it might suit your use case as a place to start.

There are some other details that might be interesting in this scenario: e.g. borgmatic list could output a short heading line displaying the path of the current repo it list instead of displaying everything in a long list (same for info etc.)

Side note: I just checked in a change to always show the repository name heading even at verbosity 0. (Previously it showed only at verbosity 1 and 2).

Thanks for the background. That certainly helps me understand the use case. > I just checked what borgmatic is doing if the hook returns a non zero return code. Currently it errors out and stops execution for the currently applied config. I thought that’s expected behaviour. Ah, yes. But that (currently) results in errors for those configurations that can't be run, e.g. due to a missing mount point. Ideally, what would borgmatic's behavior be for such configurations? Warnings? Silently succeeding? FYI, the behavior with the current code is: 1. If a source directory is missing, Borg only gives a warning, which borgmatic does not error on. 2. If a source directory is empty, Borg + borgmatic happily back it up. 3. If a repository is missing *or* empty, Borg + borgmatic error. Anyway, here is a riff on your idea of a special return value.. There could be a new hook called `should_backup` or something. It can run one or more commands, and if any of them return a non-zero exit status, the backup gets skipped without error. Example based on your example: ``` hooks: should_backup: - findmnt -rno TARGET -M /media/<user>/source_mount > /dev/null - findmnt -rno TARGET -M /media/<user>/target_mount > /dev/null ``` Let me know your thoughts. That wouldn't be as built-in as borgmatic itself probing individual source directories or repositories, but it might suit your use case as a place to start. > There are some other details that might be interesting in this scenario: e.g. borgmatic list could output a short heading line displaying the path of the current repo it list instead of displaying everything in a long list (same for info etc.) Side note: I just checked in a change to always show the repository name heading even at verbosity 0. (Previously it showed only at verbosity 1 and 2).
Author

… what would borgmatic’s behavior be for such configurations? Warnings? Silently succeeding?

I think this should be on INFO level. Since it’s expected behaviour it should not be warning or error, but the user should get notified, because it might be unintentional (e.g. because the user just forgot to mount/decrypt a volume while the disk is pluged in or something like that)

If a source directory is missing, Borg only gives a warning, which borgmatic does not error on.
If a source directory is empty, Borg + borgmatic happily back it up.

I noticed that while testing my configuration. This makes sense of course if there are multiple source directories given and some of them are empty but you still want to keep the directory structure in the backup.

In case of a single root source folder, which is a mount point this makes no sense, hence the additional test using the hook.

That wouldn’t be as built-in as borgmatic itself probing individual source directories or repositories, but it might suit your use case as a place to start.

I think using hooks is a good solution. No need to integrate this as a built-in.

There could be a new hook called should_backup

I’m not sure what would be the best place here. If I think about Subversion or GIT hook, I think hooks should exist not on a semantic (like should_backup) level but on a per step command level (like before_prune) and signal what to do with return codes (e.g. 0 – continue, 1 – exit with error, 2 – skip etc.). This would be most generic. Additionally it would be helpful to have some before_everything_in_this_config and after_… hooks.

Btw. I didn’t get from the documentation if

    # List of one or more shell commands or scripts to execute before running all
    # actions (if one of them is "create"), run once before all configuration files.
    # before_everything:
        # - echo "Starting actions."

applies to all configs in a single borgmatic run or just to everything in THIS config (but I didn’t tested). Also I’m not sure if making this depend on create would fit all useful cases. Maybe the idea I outlined above is more general.

Thanks for all your work and kind further inquiry on all the issues.

> … what would borgmatic’s behavior be for such configurations? Warnings? Silently succeeding? I think this should be on INFO level. Since it’s expected behaviour it should not be warning or error, but the user should get notified, because it might be unintentional (e.g. because the user just forgot to mount/decrypt a volume while the disk is pluged in or something like that) > If a source directory is missing, Borg only gives a warning, which borgmatic does not error on. > If a source directory is empty, Borg + borgmatic happily back it up. I noticed that while testing my configuration. This makes sense of course if there are multiple source directories given and some of them are empty but you still want to keep the directory structure in the backup. In case of a single root source folder, which is a mount point this makes no sense, hence the additional test using the hook. > That wouldn’t be as built-in as borgmatic itself probing individual source directories or repositories, but it might suit your use case as a place to start. I think using hooks is a good solution. No need to integrate this as a built-in. > There could be a new hook called `should_backup` … I’m not sure what would be the best place here. If I think about Subversion or GIT hook, I think hooks should exist not on a semantic (like `should_backup`) level but on a per step command level (like `before_prune`) and signal what to do with return codes (e.g. 0 – continue, 1 – exit with error, 2 – skip etc.). This would be most generic. Additionally it would be helpful to have some `before_everything_in_this_config` and `after_…` hooks. Btw. I didn’t get from the documentation if ``` # List of one or more shell commands or scripts to execute before running all # actions (if one of them is "create"), run once before all configuration files. # before_everything: # - echo "Starting actions." ``` applies to all configs in a single borgmatic run or just to everything in THIS config (but I didn’t tested). Also I’m not sure if making this depend on `create` would fit all useful cases. Maybe the idea I outlined above is more general. Thanks for all your work and kind further inquiry on all the issues.
Owner

Okay, I've taken a stab at this based on your feedback. The summary is that you need only return the special exit code of 75 from a before_backup (or other) hook to indicate to borgmatic that it's a "soft failure" situation. Documentation on the feature is online, although note that the feature itself is not released until the next borgmatic release.

Please let me know if you have any feedback on the approach or design or docs. It's not too late to change stuff.

I think hooks should exist not on a semantic (like should_backup) level but on a per step command level (like before_prune) ... Additionally it would be helpful to have some before_everything_in_this_config and after_… hooks.

I have this milestone for additional action hooks: https://projects.torsion.org/witten/borgmatic/milestone/4 ... Please check out the remaining open ticket in there, and then file tickets for any other hooks you'd like to see. Don't forget use cases!

Btw. I didn’t get from the documentation if ... applies to all configs in a single borgmatic run or just to everything in THIS config (but I didn’t tested). Also I’m not sure if making this depend on create would fit all useful cases. Maybe the idea I outlined above is more general.

I've just updated the comment to read as follows. Please let me know if this doesn't clear it up!

List of one or more shell commands or scripts to execute before running all actions (if one of them is "create"). These are collected from all configuration files and then run once before all of them (prior to all actions).

There are also these docs with more details: https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/

Okay, I've taken a stab at this based on your feedback. The summary is that you need only return the special exit code of 75 from a `before_backup` (or other) hook to indicate to borgmatic that it's a "soft failure" situation. [Documentation on the feature is online](https://torsion.org/borgmatic/docs/how-to/backup-to-a-removable-drive-or-an-intermittent-server/), although note that the feature itself is not released until the next borgmatic release. Please let me know if you have any feedback on the approach or design or docs. It's not too late to change stuff. > I think hooks should exist not on a semantic (like should_backup) level but on a per step command level (like before_prune) ... Additionally it would be helpful to have some before_everything_in_this_config and after_… hooks. I have this milestone for additional action hooks: https://projects.torsion.org/witten/borgmatic/milestone/4 ... Please check out the remaining open ticket in there, and then file tickets for any other hooks you'd like to see. Don't forget use cases! > Btw. I didn’t get from the documentation if ... applies to all configs in a single borgmatic run or just to everything in THIS config (but I didn’t tested). Also I’m not sure if making this depend on create would fit all useful cases. Maybe the idea I outlined above is more general. I've just updated the comment to read as follows. Please let me know if this doesn't clear it up! > List of one or more shell commands or scripts to execute before running all actions (if one of them is "create"). These are collected from all configuration files and then run once before all of them (prior to all actions). There are also these docs with more details: https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/
Author

Hi, thanks. You are really fast implementing suggestions.

Please let me know if you have any feedback on the approach or design or docs.

Docs are fine.

Sidenote: Why did you choose 75? Is their any special reason for that or just because? I googled if there is some standard and found out there is no real standard to this except some attempts to define them for C++ programs in /usr/include/sysexits.h. According to this 75 is

 *	EX_TEMPFAIL -- temporary failure, indicating something that
 *		is not really an error.  In sendmail, this means
 *		that a mailer (e.g.) could not create a connection,
 *		and the request should be reattempted later.

so this seams to be reasonable.

I’ve just updated the comment to read as follows. Please let me know if this doesn’t clear it up!

This is very clear now.

Hi, thanks. You are really fast implementing suggestions. > Please let me know if you have any feedback on the approach or design or docs. Docs are fine. Sidenote: Why did you choose 75? Is their any special reason for that or just because? I googled if there is some standard and found out there is no real standard to this except some attempts to define them for C++ programs in `/usr/include/sysexits.h`. According to this 75 is ``` * EX_TEMPFAIL -- temporary failure, indicating something that * is not really an error. In sendmail, this means * that a mailer (e.g.) could not create a connection, * and the request should be reattempted later. ``` so this seams to be reasonable. > I’ve just updated the comment to read as follows. Please let me know if this doesn’t clear it up! This is very clear now.
Owner

That's exactly why I chose 75! Good sleuthing.

That's exactly why I chose 75! Good sleuthing.
Owner

Released in borgmatic 1.5.0!

Released in borgmatic 1.5.0!
Sign in to join this conversation.
No Milestone
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#284
No description provided.