Best way to detect unavailable devices. #284
Labels
No Label
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
No Milestone
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: borgmatic-collective/borgmatic#284
Loading…
Reference in New Issue
No description provided.
Delete Branch "%!s(<nil>)"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What I'm trying to do and why
I have some backup configs where source folders or target repos are located on external devices.
I like to just call
borgmatic
to execute borgmatic for all configs in~/.config/borgmatic.d/
.Currently I use hooks to check if devices are available and only run those configurations where necessary mount points are available.
I would like to know if there is maybe a better way to do this.
Maybe it would be an interesting idea to build this directly into borgmatic.
Are there some special return values to signal expected error and skip a config without error messages (for the device not mounted case)
Environment
borgmatic version: 1.4.21
borgmatic installation method: pip
Borg version: 1.1.10
Python version: 3.5.2
Seems reasonable.. Could you say a little more about the use case though? Is the idea that you run borgmatic regularly, say to do cloud backups.. But then you want it to opportunistically backup to external devices as well, devices that may or may not be plugged in / mounted during any given borgmatic run? And you want borgmatic not to error if the external devices are absent? Something like that?
Out of curiosity, how do you make a borgmatic hook influence whether certain borgmatic configurations get run or not?
Off the top of my head, not right now, but I think I need to understand your use case a little better.
Not currently, no. Although that could certainly be one way to support this sort of thing. (There is some code that treats a Borg-specific warning as either a borgmatic warning or error in particular situations, but that's it.)
Do you get a specific shell exit code when the source directory or repository is not available / when the external device is absent?
My use case is not server backup but backup of personal used laptop.
My goal is to have a press a single button backup solution (once configured), so I don’t need to think about what goes where later anymore.
I backup to 2 external USB disks as I have to much data for the cloud (it would be too expensive).
One of these disks is stored at home and one is stored at work and I swap them regularly. They are set up with independent borg repos, so they are not binary identical copies, but as I swap them regularly I have pretty much the same recent data on both for redundancy in case of burglar, fire at one of the places etc.
One backup is for my entire home (with exceptions using
.nobackup
etc.).As my laptop disk isn’t big enough for my complete collection of music, raw image files and movies I store data like that on another USB drive I call Archive.
I use this USB drive as a single location of organization of the data, but of course I don’t want to have this data only on a single location for backup, so I also backup this disk using borgmatic if it is mounted.
So what I would like to have ideally is just running borgmatic, trying all the configs in e.g.
~/.config/borgmatic.d/*
, checking if they can get applied (i.e. checking if source and destination exist) and run all configs that can get applied.There are some other details that might be interesting in this scenario: e.g.
borgmatic list
could output a short heading line displaying the path of the current repo it list instead of displaying everything in a long list (same forinfo
etc.)I just checked what borgmatic is doing if the hook returns a non zero return code. Currently it errors out and stops execution for the currently applied config. I thought that’s expected behaviour.
Yes, but the exact return code can be set by the user in the hook script. So it doesn’t matter what my current command returns. I could return everything borgmatic needs. E.g. for the command in my previous post to return 123:
Thanks for the background. That certainly helps me understand the use case.
Ah, yes. But that (currently) results in errors for those configurations that can't be run, e.g. due to a missing mount point. Ideally, what would borgmatic's behavior be for such configurations? Warnings? Silently succeeding?
FYI, the behavior with the current code is:
Anyway, here is a riff on your idea of a special return value.. There could be a new hook called
should_backup
or something. It can run one or more commands, and if any of them return a non-zero exit status, the backup gets skipped without error. Example based on your example:Let me know your thoughts. That wouldn't be as built-in as borgmatic itself probing individual source directories or repositories, but it might suit your use case as a place to start.
Side note: I just checked in a change to always show the repository name heading even at verbosity 0. (Previously it showed only at verbosity 1 and 2).
I think this should be on INFO level. Since it’s expected behaviour it should not be warning or error, but the user should get notified, because it might be unintentional (e.g. because the user just forgot to mount/decrypt a volume while the disk is pluged in or something like that)
I noticed that while testing my configuration. This makes sense of course if there are multiple source directories given and some of them are empty but you still want to keep the directory structure in the backup.
In case of a single root source folder, which is a mount point this makes no sense, hence the additional test using the hook.
I think using hooks is a good solution. No need to integrate this as a built-in.
I’m not sure what would be the best place here. If I think about Subversion or GIT hook, I think hooks should exist not on a semantic (like
should_backup
) level but on a per step command level (likebefore_prune
) and signal what to do with return codes (e.g. 0 – continue, 1 – exit with error, 2 – skip etc.). This would be most generic. Additionally it would be helpful to have somebefore_everything_in_this_config
andafter_…
hooks.Btw. I didn’t get from the documentation if
applies to all configs in a single borgmatic run or just to everything in THIS config (but I didn’t tested). Also I’m not sure if making this depend on
create
would fit all useful cases. Maybe the idea I outlined above is more general.Thanks for all your work and kind further inquiry on all the issues.
Okay, I've taken a stab at this based on your feedback. The summary is that you need only return the special exit code of 75 from a
before_backup
(or other) hook to indicate to borgmatic that it's a "soft failure" situation. Documentation on the feature is online, although note that the feature itself is not released until the next borgmatic release.Please let me know if you have any feedback on the approach or design or docs. It's not too late to change stuff.
I have this milestone for additional action hooks: https://projects.torsion.org/witten/borgmatic/milestone/4 ... Please check out the remaining open ticket in there, and then file tickets for any other hooks you'd like to see. Don't forget use cases!
I've just updated the comment to read as follows. Please let me know if this doesn't clear it up!
There are also these docs with more details: https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/
Hi, thanks. You are really fast implementing suggestions.
Docs are fine.
Sidenote: Why did you choose 75? Is their any special reason for that or just because? I googled if there is some standard and found out there is no real standard to this except some attempts to define them for C++ programs in
/usr/include/sysexits.h
. According to this 75 isso this seams to be reasonable.
This is very clear now.
That's exactly why I chose 75! Good sleuthing.
Released in borgmatic 1.5.0!