Borgmatic fails if a single repository is not availiable #144

Closed
opened 2019-02-25 10:36:29 +00:00 by ThomasBlt · 7 comments

(Note: I'm troubleshooting my installation so one need to confirm if it's a bug of something specific to me)

Using borgmatic 1.2.15 + borg 1.1.8 with borgmatic configured to sync with multiple repositories (let's say A, B and C).
If repository B is offline, borgmatic successfully backup to A, and then fail+stop on B.
It doesn't recover of it's failure on B and thus backup is not sent to C.

Is this an expected behavior ?

(Note: I'm troubleshooting my installation so one need to confirm if it's a bug of something specific to me) Using borgmatic 1.2.15 + borg 1.1.8 with borgmatic configured to sync with multiple repositories (let's say A, B and C). If repository B is offline, borgmatic successfully backup to A, and then fail+stop on B. It doesn't recover of it's failure on B and thus backup is not sent to C. Is this an expected behavior ?
Owner

Whether it's expected behavior is perhaps a matter of opinion, but the behavior you're seeing is certainly in line with how the code is currently written: At the configuration file level, if one whole configuration file errors, it'll temporarily suppress that error and move on to the next configuration file. But borgmatic does not currently apply that same level of resilience to individual failing repositories within the same configuration file.

It sounds like you'd like borgmatic to plow on through if one repository fails, perhaps summarizing and reporting errors at the end?

Could you say a little more about the rationale behind that? I can imagine it's something like: You backup the same data to multiple repositories in different locations for redundancy, and you don't want a network or other availability error with one repository to hinder backups to another repository. But it would be great to hear your actual reasoning.

Thanks!

Whether it's expected behavior is perhaps a matter of opinion, but the behavior you're seeing is certainly in line with how the code is currently written: At the configuration file level, if one whole configuration file errors, it'll temporarily suppress that error and move on to the next configuration file. But borgmatic does not currently apply that same level of resilience to individual failing repositories within the same configuration file. It sounds like you'd like borgmatic to plow on through if one repository fails, perhaps summarizing and reporting errors at the end? Could you say a little more about the rationale behind that? I can imagine it's something like: You backup the same data to multiple repositories in different locations for redundancy, and you don't want a network or other availability error with one repository to hinder backups to another repository. But it would be great to hear your actual reasoning. Thanks!
Author

OK, I get your point and you've figured out my situation.

I'm a newbie with borgmatic (but I'm a developer). I'm aware that I can use multiple config but I didn't see a reason to do so in my scenario as I want a big backup of all my server's essential files. A single config seems to do the trick for me (IMHO).

To elaborate a little, I'll explain the full story below (sorry if it get long and boring) but in the mean time could your explain your opinion / your idea behind your implementation of the multi-repo feature ? (just to know)

full story:
I recently experienced a failure of my RAID 1 NAS : both disk died within 3 days (call it bad luck).
Therefore I decided to go for encrypted external backups with borg.
Borg is nice but scripting everything together is hard so I completed my tooling with borgmatic.

Borgmatic offer nice features like the ability to duplicate a "single" backup task to multiple repositories.

I want to have a local backup because it will always be faster to recover from it.
But if the house burns ... I want a secondary backup at a friend's place.
Yet my friend is not a geek and I tends to experience "service outages" that can last for weeks (which is the case currently, 12 days and counting).
So I added a third repo on a cloud server, in third position of my yaml.

Yet, as my friend's server is still offline, borgmatic stopped after my local repo and so I'm lacking the external backup I'm expecting

(As a workaround, I've commented the offending repo and moved this rather unstable one to the last position.)

Regards

OK, I get your point and you've figured out my situation. I'm a newbie with borgmatic (but I'm a developer). I'm aware that I can use multiple config but I didn't see a reason to do so in my scenario as I want a big backup of all my server's essential files. A single config seems to do the trick for me (IMHO). To elaborate a little, I'll explain the full story below (sorry if it get long and boring) but in the mean time could your explain your opinion / your idea behind your implementation of the multi-repo feature ? (just to know) full story: I recently experienced a failure of my RAID 1 NAS : both disk died within 3 days (call it bad luck). Therefore I decided to go for encrypted external backups with borg. Borg is nice but scripting everything together is hard so I completed my tooling with borgmatic. Borgmatic offer nice features like the ability to duplicate a "single" backup task to multiple repositories. I want to have a local backup because it will always be faster to recover from it. But if the house burns ... I want a secondary backup at a friend's place. Yet my friend is not a geek and I tends to experience "service outages" that can last for weeks (which is the case currently, 12 days and counting). So I added a third repo on a cloud server, in third position of my yaml. Yet, as my friend's server is still offline, borgmatic stopped after my local repo and so I'm lacking the external backup I'm expecting (As a workaround, I've commented the offending repo and moved this rather unstable one to the last position.) Regards
Owner

To elaborate a little, I’ll explain the full story below (sorry if it get long and boring) but in the mean time could your explain your opinion / your idea behind your implementation of the multi-repo feature ? (just to know)

Basically to enable exactly these sorts of use cases: Backup the same source directories to multiple different destinations for improved redundancy. For most cases, backups to each repository is equally reliable, so "failing fast" hasn't been a huge issue. Sounds like that's not the case here though!

Thanks for describing your use case. Having that context is actually super useful for understanding how the feature is used. I think it's reasonable to consider this a feature request: If one repository in a configuration file errors for any reason, then don't stop borgmatic from running through the remaining repositories.

> To elaborate a little, I’ll explain the full story below (sorry if it get long and boring) but in the mean time could your explain your opinion / your idea behind your implementation of the multi-repo feature ? (just to know) Basically to enable exactly these sorts of use cases: Backup the same source directories to multiple different destinations for improved redundancy. For most cases, backups to each repository is equally reliable, so "failing fast" hasn't been a huge issue. Sounds like that's not the case here though! Thanks for describing your use case. Having that context is actually super useful for understanding how the feature is used. I think it's reasonable to consider this a feature request: If one repository in a configuration file errors for any reason, then don't stop borgmatic from running through the remaining repositories.
witten added the
design finalized
label 2019-02-25 19:54:19 +00:00
Author

Thanks for developping this tool ! It is super usefull. Saved me a lot of time !

Sorry for not helping on this, I'm more a Java guy (I would probably mess things up) and my spare time is on some IoT projects these days.

Thanks for developping this tool ! It is super usefull. Saved me a lot of time ! Sorry for not helping on this, I'm more a Java guy (I would probably mess things up) and my spare time is on some IoT projects these days.
Owner

No worries.. Feature requests and bug reports are just as appreciated as pull requests! Helps me understand how actual users are using the software. And I'm glad to hear borgmatic has been useful for you.

No worries.. Feature requests and bug reports are just as appreciated as pull requests! Helps me understand how actual users are using the software. And I'm glad to hear borgmatic has been useful for you.
witten changed title from [To be confirmed] Borgmatic fails if a single repository is not availiable to Borgmatic fails if a single repository is not availiable 2019-03-05 17:11:40 +00:00
Owner

Done, at long last. Just released in borgmatic 1.3.22. Now, backup errors resulting from a single repository in a config file should not stop backups to other repositories in that same config file. And any errors are summarized at the end. Please give it a shot and let me know how it works out for you.

Thanks again for the suggestion!

Done, at long last. Just released in borgmatic 1.3.22. Now, backup errors resulting from a single repository in a config file should not stop backups to other repositories in that same config file. And any errors are summarized at the end. Please give it a shot and let me know how it works out for you. Thanks again for the suggestion!
Author

Neat bugfix, I'm updating right now
Should be running tonight

Neat bugfix, I'm updating right now Should be running tonight
Sign in to join this conversation.
No Milestone
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#144
No description provided.