Running checks on a repo asynchronously from backup to another repo? #291

New Issue

jucor_ · 2020-02-01T14:02:57Z

jucor_ commented

2020-02-01 14:02:57 +00:00

What I'm trying to do and why

I have one borgmatic file with two remote SSH repositories. It does backup and check repositories consistency, on each repo. Can the consistency check of the first repository occurs in parallel with the backup to the second repository? This would make the whole process faster.

Currently, the steps are:

A/ Backup to SSH repo B1 - takes 1 hour, involves client and remote server S1
B/ Check consistency of repo B1 - takes 5 hours, involves only remote server S1
C/ Backup to SSH repo B2 - takes 1 hour, involves client and remote server S2
D/ Check consistency of repo B2 - takes 2 hours, involves only remote server S2 (which is faster than S1)

I takes 9 hours sequentially.
It would take only 6 hours if C and D are started without waiting for B to finish.
Because B's workload is server-side, C and D's speed would not be affected.

A poor-man's way would be to have separate borgmatic files for each step, and handle the parallelization externally.

borgmatic version: 1.4.22 on client.
Borg version: 1.1.7 on client, 1.1.10 on S1, 1.1.0 on S2
Python version: 3.6.2 on client
operating system and version: Don't think it matters, but Linux 4.4.59 on Synology NAS DS218+

Thanks for any idea!

#### What I'm trying to do and why I have one borgmatic file with two remote SSH repositories. It does backup and check repositories consistency, on each repo. *Can the consistency check of the first repository occurs in parallel with the backup to the second repository?* This would make the whole process faster. Currently, the steps are: - A/ Backup to SSH repo B1 - takes 1 hour, involves client and remote server S1 - B/ Check consistency of repo B1 - takes 5 hours, involves only remote server S1 - C/ Backup to SSH repo B2 - takes 1 hour, involves client and remote server S2 - D/ Check consistency of repo B2 - takes 2 hours, involves only remote server S2 (which is faster than S1) I takes 9 hours sequentially. It would take only 6 hours if C and D are started without waiting for B to finish. Because B's workload is server-side, C and D's speed would not be affected. A poor-man's way would be to have separate borgmatic files for each step, and handle the parallelization externally. - **borgmatic version:** 1.4.22 on client. - **Borg version:** 1.1.7 on client, 1.1.10 on S1, 1.1.0 on S2 - **Python version:** 3.6.2 on client - **operating system and version:** Don't think it matters, but Linux 4.4.59 on Synology NAS DS218+ Thanks for any idea!

witten commented

2020-02-03 17:46:00 +00:00

Thanks for filing this! Adding this sort of parallelization to borgmatic would probably require: 1. Managing a pool of processes that could end at various times, 2. Somehow unifying all of the log output of the various processes, even though they might be interspersed, and 3. Potentially dealing with dependencies between processes (writing to the same repository, etc). It's certainly not out of the question, but it sounds like a fair amount of investment.

So my first instinct is to dig into the checks that are taking a while. Have you looked at tweaking the specific checks you're running to perhaps reduce the total runtime? Or, have you considered invoking checks as a separate scheduled job from your backups? That way, with checks decoupled from backups, you could run each at a different time, or even run checks on a less frequent schedule.

Related ticket: witten/borgmatic#227

Thanks for filing this! Adding this sort of parallelization to borgmatic would probably require: 1. Managing a pool of processes that could end at various times, 2. Somehow unifying all of the log output of the various processes, even though they might be interspersed, and 3. Potentially dealing with dependencies between processes (writing to the same repository, etc). It's certainly not out of the question, but it sounds like a fair amount of investment. So my first instinct is to dig into the checks that are taking a while. Have you looked at [tweaking the specific checks](https://torsion.org/borgmatic/docs/how-to/deal-with-very-large-backups/#consistency-check-configuration) you're running to perhaps reduce the total runtime? Or, have you considered invoking checks as a [separate scheduled job](https://torsion.org/borgmatic/docs/how-to/deal-with-very-large-backups/#a-la-carte-actions) from your backups? That way, with checks decoupled from backups, you could run each at a different time, or even run checks on a less frequent schedule. Related ticket: https://projects.torsion.org/witten/borgmatic/issues/227

jucor_ commented

2020-02-03 20:06:02 +00:00

Thanks Dan! Adding 1+2+3 seems absolutely insane, I agree. Better do one thing and do it well. The "A la carte" command line options in spearate cronjob is what is needed for this use case, thank you!

As to tweaking the checks themselves, already done, sticking only to the lightest repositorychecks: the remote server is just a cow. I've contacted the provider to see what can be done on their end.

Closing this ticket since the separate scheduled jobs with command line options do the trick :)
Thanks!

Thanks Dan! Adding 1+2+3 seems absolutely insane, I agree. Better do one thing and do it well. The "A la carte" command line options in spearate cronjob is what is needed for this use case, thank you! As to tweaking the checks themselves, already done, sticking only to the lightest `repository`checks: the remote server is just a cow. I've contacted the provider to see what can be done on their end. Closing this ticket since the separate scheduled jobs with command line options do the trick :) Thanks!

jucor_ closed this issue

2020-02-03 20:06:02 +00:00

witten commented

2020-02-03 20:11:54 +00:00

Glad to hear the separate jobs sound like they'll work out for you! Please continue to file tickets for anything that can be improved.

witten referenced this issue

2020-02-03 20:12:40 +00:00

Borgmatic to manage parallel support? #227

Sign in to join this conversation.