#227 Borgmatic to manage parallel support?

Open
opened 1 month ago by decentral1se · 1 comments

What I’m trying to do and why

Backups take a while but if you have a powerful machine, you could run them in parallel! This is possible to manage externally from borgmatic but it would also be nice to have a -n N flag to signal some cores and have borgmatic run what it can in parallel. The user would have to make sure that what is being backed up can be done in parallel.

This is probably a can of worms but would be nice to think through.

#### What I'm trying to do and why Backups take a while but if you have a powerful machine, you could run them in parallel! This is possible to manage externally from borgmatic but it would also be nice to have a `-n N` flag to signal some cores and have borgmatic run what it can in parallel. The user would have to make sure that what is being backed up can be done in parallel. This is probably a can of worms but would be nice to think through.
witten commented 1 month ago
Owner

Interesting. What might you want to be performed in parallel, exactly? Separate Borg invocations, say if you’re backing up to two separate repositories on different remote hosts? Is the goal here to improve performance to reduce total runtime? If so, why? (I can imagine why, but I’m interested in your thinking here.)

Do you have reason to believe that running Borg invocations in parallel would actually improve performance? Put another way, what’s the bottleneck? Is it disk I/O, or network I/O, or CPU (you mentioned cores)? And would layering on a parallel run help?

You don’t need answers to all of these now, but this is the sort of exploration that would help define/scope this feature.

Interesting. What might you want to be performed in parallel, exactly? Separate Borg invocations, say if you're backing up to two separate repositories on different remote hosts? Is the goal here to improve performance to reduce total runtime? If so, why? (I can imagine why, but I'm interested in your thinking here.) Do you have reason to believe that running Borg invocations in parallel would actually improve performance? Put another way, what's the bottleneck? Is it disk I/O, or network I/O, or CPU (you mentioned cores)? And would layering on a parallel run help? You don't need answers to all of these now, but this is the sort of exploration that would help define/scope this feature.
witten added the
waiting for response
label 3 weeks ago
Sign in to join this conversation.
No Milestone
No Assignees
2 Participants
Due Date

No due date set.

Dependencies

This issue currently doesn't have any dependencies.

Loading…
Cancel
Save
There is no content yet.