Is it possible to run two borgmatic instances in parallel, both using LVM snapshots? #957

New issue

Closed

opened 2024-12-23 09:58:45 +00:00 by aurelg · 5 comments

aurelg commented 2024-12-23 09:58:45 +00:00

Copy link

What I'm trying to do and why

Thanks for all your wonderful work!

I've tested the new lvm snapshot feature, and it's awesome!

I usually run two borgmatic instances at the same time to parallelize backups to two different repositories, each with its own password (one local storage, the other remote).

Steps to reproduce

Run two borgmatic instances in parallel, both using LVM snapshots.

Actual behavior

I noticed the following error in the second borgmatic instance after the first one is completed:

XXXXXXX: Creating archive
/run/media/XXXXX/XXXXXXX: Snapshotting LVM logical volumes
Creating archive at "/run/media/XXXXX/XXXXXXX/XXXXX::XXXXX"
/tmp/borgmatic-7zjcrits/borgmatic/lvm_snapshots/tmp/borgmatic-7zjcrits/borgmatic/bootstrap: stat: [Errno 2] No such file or directory: '/tmp/borgmatic-7zjcrits/borgmatic/lvm_snapshots/tmp/borgmatic-7zjcrits/borgmatic/bootstrap'
XXXXXXX: Pruning archives
XXXXXXX: Compacting segments
compaction freed about 332.18 MB repository space.

Expected behavior

No response

Other notes / implementation ideas

Could the first complete borgmatic instance have removed the snapshots/mounted
tree created by the second one before its completion? This line seems to target all snapshots created by borgmatic, not the one specific of the running instance?

borgmatic version

1.9.4

borgmatic installation method

pipx

Borg version

1.4.0

Python version

3.13.1

Database version (if applicable)

No response

Operating system and version

NAME="Arch Linux" PRETTY_NAME="Arch Linux" ID=arch BUILD_ID=rolling ANSI_COLOR="38;2;23;147;209" HOME_URL="https://archlinux.org/" DOCUMENTATION_URL="https://wiki.archlinux.org/" SUPPORT_URL="https://bbs.archlinux.org/" BUG_REPORT_URL="https://gitlab.archlinux.org/groups/archlinux/-/issues" PRIVACY_POLICY_URL="https://terms.archlinux.org/docs/privacy-policy/" LOGO=archlinux-logo

### What I'm trying to do and why Thanks for all your wonderful work! I've tested the new `lvm` snapshot feature, and it's awesome! I usually run two borgmatic instances at the same time to parallelize backups to two different repositories, each with its own password (one local storage, the other remote). ### Steps to reproduce Run two borgmatic instances in parallel, both using LVM snapshots. ### Actual behavior I noticed the following error in the second borgmatic instance after the first one is completed: ```text XXXXXXX: Creating archive /run/media/XXXXX/XXXXXXX: Snapshotting LVM logical volumes Creating archive at "/run/media/XXXXX/XXXXXXX/XXXXX::XXXXX" /tmp/borgmatic-7zjcrits/borgmatic/lvm_snapshots/tmp/borgmatic-7zjcrits/borgmatic/bootstrap: stat: [Errno 2] No such file or directory: '/tmp/borgmatic-7zjcrits/borgmatic/lvm_snapshots/tmp/borgmatic-7zjcrits/borgmatic/bootstrap' XXXXXXX: Pruning archives XXXXXXX: Compacting segments compaction freed about 332.18 MB repository space. ``` ### Expected behavior _No response_ ### Other notes / implementation ideas Could the first complete borgmatic instance have removed the snapshots/mounted tree created by the second one before its completion? [This line](https://projects.torsion.org/borgmatic-collective/borgmatic/src/commit/ae8a9db27d1f4029df832bcdb675936117514563/borgmatic/hooks/data_source/lvm.py#L368) seems to target all snapshots created by borgmatic, not the one specific of the running instance? ### borgmatic version 1.9.4 ### borgmatic installation method pipx ### Borg version 1.4.0 ### Python version 3.13.1 ### Database version (if applicable) _No response_ ### Operating system and version NAME="Arch Linux" PRETTY_NAME="Arch Linux" ID=arch BUILD_ID=rolling ANSI_COLOR="38;2;23;147;209" HOME_URL="https://archlinux.org/" DOCUMENTATION_URL="https://wiki.archlinux.org/" SUPPORT_URL="https://bbs.archlinux.org/" BUG_REPORT_URL="https://gitlab.archlinux.org/groups/archlinux/-/issues" PRIVACY_POLICY_URL="https://terms.archlinux.org/docs/privacy-policy/" LOGO=archlinux-logo

witten commented 2024-12-23 15:44:04 +00:00

Owner

Copy link

I'm glad to hear that the new LVM hook is (mostly) working for you! In terms of running two borgmatic instances in parallel, it's unfortunately not intentionally designed for that currently—not just in the LVM hook but throughout the rest of the application code as well. So in your particular case, yes, one borgmatic instance is removing snapshots created by the other instance. This is actually somewhat intentional; the idea is that there may be snapshots leftover from a prior borgmatic run that errored out, snapshots that we don't want to leave littering the filesystem forever.

A few ideas I can think of for addressing this:

• borgmatic could make use of the archive_name_format option when removing old snapshots. The idea is that each borgmatic run could use a configuration file with a different archive_name_format, putting that value into each snapshot path. And then that instance of borgmatic could be responsible for cleaning up a different subset of named snapshots.
• And/or borgmatic could decode the process ID from each snapshot path and only cleanup a snapshot if it's not tied to a currently running borgmatic process.

In any case, there would probably have to be an audit of the whole borgmatic source code as well to make sure that multiple simultaneous processes are really supported. Off the top of my head, I can imagine that things will break or not break depending on the configured or defaulted user runtime directory. Specifically, if borgmatic is using a path in /tmp for the runtime directory, that will conveniently use a differently named path in each borgmatic process. But if using something like /run/user/1234/borgmatic, there could be path collisions between processes.

I'm glad to hear that the new LVM hook is (mostly) working for you! In terms of running two borgmatic instances in parallel, it's unfortunately not intentionally designed for that currently—not just in the LVM hook but throughout the rest of the application code as well. So in your particular case, yes, one borgmatic instance is removing snapshots created by the other instance. This is actually somewhat intentional; the idea is that there may be snapshots leftover from a prior borgmatic run that errored out, snapshots that we don't want to leave littering the filesystem forever. A few ideas I can think of for addressing this: * borgmatic could make use of the `archive_name_format` option when removing old snapshots. The idea is that each borgmatic run could use a configuration file with a different `archive_name_format`, putting that value into each snapshot path. And then that instance of borgmatic could be responsible for cleaning up a different subset of named snapshots. * And/or borgmatic could decode the process ID from each snapshot path and only cleanup a snapshot if it's not tied to a currently running borgmatic process. In any case, there would probably have to be an audit of the whole borgmatic source code as well to make sure that multiple simultaneous processes are really supported. Off the top of my head, I can imagine that things will break or not break depending on the configured or defaulted user runtime directory. Specifically, if borgmatic is using a path in `/tmp` for the runtime directory, that will conveniently use a differently named path in each borgmatic process. But if using something like `/run/user/1234/borgmatic`, there could be path collisions between processes.

aurelg commented 2024-12-24 06:38:22 +00:00

Author

Copy link

Thanks for your kind reply! 🙏 I'll run borgmatic instances sequentially from now on. 👍

Thanks for your kind reply! 🙏 I'll run borgmatic instances sequentially from now on. 👍

👍 1

sm8ps commented 2025-10-24 18:19:31 +00:00

Copy link

I just lost a couple of hours trouble-shooting weird issues where files were not available anymore from Btrfs-snapshots. I had been running multiple Borgmatic processes in parallel because I wanted to speed up the backing up of multiple separate LXC-containers, each with a config file of its own. The background information by @witten explains the behavior perfectly well.
I only wish I had known this before! Do you see a good point to include this in the (excellent!) docu? Maybe on the page How to make per-application backups?

I just lost a couple of hours trouble-shooting weird issues where files were not available anymore from Btrfs-snapshots. I had been running multiple Borgmatic processes in parallel because I wanted to speed up the backing up of multiple separate LXC-containers, each with a config file of its own. The background information by @witten explains the behavior perfectly well. I only wish I had known this before! Do you see a good point to include this in the (excellent!) docu? Maybe on the page [How to make per-application backups](https://torsion.org/borgmatic/how-to/make-per-application-backups/)?

witten referenced this issue from a commit 2025-10-27 22:58:29 +00:00

Document borgmatic's limitations around parallelism—both its own and Borg's (#957).

witten commented 2025-10-27 22:58:41 +00:00

Owner

Copy link

Sorry to hear you lost a couple hours troubleshooting! Great suggestion.. I've just added a note about this to the per-application backups docs, which should be live shortly.

Sorry to hear you lost a couple hours troubleshooting! Great suggestion.. I've just added a note about this to the per-application backups docs, which should be live shortly.

witten closed this issue 2025-10-27 22:58:42 +00:00

sm8ps commented 2025-10-28 05:31:24 +00:00

Copy link

Thanks a lot for including this and more explanations to the docs!

Thanks a lot for including this and more explanations to the docs!

👍 1

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

parallel-borgmatic

command-hook-steps

2.1.7

2.1.6

2.1.5

2.1.4

2.1.3

2.1.2

2.1.1

2.1.0

2.0.13

2.0.12

2.0.11

2.0.10

2.0.9

2.0.8

2.0.7

2.0.6

2.0.5

2.0.4

2.0.3

2.0.2

2.0.1

2.0.0

1.9.14

1.9.13

1.9.12

1.9.11

1.9.10

1.9.9

1.9.8

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.14

1.8.13

1.8.12

1.8.11

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.24

1.5.23

1.5.22

1.5.21

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.14

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.7.dev0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.22

1.4.21

1.4.20

1.4.19

1.4.18

1.4.17

1.4.16

1.4.15

1.4.14

1.4.13

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.26

1.3.25

1.3.24

1.3.23

1.3.22

1.3.21

1.3.20

1.3.19

1.3.18

1.3.17

1.3.16

1.3.15

1.3.14

1.3.13

1.3.12

1.3.11

1.3.10

1.3.9

1.3.8

1.3.7

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.18

1.2.17

1.2.16

1.2.15

1.2.14

1.2.13

1.2.12

1.2.11

1.2.10

1.2.9

1.2.8

1.2.7

1.2.6

1.2.5

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.15

1.1.14

1.1.13

1.1.12

1.1.11

1.1.10

1.1.9

1.1.7

1.1.6

1.1.5

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.3

1.0.2

1.0.1

1.0.0

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

0.0.2

Labels

Clear labels   

blocked

  

breaking

  

bug

  

data loss

  

design finalized

  

good first issue

  

new feature area

  

question / support

  

security

  

waiting for response

No labels

blocked

breaking

bug

data loss

design finalized

good first issue

new feature area

question / support

security

waiting for response

Milestone

Clear milestone

No items

No milestone

Assignees

Clear assignees

No assignees

3 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

borgmatic-collective/borgmatic#957

No description provided.