Crash on LVM Snapshot cleanup

upbeat-eggplant commented

2025-04-13 17:09:19 +00:00

What I'm trying to do and why

I am trying to use the lvm snapshot function. Borgmatic creates the snapshot correctly, but fails to dismount it and crashes. The snapshot remains mounted.
Unmounting the snapshot with the command from the verbose log afterwards works without problems.

Steps to reproduce

No response

Actual behavior

/etc/borgmatic/config.yaml: Calling pushover hook function initialize_monitor
/etc/borgmatic/config.yaml: Calling pushover hook function ping_monitor
/etc/borgmatic/config.yaml: BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_EXIT_CODES=*** borg --version --debug --show-rc
/etc/borgmatic/config.yaml: Borg 1.2.4
/mnt/backup/test.borg: Running actions for repository
/mnt/backup/test.borg: Creating archive
/mnt/backup/test.borg: Using runtime directory /tmp/borgmatic-c051i6km/borgmatic
/mnt/backup/test.borg: Calling bootstrap hook function remove_data_source_dumps
/mnt/backup/test.borg: Looking for bootstrap manifest files to remove in /tmp/borgmatic-/borgmatic/bootstrap
/mnt/backup/test.borg: Calling btrfs hook function remove_data_source_dumps
/mnt/backup/test.borg: Calling lvm hook function remove_data_source_dumps
/mnt/backup/test.borg: lsblk --output name,path,mountpoint,type --json --list
/mnt/backup/test.borg: Looking for snapshots to remove in /tmp/borgmatic-/borgmatic/lvm_snapshots/*
/mnt/backup/test.borg: Unmounting LVM snapshot at /tmp/borgmatic-q070lklo/borgmatic/lvm_snapshots/39c1d0350dfbcc9cbb0b/data
/mnt/backup/test.borg: umount /tmp/borgmatic-q070lklo/borgmatic/lvm_snapshots/39c1d0350dfbcc9cbb0b/data
/mnt/backup/test.borg: Error running actions for repository
/mnt/backup/test.borg: [Errno 2] No such file or directory: '/tmp/borgmatic-q070lklo/borgmatic/lvm_snapshots/39c1d0350dfbcc9cbb0b'
/etc/borgmatic/config.yaml: Calling pushover hook function ping_monitor
/etc/borgmatic/config.yaml: Calling pushover hook function ping_monitor
/etc/borgmatic/config.yaml: Calling pushover hook function destroy_monitor
/etc/borgmatic/config.yaml: An error occurred

summary:
/etc/borgmatic/config.yaml: Loading configuration file
An error occurred
Error running actions for repository
[Errno 2] No such file or directory: '/tmp/borgmatic-q070lklo/borgmatic/lvm_snapshots/39c1d0350dfbcc9cbb0b'

Need some help? https://torsion.org/borgmatic/#issues

Expected behavior

The snapshot should be unmounted and delted.

Other notes / implementation ideas

Config:

verbosity: 2

source_directories:
    - /data

repositories:
    - path: /mnt/backup/test.borg

keep_daily: 7
keep_weekly: 4
keep_monthly: 6
keep_yearly: 1

lvm:

pushover:
    token: "secret"
    user: "secret"

    fail:
        message: A backup job has failed.
        priority: 1

borgmatic version

2.0.2

borgmatic installation method

pipx

Borg version

borg 1.2.4

Python version

Python 3.11.2

Database version (if applicable)

No response

Operating system and version

Debian GNU/Linux 12 (bookworm) / RaspberryPi OS

### What I'm trying to do and why I am trying to use the lvm snapshot function. Borgmatic creates the snapshot correctly, but fails to dismount it and crashes. The snapshot remains mounted. Unmounting the snapshot with the command from the verbose log afterwards works without problems. ### Steps to reproduce _No response_ ### Actual behavior /etc/borgmatic/config.yaml: Calling pushover hook function initialize_monitor /etc/borgmatic/config.yaml: Calling pushover hook function ping_monitor /etc/borgmatic/config.yaml: BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_EXIT_CODES=*** borg --version --debug --show-rc /etc/borgmatic/config.yaml: Borg 1.2.4 /mnt/backup/test.borg: Running actions for repository /mnt/backup/test.borg: Creating archive /mnt/backup/test.borg: Using runtime directory /tmp/borgmatic-c051i6km/borgmatic /mnt/backup/test.borg: Calling bootstrap hook function remove_data_source_dumps /mnt/backup/test.borg: Looking for bootstrap manifest files to remove in /tmp/borgmatic-*/borgmatic/bootstrap /mnt/backup/test.borg: Calling btrfs hook function remove_data_source_dumps /mnt/backup/test.borg: Calling lvm hook function remove_data_source_dumps /mnt/backup/test.borg: lsblk --output name,path,mountpoint,type --json --list /mnt/backup/test.borg: Looking for snapshots to remove in /tmp/borgmatic-*/borgmatic/lvm_snapshots/* **/mnt/backup/test.borg: Unmounting LVM snapshot at /tmp/borgmatic-q070lklo/borgmatic/lvm_snapshots/39c1d0350dfbcc9cbb0b/data /mnt/backup/test.borg: umount /tmp/borgmatic-q070lklo/borgmatic/lvm_snapshots/39c1d0350dfbcc9cbb0b/data /mnt/backup/test.borg: Error running actions for repository /mnt/backup/test.borg: [Errno 2] No such file or directory: '/tmp/borgmatic-q070lklo/borgmatic/lvm_snapshots/39c1d0350dfbcc9cbb0b'** /etc/borgmatic/config.yaml: Calling pushover hook function ping_monitor /etc/borgmatic/config.yaml: Calling pushover hook function ping_monitor /etc/borgmatic/config.yaml: Calling pushover hook function destroy_monitor /etc/borgmatic/config.yaml: An error occurred summary: /etc/borgmatic/config.yaml: Loading configuration file An error occurred Error running actions for repository [Errno 2] No such file or directory: '/tmp/borgmatic-q070lklo/borgmatic/lvm_snapshots/39c1d0350dfbcc9cbb0b' Need some help? https://torsion.org/borgmatic/#issues ### Expected behavior The snapshot should be unmounted and delted. ### Other notes / implementation ideas Config: ```json verbosity: 2 source_directories: - /data repositories: - path: /mnt/backup/test.borg keep_daily: 7 keep_weekly: 4 keep_monthly: 6 keep_yearly: 1 lvm: pushover: token: "secret" user: "secret" fail: message: A backup job has failed. priority: 1 ``` ### borgmatic version 2.0.2 ### borgmatic installation method pipx ### Borg version borg 1.2.4 ### Python version Python 3.11.2 ### Database version (if applicable) _No response_ ### Operating system and version Debian GNU/Linux 12 (bookworm) / RaspberryPi OS

upbeat-eggplant changed title from ~~Crash on LVM Snapshot cleaned~~ to Crash on LVM Snapshot cleanup

2025-04-13 17:09:30 +00:00

witten commented

2025-04-13 17:31:18 +00:00

Owner

Thanks for taking the time to file this. Is this log from a run where borgmatic both creates the snapshot and fails to unmount it? Or is it from a separate run afterwards where it only fails to unmount it, e.g. attempting to clean up from the previous run? Because it looks like the latter to me; I don't see anything about the snapshot being created. Do you happen to have logs from a run where the snapshot is created as well?

After borgmatic fails, can you recursively list the contents of /tmp/borgmatic-q070lklo/borgmatic lvm_snapshots/39c1d0350dfbcc9cbb0b (or similar) and paste them here? (ls -R or tree would work for this.)

If you run mount | grep lvm_snapshots after a failure, what shows up in that output?

Unmounting the snapshot with the command from the verbose log afterwards works without problems.

So you're saying that, for instance, borgmatic fails at umount /tmp/borgmatic-q070lklo/borgmatic/lvm_snapshots/39c1d0350dfbcc9cbb0b/data but if you run that same command manually afterwards, it works?

Thanks for taking the time to file this. Is this log from a run where borgmatic both creates the snapshot and fails to unmount it? Or is it from a separate run afterwards where it only fails to unmount it, e.g. attempting to clean up from the previous run? Because it looks like the latter to me; I don't see anything about the snapshot being created. Do you happen to have logs from a run where the snapshot is created as well? After borgmatic fails, can you recursively list the contents of `/tmp/borgmatic-q070lklo/borgmatic lvm_snapshots/39c1d0350dfbcc9cbb0b` (or similar) and paste them here? (`ls -R` or `tree` would work for this.) If you run `mount | grep lvm_snapshots` after a failure, what shows up in that output? > Unmounting the snapshot with the command from the verbose log afterwards works without problems. So you're saying that, for instance, borgmatic fails at `umount /tmp/borgmatic-q070lklo/borgmatic/lvm_snapshots/39c1d0350dfbcc9cbb0b/data` but if you run that same command manually afterwards, it works?

upbeat-eggplant commented

2025-04-13 18:27:06 +00:00

Author

Thank you for looking into this!
Sorry, something seems a miss with the log I posted above. This issue does indeed occur when borgmatic creates the snapshot and then fails to unmount it during the same run. I have attached a new logfile where this is now properly shown (observe lines 28–35 and 70-76).

root@raspberrypi:/mnt/backup# ls -R /tmp/borgmatic-wtirlje8/borgmatic/lvm_snapshots/39c1d0350dfbcc9cbb0b
ls: cannot access '/tmp/borgmatic-wtirlje8/borgmatic/lvm_snapshots/39c1d0350dfbcc9cbb0b': No such file or directory

It appears that the already "/tmp/borgmatic-wtirlje8" does not exist.

mount | grep lvm_snapshots returns no output. However, running lvs or lvdisplay does show the snapshot still exists. Output of lvs:

LV                             VG        Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  data                           vg_server owi-aos--- 32.00g
  root_filesystem                vg_server -wi-ao---- 24.00g
  vg_server-data_borgmatic-65384 vg_server sri-a-s--- <2.96g      data   0.23

This time the snapshot is no longer mounted, but I'm pretty sure that it remained mounted during some previous tests. Therefore running the umount command manually now produces an error (unlike I said above). I seem unable to reproduce the scenario where the snapshot remains mounted – but it does not get deleted and the crash persists.

Thank you for looking into this! Sorry, something seems a miss with the log I posted above. This issue does indeed occur when borgmatic creates the snapshot and then fails to unmount it during the same run. I have attached a new logfile where this is now properly shown (observe lines 28–35 and 70-76). ``` root@raspberrypi:/mnt/backup# ls -R /tmp/borgmatic-wtirlje8/borgmatic/lvm_snapshots/39c1d0350dfbcc9cbb0b ls: cannot access '/tmp/borgmatic-wtirlje8/borgmatic/lvm_snapshots/39c1d0350dfbcc9cbb0b': No such file or directory ``` It appears that the already "/tmp/borgmatic-wtirlje8" does not exist. `mount | grep lvm_snapshots` returns no output. However, running `lvs` or `lvdisplay` does show the snapshot still exists. Output of `lvs`: ``` LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert data vg_server owi-aos--- 32.00g root_filesystem vg_server -wi-ao---- 24.00g vg_server-data_borgmatic-65384 vg_server sri-a-s--- <2.96g data 0.23 ``` This time the snapshot is no longer mounted, but I'm pretty sure that it remained mounted during some previous tests. Therefore running the umount command manually now produces an error (unlike I said above). I seem unable to reproduce the scenario where the snapshot remains mounted – but it does not get deleted and the crash persists.

borgmatic.log

6.4 KiB

witten commented

2025-04-14 00:19:37 +00:00

Owner

This is really odd! Right before the unmount attempt, there's a check in the code to ensure the snapshot mount path exists (is a directory). If it doesn't, the unmount isn't even attempted. It's conceivable that something is removing the directory after the check and before the unmount, but I don't even know what that would be. You don't have another borgmatic process running at the same time, do you? Or anything else that could do automatic unmounts? borgmatic does remove the entire runtime directory (so /tmp/borgmatic-wtirlje8 in this case), but it only does that after the data source dumps are removed (and presumably unmounted).

This is really odd! Right before the unmount attempt, there's a check in the code to ensure the snapshot mount path exists (is a directory). If it doesn't, the unmount isn't even attempted. It's conceivable that something is removing the directory after the check and before the unmount, but I don't even know what that would be. You don't have another borgmatic process running at the same time, do you? Or anything else that could do automatic unmounts? borgmatic does remove the entire runtime directory (so `/tmp/borgmatic-wtirlje8` in this case), but it only does that *after* the data source dumps are removed (and presumably unmounted).

upbeat-eggplant commented

2025-04-14 08:06:44 +00:00

Author

I honestly wouldn't know what else could unmount the directory. It would also seem odd to me that this would always happen at the same time during the borgmatic run. If something else were trying to unmount the snapshot, wouldn't borgmatic crash sooner while trying to read the files from the snapshot?
Another thing that speaks against some other process meddeling with the mount is that when I abort borgmatic with ctrl-C after the snapshot was created but before it attempts to remove it, it remains mounted (as one would expect). If some other process were unmounting it, I would expect it to unmount it even if borgmatic is aborted.

There is no other borgmatic process running. I have so far only tried to run it manually with borgmatic create and set up no automations that would launch a second process.

Perhaps some more info about my filesystem in case that helps:

NAME                            MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINTS
sda                               8:0    1 14.5G  0 disk
└─sda1                            8:1    1 14.5G  0 part  /mnt/backup
mmcblk0                         179:0    0 59.5G  0 disk
├─mmcblk0p1                     179:1    0  512M  0 part  /boot/firmware
└─mmcblk0p2                     179:2    0   59G  0 part
  └─luks_vg                     254:0    0   59G  0 crypt
    ├─vg_server-root_filesystem 254:1    0   24G  0 lvm   /
    └─vg_server-data            254:2    0   32G  0 lvm   /data

As you can see, I have sda1 mounted to /mnt/backup. This is a USB stick on which I have the borg repository for testing purposes.
mmcblk0 is the SD-Card in the PI which has a small boot partition and a larger partition that is encrypted with LUKS. Below LUKS there are my two lvm volumes root_filesystem and data. I am trying to backup /data. I have separated my data to this lvm-volume to use the snapshoting feature.

I am not currently dumping any databases and just want to backup /data as a whole. Eventually I would like to do what the user in #790 was trying: Stop containers before the snapshot and start them immediately after. I'm unceirtain if borgmatic 2.0 has made this possible or not. At least I don't really understand how I would configure that.

I might just try to reinstall the OS and see if I can reproduce the problem if we can't figure it out.

I honestly wouldn't know what else could unmount the directory. It would also seem odd to me that this would always happen at the same time during the borgmatic run. If something else were trying to unmount the snapshot, wouldn't borgmatic crash sooner while trying to read the files from the snapshot? Another thing that speaks against some other process meddeling with the mount is that when I abort borgmatic with `ctrl-C` after the snapshot was created but before it attempts to remove it, it remains mounted (as one would expect). If some other process were unmounting it, I would expect it to unmount it even if borgmatic is aborted. There is no other borgmatic process running. I have so far only tried to run it manually with `borgmatic create` and set up no automations that would launch a second process. Perhaps some more info about my filesystem in case that helps: ``` NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sda 8:0 1 14.5G 0 disk └─sda1 8:1 1 14.5G 0 part /mnt/backup mmcblk0 179:0 0 59.5G 0 disk ├─mmcblk0p1 179:1 0 512M 0 part /boot/firmware └─mmcblk0p2 179:2 0 59G 0 part └─luks_vg 254:0 0 59G 0 crypt ├─vg_server-root_filesystem 254:1 0 24G 0 lvm / └─vg_server-data 254:2 0 32G 0 lvm /data ``` As you can see, I have `sda1` mounted to `/mnt/backup`. This is a USB stick on which I have the borg repository for testing purposes. `mmcblk0 ` is the SD-Card in the PI which has a small boot partition and a larger partition that is encrypted with LUKS. Below LUKS there are my two lvm volumes `root_filesystem` and `data`. I am trying to backup `/data`. I have separated my data to this lvm-volume to use the snapshoting feature. I am not currently dumping any databases and just want to backup /data as a whole. Eventually I would like to do what the user in #790 was trying: Stop containers before the snapshot and start them immediately after. I'm unceirtain if borgmatic 2.0 has made this possible or not. At least I don't really understand how I would configure that. I might just try to reinstall the OS and see if I can reproduce the problem if we can't figure it out.

witten commented

2025-04-14 18:06:27 +00:00

Owner

Thanks for the additional details. I agree that it doesn't seem likely that another process is meddling with the mount point here.. I'm just looking for any possible explanation here!

So my next theory is that the error isn't actually coming from the unmount, but rather the unmount is succeeding and then the No such file or directory error is happening in some code after the unmount. Especially because the path in the error isn't exactly the same as the path being unmounted.

Would you feel comfortable finding and changing the borgmatic source code on your machine to validate that theory? Since you installed via pipx, I'm guessing that the code in question is located at ~/.local/pipx/venvs/borgmatic/lib/python3.11/site-packages/borgmatic/hooks/data_source/lvm.py or thereabouts. And the line in question is line number 4110. You could try changing this:

        if not dry_run:
            shutil.rmtree(snapshots_directory)

... to this:

        if not dry_run:
            shutil.rmtree(snapshots_directory, ignore_errors=True)

My thinking is that this change should suppress the error you're seeing, which might be just fine because the snapshot is still getting unmounted and subsequently deleted.

Eventually I would like to do what the user in #790 was trying: Stop containers before the snapshot and start them immediately after. I'm unceirtain if borgmatic 2.0 has made this possible or not. At least I don't really understand how I would configure that.

That feature (command hooks around dump_data_sources) did not ultimately get released, because it turned out it did not meet that particular user's needs. But feel free to open a separate ticket and we can discuss it there.

Thanks for the additional details. I agree that it doesn't seem likely that another process is meddling with the mount point here.. I'm just looking for any possible explanation here! So my next theory is that the error isn't actually coming from the unmount, but rather the unmount is succeeding and then the `No such file or directory` error is happening in some code *after* the unmount. Especially because the path in the error isn't exactly the same as the path being unmounted. Would you feel comfortable finding and changing the borgmatic source code on your machine to validate that theory? Since you installed via `pipx`, I'm guessing that the code in question is located at `~/.local/pipx/venvs/borgmatic/lib/python3.11/site-packages/borgmatic/hooks/data_source/lvm.py` or thereabouts. And the line in question is line number 4110. You could try changing this: ```python if not dry_run: shutil.rmtree(snapshots_directory) ``` ... to this: ```python if not dry_run: shutil.rmtree(snapshots_directory, ignore_errors=True) ``` My thinking is that this change should suppress the error you're seeing, which might be just fine because the snapshot is still getting unmounted and subsequently deleted. > Eventually I would like to do what the user in #790 was trying: Stop containers before the snapshot and start them immediately after. I'm unceirtain if borgmatic 2.0 has made this possible or not. At least I don't really understand how I would configure that. That feature (command hooks around `dump_data_sources`) did not ultimately get released, because it turned out it did not meet that particular user's needs. But feel free to open a separate ticket and we can discuss it there.

witten referenced this issue from a commit

2025-04-15 05:49:18 +00:00

Fix an error in the LVM hook when removing a snapshot directory (#1071).

witten commented

2025-04-15 06:01:24 +00:00

Owner

I'm pretty confident this change will fix this for you, so I went ahead and pushed it to main and I've just released it. This means you shouldn't have to change anything locally to try this out if you upgrade to borgmatic 2.0.3. But please still report back if it solves this for you. Thanks!

witten closed this issue

2025-04-15 06:01:24 +00:00

upbeat-eggplant commented

2025-04-15 09:20:36 +00:00

Author

Thank you so much! This did indeed fix the issue and everything is working as expected now.
I would also like to take a moment and compliment the way you're handeling the reported issues here. It is unfortunately rare to find discussions in such a friendly and helpful tone – so thanks for this too.

Thank you so much! This did indeed fix the issue and everything is working as expected now. I would also like to take a moment and compliment the way you're handeling the reported issues here. It is unfortunately rare to find discussions in such a friendly and helpful tone – so thanks for this too.

witten commented

2025-04-15 15:50:55 +00:00

Owner

Awesome, I'm glad to hear that did it! And thank you for the kind words.

witten referenced this issue

2025-09-18 05:04:54 +00:00

lvm causes "No such file or directory" error #1147