borgmatic-collective/borgmatic

Watch

Fork 57

lvm causes "No such file or directory" error #1147

New issue

Closed

opened 2025-09-18 00:09:43 +00:00 by andrew-schulman · 3 comments

andrew-schulman commented

2025-09-18 00:09:43 +00:00

What I'm trying to do and why

Backing up some virtual machine disk image files (.vdi). To avoid them changing while the backup is running, I use the lvm: feature to take an LVM snapshot. Neat! The backup works fine, but at the end there's a "No such file or directory" error after the umount command for the snapshot. This causes healthcheck.io to say the job failed, although the backup did succeed.

Steps to reproduce

Here's /etc/backup.d/vm-home.yaml:

patterns_from:
  - /etc/borgmatic.d/vm-home.patterns
repositories:
  - path:  ssh://xxx@xxx.rsync.net/./borg/helium
    label: rsync.net-helium
archive_name_format: 'vm-home-{now:%Y-%m-%d}'
encryption_passcommand: 'cat /etc/borgmatic.d/passphrase/rsync.net-helium'
keep_within: 7d
remote_path: borg12
relocated_repo_access_is_ok: true
checks:
  - name:      repository
    frequency: 2 weeks
  - name:      archives
    frequency: 2 weeks
lvm:
healthchecks:
  ping_url: 'https://hc-ping.com/xxx/helium-vm-home'

And /etc/backup.d/vm-home.patterns:

R /home
+ **/*.vdi
- **

Actual behavior

Here's the whole run log. Everything works fine until near the end:

No such file or directory: '/tmp/root/borgmatic-zlv2gyjl/borgmatic/lvm_snapshots/f667480a34df1550a601'

Note that the missing directory is the parent of the snapshot directory that just got unmounted. Not sure why it went away, but ls confirms that it did.

##  borgmatic --verbosity 2 --config /etc/borgmatic.d/vm-home.yaml 2>&1
No commands to run for pre-everything hook
/etc/borgmatic.d/vm-home.yaml: cat /etc/borgmatic.d/passphrase/rsync.net-helium
/etc/borgmatic.d/vm-home.yaml: BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_REMOTE_PATH=*** BORG_PASSPHRASE_FD=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_EXIT_CODES=*** borg --version --debug --show-rc
/etc/borgmatic.d/vm-home.yaml: Borg 1.2.8
/etc/borgmatic.d/vm-home.yaml: Calling healthchecks hook function initialize_monitor
/etc/borgmatic.d/vm-home.yaml: Calling healthchecks hook function ping_monitor
/etc/borgmatic.d/vm-home.yaml: Pinging Healthchecks start
/etc/borgmatic.d/vm-home.yaml: Using Healthchecks ping URL https://hc-ping.com/rlN8bLhLdmvyWmavH2F5DQ/helium-vm-home/start
rsync.net-helium: Running actions for repository
rsync.net-helium: No commands to run for pre-actions hook
rsync.net-helium: No commands to run for pre-backup hook
rsync.net-helium: Creating archive
rsync.net-helium: Using runtime directory /tmp/root/borgmatic-zlv2gyjl/borgmatic
rsync.net-helium: Calling bootstrap hook function remove_data_source_dumps
rsync.net-helium: Looking for bootstrap manifest files to remove in /tmp/root/borgmatic-*/borgmatic/bootstrap
rsync.net-helium: Calling btrfs hook function remove_data_source_dumps
rsync.net-helium: Calling lvm hook function remove_data_source_dumps
rsync.net-helium: lsblk --output name,path,mountpoint,type --json --list
rsync.net-helium: Looking for snapshots to remove in /tmp/root/borgmatic-*/borgmatic/lvm_snapshots/*
rsync.net-helium: lvs --report-format json --options lv_name,lv_path --select lv_attr =~ ^s
File descriptor 3 (pipe:[435132]) leaked on lvs invocation. Parent PID 198885: /usr/bin/python3
rsync.net-helium: Calling mariadb hook function remove_data_source_dumps
rsync.net-helium: Removing MariaDB data source dumps
rsync.net-helium: Calling mongodb hook function remove_data_source_dumps
rsync.net-helium: Removing MongoDB data source dumps
rsync.net-helium: Calling mysql hook function remove_data_source_dumps
rsync.net-helium: Removing MySQL data source dumps
rsync.net-helium: Calling postgresql hook function remove_data_source_dumps
rsync.net-helium: Removing PostgreSQL data source dumps
rsync.net-helium: Calling sqlite hook function remove_data_source_dumps
rsync.net-helium: Removing SQLite data source dumps
rsync.net-helium: Calling zfs hook function remove_data_source_dumps
rsync.net-helium: Calling bootstrap hook function dump_data_sources
rsync.net-helium: Calling lvm hook function dump_data_sources
rsync.net-helium: Snapshotting LVM logical volumes
rsync.net-helium: lsblk --output name,path,mountpoint,type --json --list
rsync.net-helium: Creating LVM snapshot big-home_borgmatic-198885 of /home
rsync.net-helium: lvcreate --snapshot --extents 10%ORIGIN --permission r --name big-home_borgmatic-198885 /dev/mapper/big-home
File descriptor 3 (pipe:[435132]) leaked on lvcreate invocation. Parent PID 198885: /usr/bin/python3
  Logical volume "big-home_borgmatic-198885" created.
rsync.net-helium: lvs --report-format json --options lv_name,lv_path --select lv_attr =~ ^s
File descriptor 3 (pipe:[435132]) leaked on lvs invocation. Parent PID 198885: /usr/bin/python3
rsync.net-helium: Mounting LVM snapshot big-home_borgmatic-198885 at /tmp/root/borgmatic-zlv2gyjl/borgmatic/lvm_snapshots/f667480a34df1550a601/home
rsync.net-helium: mount -o ro /dev/big/big-home_borgmatic-198885 /tmp/root/borgmatic-zlv2gyjl/borgmatic/lvm_snapshots/f667480a34df1550a601/home
rsync.net-helium: Writing patterns to /tmp/root/borgmatic-zlv2gyjl/borgmatic/tmpofe4725t:
R /tmp/root/borgmatic-zlv2gyjl/borgmatic/lvm_snapshots/f667480a34df1550a601/./home
+ **/*.vdi
- **
R /etc/borgmatic.d/vm-home.yaml
R /tmp/root/borgmatic-zlv2gyjl/./borgmatic/bootstrap
rsync.net-helium: BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_REMOTE_PATH=*** BORG_PASSPHRASE_FD=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_EXIT_CODES=*** borg create --patterns-from /tmp/root/borgmatic-zlv2gyjl/borgmatic/tmpofe4725t --remote-path borg12 --debug --show-rc ssh://xxx@xxx.rsync.net/./borg/helium::vm-home-{now:%Y-%m-%d}
using builtin fallback logging configuration
33 self tests completed in 0.07 seconds
SSH command line: ['ssh', 'xxx@xxx.rsync.net', 'borg12', 'serve', '--debug']
Remote: using builtin fallback logging configuration
Remote: borg selftest disabled via BORG_SELFTEST env variable
Remote: using builtin fallback logging configuration
Remote: Initialized logging system for JSON-based protocol
Remote: Resolving repository path b'/./borg/helium'
Remote: Resolved repository path to '/data1/home/xxx/borg/helium'
Remote: Verified integrity of /data1/home/xxx/borg/helium/index.2155
TAM-verified manifest
security: read previous location 'ssh://xxx@xxx.rsync.net/./borg/helium'
security: read manifest timestamp '2025-09-17T22:23:49.573296'
security: determined newest manifest timestamp as 2025-09-17T22:23:49.573296
security: repository checks ok, allowing access
Creating archive at "ssh://xxx@xxx.rsync.net/./borg/helium::vm-home-2025-09-17"
Verified integrity of /root/.cache/borg/7b031487e00e1c8180ae19e84c68fa3185c3c664024bc905bf4f1d823827d072/chunks
Reading files cache ...
Verified integrity of /root/.cache/borg/7b031487e00e1c8180ae19e84c68fa3185c3c664024bc905bf4f1d823827d072/files
security: read previous location 'ssh://xxx@xxx.rsync.net/./borg/helium'
security: read manifest timestamp '2025-09-17T22:23:49.573296'
security: determined newest manifest timestamp as 2025-09-17T22:23:49.573296
security: repository checks ok, allowing access
Processing files ...
Remote: Cleaned up 0 uncommitted segment files (== everything after segment 2155).
Remote: Verified integrity of /data1/home/xxx/borg/helium/hints.2155
Remote: check_free_space: required bytes 751352858, free bytes 65227710976
security: saving state for 7b031487e00e1c8180ae19e84c68fa3185c3c664024bc905bf4f1d823827d072 to /root/.config/borg/security/7b031487e00e1c8180ae19e84c68fa3185c3c664024bc905bf4f1d823827d072
security: current location   ssh://xxx@xxx.rsync.net/./borg/helium
security: key type           3
security: manifest timestamp 2025-09-17T22:38:10.996069
RemoteRepository: 463.32 kB bytes sent, 3.53 kB bytes received, 14 messages sent
terminating with success status, rc 0
rsync.net-helium: Calling bootstrap hook function remove_data_source_dumps
rsync.net-helium: Looking for bootstrap manifest files to remove in /tmp/root/borgmatic-*/borgmatic/bootstrap
rsync.net-helium: Removing bootstrap manifest at /tmp/root/borgmatic-zlv2gyjl/borgmatic/bootstrap/manifest.json
rsync.net-helium: Calling btrfs hook function remove_data_source_dumps
rsync.net-helium: Calling lvm hook function remove_data_source_dumps
rsync.net-helium: lsblk --output name,path,mountpoint,type --json --list
rsync.net-helium: Looking for snapshots to remove in /tmp/root/borgmatic-*/borgmatic/lvm_snapshots/*
rsync.net-helium: Unmounting LVM snapshot at /tmp/root/borgmatic-zlv2gyjl/borgmatic/lvm_snapshots/f667480a34df1550a601/home
rsync.net-helium: umount /tmp/root/borgmatic-zlv2gyjl/borgmatic/lvm_snapshots/f667480a34df1550a601/home
rsync.net-helium: Error running actions for repository
rsync.net-helium: [Errno 2] No such file or directory: '/tmp/root/borgmatic-zlv2gyjl/borgmatic/lvm_snapshots/f667480a34df1550a601'
/etc/borgmatic.d/vm-home.yaml: Calling healthchecks hook function ping_monitor
/etc/borgmatic.d/vm-home.yaml: Pinging Healthchecks log
/etc/borgmatic.d/vm-home.yaml: Using Healthchecks ping URL https://hc-ping.com/rlN8bLhLdmvyWmavH2F5DQ/helium-vm-home/log
/etc/borgmatic.d/vm-home.yaml: No commands to run for on-error hook
/etc/borgmatic.d/vm-home.yaml: Calling healthchecks hook function ping_monitor
/etc/borgmatic.d/vm-home.yaml: Pinging Healthchecks fail
/etc/borgmatic.d/vm-home.yaml: Using Healthchecks ping URL https://hc-ping.com/rlN8bLhLdmvyWmavH2F5DQ/helium-vm-home/fail
/etc/borgmatic.d/vm-home.yaml: Calling healthchecks hook function destroy_monitor
/etc/borgmatic.d/vm-home.yaml: An error occurred
No commands to run for post-everything hook

summary:
/etc/borgmatic.d/vm-home.yaml: Loading configuration file
An error occurred
Error running actions for repository
[Errno 2] No such file or directory: '/tmp/root/borgmatic-zlv2gyjl/borgmatic/lvm_snapshots/f667480a34df1550a601'

Need some help? https://torsion.org/borgmatic/#issues

Expected behavior

No errors.

Other notes / implementation ideas

No response

borgmatic version

1.9.14

borgmatic installation method

Ubuntu package

Borg version

1.2.8

Python version

Python 3.12.3

Database version (if applicable)

No response

Operating system and version

Ubuntu 24.04.3 LTS (noble)

### What I'm trying to do and why Backing up some virtual machine disk image files (.vdi). To avoid them changing while the backup is running, I use the `lvm:` feature to take an LVM snapshot. Neat! The backup works fine, but at the end there's a "No such file or directory" error after the umount command for the snapshot. This causes healthcheck.io to say the job failed, although the backup did succeed. ### Steps to reproduce Here's /etc/backup.d/vm-home.yaml: ``` patterns_from: - /etc/borgmatic.d/vm-home.patterns repositories: - path: ssh://xxx@xxx.rsync.net/./borg/helium label: rsync.net-helium archive_name_format: 'vm-home-{now:%Y-%m-%d}' encryption_passcommand: 'cat /etc/borgmatic.d/passphrase/rsync.net-helium' keep_within: 7d remote_path: borg12 relocated_repo_access_is_ok: true checks: - name: repository frequency: 2 weeks - name: archives frequency: 2 weeks lvm: healthchecks: ping_url: 'https://hc-ping.com/xxx/helium-vm-home' ``` And /etc/backup.d/vm-home.patterns: ``` R /home + **/*.vdi - ** ``` ### Actual behavior Here's the whole run log. Everything works fine until near the end: No such file or directory: '/tmp/root/borgmatic-zlv2gyjl/borgmatic/lvm_snapshots/f667480a34df1550a601' Note that the missing directory is the parent of the snapshot directory that just got unmounted. Not sure why it went away, but `ls` confirms that it did. ``` ## borgmatic --verbosity 2 --config /etc/borgmatic.d/vm-home.yaml 2>&1 No commands to run for pre-everything hook /etc/borgmatic.d/vm-home.yaml: cat /etc/borgmatic.d/passphrase/rsync.net-helium /etc/borgmatic.d/vm-home.yaml: BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_REMOTE_PATH=*** BORG_PASSPHRASE_FD=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_EXIT_CODES=*** borg --version --debug --show-rc /etc/borgmatic.d/vm-home.yaml: Borg 1.2.8 /etc/borgmatic.d/vm-home.yaml: Calling healthchecks hook function initialize_monitor /etc/borgmatic.d/vm-home.yaml: Calling healthchecks hook function ping_monitor /etc/borgmatic.d/vm-home.yaml: Pinging Healthchecks start /etc/borgmatic.d/vm-home.yaml: Using Healthchecks ping URL https://hc-ping.com/rlN8bLhLdmvyWmavH2F5DQ/helium-vm-home/start rsync.net-helium: Running actions for repository rsync.net-helium: No commands to run for pre-actions hook rsync.net-helium: No commands to run for pre-backup hook rsync.net-helium: Creating archive rsync.net-helium: Using runtime directory /tmp/root/borgmatic-zlv2gyjl/borgmatic rsync.net-helium: Calling bootstrap hook function remove_data_source_dumps rsync.net-helium: Looking for bootstrap manifest files to remove in /tmp/root/borgmatic-*/borgmatic/bootstrap rsync.net-helium: Calling btrfs hook function remove_data_source_dumps rsync.net-helium: Calling lvm hook function remove_data_source_dumps rsync.net-helium: lsblk --output name,path,mountpoint,type --json --list rsync.net-helium: Looking for snapshots to remove in /tmp/root/borgmatic-*/borgmatic/lvm_snapshots/* rsync.net-helium: lvs --report-format json --options lv_name,lv_path --select lv_attr =~ ^s File descriptor 3 (pipe:[435132]) leaked on lvs invocation. Parent PID 198885: /usr/bin/python3 rsync.net-helium: Calling mariadb hook function remove_data_source_dumps rsync.net-helium: Removing MariaDB data source dumps rsync.net-helium: Calling mongodb hook function remove_data_source_dumps rsync.net-helium: Removing MongoDB data source dumps rsync.net-helium: Calling mysql hook function remove_data_source_dumps rsync.net-helium: Removing MySQL data source dumps rsync.net-helium: Calling postgresql hook function remove_data_source_dumps rsync.net-helium: Removing PostgreSQL data source dumps rsync.net-helium: Calling sqlite hook function remove_data_source_dumps rsync.net-helium: Removing SQLite data source dumps rsync.net-helium: Calling zfs hook function remove_data_source_dumps rsync.net-helium: Calling bootstrap hook function dump_data_sources rsync.net-helium: Calling lvm hook function dump_data_sources rsync.net-helium: Snapshotting LVM logical volumes rsync.net-helium: lsblk --output name,path,mountpoint,type --json --list rsync.net-helium: Creating LVM snapshot big-home_borgmatic-198885 of /home rsync.net-helium: lvcreate --snapshot --extents 10%ORIGIN --permission r --name big-home_borgmatic-198885 /dev/mapper/big-home File descriptor 3 (pipe:[435132]) leaked on lvcreate invocation. Parent PID 198885: /usr/bin/python3 Logical volume "big-home_borgmatic-198885" created. rsync.net-helium: lvs --report-format json --options lv_name,lv_path --select lv_attr =~ ^s File descriptor 3 (pipe:[435132]) leaked on lvs invocation. Parent PID 198885: /usr/bin/python3 rsync.net-helium: Mounting LVM snapshot big-home_borgmatic-198885 at /tmp/root/borgmatic-zlv2gyjl/borgmatic/lvm_snapshots/f667480a34df1550a601/home rsync.net-helium: mount -o ro /dev/big/big-home_borgmatic-198885 /tmp/root/borgmatic-zlv2gyjl/borgmatic/lvm_snapshots/f667480a34df1550a601/home rsync.net-helium: Writing patterns to /tmp/root/borgmatic-zlv2gyjl/borgmatic/tmpofe4725t: R /tmp/root/borgmatic-zlv2gyjl/borgmatic/lvm_snapshots/f667480a34df1550a601/./home + **/*.vdi - ** R /etc/borgmatic.d/vm-home.yaml R /tmp/root/borgmatic-zlv2gyjl/./borgmatic/bootstrap rsync.net-helium: BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_REMOTE_PATH=*** BORG_PASSPHRASE_FD=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_EXIT_CODES=*** borg create --patterns-from /tmp/root/borgmatic-zlv2gyjl/borgmatic/tmpofe4725t --remote-path borg12 --debug --show-rc ssh://xxx@xxx.rsync.net/./borg/helium::vm-home-{now:%Y-%m-%d} using builtin fallback logging configuration 33 self tests completed in 0.07 seconds SSH command line: ['ssh', 'xxx@xxx.rsync.net', 'borg12', 'serve', '--debug'] Remote: using builtin fallback logging configuration Remote: borg selftest disabled via BORG_SELFTEST env variable Remote: using builtin fallback logging configuration Remote: Initialized logging system for JSON-based protocol Remote: Resolving repository path b'/./borg/helium' Remote: Resolved repository path to '/data1/home/xxx/borg/helium' Remote: Verified integrity of /data1/home/xxx/borg/helium/index.2155 TAM-verified manifest security: read previous location 'ssh://xxx@xxx.rsync.net/./borg/helium' security: read manifest timestamp '2025-09-17T22:23:49.573296' security: determined newest manifest timestamp as 2025-09-17T22:23:49.573296 security: repository checks ok, allowing access Creating archive at "ssh://xxx@xxx.rsync.net/./borg/helium::vm-home-2025-09-17" Verified integrity of /root/.cache/borg/7b031487e00e1c8180ae19e84c68fa3185c3c664024bc905bf4f1d823827d072/chunks Reading files cache ... Verified integrity of /root/.cache/borg/7b031487e00e1c8180ae19e84c68fa3185c3c664024bc905bf4f1d823827d072/files security: read previous location 'ssh://xxx@xxx.rsync.net/./borg/helium' security: read manifest timestamp '2025-09-17T22:23:49.573296' security: determined newest manifest timestamp as 2025-09-17T22:23:49.573296 security: repository checks ok, allowing access Processing files ... Remote: Cleaned up 0 uncommitted segment files (== everything after segment 2155). Remote: Verified integrity of /data1/home/xxx/borg/helium/hints.2155 Remote: check_free_space: required bytes 751352858, free bytes 65227710976 security: saving state for 7b031487e00e1c8180ae19e84c68fa3185c3c664024bc905bf4f1d823827d072 to /root/.config/borg/security/7b031487e00e1c8180ae19e84c68fa3185c3c664024bc905bf4f1d823827d072 security: current location ssh://xxx@xxx.rsync.net/./borg/helium security: key type 3 security: manifest timestamp 2025-09-17T22:38:10.996069 RemoteRepository: 463.32 kB bytes sent, 3.53 kB bytes received, 14 messages sent terminating with success status, rc 0 rsync.net-helium: Calling bootstrap hook function remove_data_source_dumps rsync.net-helium: Looking for bootstrap manifest files to remove in /tmp/root/borgmatic-*/borgmatic/bootstrap rsync.net-helium: Removing bootstrap manifest at /tmp/root/borgmatic-zlv2gyjl/borgmatic/bootstrap/manifest.json rsync.net-helium: Calling btrfs hook function remove_data_source_dumps rsync.net-helium: Calling lvm hook function remove_data_source_dumps rsync.net-helium: lsblk --output name,path,mountpoint,type --json --list rsync.net-helium: Looking for snapshots to remove in /tmp/root/borgmatic-*/borgmatic/lvm_snapshots/* rsync.net-helium: Unmounting LVM snapshot at /tmp/root/borgmatic-zlv2gyjl/borgmatic/lvm_snapshots/f667480a34df1550a601/home rsync.net-helium: umount /tmp/root/borgmatic-zlv2gyjl/borgmatic/lvm_snapshots/f667480a34df1550a601/home rsync.net-helium: Error running actions for repository rsync.net-helium: [Errno 2] No such file or directory: '/tmp/root/borgmatic-zlv2gyjl/borgmatic/lvm_snapshots/f667480a34df1550a601' /etc/borgmatic.d/vm-home.yaml: Calling healthchecks hook function ping_monitor /etc/borgmatic.d/vm-home.yaml: Pinging Healthchecks log /etc/borgmatic.d/vm-home.yaml: Using Healthchecks ping URL https://hc-ping.com/rlN8bLhLdmvyWmavH2F5DQ/helium-vm-home/log /etc/borgmatic.d/vm-home.yaml: No commands to run for on-error hook /etc/borgmatic.d/vm-home.yaml: Calling healthchecks hook function ping_monitor /etc/borgmatic.d/vm-home.yaml: Pinging Healthchecks fail /etc/borgmatic.d/vm-home.yaml: Using Healthchecks ping URL https://hc-ping.com/rlN8bLhLdmvyWmavH2F5DQ/helium-vm-home/fail /etc/borgmatic.d/vm-home.yaml: Calling healthchecks hook function destroy_monitor /etc/borgmatic.d/vm-home.yaml: An error occurred No commands to run for post-everything hook summary: /etc/borgmatic.d/vm-home.yaml: Loading configuration file An error occurred Error running actions for repository [Errno 2] No such file or directory: '/tmp/root/borgmatic-zlv2gyjl/borgmatic/lvm_snapshots/f667480a34df1550a601' Need some help? https://torsion.org/borgmatic/#issues ``` ### Expected behavior No errors. ### Other notes / implementation ideas _No response_ ### borgmatic version 1.9.14 ### borgmatic installation method Ubuntu package ### Borg version 1.2.8 ### Python version Python 3.12.3 ### Database version (if applicable) _No response_ ### Operating system and version Ubuntu 24.04.3 LTS (noble)

witten commented

2025-09-18 05:04:54 +00:00

Owner

Good news! This was very likely already fixed in #1071, so if you can upgrade to a newer version of borgmatic (at least 2.0.3, but 2.0.7+ would be even better), you shouldn't get this error anymore. Please let me know if that's not the case.

witten added the

question / support

label

2025-09-18 05:04:59 +00:00

andrew-schulman commented

2025-09-18 08:32:33 +00:00

Author

Doh. Sorry, I neglected to look at the closed issues, or I would've spotted that. Yes, the patch that fixed #1071 also fixes this one. ignore_errors=True - nice! Thanks for your help. Closing.

Doh. Sorry, I neglected to look at the closed issues, or I would've spotted that. Yes, the patch that fixed #1071 also fixes this one. `ignore_errors=True` - nice! Thanks for your help. Closing.

andrew-schulman closed this issue

2025-09-18 08:32:34 +00:00

witten commented

2025-09-18 18:01:16 +00:00

Owner

No worries! I'm glad to hear that was it.