zfs snapshot hook: unmounting fails with nested dataset

hpfxd commented

2024-12-04 20:49:17 +00:00

What I'm trying to do and why

I am trying to setup the zfs hook to automatically snapshot a dataset before backing it up to avoid any headaches with files being changed during the backup (awesome feature btw :D) The dataset I am trying to backup is contained within another dataset.

Steps to reproduce

This is the output of zfs list with the 2 relevant datasets. Within storage, there is another dataset, storage/nextcloud (the one I am attempting to back up)

$ zfs list storage storage/nextcloud
NAME                USED  AVAIL  REFER  MOUNTPOINT
storage            15.6T  7.87T   120K  /mnt/storage
storage/nextcloud  57.2G  7.87T  57.2G  /mnt/storage/nextcloud

relevant extracts of my borgmatic configuration:

constants:
    app_home: "/srv/nextcloud"
    storage_home: "/mnt/storage/nextcloud"

source_directories:
    - "{app_home}/webroot/config"
    - "{storage_home}"

user_runtime_directory: "{app_home}/borgmatic"

mariadb_databases: ...

zfs:

Actual behavior

The creation of the snapshot works as expected:

# borgmatic -c /etc/borgmatic.d/nextcloud.yaml create --verbosity 2
... <snip>
ssh://borg@localhost/./nextcloud/: Calling zfs hook function dump_data_sources
ssh://borg@localhost/./nextcloud/: Snapshotting ZFS datasets
zfs list -H -t filesystem -o name,mountpoint,org.torsion.borgmatic:backup
ssh://borg@localhost/./nextcloud/: Creating ZFS snapshot storage/nextcloud@borgmatic-1198229
zfs snapshot storage/nextcloud@borgmatic-1198229
ssh://borg@localhost/./nextcloud/: Mounting ZFS snapshot storage/nextcloud@borgmatic-1198229 at /srv/nextcloud/borgmatic/zfs_snapshots/mnt/storage/nextcloud
mount -t zfs storage/nextcloud@borgmatic-1198229 /srv/nextcloud/borgmatic/zfs_snapshots/mnt/storage/nextcloud
... <snip>

The problem later comes when unmounting once the backup is complete:

# borgmatic -c /etc/borgmatic.d/nextcloud.yaml create --verbosity 2
... <snip>
/etc/borgmatic.d/nextcloud.yaml: Calling zfs hook function remove_data_source_dumps
zfs list -H -t filesystem -o name,mountpoint
/etc/borgmatic.d/nextcloud.yaml: Looking for snapshots to remove in /srv/nextcloud/borgmatic/zfs_snapshots
/etc/borgmatic.d/nextcloud.yaml: Unmounting ZFS snapshot at /srv/nextcloud/borgmatic/zfs_snapshots/mnt/storage
umount /srv/nextcloud/borgmatic/zfs_snapshots/mnt/storage
umount: /srv/nextcloud/borgmatic/zfs_snapshots/mnt/storage: not mounted.
/etc/borgmatic.d/nextcloud.yaml: Command 'umount /srv/nextcloud/borgmatic/zfs_snapshots/mnt/storage' returned non-zero exit status 32.
... <snip>

borgmatic looks for filesystems and their mountpoints and tries to unmount mnt/storage, when it should instead unmount mnt/storage/nextcloud

This leaves the snapshot afterwards, which needs manual unmounting and removal:

$ zfs list -t snapshot
NAME                                  USED  AVAIL  REFER  MOUNTPOINT
storage/nextcloud@borgmatic-1198229     0B      -  57.2G  -

Expected behavior

borgmatic should either:

correctly identify which dataset is mounted (perhaps keeping track of it during the earlier mount process)
sort and traverse datasets depth-first, attempting to unmount the deepest nested datasets first
not return after the umount command failure, allowing the loop to continue and succeed on the next iteration in the loop

Other notes / implementation ideas

relevant section in the borgmatic source:

     for snapshots_directory in glob.glob(snapshots_glob):
         if not os.path.isdir(snapshots_directory):
             continue
         # This might fail if the directory is already mounted, but we swallow errors here since
         # we'll try again below. The point of doing it here is that we don't want to try to unmount
         # a non-mounted directory (which *will* fail), and probing for whether a directory is
         # mounted is tough to do in a cross-platform way.
         if not dry_run:
             shutil.rmtree(snapshots_directory, ignore_errors=True)
         for _, mount_point in datasets:
             snapshot_mount_path = os.path.join(snapshots_directory, mount_point.lstrip(os.path.sep))
             if not os.path.isdir(snapshot_mount_path):
                 continue
             logger.debug(
                 f'{log_prefix}: Unmounting ZFS snapshot at {snapshot_mount_path}{dry_run_label}'
             )
             if not dry_run:
                 try:
                     unmount_snapshot(umount_command, snapshot_mount_path)
                 except FileNotFoundError:
                     logger.debug(f'{log_prefix}: Could not find "{umount_command}" command')
                     return
                 except subprocess.CalledProcessError as error:
                     logger.debug(f'{log_prefix}: {error}')
                     return

borgmatic version

1.9.3

borgmatic installation method

pipx

Borg version

borg 1.4.0

Python version

Python 3.11.2

Database version (if applicable)

No response

Operating system and version

Debian GNU/Linux 12 (bookworm)

### What I'm trying to do and why I am trying to setup the zfs hook to automatically snapshot a dataset before backing it up to avoid any headaches with files being changed during the backup (awesome feature btw :D) The dataset I am trying to backup is contained within another dataset. ### Steps to reproduce This is the output of `zfs list` with the 2 relevant datasets. Within `storage`, there is another dataset, `storage/nextcloud` (the one I am attempting to back up) ```shell-session $ zfs list storage storage/nextcloud NAME USED AVAIL REFER MOUNTPOINT storage 15.6T 7.87T 120K /mnt/storage storage/nextcloud 57.2G 7.87T 57.2G /mnt/storage/nextcloud ``` relevant extracts of my borgmatic configuration: ```yaml constants: app_home: "/srv/nextcloud" storage_home: "/mnt/storage/nextcloud" source_directories: - "{app_home}/webroot/config" - "{storage_home}" user_runtime_directory: "{app_home}/borgmatic" mariadb_databases: ... zfs: ``` ### Actual behavior The creation of the snapshot works as expected: ```shell-session # borgmatic -c /etc/borgmatic.d/nextcloud.yaml create --verbosity 2 ... <snip> ssh://borg@localhost/./nextcloud/: Calling zfs hook function dump_data_sources ssh://borg@localhost/./nextcloud/: Snapshotting ZFS datasets zfs list -H -t filesystem -o name,mountpoint,org.torsion.borgmatic:backup ssh://borg@localhost/./nextcloud/: Creating ZFS snapshot storage/nextcloud@borgmatic-1198229 zfs snapshot storage/nextcloud@borgmatic-1198229 ssh://borg@localhost/./nextcloud/: Mounting ZFS snapshot storage/nextcloud@borgmatic-1198229 at /srv/nextcloud/borgmatic/zfs_snapshots/mnt/storage/nextcloud mount -t zfs storage/nextcloud@borgmatic-1198229 /srv/nextcloud/borgmatic/zfs_snapshots/mnt/storage/nextcloud ... <snip> ``` The problem later comes when unmounting once the backup is complete: ```shell-session # borgmatic -c /etc/borgmatic.d/nextcloud.yaml create --verbosity 2 ... <snip> /etc/borgmatic.d/nextcloud.yaml: Calling zfs hook function remove_data_source_dumps zfs list -H -t filesystem -o name,mountpoint /etc/borgmatic.d/nextcloud.yaml: Looking for snapshots to remove in /srv/nextcloud/borgmatic/zfs_snapshots /etc/borgmatic.d/nextcloud.yaml: Unmounting ZFS snapshot at /srv/nextcloud/borgmatic/zfs_snapshots/mnt/storage umount /srv/nextcloud/borgmatic/zfs_snapshots/mnt/storage umount: /srv/nextcloud/borgmatic/zfs_snapshots/mnt/storage: not mounted. /etc/borgmatic.d/nextcloud.yaml: Command 'umount /srv/nextcloud/borgmatic/zfs_snapshots/mnt/storage' returned non-zero exit status 32. ... <snip> ``` borgmatic looks for filesystems and their mountpoints and tries to unmount `mnt/storage`, when it should instead unmount `mnt/storage/nextcloud` This leaves the snapshot afterwards, which needs manual unmounting and removal: ```shell-session $ zfs list -t snapshot NAME USED AVAIL REFER MOUNTPOINT storage/nextcloud@borgmatic-1198229 0B - 57.2G - ``` ### Expected behavior borgmatic should either: 1. correctly identify which dataset is mounted (perhaps keeping track of it during the earlier mount process) 2. sort and traverse datasets depth-first, attempting to unmount the deepest nested datasets first 3. not `return` after the `umount` command failure, allowing the loop to continue and succeed on the next iteration in the loop ### Other notes / implementation ideas relevant section in the borgmatic source: https://projects.torsion.org/borgmatic-collective/borgmatic/src/commit/cfff6c685566512da9b1727e2b0caf57eadbe327/borgmatic/hooks/data_source/zfs.py#L267-L295 ### borgmatic version 1.9.3 ### borgmatic installation method pipx ### Borg version borg 1.4.0 ### Python version Python 3.11.2 ### Database version (if applicable) _No response_ ### Operating system and version Debian GNU/Linux 12 (bookworm)

witten commented

2024-12-04 21:21:44 +00:00

Owner

Thanks so much for trying out the ZFS hook and reporting this issue. I believe the problem has been already fixed in ~~main~~ a branch, which will become the next release. But I'll confirm that it's fixed before closing this ticket.

In terms of the specific solution, it's basically doing your option 2 (more or less).

Thanks so much for trying out the ZFS hook and reporting this issue. I believe the problem has been already fixed in ~~main~~ a branch, which will become the next release. But I'll confirm that it's fixed before closing this ticket. In terms of the specific solution, it's basically doing your option 2 (more or less).

witten added the

bug

label

2024-12-04 21:21:55 +00:00

witten commented

2024-12-04 23:40:53 +00:00

Owner

Okay, I've confirmed this is fixed in the branch after one tweak. Here's the setup for my repro:

# zfs list
NAME           USED  AVAIL  REFER  MOUNTPOINT
pool           252K  23.8M    25K  /pool
pool/dataset    26K  23.8M    26K  /pool/dataset

borgmatic config:

source_directories:
    - /pool/dataset
...
zfs:

And then running borgmatic:

...
1.2.borg: Calling zfs hook function dump_data_sources
1.2.borg: Snapshotting ZFS datasets
zfs list -H -t filesystem -o name,mountpoint,org.torsion.borgmatic:backup
1.2.borg: Creating ZFS snapshot pool/dataset@borgmatic-136727 of /pool/dataset
zfs snapshot pool/dataset@borgmatic-136727
1.2.borg: Mounting ZFS snapshot pool/dataset@borgmatic-136727 at /tmp/borgmatic-tpeg2ek6/borgmatic/zfs_snapshots/pool/dataset
mount -t zfs pool/dataset@borgmatic-136727 /tmp/borgmatic-tpeg2ek6/borgmatic/zfs_snapshots/pool/dataset
BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_EXIT_CODES=*** /root/borg1.4.0 create --debug --show-rc 1.2.borg::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /root/tmp/test.yaml /tmp/borgmatic-tpeg2ek6/./borgmatic/bootstrap /tmp/borgmatic-tpeg2ek6/borgmatic/zfs_snapshots/./pool/dataset
...
/root/tmp/test.yaml: Calling zfs hook function remove_data_source_dumps
zfs list -H -t filesystem -o mountpoint
/root/tmp/test.yaml: Looking for snapshots to remove in /tmp/borgmatic-*/borgmatic/zfs_snapshots
/root/tmp/test.yaml: Unmounting ZFS snapshot at /tmp/borgmatic-tpeg2ek6/borgmatic/zfs_snapshots/pool/dataset
umount /tmp/borgmatic-tpeg2ek6/borgmatic/zfs_snapshots/pool/dataset
zfs list -H -t snapshot -o name
/root/tmp/test.yaml: Destroying ZFS snapshot pool/dataset@borgmatic-136727
zfs destroy pool/dataset@borgmatic-136727
...

# zfs list -t snapshot
no datasets available

Thanks again for the issue report!

Okay, I've confirmed this is fixed in the branch after one tweak. Here's the setup for my repro: ``` # zfs list NAME USED AVAIL REFER MOUNTPOINT pool 252K 23.8M 25K /pool pool/dataset 26K 23.8M 26K /pool/dataset ``` borgmatic config: ```yaml source_directories: - /pool/dataset ... zfs: ``` And then running borgmatic: ``` ... 1.2.borg: Calling zfs hook function dump_data_sources 1.2.borg: Snapshotting ZFS datasets zfs list -H -t filesystem -o name,mountpoint,org.torsion.borgmatic:backup 1.2.borg: Creating ZFS snapshot pool/dataset@borgmatic-136727 of /pool/dataset zfs snapshot pool/dataset@borgmatic-136727 1.2.borg: Mounting ZFS snapshot pool/dataset@borgmatic-136727 at /tmp/borgmatic-tpeg2ek6/borgmatic/zfs_snapshots/pool/dataset mount -t zfs pool/dataset@borgmatic-136727 /tmp/borgmatic-tpeg2ek6/borgmatic/zfs_snapshots/pool/dataset BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_EXIT_CODES=*** /root/borg1.4.0 create --debug --show-rc 1.2.borg::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /root/tmp/test.yaml /tmp/borgmatic-tpeg2ek6/./borgmatic/bootstrap /tmp/borgmatic-tpeg2ek6/borgmatic/zfs_snapshots/./pool/dataset ... /root/tmp/test.yaml: Calling zfs hook function remove_data_source_dumps zfs list -H -t filesystem -o mountpoint /root/tmp/test.yaml: Looking for snapshots to remove in /tmp/borgmatic-*/borgmatic/zfs_snapshots /root/tmp/test.yaml: Unmounting ZFS snapshot at /tmp/borgmatic-tpeg2ek6/borgmatic/zfs_snapshots/pool/dataset umount /tmp/borgmatic-tpeg2ek6/borgmatic/zfs_snapshots/pool/dataset zfs list -H -t snapshot -o name /root/tmp/test.yaml: Destroying ZFS snapshot pool/dataset@borgmatic-136727 zfs destroy pool/dataset@borgmatic-136727 ... ``` ``` # zfs list -t snapshot no datasets available ``` Thanks again for the issue report!

witten closed this issue

2024-12-04 23:41:53 +00:00

witten referenced this issue from a commit

2024-12-04 23:43:31 +00:00

Add ZFS snapshot unmount error fix to NEWS (#950).

hpfxd commented

2024-12-05 00:48:13 +00:00

Author

Thank you! ❤️ tried on 51a7f50e3a and can confirm it works for me as well :)

Thank you! :heart: tried on 51a7f50e3aa981c82ae9bf978e6753eb187bdd98 and can confirm it works for me as well :)

🎉 1

witten commented