ZFS hook fails to correctly unmount snapshots when a dataset is mounted at / #1319
Labels
No labels
blocked
breaking
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
No milestone
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
borgmatic-collective/borgmatic#1319
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What I'm trying to do and why
When a ZFS dataset is mounted at
/withcanmount=noauto(common on ZFS-on-root systems), cleanup emitsumount: not mountederrors for every child dataset. The backup completes successfully but without snapshot consistency for those datasets.Dataset layout
The problem
During cleanup,
remove_data_source_dumps()tries to unmount child datasets from inside the root dataset's snapshot directory, rather than their own.Steps to reproduce
No response
Actual behavior
Expected behavior
No response
Other notes / implementation ideas
No response
borgmatic version
2.1.6
borgmatic installation method
Gentoo package
Borg version
borg 1.4.3
Python version
Python 3.14.4
Database version (if applicable)
No response
Operating system and version
Gentoo Linux 2.18
ZFS hook produces incorrect snapshot paths when a dataset has a blank mountpoint (e.g. root dataset mounted via initrd/fstabto ZFS hook fails to correctly unmount snapshots when a dataset is mounted at /Thanks for filing this. It looks like it happens because of the way that removals work:
.../zfs_snapshots/*.../zfs_snapshots/d34db33f/var/log./var/log's actual snapshot directory is elsewhere), then it bails.That's why you're seeing child dataset snapshots being unmounted from the "wrong" snapshot directory—borgmatic is trying to unmount every combination. Normally, this doesn't result in any errors because of that existent directory check; if borgmatic constructs a non-existent directory, it just moves on. The problem comes when you have a ZFS root among your datasets. Then, when borgmatic constructs a "wrong" directory, it actually exists, because the root filesystem contains those child directories. And this results in the errors you're seeing.. although they are bypassed and therefore and don't stop borgmatic's execution.
The fix is probably to make borgmatic a little more clever about how it probes these paths. For instance, instead of probing for every combination of snapshot directory plus dataset mount point, it could maybe make sure that the hash in the snapshot directory actually matches the dataset mount point path before proceeding. I'll have to think this through a bit.
I have a potential fix, but I don't have a system with a ZFS root at the moment to test it on. (I have tested it against a ZFS dataset that's not at the root.) Since you already have a repro of the problem, do you feel comfortable modifying your local borgmatic source with a potential fix?
If so, the attached
zfs.pyis a drop-in replacement for the existing copy that's installed by your Gentoo package. Not sure exactly where that is on your system; let me know if you have trouble finding it.If you don't feel comfortable, let me know and I can look into setting up a VM to try to repro this.
Thanks!
Just tested it, all looks good to me, snapshots are created and deleted clean
Also, I don't know if you're aware but ZFS can actually manage mounting/unmounting snapshots itself. All datasets have a
.zfsdirectory at the root containing snapshots. You won't see it inls -abut it's there, and when snapshots here are accessed they are transparently mounted on demand, and unmounted when deleted. Maybe you could take advantage of that instead of handling it manually?https://theorangeone.net/posts/zfs-browsable-snapshots/
@lwatsondev wrote in #1319 (comment):
Glad to hear that worked! I'll clean this up, and then it should be part of the next release.
Good idea. Unfortunately, the
.zfsauto-mounted snapshots don't support a particular trick that borgmatic uses with Borg 1.4+ to rewrite the snapshot mount path on the way into the Borg archive. The idea is that when you access those snapshotted files within the archive, they appear at their original pre-snapshot paths. For instance, if borgmatic mounts a snapshot of the/zpoolpath at/run/borgmatic/zfs_snapshots/94adefd5ca2747d1434b/zpool, then within the Borg archive, it'll "magically" appear at/zpoolwith that ugly prefix stripped off. But due to ZFS snapshot naming limitations, I don't know of a way to accomplish that same thing with.zfsauto-mounted snapshots.This is implemented in main and will be part of the next release. Closing this ticket, but I'm happy to continue the discussion.
Released in borgmatic 2.1.7!