ZFS snapshots (#261). #944

witten · 2024-11-21T06:39:09Z

witten commented

2024-11-21 06:39:09 +00:00

This is currently beta-quality work for snapshotting ZFS filesystems (#261) from borgmatic such that those filesystems end up in a Borg archive at the original dataset path rather than the snapshot mount point path. That means that if your dataset is mounted at, say, /my/dataset, then the snapshotted data shows up in the Borg archive at my/dataset and not my/dataset/.zfs/snapshot/blah.

Dataset discovery currently works by expecting that any dataset mount points to backup are listed in borgmatic's source_directories.

Result

I have run this and it appears to work so far. Here are some logs:

# zfs list
NAME    USED  AVAIL  REFER  MOUNTPOINT
zpool   150K  23.9M    25K  /zpool

# borgmatic -c test.yaml -v 2 create
...
test2.borg: Calling zfs hook function dump_data_sources
test2.borg: Snapshotting ZFS datasets
zfs list -H -t filesystem -o name,mountpoint
test2.borg: Creating ZFS snapshot zpool@borgmatic-1994531
zfs snapshot -r zpool@borgmatic-1994531
test2.borg: Mounting ZFS snapshot zpool@borgmatic-1994531 at /tmp/borgmatic-bxntey17/borgmatic/zfs_snapshots/./zpool
mount -t zfs zpool@borgmatic-1994531 /tmp/borgmatic-bxntey17/borgmatic/zfs_snapshots/./zpool
BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_EXIT_CODES=*** /root/borg2.0.0b14 create --debug --show-rc --repo test2.borg {hostname} /root/tmp /tmp/borgmatic-bxntey17/./borgmatic /tmp/borgmatic-bxntey17/borgmatic/zfs_snapshots/./zpool
...
/root/tmp/test.yaml: Calling zfs hook function remove_data_source_dumps
zfs list -H -o name,mountpoint
/root/tmp/test.yaml: Looking for snapshots in /tmp/borgmatic-bxntey17/borgmatic/zfs_snapshots
/root/tmp/test.yaml: Unmounting ZFS snapshot at /tmp/borgmatic-bxntey17/borgmatic/zfs_snapshots/zpool
umount tmp/borgmatic-bxntey17/borgmatic/zfs_snapshots/zpool
zfs list -H -t snapshot -o name
/root/tmp/test.yaml: Destroying ZFS snapshot zpool@borgmatic-1994531
zfs destroy -r zpool@borgmatic-1994531
...

# borgmatic -c test.yaml -v 1 list --archive latest
test2.borg: Listing archive 38cdd9ba7bcd354b4493d98fe0b7d258bf2f1b87ef9a2733d308abf480ff97a8
...
drwxr-xr-x root   root          0 Wed, 2024-11-20 12:05:54 -0800 zpool
-rw-r--r-- root   root          4 Wed, 2024-11-20 12:05:54 -0800 zpool/file.txt

Still to do

all TODOs and FIXMEs
do an organizational pass on the code
documentation
get existing unit tests passing again
add tests to cover new/changed code
audit branch diff for missing test coverage
additional error handling
lots more manual testing

This is currently beta-quality work for snapshotting ZFS filesystems (#261) from borgmatic such that those filesystems end up in a Borg archive at the original dataset path rather than the snapshot mount point path. That means that if your dataset is mounted at, say, `/my/dataset`, then the snapshotted data shows up in the Borg archive at `my/dataset` and not `my/dataset/.zfs/snapshot/blah`. Dataset discovery currently works by expecting that any dataset mount points to backup are listed in borgmatic's `source_directories`. ### Result I have run this and it appears to work so far. Here are some logs: ``` # zfs list NAME USED AVAIL REFER MOUNTPOINT zpool 150K 23.9M 25K /zpool # borgmatic -c test.yaml -v 2 create ... test2.borg: Calling zfs hook function dump_data_sources test2.borg: Snapshotting ZFS datasets zfs list -H -t filesystem -o name,mountpoint test2.borg: Creating ZFS snapshot zpool@borgmatic-1994531 zfs snapshot -r zpool@borgmatic-1994531 test2.borg: Mounting ZFS snapshot zpool@borgmatic-1994531 at /tmp/borgmatic-bxntey17/borgmatic/zfs_snapshots/./zpool mount -t zfs zpool@borgmatic-1994531 /tmp/borgmatic-bxntey17/borgmatic/zfs_snapshots/./zpool BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_EXIT_CODES=*** /root/borg2.0.0b14 create --debug --show-rc --repo test2.borg {hostname} /root/tmp /tmp/borgmatic-bxntey17/./borgmatic /tmp/borgmatic-bxntey17/borgmatic/zfs_snapshots/./zpool ... /root/tmp/test.yaml: Calling zfs hook function remove_data_source_dumps zfs list -H -o name,mountpoint /root/tmp/test.yaml: Looking for snapshots in /tmp/borgmatic-bxntey17/borgmatic/zfs_snapshots /root/tmp/test.yaml: Unmounting ZFS snapshot at /tmp/borgmatic-bxntey17/borgmatic/zfs_snapshots/zpool umount tmp/borgmatic-bxntey17/borgmatic/zfs_snapshots/zpool zfs list -H -t snapshot -o name /root/tmp/test.yaml: Destroying ZFS snapshot zpool@borgmatic-1994531 zfs destroy -r zpool@borgmatic-1994531 ... # borgmatic -c test.yaml -v 1 list --archive latest test2.borg: Listing archive 38cdd9ba7bcd354b4493d98fe0b7d258bf2f1b87ef9a2733d308abf480ff97a8 ... drwxr-xr-x root root 0 Wed, 2024-11-20 12:05:54 -0800 zpool -rw-r--r-- root root 4 Wed, 2024-11-20 12:05:54 -0800 zpool/file.txt ``` ### Still to do * [x] all TODOs and FIXMEs * [x] do an organizational pass on the code * [x] documentation * [x] get existing unit tests passing again * [x] add tests to cover new/changed code * [x] audit branch diff for missing test coverage * [x] additional error handling * [x] lots more manual testing

witten added 2 commits 2024-11-21 06:39:09 +00:00

ZFS snapshots WIP (#261 ). ab43ef00ce

Comment tweaks and additional TODOs (#261 ). 5e4784991a

witten added 1 commit 2024-11-21 16:32:19 +00:00

Fix broken check action (#261 ). 2af3522902

witten added 1 commit 2024-11-21 16:36:41 +00:00

Use os.path.normpath() instead of custom list comprehension (#261 ). da8278b566

witten added 1 commit 2024-11-21 17:00:42 +00:00

Proceed gracefully in ZFS data source removal if ZFS isn't installed (#261 ). 0ed52bbc4a

witten added 1 commit 2024-11-21 18:25:46 +00:00

Prevent ZFS snapshots from ending up in the Borg archive twice (#261 ). f9e920dce9

witten added 1 commit 2024-11-21 18:52:30 +00:00