Initial work on a Btrfs hook (#251). #946

witten · 2024-11-29T02:56:59Z

witten commented

2024-11-29 02:56:59 +00:00

This is currently beta-quality work for snapshotting Btrfs subvolumes (#251) from borgmatic such that they end up in a Borg archive at the original subvolume path rather than the snapshot path. That means that if your subvolume is mounted at, say, /my/subvolume, then the snapshotted data shows up in the Borg archive at my/subvolume and not my/subvlume/some-snapshot-directory.

Subvolume discovery currently works by expecting that any subvolumes to backup are listed in borgmatic's source_directories by their paths.

Result

I have run this and it appears to work so far. Here are some logs:

# findmnt -t btrfs
TARGET SOURCE     FSTYPE OPTIONS
/mnt   /dev/loop0 btrfs  rw,relatime,ssd,space_cache=v2,subvolid=5,subvol=/

# btrfs subvolume list /mnt
ID 270 gen 95 top level 5 path subvol
ID 272 gen 74 top level 5 path foo/bar
ID 273 gen 73 top level 5 path snap
ID 274 gen 70 top level 5 path subvolsnap

# borgmatic -c test.yaml -v 2 create
...
1.2.borg: Calling btrfs hook function dump_data_sources
1.2.borg: Snapshotting Btrfs datasets
findmnt -nt btrfs
btrfs subvolume list /mnt
1.2.borg: Creating Btrfs snapshot for /mnt/subvol subvolume
btrfs subvolume snapshot /mnt/subvol /mnt/subvol/.borgmatic-snapshot-3424394/./mnt/subvol
Create snapshot of '/mnt/subvol' in '/mnt/subvol/.borgmatic-snapshot-3424394/./mnt/subvol'
BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_EXIT_CODES=*** /root/borg1.4.0 create --exclude-from /tmp/tmputbs3yf6 --debug --show-rc 1.2.borg::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /mnt/subvol/.borgmatic-snapshot-3424394/./mnt/subvol /root/tmp /tmp/borgmatic-y2uwzkra/./borgmatic/bootstrap
...
/root/tmp/test.yaml: Calling btrfs hook function remove_data_source_dumps
findmnt -nt btrfs
btrfs subvolume list /mnt
/root/tmp/test.yaml: Looking for snapshots to remove in /mnt/.borgmatic-snapshot-*/mnt
/root/tmp/test.yaml: Looking for snapshots to remove in /mnt/subvol/.borgmatic-snapshot-*/mnt/subvol
/root/tmp/test.yaml: Deleting Btrfs snapshot /mnt/subvol/.borgmatic-snapshot-3424394/mnt/subvol
btrfs subvolume delete /mnt/subvol/.borgmatic-snapshot-3424394/mnt/subvol
Delete subvolume 285 (no-commit): '/mnt/subvol/.borgmatic-snapshot-3424394/mnt/subvol'
/root/tmp/test.yaml: Looking for snapshots to remove in /mnt/foo/bar/.borgmatic-snapshot-*/mnt/foo/bar
/root/tmp/test.yaml: Looking for snapshots to remove in /mnt/snap/.borgmatic-snapshot-*/mnt/snap
/root/tmp/test.yaml: Looking for snapshots to remove in /mnt/subvolsnap/.borgmatic-snapshot-*/mnt/subvolsnap
/root/tmp/test.yaml: Looking for snapshots to remove in /mnt/subvol/.borgmatic-snapshot-*/mnt/subvol/.borgmatic-snapshot-*/mnt/subvol/.borgmatic-snapshot-*/mnt/subvol
...

# borgmatic -c test.yaml list --archive latest --path /mnt
1.2.borg: Listing archive flux-2024-11-28T18:51:09.264658
drwxr-xr-x root   root          0 Thu, 2024-11-28 18:51:08 mnt/subvol
-rw-r--r-- root   root          4 Thu, 2024-11-28 17:38:06 mnt/subvol/sub.txt

Still to do

do an organizational pass on the code
documentation
get existing unit tests passing again
add tests to cover new/changed code
audit branch diff for missing test coverage
additional error handling
lots more manual testing

This is currently beta-quality work for snapshotting Btrfs subvolumes (#251) from borgmatic such that they end up in a Borg archive at the original subvolume path rather than the snapshot path. That means that if your subvolume is mounted at, say, `/my/subvolume`, then the snapshotted data shows up in the Borg archive at `my/subvolume` and not `my/subvlume/some-snapshot-directory`. Subvolume discovery currently works by expecting that any subvolumes to backup are listed in borgmatic's `source_directories` by their paths. ### Result I have run this and it appears to work so far. Here are some logs: ``` # findmnt -t btrfs TARGET SOURCE FSTYPE OPTIONS /mnt /dev/loop0 btrfs rw,relatime,ssd,space_cache=v2,subvolid=5,subvol=/ # btrfs subvolume list /mnt ID 270 gen 95 top level 5 path subvol ID 272 gen 74 top level 5 path foo/bar ID 273 gen 73 top level 5 path snap ID 274 gen 70 top level 5 path subvolsnap # borgmatic -c test.yaml -v 2 create ... 1.2.borg: Calling btrfs hook function dump_data_sources 1.2.borg: Snapshotting Btrfs datasets findmnt -nt btrfs btrfs subvolume list /mnt 1.2.borg: Creating Btrfs snapshot for /mnt/subvol subvolume btrfs subvolume snapshot /mnt/subvol /mnt/subvol/.borgmatic-snapshot-3424394/./mnt/subvol Create snapshot of '/mnt/subvol' in '/mnt/subvol/.borgmatic-snapshot-3424394/./mnt/subvol' BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_EXIT_CODES=*** /root/borg1.4.0 create --exclude-from /tmp/tmputbs3yf6 --debug --show-rc 1.2.borg::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /mnt/subvol/.borgmatic-snapshot-3424394/./mnt/subvol /root/tmp /tmp/borgmatic-y2uwzkra/./borgmatic/bootstrap ... /root/tmp/test.yaml: Calling btrfs hook function remove_data_source_dumps findmnt -nt btrfs btrfs subvolume list /mnt /root/tmp/test.yaml: Looking for snapshots to remove in /mnt/.borgmatic-snapshot-*/mnt /root/tmp/test.yaml: Looking for snapshots to remove in /mnt/subvol/.borgmatic-snapshot-*/mnt/subvol /root/tmp/test.yaml: Deleting Btrfs snapshot /mnt/subvol/.borgmatic-snapshot-3424394/mnt/subvol btrfs subvolume delete /mnt/subvol/.borgmatic-snapshot-3424394/mnt/subvol Delete subvolume 285 (no-commit): '/mnt/subvol/.borgmatic-snapshot-3424394/mnt/subvol' /root/tmp/test.yaml: Looking for snapshots to remove in /mnt/foo/bar/.borgmatic-snapshot-*/mnt/foo/bar /root/tmp/test.yaml: Looking for snapshots to remove in /mnt/snap/.borgmatic-snapshot-*/mnt/snap /root/tmp/test.yaml: Looking for snapshots to remove in /mnt/subvolsnap/.borgmatic-snapshot-*/mnt/subvolsnap /root/tmp/test.yaml: Looking for snapshots to remove in /mnt/subvol/.borgmatic-snapshot-*/mnt/subvol/.borgmatic-snapshot-*/mnt/subvol/.borgmatic-snapshot-*/mnt/subvol ... # borgmatic -c test.yaml list --archive latest --path /mnt 1.2.borg: Listing archive flux-2024-11-28T18:51:09.264658 drwxr-xr-x root root 0 Thu, 2024-11-28 18:51:08 mnt/subvol -rw-r--r-- root root 4 Thu, 2024-11-28 17:38:06 mnt/subvol/sub.txt ``` ### Still to do * [x] do an organizational pass on the code * [x] documentation * [x] get existing unit tests passing again * [x] add tests to cover new/changed code * [x] audit branch diff for missing test coverage * [x] additional error handling * [x] lots more manual testing

witten added 1 commit 2024-11-29 02:57:00 +00:00

Initial work on a Btrfs hook (#251 ). b5b5c1fafa

witten referenced this pull request

2024-11-29 02:58:49 +00:00

Integration with BTRFS Snapshot #251

witten added 1 commit 2024-11-29 04:32:36 +00:00

Btrfs hook documentation (#251 ). 3f901c0a52

witten added 1 commit 2024-11-29 06:19:01 +00:00

Create Btrfs snapshots as read-only (#251 ). d4a02f73b5

witten added 1 commit 2024-11-29 17:37:11 +00:00

Improve Btrfs hook factoring/organization (#251 ). 84a0552277

witten added 1 commit 2024-11-30 17:33:09 +00:00