LVM snapshots (#80). #949

Merged
witten merged 33 commits from lvm-snapshots into main 2024-12-07 04:21:23 +00:00
Owner

This is currently prototype-quality work for snapshotting LVM logical volumes (#80) from borgmatic such that they end up in a Borg archive at the original logical volume path rather than the snapshot path. That means that if your logical volume is mounted at, say, /my/lvolume, then the snapshotted data shows up in the Borg archive at my/lvolume and not some snapshot directory.

Logical volume discovery currently works by expecting that any volumes to backup are listed in borgmatic's source_directories by their mount paths.

Result

I have run this and it appears to work so far. Here are some logs:

# lsblk --list
NAME                     MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
...
vgroup-lvolume           254:1    0    20M  0 lvm   /mnt/lvolume
...

# borgmatic -c test.yaml -v 2 create
...
1.2.borg: Calling lvm hook function dump_data_sources
1.2.borg: Snapshotting LVM logical volumes
1.2.borg: Creating LVM snapshot vgroup-lvolume_borgmatic-3903923
lvcreate --snapshot --extents 1 --name vgroup-lvolume_borgmatic-3903923 /dev/mapper/vgroup-lvolume
  Logical volume "vgroup-lvolume_borgmatic-3903923" created.
lvs --report-format json --options lv_name,lv_path --select lv_attr =~ ^s
1.2.borg: Mounting LVM snapshot vgroup-lvolume_borgmatic-3903923 at /tmp/borgmatic-nn1giw2w/borgmatic/lvm_snapshots/mnt/lvolume
mount -o ro /dev/vgroup/vgroup-lvolume_borgmatic-3903923 /tmp/borgmatic-nn1giw2w/borgmatic/lvm_snapshots/mnt/lvolume
BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_EXIT_CODES=*** /root/borg1.4.0 create --debug --show-rc 1.2.borg::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /root/tmp /tmp/borgmatic-nn1giw2w/./borgmatic/bootstrap /tmp/borgmatic-nn1giw2w/borgmatic/lvm_snapshots/./mnt/lvolume
...
root/tmp/test.yaml: Calling lvm hook function remove_data_source_dumps
/root/tmp/test.yaml: Looking for snapshots to remove in /tmp/borgmatic-*/borgmatic/lvm_snapshots
/root/tmp/test.yaml: Unmounting LVM snapshot at /tmp/borgmatic-nn1giw2w/borgmatic/lvm_snapshots/mnt/lvolume
umount /tmp/borgmatic-nn1giw2w/borgmatic/lvm_snapshots/mnt/lvolume
lvs --report-format json --options lv_name,lv_path --select lv_attr =~ ^s
/root/tmp/test.yaml: Deleting LVM snapshot vgroup-lvolume_borgmatic-3903923
lvremove --force /dev/vgroup/vgroup-lvolume_borgmatic-3903923
  Logical volume "vgroup-lvolume_borgmatic-3903923" successfully removed.
...

# borgmatic -c test.yaml list --archive latest --path /mnt
1.2.borg: Listing archive flux-2024-12-01T20:17:27.778344
drwxr-xr-x root   root          0 Sat, 2024-11-30 22:09:06 mnt/lvolume
drwx------ root   root          0 Sat, 2024-11-30 22:08:05 mnt/lvolume/lost+found
-rw-r--r-- root   root          4 Sat, 2024-11-30 22:09:06 mnt/lvolume/test.txt

Still to do

  • documentation
  • expand scope a bit to port the parent directory discovery logic to the ZFS and Btrfs hooks
  • do an organizational pass on the code
  • get existing unit tests passing again
  • add tests to cover new/changed code
  • consider adding some end-to-end tests
  • audit branch diff for missing test coverage
  • additional error handling
  • lots more manual testing
This is currently prototype-quality work for snapshotting LVM logical volumes (#80) from borgmatic such that they end up in a Borg archive at the original logical volume path rather than the snapshot path. That means that if your logical volume is mounted at, say, `/my/lvolume`, then the snapshotted data shows up in the Borg archive at `my/lvolume` and not some snapshot directory. Logical volume discovery currently works by expecting that any volumes to backup are listed in borgmatic's `source_directories` by their mount paths. ### Result I have run this and it appears to work so far. Here are some logs: ``` # lsblk --list NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS ... vgroup-lvolume 254:1 0 20M 0 lvm /mnt/lvolume ... # borgmatic -c test.yaml -v 2 create ... 1.2.borg: Calling lvm hook function dump_data_sources 1.2.borg: Snapshotting LVM logical volumes 1.2.borg: Creating LVM snapshot vgroup-lvolume_borgmatic-3903923 lvcreate --snapshot --extents 1 --name vgroup-lvolume_borgmatic-3903923 /dev/mapper/vgroup-lvolume Logical volume "vgroup-lvolume_borgmatic-3903923" created. lvs --report-format json --options lv_name,lv_path --select lv_attr =~ ^s 1.2.borg: Mounting LVM snapshot vgroup-lvolume_borgmatic-3903923 at /tmp/borgmatic-nn1giw2w/borgmatic/lvm_snapshots/mnt/lvolume mount -o ro /dev/vgroup/vgroup-lvolume_borgmatic-3903923 /tmp/borgmatic-nn1giw2w/borgmatic/lvm_snapshots/mnt/lvolume BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_EXIT_CODES=*** /root/borg1.4.0 create --debug --show-rc 1.2.borg::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /root/tmp /tmp/borgmatic-nn1giw2w/./borgmatic/bootstrap /tmp/borgmatic-nn1giw2w/borgmatic/lvm_snapshots/./mnt/lvolume ... root/tmp/test.yaml: Calling lvm hook function remove_data_source_dumps /root/tmp/test.yaml: Looking for snapshots to remove in /tmp/borgmatic-*/borgmatic/lvm_snapshots /root/tmp/test.yaml: Unmounting LVM snapshot at /tmp/borgmatic-nn1giw2w/borgmatic/lvm_snapshots/mnt/lvolume umount /tmp/borgmatic-nn1giw2w/borgmatic/lvm_snapshots/mnt/lvolume lvs --report-format json --options lv_name,lv_path --select lv_attr =~ ^s /root/tmp/test.yaml: Deleting LVM snapshot vgroup-lvolume_borgmatic-3903923 lvremove --force /dev/vgroup/vgroup-lvolume_borgmatic-3903923 Logical volume "vgroup-lvolume_borgmatic-3903923" successfully removed. ... # borgmatic -c test.yaml list --archive latest --path /mnt 1.2.borg: Listing archive flux-2024-12-01T20:17:27.778344 drwxr-xr-x root root 0 Sat, 2024-11-30 22:09:06 mnt/lvolume drwx------ root root 0 Sat, 2024-11-30 22:08:05 mnt/lvolume/lost+found -rw-r--r-- root root 4 Sat, 2024-11-30 22:09:06 mnt/lvolume/test.txt ``` ### Still to do * [x] documentation * [x] expand scope a bit to port the parent directory discovery logic to the ZFS and Btrfs hooks * [x] do an organizational pass on the code * [x] get existing unit tests passing again * [x] add tests to cover new/changed code * [x] consider adding some end-to-end tests * [x] audit branch diff for missing test coverage * [x] additional error handling * [x] lots more manual testing
witten added 1 commit 2024-12-02 04:20:25 +00:00
witten added 1 commit 2024-12-02 04:25:28 +00:00
witten added 1 commit 2024-12-02 05:00:26 +00:00
anarcat reviewed 2024-12-02 17:34:23 +00:00
@ -0,0 +73,4 @@
lvcreate_command,
'--snapshot',
'--extents',
'1', # The snapshot doesn't need much disk space because it's read-only.
First-time contributor

i don't think that works. snapshots take up space even if you don't write to them: writes to the parent volume will write to the snapshot (counter-intuitively, i admit) to keep a copy of the old extent.

at least, that's how i understand it: did you test this to confirm this actually works in production?

i suspect you'll have to make that number of extents configurable, and in fact i would use the --size parameter instead of --extents, unless you want to start parsing units for the user and converting that into extents...

i don't think that works. snapshots take up space even if you don't write to them: writes to the *parent* volume will write to the snapshot (counter-intuitively, i admit) to keep a *copy* of the old extent. at least, that's how i understand it: did you test this to confirm this actually works in production? i suspect you'll have to make that number of extents configurable, and in fact i would use the `--size` parameter instead of `--extents`, unless you want to start parsing units for the user and converting that into extents...
Author
Owner

Works in production? No. Works on my test machine? Yes. So are you thinking just a configuration option for setting the size of snapshots in MB? Would that really work globally for all logical volumes? Or do you think, at the risk of complicating this code, a percentage of logical volume size would work better?

And thanks for taking the time to look at this and weigh in!

Works in production? No. Works on my test machine? Yes. So are you thinking just a configuration option for setting the size of snapshots in MB? Would that really work globally for all logical volumes? Or do you think, at the risk of complicating this code, a percentage of logical volume size would work better? And thanks for taking the time to look at this and weigh in!
First-time contributor

Works in production? No. Works on my test machine? Yes.

I suspect that works for you because you never write enough to fill that
one extent.

So are you thinking just a configuration option for setting the size of snapshots in MB? Would that really work globally for all logical volumes? Or do you think, at the risk of complicating this code, a percentage of logical volume size would work better?

Essentially, yes. It's not necessarily in MB though... You specify the
unit or percentage, so you can do -L 10%ORIGIN to have a snapshot 10% of
the original size, or -L 1G to have a 1G snapshot. I'd pass that
verbatim to lvcreate, essentially.

You could even use 10% as a default, but then the problem is you need
enough free space to cover for that, which is why I think this needs to
be configured by the user.

And thanks for taking the time to look at this and weigh in!

No problem, thanks for taking a bit on that one!

> Works in production? No. Works on my test machine? Yes. I suspect that works for you because you never write enough to fill that one extent. > So are you thinking just a configuration option for setting the size of snapshots in MB? Would that really work globally for all logical volumes? Or do you think, at the risk of complicating this code, a percentage of logical volume size would work better? Essentially, yes. It's not necessarily in MB though... You specify the unit or percentage, so you can do -L 10%ORIGIN to have a snapshot 10% of the original size, or -L 1G to have a 1G snapshot. I'd pass that verbatim to lvcreate, essentially. You could even use 10% as a default, but then the problem is you need enough free space to cover for that, which is why I think this needs to be configured by the user. > And thanks for taking the time to look at this and weigh in! No problem, thanks for taking a bit on that one!
Author
Owner

Sounds good. I'll add that option as you describe, potentially with a default.

Sounds good. I'll add that option as you describe, potentially with a default.
Author
Owner

Okay, implemented!

Okay, implemented!
witten marked this conversation as resolved
witten added 1 commit 2024-12-02 19:09:29 +00:00
witten added 1 commit 2024-12-02 19:29:09 +00:00
anarcat reviewed 2024-12-02 19:42:34 +00:00
@ -148,0 +218,4 @@
Additionally, borgmatic rewrites the snapshot file paths so that they appear
at their original logical volume locations in a Borg archive. For instance, if
your logical volume is mounted at `/mnt/lvolume`, then the snapshotted files
will appear in an archive at `/mnt/lvolume` as well.
First-time contributor

i'm a bit confused by this. what are we rewriting here exactly? the files appear in /mnt/lvolume, exactly as they are in the logical volume...

i would have expected this to say something like:

Additionally, borgmatic rewrites the snapshot file paths so that they appear
at their original logical volume locations in a Borg archive. For instance, if
your logical volume is mounted at `/var`, then the snapshotted files
will appear in an archive at `/var` as well, even if, to perform the backup,
borgmatic will mount the volume in the runtime directory, in (for example) 
`/run/user/1000/lvm_snapshots/`.

that would, at least, clarify things for me.

part of the confusion i have, i think, comes from the use of /mnt in the example: that's a mountpoint i use for temporary things, not a real partition, so i thought this was where borgmatic would mount the snapshot, temporarily.

i'm a bit confused by this. what are we rewriting here exactly? the files appear in `/mnt/lvolume`, exactly as they are in the logical volume... i would have expected this to say something like: ``` Additionally, borgmatic rewrites the snapshot file paths so that they appear at their original logical volume locations in a Borg archive. For instance, if your logical volume is mounted at `/var`, then the snapshotted files will appear in an archive at `/var` as well, even if, to perform the backup, borgmatic will mount the volume in the runtime directory, in (for example) `/run/user/1000/lvm_snapshots/`. ``` that would, at least, clarify things for me. part of the confusion i have, i think, comes from the use of `/mnt` in the example: that's a mountpoint i use for temporary things, not a real partition, so i thought this was where borgmatic would mount the snapshot, temporarily.
Author
Owner

Got it. I really appreciate the feedback. I will clarify!

Got it. I really appreciate the feedback. I will clarify!
witten marked this conversation as resolved
First-time contributor

i can't figure this out from the code, but another concern i would have here is what happens when you (say) backup /var/log and /var/lib in your configuration, and that the logical volume is in /var. does it work? does it make multiple snapshots?

i can't figure this out from the code, but another concern i would have here is what happens when you (say) backup `/var/log` and `/var/lib` in your configuration, and that the logical volume is in `/var`. does it work? does it make multiple snapshots?
witten added 1 commit 2024-12-02 20:02:55 +00:00
Author
Owner

No, that doesn't work currently. borgmatic would try to find a logical volume mounted at /var/log, not find one, try to find a logical volume mounted at /var/lib, not find one, and then ask Borg to backup those two paths without any snapshotting for them.

So at minimum perhaps this should be documented more clearly, so the user knows they need to specify /var if they want it snapshotted as part of backups?

I could maybe see the case for making borgmatic smart enough to detect this case and snapshot /var since it's a common parent directory, but:

  1. I'm not sure the user would even want that 100% of the time, because let's say there's /var/log, /var/lib, and /var/humongous_ever_changing_database. Just because the user wants /var/log and /var/lib backed up doesn't mean they necessarily want to spend snapshot space on their database too.
  2. The code to implement this would be annoying. Not impossible, of course, just annoying to write and maintain.

Thoughts?

No, that doesn't work currently. borgmatic would try to find a logical volume mounted at `/var/log`, not find one, try to find a logical volume mounted at `/var/lib`, not find one, and then ask Borg to backup those two paths without any snapshotting for them. So at minimum perhaps this should be documented more clearly, so the user knows they need to specify `/var` if they want it snapshotted as part of backups? I could _maybe_ see the case for making borgmatic smart enough to detect this case and snapshot `/var` since it's a common parent directory, but: 1. I'm not sure the user would even want that 100% of the time, because let's say there's `/var/log`, `/var/lib`, and `/var/humongous_ever_changing_database`. Just because the user wants `/var/log` and `/var/lib` backed up doesn't mean they necessarily want to spend snapshot space on their database too. 2. The code to implement this would be annoying. Not impossible, of course, just annoying to write and maintain. Thoughts?
First-time contributor
  1. I'm not sure the user would even want that 100% of the time, because let's say there's /var/log, /var/lib, and /var/humongous_ever_changing_database. Just because the user wants /var/log and /var/lib backed up doesn't mean they necessarily want to spend snapshot space on their database too.

i would say that's a footgun i'd be happy to provide users: they asked for that snapshot, just give it to them.

  1. The code to implement this would be annoying. Not impossible, of course, just annoying to write and maintain.

then you might be happy to know that Someone already wrote some of that code for you. it's crap old code, and not for borg, but you might find inspiration there.

i think what you want is the find_mountpoint logic (which is good, pure-python code) and perhaps some find_device stuff (which is f-ing ugly, if you ask me, parsing /proc/mounts is probably more reliable, as the kernel won't break that interface).

then you need to deduplicate those "snapshot that directory" queries somehow.

but yes, this is the kind of stuff that made me not implement this myself, because i knew how tricky it is.

> 1. I'm not sure the user would even want that 100% of the time, because let's say there's `/var/log`, `/var/lib`, and `/var/humongous_ever_changing_database`. Just because the user wants `/var/log` and `/var/lib` backed up doesn't mean they necessarily want to spend snapshot space on their database too. i would say that's a footgun i'd be happy to provide users: they *asked* for that snapshot, just give it to them. > 2. The code to implement this would be annoying. Not impossible, of course, just annoying to write and maintain. then you might be happy to know that [Someone already wrote some of that code for you](https://gitlab.com/anarcat/bup-cron/-/blob/0ff9b75937a5e75da5a5b3cf2ec877cc8c7798f1/bup_cron/__init__.py#L296). it's crap old code, and not for borg, but you might find inspiration there. i think what you want is the find_mountpoint logic (which is good, pure-python code) and perhaps some `find_device` stuff (which is f-ing ugly, if you ask me, parsing `/proc/mounts` is probably more reliable, as the kernel won't break that interface). then you need to deduplicate those "snapshot that directory" queries somehow. but yes, this is the kind of stuff that made *me* not implement this myself, because i knew how tricky it is.
Author
Owner

Okay, thanks for the link. I'll look into what it would take to add this feature. No promises. 😄

Okay, thanks for the link. I'll look into what it would take to add this feature. No promises. 😄
witten added 1 commit 2024-12-03 04:59:24 +00:00
Author
Owner

Well it actually wasn't too bad. Didn't even have to dedup since everything was already in terms of logical volumes. It'll need lots more testing though. And I'll probably want to implement something similar for ZFS and Btrfs.

Well it actually wasn't too bad. Didn't even have to dedup since everything was already in terms of logical volumes. It'll need lots more testing though. And I'll probably want to implement something similar for ZFS and Btrfs.
witten added 1 commit 2024-12-03 05:01:48 +00:00
First-time contributor

fantastic! i'll see if i can find the time to actually test this stuff.. i have a server in the lab here that i'm running borg backups again (not with borgmatic, i'm ashamed to say) that keeps failing because a stupid log file "has changed during backup", hopefully that will knock down that annoyance!

fantastic! i'll see if i can find the time to actually test this stuff.. i have a server in the lab here that i'm running borg backups again (not with borgmatic, i'm ashamed to say) that keeps failing because a stupid log file "has changed during backup", hopefully that will knock down that annoyance!
Author
Owner

Sounds like a perfect test candidate! I'm sure I'll flush out some more bugs as well with local testing here + automated testing.

Sounds like a perfect test candidate! I'm sure I'll flush out some more bugs as well with local testing here + automated testing.
witten added 2 commits 2024-12-03 16:52:19 +00:00
witten added 1 commit 2024-12-03 19:06:26 +00:00
witten added 1 commit 2024-12-03 19:12:53 +00:00
witten added 1 commit 2024-12-03 20:15:51 +00:00
witten added 1 commit 2024-12-03 20:22:57 +00:00
witten added 1 commit 2024-12-03 23:44:14 +00:00
witten added 1 commit 2024-12-03 23:56:35 +00:00
witten added 1 commit 2024-12-04 03:13:34 +00:00
witten added 1 commit 2024-12-04 03:20:21 +00:00
witten added 1 commit 2024-12-04 22:48:29 +00:00
witten added 1 commit 2024-12-04 23:39:37 +00:00
witten added 1 commit 2024-12-04 23:43:31 +00:00
witten added 1 commit 2024-12-05 04:23:15 +00:00
witten added 1 commit 2024-12-05 04:34:21 +00:00
witten added 1 commit 2024-12-05 19:19:37 +00:00
witten added 1 commit 2024-12-06 01:36:23 +00:00
witten added 1 commit 2024-12-06 06:47:02 +00:00
witten added 1 commit 2024-12-06 17:39:51 +00:00
witten added 2 commits 2024-12-06 18:28:01 +00:00
witten added 1 commit 2024-12-06 22:00:06 +00:00
witten added 1 commit 2024-12-07 00:03:04 +00:00
witten added 1 commit 2024-12-07 00:06:03 +00:00
witten added 1 commit 2024-12-07 00:12:20 +00:00
witten changed title from WIP: LVM snapshots (#80). to LVM snapshots (#80). 2024-12-07 04:20:53 +00:00
witten merged commit 62d67cde0a into main 2024-12-07 04:21:23 +00:00
witten deleted branch lvm-snapshots 2024-12-07 04:21:39 +00:00
Sign in to join this conversation.
No Reviewers
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#949
No description provided.