#80 LVM filesystem snapshotting

The only feature borgmatic is missing for our workflow is a way to backup libvirt’s virtual disks from running virtual machines (we use LVM). borg already has all we need with the --read-special argument, it would be just a matter of creating a read-only snapshot and then remove it when we are done.

We could have something like this in the config file:

source_logical_volumes:
    # By path
    - /dev/vg1/mercury.planets.org-disk

    # By name
    - venus.planets.org-disk

    # By partition within the volume, automatically using kpartx? (maybe that's too much)
    - earth.planets.org-disk@2

Or perhaps be fancy and backup whatever disk a VM is configured with:

source_libvirt:
    # Backup all of the 'mars.planets.org' disks
    - mars.planets.org

@witten I’m willing to write a patch if you like the feature.

Just noticed #5.

I don’t think hooks are ergonomic for this use case:

• Duplication. You would have to put the config for your volume in three different places (source_directories, before_backup and after_backup)
• The chance of errors increases. If you forget to freeze a snapshot in before_backup everything seems to work, until you find out your backup is possibily corrupt when you try to restore it.
• No way to know what you’re backing up (AFAIK). I can’t seem to find the documentation for user hooks, but it looks like hooks are ran once per repository. It will be actually better to freeze and destroy the snapshot on a per source basis, because we want to get rid of it as soon as possible since it can fill up and be no longer valid.
• Will the backup run if the script in before_backup fails? If it does, it’s scenario two all over again without even realizing it.

I’m not that familiar with the libvirt backup use case. Based on what you’ve said so far, I’m gathering that it’s maybe something along these lines:

1. Create a snapshot.
2. Invoke borg --read-special ..., providing the path to the snapshot.
3. Destroy the snapshot.

Is that about right?

As for your other points:

• Duplication: Is this because the path or name for your snapshot needs to be mentioned in all three places (source_directories, before_backup and after_backup)? Or is there additional config you’d need to duplicate?
• Hooks are actually run once per config file, before all repositories are backed up, and after all repositories are backed up. I’ve filed #81 to better document hooks.
• I believe that the backup script will not run if the before_backup hook fails. It’ll instead skip backups, run the on_error hook (if any), and exits with an error.

Brainstorming on this, I can think of a couple of classes of solutions:

1. Make use of the existing hooks, and deal with the duplication.
2. Make use of the existing hooks, and somehow eliminate the duplication via parameterization of some sort. This might be difficult, given that you probably don’t want to do the snapshot create/destroy on all sources.
3. Add a feature to support before/after hooks on individual sources, perhaps with the name of the sources parameterized.
4. Add actual support for snapshots, something like your example above.

Let me know your thoughts.

One more type of solution:

5. Add support for snapshots, but instead of having a separate source_logical_volumes or source_libvert as per your example, just allow listing of logical volumes (by path?) directly in source_directories. If borgmatic detects that one such entry is actually a logical volume, it will automatically do the snapshop create/destroy for it!

Important question though for many of these options: Does the create/destroy need to be in any way configurable? Or is it pretty standard? What’s the command invocation look like for each of create and destroy?

1. Create a snapshot.
2. Invoke borg --read-special …, providing the path to the snapshot.
3. Destroy the snapshot.

Is that about right?

That’s exactly right.

You create a snapshot with: lvcreate --size SIZE --snapshot --name SNAPSHOT_NAME /dev/mapper/mercury.planets.org--disk

and remove it with lvremove /dev/mapper/SNAPSHOT_NAME

• Duplication: Is this because the path or name for your snapshot needs to be mentioned in all three places (source_directories, before_backup and after_backup)? Or is there additional config you?d need to duplicate?

No, there’s no additional config, my concern was just for those three places we’d have to maintain.

1. Add a feature to support before/after hooks on individual sources, perhaps with the name of the sources parameterized.

That would be nice and useful for other scenarios as well. Something like

 before_backup: echo $source $repository

1. Add support for snapshots, but instead of having a separate source_logical_volumes or source_libvert as per your example, just allow listing of logical volumes (by path?) directly in source_directories. If borgmatic detects that one such entry is actually a logical volume, it will automatically do the snapshop create/destroy for it!

My first reaction is “cool”, my second reaction is “it may be tricky to detect a logical volume from its path” (we’d need to check against its major and minor number, I guess) and “what happens if this heuristic fails”. I suppose it will silently fall back to a normal backup without freezing the volume.

Important question though for many of these options: Does the create/destroy need to be in any way configurable? Or is it pretty standard? What?s the command invocation look like for each of create and destroy?

The only thing that we can configure is the size of the snapshot device, which is basically the volume of changed data on the original volume the snapshot can handle before being invalidated. I think we can safely calculate that from the volume size for the most common case, or maybe use a reasonable percentage of the available size.

Okay, having had a chance to poke around with this a bit, I’m inclined to agree that “it may be tricky to detect a logical volume from it’s path”. Therefore I think your original suggestion of source_logical_volumes: sounds good to me. If you still are interested in implementing this, please feel free to take a crack at it. Otherwise, let me know. And thanks for walking me through this!

Great, I’ll come up with a patch.

On a second thought this has become “Support LVM volumes backup”, do we want to support qcow2 and raw like backup-vm does?

Does the lvcreate command differ based on the VM disk image format? If so, how? Or would you use a different command in each case?

Sorry about the confusion. lvcreate just creates LVM volumes (and their snapshots) which are block devices that may or may not be used as backend storage for virtual machines.

Virtualization platforms also support file based disk images formats such as qcow2 and raw.

Gotcha. Those seem like a reasonable thing to support, either as part of this or separately. Is your thinking that you would support generic libvirt backups, and thereby target all the various formats?

I was thinking about implementing just the LVM part for now, as it’s what I need for our workflow.

Sounds good!

I also wanted --read-special support for a similar purpose - to backup inactive disk partitions as well as LVM snapshot volumes. So, I forked borgmatic and added a read_special option to the location part of the yaml schema. The rest I do with hooks.

I can make it available on GitHub if you think it’s worth merging? Relatively straightforward to do though.

The default config is off, of course, and the template comments warn about its use, although I feel the warning might need to be stronger, and reference the borg docs.

Sure, I’d be happy to take a look at the fork and potentially merge it! It may not satisfy all of @bfabio’s asks in this ticket, but it could be a start.

Added a pull request on GitHub https://github.com/witten/borgmatic/pull/25 for your consideration.

I’m also happy to discuss my before and after backup hooks if @bfabio would like, although I don’t consider them particularly fit for general use, as they’re just some shell-fu that make certain naming convention assumptions, and don’t deal with sync issues fully.

Thanks, merged!

Also, let me know if you’d like an entry in AUTHORS (or submit a PR yourself for that).

My name’s in the commit log. That’s enough for me ;)

Sounds good. :)

The --read-special support has just been released as part of borgmatic 1.2.3.

@stevekerrison Thanks for your work. Do you think we can solve this with hooks?

1. Add support for snapshots, but instead of having a separate source_logical_volumes or source_libvert as per your example, just allow listing of logical volumes (by path?) directly in source_directories. If borgmatic detects that one such entry is actually a logical volume, it will automatically do the snapshop create/destroy for it!

As I explained in #231, that is definitely possible. I implemented this in bup-cron and detailed how that works in the other issue addressing specifically filesystem snapshots.

I’d argue that borgmatic should take any path and not assume they are directories. If they are filesystems, or block devices to be more specific, that can be detected as well, and snapshots can be made. But if you’re going to implement snapshots, I’d say: do it for everyone and allow users to snapshot /home if their filesystem setup allow it! :)

And, by the way, many linux distributions (e.g. Fedora and derivatives) ship with btrfs as a filesystem, so you do have snapshotting capability out of the box, assuming you have free space I guess. I never worked with that, so I can’t say more on the topic, but it would be interesting to support that…

I’m willing to explore automatic logical volume / mount point discovery, especially if there’s prior art for that approach. If the heuristic can be made to work reliably, then great. If not, then maybe we’ll need to resort to something more explicit as discussed above.

I’d still probably start with support for one type of snapshotting tech (say, LVM), even if ultimately I’d like to support all the things (btrfs, zfs, qcow, etc etc.).

Okay, I’m merging #231 into this ticket and re-casting this ticket as “start with LVM support”. The idea is that the design can lay the groundwork for subsequent filesystem support to follow. Since #231 has some excellent discussion as well, please refer to it for ideas on the automatic discovery/probing approach.

I manually played around with LVM on a spare USB drive (since my machine isn’t using LVM on the main disk). I have a better idea of how it all works now. Seems pretty straightforward to do snapshotting, although as outlined in this comment, the devil is in the details. And in the discovery in particular.

Pseudo-code for volume probing:

# e.g.: {'/mnt': '/dev/mapper/lvol0', ...}
mount_point_devices = {disk.mountpoint: disk.device for disk in psutil.disk_partitions()}

# e.g.: {'/dev/mapper/lvol0': '100.00m', ...}
logical_volume_device_sizes = {
    device_path: size
    for line in subprocess.check_output('lvs -o DMpath,lv_size --noheadings')
    for (device_path, size) in (line.strip().split(),)
}

for directory in configured_borgmatic_source_directories:
    logical_volume_device_path = find_device_for_path(directory, mount_point_devices)
    size = logical_volume_device_sizes.get(logical_volume_device_path)

    if not size:
        continue  # Not an LVM path. Ultimately could do lookups for btrfs, zfs, etc. here.

    maybe_calculate_a_percentage_of_the_size_for_the_snapshot(size)
    subprocess.check_output(
        'lvcreate --size {} --snapshot --permission r --name whatever {}'.format(
            size, logical_volume_device_path  # TODO: Might need to map from DMPath to lv_path here.
        )
    )

Some more thoughts beyond just volume probing: Once we have collected all of the source directories that need snapshotting, we’re potentially faced with a bit of challenge. Let’s say we’ve got a list of source directories in borgmatic’s config. Some of them are backed by LVM (and therefore can be snapshotted), and some of them aren’t (and therefore can’t be snapshotted). If we want to backup the combined set of LVM and non-LVM source directories, we’ve got a few options:

1. Rewrite LVM-backed source directory paths to their snapshot counterparts, and leave non-LVM source directories alone. This means that, for instance, borgmatic would tell Borg to backup ~/.borgmatic/snapshots/some-source (an LVM snapshot of /etc/some-source) and /etc/another-source (non-LVM source). And then when you go to restore, you’d have to deal with that mixed hierarchy, and potentially move files around per-source.
2. Rewrite all source directory paths to be in a common hierarchy regardless of whether they’re snapshottable. For example, borgmatic would tell Borg to backup ~/.borgmatic/files/some-source (an LVM snapshot of /etc/some-source) and ~/.borgmatic/files/another-source (a mount --rbind bind mount of /etc/another-source). That way, when you go to restore, all source directories are restored into a common hierarchy regardless of whether they came from LVM snapshots or not.
3. Other options?

Also, how common might it be to have mixed snapshottable and non-snapshottable source directories? Is this an edge case worth supporting?

I’ll also note that even with multiple different filesystem types that all support snapshotting (btrfs, zfs, etc.) we’d still have to mount all of their snapshots somewhere other than at the original source directory locations.

this was a tricky situation in bup-cron as well. if i remember correctly, i was able to remap things around so that the backup looked like /var was really /var and not /snap/var or whatever.. but i don’t quite remember how that worked…

this is one of the reasons why i wanted to implement this natively in borg: having support from the Archiver that crawls the archive would fix this problem upstream. but alas, thomas didn’t want to hear about it… :( maybe he could be convinced to change his mind, however for me that ship has sailed a long time ago. ;)

this was a tricky situation in bup-cron as well. if i remember correctly, i was able to remap things around so that the backup looked like /var was really /var and not /snap/var or whatever.. but i don’t quite remember how that worked…

Yeah, that seems potentially problematic.. Because if you temporary mount the snapshot of /var over the actual /var, then any running processes relying on the actual /var will be reading outdated files.. and unable to write at all, since it’s presumably a read-only snapshot.

Maybe there’s some way to create mounts such that they’re only visible to certain processes?

I’d also love to punt on this problem entirely if I can get away with not supporting mixing snapshottable and non-snapshottable directories.

this is one of the reasons why i wanted to implement this natively in borg: having support from the Archiver that crawls the archive would fix this problem upstream. but alas, thomas didn’t want to hear about it… :( maybe he could be convinced to change his mind, however for me that ship has sailed a long time ago. ;)

That would be ideal, yeah. Point me at a Borg ticket if there is one!

Maybe there’s some way to create mounts such that they’re only visible to certain processes?

Well, that was easy:

$ sudo unshare -m /bin/bash -c "mount --bind /root/tmp /mnt && mount | grep /mnt && sleep 60"
/dev/mapper/cryptroot on /mnt type ext4 (rw,noatime)

Then, while that’s sleeping, in a separate terminal:

$ sudo mount | grep /mnt
$

Meaning that the mount is only visible to the first shell.

namespaces are, indeed, an elegant solution for this, but require root unless you got the right capabilities…

Isn’t root required anyway for lvcreate?

Isn’t root required anyway for lvcreate?

true.

Hello,

Just found out about this project, and i wanted to add another filesystem to the list of those who allows snapshots : ZFS.

In my use cases i actually have both ZFS and LVM, and i’d love to be able to backup them using snapshots.

I wanted to mention ZFS as well so maybe the solution is designed with all 3 in mind (even if the support for all comes later)

Thanks !

Quick question while you’re here.. Do you do all of your snapshotting directly with ZFS, and simply use LVM use under the hood incidentally? Or do you do all of your snapshotting with LVM? Thanks!

I don’t use ZFS right now, but if I would, I would not also use LVM. ZFS has all the features of LUKS, ext4, LVM and mdadm put together, so I would only use ZFS.

+1 @anarcat I do not use both ZFS and LVM on the same machine, those are separate uses cases. ZFS has it’s own mecanism for snapshots.

Like BTRFS and LVM you have one command to take the snapshot, and another to remove it.

Maybe the integration can be simplified to only be a way to pass configured volumes (the original path) to backup to before/after scripts, and simply allow before script to pass another folder (the snapshot) instead to borg ?

Thanks for clarifying. I’ll file a separate ticket for ZFS support!

Interesting idea about making before/after scripts an official integration point for various filesystem snapshots. They would probably need to be separate hooks from the existing before_backup / after_backup hooks, just because those are for running arbitrary commands and don’t have to produce any paths. I’ll keep this approach in mind as an option when designing the general snapshot feature.

ZFS ticket: #261