LVM filesystem snapshotting #80

New Issue

Closed

opened 2018-07-28 19:40:51 +00:00 by bfabio · 45 comments

bfabio commented 2018-07-28 19:40:51 +00:00

Copy Link

The only feature borgmatic is missing for our workflow is a way to backup libvirt's virtual disks from running virtual machines (we use LVM). borg already has all we need with the --read-special argument, it would be just a matter of creating a read-only snapshot and then remove it when we are done.

We could have something like this in the config file:

source_logical_volumes:
    # By path
    - /dev/vg1/mercury.planets.org-disk

    # By name
    - venus.planets.org-disk

    # By partition within the volume, automatically using kpartx? (maybe that's too much)
    - earth.planets.org-disk@2

Or perhaps be fancy and backup whatever disk a VM is configured with:

source_libvirt:
    # Backup all of the 'mars.planets.org' disks
    - mars.planets.org

@witten I'm willing to write a patch if you like the feature.

The only feature borgmatic is missing for our workflow is a way to backup libvirt's virtual disks from running virtual machines (we use LVM). `borg` already has all we need with the `--read-special` argument, it would be just a matter of creating a read-only snapshot and then remove it when we are done. We could have something like this in the config file: ```yaml source_logical_volumes: # By path - /dev/vg1/mercury.planets.org-disk # By name - venus.planets.org-disk # By partition within the volume, automatically using kpartx? (maybe that's too much) - earth.planets.org-disk@2 ``` Or perhaps be fancy and backup whatever disk a VM is configured with: ```yaml source_libvirt: # Backup all of the 'mars.planets.org' disks - mars.planets.org ``` @witten I'm willing to write a patch if you like the feature.

bfabio commented 2018-07-28 20:17:14 +00:00

Author

Copy Link

Just noticed #5.

I don't think hooks are ergonomic for this use case:

• Duplication. You would have to put the config for your volume in three different places (source_directories, before_backup and after_backup)
• The chance of errors increases. If you forget to freeze a snapshot in before_backup everything seems to work, until you find out your backup is possibily corrupt when you try to restore it.
• No way to know what you're backing up (AFAIK). I can't seem to find the documentation for user hooks, but it looks like hooks are ran once per repository. It will be actually better to freeze and destroy the snapshot on a per source basis, because we want to get rid of it as soon as possible since it can fill up and be no longer valid.
• Will the backup run if the script in before_backup fails? If it does, it's scenario two all over again without even realizing it.

Just noticed #5. I don't think hooks are ergonomic for this use case: - Duplication. You would have to put the config for your volume in three different places (`source_directories`, `before_backup` and `after_backup`) - The chance of errors increases. If you forget to freeze a snapshot in `before_backup` everything seems to work, until you find out your backup is possibily corrupt when you try to restore it. - No way to know what you're backing up (AFAIK). I can't seem to find the documentation for user hooks, but it looks like hooks are ran once per repository. It will be actually better to freeze and destroy the snapshot on a per source basis, because we want to get rid of it as soon as possible since it can fill up and be no longer valid. - Will the backup run if the script in `before_backup` fails? If it does, it's scenario two all over again without even realizing it.

witten commented 2018-07-28 20:58:11 +00:00

Owner

Copy Link

I'm not that familiar with the libvirt backup use case. Based on what you've said so far, I'm gathering that it's maybe something along these lines:

1. Create a snapshot.
2. Invoke borg --read-special ..., providing the path to the snapshot.
3. Destroy the snapshot.

Is that about right?

As for your other points:

• Duplication: Is this because the path or name for your snapshot needs to be mentioned in all three places (source_directories, before_backup and after_backup)? Or is there additional config you'd need to duplicate?
• Hooks are actually run once per config file, before all repositories are backed up, and after all repositories are backed up. I've filed #81 to better document hooks.
• I believe that the backup script will not run if the before_backup hook fails. It'll instead skip backups, run the on_error hook (if any), and exits with an error.

Brainstorming on this, I can think of a couple of classes of solutions:

1. Make use of the existing hooks, and deal with the duplication.
2. Make use of the existing hooks, and somehow eliminate the duplication via parameterization of some sort. This might be difficult, given that you probably don't want to do the snapshot create/destroy on all sources.
3. Add a feature to support before/after hooks on individual sources, perhaps with the name of the sources parameterized.
4. Add actual support for snapshots, something like your example above.

Let me know your thoughts.

I'm not that familiar with the libvirt backup use case. Based on what you've said so far, I'm gathering that it's maybe something along these lines: 1. Create a snapshot. 2. Invoke `borg --read-special ...`, providing the path to the snapshot. 3. Destroy the snapshot. Is that about right? As for your other points: * Duplication: Is this because the path or name for your snapshot needs to be mentioned in all three places (`source_directories`, `before_backup` and `after_backup`)? Or is there additional config you'd need to duplicate? * Hooks are actually run once per config file, before all repositories are backed up, and after all repositories are backed up. I've filed #81 to better document hooks. * I believe that the backup script will not run if the `before_backup` hook fails. It'll instead skip backups, run the `on_error` hook (if any), and exits with an error. Brainstorming on this, I can think of a couple of classes of solutions: 1. Make use of the existing hooks, and deal with the duplication. 2. Make use of the existing hooks, and somehow eliminate the duplication via parameterization of some sort. This might be difficult, given that you probably don't want to do the snapshot create/destroy on all sources. 3. Add a feature to support before/after hooks on individual sources, perhaps with the name of the sources parameterized. 4. Add actual support for snapshots, something like your example above. Let me know your thoughts.

witten commented 2018-07-28 21:02:27 +00:00

Owner

Copy Link

One more type of solution:

5. Add support for snapshots, but instead of having a separate source_logical_volumes or source_libvert as per your example, just allow listing of logical volumes (by path?) directly in source_directories. If borgmatic detects that one such entry is actually a logical volume, it will automatically do the snapshop create/destroy for it!

Important question though for many of these options: Does the create/destroy need to be in any way configurable? Or is it pretty standard? What's the command invocation look like for each of create and destroy?

One more type of solution: `5.` Add support for snapshots, but instead of having a separate `source_logical_volumes` or `source_libvert` as per your example, just allow listing of logical volumes (by path?) directly in `source_directories`. If borgmatic detects that one such entry is actually a logical volume, it will automatically do the snapshop create/destroy for it! Important question though for many of these options: Does the create/destroy need to be in any way configurable? Or is it pretty standard? What's the command invocation look like for each of create and destroy?

bfabio commented 2018-08-01 12:34:36 +00:00

Author

Copy Link

1. Create a snapshot.
2. Invoke borg --read-special ..., providing the path to the snapshot.
3. Destroy the snapshot.

Is that about right?

That's exactly right.

You create a snapshot with: lvcreate --size SIZE --snapshot --name SNAPSHOT_NAME /dev/mapper/mercury.planets.org--disk

and remove it with lvremove /dev/mapper/SNAPSHOT_NAME

• Duplication: Is this because the path or name for your snapshot needs to be mentioned in all three places (source_directories, before_backup and after_backup)? Or is there additional config you?d need to duplicate?

No, there's no additional config, my concern was just for those three places we'd have to maintain.

3. Add a feature to support before/after hooks on individual sources, perhaps with the name of the sources parameterized.

That would be nice and useful for other scenarios as well. Something like

before_backup: echo $source $repository

5. Add support for snapshots, but instead of having a separate source_logical_volumes or source_libvert as per your example, just allow listing of logical volumes (by path?) directly in source_directories. If borgmatic detects that one such entry is actually a logical volume, it will automatically do the snapshop create/destroy for it!

My first reaction is "cool", my second reaction is "it may be tricky to detect a logical volume from its path" (we'd need to check against its major and minor number, I guess) and "what happens if this heuristic fails". I suppose it will silently fall back to a normal backup without freezing the volume.

Important question though for many of these options: Does the create/destroy need to be in any way configurable? Or is it pretty standard? What?s the command invocation look like for each of create and destroy?

The only thing that we can configure is the size of the snapshot device, which is basically the volume of changed data on the original volume the snapshot can handle before being invalidated. I think we can safely calculate that from the volume size for the most common case, or maybe use a reasonable percentage of the available size.

> 1. Create a snapshot. > 2. Invoke borg --read-special ..., providing the path to the snapshot. > 3. Destroy the snapshot. > Is that about right? That's exactly right. You create a snapshot with: `lvcreate --size SIZE --snapshot --name SNAPSHOT_NAME /dev/mapper/mercury.planets.org--disk` and remove it with `lvremove /dev/mapper/SNAPSHOT_NAME` > * Duplication: Is this because the path or name for your snapshot needs to be mentioned in all three places (source_directories, before_backup and after_backup)? Or is there additional config you?d need to duplicate? No, there's no additional config, my concern was just for those three places we'd have to maintain. > 3. Add a feature to support before/after hooks on individual sources, perhaps with the name of the sources parameterized. That would be nice and useful for other scenarios as well. Something like ``` before_backup: echo $source $repository ``` > 5. Add support for snapshots, but instead of having a separate source_logical_volumes or source_libvert as per your example, just allow listing of logical volumes (by path?) directly in source_directories. If borgmatic detects that one such entry is actually a logical volume, it will automatically do the snapshop create/destroy for it! My first reaction is "cool", my second reaction is "it may be tricky to detect a logical volume from its path" (we'd need to check against its major and minor number, I guess) and "what happens if this heuristic fails". I suppose it will silently fall back to a normal backup without freezing the volume. > Important question though for many of these options: Does the create/destroy need to be in any way configurable? Or is it pretty standard? What?s the command invocation look like for each of create and destroy? The only thing that we can configure is the size of the snapshot device, which is basically the volume of changed data on the original volume the snapshot can handle before being invalidated. I think we can safely calculate that from the volume size for the most common case, or maybe use a reasonable percentage of the available size.

witten commented 2018-08-12 22:32:03 +00:00

Owner

Copy Link

Okay, having had a chance to poke around with this a bit, I'm inclined to agree that "it may be tricky to detect a logical volume from it's path". Therefore I think your original suggestion of source_logical_volumes: sounds good to me. If you still are interested in implementing this, please feel free to take a crack at it. Otherwise, let me know. And thanks for walking me through this!

Okay, having had a chance to poke around with this a bit, I'm inclined to agree that "it may be tricky to detect a logical volume from it's path". Therefore I think your original suggestion of `source_logical_volumes:` sounds good to me. If you still are interested in implementing this, please feel free to take a crack at it. Otherwise, let me know. And thanks for walking me through this!

bfabio commented 2018-08-14 06:54:21 +00:00

Author

Copy Link

Great, I'll come up with a patch.

On a second thought this has become "Support LVM volumes backup", do we want to support qcow2 and raw like backup-vm does?

Great, I'll come up with a patch. On a second thought this has become "Support LVM volumes backup", do we want to support qcow2 and raw like [backup-vm](https://github.com/milkey-mouse/backup-vm) does?

witten commented 2018-08-14 15:26:10 +00:00

Owner

Copy Link

Does the lvcreate command differ based on the VM disk image format? If so, how? Or would you use a different command in each case?

Does the `lvcreate` command differ based on the VM disk image format? If so, how? Or would you use a different command in each case?

bfabio commented 2018-08-15 14:30:49 +00:00

Author

Copy Link

Sorry about the confusion. lvcreate just creates LVM volumes (and their snapshots) which are block devices that may or may not be used as backend storage for virtual machines.

Virtualization platforms also support file based disk images formats such as qcow2 and raw.

Sorry about the confusion. `lvcreate` just creates LVM volumes (and their snapshots) which are block devices that may or may not be used as backend storage for virtual machines. Virtualization platforms also support file based disk images formats such as qcow2 and raw.

witten commented 2018-08-16 05:02:07 +00:00

Owner

Copy Link

Gotcha. Those seem like a reasonable thing to support, either as part of this or separately. Is your thinking that you would support generic libvirt backups, and thereby target all the various formats?

Gotcha. Those seem like a reasonable thing to support, either as part of this or separately. Is your thinking that you would support generic libvirt backups, and thereby target all the various formats?

bfabio commented 2018-08-18 13:00:49 +00:00

Author

Copy Link

I was thinking about implementing just the LVM part for now, as it's what I need for our workflow.

I was thinking about implementing just the LVM part for now, as it's what I need for our workflow.

witten commented 2018-08-18 20:58:13 +00:00

Owner

Copy Link

Sounds good!

Sounds good!

stevekerrison commented 2018-09-06 01:40:17 +00:00

Copy Link

I also wanted --read-special support for a similar purpose - to backup inactive disk partitions as well as LVM snapshot volumes. So, I forked borgmatic and added a read_special option to the location part of the yaml schema. The rest I do with hooks.

I can make it available on GitHub if you think it's worth merging? Relatively straightforward to do though.

The default config is off, of course, and the template comments warn about its use, although I feel the warning might need to be stronger, and reference the borg docs.

I also wanted `--read-special` support for a similar purpose - to backup inactive disk partitions as well as LVM snapshot volumes. So, I forked borgmatic and added a `read_special` option to the `location` part of the yaml schema. The rest I do with hooks. I can make it available on GitHub if you think it's worth merging? Relatively straightforward to do though. The default config is off, of course, and the template comments warn about its use, although I feel the warning might need to be stronger, and reference the borg docs.

witten commented 2018-09-06 05:13:12 +00:00

Owner

Copy Link

Sure, I'd be happy to take a look at the fork and potentially merge it! It may not satisfy all of @bfabio's asks in this ticket, but it could be a start.

Sure, I'd be happy to take a look at the fork and potentially merge it! It may not satisfy all of @bfabio's asks in this ticket, but it could be a start.

stevekerrison commented 2018-09-07 08:57:56 +00:00

Copy Link

Added a pull request on GitHub https://github.com/witten/borgmatic/pull/25 for your consideration.

I'm also happy to discuss my before and after backup hooks if @bfabio would like, although I don't consider them particularly fit for general use, as they're just some shell-fu that make certain naming convention assumptions, and don't deal with sync issues fully.

Added a pull request on GitHub https://github.com/witten/borgmatic/pull/25 for your consideration. I'm also happy to discuss my before and after backup hooks if @bfabio would like, although I don't consider them particularly fit for general use, as they're just some shell-fu that make certain naming convention assumptions, and don't deal with sync issues fully.

witten commented 2018-09-09 17:41:55 +00:00

Owner

Copy Link

Thanks, merged!

Thanks, merged!

witten commented 2018-09-09 17:42:32 +00:00

Owner

Copy Link

Also, let me know if you'd like an entry in AUTHORS (or submit a PR yourself for that).

Also, let me know if you'd like an entry in `AUTHORS` (or submit a PR yourself for that).

stevekerrison commented 2018-09-10 07:44:08 +00:00

Copy Link

My name's in the commit log. That's enough for me ;)

My name's in the commit log. That's enough for me ;)

witten commented 2018-09-11 03:33:17 +00:00

Owner

Copy Link

Sounds good. :)

Sounds good. :)

witten commented 2018-09-27 04:38:49 +00:00

Owner

Copy Link

The --read-special support has just been released as part of borgmatic 1.2.3.

The `--read-special` support has just been released as part of borgmatic 1.2.3.

👍 1

witten added the

design finalized

label 2018-10-05 05:31:25 +00:00

bfabio commented 2018-12-21 11:01:39 +00:00

Author

Copy Link

@stevekerrison Thanks for your work. Do you think we can solve this with hooks?

@stevekerrison Thanks for your work. Do you think we can solve this with hooks?

anarcat commented 2019-10-24 02:54:49 +00:00

Copy Link

5. Add support for snapshots, but instead of having a separate source_logical_volumes or source_libvert as per your example, just allow listing of logical volumes (by path?) directly in source_directories. If borgmatic detects that one such entry is actually a logical volume, it will automatically do the snapshop create/destroy for it!

As I explained in #231, that is definitely possible. I implemented this in bup-cron and detailed how that works in the other issue addressing specifically filesystem snapshots.

I'd argue that borgmatic should take any path and not assume they are directories. If they are filesystems, or block devices to be more specific, that can be detected as well, and snapshots can be made. But if you're going to implement snapshots, I'd say: do it for everyone and allow users to snapshot /home if their filesystem setup allow it! :)

And, by the way, many linux distributions (e.g. Fedora and derivatives) ship with btrfs as a filesystem, so you do have snapshotting capability out of the box, assuming you have free space I guess. I never worked with that, so I can't say more on the topic, but it would be interesting to support that...

> 5. Add support for snapshots, but instead of having a separate source_logical_volumes or source_libvert as per your example, just allow listing of logical volumes (by path?) directly in source_directories. If borgmatic detects that one such entry is actually a logical volume, it will automatically do the snapshop create/destroy for it! As I explained in #231, that is definitely possible. I implemented this in bup-cron and detailed how that works in the other issue addressing specifically filesystem snapshots. I'd argue that borgmatic should take any path and not assume they are directories. If they are filesystems, or block devices to be more specific, that can be detected as well, and snapshots can be made. But if you're going to implement snapshots, I'd say: do it for *everyone* and allow users to snapshot `/home` if their filesystem setup allow it! :) And, by the way, many linux distributions (e.g. Fedora and derivatives) ship with btrfs as a filesystem, so you do have snapshotting capability out of the box, assuming you have free space I guess. I never worked with that, so I can't say more on the topic, but it would be interesting to support that...

witten commented 2019-10-24 17:18:35 +00:00

Owner

Copy Link

I'm willing to explore automatic logical volume / mount point discovery, especially if there's prior art for that approach. If the heuristic can be made to work reliably, then great. If not, then maybe we'll need to resort to something more explicit as discussed above.

I'd still probably start with support for one type of snapshotting tech (say, LVM), even if ultimately I'd like to support all the things (btrfs, zfs, qcow, etc etc.).

I'm willing to explore automatic logical volume / mount point discovery, especially if there's prior art for that approach. If the heuristic can be made to work reliably, then great. If not, then maybe we'll need to resort to something more explicit as discussed above. I'd still probably start with support for one type of snapshotting tech (say, LVM), even if ultimately I'd like to support *all the things* (btrfs, zfs, qcow, etc etc.).

witten changed title from Support libvirt backup to LVM filesystem snapshotting 2019-10-24 17:19:59 +00:00

witten commented 2019-10-24 17:22:20 +00:00

Owner

Copy Link

Okay, I'm merging witten/borgmatic#231 into this ticket and re-casting this ticket as "start with LVM support". The idea is that the design can lay the groundwork for subsequent filesystem support to follow. Since witten/borgmatic#231 has some excellent discussion as well, please refer to it for ideas on the automatic discovery/probing approach.

Okay, I'm merging https://projects.torsion.org/witten/borgmatic/issues/231 into this ticket and re-casting this ticket as "start with LVM support". The idea is that the design can lay the groundwork for subsequent filesystem support to follow. Since https://projects.torsion.org/witten/borgmatic/issues/231 has some excellent discussion as well, please refer to it for ideas on the automatic discovery/probing approach.

witten commented 2019-10-25 18:24:58 +00:00

Owner

Copy Link

I manually played around with LVM on a spare USB drive (since my machine isn't using LVM on the main disk). I have a better idea of how it all works now. Seems pretty straightforward to do snapshotting, although as outlined in this comment, the devil is in the details. And in the discovery in particular.

I manually played around with LVM on a spare USB drive (since my machine isn't using LVM on the main disk). I have a better idea of how it all works now. Seems pretty straightforward to do snapshotting, although as [outlined in this comment](https://projects.torsion.org/witten/borgmatic/issues/231#issuecomment-2011), the devil is in the details. And in the discovery in particular.

witten commented 2019-10-25 19:27:19 +00:00

Owner

Copy Link

Pseudo-code for volume probing:

# e.g.: {'/mnt': '/dev/mapper/lvol0', ...}
mount_point_devices = {disk.mountpoint: disk.device for disk in psutil.disk_partitions()}

# e.g.: {'/dev/mapper/lvol0': '100.00m', ...}
logical_volume_device_sizes = {
    device_path: size
    for line in subprocess.check_output('lvs -o DMpath,lv_size --noheadings')
    for (device_path, size) in (line.strip().split(),)
}

for directory in configured_borgmatic_source_directories:
    logical_volume_device_path = find_device_for_path(directory, mount_point_devices)
    size = logical_volume_device_sizes.get(logical_volume_device_path)

    if not size:
        continue  # Not an LVM path. Ultimately could do lookups for btrfs, zfs, etc. here.

    maybe_calculate_a_percentage_of_the_size_for_the_snapshot(size)
    subprocess.check_output(
        'lvcreate --size {} --snapshot --permission r --name whatever {}'.format(
            size, logical_volume_device_path  # TODO: Might need to map from DMPath to lv_path here.
        )
    )

Pseudo-code for volume probing: ```python # e.g.: {'/mnt': '/dev/mapper/lvol0', ...} mount_point_devices = {disk.mountpoint: disk.device for disk in psutil.disk_partitions()} # e.g.: {'/dev/mapper/lvol0': '100.00m', ...} logical_volume_device_sizes = { device_path: size for line in subprocess.check_output('lvs -o DMpath,lv_size --noheadings') for (device_path, size) in (line.strip().split(),) } for directory in configured_borgmatic_source_directories: logical_volume_device_path = find_device_for_path(directory, mount_point_devices) size = logical_volume_device_sizes.get(logical_volume_device_path) if not size: continue # Not an LVM path. Ultimately could do lookups for btrfs, zfs, etc. here. maybe_calculate_a_percentage_of_the_size_for_the_snapshot(size) subprocess.check_output( 'lvcreate --size {} --snapshot --permission r --name whatever {}'.format( size, logical_volume_device_path # TODO: Might need to map from DMPath to lv_path here. ) ) ```

👍 1

witten commented 2019-10-26 17:21:01 +00:00

Owner

Copy Link

Some more thoughts beyond just volume probing: Once we have collected all of the source directories that need snapshotting, we're potentially faced with a bit of challenge. Let's say we've got a list of source directories in borgmatic's config. Some of them are backed by LVM (and therefore can be snapshotted), and some of them aren't (and therefore can't be snapshotted). If we want to backup the combined set of LVM and non-LVM source directories, we've got a few options:

1. Rewrite LVM-backed source directory paths to their snapshot counterparts, and leave non-LVM source directories alone. This means that, for instance, borgmatic would tell Borg to backup ~/.borgmatic/snapshots/some-source (an LVM snapshot of /etc/some-source) and /etc/another-source (non-LVM source). And then when you go to restore, you'd have to deal with that mixed hierarchy, and potentially move files around per-source.
2. Rewrite all source directory paths to be in a common hierarchy regardless of whether they're snapshottable. For example, borgmatic would tell Borg to backup ~/.borgmatic/files/some-source (an LVM snapshot of /etc/some-source) and ~/.borgmatic/files/another-source (a mount --rbind bind mount of /etc/another-source). That way, when you go to restore, all source directories are restored into a common hierarchy regardless of whether they came from LVM snapshots or not.
3. Other options?

Also, how common might it be to have mixed snapshottable and non-snapshottable source directories? Is this an edge case worth supporting?

I'll also note that even with multiple different filesystem types that all support snapshotting (btrfs, zfs, etc.) we'd still have to mount all of their snapshots somewhere other than at the original source directory locations.

Some more thoughts beyond just volume probing: Once we have collected all of the source directories that need snapshotting, we're potentially faced with a bit of challenge. Let's say we've got a list of source directories in borgmatic's config. Some of them are backed by LVM (and therefore can be snapshotted), and some of them aren't (and therefore can't be snapshotted). If we want to backup the combined set of LVM and non-LVM source directories, we've got a few options: 1. Rewrite LVM-backed source directory paths to their snapshot counterparts, and leave non-LVM source directories alone. This means that, for instance, borgmatic would tell Borg to backup `~/.borgmatic/snapshots/some-source` (an LVM snapshot of `/etc/some-source`) and `/etc/another-source` (non-LVM source). And then when you go to restore, you'd have to deal with that mixed hierarchy, and potentially move files around per-source. 2. Rewrite all source directory paths to be in a common hierarchy regardless of whether they're snapshottable. For example, borgmatic would tell Borg to backup `~/.borgmatic/files/some-source` (an LVM snapshot of `/etc/some-source`) and `~/.borgmatic/files/another-source` (a `mount --rbind` [bind mount](https://unix.stackexchange.com/questions/198590/what-is-a-bind-mount) of `/etc/another-source`). That way, when you go to restore, all source directories are restored into a common hierarchy regardless of whether they came from LVM snapshots or not. 3. Other options? Also, how common might it be to have mixed snapshottable and non-snapshottable source directories? Is this an edge case worth supporting? I'll also note that even with multiple different filesystem types that all support snapshotting (btrfs, zfs, etc.) we'd still have to mount all of their snapshots somewhere other than at the original source directory locations.

anarcat commented 2019-11-01 23:09:10 +00:00

Copy Link

this was a tricky situation in bup-cron as well. if i remember correctly, i was able to remap things around so that the backup looked like /var was really /var and not /snap/var or whatever.. but i don't quite remember how that worked...

this is one of the reasons why i wanted to implement this natively in borg: having support from the Archiver that crawls the archive would fix this problem upstream. but alas, thomas didn't want to hear about it... :( maybe he could be convinced to change his mind, however for me that ship has sailed a long time ago. ;)

this was a tricky situation in bup-cron as well. if i remember correctly, i was able to remap things around so that the backup *looked* like `/var` was really `/var` and not `/snap/var` or whatever.. but i don't quite remember how that worked... this is one of the reasons why i wanted to implement this natively in borg: having support from the Archiver that crawls the archive would fix this problem upstream. but alas, thomas didn't want to hear about it... :( maybe he could be convinced to change his mind, however for me that ship has sailed a long time ago. ;)

witten commented 2019-11-04 21:32:05 +00:00

Owner

Copy Link

this was a tricky situation in bup-cron as well. if i remember correctly, i was able to remap things around so that the backup looked like /var was really /var and not /snap/var or whatever.. but i don’t quite remember how that worked…

Yeah, that seems potentially problematic.. Because if you temporary mount the snapshot of /var over the actual /var, then any running processes relying on the actual /var will be reading outdated files.. and unable to write at all, since it's presumably a read-only snapshot.

Maybe there's some way to create mounts such that they're only visible to certain processes?

I'd also love to punt on this problem entirely if I can get away with not supporting mixing snapshottable and non-snapshottable directories.

this is one of the reasons why i wanted to implement this natively in borg: having support from the Archiver that crawls the archive would fix this problem upstream. but alas, thomas didn’t want to hear about it… :( maybe he could be convinced to change his mind, however for me that ship has sailed a long time ago. ;)

That would be ideal, yeah. Point me at a Borg ticket if there is one!

> this was a tricky situation in bup-cron as well. if i remember correctly, i was able to remap things around so that the backup looked like /var was really /var and not /snap/var or whatever.. but i don’t quite remember how that worked… Yeah, that seems potentially problematic.. Because if you temporary mount the snapshot of `/var` over the actual `/var`, then any running processes relying on the actual `/var` will be reading outdated files.. and unable to write at all, since it's presumably a read-only snapshot. Maybe there's some way to create mounts such that they're only visible to certain processes? I'd also love to punt on this problem entirely if I can get away with not supporting mixing snapshottable and non-snapshottable directories. > this is one of the reasons why i wanted to implement this natively in borg: having support from the Archiver that crawls the archive would fix this problem upstream. but alas, thomas didn’t want to hear about it… :( maybe he could be convinced to change his mind, however for me that ship has sailed a long time ago. ;) That would be ideal, yeah. Point me at a Borg ticket if there is one!

witten commented 2019-11-04 21:40:38 +00:00

Owner

Copy Link

Maybe there's some way to create mounts such that they’re only visible to certain processes?

Well, that was easy:

$ sudo unshare -m /bin/bash -c "mount --bind /root/tmp /mnt && mount | grep /mnt && sleep 60"
/dev/mapper/cryptroot on /mnt type ext4 (rw,noatime)

Then, while that's sleeping, in a separate terminal:

$ sudo mount | grep /mnt
$

Meaning that the mount is only visible to the first shell.

...

EDIT: Here's how to do something similar from Python. Linux-only, same as the above Bash variant:

import os
import ctypes
import subprocess

CLONE_NEWNS = 0x00020000  # From sched.h

# Make a private namespace for mounts.
libc = ctypes.cdll.LoadLibrary('libc.so.6')
libc.unshare(CLONE_NEWNS)
subprocess.check_call('mount --make-private /'.split(' '))

# Make a bind mount and list its contents. This can only be seen by this process.
subprocess.check_call('mount --bind /root/tmp /mnt'.split(' '))
print(os.listdir('/mnt'))

> Maybe there's some way to create mounts such that they’re only visible to certain processes? Well, that was easy: ``` $ sudo unshare -m /bin/bash -c "mount --bind /root/tmp /mnt && mount | grep /mnt && sleep 60" /dev/mapper/cryptroot on /mnt type ext4 (rw,noatime) ``` Then, while that's sleeping, in a separate terminal: ``` $ sudo mount | grep /mnt $ ``` Meaning that the mount is only visible to the first shell. ... EDIT: Here's how to do something similar from Python. Linux-only, same as the above Bash variant: ```python import os import ctypes import subprocess CLONE_NEWNS = 0x00020000 # From sched.h # Make a private namespace for mounts. libc = ctypes.cdll.LoadLibrary('libc.so.6') libc.unshare(CLONE_NEWNS) subprocess.check_call('mount --make-private /'.split(' ')) # Make a bind mount and list its contents. This can only be seen by this process. subprocess.check_call('mount --bind /root/tmp /mnt'.split(' ')) print(os.listdir('/mnt')) ```

👍 1

anarcat commented 2019-11-05 15:51:46 +00:00

Copy Link

namespaces are, indeed, an elegant solution for this, but require root unless you got the right capabilities...

namespaces are, indeed, an elegant solution for this, but require root unless you got the right capabilities...

witten commented 2019-11-05 17:12:03 +00:00

Owner

Copy Link

Isn't root required anyway for lvcreate?

Isn't root required anyway for `lvcreate`?

anarcat commented 2019-11-05 19:34:04 +00:00

Copy Link

Isn’t root required anyway for lvcreate?

true.

> Isn’t root required anyway for lvcreate? true.

witten referenced this issue 2019-11-22 23:20:05 +00:00

Integration with BTRFS Snapshot #251

pixeye33 commented 2019-11-27 20:38:42 +00:00

Copy Link

Hello,

Just found out about this project, and i wanted to add another filesystem to the list of those who allows snapshots : ZFS.

In my use cases i actually have both ZFS and LVM, and i'd love to be able to backup them using snapshots.

I wanted to mention ZFS as well so maybe the solution is designed with all 3 in mind (even if the support for all comes later)

Thanks !

Hello, Just found out about this project, and i wanted to add another filesystem to the list of those who allows snapshots : ZFS. In my use cases i actually have both ZFS and LVM, and i'd love to be able to backup them using snapshots. I wanted to mention ZFS as well so maybe the solution is designed with all 3 in mind (even if the support for all comes later) Thanks !

witten commented 2019-11-27 21:32:04 +00:00

Owner

Copy Link

Quick question while you're here.. Do you do all of your snapshotting directly with ZFS, and simply use LVM use under the hood incidentally? Or do you do all of your snapshotting with LVM? Thanks!

Quick question while you're here.. Do you do all of your snapshotting directly with ZFS, and simply use LVM use under the hood incidentally? Or do you do all of your snapshotting with LVM? Thanks!

anarcat commented 2019-11-27 23:51:10 +00:00

Copy Link

I don't use ZFS right now, but if I would, I would not also use LVM. ZFS has all the features of LUKS, ext4, LVM and mdadm put together, so I would only use ZFS.

I don't use ZFS right now, but if I would, I would *not* also use LVM. ZFS has all the features of LUKS, ext4, LVM and mdadm put together, so I would only use ZFS.

pixeye33 commented 2019-11-28 07:12:48 +00:00

Copy Link

+1 @anarcat
I do not use both ZFS and LVM on the same machine, those are separate uses cases.
ZFS has it's own mecanism for snapshots.

Like BTRFS and LVM you have one command to take the snapshot, and another to remove it.

Maybe the integration can be simplified to only be a way to pass configured volumes (the original path) to backup to before/after scripts, and simply allow before script to pass another folder (the snapshot) instead to borg ?

+1 @anarcat I do not use both ZFS and LVM on the same machine, those are separate uses cases. ZFS has it's own mecanism for snapshots. Like BTRFS and LVM you have one command to take the snapshot, and another to remove it. Maybe the integration can be simplified to only be a way to pass configured volumes (the original path) to backup to before/after scripts, and simply allow before script to pass another folder (the snapshot) instead to borg ?

witten commented 2019-11-28 16:57:29 +00:00

Owner

Copy Link

Thanks for clarifying. I'll file a separate ticket for ZFS support!

Interesting idea about making before/after scripts an official integration point for various filesystem snapshots. They would probably need to be separate hooks from the existing before_backup / after_backup hooks, just because those are for running arbitrary commands and don't have to produce any paths. I'll keep this approach in mind as an option when designing the general snapshot feature.

Thanks for clarifying. I'll file a separate ticket for ZFS support! Interesting idea about making before/after scripts an official integration point for various filesystem snapshots. They would probably need to be separate hooks from the existing `before_backup` / `after_backup` hooks, just because those are for running arbitrary commands and don't have to produce any paths. I'll keep this approach in mind as an option when designing the general snapshot feature.

👍 1

witten referenced this issue 2019-11-28 17:01:41 +00:00

ZFS filesystem snapshotting #261

witten removed the

design finalized

label 2019-11-28 17:03:05 +00:00

witten commented 2019-11-28 17:03:21 +00:00

Owner

Copy Link

ZFS ticket: witten/borgmatic#261

ZFS ticket: https://projects.torsion.org/witten/borgmatic/issues/261

👍 2

witten added this to the filesystem snapshots milestone 2019-12-05 21:14:31 +00:00

witten referenced this issue 2020-01-27 00:23:47 +00:00

ZFS filesystem snapshotting #261

witten referenced this issue 2020-01-29 21:38:35 +00:00

ZFS filesystem snapshotting #261

witten commented 2020-03-22 21:55:53 +00:00

Owner

Copy Link

A user-contributed pre-backup script for creating LVM snapshots: https://pastebin.com/5cTAhL0x

A user-contributed pre-backup script for creating LVM snapshots: https://pastebin.com/5cTAhL0x

witten referenced this issue 2020-11-25 19:45:18 +00:00

Different passphrases for different repositories? #281

witten commented 2023-03-29 18:08:59 +00:00

Owner

Copy Link

FYI there has been some work on Snapper integration (which would support LVM) here: https://github.com/borgmatic-collective/borgmatic/pull/51

FYI there has been some work on Snapper integration (which would support LVM) here: https://github.com/borgmatic-collective/borgmatic/pull/51

witten added the

new feature area

label 2023-06-28 18:54:06 +00:00

witten referenced this issue 2023-11-30 22:11:47 +00:00

ZFS filesystem snapshotting #261

witten referenced this issue from a commit 2024-12-02 04:13:28 +00:00

LVM snapshots WIP (#80).

witten referenced this issue 2024-12-02 04:20:24 +00:00

LVM snapshots (#80). #949

witten commented 2024-12-02 04:23:47 +00:00

Owner

Copy Link

Well I was hoping that in the intervening 6+ years since this ticket was filed, somebody would submit a PR for this feature. Unfortunately, nobody did. The good news is then I ended up having to learn enough LVM to do it myself!

See the WIP PR for details: #949

Well I was hoping that in the intervening 6+ years since this ticket was filed, somebody would submit a PR for this feature. Unfortunately, nobody did. The good news is then I ended up having to learn enough LVM to do it myself! See the WIP PR for details: https://projects.torsion.org/borgmatic-collective/borgmatic/pulls/949

witten referenced this issue from a commit 2024-12-02 04:25:28 +00:00

Fix typo in schema comment (#80).

witten referenced this issue from a commit 2024-12-02 05:00:26 +00:00

Fix a few docstring typos (#80).

witten referenced this issue from a commit 2024-12-02 19:09:29 +00:00

Add snapshot_size option (#80).

witten referenced this issue from a commit 2024-12-02 20:02:55 +00:00

Clarify the path rewriting for LVM (but also ZFS + Btrfs) (#80).

witten referenced this issue from a commit 2024-12-03 04:59:24 +00:00

Discover parent/grandparent/etc. logical volumes of source directories (#80).

witten referenced this issue from a commit 2024-12-03 05:01:48 +00:00

Code formatting (#80).

witten referenced this issue from a commit 2024-12-03 16:52:19 +00:00

Factor out logic for finding contained source directories in a parent directory (#80).

witten referenced this issue from a commit 2024-12-03 16:52:19 +00:00

Code formatting (#80).

witten referenced this issue from a commit 2024-12-03 19:06:26 +00:00

Port the parent directory discovery logic from LVM to ZFS (#80).

witten referenced this issue from a commit 2024-12-03 19:12:53 +00:00

Use a namedtuple for logical volume metadata (#80).

witten referenced this issue from a commit 2024-12-03 20:15:51 +00:00

Port the parent directory discovery logic from LVM to Btrfs (#80).

witten referenced this issue from a commit 2024-12-03 20:22:57 +00:00

Add recent LVM and ZFS work to NEWS (#80).

witten referenced this issue from a commit 2024-12-03 23:44:14 +00:00

Fix breakage of ZFS user property auto-backup (#80).

witten referenced this issue from a commit 2024-12-03 23:56:35 +00:00

Fix a ZFS edge case in which the hook tries to unmounted a non-mounted directory (#80).

witten referenced this issue from a commit 2024-12-04 03:13:34 +00:00

Put LVM snapshots into a data structure for convenience (#80).

witten referenced this issue from a commit 2024-12-04 03:20:21 +00:00

Before unmounting, remove the snapshot mount path instead of the parent snapshot directory (#80).

witten referenced this issue from a commit 2024-12-04 22:48:29 +00:00

Fix broken ZFS tests (#80).

witten referenced this issue from a commit 2024-12-04 23:39:37 +00:00

Don't try to unmount a ZFS snapshot if it's already deleted (#80).

witten referenced this issue from a commit 2024-12-05 04:23:15 +00:00

Get existing tests passing (#80).

witten referenced this issue from a commit 2024-12-05 04:34:21 +00:00

Remove divison by zero (#80).

witten referenced this issue from a commit 2024-12-05 19:19:37 +00:00

Add an end-to-end test for the ZFS hook using a fake ZFS binary (#80).

witten referenced this issue from a commit 2024-12-06 01:36:22 +00:00

Add an end-to-end test for the Btrfs hook using a fake Btrfs binary (#80).

witten referenced this issue from a commit 2024-12-06 06:47:02 +00:00

LVM hook end-to-end tests, not quite working yet (#80).

witten referenced this issue from a commit 2024-12-06 17:39:51 +00:00

Fix LVM end-to-end tests (#80).

witten referenced this issue from a commit 2024-12-06 18:28:01 +00:00

Fix file permissions (#80).

witten referenced this issue from a commit 2024-12-06 18:28:01 +00:00

Add some missing test coverage (#80).

witten referenced this issue from a commit 2024-12-06 22:00:06 +00:00

Lots of LVM unit tests + code formatting (#80).

witten referenced this issue from a commit 2024-12-07 00:03:04 +00:00

Completed tests for LVM (#80).

witten referenced this issue from a commit 2024-12-07 00:06:03 +00:00

Organize imports (#80).

witten referenced this issue from a commit 2024-12-07 00:12:20 +00:00

Fix flake issues (#80).

witten referenced this issue from a commit 2024-12-07 04:21:24 +00:00

LVM snapshots + ZFS and Btrfs improvements (#80).

witten referenced this issue from a commit 2024-12-07 04:22:59 +00:00

Fix spelling in comment (#80).

witten commented 2024-12-07 04:25:07 +00:00

Owner

Copy Link

Implemented in main and will be part of the next release! Docs will be online shortly here: https://torsion.org/borgmatic/docs/how-to/snapshot-your-filesystems/

Implemented in main and will be part of the next release! Docs will be online shortly here: https://torsion.org/borgmatic/docs/how-to/snapshot-your-filesystems/

witten closed this issue 2024-12-07 04:25:07 +00:00

witten commented 2024-12-07 04:33:08 +00:00

Owner

Copy Link

Aaand released in borgmatic 1.9.4!

Aaand released in borgmatic 1.9.4!

witten referenced this issue from a commit 2024-12-07 04:40:03 +00:00

Add word missing from docs (#80).

bfabio commented 2024-12-07 17:06:26 +00:00

Author

Copy Link

@witten that's so cool and elegant! Thanks for working on it, you're the real MVP

@witten that's so cool and elegant! Thanks for working on it, you're the real MVP

witten commented 2024-12-07 17:07:58 +00:00

Owner

Copy Link

And thank you for your patience with this feature! 😄

And thank _you_ for your patience with this feature! 😄

❤️ 1

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

config-command-line

1.9.14

1.9.13

1.9.12

1.9.11

1.9.10

1.9.9

1.9.8

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.14

1.8.13

1.8.12

1.8.11

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.24

1.5.23

1.5.22

1.5.21

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.14

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.7.dev0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.22

1.4.21

1.4.20

1.4.19

1.4.18

1.4.17

1.4.16

1.4.15

1.4.14

1.4.13

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.26

1.3.25

1.3.24

1.3.23

1.3.22

1.3.21

1.3.20

1.3.19

1.3.18

1.3.17

1.3.16

1.3.15

1.3.14

1.3.13

1.3.12

1.3.11

1.3.10

1.3.9

1.3.8

1.3.7

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.18

1.2.17

1.2.16

1.2.15

1.2.14

1.2.13

1.2.12

1.2.11

1.2.10

1.2.9

1.2.8

1.2.7

1.2.6

1.2.5

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.15

1.1.14

1.1.13

1.1.12

1.1.11

1.1.10

1.1.9

1.1.7

1.1.6

1.1.5

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.3

1.0.2

1.0.1

1.0.0

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

0.0.2

Labels

Clear labels

blocked

bug

data loss

design finalized

good first issue

new feature area

question / support

security

waiting for response

No Label

new feature area

Milestone

No items

No Milestone filesystem snapshots

Assignees

Clear assignees

diivi import_bot witten

No Assignees

5 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#80

No description provided.