LVM Snapshot support #5

Closed
opened 2018-01-05 05:33:28 +00:00 by import_bot · 3 comments
Collaborator

This is a more difficult one, and that would be to provide support for doing an LVM snapshot of the origin volume, then mounting it as read-only prior to doing the backup. As a prerequisite, you would really need to support multiple backup "job" files as /home is probably a different LV file system then /etc.

The information that you need in order to snapshot is:

  • The LV name (if left blank, then the origin directory cannot be snapshot'd)
  • Whether it is a thin/thick LV (changes how you do the snapshot)
  • The VG name that hosts the LV
  • Amount of space (GB) to allocate for a "thick" LV snapshot

You would need to account for snapshots that fail. Or deal with "stale" snapshots from a previous run. My method in the past with bash scripts is to just re-use the same snapshot name all the time, and if it still exists on the next run, then I delete it and recreate a new one.

Nice part about snapshots is that you can mount the file system as read-only, and since it's "instant", the file system will be more consistent.


Imported from Taiga issue 4 (won't fix). Created on 2015-07-26T14:15:12+0000 by Thomas Harold.

This is a more difficult one, and that would be to provide support for doing an LVM snapshot of the origin volume, then mounting it as read-only prior to doing the backup. As a prerequisite, you would really need to support multiple backup "job" files as /home is probably a different LV file system then /etc. The information that you need in order to snapshot is: - The LV name (if left blank, then the origin directory cannot be snapshot'd) - Whether it is a thin/thick LV (changes how you do the snapshot) - The VG name that hosts the LV - Amount of space (GB) to allocate for a "thick" LV snapshot You would need to account for snapshots that fail. Or deal with "stale" snapshots from a previous run. My method in the past with bash scripts is to just re-use the same snapshot name all the time, and if it still exists on the next run, then I delete it and recreate a new one. Nice part about snapshots is that you can mount the file system as read-only, and since it's "instant", the file system will be more consistent. --- Imported from Taiga issue 4 (won't fix). Created on 2015-07-26T14:15:12+0000 by Thomas Harold.
Author
Collaborator

Just to clarify: The main benefit of this would be to get a consistent snapshot of the filesystem during the backup? Does the read-only aspect of an LVM snapshot get you anything in this case?


Comment on 2015-07-26T15:09:55+0000 by Dan Helfman.

Just to clarify: The main benefit of this would be to get a consistent snapshot of the filesystem during the backup? Does the read-only aspect of an LVM snapshot get you anything in this case? --- Comment on 2015-07-26T15:09:55+0000 by Dan Helfman.
Author
Collaborator

Mounting the file system as read-only is only done out of an abundance of caution.

The main advantage of snapshots is that the file system will be more consistent, because instead of files changing during the backup window (which might be a few hours in the cases of some backup software), the files won't be changing.

Used to backup an IMAP mail store with a few million files, and rdiff-backup (the prior solution) would take 4-6 hours to process it. So a LVM snapshot was required in order to get consistency, and the ability to verify against the disk after the backup. OTOH, borg/attic are so efficient that they could backup that mail store in only 20-25 minutes. Which means fewer files might change during the backup window, which lessens the need for an LVM snapshot.

Looking at my old bash script, the steps I used were:

  • Make a directory
  • Look for and remove old/stale snapshot
  • Create new LV snapshot
  • Mount snapshot at directory
  • Do the backup
  • Dismount the snapshot directory
  • Remove the LV snapshot

I was lazy and just used the LV's name, suffixed with "-snapshot" as the snapshot name.
LVMSNAP="$LVNAME-snapshot"

For "thick" LV snapshots, I had to use:
LVCREATE -pr -L5G --snapshot -n {LVMSNAP} {LVMVOLUMEGROUP}{LVNAME}

I was making that assumption that no more then 5GB would be written to the origin volume during the backup window. With "thin" LVs, I'm not sure that the -L5G is necessary.

Removing the snapshot:
LVREMOVE -f {LVMVOLUMEGROUP}${LVMSNAP}


Comment on 2015-07-27T10:00:47+0000 by Thomas Harold.

Mounting the file system as read-only is only done out of an abundance of caution. The main advantage of snapshots is that the file system will be more consistent, because instead of files changing during the backup window (which might be a few hours in the cases of some backup software), the files won't be changing. Used to backup an IMAP mail store with a few million files, and rdiff-backup (the prior solution) would take 4-6 hours to process it. So a LVM snapshot was required in order to get consistency, and the ability to verify against the disk after the backup. OTOH, borg/attic are so efficient that they could backup that mail store in only 20-25 minutes. Which means fewer files might change during the backup window, which lessens the need for an LVM snapshot. Looking at my old bash script, the steps I used were: - Make a directory - Look for and remove old/stale snapshot - Create new LV snapshot - Mount snapshot at directory - Do the backup - Dismount the snapshot directory - Remove the LV snapshot I was lazy and just used the LV's name, suffixed with "-snapshot" as the snapshot name. LVMSNAP="$LVNAME-snapshot" For "thick" LV snapshots, I had to use: $LVCREATE -pr -L5G --snapshot -n ${LVMSNAP} ${LVMVOLUMEGROUP}${LVNAME} I was making that assumption that no more then 5GB would be written to the origin volume during the backup window. With "thin" LVs, I'm not sure that the -L5G is necessary. Removing the snapshot: $LVREMOVE -f ${LVMVOLUMEGROUP}${LVMSNAP} --- Comment on 2015-07-27T10:00:47+0000 by Thomas Harold.
Author
Collaborator

With the new borgmatic "hooks" configuration points, I think doing this might now be possible. The way it might work is:

  • Set a "before_backup" hook that removes stale snapshots, creates a new snapshot, mounts it, etc. Exactly the first few steps you describe above.
  • Set a "after_backup" hook that unmounts the snapshot directory and removes the snapshot.

To see what the configuration looks like for hooks, you can run generate-borgmatic-config.

Also, in regards to.. "As a prerequisite, you would really need to support multiple backup 'job' files as /home is probably a different LV file system then /etc." .. That should now be possible by creating separate borgmatic configuration files in /etc/borgmatic.d/.

So I think all the pieces are here for someone to implement this with a couple of scripts and borgmatic configuration calling them. But I don't think I'm going to bake it into borgmatic itself because I don't have a lot of experience with LVM and it sounds like it should be possible with hooks.

If you or someone else would like to try the hook approach, I'd be interested to hear about how it works out.


Comment on 2017-11-04T05:08:12+0000 by Dan Helfman.

With the new borgmatic "hooks" configuration points, I think doing this might now be possible. The way it might work is: * Set a "before_backup" hook that removes stale snapshots, creates a new snapshot, mounts it, etc. Exactly the first few steps you describe above. * Set a "after_backup" hook that unmounts the snapshot directory and removes the snapshot. To see what the configuration looks like for hooks, you can run `generate-borgmatic-config`. Also, in regards to.. "As a prerequisite, you would really need to support multiple backup 'job' files as /home is probably a different LV file system then /etc." .. That should now be possible by creating separate borgmatic configuration files in `/etc/borgmatic.d/`. So I think all the pieces are here for someone to implement this with a couple of scripts and borgmatic configuration calling them. But I don't think I'm going to bake it into borgmatic itself because I don't have a lot of experience with LVM and it sounds like it should be possible with hooks. If you or someone else would like to try the hook approach, I'd be interested to hear about how it works out. --- Comment on 2017-11-04T05:08:12+0000 by Dan Helfman.
Sign in to join this conversation.
No Milestone
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#5
No description provided.