Integration with KVM VM Snapshots #252

New Issue

drewkett · 2019-11-22T22:52:43Z

drewkett commented

2019-11-22 22:52:43 +00:00

What I'm trying to do and why

I've got a couple virtual machines running using kvm and qcow2 disk images. To get a consistent snapshot of the disk, I basically create a snapshot that creates a new image file that points to the active one, which gets all new writes to the disk. I cp/rsync the now readonly disk to a separate location for backing up. And then I effectively undo the snapshot which merges any changes into the disk image since the process started. This is all handled as a pre backup script shown below (This is close to the script I use, but there is an optimization in my setup that reduce the amount that needs to be copied)

#!/usr/bin/python3

from datetime import datetime
from pathlib import Path
from subprocess import run
date = datetime.now().strftime("%Y%m%d-%H%M%S")
vmname = "sie-linux-vm3"
images = Path("/var/lib/libvirt/images-subvol")
base_vm = images / "vm3.qcow2"
snap_vm = images / "vm3-snap-{}.qcow2".format(date)
# The snapshot name is recorded with the disk image and needs to be unique,
# hence the timestamp. I haven't tried to hard but I couldn't figure out how to
# clean that up since it shows up in a list snapshots command for the disk image
# I believe
snap_name = "snap-{}".format(date)
backup_dir = Path("/.backups")
vm_backup_dir = backup_dir / "vm"
error_occured = False

# This dumps the configuration file of the running vm for backup
xml = vm_backup_dir / "vm3.xml"
with xml.open('w') as f:
    res = run(["virsh","dumpxml",vmname],stdout=f)
    error_occured |= res.returncode != 0

# This creates a snapshot which uses snap_vm as the location for it. This file
# will hold the changes to the file system while vm is copied out. 
# --quiesce here only works if you install the qemu agent in the vm
# vda would need to be an input variable as that is the block device name in the vm
res = run(["virsh","snapshot-create-as","--domain",vmname,"--name",snap_name,"--diskspec","vda,file={}".format(snap_vm),"--disk-only","--atomic","--quiesce"])
error_occured |= res.returncode != 0

# if it succeeeds, copy the image out to the /.backups folder
if res.returncode == 0:
    # Depending on the image size this can be just a copy of the file as well
    res = run(["rsync","-a",str(base_vm),str(vm_backup_dir)])
    error_occured |= res.returncode != 0

    # This commits the change. If the disk image is a chain of qcow files, then you
    # want to be careful here or you may not preserve the chain
    # --pivot tells it to use the original image file once its merged back in
    res = run(["virsh","blockcommit",vmname,"vda","--active","--verbose","--pivot"])
    error_occured |= res.returncode != 0
    if res.returncode == 0:
        # Delete the temporary file if the changes successfully get merged into vm image
        snap_vm.unlink()

if error_occured:
    exit(1)

And then I backup the /.backups/vm directory. The nice thing about this is that it entirely runs as a pre backup command which keeps the time that the vm is running against the temporary image to a minimum. A qcow2 image is required I believe for this to work.

I don't currently handle restoration, though it doesn't seem too difficult since you just need to copy the images back to the original location and then load the xml file that is backed up. I haven't yet tried it though, since I just put this code in place.

#### What I'm trying to do and why I've got a couple virtual machines running using kvm and qcow2 disk images. To get a consistent snapshot of the disk, I basically create a snapshot that creates a new image file that points to the active one, which gets all new writes to the disk. I cp/rsync the now readonly disk to a separate location for backing up. And then I effectively undo the snapshot which merges any changes into the disk image since the process started. This is all handled as a pre backup script shown below (This is close to the script I use, but there is an optimization in my setup that reduce the amount that needs to be copied) ```python #!/usr/bin/python3 from datetime import datetime from pathlib import Path from subprocess import run date = datetime.now().strftime("%Y%m%d-%H%M%S") vmname = "sie-linux-vm3" images = Path("/var/lib/libvirt/images-subvol") base_vm = images / "vm3.qcow2" snap_vm = images / "vm3-snap-{}.qcow2".format(date) # The snapshot name is recorded with the disk image and needs to be unique, # hence the timestamp. I haven't tried to hard but I couldn't figure out how to # clean that up since it shows up in a list snapshots command for the disk image # I believe snap_name = "snap-{}".format(date) backup_dir = Path("/.backups") vm_backup_dir = backup_dir / "vm" error_occured = False # This dumps the configuration file of the running vm for backup xml = vm_backup_dir / "vm3.xml" with xml.open('w') as f: res = run(["virsh","dumpxml",vmname],stdout=f) error_occured |= res.returncode != 0 # This creates a snapshot which uses snap_vm as the location for it. This file # will hold the changes to the file system while vm is copied out. # --quiesce here only works if you install the qemu agent in the vm # vda would need to be an input variable as that is the block device name in the vm res = run(["virsh","snapshot-create-as","--domain",vmname,"--name",snap_name,"--diskspec","vda,file={}".format(snap_vm),"--disk-only","--atomic","--quiesce"]) error_occured |= res.returncode != 0 # if it succeeeds, copy the image out to the /.backups folder if res.returncode == 0: # Depending on the image size this can be just a copy of the file as well res = run(["rsync","-a",str(base_vm),str(vm_backup_dir)]) error_occured |= res.returncode != 0 # This commits the change. If the disk image is a chain of qcow files, then you # want to be careful here or you may not preserve the chain # --pivot tells it to use the original image file once its merged back in res = run(["virsh","blockcommit",vmname,"vda","--active","--verbose","--pivot"]) error_occured |= res.returncode != 0 if res.returncode == 0: # Delete the temporary file if the changes successfully get merged into vm image snap_vm.unlink() if error_occured: exit(1) ``` And then I backup the /.backups/vm directory. The nice thing about this is that it entirely runs as a pre backup command which keeps the time that the vm is running against the temporary image to a minimum. A qcow2 image is required I believe for this to work. I don't currently handle restoration, though it doesn't seem too difficult since you just need to copy the images back to the original location and then load the xml file that is backed up. I haven't yet tried it though, since I just put this code in place.

witten commented

2019-11-25 17:23:38 +00:00

To get a consistent snapshot of the disk, I basically create a snapshot that creates a new image file that points to the active one, which gets all new writes to the disk. I cp/rsync the now readonly disk to a separate location for backing up. And then I effectively undo the snapshot which merges any changes into the disk image since the process started.

Again, thank you for filing this! The approach seems fairly different from my understanding of typical filesystem-level snapshots (btrfs, lvm, etc.). There, you create a read-only snapshot for a particular a point in time, without disturbing writes to the normal filesystem. But here, you're saying that with these VM-based qcow2 snapshots, it's effectively making the VM filesystem temporarily read-only until the backup is done and the writes can be merged back? Or am I misunderstanding?

Is there a way to do the qcow2 snapshotting more akin to traditional filesystem snapshotting, where the VM's filesystem is not disturbed and can continue doing writes the whole time?

> To get a consistent snapshot of the disk, I basically create a snapshot that creates a new image file that points to the active one, which gets all new writes to the disk. I cp/rsync the now readonly disk to a separate location for backing up. And then I effectively undo the snapshot which merges any changes into the disk image since the process started. Again, thank you for filing this! The approach seems fairly different from my understanding of typical filesystem-level snapshots (btrfs, lvm, etc.). There, you create a read-only snapshot for a particular a point in time, without disturbing writes to the normal filesystem. But here, you're saying that with these VM-based qcow2 snapshots, it's effectively making the VM filesystem temporarily read-only until the backup is done and the writes can be merged back? Or am I misunderstanding? Is there a way to do the qcow2 snapshotting more akin to traditional filesystem snapshotting, where the VM's filesystem is not disturbed and can continue doing writes the whole time?

drewkett commented

2019-11-27 04:58:00 +00:00

You mostly have it right, with the caveat that the VM continues to run just fine except its writes are going to a different image file. The qcow2 file format can have a chain of image files where each one is a delta relative to the underlying one (and all the underlying ones are effectively read only). This can be used to create a base image file and have multiple vm's point to the same base image but not need to have a unique copy of all the data. For snapshotting as done in the script, it's useful because you can temporarily create a new image file which points to the active image file while a vm is running. So all new changes go into the new file and the existing image is read only. So I set it up so that it creates this temporary image, and then I copy the existing image out so that I can quickly merge the temporary file into the normal vm iamge and delete the temporary image. (You could just create the temporary file and run the back up excluding the temporary file instead). It just made sense to me to keep the temporary image active for as short of time as possible since there are theoretical performance implications.

witten commented

2019-12-02 17:46:25 +00:00

The qcow2 file format can have a chain of image files where each one is a delta relative to the underlying one (and all the underlying ones are effectively read only).

Got it. It sounds very similar to the way that Docker images work at this level.

It just made sense to me to keep the temporary image active for as short of time as possible since there are theoretical performance implications.

Not just performance.. If I'm understanding all of this properly (which I may not be), then does the VM have a period of time in which any writes it does are inaccessible for read from within the VM? Even if they are merged in / pivoted at the conclusion of backups?

> The qcow2 file format can have a chain of image files where each one is a delta relative to the underlying one (and all the underlying ones are effectively read only). Got it. It sounds very similar to the way that Docker images work at this level. > It just made sense to me to keep the temporary image active for as short of time as possible since there are theoretical performance implications. Not just performance.. If I'm understanding all of this properly (which I may not be), then does the VM have a period of time in which any writes it does are inaccessible for read from within the VM? Even if they are merged in / pivoted at the conclusion of backups?

witten added this to the filesystem snapshots milestone 2019-12-05 21:14:04 +00:00

witten added the

new feature area

label 2023-06-28 18:53:35 +00:00

witten commented

2024-12-07 04:28:36 +00:00

Just checking in.. after a whole five years of this not being implemented. Is this still an ongoing need? I just developed three other filesystem snapshot hooks for borgmatic, so I'm wondering if anyone is still interested in this one—and whether any of the requirements or use cases have changed in that time. Thanks!

magma1447 commented

2025-01-24 15:19:44 +00:00

Just checking in.. after a whole five years of this not being implemented. Is this still an ongoing need? I just developed three other filesystem snapshot hooks for borgmatic, so I'm wondering if anyone is still interested in this one—and whether any of the requirements or use cases have changed in that time. Thanks!

I am not using disk images as this topic is about. But I do backup full VMs. My storage is on LVM Logical Volumes though. I assume it will be out of scope to handle that. But in case you are interested I would like to show my interested at least.

I am using https://github.com/milkey-mouse/backup-vm as it is today. The project seems fairly dead. Also it's not as elegant to configure as borgmatic. Actually, it doesn't support any config at all, everything is command line options. It does however have support to run backups (borg) in parallel using some borg-multi wrapper (33 lines of python).

Since backup-vm also is Python I found it worth mentioning. If you want to proceed you can probably find quite a lot of interesting code that handles the libvirt snapshotting part.

I do understand that borgmatic now supports LVM (and ZFS) snapshots. In fact, I just upgraded software on one server to handle zfs there. I have several more servers to go through. Those will probably use LVM when I get to it. The problem is that they don't have LVM right now, so plenty of work to make this better.

With that I also understand that in a way LVM snapshots could be used to backup the VMs I have. The advantage of backup-vm on that matter is that it understands the configuration from libvirt. A very basic command line would be `backup-vm . That's a luxary which reduces risk of human errors after adding/removing disks to VMs.

> Just checking in.. after a whole five years of this not being implemented. Is this still an ongoing need? I just developed three other filesystem snapshot hooks for borgmatic, so I'm wondering if anyone is still interested in this one—and whether any of the requirements or use cases have changed in that time. Thanks! I am not using disk images as this topic is about. But I do backup full VMs. My storage is on LVM Logical Volumes though. I assume it will be out of scope to handle that. But in case you are interested I would like to show my interested at least. I am using https://github.com/milkey-mouse/backup-vm as it is today. The project seems fairly dead. Also it's not as elegant to configure as borgmatic. Actually, it doesn't support any config at all, everything is command line options. It does however have support to run backups (borg) in parallel using some `borg-multi` wrapper (33 lines of python). Since backup-vm also is Python I found it worth mentioning. If you want to proceed you can probably find quite a lot of interesting code that handles the libvirt snapshotting part. I do understand that borgmatic now supports LVM (and ZFS) snapshots. In fact, I just upgraded software on one server to handle zfs there. I have several more servers to go through. Those will probably use LVM when I get to it. The problem is that they don't have LVM right now, so plenty of work to make this better. With that I also understand that in a way LVM snapshots could be used to backup the VMs I have. The advantage of `backup-vm` on that matter is that it understands the configuration from libvirt. A very basic command line would be `backup-vm <domain> <repository>. That's a luxary which reduces risk of human errors after adding/removing disks to VMs.

Sign in to join this conversation.

3 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#252