borgmatic-collective/borgmatic

Watch

Fork 57

Command to restore and extract with replacement of old files at the same time #1320

New issue

Open

opened 2026-06-10 23:07:15 +00:00 by cegredev · 3 comments

cegredev commented

2026-06-10 23:07:15 +00:00

What I'd like to do and why

I think there should be a command (maybe borgmatic apply-archive or something) that essentially does a full restoration of a backup, without needing to manually move files around.

This command would extract a (specified) archive (or maybe just the specified paths) and move them to the location they were backed up from (so the location their path specifies on the machine, not relative to the extraction directory). It would also do a normal restore to restore databases. Ideally, before doing this, it would move the currently existing files out of the directories the backed up files would be moved to and would create database dumps of all relevant databases, in case the backup is broken. It would then maybe even provide an option to undo this application of a backup, and restore everything to the way it was before.

This is just something I'm currently implementing myself for my home server, but I imagine a lot of people would like this feature. It just feels kind of incomplete, not to have an easy way to actually restore a full backup.

If there's interest in creating this feature, I'd also volunteer myself for implementing it somewhere in the near future (I have no experience with borgmatic internals).

Other notes / implementation ideas

No response

### What I'd like to do and why I think there should be a command (maybe `borgmatic apply-archive` or something) that essentially does a full restoration of a backup, without needing to manually move files around. This command would extract a (specified) archive (or maybe just the specified paths) and move them to the location they were backed up from (so the location their path specifies on the machine, not relative to the extraction directory). It would also do a normal `restore` to restore databases. Ideally, before doing this, it would move the currently existing files out of the directories the backed up files would be moved to and would create database dumps of all relevant databases, in case the backup is broken. It would then maybe even provide an option to undo this application of a backup, and restore everything to the way it was before. This is just something I'm currently implementing myself for my home server, but I imagine a lot of people would like this feature. It just feels kind of incomplete, not to have an easy way to actually restore a full backup. If there's interest in creating this feature, I'd also volunteer myself for implementing it somewhere in the near future (I have no experience with borgmatic internals). ### Other notes / implementation ideas _No response_

witten commented

2026-06-15 20:04:38 +00:00

Owner

Thanks for filing this. Here are my thoughts on it:

For the file extraction piece, what you're describing is basically an extract action with defaults. Specifically, the --destination / flag. Unless the archive was created with relative paths, that should put the files back where they came from originally without needing to move any files around after the fact. One thing that flag does not accomplish by itself though is deleting files on disk that were not present in the archive when it was created. Whether that matters to you depends on your use case.

I'm not super wild about the idea of moving the original source files aside as part of this, for a couple of reasons:

First, it's not an atomic operation, and if it fails part-way through, you're left with a broken system. And even if it doesn't fail part-way through and everything goes according to plan, you might have a broken system as the extract is taking place. (That might be acceptable depending on the context in which you're doing the extraction; see below.)

Second, there are much better solutions for the file move piece than anything borgmatic can implement itself via ones-twosey file moves. Namely, filesystem snapshots! The good news is that borgmatic already has support for various filesystem snapshots as part of the create action (ZFS, Btrfs, and LVM), and so in theory that could be adapted to support snapshotting the filesystem before this proposed extract/restore action, thereby allowing atomic rollbacks if something goes wrong. Or maybe even something even safer, like doing the extract/restore to a temporary directory, snapshotting it, and then promoting that snapshot to be the new root filesystem or whatever in one atomic operation.

I could also see this snapshot support coming in a second or third iteration of this feature as an additional enhancement. You wouldn't necessarily need it in an initial MVP version.

For databases, that sounds like basically a restore action without specifying a --database and relying on borgmatic to restore all of them. The one challenge is that databases would not be part of any snapshots or file moves, atomic or otherwise. So if those failed, you could be left in a broken state. No different than current behavior though.

As for the idea of a combined action that rolls up all of the above, I could see some utility. But it would basically be a borgmatic extract --destination / ; borgmatic restore under the hood. Adding a convenience flag to the existing extract action that is an alias for --destination / might get you 90% of the way there.

Some question though: Are you intending to do this general extract/restore on a fresh system or an existing system? Perhaps more importantly, why do you want to do this kind of operation? Like, walk me through the user story. For instance, did you mess up your system via an accidental mass deletion and you want to roll it back to a known-good state? Did a hard drive completely die and you're restoring a system onto a new hard drive from nothing? Something else? Do you have borgmatic config files, or do you need to extract those too? And are you intending to extract the files for an entire filesystem starting at the root? Or just a subset of the filesystem? If so, which subset?

I'd be open to a PR (or PRs) for this. My recommendation though is: Do the broad design up-front rather than in the PR. Less rework that way. And consider doing the feature iteratively, say in multiple PRs that can each get merged independently. That way it's not a giant feature that takes forever to complete and you can get something that's actually somewhat useful (if not everything you want) more quickly.

Also be aware of #309, which might be a prerequisite for this ticket depending on how extensive you want the restores to be.

Thanks for filing this. Here are my thoughts on it: For the file extraction piece, what you're describing is basically an `extract` action with defaults. Specifically, the `--destination /` flag. Unless the archive was created with relative paths, that should put the files back where they came from originally without needing to move any files around after the fact. One thing that flag does *not* accomplish by itself though is deleting files on disk that were not present in the archive when it was created. Whether that matters to you depends on your use case. I'm not super wild about the idea of moving the original source files aside as part of this, for a couple of reasons: First, it's not an atomic operation, and if it fails part-way through, you're left with a broken system. And even if it doesn't fail part-way through and everything goes according to plan, you might have a broken system *as* the extract is taking place. (That might be acceptable depending on the context in which you're doing the extraction; see below.) Second, there are much better solutions for the file move piece than anything borgmatic can implement itself via ones-twosey file moves. Namely, filesystem snapshots! The good news is that borgmatic already has support for various filesystem snapshots as part of the `create` action (ZFS, Btrfs, and LVM), and so in theory that could be adapted to support snapshotting the filesystem before this proposed `extract`/`restore` action, thereby allowing atomic rollbacks if something goes wrong. Or maybe even something even safer, like doing the `extract`/`restore` to a temporary directory, snapshotting it, and then promoting that snapshot to be the new root filesystem or whatever in one atomic operation. I could also see this snapshot support coming in a second or third iteration of this feature as an additional enhancement. You wouldn't necessarily need it in an initial MVP version. For databases, that sounds like basically a `restore` action without specifying a `--database` and relying on borgmatic to restore all of them. The one challenge is that databases would *not* be part of any snapshots or file moves, atomic or otherwise. So if those failed, you could be left in a broken state. No different than current behavior though. As for the idea of a combined action that rolls up all of the above, I could see some utility. But it would basically be a `borgmatic extract --destination / ; borgmatic restore` under the hood. Adding a convenience flag to the existing `extract` action that is an alias for `--destination /` might get you 90% of the way there. Some question though: Are you intending to do this general `extract`/`restore` on a fresh system or an existing system? Perhaps more importantly, *why* do you want to do this kind of operation? Like, walk me through the user story. For instance, did you mess up your system via an accidental mass deletion and you want to roll it back to a known-good state? Did a hard drive completely die and you're restoring a system onto a new hard drive from nothing? Something else? Do you have borgmatic config files, or do you need to extract those too? And are you intending to extract the files for an entire filesystem starting at the root? Or just a subset of the filesystem? If so, which subset? I'd be open to a PR (or PRs) for this. My recommendation though is: Do the broad design up-front rather than in the PR. Less rework that way. And consider doing the feature iteratively, say in multiple PRs that can each get merged independently. That way it's not a giant feature that takes forever to complete and you can get something that's actually somewhat useful (if not everything you want) more quickly. Also be aware of #309, which might be a prerequisite for this ticket depending on how extensive you want the restores to be.

witten added the

new feature area

label

2026-06-15 20:18:55 +00:00

cegredev commented

2026-06-15 23:19:27 +00:00

Author

My user story is basically this:

Firstly, I want one command that restores a system to the state it was in when the archive was created, i.e. all source_directories contain the same code and all databases contain the same data. I would want to be able to 1. stop all my services, 2. run the command, 3. start my services again and have a completely running server with the data of the given archive. I want this simply because of the simplicity and utility it would offer.

Secondly, I would want this command to first move all existing data (source_directories and databases (i.e. via dumps)) to a temporary directory, in case the backup was broken, and I ended up with for example a broken server or missing data. The copy would then allow me to restore the state before the backup. I can see how this is kind of weird, as it 1. replicates backup functionality in a totally different and less advanced way inside borgmatic and 2. if you're using a backup, something must be wrong with your current state. But I can see myself needing this in a case where there is some non-critical error with my current data (like some accidentally deleted files) and all my backups magically broke, meaning I can't recover any useful data and just lost everything.
Of course this wouldn't be a problem if the overwrite wasn't a part of the command, but that's where I'm coming from.

In the script I wrote for myself I also have a feature that allows me choose the source_directories and databases to restore interactively, based on which apps I selected. My server is setup in such a way, that each app (Nextcloud, Immich, Paperless) has its own config directory which is the same as the app name. The script then matches the app to source directories and databases and restores only those and also runs some scripts before and after the restore. There is generally a lot of interactive stuff in it. Some of that would probably not be fitting for borgmatic (the apps feature being too specific for example), but some of it would also be nice to have in my proposed command (interactive archive selection for example).

Maybe this helps you understand my vision. My script is probably too over the top, but I could see some of it being useful for other borgmatic users as well.

I'm not familiar with file system snapshots. I'll have a look at that!

Hey, thanks for the detailed feedback! You have a lot of valid points. I've spent the last couple days writing a script for my home setup that does exactly what I want, and I'll attach a GIF of it running below (there's like a minute of the same screen in there, couldn't get it removed, sorry). My user story is basically this: Firstly, I want one command that restores a system to the state it was in when the archive was created, i.e. all `source_directories` contain the same code and all databases contain the same data. I would want to be able to 1. stop all my services, 2. run the command, 3. start my services again and have a completely running server with the data of the given archive. I want this simply because of the simplicity and utility it would offer. Secondly, I would want this command to first move all existing data (`source_directories` and databases (i.e. via dumps)) to a temporary directory, in case the backup was broken, and I ended up with for example a broken server or missing data. The copy would then allow me to restore the state before the backup. I can see how this is kind of weird, as it 1. replicates backup functionality in a totally different and less advanced way inside borgmatic and 2. if you're using a backup, something must be wrong with your current state. But I can see myself needing this in a case where there is some non-critical error with my current data (like some accidentally deleted files) and all my backups magically broke, meaning I can't recover any useful data and just lost everything. Of course this wouldn't be a problem if the overwrite wasn't a part of the command, but that's where I'm coming from. In the script I wrote for myself I also have a feature that allows me choose the `source_directories` and databases to restore interactively, based on which apps I selected. My server is setup in such a way, that each app (Nextcloud, Immich, Paperless) has its own config directory which is the same as the app name. The script then matches the app to source directories and databases and restores only those and also runs some scripts before and after the restore. There is generally a lot of interactive stuff in it. Some of that would probably not be fitting for borgmatic (the apps feature being too specific for example), but some of it would also be nice to have in my proposed command (interactive archive selection for example). Maybe this helps you understand my vision. My script is probably too over the top, but I could see some of it being useful for other borgmatic users as well. I'm not familiar with file system snapshots. I'll have a look at that!

Recording 2026-06-16 at 00.59.11.gif

2.9 MiB

witten commented

2026-06-16 00:03:37 +00:00

Owner

Very cool script! More thoughts below.

Thanks for explaining your use case. That is helpful to see what you're trying to accomplish. It sounds like you're basically doing application extract/restore rather than system extract/restore (e.g. all of /)—which I think makes this a little easier. For instance, it sounds like you can take some application downtime while the extract/restore is happening.

You may (or may not) be interested in an experimental borgmatic feature called borgmatic browse that, similar to your script, lets you select an archive and browse what's in it. Unlike your script, it doesn't yet support extraction, although that is planned (and like half done). In theory this browse feature could be expanded to support some of these interactive use cases. Or maybe for your own use, your script already works just fine.

On the topic of file moving versus snapshots, yet another idea is to use Borg itself as your "snapshot" mechanism rather than relying on the filesystem. Here's an example scenario:

Oh crap I accidentally deleted a bunch of application files and/or database rows!
Before I start messing with extracting/restoring anything, let me create a Borg archive of the "messed up" state so I don't lose anything. This has the benefit of also including database dumps, which filesystem snapshots—or moving files aside—don't. (You can filesystem snapshot database data directories, which isn't the same thing as a proper dump.)
Now I can safely nuke my source directories and databases and completely replace them with the files and databases from the last known-good Borg archive.
If something goes wrong or my known-good archive isn't as good as I thought, I can always roll back by extracting/restoring my archive of the messed up state.

Some of the above could be automated and/or rolled into a single borgmatic action.

The concept of extracting/restoring individual "apps" is interesting. One way borgmatic kind of sort of supports apps now is with separate configuration files per app, possibly each with a separate archive name format. But currently you'd have to extract/restore each one separately instead of together in a batch. And I don't know if you want to backup separate apps like this or if you'd prefer them all in the same archive yet individually extractable/restorable...?

In any case, I can see the utility of logically grouping source files with corresponding database dumps so that you can extract/restore them as a logical unit. This could maybe be layered on as a separate feature though from some of the above (assuming it's implemented any differently from the existing per-app config files).

Very cool script! More thoughts below. Thanks for explaining your use case. That is helpful to see what you're trying to accomplish. It sounds like you're basically doing *application* extract/restore rather than system extract/restore (e.g. all of `/`)—which I think makes this a little easier. For instance, it sounds like you *can* take some application downtime while the extract/restore is happening. You may (or may not) be interested in an experimental borgmatic feature called [`borgmatic browse`](https://torsion.org/borgmatic/how-to/inspect-your-backups/#browsing-backups) that, similar to your script, lets you select an archive and browse what's in it. Unlike your script, it doesn't yet support extraction, although that is planned (and like half done). In theory this browse feature could be expanded to support some of these interactive use cases. Or maybe for your own use, your script already works just fine. On the topic of file moving versus snapshots, yet another idea is to use Borg itself as your "snapshot" mechanism rather than relying on the filesystem. Here's an example scenario: * Oh crap I accidentally deleted a bunch of application files and/or database rows! * Before I start messing with extracting/restoring anything, let me create a Borg archive of the "messed up" state so I don't lose anything. This has the benefit of also including database dumps, which filesystem snapshots—or moving files aside—don't. (You *can* filesystem snapshot database data directories, which isn't the same thing as a proper dump.) * Now I can safely nuke my source directories and databases and completely replace them with the files and databases from the last known-good Borg archive. * If something goes wrong or my known-good archive isn't as good as I thought, I can always roll back by extracting/restoring my archive of the messed up state. Some of the above could be automated and/or rolled into a single borgmatic action. The concept of extracting/restoring individual "apps" is interesting. One way borgmatic kind of sort of supports apps now is with [separate configuration files per app](https://torsion.org/borgmatic/how-to/make-per-application-backups/), possibly each with a separate [archive name format](https://torsion.org/borgmatic/reference/configuration/archive-name-format/). But currently you'd have to extract/restore each one separately instead of together in a batch. And I don't know if you want to backup separate apps like this or if you'd prefer them all in the same archive yet individually extractable/restorable...? In any case, I can see the utility of logically grouping source files with corresponding database dumps so that you can extract/restore them as a logical unit. This could maybe be layered on as a separate feature though from some of the above (assuming it's implemented any differently from the existing per-app config files).