borgmatic-collective/borgmatic
borgmatic-collective/borgmatic
Watch
9
74
Fork
You've already forked borgmatic
 57
Code Issues 71 Pull requests 3 Releases 162 Packages 1 Activity Actions 1

Data Source: Mount archives before backup #1273

New issue
Open
opened 2026-02-24 20:52:28 +00:00 by margau · 6 comments
margau commented 2026-02-24 20:52:28 +00:00
Copy link

What I'd like to do and why

Hi all,
I run borgmatic on Proxmox Servers, with the intention of efficently backup up the guest snapshops.

Proxmox usually places backups of the guests as .tar (or similar) in /var/lib/vz/dump, one tar per backup.
Even if leaving the archives uncompressed and tweaking the chunker_params, it is not really efficent in deduplicating the content.

This would be probably better if it would be possible to backup the tar "content" file-based, instead of the complete tar file as a single file.

Therefore I'd like to suggest an option to add a "archive-unpack" as a data source, mounting the archive and backing up the content file by file. This should result (in my understanding) in much better deduplication.

Right now I'm experimenting with ratarmount ( find . -type f -name '*.tar' -exec ratarmount {} \;), although that is not too successful yet.

I'm of course open for discussions, other ideas etc.

Other notes / implementation ideas

No response

### What I'd like to do and why Hi all, I run borgmatic on Proxmox Servers, with the intention of efficently backup up the guest snapshops. Proxmox usually places backups of the guests as .tar (or similar) in /var/lib/vz/dump, one tar per backup. Even if leaving the archives uncompressed and tweaking the chunker_params, it is not really efficent in deduplicating the content. This would be probably better if it would be possible to backup the tar "content" file-based, instead of the complete tar file as a single file. Therefore I'd like to suggest an option to add a "archive-unpack" as a data source, mounting the archive and backing up the content file by file. This should result (in my understanding) in much better deduplication. Right now I'm experimenting with [ratarmount](https://github.com/mxmlnkn/ratarmount) (` find . -type f -name '*.tar' -exec ratarmount {} \;`), although that is not too successful yet. I'm of course open for discussions, other ideas etc. ### Other notes / implementation ideas _No response_
witten commented 2026-02-24 21:28:32 +00:00
Owner
Copy link

Thanks for filing this. Here are my initial thoughts:

  • An "archive-unpack" data source is an interesting idea. If it worked anything like the existing data sources, its configuration would accept a single tarball path at a time (so you could list multiple). And then upon borgmatic create, it would untar each tarball—maybe into a temporary directory—and send the data to Borg (separately from any source files).
  • I'm not sure what the restore story would be for this though. Maybe restoring would produce a tarball from the stored files? And then you could use whatever Proxmox restore command you like on it?
  • Since you're presumably dumping these tarballs with vzdump (directly or indirectly), what do you think of instead having a native Proxmox/vzdump data source? That would allow borgmatic to potentially direct the dump. borgmatic still probably couldn't stream the dump directly to Borg, because storing a streamed tarball wouldn't actually solve your deduplication problem. But it could, for instance, call vzdump --stdout and pipe that into a tar extract command to write the result into a temporary directory on the fly. And then Borg could be pointed at that temp directory.
  • This kind of approach would be more in line with (not yet implemented) data sources for other VM stacks. E.g., see #252.
  • On the other hand, the benefit of an "archive-unpack" data source is it might be useful for a broader set of use cases.
Thanks for filing this. Here are my initial thoughts: * An "archive-unpack" data source is an interesting idea. If it worked anything like the existing data sources, its configuration would accept a single tarball path at a time (so you could list multiple). And then upon `borgmatic create`, it would untar each tarball—maybe into a temporary directory—and send the data to Borg (separately from any source files). * I'm not sure what the restore story would be for this though. Maybe restoring would produce a tarball from the stored files? And then you could use whatever Proxmox restore command you like on it? * Since you're presumably dumping these tarballs with [`vzdump`](https://pve.proxmox.com/pve-docs/vzdump.1.html) (directly or indirectly), what do you think of instead having a native Proxmox/vzdump data source? That would allow borgmatic to potentially direct the dump. borgmatic still probably couldn't stream the dump directly to Borg, because storing a streamed tarball wouldn't actually solve your deduplication problem. But it could, for instance, call `vzdump --stdout` and pipe that into a tar extract command to write the result into a temporary directory on the fly. And then Borg could be pointed at that temp directory. * This kind of approach would be more in line with (not yet implemented) data sources for other VM stacks. E.g., see #252. * On the other hand, the benefit of an "archive-unpack" data source is it might be useful for a broader set of use cases.
witten commented 2026-02-24 21:45:51 +00:00
Owner
Copy link

More ideas:

  • Instead of using vzdump at all, borgmatic could freeze the container, bind mount the rootfs, and point Borg at it directly. (Then unfreeze the container afterwards.) No intervening tarball to worry about that way.
More ideas: * Instead of using `vzdump` at all, borgmatic could freeze the container, bind mount the rootfs, and point Borg at it directly. (Then unfreeze the container afterwards.) No intervening tarball to worry about that way.
margau commented 2026-02-25 06:50:26 +00:00
Author
Copy link

Hi,
thanks for the thoughts!

In my point of view, vzdump and "archive-unpack" would be both independently from each other a interesting idea. vzdump directly obviously would avoid a "middle step", while being tight more direct to proxmox, thats why I came up with the tar idea which could fit more use cases.

Regarding the restore, for all ideas (including the container freeze) it would be important that a "pve-compatible" backup file (ideal: a content-side identical archive) could be restored to import the VM in another hypervisor etc. (scenario: A hypervisor had a catastrophic failure, the VM is needed up and running in another one from the backup).

Hi, thanks for the thoughts! In my point of view, vzdump and "archive-unpack" would be both independently from each other a interesting idea. vzdump directly obviously would avoid a "middle step", while being tight more direct to proxmox, thats why I came up with the tar idea which could fit more use cases. Regarding the restore, for all ideas (including the container freeze) it would be important that a "pve-compatible" backup file (ideal: a content-side identical archive) could be restored to import the VM in another hypervisor etc. (scenario: A hypervisor had a catastrophic failure, the VM is needed up and running in another one from the backup).
witten commented 2026-04-05 18:49:54 +00:00
Owner
Copy link

@margau wrote in #1273 (comment):

Hi, thanks for the thoughts!

In my point of view, vzdump and "archive-unpack" would be both independently from each other a interesting idea. vzdump directly obviously would avoid a "middle step", while being tight more direct to proxmox, thats why I came up with the tar idea which could fit more use cases.

Sounds like you have a preference for the "archive-unpack" approach at this point?

Regarding the restore, for all ideas (including the container freeze) it would be important that a "pve-compatible" backup file (ideal: a content-side identical archive) could be restored to import the VM in another hypervisor etc. (scenario: A hypervisor had a catastrophic failure, the VM is needed up and running in another one from the backup).

Could you say a little more about this? Is the idea that, for the archive unpack approach for instance, a restore would reconstruct a "pve-compatible" tarball at the same path that it started at when it was originally backed up? And it would be "pve-compatible" by virtue of being the exact same data that was in there to begin with?

@margau wrote in https://projects.torsion.org/borgmatic-collective/borgmatic/issues/1273#issuecomment-13667: > Hi, thanks for the thoughts! > > In my point of view, vzdump and "archive-unpack" would be both independently from each other a interesting idea. vzdump directly obviously would avoid a "middle step", while being tight more direct to proxmox, thats why I came up with the tar idea which could fit more use cases. Sounds like you have a preference for the "archive-unpack" approach at this point? > Regarding the restore, for all ideas (including the container freeze) it would be important that a "pve-compatible" backup file (ideal: a content-side identical archive) could be restored to import the VM in another hypervisor etc. (scenario: A hypervisor had a catastrophic failure, the VM is needed up and running in another one from the backup). Could you say a little more about this? Is the idea that, for the archive unpack approach for instance, a restore would reconstruct a "pve-compatible" tarball at the same path that it started at when it was originally backed up? And it would be "pve-compatible" by virtue of being the exact same data that was in there to begin with?
margau commented 2026-04-06 16:44:29 +00:00
Author
Copy link

Hi,
I guess archive-unpack as the "most generic version" would be the most useful also for other usecases.

Regarding the restore path:
Proxmox just wants to have the tar file "somewhere", so it can be piped by hand using pct restore or qmrestore. An automated restore probably won't be that useful, as the use case with a "restore" would likely mean moving to another host etc.

That means from my perspective:
Yes, archive-unpack would just place the archive in the same location as before the backup (especially as this is not proxmox-related, but generic). In my understanding, pve-compatible just means "the same tar archive contents" for the "archive-unpack" variant-

It could be a bit more interesting for the vzdump --stdout approach (where it would also need an archive which could be restored by the commands above, but for me it is not clear "where" that would be).

Hi, I guess archive-unpack as the "most generic version" would be the most useful also for other usecases. Regarding the restore path: Proxmox just wants to have the tar file "somewhere", so it can be piped by hand using `pct restore` or `qmrestore`. An automated restore probably won't be that useful, as the use case with a "restore" would likely mean moving to another host etc. That means from my perspective: Yes, archive-unpack would just place the archive in the same location as before the backup (especially as this is not proxmox-related, but generic). In my understanding, pve-compatible just means "the same tar archive contents" for the "archive-unpack" variant- It could be a bit more interesting for the `vzdump --stdout` approach (where it would also need an archive which could be restored by the commands above, but for me it is not clear "where" that would be).
witten commented 2026-04-10 01:52:56 +00:00
Owner
Copy link

Got it, thank you. That's helpful. Okay, here's what I'm thinking for the archive-unpack in terms of configuration UX. Something like:

tarballs:
    - name: guest1
      path: /var/lib/guest1.tar.gz
      label: Proxmox guest 1
      restore_path: /var/lib/restores/guest1.tar.gz
    - name: guest2
      path: /var/lib/guest2.tar.gz
      label: Proxmox guest 2

Only name and path would be required. The implicit restore path would default to path unless restore_path overrides it.

Let me know if you have any feedback.

Got it, thank you. That's helpful. Okay, here's what I'm thinking for the archive-unpack in terms of configuration UX. Something like: ```yaml tarballs: - name: guest1 path: /var/lib/guest1.tar.gz label: Proxmox guest 1 restore_path: /var/lib/restores/guest1.tar.gz - name: guest2 path: /var/lib/guest2.tar.gz label: Proxmox guest 2 ``` Only `name` and `path` would be required. The implicit restore path would default to `path` unless `restore_path` overrides it. Let me know if you have any feedback.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
parallel-borgmatic
command-hook-steps
2.1.7
2.1.6
2.1.5
2.1.4
2.1.3
2.1.2
2.1.1
2.1.0
2.0.13
2.0.12
2.0.11
2.0.10
2.0.9
2.0.8
2.0.7
2.0.6
2.0.5
2.0.4
2.0.3
2.0.2
2.0.1
2.0.0
1.9.14
1.9.13
1.9.12
1.9.11
1.9.10
1.9.9
1.9.8
1.9.7
1.9.6
1.9.5
1.9.4
1.9.3
1.9.2
1.9.1
1.9.0
1.8.14
1.8.13
1.8.12
1.8.11
1.8.10
1.8.9
1.8.8
1.8.7
1.8.6
1.8.5
1.8.4
1.8.3
1.8.2
1.8.1
1.8.0
1.7.15
1.7.14
1.7.13
1.7.12
1.7.11
1.7.10
1.7.9
1.7.8
1.7.7
1.7.6
1.7.5
1.7.4
1.7.3
1.7.2
1.7.1
1.7.0
1.6.6
1.6.5
1.6.4
1.6.3
1.6.2
1.6.1
1.6.0
1.5.24
1.5.23
1.5.22
1.5.21
1.5.20
1.5.19
1.5.18
1.5.17
1.5.16
1.5.15
1.5.14
1.5.13
1.5.12
1.5.11
1.5.10
1.5.9
1.5.8
1.5.7
1.5.7.dev0
1.5.6
1.5.5
1.5.4
1.5.3
1.5.2
1.5.1
1.5.0
1.4.22
1.4.21
1.4.20
1.4.19
1.4.18
1.4.17
1.4.16
1.4.15
1.4.14
1.4.13
1.4.12
1.4.11
1.4.10
1.4.9
1.4.8
1.4.7
1.4.6
1.4.5
1.4.4
1.4.3
1.4.2
1.4.1
1.4.0
1.3.26
1.3.25
1.3.24
1.3.23
1.3.22
1.3.21
1.3.20
1.3.19
1.3.18
1.3.17
1.3.16
1.3.15
1.3.14
1.3.13
1.3.12
1.3.11
1.3.10
1.3.9
1.3.8
1.3.7
1.3.6
1.3.5
1.3.4
1.3.3
1.3.2
1.3.1
1.3.0
1.2.18
1.2.17
1.2.16
1.2.15
1.2.14
1.2.13
1.2.12
1.2.11
1.2.10
1.2.9
1.2.8
1.2.7
1.2.6
1.2.5
1.2.4
1.2.3
1.2.2
1.2.1
1.2.0
1.1.15
1.1.14
1.1.13
1.1.12
1.1.11
1.1.10
1.1.9
1.1.7
1.1.6
1.1.5
1.1.4
1.1.3
1.1.2
1.1.1
1.1.0
1.0.3
1.0.2
1.0.1
1.0.0
0.1.8
0.1.7
0.1.6
0.1.5
0.1.4
0.1.3
0.1.2
0.1.1
0.1.0
0.0.7
0.0.6
0.0.5
0.0.4
0.0.3
0.0.2
Labels
Clear labels   
blocked

   
breaking

   
bug

   
data loss

   
design finalized

   
good first issue

   
new feature area

   
question / support

   
security

   
waiting for response
No labels
blocked
breaking
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
borgmatic-collective/borgmatic#1273
No description provided.