btrfs #1048
Labels
No labels
blocked
breaking
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
No milestone
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
borgmatic-collective/borgmatic#1048
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What I'm trying to do and why
Hey, I am trying to use the btrfs hook for snapshotting. My system has two drives, both have usual GPT partitions formatted with btrfs (and dm-crypt in between), one is root and the other is mounted at /media;
findmnt -t btrfs --list --jsongives:For both filesystems, I only want certain directories to be included in the backups but I don't mind btrfs snapshotting the whole filesystem.
I run into these
paths do not existerrors only when adding the btrfs hook, the backup runs fine without the hook.Thanks for any help in advance :)
Steps to reproduce
In
patternsI only have a few exludes, I don't think they are relevant.Actual behavior
Expected behavior
Create snapshots and use the respective directories from the snapshot.
Other notes / implementation ideas
#1043 might be related but I decided to post another issue because the usecases & system setups are different.
I guess I could setup btrfs-subvolumes for the directories I want to backup but I didn't really look into that yet because I feel this should also work without extra subvolumes.
borgmatic version
1.9.10
borgmatic installation method
pacman
Borg version
borg 1.4.0
Python version
Python 3.13.1
Database version (if applicable)
No response
Operating system and version
Arch Linux
Thanks for filing this. It looks like you're seeing a possible interaction between several different borgmatic features, although I'm not sure why it's resulting in an error. First, code that supports the
borgmatic config bootstrapaction is writing bootstrap metadata to/tmp/borgmatic-cq3xwb7o/borgmatic/bootstrapso that it ends up in the Borg archive. Then, the Btrfs hook is altering that file path to take into account the snapshot borgmatic makes of/(since/tmp/...is on/). Lastly, borgmatic is ensuring that all paths being passed to Borg as part ofborg createactually exist. And the error you're getting indicates that that file path doesn't exist for some reason.Since borgmatic is blowing up mid-run, you might be able to inspect the filesystem exactly as it was when the error occurred. So for instance does
/.borgmatic-snapshot-1603501/actually exist? (Substitute the actual snapshot path from the most recent run.) Does/.borgmatic-snapshot-1603501/tmp/borgmatic-cq3xwb7o/exist? (Again, update the path accordingly.) Does/.borgmatic-snapshot-1603501/tmp/borgmatic-cq3xwb7o/borgmatic/bootstrapexist?In any case, you can disable the feature that ensures paths sent to Borg actually exist by adding the following to your configuration:
... or by just commenting out the option if it's already set to
true, which I suspect it is.That may also make the error go away, although if the path doesn't exist there may be some underlying problem.
Separately, you can temporarily disable the bootstrap feature by adding the following to your configuration file:
I'd also be interested to see if that "solves" the problem.
Thanks for the quick response.
Yes, you're right, I had already noticed this while debugging (though it gets deleted by a subsequent run of borgmatic which then fails with a slightly different error, I'll post that error below just in case it has additional info for you).
No,
/.borgmatic-snapshot-1703188/tmp/is empty actually. Other directories like/etclook correctly snapshotted. I just noticed that/tmpis probably not snapshotted because/tmpis a tmpfs (on ArchLinux). So I guess the btrfs hook is replacing all paths without taking into account that some of the directories might not be part of the snapshotted filesystem?Subsequent run deleting old snapshot and failing with different error:
Ah, good find! Yes, that seems likely what's going on. And now I have a local repro as well. Since I don't have a system with a Btrfs root subvolume, I created a
tmpfsmount within a subdirectory of my one Btrfs subvolume and pointed theuser_runtime_directoryconfiguration option at thattmpfsmount point. And then I'm getting the same "do not exist" behavior you're seeing whensource_directories_must_exististrue.I'm not sure what's going on with deleting the old snapshot, but my guess it's related to it being the root subvolume. I probably should spin up a VM with a Btrfs root...
Okay, the first error should be fixed in main and will be part of the next release. For the second error, can you try updating to 1.9.14 to see if you can repro the issue? There have been a few Btrfs fixes since 1.9.10. Thanks!
Heya, sorry for not reacting in a while. I just had some time to tackle the btrfs hook again and had a successful run with it. However, after rebooting, I am getting this error:
I guess the overlay gets created by podman or something in that direction. I noticed that
findmntsometimes returns this overlay (sub)volume, sometimes not (I have a few containers running on this system but they are running at all times). I don't really understand whether it is actually a btrfs volume or not and how I could determine this 🤷♂️Current findmnt output:
For now I solved this by creating a findmnt wrapper script that removes the overlay json object. So nothing urgent, just wanted to update this issue after some time
Thanks for the update. If you run
btrfs property get -t subvol /var/lib/containers/storage/overlay ro, do you get any more of a useful error message?You could try
btrfs subvolume show /var/lib/containers/storage/overlayfor that.Possibly related: #1043.
I'm still interested in the results of the above commands on your machine, but as part of #1043, I changed the Btrfs code to swallow errors coming from
btrfs property get. So that particular subvolume should no longer cause errors for you. This is in main and will be part of the next release.Great to hear, I'll check out the next release when it's out.
Commands just return errors:
ERROR: Could not get subvolume flags: Invalid argument
ERROR: Not a Btrfs subvolume: Invalid argumentInteresting! Hopefully that will get taken care of by the next release.
Released in borgmatic 2.0.7! Feedback welcome.
Setup phase in 2.0.7 works great now, the log still shows error but they don't stop the run.
However, I am not sure how the mapping from the actual files to the snapshot works (and should work). With the btrfs hook, all files are added again even when they haven't changed since the last archive (without the hook) was created. Now it goes over all files again with
A /.borgmatic-snapshot-2824430/{file_path}. It would be kinda annoying to have to run a full backup again and I am not sure how all of that is stored in the archive (looking at storage usage) and how it would restore when I need it. Any advice or insight would be hugely appreciated :)I'm glad to hear 2.0.7 works for you now!
A couple points on your comment:
borgmatic list --archive latestand inspect the resulting file paths in your archive. Through the magic of path prefix stripping (a Borg feature that borgmatic makes use of), the actually stored file paths should not have the/.borgmatic-snapshot-2824430path prefix at all. They should reflect their original path locations on your filesystem as if they had never been snapshotted.