borgmatic not parsing all included source_directories when using btrfs #1192
Labels
No labels
blocked
breaking
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
borgmatic-collective/borgmatic#1192
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What I'm trying to do and why
Hello!
I am not entirely sure if I am seeing a bug, or misconfiguration on my side. If anything, I think btrfs documentation could use some usage examples to help understand how to configure it properly.
What I have is up-to-date Nobara Linux system (Fedora 43) with borgmatic installed using packages from Fedora, so I have pretty recent version:
What I also have, is default btrfs partition layout. I want to backup root and /home partitions:
However, unless I provide two separate configs /etc/borgmatic.d/{root,home}.yaml with exactly the same configs but different path, borgmatic always backups up only the root partition. The order in the configuration file doesn't matter - home can be first, root can be first - only root is processed. If /home is the only provided path, then /home is processed properly. When separate config files are provided in borgmatic.d folder, then both files are processed properly, but result is not exactly what I expect due to following reasons:
I do realize that I somewhat abuse borgmatic.d functionality as its main usecase is meant for separate repositories, but this workaround seems to be the only option I found thus far to get both root and /home backed up.
Thank you for this great tool. Documentation is actually outstanding and borgmatic has a lot of nice features! And I realize I might be missing something trivial here.
Steps to reproduce
Actual behavior
Expected behavior
I would expect to process both /home and root with given config file.
Other notes / implementation ideas
Nothing comes to mind :)
borgmatic version
2.0.10
borgmatic installation method
Fedora package
Borg version
1.4.2
Python version
Python 3.14.0
Database version (if applicable)
not applicable / no database being backed up
Operating system and version
Nobara Linux 43 (KDE Plasma Desktop Edition)
Thanks for all the details here! What do you get when you run
btrfs subvolume show /home? And alsobtrfs subvolume show /? That's what borgmatic uses (since 2.0.9 anyway) to determine which source directories are Btrfs subvolumes.FYI the borgmatic.d functionality isn't just for separate repositories; it can be used for any situation in which you want multiple backups with different borgmatic options. Having said that though, you ideally shouldn't have to use it here.
Hey @witten ! Thank you for very quick response. Unfortunately I couldn't get back sooner as I wasn't able to turn this PC on remotely and I had to do some troubleshooting to find that I had bad memory contact - one of the sticks needed reseating and only now I found time to do the troubleshooting. Also, I had problems previously pasting the logs as it would eat all newlines, so I converted the newlines to linux format and it seems ok now.
Anyway, to reply your question, here is the output of the commands:
Glad to hear it sounds like you've got both your memory sticks and newlines sorted out! Thanks for including the output. Based on that, I don't know why one of the subvolumes would get detected and the other wouldn't. I tried to set up a local repro with one subvolume contained inside the other—the containing subvolume with an
@name and a separate mount point—and I wasn't able to get the behavior you're seeing even with borgmatic 2.0.10. The only thing I can think of is that one of your subvolumes being at/is causing a difference here. Looking at the code though, it should handle that case.I assume that both your
btrfs subvolume showcommands above exit without error? If you're using Bash, runningecho $?immediately after a command will show its exit status (which would be0for success).What I can do is create an instrumented version of
btrfs.pyfrom borgmatic's source with additional debug logs. Then, if you feel comfortable, you can drop it into your local borgmatic install and get those additional logs.Attached is an instrumented
btrfs.pybased on borgmatic 2.0.10. You should be able to replace the existingborgmatic/hooks/data_source/btrfs.pyinstalled by your Fedora package with this altered version. (You can always reinstall the package to revert the changes.) When running borgmatic, look for new log lines containing with#1192:to verify that it's working. And then if you could include the full log output here (redacted as necessary), that would be great. Thanks for your patience!hello again,
I confirmed that both
sudo btrfs subvolume show /homeandsudo btrfs subvolume show /return with0error code.I applied changed
btrfs.pyfile here:/usr/lib/python3.14/site-packages/borgmatic/hooks/data_source/and I do see updated log entries:I also have a little feedback for default configuration. With btrfs I also seem to be affected by #1163
I disabled all security features in borgmatic.service as without this, borgmatic didn't detect any btrfs subvolumes.
Also, with default timer configuration the behavior is not something that I would expect. This is default configuration I see for timer:
This means, that by default, the backup job should be scheduled each day between 00:00 and 03:00 and my PC is never turned on at these hours. This should be remediated by
Persistent=trueparameter which will make sure, that if some instance is missed, next backup should be scheduled at next boot. But unfortunately, "RandomizedDelaySec" is also applied in such scenario, which means that next backup run might actually not happen if my PC will be turned off before 0-3h random timeout runs out (please see https://www.freedesktop.org/software/systemd/man/latest/systemd.timer.html#Persistent=).For me the solution was to change RandomizedDelaySec=3h to RandomizedDelaySec=3m, but maybe there is better solution to this case.
Thanks for doing that. So what those most recent logs indicate is that "/home" was completely absent in the patterns by the time borgmatic got to the instrumented code—not what I was suspecting! But the good news is that based on that, I tried something locally on a hunch: I put
/(not a Btrfs subvolume on my system) and/an/actual/btrfs/subvolumeinto my source directories and ran borgmatic with Btrfs enabled. The actual Btrfs subvolume vanished from patterns and only/got passed into the Btrfs hook, which is similar to the behavior you're seeing. So I might have a partial repro here. I'll dig into this further and try to determine how having/in source directories is causing patterns to vanish. (EDIT: I tracked down the problem to a function calleddeduplicate_runtime_directory_patterns(). Now the question is what to do about it.)Interesting. Were you able to narrow down exactly which options were causing problems? For instance, if you reintroduce the security options one by one (or half at a time, if you want to binary search), does that pinpoint the issue? In #1163 for LVM, it sounded like
PrivateDevices/ProtectKernelModulesmay be impacting things.Great point. Seems like a reasonable change to me. (I set this to "10m" in main.)
I will play with options and leave feedback. Is there something I could do to determine the solution in deduplicate_runtime_directory_patterns? I am not a developer, but might try to understand the logic and actual problem. Will read the code later today.
I started enabling options one by one. Enabling
PrivateDevicesorProtectKernelModulesdoes not impact detecting of btrfs volumes. SettingCapabilityBoundingSet=CAP_DAC_READ_SEARCH CAP_NET_RAWresults in borgmatic not being able to detect btrfs subvolumes.Then I downloaded default service, removed the 1m timer, changed the path to match binary location on my system, removed the "borgmatic.pw" line and then commented out
CapabilityBoundingSet=CAP_DAC_READ_SEARCH CAP_NET_RAW. Then btrfs subvolumes were detected properly.So actually, on my system, this is the only option which stops the btrfs detection.
You're welcome to have a look, but this function is unfortunately pretty thorny and complex—which is probably how a bug crept in to begin with. 😅 But the good news is I think I have a lead on a potential fix. This code isn't handling an edge case in the deduplication (when two potential duplicate patterns overlap and one of of them contains the runtime directory), and that apparently leads to the behavior you're seeing with one pattern getting over-aggressively deduplicated.
Cool, thanks for diagnosing that. I'll update the documentation / sample systemd service accordingly as part of this ticket!
Yeah... took a look, read function description and concluded that its definitely above my skills. Out of all things, all I could do is to provide mental support at this stage :) let me know if something could be tested though!
Okay, I've made a fix for this to the main branch that will be part of the next release! I'm pretty sure it will address your use case since I had a partial repro, but I would appreciate that you test it out after that release.
If you'd like to test it before then, you are welcome to as well. I've attached the updated
borgmatic/actions/pattern.pysource file that you can drop in as a replacement. (Make sure you are replacing that specific path and not any other one namedpattern.py.) This is the version of that file from main, but it looks like it's identical in 2.0.10 (modulo the fix in this ticket) and so I imagine should work as well.You can either leave the earlier
btrfs.pydebugging changes in place or not; they shouldn't impact thepattern.pychanges.Thanks!
Hello @witten! I confirm that this issue is resolved by use of attached pattern.py file. As of now both paths are properly processed and I can use single
/etc/borgmatic/config.yamlfile! Thank you!Awesome, I'm glad to hear that did it!
Released in borgmatic 2.0.13!