[DOCS] btrfs issues and related discussion. #1105
Labels
No labels
blocked
breaking
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
No milestone
No assignees
4 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
borgmatic-collective/borgmatic#1105
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What I'm trying to do and why
Hello,
Thank you for your continued work on borgmatic. I'm the Debian maintainer for btrfs-progs, the one who introduced installation to a subvolume in Debian, and a btrfs user for 11 years. I'd like to help make borgmatic effortlessly easy to use on typical btrfs-using systems. I'm sorry that the following will be direct, and I hope that you'll appreciate that I'm trying to save time with a declarative style:
In the online documentation on torsion.org, it says "Btrfs does not support snapshotting read-only subvolumes." which is reproducibly false. I'm utterly baffled by the complexity of the documentation and what it suggests about the implementation. PEP 20 says "Flat is better than nested" and I think this particularly applies to btrfs. The following is also a hard deal-breaker for all the sysadmins I know: "borgmatic rewrites the snapshot file paths". Principle aside, does borgmatic have restore functionality that unrewrites snapshot file paths and restores subvolumes? Restoring from backup is one of the elementary challenges of maintaining btrfs systems, and one that must inform the backup strategy.
The most simple case supported by any mainstream distribution is a single subvolume named "@rootfs" (Debian). The most complex topology at the time of installation is provided by SUSE. Btrfs-enabled software will tend to create a new subvolume for itself and this is [almost?] always a nested subvolume. These subvolumes must to be recreated when restoring. The most safe and simple way to handle the emergent complexity is to do what SUSE does and mount all filesystem roots to an admin location with secure permissions, and then take take out-of-band snapshots. (either to /btrfs-admin/volume-root/snapshots/@dataset_on_date directory or a /btrfs-admin/volume-root/@snapshots/@dataset_on_date).
ZFS is nice because because it has magic built-in to help cope with its complexity, but it's not elegant nor reliable to think of btrfs using ZFS semantics. Case and point: there is still no tool to recursively and deterministically create snapshots for a tree of subvolumes, when this is something that "just works" in ZFS. Yes, arguably this should be part of the core python-btrfs...
I've included info about a moderate case under "other notes / implementation ideas" as well as some possible solutions.
Steps to reproduce
No response
Actual behavior
No response
Expected behavior
No response
Other notes / implementation ideas
findmnt -t btrfs --list --json
but then this:
btrfs sub list /btrfs-admin/
btrfs sub list /btrfs-admin/Crucial/
btrfs sub list /btrfs-admin/Red/
My examples are a modest case.
Another option is to "do one thing and do it well" and instead write a plugin that calls a btrfs-specific snapshotting tool. Having spend a couple of years evaluating Snapper and Timeshift (which has an Ubuntu-specific upstream who as been described as hostile), I began to look to https://github.com/jcrd/snapback and https://github.com/hirak99/yabsnap
In terms of full btrfs support, the future will have:
As I see it, the scope of borgmatic is @4. What do you think? Would it be more reasonable to use a plugin to call a snapshot operation, and avoid the rabbit hole of all the btrfs topologies?
borgmatic version
No response
borgmatic installation method
No response
Borg version
No response
Python version
No response
Database version (if applicable)
No response
Operating system and version
No response
P.S. I decided to write this full write-up after reading #1092
Awesome, thanks for your interest and for the detailed ticket!
Maybe I'm misunderstanding the error message I get when I try that...?
I'm open to suggestions on reducing its complexity!
Could you say more about this? borgmatic rewrites snapshot file paths so that, within the Borg archive, they appear at their original location (their paths prior to snapshotting). The rationale is that borgmatic is transparently and temporarily creating these snapshots behind the scenes; they are not user-facing snapshots whose paths need preservation.
Because the backed up paths in the Borg archive appears at their original location, the restore process is the same as restoring any other non-Btrfs file (
borgmatic extractorborgmatic mount). For instance, if/etcis on Btrfs and you ask borgmatic to back it up, then it will appear in the Borg archive at/etcand will be restorable to/etc(or anywhere else you want it for that matter).Why?
I'm not sure I understand the ask in this ticket.. Is it just that borgmatic doesn't support a specific (and common) Btrfs use case / topology as you outlined in "other notes"? Could you say more about that gap?
I'm having issues with btrfs snapshot functionality and systemd service at the moment. I can't get borgmatic.service working, because as @sten noted,
findmnt -t btrfs --json --listoutput differs when run in native shell, and in systemd unit sandbox. Logic inbtrfs.pyis thrown off by this, and tries to work with non-existent subvolumes@witten On the topic of read-only subvolumes and snapshots: here are results of an experiment on my machine.
I'd have to read up on btrfs some more to weigh in on how to handle read-only subvolumes correctly.
I'll continue research sometime later, and I can create specific issues for the problems I'm having, if you prefer. That may require adjustments in
borgmatic.servicefile, which is a different topic.Hi Dan,
Sorry for the delay. Something seems to have gone wrong when I tried replying by email. It's my first time using Gitea, and I avoid using web interfaces whenever possible.
"Dan Helfman" projects@torsion.org writes:
When is this a better idea than taking a read-only snapshot, and why?
Your reproducer doesn't reproduce that error. Here is a tested reproducer that you can use to confirm this:
This is with linux-6.12.x LTS (either upstream or Debian's). I'd need to find and check my historical notes, but I'm pretty sure that this reproducer has worked since 4.4, and I vaguely remember that it has worked since SUSE-patched 3.16, which is where I started testing btrfs for safety and consistency.
btrfs property setis new and is...incomplete...to put it nicely.I'm delighted to hear that!
What are you interested to know about? In the above there are two philosophical principles, a practical/engineering one, an implementation fact, a ~10 page article, and a statement of fact about a statement of fact.
I've now tested borgmatic 2.0.6, studied the source, and more than the docs need work.
I began this ticket with what I thought was just a documentation issue, and attempted to gently demonstrate why various aspects of borgmatic 2.0.6 documented btrfs support are nonfunctional, risky, or perceived as risky (ie: when reading the documentation) and I hoped that you would look through the data I provided and have one or more engineer "aha!" moments :)
Having now tested borgmatic 2.0.6, I found its btrfs support nonfunctional
No Btrfs subvolumes found to snapshot, even when using it the way it appears to be designed to be used (configuration for human user view of the rootfs rather than operating on the actual filesystem tree). Thus, of course I read the source to find out why, and what I found is that the foundations of borgmatic's own btrfs plugin will be rewritten sooner or later.How would you like to proceed? With the information I provided you can write tests, and then code for those tests. I can also make implementation recommendations (ie: Don't use findmnt, because it will never provide an accurate btrfs view. Longterm, you'll probably switch to the more reliable btrfs python module. Short-term, calling
btrfscli command will work for the required simple cases).Meanwhile it would be significantly faster and easier to write a plugin that leverages an existing snapshotting solution, and this has the advantage of UNIX philosophy (do one thing and do it well), not destroying the hardware of non-enterprise rotational disks and users' warranties by rereading all data for every backup when using borg 1.x. (Rotational disks are now rated for volume of reads per day/week/month, criminal though this should be) This approach also allows you to ignore distribution-specific implementation challenges; here the support burden will fall on the snapshotting creation and rotation software. Ie: Snapper already handles all of the corner cases of SUSE's complex layout, so borgmatic doesn't need to reimplement that logic.
Will MRs/PRs be a high-friction series of arguments? Are there Hetzner-funded bug bounties for adding features and fixing bugs?
Regards,
Nicholas
@dmitry-t7ko Yes, research on snapshotting read-only subvolumes and a separate issue for your systemd problem would both be appreciated!
@sten
Yeah, I don't think Gitea has comment-by-email functionality like Debian's BTS has.
Above, we were discussing snapshotting read-only subvolumes (and the borgmatic documentation's claim that "Btrfs does not support snapshotting read-only subvolumes."). So I was first making an example subvolume read-only so that I could demonstrate what happened on my machine when I tried to take a read-only snapshot of it.
Unfortunately this doesn't work on my system, because I'm not running a Btrfs root. (I'm running ext4.) Here's the mount error I get from the
mount:Also, the formatting in your example is a little hard to read for me.. I couldn't determine if you were actually setting the
roproperty on your original subvolume. Assuming you were, maybe the defining difference is that you're running a Btrfs root while I'm not? And that's why my example works without error on your machine?I meant specifically: Can you say more about how this concern applies to borgmatic's Btrfs support? For instance, does borgmatic somehow assume nested subvolumes rather than flat?
Feedback is of course welcome. PRs are too.
I'm not sure I'm following all of the concerns / suggestions so far, so maybe getting that ironed out would be a good place to start! Additionally I have a few questions on your other comments below.
I think implementation recommendations would be good here, because I need to understand the approach even if I'm ultimately just reviewing a PR. You say to use the
btrfsCLI instead offindmnt. Great! Can you maybe provide a stand-alone example of how that would work? I think I usedfindmntbecause I couldn't find a way withbtrfsto list all Btrfs mount points. And I need mount points in particular in order to match those paths against source paths to backup. Similarly, if thebtrfsPython module is better, I'd appreciate an example.When the borgmatic Btrfs feature was originally being built, at least one user requested borgmatic not to take a dependency on Snapper for this: #251 (comment) ... We could revisit the decision now of course, but that's the history anyway. If I'm understanding your suggestion though, you're perhaps suggesting that borgmatic relies on Snapper's scheduled snapshots rather than taking manual (Snapper) snapshots prior to backup. I'm not sure that would work for many user's use cases, however, because then the snapshots could be stale in relation to when the backup is taken.
Having said that though, I could see adding a separate Snapper hook that relies on Snapper integration for Btrfs users who also happen to be using Snapper.
Two points here:
Just in general, I think it's good to iron out an approach prior to submitting PRs because that avoids a bunch of rework for the PR submitter. So if you're game, I'd be happy to work out the approach here first. Alternatively, if you don't mind rework, you could submit your "approach" in the form of a WIP PR.
I've been told I'm a stickler for code style considerations in PRs, although this is hopefully not a big concern because it's less substantive / requires less rework.
Nope! No bug bounties currently for borgmatic, even if Borg has 'em.
Thank you!
Quoting myself here from above:
Here's an idea from a related ticket (#1043):
Maybe something like that would work? So loop through the source directories to backup, running
btrfs subvolume showon each path to determine if it's a Btrfs subvolume? If it is, usefindmntjust on that path to get its mount point...?In #1113, I have a proposed WIP implementation of this kind of introspection. It does still relies on
findmntas a first step, but it's (hopefully) no longer conflating Btrfs subvolumes with mount points. And so it should better support "flat" (vs. nested) subvolumes.That PR has been merged now as part of #1043.
I wanted to ask if it would be possible to activate snapshots for btrfs in general.
As I understand it so far, the function in borgmatic is only implemented for subvolumes.
So far I have used borgmatic action to create a snapshot of a folder before backing it up.
This works fine, but unfortunately the path is moved to the snapshot and not the actual folder.
Could you say more about what you have in mind here? Like, what would you want snapshotted besides subvolumes, and how do you foresee that being used?
Could you also say more about this? And what version of Borg are you using? The stored snapshot path within the Borg archive should be the actual folder path if you're using Borg 1.4+. Older versions of Borg (e.g. 1.2) don't support this, so the stored archive path will contain the snapshot directory when using these older Borg versions.
I have a NAS with shared Samba folders.
Each folder has its own use and a different backup interval.
All shared folders are on the same btrfs volume and are marked as subvolumes.
When I want to make a backup of one of these folders, I create a snapshot of it so that no files are changed during the backup process.
The problem is the new path.
borgmatic saves the path to the snapshot and not the folder itself that I want to backup.
I refer to the borgmatic documentation:
https://torsion.org/borgmatic/docs/how-to/snapshot-your-filesystems/
That's what I want to achieve for my backup.
This is one of my borgmatic configurations:
But when I use your btrfs flag instead of my commands
I get this message:
And no snapshot is created during the backup.
Setup:
System: Synology NAS 1819+ (DSM 7.2.2-72806 Update 3)
Borg: 1.4.1 (via SynoCommunity 1.4.1-18)
borgmatic: 2.0.3 (via SynoCommunity 1.4.1-18)
A few things:
/volume1/Datenshow up infindmntoutput as abtrfsfilesystem?btrfs subvolume show /volume1/Datenoutput?--verbose 2would be helpful here. (Feel free to redact.)Q: I recommend upgrading borgmatic to 2.0.7 and trying again. There were a few Btrfs improvements since borgmatic 2.0.3 that may be affecting things here.
A: pipx is unfortunately not available on the Synology system. Docker will probably not work due to the access to the host system. I used to be able to install borgmatic without SynoCommunity, but that was over 4 years ago.
Q: Does /volume1/Daten show up in findmnt output as a btrfs filesystem?
A: No, because it has no mountpoint, it is more like a nested subvolume.
https://btrfs.readthedocs.io/en/latest/Subvolumes.html
Q: What btrfs subvolume show /volume1/Daten output?
btrfs subvolume show /volume1/Daten/ after creating a snapshot
Q: Seeing your borgmatic logs with --verbose 2 would be helpful here. (Feel free to redact.)
Thanks for all of the additional details. In regards to upgrading, you don't have to use pipx to install borgmatic. You can even use
pip installorpip install --userif Synology has that available.The "good news" is that you don't actually need to upgrade right now because upgrading won't fix your problem—because borgmatic doesn't yet support snapshotting unmounted Btrfs subvolumes. The bad news is any eventual fix for this will still require a borgmatic upgrade.
In terms of solutions, here's what I can think of:
Also, would you mind telling me how this unmounted subvolume was created to begin with, either here or in whatever ticket you file? That would help me reproduce your setup.
There is no such thing "unmounted Btrfs subvolumes".
findmntwill never support normal subvolumes, because normal subvolumes do not, and should not, appear in /etc/fstab, /etc/mtab, or /proc/self/mountinfo.Stop using
findmntand either callbtrfs subvolume listor (recommended) use the python module.There's no such thing as an "unmounted subvolume", and it doesn't matter if it was created with mkfs.btrfs or with
btrfs subvolume create. You mount the fs using the block device. Specifying a subvolume is a btrfs-specific implementation of an optional namespacing feature that even ext4 now supports (here it uses a directory as the / of the mount).What? Argh.. web things.. Edititing now...
Isn't Gitea the odd one out here? I assume you've used Github, Gitlab, Buzilla, etc, all of which have functional reply-by-email? ;)
Yes, I understand that, but please answer the question. Why is mucking with special properties (and then unmucking) a better idea than creating an ephemeral read-only snapshot and leaving the source untouched?
Please provide a reproducer so I can see why your machine is failing at basic functionality. At this point one explanation is that your system is affected by bugs affecting btrfs special properties...
Ah, yes my convenient setup commands assume you've booted a VM that has a btrfs rootfs. Those will need to be modified for /arbitrary/mount/point, but I'd hope that borgmatic is being tested on a real system like users use.
My reproducer demonstrates that the claim that btrfs cannot snapshot a ro subvolume is false for the special property mucking case as well as the more reliable best-practises (ie create ro snapshot of a ro snapshot) cases.
I mean, yeah, you'll need to modify the command I provided to get the block device for / if you're trying to get it for /some_mnt/. But booting off of btrfs? There is no special magic there. It will take a few minutes to install a VM and run my reproducer. Actually, on that topic, does borgmatic have access to a VM-backed CI? It seems to me that these would make good sanity checks, before doing CI on borgmatics btrfs functionality.
No, there is something else on your test-bed that is leading to idiosyncratic results. Please provide a full reproducer.
The standard btrfs CLI is
/usr/bin/btrfsand its subcommands. (ie:btrfs subvolume list /mount/point) Have you read the examples for the python module here yet: https://github.com/knorrie/python-btrfs ?Yes, that's why I called it a plugin ;) I'm not talking about replacing the difficult-to-implement-correctly-and-reliably btrfs support with support for a specific blessed snapshot manager and then calling that borgmatic's btrfs support. In addition to snapper, there is an Ubuntu-specific one, and yabsnap. Please point me in the direction of the kind of interface you'd expect in the config.
If we're talking interfaces, the excellent choice to use YAML means that a take_new_snapshot boolean property can be added globally, or per-thing :)
Yes, this is what I'm asking about. I'm interested in implementing a snapshot-manager plugin because then I don't need to teach you btrfs ;) Would you please point me to a design/schema/interfaces document? What do you mean by "approach"? What I'm proposing is arguably just enabling syntactic sugar for the YAML config, plus two or three ultra-short example configurations as documentation. The objective is to make it near effortless for users.
That's fine with me :) It's potential fundamental irreconcilable differences that I'm worried about.
Unfortunately, I can't tell you exactly how Synology creates the subvolumes.
The OS is Linux but it is very modified.
The subvolume creation is controlled by scripts.
When a Samba shared folder is created via the web interface, a subvolume with the same name is created.
What I can tell you is that Synology does not use btrfs throughout, as some features of btrfs are not yet considered stable (e.g. RAID56).
With Synology, in a RAID 5 installation, each data disk is divided into 3 partitions.
In each of these partitions, a RAID is created across all disks.
The OS is installed on RAID A, a SWAP storage is created on RAID B and an LVM VG is created in RAID C.
All hard disks with their RAID partitions and the LVM VG are called a storage pool.
LVs are then created in the LVM VG which can be formatted either with EXT4 or btrfs. These are called volumes.
By now there is a lot of text in this issue, so even I can hardly get through it,
but to make it short, subvolumes in btrfs are file extent-based.
So they are always included in the hirachy of the btrfs FS-Tree.
This results in two possibilities for dealing with subvolumes:
https://archive.kernel.org/oldwiki/btrfs.wiki.kernel.org/index.php/SysadminGuide.html#Subvolumes
No matter how the subvolume is used later, the subvolume is created as a file extent-based FS-Tree within the btrfs partition.
The command is:
btrfs subvolume create <path/name>
While “findmnt” only shows mount points and gives you the content of /proc/self/mountinfo in a fine structured form,
"btrfs subvolume show " is better suited to find out if a folder/path directs to btrfs subvolume and what properties it has (snapshots, parent subvolumes, ...).
Here is a very minimalistic python script to get a json output (Python is not my primary language):
Problems with this script:
I asked ChatGPT to help me write a script to output the herachy on the btrfs partition.
Only subvolumes, no folders or files.
Looks like it actually does; it's just not configured here.
Changing the source subvolume's properties was an attempt to demonstrate what happens when snapshotting a read-only subvolume on my system. I'm not sure how to demonstrate that without ... starting with a read-only subvolume.
Are you instead suggesting that, rather than ever snapshotting a read-only subvolume, one should instead make a read-only snapshot of a read-write subvolume? I mean, sure, that seems like a reasonable thing to do. But it also seems orthogonal to the disputed claim that "Btrfs does not support snapshotting read-only subvolumes." Which incidentally I would be happy to be proven wrong about...
Here you go... This is what I'm running as root on Linux kernel 6.12.44-3-MANJARO:
And this is what I get:
If you're suggesting that there be manual testing on a btrfs-rooted system, then no, that currently doesn't happen.
Nope, CI is currently only container-based.
See above. Hopefully you can spot what I'm doing wrong.
I'll check those out and see if I can ditch
findmntforbtrfs. But that might require doing probing along the components of a path like I think https://github.com/knorrie/python-btrfs/blob/master/examples/show_subvolumes.py is doing. Something like:btrfs subvolume showthat directory to determine if it's a subvolume (if found, done for this directory)It really depends on the needs of the plugin/hook and the functionality it's exposing. For instance, here's the config for the existing Btrfs hook:
So the minimal config needed to turn on the hook is literally:
By "approach" I just mean implementation design: What you're planning on doing in the code behind the interface, what you're planning on adding to the config schema, etc. There isn't a formal design/schema/interfaces document for writing plugins/hooks in particular, but there is this: https://torsion.org/borgmatic/docs/reference/source-code/
Also, looking at an existing plugin/hook should hopefully be pretty clear in terms of the interface. The Btrfs hook might be a good place to start: https://projects.torsion.org/borgmatic-collective/borgmatic/src/branch/main/borgmatic/hooks/data_source/btrfs.py ... The functions that need implementation for any snapshot-oriented hook are:
use_streamingdump_data_sourcesremove_data_source_dumpsmake_data_source_dump_patternsrestore_data_source_dumpFair enough.
@smhrambo Thanks for the detailed response including the script and output. A couple of points:
btrfs subvolume showas you suggest instead offindmntwould likely address your issue. See my previous comment about one way that might work in regards to borgmatic'ssource_directories.I'll keep you updated about what I find in attempting to jettison
findmnt!That's no surprise.
You are adding a snapshot or subvolume to the read-only subvolume.
Internally, btrfs attempts to create a nested subvolume in the read-only subvolume, which constitutes a change to the read-only subvolume.
try this:
Regarding
LVM:This would negate the advantage of subvolumes in btrfs.
In btrfs, all subvolumes dynamically share the entire storage space available to the BTRFS partition.
This is not the case with LVM.
Although a shared storage pool is also available, it must be "permanently" divided into LVs.
For this solution, I would have to back up all data and set up the entire NAS again.
In addition, Synology encapsulates many of the functions that Linux server administrators are familiar with.
This makes it a gamble every time you tinker with the file system.
That's why I'm staying at the last level (btrfs).
Creating a simple subvolume on the share partition is easier than moving storage space back and forth between LVs.
Yes, there is an LVM with an LV, but that would require me to back up the entire disk with more than 16TB every time, and I have a separate backup strategy for each subvolume/network share.
Regarding
source_directories:I can mount the subvolume in a different location, but that would mean interfering with the Synology system again (which should be avoided at all costs).
The root/OS file system is ext4 and has specific mounting points for the partitions on which the network shares are located.
Here is a small excerpt from one of the configuration files:
cat /etc/samba/smb.conf
I know that my next system won't be a Synology, but that's just how it is right now.
If there is a problem during bootup, the only option is to restore the system, but this will delete all data.
So booting into recovery mode and fixing the error is not an option.
it's like fuck around and find out.
Ah that makes sense.. Thanks for explaining! I'm not sure there's a generalized way for borgmatic to take advantage of this though.. e.g., for borgmatic to put its snapshot path outside of the subvolume path being snapshotted. For instance, your example works great if there's a Btrfs root filesystem (or at minimum a parent Btrfs filesystem) and the snapshot is put on it, but AFAIK it won't work if there's some other non-Btrfs root or parent. So it might be easier to just leave the no-read-only-subvolume restriction (and update the docs to clarify that it's a borgmatic limitation and not a Btrfs one).
Okay, understood!
I follow the Synology limitations, but I'm not following how this is related to the proposed
source_directoriesprobing strategy to find Btrfs subvolumes described above. Or were you talking about something else?Maybe we really did talk past each other.
I was referring to this part:
I thought you were referring to this comment:
I would prefer not to create an extra mount point in the root directory of the Synology OS.
Especially when it comes to permanent configuration (fstab).
Oh yes, about btrfs and subvolumes.
This is precisely why we avoid mounting the “root” volume directly "as much as possible".
A subvolume is usually used as the Linux root directory,
in which the directory structure is created either by folders,
nested subvolumes, or separate subvolumes from the btrfs “root” volume.
btrfs root volume
There are no limits to your imagination.
I spent a lot of time working on this last month.
Dual boot, Windows and CachyOS with secure boot, grub, and btrfs.
What a pleasure (not).
Yeah, I think we were talking past each other. (This ticket thread is too long!) I was actually referring to this comment I made earlier:
I'll check those out and see if I can ditch
findmntforbtrfs. But that might require doing probing along the components of a path like I think https://github.com/knorrie/python-btrfs/blob/master/examples/show_subvolumes.py is doing. Something like:btrfs subvolume showthat directory to determine if it's a subvolume (if found, done for this directory)Does that sound like something that will work for you?
Hah, indeed. Okay, what I'm taking from this is that I'll leave the existing borgmatic Btrfs snapshot path construction for now, but we can always revisit later.
I've removed
findmntentirely from borgmatic main. This update will be part of the next release. So in the interests of keeping things moving, I'm closing this ticket but I'd be happy to continue the discussion. I'd recommend filing new tickets to discuss any other Btrfs issues, such as topics discussed above besidesfindmntremoval.I would like to find out if the code that replaces
findmmtdoes end up working for your use cases. If not, we can certainly iterate on the solution.Released in borgmatic 2.0.9!