btrfs #1048

Closed
opened 2025-03-28 10:46:28 +00:00 by freddo · 12 comments

What I'm trying to do and why

Hey, I am trying to use the btrfs hook for snapshotting. My system has two drives, both have usual GPT partitions formatted with btrfs (and dm-crypt in between), one is root and the other is mounted at /media; findmnt -t btrfs --list --json gives:

{
   "filesystems": [
      {
         "target": "/",
         "source": "/dev/mapper/root",
         "fstype": "btrfs",
         "options": "rw,relatime,ssd,discard=async,space_cache=v2,subvolid=5,subvol=/"
      },{
         "target": "/media",
         "source": "/dev/mapper/media",
         "fstype": "btrfs",
         "options": "rw,relatime,ssd,discard=async,space_cache=v2,subvolid=5,subvol=/"
      }
   ]
}

For both filesystems, I only want certain directories to be included in the backups but I don't mind btrfs snapshotting the whole filesystem.

I run into these paths do not exist errors only when adding the btrfs hook, the backup runs fine without the hook.

Thanks for any help in advance :)

Steps to reproduce

source_directories:
    - /home
    - /etc
    - /media/paperless
    - /media/data
btrfs:

In patterns I only have a few exludes, I don't think they are relevant.

Actual behavior

storagebox: Calling btrfs hook function dump_data_sources
storagebox: Snapshotting Btrfs subvolumes
storagebox: findmnt -t btrfs --json --list
storagebox: Creating Btrfs snapshot for / subvolume
storagebox: btrfs subvolume snapshot -r / /.borgmatic-snapshot-1603501
Create readonly snapshot of '/' in '/.borgmatic-snapshot-1603501'
storagebox: Creating Btrfs snapshot for /media subvolume
storagebox: btrfs subvolume snapshot -r /media /media/.borgmatic-snapshot-1603501/media
Create readonly snapshot of '/media' in '/media/.borgmatic-snapshot-1603501/media'
storagebox: Error running actions for repository
storagebox: Source directories / root pattern paths do not exist: /.borgmatic-snapshot-1603501/./tmp/borgmatic-cq3xwb7o/./borgmatic/bootstrap
/etc/borgmatic/config.yaml: No commands to run for on-error hook
/etc/borgmatic/config.yaml: An error occurred
No commands to run for post-everything hook

summary:
/etc/borgmatic/config.yaml: Loading configuration file
An error occurred
Error running actions for repository
Source directories / root pattern paths do not exist: /.borgmatic-snapshot-1603501/./tmp/borgmatic-cq3xwb7o/./borgmatic/bootstrap

Expected behavior

Create snapshots and use the respective directories from the snapshot.

Other notes / implementation ideas

#1043 might be related but I decided to post another issue because the usecases & system setups are different.

I guess I could setup btrfs-subvolumes for the directories I want to backup but I didn't really look into that yet because I feel this should also work without extra subvolumes.

borgmatic version

1.9.10

borgmatic installation method

pacman

Borg version

borg 1.4.0

Python version

Python 3.13.1

Database version (if applicable)

No response

Operating system and version

Arch Linux

### What I'm trying to do and why Hey, I am trying to use the btrfs hook for snapshotting. My system has two drives, both have usual GPT partitions formatted with btrfs (and dm-crypt in between), one is root and the other is mounted at /media; ``findmnt -t btrfs --list --json`` gives: ``` { "filesystems": [ { "target": "/", "source": "/dev/mapper/root", "fstype": "btrfs", "options": "rw,relatime,ssd,discard=async,space_cache=v2,subvolid=5,subvol=/" },{ "target": "/media", "source": "/dev/mapper/media", "fstype": "btrfs", "options": "rw,relatime,ssd,discard=async,space_cache=v2,subvolid=5,subvol=/" } ] } ``` For both filesystems, I only want certain directories to be included in the backups but I don't mind btrfs snapshotting the whole filesystem. I run into these ``paths do not exist`` errors only when adding the btrfs hook, the backup runs fine without the hook. Thanks for any help in advance :) ### Steps to reproduce ``` source_directories: - /home - /etc - /media/paperless - /media/data btrfs: ``` In ``patterns`` I only have a few exludes, I don't think they are relevant. ### Actual behavior ``` storagebox: Calling btrfs hook function dump_data_sources storagebox: Snapshotting Btrfs subvolumes storagebox: findmnt -t btrfs --json --list storagebox: Creating Btrfs snapshot for / subvolume storagebox: btrfs subvolume snapshot -r / /.borgmatic-snapshot-1603501 Create readonly snapshot of '/' in '/.borgmatic-snapshot-1603501' storagebox: Creating Btrfs snapshot for /media subvolume storagebox: btrfs subvolume snapshot -r /media /media/.borgmatic-snapshot-1603501/media Create readonly snapshot of '/media' in '/media/.borgmatic-snapshot-1603501/media' storagebox: Error running actions for repository storagebox: Source directories / root pattern paths do not exist: /.borgmatic-snapshot-1603501/./tmp/borgmatic-cq3xwb7o/./borgmatic/bootstrap /etc/borgmatic/config.yaml: No commands to run for on-error hook /etc/borgmatic/config.yaml: An error occurred No commands to run for post-everything hook summary: /etc/borgmatic/config.yaml: Loading configuration file An error occurred Error running actions for repository Source directories / root pattern paths do not exist: /.borgmatic-snapshot-1603501/./tmp/borgmatic-cq3xwb7o/./borgmatic/bootstrap ``` ### Expected behavior Create snapshots and use the respective directories from the snapshot. ### Other notes / implementation ideas #1043 might be related but I decided to post another issue because the usecases & system setups are different. I guess I could setup btrfs-subvolumes for the directories I want to backup but I didn't really look into that yet because I feel this should also work without extra subvolumes. ### borgmatic version 1.9.10 ### borgmatic installation method pacman ### Borg version borg 1.4.0 ### Python version Python 3.13.1 ### Database version (if applicable) _No response_ ### Operating system and version Arch Linux
Owner

Thanks for filing this. It looks like you're seeing a possible interaction between several different borgmatic features, although I'm not sure why it's resulting in an error. First, code that supports the borgmatic config bootstrap action is writing bootstrap metadata to /tmp/borgmatic-cq3xwb7o/borgmatic/bootstrap so that it ends up in the Borg archive. Then, the Btrfs hook is altering that file path to take into account the snapshot borgmatic makes of / (since /tmp/... is on /). Lastly, borgmatic is ensuring that all paths being passed to Borg as part of borg create actually exist. And the error you're getting indicates that that file path doesn't exist for some reason.

Since borgmatic is blowing up mid-run, you might be able to inspect the filesystem exactly as it was when the error occurred. So for instance does /.borgmatic-snapshot-1603501/ actually exist? (Substitute the actual snapshot path from the most recent run.) Does /.borgmatic-snapshot-1603501/tmp/borgmatic-cq3xwb7o/ exist? (Again, update the path accordingly.) Does /.borgmatic-snapshot-1603501/tmp/borgmatic-cq3xwb7o/borgmatic/bootstrap exist?

In any case, you can disable the feature that ensures paths sent to Borg actually exist by adding the following to your configuration:

source_directories_must_exist: false

... or by just commenting out the option if it's already set to true, which I suspect it is.

That may also make the error go away, although if the path doesn't exist there may be some underlying problem.

Separately, you can temporarily disable the bootstrap feature by adding the following to your configuration file:

bootstrap:
    store_config_files: false

I'd also be interested to see if that "solves" the problem.

Thanks for filing this. It looks like you're seeing a possible interaction between several different borgmatic features, although I'm not sure why it's resulting in an error. First, code that supports the `borgmatic config bootstrap` action is writing bootstrap metadata to `/tmp/borgmatic-cq3xwb7o/borgmatic/bootstrap` so that it ends up in the Borg archive. Then, the Btrfs hook is altering that file path to take into account the snapshot borgmatic makes of `/` (since `/tmp/...` is on `/`). Lastly, borgmatic is ensuring that all paths being passed to Borg as part of `borg create` actually exist. And the error you're getting indicates that that file path doesn't exist for some reason. Since borgmatic is blowing up mid-run, you might be able to inspect the filesystem exactly as it was when the error occurred. So for instance does `/.borgmatic-snapshot-1603501/` actually exist? (Substitute the actual snapshot path from the most recent run.) Does `/.borgmatic-snapshot-1603501/tmp/borgmatic-cq3xwb7o/` exist? (Again, update the path accordingly.) Does `/.borgmatic-snapshot-1603501/tmp/borgmatic-cq3xwb7o/borgmatic/bootstrap` exist? In any case, you can disable the feature that ensures paths sent to Borg actually exist by adding the following to your configuration: ```yaml source_directories_must_exist: false ``` ... or by just commenting out the option if it's already set to `true`, which I suspect it is. That may also make the error go away, although if the path doesn't exist there may be some underlying problem. Separately, you can temporarily disable the bootstrap feature by adding the following to your configuration file: ```yaml bootstrap: store_config_files: false ``` I'd also be interested to see if that "solves" the problem.
Author

Thanks for the quick response.

Since borgmatic is blowing up mid-run, you might be able to inspect the filesystem exactly as it was when the error occurred.

Yes, you're right, I had already noticed this while debugging (though it gets deleted by a subsequent run of borgmatic which then fails with a slightly different error, I'll post that error below just in case it has additional info for you).

Does /.borgmatic-snapshot-1603501/tmp/borgmatic-cq3xwb7o/ exist?

No, /.borgmatic-snapshot-1703188/tmp/ is empty actually. Other directories like /etc look correctly snapshotted. I just noticed that /tmp is probably not snapshotted because /tmp is a tmpfs (on ArchLinux). So I guess the btrfs hook is replacing all paths without taking into account that some of the directories might not be part of the snapshotted filesystem?


Subsequent run deleting old snapshot and failing with different error:

toragebox: Creating archive
storagebox: Using runtime directory /tmp/borgmatic-5he28d68/borgmatic
storagebox: Calling bootstrap hook function remove_data_source_dumps
storagebox: Looking for bootstrap manifest files to remove in /tmp/borgmatic-*/borgmatic/bootstrap
storagebox: Calling btrfs hook function remove_data_source_dumps
storagebox: findmnt -t btrfs --json --list
storagebox: Looking for snapshots to remove in /media/.borgmatic-snapshot-*/media
storagebox: Deleting Btrfs snapshot /media/.borgmatic-snapshot-1603501/media
storagebox: btrfs subvolume delete /media/.borgmatic-snapshot-1603501/media
Delete subvolume 267 (no-commit): '/media/.borgmatic-snapshot-1603501/media'
storagebox: Looking for snapshots to remove in /.borgmatic-snapshot-*
storagebox: Deleting Btrfs snapshot /.borgmatic-snapshot-1603501
storagebox: btrfs subvolume delete /.borgmatic-snapshot-1603501
Delete subvolume 269 (no-commit): '//.borgmatic-snapshot-1603501'
storagebox: Error running actions for repository
storagebox: [Errno 2] No such file or directory: ''
/etc/borgmatic/config.yaml: No commands to run for on-error hook
/etc/borgmatic/config.yaml: An error occurred
No commands to run for post-everything hook

summary:
/etc/borgmatic/config.yaml: Loading configuration file
An error occurred
Error running actions for repository
[Errno 2] No such file or directory: ''
Thanks for the quick response. > Since borgmatic is blowing up mid-run, you might be able to inspect the filesystem exactly as it was when the error occurred. Yes, you're right, I had already noticed this while debugging (though it gets deleted by a subsequent run of borgmatic which then fails with a slightly different error, I'll post that error below just in case it has additional info for you). > Does /.borgmatic-snapshot-1603501/tmp/borgmatic-cq3xwb7o/ exist? No, ``/.borgmatic-snapshot-1703188/tmp/`` is empty actually. Other directories like ``/etc`` look correctly snapshotted. I just noticed that ``/tmp`` is probably not snapshotted because ``/tmp`` is a tmpfs (on ArchLinux). So I guess the btrfs hook is replacing all paths without taking into account that some of the directories might not be part of the snapshotted filesystem? --- Subsequent run deleting old snapshot and failing with different error: ``` toragebox: Creating archive storagebox: Using runtime directory /tmp/borgmatic-5he28d68/borgmatic storagebox: Calling bootstrap hook function remove_data_source_dumps storagebox: Looking for bootstrap manifest files to remove in /tmp/borgmatic-*/borgmatic/bootstrap storagebox: Calling btrfs hook function remove_data_source_dumps storagebox: findmnt -t btrfs --json --list storagebox: Looking for snapshots to remove in /media/.borgmatic-snapshot-*/media storagebox: Deleting Btrfs snapshot /media/.borgmatic-snapshot-1603501/media storagebox: btrfs subvolume delete /media/.borgmatic-snapshot-1603501/media Delete subvolume 267 (no-commit): '/media/.borgmatic-snapshot-1603501/media' storagebox: Looking for snapshots to remove in /.borgmatic-snapshot-* storagebox: Deleting Btrfs snapshot /.borgmatic-snapshot-1603501 storagebox: btrfs subvolume delete /.borgmatic-snapshot-1603501 Delete subvolume 269 (no-commit): '//.borgmatic-snapshot-1603501' storagebox: Error running actions for repository storagebox: [Errno 2] No such file or directory: '' /etc/borgmatic/config.yaml: No commands to run for on-error hook /etc/borgmatic/config.yaml: An error occurred No commands to run for post-everything hook summary: /etc/borgmatic/config.yaml: Loading configuration file An error occurred Error running actions for repository [Errno 2] No such file or directory: '' ```
Owner

No, /.borgmatic-snapshot-1703188/tmp/ is empty actually. Other directories like /etc look correctly snapshotted. I just noticed that /tmp is probably not snapshotted because /tmp is a tmpfs (on ArchLinux). So I guess the btrfs hook is replacing all paths without taking into account that some of the directories might not be part of the snapshotted filesystem?

Ah, good find! Yes, that seems likely what's going on. And now I have a local repro as well. Since I don't have a system with a Btrfs root subvolume, I created a tmpfs mount within a subdirectory of my one Btrfs subvolume and pointed the user_runtime_directory configuration option at that tmpfs mount point. And then I'm getting the same "do not exist" behavior you're seeing when source_directories_must_exist is true.

I'm not sure what's going on with deleting the old snapshot, but my guess it's related to it being the root subvolume. I probably should spin up a VM with a Btrfs root...

> No, /.borgmatic-snapshot-1703188/tmp/ is empty actually. Other directories like /etc look correctly snapshotted. I just noticed that /tmp is probably not snapshotted because /tmp is a tmpfs (on ArchLinux). So I guess the btrfs hook is replacing all paths without taking into account that some of the directories might not be part of the snapshotted filesystem? Ah, good find! Yes, that seems likely what's going on. And now I have a local repro as well. Since I don't have a system with a Btrfs root subvolume, I created a `tmpfs` mount within a subdirectory of my one Btrfs subvolume and pointed the `user_runtime_directory` configuration option at that `tmpfs` mount point. And then I'm getting the same "do not exist" behavior you're seeing when `source_directories_must_exist` is `true`. I'm not sure what's going on with deleting the old snapshot, but my guess it's related to it being the root subvolume. I probably should spin up a VM with a Btrfs root...
Owner

Okay, the first error should be fixed in main and will be part of the next release. For the second error, can you try updating to 1.9.14 to see if you can repro the issue? There have been a few Btrfs fixes since 1.9.10. Thanks!

Okay, the first error should be fixed in main and will be part of the next release. For the second error, can you try updating to 1.9.14 to see if you can repro the issue? There have been a few Btrfs fixes since 1.9.10. Thanks!
Author

Heya, sorry for not reacting in a while. I just had some time to tackle the btrfs hook again and had a successful run with it. However, after rebooting, I am getting this error:

storagebox: Error running actions for repository
storagebox: Command '('btrfs', 'property', 'get', '-t', 'subvol', '/var/lib/containers/storage/overlay', 'ro')' returned non-zero exit status 1.
/etc/borgmatic/config.yaml: Error running configuration
/etc/borgmatic/config.yaml: An error occurred

I guess the overlay gets created by podman or something in that direction. I noticed that findmnt sometimes returns this overlay (sub)volume, sometimes not (I have a few containers running on this system but they are running at all times). I don't really understand whether it is actually a btrfs volume or not and how I could determine this 🤷‍♂️

Current findmnt output:

{
   "filesystems": [
      {
         "target": "/",
         "source": "/dev/mapper/root",
         "fstype": "btrfs",
         "options": "rw,relatime,ssd,discard=async,space_cache=v2,subvolid=5,subvol=/"
      },{
         "target": "/media",
         "source": "/dev/mapper/media",
         "fstype": "btrfs",
         "options": "rw,relatime,ssd,discard=async,space_cache=v2,subvolid=5,subvol=/"
      },{
         "target": "/var/lib/containers/storage/overlay",
         "source": "/dev/mapper/root[/var/lib/containers/storage/overlay]",
         "fstype": "btrfs",
         "options": "rw,relatime,ssd,discard=async,space_cache=v2,subvolid=5,subvol=/"
      }
   ]
}

For now I solved this by creating a findmnt wrapper script that removes the overlay json object. So nothing urgent, just wanted to update this issue after some time

Heya, sorry for not reacting in a while. I just had some time to tackle the btrfs hook again and had a successful run with it. However, after rebooting, I am getting this error: ``` storagebox: Error running actions for repository storagebox: Command '('btrfs', 'property', 'get', '-t', 'subvol', '/var/lib/containers/storage/overlay', 'ro')' returned non-zero exit status 1. /etc/borgmatic/config.yaml: Error running configuration /etc/borgmatic/config.yaml: An error occurred ``` I guess the overlay gets created by podman or something in that direction. I noticed that ``findmnt`` sometimes returns this overlay (sub)volume, sometimes not (I have a few containers running on this system but they are running at all times). I don't really understand whether it is actually a btrfs volume or not and how I could determine this 🤷‍♂️ Current findmnt output: ``` { "filesystems": [ { "target": "/", "source": "/dev/mapper/root", "fstype": "btrfs", "options": "rw,relatime,ssd,discard=async,space_cache=v2,subvolid=5,subvol=/" },{ "target": "/media", "source": "/dev/mapper/media", "fstype": "btrfs", "options": "rw,relatime,ssd,discard=async,space_cache=v2,subvolid=5,subvol=/" },{ "target": "/var/lib/containers/storage/overlay", "source": "/dev/mapper/root[/var/lib/containers/storage/overlay]", "fstype": "btrfs", "options": "rw,relatime,ssd,discard=async,space_cache=v2,subvolid=5,subvol=/" } ] } ``` For now I solved this by creating a findmnt wrapper script that removes the overlay json object. So nothing urgent, just wanted to update this issue after some time
Owner

Thanks for the update. If you run btrfs property get -t subvol /var/lib/containers/storage/overlay ro, do you get any more of a useful error message?

I don't really understand whether it is actually a btrfs volume or not and how I could determine this 🤷‍♂️

You could try btrfs subvolume show /var/lib/containers/storage/overlay for that.

Possibly related: #1043.

Thanks for the update. If you run `btrfs property get -t subvol /var/lib/containers/storage/overlay ro`, do you get any more of a useful error message? > I don't really understand whether it is actually a btrfs volume or not and how I could determine this 🤷‍♂️ You could try `btrfs subvolume show /var/lib/containers/storage/overlay` for that. Possibly related: #1043.
Owner

I'm still interested in the results of the above commands on your machine, but as part of #1043, I changed the Btrfs code to swallow errors coming from btrfs property get. So that particular subvolume should no longer cause errors for you. This is in main and will be part of the next release.

I'm still interested in the results of the above commands on your machine, but as part of #1043, I changed the Btrfs code to swallow errors coming from `btrfs property get`. So that particular subvolume should no longer cause errors for you. This is in main and will be part of the next release.
Author

This is in main and will be part of the next release.

Great to hear, I'll check out the next release when it's out.

Commands just return errors:

btrfs property get -t subvol /var/lib/containers/storage/overlay ro

ERROR: Could not get subvolume flags: Invalid argument

btrfs subvolume show /var/lib/containers/storage/overlay

ERROR: Not a Btrfs subvolume: Invalid argument

> This is in main and will be part of the next release. Great to hear, I'll check out the next release when it's out. Commands just return errors: > btrfs property get -t subvol /var/lib/containers/storage/overlay ro ERROR: Could not get subvolume flags: Invalid argument > btrfs subvolume show /var/lib/containers/storage/overlay `ERROR: Not a Btrfs subvolume: Invalid argument`
Owner

Interesting! Hopefully that will get taken care of by the next release.

Interesting! Hopefully that will get taken care of by the next release.
Owner

Released in borgmatic 2.0.7! Feedback welcome.

Released in borgmatic 2.0.7! Feedback welcome.
Author

Setup phase in 2.0.7 works great now, the log still shows error but they don't stop the run.
However, I am not sure how the mapping from the actual files to the snapshot works (and should work). With the btrfs hook, all files are added again even when they haven't changed since the last archive (without the hook) was created. Now it goes over all files again with A /.borgmatic-snapshot-2824430/{file_path}. It would be kinda annoying to have to run a full backup again and I am not sure how all of that is stored in the archive (looking at storage usage) and how it would restore when I need it. Any advice or insight would be hugely appreciated :)

Setup phase in 2.0.7 works great now, the log still shows error but they don't stop the run. However, I am not sure how the mapping from the actual files to the snapshot works (and should work). With the btrfs hook, all files are added again even when they haven't changed since the last archive (without the hook) was created. Now it goes over all files again with ``A /.borgmatic-snapshot-2824430/{file_path}``. It would be kinda annoying to have to run a full backup again and I am not sure how all of that is stored in the archive (looking at storage usage) and how it would restore when I need it. Any advice or insight would be hugely appreciated :)
Owner

I'm glad to hear 2.0.7 works for you now!

A couple points on your comment:

  • Try to borgmatic list --archive latest and inspect the resulting file paths in your archive. Through the magic of path prefix stripping (a Borg feature that borgmatic makes use of), the actually stored file paths should not have the /.borgmatic-snapshot-2824430 path prefix at all. They should reflect their original path locations on your filesystem as if they had never been snapshotted.
  • Even so, there are some performance implications of the way that borgmatic stores these paths. Read more about that here: https://torsion.org/borgmatic/reference/configuration/data-sources/btrfs/#performance
I'm glad to hear 2.0.7 works for you now! A couple points on your comment: - Try to `borgmatic list --archive latest` and inspect the resulting file paths in your archive. Through the magic of path prefix stripping (a Borg feature that borgmatic makes use of), the actually stored file paths should not have the `/.borgmatic-snapshot-2824430` path prefix at all. They should reflect their original path locations on your filesystem as if they had never been snapshotted. - Even so, there are some performance implications of the way that borgmatic stores these paths. Read more about that here: https://torsion.org/borgmatic/reference/configuration/data-sources/btrfs/#performance
Sign in to join this conversation.
No milestone
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
borgmatic-collective/borgmatic#1048
No description provided.