Snapshotting of read-only `/etc` directory with systemd service and btrfs

DJsushi commented

2025-03-09 21:21:41 +00:00

What I'm trying to do and why

I am trying to set up Borgmatic with btrfs and schedule the running of borgmatic using a systemd timer since I want it to run also when the timer ticks when the PC is off. However, I have had problems doing so because of how the systemd template service is set up and how Borgmatic checks the snapshottable subvolumes.

When I run findmnt -t btrfs --list normally, this is what I get:

TARGET SOURCE                  FSTYPE OPTIONS
/      /dev/mapper/root        btrfs  rw,nodev,relatime,ssd,space_cache=v2,subvolid=5,subvol=/
/home  /dev/mapper/root[/home] btrfs  rw,relatime,ssd,space_cache=v2,subvolid=256,subvol=/home

I have a root subvolume mounted and a /home subvolume mounted as well. I have other subvolumes but they're not mounted so Borgmatic ignores them. These two are the only ones that need to be snapshotted by Borgmatic in order to back up my system.

However, when borgmatic is run inside the systemd service, it detects many other mounted filesystems. This is the output of findmnt -t btrfs --list inside of the systemd service:

TARGET               SOURCE                                                                                                   FSTYPE OPTIONS
 /                    /dev/mapper/root                                                                                         btrfs  rw,nodev,relatime,ssd,space_cache=v2,subvolid=5,subvol=/
├─/etc               /dev/mapper/root[/etc]                                                                                   btrfs  ro,nosuid,nodev,relatime,ssd,space_cache=v2,subvolid=5,subvol=/
├─/home              /dev/mapper/root[/home]                                                                                  btrfs  rw,relatime,ssd,space_cache=v2,subvolid=256,subvol=/home
├─/boot              /dev/mapper/root[/boot]                                                                                  btrfs  ro,nosuid,nodev,relatime,ssd,space_cache=v2,subvolid=5,subvol=/
├─/usr               /dev/mapper/root[/usr]                                                                                   btrfs  ro,nosuid,nodev,relatime,ssd,space_cache=v2,subvolid=5,subvol=/
├─/var/lib/borgmatic /dev/mapper/root[/var/lib/borgmatic]                                                                     btrfs  rw,nosuid,nodev,relatime,ssd,space_cache=v2,subvolid=258,subvol=/var
└─/var/tmp           /dev/mapper/root[/var/tmp/systemd-private-79bbcd3eaff942718bc84b6df0b676ca-borgmatic.service-Q4yJEj/tmp] btrfs  rw,nosuid,nodev,relatime,ssd,space_cache=v2,subvolid=261,subvol=/var/tmp

As you can see, it detects many more subvolumes and tries to snapshot them all. So, my Borgmatic systemd service fails with this error:

 borgbase-eu: Snapshotting Btrfs subvolumes
INFO borgbase-eu: Snapshotting Btrfs subvolumes
borgbase-eu: findmnt -t btrfs --json --list
borgbase-eu: Creating Btrfs snapshot for / subvolume
borgbase-eu: btrfs subvolume snapshot -r / /.borgmatic-snapshot-47392
Create readonly snapshot of '/' in '/.borgmatic-snapshot-47392'
borgbase-eu: Creating Btrfs snapshot for /etc subvolume
borgbase-eu: Error running actions for repository
CRITICAL borgbase-eu: Error running actions for repository
CRITICAL borgbase-eu: [Errno 30] Read-only file system: '/etc/.borgmatic-snapshot-47392'

Steps to reproduce

Use btrfs, download the Borgmatic systemd service from the link above and run borgmatic using the systemd service.

Actual behavior

Borgmatic fails because it tries to snapshot read only filesystems.

Expected behavior

Not sure what is to be expected. I think Borgmatic should just snapshot / and since /etc is inside of the / subvolume, it should just read the contents of /etc from /.borgmatic-snapshot-12345/etc/.

Other notes / implementation ideas

No response

borgmatic version

1.9.13

borgmatic installation method

Archlinux package

Borg version

1.4.0

Python version

3.13.2

Database version (if applicable)

No response

Operating system and version

NAME="Arch Linux" PRETTY_NAME="Arch Linux" ID=arch BUILD_ID=rolling ANSI_COLOR="38;2;23;147;209" HOME_URL="https://archlinux.org/" DOCUMENTATION_URL="https://wiki.archlinux.org/" SUPPORT_URL="https://bbs.archlinux.org/" BUG_REPORT_URL="https://gitlab.archlinux.org/groups/archlinux/-/issues" PRIVACY_POLICY_URL="https://terms.archlinux.org/docs/privacy-policy/"

### What I'm trying to do and why I am trying to set up Borgmatic with btrfs and schedule the running of borgmatic using a systemd timer since I want it to run also when the timer ticks when the PC is off. However, I have had problems doing so because of how the [systemd template service](https://projects.torsion.org/borgmatic-collective/borgmatic/raw/branch/main/sample/systemd/borgmatic.service) is set up and how Borgmatic checks the snapshottable subvolumes. When I run `findmnt -t btrfs --list` normally, this is what I get: ``` TARGET SOURCE FSTYPE OPTIONS / /dev/mapper/root btrfs rw,nodev,relatime,ssd,space_cache=v2,subvolid=5,subvol=/ /home /dev/mapper/root[/home] btrfs rw,relatime,ssd,space_cache=v2,subvolid=256,subvol=/home ``` I have a root subvolume mounted and a /home subvolume mounted as well. I have other subvolumes but they're not mounted so Borgmatic ignores them. These two are the only ones that need to be snapshotted by Borgmatic in order to back up my system. However, when `borgmatic` is run inside the systemd service, it detects many other mounted filesystems. This is the output of `findmnt -t btrfs --list` **inside** of the systemd service: ``` TARGET SOURCE FSTYPE OPTIONS / /dev/mapper/root btrfs rw,nodev,relatime,ssd,space_cache=v2,subvolid=5,subvol=/ ├─/etc /dev/mapper/root[/etc] btrfs ro,nosuid,nodev,relatime,ssd,space_cache=v2,subvolid=5,subvol=/ ├─/home /dev/mapper/root[/home] btrfs rw,relatime,ssd,space_cache=v2,subvolid=256,subvol=/home ├─/boot /dev/mapper/root[/boot] btrfs ro,nosuid,nodev,relatime,ssd,space_cache=v2,subvolid=5,subvol=/ ├─/usr /dev/mapper/root[/usr] btrfs ro,nosuid,nodev,relatime,ssd,space_cache=v2,subvolid=5,subvol=/ ├─/var/lib/borgmatic /dev/mapper/root[/var/lib/borgmatic] btrfs rw,nosuid,nodev,relatime,ssd,space_cache=v2,subvolid=258,subvol=/var └─/var/tmp /dev/mapper/root[/var/tmp/systemd-private-79bbcd3eaff942718bc84b6df0b676ca-borgmatic.service-Q4yJEj/tmp] btrfs rw,nosuid,nodev,relatime,ssd,space_cache=v2,subvolid=261,subvol=/var/tmp ``` As you can see, it detects many more subvolumes and tries to snapshot them all. So, my Borgmatic systemd service fails with this error: ``` borgbase-eu: Snapshotting Btrfs subvolumes INFO borgbase-eu: Snapshotting Btrfs subvolumes borgbase-eu: findmnt -t btrfs --json --list borgbase-eu: Creating Btrfs snapshot for / subvolume borgbase-eu: btrfs subvolume snapshot -r / /.borgmatic-snapshot-47392 Create readonly snapshot of '/' in '/.borgmatic-snapshot-47392' borgbase-eu: Creating Btrfs snapshot for /etc subvolume borgbase-eu: Error running actions for repository CRITICAL borgbase-eu: Error running actions for repository CRITICAL borgbase-eu: [Errno 30] Read-only file system: '/etc/.borgmatic-snapshot-47392' ``` ### Steps to reproduce Use btrfs, download the Borgmatic systemd service from the link above and run borgmatic using the systemd service. ### Actual behavior Borgmatic fails because it tries to snapshot read only filesystems. ### Expected behavior Not sure what is to be expected. I think Borgmatic should just snapshot `/` and since `/etc` is inside of the `/` subvolume, it should just read the contents of `/etc` from `/.borgmatic-snapshot-12345/etc/`. ### Other notes / implementation ideas _No response_ ### borgmatic version 1.9.13 ### borgmatic installation method Archlinux package ### Borg version 1.4.0 ### Python version 3.13.2 ### Database version (if applicable) _No response_ ### Operating system and version NAME="Arch Linux" PRETTY_NAME="Arch Linux" ID=arch BUILD_ID=rolling ANSI_COLOR="38;2;23;147;209" HOME_URL="https://archlinux.org/" DOCUMENTATION_URL="https://wiki.archlinux.org/" SUPPORT_URL="https://bbs.archlinux.org/" BUG_REPORT_URL="https://gitlab.archlinux.org/groups/archlinux/-/issues" PRIVACY_POLICY_URL="https://terms.archlinux.org/docs/privacy-policy/"

DJsushi changed title from ~~Snapshotting of read-only /etc directory with systemd service~~ to Snapshotting of read-only /etc directory with systemd service and btrfs

2025-03-09 21:21:53 +00:00

witten commented

2025-03-10 00:04:44 +00:00

Owner

Thanks for taking the time to file this. (I'm glad you found your way here from Reddit!) I'll look into this when I get a chance and: 1. See if I can repro the extra read-only subvolumes with systemd, and 2. Hopefully figure out a solution to them causing read-only filesystem errors. My guess is that snapshotting read-only subvolumes should actually be possible and that the Btrfs hook is doing something to break that. E.g., I see there's a makedirs() call that may actually be the cause of that error you're seeing.

Side note: Are you by chance using Snapper or anything that might be impacting the subvolumes?

Thanks for taking the time to file this. (I'm glad you found your way here from Reddit!) I'll look into this when I get a chance and: 1. See if I can repro the extra read-only subvolumes with systemd, and 2. Hopefully figure out a solution to them causing read-only filesystem errors. My _guess_ is that snapshotting read-only subvolumes should actually be possible and that the Btrfs hook is doing something to break that. E.g., I see there's a `makedirs()` call that may actually be the cause of that error you're seeing. Side note: Are you by chance using Snapper or anything that might be impacting the subvolumes?

witten referenced this issue from a commit

2025-03-10 06:05:16 +00:00

Fix an error in the Btrfs hook when attempting to snapshot a read-only subvolume (#1023).

witten commented

2025-03-10 06:08:56 +00:00

Owner

Turns out that this wasn't just a problem with making the snapshot directory; Btrfs really can't snapshot a read-only subvolume. So what I did was make borgmatic ignore any such read-only subvolumes when figuring out which subvolumes to snapshot. Now only read-write subvolumes will be considered. I didn't have a repro of your systemd setup, but I did test with both read-only and read-write subvolumes, and this fix should hopefully fix your use case.

This is implemented in main and will be part of the next release. Thanks again!

Turns out that this wasn't _just_ a problem with making the snapshot directory; Btrfs really can't snapshot a read-only subvolume. So what I did was make borgmatic ignore any such read-only subvolumes when figuring out which subvolumes to snapshot. Now only read-write subvolumes will be considered. I didn't have a repro of your systemd setup, but I did test with both read-only and read-write subvolumes, and this fix should hopefully fix your use case. This is implemented in main and will be part of the next release. Thanks again!

witten closed this issue

2025-03-10 06:08:56 +00:00

witten referenced this issue from a commit

2025-03-10 17:15:55 +00:00

Fix end-to-end tests for Btrfs (#1023).

witten added the

bug

label

2025-03-10 17:16:37 +00:00

witten commented

2025-03-10 17:29:39 +00:00

Owner