Error using ZFS snapshotting #1001

New issue

Closed

opened 2025-02-18 02:45:01 +00:00 by kinetic5 · 12 comments

kinetic5 commented 2025-02-18 02:45:01 +00:00

Copy link

What I'm trying to do and why

I'm trying to use the ZFS snapshot feature of borgmatic to backup some files. I'm having an issue where borgmatic is attempting to snapshot and mount my root filesystem when I haven't specified it to do so. I am seeing a read-only filesystem error, from where it looks like it is trying to mount my actual dataset on top of the snapshotted root filesystem (which isn't possible since it is readonly).

truncated config file:

source_directories:
    - /dpool/shares/staticfiles

source_directories_must_exist: true

compression: zstd

upload_rate_limit: 5000

one_file_system: true

zfs:

Any idea how to fix this/what I'm doing wrong?

Steps to reproduce

No response

Actual behavior

No response

Expected behavior

No response

Other notes / implementation ideas

No response

borgmatic version

1.9.10

borgmatic installation method

pip install

Borg version

1.4.0

Python version

Python 3.11.2

Database version (if applicable)

No response

Operating system and version

Debian GNU/Linux 12 (Proxmox)

### What I'm trying to do and why I'm trying to use the ZFS snapshot feature of borgmatic to backup some files. I'm having an issue where borgmatic is attempting to snapshot and mount my root filesystem when I haven't specified it to do so. I am seeing a read-only filesystem error, from where it looks like it is trying to mount my actual dataset on top of the snapshotted root filesystem (which isn't possible since it is readonly).  truncated config file: ``` source_directories: - /dpool/shares/staticfiles source_directories_must_exist: true compression: zstd upload_rate_limit: 5000 one_file_system: true zfs: ``` Any idea how to fix this/what I'm doing wrong? ### Steps to reproduce _No response_ ### Actual behavior _No response_ ### Expected behavior _No response_ ### Other notes / implementation ideas _No response_ ### borgmatic version 1.9.10 ### borgmatic installation method pip install ### Borg version 1.4.0 ### Python version Python 3.11.2 ### Database version (if applicable) _No response_ ### Operating system and version Debian GNU/Linux 12 (Proxmox)

image.png

45 KiB

witten commented 2025-02-18 03:43:36 +00:00

Owner

Copy link

Thanks for reporting this... I don't think you're necessarily doing anything wrong, and you've identified at least two different issues manifesting here:

1. borgmatic is attempting to snapshot your root filesystem when that filesystem isn't configured for backup. I'm not sure what's going wrong there, but there is logic that attempts to associate each configured source directory with one of the detected ZFS datasets, starting from the longest dataset path and then proceeding to the shortest (so, the root). In theory, if a source directory matches a longer path, it should never be associated with a shorter one. In theory.
2. borgmatic is attempting to mount your actual dataset's snapshot over the root snapshot mount point. This one is clearer. Looking at your log output, I think what's going on is that, well, I never tested with a root ZFS filesystem during development because I don't have a machine with a root ZFS filesystem. And so there just happens to be a really inconvenient overlap between the root snapshot mount point and your staticfiles snapshot mount point (because there's no final path component for a root snapshot mount point). I think the fix here may have to be a special case for any snapshotted root datasets to avoid that overlap.

Some things that might help debug the first point:

• Could I see the output of the following? zfs list -H -t filesystem -o name,mountpoint,org.torsion.borgmatic:backup
• Does the root still get snapshotted if you comment out source_directories?

Thanks for reporting this... I don't think you're necessarily doing anything wrong, and you've identified at least two different issues manifesting here: 1. borgmatic is attempting to snapshot your root filesystem when that filesystem isn't configured for backup. I'm not sure what's going wrong there, but there is logic that attempts to associate each configured source directory with one of the detected ZFS datasets, starting from the longest dataset path and then proceeding to the shortest (so, the root). In theory, if a source directory matches a longer path, it should never be associated with a shorter one. In theory. 2. borgmatic is attempting to mount your actual dataset's snapshot over the root snapshot mount point. This one is clearer. Looking at your log output, I think what's going on is that, well, I never tested with a root ZFS filesystem during development because I don't have a machine with a root ZFS filesystem. And so there just happens to be a really inconvenient overlap between the root snapshot mount point and your `staticfiles` snapshot mount point (because there's no final path component for a root snapshot mount point). I think the fix here may have to be a special case for any snapshotted root datasets to avoid that overlap. Some things that might help debug the first point: * Could I see the output of the following? `zfs list -H -t filesystem -o name,mountpoint,org.torsion.borgmatic:backup` * Does the root still get snapshotted if you comment out `source_directories`?

kinetic5 commented 2025-02-18 08:53:13 +00:00

Author

Copy link

Thanks for the quick response, here is the output from zfs list.

I currently have the org.torsion.borgmatic:backup property unset, but I have tried both methods of first using only source_directories to specify the dataset and then instead using the org.torsion.borgmatic:backup=auto property on the dataset with source_directories commented out. It seems to produce the same result in either case where it snapshots my root pool and tries to mount it with the data pool on top.

Thanks for the quick response, here is the output from zfs list.  I currently have the `org.torsion.borgmatic:backup` property unset, but I have tried both methods of first using only source_directories to specify the dataset and then instead using the `org.torsion.borgmatic:backup=auto` property on the dataset with `source_directories` commented out. It seems to produce the same result in either case where it snapshots my root pool and tries to mount it with the data pool on top.

image.png

53 KiB

witten commented 2025-02-18 18:19:28 +00:00

Owner

Copy link

Okay, I've got a system (well, a VM) with a ZFS root now, so I'll see if I can repro this. Looking at the code though, I'm a little mystified as to how the root is getting snapshotted on your system. If it comes to it, would you be comfortable installing a replacement borgmatic ZFS hook with added instrumentation/logging?

Okay, I've got a system (well, a VM) with a ZFS root now, so I'll see if I can repro this. Looking at the code though, I'm a little mystified as to how the root is getting snapshotted on your system. If it comes to it, would you be comfortable installing a replacement borgmatic ZFS hook with added instrumentation/logging?

kinetic5 commented 2025-02-18 18:39:09 +00:00

Author

Copy link

Yes, I can do that if needed.

Yes, I can do that if needed.

👍 1

witten commented 2025-02-18 20:49:38 +00:00

Owner

Copy link

A quick update: Turns out, you won't need to install any additional instrumentation. I have a full repro of both problems here on my ZFS root VM! I added a little extra logging, and it looks like the root dataset is getting included in snapshotting because of a metadata directory that borgmatic sneaks into the archive to support the borgmatic config bootstrap action. Here's the added log:

local: Directory / contains patterns: /tmp/borgmatic-y08mzwgl/./borgmatic/bootstrap

So I'll probably need to come up with a way to exclude such non-user-specified directories from the snapshotting.

A quick update: Turns out, you won't need to install any additional instrumentation. I have a full repro of both problems here on my ZFS root VM! I added a little extra logging, and it looks like the root dataset is getting included in snapshotting because of a metadata directory that borgmatic sneaks into the archive to support the `borgmatic config bootstrap` action. Here's the added log: ``` local: Directory / contains patterns: /tmp/borgmatic-y08mzwgl/./borgmatic/bootstrap ``` So I'll probably need to come up with a way to exclude such non-user-specified directories from the snapshotting.

👍 1

witten referenced this issue from a commit 2025-02-20 06:49:51 +00:00

Initial work on fixing ZFS mount errors (#1001).

witten referenced this issue from a commit 2025-02-20 21:25:46 +00:00

Add NEWS items for filesystem hook fixes/changes (#1001).

witten referenced this issue 2025-02-20 21:36:36 +00:00

Fix ZFS mount errors #1002

witten referenced this issue from a commit 2025-02-20 22:05:34 +00:00

Document ZFS snapshotting exclusion of "canmount=off" datasets (#1001).

witten referenced this issue from a commit 2025-02-21 06:58:28 +00:00

Fix ZFS and Btrfs tests (#1001).

witten referenced this issue from a commit 2025-02-21 19:34:11 +00:00

Get all unit/integration tests passing (#1001).

witten referenced this issue from a commit 2025-02-22 00:32:33 +00:00

Fix Btrfs end-to-end-test (#1001).

witten referenced this issue from a commit 2025-02-22 00:36:30 +00:00

Remove debugging prints (#1001).

witten referenced this issue from a commit 2025-02-22 02:00:20 +00:00

Don't try to unmount empty directories (#1001).

witten referenced this issue from a commit 2025-02-22 02:08:56 +00:00

Document the root pattern requirement for snapshotting (#1001).

witten referenced this issue from a commit 2025-02-22 04:13:37 +00:00

Fix ZFS mount errors (#1001).

witten added the

bug

label 2025-02-22 04:18:07 +00:00

witten commented 2025-02-22 04:18:20 +00:00

Owner

Copy link

Okay, this has been fixed in main and will be part of the next release. Here's the changelog:

• For the ZFS, Btrfs, and LVM hooks, only make snapshots for root patterns that come from a borgmatic configuration option (e.g. "source_directories")—not from other hooks within borgmatic.
• Fix a ZFS/LVM error due to colliding snapshot mount points for nested datasets or logical volumes.
• Don't try to snapshot ZFS datasets that have the "canmount=off" property.

Thanks again for the bug report!

Okay, this has been fixed in main and will be part of the next release. Here's the changelog: * For the ZFS, Btrfs, and LVM hooks, only make snapshots for root patterns that come from a borgmatic configuration option (e.g. "source_directories")—not from other hooks within borgmatic. * Fix a ZFS/LVM error due to colliding snapshot mount points for nested datasets or logical volumes. * Don't try to snapshot ZFS datasets that have the "canmount=off" property. Thanks again for the bug report!

witten closed this issue 2025-02-22 06:37:41 +00:00

witten commented 2025-02-22 22:43:02 +00:00

Owner

Copy link

Released in borgmatic 1.9.11!

Released in borgmatic 1.9.11!

kinetic5 commented 2025-02-23 19:26:11 +00:00

Author

Copy link

Thank you for working out this issue so quickly! After testing the new update, it looks like everything is working with the zfs snapshot and borg archive creation. However, I am seeing an error during the cleanup phase of the process:

It looks like zfs.py is trying to remove the temporary snapshot directory twice (lines 395 and 414). I think the issue stems from my root path '/' showing up when get_all_dataset_mount_points is run. This causes it to be picked up by snapshot_mount_path on line 383 and removed on line 395 in addition to the removal of snapshots_directory on line 414.

I added some additional debug messages prefixed by '***' to find this:

local: Calling bootstrap hook function remove_data_source_dumps
local: Looking for bootstrap manifest files to remove in /run/user/0/borgmatic/bootstrap
local: Removing bootstrap manifest at /run/user/0/borgmatic/bootstrap/manifest.json
local: Calling btrfs hook function remove_data_source_dumps
local: Calling lvm hook function remove_data_source_dumps
local: Calling mariadb hook function remove_data_source_dumps
local: Removing MariaDB data source dumps
local: Calling mongodb hook function remove_data_source_dumps
local: Removing MongoDB data source dumps
local: Calling mysql hook function remove_data_source_dumps
local: Removing MySQL data source dumps
local: Calling postgresql hook function remove_data_source_dumps
local: Removing PostgreSQL data source dumps
local: Calling sqlite hook function remove_data_source_dumps
local: Removing SQLite data source dumps
local: Calling zfs hook function remove_data_source_dumps
local: zfs list -H -t filesystem -o mountpoint
local: Looking for snapshots to remove in /run/user/0/borgmatic/zfs_snapshots/*
local: *** mount_point -> snapshot_mount_path (zfs.py, line 383): /var/lib/vz -> /run/user/0/borgmatic/zfs_snapshots/fe64c169b096a4ed742d/var/lib/vz
local: *** mount_point -> snapshot_mount_path (zfs.py, line 383): /rpool/data -> /run/user/0/borgmatic/zfs_snapshots/fe64c169b096a4ed742d/rpool/data
local: *** mount_point -> snapshot_mount_path (zfs.py, line 383): /rpool/ROOT -> /run/user/0/borgmatic/zfs_snapshots/fe64c169b096a4ed742d/rpool/ROOT
local: *** mount_point -> snapshot_mount_path (zfs.py, line 383): /rpool -> /run/user/0/borgmatic/zfs_snapshots/fe64c169b096a4ed742d/rpool
local: *** mount_point -> snapshot_mount_path (zfs.py, line 383): /dpool/shares/staticfiles -> /run/user/0/borgmatic/zfs_snapshots/fe64c169b096a4ed742d/dpool/shares/staticfiles
local: *** shutil.rmtree (zfs.py, line 395): /run/user/0/borgmatic/zfs_snapshots/fe64c169b096a4ed742d/dpool/shares/staticfiles
local: Unmounting ZFS snapshot at /run/user/0/borgmatic/zfs_snapshots/fe64c169b096a4ed742d/dpool/shares/staticfiles
local: umount /run/user/0/borgmatic/zfs_snapshots/fe64c169b096a4ed742d/dpool/shares/staticfiles
local: *** mount_point -> snapshot_mount_path (zfs.py, line 383): /dpool/shares -> /run/user/0/borgmatic/zfs_snapshots/fe64c169b096a4ed742d/dpool/shares
local: *** shutil.rmtree (zfs.py, line 395): /run/user/0/borgmatic/zfs_snapshots/fe64c169b096a4ed742d/dpool/shares
local: *** mount_point -> snapshot_mount_path (zfs.py, line 383): /dpool -> /run/user/0/borgmatic/zfs_snapshots/fe64c169b096a4ed742d/dpool
local: *** mount_point -> snapshot_mount_path (zfs.py, line 383): / -> /run/user/0/borgmatic/zfs_snapshots/fe64c169b096a4ed742d/
local: *** shutil.rmtree (zfs.py, line 395): /run/user/0/borgmatic/zfs_snapshots/fe64c169b096a4ed742d/
local: *** shutil.rmtree (zfs.py, line 414): /run/user/0/borgmatic/zfs_snapshots/fe64c169b096a4ed742d

Thank you for working out this issue so quickly! After testing the new update, it looks like everything is working with the zfs snapshot and borg archive creation. However, I am seeing an error during the cleanup phase of the process:  It looks like zfs.py is trying to remove the temporary snapshot directory twice (lines 395 and 414). I think the issue stems from my root path '/' showing up when `get_all_dataset_mount_points` is run. This causes it to be picked up by `snapshot_mount_path` on line 383 and removed on line 395 in addition to the removal of `snapshots_directory` on line 414. I added some additional debug messages prefixed by '***' to find this: ``` local: Calling bootstrap hook function remove_data_source_dumps local: Looking for bootstrap manifest files to remove in /run/user/0/borgmatic/bootstrap local: Removing bootstrap manifest at /run/user/0/borgmatic/bootstrap/manifest.json local: Calling btrfs hook function remove_data_source_dumps local: Calling lvm hook function remove_data_source_dumps local: Calling mariadb hook function remove_data_source_dumps local: Removing MariaDB data source dumps local: Calling mongodb hook function remove_data_source_dumps local: Removing MongoDB data source dumps local: Calling mysql hook function remove_data_source_dumps local: Removing MySQL data source dumps local: Calling postgresql hook function remove_data_source_dumps local: Removing PostgreSQL data source dumps local: Calling sqlite hook function remove_data_source_dumps local: Removing SQLite data source dumps local: Calling zfs hook function remove_data_source_dumps local: zfs list -H -t filesystem -o mountpoint local: Looking for snapshots to remove in /run/user/0/borgmatic/zfs_snapshots/* local: *** mount_point -> snapshot_mount_path (zfs.py, line 383): /var/lib/vz -> /run/user/0/borgmatic/zfs_snapshots/fe64c169b096a4ed742d/var/lib/vz local: *** mount_point -> snapshot_mount_path (zfs.py, line 383): /rpool/data -> /run/user/0/borgmatic/zfs_snapshots/fe64c169b096a4ed742d/rpool/data local: *** mount_point -> snapshot_mount_path (zfs.py, line 383): /rpool/ROOT -> /run/user/0/borgmatic/zfs_snapshots/fe64c169b096a4ed742d/rpool/ROOT local: *** mount_point -> snapshot_mount_path (zfs.py, line 383): /rpool -> /run/user/0/borgmatic/zfs_snapshots/fe64c169b096a4ed742d/rpool local: *** mount_point -> snapshot_mount_path (zfs.py, line 383): /dpool/shares/staticfiles -> /run/user/0/borgmatic/zfs_snapshots/fe64c169b096a4ed742d/dpool/shares/staticfiles local: *** shutil.rmtree (zfs.py, line 395): /run/user/0/borgmatic/zfs_snapshots/fe64c169b096a4ed742d/dpool/shares/staticfiles local: Unmounting ZFS snapshot at /run/user/0/borgmatic/zfs_snapshots/fe64c169b096a4ed742d/dpool/shares/staticfiles local: umount /run/user/0/borgmatic/zfs_snapshots/fe64c169b096a4ed742d/dpool/shares/staticfiles local: *** mount_point -> snapshot_mount_path (zfs.py, line 383): /dpool/shares -> /run/user/0/borgmatic/zfs_snapshots/fe64c169b096a4ed742d/dpool/shares local: *** shutil.rmtree (zfs.py, line 395): /run/user/0/borgmatic/zfs_snapshots/fe64c169b096a4ed742d/dpool/shares local: *** mount_point -> snapshot_mount_path (zfs.py, line 383): /dpool -> /run/user/0/borgmatic/zfs_snapshots/fe64c169b096a4ed742d/dpool local: *** mount_point -> snapshot_mount_path (zfs.py, line 383): / -> /run/user/0/borgmatic/zfs_snapshots/fe64c169b096a4ed742d/ local: *** shutil.rmtree (zfs.py, line 395): /run/user/0/borgmatic/zfs_snapshots/fe64c169b096a4ed742d/ local: *** shutil.rmtree (zfs.py, line 414): /run/user/0/borgmatic/zfs_snapshots/fe64c169b096a4ed742d ```

image.png

68 KiB

kinetic5 reopened this issue 2025-02-23 19:26:11 +00:00

witten commented 2025-02-23 23:46:14 +00:00

Owner

Copy link

Thanks for reopening this and providing the additional details. Unfortunately I don't have a repro here even with my ZFS VM, but here's something to try since you're already in the code. If you add , ignore_errors=True to the second rmtree() call so it looks like the first one, does that "fix" the problem? If so, I can make the change to main.

Thanks for reopening this and providing the additional details. Unfortunately I don't have a repro here even with my ZFS VM, but here's something to try since you're already in the code. If you add `, ignore_errors=True` to the second `rmtree()` call so it looks like the first one, does that "fix" the problem? If so, I can make the change to main.

kinetic5 commented 2025-02-24 00:38:11 +00:00

Author

Copy link

Yes, adding , ignore_errors=True to the second rmtree fixes the issue for me.

Yes, adding `, ignore_errors=True` to the second `rmtree` fixes the issue for me.

witten referenced this issue from a commit 2025-02-24 01:01:50 +00:00

Fix a ZFS error during snapshot cleanup (#1001).

witten commented 2025-02-24 01:02:05 +00:00

Owner

Copy link

Okay, thanks for verifying! I've made the change to main and it'll be part of the next release.

Okay, thanks for verifying! I've made the change to main and it'll be part of the next release.

witten closed this issue 2025-02-24 01:02:06 +00:00

witten commented 2025-03-03 18:23:08 +00:00

Owner

Copy link

Released in borgmatic 1.9.13!

Released in borgmatic 1.9.13!

witten referenced this issue 2025-03-20 18:26:16 +00:00

ZFS snapshot integration causing cache miss: All files reported as 'Added' in each backup #1035

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

parallel-borgmatic

command-hook-steps

2.1.7

2.1.6

2.1.5

2.1.4

2.1.3

2.1.2

2.1.1

2.1.0

2.0.13

2.0.12

2.0.11

2.0.10

2.0.9

2.0.8

2.0.7

2.0.6

2.0.5

2.0.4

2.0.3

2.0.2

2.0.1

2.0.0

1.9.14

1.9.13

1.9.12

1.9.11

1.9.10

1.9.9

1.9.8

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.14

1.8.13

1.8.12

1.8.11

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.24

1.5.23

1.5.22

1.5.21

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.14

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.7.dev0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.22

1.4.21

1.4.20

1.4.19

1.4.18

1.4.17

1.4.16

1.4.15

1.4.14

1.4.13

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.26

1.3.25

1.3.24

1.3.23

1.3.22

1.3.21

1.3.20

1.3.19

1.3.18

1.3.17

1.3.16

1.3.15

1.3.14

1.3.13

1.3.12

1.3.11

1.3.10

1.3.9

1.3.8

1.3.7

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.18

1.2.17

1.2.16

1.2.15

1.2.14

1.2.13

1.2.12

1.2.11

1.2.10

1.2.9

1.2.8

1.2.7

1.2.6

1.2.5

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.15

1.1.14

1.1.13

1.1.12

1.1.11

1.1.10

1.1.9

1.1.7

1.1.6

1.1.5

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.3

1.0.2

1.0.1

1.0.0

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

0.0.2

Labels

Clear labels   

blocked

  

breaking

  

bug

  

data loss

  

design finalized

  

good first issue

  

new feature area

  

question / support

  

security

  

waiting for response

No labels

blocked

breaking

bug

data loss

design finalized

good first issue

new feature area

question / support

security

waiting for response

Milestone

Clear milestone

No items

No milestone

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

borgmatic-collective/borgmatic#1001

No description provided.