ZFS snapshot integration causing cache miss: All files reported as 'Added' in each backup #1035

New issue

Closed

opened 2025-03-20 11:03:21 +00:00 by Flameish · 5 comments

Flameish commented 2025-03-20 11:03:21 +00:00

Copy link

What I'm trying to do and why

When using borgmatic with the zfs: configuration option to create snapshots, it fails to properly utilize the cache. All files are marked as "Added" during each backup, even if they already exist in the repository. When the zfs: option is removed, the cache works correctly and only new/changed files are added.

• The backup source is a folder on a ZFS dataset
• With zfs: in config: All files show as "Added" in each backup run
• Without zfs: in config: Only new files show as "Added" (expected behavior)

I've attached two log files from back-to-back runs on the same test repository that demonstrate the difference in behavior. (--list --verbosity 2)

Steps to reproduce

No response

Actual behavior

No response

Expected behavior

No response

Other notes / implementation ideas

No response

borgmatic version

1.9.14

borgmatic installation method

pip

Borg version

borg 1.4.0

Python version

Python 3.11.9

Database version (if applicable)

No response

Operating system and version

TrueNAS Scale 25.04-RC.1 / Debian GNU/Linux 12 (bookworm)

### What I'm trying to do and why When using borgmatic with the zfs: configuration option to create snapshots, it fails to properly utilize the cache. All files are marked as "Added" during each backup, even if they already exist in the repository. When the zfs: option is removed, the cache works correctly and only new/changed files are added. - The backup source is a folder on a ZFS dataset - With zfs: in config: All files show as "Added" in each backup run - Without zfs: in config: Only new files show as "Added" (expected behavior) I've attached two log files from back-to-back runs on the same test repository that demonstrate the difference in behavior. (--list --verbosity 2) ### Steps to reproduce _No response_ ### Actual behavior _No response_ ### Expected behavior _No response_ ### Other notes / implementation ideas _No response_ ### borgmatic version 1.9.14 ### borgmatic installation method pip ### Borg version borg 1.4.0 ### Python version Python 3.11.9 ### Database version (if applicable) _No response_ ### Operating system and version TrueNAS Scale 25.04-RC.1 / Debian GNU/Linux 12 (bookworm)

2.log

11 KiB

1.log

11 KiB

Flameish changed title from ZFS snapshot feature doesn't recognize existing backup files to ZFS snapshot integration causing cache miss: All files reported as 'Added' in each backup 2025-03-20 11:17:05 +00:00

witten commented 2025-03-20 16:41:24 +00:00

Owner

Copy link

Thanks for the detailed ticket. Both logs you've posted appear to have all As in the file listing, so are both from when zfs: was enabled? Could I see a log from a run when it's not enabled?

In any case, my guess is that you're seeing this behavior because the temporary path for the ZFS snapshot varies from run to run because of that randomly generated component after /tmp. (Even though this does appear to be a file cache miss, Borg should still be deduplicating at the block level.) If you'd like to try removing the random component from the snapshot path, try setting borgmatic's user_runtime_directory option and then running borgmatic twice with zfs: enabled. On the second run, I'd expect you to see some Ms and not just As, indicating files cache hits.

Thanks for the detailed ticket. Both logs you've posted appear to have all `A`s in the file listing, so are both from when `zfs:` was enabled? Could I see a log from a run when it's not enabled? In any case, my guess is that you're seeing this behavior because the temporary path for the ZFS snapshot varies from run to run because of that randomly generated component after `/tmp`. (Even though this does appear to be a file cache miss, Borg should still be deduplicating at the block level.) If you'd like to try removing the random component from the snapshot path, try setting borgmatic's `user_runtime_directory` option and then running borgmatic *twice* with `zfs:` enabled. On the second run, I'd expect you to see some `M`s and not just `A`s, indicating files cache hits.

witten commented 2025-03-20 16:49:00 +00:00

Owner

Copy link

Hmm, I might actually be wrong about that. The Borg docs seem to indicate that the cache keys off of file timestamp, size, and and inode—not the file path. So maybe one of those values is changing across snapshots in such a way to trigger a Borg cache miss?

EDIT: Nope, I might've been right to begin with.. Apparently I needed to keep reading. From the linked docs:

Another possible reason is that files don’t always have the same path, for example if you mount a filesystem without stable mount points for each backup or if you are running the backup from a filesystem snapshot whose name is not stable. If the directory where you mount a filesystem is different every time, Borg assumes they are different files. This is true even if you backup these files with relative pathnames - borg uses full pathnames in files cache regardless.

Hmm, I might actually be wrong about that. [The Borg docs](https://borgbackup.readthedocs.io/en/stable/faq.html#it-always-chunks-all-my-files-even-unchanged-ones) seem to indicate that the cache keys off of file timestamp, size, and and inode—not the file path. So maybe one of those values is changing across snapshots in such a way to trigger a Borg cache miss? EDIT: Nope, I might've been right to begin with.. Apparently I needed to keep reading. From the linked docs: > Another possible reason is that files don’t always have the same path, for example if you mount a filesystem without stable mount points for each backup or if you are running the backup from a filesystem snapshot whose name is not stable. If the directory where you mount a filesystem is different every time, Borg assumes they are different files. This is true even if you backup these files with relative pathnames - borg uses full pathnames in files cache regardless.

Flameish commented 2025-03-20 17:32:26 +00:00

Author

Copy link

Thank you for your swift response.

so are both from when zfs: was enabled?

Yes, both previous logs are with zfs: enabled. I've attached a log (of the second run since the first adds all files) with zfs: disabled.

Borg should still be deduplicating at the block level

It does indeed deduplicate it but my backup run time went from 2 minutes to 2 hours since it's re-adding all files.

So maybe one of those values is changing across snapshots

No, the stats command says timestamp, size, and inode are the same in the original and snapshot.

my guess is that you're seeing this behavior because the temporary path for the ZFS snapshot varies from run to run

This was my guess as well. Two test runs with your suggested user_runtime_directory configuration supports this. But this path still seems to have a random id in it, or will this always be the same? Running it multiple times did not seem to change this through.

A /mnt/pool1/backup/tmp/borgmatic/zfs_snapshots/1124e7414b499df4c9c5/mnt/pool1/backup/tmp/borgmatic/bootstrap/manifest.json`

Thank you for your swift response. > so are both from when zfs: was enabled? Yes, both previous logs are with zfs: enabled. I've attached a log (of the second run since the first adds all files) with zfs: **disabled**. > Borg should still be deduplicating at the block level It does indeed deduplicate it but my backup run time went from 2 minutes to 2 hours since it's re-adding all files. > So maybe one of those values is changing across snapshots No, the `stats` command says timestamp, size, and inode are the same in the original and snapshot. > my guess is that you're seeing this behavior because the temporary path for the ZFS snapshot varies from run to run This was my guess as well. Two test runs with your suggested `user_runtime_directory` configuration supports this. But this path still seems to have a random id in it, or will this always be the same? Running it multiple times did not seem to change this through. A /mnt/pool1/backup/tmp/borgmatic/zfs_snapshots/**1124e7414b499df4c9c5**/mnt/pool1/backup/tmp/borgmatic/bootstrap/manifest.json`

without_zfs.log

4.9 KiB

with_runtime_dir.log

6.1 KiB

witten commented 2025-03-20 18:26:16 +00:00

Owner

Copy link

Yeah, that string you bolded is a hash made from the dataset mount point path itself, so it should be constant for a given dataset and shouldn't change from run to run. It's there as a means of preventing multiple snapshots of overlapping paths (e.g. nested datasets) from colliding and causing mount/unmount problems. See #1001 for more about why this was necessary.

So if setting user_runtime_directory "fixes" the file cache issue, I should probably address this ticket by documenting the "fix." The reason that the runtime directory normally has a temporary path component is for security reasons—so another process can't interfere with the path that borgmatic is using for temporary runtime files like snapshots. But if a user sets their own runtime path, then it's assumed they know what they're doing and can secure the directory properly (unlike a shared temporary directory like /tmp).

EDIT: In Borg 1.x, it looks like Borg uses the full absolute path of the source file for the files cache key. But in Borg 2.x, Borg uses the "path as seen in the archive" instead.

Yeah, that string you bolded is a hash made from the dataset mount point path itself, so it should be constant for a given dataset and shouldn't change from run to run. It's there as a means of preventing multiple snapshots of overlapping paths (e.g. nested datasets) from colliding and causing mount/unmount problems. See #1001 for more about why this was necessary. So if [setting `user_runtime_directory`]([url](https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#runtime-directory)) "fixes" the file cache issue, I should probably address this ticket by documenting the "fix." The reason that the runtime directory normally has a temporary path component is for security reasons—so another process can't interfere with the path that borgmatic is using for temporary runtime files like snapshots. But if a user sets their own runtime path, then it's assumed they know what they're doing and can secure the directory properly (unlike a shared temporary directory like `/tmp`). EDIT: In Borg 1.x, it looks like Borg uses the full absolute path of the source file for the files cache key. But in Borg 2.x, Borg uses the "path as seen in the archive" instead.

witten referenced this issue from a commit 2025-04-07 20:51:36 +00:00

Document potential performance issues and workarounds with the ZFS, Btrfs, and LVM hooks (#1035).

witten commented 2025-04-07 20:53:53 +00:00

Owner

Copy link

Okay, I went ahead and documented this potential issue for each of the data store snapshot hooks, including ZFS. The updated docs should be live shortly: https://torsion.org/borgmatic/docs/how-to/snapshot-your-filesystems/

Interestingly, it sounds like this whole thing is a non-issue with Borg 2.x, as it's smarter about how it creates files cache keys.

Thanks again for the ticket!

Okay, I went ahead and documented this potential issue for each of the data store snapshot hooks, including ZFS. The updated docs should be live shortly: https://torsion.org/borgmatic/docs/how-to/snapshot-your-filesystems/ Interestingly, it sounds like this whole thing is a non-issue with Borg 2.x, as it's smarter about how it creates files cache keys. Thanks again for the ticket!

witten closed this issue 2025-04-07 20:53:53 +00:00

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

parallel-borgmatic

command-hook-steps

2.1.7

2.1.6

2.1.5

2.1.4

2.1.3

2.1.2

2.1.1

2.1.0

2.0.13

2.0.12

2.0.11

2.0.10

2.0.9

2.0.8

2.0.7

2.0.6

2.0.5

2.0.4

2.0.3

2.0.2

2.0.1

2.0.0

1.9.14

1.9.13

1.9.12

1.9.11

1.9.10

1.9.9

1.9.8

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.14

1.8.13

1.8.12

1.8.11

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.24

1.5.23

1.5.22

1.5.21

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.14

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.7.dev0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.22

1.4.21

1.4.20

1.4.19

1.4.18

1.4.17

1.4.16

1.4.15

1.4.14

1.4.13

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.26

1.3.25

1.3.24

1.3.23

1.3.22

1.3.21

1.3.20

1.3.19

1.3.18

1.3.17

1.3.16

1.3.15

1.3.14

1.3.13

1.3.12

1.3.11

1.3.10

1.3.9

1.3.8

1.3.7

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.18

1.2.17

1.2.16

1.2.15

1.2.14

1.2.13

1.2.12

1.2.11

1.2.10

1.2.9

1.2.8

1.2.7

1.2.6

1.2.5

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.15

1.1.14

1.1.13

1.1.12

1.1.11

1.1.10

1.1.9

1.1.7

1.1.6

1.1.5

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.3

1.0.2

1.0.1

1.0.0

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

0.0.2

Labels

Clear labels   

blocked

  

breaking

  

bug

  

data loss

  

design finalized

  

good first issue

  

new feature area

  

question / support

  

security

  

waiting for response

No labels

blocked

breaking

bug

data loss

design finalized

good first issue

new feature area

question / support

security

waiting for response

Milestone

Clear milestone

No items

No milestone

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

borgmatic-collective/borgmatic#1035

No description provided.