borgmatic-collective/borgmatic
borgmatic-collective/borgmatic
Watch
9
74
Fork
You've already forked borgmatic
 57
Code Issues 71 Pull requests 3 Releases 162 Packages 1 Activity Actions 1

spot check fails when backup contains symlinks #1050

New issue
Closed
opened 2025-04-01 01:50:34 +00:00 by itu7 · 3 comments
itu7 commented 2025-04-01 01:50:34 +00:00
Copy link

What I'm trying to do and why

I am trying to run a spot check on a recently written archive.
The spot check always fails if the backup contains symlinks.

This is part of my config:

checks:
- name: spot
  count_tolerance_percentage: 10
  data_sample_percentage: 1
  data_tolerance_percentage: 0

source_directories:
    - /etc

Steps to reproduce

  1. prepare a backup which contains symlinks
  2. run a spot check: borgmatic check -v2 --only spot
  3. try to compute remote hash of a (non matching) symlink:
# borg list --format '{xxh64} {path}{NL}' ssh://pathtorepo::archivename /etc/rc4.d/S01bluetooth
 etc/rc4.d/S01bluetooth

No hash computed - remote borg is unable to compute the hash of a symlink

Actual behavior

Example log (edited):

server: 2 non-matching spot check hashes
server: Source paths with data not matching the latest archive: /etc/rc4.d/S01bluetooth, /etc/rc6.d/K01plymouth

Inspect some files:

# file /etc/bash.bashrc /etc/rc4.d/S01bluetooth /etc/rc6.d/K01plymouth
/etc/bash.bashrc:        ASCII text
/etc/rc4.d/S01bluetooth: symbolic link to ../init.d/bluetooth
/etc/rc6.d/K01plymouth:  symbolic link to ../init.d/plymouth

# xxh64sum /etc/bash.bashrc /etc/rc4.d/S01bluetooth /etc/rc6.d/K01plymouth
1c66874322792f28  /etc/bash.bashrc
c8bbc4d25f80a462  /etc/rc4.d/S01bluetooth
404c258b857434f2  /etc/rc6.d/K01plymouth

# borg list --format '{xxh64} {path}{NL}' ssh://pathtorepo::archivename /etc/bash.bashrc /etc/rc4.d/S01bluetooth /etc/rc6.d/K01plymouth
1c66874322792f28 etc/bash.bashrc
 etc/rc4.d/S01bluetooth
 etc/rc6.d/K01plymouth

Hash is computed for non-symlink only.

Expected behavior

Spot check should not fail if borg is unable to compute the hash.

Other notes / implementation ideas

Spot check sampling routine should skip symlinks if borg is unable to compute the hash.

borgmatic version

1.9.14

borgmatic installation method

Debian package

Borg version

1.4.0 (local)/1.2.4 (remote)

Python version

2.13.2

Database version (if applicable)

No response

Operating system and version

PRETTY_NAME="Debian GNU/Linux trixie/sid" NAME="Debian GNU/Linux" VERSION_CODENAME=trixie ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/"

### What I'm trying to do and why I am trying to run a spot check on a recently written archive. The spot check always fails if the backup contains symlinks. This is part of my config: ``` checks: - name: spot count_tolerance_percentage: 10 data_sample_percentage: 1 data_tolerance_percentage: 0 source_directories: - /etc ``` ### Steps to reproduce 1. prepare a backup which contains symlinks 2. run a spot check: `borgmatic check -v2 --only spot` 3. try to compute remote hash of a (non matching) symlink: ``` # borg list --format '{xxh64} {path}{NL}' ssh://pathtorepo::archivename /etc/rc4.d/S01bluetooth etc/rc4.d/S01bluetooth ``` No hash computed - remote borg is unable to compute the hash of a symlink ### Actual behavior Example log (edited): ``` server: 2 non-matching spot check hashes server: Source paths with data not matching the latest archive: /etc/rc4.d/S01bluetooth, /etc/rc6.d/K01plymouth ``` Inspect some files: ``` # file /etc/bash.bashrc /etc/rc4.d/S01bluetooth /etc/rc6.d/K01plymouth /etc/bash.bashrc: ASCII text /etc/rc4.d/S01bluetooth: symbolic link to ../init.d/bluetooth /etc/rc6.d/K01plymouth: symbolic link to ../init.d/plymouth ``` ``` # xxh64sum /etc/bash.bashrc /etc/rc4.d/S01bluetooth /etc/rc6.d/K01plymouth 1c66874322792f28 /etc/bash.bashrc c8bbc4d25f80a462 /etc/rc4.d/S01bluetooth 404c258b857434f2 /etc/rc6.d/K01plymouth ``` ``` # borg list --format '{xxh64} {path}{NL}' ssh://pathtorepo::archivename /etc/bash.bashrc /etc/rc4.d/S01bluetooth /etc/rc6.d/K01plymouth 1c66874322792f28 etc/bash.bashrc etc/rc4.d/S01bluetooth etc/rc6.d/K01plymouth ``` Hash is computed for non-symlink only. ### Expected behavior Spot check should not fail if `borg` is unable to compute the hash. ### Other notes / implementation ideas Spot check sampling routine should skip symlinks if `borg` is unable to compute the hash. ### borgmatic version 1.9.14 ### borgmatic installation method Debian package ### Borg version 1.4.0 (local)/1.2.4 (remote) ### Python version 2.13.2 ### Database version (if applicable) _No response_ ### Operating system and version PRETTY_NAME="Debian GNU/Linux trixie/sid" NAME="Debian GNU/Linux" VERSION_CODENAME=trixie ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/"
witten commented 2025-04-01 04:24:28 +00:00
Owner
Copy link

Thanks for filing this and providing all the diagnostics! I will look into this when I get the chance.

Thanks for filing this and providing all the diagnostics! I will look into this when I get the chance.
witten commented 2025-04-01 18:49:29 +00:00
Owner
Copy link

This is fixed in main and will be part of the next release! I went with a slightly different approach than your suggestion though. The issue with skipping symlinks entirely in the spot check is it doesn't catch a few legitimate failing spot check cases:

  • A symlink present in the source directories that's missing from the archive (or present in the archive but is a plain file there).
  • A symlink present in the archive that's missing from the source directories (or present in the source directories but is a plain file there).

So instead, the fix treats symlinks in the source directories as having empty hashes, thereby matching the empty hashes Borg provides for archive symlinks and addressing the above use cases. More importantly, symlinks within an archive no longer break the spot check.

Thanks again!

This is fixed in main and will be part of the next release! I went with a slightly different approach than your suggestion though. The issue with skipping symlinks entirely in the spot check is it doesn't catch a few legitimate failing spot check cases: - A symlink present in the source directories that's missing from the archive (or present in the archive but is a plain file there). - A symlink present in the archive that's missing from the source directories (or present in the source directories but is a plain file there). So instead, the fix treats symlinks in the source directories as having empty hashes, thereby matching the empty hashes Borg provides for archive symlinks and addressing the above use cases. More importantly, symlinks within an archive no longer break the spot check. Thanks again!
witten commented 2025-04-06 15:42:50 +00:00
Owner
Copy link

Released in borgmatic 2.0.0!

Released in borgmatic 2.0.0!
🎉 1
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
parallel-borgmatic
command-hook-steps
2.1.7
2.1.6
2.1.5
2.1.4
2.1.3
2.1.2
2.1.1
2.1.0
2.0.13
2.0.12
2.0.11
2.0.10
2.0.9
2.0.8
2.0.7
2.0.6
2.0.5
2.0.4
2.0.3
2.0.2
2.0.1
2.0.0
1.9.14
1.9.13
1.9.12
1.9.11
1.9.10
1.9.9
1.9.8
1.9.7
1.9.6
1.9.5
1.9.4
1.9.3
1.9.2
1.9.1
1.9.0
1.8.14
1.8.13
1.8.12
1.8.11
1.8.10
1.8.9
1.8.8
1.8.7
1.8.6
1.8.5
1.8.4
1.8.3
1.8.2
1.8.1
1.8.0
1.7.15
1.7.14
1.7.13
1.7.12
1.7.11
1.7.10
1.7.9
1.7.8
1.7.7
1.7.6
1.7.5
1.7.4
1.7.3
1.7.2
1.7.1
1.7.0
1.6.6
1.6.5
1.6.4
1.6.3
1.6.2
1.6.1
1.6.0
1.5.24
1.5.23
1.5.22
1.5.21
1.5.20
1.5.19
1.5.18
1.5.17
1.5.16
1.5.15
1.5.14
1.5.13
1.5.12
1.5.11
1.5.10
1.5.9
1.5.8
1.5.7
1.5.7.dev0
1.5.6
1.5.5
1.5.4
1.5.3
1.5.2
1.5.1
1.5.0
1.4.22
1.4.21
1.4.20
1.4.19
1.4.18
1.4.17
1.4.16
1.4.15
1.4.14
1.4.13
1.4.12
1.4.11
1.4.10
1.4.9
1.4.8
1.4.7
1.4.6
1.4.5
1.4.4
1.4.3
1.4.2
1.4.1
1.4.0
1.3.26
1.3.25
1.3.24
1.3.23
1.3.22
1.3.21
1.3.20
1.3.19
1.3.18
1.3.17
1.3.16
1.3.15
1.3.14
1.3.13
1.3.12
1.3.11
1.3.10
1.3.9
1.3.8
1.3.7
1.3.6
1.3.5
1.3.4
1.3.3
1.3.2
1.3.1
1.3.0
1.2.18
1.2.17
1.2.16
1.2.15
1.2.14
1.2.13
1.2.12
1.2.11
1.2.10
1.2.9
1.2.8
1.2.7
1.2.6
1.2.5
1.2.4
1.2.3
1.2.2
1.2.1
1.2.0
1.1.15
1.1.14
1.1.13
1.1.12
1.1.11
1.1.10
1.1.9
1.1.7
1.1.6
1.1.5
1.1.4
1.1.3
1.1.2
1.1.1
1.1.0
1.0.3
1.0.2
1.0.1
1.0.0
0.1.8
0.1.7
0.1.6
0.1.5
0.1.4
0.1.3
0.1.2
0.1.1
0.1.0
0.0.7
0.0.6
0.0.5
0.0.4
0.0.3
0.0.2
Labels
Clear labels   
blocked

   
breaking

   
bug

   
data loss

   
design finalized

   
good first issue

   
new feature area

   
question / support

   
security

   
waiting for response
No labels
blocked
breaking
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
borgmatic-collective/borgmatic#1050
No description provided.