Spot Check Issue - Spot check failed: There are no source paths to compare against the archive #1086
Labels
No labels
blocked
breaking
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
No milestone
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
borgmatic-collective/borgmatic#1086
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What I'm trying to do and why
Please let me know how I can help troubleshoot. Over the last couple weeks i've started getting errors when a spot check is ran about "no source paths". it's only when spot check is enabled; no other errors when commenting out just that one section in my config. I've been using the same config for months, so believe its tied to a recent system upgrade). I just tried downgrading to borgmatic 1.9.14 and everything appears to be working correctly. After upgrading back to the newest (2.0.4) I get the same errors.
I do have "source_directories_must_exist: true" but i tried commenting out this to test, and got the same error/result. I also tried commenting out my "patterns_from" file, with no luck.
The one thing i'm noticing after downgrading and then ugprading back to the newest (using the same config) is when i was on 1.9.14, the list of files pulled up for the spot check are only a handful (as expected), but the output on the newest version is literally thousands and thousands of files. It doesn't appear to "respect" the "data_sample_percentage" to check but is attempting to check every file in the backup.
Furthermore as pointed out below, the first few files listed as "found in archive but not in source" have been confirmed as existing exactly where its expected on my system.
Steps to reproduce
source_directories:
- /home/jason/docker
- /home/Data/HomeAssistantBU
- /home/Data/PXAsyncthing
- /home/Media/VC
- /home/Media/Immich
- /home/Media/MusicBeets
- /home/Media/MusicHiDef
- /home/Data/PortainerBackup
repositories:
- path: ssh://#############.repo.borgbase.com/./repo
label: BorgBase
patterns_from:
- /etc/borgmatic/patterns
exclude_if_present:
- .nobackup
source_directories_must_exist: true
encryption_passphrase: "#################"
ssh_command: ssh -i /home/jason/.ssh/borgbase
archive_name_format: '{hostname}-{now:%Y-%m-%d-%H%M%S}'
match_archives: sh:-
keep_daily: 7
keep_weekly: 4
keep_monthly: 6
keep_yearly: 2
checks:
The following is my /etc/borgmatic/patterns file:
! fm:/.Trash-
! fm:/.stversions
! fm:/plex/transcode
! fm:/Plex Media Server/Logs
! fm:/Plex Media Server/Cache
! fm:/Plex Media Server/Media
! fm:/Plex Media Server/Metadata
! fm:/npm/data/logs
! fm:/jellyfin/config/metadata
! fm:/picard/config/xdg/cache
! fm:/scrutiny/influxdb
! fm:*/chromium/config
Actual behavior
❯ borgmatic check -v 2 --only spot --progress --repo BorgBase
/etc/borgmatic.d/borgbase.yaml: Calling ntfy hook function initialize_monitor
/etc/borgmatic.d/borgbase.yaml: Calling ntfy hook function ping_monitor
/etc/borgmatic.d/borgbase.yaml: BORG_RSH=*** BORG_PASSPHRASE_FD=*** BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_EXIT_CODES=*** borg --v
ersion --debug --show-rc
/etc/borgmatic.d/borgbase.yaml: Borg 1.4.1
BorgBase: Running actions for repository
BorgBase: Running consistency checks
BorgBase: BORG_RSH=*** BORG_PASSPHRASE_FD=*** BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_EXIT_CODES=*** borg info --json ssh://#######.repo.borgbase.com/./repo
BorgBase: Using runtime directory /run/user/1000/borgmatic
BorgBase: Running spot check
BorgBase: Calling bootstrap hook function use_streaming
BorgBase: Writing patterns to /run/user/1000/borgmatic/tmpepw_pskz:
R /home/jason/docker
R /home/Data/HomeAssistantBU
R /home/Data/PXAsyncthing
R /home/Media/VC
R /home/Media/Immich
R /home/Media/MusicBeets
R /home/Media/MusicHiDef
R /home/Data/PortainerBackup
! fm:/.Trash-
! fm:/.stversions
! fm:/plex/transcode
! fm:/Plex Media Server/Logs
! fm:/Plex Media Server/Cache
! fm:/Plex Media Server/Media
! fm:/Plex Media Server/Metadata
! fm:/npm/data/logs
! fm:/jellyfin/config/metadata
! fm:/picard/config/xdg/cache
! fm:/scrutiny/influxdb
! fm:/chromium/config
BorgBase: BORG_RSH=** BORG_PASSPHRASE_FD=*** BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_EXIT_CODES=*** borg create --patterns-from /r
un/user/1000/borgmatic/tmpepw_pskz --exclude-if-present .nobackup --dry-run ssh://#######.repo.borgbase.com/./repo::{hostname}-{now:%Y-%m-%d-%H%M%S}
BorgBase: 0 total source paths for spot check
BorgBase: BORG_RSH=*** BORG_PASSPHRASE_FD=*** BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_EXIT_CODES=*** borg list --last 1 --short ssh
://#######.repo.borgbase.com/./repo
BorgBase: Latest archive is ArchProx-2025-04-28-021254
BorgBase: Using archive ArchProx-2025-04-28-021254 for spot check
BorgBase: BORG_RSH=*** BORG_PASSPHRASE_FD=*** BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_EXIT_CODES=*** borg list --debug --show-rc --
format {type} {path}{NUL} ssh://#######.repo.borgbase.com/./repo::ArchProx-2025-04-28-021254
using builtin fallback logging configuration
33 self tests completed in 0.05 seconds
SSH command line: ['ssh', '-i', '/home/jason/.ssh/borgbase', 'l#######.repo.borgbase.com', 'borg', 'serve', '--debug']
Remote: using builtin fallback logging configuration
Remote: borg selftest disabled via BORG_SELFTEST env variable
Remote: using builtin fallback logging configuration
Remote: Initialized logging system for JSON-based protocol
Remote: Resolving repository path b'/./repo'
Remote: Resolved repository path to '/srv/repos/#######/repo'
Remote: Verified integrity of /srv/repos/#######/repo/index.4069
TAM-verified manifest
security: read previous location 'ssh://#######.repo.borgbase.com/./repo'
security: read manifest timestamp '2025-04-28T07:19:42.540304'
security: determined newest manifest timestamp as 2025-04-28T07:19:42.540304
security: repository checks ok, allowing access
Archive authentication DISABLED.
TAM-verified archive
RemoteRepository: 21.63 kB bytes sent, 33.66 MB bytes received, 416 messages sent
terminating with success status, rc 0
BorgBase: 167812 total archive paths for spot check
BorgBase: Paths in latest archive but not source paths: home/Media/MusicBeets/Jefferson Airplane/Surrealistic Pillow/10 White Rabbit.mp3
Long list of files (as mentioned above, with the newest version it's literally tens of thousands of "pages" of files listed, much more than expected). Furthermore, the first file mentioned "in latest archive but not source paths" is found in my system exactly where it should be.
BorgBase: Error running actions for repository
BorgBase: Spot check failed: There are no source paths to compare against the archive
/etc/borgmatic.d/borgbase.yaml: Calling ntfy hook function ping_monitor
/etc/borgmatic.d/borgbase.yaml: Calling ntfy hook function ping_monitor
/etc/borgmatic.d/borgbase.yaml: Pinging ntfy topic fnordborg
/etc/borgmatic.d/borgbase.yaml: Using Ntfy ping URL https://####
/etc/borgmatic.d/borgbase.yaml: Using basic auth with user jason for ntfy
/etc/borgmatic.d/borgbase.yaml: Calling ntfy hook function destroy_monitor
/etc/borgmatic.d/borgbase.yaml: Error running configuration
/etc/borgmatic.d/borgbase.yaml: An error occurred
/etc/borgmatic.d/borgdata.yaml: Calling ntfy hook function initialize_monitor
/etc/borgmatic.d/borgdata.yaml: Calling ntfy hook function ping_monitor
/etc/borgmatic.d/borgdata.yaml: BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_EXIT_CODES=*** borg --version --debug --show-rc
/etc/borgmatic.d/borgdata.yaml: Borg 1.4.1
BorgData: Running actions for repository
/etc/borgmatic.d/borgdata.yaml: Calling ntfy hook function ping_monitor
/etc/borgmatic.d/borgdata.yaml: Calling ntfy hook function ping_monitor
/etc/borgmatic.d/borgdata.yaml: Calling ntfy hook function destroy_monitor
summary:
/etc/borgmatic.d/borgbase.yaml: Loading configuration file
/etc/borgmatic.d/borgdata.yaml: Loading configuration file
An error occurred
Error running actions for repository
Spot check failed: There are no source paths to compare against the archive
Error running configuration
/etc/borgmatic.d/borgdata.yaml: Successfully ran configuration file
Need some help? https://torsion.org/borgmatic/#issues
Expected behavior
No response
Other notes / implementation ideas
No response
borgmatic version
2.0.4
borgmatic installation method
arch - pacman
Borg version
1.4.1
Python version
3.13.3
Database version (if applicable)
No response
Operating system and version
Arch Linux
Thanks for providing all of these details and logs. I'm not sure why you're seeing this behavior, but this it what I'd recommend trying: Run the Borg dry run command that borgmatic is running to determine the source paths. To do that, first write the combined source directories and patterns to a temporary file just like borgmatic does. Call it
patternsand put this into it:Then run the following command:
Replace
#######with your actual Borgbase repository hostname prefix. Upon running the command, there are a few possibilities:-or+. That indicates that there's a problem with the Borg output or what borgmatic is expecting of Borg's output.-or+as expected. If this happens, it indicates something is going wrong with borgmatic's parsing of Borg's output.Did exactly as requested, and got no results at all back (no errors, no results, etc... just thought for a minute or two and then showed that it ended without issue, so i assume its your #1). Played around with things for a while, including attempting the same thing on my local borg repository (it is having the same "issue" as my borgbase one; i just went with my borgbase one in testing because it is smaller/quicker).
I then edited the patterns file and simply removed all my exclusions, so that just my main folders (the ones i actually want backed up) were included in the patterns file, and exactly the same thing; nothing.
I then remembered i had downgraded both borg (to 1.4) and borgmatic (to 1.9.##) just to keep my backups running until figuring this out, so upgraded both to the most recent and retested. Same results for both borg 1.4 and 1.4.1 (meaning, no output).
I then decided to go through and comment out each of my source directories and rerun spot check to see if i could figure out which of my source directories would be causing issues. Nothing stood out.
I am at a loss. I am totally open to this being a user error lol. I just have been using the same config files for a couple months, and it was only until upgrading to borgmatic 2+ that i started having this issue. Downgrading to 1.9.14 appears to have everything fully functional (even with borg 1.4.1). it also only comes up with spot check; if i remove spot check from my config, i don't have any issues either (and that includes when i was on borgmatic 2).
Any other thoughts as to what i can test?
I don't know if it is at all helpful, but as you can see, i own every one of my source directories:
❯ ls -ld /home/jason/docker
drwxrwxrwx 27 jason jason 4096 Apr 19 19:17 /home/jason/docker
❯ ls -ld /home/Data/HomeAssistantBU
drwxrwxr-x 2 jason jason 4096 Apr 30 05:03 /home/Data/HomeAssistantBU
❯ ls -ld /home/Data/PXAsyncthing
drwxr-xr-x 14 jason jason 4096 Apr 15 11:54 /home/Data/PXAsyncthing
❯ ls -ld /home/Media/VC
drwxr-xr-x 3 jason jason 4096 Mar 22 09:18 /home/Media/VC
❯ ls -ld /home/Media/Immich
drwxr-xr-x 3 jason jason 4096 Mar 12 07:37 /home/Media/Immich
❯ ls -ld /home/Media/MusicBeets
drwxr-xr-x 1372 jason jason 61440 Mar 2 12:16 /home/Media/MusicBeets
❯ ls -ld /home/Media/MusicHiDef
drwxr-xr-x 12 jason jason 4096 Jul 18 2024 /home/Media/MusicHiDef
❯ ls -ld /home/Data/PortainerBackup
drwxr-xr-x 2 jason jason 4096 Apr 29 23:00 /home/Data/PortainerBackup
Okay, I think I've found the proximate problem, but not the underlying cause. And it doesn't appear to be user error. What I can see happening is that the
borg create --dry-runcommand that borgmatic runs on your machine is missing the--listflag—the flag that actually causes any files to be listed! I should've caught that previously. (It's also missing the--filterflag and value, which is typically used along with the--listflag.)So if you go back to your original
patternsfile contents and then run the following command, I bet you'd get plenty of files listed in the output:Now the question is why borgmatic is omitting the
--listand --filterflags from that command. Interestingly, it's not omitted on my machine, even with the same version of borgmatic (2.0.4).Aha, I think I found the underlying cause.
--progressis breaking borgmatic's spot check. That's because the borgmatic code that generates theborg createdry run command intentionally omits--listand--filterwhen--progressis set—because--progressinterferes with--listoutput.I think the fix is probably for the spot check to ignore (omit)
--progressfor purposes of constructing its dry run command. Then,--listand--filterwould get included, and the files to spot check would get properly collected.As a temporary work-around for this, simply omit
--progressfrom your borgmatic command.This is fixed in main and will be part of the next release! Thanks for filing it.
Thank you. sorry for not responding sooner. i reran sans --progress and everything appears to be functioning correctly.
Great, glad to hear it. You should be able to use
--progressagain with spot check in the next release.Released in borgmatic 2.0.6!