spot check fails due to config files not in patterns

Alexius33 commented

2025-08-22 17:53:01 +00:00

What I'm trying to do and why

Hi all!
I'm playing around a little trying to figure out the various configurations but I cannot figure out how to make the spot check work when the configurations are backed up for bootstrap, as those files are not ignored and therefore detected as a difference.

Of course, doing

bootstrap:
    store_config_files: false

or manually adding the config files among the source files solves the problem, but I'd love to backup the configurations in a transparent fashion so that I can seamlessly change the files included in my configs.

Should I manually add the config files to the source files/patterns or should those files be automatically ignored like the manifest.json itself?

Steps to reproduce

Here's a snippet of the confgurations:

encryption_passphrase: 'SomePassword'

repositories:
  - path: "/mnt/ssd/demoborgmatic"
    label: "SSD (Local)"
    encryption: repokey
    append_only: false
    make_parent_directories: true

checks:
  - name: spot
    count_tolerance_percentage: 0
    data_sample_percentage: 5
    data_tolerance_percentage: 0

patterns:
  - 'R /home/alessio/demoborg'

And here's the tree of input files

/home/alessio/demoborg
├── dirA
│   ├── dirA1
│   │   └── dirA1file
│   ├── dirA2
│   │   ├── dirA2file
│   │   ├── dirA2file2
│   │   └── dirA2file3
│   └── dirAfile
└── rootfile

4 directories, 6 files

Actual behavior

Here are the logs of the check

root@alpha:~# borgmatic check -v2
/etc/borgmatic.d/config.yaml: BORG_PASSPHRASE_FD=*** BORG_DEBUG_PASSPHRASE=*** BORG_DISPLAY_PASSPHRASE=*** BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_USE_CHUNKS_ARCHIVE=*** BORG_EXIT_CODES=*** borg --version --debug --show-rc
/etc/borgmatic.d/config.yaml: Borg 1.4.1
SSD (Local): Running actions for repository
SSD (Local): Running consistency checks
SSD (Local): BORG_PASSPHRASE_FD=*** BORG_DEBUG_PASSPHRASE=*** BORG_DISPLAY_PASSPHRASE=*** BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_USE_CHUNKS_ARCHIVE=*** BORG_EXIT_CODES=*** borg info --json /mnt/ssd/demoborgmatic
SSD (Local): Using runtime directory /tmp/borgmatic-bgv3z8qc/borgmatic
SSD (Local): Running spot check
SSD (Local): Calling bootstrap hook function use_streaming
SSD (Local): Writing patterns to /tmp/borgmatic-bgv3z8qc/borgmatic/tmplk83h_6o:
R /home/alessio/demoborg
SSD (Local): BORG_PASSPHRASE_FD=*** BORG_DEBUG_PASSPHRASE=*** BORG_DISPLAY_PASSPHRASE=*** BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_USE_CHUNKS_ARCHIVE=*** BORG_EXIT_CODES=*** borg create --patterns-from /tmp/borgmatic-bgv3z8qc/borgmatic/tmplk83h_6o --list --filter AMEx- --dry-run /mnt/ssd/demoborgmatic::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}
SSD (Local): 6 total source paths for spot check
SSD (Local): BORG_PASSPHRASE_FD=*** BORG_DEBUG_PASSPHRASE=*** BORG_DISPLAY_PASSPHRASE=*** BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_USE_CHUNKS_ARCHIVE=*** BORG_EXIT_CODES=*** borg list --last 1 --json /mnt/ssd/demoborgmatic
SSD (Local): Latest archive is alpha-2025-08-22T19:19:02.271414
SSD (Local): Using archive alpha-2025-08-22T19:19:02.271414 for spot check
SSD (Local): BORG_PASSPHRASE_FD=*** BORG_DEBUG_PASSPHRASE=*** BORG_DISPLAY_PASSPHRASE=*** BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_USE_CHUNKS_ARCHIVE=*** BORG_EXIT_CODES=*** borg list --debug --show-rc --format {type} {path}{NUL} /mnt/ssd/demoborgmatic::alpha-2025-08-22T19:19:02.271414
using builtin fallback logging configuration
33 self tests completed in 0.08 seconds
Verified integrity of /mnt/ssd/demoborgmatic/index.9
TAM-verified manifest
security: read previous location '/mnt/ssd/demoborgmatic'
security: read manifest timestamp '2025-08-22T17:19:02.407644'
security: determined newest manifest timestamp as 2025-08-22T17:19:02.407644
security: repository checks ok, allowing access
Archive authentication DISABLED.
TAM-verified archive
terminating with success status, rc 0
SSD (Local): 7 total archive paths for spot check
SSD (Local): Paths in source paths but not latest archive: none
SSD (Local): Paths in latest archive but not source paths: etc/borgmatic.d/config.yaml
SSD (Local): Error running actions for repository
SSD (Local): Spot check failed: 16.67% file count delta between source paths and latest archive (tolerance is 0%)
/etc/borgmatic.d/config.yaml: Error running configuration
/etc/borgmatic.d/config.yaml: An error occurred

summary:
/etc/borgmatic.d/config.yaml: Loading configuration file
An error occurred
Error running actions for repository
Spot check failed: 16.67% file count delta between source paths and latest archive (tolerance is 0%)
Error running configuration

Need some help? https://torsion.org/borgmatic/#issues

and here the full content of the archive

root@alpha:~# borgmatic list --archive alpha-2025-08-22T19:19:02.271414
SSD (Local): Listing archive alpha-2025-08-22T19:19:02.271414
drwxrwxr-x alessio alessio        0 Sun, 2025-08-17 17:05:53 home/alessio/demoborg
drwxrwxr-x alessio alessio        0 Sun, 2025-08-17 17:06:01 home/alessio/demoborg/dirA
drwxrwxr-x alessio alessio        0 Sun, 2025-08-17 17:06:14 home/alessio/demoborg/dirA/dirA1
-rw-rw-r-- alessio alessio        0 Sun, 2025-08-17 17:06:14 home/alessio/demoborg/dirA/dirA1/dirA1file
drwxrwxr-x alessio alessio        0 Sun, 2025-08-17 17:06:24 home/alessio/demoborg/dirA/dirA2
-rw-rw-r-- alessio alessio        0 Sun, 2025-08-17 17:06:20 home/alessio/demoborg/dirA/dirA2/dirA2file
-rw-rw-r-- alessio alessio        0 Sun, 2025-08-17 17:06:23 home/alessio/demoborg/dirA/dirA2/dirA2file2
-rw-rw-r-- alessio alessio        0 Sun, 2025-08-17 17:06:24 home/alessio/demoborg/dirA/dirA2/dirA2file3
-rw-rw-r-- alessio alessio        0 Sun, 2025-08-17 17:06:01 home/alessio/demoborg/dirA/dirAfile
-rw-rw-r-- alessio alessio        0 Sun, 2025-08-17 17:05:53 home/alessio/demoborg/rootfile
-rw-r--r-- root   root        357 Fri, 2025-08-22 19:18:29 etc/borgmatic.d/config.yaml
drwxr-xr-x root   root          0 Fri, 2025-08-22 19:19:02 borgmatic/bootstrap
-rw-r--r-- root   root         80 Fri, 2025-08-22 19:19:02 borgmatic/bootstrap/manifest.json

Expected behavior

I'd expect config files to be backed up, but ignored during spot check, just like the file manifest.json, so it passes with 0% file count delta.

Other notes / implementation ideas

Other note:

I'm executing borgmatic as root, backing up files from the user alessio.

Implementation idea:

Could it be a possible solution maybe to merge the content of the manifest.json (namely the array config_paths) to the temp file containing the patterns (/tmp/borgmatic-bgv3z8qc/borgmatic/tmplk83h_6o) before calling borg create --patterns-from ... --dry-run?

borgmatic version

2.0.7

borgmatic installation method

pipx install

Borg version

borg 1.4.1

Python version

Python 3.12.3

Database version (if applicable)

No response

Operating system and version

Ubuntu 24.04.3 LTS

### What I'm trying to do and why Hi all! I'm playing around a little trying to figure out the various configurations but I cannot figure out how to make the spot check work when the configurations are backed up for bootstrap, as those files are not ignored and therefore detected as a difference. Of course, doing ``` bootstrap: store_config_files: false ``` or manually adding the config files among the source files solves the problem, but I'd love to backup the configurations in a transparent fashion so that I can seamlessly change the files included in my configs. Should I manually add the config files to the source files/patterns or should those files be automatically ignored like the `manifest.json` itself? ### Steps to reproduce Here's a snippet of the confgurations: ``` encryption_passphrase: 'SomePassword' repositories: - path: "/mnt/ssd/demoborgmatic" label: "SSD (Local)" encryption: repokey append_only: false make_parent_directories: true checks: - name: spot count_tolerance_percentage: 0 data_sample_percentage: 5 data_tolerance_percentage: 0 patterns: - 'R /home/alessio/demoborg' ``` --- And here's the tree of input files ``` /home/alessio/demoborg ├── dirA │ ├── dirA1 │ │ └── dirA1file │ ├── dirA2 │ │ ├── dirA2file │ │ ├── dirA2file2 │ │ └── dirA2file3 │ └── dirAfile └── rootfile 4 directories, 6 files ``` ### Actual behavior Here are the logs of the check ``` root@alpha:~# borgmatic check -v2 /etc/borgmatic.d/config.yaml: BORG_PASSPHRASE_FD=*** BORG_DEBUG_PASSPHRASE=*** BORG_DISPLAY_PASSPHRASE=*** BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_USE_CHUNKS_ARCHIVE=*** BORG_EXIT_CODES=*** borg --version --debug --show-rc /etc/borgmatic.d/config.yaml: Borg 1.4.1 SSD (Local): Running actions for repository SSD (Local): Running consistency checks SSD (Local): BORG_PASSPHRASE_FD=*** BORG_DEBUG_PASSPHRASE=*** BORG_DISPLAY_PASSPHRASE=*** BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_USE_CHUNKS_ARCHIVE=*** BORG_EXIT_CODES=*** borg info --json /mnt/ssd/demoborgmatic SSD (Local): Using runtime directory /tmp/borgmatic-bgv3z8qc/borgmatic SSD (Local): Running spot check SSD (Local): Calling bootstrap hook function use_streaming SSD (Local): Writing patterns to /tmp/borgmatic-bgv3z8qc/borgmatic/tmplk83h_6o: R /home/alessio/demoborg SSD (Local): BORG_PASSPHRASE_FD=*** BORG_DEBUG_PASSPHRASE=*** BORG_DISPLAY_PASSPHRASE=*** BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_USE_CHUNKS_ARCHIVE=*** BORG_EXIT_CODES=*** borg create --patterns-from /tmp/borgmatic-bgv3z8qc/borgmatic/tmplk83h_6o --list --filter AMEx- --dry-run /mnt/ssd/demoborgmatic::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} SSD (Local): 6 total source paths for spot check SSD (Local): BORG_PASSPHRASE_FD=*** BORG_DEBUG_PASSPHRASE=*** BORG_DISPLAY_PASSPHRASE=*** BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_USE_CHUNKS_ARCHIVE=*** BORG_EXIT_CODES=*** borg list --last 1 --json /mnt/ssd/demoborgmatic SSD (Local): Latest archive is alpha-2025-08-22T19:19:02.271414 SSD (Local): Using archive alpha-2025-08-22T19:19:02.271414 for spot check SSD (Local): BORG_PASSPHRASE_FD=*** BORG_DEBUG_PASSPHRASE=*** BORG_DISPLAY_PASSPHRASE=*** BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=*** BORG_USE_CHUNKS_ARCHIVE=*** BORG_EXIT_CODES=*** borg list --debug --show-rc --format {type} {path}{NUL} /mnt/ssd/demoborgmatic::alpha-2025-08-22T19:19:02.271414 using builtin fallback logging configuration 33 self tests completed in 0.08 seconds Verified integrity of /mnt/ssd/demoborgmatic/index.9 TAM-verified manifest security: read previous location '/mnt/ssd/demoborgmatic' security: read manifest timestamp '2025-08-22T17:19:02.407644' security: determined newest manifest timestamp as 2025-08-22T17:19:02.407644 security: repository checks ok, allowing access Archive authentication DISABLED. TAM-verified archive terminating with success status, rc 0 SSD (Local): 7 total archive paths for spot check SSD (Local): Paths in source paths but not latest archive: none SSD (Local): Paths in latest archive but not source paths: etc/borgmatic.d/config.yaml SSD (Local): Error running actions for repository SSD (Local): Spot check failed: 16.67% file count delta between source paths and latest archive (tolerance is 0%) /etc/borgmatic.d/config.yaml: Error running configuration /etc/borgmatic.d/config.yaml: An error occurred summary: /etc/borgmatic.d/config.yaml: Loading configuration file An error occurred Error running actions for repository Spot check failed: 16.67% file count delta between source paths and latest archive (tolerance is 0%) Error running configuration Need some help? https://torsion.org/borgmatic/#issues ``` and here the full content of the archive ``` root@alpha:~# borgmatic list --archive alpha-2025-08-22T19:19:02.271414 SSD (Local): Listing archive alpha-2025-08-22T19:19:02.271414 drwxrwxr-x alessio alessio 0 Sun, 2025-08-17 17:05:53 home/alessio/demoborg drwxrwxr-x alessio alessio 0 Sun, 2025-08-17 17:06:01 home/alessio/demoborg/dirA drwxrwxr-x alessio alessio 0 Sun, 2025-08-17 17:06:14 home/alessio/demoborg/dirA/dirA1 -rw-rw-r-- alessio alessio 0 Sun, 2025-08-17 17:06:14 home/alessio/demoborg/dirA/dirA1/dirA1file drwxrwxr-x alessio alessio 0 Sun, 2025-08-17 17:06:24 home/alessio/demoborg/dirA/dirA2 -rw-rw-r-- alessio alessio 0 Sun, 2025-08-17 17:06:20 home/alessio/demoborg/dirA/dirA2/dirA2file -rw-rw-r-- alessio alessio 0 Sun, 2025-08-17 17:06:23 home/alessio/demoborg/dirA/dirA2/dirA2file2 -rw-rw-r-- alessio alessio 0 Sun, 2025-08-17 17:06:24 home/alessio/demoborg/dirA/dirA2/dirA2file3 -rw-rw-r-- alessio alessio 0 Sun, 2025-08-17 17:06:01 home/alessio/demoborg/dirA/dirAfile -rw-rw-r-- alessio alessio 0 Sun, 2025-08-17 17:05:53 home/alessio/demoborg/rootfile -rw-r--r-- root root 357 Fri, 2025-08-22 19:18:29 etc/borgmatic.d/config.yaml drwxr-xr-x root root 0 Fri, 2025-08-22 19:19:02 borgmatic/bootstrap -rw-r--r-- root root 80 Fri, 2025-08-22 19:19:02 borgmatic/bootstrap/manifest.json ``` ### Expected behavior I'd expect config files to be backed up, but ignored during spot check, just like the file `manifest.json`, so it passes with 0% file count delta. ### Other notes / implementation ideas ### Other note: I'm executing borgmatic as root, backing up files from the user `alessio`. ### Implementation idea: Could it be a possible solution maybe to merge the content of the `manifest.json` (namely the array `config_paths`) to the temp file containing the patterns (`/tmp/borgmatic-bgv3z8qc/borgmatic/tmplk83h_6o`) before calling `borg create --patterns-from ... --dry-run`? ### borgmatic version 2.0.7 ### borgmatic installation method pipx install ### Borg version borg 1.4.1 ### Python version Python 3.12.3 ### Database version (if applicable) _No response_ ### Operating system and version Ubuntu 24.04.3 LTS

👍 1

2rs2ts commented

2025-08-27 17:07:00 +00:00

Should I manually add the config files to the source files/patterns or should those files be automatically ignored like the manifest.json itself?

So, I'm dealing with similar problems about the spot check failing, but my math comes out such that it seems that neither manifest.json nor the config file are being ignored. I think the same is the case for you too, right? If only the config file was not ignored, then you couldn't get 16.7% for the file count delta, since you have 13 archive entries. Even if the directory entries in the archive are ignored (which honestly would make sense to do) then that would be only ~9% difference. Maybe I'm just misunderstanding how the math works out, though.

I figured I'd comment to make sure that no aspect of this gets overlooked, because right now it's really frustrating that there's no way to do a good backup and a good spot check at the same time, you either have to forgo the bootstrap functionality or allow for more count_tolerance_percentage than what you would ideally like...

> Should I manually add the config files to the source files/patterns or should those files be automatically ignored like the `manifest.json` itself? So, I'm dealing with similar problems about the spot check failing, but my math comes out such that it seems that neither `manifest.json` nor the config file are being ignored. I think the same is the case for you too, right? If only the config file was not ignored, then you couldn't get 16.7% for the file count delta, since you have 13 archive entries. Even if the directory entries in the archive are ignored (which honestly would make sense to do) then that would be only ~9% difference. Maybe I'm just misunderstanding how the math works out, though. I figured I'd comment to make sure that no aspect of this gets overlooked, because right now it's really frustrating that there's no way to do a good backup and a good spot check at the same time, you either have to forgo the bootstrap functionality or allow for more `count_tolerance_percentage` than what you would ideally like...

witten commented

2025-09-06 05:49:58 +00:00

Owner

Zero tolerances. Bold. 😄

I recognize the issue you're pointing out with configuration files that get automatically stored in an archive (to support bootstrap) getting used for spot check comparisons—when those configuration files don't exist in the source directories.

Could it be a possible solution maybe to merge the content of the manifest.json (namely the array config_paths) to the temp file containing the patterns (/tmp/borgmatic-bgv3z8qc/borgmatic/tmplk83h_6o) before calling borg create --patterns-from ... --dry-run?

Yeah, something like this might be the way to go. Because I don't think I can omit the configuration files from the list of archive paths to spot check, as a user might legitimately include their configuration files from their listed source directories. But adding the configuration files to the list of source paths to spot check (when bootstrapping is enabled) might work.

Zero tolerances. Bold. 😄 I recognize the issue you're pointing out with configuration files that get automatically stored in an archive (to support `bootstrap`) getting used for spot check comparisons—when those configuration files don't exist in the source directories. > Could it be a possible solution maybe to merge the content of the manifest.json (namely the array config_paths) to the temp file containing the patterns (/tmp/borgmatic-bgv3z8qc/borgmatic/tmplk83h_6o) before calling borg create --patterns-from ... --dry-run? Yeah, something like this might be the way to go. Because I don't think I can *omit* the configuration files from the list of archive paths to spot check, as a user might legitimately include their configuration files from their listed source directories. But adding the configuration files to the list of *source* paths to spot check (when bootstrapping is enabled) might work.

witten commented

2025-09-06 05:57:45 +00:00

Owner

In terms of the math, be aware that: 1. Yes, directories are ignored, and 2. Anything in borgmatic's runtime directory is ignored. So there should be way fewer than 13 archive entries in this case. I'm counting 7 including the config file. And 6 in the source directories.

And here's the code that produced that 16.67%:

count_delta_percentage = abs(len(source_paths) - len(archive_paths)) / len(source_paths) * 100

So I think that's abs(6 - 7) / 6 * 100 = 16.67%

In terms of the math, be aware that: 1. Yes, directories are ignored, and 2. Anything in borgmatic's runtime directory is ignored. So there should be way fewer than 13 archive entries in this case. I'm counting 7 including the config file. And 6 in the source directories. And here's the code that produced that 16.67%: ```python count_delta_percentage = abs(len(source_paths) - len(archive_paths)) / len(source_paths) * 100 ``` So I think that's `abs(6 - 7) / 6 * 100 = 16.67%`

witten added the

bug

label

2025-09-06 06:00:29 +00:00

witten referenced this issue from a commit

2025-09-08 19:18:02 +00:00

Fix the "spot" check to include borgmatic configuration files that were backed up to support the "bootstrap" action (#1133).

witten referenced this issue

2025-09-08 19:31:48 +00:00

Include bootstrapped configuration files in spot check. #1143

witten commented

2025-09-08 19:33:02 +00:00