borgmatic-collective/borgmatic
borgmatic-collective
/
borgmatic
15
57
Fork 40
Code Issues 54 Pull Requests 1 Actions Packages Releases 121 Activity

"Spot check" of source directories consistency check #656

New Issue
Closed
opened 2023-03-21 22:16:56 +00:00 by witten · 5 comments
witten commented 2023-03-21 22:16:56 +00:00
Owner
Copy Link

What I'm trying to do and why

Today, borgmatic supports a number of consistency check types, most of them implemented by Borg. They all have various trade-offs around speed and thoroughness, but one thing none of them do is check the contents of the backed up archives against the original source files—which is arguably one important way to ensure your archives contain the files you'll ultimately want to restore in the case of catastrophe (or just an accidentally deleted file). Because if you happen to misconfigure borgmatic such that you're not backing up the files you think you're backing up, every existing consistency check will still pass with flying colors—and you won't discover this problem until you go to restore.

However, automatically and exhaustively checking an archive's contents against the contents of the source directories on disk has two main problems:

  • It'd be slow: Depending on the source directories, this sort of exhaustive check would be too speed prohibitive to run on a regular basis.
  • It'd yield false negatives: Files change over time. (If they didn't, you probably wouldn't need to use Borg for backups at all.) And with changing files, that means an exhaustive consistency check of source directory contents would fail whenever a source directory changes versus a backup archive—rendering this check pretty useless.

So the proposed solution is to run a "spot check" of source directories. The code implementing such a feature would go something like this:

  1. Pick a random file from within the configured source directories.
  2. Check that the file exists within the selected backup archive and has the same contents in both places.
  3. Repeat the whole process for some percentage of total source directory files.

This approach has the benefit of being fast and hopefully not yielding too many false negatives. The main downside is it's probabilistic; it won't catch 100% of source vs. archive consistency problems on any given run. But that might be good enough given the value that it provides over time.

Additionally, to make the tradeoff between false negatives and thoroughness of the check tunable for different source data, there could be borgmatic configuration options for the check:

  • sample_percentage: The percentage of total files in the source directories to randomly sample and compare to their corresponding files in the backup archive.
  • tolerance_percentage: The percentage of total files in the source directories that can fail a sample comparison without failing the entire consistency check:

Example:

consistency:
    checks:
        - name: spot
          frequency: 2 weeks
          sample_percentage: 5
          tolerance_percentage: 2

Open questions

  • Would this proposed consistency check provide enough value over the existing Borg/borgmatic consistency checks?
  • How feasible would this proposed consistency check be to implement with Borg? It would require effectively diffing random files within archives against their corresponding files on disk. (And no, borg diff won't do that.) Is it easy to do that in a performant-enough way?
  • Is this feature just too complex, either to use as an end-user or to implement/maintain as a developer?
  • Should this proposed consistency check only check against the most recent archive in a repository, like the existing extract check does?
  • Rather than checking file contents, would it be better to simply check for existence of source directory files within the backup archive? That'd be faster, simpler, and presumably wouldn't have as many false negatives. It'd also still handle the use case of catching a borgmatic include/exclude misconfiguration.
#### What I'm trying to do and why Today, borgmatic supports a number of [consistency check types](https://torsion.org/borgmatic/docs/how-to/deal-with-very-large-backups/#consistency-check-configuration), most of them implemented by Borg. They all have various trade-offs around speed and thoroughness, but one thing none of them do is check the contents of the backed up archives against the original source files—which is arguably one important way to ensure your archives contain the files you'll ultimately want to restore in the case of catastrophe (or just an accidentally deleted file). Because if you happen to misconfigure borgmatic such that you're not backing up the files you *think* you're backing up, every existing consistency check will still pass with flying colors—and you won't discover this problem until you go to restore. However, automatically and exhaustively checking an archive's contents against the contents of the source directories on disk has two main problems: * It'd be slow: Depending on the source directories, this sort of exhaustive check would be too speed prohibitive to run on a regular basis. * It'd yield false negatives: Files change over time. (If they didn't, you probably wouldn't need to use Borg for backups at all.) And with changing files, that means an exhaustive consistency check of source directory contents would fail whenever a source directory changes versus a backup archive—rendering this check pretty useless. So the proposed solution is to run a "spot check" of source directories. The code implementing such a feature would go something like this: 1. Pick a random file from within the configured source directories. 2. Check that the file exists within the selected backup archive and has the same contents in both places. 3. Repeat the whole process for some percentage of total source directory files. This approach has the benefit of being fast and hopefully not yielding too many false negatives. The main downside is it's probabilistic; it won't catch 100% of source vs. archive consistency problems on any given run. But that might be good enough given the value that it provides over time. Additionally, to make the tradeoff between false negatives and thoroughness of the check tunable for different source data, there could be borgmatic configuration options for the check: * `sample_percentage`: The percentage of total files in the source directories to randomly sample and compare to their corresponding files in the backup archive. * `tolerance_percentage`: The percentage of total files in the source directories that can fail a sample comparison without failing the entire consistency check: Example: ``` consistency: checks: - name: spot frequency: 2 weeks sample_percentage: 5 tolerance_percentage: 2 ``` #### Open questions * Would this proposed consistency check provide enough value over the existing Borg/borgmatic consistency checks? * How feasible would this proposed consistency check be to implement with Borg? It would require effectively diffing random files within archives against their corresponding files on disk. (And no, `borg diff` won't do that.) Is it easy to do that in a performant-enough way? * Is this feature just too complex, either to use as an end-user or to implement/maintain as a developer? * Should this proposed consistency check only check against the most recent archive in a repository, like the existing `extract` check does? * Rather than checking file contents, would it be better to simply check for *existence* of source directory files within the backup archive? That'd be faster, simpler, and presumably wouldn't have as many false negatives. It'd also still handle the use case of catching a borgmatic include/exclude misconfiguration.
witten added the
new feature area
 label 2023-06-28 18:39:03 +00:00
witten commented 2024-04-15 21:35:29 +00:00
Author
Owner
Copy Link

Implemented in main and will be part of the next release! See the documentation for actual configuration options: https://torsion.org/borgmatic/docs/how-to/deal-with-very-large-backups/#spot-check

Implemented in main and will be part of the next release! See the documentation for actual configuration options: https://torsion.org/borgmatic/docs/how-to/deal-with-very-large-backups/#spot-check
witten commented 2024-04-16 17:44:52 +00:00
Author
Owner
Copy Link

Released in 1.8.10!

Released in 1.8.10!
witten referenced this issue from a commit 2024-04-16 17:50:51 +00:00
Wording tweak (#656).
witten referenced this issue from a commit 2024-04-16 17:53:06 +00:00
Wording tweak (#656).
robdavies commented 2024-04-17 02:01:46 +00:00
Copy Link

Great news this is progressing.

I'm currently using #760 and also added a script to check/verify a random set of files, reading up to a set amount of data (2GB in my usage) and a minimum set of files (I've set it to 5 for now).

I figured that if I read a large file, like a video file, it would trip the 2GB threshold and check only 1 file.

I plan to check this new spotcheck feature out today.

Great news this is progressing. I'm currently using https://projects.torsion.org/borgmatic-collective/borgmatic/issues/760 and also added a script to check/verify a random set of files, reading up to a set amount of data (2GB in my usage) and a minimum set of files (I've set it to 5 for now). I figured that if I read a large file, like a video file, it would trip the 2GB threshold and check only 1 file. I plan to check this new spotcheck feature out today.
witten commented 2024-04-17 03:44:10 +00:00
Author
Owner
Copy Link

Ah, yeah, the spot check feature doesn't currently have a threshold for file sizes, but that would be an interesting thing to add. And I'd love to hear about any feedback you have as you try out the feature!

Ah, yeah, the spot check feature doesn't currently have a threshold for file sizes, but that would be an interesting thing to add. And I'd love to hear about any feedback you have as you try out the feature!
robdavies commented 2024-04-25 08:40:18 +00:00
Copy Link

Hi, I wanted to give it a few tries before giving feedback.

My usage for spot check is for verifying immediately after backup. I ran some backups with the following settings :
count_tolerance_percentage: 0
data_sample_percentage: 1
data_tolerance_percentage: 0

I have the verbose/logging set to 1 (-v 1)

For me, this made sure that all of the files are backed up (file count) and any of the files compared don't differ (file compare). Its pretty much what my current verify scripts do, compare file count, compare a random set of files (mine is set to compare 1.5GB of files).

  1. It caught some borg files that change during the backup which caused the spot check to fail. For me, this was a success as I hadn't seen this before (my other verify scripts caught this too). This change may have occurred after an update with borg. I excluded these and ran the backup again, this time the spot check passed.

  2. When a spot check fails, it would be helpful to know how the number of the file count (backup and repo) also which files fail the compare check. I had a small amount of files fail the check during a backup, it reported 0.00% and couldn't tell where the problem was or how many files were an issue.

  3. The spot check failed one time then retried 5 more times. Is this the same retry setting? Does it need to retry, wouldn't it get the same result each time? All the other checks passed, they were reran each time. Do i need to change settings?

  4. It works on backup and check, if I run check then sometimes it fails on first try (only on one of my repositories) then passes on the second try. I couldn't find what made it fail (like number 2).

  5. I think it would be helpful to know how much data has been verified as well as number of files e.g. Verified : 1% (2000 files, 1.5GB)

  6. If the spot check fails during backup, does the backup fail and gets deleted or something else?

This is working really well.

Hi, I wanted to give it a few tries before giving feedback. My usage for spot check is for verifying immediately after backup. I ran some backups with the following settings : count_tolerance_percentage: 0 data_sample_percentage: 1 data_tolerance_percentage: 0 I have the verbose/logging set to 1 (-v 1) For me, this made sure that all of the files are backed up (file count) and any of the files compared don't differ (file compare). Its pretty much what my current verify scripts do, compare file count, compare a random set of files (mine is set to compare 1.5GB of files). 1) It caught some borg files that change during the backup which caused the spot check to fail. For me, this was a success as I hadn't seen this before (my other verify scripts caught this too). This change may have occurred after an update with borg. I excluded these and ran the backup again, this time the spot check passed. 2) When a spot check fails, it would be helpful to know how the number of the file count (backup and repo) also which files fail the compare check. I had a small amount of files fail the check during a backup, it reported 0.00% and couldn't tell where the problem was or how many files were an issue. 3) The spot check failed one time then retried 5 more times. Is this the same retry setting? Does it need to retry, wouldn't it get the same result each time? All the other checks passed, they were reran each time. Do i need to change settings? 4) It works on backup and check, if I run check then sometimes it fails on first try (only on one of my repositories) then passes on the second try. I couldn't find what made it fail (like number 2). 5) I think it would be helpful to know how much data has been verified as well as number of files e.g. Verified : 1% (2000 files, 1.5GB) 6) If the spot check fails during backup, does the backup fail and gets deleted or something else? This is working really well.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
main
spot-check
1.8.10
1.8.9
1.8.8
1.8.7
1.8.6
1.8.5
1.8.4
1.8.3
1.8.2
1.8.1
1.8.0
1.7.15
1.7.14
1.7.13
1.7.12
1.7.11
1.7.10
1.7.9
1.7.8
1.7.7
1.7.6
1.7.5
1.7.4
1.7.3
1.7.2
1.7.1
1.7.0
1.6.6
1.6.5
1.6.4
1.6.3
1.6.2
1.6.1
1.6.0
1.5.24
1.5.23
1.5.22
1.5.21
1.5.20
1.5.19
1.5.18
1.5.17
1.5.16
1.5.15
1.5.14
1.5.13
1.5.12
1.5.11
1.5.10
1.5.9
1.5.8
1.5.7
1.5.7.dev0
1.5.6
1.5.5
1.5.4
1.5.3
1.5.2
1.5.1
1.5.0
1.4.22
1.4.21
1.4.20
1.4.19
1.4.18
1.4.17
1.4.16
1.4.15
1.4.14
1.4.13
1.4.12
1.4.11
1.4.10
1.4.9
1.4.8
1.4.7
1.4.6
1.4.5
1.4.4
1.4.3
1.4.2
1.4.1
1.4.0
1.3.26
1.3.25
1.3.24
1.3.23
1.3.22
1.3.21
1.3.20
1.3.19
1.3.18
1.3.17
1.3.16
1.3.15
1.3.14
1.3.13
1.3.12
1.3.11
1.3.10
1.3.9
1.3.8
1.3.7
1.3.6
1.3.5
1.3.4
1.3.3
1.3.2
1.3.1
1.3.0
1.2.18
1.2.17
1.2.16
1.2.15
1.2.14
1.2.13
1.2.12
1.2.11
1.2.10
1.2.9
1.2.8
1.2.7
1.2.6
1.2.5
1.2.4
1.2.3
1.2.2
1.2.1
1.2.0
1.1.15
1.1.14
1.1.13
1.1.12
1.1.11
1.1.10
1.1.9
1.1.7
1.1.6
1.1.5
1.1.4
1.1.3
1.1.2
1.1.1
1.1.0
1.0.3
1.0.2
1.0.1
1.0.0
0.1.8
0.1.7
0.1.6
0.1.5
0.1.4
0.1.3
0.1.2
0.1.1
0.1.0
0.0.7
0.0.6
0.0.5
0.0.4
0.0.3
0.0.2
Labels
Clear labels   
bug

   
data loss

   
design finalized

   
good first issue

   
new feature area

   
question / support

   
security

   
waiting for response
No Label
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#656
No description provided.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?