borgmatic-collective/borgmatic
borgmatic-collective
/
borgmatic
15
57
Fork 40
Code Issues 54 Pull Requests 1 Actions Packages Releases 121 Activity

Questions: deduplication, compression, extraction #84

New Issue
Closed
opened 2018-08-01 21:08:17 +00:00 by patricko · 5 comments
patricko commented 2018-08-01 21:08:17 +00:00
Copy Link

Again, some noob questions

  1. Borg is deduplicating files, so does it mean that one data corruption can lead to all the other backups being corrupted?

Is there a backup of backups with borgmatic? like a "full backup" every 7 days or a "base backup".

  1. To use compression, we set either:
none
lz4
zlib,1-9
lzma,1-9

Is that right? I tried but I saw 0 difference in the result size though

  1. List of one or more consistency checks to run: "repository", "archives", and/or

    "extract". Defaults to "repository" and "archives". Set to "disabled" to disable

    all consistency checks. "repository" checks the consistency of the repository,

    "archive" checks all of the archives, and "extract" does an extraction dry-run

    of just the most recent archive.

    checks:
    - repository
    - archives
    - extract

I added "-extract" maybe you should add it to the conf too.
Anyway, this will verify if the last archive is corrupted or not? or what does it do exactly?

Again, some noob questions 1) Borg is deduplicating files, so does it mean that one data corruption can lead to all the other backups being corrupted? Is there a backup of backups with borgmatic? like a "full backup" every 7 days or a "base backup". 2) To use compression, we set either: ``` none lz4 zlib,1-9 lzma,1-9 ``` Is that right? I tried but I saw 0 difference in the result size though 3) # List of one or more consistency checks to run: "repository", "archives", and/or # "extract". Defaults to "repository" and "archives". Set to "disabled" to disable # all consistency checks. "repository" checks the consistency of the repository, # "archive" checks all of the archives, and "extract" does an extraction dry-run # of just the most recent archive. checks: - repository - archives - extract I added "-extract" maybe you should add it to the conf too. Anyway, this will verify if the last archive is corrupted or not? or what does it do exactly?
witten added the
question / support
 label 2018-08-02 05:08:07 +00:00
witten commented 2018-08-02 05:21:21 +00:00
Owner
Copy Link
  1. As far as I know, data corruption of a de-duplicated block can indeed screw up multiple different backup archives. If this is a concern, I recommend backing up to multiple different locations. There's not a "full backup" feature like you're describing with borgmatic or Borg. But it would be pretty easy to declare multiple repositories in your borgmatic config file, and borgmatic will backup to each repository separately, thereby getting you some of the redundancy you're looking for. (Note that unless these are stored on separate disks or servers, it won't necessarily help redundancy much.)

Also, there are a couple of Borg FAQ entries that may be of interest on this topic.

  1. That's generally right, but there are a few more compression options as well. Run borg help compression for more info. Whether compression does anything depends entirely on the source data. If your source data is already pretty well compressed, extra compression won't help much. If you suspect there's a borgmatic bug here (e.g. not passing your compression option to Borg), feel free to run borgmatic with -v 2 to up the verbosity and see exactly what it's passing to Borg.

  2. extract actually tries to do a dry-run restore of the most recent archive to make sure that it's extractable. For instance, with sufficient repository corruption, an extract check would fail and you'd know that you couldn't actually restore that most recent archive. So it's just a sort of smoke test.

1) As far as I know, data corruption of a de-duplicated block can indeed screw up multiple different backup archives. If this is a concern, I recommend backing up to multiple different locations. There's not a "full backup" feature like you're describing with borgmatic or Borg. But it would be pretty easy to declare multiple `repositories` in your borgmatic config file, and borgmatic will backup to each repository separately, thereby getting you some of the redundancy you're looking for. (Note that unless these are stored on separate disks or servers, it won't necessarily help redundancy much.) Also, there are a [couple of Borg FAQ entries](https://borgbackup.readthedocs.io/en/stable/faq.html?highlight=corruption#can-project-name-add-redundancy-to-the-backup-data-to-deal-with-hardware-malfunction) that may be of interest on this topic. 2) That's generally right, but there are a few more compression options as well. Run `borg help compression` for more info. Whether compression does anything depends entirely on the source data. If your source data is already pretty well compressed, extra compression won't help much. If you suspect there's a borgmatic bug here (e.g. not passing your compression option to Borg), feel free to run borgmatic with `-v 2` to up the verbosity and see exactly what it's passing to Borg. 3) `extract` actually tries to do a dry-run restore of the most recent archive to make sure that it's extractable. For instance, with sufficient repository corruption, an `extract` check would fail and you'd know that you couldn't actually restore that most recent archive. So it's just a sort of smoke test.
patricko commented 2018-08-11 20:30:13 +00:00
Author
Copy Link

  1. Fine!

  2. Ok thank you for that command!! So using "auto,lzma" seems like the best option to me.
    But what's the point of "auto"? to save time? isn't it risky though?

  3. Awesome! this tool is awesome
    My only concern is why not having it installable in a .deb archive too (in official distrib repos)?? too much work maybe

1) Fine! 2) Ok thank you for that command!! So using "auto,lzma" seems like the best option to me. But what's the point of "auto"? to save time? isn't it risky though? 3) Awesome! this tool is awesome My only concern is why not having it installable in a .deb archive too (in official distrib repos)?? too much work maybe
witten commented 2018-08-11 20:33:09 +00:00
Owner
Copy Link

There are .deb packages of borgbackup, but not yet for borgmatic. See that thread for more information.

There are .deb packages of [borgbackup](https://packages.debian.org/sid/borgbackup), but not yet for [borgmatic](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=887809). See that thread for more information.
witten commented 2018-08-11 20:39:01 +00:00
Owner
Copy Link

Oh, as for auto compression, I don't know. My guess is that it might save a little CPU on decompression, because it's not unnecessarily (de-)compressing if compression won't actually save any space.

Oh, as for `auto` compression, I don't know. My guess is that it might save a little CPU on decompression, because it's not unnecessarily (de-)compressing if compression won't actually save any space.
patricko commented 2018-08-12 20:19:21 +00:00
Author
Copy Link

anyway thank you!!!!

anyway thank you!!!!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
main
spot-check
1.8.10
1.8.9
1.8.8
1.8.7
1.8.6
1.8.5
1.8.4
1.8.3
1.8.2
1.8.1
1.8.0
1.7.15
1.7.14
1.7.13
1.7.12
1.7.11
1.7.10
1.7.9
1.7.8
1.7.7
1.7.6
1.7.5
1.7.4
1.7.3
1.7.2
1.7.1
1.7.0
1.6.6
1.6.5
1.6.4
1.6.3
1.6.2
1.6.1
1.6.0
1.5.24
1.5.23
1.5.22
1.5.21
1.5.20
1.5.19
1.5.18
1.5.17
1.5.16
1.5.15
1.5.14
1.5.13
1.5.12
1.5.11
1.5.10
1.5.9
1.5.8
1.5.7
1.5.7.dev0
1.5.6
1.5.5
1.5.4
1.5.3
1.5.2
1.5.1
1.5.0
1.4.22
1.4.21
1.4.20
1.4.19
1.4.18
1.4.17
1.4.16
1.4.15
1.4.14
1.4.13
1.4.12
1.4.11
1.4.10
1.4.9
1.4.8
1.4.7
1.4.6
1.4.5
1.4.4
1.4.3
1.4.2
1.4.1
1.4.0
1.3.26
1.3.25
1.3.24
1.3.23
1.3.22
1.3.21
1.3.20
1.3.19
1.3.18
1.3.17
1.3.16
1.3.15
1.3.14
1.3.13
1.3.12
1.3.11
1.3.10
1.3.9
1.3.8
1.3.7
1.3.6
1.3.5
1.3.4
1.3.3
1.3.2
1.3.1
1.3.0
1.2.18
1.2.17
1.2.16
1.2.15
1.2.14
1.2.13
1.2.12
1.2.11
1.2.10
1.2.9
1.2.8
1.2.7
1.2.6
1.2.5
1.2.4
1.2.3
1.2.2
1.2.1
1.2.0
1.1.15
1.1.14
1.1.13
1.1.12
1.1.11
1.1.10
1.1.9
1.1.7
1.1.6
1.1.5
1.1.4
1.1.3
1.1.2
1.1.1
1.1.0
1.0.3
1.0.2
1.0.1
1.0.0
0.1.8
0.1.7
0.1.6
0.1.5
0.1.4
0.1.3
0.1.2
0.1.1
0.1.0
0.0.7
0.0.6
0.0.5
0.0.4
0.0.3
0.0.2
Labels
Clear labels   
bug

   
data loss

   
design finalized

   
good first issue

   
new feature area

   
question / support

   
security

   
waiting for response
No Label
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#84
No description provided.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?