Shown deduplicated archive sizes very big #1034
Labels
No labels
blocked
breaking
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
No milestone
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
borgmatic-collective/borgmatic#1034
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What I'm trying to do and why
Since I switched the database running on my server to postgresql the stats show significantly bigger deduplicated sizes for the created archives in the logs. (I don't use borgmatic's database hook but dump the db manually to a file before backup.)
I recently figured out that if I check archive sizes with "borgmatic info", it shows significantly smaller (like 2-4x) deduplicated archive sizes than shown in the respective logs.
Steps to reproduce
I run borgmatic with the --verbosity 1 switch to let it output some information.
The destilled borg command looks like this: /usr/bin/borg create --patterns-from /tmp/borgmatic-kvb5fqfu/borgmatic/tmp7bje2tct --exclude-caches --compression auto,zstd,3 --lock-wait 30 --stats --show-rc --info --stats 'ssh://uxxxxxx-sub1@uxxxxxx.your-storagebox.de/./wolke_repo::{hostname}-{now:%Y-%m-%d_%H:%M}'
Actual behavior
All normal, but significantly bigger archive deduplicated sizes. Other shown sizes match.
(by the way: without the "--verbosity 1/--info switch" the also used switches --stats and --show-rc seem to have no effect. Is that expected behaviour?)
Expected behavior
Show the same deduplicated archive sizes after backup like with "borgmatic info".
Other notes / implementation ideas
Don't know if this is really a borgmatic issue. Seems more like a borg thingy but wanted to check back anyway.
borgmatic version
latest
borgmatic installation method
pipx
Borg version
1.4.0
Python version
No response
Database version (if applicable)
16.8
Operating system and version
Debian 12
Shown deduplicated archive sizeto Shown deduplicated archive sizes very bigborgmatic doesn't filter the
borg infooutput archive sizes, so if you're seeing a particular archive size number there, it's coming directly from Borg. Same thing forborg create --statsoutput. But if you'd like to debug this further, can I get a look at the output for bothborgmatic infoandborgmatic create --stats? Feel free to redact as necessary.I'm not sure I understand the situation here. borgmatic doesn't have a
--show-rcflag, but Borg does. And borgmatic only passes--show-rcto Borg when borgmatic's--verbosity 2is set. As forcreate --stats(which I assume you're referring to here), I get stats output regardless of the borgmatic verbosity level. Are you seeing otherwise? If so, can I see your command and configuration?I am backing up to 2 different locations so there is a "common.yaml" with the general parameters for the backup and one config file for every location with the specific parameters. Please find the common.yaml attached. As you can see there I am passing the --stats and --show-rc as "extra-borg-options". This seems to loose effect if borgmatic is called without --info switch (please see output below...).
Example for output of borg create --stats (called with borgmatic with --info switch):
Example for output of borg info (called with borgmatic) for same archive afterwards:
And this is the output if invoked without "--verbosity 1/--info switch" and with borg info as a post backup hook (like in common.yaml):
I only realized this behaviour because I removed the --verbosity 1 switch when invoking borgmatic (in my backup script) because it is the default anyway, right? So, maybe it could be a good idea than to set the --stats switch for borgmatic directly instead as a borg option? But anyway strange that the "extra-borg-options" only seem to work if the borgmatic --verbosity switch is set...
I'm not sure what's going on with the stats discrepancies you're seeing. What I'd recommend doing is seeing if you can reproduce it without borgmatic. You can do that by looking at the exact Borg commands borgmatic is running (via
--verbosity 2) and then maybe adjusting for the excludes, since the temporary patterns files won't be available if copy-pasting those commands. If you can reproduce the stats discrepancy without borgmatic, then I think you have a case for filing a ticket upstream with the Borg project.As for the other issues you're experiencing, can I get a look at your
extra_borg_optionsconfiguration? It's possible there's a conflict or interaction between those options and the flags that borgmatic passes to Borg on its own. In general, I don't recommend putting any flags inextra_borg_optionsthat borgmatic also sets. You may be interested in #790 and #793 if you're looking for a way to put some of these output-related options into the configuration file rather than having to specify them as flags on the command-line.Ok, I will give that a try.
The
extra_borg_optionsare--statsand--show-rcfor create and prune as you can see in the common.yaml I uploaded. So, yes I could easily set the--statsflag as a borgmatic flag. But for--show-rcthere is no borgmatic flag as you already mentioned. But therefore it should not hurt anyway, right?Thanks for pointing me to the upcoming new action hooks. I will have a look at that.
And also a big THANK YOU anyway for your very swift support and effort. Much appreciated!
If you want Borg to receive the
--show-rcflag (and you don't mind passing flags to borgmatic's command-line and getting high verbosity output), all you need to do is useborgmatic --verbosity 2. You're right that it shouldn't hurt anything to use theextra_borg_optionsapproach instead, but it does leave you responsible for making sure it's set individually for the right actions and also ensuring it interacts with the verbosity level as desired.