Inconsistent treatment of borg exit codes when using "retries" option #1170
Labels
No labels
blocked
breaking
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
No milestone
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
borgmatic-collective/borgmatic#1170
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What I'm trying to do and why
When using the "retries" option in borgmatic, most of the time borg error exit codes are treated as warnings and upgraded to errors only if there is no success after retrying the set number of times. This seems like the correct action to me.
However, occasionally, the exit code is tagged as an error, despite there being more retries. This does not seem consistent to me, but perhaps I am missing something subtle.
Below are two log snippets,1 hour apart, demonstrating this behavior. Note the difference in the 4th line from the bottom.
Steps to reproduce
No response
Actual behavior
Expected behavior
No response
Other notes / implementation ideas
No response
borgmatic version
1.9.14
borgmatic installation method
debian repo
Borg version
1.4.0
Python version
3.13.5
Database version (if applicable)
No response
Operating system and version
debian 13.1 (trixie)
I'm not sure why this behavior would differ from one borgmatic run to another, but a Borg exit code 81 getting tagged as an error log level is the expected behavior. The rationale is that: 1. Borg exiting with code 81 is an error from Borg's perspective, and 2. borgmatic's decision to put the error on hold and work through the retries is only made after that Borg error is logged.
A question for you though: Is the error log level causing issues for you? Or is the problem just that the behavior is inconsistent?
For the latter issue, if possible I'd recommend upgrading borgmatic (ideally to 2.0.10+) to see if that helps. If not, we can diagnose from there.
Thank you!
Ooh, I managed to get a repro of the inconsistent log levels with 2.0.10/main, so you don't need to upgrade to see if that fixes it. (It still might be a good idea to upgrade though because any fix will be on a newer version of borgmatic.)
I'll try to dig into this and see what's going on with the log level.
So here's what appears to be going on... The inconsistent logging is due to a race condition. During the normal course of operation, borgmatic consumes Borg's log output line-by-line and turns it into borgmatic log messages. Any output lines (like
terminating with error status, rc 81) that borgmatic receives before Borg exits get logged at log levelINFO, because we don't yet have an official exit status at that point (even if we have a description of one). But if borgmatic happens to receive that output line after Borg has exited, then we know for sure that an error has occurred—and therefore log that line with log levelERROR. Hence the inconsistent behavior.I'm not yet sure what to do about this. One option is always logging as
INFOfor both of the two cases described above, but that might have the side effect of making Borg errors harder to see in borgmatic output. There is the borgmatic summary logged at the end which elevates Borg log message log levels to error as needed, so maybe that's sufficient.That makes a lot of sense.
I think the decision of how to treat this hangs heavily on the actual issue borg is having. IMO, if it's a"recoverable" error, such as intermittent connection issues, and borgmatic has retries available, the borg error is more of a "warning". If it's an unrecoverable error, then borgmatic should throw an error and halt that config file. That decision would be made easier, if we had all the borg error exit codes defined, but I know that's a work in progress.
Oh well. Thank you for looking into this. Shall we close this?
Yeah, for purposes of retrying, borgmatic doesn't currently draw a distinction between recoverable and unrecoverable Borg errors. Maybe that suggests a potential new feature: A way to define what exit codes should be retried, similar to (or as part of) the separate warning/errors customization feature.
Not quite yet. I'm going to try to at least make the behavior consistent if not 100% correct. 😄
Okay, I switched the
ERRORlog level toINFOfor consistency, so now it shows up at the same log level regardless of whether the log entry comes in before/after Borg exits.It does have the unfortunate side effect of making Borg's exit status log a little harder to see since it's no longer (sometimes)
ERROR, but I rationalize that away by the fact that the summary at the bottom does still mark everything asERROR.This will be part of the next release. Thanks for filing this!
Released in borgmatic 2.0.11!