Borg warning 107 - FileNotFound #1248
Labels
No labels
blocked
breaking
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
No milestone
No project
No assignees
5 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
borgmatic-collective/borgmatic#1248
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What I'm trying to do and why
Commit
bdd2258701treats Borg's 107 (FileNotFound) as an error. For me this happens daily on large servers, especially on email servers:I know that I can configure this per server, but there are lot of servers to configure...
I kindly ask to reconsider if 107 truly is an error. For me it isn't. Feel free to close this issue if you still think it is.
Steps to reproduce
No response
Actual behavior
No response
Expected behavior
No response
Other notes / implementation ideas
No response
borgmatic version
2.1.0
borgmatic installation method
No response
Borg version
1.4.3
Python version
No response
Database version (if applicable)
No response
Operating system and version
Foobar Linux 10 (Centos stream 10)
Thanks for filing this. This is indeed a change due to #1092 in borgmatic 2.1.0. For the vanishing file in question, am I correct in understanding that it's not an explicitly listed file in
source_directories,patterns, etc., but instead it's a nested file contained somewhere within your source directories?If so, then yeah, it does seem like it shouldn't be an error condition in your situation.
When implementing #1092, I had hoped to deprecate
source_directories_must_existin favor of elevating all Borg exit code 107 "backup file not found" warnings to errors. But if it's indeed triggering for random nested files that vanish during backup, then maybe a better way to implement #1092 (and also solve your problem) would be to un-deprecatesource_directories_must_exist, make it enabled by default, and put Borg's exit code 107 back to a warning.Let me know your thoughts.
Yes, that random vanishing file is in nested sub directory, not directly listed anywhere. I have following config on all my servers:
...and then I have lot of directories (like /usr, /tmp, /home/*/.cache etc) excluded in that patterns file.
I have now run 2.1.0 for two days on ~40 servers and got 3 errors. That is my usual test bench before deploying anything for larger set of servers.
Hello,
I wonder how this borg behaviour could be useful...
It is clear: the currwnt result ist Not ok.
But still: reverting ist also Not a good alternative.
Why dies Borgmatic need to rely on Borg. Can Borgmatic Not Check If the folders specified in the config (and nothing else) exist?
@kimheino wrote:
Makes sense.. Thanks for explaining. It sounds like
source_directories_must_existbeing enabled by default shouldn't be a problem in your case—because/does exist.@henfri wrote:
That's exactly what the existing
source_directories_must_existoption does; it's just disabled by default, and I'm proposing making the default enabled.kimheino: Thank you for sharing your use case.
I would prefer it that when Borg/Borgmatic fails to backup a file, it always result in an error (by default, can be overridden using config) . However, files disappearing/changing during the backup is something that happens, and is much more likely to happen when backing up mail folders.
I believe Windows uses Shadow Copy to prevent this, and I think one could theoretically script a pre-action for Borgmatic to create a snapshot using existing filesystem tools, circumventing the issue.
How does a tool like rsync handle this situation? Does it also error?
Are there any other possible causes that can have Borg return error 107 "X is missing" besides 'a user modified/deleted a file'? (e.g.: a defective harddisk drive / unreadable sector; failed network connection to a NFS server that is being backed up; Fuse error; locale/UTF-8/UNICODE filename issues)? Because if we would change the default for error 107 to not raise an error, we might fail to catch those other causes.
Borg defines 107 as a warning, not as error. Borgmatic should honor that.
https://borgbackup.readthedocs.io/en/stable/usage/general.html#return-codes
I used rsync-based backup system (made by me) before Borg(matic). In similar situation it also gives a non-zero rc code that you must accept as OK:
Modified is return code 100, also a warning, not an error.
Here's some real examples I found from my logs:
All of those are warnings only.
@sgiebels wrote:
The filesystem snapshot idea is something that should "solve" @kimheino's use case, but I don't know if that's a reasonable solution on their particular systems.
I don't know the answer to that for sure, but looking at the Borg source code, exit code 107 maps directly from
errno.ENOENT(No such file or directory). From the errno man page: "Typically, this error results when a specified pathname does not exist, or one of the components in the directory prefix of a pathname does not exist, or the specified pathname is a dangling symbolic link." So I think it wouldn't occur for an unreadable sector (that might trigger code 106 instead) but would trigger if, say, you had an unmounted NFS mount and tried to point Borg at a missing subdirectory inside it.@sgiebels, what do you think of the solution of leaving exit code 107 as just a warning by default but then enabling the
source_directories_must_existoption by default?@kimheino wrote:
The whole discussion in #1092 was around that point; you may want to read the comments there. I actually started with that same viewpoint, but I think folks in those comments convinced me that borgmatic could/should have its own defaults that depart from Borg's.
And indeed in borgmatic 2.1.0, exit code 100 is treated as a warning by default. Same with 103.
Yes, it would. But adding 107 to
borg_exit_codessounds much easier to do than snapshots. Most of my filesystems are XFS without LVM, but not all. And that's not "universal" solution for everybody: NFS, VFAT, EXT4 (=Raspberry Pi), etc doesn't support snapshots.After reading #1092 that sounds to be the correct solution. But my opinion might be biased.
Yes, I remember that we Had convincing Arguments... I do Not remember them thought..I will try to Check tomorrow
Hi all,
We just updated to 2.1.1 and we are also encountering this issue. We are now getting lots of emails from our crons that the backup has warnings - while previously this was not the case.
The most common warning we see now is "file changed while we backed it up" - which is very logical, as logfiles for example are contineously changed.
I found the customization page ( https://torsion.org/borgmatic/how-to/customize-warnings-and-errors/ ) but I dont see an option to ignore this.
Ideally, I would do something like this, I think..?
I have attempted adding
>/dev/nullto the cron but it seems warnings are also logged to stderr, and we dont want to hide other errors. Our current solution is changing the cron to (something like) this, which works, but is of course not pretty:EDIT: We found a different way by running
borgmatic --verbosity -1- this seems to do the trick for now. Still, a way to selectively ignore only specific warnings would be great.My endgoal is that when running borgmatic at verbositoy 0, I do not see any stdout/stderr output, unless something is wrong that needs my attention. And files that change during the backup creation is something acceptable for me.
What is the best way to achieve this?
Thanks!
@NielsH That sounds like a different but related issue. "File changed while we backed it up" was a warning by default in borgmatic < 2.1.0 and it's still a warning by default in borgmatic 2.1.0+. The difference with 2.1.0+ is that anything Borg puts on stderr (which includes warnings) now stays on stderr.
I recommend filing a separate ticket for this, and we can discuss further. Thank you!
Okay, this is implemented in main and will be part of the next release. Here are the changes:
Thanks again for all the feedback here.
Released in borgmatic 2.1.3!