Borg warning 107 - FileNotFound #1248

Closed
opened 2026-01-23 09:48:24 +00:00 by kimheino · 13 comments
Contributor

What I'm trying to do and why

Commit bdd2258701 treats Borg's 107 (FileNotFound) as an error. For me this happens daily on large servers, especially on email servers:

  • backup starts, file X is there
  • file X is deleted by user (email read/deleted/moved) while backup is running
  • borg gives 107, X is missing, which is totally fine and normal

I know that I can configure this per server, but there are lot of servers to configure...

I kindly ask to reconsider if 107 truly is an error. For me it isn't. Feel free to close this issue if you still think it is.

Steps to reproduce

No response

Actual behavior

No response

Expected behavior

No response

Other notes / implementation ideas

No response

borgmatic version

2.1.0

borgmatic installation method

No response

Borg version

1.4.3

Python version

No response

Database version (if applicable)

No response

Operating system and version

Foobar Linux 10 (Centos stream 10)

### What I'm trying to do and why Commit bdd2258701 treats Borg's 107 (FileNotFound) as an error. For me this happens daily on large servers, especially on email servers: - backup starts, file X is there - file X is deleted by user (email read/deleted/moved) while backup is running - borg gives 107, X is missing, which is totally fine and normal I know that I can configure this per server, but there are lot of servers to configure... I kindly ask to reconsider if 107 truly is an error. For me it isn't. Feel free to close this issue if you still think it is. ### Steps to reproduce _No response_ ### Actual behavior _No response_ ### Expected behavior _No response_ ### Other notes / implementation ideas _No response_ ### borgmatic version 2.1.0 ### borgmatic installation method _No response_ ### Borg version 1.4.3 ### Python version _No response_ ### Database version (if applicable) _No response_ ### Operating system and version Foobar Linux 10 (Centos stream 10)
Owner

Thanks for filing this. This is indeed a change due to #1092 in borgmatic 2.1.0. For the vanishing file in question, am I correct in understanding that it's not an explicitly listed file in source_directories, patterns, etc., but instead it's a nested file contained somewhere within your source directories?

If so, then yeah, it does seem like it shouldn't be an error condition in your situation.

When implementing #1092, I had hoped to deprecate source_directories_must_exist in favor of elevating all Borg exit code 107 "backup file not found" warnings to errors. But if it's indeed triggering for random nested files that vanish during backup, then maybe a better way to implement #1092 (and also solve your problem) would be to un-deprecate source_directories_must_exist, make it enabled by default, and put Borg's exit code 107 back to a warning.

Let me know your thoughts.

Thanks for filing this. This is indeed a change due to #1092 in borgmatic 2.1.0. For the vanishing file in question, am I correct in understanding that it's not an explicitly listed file in `source_directories`, `patterns`, etc., but instead it's a nested file contained somewhere within your source directories? If so, then yeah, it does seem like it shouldn't be an error condition in your situation. When implementing #1092, I had hoped to deprecate `source_directories_must_exist` in favor of elevating all Borg exit code 107 "backup file not found" warnings to errors. But if it's indeed triggering for random nested files that vanish during backup, then maybe a better way to implement #1092 (and also solve your problem) would be to un-deprecate `source_directories_must_exist`, make it enabled by default, and put Borg's exit code 107 back to a warning. Let me know your thoughts.
Author
Contributor

Yes, that random vanishing file is in nested sub directory, not directly listed anywhere. I have following config on all my servers:

source_directories:
    - /

patterns_from:
    - /etc/borgmatic/foobar-filter.patterns

...and then I have lot of directories (like /usr, /tmp, /home/*/.cache etc) excluded in that patterns file.

I have now run 2.1.0 for two days on ~40 servers and got 3 errors. That is my usual test bench before deploying anything for larger set of servers.

Yes, that random vanishing file is in nested sub directory, not directly listed anywhere. I have following config on all my servers: ``` source_directories: - / patterns_from: - /etc/borgmatic/foobar-filter.patterns ``` ...and then I have lot of directories (like /usr, /tmp, /home/*/.cache etc) excluded in that patterns file. I have now run 2.1.0 for two days on ~40 servers and got 3 errors. That is my usual test bench before deploying anything for larger set of servers.

Hello,

I wonder how this borg behaviour could be useful...
It is clear: the currwnt result ist Not ok.
But still: reverting ist also Not a good alternative.
Why dies Borgmatic need to rely on Borg. Can Borgmatic Not Check If the folders specified in the config (and nothing else) exist?

Hello, I wonder how this borg behaviour could be useful... It is clear: the currwnt result ist Not ok. But still: reverting ist also Not a good alternative. Why dies Borgmatic need to rely on Borg. Can Borgmatic Not Check If the folders specified in the config (and nothing else) exist?
Owner

@kimheino wrote:

Yes, that random vanishing file is in nested sub directory, not directly listed anywhere. I have following config on all my servers:

source_directories:
    - /

patterns_from:
    - /etc/borgmatic/foobar-filter.patterns

...and then I have lot of directories (like /usr, /tmp, /home/*/.cache etc) excluded in that patterns file.

I have now run 2.1.0 for two days on ~40 servers and got 3 errors. That is my usual test bench before deploying anything for larger set of servers.

Makes sense.. Thanks for explaining. It sounds like source_directories_must_exist being enabled by default shouldn't be a problem in your case—because / does exist.

@henfri wrote:

I wonder how this borg behaviour could be useful...
It is clear: the currwnt result ist Not ok.
But still: reverting ist also Not a good alternative.
Why dies Borgmatic need to rely on Borg. Can Borgmatic Not Check If the folders specified in the config (and nothing else) exist?

That's exactly what the existing source_directories_must_exist option does; it's just disabled by default, and I'm proposing making the default enabled.

@kimheino wrote: > Yes, that random vanishing file is in nested sub directory, not directly listed anywhere. I have following config on all my servers: > > ``` > source_directories: > - / > > patterns_from: > - /etc/borgmatic/foobar-filter.patterns > ``` > > ...and then I have lot of directories (like /usr, /tmp, /home/*/.cache etc) excluded in that patterns file. > > I have now run 2.1.0 for two days on ~40 servers and got 3 errors. That is my usual test bench before deploying anything for larger set of servers. Makes sense.. Thanks for explaining. It sounds like `source_directories_must_exist` being enabled by default shouldn't be a problem in your case—because `/` does exist. @henfri wrote: > I wonder how this borg behaviour could be useful... > It is clear: the currwnt result ist Not ok. > But still: reverting ist also Not a good alternative. > Why dies Borgmatic need to rely on Borg. Can Borgmatic Not Check If the folders specified in the config (and nothing else) exist? That's exactly what the existing `source_directories_must_exist` option does; it's just disabled by default, and I'm proposing making the default enabled.

kimheino: Thank you for sharing your use case.

I would prefer it that when Borg/Borgmatic fails to backup a file, it always result in an error (by default, can be overridden using config) . However, files disappearing/changing during the backup is something that happens, and is much more likely to happen when backing up mail folders.

I believe Windows uses Shadow Copy to prevent this, and I think one could theoretically script a pre-action for Borgmatic to create a snapshot using existing filesystem tools, circumventing the issue.
How does a tool like rsync handle this situation? Does it also error?

Are there any other possible causes that can have Borg return error 107 "X is missing" besides 'a user modified/deleted a file'? (e.g.: a defective harddisk drive / unreadable sector; failed network connection to a NFS server that is being backed up; Fuse error; locale/UTF-8/UNICODE filename issues)? Because if we would change the default for error 107 to not raise an error, we might fail to catch those other causes.

kimheino: Thank you for sharing your use case. I would prefer it that when Borg/Borgmatic fails to backup a file, it always result in an error (by default, can be overridden using config) . However, files disappearing/changing during the backup is something that happens, and is much more likely to happen when backing up mail folders. I believe Windows uses Shadow Copy to prevent this, and I think one could theoretically script a pre-action for Borgmatic to create a snapshot using existing filesystem tools, circumventing the issue. How does a tool like rsync handle this situation? Does it also error? Are there any other possible causes that can have Borg return error 107 "X is missing" besides 'a user modified/deleted a file'? (e.g.: a defective harddisk drive / unreadable sector; failed network connection to a NFS server that is being backed up; Fuse error; locale/UTF-8/UNICODE filename issues)? Because if we would change the default for error 107 to not raise an error, we might fail to catch those other causes.
Author
Contributor

I would prefer it that when Borg/Borgmatic fails to backup a file, it always result in an error (by default, can be overridden using config) .

Borg defines 107 as a warning, not as error. Borgmatic should honor that.

https://borgbackup.readthedocs.io/en/stable/usage/general.html#return-codes

How does a tool like rsync handle this situation? Does it also error?

I used rsync-based backup system (made by me) before Borg(matic). In similar situation it also gives a non-zero rc code that you must accept as OK:

    # Check result. Following errors are OK.
    # 23: Partial transfer due to error
    # 24: Partial transfer due to vanished source files
    if ret in (23, 24):
        ret = 0

Are there any other possible causes that can have Borg return error 107 "X is missing" besides 'a user modified/deleted a file'?

Modified is return code 100, also a warning, not an error.

Here's some real examples I found from my logs:

/home/b/.config/google-chrome/BrowserMetrics/BrowserMetrics-69689A63-A79.pma: file changed while we backed it up
=> 100

/home/b/Documents/share: file type or inode changed while we backed it up (race condition, skipped file)
=> 103

/home/b/.claws-mail/clawsrc.tmp: stat: [Errno 2] No such file or directory: 'clawsrc.tmp'
=> 107

All of those are warnings only.

> I would prefer it that when Borg/Borgmatic fails to backup a file, it always result in an error (by default, can be overridden using config) . Borg defines 107 as a warning, not as error. Borgmatic should honor that. https://borgbackup.readthedocs.io/en/stable/usage/general.html#return-codes > How does a tool like rsync handle this situation? Does it also error? I used rsync-based backup system (made by me) before Borg(matic). In similar situation it also gives a non-zero rc code that you must accept as OK: ``` # Check result. Following errors are OK. # 23: Partial transfer due to error # 24: Partial transfer due to vanished source files if ret in (23, 24): ret = 0 ``` > Are there any other possible causes that can have Borg return error 107 "X is missing" besides 'a user modified/deleted a file'? Modified is return code 100, also a warning, not an error. Here's some real examples I found from my logs: ``` /home/b/.config/google-chrome/BrowserMetrics/BrowserMetrics-69689A63-A79.pma: file changed while we backed it up => 100 /home/b/Documents/share: file type or inode changed while we backed it up (race condition, skipped file) => 103 /home/b/.claws-mail/clawsrc.tmp: stat: [Errno 2] No such file or directory: 'clawsrc.tmp' => 107 ``` All of those are warnings only.
Owner

@sgiebels wrote:

kimheino: Thank you for sharing your use case.

I would prefer it that when Borg/Borgmatic fails to backup a file, it always result in an error (by default, can be overridden using config) . However, files disappearing/changing during the backup is something that happens, and is much more likely to happen when backing up mail folders.

I believe Windows uses Shadow Copy to prevent this, and I think one could theoretically script a pre-action for Borgmatic to create a snapshot using existing filesystem tools, circumventing the issue.
How does a tool like rsync handle this situation? Does it also error?

The filesystem snapshot idea is something that should "solve" @kimheino's use case, but I don't know if that's a reasonable solution on their particular systems.

Are there any other possible causes that can have Borg return error 107 "X is missing" besides 'a user modified/deleted a file'? (e.g.: a defective harddisk drive / unreadable sector; failed network connection to a NFS server that is being backed up; Fuse error; locale/UTF-8/UNICODE filename issues)? Because if we would change the default for error 107 to not raise an error, we might fail to catch those other causes.

I don't know the answer to that for sure, but looking at the Borg source code, exit code 107 maps directly from errno.ENOENT (No such file or directory). From the errno man page: "Typically, this error results when a specified pathname does not exist, or one of the components in the directory prefix of a pathname does not exist, or the specified pathname is a dangling symbolic link." So I think it wouldn't occur for an unreadable sector (that might trigger code 106 instead) but would trigger if, say, you had an unmounted NFS mount and tried to point Borg at a missing subdirectory inside it.

@sgiebels, what do you think of the solution of leaving exit code 107 as just a warning by default but then enabling the source_directories_must_exist option by default?

@kimheino wrote:

I would prefer it that when Borg/Borgmatic fails to backup a file, it always result in an error (by default, can be overridden using config) .

Borg defines 107 as a warning, not as error. Borgmatic should honor that.

The whole discussion in #1092 was around that point; you may want to read the comments there. I actually started with that same viewpoint, but I think folks in those comments convinced me that borgmatic could/should have its own defaults that depart from Borg's.

https://borgbackup.readthedocs.io/en/stable/usage/general.html#return-codes

How does a tool like rsync handle this situation? Does it also error?

I used rsync-based backup system (made by me) before Borg(matic). In similar situation it also gives a non-zero rc code that you must accept as OK:

    # Check result. Following errors are OK.
    # 23: Partial transfer due to error
    # 24: Partial transfer due to vanished source files
    if ret in (23, 24):
        ret = 0

Are there any other possible causes that can have Borg return error 107 "X is missing" besides 'a user modified/deleted a file'?

Modified is return code 100, also a warning, not an error.

And indeed in borgmatic 2.1.0, exit code 100 is treated as a warning by default. Same with 103.

Here's some real examples I found from my logs:

/home/b/.config/google-chrome/BrowserMetrics/BrowserMetrics-69689A63-A79.pma: file changed while we backed it up
=> 100

/home/b/Documents/share: file type or inode changed while we backed it up (race condition, skipped file)
=> 103

/home/b/.claws-mail/clawsrc.tmp: stat: [Errno 2] No such file or directory: 'clawsrc.tmp'
=> 107

All of those are warnings only.

@sgiebels wrote: > kimheino: Thank you for sharing your use case. > > I would prefer it that when Borg/Borgmatic fails to backup a file, it always result in an error (by default, can be overridden using config) . However, files disappearing/changing during the backup is something that happens, and is much more likely to happen when backing up mail folders. > > I believe Windows uses Shadow Copy to prevent this, and I think one could theoretically script a pre-action for Borgmatic to create a snapshot using existing filesystem tools, circumventing the issue. > How does a tool like rsync handle this situation? Does it also error? The [filesystem snapshot](https://torsion.org/borgmatic/reference/configuration/data-sources/) idea *is* something that should "solve" @kimheino's use case, but I don't know if that's a reasonable solution on their particular systems. > Are there any other possible causes that can have Borg return error 107 "X is missing" besides 'a user modified/deleted a file'? (e.g.: a defective harddisk drive / unreadable sector; failed network connection to a NFS server that is being backed up; Fuse error; locale/UTF-8/UNICODE filename issues)? Because if we would change the default for error 107 to not raise an error, we might fail to catch those other causes. I don't know the answer to that for sure, but looking at the Borg source code, exit code 107 maps directly from `errno.ENOENT` (No such file or directory). From the [errno man page](https://www.man7.org/linux/man-pages/man3/errno.3.html): "Typically, this error results when a specified pathname does not exist, or one of the components in the directory prefix of a pathname does not exist, or the specified pathname is a dangling symbolic link." So I think it wouldn't occur for an unreadable sector (that might trigger code 106 instead) but would trigger if, say, you had an unmounted NFS mount and tried to point Borg at a missing subdirectory inside it. @sgiebels, what do you think of the solution of leaving exit code 107 as just a warning by default but then enabling the `source_directories_must_exist` option by default? @kimheino wrote: > > I would prefer it that when Borg/Borgmatic fails to backup a file, it always result in an error (by default, can be overridden using config) . > > Borg defines 107 as a warning, not as error. Borgmatic should honor that. The whole discussion in #1092 was around that point; you may want to read the comments there. I actually started with that same viewpoint, but I think folks in those comments convinced me that borgmatic could/should have its own defaults that depart from Borg's. > https://borgbackup.readthedocs.io/en/stable/usage/general.html#return-codes > > > How does a tool like rsync handle this situation? Does it also error? > > I used rsync-based backup system (made by me) before Borg(matic). In similar situation it also gives a non-zero rc code that you must accept as OK: > > ``` > # Check result. Following errors are OK. > # 23: Partial transfer due to error > # 24: Partial transfer due to vanished source files > if ret in (23, 24): > ret = 0 > ``` > > > Are there any other possible causes that can have Borg return error 107 "X is missing" besides 'a user modified/deleted a file'? > > Modified is return code 100, also a warning, not an error. And indeed in borgmatic 2.1.0, exit code 100 is treated as a warning by default. Same with 103. > Here's some real examples I found from my logs: > > ``` > /home/b/.config/google-chrome/BrowserMetrics/BrowserMetrics-69689A63-A79.pma: file changed while we backed it up > => 100 > > /home/b/Documents/share: file type or inode changed while we backed it up (race condition, skipped file) > => 103 > > /home/b/.claws-mail/clawsrc.tmp: stat: [Errno 2] No such file or directory: 'clawsrc.tmp' > => 107 > ``` > > All of those are warnings only.
Author
Contributor

The filesystem snapshot idea is something that should "solve" @kimheino's use case, but I don't know if that's a reasonable solution on their particular systems.

Yes, it would. But adding 107 to borg_exit_codes sounds much easier to do than snapshots. Most of my filesystems are XFS without LVM, but not all. And that's not "universal" solution for everybody: NFS, VFAT, EXT4 (=Raspberry Pi), etc doesn't support snapshots.

@sgiebels, what do you think of the solution of leaving exit code 107 as just a warning by default but then enabling the source_directories_must_exist option by default?

After reading #1092 that sounds to be the correct solution. But my opinion might be biased.

> The [filesystem snapshot](https://torsion.org/borgmatic/reference/configuration/data-sources/) idea *is* something that should "solve" @kimheino's use case, but I don't know if that's a reasonable solution on their particular systems. Yes, it would. But adding 107 to `borg_exit_codes` sounds much easier to do than snapshots. Most of my filesystems are XFS without LVM, but not all. And that's not "universal" solution for everybody: NFS, VFAT, EXT4 (=Raspberry Pi), etc doesn't support snapshots. > @sgiebels, what do you think of the solution of leaving exit code 107 as just a warning by default but then enabling the `source_directories_must_exist` option by default? After reading #1092 that sounds to be the correct solution. But my opinion might be biased.

Yes, I remember that we Had convincing Arguments... I do Not remember them thought..I will try to Check tomorrow

Yes, I remember that we Had convincing Arguments... I do Not remember them thought..I will try to Check tomorrow

Hi all,

We just updated to 2.1.1 and we are also encountering this issue. We are now getting lots of emails from our crons that the backup has warnings - while previously this was not the case.

The most common warning we see now is "file changed while we backed it up" - which is very logical, as logfiles for example are contineously changed.

I found the customization page ( https://torsion.org/borgmatic/how-to/customize-warnings-and-errors/ ) but I dont see an option to ignore this.
Ideally, I would do something like this, I think..?

borg_exit_codes:
    - code: 100
      treat_as: success

I have attempted adding >/dev/null to the cron but it seems warnings are also logged to stderr, and we dont want to hide other errors. Our current solution is changing the cron to (something like) this, which works, but is of course not pretty:

borgmatic create 2>&1 | grep -v ': file changed while we backed it up'

EDIT: We found a different way by running borgmatic --verbosity -1 - this seems to do the trick for now. Still, a way to selectively ignore only specific warnings would be great.

My endgoal is that when running borgmatic at verbositoy 0, I do not see any stdout/stderr output, unless something is wrong that needs my attention. And files that change during the backup creation is something acceptable for me.
What is the best way to achieve this?

Thanks!

Hi all, We just updated to 2.1.1 and we are also encountering this issue. We are now getting lots of emails from our crons that the backup has warnings - while previously this was not the case. The most common warning we see now is "file changed while we backed it up" - which is very logical, as logfiles for example are contineously changed. I found the customization page ( https://torsion.org/borgmatic/how-to/customize-warnings-and-errors/ ) but I dont see an option to ignore this. Ideally, I would do something like this, I think..? ``` borg_exit_codes: - code: 100 treat_as: success ``` I have attempted adding `>/dev/null` to the cron but it seems warnings are also logged to stderr, and we dont want to hide other errors. Our current solution is changing the cron to (something like) this, which works, but is of course not pretty: ``` borgmatic create 2>&1 | grep -v ': file changed while we backed it up' ``` EDIT: We found a different way by running `borgmatic --verbosity -1` - this seems to do the trick for now. Still, a way to selectively ignore only specific warnings would be great. My endgoal is that when running borgmatic at verbositoy 0, I do not see any stdout/stderr output, unless something is wrong that needs my attention. And files that change during the backup creation is something acceptable for me. What is the best way to achieve this? Thanks!
Owner

@NielsH That sounds like a different but related issue. "File changed while we backed it up" was a warning by default in borgmatic < 2.1.0 and it's still a warning by default in borgmatic 2.1.0+. The difference with 2.1.0+ is that anything Borg puts on stderr (which includes warnings) now stays on stderr.

I recommend filing a separate ticket for this, and we can discuss further. Thank you!

@NielsH That sounds like a different but related issue. "File changed while we backed it up" was a warning by default in borgmatic < 2.1.0 and it's still a warning by default in borgmatic 2.1.0+. The difference with 2.1.0+ is that anything Borg puts on stderr (which includes warnings) now stays on stderr. I recommend filing a separate ticket for this, and we can discuss further. Thank you!
Owner

Okay, this is implemented in main and will be part of the next release. Here are the changes:

  • Go back to treating Borg "file not found" warnings (exit code 107) as warnings instead of errors. Otherwise, borgmatic can error on files that a user intentionally deletes while a backup is running. You can still override this behavior with the "borg_exit_codes" option. See the documentation for more information: https://torsion.org/borgmatic/how-to/customize-warnings-and-errors/
  • Un-deprecate the "source_directories_must_exist" option and default it to true, to compensate for Borg "file not found" warnings no longer being treated as errors.

Thanks again for all the feedback here.

Okay, this is implemented in main and will be part of the next release. Here are the changes: * Go back to treating Borg "file not found" warnings (exit code 107) as warnings instead of errors. Otherwise, borgmatic can error on files that a user intentionally deletes while a backup is running. You can still override this behavior with the "borg_exit_codes" option. See the documentation for more information: https://torsion.org/borgmatic/how-to/customize-warnings-and-errors/ * Un-deprecate the "source_directories_must_exist" option and default it to true, to compensate for Borg "file not found" warnings no longer being treated as errors. Thanks again for all the feedback here.
Owner

Released in borgmatic 2.1.3!

Released in borgmatic 2.1.3!
Sign in to join this conversation.
No milestone
No project
No assignees
5 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
borgmatic-collective/borgmatic#1248
No description provided.