[Borgmatic 2.0.8] The runtime directory overlaps with the configured excludes or patterns with excludes. #1150
Labels
No labels
blocked
breaking
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
No milestone
No assignees
5 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
borgmatic-collective/borgmatic#1150
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What I'm trying to do and why
Hi,
I've just update from Borgmatic 2.0.7 to 2.0.8 via pip, and now half the time I've got a warning reporting that runtime directory is not excluded.
Everything was working fine before this, and after a rollback on version 2.0.7 everything is working fine again.
Regards.
Steps to reproduce
Actual behavior
Here is my healthchecks mail, as you can see the backup run successfully but I've got a warning message.
The runtime directory /tmp/borgmatic-rkdj4oc4/borgmatic overlaps with the configured excludes or patterns with excludes. Please ensure the runtime directory is not excluded.Expected behavior
No response
Other notes / implementation ideas
No response
borgmatic version
2.0.8
borgmatic installation method
pip
Borg version
1.4.4
Python version
3.12.3
Database version (if applicable)
No response
Operating system and version
Ubuntu 24.04.3 LTS
The runtime directory overlaps with the configured excludes or patterns with excludes.to [Borgmatic 2.0.8] The runtime directory overlaps with the configured excludes or patterns with excludes.Thanks for filing this one! The error you're getting really shouldn't happen if you're not excluding
/tmp(which you're not), so I'm not sure what's going on here. Would it be possible for you to runborgmatic createwith--verbosity 2and post the portion of your logs containing the patterns that borgmatic is passing to Borg? Here's an example of what that section might look like in your logs:If I could get a look at that whole span of logs, that'd be helpful. Thanks!
Evening @witten ,
Thanks for the quick reply.
Please find below the full output (fully sanitized I hope) of the following command
sudo borgmatic create --verbosity 2 -c ~/borgmatic/hourly-borgbase.ymlThanks for including your logs. It looks like in that particular borgmatic run, you're not getting the error about the runtime directory overlapping with excludes. So is that error intermittent? Do you only get it, for instance, through Healthchecks but not at the command-line? In any case, would it be possible to include the logs from a run where you do get that error?
I've add a log and the --verbosity 2 to my cron, I will head back to you will all the informations on the next run.
Hi,
Here is the full log of another job working hourly but not to the same remote.
And here is the relevant log on the same remote as described on the main comment.
Another information that may have a significant impact, how I'm declaring the cron.
@hourly /usr/local/bin/borgmatic -c /etc/borgmatic.d/hourly-borgbase.yml --verbosity 2 --log-file /tmp/hourly-borgbase.log > /dev/null 2>&1I feel bad about repeatedly asking for logs, but it looks like those most recent logs aren't at the "debug" level and so don't include any log entries about the patterns borgmatic is passing to Borg. So ideally here's what I'm looking for: A complete log file for a borgmatic run that includes both those debug-level patterns (
Writing patterns to ...) and the error about the runtime directory being excluded. That way I can see if the patterns are excluding the runtime directory and if so how/why that's happening.Assuming that what you're posting is the contents of
borgbase.log, I think you should be able to get debug logs by either: 1. changing your cron entry to include--log-file-verbosity 2or by addinglog_file_verbosity: 2to yourhourly-borgbase.ymlconfiguration file.Don't feel bad at all.
I have add the
--log-file-verbosity 2param to my cron.Hi,
Here are another two logs about two different jobs, I hope full, but I'm a bit doubtful.
Meanwhile I've add
log_file_verbosity: 2on all my YAML to see if I can grab more logs.Thanks, those logs are what I was looking for. Unfortunately they don't indicate anything obvious going on with the excludes that would cause the error that you're getting...
/tmpis definitely not getting excluded and yet you're still getting the error about the runtime directory being excluded. So something else is going on here, and I'll need to dig into the code. If it comes to it, I may need to give you an instrumented version of a particular source file with additional logging to help diagnose the issue.One thing you could try in the meantime though is commenting out
one_file_system: trueto see if that changes the behavior. It probably won't, but that option affects the pattern deduplication code which is relevant here.FYI if you add
--log-file-verbosity 2to the command-line you don't also need to addlog_file_verbosity: 2to your YAML configuration; they do the same thing.Okay, I have an instrumented source file (attached) with tons of added debug logging. If you're comfortable installing it and sharing the log output, it should provide more insight into what's going on. Note that it's logging things like config and patterns, so feel free to redact as necessary. The only changes are to add logging.
The file to replace is
borgmatic/borg/create.py, and depending on your installation, it may be in something like/usr/lib/python3.12/site-packages/borgmaticor even somewhere in~/.local. And make sure it'sborg/create.pyyou're replacing, notactions/create.py.Thanks for your patience here.
Evening.
There you go.
File replace:
Differences:
I'll see if I can reproduce the error by launching the job myself instead of waiting for the hourly cron.
I have finally manage to generate a log sanitized (I hope..), and clean a bit because the original was showing every files backup, and the log was about 500 MB.
Wow, thanks for doing the work to sanitize that. For the end of the log you included, did you also truncate the file list outputted after these lines?
I realize you truncated the first time this occurred (from a prior borgmatic run), but I'm asking about this second time in particular.
If you did truncate, does
/tmp/borgmatic-heazo4ky/borgmatic/bootstrapappear in the paths that you cut out? And are there lots of other paths like ones starting with/etc,/home,/root, etc?Yeah I've truncated the file list because it was so massive.
But I've paid attention to avoid truncated any occurence of the temporary directory
borgmatic-heazo4kyin the entire log I've provided :-)But yes there are lots of other paths like
/etc,/home,/root, etc, the whole files where appearing.Okay, so if the temporary directory isn't in the
paths_outputfor that run, then the next place to look at would be what exactly borgmatic is passing to Borg. Which I did... I put all of your exact patterns from the log above into the function that borgmatic calls to produce the patterns file it passes to Borg, and the temporary directory was indeed part of that patterns file:So if that's representative of what's happening on your machine, then Borg is somehow dropping or ignoring that pattern. Some ideas of things to try (separately) if you're game:
cat /tmp/borgmatic-*/borgmatic/tmp*as the root user. That will show you whether the temporary runtime directory is indeed included in that patterns file passed to Borg.one_file_systemoption and try again. Note however that this could cause additional files to get backed up if Borg does cross filesystems.source_directoriesto see if that makes the error go away completely./src/dockerfromsource_directories.Evening,
Thanks for all the hard work for the debuging, I will look into the several solutions you have been provided ASAP.
As you noticed the problem is a bit occurring intermittently but I will be able to do some test and try to find out I hope a solution.
First thing first.
For the temporary patterns here is the result just after having launched a backup.
I just haved remove the
one_file_systemoption from two backup site, let's see tomorrow how things ends up :-)The other two options you gave me will be try after.
Good nite.
Thanks.. So that definitely confirms the behavior that I was seeing—that borgmatic is passing the temporary runtime directory to Borg as expected.
I still got errors even when removing
one_file_systemoption.So now I will test the third and fourth option you gave me.
I have just removed all the
exclude_patternsfrom one backup.I still got the error even after removing
exclude_patternsandone_file_systemat the same time.Let's try to decrease files backup and see.
It seems that the situation is more stable after excluding
/srv/dockerwhich is different from/var/lib/docker.I can't exclude this directory because it's too important, I will try to exclude some patterns in it.
More stable, as in you never get the error anymore once
/srv/dockeris excluded? Or do you sometimes get the error still, just less frequently? Or maybe you don't know yet because you don't have that many data points due to how long backups take to run...?I completely understand about you not being able to exclude the directory from your regular backups. It interesting though that in this testing you're doing,
/srv/dockerbeing excluded appears to impact whether you're getting errors.Here's one other random idea: On a previous archive created without excluding
/srv/docker, try runningborgmatic infowith--archive thatarchiveand look specifically at theNumber of files:andUtilization of max. archive size:value. My thinking is maybe you're running up against Borg archive size or file count limits that could be impacting whether the temporary directory is getting included since it's listed last in the patterns to include. EDIT: I'm seeing indications that Borg might explicitly error when you reach these archive metadata limits, so maybe this idea isn't relevant. It still might be an interesting data point though.I mean no backup in error in a 9 hours window, that means a lot.
I probably should split my backup more cleverly I guess.
And for answering your question about
Number of files:andUtilization of maximum supported archive size::Gotcha. Yeah, that's a significant difference.
Maybe. But then again Borg/borgmatic should be able to deal with an archive of this size.
Okay, so it looks like you're nowhere near running into archive metadata limits. Scratch that idea.
New theory: borgmatic is passing the right patterns (including the temporary directory) to the Borg dry-run, and Borg is producing the right file paths output (including the temporary directory) to be consumed by borgmatic—but then something is going wrong with borgmatic consuming that output such that it's not seeing the temporary directory. Maybe due to a weirdly named file in
/srv/docker.It's it possible you have filenames containing newlines in
/srv/docker? That might interfere with borgmatic consuming Borg's dry-run output. You could check by:find /srv/docker -name *$'\n'* -printAlso, you may not know the answer to this, but when you truncated the
paths_outputvalues from those logs above, did it look like all of your/srv/dockerpaths were present, or maybe were some missing from the end? For instance, did the output end in the middle of/src/dockerpaths where you might not expect it to end? Did those paths end at a different point during successful non-error runs?The following does not return a thing
find /srv/docker -name *$'\n'* -printso no newlines.I will try to debug a bit the
paths_outputbut it contains a whole bunch of files, I guess I can only compare first and last files backup.I know what you are heading to, try to find out if there is a glitch between a successful and a unsuccessful backup, and if this glitch is introduced by anything else.
Okay, then the newline thing isn't it. Thanks for checking.
Yeah, exactly. It sounds like the borgmatic temporary directory being present in the last files of
paths_outputis one notable difference between a successful and unsuccessful backup. But I'm wondering if there are other relevant differences in those last files as well.Hi,
I have rollback to version 2.0.7 in order to be able to split all my backup quietly.
But I just got the same error:
The runtime directory /tmp/borgmatic-55vjgkuj/borgmatic overlaps with the configured excludes or patterns with excludes. Please ensure the runtime directory is not excluded.So I suppose it is clearly not related with the 2.0.8 version (I was a bit confuse about that).
FWIW, I also got this error with borgmatic 2.0.8 and LVM. I backed up to version 2.0.7 and it hasn't recurred. I didn't do a full diagnosis, but if it's helpful to track this down I can try.
@jpoggi wrote:
Interesting! So with either 2.0.7 or 2.0.8, were you able to look at the end of the logged
paths_outputfiles and see if it looks like there's a difference between when there's an error and when there isn't? Like, does it appear to stop partway through the/srv/dockerfiles only when there's an error?@andrew-schulman wrote:
Thanks for weighing in with what you're finding on your machine. Yeah, help tracking this down would be super helpful since I don't have a repro here. Some questions / things to look at:
source_directories(and/or patterns) andexclude_patterns?borgmatic createwith--verbosity 2, what shows up in the logs betweenWriting patterns to ...andChecking file paths Borg plans to include?one_file_system: true? Does the behavior change if you disable it?Thanks!
Hello, I am a homebrew maintainer and recently we also have some error for the runtime path things.
There is no defined pattern for tests but we got error as follows.
@daeho-ro Thanks for jumping in here. Similar questions to the above:
source_directories(and/or patterns)... Is that correct? And do you have anything inexclude_patternsor similar?borgmatic createwith--verbosity 2, what shows up in the logs betweenWriting patterns to ...andChecking file paths Borg plans to include?one_file_system: true? Does the behavior change if you disable it?This happen from 2.0.8 with python 3.13 on macOS, it's our test step and so there is no
exclude_patterns. I will check the others when I have some time.This is
config.yml, all other options were commented.I had the same error and found that in my case it was caused by enabling JSON logging.
Error:
The runtime directory /run/user/1000/borgmatic overlaps with the configured excludes or patterns with excludes. Please ensure the runtime directory is not excluded.I looked through the repo and found that when borgmatic does the dryrun to get the planned backup paths, it expects the normal non-JSON log output (
borg/create.py,validate_planned_backup_paths()):Then it filters out any line that doesn't start with a dash.
Adding
log_json: truein the borgmatic config causes the --log-json option to be passed to that dryrun as well, changing the format and causing all files get filtered out:I'm not familiar with the repo, but I'd guess the safest and easiest option would be to ignore the --log-json option when doing the dryrun.
This issue wasn't random for me, like it was for @jpoggi, so unfortunately I'm not sure if this is going to be the same cause.
Cheers.
@daeho-ro wrote:
Thanks for providing your config! Can you also provide answers to the other questions above when you get a chance? That will help me diagnose the issue as it manifests on your machine.
Nice detective work here! Confirmed locally.
Yes, and in fact it wouldn't be the first option that gets filtered out in this manner.
Probably not unfortunately.
In any case I'll make this fix but then leave this ticket open for others to weigh in with their own details.
Okay, that particular
--log-jsonissue is fixed in main and will be part of the next release.I also have this problem, in borgmatic 2.0.8 and 2.0.9 running under systemd.
Here's my config:
And the log file:
Note that the warning about "No LVM logical volumes found to snapshot" is a separate issue, which I'll report.
I commented on the separate issue (#1163), but I think this is happening for you because of the
- **pattern that is telling Borg to exclude everything, including the runtime directory! Exactly the sort of case the error message is supposed to catch.Understood. Here is vm.patterns (sorry I didn't include it before):
I think this is a valid config. It says, in /archive and /home, back up all *.vdi files, and ignore everything else.
Now borgmatic comes along later and adds some backup roots of its own, so we get:
and the new roots collide with the exclusion.
If there's no way for borgmatic to work around that, then I can work around it by adding more inclusions in vm.patterns:
Not pretty, but I tested it this morning and it did fix the error. If that's the solution, I think it should be documented.
Ugh, I see what you're saying here. My first inclination is that maybe if borgmatic prepends its own patterns to the pattern list instead of appending them, that would sidestep this particular problem.
Yes. I think that if borgmatic prepends
That should fix this case anyway, and be safe.
Good idea. This is implemented in main now and will be part of the next release. The change applies to all data source hooks, because they were all at risk of having their pattern additions getting excluded if there happened to be a matching user-configured exclude.
To be clear, while this fix will likely address the issue that @andrew-schulman encountered, it probably won't help with @daeho-ro's or @jpoggi's issues. For those, I'll need additional information to help diagnose. Thanks!
The two changes mentioned above are now released in borgmatic 2.0.10! Feedback welcome.
Confirmed that the problem with global excludes is fixed in version 2.0.10. Thanks!
Awesome, thanks!
Related: These changes in #1150 so far have caused a potential regression in a separate use case. See #1169 for details.
sm8ps referenced this issue2025-10-25 18:25:33 +00:00
Given the lack of activity here, the length of the discussion, and the fact that several of the mentioned issues have been addressed, I'm closing this ticket for now. If some folks are still having problems, I'd recommend filing separate follow-up tickets. Thank you!
Yep. WFM now. Thanks.