crash after succeding when running under systemd with 'NoneType' object has no attribute 'read' #1341
Labels
No labels
blocked
breaking
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
No milestone
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
borgmatic-collective/borgmatic#1341
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What I'm trying to do and why
I'm trying to create a new backup, running borgmatic with the systemd unit provided by the arch linux packaged version of borgmatic.
Steps to reproduce
systemctl start borgmaticSee
borgmatic.service.txtattachment for the systemd unit file andconfig.yamlfor the/etc/borgmatic/config.yamlActual behavior
Borgmatic crashes after succeeding
See
borgmatic.logfor the --verbose 2 log outputExpected behavior
Borgmatic to not crash and move on to the next repo
Other notes / implementation ideas
No response
borgmatic version
2.1.6
borgmatic installation method
pacman -S borgmatic
Borg version
borg 1.4.5
Python version
Python 3.14.6
Database version (if applicable)
No response
Operating system and version
NAME="Arch Linux" PRETTY_NAME="Arch Linux" ID=arch BUILD_ID=rolling ANSI_COLOR="38;2;23;147;209" HOME_URL="https://archlinux.org/" DOCUMENTATION_URL="https://wiki.archlinux.org/" SUPPORT_URL="https://bbs.archlinux.org/" BUG_REPORT_URL="https://gitlab.archlinux.org/groups/archlinux/-/issues" PRIVACY_POLICY_URL="https://terms.archlinux.org/docs/privacy-policy/" LOGO=archlinux-logo
Thanks for filing this. I'm not sure why it occurs, and I don't have a repro, but I can envision a simple "fix" that would suppress the error. However, the error is actually happening within an error handler, so it probably means something else is going wrong and borgmatic isn't succeeding. If you like, I can provide you with a modified source file for you to temporarily install to suppress the error.. That should hopefully show you what else might be going wrong.
Okay, I've attached a modified copy of borgmatic's
execute.pywhere the error is occurring. It's based on borgmatic 2.1.6, and it should suppress the error you're getting while hopefully revealing the underlying error. If you feel comfortable, you can copy the file over theexecute.pyfrom the Arch package, I'm guessing it's at/usr/lib/python3.14/site-packages/borgmatic/execute.pyor similar. Then try your repro and see if you get a different error in the logs. Thanks!Okay, I've replaced the execute.py module and ran it again. Here are my findings:
First it now just hangs. This is the last 3 lines of the log
My second finding is related (I think) to another issue I also wanted to file but hadn't gotten around to yet. The code that handles database dumps has some holes when it comes to cleaning up postgresql dumps. When I noticed that it was hanging I ran
sudo systemctl status borgmatic.serviceand saw thisafter running that I also ran
ps aux | grep 'docker exec'and found even more hanged processessome of them 8 hours old. You can see I've already tried (and failed) to solve this by wrapping the dump commands in a
timeout 1hcall to no avail.Hope this helps! understand the issue. I'm open to trying more patches to debug this!
If I do
sudo kill $(ps aux | grep 'docker exec' | awk '{print $2}')it stops hanging and I get this error message.which seems to be the real error. If I run this command manually it does give me the same error that borgmatic gets. But if I first do
sudo mkdir /tmp/borgmatic/mariadb_databases/localhost/then the command succeedsIt's as if borgmatic is not creating the directories needed for extracting the databases?
if I run borgmatic from the command line instead (
sudo borgmatic create --verbosity 2) everything goes by smoothly. Directories inside/tmp/borgmaticare created, databases are dumped,docker execprocesses are cleaned up.If I had to guess this is a mix of two problems:
Interesting that you're getting hangs now (and apparently before as well!). I'd recommend first killing (or
docker rm --forceing) those lingering processes/containers in case they're interfering. Then, a few ideas:user_runtime_directoryoption and trying again. If you have multiple database processes reading from and writing to the same path (/tmp/./borgmatic/postgresql_databases/localhost/tandoorfor instance), that's a recipe for hangs. borgmatic normally generates a randomly named sub-directory for its runtime directory to avoid these sorts of collisions, but it looks like manually settinguser_runtime_directorylike you're doing may override that behavior. (This might be an area for improvement or at minimum better documentation.)source_directoriesand doing a backup of just your databases. If that still hangs, then that's a different class of problem than if the hang goes away withsource_directoriescommented out. (Some fun reading on that general topic.)On this point, you can try commenting out all the security-related systemd service options (and doing a
daemon-reload) to see if that changes the behavior. FYI that's everything fromLockPersonality=truethroughCapabilityBoundingSet=.... You can even do a binary search of commented out options if you want to find the problem child.Have you made any progress here?
Sorry, couldn't dedicate time to this until today 😅.
I think one of my issues was solved by fixing #1352 so that's nice! I'll keep monitoring my backups to see if that fixed all my issues, but by the description of that issue it sounds like it was exactly the same problem.
On the topic of the systemd unit file being too restrictive. As of 2.1.9, I have to override just
PrivateTmpto benobecause I have:This is because of my mariadb dump uses the dump command that's inside the db container
this way I avoid having to put my db user and password in the borgmatic config file, as documented.
That results in this command:
crucially
--result-filemust be a path that both the container can write to and borgmatic can read from. Hence what I've done is set theuser_runtime_directoryto/tmp/borgmaticand then mount this directory in the container/tmp/borgmatic:/tmp/borgmatic. If there is a better way to do this, please let me know.