Borg gets stuck under systemd when doing database backups #1118
Labels
No labels
blocked
breaking
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
No milestone
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
borgmatic-collective/borgmatic#1118
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What I'm trying to do and why
I updated to Ubuntu 24.04 LTS (from 22.04 LTS) and updated borgmatic to 2.0.7 (can't tell you exactly from which version, but I'd say 1.8.10/1.8.9 as an educated guess)
After that my configuration broke. I'm using database backups and borgmatic complained that the path these were created at were excluded from the backup. After fixing that I ran into this issue. running borgmatic as root user manually works fine, but as soon as I run it as systemd service borg hangs.
Steps to reproduce
Borgmatic config:
SystemD unit config (security lockdown disabled for debugging)
Actual behavior
Attaching GDB to borg shows its stuck at an openat64 syscall with the following path:
/run/borgmatic/postgresql_databases/service1-db/allThe database dump processes meanwhile died and are just zombie processes (they execute before it gets stuck so it looks like borg tries to do these backups twice)
log excerpt:
Expected behavior
It does not get stuck and works as expected
Other notes / implementation ideas
No response
borgmatic version
2.0.7
borgmatic installation method
apt-get install borgbackup && pipx install borgmatic
Borg version
1.2.8
Python version
Python 3.12.3
Database version (if applicable)
server: postgres:16 (docker container) 16.2-1.pgdg120+2 | client: psql (PostgreSQL) 16.9 (Ubuntu 16.9-1.pgdg24.04+1)
Operating system and version
Ubuntu 24.04.2 LTS
Thanks for the detailed ticket and debugging. Based on your description though, I'm not immediately sure what might be causing this. Here are some initial thoughts:
Do you have any indication about how/why the dump processes died? Anything in borgmatic's logs or PostgreSQL logs? It does make sense to me that Borg would hang on reading the database pipe if nothing is writing to it, but I don't know why the dump processes might've died—and only when running under systemd. Or did they maybe just finish executing successfully, but Borg is still expecting them to be writing output?
When you say that Borg tries to do these backups twice, what do you mean by that? Do you see for instance
/run/borgmatic/postgresql_databases/service1-db/alltwice in Borg's file list? And what about when borgmatic prints out its own list of patterns? (See the lines that come right after the log lineWriting patterns to ...)One thing you could try is commenting out the following lines from your configured
patterns:They shouldn't cause problems, and if they did I'd expect them to also cause problems when running borgmatic manually. But I'm wondering if maybe there's an interaction with borgmatic's runtime directory (where database pipes are located). And that runtime directory can differ when running borgmatic under systemd.
Speaking of which, can you show me the output of borgmatic's log line
Using runtime directory ..., both when running borgmatic manually and when running it under systemd? Thanks!They execute successfully. The parent process just does not close its fd's so they stay zombie processes temporarily.
A /run/borgmatic/postgresql_databases/service1-db/allis only written once in the output log. It probably does not try to backup it twice, but tries to access the file twice, once to backup and once after the backup for whatever reason (and that part gets stuck)Maybe this helps:
/bin/sh -c pg_dumpall .... > /run/./borgmatic/postgresql_databases/service{1,2}/allnot doing anything yetA /run/borgmatic/postgresql_databases/service1-db/allpg_dumpallis executed)/bin/sh -c pg_dumpall ... > ...process exits (is now a zombie). This also happens running under root user.x /proclog line)/run/borgmatic/postgresql_databases/service1-db/allagain for some reason and gets stuck there. This is the point of the GDB stacktrace I providedSadly this does not help
These are the patterns written by borgmatic
SystemD:
Using runtime directory /run/borgmaticroot user:
Using runtime directory /tmp/borgmatic-lnuavy8n/borgmaticI just set up a fresh debian 12 VM with postgresql, borgbackup & borgmatic with my config. there it even hangs when running under the root user, but now the runtime directory for the root user is
/run/user/0/borgmatic. I can provide you with the VM disk image if you wantI played with the patterns a bit. Seems like this only happens when the pattern list includes one of these:
R /R /runif I switch to a whitelist approach by explicitly listing /etc, /home, /root, /usr and /var it works fine again
My theory:
Since borgmatic appends this to my patterns:
the
/run/./borgmatic/postgresql_databasesdirectory is scanned twice, which causes a hang on the second scan.When I patch
borgmatic/hooks/data_source/postgresql.pyfunctiondump_data_sourcesand remove this partborg no longer hangs
I think your theory is likely right.
And this explains the difference between running manually and running under systemd—since borgmatic appends
/run/borgmaticonly under systemd, that's when it duplicates the existing+ run/borgmaticpattern and causes the hang. And with your VM scenario, my guess is that/run/user/0/borgmaticis getting duplicated byR run.As for a solution, first let me see if I can get a repro case setup. I'll let you know if I can't and whether a VM image would be helpful. But assuming that I can repro it, I'll look into why the existing deduplication code (
deduplicate_patterns()inborgmatic/actions/pattern.py)—which is designed to prevent this sort of problem—isn't successfully deduplicating the overlapping paths.I think I have a repro—and also an idea about what's going on. The existing deduplication logic only deduplicates root patterns (patterns prefixed with
R). And because you've got implicit duplication across one non-root pattern (+ run/borgmatic) and one root pattern (/run/./borgmatic/postgresql_databases), no deduplication occurs, both patterns get passed to Borg, Borg tries to read each named pipe twice, and a hang occurs.Now, the question is what to do about this!
EDIT: Looks like this initial assessment might be wrong... Still looking into this.
Okay, it looks like that initial assessment was wrong. What's actually happening is that two root patterns are not getting de-duped:
R /andR /run/./borgmatic/postgresql_databases... even though/logically contains/run/./borgmatic/postgresql_databasesand therefore only one of them should get passed to Borg (to prevent hangs).And the reason they're not getting de-duped, at least on my system, is that they're on different filesystem devices.
/is on my standard root filesystem (ext4 here), while/run/...is ontmpfs. I imagine your system is similar. Anyway, there's code indeduplicate_patterns()that intentionally skips de-duping for patterns on different filesystems. That's because theone_file_systemoption might betrue, in which case that would tell Borg not to cross filesystems implicitly—and therefore both patterns would still need to be included in order for Borg to back them up.But
one_file_systemisn't set totruein your case (or in my repro), and so perhaps this code should actually look at the value of that option to determine whether a pattern's filesystem device impacts its deduplication!Phew!
Yes this also explains why it works fine running as root user on the ubuntu system I originally had the issue on.
/tmpis not an extra mount so its part of the root file system, while/runis a tmpfs.Makes sense! So this should be fixed in main now and will be part of the next release. Thanks again for all the details here.
Released in borgmatic 2.0.8!