pg_dump: [Errno 28] No space left on device #580
Labels
No Label
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
No Milestone
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: borgmatic-collective/borgmatic#580
Loading…
Reference in New Issue
No description provided.
Delete Branch "%!s(<nil>)"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What I'm trying to do and why
I do have > 1 matrix servers up and running, deployed with https://github.com/spantaleev/matrix-docker-ansible-deploy
Included: borgmatic
Steps to reproduce (if a bug)
run the backup... (the one with the biggest database, but still not THAT big - postgres folder has a size of 52 GB
Actual behavior (if a bug)
when doing the pg_dump, backup stops after aprox. 30 minutes with.
Expected behavior (if a bug)
finish backup - as with other (smaller), too.
Other notes / implementation ideas
Source and target volumes do have plenty of room left.
Environment
borgmatic version: should be latest, since I update very often.
Use
sudo borgmatic --versionorsudo pip show borgmatic | grep ^VersionSorry - don't know how to, since these commands do not work on my machine.
borgmatic installation method: Container
Borg version: ?
Python version: ?
Database version (if applicable): PostgreSQL 14.4 on x86_64-pc-linux-musl, compiled by gcc (Alpine 11.2.1_git20220219) 11.2.1 20220219, 64-bit
operating system and version: Debian GNU/Linux 11 (bullseye)
Looking at the way that
matrix-docker-ansible-deployinstalls borgmatic, it's from a borgmatic Docker container project I haven't heard of before. I'm not sure exactly what version it's using, but the container was last built 5 months ago using Alpine latest, so my guess is it's running borgmatic 1.5.x or so. You can likely verify bydocker execing into the running borgmatic container and then runningborgmatic --version.Anyway, the container is constructed without any volumes for Borg or borgmatic cache files, which means that any of those files that Borg/borgmatic writes to the container's filesystem just get stored in RAM instead of ever hitting disk. My guess is that your RAM is filling up as Borg writes its cache files (borgmatic doesn't cache much), and therefore the borgmatic process is dying with a "disk full" error.
Now that's just a guess, but I think it's likely. You can check the hypothesis by watching the container's memory usage as borgmatic runs. E.g.:
sudo docker stats. If it's consuming tons of RAM, it might be because of the lack of cache volumes.My recommendation to solve this? Convince the container's maintainer to add formal support for cache volumes, map in your own cache volumes (
~/.cache/borgmight do it?), or use a different container. Here's the borgmatic Docker container I use on my own systems: https://hub.docker.com/r/b3vis/borgmatic/. It does include cache volume support. I realize it may not be a drop-in replacement within thematrix-docker-ansible-deployproject.I hope some of this helps! Let me know what you end up doing.
Thank you
Sound's reasonable.
Since I'm still in learnig even to read ansible and docker, I might just deactivate the "deafault" and install the b3vis/borgmatic container, mentioned by you.
Could this playbook be an simple(r) alternitive?
https://github.com/borgbase/ansible-role-borgbackup
Obviously, best would be, to fix the playbook somehow.
Thx a lot for your input!
Yes, I'd recommend that Ansible role if you'd like to use Ansible + borgmatic. It doesn't use Docker (except for tests), so the Borg cache should use your native disk. Not that there's anything wrong with using Docker; it just needs to be set up properly.