crash after succeding when running under systemd with 'NoneType' object has no attribute 'read' #1341

Open
opened 2026-08-06 22:37:25 +00:00 by mendess · 8 comments

What I'm trying to do and why

I'm trying to create a new backup, running borgmatic with the systemd unit provided by the arch linux packaged version of borgmatic.

Steps to reproduce

systemctl start borgmatic

See borgmatic.service.txt attachment for the systemd unit file and config.yaml for the /etc/borgmatic/config.yaml

Actual behavior

Borgmatic crashes after succeeding

See borgmatic.log for the --verbose 2 log output

Expected behavior

Borgmatic to not crash and move on to the next repo

Other notes / implementation ideas

No response

borgmatic version

2.1.6

borgmatic installation method

pacman -S borgmatic

Borg version

borg 1.4.5

Python version

Python 3.14.6

Database version (if applicable)

No response

Operating system and version

NAME="Arch Linux" PRETTY_NAME="Arch Linux" ID=arch BUILD_ID=rolling ANSI_COLOR="38;2;23;147;209" HOME_URL="https://archlinux.org/" DOCUMENTATION_URL="https://wiki.archlinux.org/" SUPPORT_URL="https://bbs.archlinux.org/" BUG_REPORT_URL="https://gitlab.archlinux.org/groups/archlinux/-/issues" PRIVACY_POLICY_URL="https://terms.archlinux.org/docs/privacy-policy/" LOGO=archlinux-logo

### What I'm trying to do and why I'm trying to create a new backup, running borgmatic with the systemd unit provided by the arch linux packaged version of borgmatic. ### Steps to reproduce `systemctl start borgmatic` See `borgmatic.service.txt` attachment for the systemd unit file and `config.yaml` for the `/etc/borgmatic/config.yaml` ### Actual behavior Borgmatic crashes after succeeding See `borgmatic.log` for the --verbose 2 log output ### Expected behavior Borgmatic to not crash and move on to the next repo ### Other notes / implementation ideas _No response_ ### borgmatic version 2.1.6 ### borgmatic installation method pacman -S borgmatic ### Borg version borg 1.4.5 ### Python version Python 3.14.6 ### Database version (if applicable) _No response_ ### Operating system and version NAME="Arch Linux" PRETTY_NAME="Arch Linux" ID=arch BUILD_ID=rolling ANSI_COLOR="38;2;23;147;209" HOME_URL="https://archlinux.org/" DOCUMENTATION_URL="https://wiki.archlinux.org/" SUPPORT_URL="https://bbs.archlinux.org/" BUG_REPORT_URL="https://gitlab.archlinux.org/groups/archlinux/-/issues" PRIVACY_POLICY_URL="https://terms.archlinux.org/docs/privacy-policy/" LOGO=archlinux-logo
Owner

Thanks for filing this. I'm not sure why it occurs, and I don't have a repro, but I can envision a simple "fix" that would suppress the error. However, the error is actually happening within an error handler, so it probably means something else is going wrong and borgmatic isn't succeeding. If you like, I can provide you with a modified source file for you to temporarily install to suppress the error.. That should hopefully show you what else might be going wrong.

Thanks for filing this. I'm not sure why it occurs, and I don't have a repro, but I can envision a simple "fix" that would suppress the error. However, the error is actually happening *within* an error handler, so it probably means something else is going wrong and borgmatic isn't succeeding. If you like, I can provide you with a modified source file for you to temporarily install to suppress the error.. That should hopefully show you what else might be going wrong.
Owner

Okay, I've attached a modified copy of borgmatic's execute.py where the error is occurring. It's based on borgmatic 2.1.6, and it should suppress the error you're getting while hopefully revealing the underlying error. If you feel comfortable, you can copy the file over the execute.py from the Arch package, I'm guessing it's at /usr/lib/python3.14/site-packages/borgmatic/execute.py or similar. Then try your repro and see if you get a different error in the logs. Thanks!

Okay, I've attached a modified copy of borgmatic's `execute.py` where the error is occurring. It's based on borgmatic 2.1.6, and it should suppress the error you're getting while hopefully revealing the underlying error. If you feel comfortable, you can copy the file over the `execute.py` from the Arch package, I'm guessing it's at `/usr/lib/python3.14/site-packages/borgmatic/execute.py` or similar. Then try your repro and see if you get a different error in the logs. Thanks!
Author

Okay, I've replaced the execute.py module and ran it again. Here are my findings:

First it now just hangs. This is the last 3 lines of the log

Aug 07 01:44:01 pendrellvale borgmatic[1741546]: Checking file paths Borg plans to include
Aug 07 01:44:01 pendrellvale borgmatic[1741546]: BORG_RSH=*** BORG_PASSPHRASE_FD=*** BORG_EXIT_CODES=*** borg create --patterns-from /tmp/borgmatic/tmp0j2gc_di --compression zstd,6 --read-special /backups/self/repo::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}.checkpoint --dry-run --list
Aug 07 01:44:11 pendrellvale borgmatic[1741546]: BORG_RSH=*** BORG_PASSPHRASE_FD=*** BORG_EXIT_CODES=*** borg create --patterns-from /tmp/borgmatic/tmp0j2gc_di --compression zstd,6 --read-special --stats --debug --show-rc --progress /backups/self/repo::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}.checkpoint > <object object at 0x7fa36b928d00>
Aug 07 01:44:11 pendrellvale borgmatic[1741546]: [139B blob data]

My second finding is related (I think) to another issue I also wanted to file but hadn't gotten around to yet. The code that handles database dumps has some holes when it comes to cleaning up postgresql dumps. When I noticed that it was hanging I ran sudo systemctl status borgmatic.service and saw this

     CGroup: /system.slice/borgmatic.service
             ├─1741537 systemd-inhibit --who=borgmatic --what=sleep:shutdown "--why=Prevent interrupting scheduled backup" /usr/bin/borgmatic --verbosity -2 --syslog-verbosity 2
             ├─1741546 /usr/bin/python /usr/bin/borgmatic --verbosity -2 --syslog-verbosity 2
             ├─1741586 /bin/sh -c "timeout 1h docker exec --user 1000 immich_postgres pg_dump -U postgres --no-password --clean --if-exists --format custom immich > /tmp/./borgmatic/postgresql_databases/localhost/immich"
             ├─1741587 /bin/sh -c "timeout 1h docker exec --user 1000 db pg_dump -U postgres --no-password --clean --if-exists --format custom blind_eternities > /tmp/./borgmatic/postgresql_databases/localhost/blind_eternities"
             └─1741588 /bin/sh -c "timeout 1h docker exec --user 1000 tandoor_db pg_dump -U postgres --no-password --clean --if-exists --format custom tandoor > /tmp/./borgmatic/postgresql_databases/localhost/tandoor"

after running that I also ran ps aux | grep 'docker exec' and found even more hanged processes

root     1402509  0.0  0.0   8200  1636 ?        S    Aug06   0:00 /bin/sh -c timeout 1h docker exec --user 1000 db pg_dump -U postgres --no-password --clean --if-exists --host 127.0.0.1 --format custom blind_eternities > /tmp/borgmatic-1zpvlxw9/./borgmatic/postgresql_databases/127.0.0.1/blind_eternities
root     1407886  0.0  0.0   8200  1560 ?        S    Aug06   0:00 /bin/sh -c timeout 1h docker exec --user 1000 db pg_dump -U postgres --no-password --clean --if-exists --host 127.0.0.1 --format custom blind_eternities > /tmp/borgmatic-e2286yha/./borgmatic/postgresql_databases/127.0.0.1/blind_eternities
root     1411256  0.0  0.0   8200  1572 ?        S    Aug06   0:00 /bin/sh -c timeout 1h docker exec --user 1000 db pg_dump -U postgres --no-password --clean --if-exists --host 127.0.0.1 --format custom blind_eternities > /tmp/./borgmatic/postgresql_databases/127.0.0.1/blind_eternities
root     1412815  0.0  0.0   8200  1648 ?        S    Aug06   0:00 /bin/sh -c timeout 1h docker exec --user 1000 db pg_dump -U postgres --no-password --clean --if-exists --host 127.0.0.1 --format custom blind_eternities > /tmp/./borgmatic/postgresql_databases/127.0.0.1/blind_eternities
root     1741586  0.0  0.0   8200  1640 ?        SN   01:44   0:00 /bin/sh -c timeout 1h docker exec --user 1000 immich_postgres pg_dump -U postgres --no-password --clean --if-exists --format custom immich > /tmp/./borgmatic/postgresql_databases/localhost/immich
root     1741587  0.0  0.0   8200  1580 ?        SN   01:44   0:00 /bin/sh -c timeout 1h docker exec --user 1000 db pg_dump -U postgres --no-password --clean --if-exists --format custom blind_eternities > /tmp/./borgmatic/postgresql_databases/localhost/blind_eternities
root     1741588  0.0  0.0   8200  1616 ?        SN   01:44   0:00 /bin/sh -c timeout 1h docker exec --user 1000 tandoor_db pg_dump -U postgres --no-password --clean --if-exists --format custom tandoor > /tmp/./borgmatic/postgresql_databases/localhost/tandoor
mendess  1756123  0.0  0.0   7208  2268 pts/1    S+   02:07   0:00 grep --color=auto docker exec

some of them 8 hours old. You can see I've already tried (and failed) to solve this by wrapping the dump commands in a timeout 1h call to no avail.

Hope this helps! understand the issue. I'm open to trying more patches to debug this!

Okay, I've replaced the execute.py module and ran it again. Here are my findings: First it now just hangs. This is the last 3 lines of the log ``` Aug 07 01:44:01 pendrellvale borgmatic[1741546]: Checking file paths Borg plans to include Aug 07 01:44:01 pendrellvale borgmatic[1741546]: BORG_RSH=*** BORG_PASSPHRASE_FD=*** BORG_EXIT_CODES=*** borg create --patterns-from /tmp/borgmatic/tmp0j2gc_di --compression zstd,6 --read-special /backups/self/repo::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}.checkpoint --dry-run --list Aug 07 01:44:11 pendrellvale borgmatic[1741546]: BORG_RSH=*** BORG_PASSPHRASE_FD=*** BORG_EXIT_CODES=*** borg create --patterns-from /tmp/borgmatic/tmp0j2gc_di --compression zstd,6 --read-special --stats --debug --show-rc --progress /backups/self/repo::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}.checkpoint > <object object at 0x7fa36b928d00> Aug 07 01:44:11 pendrellvale borgmatic[1741546]: [139B blob data] ``` My second finding is related (I think) to another issue I also wanted to file but hadn't gotten around to yet. The code that handles database dumps has some holes when it comes to cleaning up postgresql dumps. When I noticed that it was hanging I ran `sudo systemctl status borgmatic.service` and saw this ``` CGroup: /system.slice/borgmatic.service ├─1741537 systemd-inhibit --who=borgmatic --what=sleep:shutdown "--why=Prevent interrupting scheduled backup" /usr/bin/borgmatic --verbosity -2 --syslog-verbosity 2 ├─1741546 /usr/bin/python /usr/bin/borgmatic --verbosity -2 --syslog-verbosity 2 ├─1741586 /bin/sh -c "timeout 1h docker exec --user 1000 immich_postgres pg_dump -U postgres --no-password --clean --if-exists --format custom immich > /tmp/./borgmatic/postgresql_databases/localhost/immich" ├─1741587 /bin/sh -c "timeout 1h docker exec --user 1000 db pg_dump -U postgres --no-password --clean --if-exists --format custom blind_eternities > /tmp/./borgmatic/postgresql_databases/localhost/blind_eternities" └─1741588 /bin/sh -c "timeout 1h docker exec --user 1000 tandoor_db pg_dump -U postgres --no-password --clean --if-exists --format custom tandoor > /tmp/./borgmatic/postgresql_databases/localhost/tandoor" ``` after running that I also ran `ps aux | grep 'docker exec'` and found even more hanged processes ``` root 1402509 0.0 0.0 8200 1636 ? S Aug06 0:00 /bin/sh -c timeout 1h docker exec --user 1000 db pg_dump -U postgres --no-password --clean --if-exists --host 127.0.0.1 --format custom blind_eternities > /tmp/borgmatic-1zpvlxw9/./borgmatic/postgresql_databases/127.0.0.1/blind_eternities root 1407886 0.0 0.0 8200 1560 ? S Aug06 0:00 /bin/sh -c timeout 1h docker exec --user 1000 db pg_dump -U postgres --no-password --clean --if-exists --host 127.0.0.1 --format custom blind_eternities > /tmp/borgmatic-e2286yha/./borgmatic/postgresql_databases/127.0.0.1/blind_eternities root 1411256 0.0 0.0 8200 1572 ? S Aug06 0:00 /bin/sh -c timeout 1h docker exec --user 1000 db pg_dump -U postgres --no-password --clean --if-exists --host 127.0.0.1 --format custom blind_eternities > /tmp/./borgmatic/postgresql_databases/127.0.0.1/blind_eternities root 1412815 0.0 0.0 8200 1648 ? S Aug06 0:00 /bin/sh -c timeout 1h docker exec --user 1000 db pg_dump -U postgres --no-password --clean --if-exists --host 127.0.0.1 --format custom blind_eternities > /tmp/./borgmatic/postgresql_databases/127.0.0.1/blind_eternities root 1741586 0.0 0.0 8200 1640 ? SN 01:44 0:00 /bin/sh -c timeout 1h docker exec --user 1000 immich_postgres pg_dump -U postgres --no-password --clean --if-exists --format custom immich > /tmp/./borgmatic/postgresql_databases/localhost/immich root 1741587 0.0 0.0 8200 1580 ? SN 01:44 0:00 /bin/sh -c timeout 1h docker exec --user 1000 db pg_dump -U postgres --no-password --clean --if-exists --format custom blind_eternities > /tmp/./borgmatic/postgresql_databases/localhost/blind_eternities root 1741588 0.0 0.0 8200 1616 ? SN 01:44 0:00 /bin/sh -c timeout 1h docker exec --user 1000 tandoor_db pg_dump -U postgres --no-password --clean --if-exists --format custom tandoor > /tmp/./borgmatic/postgresql_databases/localhost/tandoor mendess 1756123 0.0 0.0 7208 2268 pts/1 S+ 02:07 0:00 grep --color=auto docker exec ``` some of them 8 hours old. You can see I've already tried (and failed) to solve this by wrapping the dump commands in a `timeout 1h` call to no avail. Hope this helps! understand the issue. I'm open to trying more patches to debug this!
Author

If I do sudo kill $(ps aux | grep 'docker exec' | awk '{print $2}') it stops hanging and I get this error message.

Aug 07 02:26:35 pendrellvale borgmatic[1767821]: Command 'docker exec nextcloud-mariadb mariadb-dump --defaults-extra-file=/tmp/credentials.cnf --default-character-set=utf8mb4 --add-drop-database --single-transaction --databases --events --routines --all-tablespaces nextcloud --result-file /tmp/./borgmatic/mariadb_databases/localhost/nextcloud' returned non-zero exit status 1.

which seems to be the real error. If I run this command manually it does give me the same error that borgmatic gets. But if I first do sudo mkdir /tmp/borgmatic/mariadb_databases/localhost/ then the command succeeds

It's as if borgmatic is not creating the directories needed for extracting the databases?

if I run borgmatic from the command line instead (sudo borgmatic create --verbosity 2) everything goes by smoothly. Directories inside /tmp/borgmatic are created, databases are dumped, docker exec processes are cleaned up.

If I had to guess this is a mix of two problems:

  • the systemd unit file is too restrictive
  • borgmatic deadlocks waiting for postgresql dump commands when another error occurs, this is a bug I have already diagnosed in the past, because one of my remote hosts was offline, which resulted in the same deadlock waiting for postgresql processes. Maybe I'll get time to propose a patch for this tomorrow
If I do `sudo kill $(ps aux | grep 'docker exec' | awk '{print $2}')` it stops hanging and I get this error message. ``` Aug 07 02:26:35 pendrellvale borgmatic[1767821]: Command 'docker exec nextcloud-mariadb mariadb-dump --defaults-extra-file=/tmp/credentials.cnf --default-character-set=utf8mb4 --add-drop-database --single-transaction --databases --events --routines --all-tablespaces nextcloud --result-file /tmp/./borgmatic/mariadb_databases/localhost/nextcloud' returned non-zero exit status 1. ``` which seems to be the real error. If I run this command manually it does give me the same error that borgmatic gets. But if I first do `sudo mkdir /tmp/borgmatic/mariadb_databases/localhost/` then the command succeeds It's as if borgmatic is not creating the directories needed for extracting the databases? if I run borgmatic from the command line instead (`sudo borgmatic create --verbosity 2`) everything goes by smoothly. Directories inside `/tmp/borgmatic` are created, databases are dumped, `docker exec` processes are cleaned up. If I had to guess this is a mix of two problems: - the systemd unit file is too restrictive - borgmatic deadlocks waiting for postgresql dump commands when another error occurs, this is a bug I have already diagnosed in the past, because one of my remote hosts was offline, which resulted in the same deadlock waiting for postgresql processes. Maybe I'll get time to propose a patch for this tomorrow
Owner

Interesting that you're getting hangs now (and apparently before as well!). I'd recommend first killing (or docker rm --forceing) those lingering processes/containers in case they're interfering. Then, a few ideas:

  1. First, try running borgmatic again after those processes are gone. If that fixes the hang, then that indicates those processes (or more specifically the files they were writing to) were interfering.
  2. Even if that does fix the hang, you may want to consider commenting out your user_runtime_directory option and trying again. If you have multiple database processes reading from and writing to the same path (/tmp/./borgmatic/postgresql_databases/localhost/tandoor for instance), that's a recipe for hangs. borgmatic normally generates a randomly named sub-directory for its runtime directory to avoid these sorts of collisions, but it looks like manually setting user_runtime_directory like you're doing may override that behavior. (This might be an area for improvement or at minimum better documentation.)
  3. If that still doesn't solve anything, then undo your config changes and try commenting out your source_directories and doing a backup of just your databases. If that still hangs, then that's a different class of problem than if the hang goes away with source_directories commented out. (Some fun reading on that general topic.)
Interesting that you're getting hangs now (and apparently before as well!). I'd recommend first killing (or `docker rm --force`ing) those lingering processes/containers in case they're interfering. Then, a few ideas: 1. First, try running borgmatic again after those processes are gone. If that fixes the hang, then that indicates those processes (or more specifically the files they were writing to) were interfering. 2. Even if that does fix the hang, you may want to consider commenting out your `user_runtime_directory` option and trying again. If you have multiple database processes reading from and writing to the same path (`/tmp/./borgmatic/postgresql_databases/localhost/tandoor` for instance), that's a recipe for hangs. borgmatic normally generates a randomly named sub-directory for its runtime directory to avoid these sorts of collisions, but it looks like manually setting `user_runtime_directory` like you're doing may override that behavior. (This might be an area for improvement or at minimum better documentation.) 3. If that still doesn't solve anything, then undo your config changes and try commenting out your `source_directories` and doing a backup of just your databases. If that still hangs, then that's a different class of problem than if the hang goes away with `source_directories` commented out. (Some fun [reading](https://torsion.org/borgmatic/how-to/backup-your-databases/#limitations) on that general topic.)
Owner
  • the systemd unit file is too restrictive

On this point, you can try commenting out all the security-related systemd service options (and doing a daemon-reload) to see if that changes the behavior. FYI that's everything from LockPersonality=true through CapabilityBoundingSet=.... You can even do a binary search of commented out options if you want to find the problem child.

> * the systemd unit file is too restrictive On this point, you can try commenting out all the security-related systemd service options (and doing a `daemon-reload`) to see if that changes the behavior. FYI that's everything from `LockPersonality=true` through `CapabilityBoundingSet=...`. You can even do a binary search of commented out options if you want to find the problem child.
Owner

Have you made any progress here?

Have you made any progress here?
Author

Sorry, couldn't dedicate time to this until today 😅.

I think one of my issues was solved by fixing #1352 so that's nice! I'll keep monitoring my backups to see if that fixed all my issues, but by the description of that issue it sounds like it was exactly the same problem.

On the topic of the systemd unit file being too restrictive. As of 2.1.9, I have to override just PrivateTmp to be no because I have:

user_runtime_directory: /tmp/borgmatic

This is because of my mariadb dump uses the dump command that's inside the db container

mariadb_databases:
  - name: nextcloud
    mariadb_dump_command: docker exec nextcloud-mariadb mariadb-dump
    options: --defaults-extra-file=/tmp/credentials.cnf --default-character-set=utf8mb4

this way I avoid having to put my db user and password in the borgmatic config file, as documented.

That results in this command:

docker exec nextcloud-mariadb mariadb-dump --defaults-extra-file=/tmp/credentials.cnf --default-character-set=utf8mb4 --add-drop-database --single-transaction --events --routines --all-tablespaces --databases nextcloud --result-file /tmp/borgmatic/borgmatic-d9nwwmv_/./borgmatic/mariadb_databases/localhost/nextcloud

crucially --result-file must be a path that both the container can write to and borgmatic can read from. Hence what I've done is set the user_runtime_directory to /tmp/borgmatic and then mount this directory in the container /tmp/borgmatic:/tmp/borgmatic. If there is a better way to do this, please let me know.

Sorry, couldn't dedicate time to this until today 😅. I think one of my issues was solved by fixing https://projects.torsion.org/borgmatic-collective/borgmatic/issues/1352 so that's nice! I'll keep monitoring my backups to see if that fixed all my issues, but by the description of that issue it sounds like it was exactly the same problem. On the topic of the systemd unit file being too restrictive. As of 2.1.9, I have to override just `PrivateTmp` to be `no` because I have: ``` user_runtime_directory: /tmp/borgmatic ``` This is because of my mariadb dump uses the dump command that's inside the db container ``` mariadb_databases: - name: nextcloud mariadb_dump_command: docker exec nextcloud-mariadb mariadb-dump options: --defaults-extra-file=/tmp/credentials.cnf --default-character-set=utf8mb4 ``` this way I avoid having to put my db user and password in the borgmatic config file, [as documented](https://torsion.org/borgmatic/how-to/backup-your-databases/). That results in this command: ``` docker exec nextcloud-mariadb mariadb-dump --defaults-extra-file=/tmp/credentials.cnf --default-character-set=utf8mb4 --add-drop-database --single-transaction --events --routines --all-tablespaces --databases nextcloud --result-file /tmp/borgmatic/borgmatic-d9nwwmv_/./borgmatic/mariadb_databases/localhost/nextcloud ``` crucially `--result-file` must be a path that both the container can write to and borgmatic can read from. Hence what I've done is set the `user_runtime_directory` to `/tmp/borgmatic` and then mount this directory in the container `/tmp/borgmatic:/tmp/borgmatic`. If there is a better way to do this, please let me know.
Sign in to join this conversation.
No milestone
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
borgmatic-collective/borgmatic#1341
No description provided.