borgmatic-collective/borgmatic
9
71
Fork 49
Code Issues 55 Pull Requests 6 Actions 1 Packages Releases 157 Activity

sqlite "database is locked" during DB dump was logged but not retried and run completed (no alert) #1223

New Issue
Closed
opened 2026-01-04 15:23:48 +00:00 by aisamu · 10 comments
aisamu commented 2026-01-04 15:23:48 +00:00
Copy Link

What I'm trying to do and why

Hi!

I run borgmatic with database dump hooks (postgres + sqlite) as part of an automated full-system backup.
I discovered a sqlite lock error in the logs but the service finished successfully and healthchecks/uptime notifications indicated success, so I was not alerted.

It'd be nice if errors during dumps were retried (for transients) and/or alerted us!

Steps to reproduce

This is unfortunately not reliably reproducible for me.

I encountered this while inspecting logs after a scheduled run, but running the service manually multiple times didn't show the issue again.

This relevant excerpt from the config:

sqlite_databases:
- name: healthchecks
  path: /var/lib/healthchecks/healthchecks.sqlite
  sqlite_command: /nix/store/awm8pnzpcj63lb26asvjm24j4kwg4d76-sqlite-3.50.4-bin/bin/sqlite3
  sqlite_restore_command: /nix/store/awm8pnzpcj63lb26asvjm24j4kwg4d76-sqlite-3.50.4-bin/bin/sqlite3

Actual behavior

borgmatic logged the sqlite lock error, did not appear to retry, continued processing (archive rename/prune/compact), reported success to healthchecks/uptime, and systemd showed the service as deactivated successfully.

Jan 03 10:39:03 aisamu-nas systemd[1]: Starting borgmatic backup...
Jan 03 10:40:08 aisamu-nas borgmatic[532114]: INFO /etc/borgmatic.d/aisamu-company.yaml: Pinging Healthchecks start
Jan 03 10:40:13 aisamu-nas borgmatic[532114]: INFO /etc/borgmatic.d/aisamu-company.yaml: Pushing Uptime Kuma push_url http://localhost:3001/api/push/YOQXtMP61I?status=up&msg=start
Jan 03 10:40:13 aisamu-nas borgmatic[532114]: INFO aisamu-company-m2: Running before repository command hook
Jan 03 10:40:13 aisamu-nas borgmatic[532114]: INFO aisamu-company-m2: Creating archive
Jan 03 10:40:13 aisamu-nas borgmatic[532114]: INFO aisamu-company-m2: Dumping PostgreSQL databases
Jan 03 10:40:13 aisamu-nas borgmatic[532114]: INFO aisamu-company-m2: Dumping SQLite databases
Jan 03 10:40:14 aisamu-nas borgmatic[532114]: WARNING aisamu-company-m2: Ignoring configured "read_special" value of false, as true is needed for database hooks.
Jan 03 10:40:15 aisamu-nas borgmatic[532114]: INFO Creating archive at "ssh://aisamu@aisamu-company//Users/aisamu/backups/aisamu-nas.borg::aisamu-nas-2026-01-03T10:40:14.806204.checkpoint"
Jan 03 10:40:16 aisamu-nas sudo[532129]:     root : PWD=/ ; USER=postgres ; COMMAND=/nix/store/a067iwfgwaabg6wkxl9a80mqdabaffsi-postgresql-16.11/bin/pg_dump --no-password --clean --if-exists --username postgres --format custom immich
Jan 03 10:40:16 aisamu-nas sudo[532129]: pam_unix(sudo:session): session opened for user postgres(uid=71) by (uid=0)
Jan 03 10:40:38 aisamu-nas sudo[532129]: pam_unix(sudo:session): session closed for user postgres
Jan 03 10:40:39 aisamu-nas borgmatic[532114]: INFO sql error: database is locked (5)
Jan 03 10:40:40 aisamu-nas borgmatic[532114]: INFO aisamu-company-m2: Renaming archive aisamu-nas-2026-01-03T10:40:14.806204.checkpoint -> aisamu-nas-2026-01-03T10:40:14.806204
Jan 03 10:40:41 aisamu-nas borgmatic[532114]: INFO aisamu-company-m2: Pruning archives
Jan 03 10:40:42 aisamu-nas borgmatic[532114]: INFO aisamu-company-m2: Compacting segments
Jan 03 10:40:43 aisamu-nas borgmatic[532114]: INFO Remote: compaction freed about 2.85 kB repository space.
Jan 03 10:40:43 aisamu-nas borgmatic[532114]: INFO aisamu-company-m2: Running consistency checks
Jan 03 10:40:45 aisamu-nas borgmatic[532114]: INFO aisamu-company-m2: Skipping repository check due to configured frequency; 26 days, 7:16:55.465493 until next check (use --force to check anyway)
Jan 03 10:40:45 aisamu-nas borgmatic[532114]: INFO aisamu-company-m2: Skipping archives check due to configured frequency; 26 days, 7:16:55.465144 until next check (use --force to check anyway)
Jan 03 10:40:45 aisamu-nas borgmatic[532114]: INFO /etc/borgmatic.d/aisamu-company.yaml: Pinging Healthchecks log
Jan 03 10:40:45 aisamu-nas borgmatic[532114]: INFO /etc/borgmatic.d/aisamu-company.yaml: Pinging Healthchecks finish
Jan 03 10:40:45 aisamu-nas borgmatic[532114]: INFO /etc/borgmatic.d/aisamu-company.yaml: Pinging ntfy topic borgmatic
Jan 03 10:40:45 aisamu-nas borgmatic[532114]: INFO /etc/borgmatic.d/aisamu-company.yaml: Pushing Uptime Kuma push_url http://localhost:3001/api/push/YOQXtMP61I?status=up&msg=finish
Jan 03 10:40:45 aisamu-nas borgmatic[532114]: INFO
Jan 03 10:40:45 aisamu-nas borgmatic[532114]: INFO summary:
Jan 03 10:40:45 aisamu-nas borgmatic[532114]: INFO /etc/borgmatic.d/aisamu-company.yaml: Successfully ran configuration file
Jan 03 10:40:45 aisamu-nas systemd[1]: borgmatic.service: Deactivated successfully.

Expected behavior

One of these (or both) would be preferable:

  • Borgmatic retries transient sqlite "database is locked" errors from DB dump hooks (configurable retry attempts/backoff), so a brief lock won't silently cause a missed DB dump.
  • Or, if a DB dump hook fails with an error that indicates the dump didn't occur, borgmatic should exit non-zero (or at least mark the run as failed) so systemd and alerting integrations detect the failure.

Either change would surface the problem to the operator instead of allowing a silent miss while healthchecks/notifications indicate success!

Other notes / implementation ideas

No response

borgmatic version

2.0.13

borgmatic installation method

NixOs

Borg version

borg 1.4.3

Python version

Python 3.13.11

Database version (if applicable)

sqlite 3.50.4 2025-07-30 19:33:53 4d8adfb30e03f9cf27f800a2c1ba3c48fb4ca1b08b0f5ed59a4d5ecbf45e20a3 (64-bit)

Operating system and version

BUILD_ID="25.11.2679.9ef261221d1e" CPE_NAME="cpe:/o:nixos:nixos:25.11" ID=nixos PRETTY_NAME="NixOS 25.11 (Xantusia)" VERSION="25.11 (Xantusia)" VERSION_CODENAME=xantusia VERSION_ID="25.11"

Happy to provide more information, logs or context!
Thanks a lot for borgmatic!

### What I'm trying to do and why Hi! I run borgmatic with database dump hooks (postgres + sqlite) as part of an automated full-system backup. I discovered a sqlite lock error in the logs but the service finished successfully and healthchecks/uptime notifications indicated success, so I was not alerted. It'd be nice if errors during dumps were retried (for transients) and/or alerted us! ### Steps to reproduce This is unfortunately not reliably reproducible for me. I encountered this while inspecting logs after a scheduled run, but running the service manually multiple times didn't show the issue again. This relevant excerpt from the config: ```yaml sqlite_databases: - name: healthchecks path: /var/lib/healthchecks/healthchecks.sqlite sqlite_command: /nix/store/awm8pnzpcj63lb26asvjm24j4kwg4d76-sqlite-3.50.4-bin/bin/sqlite3 sqlite_restore_command: /nix/store/awm8pnzpcj63lb26asvjm24j4kwg4d76-sqlite-3.50.4-bin/bin/sqlite3 ``` ### Actual behavior `borgmatic` logged the sqlite lock error, did not appear to retry, continued processing (archive rename/prune/compact), reported success to healthchecks/uptime, and systemd showed the service as deactivated successfully. ``` Jan 03 10:39:03 aisamu-nas systemd[1]: Starting borgmatic backup... Jan 03 10:40:08 aisamu-nas borgmatic[532114]: INFO /etc/borgmatic.d/aisamu-company.yaml: Pinging Healthchecks start Jan 03 10:40:13 aisamu-nas borgmatic[532114]: INFO /etc/borgmatic.d/aisamu-company.yaml: Pushing Uptime Kuma push_url http://localhost:3001/api/push/YOQXtMP61I?status=up&msg=start Jan 03 10:40:13 aisamu-nas borgmatic[532114]: INFO aisamu-company-m2: Running before repository command hook Jan 03 10:40:13 aisamu-nas borgmatic[532114]: INFO aisamu-company-m2: Creating archive Jan 03 10:40:13 aisamu-nas borgmatic[532114]: INFO aisamu-company-m2: Dumping PostgreSQL databases Jan 03 10:40:13 aisamu-nas borgmatic[532114]: INFO aisamu-company-m2: Dumping SQLite databases Jan 03 10:40:14 aisamu-nas borgmatic[532114]: WARNING aisamu-company-m2: Ignoring configured "read_special" value of false, as true is needed for database hooks. Jan 03 10:40:15 aisamu-nas borgmatic[532114]: INFO Creating archive at "ssh://aisamu@aisamu-company//Users/aisamu/backups/aisamu-nas.borg::aisamu-nas-2026-01-03T10:40:14.806204.checkpoint" Jan 03 10:40:16 aisamu-nas sudo[532129]: root : PWD=/ ; USER=postgres ; COMMAND=/nix/store/a067iwfgwaabg6wkxl9a80mqdabaffsi-postgresql-16.11/bin/pg_dump --no-password --clean --if-exists --username postgres --format custom immich Jan 03 10:40:16 aisamu-nas sudo[532129]: pam_unix(sudo:session): session opened for user postgres(uid=71) by (uid=0) Jan 03 10:40:38 aisamu-nas sudo[532129]: pam_unix(sudo:session): session closed for user postgres Jan 03 10:40:39 aisamu-nas borgmatic[532114]: INFO sql error: database is locked (5) Jan 03 10:40:40 aisamu-nas borgmatic[532114]: INFO aisamu-company-m2: Renaming archive aisamu-nas-2026-01-03T10:40:14.806204.checkpoint -> aisamu-nas-2026-01-03T10:40:14.806204 Jan 03 10:40:41 aisamu-nas borgmatic[532114]: INFO aisamu-company-m2: Pruning archives Jan 03 10:40:42 aisamu-nas borgmatic[532114]: INFO aisamu-company-m2: Compacting segments Jan 03 10:40:43 aisamu-nas borgmatic[532114]: INFO Remote: compaction freed about 2.85 kB repository space. Jan 03 10:40:43 aisamu-nas borgmatic[532114]: INFO aisamu-company-m2: Running consistency checks Jan 03 10:40:45 aisamu-nas borgmatic[532114]: INFO aisamu-company-m2: Skipping repository check due to configured frequency; 26 days, 7:16:55.465493 until next check (use --force to check anyway) Jan 03 10:40:45 aisamu-nas borgmatic[532114]: INFO aisamu-company-m2: Skipping archives check due to configured frequency; 26 days, 7:16:55.465144 until next check (use --force to check anyway) Jan 03 10:40:45 aisamu-nas borgmatic[532114]: INFO /etc/borgmatic.d/aisamu-company.yaml: Pinging Healthchecks log Jan 03 10:40:45 aisamu-nas borgmatic[532114]: INFO /etc/borgmatic.d/aisamu-company.yaml: Pinging Healthchecks finish Jan 03 10:40:45 aisamu-nas borgmatic[532114]: INFO /etc/borgmatic.d/aisamu-company.yaml: Pinging ntfy topic borgmatic Jan 03 10:40:45 aisamu-nas borgmatic[532114]: INFO /etc/borgmatic.d/aisamu-company.yaml: Pushing Uptime Kuma push_url http://localhost:3001/api/push/YOQXtMP61I?status=up&msg=finish Jan 03 10:40:45 aisamu-nas borgmatic[532114]: INFO Jan 03 10:40:45 aisamu-nas borgmatic[532114]: INFO summary: Jan 03 10:40:45 aisamu-nas borgmatic[532114]: INFO /etc/borgmatic.d/aisamu-company.yaml: Successfully ran configuration file Jan 03 10:40:45 aisamu-nas systemd[1]: borgmatic.service: Deactivated successfully. ``` ### Expected behavior One of these (or both) would be preferable: - Borgmatic retries transient sqlite "database is locked" errors from DB dump hooks (configurable retry attempts/backoff), so a brief lock won't silently cause a missed DB dump. - Or, if a DB dump hook fails with an error that indicates the dump didn't occur, borgmatic should exit non-zero (or at least mark the run as failed) so systemd and alerting integrations detect the failure. Either change would surface the problem to the operator instead of allowing a silent miss while healthchecks/notifications indicate success! ### Other notes / implementation ideas _No response_ ### borgmatic version 2.0.13 ### borgmatic installation method NixOs ### Borg version borg 1.4.3 ### Python version Python 3.13.11 ### Database version (if applicable) sqlite 3.50.4 2025-07-30 19:33:53 4d8adfb30e03f9cf27f800a2c1ba3c48fb4ca1b08b0f5ed59a4d5ecbf45e20a3 (64-bit) ### Operating system and version BUILD_ID="25.11.2679.9ef261221d1e" CPE_NAME="cpe:/o:nixos:nixos:25.11" ID=nixos PRETTY_NAME="NixOS 25.11 (Xantusia)" VERSION="25.11 (Xantusia)" VERSION_CODENAME=xantusia VERSION_ID="25.11" Happy to provide more information, logs or context! Thanks a lot for borgmatic!
witten commented 2026-01-04 18:50:04 +00:00
Owner
Copy Link

Thanks for the detailed ticket. Some points on this:

  • Even when I lock a SQLite database and try to repro this, borgmatic correctly errors with "sql error: database is locked (5)" and exits with that error. This also errors if I enable retries.
  • Can I see your borgmatic logs when running borgmatic with --verbosity 2? I'm particularly interested in the SQLite command that borgmatic is issuing. Does it contain the -bail flag?
  • If you want a more reliable repro of this on your system, you can follow the instructions here to create an exclusive lock on your SQLite database: https://stackoverflow.com/questions/14272715/how-can-i-lock-a-sqlite-database#14274452
Thanks for the detailed ticket. Some points on this: * Even when I lock a SQLite database and try to repro this, borgmatic correctly errors with "sql error: database is locked (5)" and exits with that error. This also errors if I enable retries. * Can I see your borgmatic logs when running borgmatic with `--verbosity 2`? I'm particularly interested in the SQLite command that borgmatic is issuing. Does it contain the `-bail` flag? * If you want a more reliable repro of this on your system, you can follow the instructions here to create an exclusive lock on your SQLite database: https://stackoverflow.com/questions/14272715/how-can-i-lock-a-sqlite-database#14274452
aisamu commented 2026-01-04 20:23:38 +00:00
Author
Copy Link

Can I see your borgmatic logs when running borgmatic with --verbosity 2. I'm particularly interested in the SQLite command that borgmatic is issuing. Does it contain the -bail flag

It does!

aisamu-company-m2: Dumping SQLite database at /var/lib/healthchecks/healthchecks.sqlite to /tmp/borgmatic-a7oopyzb/./borgmatic/sqlite_databases/localhost/healthchecks
aisamu-company-m2: /nix/store/awm8pnzpcj63lb26asvjm24j4kwg4d76-sqlite-3.50.4-bin/bin/sqlite3 -bail /var/lib/healthchecks/healthchecks.sqlite .dump > /tmp/borgmatic-a7oopyzb/./borgmatic/sqlite_databases/localhost/healthchecks

If you want a more reliable repro of this on your system, you can follow the instructions here to create an exclusive lock on your SQLite database: https://stackoverflow.com/questions/14272715/how-can-i-lock-a-sqlite-database#14274452

Reproduced with the lock!

Jan 04 17:16:01 aisamu-nas systemd[1]: Starting borgmatic backup...
Jan 04 17:17:05 aisamu-nas borgmatic[626981]: INFO /etc/borgmatic.d/aisamu-company.yaml: Pinging Healthchecks start
Jan 04 17:17:15 aisamu-nas borgmatic[626981]: WARNING /etc/borgmatic.d/aisamu-company.yaml: Healthchecks error: HTTPConnectionPool(host='localhost', port=port): Read timed out. (read timeout=10)
Jan 04 17:17:15 aisamu-nas borgmatic[626981]: INFO /etc/borgmatic.d/aisamu-company.yaml: Pushing Uptime Kuma push_url http://host:port/api/push/**********?status=up&msg=start
Jan 04 17:17:16 aisamu-nas borgmatic[626981]: INFO aisamu-company-m2: Running before repository command hook
Jan 04 17:17:16 aisamu-nas borgmatic[626981]: INFO aisamu-company-m2: Creating archive
Jan 04 17:17:16 aisamu-nas borgmatic[626981]: INFO aisamu-company-m2: Dumping PostgreSQL databases
Jan 04 17:17:16 aisamu-nas borgmatic[626981]: INFO aisamu-company-m2: Dumping SQLite databases
Jan 04 17:17:16 aisamu-nas borgmatic[626981]: WARNING aisamu-company-m2: Ignoring configured "read_special" value of false, as true is needed for database hooks.
Jan 04 17:17:18 aisamu-nas borgmatic[626981]: INFO Creating archive at "ssh://aisamu@aisamu-company//Users/aisamu/backups/aisamu-nas.borg::aisamu-nas-2026-01-04T17:17:17.329606.checkpoint"
Jan 04 17:17:18 aisamu-nas sudo[627074]:     root : PWD=/ ; USER=postgres ; COMMAND=/nix/store/a067iwfgwaabg6wkxl9a80mqdabaffsi-postgresql-16.11/bin/pg_dump --no-password --clean --if-exists --username postgres --format custom database
Jan 04 17:17:18 aisamu-nas sudo[627074]: pam_unix(sudo:session): session opened for user postgres(uid=71) by (uid=0)
Jan 04 17:17:41 aisamu-nas sudo[627074]: pam_unix(sudo:session): session closed for user postgres
Jan 04 17:17:41 aisamu-nas borgmatic[626981]: INFO sql error: database is locked (5)
Jan 04 17:17:43 aisamu-nas borgmatic[626981]: INFO aisamu-company-m2: Renaming archive aisamu-nas-2026-01-04T17:17:17.329606.checkpoint -> aisamu-nas-2026-01-04T17:17:17.329606
Jan 04 17:17:44 aisamu-nas borgmatic[626981]: INFO aisamu-company-m2: Pruning archives
Jan 04 17:17:46 aisamu-nas borgmatic[626981]: INFO aisamu-company-m2: Compacting segments
Jan 04 17:17:47 aisamu-nas borgmatic[626981]: INFO Remote: compaction freed about 1.72 MB repository space.
Jan 04 17:17:47 aisamu-nas borgmatic[626981]: INFO aisamu-company-m2: Running consistency checks
Jan 04 17:17:48 aisamu-nas borgmatic[626981]: INFO aisamu-company-m2: Skipping repository check due to configured frequency; 25 days, 0:39:51.955017 until next check (use --force to check anyway)
Jan 04 17:17:48 aisamu-nas borgmatic[626981]: INFO aisamu-company-m2: Skipping archives check due to configured frequency; 25 days, 0:39:51.954806 until next check (use --force to check anyway)
Jan 04 17:17:48 aisamu-nas borgmatic[626981]: INFO /etc/borgmatic.d/aisamu-company.yaml: Pinging Healthchecks log
Jan 04 17:17:58 aisamu-nas borgmatic[626981]: WARNING /etc/borgmatic.d/aisamu-company.yaml: Healthchecks error: HTTPConnectionPool(host='localhost', port=port): Read timed out. (read timeout=10)
Jan 04 17:17:58 aisamu-nas borgmatic[626981]: INFO /etc/borgmatic.d/aisamu-company.yaml: Pinging Healthchecks finish
Jan 04 17:18:08 aisamu-nas borgmatic[626981]: WARNING /etc/borgmatic.d/aisamu-company.yaml: Healthchecks error: HTTPConnectionPool(host='localhost', port=port): Read timed out. (read timeout=10)
Jan 04 17:18:08 aisamu-nas borgmatic[626981]: INFO /etc/borgmatic.d/aisamu-company.yaml: Pinging ntfy topic borgmatic
Jan 04 17:18:08 aisamu-nas borgmatic[626981]: INFO /etc/borgmatic.d/aisamu-company.yaml: Pushing Uptime Kuma push_url http://host:port/api/push/**********?status=up&msg=finish
Jan 04 17:18:08 aisamu-nas borgmatic[626981]: INFO
Jan 04 17:18:08 aisamu-nas borgmatic[626981]: INFO summary:
Jan 04 17:18:08 aisamu-nas borgmatic[626981]: INFO /etc/borgmatic.d/aisamu-company.yaml: Successfully ran configuration file
Jan 04 17:18:08 aisamu-nas systemd[1]: borgmatic.service: Deactivated successfully.
Jan 04 17:18:08 aisamu-nas systemd[1]: Finished borgmatic backup.
Jan 04 17:18:08 aisamu-nas systemd[1]: borgmatic.service: Consumed 28.653s CPU time, 95.8M memory peak, 111K incoming IP traffic, 1.7M outgoing IP traffic.

(A couple more failures here than on the original, because I created an external lock and prevented healthchecks itself from using its own database)

> Can I see your borgmatic logs when running borgmatic with --verbosity 2. I'm particularly interested in the SQLite command that borgmatic is issuing. Does it contain the -bail flag It does! ``` aisamu-company-m2: Dumping SQLite database at /var/lib/healthchecks/healthchecks.sqlite to /tmp/borgmatic-a7oopyzb/./borgmatic/sqlite_databases/localhost/healthchecks aisamu-company-m2: /nix/store/awm8pnzpcj63lb26asvjm24j4kwg4d76-sqlite-3.50.4-bin/bin/sqlite3 -bail /var/lib/healthchecks/healthchecks.sqlite .dump > /tmp/borgmatic-a7oopyzb/./borgmatic/sqlite_databases/localhost/healthchecks ``` > If you want a more reliable repro of this on your system, you can follow the instructions here to create an exclusive lock on your SQLite database: https://stackoverflow.com/questions/14272715/how-can-i-lock-a-sqlite-database#14274452 Reproduced with the lock! ``` Jan 04 17:16:01 aisamu-nas systemd[1]: Starting borgmatic backup... Jan 04 17:17:05 aisamu-nas borgmatic[626981]: INFO /etc/borgmatic.d/aisamu-company.yaml: Pinging Healthchecks start Jan 04 17:17:15 aisamu-nas borgmatic[626981]: WARNING /etc/borgmatic.d/aisamu-company.yaml: Healthchecks error: HTTPConnectionPool(host='localhost', port=port): Read timed out. (read timeout=10) Jan 04 17:17:15 aisamu-nas borgmatic[626981]: INFO /etc/borgmatic.d/aisamu-company.yaml: Pushing Uptime Kuma push_url http://host:port/api/push/**********?status=up&msg=start Jan 04 17:17:16 aisamu-nas borgmatic[626981]: INFO aisamu-company-m2: Running before repository command hook Jan 04 17:17:16 aisamu-nas borgmatic[626981]: INFO aisamu-company-m2: Creating archive Jan 04 17:17:16 aisamu-nas borgmatic[626981]: INFO aisamu-company-m2: Dumping PostgreSQL databases Jan 04 17:17:16 aisamu-nas borgmatic[626981]: INFO aisamu-company-m2: Dumping SQLite databases Jan 04 17:17:16 aisamu-nas borgmatic[626981]: WARNING aisamu-company-m2: Ignoring configured "read_special" value of false, as true is needed for database hooks. Jan 04 17:17:18 aisamu-nas borgmatic[626981]: INFO Creating archive at "ssh://aisamu@aisamu-company//Users/aisamu/backups/aisamu-nas.borg::aisamu-nas-2026-01-04T17:17:17.329606.checkpoint" Jan 04 17:17:18 aisamu-nas sudo[627074]: root : PWD=/ ; USER=postgres ; COMMAND=/nix/store/a067iwfgwaabg6wkxl9a80mqdabaffsi-postgresql-16.11/bin/pg_dump --no-password --clean --if-exists --username postgres --format custom database Jan 04 17:17:18 aisamu-nas sudo[627074]: pam_unix(sudo:session): session opened for user postgres(uid=71) by (uid=0) Jan 04 17:17:41 aisamu-nas sudo[627074]: pam_unix(sudo:session): session closed for user postgres Jan 04 17:17:41 aisamu-nas borgmatic[626981]: INFO sql error: database is locked (5) Jan 04 17:17:43 aisamu-nas borgmatic[626981]: INFO aisamu-company-m2: Renaming archive aisamu-nas-2026-01-04T17:17:17.329606.checkpoint -> aisamu-nas-2026-01-04T17:17:17.329606 Jan 04 17:17:44 aisamu-nas borgmatic[626981]: INFO aisamu-company-m2: Pruning archives Jan 04 17:17:46 aisamu-nas borgmatic[626981]: INFO aisamu-company-m2: Compacting segments Jan 04 17:17:47 aisamu-nas borgmatic[626981]: INFO Remote: compaction freed about 1.72 MB repository space. Jan 04 17:17:47 aisamu-nas borgmatic[626981]: INFO aisamu-company-m2: Running consistency checks Jan 04 17:17:48 aisamu-nas borgmatic[626981]: INFO aisamu-company-m2: Skipping repository check due to configured frequency; 25 days, 0:39:51.955017 until next check (use --force to check anyway) Jan 04 17:17:48 aisamu-nas borgmatic[626981]: INFO aisamu-company-m2: Skipping archives check due to configured frequency; 25 days, 0:39:51.954806 until next check (use --force to check anyway) Jan 04 17:17:48 aisamu-nas borgmatic[626981]: INFO /etc/borgmatic.d/aisamu-company.yaml: Pinging Healthchecks log Jan 04 17:17:58 aisamu-nas borgmatic[626981]: WARNING /etc/borgmatic.d/aisamu-company.yaml: Healthchecks error: HTTPConnectionPool(host='localhost', port=port): Read timed out. (read timeout=10) Jan 04 17:17:58 aisamu-nas borgmatic[626981]: INFO /etc/borgmatic.d/aisamu-company.yaml: Pinging Healthchecks finish Jan 04 17:18:08 aisamu-nas borgmatic[626981]: WARNING /etc/borgmatic.d/aisamu-company.yaml: Healthchecks error: HTTPConnectionPool(host='localhost', port=port): Read timed out. (read timeout=10) Jan 04 17:18:08 aisamu-nas borgmatic[626981]: INFO /etc/borgmatic.d/aisamu-company.yaml: Pinging ntfy topic borgmatic Jan 04 17:18:08 aisamu-nas borgmatic[626981]: INFO /etc/borgmatic.d/aisamu-company.yaml: Pushing Uptime Kuma push_url http://host:port/api/push/**********?status=up&msg=finish Jan 04 17:18:08 aisamu-nas borgmatic[626981]: INFO Jan 04 17:18:08 aisamu-nas borgmatic[626981]: INFO summary: Jan 04 17:18:08 aisamu-nas borgmatic[626981]: INFO /etc/borgmatic.d/aisamu-company.yaml: Successfully ran configuration file Jan 04 17:18:08 aisamu-nas systemd[1]: borgmatic.service: Deactivated successfully. Jan 04 17:18:08 aisamu-nas systemd[1]: Finished borgmatic backup. Jan 04 17:18:08 aisamu-nas systemd[1]: borgmatic.service: Consumed 28.653s CPU time, 95.8M memory peak, 111K incoming IP traffic, 1.7M outgoing IP traffic. ``` (A couple more failures here than on the original, because I created an external lock and prevented healthchecks itself from using its own database)
aisamu commented 2026-01-04 20:38:11 +00:00
Author
Copy Link

Another reproduction but with --verbosity 2

Another reproduction but with `--verbosity 2`
borgmatic.txt
24 KiB
witten commented 2026-01-04 21:34:24 +00:00
Owner
Copy Link

Super odd! What happens if you run SQLite manually while the database is locked? Example:

/nix/store/awm8pnzpcj63lb26asvjm24j4kwg4d76-sqlite-3.50.4-bin/bin/sqlite3 -bail /var/lib/healthchecks/healthchecks.sqlite .dump > temp.sql

Does that command exit with an error or not? (echo $? in bash)

Super odd! What happens if you run SQLite manually while the database is locked? Example: ```bash /nix/store/awm8pnzpcj63lb26asvjm24j4kwg4d76-sqlite-3.50.4-bin/bin/sqlite3 -bail /var/lib/healthchecks/healthchecks.sqlite .dump > temp.sql ``` Does that command exit with an error or not? (`echo $?` in bash)
aisamu commented 2026-01-04 22:20:05 +00:00
Author
Copy Link

It does not exit with an error:

/nix/store/awm8pnzpcj63lb26asvjm24j4kwg4d76-sqlite-3.50.4-bin/bin/sqlite3 -bail /var/lib/healthchecks/healthchecks.sqlite .dump
sql error: database is locked (5)
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
/**** ERROR: (5) database is locked *****/
ROLLBACK; -- due to errors

[nix-shell:~]#  echo $?
0

[nix-shell:~]#  eco $?
bash: eco: command not found

[nix-shell:~]#  echo $?
127
It does not exit with an error: ```terminal /nix/store/awm8pnzpcj63lb26asvjm24j4kwg4d76-sqlite-3.50.4-bin/bin/sqlite3 -bail /var/lib/healthchecks/healthchecks.sqlite .dump sql error: database is locked (5) PRAGMA foreign_keys=OFF; BEGIN TRANSACTION; /**** ERROR: (5) database is locked *****/ ROLLBACK; -- due to errors [nix-shell:~]# echo $? 0 [nix-shell:~]# eco $? bash: eco: command not found [nix-shell:~]# echo $? 127 ```
aisamu commented 2026-01-04 22:21:49 +00:00
Author
Copy Link

Oh, an sqlite bug was not my bingo card given their track record!
I guess I was just very very unlucky - it was recently solved but is probably not yet released
https://sqlite.org/forum/forumpost/a4f3873b0a8503ccd62b29491de01c13ca8f764a52c6032f8873802bd474caba
https://sqlite.org/src/info/850b92b634718

Thanks for holding my hand throughout this!

Oh, an sqlite bug was not my bingo card given their track record! I guess I was just very very unlucky - it was recently solved but is probably not yet released https://sqlite.org/forum/forumpost/a4f3873b0a8503ccd62b29491de01c13ca8f764a52c6032f8873802bd474caba https://sqlite.org/src/info/850b92b634718 Thanks for holding my hand throughout this!
witten commented 2026-01-04 22:47:54 +00:00
Owner
Copy Link

Wow, good find! Yeah, borgmatic gets all of its cues on database dump success from SQLite's exit code. So if it happily blunders through and returns success despite an error, so does borgmatic. 😄

Wow, good find! Yeah, borgmatic gets all of its cues on database dump success from SQLite's exit code. So if it happily blunders through and returns success despite an error, so does borgmatic. 😄
witten commented 2026-01-04 22:52:04 +00:00
Owner
Copy Link

FWIW, I do have a newer version of SQLite that apparently isn't manifesting this bug. Maybe it's not available in NixOS yet though.

sqlite3 --version
3.51.1 2025-11-28 17:28:25 281fc0e9afc38674b9b0991943b9e9d1e64c6cbdb133d35f6f5c87ff6af3alt1 (64-bit)
FWIW, I do have a newer version of SQLite that apparently isn't manifesting this bug. Maybe it's not available in NixOS yet though. ```bash sqlite3 --version 3.51.1 2025-11-28 17:28:25 281fc0e9afc38674b9b0991943b9e9d1e64c6cbdb133d35f6f5c87ff6af3alt1 (64-bit) ```
❤️ 1
witten added the question / support label 2026-01-04 22:52:51 +00:00
aisamu commented 2026-01-05 01:34:06 +00:00
Author
Copy Link

Tested with 3.51.1 on nixos-unstable and we're back on track! Thanks again!

Tested with 3.51.1 on nixos-unstable and we're back on track! Thanks again!
witten commented 2026-01-05 02:44:31 +00:00
Owner
Copy Link

Awesome, glad to hear it!

Awesome, glad to hear it!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
blocked
breaking
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
No Label question / support
Milestone
No items
No Milestone
Assignees
Clear assignees
witten
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#1223
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?