terminating with success status, rc 0 #856

New Issue

mariuspana · 2024-04-23T11:41:32Z

mariuspana commented

2024-04-23 11:41:32 +00:00

What I'm trying to do and why

Hi, Im trying to complete a backup including mysql database that never finishes.

Thanks

Steps to reproduce

Run the borgmatic command:

borgmatic -v2

Actual behavior

The backup seems to complete however there is not database backup, we have hung mysqldump processes:

[root@isp mpana]# ps -ef|grep -i mysql mysql 1061 1130 0 2023 ? 15:05:55 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mariadb/mariadb.log --pid-file=/var/run/mariadb/mariadb.pid --socket=/var/lib/mysql/mysql.sock mysql 1130 1 0 2023 ? 00:00:00 /bin/sh /usr/bin/mysqld_safe --basedir=/usr root 12416 12413 0 13:33 pts/1 00:00:00 /bin/sh -c mysqldump --quick --compress --max-allowed-packet=2G --add-drop-database --user root --databases list of databases > /home/tmp/mysql_databases/localhost/all root 12417 12416 0 13:33 pts/1 00:00:00 /bin/sh -c mysqldump --quick --compress --max-allowed-packet=2G --add-drop-database --user root --databases list of databases > /home/tmp/mysql_databases/localhost/all root 12518 12495 0 13:33 pts/2 00:00:00 grep --color=auto -i mysql root 30328 1 0 11:49 ? 00:00:00 /bin/sh -c mysqldump --quick --compress --max-allowed-packet=2G --add-drop-database --user root --databases lis of databases > /home/tmp/mysql_databases/localhost/all

And borg/borgmatic never finishes. We have a process that is waiting in this state for over 3 days now. We end up killing them.

security: saving state for ce532d87c91d01bf89453e606b34ae90834fd58d103037b0fd58c1d50c28a64a to /root/.config/borg/security/ce532d87c91d01bf89453e606b34ae90834fd58d103037b0fd58c1d50c28a64a security: current location ssh://xxxx/./xxx security: key type 2 security: manifest timestamp 2024-04-23T10:36:14.951989 RemoteRepository: 140.60 MB bytes sent, 60.22 kB bytes received, 1144 messages sent terminating with success status, rc 0

Mounting and verifying the backups does not include the databases, just the files.

Expected behavior

The job to finish and close and include the mysql dumps.

Other notes / implementation ideas

No response

borgmatic version

1.5.22

borgmatic installation method

pipx

Borg version

1.1.18

Python version

3.6.8

Database version (if applicable)

Ver 15.1 Distrib 5.5.68-MariaDB

Operating system and version

NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="7" PRETTY_NAME="CentOS Linux 7 (Core)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:7" HOME_URL="https://www.centos.org/" BUG_REPORT_URL="https://bugs.centos.org/" CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7"

### What I'm trying to do and why Hi, Im trying to complete a backup including mysql database that never finishes. Thanks ### Steps to reproduce Run the borgmatic command: # borgmatic -v2 ### Actual behavior The backup seems to complete however there is not database backup, we have hung mysqldump processes: `[root@isp mpana]# ps -ef|grep -i mysql mysql 1061 1130 0 2023 ? 15:05:55 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mariadb/mariadb.log --pid-file=/var/run/mariadb/mariadb.pid --socket=/var/lib/mysql/mysql.sock mysql 1130 1 0 2023 ? 00:00:00 /bin/sh /usr/bin/mysqld_safe --basedir=/usr root 12416 12413 0 13:33 pts/1 00:00:00 /bin/sh -c mysqldump --quick --compress --max-allowed-packet=2G --add-drop-database --user root --databases list of databases > /home/tmp/mysql_databases/localhost/all root 12417 12416 0 13:33 pts/1 00:00:00 /bin/sh -c mysqldump --quick --compress --max-allowed-packet=2G --add-drop-database --user root --databases list of databases > /home/tmp/mysql_databases/localhost/all root 12518 12495 0 13:33 pts/2 00:00:00 grep --color=auto -i mysql root 30328 1 0 11:49 ? 00:00:00 /bin/sh -c mysqldump --quick --compress --max-allowed-packet=2G --add-drop-database --user root --databases lis of databases > /home/tmp/mysql_databases/localhost/all` And borg/borgmatic never finishes. We have a process that is waiting in this state for over 3 days now. We end up killing them. `security: saving state for ce532d87c91d01bf89453e606b34ae90834fd58d103037b0fd58c1d50c28a64a to /root/.config/borg/security/ce532d87c91d01bf89453e606b34ae90834fd58d103037b0fd58c1d50c28a64a security: current location ssh://xxxx/./xxx security: key type 2 security: manifest timestamp 2024-04-23T10:36:14.951989 RemoteRepository: 140.60 MB bytes sent, 60.22 kB bytes received, 1144 messages sent terminating with success status, rc 0` Mounting and verifying the backups does not include the databases, just the files. ### Expected behavior The job to finish and close and include the mysql dumps. ### Other notes / implementation ideas _No response_ ### borgmatic version 1.5.22 ### borgmatic installation method pipx ### Borg version 1.1.18 ### Python version 3.6.8 ### Database version (if applicable) Ver 15.1 Distrib 5.5.68-MariaDB ### Operating system and version NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="7" PRETTY_NAME="CentOS Linux 7 (Core)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:7" HOME_URL="https://www.centos.org/" BUG_REPORT_URL="https://bugs.centos.org/" CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7"

witten commented

2024-04-23 15:37:46 +00:00

Thanks for taking the time to file this and provide all the details. There have been a number of improvements made to borgmatic since the version 1.5.22 that you're using, specifically around detecting and preventing hangs during database dumps. I highly suspect that's what you're encountering here and so I recommend you upgrade borgmatic and try again! It sounds like you installed borgmatic with pipx, which makes me think it should hopefully be pretty easy to upgrade. (If you didn't actually install with pipx and you're using an old OS package, uninstall that first and then install with pipx or another installation method of your choice.)

Also, check out the database hook limitations in the documentation. Some of them may be relevant in your case.

Thanks for taking the time to file this and provide all the details. There have been a number of improvements made to borgmatic since the version 1.5.22 that you're using, specifically around detecting and preventing hangs during database dumps. I highly suspect that's what you're encountering here and so I recommend you [upgrade borgmatic](https://torsion.org/borgmatic/docs/how-to/upgrade/) and try again! It sounds like you installed borgmatic with pipx, which makes me think it should hopefully be pretty easy to upgrade. (If you didn't actually install with pipx and you're using an old OS package, uninstall that first and then [install with pipx](https://torsion.org/borgmatic/docs/how-to/set-up-backups/) or [another installation method of your choice](https://torsion.org/borgmatic/docs/how-to/set-up-backups/#other-ways-to-install).) Also, check out the [database hook limitations](https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#limitations) in the documentation. Some of them may be relevant in your case.

witten added the

question / support

label 2024-04-23 15:37:58 +00:00

mariuspana commented

2024-04-23 16:16:42 +00:00

Hi,

Thank you as well.

For reasons unbeknownst to me pipx upgrade borgmatic tells me that version 1.5.22 is the latest. I will use one of the other installation methods and test things out.

I have been reading the documentation and did notice that I am using an older version here but did not bother to look into it further. Will do so now.

Regards,
Marius

Hi, Thank you as well. For reasons unbeknownst to me pipx upgrade borgmatic tells me that version 1.5.22 is the latest. I will use one of the other installation methods and test things out. I have been reading the documentation and did notice that I am using an older version here but did not bother to look into it further. Will do so now. Regards, Marius

witten commented

2024-04-23 16:30:25 +00:00

Huh, that is quite odd! There are definitely newer versions available to pipx. Maybe CentOS has modified their pipx package in some way. In any case, you might be able to install instead with sudo pip3 install borgmatic or pip3 install --user borgmatic.

Huh, that is quite odd! There are definitely [newer versions](https://pypi.org/project/borgmatic/#history) available to pipx. Maybe CentOS has modified their pipx package in some way. In any case, you might be able to install instead with `sudo pip3 install borgmatic` or `pip3 install --user borgmatic`.

mariuspana commented

2024-04-24 12:26:32 +00:00

I got updated versions installed, however the issue remains:

borg 1.2.8 borgmatic 1.8.10

I believe the problem may be that I am using a pretty old version of mariadb, which does not include mariadb-dump. I am using the mysql_databases section with mysqldump.

With the never version, the job finished succefully, it no longer hangs with "terminating with success status, rc 0", however there is not database included in the backup.

I will prepare an upgrade to a newer mariadb with the mariadb-dump program included.

Feel free to close this issue or I can report back after the upgrade with more details.

Thank you, I appreciate the support.

I got updated versions installed, however the issue remains: ` borg 1.2.8 borgmatic 1.8.10 ` I believe the problem may be that I am using a pretty old version of mariadb, which does not include mariadb-dump. I am using the mysql_databases section with mysqldump. With the never version, the job finished succefully, it no longer _hangs_ with "terminating with success status, rc 0", however there is not database included in the backup. I will prepare an upgrade to a newer mariadb with the mariadb-dump program included. Feel free to close this issue or I can report back after the upgrade with more details. Thank you, I appreciate the support.

witten commented

2024-04-24 23:19:56 +00:00

With the never version, the job finished succefully, it no longer hangs with "terminating with success status, rc 0", however there is not database included in the backup.

How are you checking whether a database is included in the backup? Do you see any indication of that in the borgmatic logs? And are you trying borgmatic list to inspect the contents of the backup archive and look for the database dump?

Feel free to close this issue or I can report back after the upgrade with more details.

I'm happy to leave this open if you're planning on reporting back!

Thank you, I appreciate the support.

Sure thing.

> With the never version, the job finished succefully, it no longer hangs with "terminating with success status, rc 0", however there is not database included in the backup. How are you checking whether a database is included in the backup? Do you see any indication of that in the borgmatic logs? And are you trying `borgmatic list` to inspect the contents of the backup archive and look for the database dump? > Feel free to close this issue or I can report back after the upgrade with more details. I'm happy to leave this open if you're planning on reporting back! > Thank you, I appreciate the support. Sure thing.

mariuspana commented

2024-04-27 17:01:41 +00:00

How are you checking whether a database is included in the backup? Do you see any indication of that in the borgmatic logs? And are you trying borgmatic list to inspect the contents of the backup archive and look for the database dump?

I am mounting the backup.

(borg) [root@isp ~] borgmatic mount --mount-point /mnt/restore --verbosity 2

And then I look for directory with .borgmatic or a mysql_datbases folder as I noticed I have on other systems where this works:

(borg) [root@isp restore]# cd isp-2024-04-27T03\:00\:03/
(borg) [root@isp isp-2024-04-27T03:00:03]# ls -la
total 0
drwxr-xr-x  1 root root 0 Apr 27 03:00 .
drwxr-xr-x  1 root root 0 Apr 27 19:42 ..
drwxr-xr-x. 1 root root 0 Apr 27 01:23 etc
drwxrwxrwx  1 root root 0 Apr 26 09:58 home

I have also tried searching with 'find' without success:

(borg) [root@isp ~]# borgmatic list --archive latest --find .borgmatic/*_databases
backupserver: Searching archives
ssh://u398645-sub4@u398645.your-storagebox.de/./isp/: Listing archive isp-2024-04-27T19:49:27

Interestingly during the backup I can see and follow (strace) the named pipe and see the mysqldump command, dumping.

I am searching for another Centos7/RH box to test.

> How are you checking whether a database is included in the backup? Do you see any indication of that in the borgmatic logs? And are you trying `borgmatic list` to inspect the contents of the backup archive and look for the database dump? I am mounting the backup. `(borg) [root@isp ~] borgmatic mount --mount-point /mnt/restore --verbosity 2` And then I look for directory with .borgmatic or a mysql_datbases folder as I noticed I have on other systems where this works: ``` (borg) [root@isp restore]# cd isp-2024-04-27T03\:00\:03/ (borg) [root@isp isp-2024-04-27T03:00:03]# ls -la total 0 drwxr-xr-x 1 root root 0 Apr 27 03:00 . drwxr-xr-x 1 root root 0 Apr 27 19:42 .. drwxr-xr-x. 1 root root 0 Apr 27 01:23 etc drwxrwxrwx 1 root root 0 Apr 26 09:58 home ``` I have also tried searching with 'find' without success: ``` (borg) [root@isp ~]# borgmatic list --archive latest --find .borgmatic/*_databases backupserver: Searching archives ssh://u398645-sub4@u398645.your-storagebox.de/./isp/: Listing archive isp-2024-04-27T19:49:27 ``` Interestingly during the backup I can see and follow (strace) the named pipe and see the mysqldump command, dumping. I am searching for another Centos7/RH box to test.

witten commented

2024-04-27 20:13:01 +00:00

Got it, thanks. It definitely does appear that there aren't any databases in that archive! I think at this point it would be useful to see the full verbose output of your borgmatic logs (redacted if necessary). That will hopefully show what's going on with the database dump command.

mariuspana commented

2024-04-29 08:06:00 +00:00

Alright, here is the the log, I "carefully" cut out all the files being uploaded and am quite certain I did not miss anything:

(borg) [root@isp ~]# borgmatic -v 2 create -c /etc/borgmatic/config.yaml --list --files --log-file /tmp/borgmaticlog.txt

[2024-04-29 10:21:45,128] INFO: backupserver: Creating archive
[2024-04-29 10:21:45,129] INFO: ssh://user@mybox.ro/./isp/: Dumping MySQL databases
[2024-04-29 10:21:45,209] WARNING: ssh://user@mybox.ro/./isp/: Ignoring configured "read_special" value of false, as true is needed for database hooks.
[2024-04-29 10:22:40,288] ANSWER: using builtin fallback logging configuration
[2024-04-29 10:22:40,408] ANSWER: 33 self tests completed in 0.12 seconds
[2024-04-29 10:22:40,409] ANSWER: SSH command line: ['ssh', '-p', '23', 'user@mybox.ro', 'borg', 'serve', '--debug']
[2024-04-29 10:22:43,051] ANSWER: Remote: using builtin fallback logging configuration
[2024-04-29 10:22:43,114] ANSWER: Remote: 33 self tests completed in 0.06 seconds
[2024-04-29 10:22:43,121] ANSWER: Remote: using builtin fallback logging configuration
[2024-04-29 10:22:43,121] ANSWER: Remote: Initialized logging system for JSON-based protocol
[2024-04-29 10:22:43,176] ANSWER: Remote: Resolving repository path b'/./isp'
[2024-04-29 10:22:43,187] ANSWER: Remote: Resolved repository path to '/home/isp'
[2024-04-29 10:22:43,305] ANSWER: Remote: Verified integrity of /home/isp/index.861
[2024-04-29 10:22:43,323] ANSWER: TAM-verified manifest
[2024-04-29 10:22:43,326] ANSWER: security: read previous location 'ssh://user@mybox.ro/./isp'
[2024-04-29 10:22:43,326] ANSWER: security: read manifest timestamp '2024-04-29T00:05:19.729505'
[2024-04-29 10:22:43,327] ANSWER: security: determined newest manifest timestamp as 2024-04-29T00:05:19.729505
[2024-04-29 10:22:43,328] ANSWER: security: repository checks ok, allowing access
[2024-04-29 10:22:43,328] ANSWER: Creating archive at "ssh://user@mybox.ro/./isp/::isp-2024-04-29T10:22:40"
[2024-04-29 10:22:43,380] ANSWER: Verified integrity of /home/tmp/ce532d87c91d01bf89453e606b34ae90834fd58d103037b0fd58c1d50c28a64a/chunks
[2024-04-29 10:22:43,380] ANSWER: Reading files cache ...
[2024-04-29 10:22:46,663] ANSWER: Verified integrity of /home/tmp/ce532d87c91d01bf89453e606b34ae90834fd58d103037b0fd58c1d50c28a64a/files
[2024-04-29 10:22:46,664] ANSWER: security: read previous location 'ssh://user@mybox.ro/./isp'
<snip>
[2024-04-29 10:23:45,008] ANSWER: M /home/mariuspana/logs/php_log
[2024-04-29 10:25:10,404] ANSWER: A /home/xxx/homes/xxx/Maildir/cur/1714374005.5945_0.isp:2,S
[2024-04-29 10:25:14,476] ANSWER: M /home/backupisp/backuplog
[2024-04-29 10:25:15,189] ANSWER: Remote: check_free_space: required bytes 751335498, free bytes 20462240726528
[2024-04-29 10:25:15,274] ANSWER: security: saving state for ce532d87c91d01bf89453e606b34ae90834fd58d103037b0fd58c1d50c28a64a to /root/.config/borg/security/ce532d87c91d01bf89453e606b34ae90834fd58d103037b0fd58c1d50c28a64a
[2024-04-29 10:25:15,275] ANSWER: security: current location   ssh://user@mybox.ro/./isp
[2024-04-29 10:25:15,275] ANSWER: security: key type           2
[2024-04-29 10:25:15,275] ANSWER: security: manifest timestamp 2024-04-29T07:25:15.078812
[2024-04-29 10:25:20,056] ANSWER: RemoteRepository: 177.56 MB bytes sent, 7.83 kB bytes received, 630 messages sent
[2024-04-29 10:25:20,154] ANSWER: terminating with warning status, rc 1

The included lines similar to these:

A /home/someone/homes/some.user/Maildir/cur/1714374173.7738_0.isp:2,RS
A /home/someone/homes/some.user/Maildir/cur/1714374466.10428_0.isp:2,S
M /home/someone/homes/some.user//.spamassassin/bayes_toks
M /home/someone/homes/some.user/.spamassassin/bayes_seen
A /home/someone/homes/some.user//Maildir/.Sent/cur/1714375951.M51276P21202.isp,S=6559,W=6742:2,S
A /home/someone/homes/some.user/Maildir/cur/1714375641.18804_0.isp:2,S

Alright, here is the the log, I "carefully" cut out all the files being uploaded and am quite certain I did not miss anything: ``` (borg) [root@isp ~]# borgmatic -v 2 create -c /etc/borgmatic/config.yaml --list --files --log-file /tmp/borgmaticlog.txt [2024-04-29 10:21:45,128] INFO: backupserver: Creating archive [2024-04-29 10:21:45,129] INFO: ssh://user@mybox.ro/./isp/: Dumping MySQL databases [2024-04-29 10:21:45,209] WARNING: ssh://user@mybox.ro/./isp/: Ignoring configured "read_special" value of false, as true is needed for database hooks. [2024-04-29 10:22:40,288] ANSWER: using builtin fallback logging configuration [2024-04-29 10:22:40,408] ANSWER: 33 self tests completed in 0.12 seconds [2024-04-29 10:22:40,409] ANSWER: SSH command line: ['ssh', '-p', '23', 'user@mybox.ro', 'borg', 'serve', '--debug'] [2024-04-29 10:22:43,051] ANSWER: Remote: using builtin fallback logging configuration [2024-04-29 10:22:43,114] ANSWER: Remote: 33 self tests completed in 0.06 seconds [2024-04-29 10:22:43,121] ANSWER: Remote: using builtin fallback logging configuration [2024-04-29 10:22:43,121] ANSWER: Remote: Initialized logging system for JSON-based protocol [2024-04-29 10:22:43,176] ANSWER: Remote: Resolving repository path b'/./isp' [2024-04-29 10:22:43,187] ANSWER: Remote: Resolved repository path to '/home/isp' [2024-04-29 10:22:43,305] ANSWER: Remote: Verified integrity of /home/isp/index.861 [2024-04-29 10:22:43,323] ANSWER: TAM-verified manifest [2024-04-29 10:22:43,326] ANSWER: security: read previous location 'ssh://user@mybox.ro/./isp' [2024-04-29 10:22:43,326] ANSWER: security: read manifest timestamp '2024-04-29T00:05:19.729505' [2024-04-29 10:22:43,327] ANSWER: security: determined newest manifest timestamp as 2024-04-29T00:05:19.729505 [2024-04-29 10:22:43,328] ANSWER: security: repository checks ok, allowing access [2024-04-29 10:22:43,328] ANSWER: Creating archive at "ssh://user@mybox.ro/./isp/::isp-2024-04-29T10:22:40" [2024-04-29 10:22:43,380] ANSWER: Verified integrity of /home/tmp/ce532d87c91d01bf89453e606b34ae90834fd58d103037b0fd58c1d50c28a64a/chunks [2024-04-29 10:22:43,380] ANSWER: Reading files cache ... [2024-04-29 10:22:46,663] ANSWER: Verified integrity of /home/tmp/ce532d87c91d01bf89453e606b34ae90834fd58d103037b0fd58c1d50c28a64a/files [2024-04-29 10:22:46,664] ANSWER: security: read previous location 'ssh://user@mybox.ro/./isp' <snip> [2024-04-29 10:23:45,008] ANSWER: M /home/mariuspana/logs/php_log [2024-04-29 10:25:10,404] ANSWER: A /home/xxx/homes/xxx/Maildir/cur/1714374005.5945_0.isp:2,S [2024-04-29 10:25:14,476] ANSWER: M /home/backupisp/backuplog [2024-04-29 10:25:15,189] ANSWER: Remote: check_free_space: required bytes 751335498, free bytes 20462240726528 [2024-04-29 10:25:15,274] ANSWER: security: saving state for ce532d87c91d01bf89453e606b34ae90834fd58d103037b0fd58c1d50c28a64a to /root/.config/borg/security/ce532d87c91d01bf89453e606b34ae90834fd58d103037b0fd58c1d50c28a64a [2024-04-29 10:25:15,275] ANSWER: security: current location ssh://user@mybox.ro/./isp [2024-04-29 10:25:15,275] ANSWER: security: key type 2 [2024-04-29 10:25:15,275] ANSWER: security: manifest timestamp 2024-04-29T07:25:15.078812 [2024-04-29 10:25:20,056] ANSWER: RemoteRepository: 177.56 MB bytes sent, 7.83 kB bytes received, 630 messages sent [2024-04-29 10:25:20,154] ANSWER: terminating with warning status, rc 1 ``` The <snip> included lines similar to these: ``` A /home/someone/homes/some.user/Maildir/cur/1714374173.7738_0.isp:2,RS A /home/someone/homes/some.user/Maildir/cur/1714374466.10428_0.isp:2,S M /home/someone/homes/some.user//.spamassassin/bayes_toks M /home/someone/homes/some.user/.spamassassin/bayes_seen A /home/someone/homes/some.user//Maildir/.Sent/cur/1714375951.M51276P21202.isp,S=6559,W=6742:2,S A /home/someone/homes/some.user/Maildir/cur/1714375641.18804_0.isp:2,S ```

witten commented

2024-04-29 16:35:28 +00:00

Okay, something's definitely going wrong between these two log lines:

[2024-04-29 10:21:45,129] INFO: ssh://user@mybox.ro/./isp/: Dumping MySQL databases
[2024-04-29 10:21:45,209] WARNING: ssh://user@mybox.ro/./isp/: Ignoring configured "read_special" value of false, as true is needed for database hooks.

Here's what I would expect to see if MySQL databases were actually getting dumped:

example: Dumping MySQL databases
example: Dumping MySQL database "test2" to /root/.borgmatic/mysql_databases/localhost/test2
MYSQL_PWD=*** mysqldump --add-drop-database --host localhost --protocol tcp --user root --databases test2 --result-file /root/.borgmatic/mysql_databases/localhost/test2
example: Ignoring configured "read_special" value of false, as true is needed for database hooks.

Specifically, I'd expect to see the Dumping MySQL database "name" to ... log entry even if there were issues calling the MySQL/MariaDB dump binary or it never exited.

Looking at the code, here are things I can think of that might result in that log entry never getting logged:

Empty MySQL database configuration in your borgmatic configuration file. What does your config actually look like there?
A configured database name of all but with borgmatic somehow unable to resolve that to actual database names to dump, maybe due to that old MariaDB version. Are you using all? If so, does borgmatic work if you instead use actual database names?

Thanks for your patience here.

Okay, something's definitely going wrong between these two log lines: ``` [2024-04-29 10:21:45,129] INFO: ssh://user@mybox.ro/./isp/: Dumping MySQL databases [2024-04-29 10:21:45,209] WARNING: ssh://user@mybox.ro/./isp/: Ignoring configured "read_special" value of false, as true is needed for database hooks. ``` Here's what I would expect to see if MySQL databases were actually getting dumped: ``` example: Dumping MySQL databases example: Dumping MySQL database "test2" to /root/.borgmatic/mysql_databases/localhost/test2 MYSQL_PWD=*** mysqldump --add-drop-database --host localhost --protocol tcp --user root --databases test2 --result-file /root/.borgmatic/mysql_databases/localhost/test2 example: Ignoring configured "read_special" value of false, as true is needed for database hooks. ``` Specifically, I'd expect to see the `Dumping MySQL database "name" to ...` log entry *even if* there were issues calling the MySQL/MariaDB dump binary or it never exited. Looking at the code, here are things I can think of that might result in that log entry never getting logged: * Empty MySQL database configuration in your borgmatic configuration file. What does your config actually look like there? * A configured database name of `all` but with borgmatic somehow unable to resolve that to actual database names to dump, maybe due to that old MariaDB version. Are you using `all`? If so, does borgmatic work if you instead use actual database names? Thanks for your patience here.

mariuspana commented

2024-04-29 18:28:25 +00:00

I do see output similar to what you showed, using all or a single database but on stdout not in the logfile specified with --log-file.

Dumping MySQL database "mariuspana" to /home/tmp/mysql_databases/localhost/mariuspana
mysqldump --single-transaction --max-allowed-packet=2G --add-drop-database --databases mariuspana --result-file /home/tmp/mysql_databases/localhost/mariuspana
gnoring configured "read_special" value of false, as true is needed for database hooks.
Collecting special file paths
...

While looking at the output I noticed where it was dumping to, or at least creating the named pipe, and realised that I am also backing up /home. I had borg_source_directory = /home/tmp ... I removed that and I now have a root/.borgmatic/mysql_databases folder with my databases. I think I remember reading that borg excludes its temporary directories automatically if they are included in the backup paths, I should have realised that it would also exclude the named pipe and mysqldums.

Man do I feel silly. Im not sure what determined me to set the borg_source_directory in the first place.

I apologise for my haste in opening this issue and really appreciate your time.

I do see output similar to what you showed, using all or a single database but on stdout not in the logfile specified with --log-file. ``` Dumping MySQL database "mariuspana" to /home/tmp/mysql_databases/localhost/mariuspana mysqldump --single-transaction --max-allowed-packet=2G --add-drop-database --databases mariuspana --result-file /home/tmp/mysql_databases/localhost/mariuspana gnoring configured "read_special" value of false, as true is needed for database hooks. Collecting special file paths ... ``` While looking at the output I noticed where it was dumping to, or at least creating the named pipe, and realised that I am also backing up /home. I had borg_source_directory = /home/tmp ... I removed that and I now have a root/.borgmatic/mysql_databases folder with my databases. I think I remember reading that borg excludes its temporary directories automatically if they are included in the backup paths, I _should_ have realised that it would also exclude the named pipe and mysqldums. Man do I feel silly. Im not sure what determined me to set the borg_source_directory in the first place. I apologise for my haste in opening this issue and really appreciate your time.

mariuspana closed this issue

2024-04-29 18:28:25 +00:00

witten commented

2024-04-29 18:32:00 +00:00

Ah that's a very tricky failure mode! I'm really glad to hear you figured it out though and that the fix was so "easy." And yeah, --log-file will only include log entries at a certain verbosity unless you increase its log level with --log-file-verbosity.

Ah that's a very tricky failure mode! I'm really glad to hear you figured it out though and that the fix was so "easy." And yeah, `--log-file` will only include log entries at a certain verbosity unless you increase its log level with `--log-file-verbosity`.

👍 1

Sign in to join this conversation.