Please add option to dump databases without streaming #587

New Issue

Daniel · 2022-09-18T05:22:55Z

Daniel commented

2022-09-18 05:22:55 +00:00

What I'm trying to do and why

Streaming of database backups was added to resolve #258, however, having to use --read-special is a major disadvantage. Today, borg got stuck in what was essentially an infinite loop consuming 100% of one CPU core and a large amount of bandwidth (both the client system and the backup server have 10Gbps networking).

I back up /var/lib/docker/volumes (storage data for Docker containers), and one of the new Docker containers I created had a device file in its volume. This was on a VPS, which generally have shared resources and strict CPU usage restrictions, for example you can't 100% of one core for a long period of time.

Because of this, I'd like the ability to disable streaming so I can avoid using --read-special. My databases are less than 1GB total, so streaming doesn't provide a major advantage.

To work around this issue, I'm currently using a before_backup hook to manually dump the MySQL databases:

mysql --defaults-extra-file=/etc/mysql/backup.cnf --skip-column-names -e 'show databases' | while read dbname; do mysqldump --defaults-extra-file=/etc/mysql/backup.cnf --single-transaction --complete-insert --add-drop-table --quick --quote-names "$dbname" > /var/backups/mysql/sqldump/$dbname.sql; done

Environment

borgmatic version: 1.6.3

borgmatic installation method: Debian package

Borg version: 1.1.16

Python version: 3.9.2

#### What I'm trying to do and why Streaming of database backups was added to resolve #258, however, having to use `--read-special` is a major disadvantage. Today, borg got stuck in what was essentially an infinite loop consuming 100% of one CPU core and a large amount of bandwidth (both the client system and the backup server have 10Gbps networking). I back up `/var/lib/docker/volumes` (storage data for Docker containers), and one of the new Docker containers I created had a device file in its volume. This was on a VPS, which generally have shared resources and strict CPU usage restrictions, for example you can't 100% of one core for a long period of time. Because of this, I'd like the ability to disable streaming so I can avoid using `--read-special`. My databases are less than 1GB total, so streaming doesn't provide a major advantage. To work around this issue, I'm currently using a `before_backup` hook to manually dump the MySQL databases: ``` mysql --defaults-extra-file=/etc/mysql/backup.cnf --skip-column-names -e 'show databases' | while read dbname; do mysqldump --defaults-extra-file=/etc/mysql/backup.cnf --single-transaction --complete-insert --add-drop-table --quick --quote-names "$dbname" > /var/backups/mysql/sqldump/$dbname.sql; done ``` #### Environment **borgmatic version:** 1.6.3 **borgmatic installation method:** Debian package **Borg version:** 1.1.16 **Python version:** 3.9.2

witten commented

2022-09-18 20:31:55 +00:00

Thank you for taking the time to file this—and persisting throuh user registeration! This is tough issue, and has a been recurring one. It's arguably a terrible user experience for borgmatic to cause Borg to unceremoneously hang on certain files whenever database hooks are enabled. We could very well work around the issue as you describe with a new option to disable database streaming, but my concern is that it doesn't solve the underlying user experience issue unless a user knows to enable the option. And even then, it's kind of annoying to have to enable a (non-default) option to avoid a hang on certain systems. We could make non-streaming the default, but then we give up performance (by default).

Here's some brainstorming on different compromise solutions:

Instead of Borg hanging when a "special" file is encountered, scan files ahead of time and abort "cleanly" with a clear error message that that file needs to be excluded. This has performance implications (scanning the filesystem), and it doesn't necessarily solve your problem if you want borgmatic to "just work"; it still requires user intervention to rectify.
Similar to above but when scanning the filesystem, instead of erroring on special files, auto-add them to the excludes list and log a warning for each. That way, borgmatic is hand-feeding Borg only the files that won't cause hangs. Similiar performance implications, but it doesn't require any user intervention if they're fine with the excludes.
Scan the files ahead of time, but instead of either erroring or excluding—auto-fallback from "turbo mode" streaming to "slow mode" non-streaming if special files are encountered in the scan (with a warning logged). This might be the best of both worlds, but it could be obnoxious if you expect your giant database dumps to stream and then one day they stop streaming because a new special file showed up on your filesystem.
Disable the ability to store database dumps and plain files in the same Borg archive. If you want to store both, borgmatic would run Borg twice and create one archive for databases and another for files. Maybe even you'd need two different configuration files. I'm not a big fan of this option, but it would completely sidestep the issue.
Do nothing and wait for Borg to solve the underlying hang issue. Doesn't seem like it's seen a lot of movement in the last two years, however.
Implement a timeout on the borgmatic side, so if Borg hangs, borgmatic eventually recovers. I'm not sure how feasible this is, however, given that it may be difficult for borgmatic to tell if Borg is legitimately hung or just taking a long time on something.

Let me know if you have any thoughts or other suggestions. I know this is more than you asked for when filing the ticket, so feel free to say "I don't know.. you figure it out!" if you like.

Thank you for taking the time to file this—and persisting throuh user registeration! This is tough issue, and has a been recurring one. It's arguably a terrible user experience for borgmatic to cause Borg to unceremoneously hang on certain files whenever database hooks are enabled. We could very well work around the issue as you describe with a new option to disable database streaming, but my concern is that it doesn't solve the underlying user experience issue *unless* a user knows to enable the option. And even then, it's kind of annoying to have to enable a (non-default) option to avoid a hang on certain systems. We could make non-streaming the default, but then we give up performance (by default). Here's some brainstorming on different compromise solutions: * Instead of Borg hanging when a "special" file is encountered, scan files ahead of time and abort "cleanly" with a clear error message that that file needs to be excluded. This has performance implications (scanning the filesystem), and it doesn't necessarily solve your problem if you want borgmatic to "just work"; it still requires user intervention to rectify. * Similar to above but when scanning the filesystem, instead of erroring on special files, auto-add them to the excludes list and log a warning for each. That way, borgmatic is hand-feeding Borg *only* the files that won't cause hangs. Similiar performance implications, but it doesn't require any user intervention if they're fine with the excludes. * Scan the files ahead of time, but instead of either erroring or excluding—auto-fallback from "turbo mode" streaming to "slow mode" non-streaming if special files are encountered in the scan (with a warning logged). This might be the best of both worlds, but it could be obnoxious if you expect your giant database dumps to stream and then one day they stop streaming because a new special file showed up on your filesystem. * Disable the ability to store database dumps and plain files in the same Borg archive. If you want to store both, borgmatic would run Borg twice and create one archive for databases and another for files. Maybe even you'd need two different configuration files. I'm not a big fan of this option, but it would completely sidestep the issue. * Do nothing and wait for Borg to solve [the underlying hang issue](https://github.com/borgbackup/borg/issues/5422). Doesn't seem like it's seen a lot of movement in the last two years, however. * Implement a timeout on the borgmatic side, so if Borg hangs, borgmatic eventually recovers. I'm not sure how feasible this is, however, given that it may be difficult for borgmatic to tell if Borg is legitimately hung or just taking a long time on something. Let me know if you have any thoughts or other suggestions. I know this is more than you asked for when filing the ticket, so feel free to say "I don't know.. you figure it out!" if you like.

Daniel commented

2022-09-19 02:36:29 +00:00

My thoughts:

Instead of Borg hanging when a "special" file is encountered, scan files ahead of time and abort "cleanly" with a clear error message

This is fine, but backups that suddenly stop working due to changes on the file system aren't ideal.

Similar to above but when scanning the filesystem, instead of erroring on special files, auto-add them to the excludes list and log a warning for each.

I think I'd be happy with this solution. It seems like a reasonable compromise. I wonder how much extra IO this would add to the backup process though. I was going to say that I wonder if Borgbackup could add this option, however I'm sure its developers would just say to stop using the --read-special option.

Scan the files ahead of time, but instead of either erroring or excluding—auto-fallback from "turbo mode" streaming to "slow mode" non-streaming if special files are encountered in the scan (with a warning logged).

This is also not ideal for the reason that the first solution isn't ideal - a change on the file system has a big impact on an unrelated part of the backup (like you mentioned). SQL takes up more space than the native storage mechanism used by the DB, so for larger DBs (say over ~1TB in size), storing the dump on disk might not even be possible.

Disable the ability to store database dumps and plain files in the same Borg archive.

This is doable. I wouldn't be happy with this as I'd need to remember about two different backup repos, but maybe it wouldn't be too bad.

Implement a timeout on the borgmatic side, so if Borg hangs, borgmatic eventually recovers.

This is not ideal because borg would be using 100% of one core until the timeout is hit.

Another issue is this: What if I want to back up the databases, and back up the metadata for the device/pipe descriptor itself? Borg's default mode (without --read-special) backs up the metadata required by mknod, mkfifo, etc, which is useful in some cases, as it'll re-create the node on restore. Out of all the options in your list, I think only the "Disable the ability to store database dumps and plain files in the same Borg archive" one would solve this use case, and I don't feel like that's a viable solution.

Some other potential options:

Change the behaviour based on the existing read_special option in Borgmatic's config, to also enable database streaming only when it's enabled. This is not really ideal as it's adding another implicit behaviour to the config.
Add a stream config option for database streaming, and throw an error if it's used without read_special: true. This removes the confusing implicit behaviour of read_special today, and requires the user to explicitly change both options (and thus be more aware of what's happening). I guess this is the same as what I originally suggested, but with validation.
Stop using a "special" file for streaming DB backups. I wonder if there's another way to do it.

My thoughts: > *Instead of Borg hanging when a "special" file is encountered, scan files ahead of time and abort "cleanly" with a clear error message* This is *fine*, but backups that suddenly stop working due to changes on the file system aren't ideal. > Similar to above but when scanning the filesystem, instead of erroring on special files, auto-add them to the excludes list and log a warning for each. I think I'd be happy with this solution. It seems like a reasonable compromise. I wonder how much extra IO this would add to the backup process though. I was going to say that I wonder if Borgbackup could add this option, however I'm sure its developers would just say to stop using the `--read-special` option. > Scan the files ahead of time, but instead of either erroring or excluding—auto-fallback from "turbo mode" streaming to "slow mode" non-streaming if special files are encountered in the scan (with a warning logged). This is also not ideal for the reason that the first solution isn't ideal - a change on the file system has a big impact on an unrelated part of the backup (like you mentioned). SQL takes up more space than the native storage mechanism used by the DB, so for larger DBs (say over ~1TB in size), storing the dump on disk might not even be possible. > Disable the ability to store database dumps and plain files in the same Borg archive. This is doable. I wouldn't be happy with this as I'd need to remember about two different backup repos, but maybe it wouldn't be too bad. > Implement a timeout on the borgmatic side, so if Borg hangs, borgmatic eventually recovers. This is not ideal because borg would be using 100% of one core until the timeout is hit. --- Another issue is this: What if I want to back up the databases, *and* back up the metadata for the device/pipe descriptor itself? Borg's default mode (without `--read-special`) backs up the metadata required by `mknod`, `mkfifo`, etc, which is useful in some cases, as it'll re-create the node on restore. Out of all the options in your list, I think only the "*Disable the ability to store database dumps and plain files in the same Borg archive*" one would solve this use case, and I don't feel like that's a viable solution. Some other potential options: - Change the behaviour based on the existing `read_special` option in Borgmatic's config, to also enable database streaming only when it's enabled. This is not really ideal as it's adding another implicit behaviour to the config. - Add a `stream` config option for database streaming, and throw an error if it's used without `read_special: true`. This removes the confusing implicit behaviour of `read_special` today, and requires the user to explicitly change both options (and thus be more aware of what's happening). I guess this is the same as what I originally suggested, but with validation. - Stop using a "special" file for streaming DB backups. I wonder if there's another way to do it.

witten commented

2022-09-19 04:31:28 +00:00

This is fine, but backups that suddenly stop working due to changes on the file system aren't ideal.

Agreed. That's basically how things work today!

Similar to above but when scanning the filesystem, instead of erroring on special files, auto-add them to the excludes list and log a warning for each.

I think I'd be happy with this solution. It seems like a reasonable compromise. I wonder how much extra IO this would add to the backup process though.

Okay, I might play around with this approach and see how bad it is. (I tried borg create --dry-run --filter bcf in an attempt to get it to list just special files it found, but that apparently doesn't work.)

Another issue is this: What if I want to back up the databases, and back up the metadata for the device/pipe descriptor itself? Borg's default mode (without --read-special) backs up the metadata required by mknod, mkfifo, etc, which is useful in some cases, as it'll re-create the node on restore. Out of all the options in your list, I think only the "Disable the ability to store database dumps and plain files in the same Borg archive" one would solve this use case, and I don't feel like that's a viable solution.

That does appear to be the case. You can achieve this today though by separating out the databases and files to backup into two separate borgmatic configuration files—which would result in a separate backup archive for each.

Change the behaviour based on the existing read_special option in Borgmatic's config, to also enable database streaming only when it's enabled. This is not really ideal as it's adding another implicit behaviour to the config.

Agreed.

Add a stream config option for database streaming, and throw an error if it's used without read_special: true. This removes the confusing implicit behaviour of read_special today, and requires the user to explicitly change both options (and thus be more aware of what's happening). I guess this is the same as what I originally suggested, but with validation.

Yeah, that's certainly more explicit, so that's nice. But it's still got the problem of a user having to know to either enable the stream option (for performance) or disable it (for not hanging on special files). I could see making it a required option with no default when using database hooks at least, so the user has to make that explicit choice. But still not ideal. (It's looking increasingly like we're going to have to choose from several non-ideal options.)

Stop using a "special" file for streaming DB backups. I wonder if there's another way to do it.

There is, but it'll only do a single file / database dump at a time: Stream via Borg's stdin. The whole named pipe streaming thing with borgmatic is a way to work around that stdin limitation, so we can stuff multiple database dumps into a single Borg archive. Which suggests another option: Switch to Borg's stdin feature for streaming, drop the use of --read-special, and only allow a single database per Borg archive. The configuration file would probably need to change to support that (since you couldn't just assume the same configured archive prefix for each database), and I'm not sure which of the archives the plain files would go in. I also have no idea how dumping "all" databases would work with this approach...

> This is fine, but backups that suddenly stop working due to changes on the file system aren't ideal. Agreed. That's basically how things work today! > > Similar to above but when scanning the filesystem, instead of erroring on special files, auto-add them to the excludes list and log a warning for each. > I think I'd be happy with this solution. It seems like a reasonable compromise. I wonder how much extra IO this would add to the backup process though. Okay, I might play around with this approach and see how bad it is. (I tried `borg create --dry-run --filter bcf` in an attempt to get it to list just special files it found, but that apparently doesn't work.) > Another issue is this: What if I want to back up the databases, and back up the metadata for the device/pipe descriptor itself? Borg's default mode (without --read-special) backs up the metadata required by mknod, mkfifo, etc, which is useful in some cases, as it'll re-create the node on restore. Out of all the options in your list, I think only the "Disable the ability to store database dumps and plain files in the same Borg archive" one would solve this use case, and I don't feel like that's a viable solution. That does appear to be the case. You can achieve this today though by separating out the databases and files to backup into two separate borgmatic configuration files—which would result in a separate backup archive for each. > * Change the behaviour based on the existing read_special option in Borgmatic's config, to also enable database streaming only when it's enabled. This is not really ideal as it's adding another implicit behaviour to the config. Agreed. > * Add a stream config option for database streaming, and throw an error if it's used without read_special: true. This removes the confusing implicit behaviour of read_special today, and requires the user to explicitly change both options (and thus be more aware of what's happening). I guess this is the same as what I originally suggested, but with validation. Yeah, that's certainly more explicit, so that's nice. But it's still got the problem of a user having to know to either enable the `stream` option (for performance) or disable it (for not hanging on special files). I could see making it a required option with no default when using database hooks at least, so the user has to make that explicit choice. But still not ideal. (It's looking increasingly like we're going to have to choose from several non-ideal options.) > * Stop using a "special" file for streaming DB backups. I wonder if there's another way to do it. There is, but it'll only do a single file / database dump at a time: Stream via Borg's stdin. The whole named pipe streaming thing with borgmatic is a way to work around that stdin limitation, so we can stuff multiple database dumps into a single Borg archive. Which suggests another option: Switch to Borg's stdin feature for streaming, drop the use of `--read-special`, and only allow a single database per Borg archive. The configuration file would probably need to change to support that (since you couldn't just assume the same configured archive prefix for each database), and I'm not sure which of the archives the plain files would go in. I also have no idea how dumping "all" databases would work with this approach...

witten commented

2022-09-19 04:35:37 +00:00

Okay, I might play around with this approach and see how bad it is. (I tried borg create --dry-run --filter bcf in an attempt to get it to list just special files it found, but that apparently doesn't work.)

I just realized this is totally not possible without borgmatic reimplementing full parsing and logic support for Borg patterns files, which can get pretty darn complex...

Ugh!

> Okay, I might play around with this approach and see how bad it is. (I tried borg create --dry-run --filter bcf in an attempt to get it to list just special files it found, but that apparently doesn't work.) I just realized this is totally not possible without borgmatic reimplementing full parsing and logic support for Borg patterns files, which can get [pretty darn complex](https://borgbackup.readthedocs.io/en/stable/usage/help.html#borg-help-patterns)... Ugh!

witten commented

2022-09-19 04:44:30 +00:00

Okay, no, here's an approach for borgmatic that might work:

Run borg create --dry-run --list along with all source directories, patterns, etc. passed to Borg.
Parse out the list of file paths that Borg displays.
Loop through those paths (omitting borgmatic's own database streaming named pipes) to find any special files (do a "stat" on each). Add them to excludes (issuing warning logs).
Run the real, non-dry-run borg create command to make a backup of files and database dumps. No hangs!

(Sorry for the comment spam!)

EDIT: For my own reference, here's apparently how to test a path for specialness in Python:

import os
import stat


def is_special(path):
    mode = os.stat(path).st_mode
    return stat.S_ISCHR(mode) or stat.S_ISBLK(mode) or stat.S_ISFIFO(mode)

Okay, no, here's an approach for borgmatic that might work: 1. Run `borg create --dry-run --list` along with all source directories, patterns, etc. passed to Borg. 2. Parse out the list of file paths that Borg displays. 3. Loop through those paths (omitting borgmatic's own database streaming named pipes) to find any special files (do a "stat" on each). Add them to excludes (issuing warning logs). 4. Run the real, non-dry-run `borg create` command to make a backup of files and database dumps. No hangs! (Sorry for the comment spam!) EDIT: For my own reference, here's apparently how to test a path for specialness in Python: ```python import os import stat def is_special(path): mode = os.stat(path).st_mode return stat.S_ISCHR(mode) or stat.S_ISBLK(mode) or stat.S_ISFIFO(mode) ```

witten referenced this issue from a commit

2022-09-20 20:52:29 +00:00

Warn when ignoring a configured "read_special" value of false, as true is needed when database hooks are enabled (#587).

witten commented

2022-09-21 16:19:26 +00:00

Well, it's not pretty, but I have it basically working and auto-excluding special files outside of ~/.borgmatic:

# borgmatic -c test.yaml -v 2 create 
...
1.2.borg: Dumping PostgreSQL database test to /root/.borgmatic/postgresql_databases/172.17.0.2/test
pg_dump --no-password --clean --if-exists --host 172.17.0.2 --username test --format custom test > /root/.borgmatic/postgresql_databases/172.17.0.2/test
borg create --one-file-system --read-special 1.2.borg::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /root/.borgmatic /root/tmp/subdir --dry-run --list
1.2.borg: Excluding special files to prevent Borg from hanging: /root/tmp/subdir/hang
borg create --one-file-system --read-special 1.2.borg::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /root/.borgmatic /root/tmp/subdir --exclude-from /tmp/tmp5yi451zl --debug --show-rc
...
test.yaml: Successfully ran configuration file

Still needs test coverage and more manual testing though.

Well, it's not pretty, but I have it basically working and auto-excluding special files outside of `~/.borgmatic`: ``` # borgmatic -c test.yaml -v 2 create ... 1.2.borg: Dumping PostgreSQL database test to /root/.borgmatic/postgresql_databases/172.17.0.2/test pg_dump --no-password --clean --if-exists --host 172.17.0.2 --username test --format custom test > /root/.borgmatic/postgresql_databases/172.17.0.2/test borg create --one-file-system --read-special 1.2.borg::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /root/.borgmatic /root/tmp/subdir --dry-run --list 1.2.borg: Excluding special files to prevent Borg from hanging: /root/tmp/subdir/hang borg create --one-file-system --read-special 1.2.borg::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /root/.borgmatic /root/tmp/subdir --exclude-from /tmp/tmp5yi451zl --debug --show-rc ... test.yaml: Successfully ran configuration file ``` Still needs test coverage and more manual testing though.

witten referenced this issue from a commit

2022-10-03 19:58:29 +00:00

When the "read_special" option is true or database hooks are enabled, auto-exclude special files for a "create" action to prevent Borg from hanging (#587).

witten commented

2022-10-03 19:59:19 +00:00

Okay, these changes are pushed to master now and will be part of the next release. Thanks for filing this!

witten closed this issue

2022-10-03 19:59:19 +00:00

witten commented

2022-10-06 17:28:00 +00:00

It occurs to me this approach is problematic, because any users who explicitly enable read_special because they actually want Borg to, well, read special files will get their wishes thwarted... because borgmatic will then go and exclude those special files from the backup.

I'm not sure about a solution here other than backing out the change, which is a non-solution. I'll reopen the ticket.

It occurs to me this approach is problematic, because any users who explicitly enable `read_special` because they actually want Borg to, well, read special files will get their wishes thwarted... because borgmatic will then go and exclude those special files from the backup. I'm not sure about a solution here other than backing out the change, which is a non-solution. I'll reopen the ticket.

witten reopened this issue

2022-10-06 17:28:00 +00:00

witten commented

2022-10-06 18:08:41 +00:00

Okay, the "fix" wasn't too bad. Here's the new logic: If database hooks are enabled, the special file exclusion kicks in. But if the user explicitly sets read_special to true (whether or not database hooks are enabled), then the special file exclusion is skipped. The idea is that the user setting read_special to true indicates they know what they're doing—specifically, requesting inclusion of special files.

Okay, the "fix" wasn't too bad. Here's the new logic: If database hooks are enabled, the special file exclusion kicks in. But if the user explicitly sets `read_special` to true (whether or not database hooks are enabled), then the special file exclusion is skipped. The idea is that the user setting `read_special` to true indicates they know what they're doing—specifically, requesting inclusion of special files.

witten closed this issue

2022-10-06 18:08:41 +00:00

witten referenced this issue from a commit

2022-10-06 18:08:46 +00:00

Skip auto-exclusion of special files when user explicitly sets read_special to true (#587).

witten commented

2022-10-12 18:02:59 +00:00

Just released in borgmatic 1.7.3!

bjo commented

2022-10-14 09:22:16 +00:00

It seems borgmatic fails now on every broken symlink.

ssh://user@user.your-storagebox.de/./mail: Creating archive
ssh://user@user.your-storagebox.de/./mail: Calling postgresql_databases hook function remove_database_dumps
ssh://user@user.your-storagebox.de/./mail: Removing PostgreSQL database dumps
ssh://user@user.your-storagebox.de/./mail: Calling mysql_databases hook function remove_database_dumps
ssh://user@user.your-storagebox.de/./mail: Removing MySQL database dumps
ssh://user@user.your-storagebox.de/./mail: Calling mongodb_databases hook function remove_database_dumps
ssh://user@user.your-storagebox.de/./mail: Removing MongoDB database dumps
ssh://user@user.your-storagebox.de/./mail: Calling postgresql_databases hook function dump_databases
ssh://user@user.your-storagebox.de/./mail: Dumping PostgreSQL databases
ssh://user@user.your-storagebox.de/./mail: Dumping PostgreSQL database all to /root/.borgmatic/postgresql_databases/localhost/all
pg_dumpall --no-password --clean --if-exists --username postgres > /root/.borgmatic/postgresql_databases/localhost/all
ssh://user@user.your-storagebox.de/./mail: Collecting special file paths
borg create --exclude-from /tmp/tmp527su9m7 --exclude-caches --exclude-if-present .nobackup --compression auto,zstd --one-file-system --read-special --files-cache ctime,size,inode ssh://user@user.your-storagebox.de/./mail::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /data /etc /home /root/.borgmatic /var/lib/dendrite /var/lib/prosody /var/lib/rspamd /var/lib/smokeping /var/lib/vaultwarden --dry-run --list
ssh://user@user.your-storagebox.de/./mail: Error running actions for repository
[Errno 2] No such file or directory: '/etc/systemd/system/multi-user.target.wants/cronie.service'
/etc/borgmatic/config.yaml: Running command for on-error hook
echo "`date` - Error while creating a backup."

It seems borgmatic fails now on every broken symlink. ``` ssh://user@user.your-storagebox.de/./mail: Creating archive ssh://user@user.your-storagebox.de/./mail: Calling postgresql_databases hook function remove_database_dumps ssh://user@user.your-storagebox.de/./mail: Removing PostgreSQL database dumps ssh://user@user.your-storagebox.de/./mail: Calling mysql_databases hook function remove_database_dumps ssh://user@user.your-storagebox.de/./mail: Removing MySQL database dumps ssh://user@user.your-storagebox.de/./mail: Calling mongodb_databases hook function remove_database_dumps ssh://user@user.your-storagebox.de/./mail: Removing MongoDB database dumps ssh://user@user.your-storagebox.de/./mail: Calling postgresql_databases hook function dump_databases ssh://user@user.your-storagebox.de/./mail: Dumping PostgreSQL databases ssh://user@user.your-storagebox.de/./mail: Dumping PostgreSQL database all to /root/.borgmatic/postgresql_databases/localhost/all pg_dumpall --no-password --clean --if-exists --username postgres > /root/.borgmatic/postgresql_databases/localhost/all ssh://user@user.your-storagebox.de/./mail: Collecting special file paths borg create --exclude-from /tmp/tmp527su9m7 --exclude-caches --exclude-if-present .nobackup --compression auto,zstd --one-file-system --read-special --files-cache ctime,size,inode ssh://user@user.your-storagebox.de/./mail::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /data /etc /home /root/.borgmatic /var/lib/dendrite /var/lib/prosody /var/lib/rspamd /var/lib/smokeping /var/lib/vaultwarden --dry-run --list ssh://user@user.your-storagebox.de/./mail: Error running actions for repository [Errno 2] No such file or directory: '/etc/systemd/system/multi-user.target.wants/cronie.service' /etc/borgmatic/config.yaml: Running command for on-error hook echo "`date` - Error while creating a backup." ```

witten commented

2022-10-14 16:32:04 +00:00

Confirmed with a local repro! As per usual, thank you for reporting this.

witten reopened this issue

2022-10-14 16:32:05 +00:00

witten commented

2022-10-14 16:42:38 +00:00

Fixed in master as part of #596. This fix will be part of the next release (hopefully soon).

witten closed this issue

2022-10-14 16:42:39 +00:00

witten commented

2022-10-14 23:28:54 +00:00