Missing source_dir = fatal error #121

New Issue

patricko · 2018-12-20T15:55:03Z

patricko commented

2018-12-20 15:55:03 +00:00

On borgmatic 1.2.0

I have two confs in borgmatic.d/

If I run borgmatic with a missing source dir, I get this error:

/my_scripts: [Errno 2] No such file or directory: '/my_scripts'

Ok fine, but other directories exist.

So the rest of the output is hard to decipher but I get this too:

Command '('borg', 'create', '/my_backups/borg_repos/files::{hostname}-{now}', '/my_scripts', '/my_programs', '/etc', '/var/log/autoxtrabackup.log',
'/var/log/sudo.log', '/var/log/daemon.log', '/var/log/alternatives.log', '/var/log/user.log', '/var/log/mail.log', '/var/log/dpkg.log', '/var/log/au
th.log', '/var/log/auth.log.1', '/var/log/daemon.log.1', '/var/log/alternatives.log.1', '/var/log/mail.log.1', '/var/log/dpkg.log.1', '/var/log/user
.log.1', '/var/log/apt/history.log', '/var/log/apt/term.log', '/var/log/unattended-upgrades/unattended-upgrades.log', '/var/log/unattended-upgrades/
unattended-upgrades-shutdown.log', '/var/log/unattended-upgrades/unattended-upgrades-dpkg.log', '/var/log/apache2/other_vhosts_access.log', '/var/lo
g/apache2/access.log', '/var/log/apache2/error.log', '/var/log/mysql/slow_queries.log', '/var/log/mysql/mysql.log', '/var/log/mysql/error.log', '/va
r/log/apache2/access.log.1', '/var/log/apache2/error.log.1', '/var/log/apache2/other_vhosts_access.log.1', '/home/my_test', '--exclude-from', '/tmp/
tmp_67cbzek', '--compression', 'lzma,6', '--remote-ratelimit', '100', '--one-file-system', '--files-cache', 'ctime,size,inode', '--umask', '77', '--
lock-wait', '5', '--info', '--stats')' returned non-zero exit status 1.

and then the second config is not ran

So I guess a missing source_dir fails everything, but that doesn't seem right, it should still work for other dirs, no?

note: also what about some colors in the verbose output??

On borgmatic 1.2.0 I have two confs in borgmatic.d/ If I run borgmatic with a missing source dir, I get this error: `/my_scripts: [Errno 2] No such file or directory: '/my_scripts'` Ok fine, but other directories exist. So the rest of the output is hard to decipher but I get this too: ``` Command '('borg', 'create', '/my_backups/borg_repos/files::{hostname}-{now}', '/my_scripts', '/my_programs', '/etc', '/var/log/autoxtrabackup.log', '/var/log/sudo.log', '/var/log/daemon.log', '/var/log/alternatives.log', '/var/log/user.log', '/var/log/mail.log', '/var/log/dpkg.log', '/var/log/au th.log', '/var/log/auth.log.1', '/var/log/daemon.log.1', '/var/log/alternatives.log.1', '/var/log/mail.log.1', '/var/log/dpkg.log.1', '/var/log/user .log.1', '/var/log/apt/history.log', '/var/log/apt/term.log', '/var/log/unattended-upgrades/unattended-upgrades.log', '/var/log/unattended-upgrades/ unattended-upgrades-shutdown.log', '/var/log/unattended-upgrades/unattended-upgrades-dpkg.log', '/var/log/apache2/other_vhosts_access.log', '/var/lo g/apache2/access.log', '/var/log/apache2/error.log', '/var/log/mysql/slow_queries.log', '/var/log/mysql/mysql.log', '/var/log/mysql/error.log', '/va r/log/apache2/access.log.1', '/var/log/apache2/error.log.1', '/var/log/apache2/other_vhosts_access.log.1', '/home/my_test', '--exclude-from', '/tmp/ tmp_67cbzek', '--compression', 'lzma,6', '--remote-ratelimit', '100', '--one-file-system', '--files-cache', 'ctime,size,inode', '--umask', '77', '-- lock-wait', '5', '--info', '--stats')' returned non-zero exit status 1. ``` and then the second config is not ran So I guess a missing source_dir fails everything, but that doesn't seem right, it should still work for other dirs, no? note: also what about some colors in the verbose output??

patricko commented

2018-12-22 11:48:40 +00:00

@witten

witten added the

question / support

label 2018-12-24 21:00:20 +00:00

witten commented

2018-12-24 21:01:48 +00:00

If I'm reading this correctly, I'm not sure there's a whole lot we can do about this limitation as the borgmatic level. That's because we pass the entire list of source directories to a single Borg invocation as part of creating a single backup archive, and Borg either succeeds or fails the whole invocation, based on whether any errors occur (such as the one you're seeing above).

You might consider filing a ticket on the Borg project if you'd like it to support (optionally) plowing through even if particular source directories don't exist.

Colors in the verbose output is a neat idea! I recommend filing that as a separate borgmatic ticket, ideally with some ideas on how you'd like it colored.

If I'm reading this correctly, I'm not sure there's a whole lot we can do about this limitation as the borgmatic level. That's because we pass the entire list of source directories to a single Borg invocation as part of creating a single backup archive, and Borg either succeeds or fails the whole invocation, based on whether any errors occur (such as the one you're seeing above). You might consider filing a ticket on the Borg project if you'd like it to support (optionally) plowing through even if particular source directories don't exist. Colors in the verbose output is a neat idea! I recommend filing that as a separate borgmatic ticket, ideally with some ideas on how you'd like it colored.

patricko commented

2018-12-25 11:10:15 +00:00

Original issue seems to be linked to this witten/borgmatic#116

Original issue seems to be linked to this https://projects.torsion.org/witten/borgmatic/issues/116

witten commented

2018-12-25 18:04:02 +00:00

It's certainly related to #116, but the difference is that with separate config files, borgmatic issues separate Borg invocations. And so we have a little more latitude to handle errors together or independently. But with a single config file and thus a single Borg invocation, we can't selectively bypass failure on individual source directories. Please let me know if I'm missing something here.. Otherwise I think I'll need to close this! Thanks.

patricko commented

2018-12-29 14:51:30 +00:00

I think you mixed two of my issues together witten, but not sure.

In this one I'm just reporting that it stops after 1 configuration failed.

I think you mixed two of my issues together witten, but not sure. In this one I'm just reporting that it stops after 1 configuration failed.

witten commented

2018-12-29 21:13:56 +00:00

I see.. So you have multiple configuration files? And borgmatic exits after one of the configuration files fails? If so, that should be fixed by #116 if you upgrade to borgmatic 1.2.14.

If that's not the case, then can you provide a little more detail about your configuration and expected versus actual behavior? Thanks.

I see.. So you have multiple configuration files? And borgmatic exits after one of the configuration files fails? If so, that should be fixed by #116 if you upgrade to borgmatic 1.2.14. If that's not the case, then can you provide a little more detail about your configuration and expected versus actual behavior? Thanks.

patricko commented

2019-01-08 16:10:27 +00:00

Ok, not sure how to retest this but yes, I updated to 1.2.14 since then so it should be fine?

patricko closed this issue

2019-01-08 16:10:27 +00:00

witten commented

2019-01-09 04:56:15 +00:00

If you wanted to try to reproduce this, you could add an intentional error into one of your configuration files so as to induce an error when processing that config file. For instance, changing the repository to a path that doesn't exist.

But yes, upgrading to 1.2.14 should hopefully fix that case.

If you wanted to try to reproduce this, you could add an intentional error into one of your configuration files so as to induce an error when processing that config file. For instance, changing the repository to a path that doesn't exist. But yes, upgrading to 1.2.14 should hopefully fix that case.

patricko commented

2019-01-11 15:19:53 +00:00

Yes, I confirm it's working. But at the end it says the error right?

/etc/borgmatic.d/config_for_databases.yaml: Error running configuration file

Yes, I confirm it's working. But at the end it says the error right? /etc/borgmatic.d/config_for_databases.yaml: Error running configuration file

witten commented

2019-01-11 21:51:38 +00:00

That's correct!

Sign in to join this conversation.