Post backup command returns non-zero exit status 255 #871

New Issue

auanasgheps · 2024-05-19T12:17:58Z

auanasgheps commented

2024-05-19 12:17:58 +00:00

What I'm trying to do and why

Hi all,
I use borgmatic for my backup routine to a remote server using SSH. It's been almost 4 years, great software.

When the backup ends, borgmatic executes this command to shut down the remote server:

Note: elements with [] have been sanitized

    after_backup:
        - /usr/sbin/borg/email-backup.sh "{configuration_filename}" "{repository}"
        - echo "Backup tasks have been completed and the remote server will be shut
          down."
        - ssh [username]@[serverdomain] -i /etc/ssh/[sshkey] -p [port] 'sudo
          /sbin/poweroff'

This used to work perfectly, but since a while this command is returning the following:

[2024-05-19 14:00:34,803] WARNING: Command 'ssh [username]@[serverdomain] -i /etc/ssh/[sshkey] -p [port] 'sudo /sbin/poweroff'' returned non-zero exit status 255.

The dommand WORKS and it shuts down the remote system as expected.

If I run the command manually in a shell, I don't get any exit status, it just executes, then I get the prompt for a new command.

Am I missing something here?

This breaks the borgmatic logic, first thinking that everything went well (backup ended), then an "error" occurs because it thinks this command din't execute correctly.

The remote server is a Synology NAS running DSM 7.

Steps to reproduce

    after_backup:
        - /usr/sbin/borg/email-backup.sh "{configuration_filename}" "{repository}"
        - echo "Backup tasks have been completed and the remote server will be shut
          down."
        - ssh [username]@[serverdomain] -i /etc/ssh/[sshkey] -p [port] 'sudo
          /sbin/poweroff'

Actual behavior

[2024-05-19 14:00:00,582] INFO: /etc/borgmatic/config.yaml: Running 3 commands for post-backup hook
[2024-05-19 14:00:00,602] WARNING: Backup tasks have been completed and the remote server will be shut down.
[2024-05-19 14:00:34,802] WARNING: ssh://[username]@[serverdomain]/./backup: Error running actions for repository
[2024-05-19 14:00:34,803] WARNING: Command 'ssh [username]@[serverdomain] -i /etc/ssh/[sshkey] -p [port] 'sudo /sbin/poweroff'' returned non-zero exit status 255.

Expected behavior

Previous behaviour:

[2023-04-22 14:19:33,642] INFO: /etc/borgmatic/config.yaml: Running 2 commands for post-everything hook
[2023-04-22 14:19:33,651] WARNING: Tasks have been completed and the remote server will be shut down.
[2023-04-22 14:19:36,123] INFO:
[2023-04-22 14:19:36,124] INFO: summary:
[2023-04-22 14:19:33,641] INFO: /etc/borgmatic/config.yaml: Successfully ran configuration file

Other notes / implementation ideas

No response

borgmatic version

1.7.4

borgmatic installation method

bullseye-backports

Borg version

1.2.3

Python version

Python 3.9.2

Database version (if applicable)

N/A

Operating system and version

Debian bullseye (OpenMediaVault 6)

### What I'm trying to do and why Hi all, I use borgmatic for my backup routine to a remote server using SSH. It's been almost 4 years, great software. When the backup ends, borgmatic executes this command to shut down the remote server: Note: elements with [] have been sanitized ```yaml after_backup: - /usr/sbin/borg/email-backup.sh "{configuration_filename}" "{repository}" - echo "Backup tasks have been completed and the remote server will be shut down." - ssh [username]@[serverdomain] -i /etc/ssh/[sshkey] -p [port] 'sudo /sbin/poweroff' ``` This used to work perfectly, but since a while this command is returning the following: `[2024-05-19 14:00:34,803] WARNING: Command 'ssh [username]@[serverdomain] -i /etc/ssh/[sshkey] -p [port] 'sudo /sbin/poweroff'' returned non-zero exit status 255.` The dommand WORKS and it shuts down the remote system as expected. If I run the command manually in a shell, I don't get any exit status, it just executes, then I get the prompt for a new command. Am I missing something here? This breaks the borgmatic logic, first thinking that everything went well (backup ended), then an "error" occurs because it thinks this command din't execute correctly. The remote server is a Synology NAS running DSM 7. ### Steps to reproduce ```yaml after_backup: - /usr/sbin/borg/email-backup.sh "{configuration_filename}" "{repository}" - echo "Backup tasks have been completed and the remote server will be shut down." - ssh [username]@[serverdomain] -i /etc/ssh/[sshkey] -p [port] 'sudo /sbin/poweroff' ``` ### Actual behavior [2024-05-19 14:00:00,582] INFO: /etc/borgmatic/config.yaml: Running 3 commands for post-backup hook [2024-05-19 14:00:00,602] WARNING: Backup tasks have been completed and the remote server will be shut down. [2024-05-19 14:00:34,802] WARNING: ssh://[username]@[serverdomain]/./backup: Error running actions for repository [2024-05-19 14:00:34,803] WARNING: Command 'ssh [username]@[serverdomain] -i /etc/ssh/[sshkey] -p [port] 'sudo /sbin/poweroff'' returned non-zero exit status 255. ### Expected behavior Previous behaviour: [2023-04-22 14:19:33,642] INFO: /etc/borgmatic/config.yaml: Running 2 commands for post-everything hook [2023-04-22 14:19:33,651] WARNING: Tasks have been completed and the remote server will be shut down. [2023-04-22 14:19:36,123] INFO: [2023-04-22 14:19:36,124] INFO: summary: [2023-04-22 14:19:33,641] INFO: /etc/borgmatic/config.yaml: Successfully ran configuration file ### Other notes / implementation ideas _No response_ ### borgmatic version 1.7.4 ### borgmatic installation method bullseye-backports ### Borg version 1.2.3 ### Python version Python 3.9.2 ### Database version (if applicable) N/A ### Operating system and version Debian bullseye (OpenMediaVault 6)

witten commented

2024-05-20 17:58:59 +00:00

I'm not exactly sure what's going on here, but is it possible that the remote machine shuts down the network before the SSH session completes, thereby resulting in the SSH error you're experiencing? That might be consistent with exit code 255.

If I run the command manually in a shell, I don't get any exit status, it just executes, then I get the prompt for a new command.

What happens if you then run echo $? in that same shell right after? Does it show an error exit code of 255?

Do you know if this changed from working to not working after a borgmatic upgrade? Or perhaps instead after an upgrade to your NAS?

If you decide that this is a "spurious" error that you'd prefer to suppress, you could add something like || true to your SSH command. However this would suppress all errors with that command.

I'm not exactly sure what's going on here, but is it possible that the remote machine shuts down the network before the SSH session completes, thereby resulting in the SSH error you're experiencing? That _might_ be consistent with exit code 255. > If I run the command manually in a shell, I don't get any exit status, it just executes, then I get the prompt for a new command. What happens if you then run `echo $?` in that same shell right after? Does it show an error exit code of 255? Do you know if this changed from working to not working after a borgmatic upgrade? Or perhaps instead after an upgrade to your NAS? If you decide that this is a "spurious" error that you'd prefer to suppress, you could add something like `|| true` to your SSH command. However this would suppress _all_ errors with that command.

witten added the

question / support

label 2024-05-20 17:59:13 +00:00

auanasgheps commented

2024-05-26 16:11:17 +00:00

Sorry for the late reply, I didn't get the email.

Do you know if this changed from working to not working after a borgmatic upgrade? Or perhaps instead after an upgrade to your NAS?

I think this behaviour has changed after a borgmatic update.

I was running a pretty old version of borgmatic (1.5.x) from the regular bullseye (Debian 11). I switched to the bullseye-backports which installed a fairly recent 1.7.4.

What happens if you then run echo $? in that same shell right after? Does it show an error exit code of 255?

root@nas:~# ssh [username]@[serverdomain] -i /etc/ssh/[sshkey] -p [port]  'sudo /sbin/poweroff'
root@nas:~# echo $?
130
root@nas:~#

Interesting, shows 130, not 255.

Sorry for the late reply, I didn't get the email. > Do you know if this changed from working to not working after a borgmatic upgrade? Or perhaps instead after an upgrade to your NAS? I think this behaviour has changed after a borgmatic update. I was running a pretty old version of borgmatic (1.5.x) from the regular bullseye (Debian 11). I switched to the bullseye-backports which installed a fairly recent 1.7.4. > > What happens if you then run `echo $?` in that same shell right after? Does it show an error exit code of 255? ```shell root@nas:~# ssh [username]@[serverdomain] -i /etc/ssh/[sshkey] -p [port] 'sudo /sbin/poweroff' root@nas:~# echo $? 130 root@nas:~# ``` Interesting, shows 130, not 255.

witten commented

2024-06-06 21:26:17 +00:00

I can't immediately think of anything that would've changed in that upgrade, but it's quite possible that the exit status handling or the precise timing in handling sub-commands changed between those versions. Although note that even 1.7.4 is pretty old at this point.. It's from 2022!

I think exit code 130 is SIGINT, which means the command is getting interrupted—possibly by the system shutting down. So I'm not sure about the 130/255 discrepancy but I think borgmatic is working "as intended" here. If it receives a non-zero exit status, it complains. You could try the || true trick if you're comfortable with all errors from that command getting swallowed.

I can't immediately think of anything that would've changed in that upgrade, but it's quite possible that the exit status handling or the precise timing in handling sub-commands changed between those versions. Although note that even 1.7.4 is pretty old at this point.. It's from 2022! I think exit code 130 is SIGINT, which means the command is getting interrupted—possibly by the system shutting down. So I'm not sure about the 130/255 discrepancy but I think borgmatic is working "as intended" here. If it receives a non-zero exit status, it complains. You could try the `|| true` trick if you're comfortable with all errors from that command getting swallowed.

auanasgheps commented

2024-06-07 09:31:07 +00:00

Hi @witten I tried the || true workaround and is working as intended. I'm fine with that, but would be nice to manage these exit codes and tell borgmatic they are fine "natively".

Although note that even 1.7.4 is pretty old at this point.. It's from 2022!

That's how Debian works... I might try to force the upgrade but I'm scared of the dependencies.

Hi @witten I tried the `|| true` workaround and is working as intended. I'm fine with that, but would be nice to manage these exit codes and tell borgmatic they are fine "natively". > Although note that even 1.7.4 is pretty old at this point.. It's from 2022! That's how Debian works... I might try to force the upgrade but I'm scared of the dependencies.

witten commented

2024-06-08 16:53:44 +00:00

Hi @witten I tried the || true workaround and is working as intended. I'm fine with that, but would be nice to manage these exit codes and tell borgmatic they are fine "natively".

There is actually an existing feature to configure how borgmatic interprets Borg's exit codes as success or failure, but that doesn't apply to other commands like those in after_backup. And I'm not sure how that would even work given that you can run any number of arbitrary commands (with different exit code semantics) in each command hook. But given that you can put arbitrary scripting within those hooks, I would suggest calling a shell script that does the exit code interpretation you want. For instance, in Bash you can check the value of $? after running a command like ssh and, based on the specific value(s), either exit with a success or error.

That's how Debian works... I might try to force the upgrade but I'm scared of the dependencies.

You could use pipx to manage the installation and dependencies, which might make things easier. Although I understand if you'd rather just stick with the Debian package.

> Hi @witten I tried the || true workaround and is working as intended. I'm fine with that, but would be nice to manage these exit codes and tell borgmatic they are fine "natively". There is actually an [existing feature](https://torsion.org/borgmatic/docs/how-to/customize-warnings-and-errors/) to configure how borgmatic interprets *Borg's* exit codes as success or failure, but that doesn't apply to other commands like those in `after_backup`. And I'm not sure how that would even work given that you can run any number of arbitrary commands (with different exit code semantics) in each command hook. But given that you can put arbitrary scripting within those hooks, I would suggest calling a shell script that does the exit code interpretation you want. For instance, in Bash you can check the value of `$?` after running a command like `ssh` and, based on the specific value(s), either exit with a success or error. > That's how Debian works... I might try to force the upgrade but I'm scared of the dependencies. You could [use pipx](https://torsion.org/borgmatic/docs/how-to/set-up-backups/) to manage the installation and dependencies, which might make things easier. Although I understand if you'd rather just stick with the Debian package.

auanasgheps commented

2024-06-11 09:39:55 +00:00

Thank you for your recommendations.

I will create a little script to make the shutdown command better and avoid the || true workaround, but most importantly I have switched to a pipx installation, thanks for telling me!

borgmatic --version
1.8.11

Since I updated borgmatic I was seeing strange things in the syslog, so I decided that was a great moment to uninstall it and go for pipx. If this issue will appear again I will open a new case.

Thanks again!

Thank you for your recommendations. I will create a little script to make the shutdown command better and avoid the `|| true` workaround, but most importantly I have switched to a pipx installation, thanks for telling me! ```shell borgmatic --version 1.8.11 ``` Since I updated borgmatic I was seeing strange things in the syslog, so I decided that was a great moment to uninstall it and go for pipx. If this issue will appear again I will open a new case. Thanks again!

auanasgheps closed this issue

2024-06-11 09:39:55 +00:00

witten commented

2024-06-11 15:27:35 +00:00

Glad to hear you've got a newer version of borgmatic!

Sign in to join this conversation.