"borgmatic check" regression in 1.7.3 #597

Commit ae036aebd7 changes execute.py:

-            command, shell=shell, env=environment, cwd=working_directory
+            command, stderr=subprocess.STDOUT, shell=shell, env=environment, cwd=working_directory

This change will break borgmatic check if there is any output to stderr from ssh:

#> borgmatic check -v 2

Ensuring legacy configuration is upgraded
borg --version --debug --show-rc
/etc/borgmatic.d/storagebox.yaml: No commands to run for pre-actions hook
/etc/borgmatic.d/storagebox.yaml: No commands to run for pre-check hook
ssh://uXXXXXX-sub14@uXXXXXX.your-storagebox.de:23/./borg: Running consistency checks
borg info --json ssh://uXXXXXX-sub14@uXXXXXX.your-storagebox.de:23/./borg
ssh://uXXXXXX-sub14@uXXXXXX.your-storagebox.de:23/./borg: Error running actions for repository
Cannot determine Borg repository ID for ssh://uXXXXXX-sub14@uXXXXXX.your-storagebox.de:23/./borg
/etc/borgmatic.d/storagebox.yaml: Running command for on-error hook

Root cause is RHEL9's default crypto policy which prohibits SHA1. Hetzner's Storagebox has RSA key with SHA1:

#> ssh uXXXXXX-sub14@u311936.your-storagebox.de -p 23
client_global_hostkeys_private_confirm: server gave bad signature for RSA key 1: error in libcrypto
+------------------------------------------------------------------+
| Welcome to your Storage Box.                                     |
...
uXXXXXX-sub14 /home >

That one extra line ("client_global_hostkeys_private_confirm") is outputted to stderr by local ssh client, not remote server. It will obviously break JSON parsing in borgmatic check.

There are workarounds to get rid of that stderr warning, for example:

Manually add remote's RSA key to known_hosts (<- this is what I did)
Allow SHA1: update-crypto-policies --set DEFAULT:SHA1

I still would recommend to revert that change, or partially revert it if JSON parsing is used.

Borgmatic 1.7.3 is currently in EPEL9-testing and this problem will be hit by multiple users in 6 days:

https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-64cd98ca6b

Commit ae036aebd76 changes execute.py: ``` - command, shell=shell, env=environment, cwd=working_directory + command, stderr=subprocess.STDOUT, shell=shell, env=environment, cwd=working_directory ``` This change will break `borgmatic check` if there is any output to stderr from ssh: ``` #> borgmatic check -v 2 Ensuring legacy configuration is upgraded borg --version --debug --show-rc /etc/borgmatic.d/storagebox.yaml: No commands to run for pre-actions hook /etc/borgmatic.d/storagebox.yaml: No commands to run for pre-check hook ssh://uXXXXXX-sub14@uXXXXXX.your-storagebox.de:23/./borg: Running consistency checks borg info --json ssh://uXXXXXX-sub14@uXXXXXX.your-storagebox.de:23/./borg ssh://uXXXXXX-sub14@uXXXXXX.your-storagebox.de:23/./borg: Error running actions for repository Cannot determine Borg repository ID for ssh://uXXXXXX-sub14@uXXXXXX.your-storagebox.de:23/./borg /etc/borgmatic.d/storagebox.yaml: Running command for on-error hook ``` Root cause is RHEL9's default crypto policy which prohibits SHA1. Hetzner's Storagebox has RSA key with SHA1: ``` #> ssh uXXXXXX-sub14@u311936.your-storagebox.de -p 23 client_global_hostkeys_private_confirm: server gave bad signature for RSA key 1: error in libcrypto +------------------------------------------------------------------+ | Welcome to your Storage Box. | ... uXXXXXX-sub14 /home > ``` That one extra line ("client_global_hostkeys_private_confirm") is outputted to stderr by local ssh client, not remote server. It will obviously break JSON parsing in `borgmatic check`. There are workarounds to get rid of that stderr warning, for example: 1) Manually add remote's RSA key to known_hosts (<- this is what I did) 2) Allow SHA1: `update-crypto-policies --set DEFAULT:SHA1` I still would recommend to revert that change, or partially revert it if JSON parsing is used. Borgmatic 1.7.3 is currently in EPEL9-testing and this problem will be hit by multiple users in 6 days: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-64cd98ca6b

Wow, thank you for reporting this and diving into the cause!

It looks like this isn't quite as straightforward as backing out the change you've helpfully pinpointed, as doing so would break the new special file detection feature. So I may need to introduce some branching logic to do one thing for check and a different thing when detecting special files.

It looks like this isn't *quite* as straightforward as backing out the change you've helpfully pinpointed, as doing so would break the new special file detection feature. So I may need to introduce some branching logic to do one thing for `check` and a different thing when detecting special files.

Fix released in borgmatic 1.7.4. Please give it a shot and let me know if it works for you. Thanks!

Works for me. Thanks for quick fix.

Labels Milestones

"borgmatic check" regression in 1.7.3 #597