Automation with systemd failing to connect over ssh #340

New Issue

wooten · 2020-07-14T22:08:36Z

wooten commented

2020-07-14 22:08:36 +00:00

What I'm trying to do and why

Automatic borgmatic backups using systemd service

Steps to reproduce (if a bug)

Contents of /etc/systemd/system/borgmatic.service

[Unit]
Description=borgmatic backup
Wants=network-online.target
After=network-online.target
ConditionACPower=true

[Service]
Type=oneshot

Nice=19
CPUSchedulingPolicy=batch
IOSchedulingClass=best-effort
IOSchedulingPriority=7
IOWeight=100

Restart=no

ExecStartPre=/bin/sleep 1m

ExecStart=/bin/systemd-inhibit --who="borgmatic" --why="Prevent interrupting scheduled backup" /home/myuser/anaconda3/bin/borgmatic --syslog-verbosity 2 --files

Contents of /etc/borgmatic.d/work.yaml

location:
    source_directories:
        - aaa

    repositories:
        - bbb@ccc.rsync.net:ddd

storage:
    encryption_passphrase: eee

retention:
    keep_daily: 7
    keep_weekly: 4
    keep_monthly: 6

consistency:
    checks:
        - repository
        - archives

hooks:
    healthchecks: fff

I have ~/.ssh/id_rsa.pub in bbb@ccc.rsync.net:.ssh/authorized_keys, and can successfully ssh bbb@ccc.rsync.net from the terminal.

Actual behavior (if a bug)

When I manually run borgmatic --verbosity 1 --files it is successful. However, when I run systemctl start borgmatic.service I get an error:

35 self tests completed in 0.06 seconds
SSH command line: ['ssh', 'bbb@ccc.rsync.net', 'borg', 'serve', '--umask=077', '--debug']
Remote: Host key verification failed.
Connection closed by remote host. Is borg working on the server?
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4253, in main

Expected behavior (if a bug)

borgmatic should be able to ssh into rsync.net when run from systemd.

Environment

borgmatic version: [version here]

1.5.8

borgmatic installation method:

pip install borgmatic (within the base conda environment)

Borg version: [version here]

1.1.5

Python version: [version here]

3.8.2

operating system and version:

Ubuntu 18.04

#### What I'm trying to do and why Automatic borgmatic backups using systemd service #### Steps to reproduce (if a bug) Contents of /etc/systemd/system/borgmatic.service ``` [Unit] Description=borgmatic backup Wants=network-online.target After=network-online.target ConditionACPower=true [Service] Type=oneshot Nice=19 CPUSchedulingPolicy=batch IOSchedulingClass=best-effort IOSchedulingPriority=7 IOWeight=100 Restart=no ExecStartPre=/bin/sleep 1m ExecStart=/bin/systemd-inhibit --who="borgmatic" --why="Prevent interrupting scheduled backup" /home/myuser/anaconda3/bin/borgmatic --syslog-verbosity 2 --files ``` Contents of /etc/borgmatic.d/work.yaml ``` location: source_directories: - aaa repositories: - bbb@ccc.rsync.net:ddd storage: encryption_passphrase: eee retention: keep_daily: 7 keep_weekly: 4 keep_monthly: 6 consistency: checks: - repository - archives hooks: healthchecks: fff ``` I have `~/.ssh/id_rsa.pub` in `bbb@ccc.rsync.net:.ssh/authorized_keys`, and can successfully `ssh bbb@ccc.rsync.net` from the terminal. #### Actual behavior (if a bug) When I manually run `borgmatic --verbosity 1 --files` it is successful. However, when I run `systemctl start borgmatic.service` I get an error: ``` 35 self tests completed in 0.06 seconds SSH command line: ['ssh', 'bbb@ccc.rsync.net', 'borg', 'serve', '--umask=077', '--debug'] Remote: Host key verification failed. Connection closed by remote host. Is borg working on the server? Traceback (most recent call last): File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4253, in main ``` #### Expected behavior (if a bug) borgmatic should be able to ssh into rsync.net when run from systemd. #### Environment **borgmatic version:** [version here] 1.5.8 **borgmatic installation method:** pip install borgmatic (within the base conda environment) **Borg version:** [version here] 1.1.5 **Python version:** [version here] 3.8.2 **operating system and version:** Ubuntu 18.04

s1shed commented

2020-07-14 22:17:02 +00:00

Remote: Host key verification failed.

The problem is likely that you've run borgmatic from a different user account than it's running with under systemd (root) and the root user doesn't have the remote server's host key yet.

To resolve this:

SSH into the remote server once as root. You'll probably get a prompt like this:

The authenticity of host 'ccc.rsync.net (1.1.1.1)' can't be established.
ED25519 key fingerprint is SHA256:HASH.
Are you sure you want to continue connecting (yes/no/[fingerprint])?

Type yes to add the host key to /root/.ssh/known_hosts.

You could also ssh_keyscan ccc.rsync.net >> /etc/ssh/ssh_known_hosts to add the host key to the global SSH configuration.

> Remote: Host key verification failed. The problem is likely that you've run borgmatic from a different user account than it's running with under systemd (root) and the root user doesn't have the remote server's host key yet. To resolve this: SSH into the remote server once as root. You'll probably get a prompt like this: ``` The authenticity of host 'ccc.rsync.net (1.1.1.1)' can't be established. ED25519 key fingerprint is SHA256:HASH. Are you sure you want to continue connecting (yes/no/[fingerprint])? ``` Type *yes* to add the host key to `/root/.ssh/known_hosts`. You could also `ssh_keyscan ccc.rsync.net >> /etc/ssh/ssh_known_hosts` to add the host key to the global SSH configuration.

👍 1

wooten commented

2020-07-14 22:32:25 +00:00

Thanks,

I ran sudo ssh ... and added the host key, but borgmatic still does not successfully ssh, giving

SSH command line: ['ssh', 'bbb@ccc.rsync.net', 'borg', 'serve', '--umask=077', '--debug']
Remote: Permission denied, please try again.
Remote: Permission denied, please try again.
Remote: Received disconnect from xxx.xxx.xxx.xxx port 22:2: Too many authentication failures
Remote: Disconnected from xxx.xxx.xxx.xxx port 22
Connection closed by remote host. Is borg working on the server?

I assume the problem is that root is not using my local user's /home/myuser/.ssh/id_rsa keypair.

Thanks, I ran `sudo ssh ...` and added the host key, but borgmatic still does not successfully ssh, giving ``` SSH command line: ['ssh', 'bbb@ccc.rsync.net', 'borg', 'serve', '--umask=077', '--debug'] Remote: Permission denied, please try again. Remote: Permission denied, please try again. Remote: Received disconnect from xxx.xxx.xxx.xxx port 22:2: Too many authentication failures Remote: Disconnected from xxx.xxx.xxx.xxx port 22 Connection closed by remote host. Is borg working on the server? ``` I assume the problem is that root is not using my local user's `/home/myuser/.ssh/id_rsa` keypair.

wooten commented

2020-07-14 22:53:48 +00:00

Aha, I just saw the config option

storage:
	ssh_command: ssh -i /home/myuser/.ssh/id_rsa

which resolves this issue.

It then raised a new issue, specific to rsync.net, which by defualt uses borg 0.29.0. I had previously set export BORG_REMOTE_PATH=/usr/local/bin/borg1/borg1 in my .bashrc, but systemd and root do not load this (conversly, export BORG_RSH="ssh -i /home/myuser/.ssh/id_rsa" likewise does not work). To overcome this, I just set the config

location:
	remote_path: /usr/local/bin/borg1/borg1

With these two changes, borgmatic.service works for me as expected. Thanks for your help!

Aha, I just saw the config option ``` storage: ssh_command: ssh -i /home/myuser/.ssh/id_rsa ``` which resolves this issue. It then raised a new issue, specific to rsync.net, which by defualt uses borg 0.29.0. I had previously set `export BORG_REMOTE_PATH=/usr/local/bin/borg1/borg1` in my .bashrc, but systemd and root do not load this (conversly, `export BORG_RSH="ssh -i /home/myuser/.ssh/id_rsa"` likewise does not work). To overcome this, I just set the config ``` location: remote_path: /usr/local/bin/borg1/borg1 ``` With these two changes, borgmatic.service works for me as expected. Thanks for your help!

wooten closed this issue

2020-07-14 22:53:50 +00:00

Sign in to join this conversation.