SSH fails when started as systemd service #188

Closed
opened 2019-05-25 20:35:05 +00:00 by confuso · 6 comments

Hi,
if I run borgmatic -v 2 in the shell, everything is working fine:

zenbook# borgmatic -v 2
Ensuring legacy configuration is upgraded
/etc/borgmatic/config.yaml: Parsing configuration file
/etc/borgmatic/config.yaml: Running command for pre-backup hook
/etc/borgmatic/config.yaml: Hook command: ssh borg-backup@nas "rm -f .prepare_to_shutdown"
borg-backup@nas.local:/volume1/homes/borg-backup/Zenbook: Pruning archives
borg prune borg-backup@nas.local:/volume1/homes/borg-backup/Zenbook --keep-daily 7 --keep-monthly 3 --keep-yearly 1 --prefix {hostname}- --remote-path /usr/local/bin/borg --stats --debug --list --show-rc
using builtin fallback logging configuration
35 self tests completed in 0.14 seconds
SSH command line: ['ssh', 'borg-backup@nas.local', '/usr/local/bin/borg', 'serve', '--umask=077', '--debug']
Remote: using builtin fallback logging configuration
Remote: 35 self tests completed in 2.26 seconds
Remote: using builtin fallback logging configuration
Remote: Initialized logging system for JSON-based protocol
Remote: Resolving repository path b'/volume1/homes/borg-backup/Zenbook'
Remote: Resolved repository path to '/volume1/homes/borg-backup/Zenbook'
Remote: Verified integrity of /volume1/homes/borg-backup/Zenbook/index.150
TAM-verified manifest
security: read previous location 'ssh://borg-backup@nas.local/volume1/homes/borg-backup/Zenbook'
security: read manifest timestamp '2019-05-25T20:13:26.358501'
security: determined newest manifest timestamp as 2019-05-25T20:13:26.358501
security: repository checks ok, allowing access

But it fails when I run systemctl start borgmatic.service:

# journalctl --unit=borgmatic.service -b | less

-- Logs begin at Tue 2019-04-30 15:04:13 CEST, end at Sat 2019-05-25 22:19:05 CEST. --
Mai 25 22:10:15 zenbook systemd[1]: Starting borgmatic backup...
Mai 25 22:10:18 zenbook borgmatic[31966]: Permission denied, please try again.
Mai 25 22:10:21 zenbook borgmatic[31966]: Permission denied, please try again.
Mai 25 22:10:23 zenbook borgmatic[31966]: borg-backup@nas: Permission denied (publickey,password).
Mai 25 22:10:26 zenbook borgmatic[31966]: Permission denied, please try again.
Mai 25 22:10:28 zenbook borgmatic[31966]: Permission denied, please try again.
Mai 25 22:10:30 zenbook borgmatic[31966]: borg-backup@nas: Permission denied (publickey,password).
Mai 25 22:10:30 zenbook borgmatic[31966]: /etc/borgmatic/config.yaml: Error running configuration file
Mai 25 22:10:30 zenbook borgmatic[31966]: Command 'ssh borg-backup@nas "rm -f .prepare_to_shutdown"' returned non-zero exit status 255.
Mai 25 22:10:30 zenbook borgmatic[31966]: Need some help? https://torsion.org/borgmatic/#issues
Mai 25 22:10:30 zenbook systemd[1]: borgmatic.service: Main process exited, code=exited, status=1/FAILURE
Mai 25 22:10:30 zenbook systemd[1]: borgmatic.service: Failed with result 'exit-code'.
Mai 25 22:10:30 zenbook systemd[1]: Failed to start borgmatic backup.``

My service file looks like this:
# cat /etc/systemd/system/borgmatic.service
[Unit]
Description=borgmatic backup

[Service]
Type=oneshot
ExecStart=/usr/bin/borgmatic -v 2

What makes the actual difference, why is it not working as a systemd service?

Hi, if I run `borgmatic -v 2` in the shell, everything is working fine: zenbook# borgmatic -v 2 Ensuring legacy configuration is upgraded /etc/borgmatic/config.yaml: Parsing configuration file /etc/borgmatic/config.yaml: Running command for pre-backup hook /etc/borgmatic/config.yaml: Hook command: ssh borg-backup@nas "rm -f .prepare_to_shutdown" borg-backup@nas.local:/volume1/homes/borg-backup/Zenbook: Pruning archives borg prune borg-backup@nas.local:/volume1/homes/borg-backup/Zenbook --keep-daily 7 --keep-monthly 3 --keep-yearly 1 --prefix {hostname}- --remote-path /usr/local/bin/borg --stats --debug --list --show-rc using builtin fallback logging configuration 35 self tests completed in 0.14 seconds SSH command line: ['ssh', 'borg-backup@nas.local', '/usr/local/bin/borg', 'serve', '--umask=077', '--debug'] Remote: using builtin fallback logging configuration Remote: 35 self tests completed in 2.26 seconds Remote: using builtin fallback logging configuration Remote: Initialized logging system for JSON-based protocol Remote: Resolving repository path b'/volume1/homes/borg-backup/Zenbook' Remote: Resolved repository path to '/volume1/homes/borg-backup/Zenbook' Remote: Verified integrity of /volume1/homes/borg-backup/Zenbook/index.150 TAM-verified manifest security: read previous location 'ssh://borg-backup@nas.local/volume1/homes/borg-backup/Zenbook' security: read manifest timestamp '2019-05-25T20:13:26.358501' security: determined newest manifest timestamp as 2019-05-25T20:13:26.358501 security: repository checks ok, allowing access But it fails when I run `systemctl start borgmatic.service`: # journalctl --unit=borgmatic.service -b | less -- Logs begin at Tue 2019-04-30 15:04:13 CEST, end at Sat 2019-05-25 22:19:05 CEST. -- Mai 25 22:10:15 zenbook systemd[1]: Starting borgmatic backup... Mai 25 22:10:18 zenbook borgmatic[31966]: Permission denied, please try again. Mai 25 22:10:21 zenbook borgmatic[31966]: Permission denied, please try again. Mai 25 22:10:23 zenbook borgmatic[31966]: borg-backup@nas: Permission denied (publickey,password). Mai 25 22:10:26 zenbook borgmatic[31966]: Permission denied, please try again. Mai 25 22:10:28 zenbook borgmatic[31966]: Permission denied, please try again. Mai 25 22:10:30 zenbook borgmatic[31966]: borg-backup@nas: Permission denied (publickey,password). Mai 25 22:10:30 zenbook borgmatic[31966]: /etc/borgmatic/config.yaml: Error running configuration file Mai 25 22:10:30 zenbook borgmatic[31966]: Command 'ssh borg-backup@nas "rm -f .prepare_to_shutdown"' returned non-zero exit status 255. Mai 25 22:10:30 zenbook borgmatic[31966]: Need some help? https://torsion.org/borgmatic/#issues Mai 25 22:10:30 zenbook systemd[1]: borgmatic.service: Main process exited, code=exited, status=1/FAILURE Mai 25 22:10:30 zenbook systemd[1]: borgmatic.service: Failed with result 'exit-code'. Mai 25 22:10:30 zenbook systemd[1]: Failed to start borgmatic backup.`` My service file looks like this: # cat /etc/systemd/system/borgmatic.service [Unit] Description=borgmatic backup [Service] Type=oneshot ExecStart=/usr/bin/borgmatic -v 2 What makes the actual difference, why is it not working as a systemd service?
Owner

Thanks for reporting this! My guess is that systemd is not running borgmatic as the correct user, and therefore you're getting permission denied. Assuming that you want to run borgmatic as root, can you try something like this in your service file?

[Unit]
Description=borgmatic backup

[Service]
Type=oneshot
ExecStart=/usr/bin/borgmatic -v 2
User=root

Let me know if that helps.

Thanks for reporting this! My guess is that systemd is not running borgmatic as the correct user, and therefore you're getting permission denied. Assuming that you want to run borgmatic as root, can you try something like this in your service file? ``` [Unit] Description=borgmatic backup [Service] Type=oneshot ExecStart=/usr/bin/borgmatic -v 2 User=root ``` Let me know if that helps.
witten added the
question / support
label 2019-05-25 20:55:45 +00:00
Contributor

We should document this! It's always a pain to setup and I do appreciate when I see doucmentation that lays out how to integrate into Systemd, which is most of the time about losing >= 1 hour configuring it ...

We should document this! It's always a pain to setup and I do appreciate when I see doucmentation that lays out how to integrate into Systemd, which is most of the time about losing >= 1 hour configuring it ...
Author

So, I figured it out since yesterday and it was my mistake. I used su to get root, which took all my environmental variables from my local user terminal session. Using ssh-copy-id to setup the ssh connection to my NAS was then probably done for the normal user, not for root. But if I run borgmatic with systemd, it doesn't have my user environmental variables. Actually, you can even get it run with systemd if you apply systemd import-environment during the session (which helped me figuring out the problem...).

I opened a new shell to login with root (and without my user environmental variables) and checked the ssh connection - which was not working, contrary to the sucase... Using ssh-copy-id in this shell, I could setup my ssh connection for root. Now it's also working fine for the systemd.service

Thank you for your support! And @decentral1se, yes, this cost me more than an hour figuring it out :D

So, I figured it out since yesterday and it was my mistake. I used `su` to get root, which took all my environmental variables from my local user terminal session. Using `ssh-copy-id` to setup the ssh connection to my NAS was then probably done for the normal user, not for root. But if I run borgmatic with systemd, it doesn't have my user environmental variables. Actually, you can even get it run with systemd if you apply `systemd import-environment` during the session (which helped me figuring out the problem...). I opened a new shell to login with root (and without my user environmental variables) and checked the ssh connection - which was not working, contrary to the `su`case... Using `ssh-copy-id` in this shell, I could setup my ssh connection for root. Now it's also working fine for the systemd.service Thank you for your support! And @decentral1se, yes, this cost me more than an hour figuring it out :D
Owner

Glad to hear you worked it out, and sorry you wasted an hour on this! If either of you have suggestions for what would have been helpful in the documentation to avoid this sort of problem, please let me know. I'm guessing that the relevant section is here: https://torsion.org/borgmatic/docs/how-to/set-up-backups/#autopilot .. Thanks.

Glad to hear you worked it out, and sorry you wasted an hour on this! If either of you have suggestions for what would have been helpful in the documentation to avoid this sort of problem, please let me know. I'm guessing that the relevant section is here: https://torsion.org/borgmatic/docs/how-to/set-up-backups/#autopilot .. Thanks.
Contributor

Aha, it's already there! Then I blame Systemd ;)

Aha, it's already there! Then I blame Systemd ;)
Owner

Still, I'm always open to ways to improve the docs!

Still, I'm always open to ways to improve the docs!
Sign in to join this conversation.
No Milestone
No Assignees
3 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#188
No description provided.