systemd service unit fails following security hardening in 1.5.19 #466
Labels
No Label
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
No Milestone
No Assignees
4 Participants
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: borgmatic-collective/borgmatic#466
Loading…
Reference in New Issue
No description provided.
Delete Branch "%!s(<nil>)"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What I'm trying to do and why
run
borgmatic.serviceSteps to reproduce (if a bug)
Install borgmatic to run
borgmaticAttempt to run
systemctl start borgmatic.serviceActual behavior (if a bug)
See attached
borgmatic-output.txtfrom l86Expected behavior (if a bug)
With
borgmatic.servicefrom here:https://projects.torsion.org/borgmatic-collective/borgmatic/raw/commit/602ad9e7ee36b2b1fdbf6083bd16919d45be8045/sample/systemd/borgmatic.service
Environment
borgmatic version:
borgmatic installation method:
sudo dnf install borgmaticFrom here:
https://copr.fedorainfracloud.org/coprs/heffer/borgmatic/repo/epel-8/heffer-borgmatic-epel-8.repo
Borg version:
Python version:
Database version (if applicable):
n/a
operating system and version:
Rocky gnu/linux 8.5
I have the simliar problem on arch.
@pseud Thanks for filing this! But I'm not sure what's going on here. So you're saying borgmatic 1.5.18 worked just fine, and starting with 1.5.19/1.5.20, borgmatic started giving permissions errors on Borg lock files? The
borgmatic.servicefile you've linked doesn't seem to have any of the security hardening changes introduced in 1.5.19. Is that your pre-1.5.19 systemd service file? Can I get a look at your updatedborgmatic.servicethat's causing problems? And possibly a (redacted) borgmatic configuration file?@smoove Same questions for you! So borgmatic stopped working around 1.5.19? Did you update your
borgmatic.servicefile around that time or not? Can I get a peek at that file and your (redacted) borgmatic configuration?Appreciate it!
Hmm, not exactly...
I installed
thusly:
sudo dnf install borgmaticFrom here:
https://copr.fedorainfracloud.org/coprs/heffer/borgmatic/repo/epel-8/heffer-borgmatic-epel-8.repo
But, the systemd service files failed, like so: borgmatic-output.txt
So, I contacted 'heffer' - the owner of heffer/borgmatic and he pointed me at
borgmatic.serviceSo, I don't know if 1.5.18 worked just fine - I've never installed it. My first install of borgmatic was 1.5.20 - I just used the replacement
borgmatic.servicefile I was pointed to. It was 'heffer's suggestion that the security hardening was the cause of the problem and that seems to have been correct (from my limited understanding). I actually wasn't aware that I was pointed to 1.5.18, but assumed it was from a pre hardening version.The
borgmatic.servicefile I'm using - I thought... was exactly as per the link above, but I see there seems to have been a comment dropped and I've changed the location of borgmatic to that installed. So, mine's attached. I've emailed the config.Oops, you asked for the
borgmatic.servicethat's causing the problems - it was whatever came with 1.5.20. From 'heffer's email to me:It's no longer on my system.
@witten i never had another version of the service installed, i installed borgmatic for the first time a few days before posting.
This is the package i use: https://archlinux.org/packages/community/any/borgmatic/ (borgmatic 1.5.21-1)
Btw.: running borgmatic manually works as expected.
The only changes to
borgmatic.servicein 1.5.19 were commented-out examples!@pseud It looks like the working
borgmatic.servicefile you're using is from borgmatic 1.5.10 or so—and doesn't include any security hardening at all. Which is fine if that works for your use case—borgmatic.serviceis intended as a sample/example file rather than the final word on systemd configuration for borgmatic.Having said that, I would like to know what setting in the 1.5.20 sample service file is causing problems. If you're willing to debug this further, my ask would be to grab a copy of the newest service file, install it on your system (don't forget
sudo systemctl daemon-reload), and confirm the problem still occurs with borgmatic. Then, comment out options in that file (thendaemon-reload) until the permissions error stops happening. Also, out of curiosity, what filesystem is your repository mounted on?@smoove Your error is different in that it doesn't appear to be a permissions issue on Borg's lock file. Instead, it looks like borgmatic isn't able to find any configuration files. I suggest a similar debugging approach as described above though: Comment out service configuration options,
daemon-reloading each time, until borgmatic starts working. That will hopefully pinpoint the problem option.Thanks for both of your help here.
@witten I commented out everything until just the
ExecStartline remained, (daemon-reload'ing every time). The same problem still persists.However, it just occured to me that the service is executed as root, and my config is in my $HOME.
This explains why no config is found:
I tried symlinking my config to /etc/borgmatic/config.yaml, after that the config is indeed found, but it fails with
CRITICAL Remote: Host key verification failed., which is probably caused by this line in my conf:encryption_passcommand: cat /home/[REDACTED]/.borg-passphrase.For some reason borgmatic has no access to my home directory, but only if it is triggered from the service.
My knowledge about systemd services is very limited, but is this simply a bug in the arch package? Should they just install the service file to /usr/lib/systemd/user instead of /usr/lib/systemd/system?
I hope atleast some of this makes sense :)
@witten Soz, haven't been here for a day or two:
Happy to help - give me another day or two :)
I have a similar problem (maybe?) with borgmatic 1.5.21 and the Systemd service, since I noticed these lines from the original poster:
In my case the process remains open even after succeeding, messing up my service timers. This issue is not reproducible: happens on some repositories only and not always.
My
ExecStartline was like this:However if i run this in a shell it works:
I think borgmatic does not like something in the Systemd's shell so what I did is
wrap borgmatic in bash like this:
Note: I am running an older version of the Systemd service unit file without the Systemd hardening options added recently.
@smoove
If the service is executed as root, then IMO the config should be either in root's $HOME or configured globally in
/etc. It shouldn't be in another user's $HOME.As for
Remote: Host key verification failed, that means that your local copy of a remote server's SSH key is somehow not matching the remote server anymore. See https://www.thegeekdiary.com/how-to-fix-the-error-host-key-verification-failed/ for some troubleshooting steps for that, keeping in mind that borgmatic is probably connecting as the root SSH user (and with the root user's SSH keys) if it's running as root.@frnmst Interesting workaround you have there. Feel free to file a separate ticket with more info if you like some help debugging.
I'm closing this ticket now due to inactivity, but please feel free to follow up here or on another ticket if anyone needs more assistance!