Suggestions for more robust borgmatic systemd service #205
Labels
No Label
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
No Milestone
No Assignees
3 Participants
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: borgmatic-collective/borgmatic#205
Loading…
Reference in New Issue
No description provided.
Delete Branch "%!s(<nil>)"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Would it make sense to use
systemd-inhibit
in the borgmatic.service file to prevent the system from being suspended or shut down while a scheduled backup is underway?eg. modifying line 6 to something like
ExecStart=systemd-inhibit --who="borgmatic" --why="Prevent interrupting scheduled backup" /root/.local/bin/borgmatic
Also, borgmatic.timer specifies
Persistent=true
which should cause borgmatic to trigger on boot if the schedule was missed. Perhaps a delay should be incorporated so that borgmatic doesn't start backing up immediately while many programs are loading. This might be bad both for system responsiveness and due to a greater chance of files being written to at that time.Both of these suggestions make sense to me. Thank you for filing them!
The one argument I could see against
systemd-inhibit
is that you may actually want the system to suspend while a backup is underway. For instance, let's say you're using a laptop and a backup is running, and you have somewhere to go, and you want to take your laptop (oops). You may not want to wait for the backup to finish before suspending. Given that Borg checkpoints every few minutes, an interrupted backup isn't so bad. Especially since the checkpoints are automatically recoverable on the next backup. And you could in theory lower the checkpoint interval if this was a common thing.Anyway, this is certainly not a deal breaker, but I wanted to think through the use cases a bit. Do you have thoughts on this? Can you talk about your use case for inhibiting suspend or shutdown when Borg is running? Thanks!
As for the suggested delay with
Persistent=true
,systemd.timer
'sRandomizedDelaySec=
may do the trick.@witten: I get your point, maybe that deserves a service level configuration option (eg: in
/etc/default/borgmatic
) ? For my own use case, I would prefer the change suggested by @adatam (especially as I use theafter_backup
hook to transfer data remotely).On a related note (sorry if off topic), I also use a few additions compared to the sample service, some security hardening stuff:
And some options to improve system responsiveness during backups:
As you can see some options are host specific, but maybe it would be useful to add it commented out ?
@witten The use case for these suggestions in my case is a desktop that is on 24/7. Backups would be scheduled for say 5AM daily, so it's unlikely that it would be rebooted, or even in use at that time. On the odd chance it were necessary, delaying the reboot a little probably wouldn't be much of an issue. Then again your point about borg's checkpoints alleviate the concern about interrupted backups, but it still might be nice to avoid.
As for the laptop use case, your point is well taken. Is there a way to override the
systemd-inhibit
? I have additional concerns about scheduled backups on laptops:How gracefully are such scenarios/failures handled?
For the second point, perhaps your suggestion of
RandomizedDelaySec=
could work. It isn't quite what I had in mind, due to its random nature, but maybe it's good enough.I am undecided whether my laptop should have scheduled backups or if I should run
borgmatic
manually. The tradeoffs in the convenience of automation versus the complexity of a reasonably robust solution are not yet clear to me. This discussion is useful to that end.@nicoulaj Thanks for the list of additional service options. Maybe the reasons and use case could be documented?
There is an interesting similar discussion about a systemd service and timer for Relax-and-Recover: https://github.com/rear/rear/issues/2139
The use cases for both ReaR and borgmatic have a lot of overlap, so the ideas proposed are likely to be relevant.
For the past couple of weeks, I've been running the following daily (prune, create) and weekly (prune, create, and the lengthier check) systemd service/timer pairs. I have not tested edge cases, though it has been working fine under "normal" use.
Most of the configuration is taken from suggestions from the ReaR issue referenced above. A couple of differences are:
systemd-inhibit
to prevent interrupting backupsSecurity hardening mentioned above is not included yet.
Here are the four files I have so far, all in
/etc/systemd/system
:####borgmatic-daily.service
####borgmatic-daily.timer
####borgmatic-weekly.service
####borgmatic-weekly.timer
Thanks for your patience on this one! I've tried to incorporate most of these ideas into the sample systemd timer/service files.
For the laptop use cases, I think scheduled backups of some sort absolutely make sense, because otherwise I won't remember to backup my laptop! However, I withdraw any concerns about
systemd-inhibit
because I just tried it on my laptop..systemd-inhibit
doesn't prevent the laptop from sleeping, which is my primary use case for "crap, I have to grab my laptop and go somewhere while a backup is running".You are correct that there are more issues with backups on laptops (network connectivity, network shares), but that doesn't change the need to have regular backups run.. It just raises the difficulty level!
Also, I omitted
ConditionACPower=true
, because I found that it prevented the timer from getting scheduled if I enabled/started it while my laptop was off AC power.. Pretty unexpected behavior!Please feel free to review the commit and let me know if I missed anything.
Note though that I left out the security hardening bits, as I don't have the systemd expertise to apply them in a generic way. For instance, I don't know if
ProtectHome=read-only
will prevent the Borg cache from getting updated. If you'd like to file a separate ticket (or a PR!) for us to work out the right general-purpose security hardening options, I'd be happy to get your help there!Maybe the
ConditionACPower=true
should be in the .service file(s), and not the .timer file(s).Good call, that worked!