Borgmatic not working with systemd, while runs fine manually #718

New Issue

yashaskm11 · 2023-06-26T13:58:30Z

yashaskm11 commented

2023-06-26 13:58:30 +00:00

Trying to setup nightly backups

Followed the steps from the documentation

---
location:
  source_directories:
    - /home
  repositories:
    - path: /mnt/External_drive/borg_backups
      label: local
retention:
  keep_daily: 7

Actual behavior (if a bug)

× borgmatic.service - borgmatic backup
     Loaded: loaded (/etc/systemd/system/borgmatic.service; static)
     Active: failed (Result: exit-code) since Mon 2023-06-26 13:22:31 UTC; 27min ago
TriggeredBy: ● borgmatic.timer
    Process: 73502 ExecStartPre=sleep 1m (code=exited, status=0/SUCCESS)
    Process: 73957 ExecStart=systemd-inhibit --who=borgmatic --what=sleep:shutdown --why=Prevent interrupting scheduled backup borgmatic --verbosity -2 --syslog-verbosity 1 (code=exited, status=1/FAILURE)
   Main PID: 73957 (code=exited, status=1/FAILURE)
        CPU: 269ms

Jun 26 13:21:31 alder systemd[1]: Starting borgmatic backup...
Jun 26 13:22:31 alder systemd-inhibit[73958]: Failed to execute : No such file or directory
Jun 26 13:22:31 alder systemd-inhibit[73957]: borgmatic failed with exit status 1.
Jun 26 13:22:31 alder systemd[1]: borgmatic.service: Main process exited, code=exited, status=1/FAILURE
Jun 26 13:22:31 alder systemd[1]: borgmatic.service: Failed with result 'exit-code'.
Jun 26 13:22:31 alder systemd[1]: Failed to start borgmatic backup.

Expected behavior (if a bug)

Other notes / implementation ideas

Environment

borgmatic version: [1.7.15]`

borgmatic installation method: [pip3, as shown in documentation]

Borg version: [borg 1.2.0]

Python version: [Python 3.10.6]

operating system and version: [Ubuntu server 22.04.2 LTS]

#### Trying to setup nightly backups #### Followed the steps from the documentation ``` --- location: source_directories: - /home repositories: - path: /mnt/External_drive/borg_backups label: local retention: keep_daily: 7 ``` #### Actual behavior (if a bug) ``` × borgmatic.service - borgmatic backup Loaded: loaded (/etc/systemd/system/borgmatic.service; static) Active: failed (Result: exit-code) since Mon 2023-06-26 13:22:31 UTC; 27min ago TriggeredBy: ● borgmatic.timer Process: 73502 ExecStartPre=sleep 1m (code=exited, status=0/SUCCESS) Process: 73957 ExecStart=systemd-inhibit --who=borgmatic --what=sleep:shutdown --why=Prevent interrupting scheduled backup borgmatic --verbosity -2 --syslog-verbosity 1 (code=exited, status=1/FAILURE) Main PID: 73957 (code=exited, status=1/FAILURE) CPU: 269ms Jun 26 13:21:31 alder systemd[1]: Starting borgmatic backup... Jun 26 13:22:31 alder systemd-inhibit[73958]: Failed to execute : No such file or directory Jun 26 13:22:31 alder systemd-inhibit[73957]: borgmatic failed with exit status 1. Jun 26 13:22:31 alder systemd[1]: borgmatic.service: Main process exited, code=exited, status=1/FAILURE Jun 26 13:22:31 alder systemd[1]: borgmatic.service: Failed with result 'exit-code'. Jun 26 13:22:31 alder systemd[1]: Failed to start borgmatic backup. ``` #### Expected behavior (if a bug) #### Other notes / implementation ideas #### Environment **borgmatic version:** [1.7.15]` **borgmatic installation method:** [pip3, as shown in documentation] **Borg version:** [borg 1.2.0] **Python version:** [Python 3.10.6] **operating system and version:** [Ubuntu server 22.04.2 LTS]

witten commented

2023-06-26 15:51:26 +00:00

Thanks for taking the time to file this. This might be a system path issue such that systemd can't find borgmatic. Or perhaps some of the systemd security settings could be interfering. Could I get a look at your complete systemd file (redacted as necessary)?

One of the things I'm interested in your last line where borgmatic is invoked. What is the path to borgmatic there? Is it /root/.local/bin/borgmatic for instance? Does borgmatic actually exist at that location? If not, then you can probably change /root/.local/bin/borgmatic to borgmatic's actual location. To find that, open a shell and type which borgmatic.

Thanks for taking the time to file this. This might be a system path issue such that systemd can't find borgmatic. Or perhaps some of the systemd security settings could be interfering. Could I get a look at your complete systemd file (redacted as necessary)? One of the things I'm interested in your last line where borgmatic is invoked. What is the path to borgmatic there? Is it `/root/.local/bin/borgmatic` for instance? Does borgmatic actually exist at that location? If not, then you can probably change `/root/.local/bin/borgmatic` to borgmatic's actual location. To find that, open a shell and type `which borgmatic`.

witten added the

question / support

label 2023-06-26 15:52:06 +00:00

yashaskm11 commented

2023-06-26 16:43:00 +00:00

Thanks for quick reply, yes the issue was due to incorrect system path for borgmatic.
Now that's fixed and I'm facing a new issue.

Jun 26 16:30:55 alder borgmatic[164247]: CRITICAL /etc/borgmatic/config.yaml: An error occurred
Jun 26 16:30:55 alder borgmatic[164247]: CRITICAL local: Error running actions for repository
Jun 26 16:30:55 alder borgmatic[164247]: CRITICAL ...
                                         Failed to create/acquire the lock /mnt/External_drive/test_it/lock.exclusive ([Errno 13] Permission denied: '/mnt/External_drive/test_it/lock.exclusive.9ssgcg8e.tmp').
                                         Traceback (most recent call last):
                                           File "/usr/lib/python3/dist-packages/borg/archiver.py", line 5089, in main
                                             exit_code = archiver.run(args)
                                           File "/usr/lib/python3/dist-packages/borg/archiver.py", line 5020, in run
                                             return set_ec(func(args))
                                           File "/usr/lib/python3/dist-packages/borg/archiver.py", line 168, in wrapper
                                             with repository:
                                           File "/usr/lib/python3/dist-packages/borg/repository.py", line 200, in __enter__
                                             self.open(self.path, bool(self.exclusive), lock_wait=self.lock_wait, lock=self.do_lock)
                                           File "/usr/lib/python3/dist-packages/borg/repository.py", line 431, in open
                                             self.lock = Lock(os.path.join(path, 'lock'), exclusive, timeout=lock_wait).acquire()
                                           File "/usr/lib/python3/dist-packages/borg/locking.py", line 384, in acquire
                                             self._wait_for_readers_finishing(remove, sleep)
                                           File "/usr/lib/python3/dist-packages/borg/locking.py", line 397, in _wait_for_readers_finishing
                                             self._lock.acquire()
                                           File "/usr/lib/python3/dist-packages/borg/locking.py", line 137, in acquire
                                             raise LockFailed(self.path, str(err)) from None
                                         borg.locking.LockFailed: Failed to create/acquire the lock /mnt/External_drive/test_it/lock.exclusive ([Errno 13] Permission denied: '/mnt/External_drive/test_it/lock.exclusive.lock.exclusive.9ssgcg8e.tmp').
                                         Platform: Linux alder 5.15.0-75-generic #82-Ubuntu SMP Tue Jun 6 23:10:23 UTC 2023 x86_64
                                         Linux: Unknown Linux
                                         Borg: 1.2.0  Python: CPython 3.10.6 msgpack: 1.0.3 fuse: pyfuse3 3.2.0 [pyfuse3,llfuse]
                                         PID: 164249  CWD: /
                                         sys.argv: ['/usr/bin/borg', 'create', '/mnt/External_drive/test_it::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}', '/etc/borgmatic/config.yaml', '/home', '/root/.borgmatic', '--info']
                                         SSH_ORIGINAL_COMMAND: None
Jun 26 16:30:55 alder borgmatic[164247]: CRITICAL Command 'borg create /mnt/External_drive/test_it::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /etc/borgmatic/config.yaml /home /root/.borgmatic --info' returned non-zero exit status 2.
Jun 26 16:30:55 alder borgmatic[164247]: CRITICAL
Jun 26 16:30:55 alder borgmatic[164247]: CRITICAL Need some help? https://torsion.org/borgmatic/#issues
Jun 26 16:30:55 alder systemd-inhibit[164246]: /root/.local/bin/borgmatic failed with exit status 1.
Jun 26 16:30:55 alder systemd[1]: borgmatic.service: Main process exited, code=exited, status=1/FAILURE
Jun 26 16:30:55 alder systemd[1]: borgmatic.service: Failed with result 'exit-code'.
Jun 26 16:30:55 alder systemd[1]: Failed to start borgmatic backup.

Again, the command from shell works without any issues

Thanks for quick reply, yes the issue was due to incorrect system path for borgmatic. Now that's fixed and I'm facing a new issue. ``` Jun 26 16:30:55 alder borgmatic[164247]: CRITICAL /etc/borgmatic/config.yaml: An error occurred Jun 26 16:30:55 alder borgmatic[164247]: CRITICAL local: Error running actions for repository Jun 26 16:30:55 alder borgmatic[164247]: CRITICAL ... Failed to create/acquire the lock /mnt/External_drive/test_it/lock.exclusive ([Errno 13] Permission denied: '/mnt/External_drive/test_it/lock.exclusive.9ssgcg8e.tmp'). Traceback (most recent call last): File "/usr/lib/python3/dist-packages/borg/archiver.py", line 5089, in main exit_code = archiver.run(args) File "/usr/lib/python3/dist-packages/borg/archiver.py", line 5020, in run return set_ec(func(args)) File "/usr/lib/python3/dist-packages/borg/archiver.py", line 168, in wrapper with repository: File "/usr/lib/python3/dist-packages/borg/repository.py", line 200, in __enter__ self.open(self.path, bool(self.exclusive), lock_wait=self.lock_wait, lock=self.do_lock) File "/usr/lib/python3/dist-packages/borg/repository.py", line 431, in open self.lock = Lock(os.path.join(path, 'lock'), exclusive, timeout=lock_wait).acquire() File "/usr/lib/python3/dist-packages/borg/locking.py", line 384, in acquire self._wait_for_readers_finishing(remove, sleep) File "/usr/lib/python3/dist-packages/borg/locking.py", line 397, in _wait_for_readers_finishing self._lock.acquire() File "/usr/lib/python3/dist-packages/borg/locking.py", line 137, in acquire raise LockFailed(self.path, str(err)) from None borg.locking.LockFailed: Failed to create/acquire the lock /mnt/External_drive/test_it/lock.exclusive ([Errno 13] Permission denied: '/mnt/External_drive/test_it/lock.exclusive.lock.exclusive.9ssgcg8e.tmp'). Platform: Linux alder 5.15.0-75-generic #82-Ubuntu SMP Tue Jun 6 23:10:23 UTC 2023 x86_64 Linux: Unknown Linux Borg: 1.2.0 Python: CPython 3.10.6 msgpack: 1.0.3 fuse: pyfuse3 3.2.0 [pyfuse3,llfuse] PID: 164249 CWD: / sys.argv: ['/usr/bin/borg', 'create', '/mnt/External_drive/test_it::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}', '/etc/borgmatic/config.yaml', '/home', '/root/.borgmatic', '--info'] SSH_ORIGINAL_COMMAND: None Jun 26 16:30:55 alder borgmatic[164247]: CRITICAL Command 'borg create /mnt/External_drive/test_it::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /etc/borgmatic/config.yaml /home /root/.borgmatic --info' returned non-zero exit status 2. Jun 26 16:30:55 alder borgmatic[164247]: CRITICAL Jun 26 16:30:55 alder borgmatic[164247]: CRITICAL Need some help? https://torsion.org/borgmatic/#issues Jun 26 16:30:55 alder systemd-inhibit[164246]: /root/.local/bin/borgmatic failed with exit status 1. Jun 26 16:30:55 alder systemd[1]: borgmatic.service: Main process exited, code=exited, status=1/FAILURE Jun 26 16:30:55 alder systemd[1]: borgmatic.service: Failed with result 'exit-code'. Jun 26 16:30:55 alder systemd[1]: Failed to start borgmatic backup. ``` Again, the command from shell works without any issues

witten commented

2023-06-26 17:01:24 +00:00

That's looking like a permission issue on writing Borg's lock file. My guess on the cause is either:

A difference between your command-line borgmatic user versus the user systemd is running borgmatic as. So do you know what those respective users are? Is this a root-level systemd service or a systemd user service? And what user is running the command-line when you run borgmatic via shell?
Or it's possible the systemd security settings in the borgmatic service file are interfering. You could try commenting out some or all of them (temporarily) to see if that helps.

That's looking like a permission issue on writing Borg's lock file. My guess on the cause is either: 1. A difference between your command-line borgmatic user versus the user systemd is running borgmatic as. So do you know what those respective users are? Is this a root-level systemd service or a systemd user service? And what user is running the command-line when you run borgmatic via shell? 2. Or it's possible the systemd security settings in the borgmatic service file are interfering. You could try commenting out some or all of them (temporarily) to see if that helps.

yashaskm11 commented

2023-06-26 17:20:44 +00:00

I use sudo to run borgmatic from command line (I've added /root/.local/bin to sudoers default path). It seems that it's root-level service as I followed the exact instructions from the docs.
I've disabled every security setting in /etc/systemd/system/borgmatic.service, still the issue persists.

1. I use sudo to run borgmatic from command line (I've added /root/.local/bin to sudoers default path). It seems that it's root-level service as I followed the exact instructions from the docs. 2. I've disabled every security setting in /etc/systemd/system/borgmatic.service, still the issue persists.

witten commented

2023-06-26 17:24:37 +00:00

Some more ideas:

Are you sure your systemd serivce file changes are taking effect? For instance, have you issued a systemd daemon-reload command?
Can I get a look at your current systemd service file?
Since you've started receiving the lock file error, can you still successfully run borgmatic from the command-line?

Some more ideas: * Are you sure your systemd serivce file changes are taking effect? For instance, have you issued a `systemd daemon-reload` command? * Can I get a look at your current systemd service file? * Since you've started receiving the lock file error, can you still successfully run borgmatic from the command-line?

yashaskm11 commented

2023-06-26 17:35:57 +00:00

[Unit]
Description=borgmatic backup
Wants=network-online.target
After=network-online.target
# Prevent borgmatic from running unless the machine is plugged into power. Remove this line if you
# want to allow borgmatic to run anytime.
ConditionACPower=true

[Service]
Type=oneshot

# Security settings for systemd running as root, optional but recommended to improve security. You
# can disable individual settings if they cause problems for your use case. For more details, see
# the systemd manual: https://www.freedesktop.org/software/systemd/man/systemd.exec.html
LockPersonality=true
# Certain borgmatic features like Healthchecks integration need MemoryDenyWriteExecute to be off.
# But you can try setting it to "yes" for improved security if you don't use those features.
MemoryDenyWriteExecute=no
NoNewPrivileges=yes
PrivateDevices=yes
PrivateTmp=yes
ProtectClock=yes
ProtectControlGroups=yes
ProtectHostname=yes
ProtectKernelLogs=yes
ProtectKernelModules=yes
ProtectKernelTunables=yes
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 AF_NETLINK
RestrictNamespaces=yes
RestrictRealtime=yes
RestrictSUIDSGID=yes
SystemCallArchitectures=native
SystemCallFilter=@system-service
SystemCallErrorNumber=EPERM
# To restrict write access further, change "ProtectSystem" to "strict" and uncomment
# "ReadWritePaths", "ReadOnlyPaths", "ProtectHome", and "BindPaths". Then add any local repository
# paths to the list of "ReadWritePaths" and local backup source paths to "ReadOnlyPaths". This
# leaves most of the filesystem read-only to borgmatic.
ProtectSystem=full
# ReadWritePaths=-/mnt/my_backup_drive
# ReadOnlyPaths=-/var/lib/my_backup_source
# This will mount a tmpfs on top of /root and pass through needed paths
# ProtectHome=tmpfs
# BindPaths=-/root/.cache/borg -/root/.config/borg -/root/.borgmatic

# May interfere with running external programs within borgmatic hooks.
CapabilityBoundingSet=CAP_DAC_READ_SEARCH CAP_NET_RAW

# Lower CPU and I/O priority.
Nice=19
CPUSchedulingPolicy=batch
IOSchedulingClass=best-effort
IOSchedulingPriority=7
IOWeight=100

Restart=no
# Prevent rate limiting of borgmatic log events. If you are using an older version of systemd that
# doesn't support this (pre-240 or so), you may have to remove this option.
LogRateLimitIntervalSec=0

# Delay start to prevent backups running during boot. Note that systemd-inhibit requires dbus and
# dbus-user-session to be installed.
ExecStartPre=sleep 1m
ExecStart=systemd-inhibit --who="borgmatic" --what="sleep:shutdown" --why="Prevent interrupting scheduled backup" /root/.local/bin/borgmatic --verbosity -2 --syslog-verbosity 1

Yes, I do reload the daemon after making changes to the service file.

Yes, I can run borgmatic successfully from the command line even now.

``` [Unit] Description=borgmatic backup Wants=network-online.target After=network-online.target # Prevent borgmatic from running unless the machine is plugged into power. Remove this line if you # want to allow borgmatic to run anytime. ConditionACPower=true [Service] Type=oneshot # Security settings for systemd running as root, optional but recommended to improve security. You # can disable individual settings if they cause problems for your use case. For more details, see # the systemd manual: https://www.freedesktop.org/software/systemd/man/systemd.exec.html LockPersonality=true # Certain borgmatic features like Healthchecks integration need MemoryDenyWriteExecute to be off. # But you can try setting it to "yes" for improved security if you don't use those features. MemoryDenyWriteExecute=no NoNewPrivileges=yes PrivateDevices=yes PrivateTmp=yes ProtectClock=yes ProtectControlGroups=yes ProtectHostname=yes ProtectKernelLogs=yes ProtectKernelModules=yes ProtectKernelTunables=yes RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 AF_NETLINK RestrictNamespaces=yes RestrictRealtime=yes RestrictSUIDSGID=yes SystemCallArchitectures=native SystemCallFilter=@system-service SystemCallErrorNumber=EPERM # To restrict write access further, change "ProtectSystem" to "strict" and uncomment # "ReadWritePaths", "ReadOnlyPaths", "ProtectHome", and "BindPaths". Then add any local repository # paths to the list of "ReadWritePaths" and local backup source paths to "ReadOnlyPaths". This # leaves most of the filesystem read-only to borgmatic. ProtectSystem=full # ReadWritePaths=-/mnt/my_backup_drive # ReadOnlyPaths=-/var/lib/my_backup_source # This will mount a tmpfs on top of /root and pass through needed paths # ProtectHome=tmpfs # BindPaths=-/root/.cache/borg -/root/.config/borg -/root/.borgmatic # May interfere with running external programs within borgmatic hooks. CapabilityBoundingSet=CAP_DAC_READ_SEARCH CAP_NET_RAW # Lower CPU and I/O priority. Nice=19 CPUSchedulingPolicy=batch IOSchedulingClass=best-effort IOSchedulingPriority=7 IOWeight=100 Restart=no # Prevent rate limiting of borgmatic log events. If you are using an older version of systemd that # doesn't support this (pre-240 or so), you may have to remove this option. LogRateLimitIntervalSec=0 # Delay start to prevent backups running during boot. Note that systemd-inhibit requires dbus and # dbus-user-session to be installed. ExecStartPre=sleep 1m ExecStart=systemd-inhibit --who="borgmatic" --what="sleep:shutdown" --why="Prevent interrupting scheduled backup" /root/.local/bin/borgmatic --verbosity -2 --syslog-verbosity 1 ``` Yes, I do reload the daemon after making changes to the service file. Yes, I can run borgmatic successfully from the command line even now.

witten commented

2023-06-26 17:38:54 +00:00

That service file looks like most of the security settings are still in place...? For instance, I'd recommend commenting out everything from LockPersonality through CapabilityBoundingSet inclusive (followed by a daemon-reload) to see if that makes a difference.

That service file looks like most of the security settings are still in place...? For instance, I'd recommend commenting out everything from `LockPersonality` through `CapabilityBoundingSet` inclusive (followed by a `daemon-reload`) to see if that makes a difference.

yashaskm11 commented

2023-06-26 18:01:21 +00:00

That worked !!
Is it okay to leave them all disabled ?

That worked !! Is it okay to leave them all disabled ?

witten commented

2023-06-26 18:47:30 +00:00

I'm glad to hear that worked! Leaving them disabled really depends on your security needs. It is probably more secure to include them, as it reduces the attack surface for any would-be intruder. If you want to try including them, you can do a binary search to find which option is causing you trouble. For instance, uncomment half of the security options and try again. If that works, uncomment another half of the options. If you start getting the permissions error again, re-comment options by half until you find the offending option. Then you can just leave it out or read up on what tweaks it might need to make things work on your system.

yashaskm11 commented

2023-06-27 07:56:31 +00:00

Finally figured it out, thanks @witten .
It was CapabilityBoundingSet=CAP_DAC_READ_SEARCH CAP_NET_RAW that was causing trouble.
So enabled everything except this.
What purpose does this serve ? safe to leave it disabled ?

Finally figured it out, thanks @witten . It was `CapabilityBoundingSet=CAP_DAC_READ_SEARCH CAP_NET_RAW` that was causing trouble. So enabled everything except this. What purpose does this serve ? safe to leave it disabled ?

witten commented

2023-06-27 17:06:44 +00:00

I'm happy to hear you've got it figured out! I believe that particular line restricts borgmatic filesystem access to read-only (except for files owned by the same user it's running as). That's likely why it failed to write to the lock file. So you could try making sure that the lock file/directory is owned by the same user borgmatic is running as (root?), or you could just leave this line out if you're comfortable with borgmatic having read-write access to your filesystem. It's really your call as to what level of security you're comfortable with.

You can read more about this here: https://www.man7.org/linux/man-pages/man5/systemd-system.conf.5.html

Also here: https://www.man7.org/linux/man-pages/man7/capabilities.7.html

I'm happy to hear you've got it figured out! I believe that particular line restricts borgmatic filesystem access to read-only (except for files owned by the same user it's running as). That's likely why it failed to write to the lock file. So you could try making sure that the lock file/directory is owned by the same user borgmatic is running as (root?), or you could just leave this line out if you're comfortable with borgmatic having read-write access to your filesystem. It's really your call as to what level of security you're comfortable with. You can read more about this here: https://www.man7.org/linux/man-pages/man5/systemd-system.conf.5.html Also here: https://www.man7.org/linux/man-pages/man7/capabilities.7.html

yashaskm11 commented

2023-06-27 17:30:15 +00:00

Indeed, the directory I was trying to write was not owned by root.
Thanks for the help mate !!

Indeed, the directory I was trying to write was not owned by root. Thanks for the help mate !!

yashaskm11 closed this issue

2023-06-27 17:30:15 +00:00

witten commented

2023-06-27 18:02:57 +00:00

Sure thing!

Sign in to join this conversation.