Improve systemd security #352

Closed
opened 2020-08-10 18:26:12 +00:00 by palto42 · 10 comments
Contributor

What I'm trying to do and why

I would lite to improve the security setiings of the systemd service for borgmartic.

Although I trust borgmatic in principle, I think it's good practice to restrict the privilidges to the minimum required.

Other notes / implementation ideas

I read about additional security settings for systemd services to apply restrictiion even if the service is running as root (which is required in my case to backup system settings etc.). Checking teh current security level with systemd-analyze security borgmatic provided a rating of → Overall exposure level for borgmatic.service: 9.6 UNSAFE 😨.

After some search I found a comment from @nicoulaj in an older ticket #205 related to systemd service which already proposed to add such security settings, but @witten finally decided to leave it out due to lack of expierience.

Can we maybe pick-up this topic again and try to find some settings which should work for majority of use cases?

There are many possible settings as you can find on systemd.exec, but I would focus on the ones with high ratings from systemd-analyze output. Note that the security option was added in systemd v240 and is not available in Ubunu 18.04 which uses v237.

I started with a few settings (less than in the above mentioned comment) which improved the score to 7.7 at least:

# Security
ProtectSystem=strict
PrivateTmp=yes
NoNewPrivileges=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes

I'm not sure if borgmatic needs write access anywhere besides the backup destination. In above comment there was a line allowing access to a few folders, but not sure if this is required (e.g. /mnt/backup semms to be the backup destination, while I use remote via ssh)

ReadWritePaths=-/root/.config -/root/.cache /mnt/backup

Environment

borgmatic version: 1.5.9

borgmatic installation method: Python venv & pip installso

Borg version: 1.1.11

Python version: 3.7.7 / 3.8.2

operating system and version: Solus / Ubuntu 20.04.1

#### What I'm trying to do and why I would lite to improve the security setiings of the systemd service for borgmartic. Although I trust borgmatic in principle, I think it's good practice to restrict the privilidges to the minimum required. #### Other notes / implementation ideas I read about additional security settings for systemd services to apply restrictiion even if the service is running as root (which is required in my case to backup system settings etc.). Checking teh current security level with `systemd-analyze security borgmatic` provided a rating of `→ Overall exposure level for borgmatic.service: 9.6 UNSAFE 😨`. After some search I found a [comment](https://projects.torsion.org/witten/borgmatic/issues/205#issuecomment-1746) from @nicoulaj in an older ticket #205 related to systemd service which already proposed to add such security settings, but @witten finally [decided to leave it out](https://projects.torsion.org/witten/borgmatic/issues/205#issuecomment-1923) due to lack of expierience. Can we maybe pick-up this topic again and try to find some settings which should work for majority of use cases? There are many possible settings as you can find on [systemd.exec](https://www.freedesktop.org/software/systemd/man/systemd.exec.html), but I would focus on the ones with high ratings from `systemd-analyze` output. Note that the `security` option was added in systemd v240 and is not available in Ubunu 18.04 which uses v237. I started with a few settings (less than in the above mentioned comment) which improved the score to 7.7 at least: ```systemd # Security ProtectSystem=strict PrivateTmp=yes NoNewPrivileges=yes ProtectKernelTunables=yes ProtectKernelModules=yes ProtectControlGroups=yes ``` I'm not sure if borgmatic needs write access anywhere besides the backup destination. In above comment there was a line allowing access to a few folders, but not sure if this is required (e.g. /mnt/backup semms to be the backup destination, while I use remote via ssh) ``` ReadWritePaths=-/root/.config -/root/.cache /mnt/backup ``` #### Environment **borgmatic version:** 1.5.9 **borgmatic installation method:** Python venv & pip installso **Borg version:** 1.1.11 **Python version:** 3.7.7 / 3.8.2 **operating system and version:** Solus / Ubuntu 20.04.1
palto42 changed title from systemd security to Improve systemd security 2020-08-10 18:32:42 +00:00
Author
Contributor

The systemd security settings will partly depend on the backup destination used and the pre/post hooks used.

Examples:

  • Local backup
    • Allow local write access to backup folder
    • Deny network access
  • Remote backup
    • Very limited local write access
    • Requires network access
  • DB backup
    • Not sure if this requires any additional capabilities/permissions?

For the 26 system capabilities it is possible to explicitely allow and/or deny them as needed. To me it seems that usually not many are required for a normal backup, but not quite sure.

In the settings of @nicoulaj there are quite a lot capabilities allowed, whould be good to understand for each of them why. Again, it may depend on the use case.

  • CAP_SYS_RESOURCE
    • Don't understand why this is needed for borgmatic
  • CAP_SYS_ADMIN
    • Very broad range of capabilities, not clear which sub-capabilies are needed.
    • Would be good to avoid and use more specific settings
  • CAP_MKNOD
    • Is bormatic creating special files with mknod?
  • CAP_DAC_READ_SEARCH
    • Required to bypass read permission checks so that borgmatic can backup all files regardless of the file permissions
  • CAP_SYS_CHROOT
    • Is chroot or setns used by borgmatic?
  • CAP_SETPCAP
    • Don't understand this capability ...

Checking the list of other settings used by @nicoulaj, most make sense to me, but some seem obsolete and some I don't understand.

  • NoNewPrivileges=yes
    • ensures that the service process and all its children can never gain new privileges through execve
  • PrivateUsers=no
    • This is the default
  • PrivateTmp=yes
    • sets up a new file system namespace for the executed processes and mounts private /tmp/ and /var/tmp/ directories inside it that are not shared by processes outside of the namespace
  • PrivateDevices=yes
    • sets up a new /dev mount for the executed processes and only adds API pseudo devices such as /dev/null, /dev/zero or /dev/random (as well as the pseudo TTY subsystem) to it, but no physical devices such as /dev/sda, system memory /dev/mem, system ports /dev/port and others
  • DevicePolicy=closed
    • This is included in PrivateDevices=yes
    • only allow types of access that are explicitly specified and to standard pseudo devices including /dev/null, /dev/zero, /dev/full, /dev/random, and /dev/urandom.
  • ProtectSystem=strict
    • If set to "strict" the entire file system hierarchy is mounted read-only, except for the API file system subtrees /dev, /proc and /sys (protect these directories using PrivateDevices=, ProtectKernelTunables=, ProtectControlGroups=)
  • ProtectHome=read-only
    • This is already covered by ProtectSystem=strict
    • If read-only, the directories /home, /root, and /run/user are made read-only for processes invoked by this unit
  • ProtectControlGroups=yes
    • the Linux Control Groups hierarchies accessible through /sys/fs/cgroup will be made read-only to all processes of the unit
  • ProtectKernelModules=yes
    • explicit module loading will be denied
  • ProtectKernelTunables=yes
    • kernel variables accessible through /proc/sys, /sys, /proc/sysrq-trigger, /proc/latency_stats, /proc/acpi, /proc/timer_stats, /proc/fs and /proc/irq will be made read-only to all processes of the unit
  • RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 AF_NETLINK
    • Restricts the set of socket address families accessible to the processes of this unit
  • RestrictRealtime=yes
    • any attempts to enable realtime scheduling in a process of the unit are refused
  • RestrictNamespaces=yes
    • access to any kind of namespacing is prohibited
  • MemoryDenyWriteExecute=yes
    • attempts to create memory mappings that are writable and executable at the same time, or to change existing memory mappings to become executable, or mapping shared memory segments as executable are prohibited
  • LockPersonality=true
    • locks down the personality system call so that the kernel execution domain may not be changed from the default or the personality selected with Personality= directive
  • SystemCallArchitectures=native
    • processes of this unit will only be permitted to call native system calls
  • SystemCallFilter=@system-service
    • A reasonable set of system calls used by common system services, excluding any special purpose calls. This is the recommended starting point for allow-listing system calls for system services, as it contains what is typically needed by system services, but excludes overly specific interfaces. For example, the following APIs are excluded: "@clock", "@mount", "@swap", "@reboot"
  • ReadWritePaths=-/root/.config -/root/.cache /mnt/backup
    • The paths prefixed with "-" will be ignored when they do not exist
    • Not sure why write access is required to /root/.config
    • Borg uses /root/.cache/borg
    • Write ccess to /mnt/backup indicates that this is used as backup destination, so this depends on the specific use case
The systemd security settings will partly depend on the backup destination used and the pre/post hooks used. Examples: * Local backup * Allow local write access to backup folder * Deny network access * Remote backup * Very limited local write access * Requires network access * DB backup * Not sure if this requires any additional capabilities/permissions? For the 26 system [capabilities](https://man7.org/linux/man-pages/man7/capabilities.7.html) it is possible to explicitely allow and/or deny them as needed. To me it seems that usually not many are required for a normal backup, but not quite sure. In the settings of @nicoulaj there are quite a lot capabilities allowed, whould be good to understand for each of them why. Again, it may depend on the use case. * CAP_SYS_RESOURCE * Don't understand why this is needed for borgmatic * CAP_SYS_ADMIN * Very broad range of capabilities, not clear which sub-capabilies are needed. * Would be good to avoid and use more specific settings * CAP_MKNOD * Is bormatic creating special files with mknod? * CAP_DAC_READ_SEARCH * Required to bypass read permission checks so that borgmatic can backup all files regardless of the file permissions * CAP_SYS_CHROOT * Is `chroot` or `setns` used by borgmatic? * CAP_SETPCAP * Don't understand this capability ... Checking the list of other settings used by @nicoulaj, most make sense to me, but some seem obsolete and some I don't understand. * NoNewPrivileges=yes * ensures that the service process and all its children can never gain new privileges through execve * PrivateUsers=no * This is the default * PrivateTmp=yes * sets up a new file system namespace for the executed processes and mounts private /tmp/ and /var/tmp/ directories inside it that are not shared by processes outside of the namespace * PrivateDevices=yes * sets up a new /dev mount for the executed processes and only adds API pseudo devices such as /dev/null, /dev/zero or /dev/random (as well as the pseudo TTY subsystem) to it, but no physical devices such as /dev/sda, system memory /dev/mem, system ports /dev/port and others * DevicePolicy=closed * This is included in `PrivateDevices=yes` * only allow types of access that are explicitly specified and to standard pseudo devices including /dev/null, /dev/zero, /dev/full, /dev/random, and /dev/urandom. * ProtectSystem=strict * If set to "strict" the entire file system hierarchy is mounted read-only, except for the API file system subtrees /dev, /proc and /sys (protect these directories using PrivateDevices=, ProtectKernelTunables=, ProtectControlGroups=) * ProtectHome=read-only * This is already covered by `ProtectSystem=strict` * If read-only, the directories /home, /root, and /run/user are made read-only for processes invoked by this unit * ProtectControlGroups=yes * the Linux Control Groups hierarchies accessible through /sys/fs/cgroup will be made read-only to all processes of the unit * ProtectKernelModules=yes * explicit module loading will be denied * ProtectKernelTunables=yes * kernel variables accessible through /proc/sys, /sys, /proc/sysrq-trigger, /proc/latency_stats, /proc/acpi, /proc/timer_stats, /proc/fs and /proc/irq will be made read-only to all processes of the unit * RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 AF_NETLINK * Restricts the set of socket address families accessible to the processes of this unit * RestrictRealtime=yes * any attempts to enable realtime scheduling in a process of the unit are refused * RestrictNamespaces=yes * access to any kind of namespacing is prohibited * MemoryDenyWriteExecute=yes * attempts to create memory mappings that are writable and executable at the same time, or to change existing memory mappings to become executable, or mapping shared memory segments as executable are prohibited * LockPersonality=true * locks down the personality system call so that the kernel execution domain may not be changed from the default or the personality selected with Personality= directive * SystemCallArchitectures=native * processes of this unit will only be permitted to call native system calls * SystemCallFilter=@system-service * A reasonable set of system calls used by common system services, excluding any special purpose calls. This is the recommended starting point for allow-listing system calls for system services, as it contains what is typically needed by system services, but excludes overly specific interfaces. For example, the following APIs are excluded: "@clock", "@mount", "@swap", "@reboot" * ReadWritePaths=-/root/.config -/root/.cache /mnt/backup * The paths prefixed with "-" will be ignored when they do not exist * Not sure why write access is required to `/root/.config` * Borg uses `/root/.cache/borg` * Write ccess to `/mnt/backup` indicates that this is used as backup destination, so this depends on the specific use case
Author
Contributor

With some try & error I tuned the security settings for my use case and reached an OK rating from systemd-analyze security borgmatic.

[Service]

# Security
ProtectSystem=strict
PrivateTmp=yes
PrivateDevices=yes
NoNewPrivileges=yes
LockPersonality=true
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
MemoryDenyWriteExecute=yes
RestrictRealtime=yes
RestrictNamespaces=yes
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 AF_NETLINK
ReadWritePaths=-/root/.config -/root/.cache  -/root/.borgmatic -/mnt/some_backup_drive
SystemCallArchitectures=native
SystemCallFilter=@system-service
ProtectClock=yes
ProtectHome=read-only
ProtectKernelLogs=yes
RestrictSUIDSGID=yes
ProtectHostname=yes

CapabilityBoundingSet=CAP_DAC_READ_SEARCH CAP_NET_RAW

Write access to /root/.borgmatic is required for the mysql backup.

The CAP_NET_RAW is required for my pre-backup hook which uses ping to check the reachability of the remote server.

Interestingly the value calculated by systemd-analyze security borgmatic differs between Ubuntu 20.04.1 and Solus (latest version), even though both are on systemd v245 release. For some reason the values per security setting differ, e.g. PrivateNetwork on Solus is rated with 0.6 and on Ubuntu with 0.5 and some settinsg are ignored on Solus.

The systemd seems to use different settings on each system:

# Ubuntu 20.04.1
systemd 245 (245.4-4ubuntu3.2)
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid

# Solus
systemd 245 (245)
+PAM -AUDIT -SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID -ELFUTILS +KMOD -IDN2 -IDN -PCRE2 default-hierarchy=legacy

On Solus I get a rating of 4.4 (OK) and on Ubuntu 2.5 (OK).

With some try & error I tuned the security settings for my use case and reached an `OK` rating from `systemd-analyze security borgmatic`. ``` [Service] # Security ProtectSystem=strict PrivateTmp=yes PrivateDevices=yes NoNewPrivileges=yes LockPersonality=true ProtectKernelTunables=yes ProtectKernelModules=yes ProtectControlGroups=yes MemoryDenyWriteExecute=yes RestrictRealtime=yes RestrictNamespaces=yes RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 AF_NETLINK ReadWritePaths=-/root/.config -/root/.cache -/root/.borgmatic -/mnt/some_backup_drive SystemCallArchitectures=native SystemCallFilter=@system-service ProtectClock=yes ProtectHome=read-only ProtectKernelLogs=yes RestrictSUIDSGID=yes ProtectHostname=yes CapabilityBoundingSet=CAP_DAC_READ_SEARCH CAP_NET_RAW ``` Write access to `/root/.borgmatic` is required for the `mysql` backup. The `CAP_NET_RAW` is required for my pre-backup hook which uses `ping` to check the reachability of the remote server. Interestingly the value calculated by `systemd-analyze security borgmatic` differs between Ubuntu 20.04.1 and Solus (latest version), even though both are on systemd v245 release. For some reason the values per security setting differ, e.g. `PrivateNetwork` on Solus is rated with `0.6` and on Ubuntu with `0.5` and some settinsg are ignored on Solus. The systemd seems to use different settings on each system: ``` # Ubuntu 20.04.1 systemd 245 (245.4-4ubuntu3.2) +PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid # Solus systemd 245 (245) +PAM -AUDIT -SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID -ELFUTILS +KMOD -IDN2 -IDN -PCRE2 default-hierarchy=legacy ``` On Solus I get a rating of 4.4 (OK) and on Ubuntu 2.5 (OK).
Owner

I appreciate all the work you are putting in on this! I'll try to answer some of the questions that you've raised during your investigations:

I’m not sure if borgmatic needs write access anywhere besides the backup destination.

Yeah, as you discovered ~/.borgmatic is needed for backup dumps.

DB backup
Not sure if this requires any additional capabilities/permissions?

In terms of read-write disk access, ~/.borgmatic is it. But connecting to the database does use local or remote network.

Is bormatic creating special files with mknod?

It's not using the mknod command. It is creating named pipes for use in streaming database dumps.

Is chroot or setns used by borgmatic?

Nope.

Also, as for some of the particular settings:

ReadWritePaths=-/root/.config -/root/.cache ...

You could probably restrict these further to /root/.config/borg and /root/.cache/borg respectively.

The rest of the settings look fine to me, just giving a skim of the systemd.exec man page, and given my limited systemd knowledge. I think the main challenge will be for users to remember to update ReadWritePaths with any repository destinations.

Did you want to submit a PR for all of these additions?

I appreciate all the work you are putting in on this! I'll try to answer some of the questions that you've raised during your investigations: > I’m not sure if borgmatic needs write access anywhere besides the backup destination. Yeah, as you discovered `~/.borgmatic` is needed for backup dumps. > DB backup > Not sure if this requires any additional capabilities/permissions? In terms of read-write disk access, `~/.borgmatic` is it. But connecting to the database does use local or remote network. > Is bormatic creating special files with mknod? It's not using the `mknod` command. It is creating named pipes for use in streaming database dumps. > Is chroot or setns used by borgmatic? Nope. Also, as for some of the particular settings: > ReadWritePaths=-/root/.config -/root/.cache ... You could probably restrict these further to `/root/.config/borg` and `/root/.cache/borg` respectively. The rest of the settings look fine to me, just giving a skim of the `systemd.exec` man page, and given my limited systemd knowledge. I think the main challenge will be for users to remember to update `ReadWritePaths` with any repository destinations. Did you want to submit a PR for all of these additions?
witten added the
security
label 2020-08-12 18:11:31 +00:00
Author
Contributor

@witten thanks for your feedback and answers.

I will continue my test for a while (1-2 weeks maybe) to ensure that nothing strange happens or anyone else has a comment/suggestion, before submitting a PR.

Besides updating the template file (with some comments). Shoudl I comment-out all or most of this settings by default to ensure that it doesn't break anything before it is proprely customised? Alternative would be to expect that the user checks the settings and adjusts them if needed.

In addition some brief explanation to the documentation How to set up backups with borgmatic > systmd could help, maybe just a hint that the borgmatic.service file requires some customisation to improve the secuirity.

@witten thanks for your feedback and answers. I will continue my test for a while (1-2 weeks maybe) to ensure that nothing strange happens or anyone else has a comment/suggestion, before submitting a PR. Besides updating the template file (with some comments). Shoudl I comment-out all or most of this settings by default to ensure that it doesn't break anything before it is proprely customised? Alternative would be to expect that the user checks the settings and adjusts them if needed. In addition some brief explanation to the documentation [How to set up backups with borgmatic > systmd](https://torsion.org/borgmatic/docs/how-to/set-up-backups/#systemd) could help, maybe just a hint that the `borgmatic.service` file requires some customisation to improve the secuirity.
Owner

Sounds good. I'm okay if you want to leave these settings uncommented.. This is only a sample configuration file anyway, so it's okay in my opinion if the user has to do some customization instead of copying it blindly. Your choice though.

In any case, it might be nice to update the comment in borgmatic/config/schema.yaml for the repositories option to remind the user to update the systemd ReadWritePaths for any local repositories if they are in fact using systemd.

In addition some brief explanation to the documentation How to set up backups with borgmatic > systmd could help, maybe just a hint that the borgmatic.service file requires some customisation to improve the secuirity.

Yes, that makes sense to me!

Sounds good. I'm okay if you want to leave these settings uncommented.. This is only a sample configuration file anyway, so it's okay in my opinion if the user has to do some customization instead of copying it blindly. Your choice though. In any case, it might be nice to update the comment in `borgmatic/config/schema.yaml` for the `repositories` option to remind the user to update the systemd `ReadWritePaths` for any local repositories if they are in fact using systemd. > In addition some brief explanation to the documentation How to set up backups with borgmatic > systmd could help, maybe just a hint that the borgmatic.service file requires some customisation to improve the secuirity. Yes, that makes sense to me!
Owner

Merged into master!

Merged into master!
Contributor

Thanks for your (collective) work on this. :)

I've enabled these settings on one system, so far, and had to make one modification. Pinging to healthchecks I have to change MemoryDenyWriteExecute to no, otherwise I see the following:

MemoryError: Cannot allocate write+execute memory for ffi.callback(). You might be running on a system that prevents this. For more information, se
e https://cffi.readthedocs.io/en/latest/using.html#callbacks

(Just for the sake of documenting it…)

Thanks for your (collective) work on this. :) I've enabled these settings on one system, so far, and had to make one modification. Pinging to healthchecks I have to change `MemoryDenyWriteExecute` to no, otherwise I see the following: ``` MemoryError: Cannot allocate write+execute memory for ffi.callback(). You might be running on a system that prevents this. For more information, se e https://cffi.readthedocs.io/en/latest/using.html#callbacks ``` (Just for the sake of documenting it…)
Owner

Thanks for the testing and reporting back! I'll go ahead and make that change to master so as not to prevent Healthchecks pings.

Thanks for the testing and reporting back! I'll go ahead and make that change to master so as not to prevent Healthchecks pings.
Author
Contributor

Thanks for testing and updating this setting. I'm not using healthchecks, that's why it worked for me with MemoryDenyWriteExecute=yes.

The systemd manual MemoryDenyWriteExecute mentions some cases where it causes issues:

Note that this option is incompatible with programs and libraries that generate program code dynamically at runtime, including JIT execution engines, executable stacks, and code "trampoline" feature of various C compilers.

Assume that this is affecting the Healthchecks function.

Maybe add a comment in the service sample that this function may be set to "yes" to further improve security but conflicts with the Healthchecks hook and maybe other features.

Looking at the code for the Healthchecks hook it might be that requests causes the issue incirectly. In a Google search I found a hint that the certificat verification of the https request may cause it by using a C library: mgr/dashboard: openssl exception when verifying certificates of HTTPS requests. This means that any hook that uses https would be impacted.

Thanks for testing and updating this setting. I'm not using healthchecks, that's why it worked for me with `MemoryDenyWriteExecute=yes`. The systemd manual [MemoryDenyWriteExecute](https://www.freedesktop.org/software/systemd/man/systemd.exec.html#MemoryDenyWriteExecute=) mentions some cases where it causes issues: > Note that this option is incompatible with programs and libraries that generate program code dynamically at runtime, including JIT execution engines, executable stacks, and code "trampoline" feature of various C compilers. Assume that this is affecting the Healthchecks function. Maybe add a comment in the service sample that this function may be set to "yes" to further improve security but conflicts with the Healthchecks hook and maybe other features. Looking at the code for the Healthchecks hook it might be that `requests` causes the issue incirectly. In a Google search I found a hint that the certificat verification of the https request may cause it by using a C library: [mgr/dashboard: openssl exception when verifying certificates of HTTPS requests](https://tracker.ceph.com/issues/39628). This means that any hook that uses https would be impacted.
Owner

Good call. Done!

Good call. Done!
Sign in to join this conversation.
No Milestone
No Assignees
3 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#352
No description provided.