How to run borgmatic with launchd on macOS #293

New Issue

oskapt · 2020-02-07T12:51:36Z

oskapt commented

2020-02-07 12:51:36 +00:00

Hi, Dan. I opened #265 a while back, and after much back and forth, I've landed on a working solution for running borgmatic on macOS (tested on 10.14.6).

The Problem

macOS has a security boundary that prevents applications and scripts from reading private data unless they've been granted "Full Disk Access" (FDA) in the Security preferences pane.

FDA is a bit of a black box. Even the developers don't understand how it works. FDA privileges can pass from one process to other processes that it spawns, but only if the parent process is a bundled application that resides in the Applications folder. It does not apply to scripts, even if they're compiled into binaries. It may not pass through borgmatic to borg, but it's a right pain to troubleshoot because of the lack of logs.

Because of all these opaque conditions, the solution I'm proposing may have more steps than are necessary. I've tried to eliminate variables one at a time to arrive at the minimum number of steps.

The Solution

Install borg and borgmatic
Create a script called run-borgmatic.sh with simple contents:
```
#!/bin/bash

./borgmatic --verbosity 1
```
Use platypus to turn the script into a bundled application (platypus-values.png)

select the script first (because it changes the app name)
set script type (bash)
set app name (Borgmatic)
set Interface to None
set to Run in background
add /usr/local/Cellar/borgmatic/{VERSION}/libexec/bin/borgmatic to the list of bundled files (platypus-adding-borgmatic.png)
click 'Create App'

This will create Borgmatic.app - move this into /Applications
Open the Security & Privacy Preferences pane and unlock it if necessary
Select Full Disk Access
Click the + to add an application and select Borgmatic.app in /Applications

You may have to also add /usr/local/Caskroom/borgbackup/{VERSION}/borg-macosx64, but I'm unsure if it's needed here. Try without it, and if it fails, add it.

Create /Library/LaunchDaemons/com.whatever.borgmatic.plist from the contents of borgmatic.plist (I tried to attach this as a .txt but Gitea won't let me.)

Label: unimportant. I generated this plist from Zerowidth. This label is used if you use launchctl start/stop to kick off or kill the process manually.
ProgramArguments: Set this to the location of the app, but point it to the borgmatic binary that was bundled into the app. The only part that needs to change is the app name (the part that ends in .app) if the app was named something other than borgmatic.app.
Standard(Out|Error)Path: I'm writing out to /var/log/borgmatic.log and rotating that file automatically w/ the rest of the macOS logfiles.
StartCalendarInterval: When the job should run. The main reason for using launchd over cron is that launchd will run a process later if the start time has already passed, and it will do it exactly once. If my laptop is off for 3 days, and I turn it on at 10am on the 4th day, launchd will run exactly one borgmatic job as soon as the machine boots up, even though 4 jobs were skipped.
UserName: if not root, borgmatic/borg will error on files that it can't copy.

Load the plist:

sudo launchctl load -w /Library/LaunchDaemons/com.whatever.borgmatic.plist`

If you want to start a job right away, you can do so with:

sudo launchctl start com.zerowidth.launched.borgmatic
sudo tail -f /var/log/borgmatic.log

You've probably noticed that we don't actually use the script that we created. That's because launchd needs to call the actual binary inside the bundle, but Platypus builds bundles from scripts. We can still double-click the application to kick off a manual borgmatic run, but then you'll be running it as yourself, not as root.

There might be a better way to do this with a different bundler. It's voodoo. I thought that my previous method wasn't working because it wouldn't log anything to anywhere, but then I discovered that it was actually making backups. I don't know if those backups include restricted content. I prefer this method over the previous one because this method writes out to the logfile.

Hi, Dan. I opened #265 a while back, and after much back and forth, I've landed on a working solution for running borgmatic on macOS (tested on 10.14.6). ## The Problem macOS has a security boundary that prevents applications and scripts from reading private data unless they've been granted "Full Disk Access" (FDA) in the Security preferences pane. FDA is a bit of a black box. [Even the developers don't understand](https://forums.developer.apple.com/thread/107546) how it works. FDA privileges can pass from one process to other processes that it spawns, but only if the parent process is a bundled application that resides in the Applications folder. It does not apply to scripts, even if they're compiled into binaries. It _may_ not pass through `borgmatic` to `borg`, but it's a right pain to troubleshoot because of the lack of logs. Because of all these opaque conditions, the solution I'm proposing may have more steps than are necessary. I've tried to eliminate variables one at a time to arrive at the minimum number of steps. ## The Solution 1. Install borg and borgmatic 2. Create a script called `run-borgmatic.sh` with simple contents: ``` #!/bin/bash ./borgmatic --verbosity 1 ``` 3. Use [platypus](https://sveinbjorn.org/platypus) to turn the script into a bundled application (platypus-values.png) - select the script first (because it changes the app name) - set script type (bash) - set app name (Borgmatic) - set Interface to `None` - set to `Run in background` - add `/usr/local/Cellar/borgmatic/{VERSION}/libexec/bin/borgmatic` to the list of bundled files (platypus-adding-borgmatic.png) - click 'Create App' 4. This will create `Borgmatic.app` - move this into `/Applications` 5. Open the Security & Privacy Preferences pane and unlock it if necessary 6. Select Full Disk Access 7. Click the `+` to add an application and select Borgmatic.app in `/Applications` - You may have to also add `/usr/local/Caskroom/borgbackup/{VERSION}/borg-macosx64`, but I'm unsure if it's needed here. Try without it, and if it fails, add it. 8. Create `/Library/LaunchDaemons/com.whatever.borgmatic.plist` from the contents of [borgmatic.plist](https://gist.github.com/oskapt/21b9d80dcfd7bb278e20fa38e2d72f53) (I tried to attach this as a `.txt` but Gitea won't let me.) - `Label`: unimportant. I generated this plist from [Zerowidth](http://launched.zerowidth.com/). This label is used if you use `launchctl start/stop` to kick off or kill the process manually. - `ProgramArguments`: Set this to the location of the app, but point it to the `borgmatic` binary that was bundled _into_ the app. The only part that needs to change is the app name (the part that ends in `.app`) if the app was named something other than `borgmatic.app`. - `Standard(Out|Error)Path`: I'm writing out to `/var/log/borgmatic.log` and rotating that file automatically w/ the rest of the macOS logfiles. - `StartCalendarInterval`: When the job should run. The main reason for using `launchd` over `cron` is that `launchd` will run a process later if the start time has already passed, and it will do it exactly once. If my laptop is off for 3 days, and I turn it on at 10am on the 4th day, `launchd` will run exactly one borgmatic job as soon as the machine boots up, even though 4 jobs were skipped. - `UserName`: if not root, borgmatic/borg will error on files that it can't copy. 9. Load the plist: ```bash sudo launchctl load -w /Library/LaunchDaemons/com.whatever.borgmatic.plist` ``` 10. If you want to start a job right away, you can do so with: ```bash sudo launchctl start com.zerowidth.launched.borgmatic sudo tail -f /var/log/borgmatic.log ``` You've probably noticed that we don't actually use the script that we created. That's because launchd needs to call the actual binary inside the bundle, but Platypus builds bundles from scripts. We can still double-click the application to kick off a manual borgmatic run, but then you'll be running it as yourself, not as root. There might be a better way to do this with a different bundler. It's voodoo. I thought that my previous method wasn't working because it wouldn't log anything to anywhere, but then I discovered that it was actually making backups. I don't know if those backups include restricted content. I prefer this method over the previous one because this method writes out to the logfile.

platypus-adding-borgmatic.png

312 KiB

platypus-values.png

85 KiB

oskapt commented

2020-02-07 13:04:06 +00:00

An awesome caveat here is that if borgmatic is updated, the app has to be re-bundled with the new binary, and that may also require removing and re-adding it to the FDA section in Security Preferences. FDA tends to respond poorly to things changing after they've been granted permission.

An awesome caveat here is that if `borgmatic` is updated, the app has to be re-bundled with the new binary, and that _may_ also require removing and re-adding it to the FDA section in Security Preferences. FDA tends to respond poorly to things changing after they've been granted permission.

witten commented

2020-02-07 18:07:08 +00:00

Wow, thanks for figuring out this process and writing it up! Super obnoxious though that you have to go through this to give borgmatic full disk access on macOS. I think our best bet at this point is to link to your series of steps from the borgmatic docs, unless you have other ideas.

Side note: .txt file uploads may be permitted now. (Gitea config change.)

Wow, thanks for figuring out this process and writing it up! Super obnoxious though that you have to go through this to give borgmatic full disk access on macOS. I think our best bet at this point is to link to your series of steps from the borgmatic docs, unless you have other ideas. Side note: `.txt` file uploads may be permitted now. (Gitea config change.)

oskapt commented

2020-02-09 00:01:57 +00:00

You're welcome, and yes, it's frustrating. :) Feel free to link the docs to this issue (and close it), and if I come up with a better way, I'll let you know.

witten referenced this issue from a commit

2020-04-26 23:12:51 +00:00

Documentation on macOS launchd permissions issues with work-around for Full Disk Access (#293).

witten commented

2020-04-26 23:37:09 +00:00

I've linked to this ticket from the docs! Thanks again for the investigation and write-up.

I've linked to this ticket from [the docs](https://torsion.org/borgmatic/docs/how-to/set-up-backups/#launchd-in-macos)! Thanks again for the investigation and write-up.

witten closed this issue

2020-04-26 23:37:09 +00:00

Fackelmann commented

2020-08-12 03:02:32 +00:00

Hi,

Thanks for the write up! Unfortunately this doesn't work for me... I still get the feared:

[Errno 2] No such file or directory: 'borg'

I have added both the platypus binary and the borg binary (in my case located in /usr/local/bin/borg) to FDA, but no luck. It's a shame that Apple make it so hard to use our computers however we want. No luck with cron either.

One minor thing, since the user we execute is root, I had the create a config file in /etc/borgmatic/config.yaml. It couldn't find one otherwise.

EDIT: I'll have to give Vorta a try.

Hi, Thanks for the write up! Unfortunately this doesn't work for me... I still get the feared: `[Errno 2] No such file or directory: 'borg'` I have added both the platypus binary and the borg binary (in my case located in /usr/local/bin/borg) to FDA, but no luck. It's a shame that Apple make it so hard to use our computers however we want. No luck with cron either. One minor thing, since the user we execute is root, I had the create a config file in `/etc/borgmatic/config.yaml`. It couldn't find one otherwise. EDIT: I'll have to give Vorta a try.

melissaboiko commented

2020-08-18 14:58:29 +00:00

Same as @Fackelmann – getting no such file for borg despite having bundled (from /usr/local/bin, there’s no such path as described in the tutorial after brew’ing it in my system).

I will also resort to Vorta, but I wanted to backup more than just the user folder, and Vorta apparently doesn’t even run as root, so it’s not a solution to the FDA problem this scheme is trying to solve. I even considered triggering borgmatic via ssh from the remote borg server (doing an ssh round-trip), just to be able to use cron in a sane system—but then of course you lose the benefit of anacron-like behaviour, unless you reinvent anacron over ssh.

Same as @Fackelmann – getting no such file for `borg` despite having bundled (from `/usr/local/bin`, there’s no such path as described in the tutorial after brew’ing it in my system). I will also resort to Vorta, but I wanted to backup more than just the user folder, and Vorta apparently doesn’t even run as root, so it’s not a solution to the FDA problem this scheme is trying to solve. I even considered triggering borgmatic via ssh _from_ the remote borg server (doing an ssh round-trip), just to be able to use cron in a sane system—but then of course you lose the benefit of anacron-like behaviour, unless you reinvent anacron over ssh.

witten commented

2020-08-18 17:00:08 +00:00

I'm not a macOS user and so it's entirely possible I don't know what I'm talking about here, but have you tried setting the local_path configuration option in borgmatic to the full path of your Borg binary? Example:

location:
    ...
    local_path: /usr/local/bin/borg

I'm not a macOS user and so it's entirely possible I don't know what I'm talking about here, but have you tried setting the `local_path` configuration option in borgmatic to the full path of your Borg binary? Example: ```yaml location: ... local_path: /usr/local/bin/borg ```

melissaboiko commented

2020-08-20 07:00:46 +00:00

@witten That’s a good idea. I don’t know anything about this OS either. The questions are, can you access commands in the root fs from ‘inside’ the Native Mac Application Bundle? And if you can, will the FDA thing ‘pass’ to the borg script?

I will try it for science when I have access to that hellish maschine again.

@witten That’s a good idea. I don’t know anything about this OS either. The questions are, _can_ you access commands in the root fs from ‘inside’ the Native Mac Application Bundle? And if you can, will the FDA thing ‘pass’ to the borg script? I will try it for science when I have access to that hellish maschine again.

oskapt commented

2020-08-20 10:25:53 +00:00

It is very frustrating and behaves contrary to logic. I've actually stopped using borgmatic on my Macs because, while I trust borgmatic implicitly, I don't trust MacOS permissions allowing a restore to happen. Catalina has the whole read-only rootfs with overlayfs mounts to subdirectories...I just went back to Time Machine.

mconigliaro commented

2021-01-18 22:40:38 +00:00

I'm not sure why this works, but another solution is to add /bin/bash to the list of apps that get full disk access.

I'm not sure why this works, but another solution is to add `/bin/bash` to the list of apps that get full disk access.

notDavid commented

2023-08-10 16:32:11 +00:00

Hello all, i saw this referenced from the Docs, so i thought it might be useful to mention, the way i do this;

Create a LaunchDaemon with the LaunchControl app, and then my script uses fdautil, creates an APFS snapshot of the whole drive, and then runs the backup with Borgmatic.

Here's a script that should give an overview of how to do it, it needs to be adjusted for Borgmatic of course: https://gist.github.com/QuantumGhost/1aae8eb8527c9d522fe2a57f214f6ee5

Hello all, i saw this referenced from the Docs, so i thought it might be useful to mention, the way i do this; Create a LaunchDaemon with the [LaunchControl](https://www.soma-zone.com/LaunchControl/) app, and then my script uses [fdautil](https://www.soma-zone.com/LaunchControl/FAQ.html), creates an APFS snapshot of the whole drive, and then runs the backup with Borgmatic. Here's a script that should give an overview of how to do it, it needs to be adjusted for Borgmatic of course: https://gist.github.com/QuantumGhost/1aae8eb8527c9d522fe2a57f214f6ee5

👍 1

Sign in to join this conversation.