Hi, Dan. I opened #265 a while back, and after much back and forth, I've landed on a working solution for running borgmatic on macOS (tested on 10.14.6).
The Problem
macOS has a security boundary that prevents applications and scripts from reading private data unless they've been granted "Full Disk Access" (FDA) in the Security preferences pane.
FDA is a bit of a black box. Even the developers don't understand how it works. FDA privileges can pass from one process to other processes that it spawns, but only if the parent process is a bundled application that resides in the Applications folder. It does not apply to scripts, even if they're compiled into binaries. It may not pass through borgmatic to borg, but it's a right pain to troubleshoot because of the lack of logs.
Because of all these opaque conditions, the solution I'm proposing may have more steps than are necessary. I've tried to eliminate variables one at a time to arrive at the minimum number of steps.
The Solution
Install borg and borgmatic
Create a script called run-borgmatic.sh with simple contents:
#!/bin/bash
./borgmatic --verbosity 1
Use platypus to turn the script into a bundled application (platypus-values.png)
select the script first (because it changes the app name)
set script type (bash)
set app name (Borgmatic)
set Interface to None
set to Run in background
add /usr/local/Cellar/borgmatic/{VERSION}/libexec/bin/borgmatic to the list of bundled files (platypus-adding-borgmatic.png)
click 'Create App'
This will create Borgmatic.app - move this into /Applications
Open the Security & Privacy Preferences pane and unlock it if necessary
Select Full Disk Access
Click the + to add an application and select Borgmatic.app in /Applications
You may have to also add /usr/local/Caskroom/borgbackup/{VERSION}/borg-macosx64, but I'm unsure if it's needed here. Try without it, and if it fails, add it.
Create /Library/LaunchDaemons/com.whatever.borgmatic.plist from the contents of borgmatic.plist (I tried to attach this as a .txt but Gitea won't let me.)
Label: unimportant. I generated this plist from Zerowidth. This label is used if you use launchctl start/stop to kick off or kill the process manually.
ProgramArguments: Set this to the location of the app, but point it to the borgmatic binary that was bundled into the app. The only part that needs to change is the app name (the part that ends in .app) if the app was named something other than borgmatic.app.
Standard(Out|Error)Path: I'm writing out to /var/log/borgmatic.log and rotating that file automatically w/ the rest of the macOS logfiles.
StartCalendarInterval: When the job should run. The main reason for using launchd over cron is that launchd will run a process later if the start time has already passed, and it will do it exactly once. If my laptop is off for 3 days, and I turn it on at 10am on the 4th day, launchd will run exactly one borgmatic job as soon as the machine boots up, even though 4 jobs were skipped.
UserName: if not root, borgmatic/borg will error on files that it can't copy.
You've probably noticed that we don't actually use the script that we created. That's because launchd needs to call the actual binary inside the bundle, but Platypus builds bundles from scripts. We can still double-click the application to kick off a manual borgmatic run, but then you'll be running it as yourself, not as root.
There might be a better way to do this with a different bundler. It's voodoo. I thought that my previous method wasn't working because it wouldn't log anything to anywhere, but then I discovered that it was actually making backups. I don't know if those backups include restricted content. I prefer this method over the previous one because this method writes out to the logfile.
Hi, Dan. I opened #265 a while back, and after much back and forth, I've landed on a working solution for running borgmatic on macOS (tested on 10.14.6).
## The Problem
macOS has a security boundary that prevents applications and scripts from reading private data unless they've been granted "Full Disk Access" (FDA) in the Security preferences pane.
FDA is a bit of a black box. [Even the developers don't understand](https://forums.developer.apple.com/thread/107546) how it works. FDA privileges can pass from one process to other processes that it spawns, but only if the parent process is a bundled application that resides in the Applications folder. It does not apply to scripts, even if they're compiled into binaries. It _may_ not pass through `borgmatic` to `borg`, but it's a right pain to troubleshoot because of the lack of logs.
Because of all these opaque conditions, the solution I'm proposing may have more steps than are necessary. I've tried to eliminate variables one at a time to arrive at the minimum number of steps.
## The Solution
1. Install borg and borgmatic
2. Create a script called `run-borgmatic.sh` with simple contents:
```
#!/bin/bash
./borgmatic --verbosity 1
```
3. Use [platypus](https://sveinbjorn.org/platypus) to turn the script into a bundled application (platypus-values.png)
- select the script first (because it changes the app name)
- set script type (bash)
- set app name (Borgmatic)
- set Interface to `None`
- set to `Run in background`
- add `/usr/local/Cellar/borgmatic/{VERSION}/libexec/bin/borgmatic` to the list of bundled files (platypus-adding-borgmatic.png)
- click 'Create App'
4. This will create `Borgmatic.app` - move this into `/Applications`
5. Open the Security & Privacy Preferences pane and unlock it if necessary
6. Select Full Disk Access
7. Click the `+` to add an application and select Borgmatic.app in `/Applications`
- You may have to also add `/usr/local/Caskroom/borgbackup/{VERSION}/borg-macosx64`, but I'm unsure if it's needed here. Try without it, and if it fails, add it.
8. Create `/Library/LaunchDaemons/com.whatever.borgmatic.plist` from the contents of [borgmatic.plist](https://gist.github.com/oskapt/21b9d80dcfd7bb278e20fa38e2d72f53) (I tried to attach this as a `.txt` but Gitea won't let me.)
- `Label`: unimportant. I generated this plist from [Zerowidth](http://launched.zerowidth.com/). This label is used if you use `launchctl start/stop` to kick off or kill the process manually.
- `ProgramArguments`: Set this to the location of the app, but point it to the `borgmatic` binary that was bundled _into_ the app. The only part that needs to change is the app name (the part that ends in `.app`) if the app was named something other than `borgmatic.app`.
- `Standard(Out|Error)Path`: I'm writing out to `/var/log/borgmatic.log` and rotating that file automatically w/ the rest of the macOS logfiles.
- `StartCalendarInterval`: When the job should run. The main reason for using `launchd` over `cron` is that `launchd` will run a process later if the start time has already passed, and it will do it exactly once. If my laptop is off for 3 days, and I turn it on at 10am on the 4th day, `launchd` will run exactly one borgmatic job as soon as the machine boots up, even though 4 jobs were skipped.
- `UserName`: if not root, borgmatic/borg will error on files that it can't copy.
9. Load the plist:
```bash
sudo launchctl load -w /Library/LaunchDaemons/com.whatever.borgmatic.plist`
```
10. If you want to start a job right away, you can do so with:
```bash
sudo launchctl start com.zerowidth.launched.borgmatic
sudo tail -f /var/log/borgmatic.log
```
You've probably noticed that we don't actually use the script that we created. That's because launchd needs to call the actual binary inside the bundle, but Platypus builds bundles from scripts. We can still double-click the application to kick off a manual borgmatic run, but then you'll be running it as yourself, not as root.
There might be a better way to do this with a different bundler. It's voodoo. I thought that my previous method wasn't working because it wouldn't log anything to anywhere, but then I discovered that it was actually making backups. I don't know if those backups include restricted content. I prefer this method over the previous one because this method writes out to the logfile.
An awesome caveat here is that if borgmatic is updated, the app has to be re-bundled with the new binary, and that may also require removing and re-adding it to the FDA section in Security Preferences. FDA tends to respond poorly to things changing after they've been granted permission.
An awesome caveat here is that if `borgmatic` is updated, the app has to be re-bundled with the new binary, and that _may_ also require removing and re-adding it to the FDA section in Security Preferences. FDA tends to respond poorly to things changing after they've been granted permission.
Wow, thanks for figuring out this process and writing it up! Super obnoxious though that you have to go through this to give borgmatic full disk access on macOS. I think our best bet at this point is to link to your series of steps from the borgmatic docs, unless you have other ideas.
Side note: .txt file uploads may be permitted now. (Gitea config change.)
Wow, thanks for figuring out this process and writing it up! Super obnoxious though that you have to go through this to give borgmatic full disk access on macOS. I think our best bet at this point is to link to your series of steps from the borgmatic docs, unless you have other ideas.
Side note: `.txt` file uploads may be permitted now. (Gitea config change.)
You're welcome, and yes, it's frustrating. :) Feel free to link the docs to this issue (and close it), and if I come up with a better way, I'll let you know.
You're welcome, and yes, it's frustrating. :) Feel free to link the docs to this issue (and close it), and if I come up with a better way, I'll let you know.
I've linked to this ticket from the docs! Thanks again for the investigation and write-up.
I've linked to this ticket from [the docs](https://torsion.org/borgmatic/docs/how-to/set-up-backups/#launchd-in-macos)! Thanks again for the investigation and write-up.
Thanks for the write up! Unfortunately this doesn't work for me... I still get the feared:
[Errno 2] No such file or directory: 'borg'
I have added both the platypus binary and the borg binary (in my case located in /usr/local/bin/borg) to FDA, but no luck. It's a shame that Apple make it so hard to use our computers however we want. No luck with cron either.
One minor thing, since the user we execute is root, I had the create a config file in /etc/borgmatic/config.yaml. It couldn't find one otherwise.
EDIT: I'll have to give Vorta a try.
Hi,
Thanks for the write up! Unfortunately this doesn't work for me... I still get the feared:
`[Errno 2] No such file or directory: 'borg'`
I have added both the platypus binary and the borg binary (in my case located in /usr/local/bin/borg) to FDA, but no luck. It's a shame that Apple make it so hard to use our computers however we want. No luck with cron either.
One minor thing, since the user we execute is root, I had the create a config file in `/etc/borgmatic/config.yaml`. It couldn't find one otherwise.
EDIT: I'll have to give Vorta a try.
Same as @Fackelmann – getting no such file for borg despite having bundled (from /usr/local/bin, there’s no such path as described in the tutorial after brew’ing it in my system).
I will also resort to Vorta, but I wanted to backup more than just the user folder, and Vorta apparently doesn’t even run as root, so it’s not a solution to the FDA problem this scheme is trying to solve. I even considered triggering borgmatic via ssh from the remote borg server (doing an ssh round-trip), just to be able to use cron in a sane system—but then of course you lose the benefit of anacron-like behaviour, unless you reinvent anacron over ssh.
Same as @Fackelmann – getting no such file for `borg` despite having bundled (from `/usr/local/bin`, there’s no such path as described in the tutorial after brew’ing it in my system).
I will also resort to Vorta, but I wanted to backup more than just the user folder, and Vorta apparently doesn’t even run as root, so it’s not a solution to the FDA problem this scheme is trying to solve. I even considered triggering borgmatic via ssh _from_ the remote borg server (doing an ssh round-trip), just to be able to use cron in a sane system—but then of course you lose the benefit of anacron-like behaviour, unless you reinvent anacron over ssh.
I'm not a macOS user and so it's entirely possible I don't know what I'm talking about here, but have you tried setting the local_path configuration option in borgmatic to the full path of your Borg binary? Example:
location:...local_path:/usr/local/bin/borg
I'm not a macOS user and so it's entirely possible I don't know what I'm talking about here, but have you tried setting the `local_path` configuration option in borgmatic to the full path of your Borg binary? Example:
```yaml
location:
...
local_path: /usr/local/bin/borg
```
@witten That’s a good idea. I don’t know anything about this OS either. The questions are, can you access commands in the root fs from ‘inside’ the Native Mac Application Bundle? And if you can, will the FDA thing ‘pass’ to the borg script?
I will try it for science when I have access to that hellish maschine again.
@witten That’s a good idea. I don’t know anything about this OS either. The questions are, _can_ you access commands in the root fs from ‘inside’ the Native Mac Application Bundle? And if you can, will the FDA thing ‘pass’ to the borg script?
I will try it for science when I have access to that hellish maschine again.
It is very frustrating and behaves contrary to logic. I've actually stopped using borgmatic on my Macs because, while I trust borgmatic implicitly, I don't trust MacOS permissions allowing a restore to happen. Catalina has the whole read-only rootfs with overlayfs mounts to subdirectories...I just went back to Time Machine.
It is very frustrating and behaves contrary to logic. I've actually stopped using borgmatic on my Macs because, while I trust borgmatic implicitly, I don't trust MacOS permissions allowing a restore to happen. Catalina has the whole read-only rootfs with overlayfs mounts to subdirectories...I just went back to Time Machine.
Hi, Dan. I opened #265 a while back, and after much back and forth, I've landed on a working solution for running borgmatic on macOS (tested on 10.14.6).
The Problem
macOS has a security boundary that prevents applications and scripts from reading private data unless they've been granted "Full Disk Access" (FDA) in the Security preferences pane.
FDA is a bit of a black box. Even the developers don't understand how it works. FDA privileges can pass from one process to other processes that it spawns, but only if the parent process is a bundled application that resides in the Applications folder. It does not apply to scripts, even if they're compiled into binaries. It may not pass through
borgmatictoborg, but it's a right pain to troubleshoot because of the lack of logs.Because of all these opaque conditions, the solution I'm proposing may have more steps than are necessary. I've tried to eliminate variables one at a time to arrive at the minimum number of steps.
The Solution
run-borgmatic.shwith simple contents:NoneRun in background/usr/local/Cellar/borgmatic/{VERSION}/libexec/bin/borgmaticto the list of bundled files (platypus-adding-borgmatic.png)Borgmatic.app- move this into/Applications+to add an application and select Borgmatic.app in/Applications/usr/local/Caskroom/borgbackup/{VERSION}/borg-macosx64, but I'm unsure if it's needed here. Try without it, and if it fails, add it./Library/LaunchDaemons/com.whatever.borgmatic.plistfrom the contents of borgmatic.plist (I tried to attach this as a.txtbut Gitea won't let me.)Label: unimportant. I generated this plist from Zerowidth. This label is used if you uselaunchctl start/stopto kick off or kill the process manually.ProgramArguments: Set this to the location of the app, but point it to theborgmaticbinary that was bundled into the app. The only part that needs to change is the app name (the part that ends in.app) if the app was named something other thanborgmatic.app.Standard(Out|Error)Path: I'm writing out to/var/log/borgmatic.logand rotating that file automatically w/ the rest of the macOS logfiles.StartCalendarInterval: When the job should run. The main reason for usinglaunchdovercronis thatlaunchdwill run a process later if the start time has already passed, and it will do it exactly once. If my laptop is off for 3 days, and I turn it on at 10am on the 4th day,launchdwill run exactly one borgmatic job as soon as the machine boots up, even though 4 jobs were skipped.UserName: if not root, borgmatic/borg will error on files that it can't copy.You've probably noticed that we don't actually use the script that we created. That's because launchd needs to call the actual binary inside the bundle, but Platypus builds bundles from scripts. We can still double-click the application to kick off a manual borgmatic run, but then you'll be running it as yourself, not as root.
There might be a better way to do this with a different bundler. It's voodoo. I thought that my previous method wasn't working because it wouldn't log anything to anywhere, but then I discovered that it was actually making backups. I don't know if those backups include restricted content. I prefer this method over the previous one because this method writes out to the logfile.
An awesome caveat here is that if
borgmaticis updated, the app has to be re-bundled with the new binary, and that may also require removing and re-adding it to the FDA section in Security Preferences. FDA tends to respond poorly to things changing after they've been granted permission.Wow, thanks for figuring out this process and writing it up! Super obnoxious though that you have to go through this to give borgmatic full disk access on macOS. I think our best bet at this point is to link to your series of steps from the borgmatic docs, unless you have other ideas.
Side note:
.txtfile uploads may be permitted now. (Gitea config change.)You're welcome, and yes, it's frustrating. :) Feel free to link the docs to this issue (and close it), and if I come up with a better way, I'll let you know.
I've linked to this ticket from the docs! Thanks again for the investigation and write-up.
Hi,
Thanks for the write up! Unfortunately this doesn't work for me... I still get the feared:
[Errno 2] No such file or directory: 'borg'I have added both the platypus binary and the borg binary (in my case located in /usr/local/bin/borg) to FDA, but no luck. It's a shame that Apple make it so hard to use our computers however we want. No luck with cron either.
One minor thing, since the user we execute is root, I had the create a config file in
/etc/borgmatic/config.yaml. It couldn't find one otherwise.EDIT: I'll have to give Vorta a try.
Same as @Fackelmann – getting no such file for
borgdespite having bundled (from/usr/local/bin, there’s no such path as described in the tutorial after brew’ing it in my system).I will also resort to Vorta, but I wanted to backup more than just the user folder, and Vorta apparently doesn’t even run as root, so it’s not a solution to the FDA problem this scheme is trying to solve. I even considered triggering borgmatic via ssh from the remote borg server (doing an ssh round-trip), just to be able to use cron in a sane system—but then of course you lose the benefit of anacron-like behaviour, unless you reinvent anacron over ssh.
I'm not a macOS user and so it's entirely possible I don't know what I'm talking about here, but have you tried setting the
local_pathconfiguration option in borgmatic to the full path of your Borg binary? Example:@witten That’s a good idea. I don’t know anything about this OS either. The questions are, can you access commands in the root fs from ‘inside’ the Native Mac Application Bundle? And if you can, will the FDA thing ‘pass’ to the borg script?
I will try it for science when I have access to that hellish maschine again.
It is very frustrating and behaves contrary to logic. I've actually stopped using borgmatic on my Macs because, while I trust borgmatic implicitly, I don't trust MacOS permissions allowing a restore to happen. Catalina has the whole read-only rootfs with overlayfs mounts to subdirectories...I just went back to Time Machine.
I'm not sure why this works, but another solution is to add
/bin/bashto the list of apps that get full disk access.