Borgmatic fails with strange error when running under launchd #265

Closed
opened 2019-12-02 13:47:29 +00:00 by oskapt · 3 comments

What I'm trying to do and why

Hello! I've moved borgmatic from cron to launchd (MacOS Mojave), and it won't run. It runs fine on the command-line, but when running out of launchd, it reports:

/etc/borgmatic/config.yaml: Error running configuration file
tm_user@nas:/volume2/borg/ferdinand: Error running actions for repository
[Errno 2] No such file or directory: 'borg': 'borg'

The relevant part of the config is this:

    repositories:
        - tm_user@nas:/volume2/borg/ferdinand

This directory exists and holds successful manual and cron-initiated backups.

Sniffing on the interface shows that it never actually makes the SSH connection, so the error message is masking some other problem - any thoughts?

Steps to reproduce (if a bug)

Include (sanitized) borgmatic configuration files if applicable.

Actual behavior (if a bug)

Include (sanitized) --verbosity 2 output if applicable.

Expected behavior (if a bug)

Other notes / implementation ideas

I suspect that this is an issue with MacOS process security, where something in the chain is not authorized to call something else. I've also made it an app with script editor and added that to full disk access (FDA), but it reports the same error, just in a popup instead of in the log.

Environment

borgmatic version: 1.4.15 (though was a problem with 1.4.0 as well)

borgmatic installation method: Homebrew

Borg version: 1.1.0

Python version: 3.7.5

operating system and version: MacOS 10.14.6

#### What I'm trying to do and why Hello! I've moved borgmatic from cron to launchd (MacOS Mojave), and it won't run. It runs fine on the command-line, but when running out of launchd, it reports: ```bash /etc/borgmatic/config.yaml: Error running configuration file tm_user@nas:/volume2/borg/ferdinand: Error running actions for repository [Errno 2] No such file or directory: 'borg': 'borg' ``` The relevant part of the config is this: ```yaml repositories: - tm_user@nas:/volume2/borg/ferdinand ``` This directory exists and holds successful manual and cron-initiated backups. Sniffing on the interface shows that it never actually makes the SSH connection, so the error message is masking some other problem - any thoughts? #### Steps to reproduce (if a bug) Include (sanitized) borgmatic configuration files if applicable. #### Actual behavior (if a bug) Include (sanitized) `--verbosity 2` output if applicable. #### Expected behavior (if a bug) #### Other notes / implementation ideas I suspect that this is an issue with MacOS process security, where something in the chain is not authorized to call something else. I've also made it an app with script editor and added that to full disk access (FDA), but it reports the same error, just in a popup instead of in the log. #### Environment **borgmatic version:** 1.4.15 (though was a problem with 1.4.0 as well) **borgmatic installation method:** Homebrew **Borg version:** 1.1.0 **Python version:** 3.7.5 **operating system and version:** MacOS 10.14.6
Owner

I appreciate the detailed ticket. The error [Errno 2] No such file or directory: 'borg': 'borg' suggests to me that borgmatic cannot find the borg executable. I haven't used launchd before, but if it's anything like cron, then the problem may be as simple as a default path issue rather than any sort of process security. (I've seen similar errors with cron when the default path doesn't contain borg's parent directory.)

So, one approach is to try setting the PATH environment variable to contain borg's parent directory. Note that the /etc/launchd.conf method is no longer supported for this.

If that doesn't do it, then you could just configure borgmatic to use Borg's absolute path. For instance, set the local_path option in borgmatic's configpuration to /your/absolute/path/to/borg, whatever that happens to be, and then borgmatic should be able to find it.

Let me know if any of those approaches work out for you.

I appreciate the detailed ticket. The error `[Errno 2] No such file or directory: 'borg': 'borg'` suggests to me that borgmatic cannot find the `borg` executable. I haven't used launchd before, but if it's anything like cron, then the problem *may* be as simple as a default path issue rather than any sort of process security. (I've seen similar errors with cron when the default path doesn't contain `borg`'s parent directory.) So, one approach is to try [setting the `PATH` environment](http://www.felfelworld.com/2014/04/14/adding-to-the-path-variable-on-mac-os-x/) variable to contain `borg`'s parent directory. Note that the `/etc/launchd.conf` method is [no longer supported](https://apple.stackexchange.com/questions/106355/setting-the-system-wide-path-environment-variable-in-mavericks) for this. If that doesn't do it, then you could just configure borgmatic to use Borg's absolute path. For instance, set the `local_path` option in borgmatic's configpuration to `/your/absolute/path/to/borg`, whatever that happens to be, and then borgmatic should be able to find it. Let me know if any of those approaches work out for you.
witten added the
question / support
label 2019-12-02 17:21:50 +00:00
Author

Ah! So obvious! Thanks. 😊

Adding it to /etc/borgmatic/config.yaml with the full path solved the error message, but it left me with additional launchd/FDA issues, which I'm working on solving.

Launchd is similar to cron, but cron is deprecated and will be removed in a future release (if not already gone in 10.15). Starting in 10.13 or 10.14 Apple added draconian security to MacOS that restricts what programs are allowed to access by default. There's a "Full Disk Access" list in System Preferences>Security, but no one really knows how it works. Without the correct program in this list, you'll get errors like:

/Users/monachus/Library/Calendars: scandir: 
  [Errno 1] Operation not permitted: '/Users/monachus/Library/Calendars'

This will happen with things that are considered sensitive, like user home directories, which makes it hard to back them up. I have borg in that list, but it still fails. This is where the Internet is unclear - I may have to go all the way up to launchd, or the sh that is kicking off the borgmatic process. What's stupid is that if I give full disk access to sh so that I can run backups, I've neutered the entire reason for having the process-level security in place at all.

They might want people to only run signed applications and collect the developer program fee, or maybe they've forgotten their *nix origins.

Well done, Apple.

I'll figure it out and submit some docs for borgmatic that spare others the suffering.

I see you've been hard at work on updates recently - well done! I've been manually updating the container monachus/borgmatic with each new release, but your pace has shown me that I really need to automate that. 😄

Ah! So obvious! Thanks. :blush: Adding it to `/etc/borgmatic/config.yaml` with the full path solved the error message, but it left me with additional launchd/FDA issues, which I'm working on solving. Launchd is similar to cron, but cron is deprecated and will be removed in a future release (if not already gone in 10.15). Starting in 10.13 or 10.14 Apple added draconian security to MacOS that restricts what programs are allowed to access by default. There's a "Full Disk Access" list in **System Preferences>Security**, but no one really knows how it works. Without the correct program in this list, you'll get errors like: ``` /Users/monachus/Library/Calendars: scandir: [Errno 1] Operation not permitted: '/Users/monachus/Library/Calendars' ``` This will happen with things that are considered sensitive, like user home directories, which makes it hard to back them up. I have `borg` in that list, but it still fails. This is where the Internet is unclear - I may have to go all the way up to `launchd`, or the `sh` that is kicking off the `borgmatic` process. What's stupid is that if I give full disk access to `sh` so that I can run backups, I've neutered the entire reason for having the process-level security in place at all. They might want people to only run signed applications and collect the developer program fee, or maybe they've forgotten their *nix origins. Well done, Apple. I'll figure it out and submit some docs for `borgmatic` that spare others the suffering. I see you've been hard at work on updates recently - well done! I've been manually updating the container `monachus/borgmatic` with each new release, but your pace has shown me that I really need to automate that. :smile:
Owner

I'm glad that solved the immediate path problem! As for the permission issue.. ugh, that sounds super frustrating. Here's an open ticket for some other backup software that may provide some ideas for work-arounds: https://github.com/restic/restic/issues/2051

If/when you do figure it out, I would love to get that included in borgmatic docs.

I know the rapid pace of releases can be obnoxious for packagers.. I'm trying to get fixes and improvements out with as low of turn-around time as I can.

I'm glad that solved the immediate path problem! As for the permission issue.. ugh, that sounds super frustrating. Here's an open ticket for some other backup software that may provide some ideas for work-arounds: https://github.com/restic/restic/issues/2051 If/when you do figure it out, I would love to get that included in borgmatic docs. I know the rapid pace of releases can be obnoxious for packagers.. I'm trying to get fixes and improvements out with as low of turn-around time as I can.
Sign in to join this conversation.
No Milestone
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#265
No description provided.