challenge from passcommand can end up hidden #961
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: borgmatic-collective/borgmatic#961
Loading…
x
Reference in New Issue
Block a user
No description provided.
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What I'm trying to do and why
I'm just starting with borg and borgmatic. The setup I'm trying is local borgmatic -> remote authorized key with
borg serveas forced-command -> remote borg repo, set up with repokey encryption.I'm trying to use
encryption_passcommand, pointed at my password manager, because i feel uneasy about leaving my repo passphrase around (in plaintext) in env vars or files on disk, in terms of defense-in-depth/lateral movement risk. I'm not sure how justified that feeling is (putting a repokey passphrase behind a different password at best adds an extra hurdle to someone who's only compromised my server), nor how practical this approach is (requiring interactive passcommand is at odds with the usual endgame of automating backups). I may well have an X-Y problem.I'm using
borg list/borgmatic repo-list -a '*'as my roundtrip test for remote borg and remote borgmatic.borgmatic repo-listandborgmatic repo-list -a '*'hang indefinitely, without any prompt for action from me. If I enter the password to my password manager, the command continues and gives me the list of archives in the repo.My impression is that it is trying to get my repo passphrase for a 2nd time (though this invocation already had it), and not showing the password prompt from my password manager.
Steps to reproduce
Given the configs below, the test command is:
my borgmatic config, at
/home/localuser/.config/borgmatic/config.yamllocally:
remote:
Actual behavior
The command hangs indefinitely:
I waited about 3 hours before I started messing with it.
If I type some stuff and press enter, borgmatic dies with stacktrace.
You can see in the above that the first line after where it hangs is my password manager asking for a password.
If I re-run, entering the password for my password database, the command completes normally, listing archives in the repo:
The above plaintext doesn't show terminal colors, but the colors of the two password prompts are different:
but:
That color difference is more noticeable with verbosity off:
Expected behavior
The password challenge issued from my (local) password manager executable:
Ideally, the borgmatic invocation would reuse the repo passphrase it's already retrieved and not need to issue a 2nd challenge. But I'm not sure if it's supposed to work this way.
Other notes / implementation ideas
No response
borgmatic version
1.9.5
borgmatic installation method
pip install
Borg version
borg 1.4.0
Python version
Python 3.12.3
Database version (if applicable)
n/a
Operating system and version
Ubuntu 24.04.1 LTS
Thanks for giving borgmatic a shot!
Yeah, it's really a question of the risk level you're comfortable with. And of course the tradeoff between security and convenience.
Any given invocation of a borgmatic action may call Borg multiple times. And in this particular case, it's calling
borg repo-listonce with--jsonin order to check for too-aggressive archive flags and once without--jsonto satisfy the user's ask to actually list the repo. Also, borgmatic doesn't actually do anything with your configured passcommand itself; that gets passed to Borg, which is responsible for calling the passcommand and prompting the user. The call order looks something like this:borgmatic repo-list ...borg repo-list --jsonwithBORG_PASSCOMMANDenvironment variableborg repo-listwithBORG_PASSCOMMANDenvironment variableSo what I think is going on here with the hang is that borgmatic's second call to
borg repo-listis collecting Borg's output, so as to log each line at the appropriate colored log level. And that includes Borg's call to your passcommand. However, a side effect of collecting the output is that anything that doesn't log a complete line—like the interactive prompt of your passcommand—doesn't show up. Which makes it appear that borgmatic is hanging since that prompt is invisible.One solution I originally thought of here is to disable the output collection for logging when a passcommand is configured. That works for your
repo-listcase, but not for other use cases. For instance,borgmatic restorefor database restoration currently callsborg extractand collects its output directly as part of the streaming database restore. So it wouldn't be possible to disable the output collection when a passcommand is set without breaking database restoration streaming.This suggests another potential solution. borgmatic could potentially not delegate passcommand invocation to Borg like it does now and instead call the passcommand itself. Then, borgmatic would have to somehow provide the passphrase to each individual invocation of Borg in a secure manner. The natural way would be using an environment variable, but my guess is that anyone who's bothering to use a passcommand probably wouldn't want their plaintext passphrase stuffed unceremoneously into an environment variable. Borg does however support passing in a passphrase by file descriptor, so that may be a sufficiently secure way to provide the passphrase to Borg.
The upshot is the call order in our example would look something like:
borgmatic repo-list ...borg repo-list --jsonwithBORG_PASSPHRASE_FD(file descriptor) environment variableborg repo-listwithBORG_PASSPHRASE_FD(file descriptor) environment variable... meaning that there would be: 1. only a single passcommand prompt regardless of how many times borgmatic runs Borg, and 2. the prompt would actually be visible.
Please let me know if you have thoughts on any of this.
Hi, thanks for the speedy reply (during the holidays), and sorry I didn't reply sooner.
Ah, makes sense.
Ohhhkaaaay. Yeah, as a beginner I haven't used/interacted much with the richer functionality, like backing up databases. If I understand correctly, the blocker is more or less that the various data sources don't really produce output/logging/errors in any one way more portable than STDIN/STDERR/exit code, so borgmatic needs to only rely on these, and in so doing it consumes these, whether they are presenting data or prompts. Makes sense.
This is why it took me so long to reply.... I had seen mention of
BORG_PASSPHRASE_FDwhen getting started, didn't really understand the use case, and shelved it as "I'd like to know but I will never bother to find out.". It took me a bit to google the right terms to develop an impression around it. As a responsible forum user, I'll give a bit of a summary of those notes below (apologies if it's spammy).I agree with your reasoning, that an env var would raise alarm bells for some folks.
For my use case, the proposal sounds good. As you say, hoisting passcommand execution into
borgmaticsolves multi-prompting as well as swallowed password prompts.I couldn't think of any downside. It adds a little bit of complexity to borgmatic, and it's hard to predict how many people would make use of it.
Notes on using file descriptors for ephemeral handling of secrets:
Best TLDR is https://security.stackexchange.com/a/190075 , mainly this quote
Places where these can leak:
commandline arguments:
ps,{,h}top, and/proc/$pid/cmdlinesecrets via env var
/proc/$pid/environdr-xr-xr-xandsecrets via FD
On my Ubuntu 24.04.1 LTS host:
https://lackingrhoticity.blogspot.com/2015/05/passing-fds-handles-between-processes.html
There are standard ways for a process to share its FDs with another process over a unix domain socket, with sample code published in one of those big white Stevens books. They need to be initiated by the owning process, and they seem rather fussy https://stackoverflow.com/a/28005250 . Some comments on https://unix.stackexchange.com/a/429028 look informative.
Since 2020 (linux 5.6), the
pidfd_getfdsyscall exists, for a process to duplicate an FD from another process, but it requires advanced access (PTRACE_MODE_ATTACH_REALCREDS, see https://www.man7.org/linux/man-pages/man2/ptrace.2.html ). sample code at https://stackoverflow.com/a/72135834 .I took some fun wrong turns that might be related:
gpgalso allows you to give password via FD, with--passphrase-fd ngdbmemfd_secretsyscall sounds like a riot:memfd_create/fcntlThanks for digging into the security aspects of this. I haven't looked into some of the fancier options like
memfd_secret(), but my instinct right now is that the anonymous pipe FD option is probably a good balance of convenience and security. Of course, my instinct may be wrong. There's also the consideration that borgmatic users are not all Linux users, as I've discovered! Anyway, I'll consider the resolution of this point an (important) implementation detail of this ticket.Additional note, mostly for myself or whoever implements this: #966 recently came up with a generalized way to load credentials within borgmatic's config file, which on its face has some vague adjacency to this ticket even if the specific approach is very different. At the risk of over-engineering, it makes me think that it may make sense to introduce the notion of borgmatic credential "providers" or "hooks." For instance, there could be one Borg passphrase provider, prompting for the passphrase and passing it to Borg as described in this ticket. And then there could be another provider for loading systemd credentials instead as described in #966. Additional credential providers/hooks could be added in the future as needed.
I have an initial implementation going! #984
Gee whiz that was fast.
This is implemented in main and will be part of the next release! I've tested it with
keepassxc-cliand it seems to work great.. It only prompts for a passphrase to unlock the kdbx file once now even though Borg is called multiple times. And no more apparent hangs. To keep things (relatively) simple, I ended up going with the file descriptor approach for passing the passphrase to Borg when a passcommand is used. We can always revisit in the future if needed.That's funny.. From my end it seemed like it took way longer than it should've. 😄
Released in borgmatic 1.9.9!