borgmatic-collective/borgmatic
borgmatic-collective/borgmatic
9
65
Fork 31
Code Issues 45 Pull Requests 4 Actions Packages Releases 147 Activity

Securely load ntfy access token or password #966

New Issue
Closed
opened 2025-01-12 18:19:10 +00:00 by mgrecar · 15 comments
mgrecar commented 2025-01-12 18:19:10 +00:00
Copy Link

What I'm trying to do and why

I managed to get my borgmatic config working with systemd credentials for the borg repository password, but I'm not finding a way to securely load the ntfy access token or password for the ntfy hook. I'd like to avoid storing it in plain text in the config file, and instead load it securely.

Steps to reproduce

No response

Actual behavior

No response

Expected behavior

No response

Other notes / implementation ideas

No response

borgmatic version

No response

borgmatic installation method

No response

Borg version

No response

Python version

No response

Database version (if applicable)

No response

Operating system and version

No response

### What I'm trying to do and why I managed to get my borgmatic config working with systemd credentials for the borg repository password, but I'm not finding a way to securely load the ntfy access token or password for the ntfy hook. I'd like to avoid storing it in plain text in the config file, and instead load it securely. ### Steps to reproduce _No response_ ### Actual behavior _No response_ ### Expected behavior _No response_ ### Other notes / implementation ideas _No response_ ### borgmatic version _No response_ ### borgmatic installation method _No response_ ### Borg version _No response_ ### Python version _No response_ ### Database version (if applicable) _No response_ ### Operating system and version _No response_
witten commented 2025-01-12 19:22:59 +00:00
Owner
Copy Link

Yeah, as you've probably seen, the way that systemd credentials get to Borg in the docs is via Borg running a "passcommand" ... which doesn't exist for ntfy.

I haven't tried it, but have you considered something like the following for the ntfy use case?

ntfy:
    topic: my-unique-topic
    server: https://ntfy.my-domain.com
    username: myuser
    password: !include "${CREDENTIALS_DIRECTORY}/borgmatic_ntfypass"
Yeah, as you've [probably seen](https://torsion.org/borgmatic/docs/how-to/provide-your-passwords/), the way that systemd credentials get to Borg in the docs is via Borg running a "passcommand" ... which doesn't exist for ntfy. I haven't tried it, but have you considered something like the following for the ntfy use case? ```yaml ntfy: topic: my-unique-topic server: https://ntfy.my-domain.com username: myuser password: !include "${CREDENTIALS_DIRECTORY}/borgmatic_ntfypass" ```
mgrecar commented 2025-01-12 20:28:36 +00:00
Author
Copy Link

Interesting idea, unfortunately, I couldn't get it to work. After wrestling with getting multiple systemd-creds to actually work, it doesn't seem that it's able to find the file to include it properly:

Could not find include ${CREDENTIALS_DIRECTORY}/borgmatic_ntfy.yaml at /etc/borgmatic/${CREDENTIALS_DIRECTORY}/borgmatic_ntfy.yaml or /${CREDENTIALS_DIRECTORY}/borgmatic_ntfy.yaml

from attempting to include it like:

access_token: !include "${CREDENTIALS_DIRECTORY}/borgmatic_ntfy.yaml"
Interesting idea, unfortunately, I couldn't get it to work. After wrestling with getting multiple `systemd-creds` to actually work, it doesn't seem that it's able to find the file to include it properly: ``` Could not find include ${CREDENTIALS_DIRECTORY}/borgmatic_ntfy.yaml at /etc/borgmatic/${CREDENTIALS_DIRECTORY}/borgmatic_ntfy.yaml or /${CREDENTIALS_DIRECTORY}/borgmatic_ntfy.yaml ``` from attempting to include it like: ``` access_token: !include "${CREDENTIALS_DIRECTORY}/borgmatic_ntfy.yaml" ```
mgrecar commented 2025-01-12 20:41:48 +00:00
Author
Copy Link

Well, I find that if I instead try:

access_token: !include "/run/credentials/borgmatic.service/borgmatic_ntfy.yaml"

It actually works. That path is where the decrypted content seems to get mounted, and so the include is able to resolve if you don't depend on environment variable interpolation.

That being said, that feels real bad, and you lose your ability to validate your config file, because that path is only legitimate when the systemd job is running.

Well, I find that if I instead try: ``` access_token: !include "/run/credentials/borgmatic.service/borgmatic_ntfy.yaml" ``` It actually works. That path is where the decrypted content seems to get mounted, and so the include is able to resolve if you don't depend on environment variable interpolation. That being said, that feels real bad, and you lose your ability to validate your config file, because that path is only legitimate when the systemd job is running.
witten commented 2025-01-12 20:48:10 +00:00
Owner
Copy Link

Ah, unfortunately it looks like environment variable interpolation is currently done after configuration includes are applied—which means you can't use environment variables within include paths. So that would somehow have to change in order to support this particular solution to your use case.

Ah, unfortunately it looks like environment variable interpolation is currently done *after* configuration includes are applied—which means you can't use environment variables within include paths. So that would somehow have to change in order to support this particular solution to your use case.
witten commented 2025-01-12 20:52:05 +00:00
Owner
Copy Link

It actually works. That path is where the decrypted content seems to get mounted, and so the include is able to resolve if you don't depend on environment variable interpolation.

Interesting! Yeah, it makes sense given the sequencing mentioned above about environment variable interpolation and includes. I agree that it would be better for that path not to be hard-coded. Although it seems like the config file would only work under systemd, regardless of whether an environment variable is used there, and I'm not sure of a way around that.. Maybe a concept other than !include that's like include but doesn't get resolved at the point that the configuration file is loaded, but instead gets resolved later on closer to the time it's consumed (by ntfy in this case), similar to Borg's passcommand? I'm not sure exactly how that would work in practice, but I'm open to ideas.

> It actually works. That path is where the decrypted content seems to get mounted, and so the include is able to resolve if you don't depend on environment variable interpolation. Interesting! Yeah, it makes sense given the sequencing mentioned above about environment variable interpolation and includes. I agree that it would be better for that path not to be hard-coded. Although it seems like the config file would only work under systemd, regardless of whether an environment variable is used there, and I'm not sure of a way around that.. Maybe a concept other than `!include` that's like include but doesn't get resolved at the point that the configuration file is loaded, but instead gets resolved later on closer to the time it's consumed (by ntfy in this case), similar to Borg's passcommand? I'm not sure exactly how that would work in practice, but I'm open to ideas.
witten commented 2025-01-12 21:22:11 +00:00
Owner
Copy Link

Okay, a little brainstorming on how this might look:

access_token: !load_credential "${CREDENTIALS_DIRECTORY}/borgmatic_ntfy"

Despite the syntax, the !load_credential tag would have to be ignored by the configuration loading machinery, so it didn't trigger when configuration files are loaded. Presumably environment variable interpolation would occur just fine afterwards, filling out that credentials directory path. And then within the ntfy hook (and any other hook accepting a credential), the hook could check the credential value for whether it started with !load_credential and if so, load the credential from file similar to Borg's passcommand.

Okay, a little brainstorming on how this might look: ```yaml access_token: !load_credential "${CREDENTIALS_DIRECTORY}/borgmatic_ntfy" ``` Despite the syntax, the `!load_credential` tag would have to be ignored by the configuration loading machinery, so it didn't trigger when configuration files are loaded. Presumably environment variable interpolation would occur just fine afterwards, filling out that credentials directory path. And then within the ntfy hook (and any other hook accepting a credential), the hook could check the credential value for whether it started with `!load_credential` and if so, load the credential from file similar to Borg's passcommand.
👍 1
mgrecar commented 2025-01-12 22:22:36 +00:00
Author
Copy Link

Although it seems like the config file would only work under systemd, regardless of whether an environment variable is used there, and I'm not sure of a way around that..

Oh yeah, this is quite true. It's implicitly relying on the fact that, in my specific case, I'm trying to run borgmatic as a systemd service. Totally not a portable solution.

Maybe a concept other than !include that's like include but doesn't get resolved at the point that the configuration file is loaded, but instead gets resolved later on closer to the time it's consumed (by ntfy in this case), similar to Borg's passcommand? I'm not sure exactly how that would work in practice, but I'm open to ideas.

Yeah, that's kind of what I was imagining. I think relying on !include feels like it works accidentally and is brittle and subject to accidentally breaking, rather than an intentional configuration option.

Despite the syntax, the !load_credential tag would have to be ignored by the configuration loading machinery, so it didn't trigger when configuration files are loaded. Presumably environment variable interpolation would occur just fine afterwards, filling out that credentials directory path. And then within the ntfy hook (and any other hook accepting a credential), the hook could check the credential value for whether it started with !load_credential and if so, load the credential from file similar to Borg's passcommand.

In mulling this over, how might this work with other non-systemd credential providers? I had a thought that perhaps the config file might let you select a credential provider, perhaps like:

credential_provider: systemd

Or perhaps pass, or env vars, or other solutions for retrieving credentials. Based on that, borgmatic would understand the standard way to pull credentials for that solution. Then, maybe the syntax could get cleaned up to something like:

access_token: !load_credential "borgmatic_ntfy"

Since, in this case, it understands the expectations around that pathing. I'm definitely out of my zone of experience on this, I was just thinking of ways to address the need without necessarily over-fitting to my specific situation.

> Although it seems like the config file would only work under systemd, regardless of whether an environment variable is used there, and I'm not sure of a way around that.. Oh yeah, this is quite true. It's implicitly relying on the fact that, in my specific case, I'm trying to run borgmatic as a systemd service. Totally not a portable solution. >Maybe a concept other than !include that's like include but doesn't get resolved at the point that the configuration file is loaded, but instead gets resolved later on closer to the time it's consumed (by ntfy in this case), similar to Borg's passcommand? I'm not sure exactly how that would work in practice, but I'm open to ideas. Yeah, that's kind of what I was imagining. I think relying on `!include` feels like it works accidentally and is brittle and subject to accidentally breaking, rather than an intentional configuration option. > Despite the syntax, the !load_credential tag would have to be ignored by the configuration loading machinery, so it didn't trigger when configuration files are loaded. Presumably environment variable interpolation would occur just fine afterwards, filling out that credentials directory path. And then within the ntfy hook (and any other hook accepting a credential), the hook could check the credential value for whether it started with !load_credential and if so, load the credential from file similar to Borg's passcommand. In mulling this over, how might this work with other non-systemd credential providers? I had a thought that perhaps the config file might let you select a credential provider, perhaps like: ``` credential_provider: systemd ``` Or perhaps pass, or env vars, or other solutions for retrieving credentials. Based on that, borgmatic would understand the standard way to pull credentials for that solution. Then, maybe the syntax could get cleaned up to something like: ``` access_token: !load_credential "borgmatic_ntfy" ``` Since, in this case, it understands the expectations around that pathing. I'm definitely out of my zone of experience on this, I was just thinking of ways to address the need without necessarily over-fitting to my specific situation.
mgrecar commented 2025-01-13 01:36:05 +00:00
Author
Copy Link

As I was working further on my configuration, I found an alternative, sparked from the insight you brought to my attention about !include.

In order to store my configuration in Git, I moved everything over (system timer, service and borgmatic config) to it's own project repo. I created a file ntfy_access_token.yaml, put the access token inside it, and locked it down with chmod 400 ntfy_access_token.yaml. From there, I set my .gitignore file to include that specific file, and my config.yaml gets updated to use:

    access_token: !include ./ntfy_access_code.yaml

After setting up my systemd service with this from within my project repo:

sudo systemctl link ./borgmatic.service
sudo systemctl enable --now ./borgmatic.timer

Borgmatic is able to properly locate and pull in the include, and I can commit my config to Git without worrying about publishing secrets.

I suppose I could even sudo chown root:root ntfy_access_code.yaml to lock it down even further. It's not encrypted on disk, but it's about the best I can come up with at the moment.

As I was working further on my configuration, I found an alternative, sparked from the insight you brought to my attention about `!include`. In order to store my configuration in Git, I moved everything over (system timer, service and borgmatic config) to it's own project repo. I created a file `ntfy_access_token.yaml`, put the access token inside it, and locked it down with `chmod 400 ntfy_access_token.yaml`. From there, I set my `.gitignore` file to include that specific file, and my `config.yaml` gets updated to use: ``` access_token: !include ./ntfy_access_code.yaml ``` After setting up my systemd service with this from within my project repo: ``` sudo systemctl link ./borgmatic.service sudo systemctl enable --now ./borgmatic.timer ``` Borgmatic is able to properly locate and pull in the include, and I can commit my config to Git without worrying about publishing secrets. I suppose I could even `sudo chown root:root ntfy_access_code.yaml` to lock it down even further. It's not encrypted on disk, but it's about the best I can come up with at the moment.
witten commented 2025-01-19 16:23:53 +00:00
Owner
Copy Link

In mulling this over, how might this work with other non-systemd credential providers? I had a thought that perhaps the config file might let you select a credential provider, perhaps like:

credential_provider: systemd

Or perhaps pass, or env vars, or other solutions for retrieving credentials. Based on that, borgmatic would understand the standard way > to pull credentials for that solution. Then, maybe the syntax could get cleaned up to something like:

access_token: !load_credential "borgmatic_ntfy"

Since, in this case, it understands the expectations around that pathing. I'm definitely out of my zone of experience on this, I was
just thinking of ways to address the need without necessarily over-fitting to my specific situation.

I like the idea of borgmatic understanding how to pull credentials for particular credentials providers, but I'm not sure the provider selection should be global for the whole config file. A user might have multiple credentials providers. So how about a variant like this?

access_token: !load_credential systemd "borgmatic_ntfy"

Or even:

access_token: !systemd_credential "borgmatic_ntfy"

Thoughts?

As I was working further on my configuration, I found an alternative, sparked from the insight you brought to my attention about !include.

I'm glad you have a working alternative for the time being!

> In mulling this over, how might this work with other non-systemd credential providers? I had a thought that perhaps the config file might let you select a credential provider, perhaps like: > > ```yaml > credential_provider: systemd > ``` > > Or perhaps pass, or env vars, or other solutions for retrieving credentials. Based on that, borgmatic would understand the standard way > to pull credentials for that solution. Then, maybe the syntax could get cleaned up to something like: > > ```yaml > access_token: !load_credential "borgmatic_ntfy" > ``` > > Since, in this case, it understands the expectations around that pathing. I'm definitely out of my zone of experience on this, I was > just thinking of ways to address the need without necessarily over-fitting to my specific situation. I like the idea of borgmatic understanding how to pull credentials for particular credentials providers, but I'm not sure the provider selection should be global for the whole config file. A user might have multiple credentials providers. So how about a variant like this? ```yaml access_token: !load_credential systemd "borgmatic_ntfy" ``` Or even: ```yaml access_token: !systemd_credential "borgmatic_ntfy" ``` Thoughts? > As I was working further on my configuration, I found an alternative, sparked from the insight you brought to my attention about `!include.` I'm glad you have a working alternative for the time being!
mgrecar commented 2025-01-20 13:21:00 +00:00
Author
Copy Link

I'm not sure the provider selection should be global for the whole config file. A user might have multiple credentials providers.

That's a good point. There's no good reason to restrict all credentials to be from the same provider.

Of the two, I think I personally prefer

access_token: !load_credential systemd "borgmatic_ntfy"

just a bit better. I think it flows a bit better as you read the config, but that's super subjective.

> I'm not sure the provider selection should be global for the whole config file. A user might have multiple credentials providers. That's a good point. There's no good reason to restrict all credentials to be from the same provider. Of the two, I think I personally prefer ``` access_token: !load_credential systemd "borgmatic_ntfy" ``` just a bit better. I think it flows a bit better as you read the config, but that's super subjective.
👍 1
witten added the
design finalized
 label 2025-01-24 18:28:51 +00:00
witten commented 2025-01-24 19:21:50 +00:00
Owner
Copy Link

Makes sense!

I added a note to #961 about some potential implementation detail crossover between these respective tickets.

Makes sense! I added [a note](https://projects.torsion.org/borgmatic-collective/borgmatic/issues/961#issuecomment-9597) to #961 about some potential implementation detail crossover between these respective tickets.
witten referenced this issue from a commit 2025-02-09 03:28:26 +00:00
Flake fixes (#966).
witten commented 2025-02-10 22:28:21 +00:00
Owner
Copy Link

This is now implemented in main and will be part of the next release! Updated documentation will be online shortly. In terms of the syntax, I ended up going with !credential systemd credentialname for brevity. And loading of the credential is delayed as long as possible, so for instance if you use !credential for an Ntfy password, you should still be able to run borgmatic config validate or even borgmatic list outside of systemd. Really, anything that doesn't try to hit Ntfy and therefore need to load the credential.

Feedback welcome!

This is now implemented in main and will be part of the next release! [Updated documentation](https://torsion.org/borgmatic/docs/how-to/provide-your-passwords/#using-systemd-service-credentials) will be online shortly. In terms of the syntax, I ended up going with `!credential systemd credentialname` for brevity. And loading of the credential is delayed as long as possible, so for instance if you use `!credential` for an Ntfy password, you should still be able to run `borgmatic config validate` or even `borgmatic list` outside of systemd. Really, anything that doesn't try to hit Ntfy and therefore need to load the credential. Feedback welcome!
witten commented 2025-02-11 00:17:12 +00:00
Owner
Copy Link

Reopening this. I'm having second thoughts about the syntax, and I want to investigate an alternative before I commit one way or the other, as it's much harder to change it once it's released.

I'm thinking maybe something like this:

access_token: "{credential systemd borgmatic_ntfy}"

My rationale is that it's more in line with the existing constants syntax, and feels less like a weird one off. Plus, semantically this is like plugging in a "constant" that happens to be a credential. The !credential syntax is more similar to YAML includes, but this isn't a YAML tag even if it looks like one; it's evaluated as late as possible, whereas YAML tags are evaluated early—at the time the configuration file is parsed.

I dunno, maybe I'm overthinking/overcooking this. But I thought it's worth at least investigating.

Reopening this. I'm having second thoughts about the syntax, and I want to investigate an alternative before I commit one way or the other, as it's much harder to change it once it's released. I'm thinking maybe something like this: ```yaml access_token: "{credential systemd borgmatic_ntfy}" ``` My rationale is that it's more in line with the existing [constants syntax](https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/#constant-interpolation), and feels less like a weird one off. Plus, semantically this is like plugging in a "constant" that happens to be a credential. The `!credential` syntax is more similar to YAML includes, but this *isn't* a YAML tag even if it looks like one; it's evaluated as late as possible, whereas YAML tags are evaluated early—at the time the configuration file is parsed. I dunno, maybe I'm overthinking/overcooking this. But I thought it's worth at least investigating.
witten reopened this issue 2025-02-11 00:18:09 +00:00
witten commented 2025-02-11 06:30:34 +00:00
Owner
Copy Link

Okay, I did end up going with the {credential ...} syntax for consistency with constants (and also environment variables). Docs will be updated soon.

Okay, I did end up going with the `{credential ...}` syntax for consistency with constants (and also environment variables). Docs will be updated soon.
witten commented 2025-02-11 06:35:29 +00:00
Owner
Copy Link

Released in borgmatic 1.9.10!

Released in borgmatic 1.9.10!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
blocked
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
No Label
design finalized
Milestone
No items
No Milestone
Assignees
Clear assignees
diivi import_bot witten
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#966
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?