Setting prefix of repository path from commandline #782

New Issue

vboehm · 2023-11-03T20:42:36Z

vboehm commented

2023-11-03 20:42:36 +00:00

What I'm trying to do and why

I'm using borgmatic 1.8.4 on debian 12 and try to replace a prefix to the repository name from command line.

According to your documentation you support default substitution on environment variables.

Interpolation defaults

If you'd like to set a default for your environment variables, you can do so with the following syntax:

encryption_passphrase: ${YOUR_PASSPHRASE:-defaultpass}

Here, "defaultpass" is the default passphrase if the YOUR_PASSPHRASE environment variable is not set. Without a default, if the environment variable doesn't exist, borgmatic will error.

The standard call to make a backup should be
borgmatic create
But sometimes I want to use the same conf files in /etc/borgmatic.d to use a different disk on a different computer and start the backup with
BORGMATIC_PREFIX='ssh://user@myothercomputer/someother/path' borgmatic create

So I created the conf files like this:

source_directories:
  - /
one_file_system: true
repositories:
  - path: "${BORGPREFIX:-ssh://user@computer/some/path}/name.borg"
    label: system
archive_name_format: '{hostname}-system-{now}'
keep_daily: 10
keep_weekly: 8
keep_monthly: 8
keep_yearly: 2

But I always get the same error message

system: Error running actions for repository
Command '('borg', 'list', '--glob-archives', '{hostname}-system-*', '--json', 'ssh://$BORG_PREFIX/./-ssh//user@computer/some/path}/name.borg')' returned non-zero exit status 2.
/etc/borgmatic.d/system.yaml: An error occurred

with or without setting the environment variable.

As soon as I replace the environment default expression ${BORGPREFIX:-ssh://...} with a simple expression ${BORG_PREFIX} everything works fine. But i have no default value :-(

I also tried setting a constant

constants;
    borg_prefix: 'ssh://user@computer/some/path'
...
repositories:
  - path: "{borg_prefix}/name.borg"
    label: system

and overriding it with --override 'constants.borg_prefix=ssh://user@myothercomputer/someother/path' but this has no effect: borgmatic accesses the original path.

Can you tell me what I'm doing wrong? Or do you have another solution?

regards Volker

Steps to reproduce

see above

Actual behavior

see above

Expected behavior

see above

Other notes / implementation ideas

No response

borgmatic version

1.8.4

borgmatic installation method

debian package

Borg version

1.2.4

Python version

3.11.2

Database version (if applicable)

No response

Operating system and version

Debian 12

### What I'm trying to do and why I'm using borgmatic 1.8.4 on debian 12 and try to replace a prefix to the repository name from command line. According to your documentation you support default substitution on environment variables. ``` Interpolation defaults If you'd like to set a default for your environment variables, you can do so with the following syntax: encryption_passphrase: ${YOUR_PASSPHRASE:-defaultpass} Here, "defaultpass" is the default passphrase if the YOUR_PASSPHRASE environment variable is not set. Without a default, if the environment variable doesn't exist, borgmatic will error. ``` The standard call to make a backup should be `borgmatic create` But sometimes I want to use the same conf files in `/etc/borgmatic.d` to use a different disk on a different computer and start the backup with `BORGMATIC_PREFIX='ssh://user@myothercomputer/someother/path' borgmatic create` So I created the conf files like this: ``` source_directories: - / one_file_system: true repositories: - path: "${BORGPREFIX:-ssh://user@computer/some/path}/name.borg" label: system archive_name_format: '{hostname}-system-{now}' keep_daily: 10 keep_weekly: 8 keep_monthly: 8 keep_yearly: 2 ``` But I always get the same error message ``` system: Error running actions for repository Command '('borg', 'list', '--glob-archives', '{hostname}-system-*', '--json', 'ssh://$BORG_PREFIX/./-ssh//user@computer/some/path}/name.borg')' returned non-zero exit status 2. /etc/borgmatic.d/system.yaml: An error occurred ``` with or without setting the environment variable. As soon as I replace the environment default expression `${BORGPREFIX:-ssh://...}` with a simple expression `${BORG_PREFIX}` everything works fine. But i have no default value :-( I also tried setting a constant ``` constants; borg_prefix: 'ssh://user@computer/some/path' ... repositories: - path: "{borg_prefix}/name.borg" label: system ``` and overriding it with `--override 'constants.borg_prefix=ssh://user@myothercomputer/someother/path'` but this has no effect: borgmatic accesses the original path. Can you tell me what I'm doing wrong? Or do you have another solution? regards Volker ### Steps to reproduce see above ### Actual behavior see above ### Expected behavior see above ### Other notes / implementation ideas _No response_ ### borgmatic version 1.8.4 ### borgmatic installation method debian package ### Borg version 1.2.4 ### Python version 3.11.2 ### Database version (if applicable) _No response_ ### Operating system and version Debian 12

witten commented

2023-11-03 21:34:51 +00:00

Thanks for taking the time to file this! For this first issue with this configuration ...

  - path: "${BORGPREFIX:-ssh://user@computer/some/path}/name.borg"
    label: system

... I was able to reproduce the behavior you're seeing. However, the environment variable substitution—even falling back to a default value—works just fine for me when I make use of the environment variable anywhere other than for the path. For instance, try making use of an environment variable (with or without a default) for the label value, and you'll see that it works just fine. I think what's going on with the path though is an unfortunate interaction with the logic that rewrites non-ssh:// style repository URLs. That logic is triggering before the environment variable substitution happens, thereby screwing everything up.

Side note: You have BORGMATIC_PREFIX in your command-line invocation but BORGPREFIX in your configuration file, so make sure those environment variable names match up!

As for the --override 'constants.borg_prefix=ssh://user@myothercomputer/someother/path' issue, I expect you've found another unfortunate interaction: Most likely, overrides don't work with constants because constant interpolation happens before overrides kick in. I'll have to look into whether it's possible to fix that or not.

I'm going to go ahead and mark this ticket as a bug, but I don't yet know what all is fixable or work-aroundable. In the meantime, I recommend using a separate configuration file for your second repository. You can take advantage of configuration includes if you don't want to duplicate settings. In fact, this may ultimately be a more convenient longer-term approach, so you can just select the configuration file you want to use instead of typing so much on the command-line. Having said that, I still want to address the two issues you've highlighted.

Thanks for taking the time to file this! For this first issue with this configuration ... ```yaml - path: "${BORGPREFIX:-ssh://user@computer/some/path}/name.borg" label: system ``` ... I was able to reproduce the behavior you're seeing. However, the environment variable substitution—even falling back to a default value—works just fine for me when I make use of the environment variable anywhere *other than* for the `path`. For instance, try making use of an environment variable (with or without a default) for the `label` value, and you'll see that it works just fine. I think what's going on with the `path` though is an unfortunate interaction with the logic that rewrites [non-`ssh://` style](https://borgbackup.readthedocs.io/en/stable/usage/general.html#repository-urls) repository URLs. That logic is triggering before the environment variable substitution happens, thereby screwing everything up. Side note: You have `BORGMATIC_PREFIX` in your command-line invocation but `BORGPREFIX` in your configuration file, so make sure those environment variable names match up! As for the `--override 'constants.borg_prefix=ssh://user@myothercomputer/someother/path'` issue, I expect you've found another unfortunate interaction: Most likely, overrides don't work with constants because constant interpolation happens _before_ overrides kick in. I'll have to look into whether it's possible to fix that or not. I'm going to go ahead and mark this ticket as a bug, but I don't yet know what all is fixable or work-aroundable. In the meantime, I recommend using a separate configuration file for your second repository. You can take advantage of [configuration includes](https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/#configuration-includes) if you don't want to duplicate settings. In fact, this may ultimately be a more convenient longer-term approach, so you can just select the configuration file you want to use instead of typing so much on the command-line. Having said that, I still want to address the two issues you've highlighted.

witten added the

bug

label 2023-11-03 21:34:57 +00:00

witten referenced this issue from a commit

2023-11-04 04:16:17 +00:00

Fix environment variable interpolation within configured repository paths (#782).

witten commented

2023-11-04 04:24:02 +00:00

An update: I've fixed the first issue in main by swapping the order of environment variable interpolation and normalization (like repository URL rewriting), so environment variable interpolation comes first.

Still to do: Figure out what to do for the other issue with the overrides.

An update: I've fixed the first issue in main by swapping the order of environment variable interpolation and normalization (like repository URL rewriting), so environment variable interpolation comes first. Still to do: Figure out what to do for the other issue with the overrides.

witten commented

2023-11-20 05:17:23 +00:00

The override portion of this ticket has been fixed in borgmatic 1.8.5, just released. (This also includes the previous fix for the environment variable issue.) Thanks!

witten closed this issue

2023-11-20 05:17:23 +00:00

witten referenced this issue from a commit

2023-11-20 05:22:40 +00:00

Constants support includes and command-line overrides (#745, #782)

Sign in to join this conversation.