borgmatic-collective/borgmatic

Watch

Fork 57

Environment variable within `repositories` path no longer supported #1062

New issue

Closed

opened 2025-04-08 16:40:40 +00:00 by wjam · 2 comments

wjam commented

2025-04-08 16:40:40 +00:00

What I'm trying to do and why

I have a local server running a Docker compose-based setup that makes use of Borgmatic to backup the various databases running on the server. Given the complexity of the set up, and to make it simpler to update the services that I'm using, I run a number of automated tests that include running all of the services as well as attempting to restore a pre-existing backup.

To make it simpler to switch from the 'real' backup location to a test one, I've set path of the repository to be ${REMOTE_REPO} and so allowing the CI to set it to one value (a local directory) while the server sets it to another (a remote service). When upgrading from ghcr.io/borgmatic-collective/borgmatic:1.9.14 to ghcr.io/borgmatic-collective/borgmatic:2.0.1, the tests that were previously passing started failing.

Steps to reproduce

The config.yaml looks like this:

# https://torsion.org/borgmatic/docs/reference/configuration/

# Required for database hooks
read_special: true

# List of source directories to backup.
source_directories:
  - /mnt/source/grafana_data # Assorted paths shared through Docker

exclude_patterns:
  # Exclude SQLite db, WAL, etc. files from the backup
  - /mnt/source/grafana_data/grafana.db* # Various database-ish or temporary files that shouldn't be backed up

# Paths of local or remote repositories to backup to.
repositories:
  - path: ${REMOTE_REPO}
    label: remote

# Rest omitted

The Docker compose file looks like this:

services:
  borgmatic:
    image: ghcr.io/borgmatic-collective/borgmatic:2.0.1@sha256:5501c36599a3e809eba75a9b17705d0562d51775ea03dc07b8ba4d43fd6a312a
    container_name: borgmatic
    restart: always
    environment:
      BACKUP_CRON: "0 23 * * *"
    volumes:
      - ./grafana/data:/mnt/source/grafana_data # Plus other shared paths
    env_file: "borgmatic.env"

Actual behavior

Attempting to run a borgmatic command, such as attempting to restore a previous created backup (docker compose run --rm borgmatic restore --archive latest), results in this error:

     command.go:206: Invalid placeholder "REMOTE_REPO" in string: ${REMOTE_REPO}
    command.go:206: remote: Error running actions for repository
    command.go:206: remote: Command '('borg', 'list', '--last', '1', '--short', '${REMOTE_REPO}')' returned non-zero exit status 6.
    command.go:206: /etc/borgmatic.d/config.yaml: An error occurred
    command.go:206: 
    command.go:206: summary:
    command.go:206: An error occurred
    command.go:206: Error running actions for repository
    command.go:206: Command '('borg', 'list', '--last', '1', '--short', '${REMOTE_REPO}')' returned non-zero exit status 6.
    command.go:206: 
    command.go:206: Need some help? https://torsion.org/borgmatic/#issues
    docker_compose.go:31: error while running command: exit status 1;  Network testrestore_default  Creating
         Network testrestore_default  Created
         Container testrestore-immich-database-1  Creating
         Container testrestore-miniflux-db-1  Creating
         Container testrestore-miniflux-db-1  Created
         Container testrestore-immich-database-1  Created
         Container testrestore-immich-database-1  Starting
         Container testrestore-miniflux-db-1  Starting
         Container testrestore-miniflux-db-1  Started
         Container testrestore-immich-database-1  Started
        Invalid placeholder "REMOTE_REPO" in string: ${REMOTE_REPO}
        remote: Error running actions for repository
        remote: Command '('borg', 'list', '--last', '1', '--short', '${REMOTE_REPO}')' returned non-zero exit status 6.
        /etc/borgmatic.d/config.yaml: An error occurred
        
        summary:
        An error occurred
        Error running actions for repository
        Command '('borg', 'list', '--last', '1', '--short', '${REMOTE_REPO}')' returned non-zero exit status 6.
        
        Need some help? https://torsion.org/borgmatic/#issues

Running docker compose --file=compose.yaml run --rm --entrypoint env borgmatic shows the environment variables available inside the container are these (with passwords & passphrases being test data) - note the REMOTE_REPO is present:

PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=4f16ebf43e8d
TERM=xterm
IMMICH_PASSWORD=
BACKUP_CRON=0 23 * * *
MINIFLUX_PASSWORD=password
BORG_PASSPHRASE=SomeRepositoryPassPhrase
REMOTE_REPO=/mnt/borg-repository
LANG=en_US.UTF-8
GPG_KEY=7169605F62C751356D054A26A821E680E5FA6305
PYTHON_VERSION=3.12.9
PYTHON_SHA256=7220835d9f90b37c006e9842a8dff4580aaca4318674f947302b8d28f3f81112
S6_OVERLAY_ARCH=x86_64
LANGUAGE=en_US.UTF-8
S6_LOGGING=1
S6_VERBOSITY=0
S6_CMD_WAIT_FOR_SERVICES_MAXTIME=0
TZ=Europe/London
HOME=/root

Expected behavior

Borgmatic should be able to replace the repository path with an environment variable value.

Other notes / implementation ideas

No response

borgmatic version

ghcr.io/borgmatic-collective/borgmatic:2.0.1

borgmatic installation method

Container

Borg version

Packaged in the official container image

Python version

Packaged in the official container image

Database version (if applicable)

n/a

Operating system and version

Server is Ubuntu 24.04, although the failure was spotted on a GitHub Action public runner

### What I'm trying to do and why I have a local server running a Docker compose-based setup that makes use of Borgmatic to backup the various databases running on the server. Given the complexity of the set up, and to make it simpler to update the services that I'm using, I run a number of automated tests that include running all of the services as well as attempting to restore a pre-existing backup. To make it simpler to switch from the 'real' backup location to a test one, I've set `path` of the repository to be `${REMOTE_REPO}` and so allowing the CI to set it to one value (a local directory) while the server sets it to another (a remote service). When upgrading from `ghcr.io/borgmatic-collective/borgmatic:1.9.14` to `ghcr.io/borgmatic-collective/borgmatic:2.0.1`, the tests that were previously passing started failing. ### Steps to reproduce The `config.yaml` looks like this: ```yaml # https://torsion.org/borgmatic/docs/reference/configuration/ # Required for database hooks read_special: true # List of source directories to backup. source_directories: - /mnt/source/grafana_data # Assorted paths shared through Docker exclude_patterns: # Exclude SQLite db, WAL, etc. files from the backup - /mnt/source/grafana_data/grafana.db* # Various database-ish or temporary files that shouldn't be backed up # Paths of local or remote repositories to backup to. repositories: - path: ${REMOTE_REPO} label: remote # Rest omitted ``` The Docker compose file looks like this: ```yaml services: borgmatic: image: ghcr.io/borgmatic-collective/borgmatic:2.0.1@sha256:5501c36599a3e809eba75a9b17705d0562d51775ea03dc07b8ba4d43fd6a312a container_name: borgmatic restart: always environment: BACKUP_CRON: "0 23 * * *" volumes: - ./grafana/data:/mnt/source/grafana_data # Plus other shared paths env_file: "borgmatic.env" ``` ### Actual behavior Attempting to run a borgmatic command, such as attempting to restore a previous created backup (`docker compose run --rm borgmatic restore --archive latest`), results in this error: ``` command.go:206: Invalid placeholder "REMOTE_REPO" in string: ${REMOTE_REPO} command.go:206: remote: Error running actions for repository command.go:206: remote: Command '('borg', 'list', '--last', '1', '--short', '${REMOTE_REPO}')' returned non-zero exit status 6. command.go:206: /etc/borgmatic.d/config.yaml: An error occurred command.go:206: command.go:206: summary: command.go:206: An error occurred command.go:206: Error running actions for repository command.go:206: Command '('borg', 'list', '--last', '1', '--short', '${REMOTE_REPO}')' returned non-zero exit status 6. command.go:206: command.go:206: Need some help? https://torsion.org/borgmatic/#issues docker_compose.go:31: error while running command: exit status 1; Network testrestore_default Creating Network testrestore_default Created Container testrestore-immich-database-1 Creating Container testrestore-miniflux-db-1 Creating Container testrestore-miniflux-db-1 Created Container testrestore-immich-database-1 Created Container testrestore-immich-database-1 Starting Container testrestore-miniflux-db-1 Starting Container testrestore-miniflux-db-1 Started Container testrestore-immich-database-1 Started Invalid placeholder "REMOTE_REPO" in string: ${REMOTE_REPO} remote: Error running actions for repository remote: Command '('borg', 'list', '--last', '1', '--short', '${REMOTE_REPO}')' returned non-zero exit status 6. /etc/borgmatic.d/config.yaml: An error occurred summary: An error occurred Error running actions for repository Command '('borg', 'list', '--last', '1', '--short', '${REMOTE_REPO}')' returned non-zero exit status 6. Need some help? https://torsion.org/borgmatic/#issues ``` Running `docker compose --file=compose.yaml run --rm --entrypoint env borgmatic` shows the environment variables available inside the container are these (with passwords & passphrases being test data) - note the `REMOTE_REPO` is present: ``` PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=4f16ebf43e8d TERM=xterm IMMICH_PASSWORD= BACKUP_CRON=0 23 * * * MINIFLUX_PASSWORD=password BORG_PASSPHRASE=SomeRepositoryPassPhrase REMOTE_REPO=/mnt/borg-repository LANG=en_US.UTF-8 GPG_KEY=7169605F62C751356D054A26A821E680E5FA6305 PYTHON_VERSION=3.12.9 PYTHON_SHA256=7220835d9f90b37c006e9842a8dff4580aaca4318674f947302b8d28f3f81112 S6_OVERLAY_ARCH=x86_64 LANGUAGE=en_US.UTF-8 S6_LOGGING=1 S6_VERBOSITY=0 S6_CMD_WAIT_FOR_SERVICES_MAXTIME=0 TZ=Europe/London HOME=/root ``` ### Expected behavior Borgmatic should be able to replace the repository path with an environment variable value. ### Other notes / implementation ideas _No response_ ### borgmatic version ghcr.io/borgmatic-collective/borgmatic:2.0.1 ### borgmatic installation method Container ### Borg version Packaged in the official container image ### Python version Packaged in the official container image ### Database version (if applicable) n/a ### Operating system and version Server is Ubuntu 24.04, although the failure was spotted on a GitHub Action public runner

witten added the

bug

label

2025-04-08 18:33:07 +00:00

witten referenced this issue from a commit

2025-04-08 18:55:38 +00:00

Fix a regression that broke environment variable interpolation (#1062).

witten commented

2025-04-08 18:55:45 +00:00

Owner

Thanks for filing this. This is now fixed in main (along with a new end-to-end test to catch this sort of issue) and will be part of the next release! Probably pretty soon since this is kind of an egregious bug...

👍 1

witten closed this issue

2025-04-08 18:55:45 +00:00

witten commented

2025-04-08 19:14:00 +00:00

Owner

Released in borgmatic 2.0.2!