No valid configuration files found #662

New Issue

modem7 · 2023-03-28T11:20:47Z

modem7 commented

2023-03-28 11:20:47 +00:00

What I'm trying to do and why

Create backup

Steps to reproduce (if a bug)

Was working until upgrade to 1.7.10.
Error:

/etc/borgmatic/config.yaml /etc/borgmatic.d /root/.config/borgmatic/config.yaml /root/.config/borgmatic.d: No valid configuration files found

summary:
/etc/borgmatic.d/config.yml: Error parsing configuration file
An error occurred while parsing a configuration file at /etc/borgmatic.d/config.yml:
Unknown repository in the "consistency" section's "check_repositories": /mnt/borg-repository
/etc/borgmatic/config.yaml /etc/borgmatic.d /root/.config/borgmatic/config.yaml /root/.config/borgmatic.d: No valid configuration files found

Need some help? https://torsion.org/borgmatic/#issues

Location of yaml: /etc/borgmatic.d/config.yaml

Config contents:

location:
    source_directories:
         - ${BORG_SOURCE_1}
         - ${BORG_SOURCE_2}
    repositories:
         - ${BORG_REPO_LOCAL}
         - ${BORG_REPO_REMOTE}
    one_file_system: true
    exclude_caches: true

storage:
#   Passphase is set in variable $BORG_PASSPHRASE
    compression: lz4
    archive_name_format: 'backup-{now}'

retention:
    keep_hourly: 0
    keep_daily: 7
    keep_weekly: 4
    keep_monthly: 12
    keep_yearly: 1
    prefix: 'backup-'

consistency:
    check_repositories:
      - ${BORG_REPO_LOCAL}
    checks:
      - name: repository
        frequency: 1 week
      - name: archives
        frequency: always
      - name: extract
        frequency: 2 weeks
      - name: data
        frequency: 1 month
    prefix: 'backup-'

hooks:
    before_everything:
        - borgmatic break-lock
        - echo "Starting a backup job."
        - echo "Stopping containers."
        - exec /borgscripts/docker-stop.sh
    after_everything:
        - echo "Starting containers."
        - exec /borgscripts/docker-start.sh
        - echo "Backup created."
    on_error:
        - echo "Error while creating a backup."
        - exec /borgscripts/docker-start.sh
    # https://torsion.org/borgmatic/docs/how-to/backup-your-databases/

    # Third-party services to notify you if backups aren't happening.
    healthchecks: ${BORG_HEALTHCHECK_URL}

    # https://torsion.org/borgmatic/docs/reference/configuration/

Docker-compose:

  borgmatic:
    # image: modem7/borgmatic-docker:dockercli
    image: docker.modem7.com/modem7/borgmatic-docker:dockercli
    # image: modem7/borgmatic-docker
    init: true
    container_name: Borgmatic
    environment:
      TZ: $TZ
      BORG_PASSPHRASE: $BORG_PASSPHRASE
      BORG_SOURCE_1: $BORG_SOURCE_1
      BORG_SOURCE_2: $BORG_SOURCE_2
      BORG_REPO_LOCAL: $BORG_REPO_LOCAL
      BORG_REPO_REMOTE: $BORG_REPO_REMOTE
      BORG_HEALTHCHECK_URL: $BORG_HEALTHCHECK_URL
      CRON: $BORG_CRON
      CRON_COMMAND: $BORG_CRON_COMMAND
      # EXTRA_CRON: |-
      #   0 5 2 * * command1
      #   0 7 1 * * command2
      # - LIVEINSTALL=docker-cli nano
    logging:
      driver: "local"
      options:
        max-size: 10m
        max-file: "3"
    volumes:
      - $BORGHOMESOURCEDIR:/mnt/source/
      - $USERDIR:/mnt/source/DockerApps/
      # - $CRONTAB:/mnt/source/Cron
      # - Pihole:/mnt/source/Pihole/Pihole
      # - Dnsmasq:/mnt/source/Pihole/Dnsmasq
      - $BORGSERVBACKUPDIR/Database:/mnt/borg-DBrepository
      - $BORGSERVBACKUPDIR/Docker:/mnt/borg-repository
      - $RAMBACKUP/Backup:/mnt/ramdrive
      - $USERDIR/Borgmatic/borgmatic.d/:/etc/borgmatic.d/
      - $USERDIR/Borgmatic/.config/borg/:/root/.config/borg
      - $USERDIR/Borgmatic/.ssh/:/root/.ssh
      - $USERDIR/Borgmatic/.state/:/root/.borgmatic
      - $USERDIR/Borgmatic/.cache/borg/:/root/.cache/borg
      - $BORGSCRIPTS:/borgscripts
      - /var/run/docker.sock:/var/run/docker.sock # So we can run scripts
    networks:
      isonet:
      isolated:
    restart: always

Logs:

docker 20.10.21
borgmatic 1.7.10
borg 1.2.4
apprise 1.3.0
python 3.11.2
Applying custom cron
Cron job set as: 
0 5 * * * borgmatic --stats -v 0 2>&1

crond: crond (busybox 1.35.0) started, log level 8
crond: USER root pid 1512 cmd borgmatic --stats -v 0 2>&1
/etc/borgmatic/config.yaml /etc/borgmatic.d /root/.config/borgmatic/config.yaml /root/.config/borgmatic.d: No valid configuration files found

OS: Ubuntu 22.04
Docker container.

Dockerfile: https://github.com/modem7/docker-borgmatic/blob/master/base-fullbuild/Dockerfile

#### What I'm trying to do and why Create backup #### Steps to reproduce (if a bug) Was working until upgrade to 1.7.10. Error: ``` /etc/borgmatic/config.yaml /etc/borgmatic.d /root/.config/borgmatic/config.yaml /root/.config/borgmatic.d: No valid configuration files found summary: /etc/borgmatic.d/config.yml: Error parsing configuration file An error occurred while parsing a configuration file at /etc/borgmatic.d/config.yml: Unknown repository in the "consistency" section's "check_repositories": /mnt/borg-repository /etc/borgmatic/config.yaml /etc/borgmatic.d /root/.config/borgmatic/config.yaml /root/.config/borgmatic.d: No valid configuration files found Need some help? https://torsion.org/borgmatic/#issues ``` Location of yaml: /etc/borgmatic.d/config.yaml Config contents: ```yaml location: source_directories: - ${BORG_SOURCE_1} - ${BORG_SOURCE_2} repositories: - ${BORG_REPO_LOCAL} - ${BORG_REPO_REMOTE} one_file_system: true exclude_caches: true storage: # Passphase is set in variable $BORG_PASSPHRASE compression: lz4 archive_name_format: 'backup-{now}' retention: keep_hourly: 0 keep_daily: 7 keep_weekly: 4 keep_monthly: 12 keep_yearly: 1 prefix: 'backup-' consistency: check_repositories: - ${BORG_REPO_LOCAL} checks: - name: repository frequency: 1 week - name: archives frequency: always - name: extract frequency: 2 weeks - name: data frequency: 1 month prefix: 'backup-' hooks: before_everything: - borgmatic break-lock - echo "Starting a backup job." - echo "Stopping containers." - exec /borgscripts/docker-stop.sh after_everything: - echo "Starting containers." - exec /borgscripts/docker-start.sh - echo "Backup created." on_error: - echo "Error while creating a backup." - exec /borgscripts/docker-start.sh # https://torsion.org/borgmatic/docs/how-to/backup-your-databases/ # Third-party services to notify you if backups aren't happening. healthchecks: ${BORG_HEALTHCHECK_URL} # https://torsion.org/borgmatic/docs/reference/configuration/ ``` Docker-compose: ```yaml borgmatic: # image: modem7/borgmatic-docker:dockercli image: docker.modem7.com/modem7/borgmatic-docker:dockercli # image: modem7/borgmatic-docker init: true container_name: Borgmatic environment: TZ: $TZ BORG_PASSPHRASE: $BORG_PASSPHRASE BORG_SOURCE_1: $BORG_SOURCE_1 BORG_SOURCE_2: $BORG_SOURCE_2 BORG_REPO_LOCAL: $BORG_REPO_LOCAL BORG_REPO_REMOTE: $BORG_REPO_REMOTE BORG_HEALTHCHECK_URL: $BORG_HEALTHCHECK_URL CRON: $BORG_CRON CRON_COMMAND: $BORG_CRON_COMMAND # EXTRA_CRON: |- # 0 5 2 * * command1 # 0 7 1 * * command2 # - LIVEINSTALL=docker-cli nano logging: driver: "local" options: max-size: 10m max-file: "3" volumes: - $BORGHOMESOURCEDIR:/mnt/source/ - $USERDIR:/mnt/source/DockerApps/ # - $CRONTAB:/mnt/source/Cron # - Pihole:/mnt/source/Pihole/Pihole # - Dnsmasq:/mnt/source/Pihole/Dnsmasq - $BORGSERVBACKUPDIR/Database:/mnt/borg-DBrepository - $BORGSERVBACKUPDIR/Docker:/mnt/borg-repository - $RAMBACKUP/Backup:/mnt/ramdrive - $USERDIR/Borgmatic/borgmatic.d/:/etc/borgmatic.d/ - $USERDIR/Borgmatic/.config/borg/:/root/.config/borg - $USERDIR/Borgmatic/.ssh/:/root/.ssh - $USERDIR/Borgmatic/.state/:/root/.borgmatic - $USERDIR/Borgmatic/.cache/borg/:/root/.cache/borg - $BORGSCRIPTS:/borgscripts - /var/run/docker.sock:/var/run/docker.sock # So we can run scripts networks: isonet: isolated: restart: always ``` Logs: ``` docker 20.10.21 borgmatic 1.7.10 borg 1.2.4 apprise 1.3.0 python 3.11.2 Applying custom cron Cron job set as: 0 5 * * * borgmatic --stats -v 0 2>&1 crond: crond (busybox 1.35.0) started, log level 8 crond: USER root pid 1512 cmd borgmatic --stats -v 0 2>&1 /etc/borgmatic/config.yaml /etc/borgmatic.d /root/.config/borgmatic/config.yaml /root/.config/borgmatic.d: No valid configuration files found ``` OS: Ubuntu 22.04 Docker container. Dockerfile: https://github.com/modem7/docker-borgmatic/blob/master/base-fullbuild/Dockerfile

diivi commented

2023-03-28 13:00:55 +00:00

Nice, thanks for filing this!

For context, I recently worked on this PR - https://github.com/borgmatic-collective/borgmatic/pull/57, which allows users to tag/label a repo, hence changing the data structure of repositories throughout the codebase ( from a str to a dict with keys path and label).

Now I knew I could miss something out and that's exactly what happened 😅.

In this case particularly - validate.py::apply_logical_validation() fails because it expects the repositories in the parsed configuration to be a list of strings, which we changed to a list of dicts.

def apply_logical_validation(config_filename, parsed_configuration):
    '''
    Given a parsed and schematically valid configuration as a data structure of nested dicts (see
    below), run through any additional logical validation checks. If there are any such validation
    problems, raise a Validation_error.
    '''
    location_repositories = parsed_configuration.get('location', {}).get('repositories')
    check_repositories = parsed_configuration.get('consistency', {}).get('check_repositories', [])
    for repository in check_repositories:
        print("parsed config repositories= ",location_repositories,"check repositories repo= ", repository)
        if repository not in location_repositories:
            raise Validation_error(
                config_filename,
                (
                    f'Unknown repository in the "consistency" section\'s "check_repositories": {repository}',
                ),
            )

This is what the quick print statement results in:

parsed config repositories=  [{'path': '/home/divyansh/Desktop/bkt2', 'label': 'my_repo'}] check repositories repo=  /home/divyansh/Desktop/bkt2

Hence, the error that you are facing. @witten I'll patch a fix for this, looking for other repository references in the validate.py file too. Do you know any other place in the config where a reference to the repositories key is made?

This means I can add label support to the check_repositories array too. Again, really helpful ticket, I was looking for community feedback and got it in a single day!

Nice, thanks for filing this! For context, I recently worked on this PR - https://github.com/borgmatic-collective/borgmatic/pull/57, which allows users to tag/label a repo, hence changing the data structure of repositories throughout the codebase ( from a `str` to a `dict` with keys `path` and `label`). Now I knew I could miss something out and that's exactly what happened 😅. In this case particularly - `validate.py::apply_logical_validation()` fails because it expects the repositories in the parsed configuration to be a list of strings, which we changed to a list of dicts. ```python3 def apply_logical_validation(config_filename, parsed_configuration): ''' Given a parsed and schematically valid configuration as a data structure of nested dicts (see below), run through any additional logical validation checks. If there are any such validation problems, raise a Validation_error. ''' location_repositories = parsed_configuration.get('location', {}).get('repositories') check_repositories = parsed_configuration.get('consistency', {}).get('check_repositories', []) for repository in check_repositories: print("parsed config repositories= ",location_repositories,"check repositories repo= ", repository) if repository not in location_repositories: raise Validation_error( config_filename, ( f'Unknown repository in the "consistency" section\'s "check_repositories": {repository}', ), ) ``` This is what the quick `print` statement results in: ```bash parsed config repositories= [{'path': '/home/divyansh/Desktop/bkt2', 'label': 'my_repo'}] check repositories repo= /home/divyansh/Desktop/bkt2 ``` Hence, the error that you are facing. @witten I'll patch a fix for this, looking for other `repository` references in the `validate.py` file too. Do you know any other place in the config where a reference to the repositories key is made? This means I can add label support to the `check_repositories` array too. Again, really helpful ticket, I was looking for community feedback and got it in a single day!

❤️ 1

modem7 commented

2023-03-28 13:04:56 +00:00

Oh thank god - I thought I screwed something up (usually the case), especially as two other VMs (albeit with one repo attached rather than multiple) were working!

Thanks for the quick response and I'm glad it was a productive ticket!!

Oh thank god - I thought I screwed something up (usually the case), especially as two other VMs (albeit with one repo attached rather than multiple) were working! Thanks for the quick response and I'm glad it was a productive ticket!!

witten added the

bug

label 2023-03-28 16:26:51 +00:00

witten commented

2023-03-28 16:29:13 +00:00

Yes, thanks for jumping in @diivi. And @modem7, I appreciate the detailed ticket. You can likely comment out check_repositories in order to get borgmatic working again, at least in the interim. We'll keep you updated.

Yes, thanks for jumping in @diivi. And @modem7, I appreciate the detailed ticket. You can likely comment out `check_repositories` in order to get borgmatic working again, at least in the interim. We'll keep you updated.

witten commented

2023-03-28 16:52:21 +00:00

PR: https://github.com/borgmatic-collective/borgmatic/pull/63

witten commented

2023-03-28 18:43:28 +00:00

Okay, @diivi has fixed this in master and it will be part of the next release. I was hoping not to do an immediate fix release, but we'll see if any other bugs crop up. Thanks again for reporting this.

witten closed this issue

2023-03-28 18:43:28 +00:00

witten referenced this issue from a commit

2023-03-28 18:46:23 +00:00

fix: make check repositories work with dict and str repositories (#662).

witten referenced this issue from a commit

2023-03-28 18:46:23 +00:00

Add check_repositories regression fix to NEWS (#662).

modem7 commented

2023-03-28 18:47:01 +00:00

Thank you guys for this amazingly quick turnaround!

👍 2