borgmatic-collective/borgmatic
11
70
Fork 40
Code Issues 49 Pull Requests 7 Actions 1 Packages Releases 152 Activity

No valid configuration files found #662

New Issue
Closed
opened 2023-03-28 11:20:47 +00:00 by modem7 · 6 comments
modem7 commented 2023-03-28 11:20:47 +00:00
Copy Link

What I'm trying to do and why

Create backup

Steps to reproduce (if a bug)

Was working until upgrade to 1.7.10.
Error:

/etc/borgmatic/config.yaml /etc/borgmatic.d /root/.config/borgmatic/config.yaml /root/.config/borgmatic.d: No valid configuration files found

summary:
/etc/borgmatic.d/config.yml: Error parsing configuration file
An error occurred while parsing a configuration file at /etc/borgmatic.d/config.yml:
Unknown repository in the "consistency" section's "check_repositories": /mnt/borg-repository
/etc/borgmatic/config.yaml /etc/borgmatic.d /root/.config/borgmatic/config.yaml /root/.config/borgmatic.d: No valid configuration files found

Need some help? https://torsion.org/borgmatic/#issues

Location of yaml: /etc/borgmatic.d/config.yaml

Config contents:

location:
    source_directories:
         - ${BORG_SOURCE_1}
         - ${BORG_SOURCE_2}
    repositories:
         - ${BORG_REPO_LOCAL}
         - ${BORG_REPO_REMOTE}
    one_file_system: true
    exclude_caches: true

storage:
#   Passphase is set in variable $BORG_PASSPHRASE
    compression: lz4
    archive_name_format: 'backup-{now}'

retention:
    keep_hourly: 0
    keep_daily: 7
    keep_weekly: 4
    keep_monthly: 12
    keep_yearly: 1
    prefix: 'backup-'

consistency:
    check_repositories:
      - ${BORG_REPO_LOCAL}
    checks:
      - name: repository
        frequency: 1 week
      - name: archives
        frequency: always
      - name: extract
        frequency: 2 weeks
      - name: data
        frequency: 1 month
    prefix: 'backup-'

hooks:
    before_everything:
        - borgmatic break-lock
        - echo "Starting a backup job."
        - echo "Stopping containers."
        - exec /borgscripts/docker-stop.sh
    after_everything:
        - echo "Starting containers."
        - exec /borgscripts/docker-start.sh
        - echo "Backup created."
    on_error:
        - echo "Error while creating a backup."
        - exec /borgscripts/docker-start.sh
    # https://torsion.org/borgmatic/docs/how-to/backup-your-databases/

    # Third-party services to notify you if backups aren't happening.
    healthchecks: ${BORG_HEALTHCHECK_URL}

    # https://torsion.org/borgmatic/docs/reference/configuration/

Docker-compose:

  borgmatic:
    # image: modem7/borgmatic-docker:dockercli
    image: docker.modem7.com/modem7/borgmatic-docker:dockercli
    # image: modem7/borgmatic-docker
    init: true
    container_name: Borgmatic
    environment:
      TZ: $TZ
      BORG_PASSPHRASE: $BORG_PASSPHRASE
      BORG_SOURCE_1: $BORG_SOURCE_1
      BORG_SOURCE_2: $BORG_SOURCE_2
      BORG_REPO_LOCAL: $BORG_REPO_LOCAL
      BORG_REPO_REMOTE: $BORG_REPO_REMOTE
      BORG_HEALTHCHECK_URL: $BORG_HEALTHCHECK_URL
      CRON: $BORG_CRON
      CRON_COMMAND: $BORG_CRON_COMMAND
      # EXTRA_CRON: |-
      #   0 5 2 * * command1
      #   0 7 1 * * command2
      # - LIVEINSTALL=docker-cli nano
    logging:
      driver: "local"
      options:
        max-size: 10m
        max-file: "3"
    volumes:
      - $BORGHOMESOURCEDIR:/mnt/source/
      - $USERDIR:/mnt/source/DockerApps/
      # - $CRONTAB:/mnt/source/Cron
      # - Pihole:/mnt/source/Pihole/Pihole
      # - Dnsmasq:/mnt/source/Pihole/Dnsmasq
      - $BORGSERVBACKUPDIR/Database:/mnt/borg-DBrepository
      - $BORGSERVBACKUPDIR/Docker:/mnt/borg-repository
      - $RAMBACKUP/Backup:/mnt/ramdrive
      - $USERDIR/Borgmatic/borgmatic.d/:/etc/borgmatic.d/
      - $USERDIR/Borgmatic/.config/borg/:/root/.config/borg
      - $USERDIR/Borgmatic/.ssh/:/root/.ssh
      - $USERDIR/Borgmatic/.state/:/root/.borgmatic
      - $USERDIR/Borgmatic/.cache/borg/:/root/.cache/borg
      - $BORGSCRIPTS:/borgscripts
      - /var/run/docker.sock:/var/run/docker.sock # So we can run scripts
    networks:
      isonet:
      isolated:
    restart: always

Logs:

docker 20.10.21
borgmatic 1.7.10
borg 1.2.4
apprise 1.3.0
python 3.11.2
Applying custom cron
Cron job set as: 
0 5 * * * borgmatic --stats -v 0 2>&1

crond: crond (busybox 1.35.0) started, log level 8
crond: USER root pid 1512 cmd borgmatic --stats -v 0 2>&1
/etc/borgmatic/config.yaml /etc/borgmatic.d /root/.config/borgmatic/config.yaml /root/.config/borgmatic.d: No valid configuration files found

OS: Ubuntu 22.04
Docker container.

Dockerfile: https://github.com/modem7/docker-borgmatic/blob/master/base-fullbuild/Dockerfile

#### What I'm trying to do and why Create backup #### Steps to reproduce (if a bug) Was working until upgrade to 1.7.10. Error: ``` /etc/borgmatic/config.yaml /etc/borgmatic.d /root/.config/borgmatic/config.yaml /root/.config/borgmatic.d: No valid configuration files found summary: /etc/borgmatic.d/config.yml: Error parsing configuration file An error occurred while parsing a configuration file at /etc/borgmatic.d/config.yml: Unknown repository in the "consistency" section's "check_repositories": /mnt/borg-repository /etc/borgmatic/config.yaml /etc/borgmatic.d /root/.config/borgmatic/config.yaml /root/.config/borgmatic.d: No valid configuration files found Need some help? https://torsion.org/borgmatic/#issues ``` Location of yaml: /etc/borgmatic.d/config.yaml Config contents: ```yaml location: source_directories: - ${BORG_SOURCE_1} - ${BORG_SOURCE_2} repositories: - ${BORG_REPO_LOCAL} - ${BORG_REPO_REMOTE} one_file_system: true exclude_caches: true storage: # Passphase is set in variable $BORG_PASSPHRASE compression: lz4 archive_name_format: 'backup-{now}' retention: keep_hourly: 0 keep_daily: 7 keep_weekly: 4 keep_monthly: 12 keep_yearly: 1 prefix: 'backup-' consistency: check_repositories: - ${BORG_REPO_LOCAL} checks: - name: repository frequency: 1 week - name: archives frequency: always - name: extract frequency: 2 weeks - name: data frequency: 1 month prefix: 'backup-' hooks: before_everything: - borgmatic break-lock - echo "Starting a backup job." - echo "Stopping containers." - exec /borgscripts/docker-stop.sh after_everything: - echo "Starting containers." - exec /borgscripts/docker-start.sh - echo "Backup created." on_error: - echo "Error while creating a backup." - exec /borgscripts/docker-start.sh # https://torsion.org/borgmatic/docs/how-to/backup-your-databases/ # Third-party services to notify you if backups aren't happening. healthchecks: ${BORG_HEALTHCHECK_URL} # https://torsion.org/borgmatic/docs/reference/configuration/ ``` Docker-compose: ```yaml borgmatic: # image: modem7/borgmatic-docker:dockercli image: docker.modem7.com/modem7/borgmatic-docker:dockercli # image: modem7/borgmatic-docker init: true container_name: Borgmatic environment: TZ: $TZ BORG_PASSPHRASE: $BORG_PASSPHRASE BORG_SOURCE_1: $BORG_SOURCE_1 BORG_SOURCE_2: $BORG_SOURCE_2 BORG_REPO_LOCAL: $BORG_REPO_LOCAL BORG_REPO_REMOTE: $BORG_REPO_REMOTE BORG_HEALTHCHECK_URL: $BORG_HEALTHCHECK_URL CRON: $BORG_CRON CRON_COMMAND: $BORG_CRON_COMMAND # EXTRA_CRON: |- # 0 5 2 * * command1 # 0 7 1 * * command2 # - LIVEINSTALL=docker-cli nano logging: driver: "local" options: max-size: 10m max-file: "3" volumes: - $BORGHOMESOURCEDIR:/mnt/source/ - $USERDIR:/mnt/source/DockerApps/ # - $CRONTAB:/mnt/source/Cron # - Pihole:/mnt/source/Pihole/Pihole # - Dnsmasq:/mnt/source/Pihole/Dnsmasq - $BORGSERVBACKUPDIR/Database:/mnt/borg-DBrepository - $BORGSERVBACKUPDIR/Docker:/mnt/borg-repository - $RAMBACKUP/Backup:/mnt/ramdrive - $USERDIR/Borgmatic/borgmatic.d/:/etc/borgmatic.d/ - $USERDIR/Borgmatic/.config/borg/:/root/.config/borg - $USERDIR/Borgmatic/.ssh/:/root/.ssh - $USERDIR/Borgmatic/.state/:/root/.borgmatic - $USERDIR/Borgmatic/.cache/borg/:/root/.cache/borg - $BORGSCRIPTS:/borgscripts - /var/run/docker.sock:/var/run/docker.sock # So we can run scripts networks: isonet: isolated: restart: always ``` Logs: ``` docker 20.10.21 borgmatic 1.7.10 borg 1.2.4 apprise 1.3.0 python 3.11.2 Applying custom cron Cron job set as: 0 5 * * * borgmatic --stats -v 0 2>&1 crond: crond (busybox 1.35.0) started, log level 8 crond: USER root pid 1512 cmd borgmatic --stats -v 0 2>&1 /etc/borgmatic/config.yaml /etc/borgmatic.d /root/.config/borgmatic/config.yaml /root/.config/borgmatic.d: No valid configuration files found ``` OS: Ubuntu 22.04 Docker container. Dockerfile: https://github.com/modem7/docker-borgmatic/blob/master/base-fullbuild/Dockerfile
diivi commented 2023-03-28 13:00:55 +00:00
Collaborator
Copy Link

Nice, thanks for filing this!

For context, I recently worked on this PR - https://github.com/borgmatic-collective/borgmatic/pull/57, which allows users to tag/label a repo, hence changing the data structure of repositories throughout the codebase ( from a str to a dict with keys path and label).

Now I knew I could miss something out and that's exactly what happened 😅.

In this case particularly - validate.py::apply_logical_validation() fails because it expects the repositories in the parsed configuration to be a list of strings, which we changed to a list of dicts.

def apply_logical_validation(config_filename, parsed_configuration):
    '''
    Given a parsed and schematically valid configuration as a data structure of nested dicts (see
    below), run through any additional logical validation checks. If there are any such validation
    problems, raise a Validation_error.
    '''
    location_repositories = parsed_configuration.get('location', {}).get('repositories')
    check_repositories = parsed_configuration.get('consistency', {}).get('check_repositories', [])
    for repository in check_repositories:
        print("parsed config repositories= ",location_repositories,"check repositories repo= ", repository)
        if repository not in location_repositories:
            raise Validation_error(
                config_filename,
                (
                    f'Unknown repository in the "consistency" section\'s "check_repositories": {repository}',
                ),
            )

This is what the quick print statement results in:

parsed config repositories=  [{'path': '/home/divyansh/Desktop/bkt2', 'label': 'my_repo'}] check repositories repo=  /home/divyansh/Desktop/bkt2

Hence, the error that you are facing. @witten I'll patch a fix for this, looking for other repository references in the validate.py file too. Do you know any other place in the config where a reference to the repositories key is made?

This means I can add label support to the check_repositories array too. Again, really helpful ticket, I was looking for community feedback and got it in a single day!

Nice, thanks for filing this! For context, I recently worked on this PR - https://github.com/borgmatic-collective/borgmatic/pull/57, which allows users to tag/label a repo, hence changing the data structure of repositories throughout the codebase ( from a `str` to a `dict` with keys `path` and `label`). Now I knew I could miss something out and that's exactly what happened 😅. In this case particularly - `validate.py::apply_logical_validation()` fails because it expects the repositories in the parsed configuration to be a list of strings, which we changed to a list of dicts. ```python3 def apply_logical_validation(config_filename, parsed_configuration): ''' Given a parsed and schematically valid configuration as a data structure of nested dicts (see below), run through any additional logical validation checks. If there are any such validation problems, raise a Validation_error. ''' location_repositories = parsed_configuration.get('location', {}).get('repositories') check_repositories = parsed_configuration.get('consistency', {}).get('check_repositories', []) for repository in check_repositories: print("parsed config repositories= ",location_repositories,"check repositories repo= ", repository) if repository not in location_repositories: raise Validation_error( config_filename, ( f'Unknown repository in the "consistency" section\'s "check_repositories": {repository}', ), ) ``` This is what the quick `print` statement results in: ```bash parsed config repositories= [{'path': '/home/divyansh/Desktop/bkt2', 'label': 'my_repo'}] check repositories repo= /home/divyansh/Desktop/bkt2 ``` Hence, the error that you are facing. @witten I'll patch a fix for this, looking for other `repository` references in the `validate.py` file too. Do you know any other place in the config where a reference to the repositories key is made? This means I can add label support to the `check_repositories` array too. Again, really helpful ticket, I was looking for community feedback and got it in a single day!
❤️ 1
modem7 commented 2023-03-28 13:04:56 +00:00
Author
Copy Link

Oh thank god - I thought I screwed something up (usually the case), especially as two other VMs (albeit with one repo attached rather than multiple) were working!

Thanks for the quick response and I'm glad it was a productive ticket!!

Oh thank god - I thought I screwed something up (usually the case), especially as two other VMs (albeit with one repo attached rather than multiple) were working! Thanks for the quick response and I'm glad it was a productive ticket!!
witten added the bug label 2023-03-28 16:26:51 +00:00
witten commented 2023-03-28 16:29:13 +00:00
Owner
Copy Link

Yes, thanks for jumping in @diivi. And @modem7, I appreciate the detailed ticket. You can likely comment out check_repositories in order to get borgmatic working again, at least in the interim. We'll keep you updated.

Yes, thanks for jumping in @diivi. And @modem7, I appreciate the detailed ticket. You can likely comment out `check_repositories` in order to get borgmatic working again, at least in the interim. We'll keep you updated.
witten commented 2023-03-28 16:52:21 +00:00
Owner
Copy Link

PR: https://github.com/borgmatic-collective/borgmatic/pull/63

PR: https://github.com/borgmatic-collective/borgmatic/pull/63
witten commented 2023-03-28 18:43:28 +00:00
Owner
Copy Link

Okay, @diivi has fixed this in master and it will be part of the next release. I was hoping not to do an immediate fix release, but we'll see if any other bugs crop up. Thanks again for reporting this.

Okay, @diivi has fixed this in master and it will be part of the next release. I was hoping not to do an immediate fix release, but we'll see if any other bugs crop up. Thanks again for reporting this.
modem7 commented 2023-03-28 18:47:01 +00:00
Author
Copy Link

Thank you guys for this amazingly quick turnaround!

Thank you guys for this amazingly quick turnaround!
👍 2
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
blocked
breaking
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
No Label bug
Milestone
No items
No Milestone
Assignees
Clear assignees
witten
No Assignees
3 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#662
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?