borgmatic-collective/borgmatic

Watch

Fork 57

"borgmatic generate config" output doesn't validate with "borgmatic config validate" #1091

New issue

Closed

opened 2025-05-07 15:09:51 +00:00 by edrulrd · 3 comments

edrulrd commented

2025-05-07 15:09:51 +00:00

What I'm trying to do and why

Issue summary: a working configuration file that is generated by

borgmatic generate config  --source file --destination fileout

doesn't validate with

borgmatic config validate --config fileout

Systems affected: borgmatic --versions from at least 2.0.3 to 2.0.6.dev0

Proof follows:

With the recently upgraded borgmatic to 2.0.3, (and confirmed that the issue still exists in 2.0.6.dev0) I am now getting the following warning messages:

summary:
/tmp/config: before_backup is deprecated and support will be removed from a future release. Use commands: instead.
/tmp/config: after_backup is deprecated and support will be removed from a future release. Use commands: instead.
/tmp/config: before_prune is deprecated and support will be removed from a future release. Use commands: instead.
/tmp/config: after_prune is deprecated and support will be removed from a future release. Use commands: instead.
/tmp/config: before_check is deprecated and support will be removed from a future release. Use commands: instead.
/tmp/config: after_check is deprecated and support will be removed from a future release. Use commands: instead.
/tmp/config: before_extract is deprecated and support will be removed from a future release. Use commands: instead.
/tmp/config: after_extract is deprecated and support will be removed from a future release. Use commands: instead.
/tmp/config: on_error is deprecated and support will be removed from a future release. Use commands: instead.
/tmp/config: before_everything is deprecated and support will be removed from a future release. Use commands: instead.
/tmp/config: after_everything is deprecated and support will be removed from a future release. Use commands: instead.
All configuration files are valid

I have been successfully using the following configuration for some time: (Note all commented lines deleted for readability)

   source_directories:
        - /var/services/homes/edrulrd
    repositories:
        - path: /volumeUSB2/usbshare/BorgBackup/nas1
    remote_path: /usr/local/bin/borg
    patterns:
        - '+ /var/services/homes/admin'
        - '+ /var/services/homes/edrulrd'
    exclude_patterns:
        - '**.pyc'
        - '**/.vim*.tmp'
        - '**/.cache'
        - '**/.mozilla'
        - '**/.thumbnails'
        - '**/.Trash*'
        - '**/Trash*'
        - '**/cache'
        - '**/downloads'
        - '**/Downloads'
        - '**/*image.gz'
        - '**/*gz'
        - /config
        - /dump.rdb
        - /dev
        - /etc.defaults
        - /mnt
        - /proc
        - /run
        - /sbin
        - /snap
        - /sys
        - /tmp
        - /var/run
        - /var/tmp/*
        - /usr
        - /var
        - /var.defaults
        - /volume1/data/Ed/Images/
        - /volumeUSB1
        - /volumeUSB2
    compression: zlib
    temporary_directory: /tmp
    archive_name_format: '{hostname}-{now}'
    extra_borg_options:
        create: --list --filter AME
    keep_daily: 7
    keep_weekly: 4
    keep_monthly: -1
    keep_yearly: 1
    before_backup:
        - echo "Starting a backup."
        - echo "Current backup media space utilization:"
        - df -h /volumeUSB2/usbshare
        - echo
    before_prune:
        - echo "Starting pruning."
    before_check:
        - echo "Starting checks."
    before_extract:
        - echo "Starting extracting."
    after_backup:
        - echo "Finished a backup."
    after_prune:
        - echo "Finished pruning."
    after_check:
        - echo "Finished checks."
    after_extract:
        - echo "Finished extracting."
    on_error:
        - echo "Error during prune/create/check."
    before_everything:
        - echo "Starting actions."
    after_everything:
        - echo "Completed actions."
        - echo "Current backup media space utilization:"
        - df -h /volumeUSB2/usbshare

So, I used the command borgmatic config generate against the config file, and it generated:

source_directories:
    - /var/services/homes/edrulrd
repositories:
    - path: /volumeUSB2/usbshare/BorgBackup/nas1
remote_path: /usr/local/bin/borg
patterns:
    - + /var/services/homes/admin
    - + /var/services/homes/edrulrd
exclude_patterns:
    - '**.pyc'
    - '**/.vim*.tmp'
    - '**/.cache'
    - '**/.mozilla'
    - '**/.thumbnails'
    - '**/.Trash*'
    - '**/Trash*'
    - '**/cache'
    - '**/downloads'
    - '**/Downloads'
    - '**/*image.gz'
    - '**/*gz'
    - /config
    - /dump.rdb
    - /dev
    - /etc.defaults
    - /mnt
    - /proc
    - /run
    - /sbin
    - /snap
    - /sys
    - /tmp
    - /var/run
    - /var/tmp/*
    - /usr
    - /var
    - /var.defaults
    - /volume1/data/Ed/Images/
    - /volumeUSB1
    - /volumeUSB2
compression: zlib
temporary_directory: /tmp
archive_name_format: '{hostname}-{now}'
extra_borg_options:
    create: --list --filter AME
keep_daily: 7
keep_weekly: 4
keep_monthly: -1
keep_yearly: 1
commands:
    - before: action
      after: action
      when:
          - create
      run:
          - echo "Starting a backup."
          - echo "Current backup media space utilization:"
          - df -h /volumeUSB2/usbshare
          - echo
bootstrap:

Then I ran the validate command: borgmatic config validate against the the generated config. And the following error messages were issued:

summary:
/tmp/config-new: Error parsing configuration file
An error occurred while parsing a configuration file at /tmp/config-new:
At 'bootstrap': None is not of type 'object'
At 'commands[0]': {'after': 'action', 'before': 'action', 'run': ['echo "Starting a backup."', 'echo "Current backup media space utilization:"', 'df -h /volumeUSB2/usbshare', 'echo'], 'when': ['create']} is not valid under any of the given schemas
Configuration validation failed

Need some help? https://torsion.org/borgmatic/#issues

I have found that if I change the commands: section to be set to the following, and commented out the bootstrap line, that the configuration validates (and borgmatic successfully runs):

commands:
     - before: everything
       when:
           - create
       run:
           - echo "Starting actions."
           - echo "Current backup media space utilization:"
           - df -h /volumeUSB2/usbshare
     - after: everything
       when:
           - create
       run:
           - echo "Completed actions."
           - echo "Current backup media space utilization:"
           - df -h /volumeUSB2/usbshare

Steps to reproduce

see above

Actual behavior

No response

Expected behavior

a config file that is generated by borgmatic should be able to successfully validate it (pretty obvious :-)

Other notes / implementation ideas

No response

borgmatic version

2.0.3 and 2.0.6.dev0

borgmatic installation method

pip install

Borg version

1.2.4 and 1.4.1

Python version

3.11.2 and 3.8.15

Database version (if applicable)

N/A

Operating system and version

PRETTY_NAME="Debian GNU/Linux 12 (bookworm)" NAME="Debian GNU/Linux" VERSION_ID="12" VERSION="12 (bookworm)" VERSION_CODENAME=bookworm ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/"

### What I'm trying to do and why Issue summary: a working configuration file that is generated by ``` borgmatic generate config --source file --destination fileout ``` doesn't validate with ``` borgmatic config validate --config fileout ``` Systems affected: ```borgmatic --versions from at least 2.0.3 to 2.0.6.dev0``` Proof follows: With the recently upgraded borgmatic to 2.0.3, (and confirmed that the issue still exists in 2.0.6.dev0) I am now getting the following warning messages: ``` summary: /tmp/config: before_backup is deprecated and support will be removed from a future release. Use commands: instead. /tmp/config: after_backup is deprecated and support will be removed from a future release. Use commands: instead. /tmp/config: before_prune is deprecated and support will be removed from a future release. Use commands: instead. /tmp/config: after_prune is deprecated and support will be removed from a future release. Use commands: instead. /tmp/config: before_check is deprecated and support will be removed from a future release. Use commands: instead. /tmp/config: after_check is deprecated and support will be removed from a future release. Use commands: instead. /tmp/config: before_extract is deprecated and support will be removed from a future release. Use commands: instead. /tmp/config: after_extract is deprecated and support will be removed from a future release. Use commands: instead. /tmp/config: on_error is deprecated and support will be removed from a future release. Use commands: instead. /tmp/config: before_everything is deprecated and support will be removed from a future release. Use commands: instead. /tmp/config: after_everything is deprecated and support will be removed from a future release. Use commands: instead. All configuration files are valid ``` I have been successfully using the following configuration for some time: (Note all commented lines deleted for readability) ``` source_directories: - /var/services/homes/edrulrd repositories: - path: /volumeUSB2/usbshare/BorgBackup/nas1 remote_path: /usr/local/bin/borg patterns: - '+ /var/services/homes/admin' - '+ /var/services/homes/edrulrd' exclude_patterns: - '**.pyc' - '**/.vim*.tmp' - '**/.cache' - '**/.mozilla' - '**/.thumbnails' - '**/.Trash*' - '**/Trash*' - '**/cache' - '**/downloads' - '**/Downloads' - '**/*image.gz' - '**/*gz' - /config - /dump.rdb - /dev - /etc.defaults - /mnt - /proc - /run - /sbin - /snap - /sys - /tmp - /var/run - /var/tmp/* - /usr - /var - /var.defaults - /volume1/data/Ed/Images/ - /volumeUSB1 - /volumeUSB2 compression: zlib temporary_directory: /tmp archive_name_format: '{hostname}-{now}' extra_borg_options: create: --list --filter AME keep_daily: 7 keep_weekly: 4 keep_monthly: -1 keep_yearly: 1 before_backup: - echo "Starting a backup." - echo "Current backup media space utilization:" - df -h /volumeUSB2/usbshare - echo before_prune: - echo "Starting pruning." before_check: - echo "Starting checks." before_extract: - echo "Starting extracting." after_backup: - echo "Finished a backup." after_prune: - echo "Finished pruning." after_check: - echo "Finished checks." after_extract: - echo "Finished extracting." on_error: - echo "Error during prune/create/check." before_everything: - echo "Starting actions." after_everything: - echo "Completed actions." - echo "Current backup media space utilization:" - df -h /volumeUSB2/usbshare ``` So, I used the command ```borgmatic config generate``` against the config file, and it generated: ``` source_directories: - /var/services/homes/edrulrd repositories: - path: /volumeUSB2/usbshare/BorgBackup/nas1 remote_path: /usr/local/bin/borg patterns: - + /var/services/homes/admin - + /var/services/homes/edrulrd exclude_patterns: - '**.pyc' - '**/.vim*.tmp' - '**/.cache' - '**/.mozilla' - '**/.thumbnails' - '**/.Trash*' - '**/Trash*' - '**/cache' - '**/downloads' - '**/Downloads' - '**/*image.gz' - '**/*gz' - /config - /dump.rdb - /dev - /etc.defaults - /mnt - /proc - /run - /sbin - /snap - /sys - /tmp - /var/run - /var/tmp/* - /usr - /var - /var.defaults - /volume1/data/Ed/Images/ - /volumeUSB1 - /volumeUSB2 compression: zlib temporary_directory: /tmp archive_name_format: '{hostname}-{now}' extra_borg_options: create: --list --filter AME keep_daily: 7 keep_weekly: 4 keep_monthly: -1 keep_yearly: 1 commands: - before: action after: action when: - create run: - echo "Starting a backup." - echo "Current backup media space utilization:" - df -h /volumeUSB2/usbshare - echo bootstrap: ``` Then I ran the validate command: ```borgmatic config validate ``` against the the generated config. And the following error messages were issued: ``` summary: /tmp/config-new: Error parsing configuration file An error occurred while parsing a configuration file at /tmp/config-new: At 'bootstrap': None is not of type 'object' At 'commands[0]': {'after': 'action', 'before': 'action', 'run': ['echo "Starting a backup."', 'echo "Current backup media space utilization:"', 'df -h /volumeUSB2/usbshare', 'echo'], 'when': ['create']} is not valid under any of the given schemas Configuration validation failed Need some help? https://torsion.org/borgmatic/#issues ``` I have found that if I change the ```commands:``` section to be set to the following, and commented out the ``` bootstrap``` line, that the configuration validates (and borgmatic successfully runs): ``` commands: - before: everything when: - create run: - echo "Starting actions." - echo "Current backup media space utilization:" - df -h /volumeUSB2/usbshare - after: everything when: - create run: - echo "Completed actions." - echo "Current backup media space utilization:" - df -h /volumeUSB2/usbshare ``` Ed ### Steps to reproduce see above ### Actual behavior _No response_ ### Expected behavior a config file that is generated by borgmatic should be able to successfully validate it (pretty obvious :-) ### Other notes / implementation ideas _No response_ ### borgmatic version 2.0.3 and 2.0.6.dev0 ### borgmatic installation method pip install ### Borg version 1.2.4 and 1.4.1 ### Python version 3.11.2 and 3.8.15 ### Database version (if applicable) N/A ### Operating system and version PRETTY_NAME="Debian GNU/Linux 12 (bookworm)" NAME="Debian GNU/Linux" VERSION_ID="12" VERSION="12 (bookworm)" VERSION_CODENAME=bookworm ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/"

witten commented

2025-05-07 17:20:07 +00:00

Owner

Thanks for taking the time to file this and provide all these details! I've got a repro locally, so I'll dig in when I get a chance and figure out what's going wrong.

witten added the

bug

label

2025-05-07 17:20:14 +00:00

witten referenced this issue from a commit

2025-05-08 20:00:52 +00:00

Fix for the "config generate" action generating invalid configuration when upgrading deprecated command hooks (#1091).

witten commented

2025-05-08 20:01:26 +00:00

Owner

I've implemented the fix for this in main, and it'll be part of the next release! Thanks again for bringing this to my attention.

witten closed this issue