hooks command parameter spaces not allowed #414

New Issue

Cartache · 2021-05-03T16:28:25Z

Cartache commented

2021-05-03 16:28:25 +00:00

What I'm trying to do and why

I am running borgmatic in a docker container on the latest version of unraid.
I have created a script to send a notification to unraid in case of error from within the container. The command is executed through a ssh to unraid. The script expects several parameters. Everything works as exepected when I do not put spaces in the parameters. If I try to insert parameters enclosed within '' or "" or " escaped or ' escaped, the borg job crashes telling that the yaml file is not correctly built.

I have tried the same script from a shell within the container and it works with spaces in the parameters enclosed within "".

Actual behavior (if a bug)

WORKING:

    on_error:
        - echo "Error during prune/create/check."
        - ssh 192.168.10.222 /scripts/docker/docker_start.sh -n ha-dockermon
        - ssh 192.168.10.222 /scripts/unraid/unraid_notify.sh -e Backup -i error -s ha-dockermon -d Error

This way the job runs smoothly without any issue

NOT WORKING

    on_error:
#    <<: !include /etc/borgmatic.d/includes/hooks_on_error_common.yaml
        - echo "Error during prune/create/check."
        - ssh 192.168.10.222 /scripts/docker/docker_start.sh -n ha-dockermon
        - ssh 192.168.10.222 "/scripts/unraid/unraid_notify.sh -e 'Backup Notification' -i 'alert' -s 'Borgmatic HA-Dockermon: Error in Backup' -d 'Borg is in error'"

Error generated:

# PATH=$PATH:/usr/bin /usr/bin/flock /opt/cron.lock /usr/bin/borgmatic --log-file /borg_logs/docker_ha-dockermon.log -c /etc/borgmatic.d/config_docker_ha-dockermon.yaml prune
 create -v 2 --stats 2>&1
validation.invalid
 --- All found errors ---
['Value \'{\'ssh 192.168.10.222 "/scripts/unraid/unraid_notify.sh -e \\\'Backup Notification\\\' -i \\\'alert\\\' -s \\\'Borgmatic HA-Dockermon\': \'Error in Backup\\\' -d \\\'Borg is in error\\\'"\'}\' is not of type \'str\'. Path: \'/hooks/on_error/2\'']
Errors found but will not raise exception...
Ensuring legacy configuration is upgraded
/etc/borgmatic.d/config_docker_ha-dockermon.yaml: No valid configuration files found

summary:
/etc/borgmatic.d/config_docker_ha-dockermon.yaml: Error parsing configuration file
An error occurred while parsing a configuration file at /etc/borgmatic.d/config_docker_ha-dockermon.yaml:
Value '{'ssh 192.168.10.222 "/scripts/unraid/unraid_notify.sh -e \'Backup Notification\' -i \'alert\' -s \'Borgmatic HA-Dockermon': 'Error in Backup\' -d \'Borg is in error\'"'}' is not of type 'str'. Path: '/hooks/on_error/2'
/etc/borgmatic.d/config_docker_ha-dockermon.yaml: No valid configuration files found

Need some help? https://torsion.org/borgmatic/#issues

Log File content:

[2021-05-03 18:20:26,541] CRITICAL: /etc/borgmatic.d/config_docker_ha-dockermon.yaml: No valid configuration files found
[2021-05-03 18:20:26,541] CRITICAL: 
[2021-05-03 18:20:26,541] CRITICAL: summary:
[2021-05-03 18:20:26,443] CRITICAL: /etc/borgmatic.d/config_docker_ha-dockermon.yaml: Error parsing configuration file
[2021-05-03 18:20:26,443] CRITICAL: An error occurred while parsing a configuration file at /etc/borgmatic.d/config_docker_ha-dockermon.yaml:
Value '{'ssh 192.168.10.222 "/scripts/unraid/unraid_notify.sh -e \'Backup Notification\' -i \'alert\' -s \'Borgmatic HA-Dockermon': 'Error in Backup\' -d \'Borg is in error\'"'}' is not of type 'str'. Path: '/hooks/on_error/2'
[2021-05-03 18:20:26,541] CRITICAL: /etc/borgmatic.d/config_docker_ha-dockermon.yaml: No valid configuration files found
[2021-05-03 18:20:26,541] CRITICAL: 
[2021-05-03 18:20:26,542] CRITICAL: Need some help? https://torsion.org/borgmatic/#issues
[2021-05-03 18:26:31,533] DEBUG: Ensuring legacy configuration is upgraded
[2021-05-03 18:26:31,533] CRITICAL: /etc/borgmatic.d/config_docker_ha-dockermon.yaml: No valid configuration files found
[2021-05-03 18:26:31,533] CRITICAL: 
[2021-05-03 18:26:31,534] CRITICAL: summary:
[2021-05-03 18:26:31,533] CRITICAL: /etc/borgmatic.d/config_docker_ha-dockermon.yaml: Error parsing configuration file
[2021-05-03 18:26:31,533] CRITICAL: An error occurred while parsing a configuration file at /etc/borgmatic.d/config_docker_ha-dockermon.yaml:
Value '{'ssh 192.168.10.222 "/scripts/unraid/unraid_notify.sh -e \'Backup Notification\' -i \'alert\' -s \'Borgmatic HA-Dockermon': 'Error in Backup\' -d \'Borg is in error\'"'}' is not of type 'str'. Path: '/hooks/on_error/2'
[2021-05-03 18:26:31,533] CRITICAL: /etc/borgmatic.d/config_docker_ha-dockermon.yaml: No valid configuration files found
[2021-05-03 18:26:31,534] CRITICAL: 
[2021-05-03 18:26:31,534] CRITICAL: Need some help? https://torsion.org/borgmatic/#issues

Environment

borgmatic version: 1.5.12

Use sudo borgmatic --version or sudo pip show borgmatic | grep ^Version

borgmatic installation method: Docker Container

Borg version: 1.1.15

Use sudo borg --version

Python version: 3.8.5

Use python3 --version

Database version (if applicable): psql 12.5

Use psql --version or mysql --version on client and server.

operating system and version: Unraid 6.8.2

#### What I'm trying to do and why I am running borgmatic in a docker container on the latest version of unraid. I have created a script to send a notification to unraid in case of error from within the container. The command is executed through a ssh to unraid. The script expects several parameters. Everything works as exepected when I do not put spaces in the parameters. If I try to insert parameters enclosed within '' or "" or " escaped or ' escaped, the borg job crashes telling that the yaml file is not correctly built. I have tried the same script from a shell within the container and it works with spaces in the parameters enclosed within "". #### Actual behavior (if a bug) ## WORKING: ``` on_error: - echo "Error during prune/create/check." - ssh 192.168.10.222 /scripts/docker/docker_start.sh -n ha-dockermon - ssh 192.168.10.222 /scripts/unraid/unraid_notify.sh -e Backup -i error -s ha-dockermon -d Error ``` This way the job runs smoothly without any issue ## NOT WORKING ``` on_error: # <<: !include /etc/borgmatic.d/includes/hooks_on_error_common.yaml - echo "Error during prune/create/check." - ssh 192.168.10.222 /scripts/docker/docker_start.sh -n ha-dockermon - ssh 192.168.10.222 "/scripts/unraid/unraid_notify.sh -e 'Backup Notification' -i 'alert' -s 'Borgmatic HA-Dockermon: Error in Backup' -d 'Borg is in error'" ``` Error generated: ``` # PATH=$PATH:/usr/bin /usr/bin/flock /opt/cron.lock /usr/bin/borgmatic --log-file /borg_logs/docker_ha-dockermon.log -c /etc/borgmatic.d/config_docker_ha-dockermon.yaml prune create -v 2 --stats 2>&1 validation.invalid --- All found errors --- ['Value \'{\'ssh 192.168.10.222 "/scripts/unraid/unraid_notify.sh -e \\\'Backup Notification\\\' -i \\\'alert\\\' -s \\\'Borgmatic HA-Dockermon\': \'Error in Backup\\\' -d \\\'Borg is in error\\\'"\'}\' is not of type \'str\'. Path: \'/hooks/on_error/2\''] Errors found but will not raise exception... Ensuring legacy configuration is upgraded /etc/borgmatic.d/config_docker_ha-dockermon.yaml: No valid configuration files found summary: /etc/borgmatic.d/config_docker_ha-dockermon.yaml: Error parsing configuration file An error occurred while parsing a configuration file at /etc/borgmatic.d/config_docker_ha-dockermon.yaml: Value '{'ssh 192.168.10.222 "/scripts/unraid/unraid_notify.sh -e \'Backup Notification\' -i \'alert\' -s \'Borgmatic HA-Dockermon': 'Error in Backup\' -d \'Borg is in error\'"'}' is not of type 'str'. Path: '/hooks/on_error/2' /etc/borgmatic.d/config_docker_ha-dockermon.yaml: No valid configuration files found Need some help? https://torsion.org/borgmatic/#issues ``` Log File content: ``` [2021-05-03 18:20:26,541] CRITICAL: /etc/borgmatic.d/config_docker_ha-dockermon.yaml: No valid configuration files found [2021-05-03 18:20:26,541] CRITICAL: [2021-05-03 18:20:26,541] CRITICAL: summary: [2021-05-03 18:20:26,443] CRITICAL: /etc/borgmatic.d/config_docker_ha-dockermon.yaml: Error parsing configuration file [2021-05-03 18:20:26,443] CRITICAL: An error occurred while parsing a configuration file at /etc/borgmatic.d/config_docker_ha-dockermon.yaml: Value '{'ssh 192.168.10.222 "/scripts/unraid/unraid_notify.sh -e \'Backup Notification\' -i \'alert\' -s \'Borgmatic HA-Dockermon': 'Error in Backup\' -d \'Borg is in error\'"'}' is not of type 'str'. Path: '/hooks/on_error/2' [2021-05-03 18:20:26,541] CRITICAL: /etc/borgmatic.d/config_docker_ha-dockermon.yaml: No valid configuration files found [2021-05-03 18:20:26,541] CRITICAL: [2021-05-03 18:20:26,542] CRITICAL: Need some help? https://torsion.org/borgmatic/#issues [2021-05-03 18:26:31,533] DEBUG: Ensuring legacy configuration is upgraded [2021-05-03 18:26:31,533] CRITICAL: /etc/borgmatic.d/config_docker_ha-dockermon.yaml: No valid configuration files found [2021-05-03 18:26:31,533] CRITICAL: [2021-05-03 18:26:31,534] CRITICAL: summary: [2021-05-03 18:26:31,533] CRITICAL: /etc/borgmatic.d/config_docker_ha-dockermon.yaml: Error parsing configuration file [2021-05-03 18:26:31,533] CRITICAL: An error occurred while parsing a configuration file at /etc/borgmatic.d/config_docker_ha-dockermon.yaml: Value '{'ssh 192.168.10.222 "/scripts/unraid/unraid_notify.sh -e \'Backup Notification\' -i \'alert\' -s \'Borgmatic HA-Dockermon': 'Error in Backup\' -d \'Borg is in error\'"'}' is not of type 'str'. Path: '/hooks/on_error/2' [2021-05-03 18:26:31,533] CRITICAL: /etc/borgmatic.d/config_docker_ha-dockermon.yaml: No valid configuration files found [2021-05-03 18:26:31,534] CRITICAL: [2021-05-03 18:26:31,534] CRITICAL: Need some help? https://torsion.org/borgmatic/#issues ``` #### Environment **borgmatic version:** 1.5.12 Use `sudo borgmatic --version` or `sudo pip show borgmatic | grep ^Version` **borgmatic installation method:** Docker Container **Borg version:** 1.1.15 Use `sudo borg --version` **Python version:** 3.8.5 Use `python3 --version` **Database version (if applicable):** psql 12.5 Use `psql --version` or `mysql --version` on client and server. **operating system and version:** Unraid 6.8.2

witten commented

2021-05-03 16:46:43 +00:00

Thank you for the detailed bug report! It really helps in diagnosing.

So here's the thing—the spaces aren't the problem! It's the colon. Specifically, the one in HA-Dockermon: (I think that makes YAML interpret your command as a YAML dictionary rather than a string.)

So, a couple of ways around this:

Double quote the entire string from the very start, effectively telling YAML that it shouldn't try to interpret it at all. This would be a little obnoxious, however, because you'd have to backslash any double quotes you have within that string.
Or, just replace the colon with something else. A dash works, for instance.

Let me know if either of those ends up working for you! YAML has its strengths, but it's not perfect for embedding moderately complicated shell commands.

Thank you for the detailed bug report! It really helps in diagnosing. So here's the thing—the spaces aren't the problem! It's the colon. Specifically, the one in `HA-Dockermon:` (I think that makes YAML interpret your command as a YAML dictionary rather than a string.) So, a couple of ways around this: 1. Double quote the entire string from the very start, effectively telling YAML that it shouldn't try to interpret it at all. This would be a little obnoxious, however, because you'd have to backslash any double quotes you have *within* that string. 2. Or, just replace the colon with something else. A dash works, for instance. Let me know if either of those ends up working for you! YAML has its strengths, but it's not perfect for embedding moderately complicated shell commands.

witten added the

question / support

label 2021-05-03 16:46:49 +00:00

Cartache commented

2021-05-03 17:03:55 +00:00

Dear witten,

Thank you for this rapid feedback. I really appreciate the time you spent answering my request. Morevover when your reply and propsal is working as expected.

I simply removed the colon and managed to make it work.

Thanks again and have a nice week.

Denis

Dear witten, Thank you for this rapid feedback. I really appreciate the time you spent answering my request. Morevover when your reply and propsal is working as expected. I simply removed the colon and managed to make it work. Thanks again and have a nice week. Denis

witten commented

2021-05-03 19:34:38 +00:00

I'm glad to hear that did it!

witten closed this issue

2021-05-03 19:34:39 +00:00

Sign in to join this conversation.