on_error and Apprise causing Borgmatic to fail backups #839

New Issue

Closed

opened 2024-03-08 20:43:45 +00:00 by ericswpark · 7 comments

ericswpark commented 2024-03-08 20:43:45 +00:00

Copy Link

What I'm trying to do and why

I'm trying to figure out why my Borgmatic + Apprise configuration stopped working from a Borgmatic update, when it was working fine before. The problematic config lines are the following:

on_error:
  - apprise --title="Borgmatic" --body="An error occurred.\n{error}\n{output}" -e

Steps to reproduce

Try and run borgmatic with the configuration that includes the lines provided above

Actual behavior

It produces the following error:

/etc/borgmatic.d/config.yaml: Error running on-error hook
Usage: apprise [OPTIONS] SERVER_URL [SERVER_URL2 [SERVER_URL3]]
Try 'apprise -h' for help.
Error: No such option: --keep-daily
Command 'apprise --title="Borgmatic" --body="An error occurred.\n'Command '"'"'borg prune --keep-daily 7 --keep-hourly 24 --keep-monthly 6 --keep-yearly 5 --glob-archives server-1-* --stats --info --list ssh://borg@server-2/./server-1.borg'"'"' returned non-zero exit status 2.'\n'...
  File "/usr/lib/python3/dist-packages/borg/remote.py", line 368, in open
(...)

It seems like the {error} variable does some quote escaping, which manages to escape out of the --body="" quotes.

Expected behavior

Borgmatic should not fail to run with this configuration, by doing the quote escaping properly or not at all.

Other notes / implementation ideas

Transferred from https://github.com/borgmatic-collective/docker-borgmatic/issues/306

borgmatic version

1.8.8

borgmatic installation method

Docker container

Borg version

1.2.7

Python version

3.11.5

Database version (if applicable)

N/A

Operating system and version

Slackware 15.0 (unRAID)

### What I'm trying to do and why I'm trying to figure out why my Borgmatic + Apprise configuration stopped working from a Borgmatic update, when it was working fine before. The problematic config lines are the following: ``` on_error: - apprise --title="Borgmatic" --body="An error occurred.\n{error}\n{output}" -e ``` ### Steps to reproduce Try and run borgmatic with the configuration that includes the lines provided above ### Actual behavior It produces the following error: ``` /etc/borgmatic.d/config.yaml: Error running on-error hook Usage: apprise [OPTIONS] SERVER_URL [SERVER_URL2 [SERVER_URL3]] Try 'apprise -h' for help. Error: No such option: --keep-daily Command 'apprise --title="Borgmatic" --body="An error occurred.\n'Command '"'"'borg prune --keep-daily 7 --keep-hourly 24 --keep-monthly 6 --keep-yearly 5 --glob-archives server-1-* --stats --info --list ssh://borg@server-2/./server-1.borg'"'"' returned non-zero exit status 2.'\n'... File "/usr/lib/python3/dist-packages/borg/remote.py", line 368, in open (...) ``` It seems like the `{error}` variable does some quote escaping, which manages to escape out of the `--body=""` quotes. ### Expected behavior Borgmatic should not fail to run with this configuration, by doing the quote escaping properly or not at all. ### Other notes / implementation ideas Transferred from https://github.com/borgmatic-collective/docker-borgmatic/issues/306 ### borgmatic version 1.8.8 ### borgmatic installation method Docker container ### Borg version 1.2.7 ### Python version 3.11.5 ### Database version (if applicable) N/A ### Operating system and version Slackware 15.0 (unRAID)

witten commented 2024-03-08 21:36:14 +00:00

Owner

Copy Link

Thanks for (re)filing this! Since borgmatic #810 in 1.8.7, interpolated hook variables are escaped to prevent shell injection attacks, and so you are correct that escaping of the {error} value is what's causing problems here. Specifically, the problem looks to be that {error} is escaped as if it was a single argument when in fact it's contained in quotes within your --body value, ironically resulting in the "break-out" that causes the Apprise error you see.

Unfortunately I'm not sure of a good way around this at the escaping level. That's because the escaping function I'm using (shlex.escape()) isn't smart enough to know about the context in which it's used. I'm open to suggestions though.

However, I do have an idea for an alternative. If borgmatic's existing Apprise hook included support for sending logs (and therefore errors) to Apprise notification services, would that address your need here? I'm thinking of adding an optional send_logs flag that could be used as follows:

apprise:
    services:
        - url: gotify://hostname/token
          label: gotify
    send_logs: true
    ...
    fail:
        title: Uh oh!
        body: Your backups have failed. See logs for details.

That would cause full borgmatic logs to be added automatically to the Apprise body upon send.

Anyway, let me know your thoughts!

EDIT: Moved send_logs: up a level, because I don't think it needs to be configurable per state.

Thanks for (re)filing this! Since borgmatic #810 in 1.8.7, interpolated hook variables are escaped to prevent shell injection attacks, and so you are correct that escaping of the `{error}` value is what's causing problems here. Specifically, the problem looks to be that `{error}` is escaped as if it was a single argument when in fact it's contained in quotes within your `--body` value, ironically resulting in the "break-out" that causes the Apprise error you see. Unfortunately I'm not sure of a good way around this at the escaping level. That's because the escaping function I'm using (`shlex.escape()`) isn't smart enough to know about the context in which it's used. I'm open to suggestions though. However, I do have an idea for an alternative. If borgmatic's existing [Apprise hook](https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#apprise-hook) included support for sending logs (and therefore errors) to Apprise notification services, would that address your need here? I'm thinking of adding an optional `send_logs` flag that could be used as follows: ```yaml apprise: services: - url: gotify://hostname/token label: gotify send_logs: true ... fail: title: Uh oh! body: Your backups have failed. See logs for details. ``` That would cause full borgmatic logs to be added automatically to the Apprise `body` upon send. Anyway, let me know your thoughts! EDIT: Moved `send_logs:` up a level, because I don't think it needs to be configurable per state.

ericswpark commented 2024-03-08 21:46:32 +00:00

Author

Copy Link

Yup, the send_logs flag would definitely be very useful! The main reason I want the log included in the notifying email is because it's a bit of a chore to dig through the Docker logs to figure out where the error is, especially if Borgmatic has since done more automated backups following the schedule.

Yup, the `send_logs` flag would definitely be very useful! The main reason I want the log included in the notifying email is because it's a bit of a chore to dig through the Docker logs to figure out where the error is, especially if Borgmatic has since done more automated backups following the schedule.

witten commented 2024-03-08 21:56:33 +00:00

Owner

Copy Link

Okay, thanks. I'll work on getting send_logs supported. (It would still probably be a good idea to fix the escaping, but right now I don't have line of sight to a solution for that part.)

Okay, thanks. I'll work on getting `send_logs` supported. (It would still probably be a good idea to fix the escaping, but right now I don't have line of sight to a solution for that part.)

❤️ 1

witten commented 2024-03-10 23:31:53 +00:00

Owner

Copy Link

send_logs is now implemented in main, enabled by default without having to configure anything additional, and will be part of the next release! I also added a logs_size_limit option under apprise: which may be of use.

Documentation is here: https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#apprise-hook

Feedback welcome.

I'm going to leave this ticket open a bit longer though to look at the original escaping issue when I get a chance.

`send_logs` is now implemented in main, enabled by default without having to configure anything additional, and will be part of the next release! I also added a `logs_size_limit` option under `apprise:` which may be of use. Documentation is here: https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#apprise-hook Feedback welcome. I'm going to leave this ticket open a bit longer though to look at the original escaping issue when I get a chance.

🚀 2

witten referenced this issue from a commit 2024-03-11 17:43:10 +00:00

Document a potentially breaking shell quoting edge case within error hooks (#839).

witten commented 2024-03-11 17:57:22 +00:00

Owner

Copy Link

Well, I ended up "fixing" the escaping issue by documenting around it. (See the bottom of that section.)

Hopefully though it's no longer as much of an issue now that the Apprise hook can do its own logging.

Calling this done for now.

Well, I ended up "fixing" the escaping issue by [documenting around it](https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#error-hooks). (See the bottom of that section.) Hopefully though it's no longer as much of an issue now that the Apprise hook can do its own logging. Calling this done for now.

witten closed this issue 2024-03-11 17:57:22 +00:00

witten commented 2024-03-11 20:30:30 +00:00

Owner

Copy Link

Released in borgmatic 1.8.9!

Released in borgmatic 1.8.9!

ericswpark commented 2024-03-11 21:24:22 +00:00

Author

Copy Link

Thank you @witten! Looking forward to testing it once it hits the Docker registry and pulls to my server :)

Thank you @witten! Looking forward to testing it once it hits the Docker registry and pulls to my server :)

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

lvm-snapshots

1.9.3

1.9.2

1.9.1

1.9.0

1.8.14

1.8.13

1.8.12

1.8.11

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.24

1.5.23

1.5.22

1.5.21

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.14

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.7.dev0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.22

1.4.21

1.4.20

1.4.19

1.4.18

1.4.17

1.4.16

1.4.15

1.4.14

1.4.13

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.26

1.3.25

1.3.24

1.3.23

1.3.22

1.3.21

1.3.20

1.3.19

1.3.18

1.3.17

1.3.16

1.3.15

1.3.14

1.3.13

1.3.12

1.3.11

1.3.10

1.3.9

1.3.8

1.3.7

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.18

1.2.17

1.2.16

1.2.15

1.2.14

1.2.13

1.2.12

1.2.11

1.2.10

1.2.9

1.2.8

1.2.7

1.2.6

1.2.5

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.15

1.1.14

1.1.13

1.1.12

1.1.11

1.1.10

1.1.9

1.1.7

1.1.6

1.1.5

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.3

1.0.2

1.0.1

1.0.0

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

0.0.2

Labels

Clear labels   

blocked

  

bug

  

data loss

  

design finalized

  

good first issue

  

new feature area

  

question / support

  

security

  

waiting for response

No Label

blocked

bug

data loss

design finalized

good first issue

new feature area

question / support

security

waiting for response

Milestone

Clear milestone

No items

No Milestone

Assignees

Clear assignees

No Assignees

2 Participants

Notifications

Subscribe

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#839

No description provided.