borgmatic-collective/borgmatic
borgmatic-collective/borgmatic
Watch
9
74
Fork
You've already forked borgmatic
 57
Code Issues 71 Pull requests 3 Releases 162 Packages 1 Activity Actions 1

Borgmatic sends Healthchecks.io OK ping even when there was an error #1065

New issue
Closed
opened 2025-04-09 16:47:40 +00:00 by raxod502 · 6 comments
raxod502 commented 2025-04-09 16:47:40 +00:00
Copy link

What I'm trying to do and why

I'm using the configuration in #525 to only have a ping sent to Healthchecks.io when my backup operations have succeeded, for similar reasons as the ones described in that issue (want to only be alerted if backups are persistently down, not just having a transient error).

Steps to reproduce

source_directories:
  - "/redacted"

repositories:
  - path: "ssh://redacted"

exclude_patterns:
  - "*/.snapshots"

compression: auto,zstd
encryption_passphrase: "redacted"
ssh_command: "ssh -oIdentitiesOnly=yes -oControlPath=none -i ~/.ssh/redacted"
checkpoint_interval: 600

keep_within: 2w
keep_daily: 30
keep_weekly: 8
keep_monthly: 12

checks:
  - name: spot
    count_tolerance_percentage: 10
    data_sample_percentage: 0.01
    data_tolerance_percentage: 0.005

skip_actions:
  - compact

healthchecks:
  ping_url: "https://hc-ping.com/redacted"
  states:
    - finish

Actual behavior

Here's the output from the command:

ssh://redacted: Error running actions for repository
ssh://redacted: Command 'borg create --patterns-from /run/user/1000/borgmatic/tmp4_mkexlh --checkpoint-interval 600 --compression auto,zstd --info --progress ssh://redacted::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}' returned non-zero exit status 2.
/home/raxod502/.config/borgmatic.d/redacted.yaml: Skipping Healthchecks log ping due to configured states
/home/raxod502/.config/borgmatic.d/redacted.yaml: Pinging Healthchecks finish
/home/raxod502/.config/borgmatic.d/redacted.yaml: An error occurred

summary:
An error occurred
Error running actions for repository
Command 'borg create --patterns-from /run/user/1000/borgmatic/tmp4_mkexlh --checkpoint-interval 600 --compression auto,zstd --info --progress ssh://redacted::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}' returned non-zero exit status 2.

The real smoking gun for me is this though:

image.png

Why is Borgmatic sending an OK ping with a fatal error message in the body?

Expected behavior

I want an OK ping to be sent when there are no errors. Otherwise, I want nothing to be sent, or at the very least, a failure ping should be sent instead of an OK ping - the behavior of sending an OK ping unconditionally effectively means that alerting is silently disabled.

Other notes / implementation ideas

No response

borgmatic version

2.0.2

borgmatic installation method

pip install

Borg version

1.2.8

Python version

3.12.3

Database version (if applicable)

No response

Operating system and version

Ubuntu 24.04.2 LTS

### What I'm trying to do and why I'm using the configuration in https://projects.torsion.org/borgmatic-collective/borgmatic/issues/525 to only have a ping sent to Healthchecks.io when my backup operations have succeeded, for similar reasons as the ones described in that issue (want to only be alerted if backups are persistently down, not just having a transient error). ### Steps to reproduce ```yaml source_directories: - "/redacted" repositories: - path: "ssh://redacted" exclude_patterns: - "*/.snapshots" compression: auto,zstd encryption_passphrase: "redacted" ssh_command: "ssh -oIdentitiesOnly=yes -oControlPath=none -i ~/.ssh/redacted" checkpoint_interval: 600 keep_within: 2w keep_daily: 30 keep_weekly: 8 keep_monthly: 12 checks: - name: spot count_tolerance_percentage: 10 data_sample_percentage: 0.01 data_tolerance_percentage: 0.005 skip_actions: - compact healthchecks: ping_url: "https://hc-ping.com/redacted" states: - finish ``` ### Actual behavior Here's the output from the command: ``` ssh://redacted: Error running actions for repository ssh://redacted: Command 'borg create --patterns-from /run/user/1000/borgmatic/tmp4_mkexlh --checkpoint-interval 600 --compression auto,zstd --info --progress ssh://redacted::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}' returned non-zero exit status 2. /home/raxod502/.config/borgmatic.d/redacted.yaml: Skipping Healthchecks log ping due to configured states /home/raxod502/.config/borgmatic.d/redacted.yaml: Pinging Healthchecks finish /home/raxod502/.config/borgmatic.d/redacted.yaml: An error occurred summary: An error occurred Error running actions for repository Command 'borg create --patterns-from /run/user/1000/borgmatic/tmp4_mkexlh --checkpoint-interval 600 --compression auto,zstd --info --progress ssh://redacted::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}' returned non-zero exit status 2. ``` The real smoking gun for me is this though: <img width="632" alt="image.png" src="attachments/00f709cc-59ee-4569-8858-931c72d80435"> Why is Borgmatic sending an OK ping with a fatal error message in the body? ### Expected behavior I want an OK ping to be sent when there are no errors. Otherwise, I want nothing to be sent, or **at the very least**, a failure ping should be sent instead of an OK ping - the behavior of sending an OK ping unconditionally effectively means that alerting is silently disabled. ### Other notes / implementation ideas _No response_ ### borgmatic version 2.0.2 ### borgmatic installation method pip install ### Borg version 1.2.8 ### Python version 3.12.3 ### Database version (if applicable) _No response_ ### Operating system and version Ubuntu 24.04.2 LTS
image.png
94 KiB
raxod502 commented 2025-04-09 17:00:49 +00:00
Author
Copy link

Looking at the code here and here it looks like my expected behavior is the intention of the code, but for some reason the error visible in the Borgmatic logs is not getting passed to Monitoring_hooks.__exit__ and thus Borgmatic does not realize there is an error here.

Looking at the code [here](https://projects.torsion.org/borgmatic-collective/borgmatic/src/commit/0b4aff9277eba86c53098121583c6190cad96efb/borgmatic/commands/borgmatic.py#L157-L165) and [here](https://projects.torsion.org/borgmatic-collective/borgmatic/src/commit/0b4aff9277eba86c53098121583c6190cad96efb/borgmatic/hooks/monitoring/healthchecks.py#L57-L59) it looks like my expected behavior is the intention of the code, but for some reason the error visible in the Borgmatic logs is not getting passed to `Monitoring_hooks.__exit__` and thus Borgmatic does not realize there is an error here.
raxod502 commented 2025-04-09 17:04:30 +00:00
Author
Copy link

Okay, the bug is easy to understand: the exception is caught by this code fully inside of the context manager, thus the context manager doesn't get the exception passed to it. Should be a simple refactor to fix. Also maybe worth adding a test to ensure this doesn't regress in the future? Having the backup monitoring silently disabled is a really scary failure condition.

Okay, the bug is easy to understand: the exception is caught [by this code](https://projects.torsion.org/borgmatic-collective/borgmatic/src/commit/0b4aff9277eba86c53098121583c6190cad96efb/borgmatic/commands/borgmatic.py#L256-L280) fully inside of [the context manager](https://projects.torsion.org/borgmatic-collective/borgmatic/src/commit/0b4aff9277eba86c53098121583c6190cad96efb/borgmatic/commands/borgmatic.py#L211), thus the context manager doesn't get the exception passed to it. Should be a simple refactor to fix. Also maybe worth adding a test to ensure this doesn't regress in the future? Having the backup monitoring silently disabled is a really scary failure condition.
raxod502 changed title from Borgmatic sends Healthchecks.io ping even when there was an error to Borgmatic sends Healthchecks.io OK ping even when there was an error 2025-04-09 17:05:33 +00:00
witten commented 2025-04-09 17:09:03 +00:00
Owner
Copy link

Thanks for filing this! I think I might see the problem in code: There is logic that sends a failure ping instead of a OK ping if an error is encountered—but the error is getting swallowed by some other code before it ever makes its way to that logic. I'm not sure of the solution if that's the case, but first I'll start by seeing if I can repro the behavior you're seeing.

Thanks for filing this! I think I might see the problem in code: There _is_ logic that sends a failure ping instead of a OK ping if an error is encountered—but the error is getting swallowed by some other code before it ever makes its way to that logic. I'm not sure of the solution if that's the case, but first I'll start by seeing if I can repro the behavior you're seeing.
witten commented 2025-04-09 17:11:45 +00:00
Owner
Copy link

Oh, hah, just saw your comments. Look like you figured it out first! And yes, I agree that: 1. A refactor might fix this, and 2. Absolutely a regression test would be a good idea.

Oh, hah, just saw your comments. Look like you figured it out first! And yes, I agree that: 1. A refactor might fix this, and 2. Absolutely a regression test would be a good idea.
❤️ 1
witten commented 2025-04-09 18:35:30 +00:00
Owner
Copy link

This is fixed in main and will be part of the next release. The solution I went with was re-raising the exception after all repositories had a chance to run, so that one repository erroring doesn't interfere with other repositories running. And there's a regression test as well.

Thanks again!

This is fixed in main and will be part of the next release. The solution I went with was re-raising the exception after all repositories had a chance to run, so that one repository erroring doesn't interfere with other repositories running. And there's a regression test as well. Thanks again!
❤️ 1
witten commented 2025-04-15 06:02:24 +00:00
Owner
Copy link

Released in borgmatic 2.0.3!

Released in borgmatic 2.0.3!
❤️ 1
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
parallel-borgmatic
command-hook-steps
2.1.7
2.1.6
2.1.5
2.1.4
2.1.3
2.1.2
2.1.1
2.1.0
2.0.13
2.0.12
2.0.11
2.0.10
2.0.9
2.0.8
2.0.7
2.0.6
2.0.5
2.0.4
2.0.3
2.0.2
2.0.1
2.0.0
1.9.14
1.9.13
1.9.12
1.9.11
1.9.10
1.9.9
1.9.8
1.9.7
1.9.6
1.9.5
1.9.4
1.9.3
1.9.2
1.9.1
1.9.0
1.8.14
1.8.13
1.8.12
1.8.11
1.8.10
1.8.9
1.8.8
1.8.7
1.8.6
1.8.5
1.8.4
1.8.3
1.8.2
1.8.1
1.8.0
1.7.15
1.7.14
1.7.13
1.7.12
1.7.11
1.7.10
1.7.9
1.7.8
1.7.7
1.7.6
1.7.5
1.7.4
1.7.3
1.7.2
1.7.1
1.7.0
1.6.6
1.6.5
1.6.4
1.6.3
1.6.2
1.6.1
1.6.0
1.5.24
1.5.23
1.5.22
1.5.21
1.5.20
1.5.19
1.5.18
1.5.17
1.5.16
1.5.15
1.5.14
1.5.13
1.5.12
1.5.11
1.5.10
1.5.9
1.5.8
1.5.7
1.5.7.dev0
1.5.6
1.5.5
1.5.4
1.5.3
1.5.2
1.5.1
1.5.0
1.4.22
1.4.21
1.4.20
1.4.19
1.4.18
1.4.17
1.4.16
1.4.15
1.4.14
1.4.13
1.4.12
1.4.11
1.4.10
1.4.9
1.4.8
1.4.7
1.4.6
1.4.5
1.4.4
1.4.3
1.4.2
1.4.1
1.4.0
1.3.26
1.3.25
1.3.24
1.3.23
1.3.22
1.3.21
1.3.20
1.3.19
1.3.18
1.3.17
1.3.16
1.3.15
1.3.14
1.3.13
1.3.12
1.3.11
1.3.10
1.3.9
1.3.8
1.3.7
1.3.6
1.3.5
1.3.4
1.3.3
1.3.2
1.3.1
1.3.0
1.2.18
1.2.17
1.2.16
1.2.15
1.2.14
1.2.13
1.2.12
1.2.11
1.2.10
1.2.9
1.2.8
1.2.7
1.2.6
1.2.5
1.2.4
1.2.3
1.2.2
1.2.1
1.2.0
1.1.15
1.1.14
1.1.13
1.1.12
1.1.11
1.1.10
1.1.9
1.1.7
1.1.6
1.1.5
1.1.4
1.1.3
1.1.2
1.1.1
1.1.0
1.0.3
1.0.2
1.0.1
1.0.0
0.1.8
0.1.7
0.1.6
0.1.5
0.1.4
0.1.3
0.1.2
0.1.1
0.1.0
0.0.7
0.0.6
0.0.5
0.0.4
0.0.3
0.0.2
Labels
Clear labels   
blocked

   
breaking

   
bug

   
data loss

   
design finalized

   
good first issue

   
new feature area

   
question / support

   
security

   
waiting for response
No labels
blocked
breaking
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
borgmatic-collective/borgmatic#1065
No description provided.