borgmatic-collective/borgmatic
borgmatic-collective/borgmatic
Watch
9
74
Fork
You've already forked borgmatic
 57
Code Issues 71 Pull requests 3 Releases 162 Packages 1 Activity Actions 1

Healthchecks: do not ping "[successful] finish" state after warning/error occured #888

New issue
Closed
opened 2024-06-25 16:16:20 +00:00 by chrisblech · 3 comments
chrisblech commented 2024-06-25 16:16:20 +00:00
Copy link

What I'm trying to do and why

I'm using healthchecks.io (and its status-badge PNG) to monitor the success of my backups.

When where is an error/warning during backup, borgmatic pings the healthchecks "fail" endpoint. This is correct: healthchecks sends an alert, and its status-badge turns to red. This is exactly how it should work.

Some seconds later, when the backup task completes, borgmatic pings healthchecks again, now to the "finish" endpoint. This makes healthchecks think the failure-state has recovered: it sends an "recovered" alert, and the status-badge PNG turns back to green.

Steps to reproduce

location:
    source_directories:
        - /var/discourse
        - /var/backup-docker-volumes
    repositories:
        - ssh://[redacted].your-storagebox.de:23/./borg
storage:
    encryption_passphrase: [redacted]
retention:
    keep_daily: 7
    keep_weekly: 4
    keep_monthly: 12
consistency:
    checks:
        - name: data
hooks:
    before_everything:
        - backup-docker-volumes.sh all
    healthchecks:
        ping_url: https://hc.hosrv.de/ping/[redacted]
        ping_body_limit: 0

Actual behavior

Ping #305 (received via the /fail endpoint)
2024-06-20T02:20:52.294315+00:00:

ssh://[redacted].your-storagebox.de:23/./borg: Error running actions for repository
Command 'borg prune --keep-daily 7 --keep-weekly 4 --keep-monthly 12 --glob-archives {hostname}-* ssh://[redacted].your-storagebox.de:23/./borg' returned non-zero exit status 2

Ping #306: OK
2024-06-20T02:53:35.110450+00:00

Expected behavior

After an error occured (and healthchecks "fail" endpoint was pinged), bormatic should not ping healthchecks after the backup finished, or include status code in finish endpoint (see https://healthchecks.io/docs/signaling_failures/ )

Other notes / implementation ideas

No response

borgmatic version

1.7.7

borgmatic installation method

Debian package

Borg version

borg 1.2.4

Python version

Python 3.11.2

Database version (if applicable)

No response

Operating system and version

Debian GNU/Linux 12 (bookworm)

### What I'm trying to do and why I'm using healthchecks.io (and its status-badge PNG) to monitor the success of my backups. When where is an error/warning during backup, borgmatic pings the healthchecks "fail" endpoint. This is correct: healthchecks sends an alert, and its status-badge turns to red. This is exactly how it should work. Some seconds later, when the backup task completes, borgmatic pings healthchecks again, now to the "finish" endpoint. This makes healthchecks think the failure-state has recovered: it sends an "recovered" alert, and the status-badge PNG turns back to green. ### Steps to reproduce ``` location: source_directories: - /var/discourse - /var/backup-docker-volumes repositories: - ssh://[redacted].your-storagebox.de:23/./borg storage: encryption_passphrase: [redacted] retention: keep_daily: 7 keep_weekly: 4 keep_monthly: 12 consistency: checks: - name: data hooks: before_everything: - backup-docker-volumes.sh all healthchecks: ping_url: https://hc.hosrv.de/ping/[redacted] ping_body_limit: 0 ``` ### Actual behavior Ping #305 (received via the /fail endpoint) 2024-06-20T02:20:52.294315+00:00: ssh://[redacted].your-storagebox.de:23/./borg: Error running actions for repository Command 'borg prune --keep-daily 7 --keep-weekly 4 --keep-monthly 12 --glob-archives {hostname}-* ssh://[redacted].your-storagebox.de:23/./borg' returned non-zero exit status 2 Ping #306: OK 2024-06-20T02:53:35.110450+00:00 ### Expected behavior After an error occured (and healthchecks "fail" endpoint was pinged), bormatic should not ping healthchecks after the backup finished, or include status code in finish endpoint (see https://healthchecks.io/docs/signaling_failures/ ) ### Other notes / implementation ideas _No response_ ### borgmatic version 1.7.7 ### borgmatic installation method Debian package ### Borg version borg 1.2.4 ### Python version Python 3.11.2 ### Database version (if applicable) _No response_ ### Operating system and version Debian GNU/Linux 12 (bookworm)
witten commented 2024-06-25 17:25:06 +00:00
Owner
Copy link

Interesting... I've looked at the code in question from borgmatic 1.7.7, and the behavior you're seeing shouldn't be possible. However there is a theoretical code path where the following related scenario can happen:

  1. borgmatic finishes normally and pings the Healthchecks "finish" endpoint.
  2. borgmatic encounters an error in the act of pinging that "finish" endpoint. (Network error, etc.)
  3. borgmatic pings the Healthchecks "fail" endpoint.

(Note that this same scenario does not work the same way in newer versions of borgmatic.)

So are you sure about the "fail" then "finish" ordering?

Also can I get a look at your (redacted) borgmatic logs? That might help pinpoint what's going on. Thanks!

Interesting... I've looked at the code in question from borgmatic 1.7.7, and the behavior you're seeing _shouldn't_ be possible. However there is a theoretical code path where the following related scenario can happen: 1. borgmatic finishes normally and pings the Healthchecks "finish" endpoint. 2. borgmatic encounters an error in the act of pinging that "finish" endpoint. (Network error, etc.) 3. borgmatic pings the Healthchecks "fail" endpoint. (Note that this same scenario does not work the same way in newer versions of borgmatic.) So are you sure about the "fail" _then_ "finish" ordering? Also can I get a look at your (redacted) borgmatic logs? That might help pinpoint what's going on. Thanks!
chrisblech commented 2024-06-27 13:37:34 +00:00
Author
Copy link

While collecting logs (see attachment), I realized that borgmatic seemed to start twice on this host: once via /etc/crontab and once via systemd.timer.

This should explain the cause of the error (could not aquire exclusive lock) and also the order of the healthchecks ping events.

So I deleted the crontab-line and close this issue.

While collecting logs (see attachment), I realized that borgmatic seemed to start twice on this host: once via `/etc/crontab` and once via systemd.timer. This should explain the cause of the error (could not aquire exclusive lock) and also the order of the healthchecks ping events. So I deleted the crontab-line and close this issue.
all-logs.txt
5.7 KiB
witten commented 2024-06-27 16:36:21 +00:00
Owner
Copy link

Wow, good sleuthing! I'm glad to hear that resolved it.

Wow, good sleuthing! I'm glad to hear that resolved it.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
parallel-borgmatic
command-hook-steps
2.1.7
2.1.6
2.1.5
2.1.4
2.1.3
2.1.2
2.1.1
2.1.0
2.0.13
2.0.12
2.0.11
2.0.10
2.0.9
2.0.8
2.0.7
2.0.6
2.0.5
2.0.4
2.0.3
2.0.2
2.0.1
2.0.0
1.9.14
1.9.13
1.9.12
1.9.11
1.9.10
1.9.9
1.9.8
1.9.7
1.9.6
1.9.5
1.9.4
1.9.3
1.9.2
1.9.1
1.9.0
1.8.14
1.8.13
1.8.12
1.8.11
1.8.10
1.8.9
1.8.8
1.8.7
1.8.6
1.8.5
1.8.4
1.8.3
1.8.2
1.8.1
1.8.0
1.7.15
1.7.14
1.7.13
1.7.12
1.7.11
1.7.10
1.7.9
1.7.8
1.7.7
1.7.6
1.7.5
1.7.4
1.7.3
1.7.2
1.7.1
1.7.0
1.6.6
1.6.5
1.6.4
1.6.3
1.6.2
1.6.1
1.6.0
1.5.24
1.5.23
1.5.22
1.5.21
1.5.20
1.5.19
1.5.18
1.5.17
1.5.16
1.5.15
1.5.14
1.5.13
1.5.12
1.5.11
1.5.10
1.5.9
1.5.8
1.5.7
1.5.7.dev0
1.5.6
1.5.5
1.5.4
1.5.3
1.5.2
1.5.1
1.5.0
1.4.22
1.4.21
1.4.20
1.4.19
1.4.18
1.4.17
1.4.16
1.4.15
1.4.14
1.4.13
1.4.12
1.4.11
1.4.10
1.4.9
1.4.8
1.4.7
1.4.6
1.4.5
1.4.4
1.4.3
1.4.2
1.4.1
1.4.0
1.3.26
1.3.25
1.3.24
1.3.23
1.3.22
1.3.21
1.3.20
1.3.19
1.3.18
1.3.17
1.3.16
1.3.15
1.3.14
1.3.13
1.3.12
1.3.11
1.3.10
1.3.9
1.3.8
1.3.7
1.3.6
1.3.5
1.3.4
1.3.3
1.3.2
1.3.1
1.3.0
1.2.18
1.2.17
1.2.16
1.2.15
1.2.14
1.2.13
1.2.12
1.2.11
1.2.10
1.2.9
1.2.8
1.2.7
1.2.6
1.2.5
1.2.4
1.2.3
1.2.2
1.2.1
1.2.0
1.1.15
1.1.14
1.1.13
1.1.12
1.1.11
1.1.10
1.1.9
1.1.7
1.1.6
1.1.5
1.1.4
1.1.3
1.1.2
1.1.1
1.1.0
1.0.3
1.0.2
1.0.1
1.0.0
0.1.8
0.1.7
0.1.6
0.1.5
0.1.4
0.1.3
0.1.2
0.1.1
0.1.0
0.0.7
0.0.6
0.0.5
0.0.4
0.0.3
0.0.2
Labels
Clear labels   
blocked

   
breaking

   
bug

   
data loss

   
design finalized

   
good first issue

   
new feature area

   
question / support

   
security

   
waiting for response
No labels
blocked
breaking
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
borgmatic-collective/borgmatic#888
No description provided.