borgmatic-collective/borgmatic
borgmatic-collective
/
borgmatic
15
56
Fork 39
Code Issues 52 Pull Requests 1 Actions Packages Releases 121 Activity

healthchecks hook failure blocks backup #439

New Issue
Closed
opened 2021-08-05 09:30:23 +00:00 by BastienDurel · 6 comments
BastienDurel commented 2021-08-05 09:30:23 +00:00
Copy Link

What I'm trying to do and why

I've included an healthchecks hook to monitor my backups.
For some reason, healthchecks.io does not respond to HTTP from my IP since yesterday
As borgmatic calls the /start endpoint on start, the whole backup fails

Steps to reproduce (if a bug)

Have a borgmatic backup running with healthchecks hook

hooks:
    healthchecks: https://hc-ping.com/UUID

Block access to hc-ping.com via your firewall (DROP)

Actual behavior (if a bug)

The backup fails at pre hook phase

août 04 00:52:24 corrin systemd[1]: Starting borgmatic backup...
août 04 00:57:25 corrin borgmatic[31259]: INFO /etc/borgmatic/config.yaml: Pinging Healthchecks start
août 04 00:59:27 corrin systemd-inhibit[31258]: /etc/borgmatic/config.yaml: Error running pre hook
août 04 00:59:27 corrin borgmatic[31259]: CRITICAL /etc/borgmatic/config.yaml: Error running pre hook
août 04 00:59:27 corrin systemd-inhibit[31258]: HTTPSConnectionPool(host='hc-ping.com', port=443): Max retries exceeded with url: /UUID/start (Caused by SSLError(SSLError("bad handshake: SysCallError(104, '
août 04 00:59:27 corrin borgmatic[31259]: CRITICAL HTTPSConnectionPool(host='hc-ping.com', port=443): Max retries exceeded with url: /UUID/start (Caused by SSLError(SSLError("bad handshake: SysCallError(104
août 04 00:59:27 corrin borgmatic[31259]: INFO /etc/borgmatic/config.yaml: Running 2 commands for on-error hook
août 04 00:59:27 corrin systemd-inhibit[31258]: Wed 04 Aug 2021 12:59:27 AM CEST - Error while creating a backup.
août 04 00:59:27 corrin borgmatic[31259]: ERROR Wed 04 Aug 2021 12:59:27 AM CEST - Error while creating a backup.
août 04 00:59:27 corrin borgmatic[31259]: INFO /etc/borgmatic/config.yaml: Pinging Healthchecks fail
août 04 01:03:25 corrin systemd-inhibit[31258]: /etc/borgmatic/config.yaml: Error running on-error hook
août 04 01:03:25 corrin borgmatic[31259]: CRITICAL /etc/borgmatic/config.yaml: Error running on-error hook
août 04 01:03:25 corrin systemd-inhibit[31258]: HTTPSConnectionPool(host='hc-ping.com', port=443): Max retries exceeded with url: /UUID/fail (Caused by SSLError(SSLError("bad handshake: SysCallError(104, 'E
août 04 01:03:25 corrin borgmatic[31259]: CRITICAL HTTPSConnectionPool(host='hc-ping.com', port=443): Max retries exceeded with url: /UUID/fail (Caused by SSLError(SSLError("bad handshake: SysCallError(104,
août 04 01:03:25 corrin systemd-inhibit[31258]: /etc/borgmatic/config.yaml: Error running configuration file
août 04 01:03:25 corrin borgmatic[31259]: CRITICAL /etc/borgmatic/config.yaml: Error running configuration file
août 04 01:03:25 corrin borgmatic[31259]: CRITICAL
août 04 01:03:25 corrin systemd-inhibit[31258]: summary:
août 04 01:03:25 corrin borgmatic[31259]: CRITICAL summary:
août 04 01:03:25 corrin systemd-inhibit[31258]: /etc/borgmatic/config.yaml: Error running configuration file
août 04 01:03:25 corrin borgmatic[31259]: CRITICAL /etc/borgmatic/config.yaml: Error running configuration file
août 04 01:03:25 corrin systemd-inhibit[31258]: /etc/borgmatic/config.yaml: Error running pre hook
août 04 01:03:25 corrin borgmatic[31259]: CRITICAL /etc/borgmatic/config.yaml: Error running pre hook
août 04 01:03:25 corrin systemd-inhibit[31258]: HTTPSConnectionPool(host='hc-ping.com', port=443): Max retries exceeded with url: /UUID/start (Caused by SSLError(SSLError("bad handshake: SysCallError(104, '
août 04 01:03:25 corrin borgmatic[31259]: CRITICAL HTTPSConnectionPool(host='hc-ping.com', port=443): Max retries exceeded with url: /UUID/start (Caused by SSLError(SSLError("bad handshake: SysCallError(104
août 04 01:03:25 corrin systemd-inhibit[31258]: /etc/borgmatic/config.yaml: Error running on-error hook
août 04 01:03:25 corrin borgmatic[31259]: CRITICAL /etc/borgmatic/config.yaml: Error running on-error hook
août 04 01:03:25 corrin systemd-inhibit[31258]: HTTPSConnectionPool(host='hc-ping.com', port=443): Max retries exceeded with url: /UUID/fail (Caused by SSLError(SSLError("bad handshake: SysCallError(104, 'E
août 04 01:03:25 corrin borgmatic[31259]: CRITICAL HTTPSConnectionPool(host='hc-ping.com', port=443): Max retries exceeded with url: /UUID/fail (Caused by SSLError(SSLError("bad handshake: SysCallError(104,
août 04 01:03:25 corrin borgmatic[31259]: CRITICAL
août 04 01:03:25 corrin systemd-inhibit[31258]: Need some help? https://torsion.org/borgmatic/#issues
août 04 01:03:25 corrin borgmatic[31259]: CRITICAL Need some help? https://torsion.org/borgmatic/#issues
août 04 01:03:25 corrin systemd-inhibit[31258]: /usr/bin/borgmatic failed with exit status 1.
août 04 01:03:25 corrin systemd[1]: borgmatic.service: Main process exited, code=exited, status=1/FAILURE
août 04 01:03:25 corrin systemd[1]: borgmatic.service: Failed with result 'exit-code'.

Other notes / implementation ideas

Backup shoud be done even if healthchecks hook fails. (ignore failure on this hook); it's better to have a backup done reported missing by monitoring than no backup at all (with clear log about reporting failure)

Environment

borgmatic version:

1.5.1

borgmatic installation method:

Debian package

Borg version: [version here]

1.1.9

Python version: [version here]

3.7.3

operating system and version: [OS here]

Debian 10

#### What I'm trying to do and why I've included an `healthchecks` hook to monitor my backups. For some reason, healthchecks.io does not respond to HTTP from my IP since yesterday As borgmatic calls the /start endpoint on start, the whole backup fails #### Steps to reproduce (if a bug) Have a borgmatic backup running with healthchecks hook ``` hooks: healthchecks: https://hc-ping.com/UUID ``` Block access to hc-ping.com via your firewall (DROP) #### Actual behavior (if a bug) The backup fails at pre hook phase ``` août 04 00:52:24 corrin systemd[1]: Starting borgmatic backup... août 04 00:57:25 corrin borgmatic[31259]: INFO /etc/borgmatic/config.yaml: Pinging Healthchecks start août 04 00:59:27 corrin systemd-inhibit[31258]: /etc/borgmatic/config.yaml: Error running pre hook août 04 00:59:27 corrin borgmatic[31259]: CRITICAL /etc/borgmatic/config.yaml: Error running pre hook août 04 00:59:27 corrin systemd-inhibit[31258]: HTTPSConnectionPool(host='hc-ping.com', port=443): Max retries exceeded with url: /UUID/start (Caused by SSLError(SSLError("bad handshake: SysCallError(104, ' août 04 00:59:27 corrin borgmatic[31259]: CRITICAL HTTPSConnectionPool(host='hc-ping.com', port=443): Max retries exceeded with url: /UUID/start (Caused by SSLError(SSLError("bad handshake: SysCallError(104 août 04 00:59:27 corrin borgmatic[31259]: INFO /etc/borgmatic/config.yaml: Running 2 commands for on-error hook août 04 00:59:27 corrin systemd-inhibit[31258]: Wed 04 Aug 2021 12:59:27 AM CEST - Error while creating a backup. août 04 00:59:27 corrin borgmatic[31259]: ERROR Wed 04 Aug 2021 12:59:27 AM CEST - Error while creating a backup. août 04 00:59:27 corrin borgmatic[31259]: INFO /etc/borgmatic/config.yaml: Pinging Healthchecks fail août 04 01:03:25 corrin systemd-inhibit[31258]: /etc/borgmatic/config.yaml: Error running on-error hook août 04 01:03:25 corrin borgmatic[31259]: CRITICAL /etc/borgmatic/config.yaml: Error running on-error hook août 04 01:03:25 corrin systemd-inhibit[31258]: HTTPSConnectionPool(host='hc-ping.com', port=443): Max retries exceeded with url: /UUID/fail (Caused by SSLError(SSLError("bad handshake: SysCallError(104, 'E août 04 01:03:25 corrin borgmatic[31259]: CRITICAL HTTPSConnectionPool(host='hc-ping.com', port=443): Max retries exceeded with url: /UUID/fail (Caused by SSLError(SSLError("bad handshake: SysCallError(104, août 04 01:03:25 corrin systemd-inhibit[31258]: /etc/borgmatic/config.yaml: Error running configuration file août 04 01:03:25 corrin borgmatic[31259]: CRITICAL /etc/borgmatic/config.yaml: Error running configuration file août 04 01:03:25 corrin borgmatic[31259]: CRITICAL août 04 01:03:25 corrin systemd-inhibit[31258]: summary: août 04 01:03:25 corrin borgmatic[31259]: CRITICAL summary: août 04 01:03:25 corrin systemd-inhibit[31258]: /etc/borgmatic/config.yaml: Error running configuration file août 04 01:03:25 corrin borgmatic[31259]: CRITICAL /etc/borgmatic/config.yaml: Error running configuration file août 04 01:03:25 corrin systemd-inhibit[31258]: /etc/borgmatic/config.yaml: Error running pre hook août 04 01:03:25 corrin borgmatic[31259]: CRITICAL /etc/borgmatic/config.yaml: Error running pre hook août 04 01:03:25 corrin systemd-inhibit[31258]: HTTPSConnectionPool(host='hc-ping.com', port=443): Max retries exceeded with url: /UUID/start (Caused by SSLError(SSLError("bad handshake: SysCallError(104, ' août 04 01:03:25 corrin borgmatic[31259]: CRITICAL HTTPSConnectionPool(host='hc-ping.com', port=443): Max retries exceeded with url: /UUID/start (Caused by SSLError(SSLError("bad handshake: SysCallError(104 août 04 01:03:25 corrin systemd-inhibit[31258]: /etc/borgmatic/config.yaml: Error running on-error hook août 04 01:03:25 corrin borgmatic[31259]: CRITICAL /etc/borgmatic/config.yaml: Error running on-error hook août 04 01:03:25 corrin systemd-inhibit[31258]: HTTPSConnectionPool(host='hc-ping.com', port=443): Max retries exceeded with url: /UUID/fail (Caused by SSLError(SSLError("bad handshake: SysCallError(104, 'E août 04 01:03:25 corrin borgmatic[31259]: CRITICAL HTTPSConnectionPool(host='hc-ping.com', port=443): Max retries exceeded with url: /UUID/fail (Caused by SSLError(SSLError("bad handshake: SysCallError(104, août 04 01:03:25 corrin borgmatic[31259]: CRITICAL août 04 01:03:25 corrin systemd-inhibit[31258]: Need some help? https://torsion.org/borgmatic/#issues août 04 01:03:25 corrin borgmatic[31259]: CRITICAL Need some help? https://torsion.org/borgmatic/#issues août 04 01:03:25 corrin systemd-inhibit[31258]: /usr/bin/borgmatic failed with exit status 1. août 04 01:03:25 corrin systemd[1]: borgmatic.service: Main process exited, code=exited, status=1/FAILURE août 04 01:03:25 corrin systemd[1]: borgmatic.service: Failed with result 'exit-code'. ``` #### Other notes / implementation ideas Backup shoud be done even if healthchecks hook fails. (ignore failure on this hook); it's better to have a backup done reported missing by monitoring than no backup at all (with clear log about reporting failure) #### Environment **borgmatic version:** 1.5.1 **borgmatic installation method:** Debian package **Borg version:** [version here] 1.1.9 **Python version:** [version here] 3.7.3 **operating system and version:** [OS here] Debian 10
witten commented 2021-08-05 16:24:35 +00:00
Owner
Copy Link

This behavior is actually by design! The thinking is that it could be really bad if your backups suddenly stop being monitored when you expect them to be, because then you could go weeks or months with broken backups. So given a sudden lack of monitoring, the only way to communicate that issue to the user is by failing the backup as a whole. That way, they are immediately made aware that their backups are unmonitored and can hopefully rectify the issue.

Let me know your thoughts on this and whether you have any other ideas. And thanks for taking the time to file ticket!

This behavior is actually by design! The thinking is that it could be really bad if your backups suddenly stop being monitored when you expect them to be, because then you could go weeks or months with broken backups. So given a sudden lack of monitoring, the only way to communicate that issue to the user is by failing the backup as a whole. That way, they are immediately made aware that their backups are unmonitored and can hopefully rectify the issue. Let me know your thoughts on this and whether you have any other ideas. And thanks for taking the time to file ticket!
BastienDurel commented 2021-08-06 12:33:31 +00:00
Author
Copy Link

If the connection to the minitoring system fails, the monitoring should alert me it wasn't contacted. So I will be notified of the problem, backup done or not.
Or the monitoring system itself is down, and I won't be notified, backup failed or not.

Therefore I did not see how failling the backup should lead to a better state.

If the connection to the minitoring system fails, the monitoring should alert me it wasn't contacted. So I will be notified of the problem, backup done or not. Or the monitoring system itself is down, and I won't be notified, backup failed or not. Therefore I did not see how failling the backup should lead to a better state.
andreasferber commented 2021-09-28 22:18:53 +00:00
Copy Link

I don't think the reasoning behind failing the backup is sound, at least if you aren't using multiple monitoring services in parallel. If the single monitoring service is down/unreachable so you can't report a backup success, then you most likely can't report a backup failure through the service either. Therefore causing the backup to fail actually won't make you notice the monitoring failure immediately unless you always manually check that backups have been created (but if you do that anyway, why have a monitoring service in the first place?). The only way you'd notice it outside of manual checks is if the monitoring service notifies you after a while that there haven't been any status updates submitted recently, but that would be completely independent of whether you make the backup jobs fail because of monitoring or not.

So what you are actually causing is silently failing backups. In my opinion monitoring functions should never influence the actual work of the system being monitored (other than say causing a logged warning about the failed status submission). Monitoring failures should be detected by the monitoring system itself (eg. through a notification if there hasn't been any status report for x amount of time), not by the monitored system. But if you insist on making the jobs fail at least do it after the new borg archive has been written, so that you still end up with a functional backup archive even if the job is ultimately reported as failed.

I don't think the reasoning behind failing the backup is sound, at least if you aren't using multiple monitoring services in parallel. If the single monitoring service is down/unreachable so you can't report a backup success, then you most likely can't report a backup failure through the service either. Therefore causing the backup to fail actually *won't* make you notice the monitoring failure immediately unless you always manually check that backups have been created (but if you do that anyway, why have a monitoring service in the first place?). The only way you'd notice it outside of manual checks is if the monitoring service notifies you after a while that there haven't been any status updates submitted recently, but that would be completely independent of whether you make the backup jobs fail because of monitoring or not. So what you are actually causing is silently failing backups. In my opinion monitoring functions should never influence the actual work of the system being monitored (other than say causing a logged warning about the failed status submission). Monitoring failures should be detected by the monitoring system itself (eg. through a notification if there hasn't been any status report for x amount of time), not by the monitored system. But if you insist on making the jobs fail at least do it *after* the new borg archive has been written, so that you still end up with a functional backup archive even if the job is ultimately reported as failed.
👍 2
witten commented 2022-05-24 22:51:54 +00:00
Owner
Copy Link

I apologize for the lengthy delay in getting back to this. The feature is implemented now in master—connection errors in monitoring hooks are now treated as warnings instead of errors. This will be part of the next release. Thanks for suggesting this!

I apologize for the lengthy delay in getting back to this. The feature is implemented now in master—connection errors in monitoring hooks are now treated as warnings instead of errors. This will be part of the next release. Thanks for suggesting this!
witten commented 2022-05-24 23:25:53 +00:00
Owner
Copy Link

Released in borgmatic 1.6.1!

Released in borgmatic 1.6.1!
BastienDurel commented 2022-05-25 07:44:40 +00:00
Author
Copy Link

Thanks :)

Thanks :)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
main
spot-check
1.8.10
1.8.9
1.8.8
1.8.7
1.8.6
1.8.5
1.8.4
1.8.3
1.8.2
1.8.1
1.8.0
1.7.15
1.7.14
1.7.13
1.7.12
1.7.11
1.7.10
1.7.9
1.7.8
1.7.7
1.7.6
1.7.5
1.7.4
1.7.3
1.7.2
1.7.1
1.7.0
1.6.6
1.6.5
1.6.4
1.6.3
1.6.2
1.6.1
1.6.0
1.5.24
1.5.23
1.5.22
1.5.21
1.5.20
1.5.19
1.5.18
1.5.17
1.5.16
1.5.15
1.5.14
1.5.13
1.5.12
1.5.11
1.5.10
1.5.9
1.5.8
1.5.7
1.5.7.dev0
1.5.6
1.5.5
1.5.4
1.5.3
1.5.2
1.5.1
1.5.0
1.4.22
1.4.21
1.4.20
1.4.19
1.4.18
1.4.17
1.4.16
1.4.15
1.4.14
1.4.13
1.4.12
1.4.11
1.4.10
1.4.9
1.4.8
1.4.7
1.4.6
1.4.5
1.4.4
1.4.3
1.4.2
1.4.1
1.4.0
1.3.26
1.3.25
1.3.24
1.3.23
1.3.22
1.3.21
1.3.20
1.3.19
1.3.18
1.3.17
1.3.16
1.3.15
1.3.14
1.3.13
1.3.12
1.3.11
1.3.10
1.3.9
1.3.8
1.3.7
1.3.6
1.3.5
1.3.4
1.3.3
1.3.2
1.3.1
1.3.0
1.2.18
1.2.17
1.2.16
1.2.15
1.2.14
1.2.13
1.2.12
1.2.11
1.2.10
1.2.9
1.2.8
1.2.7
1.2.6
1.2.5
1.2.4
1.2.3
1.2.2
1.2.1
1.2.0
1.1.15
1.1.14
1.1.13
1.1.12
1.1.11
1.1.10
1.1.9
1.1.7
1.1.6
1.1.5
1.1.4
1.1.3
1.1.2
1.1.1
1.1.0
1.0.3
1.0.2
1.0.1
1.0.0
0.1.8
0.1.7
0.1.6
0.1.5
0.1.4
0.1.3
0.1.2
0.1.1
0.1.0
0.0.7
0.0.6
0.0.5
0.0.4
0.0.3
0.0.2
Labels
Clear labels   
bug

   
data loss

   
design finalized

   
good first issue

   
new feature area

   
question / support

   
security

   
waiting for response
No Label
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
3 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#439
No description provided.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?