Multiple calls to healthchecks.io fail with "Max retries exceeded" #377

New Issue

Closed

opened 2020-12-07 12:52:53 +00:00 by GeorgeGedox · 2 comments

GeorgeGedox commented 2020-12-07 12:52:53 +00:00

Copy Link

What I'm trying to do and why

I've got 3 config files on a host backing up in 3 repositories and pinging 3 healthchecks.io endpoints via cron, the cronjob runs hourly and goes trough all configs.

Most of the time the hook runs correctly but after a while it'll start failing for one of the configs or even for all of them with a Max retries exceeded, my theory is due to a lack of delayed tries in the integration, it should try at least 5 times to ping the endpoints with increased delay so it avoids concurrent connections.

Steps to reproduce (if a bug)

Ping multiple healtchecks.io endpoints in a short amount of time

Actual behavior (if a bug)

[2020-12-07 13:07:28,602] INFO: Archive consistency check complete, no problems found.
[2020-12-07 13:07:28,603] INFO: /etc/borgmatic.d/nextcloud.yaml: Pinging Healthchecks finish
[2020-12-07 13:07:28,825] CRITICAL:
[2020-12-07 13:07:28,825] CRITICAL: summary:
[2020-12-07 13:00:16,977] CRITICAL: /etc/borgmatic.d/containers.yaml: Error running configuration file
[2020-12-07 13:00:11,815] CRITICAL: /etc/borgmatic.d/containers.yaml: Error running pre hook
[2020-12-07 13:00:11,815] CRITICAL: HTTPSConnectionPool(host='hc-ping.com', port=443): Max retries exceeded with url: /<endpoint>/start (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f7b27b1c7f0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))

Expected behavior (if a bug)

Hook should retry a few times to ping an endpoint if it fails to avoid false-positives due to concurent connections

Other notes / implementation ideas

Integration should try at least 5 times to ping the endpoints with a delay of +1 second for each failed ping.

Environment

borgmatic version: 1.5.12

borgmatic installation method: pip

Borg version: 1.11.1

Python version: 3.8.5

operating system and version: Ubuntu 20.04

#### What I'm trying to do and why I've got 3 config files on a host backing up in 3 repositories and pinging 3 healthchecks.io endpoints via cron, the cronjob runs hourly and goes trough all configs. Most of the time the hook runs correctly but after a while it'll start failing for one of the configs or even for all of them with a `Max retries exceeded`, my theory is due to a lack of delayed tries in the integration, it should try at least 5 times to ping the endpoints with increased delay so it avoids concurrent connections. #### Steps to reproduce (if a bug) Ping multiple healtchecks.io endpoints in a short amount of time #### Actual behavior (if a bug) ``` [2020-12-07 13:07:28,602] INFO: Archive consistency check complete, no problems found. [2020-12-07 13:07:28,603] INFO: /etc/borgmatic.d/nextcloud.yaml: Pinging Healthchecks finish [2020-12-07 13:07:28,825] CRITICAL: [2020-12-07 13:07:28,825] CRITICAL: summary: [2020-12-07 13:00:16,977] CRITICAL: /etc/borgmatic.d/containers.yaml: Error running configuration file [2020-12-07 13:00:11,815] CRITICAL: /etc/borgmatic.d/containers.yaml: Error running pre hook [2020-12-07 13:00:11,815] CRITICAL: HTTPSConnectionPool(host='hc-ping.com', port=443): Max retries exceeded with url: /<endpoint>/start (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f7b27b1c7f0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')) ``` #### Expected behavior (if a bug) Hook should retry a few times to ping an endpoint if it fails to avoid false-positives due to concurent connections #### Other notes / implementation ideas Integration should try at least 5 times to ping the endpoints with a delay of +1 second for each failed ping. #### Environment **borgmatic version:** 1.5.12 **borgmatic installation method:** pip **Borg version:** 1.11.1 **Python version:** 3.8.5 **operating system and version:** Ubuntu 20.04

GeorgeGedox changed title from Multiple calls to healtchecks fail with "Max retries exceeded" to Multiple calls to healthchecks.io fail with "Max retries exceeded" 2020-12-07 12:53:18 +00:00

witten commented 2022-05-24 23:18:14 +00:00

Owner

Copy Link

I apologize for the lengthy delay here. I just implemented a related ticket (#439) that changes Healthchecks connection failures from errors to mere warnings. That won't do any retries, but it will make occasional Healthchecks connections errors less of a big deal—they'll no longer block any backups from happening. So do you think that's sufficient for your particular use case? Thanks!

I apologize for the lengthy delay here. I just implemented a related ticket (#439) that changes Healthchecks connection failures from errors to mere warnings. That won't do any retries, but it will make occasional Healthchecks connections errors less of a big deal—they'll no longer block any backups from happening. So do you think that's sufficient for your particular use case? Thanks!

witten added the

waiting for response

label 2022-05-24 23:18:35 +00:00

witten commented 2022-10-05 06:11:57 +00:00

Owner

Copy Link

Closing this for now, but feel free to follow up. We can always reopen it. Thanks!

Closing this for now, but feel free to follow up. We can always reopen it. Thanks!

witten closed this issue 2022-10-05 06:11:57 +00:00

witten removed the

waiting for response

label 2022-10-05 06:12:13 +00:00

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

spot-check

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.24

1.5.23

1.5.22

1.5.21

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.14

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.7.dev0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.22

1.4.21

1.4.20

1.4.19

1.4.18

1.4.17

1.4.16

1.4.15

1.4.14

1.4.13

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.26

1.3.25

1.3.24

1.3.23

1.3.22

1.3.21

1.3.20

1.3.19

1.3.18

1.3.17

1.3.16

1.3.15

1.3.14

1.3.13

1.3.12

1.3.11

1.3.10

1.3.9

1.3.8

1.3.7

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.18

1.2.17

1.2.16

1.2.15

1.2.14

1.2.13

1.2.12

1.2.11

1.2.10

1.2.9

1.2.8

1.2.7

1.2.6

1.2.5

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.15

1.1.14

1.1.13

1.1.12

1.1.11

1.1.10

1.1.9

1.1.7

1.1.6

1.1.5

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.3

1.0.2

1.0.1

1.0.0

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

0.0.2

Labels

Clear labels   

bug

  

data loss

  

design finalized

  

good first issue

  

new feature area

  

question / support

  

security

  

waiting for response

No Label

bug

data loss

design finalized

good first issue

new feature area

question / support

security

waiting for response

Milestone

Clear milestone

No items

No Milestone

Assignees

Clear assignees

No Assignees

2 Participants

Notifications

Subscribe

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#377

No description provided.