Healthchecks integration for independent operations

lgp171188 commented

2020-11-06 10:18:32 +00:00

What I'm trying to do and why

(This is related to witten/borgmatic#255)

I have 2 cron jobs, one that runs the prune and create operations multiple times a day and a check job that runs once a week at a given time.

I'd like to monitor both using the healthchecks integration.

Steps to reproduce (if a bug)

The healthchecks integration pings the healthchecks URL for both these cron jobs but it is impossible to set a schedule on a single healthchecks monitor that covers the expected starting times of both these cron jobs.

If I set the schedule of the monitor to match the multiple-times-a-day cron job, pings from the check job will be extraneous and will not trigger alerts when missed and vice-versa if I use the schedule of the 2nd cron job.

While I could stop using the integration and use manual curl commands to make it work, I then lose the benefits of the integration which provides more than just pinging URLs.

Actual behavior (if a bug)

See the previous section

Expected behavior (if a bug)

TBD as this is a feature request but it will be helpful to have a way to override/specify the healthchecks URL for specific actions.

Other notes / implementation ideas

Environment

borgmatic version: 1.5.10

borgmatic installation method: Installed in a virtualenv environment from PyPI.

Borg version: 1.1.14

Python version: 3.7.3

operating system and version: Debian 10 Buster

#### What I'm trying to do and why (This is related to https://projects.torsion.org/witten/borgmatic/issues/255) I have 2 cron jobs, one that runs the `prune` and `create` operations multiple times a day and a `check` job that runs once a week at a given time. I'd like to monitor both using the healthchecks integration. #### Steps to reproduce (if a bug) The healthchecks integration pings the healthchecks URL for both these cron jobs but it is impossible to set a schedule on a single healthchecks monitor that covers the expected starting times of both these cron jobs. If I set the schedule of the monitor to match the multiple-times-a-day cron job, pings from the `check` job will be extraneous and will not trigger alerts when missed and vice-versa if I use the schedule of the 2nd cron job. While I could stop using the integration and use manual `curl` commands to make it work, I then lose the benefits of the integration which provides more than just pinging URLs. #### Actual behavior (if a bug) See the previous section #### Expected behavior (if a bug) TBD as this is a feature request but it will be helpful to have a way to override/specify the healthchecks URL for specific actions. #### Other notes / implementation ideas #### Environment **borgmatic version:** 1.5.10 **borgmatic installation method:** Installed in a `virtualenv` environment from PyPI. **Borg version:** 1.1.14 **Python version:** 3.7.3 **operating system and version:** Debian 10 Buster

witten commented

2020-11-06 17:57:49 +00:00

Owner

Thank you for taking the time to file this! A couple of thoughts here..

One way I could see to satisfy this need would be to expand the existing Healthchecks configuration schema to encompass different ping URLs per action. Made-up example:

hooks:
    healthchecks:
        create: https://hc-ping.com/uuid-here
        check: https://hc-ping.com/other-uuid-here
        prune: https://hc-ping.com/yet-another-uuid-here

Another idea is to leave the existing configuration schema as-is, but then override the ping URL on the command-line when running each action in a cron job. And use a separate ping URL / Healthchecks monitor for each cron job. Example:

borgmatic check --override hooks.healthchecks=https://hc-ping.com/your-uuid-here

This approach has the benefit of working today without any changes to borgmatic. The downside of course is you need to put the ping URL in your cron job.

Let me know your thoughts!

Thank you for taking the time to file this! A couple of thoughts here.. One way I could see to satisfy this need would be to expand the existing Healthchecks configuration schema to encompass different ping URLs per action. Made-up example: ```yaml hooks: healthchecks: create: https://hc-ping.com/uuid-here check: https://hc-ping.com/other-uuid-here prune: https://hc-ping.com/yet-another-uuid-here ``` Another idea is to leave the existing configuration schema as-is, but then override the ping URL on the command-line when running each action in a cron job. And use a separate ping URL / Healthchecks monitor for each cron job. Example: ```bash borgmatic check --override hooks.healthchecks=https://hc-ping.com/your-uuid-here ``` This approach has the benefit of working today without any changes to borgmatic. The downside of course is you need to put the ping URL in your cron job. Let me know your thoughts!

lgp171188 commented

2020-11-06 19:06:08 +00:00

Author

@witten, thanks for the very quick response 🙂

Another idea is to leave the existing configuration schema as-is, but then override the ping URL on the command-line when running each action in a cron job. And use a separate ping URL / Healthchecks monitor for each cron job. Example:

This approach has the benefit of working today without any changes to borgmatic. The downside of course is you need to put the ping URL in your cron job.

I'm perfectly fine with using this solution for my needs if a lot of effort and thought has to be put into implementing the former. It will be good to document this scenario and solution.

One way I could see to satisfy this need would be to expand the existing Healthchecks configuration schema to encompass different ping URLs per action. Made-up example:

It might be easier to declare a default URL and override only the ones that have to be different.

@witten, thanks for the very quick response 🙂 > Another idea is to leave the existing configuration schema as-is, but then override the ping URL on the command-line when running each action in a cron job. And use a separate ping URL / Healthchecks monitor for each cron job. Example: > This approach has the benefit of working today without any changes to borgmatic. The downside of course is you need to put the ping URL in your cron job. I'm perfectly fine with using this solution for my needs if a lot of effort and thought has to be put into implementing the former. It will be good to document this scenario and solution. > One way I could see to satisfy this need would be to expand the existing Healthchecks configuration schema to encompass different ping URLs per action. Made-up example: It might be easier to declare a default URL and override only the ones that have to be different.

witten referenced this issue

2022-04-22 16:48:25 +00:00

Different health check endpoints for different commands #518

witten commented

2022-04-22 16:49:18 +00:00

Owner

#518 is a very similar change on the Cronitor monitoring hook. It's possible that work there could apply to Healthchecks as well.

witten referenced this issue

2022-07-04 05:46:38 +00:00

[Question] Does borgmatic check repeat the repo check if I have multiple config files? #558

witten referenced this issue

2022-10-27 16:24:45 +00:00

Hook "on_warning" #606

witten referenced this issue

2023-05-08 21:58:53 +00:00

Different health check endpoints for different commands #518

kaysond commented

2023-05-22 19:01:56 +00:00

Subscribing since I have the same issue!

👍 1

witten referenced this issue

2023-07-07 03:56:59 +00:00

WIP: Separate endpoint for each primary action for monitor hooks #660

joef42 commented

2024-08-26 11:12:37 +00:00

Is anybody currently working on this? If not, I'm thinking about taking a look at it, because I would be really interested in having this.

👍 1

witten commented

2024-08-26 15:52:54 +00:00

Owner

As far as I know, nobody is currently working on this! Please let me know if you have any questions as you dig into this.

joef42 commented

2024-08-27 11:39:22 +00:00

Thanks @witten. I will give it a try and get back to your offer if needed.

One thing I would already like to clarify. I would like to have the ping for check only trigger, if a check was actually executed (i.e. not skipped due to frequency). I think this would allow for a more accurate health monitoring, e.g. if the check frequency is wrongly configured.

Do you think this makes sense in general? Would this be too complicated from an implementation perspective?

Thanks @witten. I will give it a try and get back to your offer if needed. One thing I would already like to clarify. I would like to have the ping for check only trigger, if a check was actually executed (i.e. not skipped due to frequency). I think this would allow for a more accurate health monitoring, e.g. if the check frequency is wrongly configured. Do you think this makes sense in general? Would this be too complicated from an implementation perspective?

witten commented

2024-08-29 22:41:00 +00:00

Owner

Here are some thoughts on that:

Today, the "API" between borgmatic and the Healthchecks hook pinging your configured Healthchecks server is function called ping_monitor() that takes several arguments—but none of them contain the current borgmatic actions.
So it stands to reason that this function/API would have to be expanded to pass the current borgmatic actions so that the Healthchecks hook can figure out which configured ping URLs to hit.
Per your question, would it make sense to skip a ping URL call for check if no checks actually ran? I'm not sure how you'd do that since at the point ping_monitor() first runs (the ping start), checks haven't even attempted to run yet. All we know is that check is among the actions to run. And on the other side (ping finish), you presumably wouldn't have any access to the check results.

So my recommendation would be to call the check ping URL regardless of which check(s) do or don't run. And then if you need more details about what specific checks ran, that'll show up in both your borgmatic system logs and your Healtchecks logs. I realize though that that won't allow for easy programmatic checking that a check's frequency is configured correctly.

However this is just a recommendation based on what I remember of the code; if you find a decent way to accomplish this anyway, I'd be happy to hear about it.

Here are some thoughts on that: - Today, the "API" between borgmatic and the Healthchecks hook pinging your configured Healthchecks server is function called `ping_monitor()` that takes several arguments—but none of them contain the current borgmatic actions. - So it stands to reason that this function/API would have to be expanded to pass the current borgmatic actions so that the Healthchecks hook can figure out which configured ping URLs to hit. - Per your question, would it make sense to skip a ping URL call for `check` if no checks actually ran? I'm not sure how you'd do that since at the point `ping_monitor()` first runs (the ping start), checks haven't even attempted to run yet. All we know is that `check` is among the actions to run. And on the other side (ping finish), you presumably wouldn't have any access to the check results. So my recommendation would be to call the `check` ping URL regardless of which check(s) do or don't run. And then if you need more details about what specific checks ran, that'll show up in both your borgmatic system logs and your Healtchecks logs. I realize though that that won't allow for easy programmatic checking that a check's frequency is configured correctly. However this is just a recommendation based on what I remember of the code; if you find a decent way to accomplish this anyway, I'd be happy to hear about it.

Rows
Columns

Healthchecks integration for independent operations #366

What I'm trying to do and why

Steps to reproduce (if a bug)

Actual behavior (if a bug)

Expected behavior (if a bug)

Other notes / implementation ideas

Environment