#245 Feature request: PagerDuty Notifications

Closed
opened 7 months ago by earthquakesan · 8 comments

What I’m trying to do and why

I want to have embedded pagerduty notifications (the same as healthchecks etc.)

Steps to reproduce

As a work around I have implemented a bash script for my setup atm. See the source on the Github: https://github.com/earthquakesan/borg-lab/blob/master/playbooks/roles/borgmatic/templates/pagerduty-notification.sh.j2

Expected behavior

hooks:
    pagerduty:
        routing_key: "blabla"

When there is an error, borgmatic should automatically send incident to pagerduty on that routing_key.

Pagerduty can only create triggers or incidents (it is incident responding tool). routing_key is a unique key to send notifications to a Service (internal pagerduty object).

#### What I'm trying to do and why I want to have embedded pagerduty notifications (the same as healthchecks etc.) #### Steps to reproduce As a work around I have implemented a bash script for my setup atm. See the source on the Github: https://github.com/earthquakesan/borg-lab/blob/master/playbooks/roles/borgmatic/templates/pagerduty-notification.sh.j2 #### Expected behavior ``` hooks: pagerduty: routing_key: "blabla" ``` When there is an error, borgmatic should automatically send incident to pagerduty on that routing_key. Pagerduty can only create triggers or incidents (it is incident responding tool). `routing_key` is a unique key to send notifications to a Service (internal pagerduty object).
witten commented 7 months ago
Owner

I’m certainly not opposed to this, but are you aware that Healthchecks, Cronitor, and Cronhub all have PagerDuty integration? So today, you could configure borgmatic to ping one of Healthchecks/Cronitor/Cronhub, and then configure that to notify you of failures via PagerDuty.

Is the problem that you don’t want to use a service like Healthchecks/Cronitor/Cronhub, and just want to get PagerDuty notifications directly? Thanks!

I'm certainly not opposed to this, but are you aware that Healthchecks, Cronitor, and Cronhub all have PagerDuty integration? So today, you could configure borgmatic to ping one of Healthchecks/Cronitor/Cronhub, and then configure that to notify you of failures via PagerDuty. Is the problem that you don't want to use a service like Healthchecks/Cronitor/Cronhub, and just want to get PagerDuty notifications directly? Thanks!
earthquakesan commented 7 months ago
Poster

Those are all additional services --> management overhead + need to pay for them. While I just need simply notifications by PagerDuty if a backup has failed.

I could potentially run healthchecks inside our data center and ping PagerDuty from there. But I don’t have to deploy it just for this purpose. Sounds like an overkill for me.

Those are all additional services --> management overhead + need to pay for them. While I just need simply notifications by PagerDuty if a backup has failed. I could potentially run healthchecks inside our data center and ping PagerDuty from there. But I don't have to deploy it just for this purpose. Sounds like an overkill for me.
witten commented 7 months ago
Owner

Got it, understood. Thanks for the explanation. This looks like it may be the place to start: https://v2.developer.pagerduty.com/docs/events-api-v2

Got it, understood. Thanks for the explanation. This looks like it may be the place to start: https://v2.developer.pagerduty.com/docs/events-api-v2
earthquakesan commented 7 months ago
Poster

ok, thx, I will implement when I have time. Also will look into Healthchecks in parallel as it still might be useful.

ok, thx, I will implement when I have time. Also will look into Healthchecks in parallel as it still might be useful.
witten commented 7 months ago
Owner

I put that API link there for whomever picks up the ticket.. wasn’t suggesting that had to be you! But PRs of course are more than welcome.

I put that API link there for whomever picks up the ticket.. wasn't suggesting that had to be you! But PRs of course are more than welcome.
witten commented 5 months ago
Owner

Implemented, and released in borgmatic 1.5.0! I went with this configuration schema for simplicity:

hooks:
    pagerduty: a177cad45bd374409f78906a810a3074

Let me know how it works out for you in practice, and whether you’d like any tweaks to how it works. Thanks for suggesting the feature!

Full docs here: https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#pagerduty-hook

Implemented, and released in borgmatic 1.5.0! I went with this configuration schema for simplicity: ``` hooks: pagerduty: a177cad45bd374409f78906a810a3074 ``` Let me know how it works out for you in practice, and whether you'd like any tweaks to how it works. Thanks for suggesting the feature! Full docs here: https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#pagerduty-hook
witten closed this issue 5 months ago
witten commented 4 months ago
Owner

And now borgmatic is listed on PagerDuty’s integrations page! https://www.pagerduty.com/integrations/

And now borgmatic is listed on PagerDuty's integrations page! https://www.pagerduty.com/integrations/
earthquakesan commented 4 months ago
Poster

@witten, thanks a lot for implementing!

@witten, thanks a lot for implementing!
Sign in to join this conversation.
No Milestone
No Assignees
2 Participants
Notifications
Due Date

No due date set.

Dependencies

This issue currently doesn't have any dependencies.

Loading…
There is no content yet.