Feature request: PagerDuty Notifications #245

Closed
opened 2019-11-13 09:36:40 +00:00 by earthquakesan · 8 comments

What I'm trying to do and why

I want to have embedded pagerduty notifications (the same as healthchecks etc.)

Steps to reproduce

As a work around I have implemented a bash script for my setup atm. See the source on the Github: https://github.com/earthquakesan/borg-lab/blob/master/playbooks/roles/borgmatic/templates/pagerduty-notification.sh.j2

Expected behavior

hooks:
    pagerduty:
        routing_key: "blabla"

When there is an error, borgmatic should automatically send incident to pagerduty on that routing_key.

Pagerduty can only create triggers or incidents (it is incident responding tool). routing_key is a unique key to send notifications to a Service (internal pagerduty object).

#### What I'm trying to do and why I want to have embedded pagerduty notifications (the same as healthchecks etc.) #### Steps to reproduce As a work around I have implemented a bash script for my setup atm. See the source on the Github: https://github.com/earthquakesan/borg-lab/blob/master/playbooks/roles/borgmatic/templates/pagerduty-notification.sh.j2 #### Expected behavior ``` hooks: pagerduty: routing_key: "blabla" ``` When there is an error, borgmatic should automatically send incident to pagerduty on that routing_key. Pagerduty can only create triggers or incidents (it is incident responding tool). `routing_key` is a unique key to send notifications to a Service (internal pagerduty object).
Owner

I'm certainly not opposed to this, but are you aware that Healthchecks, Cronitor, and Cronhub all have PagerDuty integration? So today, you could configure borgmatic to ping one of Healthchecks/Cronitor/Cronhub, and then configure that to notify you of failures via PagerDuty.

Is the problem that you don't want to use a service like Healthchecks/Cronitor/Cronhub, and just want to get PagerDuty notifications directly? Thanks!

I'm certainly not opposed to this, but are you aware that Healthchecks, Cronitor, and Cronhub all have PagerDuty integration? So today, you could configure borgmatic to ping one of Healthchecks/Cronitor/Cronhub, and then configure that to notify you of failures via PagerDuty. Is the problem that you don't want to use a service like Healthchecks/Cronitor/Cronhub, and just want to get PagerDuty notifications directly? Thanks!
Author

Those are all additional services --> management overhead + need to pay for them. While I just need simply notifications by PagerDuty if a backup has failed.

I could potentially run healthchecks inside our data center and ping PagerDuty from there. But I don't have to deploy it just for this purpose. Sounds like an overkill for me.

Those are all additional services --> management overhead + need to pay for them. While I just need simply notifications by PagerDuty if a backup has failed. I could potentially run healthchecks inside our data center and ping PagerDuty from there. But I don't have to deploy it just for this purpose. Sounds like an overkill for me.
Owner

Got it, understood. Thanks for the explanation. This looks like it may be the place to start: https://v2.developer.pagerduty.com/docs/events-api-v2

Got it, understood. Thanks for the explanation. This looks like it may be the place to start: https://v2.developer.pagerduty.com/docs/events-api-v2
Author

ok, thx, I will implement when I have time. Also will look into Healthchecks in parallel as it still might be useful.

ok, thx, I will implement when I have time. Also will look into Healthchecks in parallel as it still might be useful.
Owner

I put that API link there for whomever picks up the ticket.. wasn't suggesting that had to be you! But PRs of course are more than welcome.

I put that API link there for whomever picks up the ticket.. wasn't suggesting that had to be you! But PRs of course are more than welcome.
Owner

Implemented, and released in borgmatic 1.5.0! I went with this configuration schema for simplicity:

hooks:
    pagerduty: a177cad45bd374409f78906a810a3074

Let me know how it works out for you in practice, and whether you'd like any tweaks to how it works. Thanks for suggesting the feature!

Full docs here: https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#pagerduty-hook

Implemented, and released in borgmatic 1.5.0! I went with this configuration schema for simplicity: ``` hooks: pagerduty: a177cad45bd374409f78906a810a3074 ``` Let me know how it works out for you in practice, and whether you'd like any tweaks to how it works. Thanks for suggesting the feature! Full docs here: https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#pagerduty-hook
Owner

And now borgmatic is listed on PagerDuty's integrations page! https://www.pagerduty.com/integrations/

And now borgmatic is listed on PagerDuty's integrations page! https://www.pagerduty.com/integrations/
Author

@witten, thanks a lot for implementing!

@witten, thanks a lot for implementing!
Sign in to join this conversation.
No Milestone
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#245
No description provided.