Sentry Cron Monitoring support #855

New Issue

Closed

opened 2024-04-22 10:01:20 +00:00 by SCjona · 6 comments

SCjona commented 2024-04-22 10:01:20 +00:00

Copy Link

What I'd like to do and why

I'd like to use Sentry Cron Monitoring to monitor my backups.
See the documentation here: https://docs.sentry.io/product/crons/getting-started/http/#check-ins-recommended

Other notes / implementation ideas

You'd configure the monitoring URL (variable SENTRY_CRONS in the Sentry docs) in the borgmatic config and borgmatic should send the correct HTTP requests.

I tried implementing this with hooks (before_everything => send status=in_progress, on_error => send status=error and after_everything => send status=ok), which works fine for the success case (before_everything and after_everything is run), but in the error case it will run all three hooks and after_everything will cause the monitoring status to switch from error to ok. So alternatively I would like to request an on_success hook.

### What I'd like to do and why I'd like to use Sentry Cron Monitoring to monitor my backups. See the documentation here: https://docs.sentry.io/product/crons/getting-started/http/#check-ins-recommended ### Other notes / implementation ideas You'd configure the monitoring URL (variable `SENTRY_CRONS` in the Sentry docs) in the borgmatic config and borgmatic should send the correct HTTP requests. I tried implementing this with hooks (`before_everything` => send `status=in_progress`, `on_error` => send `status=error` and `after_everything` => send `status=ok`), which works fine for the success case (`before_everything` and `after_everything` is run), but in the error case it will run all three hooks and `after_everything` will cause the monitoring status to switch from `error` to `ok`. So alternatively I would like to request an `on_success` hook.

👍 1

witten commented 2024-04-23 16:00:49 +00:00

Owner

Copy Link

Thanks for filing this. Sentry support seems like a reasonable addition to borgmatic. But just to confirm, should the monitoring URL (SENTRY_CRONS from the docs you linked) be the same value as SENTRY_DSN, provided when creating a cron in Sentry?

And yeah, I would not recommend using before_everything/after_everything for sending monitoring pings, because after_everything runs even if an error occurs during the backup. (before_actions/after_actions may make more sense for this.)

Thanks for filing this. Sentry support seems like a reasonable addition to borgmatic. But just to confirm, should the monitoring URL (`SENTRY_CRONS` from the docs you linked) be the same value as `SENTRY_DSN`, provided when creating a cron in Sentry? And yeah, I would not recommend using `before_everything`/`after_everything` for sending monitoring pings, because `after_everything` runs even if an error occurs during the backup. (`before_actions`/`after_actions` may make more sense for this.)

SCjona commented 2024-04-24 08:04:43 +00:00

Author

Copy Link

SENTRY_DSN is something else, it's basically just an project identifer used for error and performance reporting. They created special endpoints for cron monitoring.

Example values:
SENTRY_DSN: https://<random-hex-token>@<sentry-host-name>/<project-id> => https://ada43a0c496f5b2439866a6c20398628@sentry.hostname.tld/2
SENTRY_CRONS: https://<sentry-host-name>/api/<project-id>/cron/<cron-name>/<hex-token-from-DSN>/ => https://sentry.hostname.tld/api/2/cron/my-backups/ada43a0c496f5b2439866a6c20398628/

As you can see SENTRY_CRONS is basically just the info in SENTRY_DSN combined with the Cron monitor name (called monitor slug in Sentry UI). Might be easier to just require the users to configure those two as they are easily found in Sentry UI.

Where to find the DSN => https://docs.sentry.io/product/sentry-basics/concepts/dsn-explainer/
Where to find the monitor slug =>
(bottom right in image => monitor slug)

You could also consider doing a complete Sentry Integration, not just for the Cron monitoring part. You could report errors directly to Sentry and track performance (which backup step took how long for example) with it, however I'm not sure if adding a new dependency (Sentry Python SDK) is reasonable/acceptable for the project. It's probably overkill anyways.

Also FYI there is self-hosted sentry, pretty easy to set up if you know your way around Linux and Docker: https://develop.sentry.dev/self-hosted/

`SENTRY_DSN` is something else, it's basically just an project identifer used for error and performance reporting. They created special endpoints for cron monitoring. Example values: `SENTRY_DSN`: `https://<random-hex-token>@<sentry-host-name>/<project-id>` => `https://ada43a0c496f5b2439866a6c20398628@sentry.hostname.tld/2` `SENTRY_CRONS`: `https://<sentry-host-name>/api/<project-id>/cron/<cron-name>/<hex-token-from-DSN>/` => `https://sentry.hostname.tld/api/2/cron/my-backups/ada43a0c496f5b2439866a6c20398628/` As you can see `SENTRY_CRONS` is basically just the info in `SENTRY_DSN` combined with the Cron monitor name (called `monitor slug` in Sentry UI). Might be easier to just require the users to configure those two as they are easily found in Sentry UI. Where to find the DSN => https://docs.sentry.io/product/sentry-basics/concepts/dsn-explainer/ Where to find the monitor slug =>  (bottom right in image => `monitor slug`) You could also consider doing a complete Sentry Integration, not just for the Cron monitoring part. You could report errors directly to Sentry and track performance (which backup step took how long for example) with it, however I'm not sure if adding a new dependency ([Sentry Python SDK](https://docs.sentry.io/platforms/python/)) is reasonable/acceptable for the project. It's probably overkill anyways. Also FYI there is self-hosted sentry, pretty easy to set up if you know your way around Linux and Docker: https://develop.sentry.dev/self-hosted/

image.png

77 KiB

SCjona commented 2024-04-24 11:10:46 +00:00

Author

Copy Link

Also thank you for the hint with before_actions / after_actions. I got the cron monitoring setup working with before_actions => in_progress, on_error => error and after_actions => ok.

Also thank you for the hint with `before_actions` / `after_actions`. I got the cron monitoring setup working with `before_actions` => `in_progress`, `on_error` => `error` and `after_actions` => `ok`.

witten commented 2024-06-23 22:10:01 +00:00

Owner

Copy Link

Thanks for the clarification, and I'm glad to hear you have a work-around with before_actions/after_actions!

Thanks for the clarification, and I'm glad to hear you have a work-around with `before_actions`/`after_actions`!

witten added the new feature area label 2024-06-23 22:10:12 +00:00

witten referenced this issue from a commit 2025-01-22 01:24:39 +00:00

Add a Sentry monitoring hook (#855).

witten referenced this issue from a commit 2025-01-22 01:24:39 +00:00

Add a Sentry monitoring hook (#855).

witten commented 2025-01-22 01:29:50 +00:00

Owner

Copy Link

This is implemented in main and will be part of the next release! The documentation should be available online shortly: https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#sentry-hook

Here's the basic configuration I ended up going with, based on your suggestion:

sentry:
    data_source_name_url: https://5f80ec@o294220.ingest.us.sentry.io/203069
    monitor_slug: mymonitor

I didn't do a "complete" Sentry integration; only the cron monitor portion is integrated.

Thanks for your patience here!

This is implemented in main and will be part of the next release! The documentation should be available online shortly: https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#sentry-hook Here's the basic configuration I ended up going with, based on your suggestion: ```yaml sentry: data_source_name_url: https://5f80ec@o294220.ingest.us.sentry.io/203069 monitor_slug: mymonitor ``` I didn't do a "complete" Sentry integration; only the cron monitor portion is integrated. Thanks for your patience here!

witten closed this issue 2025-01-22 01:29:50 +00:00

witten referenced this issue from a commit 2025-01-22 01:30:16 +00:00

Link to Sentry's DSN documentation (#855).

witten commented 2025-01-23 20:15:52 +00:00

Owner

Copy Link

Released in borgmatic 1.9.7!

Released in borgmatic 1.9.7!

❤️ 1

witten referenced this issue from a commit 2025-01-26 17:30:26 +00:00

Add missing test coverage and fix incorrect test expectations (#855).

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

command-hook-steps

2.1.2

2.1.1

2.1.0

2.0.13

2.0.12

2.0.11

2.0.10

2.0.9

2.0.8

2.0.7

2.0.6

2.0.5

2.0.4

2.0.3

2.0.2

2.0.1

2.0.0

1.9.14

1.9.13

1.9.12

1.9.11

1.9.10

1.9.9

1.9.8

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.14

1.8.13

1.8.12

1.8.11

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.24

1.5.23

1.5.22

1.5.21

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.14

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.7.dev0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.22

1.4.21

1.4.20

1.4.19

1.4.18

1.4.17

1.4.16

1.4.15

1.4.14

1.4.13

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.26

1.3.25

1.3.24

1.3.23

1.3.22

1.3.21

1.3.20

1.3.19

1.3.18

1.3.17

1.3.16

1.3.15

1.3.14

1.3.13

1.3.12

1.3.11

1.3.10

1.3.9

1.3.8

1.3.7

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.18

1.2.17

1.2.16

1.2.15

1.2.14

1.2.13

1.2.12

1.2.11

1.2.10

1.2.9

1.2.8

1.2.7

1.2.6

1.2.5

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.15

1.1.14

1.1.13

1.1.12

1.1.11

1.1.10

1.1.9

1.1.7

1.1.6

1.1.5

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.3

1.0.2

1.0.1

1.0.0

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

0.0.2

Labels

Clear labels

blocked

breaking

bug

data loss

design finalized

good first issue

new feature area

question / support

security

waiting for response

No Label new feature area

Milestone

No items

No Milestone

Assignees

Clear assignees

witten

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#855