[Question] Monitoring borgmatic check #255

New Issue

Closed

opened 2019-11-27 17:18:29 +00:00 by mhaamann · 11 comments

mhaamann commented 2019-11-27 17:18:29 +00:00

Copy Link

What I'm trying to do and why

I am running borgmatic --check every month to check on my backups. The backups are large (> 1tb) so I run borgmatic --check and borgmatic --create at different intervals. Both commands are run using cron.

Question

How can see the results from borgmatic --check or be notified if the check fails. It does not seem to be supported by the hooks?

Environment

borgmatic version: 1.3.26

Borg version: borg 1.1.10

Python version: Python 3.6.8

#### What I'm trying to do and why I am running borgmatic --check every month to check on my backups. The backups are large (> 1tb) so I run borgmatic --check and borgmatic --create at different intervals. Both commands are run using cron. #### Question How can see the results from `borgmatic --check` or be notified if the check fails. It does not seem to be supported by the hooks? #### Environment **borgmatic version:** 1.3.26 **Borg version:** borg 1.1.10 **Python version:** Python 3.6.8

witten added the question / support label 2019-11-27 17:50:20 +00:00

witten commented 2019-11-27 17:56:56 +00:00

Owner

Copy Link

The existing on_error command hook will trigger when borgmatic check fails! Additionally, if you're using one of the monitoring services hooks (Healthchecks, Cronitor, Cronhub), then they will be notified if a borgmatic check invocation fails. Or did you have another type of notification in mind?

Also, perhaps the docs need clarification on this point?

The existing `on_error` command hook will trigger when `borgmatic check` fails! Additionally, if you're using one of the monitoring services hooks ([Healthchecks](https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#healthchecks-hook), Cronitor, Cronhub), then they will be notified if a `borgmatic check` invocation fails. Or did you have another type of notification in mind? Also, perhaps the docs need clarification on this point?

mhaamann commented 2019-11-28 14:37:35 +00:00

Author

Copy Link

Thanks!
I guess what happened was that borgmatic check was running when another process attempted a borgmatic create. The last failed and fired of the on_error hook.

While debugging I couldn't see in the logs that the borgmatic check had started/finished so my conclusion was that it had somehow failed.

After re-reading the documentation this section is important:

before_everything hooks collected from all borgmatic configuration files run once before all configuration files (prior to all actions), but only if there is a create action.

Which means that there is no way for me to see whether the borgmatic check started or not.
It seems that the health checks follows the same logic. No notification is done if create is not included.

Do you have any recommendation for monitoring the borgmatic check??

Thanks! I guess what happened was that `borgmatic check` was running when another process attempted a `borgmatic create`. The last failed and fired of the `on_error ` hook. While debugging I couldn't see in the logs that the `borgmatic check` had started/finished so my conclusion was that it had somehow failed. After re-reading the documentation this section is important: ```before_everything hooks collected from all borgmatic configuration files run once before all configuration files (prior to all actions), but only if there is a create action.``` Which means that there is no way for me to see whether the `borgmatic check` started or not. It seems that the health checks follows the same logic. No notification is done if `create` is not included. Do you have any recommendation for monitoring the `borgmatic check`??

witten commented 2019-11-28 21:41:53 +00:00

Owner

Copy Link

As far as the multiple borgmatic processes running, you may be interested in this ticket: witten/borgmatic#250

In terms of monitoring of borgmatic check (and specifically starting/ending), one option would be to update the existing logic to fire start/end hooks for borgmatic check in addition to borgmatic create. The downside though of that approach is that it might be confusing whether a create or check is happening in any given hook. Thoughts on that?

It's also possible that logging on borgmatic check start/end could be increased a bit.

Ideally, how are you wanting to monitor borgmatic check specifically?

As far as the multiple borgmatic processes running, you may be interested in this ticket: https://projects.torsion.org/witten/borgmatic/issues/250 In terms of monitoring of `borgmatic check` (and specifically starting/ending), one option would be to update the existing logic to fire start/end hooks for `borgmatic check` in addition to `borgmatic create`. The downside though of that approach is that it might be confusing whether a `create` or `check` is happening in any given hook. Thoughts on that? It's also possible that logging on `borgmatic check` start/end could be increased a bit. Ideally, how are you wanting to monitor `borgmatic check` specifically?

mhaamann commented 2019-11-29 08:24:12 +00:00

Author

Copy Link

As far as the multiple borgmatic processes running, you may be interested in this ticket: #250

Thanks, I subscribed.

In terms of monitoring of borgmatic check (and specifically starting/ending), one option would be to update the existing logic to fire start/end hooks for borgmatic check in addition to borgmatic create. The downside though of that approach is that it might be confusing whether a create or check is happening in any given hook. Thoughts on that?

One solution could be to separate the hooks for create and check. In this way you could monitor with external health check services that create runs every night and check runs every month. Thoughts?

The new check added could be named after_check.

It’s also possible that logging on borgmatic check start/end could be increased a bit.

That would help debugging for sure, but I believe that most users should also be monitoring that the check runs as scheduled.

Ideally, how are you wanting to monitor borgmatic check specifically?

I would POST (using curl) to our monitoring service NodePing, so both check and create is monitored.

>As far as the multiple borgmatic processes running, you may be interested in this ticket: #250 Thanks, I subscribed. >In terms of monitoring of borgmatic check (and specifically starting/ending), one option would be to update the existing logic to fire start/end hooks for borgmatic check in addition to borgmatic create. The downside though of that approach is that it might be confusing whether a create or check is happening in any given hook. Thoughts on that? One solution could be to separate the hooks for `create` and `check`. In this way you could monitor with external health check services that `create` runs every night and `check` runs every month. Thoughts? The new check added could be named `after_check`. >It’s also possible that logging on borgmatic check start/end could be increased a bit. That would help debugging for sure, but I believe that most users should also be monitoring that the `check` runs as scheduled. >Ideally, how are you wanting to monitor borgmatic check specifically? I would POST (using curl) to our monitoring service `NodePing`, so both `check` and `create` is monitored.

witten commented 2019-12-02 18:26:53 +00:00

Owner

Copy Link

One solution could be to separate the hooks for create and check. In this way you could monitor with external health check services that create runs every night and check runs every month. Thoughts?

That could certain work. However, note that currently the after_* hooks only fire on success, and on_error fires on failure/error. So we'd probably also have to separate out on_check_error or similar, assuming that you want separate failure notifications for create and check.

I would POST (using curl) to our monitoring service NodePing, so both check and create is monitored.

That makes me wonder: Would a more formal NodePing integration be desirable here? Similar to the existing Healthchecks/Cronitor/Cronhub integrations? That way you wouldn't have to craft curl commands. It sounds like for use case, you'd need it to hit different endpoints for each of create and check, and indicate successes/failures separately for each.

> One solution could be to separate the hooks for create and check. In this way you could monitor with external health check services that create runs every night and check runs every month. Thoughts? That could certain work. However, note that currently the `after_*` hooks only fire on success, and `on_error` fires on failure/error. So we'd probably also have to separate out `on_check_error` or similar, assuming that you want separate failure notifications for `create` and `check`. > I would POST (using curl) to our monitoring service NodePing, so both check and create is monitored. That makes me wonder: Would a more formal NodePing integration be desirable here? Similar to the existing Healthchecks/Cronitor/Cronhub integrations? That way you wouldn't have to craft `curl` commands. It sounds like for use case, you'd need it to hit different endpoints for each of `create` and `check`, and indicate successes/failures separately for each.

witten referenced this issue 2019-12-04 23:52:26 +00:00

External integrations for borgmatic check and prune #249

witten commented 2019-12-04 23:52:37 +00:00

Owner

Copy Link

Related issue: #249.

Related issue: #249.

mhaamann commented 2019-12-05 08:00:29 +00:00

Author

Copy Link

That makes me wonder: Would a more formal NodePing integration be desirable here? Similar to the existing Healthchecks/Cronitor/Cronhub integrations? That way you wouldn’t have to craft curl commands. It sounds like for use case, you’d need it to hit different endpoints for each of create and check, and indicate successes/failures separately for each.

Sure, with Nodeping I would have to create 3 separate checks: Create, Check and Prune but I think its easier for Borgmatic to maintain after_* hooks for the above rather than trying to tightly integrate with all the different services. I believe crafting a curl command is easier for the community compared to maintaining X number of tight integrations with different providers.

I believe my scenario would be solved by adding additional hooks for after_* and providing better logs on checks.

>That makes me wonder: Would a more formal NodePing integration be desirable here? Similar to the existing Healthchecks/Cronitor/Cronhub integrations? That way you wouldn’t have to craft curl commands. It sounds like for use case, you’d need it to hit different endpoints for each of create and check, and indicate successes/failures separately for each. Sure, with Nodeping I would have to create 3 separate checks: Create, Check and Prune but I think its easier for Borgmatic to maintain `after_*` hooks for the above rather than trying to tightly integrate with all the different services. I believe crafting a curl command is easier for the community compared to maintaining X number of tight integrations with different providers. I believe my scenario would be solved by adding additional hooks for `after_*` and providing better logs on `checks`.

witten commented 2019-12-05 18:49:34 +00:00

Owner

Copy Link

Why not both? 😃 Both being: Three separate checks + a NodePing integration. I'm fine starting this ticket at the former however.

Why not both? :smiley: Both being: Three separate checks + a NodePing integration. I'm fine starting this ticket at the former however.

witten added this to the per-action hooks milestone 2019-12-05 21:16:16 +00:00

witten referenced this issue from a commit 2020-01-27 19:11:48 +00:00

Add per-action hooks: "before_prune", "after_prune", "before_check", and "after_check" (#255).

witten commented 2020-01-27 19:13:15 +00:00

Owner

Copy Link

Okay, the following per-action hooks are implemented in master: "before_prune", "after_prune", "before_check", and "after_check". I'll post here when it's released. Thanks again for the idea!

Okay, the following per-action hooks are implemented in master: "before_prune", "after_prune", "before_check", and "after_check". I'll post here when it's released. Thanks again for the idea!

👍 1

witten closed this issue 2020-01-27 19:13:15 +00:00

witten commented 2020-01-27 23:42:26 +00:00

Owner

Copy Link

Released in borgmatic 1.5.0!

Released in borgmatic 1.5.0!

mhaamann commented 2020-01-28 08:06:39 +00:00

Author

Copy Link

Awesome. Looking forward to try it out

Awesome. Looking forward to try it out

lgp171188 referenced this issue 2020-11-06 10:18:33 +00:00

Healthchecks integration for independent operations #366

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

command-hook-steps

2.1.2

2.1.1

2.1.0

2.0.13

2.0.12

2.0.11

2.0.10

2.0.9

2.0.8

2.0.7

2.0.6

2.0.5

2.0.4

2.0.3

2.0.2

2.0.1

2.0.0

1.9.14

1.9.13

1.9.12

1.9.11

1.9.10

1.9.9

1.9.8

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.14

1.8.13

1.8.12

1.8.11

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.24

1.5.23

1.5.22

1.5.21

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.14

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.7.dev0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.22

1.4.21

1.4.20

1.4.19

1.4.18

1.4.17

1.4.16

1.4.15

1.4.14

1.4.13

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.26

1.3.25

1.3.24

1.3.23

1.3.22

1.3.21

1.3.20

1.3.19

1.3.18

1.3.17

1.3.16

1.3.15

1.3.14

1.3.13

1.3.12

1.3.11

1.3.10

1.3.9

1.3.8

1.3.7

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.18

1.2.17

1.2.16

1.2.15

1.2.14

1.2.13

1.2.12

1.2.11

1.2.10

1.2.9

1.2.8

1.2.7

1.2.6

1.2.5

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.15

1.1.14

1.1.13

1.1.12

1.1.11

1.1.10

1.1.9

1.1.7

1.1.6

1.1.5

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.3

1.0.2

1.0.1

1.0.0

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

0.0.2

Labels

Clear labels

blocked

breaking

bug

data loss

design finalized

good first issue

new feature area

question / support

security

waiting for response

No Label question / support

Milestone

No items

No Milestone per-action hooks

Assignees

Clear assignees

witten

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#255