Grafana loki support #743

New Issue

IBims1NicerTobi · 2023-08-14T17:31:52Z

IBims1NicerTobi commented

2023-08-14 17:31:52 +00:00

What I'd like to do and why

It would be great if we had the option to push the logs and the stats of the backup to loki. I would be willing to write the integration for that if someone would be so kind and tell me what I would have to add in terms of code and how integration testing works in such cases. I personally use the entrie grafana stack for monitoring and it would be great to have backup status and monitoring in there too.

As far as I am aware I would only have to implement a new hook for this, so something like this should be enough.

Other notes / implementation ideas

No response

### What I'd like to do and why It would be great if we had the option to push the logs and the stats of the backup to [loki](https://grafana.com/docs/loki/latest/api/). I would be willing to write the integration for that if someone would be so kind and tell me what I would have to add in terms of code and how integration testing works in such cases. I personally use the entrie grafana stack for monitoring and it would be great to have backup status and monitoring in there too. As far as I am aware I would only have to implement a new hook for this, so something like [this](https://projects.torsion.org/borgmatic-collective/borgmatic/src/branch/main/borgmatic/hooks/pagerduty.py) should be enough. ### Other notes / implementation ideas _No response_

witten commented

2023-08-14 17:51:26 +00:00

Sounds like that'd be a great addition to borgmatic's monitoring capabilities! PRs are definitely welcome. And yes, implementing a new hook like one of the existing ones is the way to go. Most of the work would be in that Python hook file, but here are a few other places you'd need to touch as well:

borgmatic/config/schema.yaml: For adding the new configuration options for this hook.
borgmatic/hooks/dispatch.py and monitor.py: So the hook can actually get called by the borgmatic code.

In terms of automated testing, there are currently no any end-to-end tests that actually hit any monitoring services. But if you are keen on doing that anyway, I'd imagine they'd work like the existing database end-to-end tests (tests/end-to-end/) that spin up a database server with Docker Compose to test against. This is totally not a requirement for completing this hook IMO.

For unit tests, those would definitely be a good idea IMO. Take a look at the existing tests in tests/unit/hooks/test_pagerduty.py or other monitoring hooks for examples.

Note that there are a few different "styles" of monitoring hooks in borgmatic right now. Some, like the PagerDuty hook, only log failures. Others, like Cronhub, log several different states (start, finish, failure). So which style you'd like to write probably depends on what Loki expects.

And documentation: You're welcome to add docs for this new hook, but I'd also be happy to take that on if you'd prefer to focus on the code.

Finally, check out the following borgmatic dev docs:

Hope that helps! Let me know if you have any questions.

Sounds like that'd be a great addition to borgmatic's monitoring capabilities! PRs are definitely welcome. And yes, implementing a new hook like one of the existing ones is the way to go. Most of the work would be in that Python hook file, but here are a few other places you'd need to touch as well: - `borgmatic/config/schema.yaml`: For adding the new configuration options for this hook. - `borgmatic/hooks/dispatch.py` and `monitor.py`: So the hook can actually get called by the borgmatic code. In terms of automated testing, there are currently no any end-to-end tests that actually hit any monitoring services. But if you are keen on doing that anyway, I'd imagine they'd work like the existing database end-to-end tests (`tests/end-to-end/`) that spin up a database server with Docker Compose to test against. This is totally not a requirement for completing this hook IMO. For unit tests, those would definitely be a good idea IMO. Take a look at the existing tests in `tests/unit/hooks/test_pagerduty.py` or other monitoring hooks for examples. Note that there are a few different "styles" of monitoring hooks in borgmatic right now. Some, like the PagerDuty hook, only log failures. Others, like Cronhub, log several different states (start, finish, failure). So which style you'd like to write probably depends on what Loki expects. And documentation: You're welcome to add docs for this new hook, but I'd also be happy to take that on if you'd prefer to focus on the code. Finally, check out the following borgmatic dev docs: - https://torsion.org/borgmatic/docs/how-to/develop-on-borgmatic/ - https://torsion.org/borgmatic/docs/reference/source-code/ Hope that helps! Let me know if you have any questions.

IBims1NicerTobi commented

2023-08-21 14:12:50 +00:00

@witten So I have basically finished the core implementation now and I'm working on the configuration and have a question about the schema. Is it possible to allow a dict of arbitrary strings as key/value pairs? Right now the schema looks something like this:
loki: url: http://localhost:3100/loki/api/v1/push labels: - label: hostname value: test
What I am trying to achieve is having the labels as a normal yaml dict (like for example in ansible)
loki: url: http://localhost:3100/loki/api/v1/push labels: - hostname: test
Is that possible?

@witten So I have basically finished the core implementation now and I'm working on the configuration and have a question about the schema. Is it possible to allow a dict of arbitrary strings as key/value pairs? Right now the schema looks something like this: ` loki: url: http://localhost:3100/loki/api/v1/push labels: - label: hostname value: test ` What I am trying to achieve is having the labels as a normal yaml dict (like for example in ansible) ` loki: url: http://localhost:3100/loki/api/v1/push labels: - hostname: test ` Is that possible?

witten commented

2023-08-21 18:48:34 +00:00

I'm not 100% sure as I haven't done this before, but I'd start by trying something like this in the schema under properties:

labels:
    type: object
    additionalProperties: true
    patternProperties:
        "":
            type: string

Docs here: https://json-schema.org/understanding-json-schema/reference/object.html

I'm not 100% sure as I haven't done this before, but I'd start by trying something like this in the schema under `properties`: ``` labels: type: object additionalProperties: true patternProperties: "": type: string ``` Docs here: https://json-schema.org/understanding-json-schema/reference/object.html

IBims1NicerTobi commented

2023-08-22 01:16:32 +00:00

The basics are done in the pull request.

witten referenced this issue from a commit

2023-08-25 16:28:22 +00:00

Added support for grafana loki (#743).

witten commented

2023-08-25 16:54:34 +00:00

Merged and will be part of the next release. Thanks again!

witten closed this issue

2023-08-25 16:54:35 +00:00

witten referenced this issue from a commit

2023-08-25 17:52:23 +00:00

Add documentation for Grafana Loki hook (#743).

witten commented

2023-08-25 17:59:51 +00:00

Docs are live (even though the code isn't released yet)! https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#loki-hook

Let me know if anything's inaccurate or otherwise needs tweaking.

Docs are live (even though the code isn't released yet)! https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#loki-hook Let me know if anything's inaccurate or otherwise needs tweaking.

IBims1NicerTobi commented

2023-08-27 20:43:58 +00:00

The docs you added look great. I don't think I have anything to add so far but I would like to say that I have only tested the hook against a local grafana loki instance and I have not yet confirmed that it works with a grafana cloud account. Other than that I will add end-to-end tests with a docker container as soon as I get to it.
Thank you for the great help.

The docs you added look great. I don't think I have anything to add so far but I would like to say that I have only tested the hook against a local grafana loki instance and I have not yet confirmed that it works with a grafana cloud account. Other than that I will add end-to-end tests with a docker container as soon as I get to it. Thank you for the great help.

witten commented

2023-08-28 03:23:51 +00:00

Sounds good!

witten referenced this issue from a commit

2023-08-28 03:30:44 +00:00

Add borgmatic version introducing Loki hook to docs (#743).

witten commented

2023-09-30 20:40:04 +00:00

This has been released as part of borgmatic 1.8.3!

Sign in to join this conversation.