borgmatic-collective/borgmatic

Watch

Fork 57

Export borgmatic backup statistics in prometheus format #1087

New issue

Open

opened 2025-04-30 18:59:54 +00:00 by vitoyucepi · 5 comments

vitoyucepi commented

2025-04-30 18:59:54 +00:00

What I'd like to do and why

I would like to monitor Borgmatic with Prometheus.
Borgmatic doesn't have a built-in Prometheus exporter, and it doesn't have a continuously running process, so in my opinion the best solutions would be to collect metrics with

Other notes / implementation ideas

The online exporters are:

I don't know for sure, but all of them will try to update the parameters on each metrics request, creating an unnecessary load.

There's also a variant of a text file exporter based on hooks: https://margau.net/posts/2024-08-02-borgmatic-prometheus/. But it's not very useful, because it doesn't know anything about the storage.

### What I'd like to do and why I would like to monitor Borgmatic with Prometheus. Borgmatic doesn't have a built-in Prometheus exporter, and it doesn't have a continuously running process, so in my opinion the best solutions would be to collect metrics with - [textfile collector](https://github.com/prometheus/node_exporter#textfile-collector) - [push gateway](https://github.com/prometheus/pushgateway) ### Other notes / implementation ideas The online exporters are: - https://github.com/fishgrimsby/borgmatic-exporter - https://github.com/heartshare/borgmatic-exporter2 - https://github.com/maxim-mityutko/borgmatic-exporter I don't know for sure, but all of them will try to update the parameters on each metrics request, creating an unnecessary load. There's also a variant of a text file exporter based on hooks: https://margau.net/posts/2024-08-02-borgmatic-prometheus/. But it's not very useful, because it doesn't know anything about the storage.

witten commented

2025-04-30 23:27:31 +00:00

Owner

Thanks for filing this and providing some helpful leads. For context, borgmatic's existing monitoring hooks generally push information like some (or all) of the following:

whether the backup succeeded or failed
an associated title or message about that
tags to attach to the message
a priority for the message
targeting info about what devices to send the message to
borgmatic log output

None of this is by its nature time series data, but I know Prometheus is often much more oriented around metrics and time series. So could you say a little more about what (if anything) you'd expect borgmatic to push to Prometheus when borgmatic runs? Are you envisioning more like metric data, as per some of the existing exporters you linked? Or just backup success/fail status?

Thanks for filing this and providing some helpful leads. For context, borgmatic's existing monitoring hooks generally push information like some (or all) of the following: - whether the backup succeeded or failed - an associated title or message about that - tags to attach to the message - a priority for the message - targeting info about what devices to send the message to - borgmatic log output None of this is by its nature time series data, but I know Prometheus is often much more oriented around metrics and time series. So could you say a little more about what (if anything) you'd expect borgmatic to push to Prometheus when borgmatic runs? Are you envisioning more like metric data, as per some of the existing exporters you linked? Or just backup success/fail status?

witten added the

new feature area

label

2025-04-30 23:27:38 +00:00

vitoyucepi commented

2025-05-01 07:44:19 +00:00

Author

Hi @witten,
I think the other exporters expose all the necessary information about the backup process. The main problem with their behavior is that they have no idea about the backup process itself.

As for the metrics, I can divide them into the following groups:

Setup information

borgmatic_version - Installed version of Borgmatic.
Of type info, or a gauge with version as a parameter and value of 1.
borg_version - Installed version of Borg.
Of type info, or a gauge with version as a parameter and value of 1.
borg_remote_version - Installed version of Borg in the remote repository.
Of type info, or a gauge with version as a parameter and value of 1.
For each remote repository.

Repository statistics

borg_archives - Number of archives in the repository.
Of type gauge.
For each remote repository.
borg_unique_chunks - Number of unique chunks in the repository.
Of type gauge.
For each remote repository.
borg_chunks - Number of chunks in the repository.
Of type gauge.
For each remote repository.
borg_size - Original size of the backup data.
Of type gauge.
For each remote repository.
borg_compressed_size - Compressed size of the backup data.
Of type gauge.
For each remote repository.
borg_deduplicated_size - Deduplicated size of the backup data.
Of type gauge.
For each remote repository.
borg_maximum_size - Maximum size of the repository.
Of type gauge.
For each remote repository.

Last backup statistics

borg_last_backup_timestamp - Timestamp of the last backup.
Of type timestamp, or a gauge. There are some problems with the type.
borg_last_backup_duration - Duration of the last backup.
Of type gauge.
borg_last_backup_files - Number of files in the last backup.
Of type gauge.
borg_last_backup_size - Original size of the last backup.
Of type gauge.
borg_last_backup_compressed_size - Compressed size of the last backup.
Of type gauge.
borg_last_backup_deduplicated_size - Deduplicated size of the last backup.
Of type gauge.

I think it's possible to combine metrics into one with parameters

borg_chunks{type="unique", repository="remote"} 123
borg_chunks{type="total", repository="remote"} 123

borg_size{type="original", repository="remote"} 123
borg_size{type="compressed", repository="remote"} 123
borg_size{type="deduplicated", repository="remote"} 123

borg_last_backup_size{type="original"} 123
borg_last_backup_size{type="compressed"} 123
borg_last_backup_size{type="deduplicated"} 123

The timestamp parameter is problematic because

Prometheus uses float64 to parse data.
The textfile collector doesn't support timestamps.
Prometheus library for Python outputs each metric as a float.

Hi @witten, I think the other exporters expose all the necessary information about the backup process. The main problem with their behavior is that they have no idea about the backup process itself. As for the metrics, I can divide them into the following groups: ### Setup information - `borgmatic_version` - Installed version of Borgmatic. Of type info, or a gauge with version as a parameter and value of 1. - `borg_version` - Installed version of Borg. Of type info, or a gauge with version as a parameter and value of 1. - `borg_remote_version` - Installed version of Borg in the remote repository. Of type info, or a gauge with version as a parameter and value of 1. For each remote repository. ### Repository statistics - `borg_archives` - Number of archives in the repository. Of type gauge. For each remote repository. - `borg_unique_chunks` - Number of unique chunks in the repository. Of type gauge. For each remote repository. - `borg_chunks` - Number of chunks in the repository. Of type gauge. For each remote repository. - `borg_size` - Original size of the backup data. Of type gauge. For each remote repository. - `borg_compressed_size` - Compressed size of the backup data. Of type gauge. For each remote repository. - `borg_deduplicated_size` - Deduplicated size of the backup data. Of type gauge. For each remote repository. - `borg_maximum_size` - Maximum size of the repository. Of type gauge. For each remote repository. ### Last backup statistics - `borg_last_backup_timestamp` - Timestamp of the last backup. Of type timestamp, or a gauge. There are some problems with the type. - `borg_last_backup_duration` - Duration of the last backup. Of type gauge. - `borg_last_backup_files` - Number of files in the last backup. Of type gauge. - `borg_last_backup_size` - Original size of the last backup. Of type gauge. - `borg_last_backup_compressed_size` - Compressed size of the last backup. Of type gauge. - `borg_last_backup_deduplicated_size` - Deduplicated size of the last backup. Of type gauge. I think it's possible to combine metrics into one with parameters ```prom borg_chunks{type="unique", repository="remote"} 123 borg_chunks{type="total", repository="remote"} 123 borg_size{type="original", repository="remote"} 123 borg_size{type="compressed", repository="remote"} 123 borg_size{type="deduplicated", repository="remote"} 123 borg_last_backup_size{type="original"} 123 borg_last_backup_size{type="compressed"} 123 borg_last_backup_size{type="deduplicated"} 123 ``` The timestamp parameter is problematic because 1. Prometheus uses `float64` to parse data. 2. The textfile collector doesn't support timestamps. 3. Prometheus library for Python outputs each metric as a float.

witten commented

2025-05-05 18:07:37 +00:00

Owner

Thanks, that's helpful.

I think the other exporters expose all the necessary information about the backup process. The main problem with their behavior is that they have no idea about the backup process itself.

To clarify: Are you referring to the "last backup statistics" here? Or are you referring to the more traditional monitoring hook information about backup success/failure? In any case, do you see that fitting in here somewhere or are you mostly just interested in the metrics?

Related: #617

Thanks, that's helpful. > I think the other exporters expose all the necessary information about the backup process. The main problem with their behavior is that they have no idea about the backup process itself. To clarify: Are you referring to the "last backup statistics" here? Or are you referring to the more traditional monitoring hook information about backup success/failure? In any case, do you see that fitting in here somewhere or are you mostly just interested in the metrics? Related: #617

vitoyucepi commented

2025-05-05 18:48:02 +00:00

Author

In all the examples I've seen, success and failure were defined by the last backup timestamp.
Nevertheless, metrics similar to borgmatic_last_action_success of type gauge with values 0 and 1 could be a great addition because they can trigger alerts directly.

borgmatic_last_action_success{type="backup"} 1
borgmatic_last_action_success{type="compact"} 1
borgmatic_last_action_success{type="check"} 1
borgmatic_last_action_success{type="prune"} 1

borgmatic_last_action_timestamp{type="backup"} 123
borgmatic_last_action_timestamp{type="compact"} 123
borgmatic_last_action_timestamp{type="check"} 123
borgmatic_last_action_timestamp{type="prune"} 123

In all the examples I've seen, success and failure were defined by the last backup timestamp. Nevertheless, metrics similar to `borgmatic_last_action_success` of type gauge with values `0` and `1` could be a great addition because they can trigger alerts directly. ``` borgmatic_last_action_success{type="backup"} 1 borgmatic_last_action_success{type="compact"} 1 borgmatic_last_action_success{type="check"} 1 borgmatic_last_action_success{type="prune"} 1 borgmatic_last_action_timestamp{type="backup"} 123 borgmatic_last_action_timestamp{type="compact"} 123 borgmatic_last_action_timestamp{type="check"} 123 borgmatic_last_action_timestamp{type="prune"} 123 ```

witten commented

2025-05-05 20:22:38 +00:00

Owner

Makes sense!