Tune verbose check output to be suitable for email report #31

New Issue

import_bot · 2018-01-05T05:34:16Z

import_bot commented

2018-01-05 05:34:16 +00:00

Hi I'm using borgmatic with systemd on archlinux. Unfortunately it's not so easy to convince systemd to send an email after running the backup job ... so I wonder if you would consider adding notification functionality to borgmatic?

Imported from Taiga issue 30 (to do). Created on 2017-07-06T15:26:08+0000 by Markus Hubig.

Hi I'm using borgmatic with systemd on archlinux. Unfortunately it's not so easy to convince systemd to send an email after running the backup job ... so I wonder if you would consider adding notification functionality to borgmatic? --- Imported from Taiga issue 30 (to do). Created on 2017-07-06T15:26:08+0000 by Markus Hubig.

import_bot commented

2018-01-05 05:34:16 +00:00

What sort of notifications are you looking for? Completion? Only failure? And what content would you expect to see in the emails?

Comment on 2017-07-23T05:00:52+0000 by Dan Helfman.

What sort of notifications are you looking for? Completion? Only failure? And what content would you expect to see in the emails? --- Comment on 2017-07-23T05:00:52+0000 by Dan Helfman.

import_bot commented

2018-01-05 05:34:17 +00:00

Failure and Completion. The -v 1 output would be sufficient.

A really cool option would be to be able to only send failure notifications, but to be able to run borgmatic in a check only mode where it would not make a backup, but instead sends a notification containing the -v 1 or -v 2 output.

This way I could schedule it to make the daily backups and notify me on failures. And once a week I would get an overview on the state of my backup archive ...

Comment on 2017-07-25T16:13:08+0000 by Markus Hubig.

Failure and Completion. The `-v 1` output would be sufficient. A really cool option would be to be able to only send failure notifications, but to be able to run `borgmatic` in a `check only mode` where it would not make a backup, but instead sends a notification containing the `-v 1` or `-v 2` output. This way I could schedule it to make the daily backups and notify me on failures. And once a week I would get an overview on the state of my backup archive ... --- Comment on 2017-07-25T16:13:08+0000 by Markus Hubig.

import_bot commented

2018-01-05 05:34:17 +00:00

Interesting idea.. Might have some overlap with https://tree.taiga.io/project/witten-borgmatic/issue/17 .. I've mentioned it there.

Comment on 2017-07-29T03:45:28+0000 by Dan Helfman.

Interesting idea.. Might have some overlap with https://tree.taiga.io/project/witten-borgmatic/issue/17 .. I've mentioned it there. --- Comment on 2017-07-29T03:45:28+0000 by Dan Helfman.

import_bot commented

2018-01-05 05:34:17 +00:00

Check-only mode is implemented in borgmatic 1.1.4 with three new flags: "--prune", "--create", and "--check". So you can configure one cron job with "--prune --create" and a separate job with a different frequency with just "--check". Or mix and match those flags how ever you like. Thanks for the suggestion!

I still haven't implemented the email portion of this ticket, however.

Comment on 2017-07-29T05:26:21+0000 by Dan Helfman.

Check-only mode is implemented in borgmatic 1.1.4 with three new flags: "--prune", "--create", and "--check". So you can configure one cron job with "--prune --create" and a separate job with a different frequency with just "--check". Or mix and match those flags how ever you like. Thanks for the suggestion! I still haven't implemented the email portion of this ticket, however. --- Comment on 2017-07-29T05:26:21+0000 by Dan Helfman.

import_bot commented

2018-01-05 05:34:18 +00:00

I'll tried those variants:

$ borgmatic --check -v 1 |mail -s "Backup archive consistency check" mh@example.com
$ borgmatic --check -v 2 |mail -s "Backup archive consistency check" mh@example.com
$ borgmatic --check |mail -s "Backup archive consistency check" mh@example.com

but found that -v 1 and -v 2 produce almost the same output and the third variant produces no output at all.

Starting repository check
Checking segments 0.0%
[... skipping 1000 lines ...]
Checking segments 99.9%
Completed repository check, no problems found.
Starting archive consistency check...
Analyzing archive fileserver-2017-07-31T22:00:25.341480 (10/10)
Analyzing archive fileserver-2017-07-28T22:00:22.798839 (9/10)
Analyzing archive fileserver-2017-07-27T22:00:23.321103 (8/10)
Analyzing archive fileserver-2017-07-26T22:00:07.470558 (7/10)
Analyzing archive fileserver-2017-07-25T22:00:19.408762 (6/10)
Analyzing archive fileserver-2017-07-24T22:00:14.371046 (5/10)
Analyzing archive fileserver-2017-07-21T22:00:17.047039 (4/10)
Analyzing archive fileserver-2017-07-20T22:00:17.048877 (3/10)
Analyzing archive fileserver-2017-07-14T22:00:17.035007 (2/10)
Analyzing archive fileserver-2017-07-07T22:00:30.853421 (1/10)
Archive consistency check complete, no problems found.

Maybe you can detect if you have a terminal and if not just skip the Checking segments stuff?
And Maybe you could add some more checks to -v 2, like archive size and free storage?

But ultimately I think you don't need to add email support since piping through mail is sufficient!

Comment on 2017-08-01T14:34:55+0000 by Markus Hubig.

I'll tried those variants: ``` $ borgmatic --check -v 1 |mail -s "Backup archive consistency check" mh@example.com $ borgmatic --check -v 2 |mail -s "Backup archive consistency check" mh@example.com $ borgmatic --check |mail -s "Backup archive consistency check" mh@example.com ``` but found that `-v 1` and `-v 2` produce almost the same output and the third variant produces no output at all. ``` Starting repository check Checking segments 0.0% [... skipping 1000 lines ...] Checking segments 99.9% Completed repository check, no problems found. Starting archive consistency check... Analyzing archive fileserver-2017-07-31T22:00:25.341480 (10/10) Analyzing archive fileserver-2017-07-28T22:00:22.798839 (9/10) Analyzing archive fileserver-2017-07-27T22:00:23.321103 (8/10) Analyzing archive fileserver-2017-07-26T22:00:07.470558 (7/10) Analyzing archive fileserver-2017-07-25T22:00:19.408762 (6/10) Analyzing archive fileserver-2017-07-24T22:00:14.371046 (5/10) Analyzing archive fileserver-2017-07-21T22:00:17.047039 (4/10) Analyzing archive fileserver-2017-07-20T22:00:17.048877 (3/10) Analyzing archive fileserver-2017-07-14T22:00:17.035007 (2/10) Analyzing archive fileserver-2017-07-07T22:00:30.853421 (1/10) Archive consistency check complete, no problems found. ``` * Maybe you can detect if you have a terminal and if not just skip the `Checking segments` stuff? * And Maybe you could add some more checks to `-v 2`, like archive size and free storage? But ultimately I think you don't need to add email support since piping through mail is sufficient! --- Comment on 2017-08-01T14:34:55+0000 by Markus Hubig.

import_bot commented

2018-01-05 05:34:18 +00:00

Here are the systemd files I'm using:

borgmatic-check.service

[Unit]
Description=borgmatic backup archive consistency check

[Service]
Type=oneshot
ExecStart=/bin/bash -c 'exec /usr/bin/borgmatic --check -v 2 |mail -s "Backup archive consistency check" mh@example.com'

borgmatic-check.timer

[Unit]
Description=Run borgmatic backup archive consistency check

[Timer]
OnCalendar=Sat *-*-* 22:00
Persistent=true

[Install]
WantedBy=timers.target

Comment on 2017-08-01T14:51:40+0000 by Markus Hubig.

Here are the systemd files I'm using: borgmatic-check.service ``` [Unit] Description=borgmatic backup archive consistency check [Service] Type=oneshot ExecStart=/bin/bash -c 'exec /usr/bin/borgmatic --check -v 2 |mail -s "Backup archive consistency check" mh@example.com' ``` borgmatic-check.timer ``` [Unit] Description=Run borgmatic backup archive consistency check [Timer] OnCalendar=Sat *-*-* 22:00 Persistent=true [Install] WantedBy=timers.target ``` --- Comment on 2017-08-01T14:51:40+0000 by Markus Hubig.

import_bot commented

2018-01-05 05:34:19 +00:00

Good idea. I've updated the title of the ticket accordingly. I'm not certain though how possible tweaking the output of "Checking segments", since that comes direct from Borg. Might be able to filter it anyway.

Comment on 2017-08-06T17:05:59+0000 by Dan Helfman.

Good idea. I've updated the title of the ticket accordingly. I'm not certain though how possible tweaking the output of "Checking segments", since that comes direct from Borg. Might be able to filter it anyway. --- Comment on 2017-08-06T17:05:59+0000 by Dan Helfman.

import_bot commented

2018-01-05 05:34:19 +00:00

FYI - The output of "Checking segments" is fixed in borgbackup 1.1.

See github.com/borgbackup/borg/issues/2201 and https://github.com/borgbackup/borg/issues/1590

Comment on 2017-10-14T12:54:51+0000 by Simon Dellenbach.

FYI - The output of "Checking segments" is fixed in borgbackup 1.1. See github.com/borgbackup/borg/issues/2201 and https://github.com/borgbackup/borg/issues/1590 --- Comment on 2017-10-14T12:54:51+0000 by Simon Dellenbach.

witten commented

2018-08-19 19:17:36 +00:00

Now that the Borg "checking segments" issue is fixed, and borgmatic has --list and --info flags to show individual and aggregate archive stats, respectively, I'm calling this ticket done! The one thing not supported is showing free disk space, but you can easily get that with df.

If there are other features that borgmatic doesn't support yet that would help out in email reports, please file follow-up tickets with details! Thanks.

Now that the Borg "checking segments" issue is fixed, and borgmatic has `--list` and `--info` flags to show individual and aggregate archive stats, respectively, I'm calling this ticket done! The one thing not supported is showing free disk space, but you can easily get that with `df`. If there are other features that borgmatic doesn't support yet that would help out in email reports, please file follow-up tickets with details! Thanks.

witten closed this issue

2018-08-19 19:17:36 +00:00

Sign in to join this conversation.