borgmatic-collective/borgmatic
borgmatic-collective
/
borgmatic
15
57
Fork 40
Code Issues 54 Pull Requests 1 Actions Packages Releases 121 Activity

Give access to results details in case of succeeded archive creation #385

New Issue
Closed
opened 2021-01-16 15:30:00 +00:00 by fpoulain · 5 comments
fpoulain commented 2021-01-16 15:30:00 +00:00
Contributor
Copy Link

What I'm trying to do and why

I am monitoring borgmatics logs using icinga and a python script (small log parser).

In order to be reliable, I made use of borgmatic hooks.

I would like to get extra infos, like e.g. duration of the archive creation. It would be nice to get all ouput or the creation sumary in hooks variables.

Currently, only on_error hooks is getting extra infos (output, error).

Other notes / implementation ideas

I suggest making available:

  • configuration_filename
  • output
  • summary

It would be great if summary is parsed into variables:

  • archive_name
  • archive_fingerprint
  • start_time
  • end_time
  • duration
  • number_of_files
#### What I'm trying to do and why I am monitoring borgmatics logs using icinga and a python script (small log parser). In order to be reliable, I made use of borgmatic hooks. I would like to get extra infos, like e.g. duration of the archive creation. It would be nice to get all ouput or the creation sumary in hooks variables. Currently, only `on_error` hooks is getting extra infos (output, error). #### Other notes / implementation ideas I suggest making available: * configuration_filename * output * summary It would be great if summary is parsed into variables: * archive_name * archive_fingerprint * start_time * end_time * duration * number_of_files
witten commented 2021-01-16 17:24:01 +00:00
Owner
Copy Link

Thank you for filing this. The output and summary are currently sent to the logging system (syslog or journalctl). Is the problem that it's just not convenient or reliable to parse them from there?

As for the duration of archive creation and similar data, have you considered running borgmatic with the --json flag? That of course won't make the data available to hooks, but it does return easily parseable info like archive duration.

Let me know your thoughts!

Thank you for filing this. The output and summary are currently sent to the logging system (`syslog` or `journalctl`). Is the problem that it's just not convenient or reliable to parse them from there? As for the duration of archive creation and similar data, have you considered running borgmatic with the `--json` flag? That of course won't make the data available to hooks, but it does return easily parseable info like archive duration. Let me know your thoughts!
palto42 commented 2021-01-17 09:15:47 +00:00
Contributor
Copy Link

I'm using a small script which parses journalctl as per my needs after borgmatic completed the backup. It mainly filters on keywords like "Successful" plus a list of desired log level data like "WARNING" or "ERROR".

I'm running the script as "ExecStopPost" in the systemd service, but you can also start it via the "after_backup" hook from Borgmatic itself.

Another method I used previously was to write the log in parallel to a temporary file (with separate log level), but then I still wanted some post processing/filter so that I now simply use the journalctl.

I'm using a small script which parses journalctl as per my needs after borgmatic completed the backup. It mainly filters on keywords like "Successful" plus a list of desired log level data like "WARNING" or "ERROR". I'm running the script as "ExecStopPost" in the systemd service, but you can also start it via the "after_backup" hook from Borgmatic itself. Another method I used previously was to write the log in parallel to a temporary file (with separate log level), but then I still wanted some post processing/filter so that I now simply use the journalctl.
fpoulain commented 2021-01-17 13:44:11 +00:00
Author
Contributor
Copy Link

As for the duration of archive creation and similar data, have you considered running borgmatic with the --json flag? That of course won't make the data available to hooks, but it does return easily parseable info like archive duration.

Well, I see. Actually I didn't throught about calling borgmatic with the --json flag. It gives the expected result.

E.g. I can get many info using:

borgmatic --info --archive latest --json

And if I am only interested in durations I can get those with:

borgmatic --info --archive latest --json | jq '[.[] | .archives | last | .duration ]'

Currently it appears that it is ok to call borgmatic --info or borgmatic --list into a hook. Do you advise against it ?

> As for the duration of archive creation and similar data, have you considered running borgmatic with the `--json` flag? That of course won't make the data available to hooks, but it does return easily parseable info like archive duration. Well, I see. Actually I didn't throught about calling borgmatic with the `--json` flag. It gives the expected result. E.g. I can get many info using: ``` borgmatic --info --archive latest --json ``` And if I am only interested in durations I can get those with: ``` borgmatic --info --archive latest --json | jq '[.[] | .archives | last | .duration ]' ``` Currently it appears that it is ok to call `borgmatic --info` or `borgmatic --list` into a hook. Do you advise against it ?
witten commented 2023-03-27 04:08:07 +00:00
Owner
Copy Link

I apologize for the super lengthy delay in getting back to this. I generally don't recommend calling borgmatic from borgmatic (especially because of the infinite loop risk), but in this case I don't see a better option given the current feature set.

Assuming this is still a relevant issue to you, how can/does your monitoring software consume information from the software it's monitoring? For instance, if borgmatic had a built-in feature to write some of the stats you're looking for to a JSON file (as in #617), would that be easier to monitor?

I apologize for the super lengthy delay in getting back to this. I generally don't recommend calling borgmatic from borgmatic (especially because of the infinite loop risk), but in this case I don't see a better option given the current feature set. Assuming this is still a relevant issue to you, how can/does your monitoring software consume information from the software it's monitoring? For instance, if borgmatic had a built-in feature to write some of the stats you're looking for to a JSON file (as in #617), would that be easier to monitor?
witten commented 2023-06-28 18:48:50 +00:00
Owner
Copy Link

I'm closing this one for now due to inactivity, but I'd be happy to open it up again if you have further thoughts. Thank you!

I'm closing this one for now due to inactivity, but I'd be happy to open it up again if you have further thoughts. Thank you!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
main
spot-check
1.8.10
1.8.9
1.8.8
1.8.7
1.8.6
1.8.5
1.8.4
1.8.3
1.8.2
1.8.1
1.8.0
1.7.15
1.7.14
1.7.13
1.7.12
1.7.11
1.7.10
1.7.9
1.7.8
1.7.7
1.7.6
1.7.5
1.7.4
1.7.3
1.7.2
1.7.1
1.7.0
1.6.6
1.6.5
1.6.4
1.6.3
1.6.2
1.6.1
1.6.0
1.5.24
1.5.23
1.5.22
1.5.21
1.5.20
1.5.19
1.5.18
1.5.17
1.5.16
1.5.15
1.5.14
1.5.13
1.5.12
1.5.11
1.5.10
1.5.9
1.5.8
1.5.7
1.5.7.dev0
1.5.6
1.5.5
1.5.4
1.5.3
1.5.2
1.5.1
1.5.0
1.4.22
1.4.21
1.4.20
1.4.19
1.4.18
1.4.17
1.4.16
1.4.15
1.4.14
1.4.13
1.4.12
1.4.11
1.4.10
1.4.9
1.4.8
1.4.7
1.4.6
1.4.5
1.4.4
1.4.3
1.4.2
1.4.1
1.4.0
1.3.26
1.3.25
1.3.24
1.3.23
1.3.22
1.3.21
1.3.20
1.3.19
1.3.18
1.3.17
1.3.16
1.3.15
1.3.14
1.3.13
1.3.12
1.3.11
1.3.10
1.3.9
1.3.8
1.3.7
1.3.6
1.3.5
1.3.4
1.3.3
1.3.2
1.3.1
1.3.0
1.2.18
1.2.17
1.2.16
1.2.15
1.2.14
1.2.13
1.2.12
1.2.11
1.2.10
1.2.9
1.2.8
1.2.7
1.2.6
1.2.5
1.2.4
1.2.3
1.2.2
1.2.1
1.2.0
1.1.15
1.1.14
1.1.13
1.1.12
1.1.11
1.1.10
1.1.9
1.1.7
1.1.6
1.1.5
1.1.4
1.1.3
1.1.2
1.1.1
1.1.0
1.0.3
1.0.2
1.0.1
1.0.0
0.1.8
0.1.7
0.1.6
0.1.5
0.1.4
0.1.3
0.1.2
0.1.1
0.1.0
0.0.7
0.0.6
0.0.5
0.0.4
0.0.3
0.0.2
Labels
Clear labels   
bug

   
data loss

   
design finalized

   
good first issue

   
new feature area

   
question / support

   
security

   
waiting for response
No Label
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
3 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#385
No description provided.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?