Send --stats output to healthchecks #524
Loading…
x
Reference in New Issue
Block a user
No description provided.
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What I'm trying to do and why
Right now when I ping Healthchecks, it only sends the output of the actions specificed in the hooks: section to the healthchecks log. if the --stats option is specified, it would be very nice to send this as log output to Healthchecks... much more helpful than seeing the output of my MongoDB dump script.
Send summary to healthchecksto Send --stats output to healthchecksThanks for filing this! Have you tried tried tweaking your
--monitoring-verbosity
? For instance:borgmatic create --monitoring-verbosity 2
?I wasn't aware of that... I had just been piping the shell output to a log file. After playing with it I still have a few questions.
I like to use the --files option on for logging so I can go back and see if a particular file is being backed up as expected.
it seems that mointoring-verbosity=0 supresses everything including the --stats output. monitoring-verbosity=1 shows both stats and the complete file listing, which I don't want transmitted for privacy reasons.
Is there a way to suppress the changed file listing in the monitor log?
Also I don't know if it's a bug but the following commands produce different monitoring output:
edit: Is there a difference between level -1 and level 0 logging?
Not without suppressing it entirely (e.g., ommitting
--files
). That's because--files
and--stats
output are currently logged at the same log level.Some of this might be "by design" or at least a side-effect of the design. Since Borg doesn't have the concept of multiple logging backends (file, monitoring, etc.), borgmatic determines the highest log level requested (so, "2" in the second command) and in turn requests that log level of Borg. Therefore, the resultant logs sent to monitoring may be more verbose than they would be if the highest log level was "1".
It seems like the Borgmatic -> Borg log levels are generally:
Level -1: Nothing
Level 0: CRITCAL + ERROR + WARNING
Level 1: CRITCAL + ERROR + WARNING + INFO
Level 2: CRITCAL + ERROR + WARNING + INFO + DEBUG
But some stuff like stats or pruning details change from INFO to WARNING depending on the requested log level. And the --verbosity, --log-file-verbosity, and --monitoring-verbosity options are not independent... for instance changing the monitoring verbosity changes what's shown in the console also.
The only implementation suggestion/change I would suggest would be to set the log level to DEBUG in the underlying Borg command, then filter the output to the various log backends, maybe per the table in my previous post.
It's something to consider as a logging enhancement, independent of the request to send the stats and not files to the monitoring back-end. It would certainly make the behavior much easier to understand.
The file/stats question may be a request for Borg to reclassify --files as DEBUG output and --stats as INFO output.
For what it's worth, I think I'm going to migrate to a self-hosted Healthchecks implementation. That way I don't care if I send the more verbose logs including personal info.
Is it possible to have the logs not append to each other? My cron job automatically rolls the log files weekly, but it doesn't work if the logs are appended instead of overwritten.
borgmatic prune create --log-file /mnt/logs/backup-"
date +"%a-%H"
".logI don't really need my logs hanging around forever...
That's correct, with the exception that level "-1" is critical + errors only.
This is done very intentionally, although you could perhaps call it a hack. The behavior is:
--stats
command-line flag.If you have ideas for a better approach given these constraints, I'm open!
Yup, that's due to the logic described above.
Could you talk a little more about how that might work? Individual lines of the output that borgmatic receives from Borg are log-level-less. borgmatic simply requests either
--debug
or--info
or whatever from Borg.. and then receives plain ol' stdout/stderr back.. and that's it. There's no parsing of the output that comes back beyond that.Makes sense!
What's the problem specifically with appending? How does it break log rotation? Thanks.
I have seen this behaviour (sending just stats) from
b3vis/borgmatic
Docker container, but I've not been able to replicate it myself yet.The crontab entry doesn't look that out of the ordinary to me, so I'm unsure where it's coming from:
53 */4 * * * PATH=$PATH:/usr/bin /usr/bin/borgmatic --stats -v 0 2>&1
The issue I was having with appending logfiles is that my cron job was piping console output to log files named "backup-Mon", "backup-Tue", etc. that were overwritten the following week just by nature of having the same name. Using the --log-file option causes the output to append, so I have 7 logfilles that just grow forever.
To solve the logfile rotation issue, I had theborgmatic docker container maintainer (b3vis) add "logrotate" to the container. Now my cron job looks like "logrotate --opts; borgmatic --opts" where I'm simply using "backup.log" as my backup log filename. This seems like a proper resolution to this issue, though an option for "--overwrite-logs" might be nice.
The idea I had for the log file verbosity was to have the maximum detail coming out of Borg at all times, then filter the output to screen/syslog/file-log/monitor-log per the Borgmatic command-line options.
I've solved this issue by writing my own hc-ping script and disabling the built-in HC feature in the Borgmatic config file. My Borgmatic command is using a relatively verbose setting, then this "hc-ping" script filters the output down before posting to Healthchecks. My config.yaml has start/stop/error hooks similar to: "hc-ping -c finish -u XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX -l /mnt/logs/backup.log"
If this explicit filtering were built into Borgmatic, I could revert to the built-in HC support. (I want add/modify file details in my log files, but I don't want to send that level of detail to Healthchecks. In my instance, I'm self-hosting Healthchecks so my $hc_base is different. Splitting out the $curl_cmd here has another benefit in that I can specify a proxy, or even use "ssh user@host /usr/bin/curl" if my borgmatic container doesn't have internet access.
#518
#512
#525
Other related recent issues that might find this custom script useful.
Glad to hear that you've got logrotate going to solve the logging issue. I would prefer to lean on external tools like logrotate for that rather than putting that functionality into borgmatic. Also note that you could use a logging daemon like systemd, which borgmatic supports natively.. although I'm not sure the Docker image does. And even with the Docker image, I imagine logs (if not redirected to file) go to Docker's own logging system.
I'm also glad to hear you have a work-around for the Healthchecks output issue!
Could you say a little more about how that would work? With
borgmatic --verbosity 2
now, the maximum detail will come out of Borg. But then how would the output get filtered further?Thanks!
Closing this for now, but feel free to follow up. We can always reopen it. Thanks!