With multiple config files in borgmatic.d, cumulative logs are sent to the healthchecks endpoints #328
Labels
No Label
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
No Milestone
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: borgmatic-collective/borgmatic#328
Loading…
Reference in New Issue
No description provided.
Delete Branch "%!s(<nil>)"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Apologies for the potentially unclear bug title (and thank you very much for your work on this excellent tool!)
What I'm trying to do and why
I have multiple config files in
/etc/borgmatic.d
. Each config file has its own uniquehooks.healthchecks
configuration.Actual behavior (if a bug)
Assuming all borg runs / hooks generate output that is
POST
ed to healthchecks, each subsequent healthcheck ping will include the previous logs and its own output (if any).Expected behavior (if a bug)
Each submission to healthchecks should consist of the data generated by that borg run itself.
Other notes
With three borgmatic config files in
/etc/borgmatic.d
that echo a string in thebefore_backup
andafter_backup
hooks, healthchecks gets the following payload from the last (3rd) backup job when running borgmatic without specifying a config file with the--config
parameter:The first job sent the first line to its healthchecks endpoint. The second job sent the first three lines. The third job sent the five lines shown here. The
after_backup
text of the third job was swallowed.If I run the first job by itself using
borgmatic --config /etc/borgmatic.d/conf.yaml
only thebefore_backup
hook's text is sent to healthchecks.Environment
borgmatic version: 1.5.6
borgmatic installation method:
pip3 install --user borgmatic
Borg version: 1.1.5
Python version: Python 3.6.9
operating system and version: Ubuntu 16.04
I appreciate the bug report. I think I know what's going on here, but the way to fix it isn't entirely obvious:
after_backups
output getting omitted from what's sent to Healthchecks: Since the implementation of #292 the Healthchecks "finish" ping (which is what sends accumulated logs to Healthchecks) was moved to happen before theafter_backups
hook, meaning that any output from theafter_backups
hook doesn't get sent to Healthchecks.The rationale is that #292 moved the Healthchecks "start" ping to happen after the
before_backups
hook, and so it made sense for the before/after hooks to effectively "wrap" the Healthchecks interactions. The current ordering is: 1. run before hooks, 2. do Healthchecks start ping, 3. run actions, 4. do Healthchecks finish ping (and transmit logs), 5. run after hooks.I could swap the order of the Healthchecks finish and the after hooks, but that might be a little unexpected, given the order of the before hooks and Healthchecks start. Thoughts? Do you need to see the output of the
after_backups
hook on Healthchecks?The reason that you're still seeing the output of the
before_backup
output is that the Healthchecks log is initialized even beforebefore_backup
runs, so its output is still captured and sent to Healthchecks... Which leads me to the second problem you're seeing:Healthchecks logs from previous borgmatic configuration file runs bleed over to subsequent configuration files: When the first configuration file initializes Healthchecks logging, that same logger ends up implicitly used for each subseqent configuration file, and the logger never gets cleared. That's why you're seeing output for all configuration files sent for the final "job".
The fix here is either to clear logs upon sending a finish ping to Healthchecks, or implement the concept of hook cleanup to correspond to the existing hook initialization.
This should be fixed in master. I'm now destroying the Healthchecks logging handler after each configuration file is processed, so it doesn't carry over to the logs of the next configuration file.
I've also moved the Healthchecks "finish" ping to after borgmatic's
after_*
hooks, so their output should now get sent to Healthchecks.Thanks again for reporting. I post here when it's released.
Released in borgmatic 1.5.7!