From my perspective, a significant weakness of borg is, that no logging is done. This must be handled by the wrapper scripts.
As borgmatic is one of them, I write here :-)
As far as I see, borgmatic is also not writing logs. I would suggest two logs:
One that logs each run of borgmatic and shows top level information (one line!):
Date, Success, Folder, Size, Transferred Data
And another (!) logfile per run of borgmatic that holds all information that borg threw on the stdout and stderr.
What is your view on this?
Something like this seems pretty useful, yes. You can simulate it now by redirecting borgmatic -v 2 output to your own log file. That’s not as good as real logging, but it’s a start.
borgmatic -v 2
I’m trying to implement logging but I’m not sure how to do it properly.
For now, I’m just writing json ouput to files:
I think the most relevant information we can use is the return code of each command (and the attached error if any)
However, I can’t find a way to catch return codes or this kind of error: Repository /tmp/borg002 does not exist.
Repository /tmp/borg002 does not exist.
What do you think ? Do you have any ideas of how we could properly handle logging ?
While I haven’t spend much time looking into it, my first thought was to use something like the Python syslog module to add support for (optional) logging to syslog. That way, we don’t have to build all of our own support for: multiple-file logging, log rotation, log levels, etc. We can just log everything to syslog and let decades of Unix technology handle the rest.
In terms of what to log, my inclination is: Everything. Pick a log level (such as INFO or DEBUG or make it configurable) and set up a Python log handler to just log all of the existing logger.whatever() messages to syslog. This would include the return code of each command and any errors. The benefit of using the existing Python logging system is that we don’t have to log twice for each of: file logging, and standard out/error logging. We just log once and let the configured log handlers send everything to the right places.
One way to catch return codes would be to use the existing subprocess.check_call() or the new Python 3.5+ subprocess.run(check=True, ...) and catch any exception, pulling the exit code off of the caught subprocess.CalledProcessError instance, specifically the returncode attribute.
I could also see in the future supporting other logging backends beside just syslog. But let’s not get ahead of ourselves.
Lastly, you should be aware of some related efforts that may affect logging: #94 and #100. Hope this helps. Let me know if you have any other questions.
I understand your thoughts.
The problem I see is:
Would you see that borgmatic has thrown an error at each backup during the last four weeks if it is hidden in hunderts of other lines of messages.
I suggest to use https://docs.python.org/3/library/logging.handlers.html#module-logging.handlers
The idea here is, that you use the python module ‘logger’ (as implemented already) and then you or the user can later decide whether to log to a file or to syslog by chosing the handler.
But what I really need is a simple piece of information:
1) When was the last successful backup
2) How many unsuccessful backups did I have since then
And this, I need for each target in one always up to date file.
This is not ‘logging’ but more a ‘cockpit’ if you like.
Use this for Inspiration https://i1.wp.com/www.accuratereviews.com/wordpress/wp-content/uploads/2015/09/crashplan.jpg
(I am not asking for a GUI. Just look for the content)
Maybe the “last successful backup” part you’re looking for is what this ticket talks about? #86
Do you think if that ticket was expanded to also include “how many unsuccessful backups did I have since then”, then your “cockpit” needs would be met?
I’m not trying to get out of adding real logging support to borgmatic. I’m just wondering if a more targeted feature (e.g., that linked ticket) would get closer to what you’re looking for.
yes, that comes close to what I have in mind and I am happy with splitting the issue here (#52) to the pure logging and continuing the discussion on the cockpit in #86 or in a new issue.
No due date set.
This issue currently doesn't have any dependencies.
Deleting a branch is permanent. It CANNOT be undone. Continue?