#53 Add logging

Open
opened 1 year ago by henfri · 7 comments
henfri commented 1 year ago

From my perspective, a significant weakness of borg is, that no logging is done. This must be handled by the wrapper scripts. As borgmatic is one of them, I write here :-)

As far as I see, borgmatic is also not writing logs. I would suggest two logs: One that logs each run of borgmatic and shows top level information (one line!): Date, Success, Folder, Size, Transferred Data

And another (!) logfile per run of borgmatic that holds all information that borg threw on the stdout and stderr.

What is your view on this?

Greetings,

Hendrik

From my perspective, a significant weakness of borg is, that no logging is done. This must be handled by the wrapper scripts. As borgmatic is one of them, I write here :-) As far as I see, borgmatic is also not writing logs. I would suggest two logs: One that logs each run of borgmatic and shows top level information (one line!): Date, Success, Folder, Size, Transferred Data And another (!) logfile per run of borgmatic that holds all information that borg threw on the stdout and stderr. What is your view on this? Greetings, Hendrik
witten commented 1 year ago
Owner

Something like this seems pretty useful, yes. You can simulate it now by redirecting borgmatic -v 2 output to your own log file. That’s not as good as real logging, but it’s a start.

Something like this seems pretty useful, yes. You can simulate it now by redirecting `borgmatic -v 2` output to your own log file. That's not as good as real logging, but it's a start.
jk commented 3 months ago

Hello,

I’m trying to implement logging but I’m not sure how to do it properly.

For now, I’m just writing json ouput to files:

  • --create output goes to a new file every run (to get history of backups created)
  • --info / --list output overwrite a file every run (to get the current state of the repo)

I think the most relevant information we can use is the return code of each command (and the attached error if any)

However, I can’t find a way to catch return codes or this kind of error: Repository /tmp/borg002 does not exist.

What do you think ? Do you have any ideas of how we could properly handle logging ?

Hello, I'm trying to implement logging but I'm not sure how to do it properly. For now, I'm just writing json ouput to files: - --create output goes to a new file every run (to get history of backups created) - --info / --list output overwrite a file every run (to get the current state of the repo) I think the most relevant information we can use is the return code of each command (and the attached error if any) However, I can't find a way to catch return codes or this kind of error: `Repository /tmp/borg002 does not exist.` What do you think ? Do you have any ideas of how we could properly handle logging ?
witten commented 3 months ago
Owner

While I haven’t spend much time looking into it, my first thought was to use something like the Python syslog module to add support for (optional) logging to syslog. That way, we don’t have to build all of our own support for: multiple-file logging, log rotation, log levels, etc. We can just log everything to syslog and let decades of Unix technology handle the rest.

In terms of what to log, my inclination is: Everything. Pick a log level (such as INFO or DEBUG or make it configurable) and set up a Python log handler to just log all of the existing logger.whatever() messages to syslog. This would include the return code of each command and any errors. The benefit of using the existing Python logging system is that we don’t have to log twice for each of: file logging, and standard out/error logging. We just log once and let the configured log handlers send everything to the right places.

One way to catch return codes would be to use the existing subprocess.check_call() or the new Python 3.5+ subprocess.run(check=True, ...) and catch any exception, pulling the exit code off of the caught subprocess.CalledProcessError instance, specifically the returncode attribute.

I could also see in the future supporting other logging backends beside just syslog. But let’s not get ahead of ourselves.

Lastly, you should be aware of some related efforts that may affect logging: #94 and #100. Hope this helps. Let me know if you have any other questions.

While I haven't spend much time looking into it, my first thought was to use something like the Python [syslog module](https://docs.python.org/3/library/syslog.html) to add support for (optional) logging to syslog. That way, we don't have to build all of our own support for: multiple-file logging, log rotation, log levels, etc. We can just log everything to syslog and let decades of Unix technology handle the rest. In terms of what to log, my inclination is: Everything. Pick a log level (such as `INFO` or `DEBUG` or make it configurable) and set up a Python log handler to just log all of the existing `logger.whatever()` messages to syslog. This would include the return code of each command *and* any errors. The benefit of using the existing Python logging system is that we don't have to log twice for each of: file logging, and standard out/error logging. We just log once and let the configured log handlers send everything to the right places. One way to catch return codes would be to use the existing `subprocess.check_call()` or the new Python 3.5+ `subprocess.run(check=True, ...)` and catch any exception, pulling the exit code off of the caught [`subprocess.CalledProcessError`](https://docs.python.org/3/library/subprocess.html#subprocess.CalledProcessError) instance, specifically the `returncode` attribute. I could also see in the future supporting other logging backends beside just syslog. But let's not get ahead of ourselves. Lastly, you should be aware of some related efforts that may affect logging: #94 and #100. Hope this helps. Let me know if you have any other questions.
henfri commented 1 month ago
Poster

Hello,

I understand your thoughts. The problem I see is: Would you see that borgmatic has thrown an error at each backup during the last four weeks if it is hidden in hunderts of other lines of messages.

I suggest to use https://docs.python.org/3/library/logging.handlers.html#module-logging.handlers

The idea here is, that you use the python module ‘logger’ (as implemented already) and then you or the user can later decide whether to log to a file or to syslog by chosing the handler.

But what I really need is a simple piece of information: 1) When was the last successful backup 2) How many unsuccessful backups did I have since then

And this, I need for each target in one always up to date file. This is not ‘logging’ but more a ‘cockpit’ if you like. Use this for Inspiration https://i1.wp.com/www.accuratereviews.com/wordpress/wp-content/uploads/2015/09/crashplan.jpg (I am not asking for a GUI. Just look for the content)

Regards, Hendrik

Hello, I understand your thoughts. The problem I see is: Would you see that borgmatic has thrown an error at each backup during the last four weeks if it is hidden in hunderts of other lines of messages. I suggest to use https://docs.python.org/3/library/logging.handlers.html#module-logging.handlers The idea here is, that you use the python module 'logger' (as implemented already) and then you or the user can later decide whether to log to a file or to syslog by chosing the handler. But what I really need is a simple piece of information: 1) When was the last successful backup 2) How many unsuccessful backups did I have since then And this, I need for each target in one always up to date file. This is not 'logging' but more a 'cockpit' if you like. Use this for Inspiration https://i1.wp.com/www.accuratereviews.com/wordpress/wp-content/uploads/2015/09/crashplan.jpg (I am not asking for a GUI. Just look for the content) Regards, Hendrik
witten commented 1 month ago
Owner

Maybe the “last successful backup” part you’re looking for is what this ticket talks about? #86

Do you think if that ticket was expanded to also include “how many unsuccessful backups did I have since then”, then your “cockpit” needs would be met?

I’m not trying to get out of adding real logging support to borgmatic. I’m just wondering if a more targeted feature (e.g., that linked ticket) would get closer to what you’re looking for.

Maybe the "last successful backup" part you're looking for is what this ticket talks about? https://projects.torsion.org/witten/borgmatic/issues/86 Do you think if that ticket was expanded to also include "how many unsuccessful backups did I have since then", then your "cockpit" needs would be met? I'm not trying to get out of adding real logging support to borgmatic. I'm just wondering if a more targeted feature (e.g., that linked ticket) would get closer to what you're looking for.
henfri commented 1 month ago
Poster

Hello,

yes, that comes close to what I have in mind and I am happy with splitting the issue here (#52) to the pure logging and continuing the discussion on the cockpit in #86 or in a new issue.

Regards, Hendrik

Hello, yes, that comes close to what I have in mind and I am happy with splitting the issue here (#52) to the pure logging and continuing the discussion on the cockpit in #86 or in a new issue. Regards, Hendrik
witten commented 1 month ago
Owner

Sounds good!

Sounds good!
Sign in to join this conversation.
No Milestone
No Assignees
3 Participants
Due Date

No due date set.

Dependencies

This issue currently doesn't have any dependencies.

Loading…
Cancel
Save
There is no content yet.