How can I disable timestamps on each line of borgmatic output #658

New Issue

benibilme · 2023-03-23T05:51:33Z

benibilme commented

2023-03-23 05:51:33 +00:00

I am using borgmatic 1.7.6 on manjaro machine. I log borgmatic output to a file and email it to me. For each line of borgmatic output there is a timestamp and ANSWER word. How can I get rid of this timestamps and the word "ANSWER". I do not need them. Is there an option to do it? I provide a sample below.

[2023-03-23 06:05:28,094] ANSWER: Time (start): Thu, 2023-03-23 06:04:52
[2023-03-23 06:05:28,094] ANSWER: Time (end): Thu, 2023-03-23 06:05:24
[2023-03-23 06:05:28,094] ANSWER: Duration: 32.19 seconds
[2023-03-23 06:05:28,094] ANSWER: Number of files: 37628
[2023-03-23 06:05:28,094] ANSWER: Utilization of max. archive size: 0%

I am using borgmatic 1.7.6 on manjaro machine. I log borgmatic output to a file and email it to me. For each line of borgmatic output there is a timestamp and ANSWER word. How can I get rid of this timestamps and the word "ANSWER". I do not need them. Is there an option to do it? I provide a sample below. [2023-03-23 06:05:28,094] ANSWER: Time (start): Thu, 2023-03-23 06:04:52 [2023-03-23 06:05:28,094] ANSWER: Time (end): Thu, 2023-03-23 06:05:24 [2023-03-23 06:05:28,094] ANSWER: Duration: 32.19 seconds [2023-03-23 06:05:28,094] ANSWER: Number of files: 37628 [2023-03-23 06:05:28,094] ANSWER: Utilization of max. archive size: 0%

diivi commented

2023-03-24 06:32:15 +00:00

For your particular use case, maybe you can use sed to strip this information from the file before you email it to yourself?

I tried this and it seems to work:

sed 's/^\[[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\} [0-9]\{2\}:[0-9]\{2\}:[0-9]\{2\},[0-9]\{3\}\] ANSWER: //' input_file > output_file

Otherwise, could you help me understand why you need to remove these prefixes?

For your particular use case, maybe you can use `sed` to strip this information from the file before you email it to yourself? I tried this and it seems to work: ```bash sed 's/^\[[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\} [0-9]\{2\}:[0-9]\{2\}:[0-9]\{2\},[0-9]\{3\}\] ANSWER: //' input_file > output_file ``` Otherwise, could you help me understand why you need to remove these prefixes?

witten commented

2023-03-24 19:26:26 +00:00

Yeah, as you've discovered, Python's logging includes two things on each log line besides the log message:

the log timestamp
the log level (in this case, "ANSWER", which is a custom log level used in borgmatic)

So does the sed filtering @diivi suggested work for your use case? Or do you need finer-grained control on what actually gets logged? Understanding your use case / how you want to consume these logs and why would help. Thanks!

Yeah, as you've discovered, Python's logging includes two things on each log line besides the log message: * the log timestamp * the log level (in this case, "ANSWER", which is a custom log level used in borgmatic) So does the `sed` filtering @diivi suggested work for your use case? Or do you need finer-grained control on what actually gets logged? Understanding your use case / how you want to consume these logs and why would help. Thanks!

witten added the

waiting for response

label 2023-03-28 22:40:58 +00:00

benibilme commented

2023-03-29 06:25:53 +00:00

Thank you all for the reply. Sorry about not being able to the respond quickly. I wanted the test the command you provided before respond.

I receive mails from my server and checking these borgmatic logs of many repositories on mobile phone screen is difficult. I often have to change the screen orientation to read the mail, but also scroll to see the whole lines. I am over fifty and my eyes are not as good as they used to be. Because of that I needed this.

Thank you providing the regular expression, it would have taken sometime for me to get it right.

I tested the regular expression. It works. I little bit enhanced it to include all similar time stamps and INFO logging level as well. My modified version is as follows. Resulting output is more terse and understandable for me.

I hope future changes in borgmatic does not break it.

 cat $LOGFILE | sed -E 's/^\[[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}\]//' | sed -E 's/ANSWER:|INFO://' | mail -s "$LOGFILE" me@me.com

One more thing, I knew, I could have filter the output with sed/awk, as you suggested, but I asked to see if there were a way to do it in borgmatic. I know that these are just format issues for the logging library and maybe similar to passing raw borg commands with --borg option, it is possible to pass user supplied raw log formats to the logging library. I do not code in python, but setting log format is almost universally available feature in every logging library in any language that I had encountered, and I used quite many languages. I bet python stdlib logging has it as well.

Thanks again for replies.

Thank you all for the reply. Sorry about not being able to the respond quickly. I wanted the test the command you provided before respond. I receive mails from my server and checking these borgmatic logs of many repositories on mobile phone screen is difficult. I often have to change the screen orientation to read the mail, but also scroll to see the whole lines. I am over fifty and my eyes are not as good as they used to be. Because of that I needed this. Thank you providing the regular expression, it would have taken sometime for me to get it right. I tested the regular expression. It works. I little bit enhanced it to include all similar time stamps and INFO logging level as well. My modified version is as follows. Resulting output is more terse and understandable for me. I hope future changes in borgmatic does not break it. ```` cat $LOGFILE | sed -E 's/^\[[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}\]//' | sed -E 's/ANSWER:|INFO://' | mail -s "$LOGFILE" me@me.com ```` One more thing, I knew, I could have filter the output with sed/awk, as you suggested, but I asked to see if there were a way to do it in borgmatic. I know that these are just format issues for the logging library and maybe similar to passing raw borg commands with --borg option, it is possible to pass user supplied raw log formats to the logging library. I do not code in python, but setting log format is almost universally available feature in every logging library in any language that I had encountered, and I used quite many languages. I bet python stdlib logging has it as well. Thanks again for replies.

diivi commented

2023-03-29 10:38:57 +00:00

Glad it worked out!

I asked to see if there were a way to do it in borgmatic.

I don't think so.

I know that these are just format issues for the logging library and maybe similar to passing raw borg commands with --borg option, it is possible to pass user supplied raw log formats to the logging library.

Do you have an example of how you'd like to use this? In case we do end up implementing it, it would help to finalise a design.
Like CLI options, config options, where exactly would you like to supply a format for the logger and what it would consist of.

Also, I don't think the logging format would change a lot with new Borgmatic features, so you can keep this sed command running, and it might not cause problems. @witten will be able to tell you more though.

Thanks!

Glad it worked out! > I asked to see if there were a way to do it in borgmatic. I don't think so. > I know that these are just format issues for the logging library and maybe similar to passing raw borg commands with --borg option, it is possible to pass user supplied raw log formats to the logging library. Do you have an example of how you'd like to use this? In case we do end up implementing it, it would help to finalise a design. Like CLI options, config options, where exactly would you like to supply a format for the logger and what it would consist of. Also, I don't think the logging format would change a lot with new Borgmatic features, so you can keep this sed command running, and it might not cause problems. @witten will be able to tell you more though. Thanks!

witten commented

2023-03-29 23:25:57 +00:00

Yeah, there haven't been any chances to this log format since it was introduced.

Some logging details for your consideration though: In Python, log records have a particular set of supported data points, and log formatters typically take a format string that might look like: "{level} {msg}".

Yeah, there haven't been any chances to this log format since it was introduced. Some logging details for your consideration though: In Python, log records have a [particular set of supported data points](https://docs.python.org/3/library/logging.html#logging.LogRecord), and log formatters typically take a format string that might look like: "{level} {msg}".

benibilme commented

2023-04-02 22:36:27 +00:00

Hello,

I have always despised timestamps for each line of log for the most task, even though they could be useful in long running tasks, when only errors etc are displayed or if one is doing forensic or datascience on logs. However for my daily tasks, such as running a system backup script and sending the log via e-mail, I never needed times stamps for each log message, and many other data that comes in default settings. I remember always setting log format for any project, little or big as the first setting in my previous works generally to get rid of these timestamps. I remember doing it log4cpp, log4j, ruby, perl, etc. I have not done it python I guess, but I am almost sure that a formatter object can be passed to logger configuration. I believe log format can be an option borgmatic yaml file, raw format string in whatever python syntax can be passed directly to the underlying infrastructure. For example, user may pass this string in yaml file if really wishes to do it.

It is really up-to-you. It may require more work than the benefit, I can not evaluate. However, as you have shown, stripping or filtering logs is not the real problem, but enhancing it. If you do it, each yaml configuration may have different log formats, may enhance the logs various ways if one really needs it. I have no such need currently so I can not create scenario which it does it my mind right now.

My problem has been solved, but I am not sure, if you would like to keep this issue open for further discussion. So I leave it to you to close the issue.

Thanks all the support and for the great piece of software.

However, before closure, I can not pass stating something, even though it is not the subject of this discussion. I really hope borgmatic and borg are not two separate projects and become united as a part of borg. Maybe borg team can include into their software and two projects can merge. There will always be discrepancies between two projects, and you will have to follow them. For example this issue could be possibly handled better if these two software were managed together. I started using borg by itself, but I did not enjoy writing scripts to handle backups even though it was not hard. I have found borgmatic, and have been using it for several years now. Now new borg version is coming which breaks things, I believe, I may have problem with borg vs. borgmatic. But there are new projects such as kopia which includes backups along with backup profiles, a combination of borg and borgmatic and it claims to faster then borg.

Anyhow, again thank you all for the responses, and kind assistance.

Hello, I have always despised timestamps for each line of log for the most task, even though they could be useful in long running tasks, when only errors etc are displayed or if one is doing forensic or datascience on logs. However for my daily tasks, such as running a system backup script and sending the log via e-mail, I never needed times stamps for each log message, and many other data that comes in default settings. I remember always setting log format for any project, little or big as the first setting in my previous works generally to get rid of these timestamps. I remember doing it log4cpp, log4j, ruby, perl, etc. I have not done it python I guess, but I am almost sure that a formatter object can be passed to logger configuration. I believe log format can be an option borgmatic yaml file, raw format string in whatever python syntax can be passed directly to the underlying infrastructure. For example, user may pass this string in yaml file if really wishes to do it. It is really up-to-you. It may require more work than the benefit, I can not evaluate. However, as you have shown, stripping or filtering logs is not the real problem, but enhancing it. If you do it, each yaml configuration may have different log formats, may enhance the logs various ways if one really needs it. I have no such need currently so I can not create scenario which it does it my mind right now. My problem has been solved, but I am not sure, if you would like to keep this issue open for further discussion. So I leave it to you to close the issue. Thanks all the support and for the great piece of software. However, before closure, I can not pass stating something, even though it is not the subject of this discussion. I really hope borgmatic and borg are not two separate projects and become united as a part of borg. Maybe borg team can include into their software and two projects can merge. There will always be discrepancies between two projects, and you will have to follow them. For example this issue could be possibly handled better if these two software were managed together. I started using borg by itself, but I did not enjoy writing scripts to handle backups even though it was not hard. I have found borgmatic, and have been using it for several years now. Now new borg version is coming which breaks things, I believe, I may have problem with borg vs. borgmatic. But there are new projects such as kopia which includes backups along with backup profiles, a combination of borg and borgmatic and it claims to faster then borg. Anyhow, again thank you all for the responses, and kind assistance.

witten referenced this issue from a commit

2023-04-03 06:06:48 +00:00

Add "--log-file-format" flag for customizing the log message format (#658).

witten commented

2023-04-03 06:13:06 +00:00

I went ahead and implemented a --log-file-format flag that does what you want (strip out the timestamp and the log level name) as follows:

borgmatic --log-file /your/log/file --log-file-format "{message}"

Full documentation will be here shortly: https://torsion.org/borgmatic/docs/how-to/inspect-your-backups/#logging-to-file

And this feature will be part of the next release. Thanks for the suggestion!

As for combining Borg + borgmatic, that was actually something I suggested pretty early on (in borgmatic's history) to the Borg maintainers. And they weren't really interested in that. If I recall correctly, it was because they saw the functionality that borgmatic provided (configuration, etc.) as outside the purview of Borg itself. These days, it seems even less likely to combine the two projects given how many Borg wrappers there are: borgmatic, Emborg, Vorta, PikaBackup, etc.

If you have any specific suggestions for making borgmatic better integrate with Borg, please feel free to file those tickets. Note that borgmatic already has full Borg 2 support built-in.

I went ahead and implemented a `--log-file-format` flag that does what you want (strip out the timestamp and the log level name) as follows: ```bash borgmatic --log-file /your/log/file --log-file-format "{message}" ``` Full documentation will be here shortly: https://torsion.org/borgmatic/docs/how-to/inspect-your-backups/#logging-to-file And this feature will be part of the next release. Thanks for the suggestion! As for combining Borg + borgmatic, that was actually something I suggested pretty early on (in borgmatic's history) to the Borg maintainers. And they weren't really interested in that. If I recall correctly, it was because they saw the functionality that borgmatic provided (configuration, etc.) as outside the purview of Borg itself. These days, it seems even less likely to combine the two projects given how many Borg wrappers there are: borgmatic, Emborg, Vorta, PikaBackup, etc. If you have any specific suggestions for making borgmatic better integrate with Borg, please feel free to file those tickets. Note that borgmatic already has full Borg 2 support built-in.

witten closed this issue

2023-04-03 06:13:06 +00:00

witten removed the

waiting for response

label 2023-04-18 16:48:59 +00:00

Sign in to join this conversation.