With --log-json, format borgmatic logs as JSON, not just Borg logs #858

New Issue

Open

opened 2024-04-26 01:24:27 +00:00 by lingfish · 3 comments

lingfish commented 2024-04-26 01:24:27 +00:00

Copy Link

What I'm trying to do and why

Trying to push logs to Graylog via JSON.

Steps to reproduce

borgmatic --log-json info --repo usbdisk

Actual behavior

Outputs "normal" text, instead of JSON.

Expected behavior

Log messages in JSON format.

Other notes / implementation ideas

If you run the same with -v 2, amongst normal text output, you can see borg's JSON output.

borgmatic version

1.8.8

borgmatic installation method

pipx

Borg version

borg 1.2.7

Python version

Python 3.11.2

Database version (if applicable)

No response

Operating system and version

PRETTY_NAME="Debian GNU/Linux 12 (bookworm)" NAME="Debian GNU/Linux" VERSION_ID="12" VERSION="12 (bookworm)" VERSION_CODENAME=bookworm ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/"

### What I'm trying to do and why Trying to push logs to Graylog via JSON. ### Steps to reproduce `borgmatic --log-json info --repo usbdisk` ### Actual behavior Outputs "normal" text, instead of JSON. ### Expected behavior Log messages in JSON format. ### Other notes / implementation ideas If you run the same with `-v 2`, amongst normal text output, you can see `borg`'s JSON output. ### borgmatic version 1.8.8 ### borgmatic installation method pipx ### Borg version borg 1.2.7 ### Python version Python 3.11.2 ### Database version (if applicable) _No response_ ### Operating system and version PRETTY_NAME="Debian GNU/Linux 12 (bookworm)" NAME="Debian GNU/Linux" VERSION_ID="12" VERSION="12 (bookworm)" VERSION_CODENAME=bookworm ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/"

witten commented 2024-04-26 06:20:48 +00:00

Owner

Copy Link

Thanks for filing this! However I'm pretty sure the behavior you're seeing is a Borg limitation, even without using borgmatic. For instance, here's the equivalent Borg command to using borgmatic's --verbosity 2:

borg info --log-json --glob-archives {hostname}-* yourrepo

(The --glob-archives value may vary based on your configuration, and yourrepo will depend on the path to your Borg repository.)

What that command produces is "normal" non-JSON repository information, I guess because Borg doesn't consider its info output as log entries subject to --log-json. And there are no other log entries to display (as JSON) at that verbosity level.

But if you add --debug, that's the equivalent to borgmatic's --verbosity 2:

borg info --debug --log-json --glob-archives {hostname}-* yourrepo

That produces the mixed output you're observing:

• borgmatic verbose log entries show as non-JSON, because --log-json currently only applies to Borg logging and not borgmatic's own logs.
• Borg logs its entries as JSON, because Borg applies --log-json to them.
• Borg info output shows as "normal" text, because again, Borg doesn't consider that as part of its log entries.

However, switching back to borgmatic, what you can do is is add the --json flag to --log-json but don't use --verbosity 2. Here's what that does:

• borgmatic (non-JSON) verbose log entries get suppressed since you're omitting --verbosity 2.
• Borg logs its entries as JSON, because Borg applies --log-json to them.
• Borg info output shows as JSON, because of the --json flag.

Maybe that's more along the lines of what you're looking for?

Thanks for filing this! However I'm pretty sure the behavior you're seeing is a Borg limitation, even without using borgmatic. For instance, here's the equivalent Borg command to using borgmatic's `--verbosity 2`: ```bash borg info --log-json --glob-archives {hostname}-* yourrepo ``` (The `--glob-archives` value may vary based on your configuration, and `yourrepo` will depend on the path to your Borg repository.) What that command produces is "normal" non-JSON repository information, I guess because Borg doesn't consider its `info` output as log entries subject to `--log-json`. And there are no other log entries to display (as JSON) at that verbosity level. But if you add `--debug`, that's the equivalent to borgmatic's `--verbosity 2`: ```bash borg info --debug --log-json --glob-archives {hostname}-* yourrepo ``` That produces the mixed output you're observing: * borgmatic verbose log entries show as non-JSON, because `--log-json` currently only applies to Borg logging and not borgmatic's own logs. * Borg logs its entries as JSON, because Borg applies `--log-json` to them. * Borg `info` output shows as "normal" text, because again, Borg doesn't consider that as part of its log entries. However, switching back to borgmatic, what you _can_ do is is add the `--json` flag to `--log-json` but don't use `--verbosity 2`. Here's what that does: * borgmatic (non-JSON) verbose log entries get suppressed since you're omitting `--verbosity 2`. * Borg logs its entries as JSON, because Borg applies `--log-json` to them. * Borg `info` output shows as JSON, because of the `--json` flag. Maybe that's more along the lines of what you're looking for?

witten added the

question / support

label 2024-04-26 06:20:57 +00:00

lingfish commented 2024-04-29 01:49:55 +00:00

Author

Copy Link

Thanks for responding @witten. Excellent clarification, thanks heaps. You're right, your final suggestion does the job, but I've now realised it's basically only a summary (without --progress of course), and I was barking up the wrong tree.

I think I'd like to see borgmatic's logs turned into a structure, moreso than borg. borg JSON is essentially for GUI authors, and mostly resolves therefore around progress, not full state.

I'll give you a couple of my use cases/wishes for this:

1. Having a key/identifier for the current repo/backup name (in JSON), so that...
2. One can search for that label, to then see things like duration of the backup, and anything else associated with it in, say, Graylog

What I mean by repo/backup name is each item listed in the borgmatic repositories: key (preferably even by borgmatic's label!)

I guess the above suggested run method gives some of that, we then miss out on also doing the "human" logging in the same run. One can then do an info with JSON after, but that feels clunky.

As I type this, perhaps it even belongs in the hooks framework.

Using just --json, I do see some useful stuff:

        "repository": {
            "id": "xxxxxxxx",
            "last_modified": "2024-04-27T12:31:13.000000",
            "location": "ssh://user@host/./some_repo",
            "label": "label-from-borgmatic-config"
        }

... but again, it'd be nice to push that into Graylog, but also see the usual multiline logging (and even push that somewhere, which I'm currently doing to Graylog).

Seeing borgmatic informational stuff, also structured (probably just a "message" field), would also be useful, so that the structured logging is complete.

On top of all of this then would be wrapping it all in GELF.

I dunno, a lot of random spitballing here; I just struggle to keep an eye on backups when viewing the normal logging, and trying to think of better ways.

Thanks for responding @witten. Excellent clarification, thanks heaps. You're right, your final suggestion does the job, but I've now realised it's basically only a summary (without `--progress` of course), and I was barking up the wrong tree. I think I'd like to see borgmatic's logs turned into a structure, moreso than borg. borg JSON is essentially for GUI authors, and mostly resolves therefore around progress, not full state. I'll give you a couple of my use cases/wishes for this: 1. Having a key/identifier for the current repo/backup name (in JSON), so that... 2. One can search for that label, to then see things like duration of the backup, and anything else associated with it in, say, Graylog What I mean by repo/backup name is each item listed in the borgmatic `repositories:` key (preferably even by borgmatic's label!) I guess the above suggested run method gives some of that, we then miss out on also doing the "human" logging in the same run. One can then do an `info` with JSON after, but that feels clunky. As I type this, perhaps it even belongs in the hooks framework. Using just `--json`, I do see some useful stuff: ``` "repository": { "id": "xxxxxxxx", "last_modified": "2024-04-27T12:31:13.000000", "location": "ssh://user@host/./some_repo", "label": "label-from-borgmatic-config" } ``` ... but again, it'd be nice to push that into Graylog, but also see the usual multiline logging (and even push that somewhere, which I'm currently doing to Graylog). Seeing borgmatic informational stuff, also structured (probably just a "message" field), would also be useful, so that the structured logging is complete. On top of all of this then would be wrapping it all in GELF. I dunno, a lot of random spitballing here; I just struggle to keep an eye on backups when viewing the normal logging, and trying to think of better ways.

witten commented 2024-06-23 22:30:38 +00:00

Owner

Copy Link

(I apologize for the delay in getting back to this.) My take is that it's a reasonable ask for borgmatic logs to (optionally) be in JSON format themselves.. especially since Borg sorta kinds supports JSON logging. I could see automatically keying that off of the existing --log-json flag. And starting with just a message field would be the easiest way to get going, although I could imagine adding more fields later on (configuration_file, repository_path, etc.). However the one caveat would be that these additional fields to search for would only be in borgmatic's own log entries and not in any associated Borg JSON logs—unless borgmatic injected them there.

(I apologize for the delay in getting back to this.) My take is that it's a reasonable ask for borgmatic logs to (optionally) be in JSON format themselves.. especially since Borg sorta kinds supports JSON logging. I could see automatically keying that off of the existing `--log-json` flag. And starting with just a `message` field would be the easiest way to get going, although I could imagine adding more fields later on (`configuration_file`, `repository_path`, etc.). However the one caveat would be that these additional fields to search for would only be in borgmatic's own log entries and not in any associated Borg JSON logs—unless borgmatic injected them there.

witten added

new feature area

and removed

question / support

labels 2024-06-23 22:30:59 +00:00

witten changed title from log-json seems to do nothing except in debug mode to With --log-json, format borgmatic logs as JSON, not just Borg logs 2024-06-23 22:32:08 +00:00

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

No results found.

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.14

1.8.13

1.8.12

1.8.11

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.24

1.5.23

1.5.22

1.5.21

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.14

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.7.dev0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.22

1.4.21

1.4.20

1.4.19

1.4.18

1.4.17

1.4.16

1.4.15

1.4.14

1.4.13

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.26

1.3.25

1.3.24

1.3.23

1.3.22

1.3.21

1.3.20

1.3.19

1.3.18

1.3.17

1.3.16

1.3.15

1.3.14

1.3.13

1.3.12

1.3.11

1.3.10

1.3.9

1.3.8

1.3.7

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.18

1.2.17

1.2.16

1.2.15

1.2.14

1.2.13

1.2.12

1.2.11

1.2.10

1.2.9

1.2.8

1.2.7

1.2.6

1.2.5

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.15

1.1.14

1.1.13

1.1.12

1.1.11

1.1.10

1.1.9

1.1.7

1.1.6

1.1.5

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.3

1.0.2

1.0.1

1.0.0

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

0.0.2

Labels

Clear labels

blocked

bug

data loss

design finalized

good first issue

new feature area

question / support

security

waiting for response

No Label

new feature area

Milestone

No items

No Milestone

Assignees

Clear assignees

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#858

No description provided.