Cannot Parse json output: jq: parse error: Invalid numeric literal #827

New Issue

Closed

opened 2024-02-10 23:19:19 +00:00 by xLogiiCx · 6 comments

xLogiiCx commented 2024-02-10 23:19:19 +00:00

Copy Link

What I'm trying to do and why

Hi, i am trying to write a Python Script to export Data about my Borg Repositories.
But i am having Problems parsing the output.

I tried following:

def get_repo_info(repository):
    # Build the borgmatic list command
    # repo_info_cmd = ['borgmatic', 'rinfo', '--json', '--repository', repository]
    cmd = f'borgmatic rinfo --json --repository {repository} | jq'
    sd = os.system(cmd)

    return sd

get_repo_info('/media/user/b/b.borg')

but i get following error:
jq: parse error: Invalid numeric literal at line 1, column 649

i also tried with subprocess and json.dumps but i get a simmilar error.
is there something wrong on my site?
i got a [ESC], % and some other weird things at the end of my json output string.

Steps to reproduce

No response

Actual behavior

No response

Expected behavior

No response

Other notes / implementation ideas

No response

borgmatic version

1.8.8

borgmatic installation method

AUR

Borg version

borg 1.2.7

Python version

3.11.7

Database version (if applicable)

No response

Operating system and version

No response

### What I'm trying to do and why Hi, i am trying to write a Python Script to export Data about my Borg Repositories. But i am having Problems parsing the output. I tried following: ```python def get_repo_info(repository): # Build the borgmatic list command # repo_info_cmd = ['borgmatic', 'rinfo', '--json', '--repository', repository] cmd = f'borgmatic rinfo --json --repository {repository} | jq' sd = os.system(cmd) return sd get_repo_info('/media/user/b/b.borg') ``` but i get following error: `jq: parse error: Invalid numeric literal at line 1, column 649` i also tried with subprocess and json.dumps but i get a simmilar error. is there something wrong on my site? i got a `[ESC]`, `%` and some other weird things at the end of my json output string. ### Steps to reproduce _No response_ ### Actual behavior _No response_ ### Expected behavior _No response_ ### Other notes / implementation ideas _No response_ ### borgmatic version 1.8.8 ### borgmatic installation method AUR ### Borg version borg 1.2.7 ### Python version 3.11.7 ### Database version (if applicable) _No response_ ### Operating system and version _No response_

witten commented 2024-02-11 03:56:16 +00:00

Owner

Copy Link

Thanks for filing this, but so far I'm not able to reproduce it. Can I get a look at your borgmatic JSON output including the weird characters? If they're not showing up with copy and paste, feel free to redirect to a file and then attach the file here. And you can redact anything sensitive, of course. Thank you!

Thanks for filing this, but so far I'm not able to reproduce it. Can I get a look at your borgmatic JSON output including the weird characters? If they're not showing up with copy and paste, feel free to redirect to a file and then attach the file here. And you can redact anything sensitive, of course. Thank you!

xLogiiCx commented 2024-02-11 09:13:16 +00:00

Author

Copy Link

b'[{"cache": {"path": "/root/.cache/borg/red", "stats": {"total_chunks": 3808212, "total_csize": 1270077190113, "total_size": 1724789507361, "total_unique_chunks": 569956, "unique_csize": 92260417644, "unique_size": 107385743183}}, "encryption": {"mode": "repokey-blake2"}, "repository": {"id": "red", "last_modified": "2024-02-11T00:25:36.000000", "location": "/media/user/borg/pc.borg", "label": "local"}, "security_dir": "/root/.config/borg/security/red"}]\x1b[0m'

is what i get from following for example:

def get_info(repository):
    # Build the borgmatic info command
    cmd = ['borgmatic', 'rinfo', '--json', '--repository', repository]

    # Run the command and capture the output
    output = subprocess.check_output(cmd, stderr=subprocess.STDOUT)
    print(output)
    # Parse the JSON output
    data = json.loads(output)

    return data


get_info('/media/user/borg/pc.borg')

it kinda works with jq but there is still an Error:

def get_repo_info(repository):
    # Build the borgmatic list command
    # repo_info_cmd = ['borgmatic', 'rinfo', '--json', '--repository', repository]
    cmd = f'borgmatic rinfo --json --repository {repository} | jq'
    sd = os.system(cmd)
    
    return sd


get_repo_info('/media/user/borg/pc.borg')

jq: parse error: Invalid numeric literal at line 1, column 649
[
  {
    "cache": {
      "path": "/root/.cache/borg/red",
      "stats": {
        "total_chunks": 3808212,
        "total_csize": 1270077190113,
        "total_size": 1724789507361,
        "total_unique_chunks": 569956,
        "unique_csize": 92260417644,
        "unique_size": 107385743183
      }
    },
    "encryption": {
      "mode": "repokey-blake2"
    },
    "repository": {
      "id": "red",
      "last_modified": "2024-02-11T00:25:36.000000",
      "location": "/media/user/borg/pc.borg",
      "label": "local"
    },
    "security_dir": "/root/.config/borg/security/red"
  }
]

cmdline:

~ » sudo borgmatic rinfo --json --repository /media/user/borg/pc.borg > test.txt                                                                                                                                          
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
~ » cat test.txt                                                                                                                                                                                                                 
[{"cache": {"path": "/root/.cache/borg/red", "stats": {"total_chunks": 3808212, "total_csize": 1270077190113, "total_size": 1724789507361, "total_unique_chunks": 569956, "unique_csize": 92260417644, "unique_size": 107385743183}}, "encryption": {"mode": "repokey-blake2"}, "repository": {"id": "red", "last_modified": "2024-02-11T00:25:36.000000", "location": "/media/user/borg/pc.borg", "label": "local"}, "security_dir": "/root/.config/borg/security/red"}]%                                                      --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
~ »

```json b'[{"cache": {"path": "/root/.cache/borg/red", "stats": {"total_chunks": 3808212, "total_csize": 1270077190113, "total_size": 1724789507361, "total_unique_chunks": 569956, "unique_csize": 92260417644, "unique_size": 107385743183}}, "encryption": {"mode": "repokey-blake2"}, "repository": {"id": "red", "last_modified": "2024-02-11T00:25:36.000000", "location": "/media/user/borg/pc.borg", "label": "local"}, "security_dir": "/root/.config/borg/security/red"}]\x1b[0m' ``` is what i get from following for example: ``` def get_info(repository): # Build the borgmatic info command cmd = ['borgmatic', 'rinfo', '--json', '--repository', repository] # Run the command and capture the output output = subprocess.check_output(cmd, stderr=subprocess.STDOUT) print(output) # Parse the JSON output data = json.loads(output) return data get_info('/media/user/borg/pc.borg') ``` it kinda works with jq but there is still an Error: ``` def get_repo_info(repository): # Build the borgmatic list command # repo_info_cmd = ['borgmatic', 'rinfo', '--json', '--repository', repository] cmd = f'borgmatic rinfo --json --repository {repository} | jq' sd = os.system(cmd) return sd get_repo_info('/media/user/borg/pc.borg') ``` ```json jq: parse error: Invalid numeric literal at line 1, column 649 [ { "cache": { "path": "/root/.cache/borg/red", "stats": { "total_chunks": 3808212, "total_csize": 1270077190113, "total_size": 1724789507361, "total_unique_chunks": 569956, "unique_csize": 92260417644, "unique_size": 107385743183 } }, "encryption": { "mode": "repokey-blake2" }, "repository": { "id": "red", "last_modified": "2024-02-11T00:25:36.000000", "location": "/media/user/borg/pc.borg", "label": "local" }, "security_dir": "/root/.config/borg/security/red" } ] ``` cmdline: ``` ~ » sudo borgmatic rinfo --json --repository /media/user/borg/pc.borg > test.txt -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ~ » cat test.txt [{"cache": {"path": "/root/.cache/borg/red", "stats": {"total_chunks": 3808212, "total_csize": 1270077190113, "total_size": 1724789507361, "total_unique_chunks": 569956, "unique_csize": 92260417644, "unique_size": 107385743183}}, "encryption": {"mode": "repokey-blake2"}, "repository": {"id": "red", "last_modified": "2024-02-11T00:25:36.000000", "location": "/media/user/borg/pc.borg", "label": "local"}, "security_dir": "/root/.config/borg/security/red"}]% -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ~ » ```

xLogiiCx commented 2024-02-11 10:44:14 +00:00

Author

Copy Link

Got it woking by removing the ANSI from the String with a regex.
Maybe this is because of the Colored output?

def get_repo_info(repository):
    # Build the borgmatic list command
    cmd = ['borgmatic', '-nc', 'rinfo', '--json']

    rinfo = subprocess.check_output(cmd, text=True)
    regex = re.compile(r'\x1B(?:[@-Z\\-_]|\[[0-?]*[ -/]*[@-~])')
    result = regex.sub('', rinfo)
    rinfo_json = json.loads(result)

    return rinfo_json

Got it woking by removing the ANSI from the String with a regex. Maybe this is because of the Colored output? ```python3 def get_repo_info(repository): # Build the borgmatic list command cmd = ['borgmatic', '-nc', 'rinfo', '--json'] rinfo = subprocess.check_output(cmd, text=True) regex = re.compile(r'\x1B(?:[@-Z\\-_]|\[[0-?]*[ -/]*[@-~])') result = regex.sub('', rinfo) rinfo_json = json.loads(result) return rinfo_json ```

witten commented 2024-02-11 17:14:43 +00:00

Owner

Copy Link

Thanks for the additional details. I don't have a local repro still, but I think I know what might be going on. That code you're getting is a "styling reset" escape code, so even though colors are disabled automatically when --json is given, that reset code is still coming through for some reason. I'll mark this as a bug and look into suppressing the code.

Thanks for the additional details. I don't have a local repro still, but I think I know what might be going on. That code you're getting is a "styling reset" escape code, so even though colors are disabled automatically when `--json` is given, that reset code is still coming through for some reason. I'll mark this as a bug and look into suppressing the code.

witten added the

bug

label 2024-02-11 17:14:48 +00:00

witten referenced this issue from a commit 2024-02-12 01:44:58 +00:00

When the "--json" flag is given, suppress console escape codes so as not to interfere with JSON output (#827).

witten commented 2024-02-12 01:45:39 +00:00

Owner

Copy Link

Okay, this should hopefully be fixed in main and will be part of the next release!

Okay, this should hopefully be fixed in main and will be part of the next release!

👍 1

witten closed this issue 2024-02-12 01:45:39 +00:00

witten commented 2024-03-11 20:30:08 +00:00

Owner

Copy Link

Released in borgmatic 1.8.9!

Released in borgmatic 1.8.9!

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

spot-check

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.24

1.5.23

1.5.22

1.5.21

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.14

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.7.dev0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.22

1.4.21

1.4.20

1.4.19

1.4.18

1.4.17

1.4.16

1.4.15

1.4.14

1.4.13

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.26

1.3.25

1.3.24

1.3.23

1.3.22

1.3.21

1.3.20

1.3.19

1.3.18

1.3.17

1.3.16

1.3.15

1.3.14

1.3.13

1.3.12

1.3.11

1.3.10

1.3.9

1.3.8

1.3.7

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.18

1.2.17

1.2.16

1.2.15

1.2.14

1.2.13

1.2.12

1.2.11

1.2.10

1.2.9

1.2.8

1.2.7

1.2.6

1.2.5

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.15

1.1.14

1.1.13

1.1.12

1.1.11

1.1.10

1.1.9

1.1.7

1.1.6

1.1.5

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.3

1.0.2

1.0.1

1.0.0

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

0.0.2

Labels

Clear labels   

bug

  

data loss

  

design finalized

  

good first issue

  

new feature area

  

question / support

  

security

  

waiting for response

No Label

bug

data loss

design finalized

good first issue

new feature area

question / support

security

waiting for response

Milestone

Clear milestone

No items

No Milestone

Assignees

Clear assignees

No Assignees

2 Participants

Notifications

Subscribe

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#827

No description provided.