borgmatic-collective/borgmatic
borgmatic-collective
/
borgmatic
15
57
Fork 40
Code Issues 53 Pull Requests 1 Actions Packages Releases 121 Activity

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 118: invalid start byte #489

New Issue
Closed
opened 2022-01-10 20:08:15 +00:00 by Alexander-Shukaev · 3 comments
Alexander-Shukaev commented 2022-01-10 20:08:15 +00:00
Copy Link

What I'm trying to do and why

Steps to reproduce (if a bug)

After backing up several thousands of files, for whatever reason, the remote timed out, which can happen, that's fine. Some of the last files had non-latin characters. As a result, when borgmatic print error trace tail, it does capture a few A /... lines, where some of them also having non-latin characters are output just fine, but then suddenly it crashes on some file name (see below).

Actual behavior (if a bug)

borgmatic[999048]: Traceback (most recent call last):
borgmatic[999048]:   File "/usr/bin/borgmatic", line 33, in <module>
borgmatic[999048]:     sys.exit(load_entry_point('borgmatic==1.5.21', 'console_scripts', 'borgmatic')())
borgmatic[999048]:   File "/usr/lib/python3.10/site-packages/borgmatic/commands/borgmatic.py", line 823, in main
borgmatic[999048]:     summary_logs = parse_logs + list(collect_configuration_run_summary_logs(configs, arguments))
borgmatic[999048]:   File "/usr/lib/python3.10/site-packages/borgmatic/commands/borgmatic.py", line 721, in collect_configuration_run_summary_logs
borgmatic[999048]:     results = list(run_configuration(config_filename, config, arguments))
borgmatic[999048]:   File "/usr/lib/python3.10/site-packages/borgmatic/commands/borgmatic.py", line 233, in run_configuration
borgmatic[999048]:     command.execute_hook(
borgmatic[999048]:   File "/usr/lib/python3.10/site-packages/borgmatic/hooks/command.py", line 65, in execute_hook
borgmatic[999048]:     execute.execute_command(
borgmatic[999048]:   File "/usr/lib/python3.10/site-packages/borgmatic/execute.py", line 217, in execute_command
borgmatic[999048]:     log_outputs(
borgmatic[999048]:   File "/usr/lib/python3.10/site-packages/borgmatic/execute.py", line 90, in log_outputs
borgmatic[999048]:     line = ready_buffer.readline().rstrip().decode()
borgmatic[999048]: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 118: invalid start byte
borgmatic[999047]: /usr/bin/borgmatic failed with exit status 1.

Expected behavior (if a bug)

I'm not sure we can really prevent this from happening universally, I would simply suggest to make this piece of code more robust as a first step at least. That is encoding error in outputing just one of the lines should not crash the whole story. Either we catch it and skip to next line or, even better (but more involved), replace the offending symbol(s) with some filler and continue.

Environment

borgmatic version: 1.5.21

Borg version: 1.1.17

Python version: 3.10.1

#### What I'm trying to do and why #### Steps to reproduce (if a bug) After backing up several thousands of files, for whatever reason, the remote timed out, which can happen, that's fine. Some of the last files had non-latin characters. As a result, when `borgmatic` print error trace tail, it does capture a few `A /...` lines, where some of them also having non-latin characters are output just fine, but then suddenly it crashes on some file name (see below). #### Actual behavior (if a bug) ``` borgmatic[999048]: Traceback (most recent call last): borgmatic[999048]: File "/usr/bin/borgmatic", line 33, in <module> borgmatic[999048]: sys.exit(load_entry_point('borgmatic==1.5.21', 'console_scripts', 'borgmatic')()) borgmatic[999048]: File "/usr/lib/python3.10/site-packages/borgmatic/commands/borgmatic.py", line 823, in main borgmatic[999048]: summary_logs = parse_logs + list(collect_configuration_run_summary_logs(configs, arguments)) borgmatic[999048]: File "/usr/lib/python3.10/site-packages/borgmatic/commands/borgmatic.py", line 721, in collect_configuration_run_summary_logs borgmatic[999048]: results = list(run_configuration(config_filename, config, arguments)) borgmatic[999048]: File "/usr/lib/python3.10/site-packages/borgmatic/commands/borgmatic.py", line 233, in run_configuration borgmatic[999048]: command.execute_hook( borgmatic[999048]: File "/usr/lib/python3.10/site-packages/borgmatic/hooks/command.py", line 65, in execute_hook borgmatic[999048]: execute.execute_command( borgmatic[999048]: File "/usr/lib/python3.10/site-packages/borgmatic/execute.py", line 217, in execute_command borgmatic[999048]: log_outputs( borgmatic[999048]: File "/usr/lib/python3.10/site-packages/borgmatic/execute.py", line 90, in log_outputs borgmatic[999048]: line = ready_buffer.readline().rstrip().decode() borgmatic[999048]: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 118: invalid start byte borgmatic[999047]: /usr/bin/borgmatic failed with exit status 1. ``` #### Expected behavior (if a bug) I'm not sure we can really prevent this from happening universally, I would simply suggest to make this piece of code more robust as a first step at least. That is encoding error in outputing just one of the lines should not crash the whole story. Either we catch it and skip to next line or, even better (but more involved), replace the offending symbol(s) with some filler and continue. #### Environment **borgmatic version:** 1.5.21 **Borg version:** 1.1.17 **Python version:** 3.10.1
witten commented 2022-01-10 21:12:51 +00:00
Owner
Copy Link

Thank you for taking the time to report this! I totally agree that borgmatic shouldn't blow up with a traceback on "invalid" input. A few questions for you:

  • Do you have an example of a filename that causes this sort of problem, so that I can try to reproduce it?
  • Can I get a look at a redacted copy of your borgmatic configuration? In particular, I'm interested in seeing your hook configuration (if any).
  • Also, is your terminal where you're running this using UTF-8? Running echo $LANG may help determine that.
Thank you for taking the time to report this! I totally agree that borgmatic shouldn't blow up with a traceback on "invalid" input. A few questions for you: * Do you have an example of a filename that causes this sort of problem, so that I can try to reproduce it? * Can I get a look at a redacted copy of your borgmatic configuration? In particular, I'm interested in seeing your hook configuration (if any). * Also, is your terminal where you're running this using UTF-8? Running `echo $LANG` may help determine that.
witten added the
bug
 label 2022-01-10 21:12:58 +00:00
witten commented 2022-01-19 04:44:31 +00:00
Owner
Copy Link

Can I get a look at your /etc/borgmatic.d/._backup1.yaml configuration file? Or is that the same as config.yaml above?

And would it possible to see your borgmatic --verbosity 2 output? That might help pinpoint where the unicode error occurs.

Thank you!

Can I get a look at your `/etc/borgmatic.d/._backup1.yaml` configuration file? Or is that the same as `config.yaml` above? And would it possible to see your `borgmatic --verbosity 2` output? That might help pinpoint where the unicode error occurs. Thank you!
witten added the
waiting for response
 label 2022-03-06 00:14:02 +00:00
witten commented 2022-04-28 20:44:01 +00:00
Owner
Copy Link

I'm closing this now due to inactivity, but please feel free to re-open if you'd like to post follow-up information at any point. Thanks!

I'm closing this now due to inactivity, but please feel free to re-open if you'd like to post follow-up information at any point. Thanks!
witten removed the
waiting for response
 label 2022-04-28 20:44:14 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
main
spot-check
1.8.10
1.8.9
1.8.8
1.8.7
1.8.6
1.8.5
1.8.4
1.8.3
1.8.2
1.8.1
1.8.0
1.7.15
1.7.14
1.7.13
1.7.12
1.7.11
1.7.10
1.7.9
1.7.8
1.7.7
1.7.6
1.7.5
1.7.4
1.7.3
1.7.2
1.7.1
1.7.0
1.6.6
1.6.5
1.6.4
1.6.3
1.6.2
1.6.1
1.6.0
1.5.24
1.5.23
1.5.22
1.5.21
1.5.20
1.5.19
1.5.18
1.5.17
1.5.16
1.5.15
1.5.14
1.5.13
1.5.12
1.5.11
1.5.10
1.5.9
1.5.8
1.5.7
1.5.7.dev0
1.5.6
1.5.5
1.5.4
1.5.3
1.5.2
1.5.1
1.5.0
1.4.22
1.4.21
1.4.20
1.4.19
1.4.18
1.4.17
1.4.16
1.4.15
1.4.14
1.4.13
1.4.12
1.4.11
1.4.10
1.4.9
1.4.8
1.4.7
1.4.6
1.4.5
1.4.4
1.4.3
1.4.2
1.4.1
1.4.0
1.3.26
1.3.25
1.3.24
1.3.23
1.3.22
1.3.21
1.3.20
1.3.19
1.3.18
1.3.17
1.3.16
1.3.15
1.3.14
1.3.13
1.3.12
1.3.11
1.3.10
1.3.9
1.3.8
1.3.7
1.3.6
1.3.5
1.3.4
1.3.3
1.3.2
1.3.1
1.3.0
1.2.18
1.2.17
1.2.16
1.2.15
1.2.14
1.2.13
1.2.12
1.2.11
1.2.10
1.2.9
1.2.8
1.2.7
1.2.6
1.2.5
1.2.4
1.2.3
1.2.2
1.2.1
1.2.0
1.1.15
1.1.14
1.1.13
1.1.12
1.1.11
1.1.10
1.1.9
1.1.7
1.1.6
1.1.5
1.1.4
1.1.3
1.1.2
1.1.1
1.1.0
1.0.3
1.0.2
1.0.1
1.0.0
0.1.8
0.1.7
0.1.6
0.1.5
0.1.4
0.1.3
0.1.2
0.1.1
0.1.0
0.0.7
0.0.6
0.0.5
0.0.4
0.0.3
0.0.2
Labels
Clear labels   
bug

   
data loss

   
design finalized

   
good first issue

   
new feature area

   
question / support

   
security

   
waiting for response
No Label
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#489
No description provided.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?