bad character Local Exception #899

New Issue

Closed

opened 2024-07-12 16:50:06 +00:00 by sshaikh · 9 comments

sshaikh commented 2024-07-12 16:50:06 +00:00

Copy Link

What I'm trying to do and why

The regular backup I make has suddenly started failing

Steps to reproduce

No response

Actual behavior

$ sudo borgmatic -c config.yaml
Sending message 'BORG: Finished backup to repo1:
35.496411 seconds
36 / 307270 files changed' 
Done!
Sending message 'BORG: Finished backup to repo2:
34.868918 seconds
36 / 307270 files changed' 
Done!
Local Exception
Traceback (most recent call last):
  File "borg/archiver.py", line 5401, in main
  File "borg/archiver.py", line 5321, in run
  File "borg/archiver.py", line 190, in wrapper
  File "borg/archiver.py", line 1395, in do_list
  File "borg/archiver.py", line 1403, in _list_archive
  File "borg/archiver.py", line 270, in build_matcher
  File "borg/patterns.py", line 128, in add_includepaths
  File "borg/patterns.py", line 128, in <listcomp>
  File "borg/patterns.py", line 357, in parse_pattern
  File "borg/patterns.py", line 190, in __init__
  File "borg/patterns.py", line 290, in _prepare
  File "re.py", line 252, in compile
  File "re.py", line 304, in _compile
  File "sre_compile.py", line 788, in compile
  File "sre_parse.py", line 955, in parse
  File "sre_parse.py", line 444, in _parse_sub
  File "sre_parse.py", line 599, in _parse
re.error: bad character range 4-1 at position 120

Platform: Linux nas 6.1.0-21-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.90-1 (2024-05-03) x86_64
Linux: Unknown Linux
Borg: 1.2.8  Python: CPython 3.9.19 msgpack: 1.0.8 fuse: llfuse 1.5.0 [pyfuse3,llfuse]

and in the rest of the output:

repo2: Error running actions for repository
Command '('borg', 'list', '--format', '{xxh64} /{path}{NL}', 'ssh://repo2 borg::xxx', 'file1', 'etc' ... )

Expected behavior

For the config to complete

Other notes / implementation ideas

No response

borgmatic version

1.8.12

borgmatic installation method

pipx

Borg version

1.2.8

Python version

3.9.19

Database version (if applicable)

No response

Operating system and version

Debian GNU/Linux 12 (bookworm)

### What I'm trying to do and why The regular backup I make has suddenly started failing ### Steps to reproduce _No response_ ### Actual behavior ``` $ sudo borgmatic -c config.yaml Sending message 'BORG: Finished backup to repo1: 35.496411 seconds 36 / 307270 files changed' Done! Sending message 'BORG: Finished backup to repo2: 34.868918 seconds 36 / 307270 files changed' Done! Local Exception Traceback (most recent call last): File "borg/archiver.py", line 5401, in main File "borg/archiver.py", line 5321, in run File "borg/archiver.py", line 190, in wrapper File "borg/archiver.py", line 1395, in do_list File "borg/archiver.py", line 1403, in _list_archive File "borg/archiver.py", line 270, in build_matcher File "borg/patterns.py", line 128, in add_includepaths File "borg/patterns.py", line 128, in <listcomp> File "borg/patterns.py", line 357, in parse_pattern File "borg/patterns.py", line 190, in __init__ File "borg/patterns.py", line 290, in _prepare File "re.py", line 252, in compile File "re.py", line 304, in _compile File "sre_compile.py", line 788, in compile File "sre_parse.py", line 955, in parse File "sre_parse.py", line 444, in _parse_sub File "sre_parse.py", line 599, in _parse re.error: bad character range 4-1 at position 120 Platform: Linux nas 6.1.0-21-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.90-1 (2024-05-03) x86_64 Linux: Unknown Linux Borg: 1.2.8 Python: CPython 3.9.19 msgpack: 1.0.8 fuse: llfuse 1.5.0 [pyfuse3,llfuse] ``` and in the rest of the output: ``` repo2: Error running actions for repository Command '('borg', 'list', '--format', '{xxh64} /{path}{NL}', 'ssh://repo2 borg::xxx', 'file1', 'etc' ... ) ``` ### Expected behavior For the config to complete ### Other notes / implementation ideas _No response_ ### borgmatic version 1.8.12 ### borgmatic installation method pipx ### Borg version 1.2.8 ### Python version 3.9.19 ### Database version (if applicable) _No response_ ### Operating system and version Debian GNU/Linux 12 (bookworm)

witten commented 2024-07-12 17:01:40 +00:00

Owner

Copy Link

That error looks like it's coming from Borg itself. I'm guessing that you have the spot check enabled in borgmatic's configuration? What I'd recommend is try running borgmatic with --verbosity 2 to get the full Borg borg list --format ... command that borgmatic is running—or as much of it as it'll show you since it's probably really long. Then you can try running that Borg command directly to see if you can get the error to reproduce without borgmatic. If so, you'd have a good case to file a bug with the Borg project itself.

Hope that helps!

That error looks like it's coming from Borg itself. I'm guessing that you have the `spot` check enabled in borgmatic's configuration? What I'd recommend is try running borgmatic with `--verbosity 2` to get the full Borg `borg list --format ...` command that borgmatic is running—or as much of it as it'll show you since it's probably really long. Then you can try running that Borg command directly to see if you can get the error to reproduce without borgmatic. If so, you'd have a good case to file a bug with the Borg project itself. Hope that helps!

witten added the

question / support

label 2024-07-12 17:01:52 +00:00

sshaikh commented 2024-07-12 17:18:05 +00:00

Author

Copy Link

Running create, prune, compact, check separately resulted in a failed check, but then retrying that succeeded. So this seems transient.

If a check fails, should that take out the whole of borgmatic?

EDIT: whoops, didn't see the reply before posting. Yes, that seems in line. Will try to recreate the error consistently - I'm assuming there's no way to use the same spot that was used in the error case?

Running `create`, `prune`, `compact`, `check` separately resulted in a failed `check`, but then retrying that succeeded. So this seems transient. If a `check` fails, should that take out the whole of borgmatic? EDIT: whoops, didn't see the reply before posting. Yes, that seems in line. Will try to recreate the error consistently - I'm assuming there's no way to use the same `spot` that was used in the error case?

witten commented 2024-07-12 17:48:56 +00:00

Owner

Copy Link

Running create, prune, compact, check separately resulted in a failed check, but then retrying that succeeded. So this seems transient.

Yeah, the spot check is probabilistic (more info here), so it makes sense that you'd see different results with different runs.

If a check fails, should that take out the whole of borgmatic?

check runs last (by default), so it doesn't prevent any of the other actions from completing even if the check fails. But borgmatic can only return a single exit code to the operating system, so if any of those steps fail, the entire borgmatic run fails. You can always run borgmatic actions individually (e.g., borgmatic check) if you don't want the failure of one action to impact the others.

EDIT: whoops, didn't see the reply before posting. Yes, that seems in line. Will try to recreate the error consistently - I'm assuming there's no way to use the same spot that was used in the error case?

Correct. But if you run it enough times, it should hopefully reproduce eventually!

> Running create, prune, compact, check separately resulted in a failed check, but then retrying that succeeded. So this seems transient. Yeah, the `spot` check is probabilistic ([more info here](https://torsion.org/borgmatic/docs/how-to/deal-with-very-large-backups/#spot-check)), so it makes sense that you'd see different results with different runs. > If a check fails, should that take out the whole of borgmatic? `check` runs last (by default), so it doesn't prevent any of the other actions from completing even if the `check` fails. But borgmatic can only return a single exit code to the operating system, so if any of those steps fail, the entire borgmatic run fails. You can always run borgmatic actions individually (e.g., `borgmatic check`) if you don't want the failure of one action to impact the others. > EDIT: whoops, didn't see the reply before posting. Yes, that seems in line. Will try to recreate the error consistently - I'm assuming there's no way to use the same spot that was used in the error case? Correct. But if you run it enough times, it should hopefully reproduce eventually!

sshaikh commented 2024-07-14 18:25:00 +00:00

Author

Copy Link

check found issues. Thanks for the tip - closing.

`check` found issues. Thanks for the tip - closing.

sshaikh closed this issue 2024-07-14 18:25:01 +00:00

witten commented 2024-07-14 21:35:21 +00:00

Owner

Copy Link

Sure thing.. Good luck with your repo!

Sure thing.. Good luck with your repo!

nicolasavru commented 2024-07-25 17:09:32 +00:00

Copy Link

I hit this issue issue today and I think I found the problem. The file paths being passed to borg from the borgmatic spot check are of the form sh:/path/to/file. The sh: prefix tells borg to treat the pattern as a shell-style pattern, which includes things like [a..z] for ranges. In my case, I had a file path that looked structurally like some_text[zxy-cba]/more_text and borg (ultimately python's re module) complained that y-c was a bad character range. So I think we just want to drop the sh: prefix and have borg treat the paths as literals, not patterns.

I hit this issue issue today and I think I found the problem. The file paths being passed to borg from the borgmatic spot check are of the form `sh:/path/to/file`. The `sh:` prefix tells borg to treat the pattern as a shell-style pattern, which includes things like `[a..z]` for ranges. In my case, I had a file path that looked structurally like `some_text[zxy-cba]/more_text` and borg (ultimately python's re module) complained that `y-c` was a bad character range. So I think we just want to drop the `sh:` prefix and have borg treat the paths as literals, not patterns.

👍 1

witten commented 2024-07-25 17:42:46 +00:00

Owner

Copy Link

Good sleuthing! That makes sense to me. I'll reopen this ticket to cover that change.

Good sleuthing! That makes sense to me. I'll reopen this ticket to cover that change.

witten reopened this issue 2024-07-25 17:42:46 +00:00

witten added the

bug

label 2024-07-25 17:42:58 +00:00

witten referenced this issue from a commit 2024-07-25 18:34:44 +00:00

Fix for a "bad character" Borg error in which the "spot" check fed Borg an invalid pattern (#899).

witten commented 2024-07-25 18:34:47 +00:00

Owner

Copy Link

This has been fixed in main and will be part of the next release!

This has been fixed in main and will be part of the next release!

witten closed this issue 2024-07-25 18:34:47 +00:00

witten commented 2024-08-30 00:46:40 +00:00

Owner

Copy Link

Released in borgmatic 1.8.14!

Released in borgmatic 1.8.14!

witten referenced this issue 2024-10-08 03:33:04 +00:00

no activity during borgmatic check #920

sshaikh referenced this issue 2024-10-08 17:03:45 +00:00

no activity during borgmatic check #920

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

config-command-line

1.9.14

1.9.13

1.9.12

1.9.11

1.9.10

1.9.9

1.9.8

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.14

1.8.13

1.8.12

1.8.11

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.24

1.5.23

1.5.22

1.5.21

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.14

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.7.dev0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.22

1.4.21

1.4.20

1.4.19

1.4.18

1.4.17

1.4.16

1.4.15

1.4.14

1.4.13

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.26

1.3.25

1.3.24

1.3.23

1.3.22

1.3.21

1.3.20

1.3.19

1.3.18

1.3.17

1.3.16

1.3.15

1.3.14

1.3.13

1.3.12

1.3.11

1.3.10

1.3.9

1.3.8

1.3.7

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.18

1.2.17

1.2.16

1.2.15

1.2.14

1.2.13

1.2.12

1.2.11

1.2.10

1.2.9

1.2.8

1.2.7

1.2.6

1.2.5

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.15

1.1.14

1.1.13

1.1.12

1.1.11

1.1.10

1.1.9

1.1.7

1.1.6

1.1.5

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.3

1.0.2

1.0.1

1.0.0

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

0.0.2

Labels

Clear labels

blocked

bug

data loss

design finalized

good first issue

new feature area

question / support

security

waiting for response

No Label

bug

question / support

Milestone

No items

No Milestone

Assignees

Clear assignees

diivi import_bot witten

No Assignees

3 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#899

No description provided.