Error parsing arguments with multiple verbosity flags #716

New Issue

Closed

opened 2023-06-23 17:26:00 +00:00 by witten · 4 comments

witten commented 2023-06-23 17:26:00 +00:00

Owner

Copy Link

What I'm trying to do and why

Test multiple verbosity flags at once during development. (This came up while doing repros on #713.)

Steps to reproduce

borgmatic -c test.yaml create --verbosity 2 --log-file-verbosity 2

Actual behavior

usage: borgmatic [-h] [-c [CONFIG_PATHS ...]] [--excludes EXCLUDES_FILENAME] [-n] [-nc] [-v {-2,-1,0,1,2}] [--syslog-verbosity {-2,-1,0,1,2}]
                 [--log-file-verbosity {-2,-1,0,1,2}] [--monitoring-verbosity {-2,-1,0,1,2}] [--log-file LOG_FILE]
                 [--log-file-format LOG_FILE_FORMAT] [--log-json] [--override SECTION.OPTION=VALUE [SECTION.OPTION=VALUE ...]]
                 [--no-environment-interpolation] [--bash-completion] [--fish-completion] [--version]
                 ...
borgmatic: error: argument --log-file-verbosity: expected one argument

Expected behavior

The create action runs without argument parse errors.

Other notes / implementation ideas

Environment

borgmatic version: main (this does not repro in 1.7.14)

borgmatic installation method: editable

Borg version: 1.2.4

Python version: 3.10.10

operating system and version: Manjaro stable

#### What I'm trying to do and why Test multiple verbosity flags at once during development. (This came up while doing repros on #713.) #### Steps to reproduce ``` borgmatic -c test.yaml create --verbosity 2 --log-file-verbosity 2 ``` #### Actual behavior ``` usage: borgmatic [-h] [-c [CONFIG_PATHS ...]] [--excludes EXCLUDES_FILENAME] [-n] [-nc] [-v {-2,-1,0,1,2}] [--syslog-verbosity {-2,-1,0,1,2}] [--log-file-verbosity {-2,-1,0,1,2}] [--monitoring-verbosity {-2,-1,0,1,2}] [--log-file LOG_FILE] [--log-file-format LOG_FILE_FORMAT] [--log-json] [--override SECTION.OPTION=VALUE [SECTION.OPTION=VALUE ...]] [--no-environment-interpolation] [--bash-completion] [--fish-completion] [--version] ... borgmatic: error: argument --log-file-verbosity: expected one argument ``` #### Expected behavior The `create` action runs without argument parse errors. #### Other notes / implementation ideas #### Environment **borgmatic version:** main (this does *not* repro in 1.7.14) **borgmatic installation method:** editable **Borg version:** 1.2.4 **Python version:** 3.10.10 **operating system and version:** Manjaro stable

witten added the bug label 2023-06-23 17:26:06 +00:00

witten referenced this issue 2023-06-23 17:26:32 +00:00

'Not a directory' errors when running consistency check #713

witten referenced this issue 2023-06-23 17:35:14 +00:00

'Not a directory' errors when running consistency check #713

witten commented 2023-06-23 18:03:22 +00:00

Author

Owner

Copy Link

Localized the problem to get_unparsable_arguments().. It's effectively dropping arguments as part of unique-ifying them. So interestingly this bug doesn't repro if all the verbosity values are different!

Localized the problem to `get_unparsable_arguments()`.. It's effectively dropping arguments as part of unique-ifying them. So interestingly this bug doesn't repro if all the verbosity values are different!

witten commented 2023-06-23 18:13:30 +00:00

Author

Owner

Copy Link

Looking back at the PR where I proposed the approach to begin with, I guess I was onto something about this potentially breaking stuff....

The one thing I'm not entirely sure about is the unique-ifying done with the dict.fromkeys() call. It's necessary so you don't end up with a duplicated result like ['--wtf', '--wtf', '--wtf', '--wtf'], but I haven't thought through whether that would break other stuff. ("Duplicated" arguments like extract in borgmatic --progress check --only extract extract --archive latest do not break, because they don't end up in remaining_subparser_arguments as they're consumed by the subparsers.)

Looking back at [the PR](https://github.com/borgmatic-collective/borgmatic/pull/71#issuecomment-1577651275) where I proposed the approach to begin with, I guess I was onto something about this potentially breaking stuff.... > The one thing I'm not entirely sure about is the unique-ifying done with the dict.fromkeys() call. It's necessary so you don't end up with a duplicated result like ['--wtf', '--wtf', '--wtf', '--wtf'], but I haven't thought through whether that would break other stuff. ("Duplicated" arguments like extract in borgmatic --progress check --only extract extract --archive latest do not break, because they don't end up in remaining_subparser_arguments as they're consumed by the subparsers.)

witten referenced this issue from a commit 2023-06-24 21:11:37 +00:00

Fix error parsing arguments with multiple verbosity flags (#716).

witten commented 2023-06-24 21:19:19 +00:00

Author

Owner

Copy Link

Fixed.

The old approach was to get unparsable arguments after all actions parse args but before the global parser parses them. That could leave the global parser with deduped/uniqued arguments, which you didn't want if you had two flags (like --verbosity and --log-file-verbosity) with the exact same value. Because that same duplicated value would get de-duplicated and then you'd have a bad time.

The new approach is to still call that same get_unparsable_arguments() function, but do it after all arguments have been parsed, including for each action and at the global level. Then, get_unparsable_arguments() still exists but strictly to find unknown arguments that no parser knows what to do with (e.g., borgmatic --wtf). Then the deduplication doesn't matter. (If someone calls borgmatic --wtf --wtf, borgmatic will still error as expected.)

Making this change though required several other changes like switching the global parser to using parse_known_args() instead of parse_args(), which itself had upstream implications on the construction of the parsers and even how --help was rendered.

Long story short, I hate argparse and this should be fixed now.

Fixed. The old approach was to get unparsable arguments after all actions parse args but before the global parser parses them. That could leave the global parser with deduped/uniqued arguments, which you *didn't* want if you had two flags (like `--verbosity` and `--log-file-verbosity`) with the exact same value. Because that same duplicated value would get de-duplicated and then you'd have a bad time. The new approach is to still call that same `get_unparsable_arguments()` function, but do it after *all* arguments have been parsed, including for each action *and* at the global level. Then, `get_unparsable_arguments()` still exists but strictly to find unknown arguments that no parser knows what to do with (e.g., `borgmatic --wtf`). Then the deduplication doesn't matter. (If someone calls `borgmatic --wtf --wtf`, borgmatic will still error as expected.) Making this change though required several other changes like switching the global parser to using `parse_known_args()` instead of `parse_args()`, which itself had upstream implications on the construction of the parsers and even how `--help` was rendered. Long story short, I hate `argparse` and this should be fixed now.

witten closed this issue 2023-06-24 21:19:26 +00:00

witten referenced this issue from a commit 2023-06-24 21:29:20 +00:00

Remove duplicated tests (#716).

witten referenced this issue from a commit 2023-06-24 22:36:12 +00:00

Fix edge case in which "--config somepath.yaml" followed by an action alias (e.g. init for rcreate) wasn't parsed correctly (#716).

witten commented 2023-06-25 05:31:31 +00:00

Author

Owner

Copy Link

Just released in borgmatic 1.7.15.

Just released in borgmatic 1.7.15.

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

command-hook-steps

2.1.3

2.1.2

2.1.1

2.1.0

2.0.13

2.0.12

2.0.11

2.0.10

2.0.9

2.0.8

2.0.7

2.0.6

2.0.5

2.0.4

2.0.3

2.0.2

2.0.1

2.0.0

1.9.14

1.9.13

1.9.12

1.9.11

1.9.10

1.9.9

1.9.8

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.14

1.8.13

1.8.12

1.8.11

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.24

1.5.23

1.5.22

1.5.21

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.14

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.7.dev0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.22

1.4.21

1.4.20

1.4.19

1.4.18

1.4.17

1.4.16

1.4.15

1.4.14

1.4.13

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.26

1.3.25

1.3.24

1.3.23

1.3.22

1.3.21

1.3.20

1.3.19

1.3.18

1.3.17

1.3.16

1.3.15

1.3.14

1.3.13

1.3.12

1.3.11

1.3.10

1.3.9

1.3.8

1.3.7

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.18

1.2.17

1.2.16

1.2.15

1.2.14

1.2.13

1.2.12

1.2.11

1.2.10

1.2.9

1.2.8

1.2.7

1.2.6

1.2.5

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.15

1.1.14

1.1.13

1.1.12

1.1.11

1.1.10

1.1.9

1.1.7

1.1.6

1.1.5

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.3

1.0.2

1.0.1

1.0.0

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

0.0.2

Labels

Clear labels

blocked

breaking

bug

data loss

design finalized

good first issue

new feature area

question / support

security

waiting for response

No Label bug

Milestone

No items

No Milestone

Assignees

Clear assignees

witten

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#716