borgmatic seems to be confused re --glob-archives and --match-archives #666

New Issue

Closed

opened 2023-04-04 01:52:26 +00:00 by lingfish · 9 comments

lingfish commented 2023-04-04 01:52:26 +00:00

Contributor

Copy Link

What I'm trying to do and why

Trying to run borgmatic info with the -a argument.

Steps to reproduce (if a bug)

Run borgmatic -v2 -c /etc/borgmatic/config.yaml info -a '*a*'

Actual behavior (if a bug)

Ensuring legacy configuration is upgraded
borg --version --debug --show-rc
/etc/borgmatic/config.yaml: No commands to run for pre-actions hook
ssh://host/./NAS: Displaying archive summary information
borg info --debug --show-rc --match-archives *a* ssh://host/./NAS
usage: borg [-V] [-h] [--critical] [--error] [--warning] [--info] [--debug]
            [--debug-topic TOPIC] [-p] [--iec] [--log-json]
            [--lock-wait SECONDS] [--bypass-lock] [--show-version] [--show-rc]
            [--umask M] [--remote-path PATH] [--remote-ratelimit RATE]
            [--upload-ratelimit RATE] [--remote-buffer UPLOAD_BUFFER]
            [--upload-buffer UPLOAD_BUFFER] [--consider-part-files]
            [--debug-profile FILE] [--rsh RSH]
            <command> ...
borg: error: unrecognized arguments: --match-archives ssh://host/./NAS
ssh://host/./NAS: Error running actions for repository
Command 'borg info --debug --show-rc --match-archives *a* ssh://host/./NAS' returned non-zero exit status 2.
/etc/borgmatic/config.yaml: Error running configuration file

As you can see, I'm using borg 1, but the newer --match-archives is being supplied.

Expected behavior (if a bug)

Normal listing.

Other notes / implementation ideas

When run in an identical PyCharm environment, the feature args get even weirder:

borg info --debug --show-rc --glob-archives {hostname}-nas-* --match-archives '*a*' ssh://host/./NAS

Environment

borgmatic version: 1.7.9 and 1.7.11

borgmatic installation method: Python venv

Borg version: 1.2.2

Python version: 3.9.2

Database version (if applicable): [version here]

operating system and version: Debian 11

#### What I'm trying to do and why Trying to run borgmatic info with the `-a` argument. #### Steps to reproduce (if a bug) Run `borgmatic -v2 -c /etc/borgmatic/config.yaml info -a '*a*'` #### Actual behavior (if a bug) ``` Ensuring legacy configuration is upgraded borg --version --debug --show-rc /etc/borgmatic/config.yaml: No commands to run for pre-actions hook ssh://host/./NAS: Displaying archive summary information borg info --debug --show-rc --match-archives *a* ssh://host/./NAS usage: borg [-V] [-h] [--critical] [--error] [--warning] [--info] [--debug] [--debug-topic TOPIC] [-p] [--iec] [--log-json] [--lock-wait SECONDS] [--bypass-lock] [--show-version] [--show-rc] [--umask M] [--remote-path PATH] [--remote-ratelimit RATE] [--upload-ratelimit RATE] [--remote-buffer UPLOAD_BUFFER] [--upload-buffer UPLOAD_BUFFER] [--consider-part-files] [--debug-profile FILE] [--rsh RSH] <command> ... borg: error: unrecognized arguments: --match-archives ssh://host/./NAS ssh://host/./NAS: Error running actions for repository Command 'borg info --debug --show-rc --match-archives *a* ssh://host/./NAS' returned non-zero exit status 2. /etc/borgmatic/config.yaml: Error running configuration file ``` As you can see, I'm using borg 1, but the newer --match-archives is being supplied. #### Expected behavior (if a bug) Normal listing. #### Other notes / implementation ideas When run in an identical PyCharm environment, the feature args get even weirder: `borg info --debug --show-rc --glob-archives {hostname}-nas-* --match-archives '*a*' ssh://host/./NAS` #### Environment **borgmatic version:** 1.7.9 and 1.7.11 **borgmatic installation method:** Python venv **Borg version:** 1.2.2 **Python version:** 3.9.2 **Database version (if applicable):** [version here] **operating system and version:** Debian 11

witten commented 2023-04-04 03:22:16 +00:00

Owner

Copy Link

Thank you so much for reporting this! I've confirmed the behavior with Borg 1.2. I'm not quite sure what's going on yet, because the code is supposed to check the local Borg version and use --match-archives or --glob-archives as appropriate. I'll dig into this and let you know what I find.

Thank you so much for reporting this! I've confirmed the behavior with Borg 1.2. I'm not quite sure what's going on yet, because the code is *supposed* to check the local Borg version and use `--match-archives` or `--glob-archives` as appropriate. I'll dig into this and let you know what I find.

witten added the bug label 2023-04-04 03:22:22 +00:00

lingfish commented 2023-04-04 03:37:31 +00:00

Author

Contributor

Copy Link

Yep, agreed... I did see the cool stuff being done in borg/feature.py etc, and even ran tests via tox -- even made another test as below and it strangely passed!

def test_version_checker():
    assert not module.available(module.Feature.MATCH_ARCHIVES, '1.2.2')

Yep, agreed... I did see the cool stuff being done in `borg/feature.py` etc, and even ran tests via `tox` -- even made another test as below and it strangely passed! ``` def test_version_checker(): assert not module.available(module.Feature.MATCH_ARCHIVES, '1.2.2') ```

diivi commented 2023-04-04 04:10:31 +00:00

Collaborator

Copy Link

I Was able to debug this:

The problem is with this line in info.py::display_archives_info() (line55):

flags.make_flags_from_arguments(
            info_arguments, excludes=('repository', 'archive', 'prefix')
        )

When I added match_prefix to the excludes argument for make_flags_from_arguments, the error did not occur anymore.

I Was able to debug this: The problem is with this line in `info.py::display_archives_info()` (line55): ```python flags.make_flags_from_arguments( info_arguments, excludes=('repository', 'archive', 'prefix') ) ``` When I added `match_prefix` to the `excludes` argument for `make_flags_from_arguments`, the error did not occur anymore.

witten commented 2023-04-04 04:42:36 +00:00

Owner

Copy Link

Aha! Nice catch. I'll make that fix and see if there's a way to catch this sort of issue in an integration test.

Aha! Nice catch. I'll make that fix and see if there's a way to catch this sort of issue in an integration test.

👍 1

lingfish commented 2023-04-04 04:51:51 +00:00

Author

Contributor

Copy Link

I Was able to debug this:

The problem is with this line in info.py::display_archives_info() (line55):

flags.make_flags_from_arguments(
            info_arguments, excludes=('repository', 'archive', 'prefix')
        )

When I added match_prefix to the excludes argument for make_flags_from_arguments, the error did not occur anymore.

Not being fully immersed in the code, would this fix not need to be in other lines too, wherever --archive is accepted?

> I Was able to debug this: > > The problem is with this line in `info.py::display_archives_info()` (line55): > > ```python > flags.make_flags_from_arguments( > info_arguments, excludes=('repository', 'archive', 'prefix') > ) > ``` > > When I added `match_prefix` to the `excludes` argument for `make_flags_from_arguments`, the error did not occur anymore. Not being fully immersed in the code, would this fix not need to be in other lines too, wherever `--archive` is accepted?

witten commented 2023-04-04 05:41:50 +00:00

Owner

Copy Link

Yes, this is likely a bug in multiple different actions. And what's worse, it looks like even once fixed, the -a/--match-archives/--glob-archives flag on the command-line isn't correctly overriding any of the other places that value can come from. Fortunately the fix should be pretty straightforward.

Yes, this is likely a bug in multiple different actions. And what's worse, it looks like even once fixed, the `-a`/`--match-archives`/`--glob-archives` flag on the command-line isn't correctly overriding any of the [other places](https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/#archive-naming) that value can come from. Fortunately the fix should be pretty straightforward.

👍 2

witten referenced this issue from a commit 2023-04-05 04:26:05 +00:00

Fix error with "info --match-archives" and fix "--match-archives" overriding logic (#666).

witten commented 2023-04-05 04:29:31 +00:00

Owner

Copy Link

This has been fixed in master for all relevant actions. I also fixed the related --match-archives override problem mentioned above. Lastly, I added a few integration tests that lightly "fuzz" some of the the functions that produce Borg arguments to hopefully catch related issues in the future.

These fixes will be part of the next release. Thanks again for taking the time to file this. And thanks to @diivi for diagnosing!

This has been fixed in master for all relevant actions. I also fixed the related `--match-archives` override problem mentioned above. Lastly, I added a few integration tests that lightly "fuzz" some of the the functions that produce Borg arguments to hopefully catch related issues in the future. These fixes will be part of the next release. Thanks again for taking the time to file this. And thanks to @diivi for diagnosing!

🚀 1

witten closed this issue 2023-04-05 04:29:31 +00:00

lingfish commented 2023-04-05 04:41:19 +00:00

Author

Contributor

Copy Link

You're awesome Dan, always a pleasure dealing with you :)

You're awesome Dan, always a pleasure dealing with you :)

👍 2

witten referenced this issue from a commit 2023-04-05 05:12:05 +00:00

Fix integration tests to actually assert (#666).

witten referenced this issue 2023-04-05 20:53:43 +00:00

Prune action with prefix sends duplicate --glob-archives flags to Borg #668

witten referenced this issue 2023-04-09 16:58:07 +00:00

`borgmatic info --archive latest --json` creates invalid borg invocation including duplicate `--glob-archives` argument #670

witten commented 2023-04-13 22:18:13 +00:00

Owner

Copy Link

This fix has been released as part of borgmatic 1.7.12!

This fix has been released as part of borgmatic 1.7.12!

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

command-hook-steps

2.1.2

2.1.1

2.1.0

2.0.13

2.0.12

2.0.11

2.0.10

2.0.9

2.0.8

2.0.7

2.0.6

2.0.5

2.0.4

2.0.3

2.0.2

2.0.1

2.0.0

1.9.14

1.9.13

1.9.12

1.9.11

1.9.10

1.9.9

1.9.8

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.14

1.8.13

1.8.12

1.8.11

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.24

1.5.23

1.5.22

1.5.21

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.14

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.7.dev0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.22

1.4.21

1.4.20

1.4.19

1.4.18

1.4.17

1.4.16

1.4.15

1.4.14

1.4.13

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.26

1.3.25

1.3.24

1.3.23

1.3.22

1.3.21

1.3.20

1.3.19

1.3.18

1.3.17

1.3.16

1.3.15

1.3.14

1.3.13

1.3.12

1.3.11

1.3.10

1.3.9

1.3.8

1.3.7

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.18

1.2.17

1.2.16

1.2.15

1.2.14

1.2.13

1.2.12

1.2.11

1.2.10

1.2.9

1.2.8

1.2.7

1.2.6

1.2.5

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.15

1.1.14

1.1.13

1.1.12

1.1.11

1.1.10

1.1.9

1.1.7

1.1.6

1.1.5

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.3

1.0.2

1.0.1

1.0.0

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

0.0.2

Labels

Clear labels

blocked

breaking

bug

data loss

design finalized

good first issue

new feature area

question / support

security

waiting for response

No Label bug

Milestone

No items

No Milestone

Assignees

Clear assignees

witten

No Assignees

3 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#666