How to prevent repeating output of the same by list, info, ... commands #479

New Issue

borg.haefemeier · 2021-12-11T14:41:08Z

borg.haefemeier commented

2021-12-11 14:41:08 +00:00

What I'm trying to do and why

I am curious how to prevent the same output several times if I use list, info, ... in case multiple configurations available for same repository. My configuration follows the manual to get a per-application backup.
My setup:

three configurations (two in /etc/borgmatic.d and one global /etc/borgmatic).
borgmatic create ... creates for each config a separate archive (with different pattern/name) as expected in same repository
borgmatic list list all archives in this repository but repeats the list in my case three times with same content

What did I miss to prevent the repeated output?

Other notes / implementation ideas

Environment

borgmatic version: 1.5.20

borgmatic installation method: dnf (rpm)

Borg version: 1.1.17

Python version: 3.10.0

Database version (if applicable): none

operating system and version: Linux 5.14.18-300.fc35.x86_64

#### What I'm trying to do and why I am curious how to prevent the same output several times if I use list, info, ... in case multiple configurations available for same repository. My configuration follows the manual to get a per-application backup. My setup: * three configurations (two in `/etc/borgmatic.d` and one global `/etc/borgmatic`). * `borgmatic create ...` creates for each config a separate archive (with different pattern/name) as expected in same repository * `borgmatic list` list all archives in this repository but repeats the list in my case three times with same content What did I miss to prevent the repeated output? #### Other notes / implementation ideas #### Environment **borgmatic version:** 1.5.20 **borgmatic installation method:** dnf (rpm) **Borg version:** 1.1.17 **Python version:** 3.10.0 **Database version (if applicable):** none **operating system and version:** Linux 5.14.18-300.fc35.x86_64

s1shed commented

2021-12-11 15:21:42 +00:00

I have a similar set-up, with multiple config files for each individual application.

I list the archives for a specific application by providing the config file used for that application.

borgmatic --config /path/to/config_file.yaml list

I have a similar set-up, with multiple config files for each individual application. I list the archives for a specific application by providing the config file used for that application. ``` borgmatic --config /path/to/config_file.yaml list ```

borg.haefemeier commented

2021-12-11 21:13:24 +00:00

I have a similar set-up, with multiple config files for each individual application.

I list the archives for a specific application by providing the config file used for that application.
borgmatic --config /path/to/config_file.yaml list

It list all archives in the repository once. But I would expected in this case only the archives of this configured application. I configured different archive names, so I have to use additionally a pattern with -P.

Sure it can be done in this way, but it isn't practical. Because I have to know each configuration details.

I would propose the following logic:

borgmatic list (without any parameter) display all archives of the repository but only once per repository or
repeat the output but restrict it each time with the archive pattern, which is configured (if present)

Similar logic in case of info command.

> I have a similar set-up, with multiple config files for each individual application. > > I list the archives for a specific application by providing the config file used for that application. > > ``` > borgmatic --config /path/to/config_file.yaml list > ``` It list **all** archives in the repository once. But I would expected in this case only the archives of this configured application. I configured different archive names, so I have to use additionally a pattern with `-P`. Sure it can be done in this way, but it isn't practical. Because I have to know each configuration details. I would propose the following logic: * `borgmatic list` (without any parameter) display all archives of the repository but only once per repository *or* * repeat the output but restrict it each time with the archive pattern, which is configured (if present) Similar logic in case of `info` command.

borg.haefemeier commented

2021-12-12 20:09:17 +00:00

Starting to experience more details about my setup, diff do the same repeatings, which makes no sense. Diff compare two archives related to a specific repository. If I don't define a configuration file I get the same result several times (as much configs as I have).

What does it mean in detail to specify a configuration file? I would expect the command is related to the configuration, but the command is working whatever configuration I specify. But if I use a per-application file, the command is maybe not working, because not all details are set.

Side note: Currently the whole setup (in my environment) is working, because it seems to merge the global configuration /etc/borgmatic with per-application config from /etc/borgmatic.d. I have specified not all parameter in all. For example the parameter encryption_passcommand is only defined in global config. But it is used (automatically without include) for all configurations during a backup run.

Maybe it is a bug, but it is a good one for me;-). Because it makes it easier to define seperated configs, without forgetting basic stuff respectively to force config parameter globally. An include statement can be forgotten...

Otherwise if I use a different repo location in one config file, I have to use the right one in case to call diff for example.

Just my 50 cents. Hope I could made my thoughts clear and I am open for discussion...

Starting to experience more details about my setup, `diff` do the same repeatings, which makes no sense. Diff compare two archives related to a specific repository. If I don't define a configuration file I get the same result several times (as much configs as I have). What does it mean in detail to specify a configuration file? I would expect the command is related to the configuration, but the command is working whatever configuration I specify. But if I use a per-application file, the command is maybe not working, because not all details are set. Side note: *Currently the whole setup (in my environment) is working, because it seems to merge the global configuration `/etc/borgmatic` with per-application config from `/etc/borgmatic.d`. I have specified not all parameter in all. For example the parameter `encryption_passcommand` is only defined in global config. But it is used (automatically without include) for all configurations during a backup run.* Maybe it is a bug, but it is a good one for me;-). Because it makes it easier to define seperated configs, without forgetting basic stuff respectively to force config parameter globally. An include statement can be forgotten... Otherwise if I use a different repo location in one config file, I have to use the right one in case to call `diff` for example. Just my 50 cents. Hope I could made my thoughts clear and I am open for discussion...

witten commented

2023-03-28 23:12:36 +00:00

I apologize for the delay in getting to this. This does sound to me like an obnoxious "feature" of using the same repository in multiple borgmatic configuration files, and I agree that it's something borgmatic should address.

I would propose the following logic:

borgmatic list (without any parameter) display all archives of the repository
but only once per repository or

The tricky part here is that each configuration file, even if it contains the exact same repository, has different options that can affect the repository listing/info. For instance if each configuration file has a different local_path or remote_path or lock_wait, it's not immediately clear which values to use when listing a de-duplicated repository. So that's why borgmatic currently executes list, info, and other actions within the context of a single configuration file at a time.

repeat the output but restrict it each time with the archive pattern, which is
configured (if present)

That sounds reasonable, but this is also a little tricky! The archive_name_format can be arbitrary complex, and so currently it's up to the user to define a consistency and/or retention prefix that matches the archive names resulting from the configured format.

So do you think it would solve your use case if there was another prefix (or glob_archives/match_archives) option that you could configure in each configuration file to influence how borgmatic ran list or info for the repository in that file? The main downside is it would be up to you to configure it.

An alternative would be for borgmatic to automatically try to parse archive_name_format and construct a glob pattern that matches archives named with it. But without some experimentation, I'm not 100% sure such an approach would work. The upside is that the user wouldn't have to go configuring prefixes or other options.

For example the parameter encryption_passcommand is only defined in global config. But it is used (automatically without include) for all configurations during a backup run.

I believe this was a bug that (unfortunately) has since been fixed (#555)!

Please let me know your thoughts when you get a chance.

I apologize for the delay in getting to this. This does sound to me like an obnoxious "feature" of using the same repository in multiple borgmatic configuration files, and I agree that it's something borgmatic should address. > I would propose the following logic: > > borgmatic list (without any parameter) display all archives of the repository > but only once per repository or The tricky part here is that each configuration file, even if it contains the exact same repository, has different options that can affect the repository listing/info. For instance if each configuration file has a different `local_path` or `remote_path` or `lock_wait`, it's not immediately clear which values to use when listing a de-duplicated repository. So that's why borgmatic currently executes `list`, `info`, and other actions within the context of a single configuration file at a time. > repeat the output but restrict it each time with the archive pattern, which is > configured (if present) That sounds reasonable, but this is also a little tricky! The `archive_name_format` can be arbitrary complex, and so currently it's up to the user to define a `consistency` and/or `retention` `prefix` that matches the archive names resulting from the configured format. So do you think it would solve your use case if there was another `prefix` (or `glob_archives`/`match_archives`) option that you could configure in each configuration file to influence how borgmatic ran `list` or `info` for the repository in that file? The main downside is it would be up to you to configure it. An alternative would be for borgmatic to automatically try to parse `archive_name_format` and construct a glob pattern that matches archives named with it. But without some experimentation, I'm not 100% sure such an approach would work. The upside is that the user wouldn't have to go configuring prefixes or other options. > For example the parameter encryption_passcommand is only defined in global config. But it is used (automatically without include) for all configurations during a backup run. I believe this was a bug that (unfortunately) has since been fixed (#555)! Please let me know your thoughts when you get a chance.

witten referenced this issue

2023-03-29 18:54:06 +00:00

Suggestion: Limiting archive listings according to what's specified in the config file #334

witten referenced this issue from a commit

2023-03-31 22:21:26 +00:00

Automatically use the "archive_name_format" option to filter which archives get used for borgmatic actions that operate on multiple archives (#479).

witten commented

2023-03-31 22:31:19 +00:00

At long last, this is implemented in master and will be part of the next release. Rather than doubling down on prefix, I went with this approach:

An alternative would be for borgmatic to automatically try to parse archive_name_format and construct a glob pattern that matches archives named with it. ... The upside is that the user wouldn't have to go configuring prefixes or other options.

Here's an example of it in use:

$ ag ' repositories:' -A 1 test.yaml other.yaml
test.yaml
21:    repositories:
22-        - 1.2.borg

other.yaml
21:    repositories:
22-        - 1.2.borg

$ ag archive_name_format: test.yaml other.yaml
test.yaml
234:    archive_name_format: 'first-documents-{now}'

other.yaml
234:    archive_name_format: 'second-documents-{now}'

$ borgmatic rlist -c test.yaml other.yaml -v 1
1.2.borg: Listing repository
first-documents-2023-03-29T17:47:01  Wed, 2023-03-29 17:47:01 [...]
first-documents-2023-03-29T17:47:28  Wed, 2023-03-29 17:47:28 [...]
first-documents-2023-03-30T11:59:24  Thu, 2023-03-30 11:59:24 [...]
first-documents-2023-03-30T22:01:27  Thu, 2023-03-30 22:01:27 [...]
1.2.borg: Listing repository
second-documents-2023-03-29T17:47:02 Wed, 2023-03-29 17:47:03 [...]
second-documents-2023-03-29T17:47:30 Wed, 2023-03-29 17:47:30 [...]
second-documents-2023-03-30T11:59:25 Thu, 2023-03-30 11:59:26 [...]

The documentation on this feature is here: https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/#archive-naming

Thanks for filing this ticket to being with!

Also see #588, which is related and will probably be implemented as a way to override the feature in this ticket (#479).

At long last, this is implemented in master and will be part of the next release. Rather than doubling down on `prefix`, I went with this approach: > An alternative would be for borgmatic to automatically try to parse archive_name_format and construct a glob pattern that matches archives named with it. ... The upside is that the user wouldn't have to go configuring prefixes or other options. Here's an example of it in use: ```bash $ ag ' repositories:' -A 1 test.yaml other.yaml test.yaml 21: repositories: 22- - 1.2.borg other.yaml 21: repositories: 22- - 1.2.borg $ ag archive_name_format: test.yaml other.yaml test.yaml 234: archive_name_format: 'first-documents-{now}' other.yaml 234: archive_name_format: 'second-documents-{now}' $ borgmatic rlist -c test.yaml other.yaml -v 1 1.2.borg: Listing repository first-documents-2023-03-29T17:47:01 Wed, 2023-03-29 17:47:01 [...] first-documents-2023-03-29T17:47:28 Wed, 2023-03-29 17:47:28 [...] first-documents-2023-03-30T11:59:24 Thu, 2023-03-30 11:59:24 [...] first-documents-2023-03-30T22:01:27 Thu, 2023-03-30 22:01:27 [...] 1.2.borg: Listing repository second-documents-2023-03-29T17:47:02 Wed, 2023-03-29 17:47:03 [...] second-documents-2023-03-29T17:47:30 Wed, 2023-03-29 17:47:30 [...] second-documents-2023-03-30T11:59:25 Thu, 2023-03-30 11:59:26 [...] ``` The documentation on this feature is here: https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/#archive-naming Thanks for filing this ticket to being with! Also see #588, which is related and will probably be implemented as a way to override the feature in this ticket (#479).

👍 1

witten closed this issue

2023-03-31 22:31:19 +00:00

witten referenced this issue from a commit

2023-03-31 22:37:16 +00:00

Document wording tweaks (#479).

witten referenced this issue

2023-06-26 05:33:15 +00:00

borgmatic list iterates over each config but shows the same (entire) archive list each time #365

Sign in to join this conversation.