borgmatic list iterates over each config but shows the same (entire) archive list each time #365

Closed
opened 2020-10-31 11:10:45 +00:00 by fallen · 10 comments

What I'm trying to do and why

I am trying to list my borg repo archives.

Steps to reproduce (if a bug)

I created my borgmatic configs using the provided ansible modules.

Actual behavior (if a bug)

It iterates and runs "borg list --debug --show-rc --lock-wait 5 REPOURL" N times.
with N being the number of configs.
So that it shows N times the same (big) archive list.

Expected behavior (if a bug)

I would expect to either

  • have N commands run, each showing the archives related to a config file
    or
  • have only 1 command run which shows all the archives

Other notes / implementation ideas

Environment

borgmatic version: 1.5.10

borgmatic installation method: ansible

Borg version: 1.1.14

Python version: 3.5.3

Database version (if applicable): mysql Ver 15.1 Distrib 10.1.45-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2

operating system and version: Linux x86_64 4.9.185, Debian stretch 9.13

#### What I'm trying to do and why I am trying to list my borg repo archives. #### Steps to reproduce (if a bug) I created my borgmatic configs using the provided ansible modules. #### Actual behavior (if a bug) It iterates and runs "borg list --debug --show-rc --lock-wait 5 REPOURL" N times. with N being the number of configs. So that it shows N times the same (big) archive list. #### Expected behavior (if a bug) I would expect to either * have N commands run, each showing the archives related to a config file or * have only 1 command run which shows all the archives #### Other notes / implementation ideas #### Environment **borgmatic version:** 1.5.10 **borgmatic installation method:** ansible **Borg version:** 1.1.14 **Python version:** 3.5.3 **Database version (if applicable):** mysql Ver 15.1 Distrib 10.1.45-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2 **operating system and version:** Linux x86_64 4.9.185, Debian stretch 9.13
fallen changed title from borgmatic list iterates over each config but shows the same (entire) archive list to borgmatic list iterates over each config but shows the same (entire) archive list each time 2020-10-31 11:52:07 +00:00
Owner

Thanks for taking the time to file this one! Would you mind either including your relevant borgmatic configuration files (with sensitive stuff redacted), or just paste in the list of repositories in each config? For bonus points, an example of the command-line input and output you're seeing would help too. I'm just trying to understand the problem fully.

For instance, do you have the same repository listed in multiple different borgmatic configuration files?

Thanks for taking the time to file this one! Would you mind either including your relevant borgmatic configuration files (with sensitive stuff redacted), or just paste in the list of repositories in each config? For bonus points, an example of the command-line input and output you're seeing would help too. I'm just trying to understand the problem fully. For instance, do you have the same repository listed in multiple different borgmatic configuration files?
Author

For instance, do you have the same repository listed in multiple different borgmatic configuration files?

Yes it's the same repo in all the config files.
I have one config file for each service, so basically they have different "source_directories", and different prefix.

For instance I've put as attached files 2 config files (out of the 6 I have).

> For instance, do you have the same repository listed in multiple different borgmatic configuration files? Yes it's the same repo in all the config files. I have one config file for each service, so basically they have different "source_directories", and different prefix. For instance I've put as attached files 2 config files (out of the 6 I have).
Author
~# borgmatic list -v 2
Ensuring legacy configuration is upgraded
user@host:repo: Listing archives
borg list --debug --show-rc --lock-wait 5 user@host:repo
using builtin fallback logging configuration
35 self tests completed in 0.51 seconds
SSH command line: ['ssh', 'user@host', 'borg', 'serve', '--umask=077', '--debug']
Remote: using builtin fallback logging configuration
Remote: 35 self tests completed in 7.13 seconds
Remote: using builtin fallback logging configuration
Remote: Initialized logging system for JSON-based protocol
Remote: Resolving repository path b'repo'
Remote: Resolved repository path to '/srv/repos/user/repo'
Remote: Verified integrity of /srv/repos/user/repo/index.448
TAM-verified manifest
security: read previous location 'ssh://user@host/./repo'
security: read manifest timestamp '2020-10-31T04:08:52.171073'
security: determined newest manifest timestamp as 2020-10-31T04:08:52.171073
security: repository checks ok, allowing access
nginx-2020-10-17T20:59:21            Sat, 2020-10-17 20:59:23 [f3da9ab3c4235afacf69882670e0c12f0e33ecf88fc30499a3f10b7ef99c205e]
uwsgi-2020-10-17T21:01:48            Sat, 2020-10-17 21:02:00 [45c79f0b7acd18dff87cd78794975a13863e44a7cc5bb42963dd5d4c7b7d92a1]
nginx-2020-10-17T21:12:20            Sat, 2020-10-17 21:12:22 [0c0db788a1f4f1cae32cecf93fd4401cf5a146f9f085eefd76f26be858c809b1]
[...]
RemoteRepository: 213 B bytes sent, 7.12 kB bytes received, 5 messages sent
terminating with success status, rc 0
user@host:repo: Listing archives

and then it goes again with same output
it happens multiple times and finally ends up with:

summary:
/etc/borgmatic.d/civicrm.yaml: Successfully ran configuration file
/etc/borgmatic.d/mysql.yaml: Successfully ran configuration file
/etc/borgmatic.d/nginx.yaml: Successfully ran configuration file
/etc/borgmatic.d/onarretetoutes.yaml: Successfully ran configuration file
/etc/borgmatic.d/peertube_config.yaml: Successfully ran configuration file
/etc/borgmatic.d/uwsgi.yaml: Successfully ran configuration file
``` ~# borgmatic list -v 2 Ensuring legacy configuration is upgraded user@host:repo: Listing archives borg list --debug --show-rc --lock-wait 5 user@host:repo using builtin fallback logging configuration 35 self tests completed in 0.51 seconds SSH command line: ['ssh', 'user@host', 'borg', 'serve', '--umask=077', '--debug'] Remote: using builtin fallback logging configuration Remote: 35 self tests completed in 7.13 seconds Remote: using builtin fallback logging configuration Remote: Initialized logging system for JSON-based protocol Remote: Resolving repository path b'repo' Remote: Resolved repository path to '/srv/repos/user/repo' Remote: Verified integrity of /srv/repos/user/repo/index.448 TAM-verified manifest security: read previous location 'ssh://user@host/./repo' security: read manifest timestamp '2020-10-31T04:08:52.171073' security: determined newest manifest timestamp as 2020-10-31T04:08:52.171073 security: repository checks ok, allowing access nginx-2020-10-17T20:59:21 Sat, 2020-10-17 20:59:23 [f3da9ab3c4235afacf69882670e0c12f0e33ecf88fc30499a3f10b7ef99c205e] uwsgi-2020-10-17T21:01:48 Sat, 2020-10-17 21:02:00 [45c79f0b7acd18dff87cd78794975a13863e44a7cc5bb42963dd5d4c7b7d92a1] nginx-2020-10-17T21:12:20 Sat, 2020-10-17 21:12:22 [0c0db788a1f4f1cae32cecf93fd4401cf5a146f9f085eefd76f26be858c809b1] [...] RemoteRepository: 213 B bytes sent, 7.12 kB bytes received, 5 messages sent terminating with success status, rc 0 user@host:repo: Listing archives ``` and then it goes again with same output it happens multiple times and finally ends up with: ``` summary: /etc/borgmatic.d/civicrm.yaml: Successfully ran configuration file /etc/borgmatic.d/mysql.yaml: Successfully ran configuration file /etc/borgmatic.d/nginx.yaml: Successfully ran configuration file /etc/borgmatic.d/onarretetoutes.yaml: Successfully ran configuration file /etc/borgmatic.d/peertube_config.yaml: Successfully ran configuration file /etc/borgmatic.d/uwsgi.yaml: Successfully ran configuration file ```
Author

I guess I'm doing this wrong?

I guess I'm doing this wrong?
Owner

Sorry for the delay in getting back to you! Having looked at your configuration files, I wouldn't say that you're doing this "wrong" at all. What's happening is that borgmatic runs borg list once for each configuration file, oblivious to the fact that you're using the same repository in both.

Your "expected behavior" make sense to me, but here are some challenges for each one:

  • have N commands run, each showing the archives related to a config file or

There's an implementation detail issue here, which is that borgmatic list is already making use of Borg's --glob-archives flag to exclude checkpoint archives. And it just so turns out that --glob-archives is incompatible with the --prefix flag necessary to filter down Borg's output to the archives related to a particular config file.

One solution I could see would be to stop excluding checkpoints when the borg list --prefix flag needs to used. But this seems like pretty unexpected behavior even if it were documented.

Another solution would be to drop --glob-archives entirely and tell anyone who wants their checkpoint archives to conitnue to be omitted to upgrade to Borg 1.1.14 which appears to do that by default (unverified).

  • have only 1 command run which shows all the archives

The challenge here is that a borg list command could in theory be different for each configuration file even if the repository is identical. For instance, the requested local/remote Borg file paths or lock wait value could differ across configuration files. And therefore result in different borg list flags for each configuration file.

One potential solution could be to skip subsequent runs of borg list if the exact command flags would be identical.

Let me ask you this though: Is there a reason you've got things split across two configuration files like you do now? Just to be able to split archives by function for ease of restore? Seems like a reasonable thing to do, but I just wanted to make sure you realize you could do it all with a single config file.. losing the separate archive prefixes in the process—that would be the cost of doing things that way.

In any case, let me know your thoughts on all of this.

Sorry for the delay in getting back to you! Having looked at your configuration files, I wouldn't say that you're doing this "wrong" at all. What's happening is that borgmatic runs `borg list` once for each configuration file, oblivious to the fact that you're using the same repository in both. Your "expected behavior" make sense to me, but here are some challenges for each one: > * have N commands run, each showing the archives related to a config file or There's an implementation detail issue here, which is that `borgmatic list` is already making use of Borg's `--glob-archives` flag to exclude checkpoint archives. And it just so turns out that `--glob-archives` is incompatible with the `--prefix` flag necessary to filter down Borg's output to the archives related to a particular config file. One solution I could see would be to stop excluding checkpoints when the `borg list --prefix` flag needs to used. But this seems like pretty unexpected behavior even if it were documented. Another solution would be to drop `--glob-archives` entirely and tell anyone who wants their checkpoint archives to conitnue to be omitted to upgrade to [Borg 1.1.14](https://github.com/borgbackup/borg/pull/4918/files) which appears to do that by default (unverified). > * have only 1 command run which shows all the archives The challenge here is that a `borg list` command could in theory be different for each configuration file even if the repository is identical. For instance, the requested local/remote Borg file paths or lock wait value could differ across configuration files. And therefore result in different `borg list` flags for each configuration file. One potential solution could be to skip subsequent runs of `borg list` if the exact command flags would be identical. Let me ask you this though: Is there a reason you've got things split across two configuration files like you do now? Just to be able to split archives by function for ease of restore? Seems like a reasonable thing to do, but I just wanted to make sure you realize you *could* do it all with a single config file.. losing the separate archive prefixes in the process—that would be the cost of doing things that way. In any case, let me know your thoughts on all of this.
Author

Thanks for getting back to me!
After reading you, I now realize this might be hard to implement.

Anyway the only reason I used several config file was just to separate each "service" to backup with its own borgmatic config file.
Just as a matter of cleanliness.
This way it's pretty clear which service needs which resource.
This helps getting my head clear about what needs to be backuped up (and therefore restored) for each individual service.

Thanks for getting back to me! After reading you, I now realize this might be hard to implement. Anyway the only reason I used several config file was just to separate each "service" to backup with its own borgmatic config file. Just as a matter of cleanliness. This way it's pretty clear which service needs which resource. This helps getting my head clear about what needs to be backuped up (and therefore restored) for each individual service.
Owner

Totally understood about the cleanliness and separate archives. Thanks for clarifying. I'll have to think about which of the possible solutions would work best here.

Totally understood about the cleanliness and separate archives. Thanks for clarifying. I'll have to think about which of the possible solutions would work best here.
Owner

A little late, but this looks like it's already been implemented in #479, released in borgmatic 1.7.11! More documentation on the feature is here: https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/#archive-naming

A little late, but this looks like it's already been implemented in #479, released in borgmatic 1.7.11! More documentation on the feature is here: https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/#archive-naming
Author

Awesome! Thanks a lot @witten :)

Awesome! Thanks a lot @witten :)
Owner

And thank you for your patience. 😄

And thank you for your patience. 😄
Sign in to join this conversation.
No Milestone
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#365
No description provided.