borgmatic-collective/borgmatic
10
71
Fork 43
Code Issues 48 Pull Requests 6 Actions 1 Packages Releases 153 Activity

Listing latest archive with multiple repositories conifgured only works for first one #706

New Issue
Closed
opened 2023-05-29 10:48:15 +00:00 by count · 4 comments
count commented 2023-05-29 10:48:15 +00:00
Copy Link

What I'm trying to do and why

I've got machines sending backups to multiple repositories and wanted to set up some monitoring for them. Only the archive information for the first one is shown.

Steps to reproduce (if a bug)

Include (sanitized) borgmatic configuration files if applicable.

location:
    source_directories:
     - /whatever
    repositories:
     - ssh://borgbackup@host1/backupstore
     - ssh://borgbackup@host2/backupstore
     - ssh://borgbackup@host3/backupstore

After backups:

$ borgmatic info --archive latest
ssh://borgbackup@host1/backupstore: Displaying archive summary information
Archive name: whatever-2023-05-29T12:14:25.854541
Archive fingerprint: XXX
Comment:
Hostname: whatever
Username: root
Time (start): Mon, 2023-05-29 12:14:27
Time (end): Mon, 2023-05-29 12:16:02
Duration: 1 minutes 35.87 seconds
Number of files: 667748
Command line: /usr/local/bin/borg create --exclude-from /tmp/tmpei_9zj40 --compression zstd --one-file-system --read-special 'ssh://borgbackup@host1/backupstore::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}' /whatever /root/.borgmatic --stats
Utilization of maximum supported archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:               20.34 GB             19.94 GB              2.02 MB
All archives:              819.66 GB            807.02 GB             20.05 GB
                       Unique chunks         Total chunks
Chunk index:                  673525             27030814
ssh://borgbackup@host2/backupstore: Displaying archive summary information
ssh://borgbackup@host3/backupstore: Displaying archive summary information

Note the lack of output for the secondary / tertiary repository. Also empty in the JSON output.

Expected output would be more like this:

$ borgmatic info --archive latest --json | jq -r .[].repository.location
ssh://borgbackup@host1/backupstore
ssh://borgbackup@host2/backupstore
ssh://borgbackup@host3/backupstore
$ borgmatic info --archive latest --json | jq -r .[].repository.location | xargs -n1 borgmatic info --archive latest --repository
ssh://borgbackup@host1/backupstore: Displaying archive summary information
Archive name: whatever-2023-05-29T12:14:25.854541
Archive fingerprint: XXX
Comment:
Hostname: whatever
Username: root
Time (start): Mon, 2023-05-29 12:14:27
Time (end): Mon, 2023-05-29 12:16:02
Duration: 1 minutes 35.87 seconds
Number of files: 667748
Command line: /usr/local/bin/borg create --exclude-from /tmp/tmpei_9zj40 --compression zstd --one-file-system --read-special 'ssh://host1/backupstore::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}' /whatever /root/.borgmatic --stats
Utilization of maximum supported archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:               20.34 GB             19.94 GB              2.02 MB
All archives:              819.66 GB            807.02 GB             20.05 GB
                       Unique chunks         Total chunks
Chunk index:                  673525             27030814
ssh://borgbackup@host2/backupstore: Displaying archive summary information
Archive name: whatever-2023-05-29T12:16:52.204399
Archive fingerprint: XXX
Comment:
Hostname: whatever
Username: root
Time (start): Mon, 2023-05-29 12:16:57
Time (end): Mon, 2023-05-29 12:18:34
Duration: 1 minutes 37.19 seconds
Number of files: 667748
Command line: /usr/local/bin/borg create --exclude-from /tmp/tmpohp18gtg --compression zstd --one-file-system --read-special 'ssh://borgbackup@host2/backupstore::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}' /whatever /root/.borgmatic --stats
Utilization of maximum supported archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:               20.34 GB             19.94 GB              2.47 MB
All archives:              801.23 GB            790.48 GB             20.06 GB
                       Unique chunks         Total chunks
Chunk index:                  673589             26205744
ssh://borgbackup@host3/backupstore
: Displaying archive summary information
Archive name: whatever-2023-05-29T12:19:51.427329
Archive fingerprint: XXX
Comment:
Hostname: whatever
Username: root
Time (start): Mon, 2023-05-29 12:19:55
Time (end): Mon, 2023-05-29 12:21:28
Duration: 1 minutes 33.29 seconds
Number of files: 667748
Command line: /usr/local/bin/borg create --exclude-from /tmp/tmpor2oyviu --compression zstd --one-file-system --read-special 'ssh://borgbackup@host3/backupstore
::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}' /whatever /root/.borgmatic --stats
Utilization of maximum supported archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:               20.34 GB             19.94 GB              2.06 MB
All archives:              762.41 GB            752.12 GB             20.05 GB
                       Unique chunks         Total chunks
Chunk index:                  673430             24939879

Actual behavior (if a bug)

Latest archive on first repository only is shown / represented in output/json.

Expected behavior (if a bug)

Latest archive on each repository is shown / represented in output/json.

Other notes / implementation ideas

My guess would be that the "latest" archive information is cached based on what's found on the first repository, and also tried to be shown from the second and third repository - instead of being refetched.

Environment

borgmatic version: [version here]

1.7.13

borgmatic installation method: [e.g., Debian package, Docker container, etc.]

https://docs.mailcow.email/third_party/borgmatic/third_party-borgmatic/

Borg version: [version here]

borg 1.2.4

Python version: [version here]

Python 3.11.2

Database version (if applicable): [version here]

mysql Ver 15.1 Distrib 10.6.13-MariaDB, for Linux (x86_64) using readline 5.1 c

operating system and version: [OS here]

Alpine 3.17.3

#### What I'm trying to do and why I've got machines sending backups to multiple repositories and wanted to set up some monitoring for them. Only the archive information for the first one is shown. #### Steps to reproduce (if a bug) Include (sanitized) borgmatic configuration files if applicable. ``` location: source_directories: - /whatever repositories: - ssh://borgbackup@host1/backupstore - ssh://borgbackup@host2/backupstore - ssh://borgbackup@host3/backupstore ```` After backups: ``` $ borgmatic info --archive latest ssh://borgbackup@host1/backupstore: Displaying archive summary information Archive name: whatever-2023-05-29T12:14:25.854541 Archive fingerprint: XXX Comment: Hostname: whatever Username: root Time (start): Mon, 2023-05-29 12:14:27 Time (end): Mon, 2023-05-29 12:16:02 Duration: 1 minutes 35.87 seconds Number of files: 667748 Command line: /usr/local/bin/borg create --exclude-from /tmp/tmpei_9zj40 --compression zstd --one-file-system --read-special 'ssh://borgbackup@host1/backupstore::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}' /whatever /root/.borgmatic --stats Utilization of maximum supported archive size: 0% ------------------------------------------------------------------------------ Original size Compressed size Deduplicated size This archive: 20.34 GB 19.94 GB 2.02 MB All archives: 819.66 GB 807.02 GB 20.05 GB Unique chunks Total chunks Chunk index: 673525 27030814 ssh://borgbackup@host2/backupstore: Displaying archive summary information ssh://borgbackup@host3/backupstore: Displaying archive summary information ``` Note the lack of output for the secondary / tertiary repository. Also empty in the JSON output. Expected output would be more like this: ``` $ borgmatic info --archive latest --json | jq -r .[].repository.location ssh://borgbackup@host1/backupstore ssh://borgbackup@host2/backupstore ssh://borgbackup@host3/backupstore $ borgmatic info --archive latest --json | jq -r .[].repository.location | xargs -n1 borgmatic info --archive latest --repository ssh://borgbackup@host1/backupstore: Displaying archive summary information Archive name: whatever-2023-05-29T12:14:25.854541 Archive fingerprint: XXX Comment: Hostname: whatever Username: root Time (start): Mon, 2023-05-29 12:14:27 Time (end): Mon, 2023-05-29 12:16:02 Duration: 1 minutes 35.87 seconds Number of files: 667748 Command line: /usr/local/bin/borg create --exclude-from /tmp/tmpei_9zj40 --compression zstd --one-file-system --read-special 'ssh://host1/backupstore::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}' /whatever /root/.borgmatic --stats Utilization of maximum supported archive size: 0% ------------------------------------------------------------------------------ Original size Compressed size Deduplicated size This archive: 20.34 GB 19.94 GB 2.02 MB All archives: 819.66 GB 807.02 GB 20.05 GB Unique chunks Total chunks Chunk index: 673525 27030814 ssh://borgbackup@host2/backupstore: Displaying archive summary information Archive name: whatever-2023-05-29T12:16:52.204399 Archive fingerprint: XXX Comment: Hostname: whatever Username: root Time (start): Mon, 2023-05-29 12:16:57 Time (end): Mon, 2023-05-29 12:18:34 Duration: 1 minutes 37.19 seconds Number of files: 667748 Command line: /usr/local/bin/borg create --exclude-from /tmp/tmpohp18gtg --compression zstd --one-file-system --read-special 'ssh://borgbackup@host2/backupstore::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}' /whatever /root/.borgmatic --stats Utilization of maximum supported archive size: 0% ------------------------------------------------------------------------------ Original size Compressed size Deduplicated size This archive: 20.34 GB 19.94 GB 2.47 MB All archives: 801.23 GB 790.48 GB 20.06 GB Unique chunks Total chunks Chunk index: 673589 26205744 ssh://borgbackup@host3/backupstore : Displaying archive summary information Archive name: whatever-2023-05-29T12:19:51.427329 Archive fingerprint: XXX Comment: Hostname: whatever Username: root Time (start): Mon, 2023-05-29 12:19:55 Time (end): Mon, 2023-05-29 12:21:28 Duration: 1 minutes 33.29 seconds Number of files: 667748 Command line: /usr/local/bin/borg create --exclude-from /tmp/tmpor2oyviu --compression zstd --one-file-system --read-special 'ssh://borgbackup@host3/backupstore ::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}' /whatever /root/.borgmatic --stats Utilization of maximum supported archive size: 0% ------------------------------------------------------------------------------ Original size Compressed size Deduplicated size This archive: 20.34 GB 19.94 GB 2.06 MB All archives: 762.41 GB 752.12 GB 20.05 GB Unique chunks Total chunks Chunk index: 673430 24939879 ``` #### Actual behavior (if a bug) Latest archive _on first repository only_ is shown / represented in output/json. #### Expected behavior (if a bug) Latest archive _on each repository_ is shown / represented in output/json. #### Other notes / implementation ideas My guess would be that the "latest" archive information is cached based on what's found on the first repository, and also tried to be shown from the second and third repository - instead of being refetched. #### Environment **borgmatic version:** [version here] `1.7.13` **borgmatic installation method:** [e.g., Debian package, Docker container, etc.] https://docs.mailcow.email/third_party/borgmatic/third_party-borgmatic/ **Borg version:** [version here] `borg 1.2.4` **Python version:** [version here] `Python 3.11.2` **Database version (if applicable):** [version here] `mysql Ver 15.1 Distrib 10.6.13-MariaDB, for Linux (x86_64) using readline 5.1 c` **operating system and version:** [OS here] `Alpine 3.17.3`
witten commented 2023-05-30 23:04:04 +00:00
Owner
Copy Link

Thanks so much for taking the time to file this. I have confirmed the behavior you're seeing, and I believe a similar problem occurs with the list action as well. Running info with --verbosity 2, it does look like borgmatic is passing the exact same archive name to the commands for each repository. So that will need to be addressed to solve this.

Thanks so much for taking the time to file this. I have confirmed the behavior you're seeing, and I believe a similar problem occurs with the `list` action as well. Running `info` with `--verbosity 2`, it does look like borgmatic is passing the exact same archive name to the commands for each repository. So that will need to be addressed to solve this.
witten added the bug label 2023-05-30 23:04:10 +00:00
witten commented 2023-05-30 23:56:21 +00:00
Owner
Copy Link

This is fixed in main and will be part of the next release. (The problem turned out not to be an intentionally cached value, but rather accidental caching due to code overwriting a value at global scope.) Thanks again!

This is fixed in main and will be part of the next release. (The problem turned out not to be an intentionally cached value, but rather accidental caching due to code overwriting a value at global scope.) Thanks again!
count commented 2023-06-03 23:30:04 +00:00
Author
Copy Link

wow, that was a quick turnaround - thank you very much! :)

wow, that was a quick turnaround - thank you very much! :)
👍 1
witten commented 2023-06-04 05:30:27 +00:00
Owner
Copy Link

Just released as part of borgmatic 1.7.14!

Just released as part of borgmatic 1.7.14!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
blocked
breaking
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
No Label bug
Milestone
No items
No Milestone
Assignees
Clear assignees
witten
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#706
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?