borgmatic list iterates over each config but shows the same (entire) archive list each time #365
Labels
No Label
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
No Milestone
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: borgmatic-collective/borgmatic#365
Loading…
Reference in New Issue
No description provided.
Delete Branch "%!s(<nil>)"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What I'm trying to do and why
I am trying to list my borg repo archives.
Steps to reproduce (if a bug)
I created my borgmatic configs using the provided ansible modules.
Actual behavior (if a bug)
It iterates and runs "borg list --debug --show-rc --lock-wait 5 REPOURL" N times.
with N being the number of configs.
So that it shows N times the same (big) archive list.
Expected behavior (if a bug)
I would expect to either
or
Other notes / implementation ideas
Environment
borgmatic version: 1.5.10
borgmatic installation method: ansible
Borg version: 1.1.14
Python version: 3.5.3
Database version (if applicable): mysql Ver 15.1 Distrib 10.1.45-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2
operating system and version: Linux x86_64 4.9.185, Debian stretch 9.13
borgmatic list iterates over each config but shows the same (entire) archive listto borgmatic list iterates over each config but shows the same (entire) archive list each timeThanks for taking the time to file this one! Would you mind either including your relevant borgmatic configuration files (with sensitive stuff redacted), or just paste in the list of repositories in each config? For bonus points, an example of the command-line input and output you're seeing would help too. I'm just trying to understand the problem fully.
For instance, do you have the same repository listed in multiple different borgmatic configuration files?
Yes it's the same repo in all the config files.
I have one config file for each service, so basically they have different "source_directories", and different prefix.
For instance I've put as attached files 2 config files (out of the 6 I have).
and then it goes again with same output
it happens multiple times and finally ends up with:
I guess I'm doing this wrong?
Sorry for the delay in getting back to you! Having looked at your configuration files, I wouldn't say that you're doing this "wrong" at all. What's happening is that borgmatic runs
borg list
once for each configuration file, oblivious to the fact that you're using the same repository in both.Your "expected behavior" make sense to me, but here are some challenges for each one:
There's an implementation detail issue here, which is that
borgmatic list
is already making use of Borg's--glob-archives
flag to exclude checkpoint archives. And it just so turns out that--glob-archives
is incompatible with the--prefix
flag necessary to filter down Borg's output to the archives related to a particular config file.One solution I could see would be to stop excluding checkpoints when the
borg list --prefix
flag needs to used. But this seems like pretty unexpected behavior even if it were documented.Another solution would be to drop
--glob-archives
entirely and tell anyone who wants their checkpoint archives to conitnue to be omitted to upgrade to Borg 1.1.14 which appears to do that by default (unverified).The challenge here is that a
borg list
command could in theory be different for each configuration file even if the repository is identical. For instance, the requested local/remote Borg file paths or lock wait value could differ across configuration files. And therefore result in differentborg list
flags for each configuration file.One potential solution could be to skip subsequent runs of
borg list
if the exact command flags would be identical.Let me ask you this though: Is there a reason you've got things split across two configuration files like you do now? Just to be able to split archives by function for ease of restore? Seems like a reasonable thing to do, but I just wanted to make sure you realize you could do it all with a single config file.. losing the separate archive prefixes in the process—that would be the cost of doing things that way.
In any case, let me know your thoughts on all of this.
Thanks for getting back to me!
After reading you, I now realize this might be hard to implement.
Anyway the only reason I used several config file was just to separate each "service" to backup with its own borgmatic config file.
Just as a matter of cleanliness.
This way it's pretty clear which service needs which resource.
This helps getting my head clear about what needs to be backuped up (and therefore restored) for each individual service.
Totally understood about the cleanliness and separate archives. Thanks for clarifying. I'll have to think about which of the possible solutions would work best here.
A little late, but this looks like it's already been implemented in #479, released in borgmatic 1.7.11! More documentation on the feature is here: https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/#archive-naming
Awesome! Thanks a lot @witten :)
And thank you for your patience. 😄