`bootstrap` hard-codes `--glob-archives`

lingfish commented

2026-05-22 03:10:46 +00:00

Contributor

What I'd like to do and why

Off the back of recent bootstrap work, I've discovered that, I think, it hard-codes --glob-archives:

BORG_EXIT_CODES=*** borg --version --log-json --debug --show-rc
BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_EXIT_CODES=*** borg list --remote-path borg14 --log-json --glob-archives {hostname}-* --last 1 --json ssh://some-repo

When something different is used for archive_name_format, this makes bootstrapping difficult.

--archive of course works, but it's a hard catch-22 guessing what latest might be ;)

Other notes / implementation ideas

No response

### What I'd like to do and why Off the back of recent `bootstrap` work, I've discovered that, I think, it hard-codes `--glob-archives`: ``` BORG_EXIT_CODES=*** borg --version --log-json --debug --show-rc BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_EXIT_CODES=*** borg list --remote-path borg14 --log-json --glob-archives {hostname}-* --last 1 --json ssh://some-repo ``` When something different is used for `archive_name_format`, this makes bootstrapping difficult. `--archive` of course works, but it's a hard catch-22 guessing what `latest` might be ;) ### Other notes / implementation ideas _No response_

witten commented

2026-05-22 22:52:18 +00:00

Owner

Yeah, the --glob-archives default value is {hostname}-* in borgmatic if archive_name_format is not set. And so, because the bootstrap action is starting from a place of zero configuration, there's no archive_name_format and it uses the default value.

This is of course a problem when the user has been using a non-default archive_name_format to create their backups, as you've discovered. And there could even be multiple formats in a single repository.

One idea here is to add an --archive-name-format flag to the bootstrap action, which would require a user to reproduce the full format, so bootstrap can go and grab the corresponding latest archive. That could work, but it's not great—in that it requires the user to remember a potentially complex string of characters (or get it from another source).

Another idea is for bootstrap to try to detect the multiple different "latest" archives in the repository, one for each format used, and present them to the user to select from. Not with an interactive UI, but just erroring out with: "Which of these latest archives do you actually want? Add an --archive flag to select one." However detecting this seems potentially error-prone. Borg does store the archive name "format", but not directly. It stores the command-line used to create each archive. E.g.: Command line: /usr/bin/borg create --patterns-from /run/user/1000/borgmatic/tmp8h6zlbt6 --log-json --debug --show-rc 'test.borg::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}' But maybe that could be used to back-calculate the archive_name_format, at least to a degree necessary to group archives and find all the latests.

Let me know your thoughts. (I'm open to other ideas too!)

Yeah, the `--glob-archives` default value is `{hostname}-*` in borgmatic if `archive_name_format` is not set. And so, because the `bootstrap` action is starting from a place of zero configuration, there's no `archive_name_format` and it uses the default value. This is of course a problem when the user has been using a non-default `archive_name_format` to create their backups, as you've discovered. And there could even be multiple formats in a single repository. One idea here is to add an `--archive-name-format` flag to the `bootstrap` action, which would require a user to reproduce the full format, so `bootstrap` can go and grab the corresponding `latest` archive. That could work, but it's not great—in that it requires the user to remember a potentially complex string of characters (or get it from another source). Another idea is for bootstrap to try to detect the multiple different "latest" archives in the repository, one for each format used, and present them to the user to select from. Not with an interactive UI, but just erroring out with: "Which of these latest archives do you actually want? Add an --archive flag to select one." However detecting this seems potentially error-prone. Borg does store the archive name "format", but not directly. It stores the command-line used to create each archive. E.g.: `Command line: /usr/bin/borg create --patterns-from /run/user/1000/borgmatic/tmp8h6zlbt6 --log-json --debug --show-rc 'test.borg::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}'` But maybe that could be used to back-calculate the `archive_name_format`, at least to a degree necessary to group archives and find all the latests. Let me know your thoughts. (I'm open to other ideas too!)

lingfish commented

2026-05-23 00:35:26 +00:00

Author

Contributor

@witten wrote in #1310 (comment):

Yeah, the --glob-archives default value is {hostname}-* in borgmatic if archive_name_format is not set.

Indeed, and as I've discovered, that {hostname} is resolved from the machine borg{matic} is being run from; see next comment.

@witten wrote in #1310 (comment):

This is of course a problem when the user has been using a non-default archive_name_format to create their backups, as you've discovered.

Or, in my case, attempting to restore/mount/etc a backup on a different machine, with a different {hostname}! So even when using a default for archive_name_format, it breaks.

@witten wrote in #1310 (comment):

That could work, but it's not great—in that it requires the user to remember a potentially complex string of characters

Yeah, not good.

@witten wrote in #1310 (comment):

Another idea is for bootstrap to try to detect the multiple different "latest" archives in the repository

Agree, this starts to sound complex/messy too.

My thoughts right now (and I'll mull over it more), is that it should simply resolve --glob-archives to '*' if in latest mode, and be done with it. If a user has mixed naming or other complexities, they'll have to manage that themselves (to avoid over-reaching in borgmatic).

They either get the freedom to specify a specific archive (in full), or they just get whatever '*' and --last 1 resolves to.

@witten wrote in https://projects.torsion.org/borgmatic-collective/borgmatic/issues/1310#issuecomment-14083: > Yeah, the `--glob-archives` default value is `{hostname}-*` in borgmatic if `archive_name_format` is not set. Indeed, and as I've discovered, that `{hostname}` is resolved from the machine borg{matic} is _being run from_; see next comment. @witten wrote in https://projects.torsion.org/borgmatic-collective/borgmatic/issues/1310#issuecomment-14083: > This is of course a problem when the user has been using a non-default `archive_name_format` to create their backups, as you've discovered. Or, in my case, attempting to restore/mount/etc a backup on a different machine, with a different `{hostname}`! So even when using a default for `archive_name_format`, it breaks. @witten wrote in https://projects.torsion.org/borgmatic-collective/borgmatic/issues/1310#issuecomment-14083: > That could work, but it's not great—in that it requires the user to remember a potentially complex string of characters Yeah, not good. @witten wrote in https://projects.torsion.org/borgmatic-collective/borgmatic/issues/1310#issuecomment-14083: > Another idea is for bootstrap to try to detect the multiple different "latest" archives in the repository Agree, this starts to sound complex/messy too. My thoughts right now (and I'll mull over it more), is that it should simply resolve `--glob-archives` to `'*'` if in `latest` mode, and be done with it. If a user has mixed naming or other complexities, they'll have to manage that themselves (to avoid over-reaching in `borgmatic`). They either get the freedom to specify a specific archive (in full), or they just get whatever `'*'` and `--last 1` resolves to.

witten referenced this issue from a commit

2026-05-23 02:33:34 +00:00

Proof of concept for a helpful error when bootstrapping with multiple "latest" archives (#1310).

witten referenced this issue

2026-05-23 02:34:12 +00:00

Add a helpful error when bootstrapping with multiple "latest" archives (#1310). #1313

witten commented

2026-05-23 02:37:47 +00:00

Owner

@lingfish wrote in #1310 (comment):

Another idea is for bootstrap to try to detect the multiple different "latest" archives in the repository

Agree, this starts to sound complex/messy too.

So I tried this for fun. It actually wasn't too terrible given that I could make use of borg info's JSON output. Proof of concept here: #1313

Example actual output:

$ borgmatic -c test.yaml -v 2 config bootstrap --repository test.borg
BORG_EXIT_CODES=*** borg --version --log-json --debug --show-rc
BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_EXIT_CODES=*** borg info --remote-path borg --log-json --glob-archives {hostname}-* --json test.borg
Enter passphrase for key /home/witten/Downloads/tmp/test.borg: 
The repository appears to have multiple "latest" archives, each with a different archive name format: flux-2026-05-22T14:33:56.698164, flux-documents-2026-05-22T19:30:54. Please select one with --archive.

@lingfish wrote in #1310 (comment):

My thoughts right now (and I'll mull over it more), is that it should simply resolve --glob-archives to '*' if in latest mode, and be done with it. If a user has mixed naming or other complexities, they'll have to manage that themselves (to avoid over-reaching in borgmatic).

That could work. But it would hide the archive_name_format complexities in a way that means the user could select the wrong archive on accident.

@lingfish wrote in https://projects.torsion.org/borgmatic-collective/borgmatic/issues/1310#issuecomment-14084: > > Another idea is for bootstrap to try to detect the multiple different "latest" archives in the repository > > Agree, this starts to sound complex/messy too. So I tried this for fun. It actually wasn't *too* terrible given that I could make use of `borg info`'s JSON output. Proof of concept here: https://projects.torsion.org/borgmatic-collective/borgmatic/pulls/1313 Example actual output: ``` $ borgmatic -c test.yaml -v 2 config bootstrap --repository test.borg BORG_EXIT_CODES=*** borg --version --log-json --debug --show-rc BORG_RELOCATED_REPO_ACCESS_IS_OK=*** BORG_EXIT_CODES=*** borg info --remote-path borg --log-json --glob-archives {hostname}-* --json test.borg Enter passphrase for key /home/witten/Downloads/tmp/test.borg: The repository appears to have multiple "latest" archives, each with a different archive name format: flux-2026-05-22T14:33:56.698164, flux-documents-2026-05-22T19:30:54. Please select one with --archive. ``` @lingfish wrote in https://projects.torsion.org/borgmatic-collective/borgmatic/issues/1310#issuecomment-14084: > My thoughts right now (and I'll mull over it more), is that it should simply resolve `--glob-archives` to `'*'` if in `latest` mode, and be done with it. If a user has mixed naming or other complexities, they'll have to manage that themselves (to avoid over-reaching in `borgmatic`). That could work. But it would hide the `archive_name_format` complexities in a way that means the user could select the wrong archive on accident.

lingfish commented

2026-05-26 06:50:48 +00:00

Author

Contributor

@witten wrote in #1310 (comment):

Proof of concept here

Ok, that's kinda cute, but I wonder if there are many different "latest"'s, how the output would look. For me personally, I wouldn't have thought that one repo would have many, as a typical user use-case... I tend to split mine out (finer-grain retention control etc)... but I guess borgmatic should cater for any and all.

And what about the case (my case actually) of having one repo, one archive prefix, that merely doesn't resolve to the original hostname (as I'm restoring on another host)? Without staring at the code for a bit, would it still claim there's multiples, yet listing only one? Perhaps with the PoC, if it sees only one, that differs from currently-running hostname, it could auto-select that at least?

@witten wrote in #1310 (comment):

the user could select the wrong archive on accident

True, but couldn't that technically happen as the prod code is now? Say you have (as yours above):

flux-2026-05-22T14:33:56.698164
flux-documents-2026-05-22T19:30:54 (note this is technically "latest")

... and you're restoring from a hostname of flux, well in latest mode you're kinda technically getting the latest in the flux train, but not the latest in the repo overall, and that'd be silent to the user too.

@witten wrote in https://projects.torsion.org/borgmatic-collective/borgmatic/issues/1310#issuecomment-14088: > Proof of concept here Ok, that's kinda cute, but I wonder if there are many different "latest"'s, how the output would look. For me personally, I wouldn't have thought that one repo would have many, as a typical user use-case... I tend to split mine out (finer-grain retention control etc)... but I guess `borgmatic` should cater for any and all. And what about the case (my case actually) of having one repo, one archive prefix, that merely doesn't resolve to the original hostname (as I'm restoring on another host)? Without staring at the code for a bit, would it still claim there's multiples, yet listing only one? Perhaps with the PoC, if it sees only one, that differs from currently-running `hostname`, it could auto-select that at least? @witten wrote in https://projects.torsion.org/borgmatic-collective/borgmatic/issues/1310#issuecomment-14088: > the user could select the wrong archive on accident True, but couldn't that technically happen as the prod code is now? Say you have (as yours above): - `flux-2026-05-22T14:33:56.698164` - `flux-documents-2026-05-22T19:30:54` (note this is technically "latest") ... and you're restoring from a `hostname` of `flux`, well in `latest` mode you're kinda technically getting the latest in the `flux` train, but not the latest in the repo overall, and that'd be silent to the user too.

witten commented

2026-05-27 20:17:02 +00:00

Owner

@lingfish wrote in #1310 (comment):

@witten wrote in #1310 (comment):

Proof of concept here

Ok, that's kinda cute, but I wonder if there are many different "latest"'s, how the output would look. For me personally, I wouldn't have thought that one repo would have many, as a typical user use-case... I tend to split mine out (finer-grain retention control etc)... but I guess borgmatic should cater for any and all.

Yeah, I know at least some users use a different archive_name_format for each of several different application backups written to the same repository.

And what about the case (my case actually) of having one repo, one archive prefix, that merely doesn't resolve to the original hostname (as I'm restoring on another host)? Without staring at the code for a bit, would it still claim there's multiples, yet listing only one? Perhaps with the PoC, if it sees only one, that differs from currently-running hostname, it could auto-select that at least?

Actually for that case, yeah, the PoC would see only one. And right now it would silently just try to use the same wrong default archive_name_format to fetch the latest archive. But it would be pretty easy to change the PoC it to ignore the default archive_name_format and always fetch the latest archive if there's only one. And I think that would solve your particular use case (while also erroring when there are multiple latests and it can't pick just one).

@witten wrote in #1310 (comment):

the user could select the wrong archive on accident

True, but couldn't that technically happen as the prod code is now? Say you have (as yours above):
* `flux-2026-05-22T14:33:56.698164`

* `flux-documents-2026-05-22T19:30:54` (note this is technically "latest")
... and you're restoring from a hostname of flux, well in latest mode you're kinda technically getting the latest in the flux train, but not the latest in the repo overall, and that'd be silent to the user too.

Yeah, I think we both agree that the released code behavior is problematic in this regard. There's no guarantee that any archive starts with the hostname. That's just the default / a common archive_name_format prefix. I've seen users use formats like myapp-{now...}.

@lingfish wrote in https://projects.torsion.org/borgmatic-collective/borgmatic/issues/1310#issuecomment-14112: > @witten wrote in #1310 (comment): > > > Proof of concept here > > Ok, that's kinda cute, but I wonder if there are many different "latest"'s, how the output would look. For me personally, I wouldn't have thought that one repo would have many, as a typical user use-case... I tend to split mine out (finer-grain retention control etc)... but I guess `borgmatic` should cater for any and all. Yeah, I know at least some users use a different `archive_name_format` for each of several different [application backups](https://torsion.org/borgmatic/how-to/make-per-application-backups/) written to the same repository. > And what about the case (my case actually) of having one repo, one archive prefix, that merely doesn't resolve to the original hostname (as I'm restoring on another host)? Without staring at the code for a bit, would it still claim there's multiples, yet listing only one? Perhaps with the PoC, if it sees only one, that differs from currently-running `hostname`, it could auto-select that at least? Actually for that case, yeah, the PoC would see only one. And right now it would silently just try to use the same wrong default `archive_name_format` to fetch the latest archive. But it would be pretty easy to change the PoC it to ignore the default `archive_name_format` and always fetch the latest archive if there's only one. And I think that would solve your particular use case (while also erroring when there are multiple latests and it can't pick just one). > > @witten wrote in #1310 (comment): > > > the user could select the wrong archive on accident > > True, but couldn't that technically happen as the prod code is now? Say you have (as yours above): > > * `flux-2026-05-22T14:33:56.698164` > > * `flux-documents-2026-05-22T19:30:54` (note this is technically "latest") > > > ... and you're restoring from a `hostname` of `flux`, well in `latest` mode you're kinda technically getting the latest in the `flux` train, but not the latest in the repo overall, and that'd be silent to the user too. Yeah, I think we both agree that the released code behavior is problematic in this regard. There's no guarantee that any archive starts with the hostname. That's just the default / a common `archive_name_format` prefix. I've seen users use formats like `myapp-{now...}`.

quietquark commented

2026-05-29 11:09:20 +00:00

Just wanted to add an upstream borgbackup request that I have open as well:

https://github.com/borgbackup/borg/issues/9651

(borg2 already supports this).

This simplifies some use cases with borg (as documented in the issue).

I think it is equally useful for borgmatic to support custom hostnames when one "controller" impersonates othe machines.

[Please advise if this should be split out into a separate issue]

Just wanted to add an upstream borgbackup request that I have open as well: https://github.com/borgbackup/borg/issues/9651 (borg2 already supports this). This simplifies some use cases with borg (as documented in the issue). I think it is equally useful for borgmatic to support custom hostnames when one "controller" impersonates othe machines. [Please advise if this should be split out into a separate issue]

witten commented

2026-05-29 18:14:37 +00:00

Owner

Seems reasonable. Maybe a hostname option (and corresponding flag) that gets passed to Borg as --hostname. I think that would allow you to address your use case—but only if you're aware of the problem. I'm thinking that something like the PoC or the --glob-archives * during bootstrap would still be necessary to avoid some of the pitfalls in the general case discussed above. So I think the hostname thing should probably be a separate ticket.

Seems reasonable. Maybe a `hostname` option (and corresponding flag) that gets passed to Borg as `--hostname`. I think that would allow you to address your use case—but only if you're aware of the problem. I'm thinking that something like the PoC or the `--glob-archives *` during bootstrap would still be necessary to avoid some of the pitfalls in the general case discussed above. So I think the hostname thing should probably be a separate ticket.

quietquark referenced this issue

2026-05-30 06:08:54 +00:00

Support Overriding System Hostname #1317

witten commented

2026-06-16 02:19:16 +00:00

Owner

#1317 is implemented now as an archive_hostname option (and corresponding CLI flag). It requires Borg 1.4.5+ though.

#1317 is implemented now as an `archive_hostname` option (and corresponding CLI flag). It requires Borg 1.4.5+ though.

❤️ 1

lingfish commented

2026-06-18 07:56:03 +00:00

Author

Contributor

Nice one Dan, thanks!

👍 1

lingfish closed this issue

2026-06-18 07:56:03 +00:00

witten commented

2026-06-18 17:58:32 +00:00

Owner

Reopening this, as I think I still have to do something about the incorrect "latest" archive. Maybe not for your use case, but in general.

witten reopened this issue

2026-06-18 17:58:32 +00:00

witten referenced this issue from a commit

2026-08-06 22:19:38 +00:00

For the "bootstrap" action, stop using the default achive name format. Also add tests (#1310).

witten referenced this issue from a commit

2026-08-07 02:52:30 +00:00

Add a helpful error when bootstrapping with multiple "latest" archives (#1310).

witten commented

2026-08-07 02:55:08 +00:00

Owner

Okay, I cleaned up the proof of concept, made bootstrap not use the default archive name format, and merged all that to main. It'll be part of the next release.

Okay, I cleaned up the proof of concept, made `bootstrap` _not_ use the default archive name format, and merged all that to main. It'll be part of the next release.

witten closed this issue

2026-08-07 02:55:08 +00:00

witten commented

2026-08-14 20:18:37 +00:00

Owner

Released in borgmatic 2.1.7!

Rows
Columns

bootstrap hard-codes --glob-archives #1310

What I'd like to do and why

Other notes / implementation ideas

`bootstrap` hard-codes `--glob-archives` #1310