Support for multiple prefixes/archive/glob matchers? #599

New Issue

Closed

opened 2022-10-16 12:52:37 +00:00 by ams_tschoening · 4 comments

ams_tschoening commented 2022-10-16 12:52:37 +00:00

Copy Link

What I'm trying to do and why

I have multipel repos containing files for multiple different hosts and have named my archives within the repos on host names in reversed order. So host foobar.example.org became archive name org.example.foobar, though, some of my real host names are much longer. Therefore I considered implementing a shorter naming scheme, e.g. like ex.foo, which would still be unique enough in my cases.

The problem is that BorgMatic seems to only maintain one prefix for operations like prune etc., so I would need to rename all available archives, which is a lot of work. Borg 2 introduced support for regular expressions, which would allow me to use | in theory to support multiple different archvie names. But Borg 2 shouldn't be used in production yet and even if that changes at some point, one needs to transfer existing repos first. So I most likely not switch easily. Additionally, having one large regular expression with | might be difficult to maintain as well.

So from my point of view it would be great if BorgMatic would be able to support multiple prefixes and simply process one after another. In my case I don't even need different settings for prune, it's really only about the names. And I most likely wouldn't care too much as well if prune is actually executed multiple times one after another. For Borg 2, this might make handling multiple RE-matchers easier as well.

Any thoughts? Thanks!

Environment

borgmatic version: 1.6.1
borgmatic installation method: PIP, system wide
Borg version: 1.2.1
Python version: 3.8.10
operating system and version: Ubuntu 20.04

#### What I'm trying to do and why I have multipel repos containing files for multiple different hosts and have named my archives within the repos on host names in reversed order. So host `foobar.example.org` became archive name `org.example.foobar`, though, some of my real host names are much longer. Therefore I considered implementing a shorter naming scheme, e.g. like `ex.foo`, which would still be unique enough in my cases. The problem is that BorgMatic seems to only maintain one prefix for operations like `prune` etc., so I would need to rename all available archives, which is a lot of work. [Borg 2](https://projects.torsion.org/borgmatic-collective/borgmatic/issues/591) introduced support for regular expressions, which would allow me to use `|` in theory to support multiple different archvie names. But Borg 2 shouldn't be used in production yet and even if that changes at some point, one needs to transfer existing repos first. So I most likely not switch easily. Additionally, having one large regular expression with `|` might be difficult to maintain as well. So from my point of view it would be great if BorgMatic would be able to support multiple prefixes and simply process one after another. In my case I don't even need different settings for prune, it's really only about the names. And I most likely wouldn't care too much as well if prune is actually executed multiple times one after another. For Borg 2, this might make handling multiple RE-matchers easier as well. Any thoughts? Thanks! #### Environment **borgmatic version:** 1.6.1 **borgmatic installation method:** PIP, system wide **Borg version:** 1.2.1 **Python version:** 3.8.10 **operating system and version:** Ubuntu 20.04

witten commented 2022-10-16 16:33:35 +00:00

Owner

Copy Link

Generally, there's a convention that one borgmatic configuration file corresponds to one borgmatic invocation (per action). That convention isn't written in stone of course, but I am a little hesitant to add features that would be obviated by the eventual release of Borg 2 and its --match-archives feature. At the rate borgmatic feature development goes, Borg 2 might be used in production well before this or any other ticket makes it into a borgmatic release! #588 specifically is a feature to add a match_archives option (or options) to borgmatic configuration, so you would be able to use it very similiar to prefix today.

Alternatively, if you don't want to wait for Borg 2, you could make multiple different prune-only configuration files for borgmatic, one for each hostname. And then optionally you could use borgmatic includes so as to cut down on duplication.

I do wonder though: Why not prune all hosts at the same time with a single borgmatic/Borg invocation? In other words, why do you need to only prune the archives for a particular host instead of all hosts?

Let me know your thoughts!

Generally, there's a convention that one borgmatic configuration file corresponds to one borgmatic invocation (per action). That convention isn't written in stone of course, but I am a little hesitant to add features that would be obviated by the eventual release of Borg 2 and its `--match-archives` feature. At the rate borgmatic feature development goes, Borg 2 might be used in production well before this or any other ticket makes it into a borgmatic release! #588 specifically is a feature to add a `match_archives` option (or options) to borgmatic configuration, so you would be able to use it very similiar to `prefix` today. Alternatively, if you don't want to wait for Borg 2, you could make multiple different prune-only configuration files for borgmatic, one for each hostname. And then optionally you could use borgmatic includes so as to cut down on duplication. I do wonder though: Why not prune all hosts at the same time with a single borgmatic/Borg invocation? In other words, why do you need to only prune the archives for a particular host instead of all hosts? Let me know your thoughts!

ams_tschoening changed title from Support for multiple prefixes/archive&glob matchers? to Support for multiple prefixes/archive/glob matchers? 2022-10-16 19:03:20 +00:00

ams_tschoening commented 2022-10-16 19:17:57 +00:00

Author

Copy Link

I do wonder though: Why not prune all hosts at the same time with a single borgmatic/Borg invocation? In other words, why do you need to only prune the archives for a particular host instead of all hosts?

Different hosts are differently regularly backed up and need different prune settings because of space consumption and stuff. With the current approach, everything regarding one host is handled by one YAML file for that host, which I need anyway for its backup. If the host is not backed up anymore, either because of some error or for some special reason, it's not even pruned by accident too much. Otherwise I would need at least one additional prune YAML per repo, I need to consider the same for archive checks, as those aren't much different from pruning if pruning is outsourced etc. I currently like the lifecycle of checking+pruning+backing per host on each and every invocation pretty much.

But I get your point, might simply not be worth it. I am living with the "wrong" archive names for some time now already and most likely can longer. :-)

> I do wonder though: Why not prune all hosts at the same time with a single borgmatic/Borg invocation? In other words, why do you need to only prune the archives for a particular host instead of all hosts? Different hosts are differently regularly backed up and need different prune settings because of space consumption and stuff. With the current approach, everything regarding one host is handled by one YAML file for that host, which I need anyway for its backup. If the host is not backed up anymore, either because of some error or for some special reason, it's not even pruned by accident too much. Otherwise I would need at least one additional prune YAML per repo, I need to consider the same for archive checks, as those aren't much different from pruning if pruning is outsourced etc. I currently like the lifecycle of checking+pruning+backing per host on each and every invocation pretty much. But I get your point, might simply not be worth it. I am living with the "wrong" archive names for some time now already and most likely can longer. :-)

ams_tschoening closed this issue 2022-10-16 19:17:58 +00:00

witten commented 2022-10-16 20:25:06 +00:00

Owner

Copy Link

Hmm.. I think I'm not quite understanding your setup. If you already have one YAML file per host (which you need anyway for its backup, as you say), then doesn't borgmatic already support host-specific retention settings? For instance, in a given configuration file, you would have the all the backup and retention settings for that host—including a hard-coded host-specific retention prefix.

Or are you saying that you'd ideally like to collapse all those settings for all hosts into a single configuration file?

Hmm.. I think I'm not quite understanding your setup. If you already have one YAML file per host (which you need anyway for its backup, as you say), then doesn't borgmatic already support host-specific retention settings? For instance, in a given configuration file, you would have the all the backup and retention settings for that host—*including* a hard-coded host-specific retention `prefix`. Or are you saying that you'd ideally like to collapse all those settings for all hosts into a single configuration file?

ams_tschoening commented 2022-10-17 06:16:12 +00:00

Author

Copy Link

[...]including a hard-coded host-specific retention prefix.

Yes, but I need two of those prefixes when I change the naming scheme for archives for one and the same host.

> [...]including a hard-coded host-specific retention prefix. Yes, but I need two of those prefixes when I change the naming scheme for archives for one and the same host.

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

2.0.3

2.0.2

2.0.1

2.0.0

1.9.14

1.9.13

1.9.12

1.9.11

1.9.10

1.9.9

1.9.8

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.14

1.8.13

1.8.12

1.8.11

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.24

1.5.23

1.5.22

1.5.21

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.14

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.7.dev0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.22

1.4.21

1.4.20

1.4.19

1.4.18

1.4.17

1.4.16

1.4.15

1.4.14

1.4.13

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.26

1.3.25

1.3.24

1.3.23

1.3.22

1.3.21

1.3.20

1.3.19

1.3.18

1.3.17

1.3.16

1.3.15

1.3.14

1.3.13

1.3.12

1.3.11

1.3.10

1.3.9

1.3.8

1.3.7

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.18

1.2.17

1.2.16

1.2.15

1.2.14

1.2.13

1.2.12

1.2.11

1.2.10

1.2.9

1.2.8

1.2.7

1.2.6

1.2.5

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.15

1.1.14

1.1.13

1.1.12

1.1.11

1.1.10

1.1.9

1.1.7

1.1.6

1.1.5

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.3

1.0.2

1.0.1

1.0.0

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

0.0.2

Labels

Clear labels

blocked

bug

data loss

design finalized

good first issue

new feature area

question / support

security

waiting for response

No Label

Milestone

No items

No Milestone

Assignees

Clear assignees

diivi import_bot witten

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#599

No description provided.