Dynamic creation of database backup configuration

apollo13 commented

2025-04-09 13:34:55 +00:00

Contributor

What I'd like to do and why

I'd like to have an ability to dynamically construct the database backup configurations. In my case I have an external source where they are stored but so far I do not see any possibility to incorpate it.

What I managed so far is dynamic creation of patterns_from via a custom command set to before:action, when: [create]. It would be lovely if a similar approach would work for database backups, maybe a databases_backups_from, or a database_backup_config_command option?

If you can provide some guidance I on how you would like the options to look like I might be able to provide a PR.

Other notes / implementation ideas

No response

### What I'd like to do and why I'd like to have an ability to dynamically construct the database backup configurations. In my case I have an external source where they are stored but so far I do not see any possibility to incorpate it. What I managed so far is dynamic creation of `patterns_from` via a custom command set to `before:action, when: [create]`. It would be lovely if a similar approach would work for database backups, maybe a `databases_backups_from`, or a `database_backup_config_command` option? If you can provide some guidance I on how you would like the options to look like I might be able to provide a PR. ### Other notes / implementation ideas _No response_

witten commented

2025-04-09 16:35:59 +00:00

Owner

Thanks for filing this.. It's an interesting idea! Can you say a little more about your external source of database configurations? For instance:

Where do they come from? What is the external source?
Why are they dynamic / only known at runtime?
Or are they fixed, but for example you just don't want to have to duplicate them in borgmatic's configuration file?
Do database credentials also come from this external source?

The answers to these will hopefully help guide the solution design.

Thanks for filing this.. It's an interesting idea! Can you say a little more about your external source of database configurations? For instance: * Where do they come from? What is the external source? * Why are they dynamic / only known at runtime? * Or are they fixed, but for example you just don't want to have to duplicate them in borgmatic's configuration file? * Do database credentials also come from this external source? The answers to these will hopefully help guide the solution design.

witten added the

new feature area

label

2025-04-09 16:36:25 +00:00

apollo13 commented

2025-04-09 16:49:02 +00:00

Author

Contributor

Hi @witten, thank you for your thoughtful response.

Where do they come from? What is the external source?

Some JSON files but could as well be HashiCorp Vault for instance.

Why are they dynamic / only known at runtime?

In my case they are not actually dynamic and are kinda known beforehand. As a "workaround" I will be writing them into an extra file and include that from the main configuration file. But with Vault one could use dynamic credentials that only have a relatively short lifetime (a day or so) which would need renewal (but again, with a renewal agent I can write those to a file).

To expand a little bit on what I am doing: I am writing my own "PaaS" for home servers / single VMs with some fancy docker-compose trickery. As part of that I can define labels on containers and volumes so I can determine what to backup. In that sense the information is not highly dynamic. So instead of determining the databases every time borgmatic runs I am now going to query that information when I deploy a compose file and put it into a file that borgmatic can include. It is a bit of an inversion of control, but should be fine either way.

I am leaving this open for now, maybe someone finds it useful -- but feel free to close.

Hi @witten, thank you for your thoughtful response. > Where do they come from? What is the external source? Some JSON files but could as well be HashiCorp Vault for instance. > Why are they dynamic / only known at runtime? In my case they are not actually dynamic and are kinda known beforehand. As a "workaround" I will be writing them into an extra file and include that from the main configuration file. But with Vault one could use dynamic credentials that only have a relatively short lifetime (a day or so) which would need renewal (but again, with a renewal agent I can write those to a file). To expand a little bit on what I am doing: I am writing my own "PaaS" for home servers / single VMs with some fancy docker-compose trickery. As part of that I can define labels on containers and volumes so I can determine what to backup. In that sense the information is not highly dynamic. So instead of determining the databases every time borgmatic runs I am now going to query that information when I deploy a compose file and put it into a file that borgmatic can include. It is a bit of an inversion of control, but should be fine either way. I am leaving this open for now, maybe someone finds it useful -- but feel free to close.

witten commented

2025-04-09 19:08:44 +00:00

Owner

Interesting. What you're describing sounds a little like #685, but for discovering container databases to backup instead of (or in addition to) container volumes to backup.

The HashiCorp Vault credentials makes me think of borgmatic's existing credential hooks. Like one could pretty easily (I'm assuming) add a new credential hook to integrate with HashiCorp's Vault. I can envision how that would work with static / in-borgmatic-config database configuration... Maybe something like the existing KeePassXC credential hook's config. But I'm not sure how that would interface with potentially dynamic config.

Anyway, to keep this a little simpler and more tractable at first, I'll disregard the Vault piece for purposes of this ticket. (Feel free to file it separately.) That leaves "dynamically" determining the databases to backup.

A couple more questions:

What's the benefit in your mind to an approach like databases_backups_from or database_backup_config_command vs. what it sounds like you're doing now which is generating a borgmatic include file with databases to backup?
What do you think of borgmatic itself looking for container labels to determine which database containers to backup?
Any thoughts/preferences for how those labels would express the various database configuration options to borgmatic?

Interesting. What you're describing sounds a little like #685, but for discovering container databases to backup instead of (or in addition to) container volumes to backup. The HashiCorp Vault credentials makes me think of [borgmatic's existing credential hooks](https://torsion.org/borgmatic/docs/how-to/provide-your-passwords/). Like one could pretty easily (I'm assuming) add a new credential hook to integrate with HashiCorp's Vault. I can envision how that would work with static / in-borgmatic-config database configuration... Maybe something like the existing [KeePassXC credential hook's config](https://torsion.org/borgmatic/docs/how-to/provide-your-passwords/#keepassxc-passwords). But I'm not sure how that would interface with potentially dynamic config. Anyway, to keep this a little simpler and more tractable at first, I'll disregard the Vault piece for purposes of this ticket. (Feel free to file it separately.) That leaves "dynamically" determining the databases to backup. A couple more questions: * What's the benefit in your mind to an approach like `databases_backups_from` or `database_backup_config_command` vs. what it sounds like you're doing now which is generating a borgmatic include file with databases to backup? * What do you think of borgmatic itself looking for container labels to determine which database containers to backup? * Any thoughts/preferences for how those labels would express the various database configuration options to borgmatic?

apollo13 commented

2025-04-09 19:33:06 +00:00

Author

Contributor

Yes the referenced ticket kinda sounds similar.

What's the benefit in your mind to an approach like databases_backups_from or database_backup_config_command vs. what it sounds like you're doing now which is generating a borgmatic include file with databases to backup?

Probably no real benefit, especially since my current approach to generate a pattern file in a before command is also kind of a hack and probably only works by "accident" since borgmatic does not read the file early on. What I do like about that "hack" though is that it gives me the possibility to output from the command which data I am going to backup etc, which is nice to see: https://gitlab.com/rocketduck/paas/-/blob/main/compose/paas-backup/scripts/before_create.py -- but I can also simply write that to a file beforehand and than just "cat" it from a command.

Another thing is that the current include mechanism feels kinda clunky. I am used to other tools where you can say something along the lines of include: /some/dir/*.conf and don't have to bother whether any files exist etc… That said I understand that you are working within the boundaries of yaml itself here.

What do you think of borgmatic itself looking for container labels to determine which database containers to backup?

Not much to be honest since labels are not a good place to transport sensitive data (assuming you want full label based control you'd somehow have to transport the password via labels as well). Sure you could set a label which tells you which env variable or file contains the password and then gather that via exec calls into the container, but that gets ugly as hell. My current approach is something along the lines of this (this is still mainly a PoC and has some very harsh edges, but it already works impressively well in first tests):

gitlab.com/rocketduck/paas/-@250be9df5a/compose/paas-postgres/compose.yaml

As you can see I am using compose extensions x-* to read out the data, and am also able to interpolate variables etc in there. The "orchestrator" takes this metadata and saves it away in it's database. This data can then be used to discover connection details for postgres etc…

Any thoughts/preferences for how those labels would express the various database configuration options to borgmatic?

You can always json-encode the data and put it in as label value. But that is ugly as hell :D

Hope this all makes sense…

Yes the referenced ticket kinda sounds similar. > What's the benefit in your mind to an approach like `databases_backups_from` or `database_backup_config_command` vs. what it sounds like you're doing now which is generating a borgmatic include file with databases to backup? Probably no real benefit, especially since my current approach to generate a pattern file in a before command is also kind of a hack and probably only works by "accident" since borgmatic does not read the file early on. What I do like about that "hack" though is that it gives me the possibility to output from the command which data I am going to backup etc, which is nice to see: https://gitlab.com/rocketduck/paas/-/blob/main/compose/paas-backup/scripts/before_create.py -- but I can also simply write that to a file beforehand and than just "cat" it from a command. Another thing is that the current include mechanism feels kinda clunky. I am used to other tools where you can say something along the lines of `include: /some/dir/*.conf` and don't have to bother whether any files exist etc… That said I understand that you are working within the boundaries of yaml itself here. > What do you think of borgmatic itself looking for container labels to determine which database containers to backup? Not much to be honest since labels are not a good place to transport sensitive data (assuming you want full label based control you'd somehow have to transport the password via labels as well). Sure you could set a label which tells you which env variable or file contains the password and then gather that via exec calls into the container, but that gets ugly as hell. My current approach is something along the lines of this (this is still mainly a PoC and has some very harsh edges, but it already works impressively well in first tests): https://gitlab.com/rocketduck/paas/-/blob/250be9df5a53d06c11d7adc44319864dea62212b/compose/paas-postgres/compose.yaml As you can see I am using compose extensions `x-*` to read out the data, and am also able to interpolate variables etc in there. The "orchestrator" takes this metadata and saves it away in it's database. This data can then be used to discover connection details for postgres etc… > Any thoughts/preferences for how those labels would express the various database configuration options to borgmatic? You can always json-encode the data and put it in as label value. But that is ugly as hell :D Hope this all makes sense…

witten commented

2025-04-10 06:10:02 +00:00

Owner

Probably no real benefit, especially since my current approach to generate a pattern file in a before command is also kind of a hack and probably only works by "accident" since borgmatic does not read the file early on. What I do like about that "hack" though is that it gives me the possibility to output from the command which data I am going to backup etc, which is nice to see: https://gitlab.com/rocketduck/paas/-/blob/main/compose/paas-backup/scripts/before_create.py -- but I can also simply write that to a file beforehand and than just "cat" it from a command.

This does look a lot like the proposed design for #685, although as an external command hook script rather than built into borgmatic. But the general approaches in both have a lot in common.

As for outputting the patterns, modern versions of borgmatic do that by default at --verbosity 2. Example:

testrepo: Writing patterns to /tmp/borgmatic-42n4n2k2/borgmatic/tmpkapwt1cg:
R /root
! fm:/root/tmp/*.borg
! fm:/root/tmp/tmp
R /root/tmp/tmp/file.txt
R /tmp/borgmatic-42n4n2k2/./borgmatic/bootstrap

(Some of those patterns are from a patterns_from file, some are from source_directories, some are from exclude_patterns, and some are from hooks.)

Another thing is that the current include mechanism feels kinda clunky. I am used to other tools where you can say something along the lines of include: /some/dir/*.conf and don't have to bother whether any files exist etc… That said I understand that you are working within the boundaries of yaml itself here.

Not really, actually.. The include mechanism is totally custom even if the syntax is somewhat standard YAML. So you're welcome to file tickets for improvements to it.

Not much to be honest since labels are not a good place to transport sensitive data (assuming you want full label based control you'd somehow have to transport the password via labels as well). Sure you could set a label which tells you which env variable or file contains the password and then gather that via exec calls into the container, but that gets ugly as hell. My current approach is something along the lines of this (this is still mainly a PoC and has some very harsh edges, but it already works impressively well in first tests):

Well that gets back to the whole credential thing, so I guess we can't really leave it out of this discussion. But yeah, I wouldn't expect literal credentials within container labels! But what about something like this on a container:

labels:
    - "org.torsion.borgmatic.postgresql_databases.name=mydb"
    - "org.torsion.borgmatic.postgresql_databases.username=user"
    - "org.torsion.borgmatic.postgresql_databases.password={credential vault pg/creds/blah}"

So the credential never touches the label or even an env var; this just indicates to borgmatic that it needs to load the credential from a specific location. I'm not wedded to this of course, but it's similar to both what's proposed in #685 and other non-borgmatic container-centric apps (Traefik, etc.).

As you can see I am using compose extensions x-* to read out the data, and am also able to interpolate variables etc in there. The "orchestrator" takes this metadata and saves it away in it's database. This data can then be used to discover connection details for postgres etc…

Ah, clever. So if I'm reading it correctly, you're currently dumping databases manually rather than using the borgmatic hook to do it. But you'd like some way to feed database config into borgmatic's database hook(s) so you don't have to dump databases manually...?

You can always json-encode the data and put it in as label value. But that is ugly as hell :D

Agreed. :D

Hope this all makes sense…

It does. I'm just trying to figure out how borgmatic can fit in without creating even more work for you!

> Probably no real benefit, especially since my current approach to generate a pattern file in a before command is also kind of a hack and probably only works by "accident" since borgmatic does not read the file early on. What I do like about that "hack" though is that it gives me the possibility to output from the command which data I am going to backup etc, which is nice to see: https://gitlab.com/rocketduck/paas/-/blob/main/compose/paas-backup/scripts/before_create.py -- but I can also simply write that to a file beforehand and than just "cat" it from a command. This does look a lot like the proposed design for #685, although as an external command hook script rather than built into borgmatic. But the general approaches in both have a lot in common. As for outputting the patterns, modern versions of borgmatic do that by default at `--verbosity 2`. Example: ``` testrepo: Writing patterns to /tmp/borgmatic-42n4n2k2/borgmatic/tmpkapwt1cg: R /root ! fm:/root/tmp/*.borg ! fm:/root/tmp/tmp R /root/tmp/tmp/file.txt R /tmp/borgmatic-42n4n2k2/./borgmatic/bootstrap ``` (Some of those patterns are from a `patterns_from` file, some are from `source_directories`, some are from `exclude_patterns`, and some are from hooks.) > Another thing is that the current include mechanism feels kinda clunky. I am used to other tools where you can say something along the lines of include: /some/dir/*.conf and don't have to bother whether any files exist etc… That said I understand that you are working within the boundaries of yaml itself here. Not really, actually.. The include mechanism is totally custom even if the syntax is somewhat standard YAML. So you're welcome to file tickets for improvements to it. > Not much to be honest since labels are not a good place to transport sensitive data (assuming you want full label based control you'd somehow have to transport the password via labels as well). Sure you could set a label which tells you which env variable or file contains the password and then gather that via exec calls into the container, but that gets ugly as hell. My current approach is something along the lines of this (this is still mainly a PoC and has some very harsh edges, but it already works impressively well in first tests): Well that gets back to the whole credential thing, so I guess we can't really leave it out of this discussion. But yeah, I wouldn't expect literal credentials within container labels! But what about something like this on a container: ```yaml labels: - "org.torsion.borgmatic.postgresql_databases.name=mydb" - "org.torsion.borgmatic.postgresql_databases.username=user" - "org.torsion.borgmatic.postgresql_databases.password={credential vault pg/creds/blah}" ``` So the credential never touches the label or even an env var; this just indicates to borgmatic that it needs to load the credential from a specific location. I'm not wedded to this of course, but it's similar to both what's proposed in #685 and other non-borgmatic container-centric apps (Traefik, etc.). > As you can see I am using compose extensions x-* to read out the data, and am also able to interpolate variables etc in there. The "orchestrator" takes this metadata and saves it away in it's database. This data can then be used to discover connection details for postgres etc… Ah, clever. So if I'm reading it correctly, you're currently dumping databases manually rather than using the borgmatic hook to do it. But you'd like some way to feed database config into borgmatic's database hook(s) so you don't have to dump databases manually...? > You can always json-encode the data and put it in as label value. But that is ugly as hell :D Agreed. :D > Hope this all makes sense… It does. I'm just trying to figure out how borgmatic can fit in without creating even more work for you!

apollo13 commented

2025-04-10 06:52:40 +00:00

Author

Contributor

As for outputting the patterns, modern versions of borgmatic do that by default at --verbosity 2. Example:

Agreed, but that verbosity level is relatively noisy and I am thinking of something more readable like "Adding volume 'x' from project 'y' to backup list" -- at least that is what my code currently does.

The include mechanism is totally custom even if the syntax is somewhat standard YAML.

Interesting, didn't think to much about it, but I interpreted <<: as the standard merge mechanism form YAML 1.1 and !include as a custom tag.

So you're welcome to file tickets for improvements to it.

I'd love include: /path/with_some_glob/* on the top level and let that get (deep?) merged with the existing configuration. Not sure how that interacts with the schema validation though, but maybe merge first and then validate?

- "org.torsion.borgmatic.postgresql_databases.password={credential vault pg/creds/blah}"

Ah yes that would indeed work. Though at this point you probably want to make credential provider registration plugable 🙊

Ah, clever. So if I'm reading it correctly, you're currently dumping databases manually rather than using the borgmatic hook to do it.

Yes, but I reworked a bit of the logic yesterday and should be able to dynamically generate the relevant pattern files and config for borgmatic, will link the result.

But you'd like some way to feed database config into borgmatic's database hook(s) so you don't have to dump databases manually...?

Yeah, to be honest I haven't tried the database support in borgmatic yet myself, but thought it would be nice to have the dump just in the backup and not having to go and roundtrip via the filesystem.

It does. I'm just trying to figure out how borgmatic can fit in without creating even more work for you!

That is really kind of you, but I understand am fine with doing the work and don't want to create work for you/us just to make my life easier. I think anything added to borgmatic should follow a higher goal (with regards to how useful it would be to how many people etc…). Currently I think a common middleground would be to make a hook API public so someone can implement something like postgresql_databases on their own by providing the necessary hook scripts to borgmatic. This way people could experiment and if a hook shows that it is really useful (ie used by many) it could get moved into borgmatic properly.

> As for outputting the patterns, modern versions of borgmatic do that by default at --verbosity 2. Example: Agreed, but that verbosity level is relatively noisy and I am thinking of something more readable like "Adding volume 'x' from project 'y' to backup list" -- at least that is what my code currently does. > The include mechanism is totally custom even if the syntax is somewhat standard YAML. Interesting, didn't think to much about it, but I interpreted `<<:` as the standard merge mechanism form YAML 1.1 and `!include` as a custom tag. > So you're welcome to file tickets for improvements to it. I'd love `include: /path/with_some_glob/*` on the top level and let that get (deep?) merged with the existing configuration. Not sure how that interacts with the schema validation though, but maybe merge first and then validate? > - "org.torsion.borgmatic.postgresql_databases.password={credential vault pg/creds/blah}" Ah yes that would indeed work. Though at this point you probably want to make credential provider registration plugable 🙊 > Ah, clever. So if I'm reading it correctly, you're currently dumping databases manually rather than using the borgmatic hook to do it. Yes, but I reworked a bit of the logic yesterday and should be able to dynamically generate the relevant pattern files and config for borgmatic, will link the result. > But you'd like some way to feed database config into borgmatic's database hook(s) so you don't have to dump databases manually...? Yeah, to be honest I haven't tried the database support in borgmatic yet myself, but thought it would be nice to have the dump just in the backup and not having to go and roundtrip via the filesystem. > It does. I'm just trying to figure out how borgmatic can fit in without creating even more work for you! That is really kind of you, but I understand am fine with doing the work and don't want to create work for you/us just to make my life easier. I think anything added to borgmatic should follow a higher goal (with regards to how useful it would be to how many people etc…). Currently I think a common middleground would be to make a hook API public so someone can implement something like `postgresql_databases` on their own by providing the necessary hook scripts to borgmatic. This way people could experiment and if a hook shows that it is really useful (ie used by many) it could get moved into borgmatic properly.

apollo13 commented

2025-04-10 08:07:45 +00:00

Author

Contributor

Implemented it via the include mechanism now: gitlab.com/rocketduck/paas/-@28450939d5

Kinda happy with the outcome:

$ borgmatic create -v1
ssh://user@host/./paas_test.borg: Running before action command hook
Adding the following volumes to the backup:
	paas-example: test2
Adding postgres server postgres.service.local to the backup.
ssh://user@host/./paas_test.borg: Running before action command hook
ssh://user@host/./paas_test.borg: Creating archive
ssh://user@host/./paas_test.borg: Dumping PostgreSQL databases
Creating archive at "ssh://user@host/./paas_test.borg::backup-2025-04-10T10:07:00"

summary:
/etc/borgmatic.d/config.yaml: Successfully ran configuration file

Implemented it via the include mechanism now: https://gitlab.com/rocketduck/paas/-/commit/28450939d5737d36eb8f8e91b2dafc9fcf453428 Kinda happy with the outcome: ``` $ borgmatic create -v1 ssh://user@host/./paas_test.borg: Running before action command hook Adding the following volumes to the backup: paas-example: test2 Adding postgres server postgres.service.local to the backup. ssh://user@host/./paas_test.borg: Running before action command hook ssh://user@host/./paas_test.borg: Creating archive ssh://user@host/./paas_test.borg: Dumping PostgreSQL databases Creating archive at "ssh://user@host/./paas_test.borg::backup-2025-04-10T10:07:00" summary: /etc/borgmatic.d/config.yaml: Successfully ran configuration file ```

witten commented

2025-04-10 17:22:50 +00:00

Owner

I'm glad to hear you have an outcome you're happy with!

Agreed, but that verbosity level is relatively noisy and I am thinking of something more readable like "Adding volume 'x' from project 'y' to backup list" -- at least that is what my code currently does.

Ahh gotcha. Yeah, that seems nicer.

The include mechanism is totally custom even if the syntax is somewhat standard YAML.

Interesting, didn't think to much about it, but I interpreted <<: as the standard merge mechanism form YAML 1.1 and !include as a custom tag.

It's all custom under the hood, even if it's standard YAML syntax.

I'd love include: /path/with_some_glob/* on the top level and let that get (deep?) merged with the existing configuration. Not sure how that interacts with the schema validation though, but maybe merge first and then validate?

That's how it works now—deep merge first and then validate. So glob includes should be possible.

Ah yes that would indeed work. Though at this point you probably want to make credential provider registration plugable 🙊

It already is! At least within the borgmatic monolith. Here are the various providers: https://projects.torsion.org/borgmatic-collective/borgmatic/src/branch/main/borgmatic/hooks/credential

Yes, but I reworked a bit of the logic yesterday and should be able to dynamically generate the relevant pattern files and config for borgmatic, will link the result.

Cool!

Yeah, to be honest I haven't tried the database support in borgmatic yet myself, but thought it would be nice to have the dump just in the backup and not having to go and roundtrip via the filesystem.

Yeah, that's the main benefit of the database hooks (from a performance perspective).

That is really kind of you, but I understand am fine with doing the work and don't want to create work for you/us just to make my life easier. I think anything added to borgmatic should follow a higher goal (with regards to how useful it would be to how many people etc…). Currently I think a common middleground would be to make a hook API public so someone can implement something like postgresql_databases on their own by providing the necessary hook scripts to borgmatic. This way people could experiment and if a hook shows that it is really useful (ie used by many) it could get moved into borgmatic properly.

Totally. But often, needs like these are representative of many other users as well. It's not that I think every borgmatic user is making a custom PaaS. There are however lots of users who have a need to easily dump and backup databases running in containers. (Incidentally, I'm one of them.)

Anyway, as for a hook API for implementing something like postgresql_databases.. I assume you're suggesting a command hook API. But the difficulty there is that Bash commands or scripts are not great for passing structured information. What you're describing could be generalized though. Like broadly speaking you need a way to inject custom, structured configuration into borgmatic's config files. Just brainstorming ways to do that:

includes (already implemented, as you've discovered)
command-line config overrides (already implemented)
environment variables config overrides (e.g. something like BORGMATIC_POSTGRESQL_DATABASES.NAME=blah)
container label introspection (as floated above)
maybe even a command hook API returning a JSON blob or something that gets merged into the config?
other options?

I'm glad to hear you have an outcome you're happy with! > Agreed, but that verbosity level is relatively noisy and I am thinking of something more readable like "Adding volume 'x' from project 'y' to backup list" -- at least that is what my code currently does. Ahh gotcha. Yeah, that seems nicer. > > The include mechanism is totally custom even if the syntax is somewhat standard YAML. > > Interesting, didn't think to much about it, but I interpreted `<<:` as the standard merge mechanism form YAML 1.1 and `!include` as a custom tag. It's all custom under the hood, even if it's standard YAML syntax. > I'd love `include: /path/with_some_glob/*` on the top level and let that get (deep?) merged with the existing configuration. Not sure how that interacts with the schema validation though, but maybe merge first and then validate? That's how it works now—deep merge first and then validate. So glob includes should be possible. > Ah yes that would indeed work. Though at this point you probably want to make credential provider registration plugable 🙊 It already is! At least within the borgmatic monolith. Here are the various providers: https://projects.torsion.org/borgmatic-collective/borgmatic/src/branch/main/borgmatic/hooks/credential > Yes, but I reworked a bit of the logic yesterday and should be able to dynamically generate the relevant pattern files and config for borgmatic, will link the result. Cool! > Yeah, to be honest I haven't tried the database support in borgmatic yet myself, but thought it would be nice to have the dump just in the backup and not having to go and roundtrip via the filesystem. Yeah, that's the main benefit of the database hooks (from a performance perspective). > That is really kind of you, but I understand am fine with doing the work and don't want to create work for you/us just to make my life easier. I think anything added to borgmatic should follow a higher goal (with regards to how useful it would be to how many people etc…). Currently I think a common middleground would be to make a hook API public so someone can implement something like `postgresql_databases` on their own by providing the necessary hook scripts to borgmatic. This way people could experiment and if a hook shows that it is really useful (ie used by many) it could get moved into borgmatic properly. Totally. But often, needs like these are representative of many other users as well. It's not that I think every borgmatic user is making a custom PaaS. There are however lots of users who have a need to easily dump and backup databases running in containers. (Incidentally, I'm one of them.) Anyway, as for a hook API for implementing something like `postgresql_databases`.. I assume you're suggesting a *command hook* API. But the difficulty there is that Bash commands or scripts are not great for passing structured information. What you're describing *could* be generalized though. Like broadly speaking you need a way to inject custom, structured configuration into borgmatic's config files. Just brainstorming ways to do that: - includes (already implemented, as you've discovered) - [command-line config overrides]([url](https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/#configuration-overrides)) (already implemented) - environment variables config overrides (e.g. something like `BORGMATIC_POSTGRESQL_DATABASES.NAME=blah`) - container label introspection (as floated above) - maybe even a command hook API returning a JSON blob or something that gets merged into the config? - other options?

apollo13 commented

2025-04-10 17:43:50 +00:00

Author

Contributor

I assume you're suggesting a command hook API.

Nope, I actually thought of documenting the python hook API and providing a way that people could provide their own hook implementations via python modules.

> I assume you're suggesting a command hook API. Nope, I actually thought of documenting the python hook API and providing a way that people could provide their own hook implementations via python modules.

witten commented

2025-04-10 18:16:34 +00:00

Owner

Oh gotcha! I'm not necessarily against that in the fullness of time, but honestly I'm not sure I'm quite ready for that right now. My rationale is that the now-internal hook API can and does change quite often, and allowing it to change makes borgmatic more able to be responsive to new and changing user needs. But if the API was made public, it'd have to be made more fixed, and therefore it'd be that much harder to make borgmatic evolve over time.

That's one reason I'm more a proponent of "config as API" here, whether that config is coming from includes or container labels or somewhere else. Config is declarative data, and so I feel pretty confident that I can evolve its schema over time to meet changing requirements—because I've already had to do it many times over the history of borgmatic.

Oh gotcha! I'm not necessarily against that in the fullness of time, but honestly I'm not sure I'm quite ready for that right now. My rationale is that the now-internal hook API can and does change quite often, and allowing it to change makes borgmatic more able to be responsive to new and changing user needs. But if the API was made public, it'd have to be made more fixed, and therefore it'd be that much harder to make borgmatic evolve over time. That's one reason I'm more a proponent of "config as API" here, whether that config is coming from includes or container labels or somewhere else. Config is declarative data, and so I feel pretty confident that I can evolve its schema over time to meet changing requirements—because I've already had to do it many times over the history of borgmatic.

👍 1

Rows
Columns

Dynamic creation of database backup configuration #1064

What I'd like to do and why

Other notes / implementation ideas