[containers] Allow database backups via `container_name` instead of `hostname`

apollo13 commented

2025-07-01 05:52:59 +00:00

Contributor

What I'd like to do and why

Currently container database backups are supported by specifying the container name as hostname. This only works if borgmatic is running in the same container network though. My borgmatic runs on the host and I'd like to backup databases from multiple compose apps (which all have their own network). If I could specify container_name instead of hostname and let borgmatic run a docker inspect *container_name* to get the IP I would be able to address the container properly. Is this something you would be willing to support -- if yes I can provide the code for it?

Other notes / implementation ideas

No response

### What I'd like to do and why Currently container database backups are supported by specifying the container name as hostname. This only works if borgmatic is running in the same container network though. My borgmatic runs on the host and I'd like to backup databases from multiple compose apps (which all have their own network). If I could specify `container_name` instead of `hostname` and let borgmatic run a `docker inspect *container_name*` to get the IP I would be able to address the container properly. Is this something you would be willing to support -- if yes I can provide the code for it? ### Other notes / implementation ideas _No response_

witten commented

2025-07-01 06:13:47 +00:00

Owner

I'd be willing to support something like this, but I'm not exactly sure how it would work in practice. For instance, as with the current feature for borgmatic running on the host and connecting to a database in a container, container database ports would have to be mapped to the host somehow for borgmatic on the host to be able to connect. Would that be a prerequisite for this proposed feature? And would that port be inspected as well?

Also, instead of connecting to the database by host/port, have you considered dumping directly from a database client running in a container to the host? That might make it easier to just specify a container_name and have borgmatic do the right thing.

Also note that, as per #685, borgmatic should probably support both Docker and Podman.

I'd be willing to support something like this, but I'm not exactly sure how it would work in practice. For instance, as with the current feature for [borgmatic running on the host and connecting to a database in a container](https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#database-client-on-the-host), container database ports would have to be mapped to the host somehow for borgmatic on the host to be able to connect. Would that be a prerequisite for this proposed feature? And would that port be `inspect`ed as well? Also, instead of connecting to the database by host/port, have you considered dumping [directly from a database client running in a container]( https://torsion.org/borgmatic/docs/how-to/backup-your-databases/#database-client-in-a-running-container) to the host? That might make it easier to just specify a `container_name` and have borgmatic do the right thing. Also note that, as per #685, borgmatic should probably support both Docker and Podman.

apollo13 commented

2025-07-01 06:53:04 +00:00

Author

Contributor

No, the port would not need to be mapped and the docker/podman inspect would just figure out the IP. Then borgmatic can connect directly to the IP (host services are allowed to connect to all container IPs by default) and the port configured in the config -- so essentially:

postgresql_databases:
    - name: users
      container_name: my_postgres_container
      port: 5433
      username: postgres
      password: trustsome1

Also, instead of connecting to the database by host/port, have you considered dumping directly from a database client running in a container to the host?

I haven't not yet tried it because I am a bit afraid of the extra processes in the mix (more complexity). Does that work with all the features (ie streaming directly into the backup). If it works with all the feature then it is probably already 99% what I want. The only problem I see when using the container exec command is that the hostname will be the same, so if I do something like this:

postgresql_databases:
    - name: all
      hostname: 127.0.0.1
      port: 5433
      username: postgres
      password: trustsome1
      pg_dump_command: docker exec my_pg_container_1 pg_dump

    - name: all
      hostname: 127.0.0.1
      port: 5433
      username: postgres
      password: trustsome1
      pg_dump_command: docker exec my_pg_container_2 pg_dump

then I will probably have a conflict in the storage names, no?

No, the port would not need to be mapped and the `docker/podman inspect` would just figure out the IP. Then borgmatic can connect directly to the IP (host services are allowed to connect to all container IPs by default) and the port configured in the config -- so essentially: ``` postgresql_databases: - name: users container_name: my_postgres_container port: 5433 username: postgres password: trustsome1 ``` > Also, instead of connecting to the database by host/port, have you considered dumping directly from a database client running in a container to the host? I haven't not yet tried it because I am a bit afraid of the extra processes in the mix (more complexity). Does that work with all the features (ie streaming directly into the backup). If it works with all the feature then it is probably already 99% what I want. The only problem I see when using the container exec command is that the hostname will be the same, so if I do something like this: ``` postgresql_databases: - name: all hostname: 127.0.0.1 port: 5433 username: postgres password: trustsome1 pg_dump_command: docker exec my_pg_container_1 pg_dump - name: all hostname: 127.0.0.1 port: 5433 username: postgres password: trustsome1 pg_dump_command: docker exec my_pg_container_2 pg_dump ``` then I will probably have a conflict in the storage names, no?

witten commented

2025-07-19 21:42:01 +00:00

Owner

Thanks for the additional details about the port and IP.

I haven't not yet tried it because I am a bit afraid of the extra processes in the mix (more complexity). Does that work with all the features (ie streaming directly into the backup).

Yes, I believe so, as long as you volume mount the runtime directory as described in the docs.

If it works with all the feature then it is probably already 99% what I want. The only problem I see when using the container exec command is that the hostname will be the same, so if I do something like this:

[...]

then I will probably have a conflict in the storage names, no?

You are correct! That would indeed be a problem with this approach.

Thanks for the additional details about the port and IP. > I haven't not yet tried it because I am a bit afraid of the extra processes in the mix (more complexity). Does that work with all the features (ie streaming directly into the backup). Yes, I believe so, as long as you volume mount the runtime directory as described in the docs. > If it works with all the feature then it is probably already 99% what I want. The only problem I see when using the container exec command is that the hostname will be the same, so if I do something like this: [...] > then I will probably have a conflict in the storage names, no? You are correct! That would indeed be a problem with this approach.

apollo13 commented

2025-07-20 17:40:53 +00:00

Author

Contributor

Yes, I believe so, as long as you volume mount the runtime directory as described in the docs.

I'd like to avoid modifying the containers.

You are correct! That would indeed be a problem with this approach.

Can I read that as an "ok, go ahead with a PR/MR"?

> Yes, I believe so, as long as you volume mount the runtime directory as described in the docs. I'd like to avoid modifying the containers. > You are correct! That would indeed be a problem with this approach. Can I read that as an "ok, go ahead with a PR/MR"?

witten commented

2025-07-21 00:55:31 +00:00

Owner

For doing the container_name approach? Sure, a PR would be great. Be aware though that there have been some (hopefully good) dev stack changes since you last submitted a PR.

For doing the `container_name` approach? Sure, a PR would be great. Be aware though that there have been some (hopefully good) [dev stack changes](https://torsion.org/borgmatic/docs/how-to/develop-on-borgmatic/) since you last submitted a PR.

witten commented

2025-07-21 01:04:33 +00:00

Owner

One variant on your implementation idea—and I'll emphasize that I'm just spitballing here—is that instead of a dedicated container_name option, there could be a {container ip ...} interpolation syntax, much like the existing {credential ...} syntax for another use case. Example usage:

postgresql_databases:
    - name: users
      hostname: "{container ip my_postgres_container}"
      port: 5433
      username: postgres
      password: trustsome1

The idea is that it would allow the container IP to get looked up and plugged in anywhere a hostname was needed in borgmatic's vast config file. So, that means in the hostname option for all other database hooks, all monitoring hooks, etc. And that's without needing (major) changes to any of those hooks to support it.

The downside is that the container name wouldn't be a first-class concept in those hooks. The hooks would only get an IP address for the container and that's it. And then the existing per-hook hostname machinery could go on its merry way using that IP. But that might be enough?

One variant on your implementation idea—and I'll emphasize that I'm just spitballing here—is that instead of a dedicated `container_name` option, there could be a `{container ip ...}` interpolation syntax, much like the existing [`{credential ...}` syntax](https://torsion.org/borgmatic/docs/how-to/provide-your-passwords/#container-secrets) for another use case. Example usage: ```yaml postgresql_databases: - name: users hostname: "{container ip my_postgres_container}" port: 5433 username: postgres password: trustsome1 ``` The idea is that it would allow the container IP to get looked up and plugged in anywhere a hostname was needed in borgmatic's vast config file. So, that means in the hostname option for all other database hooks, all monitoring hooks, etc. And that's without needing (major) changes to any of those hooks to support it. The downside is that the container name wouldn't be a first-class concept in those hooks. The hooks would only get an IP address for the container and that's it. And then the existing per-hook hostname machinery could go on its merry way using that IP. But that might be enough?

apollo13 commented

2025-07-21 06:29:40 +00:00

Author

Contributor

Oh, that is most certainly an interesting idea. The only issue I can see is that container ips are not necessarily stable (container restarts usually keep the ip, but destroying and running a new container might not) leading to a potentially new directory in the backup every time and you might have a hard time to figure out which IP was the correct one :D

One could fix that with a label option to the database backup hooks that would be used to keep a stable name.

In the end {container ip my_postgres_container} and a label or solely a container_name both work for me. I think the interpolation is interesting (+label) and looks more powerful. I am not sure how useful it would be for the other hostname options in practice, but it might help?

Let me know which approach you'd prefer and I get hacking.

Oh, that is most certainly an interesting idea. The only issue I can see is that container ips are not necessarily stable (container restarts usually keep the ip, but destroying and running a new container might not) leading to a potentially new directory in the backup every time and you might have a hard time to figure out which IP was the correct one :D One could fix that with a `label` option to the database backup hooks that would be used to keep a stable name. In the end `{container ip my_postgres_container}` and a `label` or solely a `container_name` both work for me. I think the interpolation is interesting (+label) and looks more powerful. I am not sure how useful it would be for the other hostname options in practice, but it might help? Let me know which approach you'd prefer and I get hacking.

witten commented

2025-07-21 18:24:13 +00:00

Owner

Oh, that is most certainly an interesting idea. The only issue I can see is that container ips are not necessarily stable (container restarts usually keep the ip, but destroying and running a new container might not) leading to a potentially new directory in the backup every time and you might have a hard time to figure out which IP was the correct one :D

That is a very good point!

One could fix that with a label option to the database backup hooks that would be used to keep a stable name.

Great idea.

In the end {container ip my_postgres_container} and a label or solely a container_name both work for me. I think the interpolation is interesting (+label) and looks more powerful. I am not sure how useful it would be for the other hostname options in practice, but it might help?

Yeah, I could see interpolation + label: being useful for the other database hooks (except for SQLite). But label: wouldn't be needed for the monitoring hooks, as there's no constructed path stored in the Borg archive for those. (I think the interpolation feature could even be skipped for the monitoring hooks altogether until a need arises.)

So I think I have a slight preference for interpolation + label: over just container_name:, but I don't feel strongly.

Thank you!

> Oh, that is most certainly an interesting idea. The only issue I can see is that container ips are not necessarily stable (container restarts usually keep the ip, but destroying and running a new container might not) leading to a potentially new directory in the backup every time and you might have a hard time to figure out which IP was the correct one :D That is a very good point! > One could fix that with a label option to the database backup hooks that would be used to keep a stable name. Great idea. > In the end {container ip my_postgres_container} and a label or solely a container_name both work for me. I think the interpolation is interesting (+label) and looks more powerful. I am not sure how useful it would be for the other hostname options in practice, but it might help? Yeah, I could see interpolation + `label:` being useful for the other database hooks (except for SQLite). But `label:` wouldn't be needed for the monitoring hooks, as there's no constructed path stored in the Borg archive for those. (I think the interpolation feature could even be skipped for the monitoring hooks altogether until a need arises.) So I think I have a slight preference for interpolation + `label:` over just `container_name:`, but I don't feel strongly. Thank you!

apollo13 commented

2025-07-24 06:58:26 +00:00

Author

Contributor

Hi @witten, I have created a draft PR in #1121 to talk about the design.

What I did for now is adding label: which is a no-brainer. I then opted for container: instead of an interpolation because the reuse potential seems low and an interpolation might want to allow reading labels etc in the future so we should brainstorm it's design first (like {container label=x container_name} might return the label value of label x or similar…

I also opted to "shell out" and use the docker/podman CLIs. In theory I could call the relevant API endpoints directly but that would:

either require an extra dependency on the docker python library
require an extra dependency so requests can talk to unix domain sockets + plenty of extra code to support environment variables like DOCKER_HOST/DOCKER_CONTEXT which the CLI handles out of the box

I would appreciate a "design"-review before I round it up.

Hi @witten, I have created a draft PR in https://projects.torsion.org/borgmatic-collective/borgmatic/pulls/1121 to talk about the design. What I did for now is adding `label:` which is a no-brainer. I then opted for `container:` instead of an interpolation because the reuse potential seems low and an interpolation might want to allow reading labels etc in the future so we should brainstorm it's design first (like `{container label=x container_name}` might return the label value of label `x` or similar… I also opted to "shell out" and use the docker/podman CLIs. In theory I could call the relevant API endpoints directly but that would: * either require an extra dependency on the docker python library * require an extra dependency so requests can talk to unix domain sockets + plenty of extra code to support environment variables like `DOCKER_HOST`/`DOCKER_CONTEXT` which the CLI handles out of the box I would appreciate a "design"-review before I round it up.

👍 1

witten referenced this issue from a commit

2025-09-17 23:03:00 +00:00

Add support for a database backup label instead of host:port (#1116)

witten referenced this issue from a commit

2025-09-17 23:03:00 +00:00

Add dumping of database containers via their container name and for labels in database configuration (#1116).

witten referenced this issue from a commit

2025-09-17 23:25:52 +00:00

Spelling out the new container database features in NEWS a little more and making the documentation on it more verbose as well (#1116).

witten referenced this issue from a commit

2025-09-18 04:56:36 +00:00

Make test code style more consistent with the rest of the codebase (#1116).

witten commented

2025-09-18 04:56:58 +00:00

Owner

This will be part of the next release. Thanks again!

witten closed this issue

2025-09-18 04:56:58 +00:00

witten referenced this issue from a commit

2025-09-18 23:13:50 +00:00

Clarify "container:" documentation about borgmatic running inside a container (#1116).

witten referenced this issue from a commit

2025-09-19 02:23:18 +00:00

No longer defaulting Dump(hostname=...) to "localhost" (#1116).

witten commented

2025-09-20 00:42:42 +00:00

Owner

Released in borgmatic 2.0.8!

apollo13 commented

2025-09-22 13:17:06 +00:00

Author

Contributor

Thank you, is there a timeline when the docker container with borgmatic will get updated?

witten commented

2025-09-22 16:24:16 +00:00

Owner

That's actually a separate project that other folks manage.. It's usually released with a couple days of a borgmatic release, but it really depends on when they get to it.

That's actually a [separate project](https://github.com/borgmatic-collective/docker-borgmatic) that other folks manage.. It's usually released with a couple days of a borgmatic release, but it really depends on when they get to it.

Rows
Columns

[containers] Allow database backups via container_name instead of hostname #1116

What I'd like to do and why

Other notes / implementation ideas

[containers] Allow database backups via `container_name` instead of `hostname` #1116