borgmatic-collective/borgmatic

Watch

Fork 57

Generic "cmd" data source #1344

New issue

Open

opened 2026-09-01 01:41:58 +00:00 by l00ptr · 4 comments

l00ptr commented

2026-09-01 01:41:58 +00:00

What I'd like to do and why

I use tools that write important data to stdout, i would like to take backup of those information; Ideally, I would like to back up this information without writing to a local file first (I would like to avoid command hooks to write to temporary files) I am wondering whether it would be possible to have some kind of generic data source and whether it would be useful.

E.g.:

cmd_pipe:
    - cmd: ["ls", "-ahltr"]
      named_pipe: /tmp/my_cmd_stdout

This will make Borgmatic run those commands and use the same mechanism as the other data source to back up the resulting stream or the content of the named pipe, as long as it is being fed. What do you think of this use case, and do you think it's a good way to start experimenting with Borgmatic?

Other notes / implementation ideas

No response

### What I'd like to do and why I use tools that write important data to stdout, i would like to take backup of those information; Ideally, I would like to back up this information without writing to a local file first (I would like to avoid command hooks to write to temporary files) I am wondering whether it would be possible to have some kind of generic data source and whether it would be useful. E.g.: ``` yaml cmd_pipe: - cmd: ["ls", "-ahltr"] named_pipe: /tmp/my_cmd_stdout ``` This will make Borgmatic run those commands and use the same mechanism as the other data source to back up the resulting stream or the content of the named pipe, as long as it is being fed. What do you think of this use case, and do you think it's a good way to start experimenting with Borgmatic? ### Other notes / implementation ideas _No response_

l00ptr changed title from ~~generic data source~~ to Generic "cmd" data source

2026-09-01 01:42:46 +00:00

witten commented

2026-09-01 06:18:45 +00:00

Owner

Interesting idea! Thanks for filing it. A few questions:

Could you say a little more about the why here? For instance, why do you want to backup the output of that particular ls command and other commands?
Once it's backed up, what do you plan on doing with this data? How would you like to consume it from a Borg archive? When would you consume it?
Is there a "restore" analog for this data? Or do you just want it in the archive to manually look at?
Are there other command examples you have in mind besides ls?

As for the configuration design, I'd actually recommend modeling it after database data sources a bit. Something like:

command_outputs:
    - name: listing
      command: ls -ahltr
      stderr: true
    - name: packages
      command: apt list

I don't think it should be necessary for the user to specify the pipe path. borgmatic would presumably put that in a canonical location inside the Borg archive (via a named pipe in borgmatic's runtime directory). So for instance, the example above might create files in the archive at borgmatic/command_outputs/listing and borgmatic/command_outputs/packages.

Interesting idea! Thanks for filing it. A few questions: - Could you say a little more about the *why* here? For instance, why do you want to backup the output of that particular `ls` command and other commands? - Once it's backed up, what do you plan on doing with this data? How would you like to consume it from a Borg archive? When would you consume it? - Is there a "restore" analog for this data? Or do you just want it in the archive to manually look at? - Are there other command examples you have in mind besides `ls`? As for the configuration design, I'd actually recommend modeling it after database data sources a bit. Something like: ```yaml command_outputs: - name: listing command: ls -ahltr stderr: true - name: packages command: apt list ``` I don't think it should be necessary for the user to specify the pipe path. borgmatic would presumably put that in a [canonical location](https://torsion.org/borgmatic/how-to/backup-your-databases/#manual-restoration) inside the Borg archive (via a named pipe in borgmatic's runtime directory). So for instance, the example above might create files in the archive at `borgmatic/command_outputs/listing` and `borgmatic/command_outputs/packages`.

witten added the

new feature area

label

2026-09-01 06:18:53 +00:00

l00ptr commented

2026-09-01 07:03:50 +00:00

Author

Hi Witten!

thx for your promt reply, here are a few anwser to your question:

@witten wrote in #1344 (comment):

Interesting idea! Thanks for filing it. A few questions:

* Could you say a little more about the _why_ here? For instance, why do you want to backup the output of that particular `ls` command and other commands?

I have different source of data, some of them can be backup-ed with usual file access, other need docker compose run command to executed before and others require to requests API.

Of course, I can write them to local file first and then take a backup, but I don't like the idea of potentially leaving unencrypted data behind me; So I think IPC/Pipes could be useful here. Just to let you know, here are two example of command we actually use in our backup process:

$ docker compose run --rm app tar cf   -C /usr/local/app data > /var/backups/files.tar
$ docker compose run --rm db \
           sh -c 'mariadb-dump --password=${MYSQL_ROOT_PASSWORD} --singe-transaction --host=db --database app' > \
          /var/backups/database_dump.sql
$ curl http://.../config > /var/backups/config

You might be wondering why we don't run a dedicated container for Borg/Bormatic. Personally, I think it's overkill. Having such a sidecar or dedicated container would mean having a ton of containers just for backups, when we could simply orchestrate the backups from the Docker host. If you want, I can probably find and share other examples, non related to docker / container limitation.

* Once it's backed up, what do you plan on doing with this data? How would you like to consume it from a Borg archive? When would you consume it?

The idea is that we should be able to restore them. We don't want to let Borgmatic/Borg handle advanced use cases for restoration. In my opinion, when using this kind of composable approach, it is the responsibility of the user to handle restoring from stdin. Therefore, we only need to print or restore the backup content to stdout.

* Is there a "restore" analog for this data? Or do you just want it in the archive to manually look at?

More information on my answer to the previous question :)

* Are there other command examples you have in mind besides `ls`?

Yes, a lot of different command I actually use to dump data locally which are then backup-ed with Borgmatic / Borg.

As for the configuration design, I'd actually recommend modeling it after database data sources a bit. Something like:
command_outputs:
    - name: listing
      command: ls -ahltr
      stderr: true
    - name: packages
      command: apt list
I don't think it should be necessary for the user to specify the pipe path. borgmatic would presumably put that in a canonical location inside the Borg archive (via a named pipe in borgmatic's runtime directory). So for instance, the example above might create files in the archive at borgmatic/command_outputs/listing and borgmatic/command_outputs/packages.

Ok thanks for the information, I will dig deeper into the code and try a first implementation!

Hi Witten! thx for your promt reply, here are a few anwser to your question: @witten wrote in https://projects.torsion.org/borgmatic-collective/borgmatic/issues/1344#issuecomment-14422: > Interesting idea! Thanks for filing it. A few questions: > > * Could you say a little more about the _why_ here? For instance, why do you want to backup the output of that particular `ls` command and other commands? I have different source of data, some of them can be backup-ed with usual file access, other need `docker compose run` command to executed before and others require to requests API. Of course, I can write them to local file first and then take a backup, but I don't like the idea of potentially leaving unencrypted data behind me; So I think IPC/Pipes could be useful here. Just to let you know, here are two example of command we actually use in our backup process: ``` console $ docker compose run --rm app tar cf -C /usr/local/app data > /var/backups/files.tar $ docker compose run --rm db \ sh -c 'mariadb-dump --password=${MYSQL_ROOT_PASSWORD} --singe-transaction --host=db --database app' > \ /var/backups/database_dump.sql $ curl http://.../config > /var/backups/config ``` You might be wondering why we don't run a dedicated container for Borg/Bormatic. Personally, I think it's overkill. Having such a sidecar or dedicated container would mean having a ton of containers just for backups, when we could simply orchestrate the backups from the Docker host. If you want, I can probably find and share other examples, non related to docker / container limitation. > > * Once it's backed up, what do you plan on doing with this data? How would you like to consume it from a Borg archive? When would you consume it? The idea is that we should be able to restore them. We don't want to let Borgmatic/Borg handle advanced use cases for restoration. In my opinion, when using this kind of composable approach, it is the responsibility of the user to handle restoring from stdin. Therefore, we only need to print or restore the backup content to stdout. > > * Is there a "restore" analog for this data? Or do you just want it in the archive to manually look at? More information on my answer to the previous question :) > > * Are there other command examples you have in mind besides `ls`? Yes, a lot of different command I actually use to dump data locally which are then backup-ed with Borgmatic / Borg. > > As for the configuration design, I'd actually recommend modeling it after database data sources a bit. Something like: > > ```yaml > command_outputs: > - name: listing > command: ls -ahltr > stderr: true > - name: packages > command: apt list > ``` > > I don't think it should be necessary for the user to specify the pipe path. borgmatic would presumably put that in a [canonical location](https://torsion.org/borgmatic/how-to/backup-your-databases/#manual-restoration) inside the Borg archive (via a named pipe in borgmatic's runtime directory). So for instance, the example above might create files in the archive at `borgmatic/command_outputs/listing` and `borgmatic/command_outputs/packages`. Ok thanks for the information, I will dig deeper into the code and try a first implementation!

witten commented

2026-09-01 19:20:58 +00:00

Owner

I have different source of data, some of them can be backup-ed with usual file access, other need docker compose run command to executed before and others require to requests API.

Of course, I can write them to local file first and then take a backup, but I don't like the idea of potentially leaving unencrypted data behind me; So I think IPC/Pipes could be useful here. Just to let you know, here are two example of command we actually use in our backup process:
$ docker compose run --rm app tar cf   -C /usr/local/app data > /var/backups/files.tar
$ docker compose run --rm db \
           sh -c 'mariadb-dump --password=${MYSQL_ROOT_PASSWORD} --singe-transaction --host=db --database app' > \
          /var/backups/database_dump.sql
$ curl http://.../config > /var/backups/config

Makes sense. Thanks for explaining. Note that for the database dump in particular, you might also be able to do that with borgmatic's built-in database hooks, including the dumping from a container part.

You might be wondering why we don't run a dedicated container for Borg/Bormatic. Personally, I think it's overkill. Having such a sidecar or dedicated container would mean having a ton of containers just for backups, when we could simply orchestrate the backups from the Docker host. If you want, I can probably find and share other examples, non related to docker / container limitation.

Nope, I think that's enough to give me an idea. And running borgmatic on the host makes sense to me.

The idea is that we should be able to restore them. We don't want to let Borgmatic/Borg handle advanced use cases for restoration. In my opinion, when using this kind of composable approach, it is the responsibility of the user to handle restoring from stdin. Therefore, we only need to print or restore the backup content to stdout.

Okay, fair enough. As a future enhancement, I could see adding a configurable restore_command: that allows the user to write a command that takes the backed up data as input. But for an initial version, maybe a borgmatic extract --stdout flag would be sufficient. (Note that --stdout would have to be added.. Should be easy to do as Borg supports it already.)

Ok thanks for the information, I will dig deeper into the code and try a first implementation!

Great! Let me know if you have any questions or problems along the way.

> I have different source of data, some of them can be backup-ed with usual file access, other need `docker compose run` command to executed before and others require to requests API. > > Of course, I can write them to local file first and then take a backup, but I don't like the idea of potentially leaving unencrypted data behind me; So I think IPC/Pipes could be useful here. Just to let you know, here are two example of command we actually use in our backup process: > > ```console > $ docker compose run --rm app tar cf -C /usr/local/app data > /var/backups/files.tar > $ docker compose run --rm db \ > sh -c 'mariadb-dump --password=${MYSQL_ROOT_PASSWORD} --singe-transaction --host=db --database app' > \ > /var/backups/database_dump.sql > $ curl http://.../config > /var/backups/config > ``` Makes sense. Thanks for explaining. Note that for the database dump in particular, you might also be able to do that with borgmatic's built-in database hooks, [including the dumping from a container part](https://torsion.org/borgmatic/how-to/backup-your-databases/#database-containers). > You might be wondering why we don't run a dedicated container for Borg/Bormatic. Personally, I think it's overkill. Having such a sidecar or dedicated container would mean having a ton of containers just for backups, when we could simply orchestrate the backups from the Docker host. If you want, I can probably find and share other examples, non related to docker / container limitation. Nope, I think that's enough to give me an idea. And running borgmatic on the host makes sense to me. > The idea is that we should be able to restore them. We don't want to let Borgmatic/Borg handle advanced use cases for restoration. In my opinion, when using this kind of composable approach, it is the responsibility of the user to handle restoring from stdin. Therefore, we only need to print or restore the backup content to stdout. Okay, fair enough. As a future enhancement, I could see adding a configurable `restore_command:` that allows the user to write a command that takes the backed up data as input. But for an initial version, maybe a `borgmatic extract --stdout` flag would be sufficient. (Note that `--stdout` would have to be added.. Should be easy to do as Borg supports it already.) > Ok thanks for the information, I will dig deeper into the code and try a first implementation! Great! Let me know if you have any questions or problems along the way.

witten commented

2026-09-01 19:30:21 +00:00

Owner

That also makes me wonder whether a "native" curl data source hook might be interesting. So maybe instead or in addition to the command data source hook described in this ticket, there could be: a tarball data source hook (#1273), a MariaDB data source hook (already exists), and a URL data source hook.

Just brainstorming here. I'm not attached to any one approach.

One note about tarballs in particular: I don't think you'd get any of the benefits of Borg's deduplication if storing tarballs directly in an archive. It's generally preferred to store individual files that came from the tarball to get those dedup benefits. So for that particular use case, it may be preferable to volume-mount your files onto the host and then backup source directories from there. Alternatively, there's #1273, which describes a feature to natively take tarballs as a data source. As spec'd, it doesn't currently support `docker run` before `tar`, but it totally could, much in the way that the existing database hooks support `docker run`. That also makes me wonder whether a "native" curl data source hook might be interesting. So maybe instead or in addition to the command data source hook described in this ticket, there could be: a tarball data source hook (#1273), a MariaDB data source hook (already exists), and a URL data source hook. Just brainstorming here. I'm not attached to any one approach.