#42 stdin or piped input

Closed
opened 2 years ago by import_bot · 7 comments

borg has support for piped input, great for mysqldump, but I don’t see a way to configure with borgmatic.


Imported from Taiga issue 41 (to do). Created on 2017-09-08T13:41:37+0000 by asdfasdfoaijsdf.

borg has support for piped input, great for mysqldump, but I don't see a way to configure with borgmatic. --- Imported from Taiga issue 41 (to do). Created on 2017-09-08T13:41:37+0000 by asdfasdfoaijsdf.
import_bot commented 2 years ago
Poster

I’m not familiar with this feature, but that doesn’t mean it’s not a great idea. :) Could you tell me a little more about how you use piped input and why you use it? Maybe give an example with Borg?


Comment on 2017-09-09T04:04:23+0000 by Dan Helfman.

I'm not familiar with this feature, but that doesn't mean it's not a great idea. :) Could you tell me a little more about how you use piped input and why you use it? Maybe give an example with Borg? --- Comment on 2017-09-09T04:04:23+0000 by Dan Helfman.
import_bot commented 2 years ago
Poster

Here’s the documentation on the feature in borg

http://borgbackup.readthedocs.io/en/stable/usage.html?highlight=stdin#id3

I got the idea from a similar backup program called restic:

http://restic.readthedocs.io/en/stable/manual.html#reading-data-from-stdin

I use it pretty much exactly as in the example of restic, to pipe the output of mysqldump directly into a borg repo.


Comment on 2017-09-09T11:12:36+0000 by jorisjh.

Here's the documentation on the feature in borg http://borgbackup.readthedocs.io/en/stable/usage.html?highlight=stdin#id3 I got the idea from a similar backup program called restic: http://restic.readthedocs.io/en/stable/manual.html#reading-data-from-stdin I use it pretty much exactly as in the example of restic, to pipe the output of mysqldump directly into a borg repo. --- Comment on 2017-09-09T11:12:36+0000 by jorisjh.
import_bot commented 2 years ago
Poster

Got it, thanks. That makes sense.. I can envision two different ways that borgmatic could implement that feature:

  1. borgmatic config could support specifying a command to run (e.g., you could specify mysqldump there), the output of which borgmatic pipes into the borg create invocation.
  2. Or, borgmatic could optionally accept stdin piping on the command-line, and just pass that stream through to the borg create invocation. Then, you’d have to invoke borgmatic with the command you want to use (e.g. mysqldump | borgmatic --stdin or whatever).

The second option would have some challenges, in that it’d be unclear which of the multiple borgmatic config files should consume that stdin. Like, if you’ve got five borgmatic config files and thus five Borg repositories, you probably only want that stdin to go to one of them. So that might mean option one is a more natural fit.


Comment on 2017-09-09T17:01:44+0000 by Dan Helfman.

Got it, thanks. That makes sense.. I can envision two different ways that borgmatic could implement that feature: 1. borgmatic config could support specifying a command to run (e.g., you could specify `mysqldump` there), the output of which borgmatic pipes into the `borg create` invocation. 2. Or, borgmatic could optionally accept stdin piping on the command-line, and just pass that stream through to the `borg create` invocation. Then, you'd have to invoke borgmatic with the command you want to use (e.g. `mysqldump | borgmatic --stdin` or whatever). The second option would have some challenges, in that it'd be unclear which of the multiple borgmatic config files should consume that stdin. Like, if you've got five borgmatic config files and thus five Borg repositories, you probably only want that stdin to go to one of them. So that might mean option one is a more natural fit. --- Comment on 2017-09-09T17:01:44+0000 by Dan Helfman.
witten commented 1 year ago
Owner

Now that borgmatic has a hooks system in place, option number 1 may be doable with a hook. For instance, there could be a new backup_input hook that runs an arbitrary command and pipes its output to Borg’s stdin pretty easily.

Support for the borg create --stdin-name flag might also be nice to have.

Now that borgmatic has a [hooks system](https://torsion.org/borgmatic/#hooks) in place, option number 1 may be doable with a hook. For instance, there could be a new `backup_input` hook that runs an arbitrary command and pipes its output to Borg's stdin pretty easily. Support for the `borg create --stdin-name` flag might also be nice to have.
witten added the
design finalized
label 1 year ago
angristan commented 1 year ago

Hello!

I just wanted to say that I’m interested in this feature 🙂

Hello! I just wanted to say that I'm interested in this feature 🙂
witten commented 1 year ago
Owner

Wow, it looks like this might “just work”. I tried adding a borgmatic source directory of "-" (with quotes) in the config file to indicate stdin. Then I was able to pipe input in via the command-line:

echo "hi" | borgmatic --create

I did an extract and confirmed that a file named “stdin” with those contents was extracted.

So, work still to-do before calling this feature done:

  1. Document it.
  2. Figure out what to do when there are multiple configuration files with "-" declared. It appears that the first instance of "-" consumes all the stdin, and the second instance silently consumes nothing. Which doesn’t seem like great behavior. One idea to deal with this is to guard that "-" is present at most once in all source directories across all config files. Error if it appears twice or more.
  3. Probably add support for --stdin-name.
Wow, it looks like this might "just work". I tried adding a borgmatic source directory of `"-"` (with quotes) in the config file to indicate stdin. Then I was able to pipe input in via the command-line: ```bash echo "hi" | borgmatic --create ``` I did an extract and confirmed that a file named "stdin" with those contents was extracted. So, work still to-do before calling this feature done: 1. Document it. 2. Figure out what to do when there are multiple configuration files with `"-"` declared. It appears that the first instance of `"-"` consumes all the stdin, and the second instance silently consumes nothing. Which doesn't seem like great behavior. One idea to deal with this is to guard that `"-"` is present at most once in all source directories across all config files. Error if it appears twice or more. 3. Probably add support for `--stdin-name`.
witten commented 2 months ago
Owner

With #258 implemented, adding separate support for stdin streaming should no longer be necessary. That’s because with the new automatic database -> Borg streaming support in #258, you shouldn’t have to manually execute database dumps and stream them into borgmatic.

Having said that, if there’s a use case I’ve missed with stdin support, please feel free to file a new ticket or reopen this one.

With #258 implemented, adding separate support for stdin streaming should no longer be necessary. That's because with the new automatic database -> Borg streaming support in #258, you shouldn't have to manually execute database dumps and stream them into borgmatic. Having said that, if there's a use case I've missed with stdin support, please feel free to file a new ticket or reopen this one.
witten closed this issue 2 months ago
Sign in to join this conversation.
No Milestone
No Assignees
3 Participants
Notifications
Due Date

No due date set.

Dependencies

This issue currently doesn't have any dependencies.

Loading…
There is no content yet.