stdin or piped input #42

Closed
opened 2018-01-05 05:34:40 +00:00 by import_bot · 7 comments
Collaborator

borg has support for piped input, great for mysqldump, but I don't see a way to configure with borgmatic.


Imported from Taiga issue 41 (to do). Created on 2017-09-08T13:41:37+0000 by asdfasdfoaijsdf.

borg has support for piped input, great for mysqldump, but I don't see a way to configure with borgmatic. --- Imported from Taiga issue 41 (to do). Created on 2017-09-08T13:41:37+0000 by asdfasdfoaijsdf.
Author
Collaborator

I'm not familiar with this feature, but that doesn't mean it's not a great idea. :) Could you tell me a little more about how you use piped input and why you use it? Maybe give an example with Borg?


Comment on 2017-09-09T04:04:23+0000 by Dan Helfman.

I'm not familiar with this feature, but that doesn't mean it's not a great idea. :) Could you tell me a little more about how you use piped input and why you use it? Maybe give an example with Borg? --- Comment on 2017-09-09T04:04:23+0000 by Dan Helfman.
Author
Collaborator

Here's the documentation on the feature in borg

http://borgbackup.readthedocs.io/en/stable/usage.html?highlight=stdin#id3

I got the idea from a similar backup program called restic:

http://restic.readthedocs.io/en/stable/manual.html#reading-data-from-stdin

I use it pretty much exactly as in the example of restic, to pipe the output of mysqldump directly into a borg repo.


Comment on 2017-09-09T11:12:36+0000 by jorisjh.

Here's the documentation on the feature in borg http://borgbackup.readthedocs.io/en/stable/usage.html?highlight=stdin#id3 I got the idea from a similar backup program called restic: http://restic.readthedocs.io/en/stable/manual.html#reading-data-from-stdin I use it pretty much exactly as in the example of restic, to pipe the output of mysqldump directly into a borg repo. --- Comment on 2017-09-09T11:12:36+0000 by jorisjh.
Author
Collaborator

Got it, thanks. That makes sense.. I can envision two different ways that borgmatic could implement that feature:

  1. borgmatic config could support specifying a command to run (e.g., you could specify mysqldump there), the output of which borgmatic pipes into the borg create invocation.
  2. Or, borgmatic could optionally accept stdin piping on the command-line, and just pass that stream through to the borg create invocation. Then, you'd have to invoke borgmatic with the command you want to use (e.g. mysqldump | borgmatic --stdin or whatever).

The second option would have some challenges, in that it'd be unclear which of the multiple borgmatic config files should consume that stdin. Like, if you've got five borgmatic config files and thus five Borg repositories, you probably only want that stdin to go to one of them. So that might mean option one is a more natural fit.


Comment on 2017-09-09T17:01:44+0000 by Dan Helfman.

Got it, thanks. That makes sense.. I can envision two different ways that borgmatic could implement that feature: 1. borgmatic config could support specifying a command to run (e.g., you could specify `mysqldump` there), the output of which borgmatic pipes into the `borg create` invocation. 2. Or, borgmatic could optionally accept stdin piping on the command-line, and just pass that stream through to the `borg create` invocation. Then, you'd have to invoke borgmatic with the command you want to use (e.g. `mysqldump | borgmatic --stdin` or whatever). The second option would have some challenges, in that it'd be unclear which of the multiple borgmatic config files should consume that stdin. Like, if you've got five borgmatic config files and thus five Borg repositories, you probably only want that stdin to go to one of them. So that might mean option one is a more natural fit. --- Comment on 2017-09-09T17:01:44+0000 by Dan Helfman.
Owner

Now that borgmatic has a hooks system in place, option number 1 may be doable with a hook. For instance, there could be a new backup_input hook that runs an arbitrary command and pipes its output to Borg's stdin pretty easily.

Support for the borg create --stdin-name flag might also be nice to have.

Now that borgmatic has a [hooks system](https://torsion.org/borgmatic/#hooks) in place, option number 1 may be doable with a hook. For instance, there could be a new `backup_input` hook that runs an arbitrary command and pipes its output to Borg's stdin pretty easily. Support for the `borg create --stdin-name` flag might also be nice to have.
witten added the
design finalized
label 2018-10-05 05:27:23 +00:00

Hello!

I just wanted to say that I'm interested in this feature 🙂

Hello! I just wanted to say that I'm interested in this feature 🙂
Owner

Wow, it looks like this might "just work". I tried adding a borgmatic source directory of "-" (with quotes) in the config file to indicate stdin. Then I was able to pipe input in via the command-line:

echo "hi" | borgmatic --create

I did an extract and confirmed that a file named "stdin" with those contents was extracted.

So, work still to-do before calling this feature done:

  1. Document it.
  2. Figure out what to do when there are multiple configuration files with "-" declared. It appears that the first instance of "-" consumes all the stdin, and the second instance silently consumes nothing. Which doesn't seem like great behavior. One idea to deal with this is to guard that "-" is present at most once in all source directories across all config files. Error if it appears twice or more.
  3. Probably add support for --stdin-name.
Wow, it looks like this might "just work". I tried adding a borgmatic source directory of `"-"` (with quotes) in the config file to indicate stdin. Then I was able to pipe input in via the command-line: ```bash echo "hi" | borgmatic --create ``` I did an extract and confirmed that a file named "stdin" with those contents was extracted. So, work still to-do before calling this feature done: 1. Document it. 2. Figure out what to do when there are multiple configuration files with `"-"` declared. It appears that the first instance of `"-"` consumes all the stdin, and the second instance silently consumes nothing. Which doesn't seem like great behavior. One idea to deal with this is to guard that `"-"` is present at most once in all source directories across all config files. Error if it appears twice or more. 3. Probably add support for `--stdin-name`.
Owner

With #258 implemented, adding separate support for stdin streaming should no longer be necessary. That's because with the new automatic database -> Borg streaming support in #258, you shouldn't have to manually execute database dumps and stream them into borgmatic.

Having said that, if there's a use case I've missed with stdin support, please feel free to file a new ticket or reopen this one.

With #258 implemented, adding separate support for stdin streaming should no longer be necessary. That's because with the new automatic database -> Borg streaming support in #258, you shouldn't have to manually execute database dumps and stream them into borgmatic. Having said that, if there's a use case I've missed with stdin support, please feel free to file a new ticket or reopen this one.
Sign in to join this conversation.
No Milestone
No Assignees
3 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#42
No description provided.