stdin or piped input #42
Labels
No Label
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
No Milestone
No Assignees
3 Participants
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: borgmatic-collective/borgmatic#42
Loading…
Reference in New Issue
No description provided.
Delete Branch "%!s(<nil>)"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
borg has support for piped input, great for mysqldump, but I don't see a way to configure with borgmatic.
Imported from Taiga issue 41 (to do). Created on 2017-09-08T13:41:37+0000 by asdfasdfoaijsdf.
I'm not familiar with this feature, but that doesn't mean it's not a great idea. :) Could you tell me a little more about how you use piped input and why you use it? Maybe give an example with Borg?
Comment on 2017-09-09T04:04:23+0000 by Dan Helfman.
Here's the documentation on the feature in borg
http://borgbackup.readthedocs.io/en/stable/usage.html?highlight=stdin#id3
I got the idea from a similar backup program called restic:
http://restic.readthedocs.io/en/stable/manual.html#reading-data-from-stdin
I use it pretty much exactly as in the example of restic, to pipe the output of mysqldump directly into a borg repo.
Comment on 2017-09-09T11:12:36+0000 by jorisjh.
Got it, thanks. That makes sense.. I can envision two different ways that borgmatic could implement that feature:
mysqldump
there), the output of which borgmatic pipes into theborg create
invocation.borg create
invocation. Then, you'd have to invoke borgmatic with the command you want to use (e.g.mysqldump | borgmatic --stdin
or whatever).The second option would have some challenges, in that it'd be unclear which of the multiple borgmatic config files should consume that stdin. Like, if you've got five borgmatic config files and thus five Borg repositories, you probably only want that stdin to go to one of them. So that might mean option one is a more natural fit.
Comment on 2017-09-09T17:01:44+0000 by Dan Helfman.
Now that borgmatic has a hooks system in place, option number 1 may be doable with a hook. For instance, there could be a new
backup_input
hook that runs an arbitrary command and pipes its output to Borg's stdin pretty easily.Support for the
borg create --stdin-name
flag might also be nice to have.Hello!
I just wanted to say that I'm interested in this feature 🙂
Wow, it looks like this might "just work". I tried adding a borgmatic source directory of
"-"
(with quotes) in the config file to indicate stdin. Then I was able to pipe input in via the command-line:I did an extract and confirmed that a file named "stdin" with those contents was extracted.
So, work still to-do before calling this feature done:
"-"
declared. It appears that the first instance of"-"
consumes all the stdin, and the second instance silently consumes nothing. Which doesn't seem like great behavior. One idea to deal with this is to guard that"-"
is present at most once in all source directories across all config files. Error if it appears twice or more.--stdin-name
.witten referenced this issue2019-12-02 17:51:56 +00:00
With #258 implemented, adding separate support for stdin streaming should no longer be necessary. That's because with the new automatic database -> Borg streaming support in #258, you shouldn't have to manually execute database dumps and stream them into borgmatic.
Having said that, if there's a use case I've missed with stdin support, please feel free to file a new ticket or reopen this one.