read_special defaults to true if unset #315
Labels
No Label
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
No Milestone
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: borgmatic-collective/borgmatic#315
Loading…
Reference in New Issue
No description provided.
Delete Branch "%!s(<nil>)"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What I'm trying to do and why
Steps to reproduce (if a bug)
run borgmatic with the attached config - with unset read_special.
Actual behavior (if a bug)
Backup is created including --read-special (which included a /dev/random device of a Collabora Online installation until the destination was full).
Edit: Even with
read_special: false
it sets--read-special
as a parameter.Expected behavior (if a bug)
No inclusion of special devices when read_special is not set to true.
Environment
borgmatic version: 1.5.4
borgmatic installation method: Arch Linux package
Borg version: 1.1.11
Python version: 3.8.2
operating system and version: Arch Linux
read_special defaults to trueto read_special defaults to true if unsetUgh! Thank you for filing this. Here's what I think is going on: The new database integration uses named pipes on the filesystem to stream dumps directly to Borg. This means that whenever a database hook is enabled, borgmatic automatically/implicitly enables Borg's
--read-special
, so that reading from named pipes works.Obviously, this has some pretty bad side-effects.. Like
--read-special
being active when reading/dev/random
, as you discovered.So, brainstorming potential solutions... What do you think of: 1. Better documentation on this, so users are aware of it, and 2. Perhaps borgmatic automatically excluding paths like
/dev
to hopefully sidestep this. At least when a database hook is in use.In the meantime, you could try adding a manual exclude for
/dev
to see if that sidesteps the issue./dev/random
did not get backed up here, as I useone_file_system = true
. But Collabora Online puts somehow a random and an urandom device into/opt/lool/child-roots/<somehash>/dev
which isn't a extra mounted devfs, so it gets included when backing up/
.I think solution 2 would be the better one, maybe with an extra documentated option if someone who backs up
/
wants really to backup his/her devfs. Ad hoc I have no idea of a usecase why somebody would want to do this.It sounds like solution 2 (borgmatic auto-excluding
/dev
) won't address your issue though, because/opt/lool/child-roots/<somehash>/dev
isn't located there. I suppose you could manually exclude/opt/lool/child-roots/
, but that's annoying for you to have to do. Thoughts?I agree about someone wanting to backup
/dev
.. Seems pretty low-utility.FYI I tried backing up
/
on my host (Manjaro Linux) with--read-special
implicitly enabled, and I had to exclude all of the following directories to get the backup not to hang:Not using
one_file_system
, which would take care of most of that. Which leads me to another idea: Maybe borgmatic should automatically enableone_file_system
whenever automatically enabling--read-special
? That might "solve" most of this.. At the cost of limiting what you can backup when database hooks are enabled. And of course not actually solving your/opt/.../dev
issue.It's excluded now manually which is fine.
Okay. In that case I'll go with the implicit
one_file_system
approach, along with documentating the hell out of this behavior. I'll leave this ticket open til that's done. Thanks again for the bug report.Would it be a big task to make the DB-backup behaviour switchable, turning backup via pipe on/off? I know streaming database backups was a wish from #258, but if space is not such an issue, the "usual" behaviour without the read-special side effects is IMHO sufficient. Using this as default and a big warning regarding piped database backups and it's side-effects as special devices aren't excluded any more (if not one_file_system is set) would be nice.
I think that's certainly a backstop if there are too many more issues with the auto-streaming behavior. But I was hoping that I could get away with just one supported database dump/restore approach, and a single code path for it. (Although I've already had to stray from that somewhat to continue supporting PostgreSQL's directory dump format, which by its nature can't stream.)
Okay, the "fix" of forcing
one_file_system
is released with borgmatic 1.5.5!Strange issue now with the storage dir of Knot DNS
It contains some certs and LMDB files and borg seems to get a timeout. Are LMDB files considered special or is it another issue?
I'm not familiar with LMDB, but here's something that might be relevant from http://www.lmdb.tech/doc/:
So the way I'm reading that is that one shouldn't access an LMDB database from multiple different processes. Although that's contradicted by the Wikipedia entry. In any case, borgmatic is just reading from the filesystem and not "connecting" to LMDB of course.
Another idea is that LMDB has some named pipes or other devices on its filesystem that borgmatic can't read without hanging.
What's the directory listing look like where the problem is occurring?
When you say that Borg gets a timeout, what's that look like? Is it an actual remote timeout error?
If I remember correctly, the output tells about the last included file? So
lock.mdb
seems to be included:But it seems borg went further after some time, so it's no real lockup.
So is there still a problem here? How big is that lock file?
It might be the case that the preferred way of backing up an LMDB database is by exporting/dumping it to file, just like with PostgreSQL and MySQL databases. Here's a command I found: http://www.lmdb.tech/doc/man1/mdb_dump_1.html
It's possible that you could exclude the entire LMDB database directory, and then back it up via that dump command instead.