#315 read_special defaults to true if unset

Ugh! Thank you for filing this. Here’s what I think is going on: The new database integration uses named pipes on the filesystem to stream dumps directly to Borg. This means that whenever a database hook is enabled, borgmatic automatically/implicitly enables Borg’s --read-special, so that reading from named pipes works.

Obviously, this has some pretty bad side-effects.. Like --read-special being active when reading /dev/random, as you discovered.

So, brainstorming potential solutions... What do you think of: 1. Better documentation on this, so users are aware of it, and 2. Perhaps borgmatic automatically excluding paths like /dev to hopefully sidestep this. At least when a database hook is in use.

In the meantime, you could try adding a manual exclude for /dev to see if that sidesteps the issue.

/dev/random did not get backed up here, as I use one_file_system = true. But Collabora Online puts somehow a random and an urandom device into /opt/lool/child-roots/<somehash>/dev which isn’t a extra mounted devfs, so it gets included when backing up /.
I think solution 2 would be the better one, maybe with an extra documentated option if someone who backs up / wants really to backup his/her devfs. Ad hoc I have no idea of a usecase why somebody would want to do this.

It sounds like solution 2 (borgmatic auto-excluding /dev) won’t address your issue though, because /opt/lool/child-roots/<somehash>/dev isn’t located there. I suppose you could manually exclude /opt/lool/child-roots/, but that’s annoying for you to have to do. Thoughts?

I agree about someone wanting to backup /dev.. Seems pretty low-utility.

FYI I tried backing up / on my host (Manjaro Linux) with --read-special implicitly enabled, and I had to exclude all of the following directories to get the backup not to hang:

• /dev
• /sys
• /proc
• /run

Not using one_file_system, which would take care of most of that. Which leads me to another idea: Maybe borgmatic should automatically enable one_file_system whenever automatically enabling --read-special? That might “solve” most of this.. At the cost of limiting what you can backup when database hooks are enabled. And of course not actually solving your /opt/.../dev issue.

It sounds like solution 2 (borgmatic auto-excluding /dev) won’t address your issue though, because /opt/lool/child-roots/<somehash>/dev isn’t located there. I suppose you could manually exclude /opt/lool/child-roots/, but that’s annoying for you to have to do. Thoughts?

It’s excluded now manually which is fine.

Okay. In that case I’ll go with the implicit one_file_system approach, along with documentating the hell out of this behavior. I’ll leave this ticket open til that’s done. Thanks again for the bug report.

Would it be a big task to make the DB-backup behaviour switchable, turning backup via pipe on/off? I know streaming database backups was a wish from #258, but if space is not such an issue, the “usual” behaviour without the read-special side effects is IMHO sufficient. Using this as default and a big warning regarding piped database backups and it’s side-effects as special devices aren’t excluded any more (if not one_file_system is set) would be nice.

Would it be a big task to make the DB-backup behaviour switchable, turning backup via pipe on/off?

I think that’s certainly a backstop if there are too many more issues with the auto-streaming behavior. But I was hoping that I could get away with just one supported database dump/restore approach, and a single code path for it. (Although I’ve already had to stray from that somewhat to continue supporting PostgreSQL’s directory dump format, which by its nature can’t stream.)

Okay, the “fix” of forcing one_file_system is released with borgmatic 1.5.5!

Strange issue now with the storage dir of Knot DNS
It contains some certs and LMDB files and borg seems to get a timeout. Are LMDB files considered special or is it another issue?

I’m not familiar with LMDB, but here’s something that might be relevant from http://www.lmdb.tech/doc/:

Do not use LMDB databases on remote filesystems, even between processes on the same host. This breaks flock() on some OSes, possibly memory map sync, and certainly sync between programs on different hosts.

So the way I’m reading that is that one shouldn’t access an LMDB database from multiple different processes. Although that’s contradicted by the Wikipedia entry. In any case, borgmatic is just reading from the filesystem and not “connecting” to LMDB of course.

Another idea is that LMDB has some named pipes or other devices on its filesystem that borgmatic can’t read without hanging.

What’s the directory listing look like where the problem is occurring?

When you say that Borg gets a timeout, what’s that look like? Is it an actual remote timeout error?

If I remember correctly, the output tells about the last included file? So lock.mdb seems to be included:

SSH command line: ['ssh', '12345@ch-s012.rsync.net', 'borg1', 'serve', '--umask=077', '--debug']
Remote: using builtin fallback logging configuration
Remote: using builtin fallback logging configuration
Remote: Initialized logging system for JSON-based protocol
Remote: Resolving repository path b'mail'
Remote: Resolved repository path to '/data2/home/12345/mail'
Remote: Verified integrity of /data2/home/12345/mail/index.1650
TAM-verified manifest
security: read previous location 'ssh://12345@ch-s012.rsync.net/./mail'
security: read manifest timestamp '2020-06-19T07:26:46.328936'
security: determined newest manifest timestamp as 2020-06-19T07:26:46.328936
security: repository checks ok, allowing access
Creating archive at "12345@ch-s012.rsync.net:mail::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}"
Verified integrity of /root/.cache/borg/806c460e59a4dcc340f6a3ab2e76d2102ab899f468189d77dbb17ec4bf5cfb00/chunks
Reading files cache ...
Verified integrity of /root/.cache/borg/806c460e59a4dcc340f6a3ab2e76d2102ab899f468189d77dbb17ec4bf5cfb00/files
security: read previous location 'ssh://12345@ch-s012.rsync.net/./mail'
security: read manifest timestamp '2020-06-19T07:26:46.328936'
security: determined newest manifest timestamp as 2020-06-19T07:26:46.328936
security: repository checks ok, allowing access
Processing files ...
Remote: Cleaned up 0 uncommitted segment files (== everything after segment 1650).0876dd46af6f9efc6d4a20b9                                                                                                         
Remote: Verified integrity of /data2/home/12345/mail/hints.1650
35.28 GB O 33.14 GB C 22.56 MB D 27622 N var/lib/knot/journal/lock.mdb  

But it seems borg went further after some time, so it’s no real lockup.

So is there still a problem here? How big is that lock file?

It might be the case that the preferred way of backing up an LMDB database is by exporting/dumping it to file, just like with PostgreSQL and MySQL databases. Here’s a command I found: http://www.lmdb.tech/doc/man1/mdb_dump_1.html

It’s possible that you could exclude the entire LMDB database directory, and then back it up via that dump command instead.