Errno 2 for symlinks of nspawn containers #596

What I'm trying to do and why

Steps to reproduce (if a bug)

Create nspawn container and backup from underlying host
Symlinks for "/run/lock", "/etc/mtab", etc are to tmpfs folders which aren't visible to the host running the containers.

Alternatively you could create a broken symlink manually

Actual behavior (if a bug)

ssh:/XXX/./backups: Collecting special file paths
borg create --exclude-from /tmp/tmp1_az7fgy --compression zstd --one-file-system --read-special --remote-path /XXX/bin/borg ssh:/XXX/./backups::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /var/lib/machines/XXX --dry-run --list
ssh://XXX/./backups: Error running actions for repository
[Errno 2] No such file or directory: '/var/lib/machines/XXX/var/lock'

After createing the folder /var/lib/machines/XXX/run/lock on host i got the next issue with mtab

ssh://XXX/./backups: Error running actions for repository
[Errno 2] No such file or directory: '/var/lib/machines/XXX/etc/mtab'

Expected behavior (if a bug)

Same as with Version 1.7.1 - just backup the symlink like before the last update ;)

Other notes / implementation ideas

Environment

borgmatic version: 1.7.2

borgmatic installation method: pacman

Borg version: 12.2.2

Python version: 3.10.8

operating system and version: Arch Linux

#### What I'm trying to do and why #### Steps to reproduce (if a bug) Create nspawn container and backup from underlying host Symlinks for "/run/lock", "/etc/mtab", etc are to tmpfs folders which aren't visible to the host running the containers. Alternatively you could create a broken symlink manually #### Actual behavior (if a bug) ssh:/XXX/./backups: Collecting special file paths borg create --exclude-from /tmp/tmp1_az7fgy --compression zstd --one-file-system --read-special --remote-path /XXX/bin/borg ssh:/XXX/./backups::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /var/lib/machines/XXX --dry-run --list ssh://XXX/./backups: Error running actions for repository [Errno 2] No such file or directory: '/var/lib/machines/XXX/var/lock' After createing the folder /var/lib/machines/XXX/run/lock on host i got the next issue with mtab ssh://XXX/./backups: Error running actions for repository [Errno 2] No such file or directory: '/var/lib/machines/XXX/etc/mtab' #### Expected behavior (if a bug) Same as with Version 1.7.1 - just backup the symlink like before the last update ;) #### Other notes / implementation ideas #### Environment **borgmatic version:** 1.7.2 **borgmatic installation method:** pacman **Borg version:** 12.2.2 **Python version:** 3.10.8 **operating system and version:** Arch Linux

Thank you for taking the time to report this! I just confirmed this behavior with a manually created broken symlink.

Okay, this is fixed in master by treating broken symlinks as non-special files. The fix will be part of the next release (hopefully soon). Thanks again!

Fix released in borgmatic 1.7.4. Please give it a shot and let me know if it works for you. Thanks!

Looks pretty good - "pretty" because the file exclusion triggered for a masked systemd service file in my nspawn container

ssh://XXX/./backups: Excluding special files to prevent Borg from hanging: /var/lib/machines/XXX/etc/systemd/system/dnsmasq.service

ls -ln /var/lib/machines/XXX/etc/systemd/system/dnsmasq.service
lrwxrwxrwx 1 1398865920 1398865920 9 Apr 11 2022 /var/lib/machines/XXX/etc/systemd/system/dnsmasq.service -> /dev/null

Looks pretty good - "pretty" because the file exclusion triggered for a masked systemd service file in my nspawn container ssh://XXX/./backups: Excluding special files to prevent Borg from hanging: /var/lib/machines/XXX/etc/systemd/system/dnsmasq.service ls -ln /var/lib/machines/XXX/etc/systemd/system/dnsmasq.service lrwxrwxrwx 1 1398865920 1398865920 9 Apr 11 2022 /var/lib/machines/XXX/etc/systemd/system/dnsmasq.service -> /dev/null

Yeah, I think that's due to the symlink's destination—/dev/null, a character device.. which would cause Borg to hang if not excluded. Is it a problem for you that it does get excluded? For instance, did you want/expect the symlink itself to get backed up?

Yeah, I think that's due to the symlink's destination—`/dev/null`, a character device.. which would cause Borg to hang if not excluded. Is it a problem for you that it does get excluded? For instance, did you want/expect the symlink itself to get backed up?

For everything in /etc/systemd/system/ i'd expect the symlink to be backed up.
Same for the /var/lib/machines/XXX/etc/systemd/system/ because that's the default location for systmed-nspawn containers.

For me personally it wouldn't amount to too much of an issue because i use nagios and would be notified when bind could not start because dnsmasq is already using the port. BUT it would lead to issues when i restore this container until I get the nagios notification ;)

Theoretically the same could happen for others who store their containers somewhere else. So maybe regex for everything with /etc/systemd/system - with and without prefix path - to backup symlinks instead of the destination?

For everything in /etc/systemd/system/ i'd expect the symlink to be backed up. Same for the /var/lib/machines/XXX/etc/systemd/system/ because that's the default location for systmed-nspawn containers. For me personally it wouldn't amount to too much of an issue because i use nagios and would be notified when bind could not start because dnsmasq is already using the port. BUT it would lead to issues when i restore this container until I get the nagios notification ;) Theoretically the same could happen for others who store their containers somewhere else. So maybe regex for everything with /etc/systemd/system - with and without prefix path - to backup symlinks instead of the destination?

Hmm.. I'm not sure of a way to do this! When --read-special is used with Borg (which borgmatic implictly enables when database hooks are configured), Borg follows all symlinks that point to special files. Meaning it tries to read those special files and will hang on them unless they're excluded. The only way I can think of to back up those symlinks instead of their destinations would be to avoid using --read-special. Maybe I should just document this as another limitation of the database hooks?

Hmm.. I'm not sure of a way to do this! When `--read-special` is used with Borg (which borgmatic implictly enables when database hooks are configured), Borg *follows* all symlinks that point to special files. Meaning it tries to read those special files and will hang on them unless they're excluded. The only way I can think of to back up those symlinks instead of their destinations would be to avoid using `--read-special`. Maybe I should just document this as another limitation of the database hooks?

That would be one possible solution.

I've got another idea but please take it as something from someone without knowledge of borg itself.

Would it be possible to append to an existing archive?
If yes, it could be possible to separate the filesystem and db part in their own backup processes and just write them in the same archive

That would be one possible solution. I've got another idea but please take it as something from someone without knowledge of borg itself. Would it be possible to append to an existing archive? If yes, it could be possible to separate the filesystem and db part in their own backup processes and just write them in the same archive

Would it be possible to append to an existing archive? If yes, it could be possible to separate the filesystem and db part in their own backup processes and just write them in the same archive

If only Borg supported that! With Borg's current feature set, you'd have to make two separate archives, one for the databases and one for everything else. You could achieve that today with two separate borgmatic configuration files.

> Would it be possible to append to an existing archive? If yes, it could be possible to separate the filesystem and db part in their own backup processes and just write them in the same archive If only Borg supported that! With Borg's current feature set, you'd have to make two separate archives, one for the databases and one for everything else. You could achieve that today with two separate borgmatic configuration files.

Labels Milestones