Unexpected filesystem-crossing using glob #520

New Issue

rgaudin · 2022-04-12T11:35:10Z

rgaudin commented

2022-04-12 11:35:10 +00:00

What I'm trying to do and why

I want to backup both a number of directories and a database at the same time.

I am doing this via a docker container, so my files are mounted, on different paths (subpath of my source_directories).

Steps to reproduce (if a bug)

mount | grep /storage
grpcfuse on /storage/config type fuse.grpcfuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=1048576)
grpcfuse on /storage/downloads type fuse.grpcfuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=1048576)
grpcfuse on /storage/images type fuse.grpcfuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=1048576)

The key here is the * in source_directories

location:
    source_directories:
        - /storage/*
    repositories:
        - user@server:repo
storage:
    encryption_passphrase: ""
    relocated_repo_access_is_ok: true
    unknown_unencrypted_repo_access_is_ok: true
    borg_base_directory: "/repo"
    borg_cache_directory: "/cache"
    archive_name_format: 'doesn-matter__backup__{now}'
retention:
    keep_daily: 7
    keep_weekly: 5
    keep_monthly: 12
    keep_yearly: 1
    prefix: doesn-matter__backup__
hooks:
    mysql_databases:
        - name: mysqluser
          username : mysqluser
          password : mysqluser
          hostname : mysqlhost
          port : 3306
          options : "--single-transaction --quick --lock-tables=false --max-allowed-packet=128M"

Actual behavior (if a bug)

Files from all the mount points are added to the repo

security: repository checks ok, allowing access
Processing files ...
A /root/.borgmatic/mysql_databases/192.168.5.80/db
M /storage/config/settings.conf
M /storage/downloads/some-file.zip
M /storage/downloads/some-file2.zip
M /storage/images/a/aaaaa.png
M /storage/images/b/bbbbb.png
A /storage/images/.hidden/img-message
Remote: Cleaned up 0 uncommitted segment files (== everything after segment 126).

Expected behavior (if a bug)

I should not be able to backup files from those sub directories as those are on different filesystems, given what the doc says:

    # Stay in same file system (do not cross mount points).
    # Defaults to false. But when a database hook is used, the
    # setting here is ignored and one_file_system is considered
    # true.
    # one_file_system: true

This quirk (using a * glob) currently allows me to do what I intend to but I am afraid this behavior might get fixed in the future and that I would silently stop backuping my files.

Also, using * implies leaving any empty files in that folder aside.

Environment

borgmatic version: 1.5.24

borgmatic installation method: pip

Borg version: 1.1.16

Python version: 3.9.2

Database version (if applicable): 10.4

operating system and version: debian:bulleyes-slim docker image

#### What I'm trying to do and why I want to backup both a number of directories and a database at the same time. I am doing this via a docker container, so my files are mounted, on different paths (subpath of my source_directories). #### Steps to reproduce (if a bug) ``` mount | grep /storage grpcfuse on /storage/config type fuse.grpcfuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=1048576) grpcfuse on /storage/downloads type fuse.grpcfuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=1048576) grpcfuse on /storage/images type fuse.grpcfuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=1048576) ``` The key here is the **`*`** in `source_directories` ``` location: source_directories: - /storage/* repositories: - user@server:repo storage: encryption_passphrase: "" relocated_repo_access_is_ok: true unknown_unencrypted_repo_access_is_ok: true borg_base_directory: "/repo" borg_cache_directory: "/cache" archive_name_format: 'doesn-matter__backup__{now}' retention: keep_daily: 7 keep_weekly: 5 keep_monthly: 12 keep_yearly: 1 prefix: doesn-matter__backup__ hooks: mysql_databases: - name: mysqluser username : mysqluser password : mysqluser hostname : mysqlhost port : 3306 options : "--single-transaction --quick --lock-tables=false --max-allowed-packet=128M" ``` #### Actual behavior (if a bug) Files from all the mount points are added to the repo ``` security: repository checks ok, allowing access Processing files ... A /root/.borgmatic/mysql_databases/192.168.5.80/db M /storage/config/settings.conf M /storage/downloads/some-file.zip M /storage/downloads/some-file2.zip M /storage/images/a/aaaaa.png M /storage/images/b/bbbbb.png A /storage/images/.hidden/img-message Remote: Cleaned up 0 uncommitted segment files (== everything after segment 126). ``` #### Expected behavior (if a bug) I should not be able to backup files from those sub directories as those are on different filesystems, given what [the doc](https://torsion.org/borgmatic/docs/reference/configuration/) says: ``` # Stay in same file system (do not cross mount points). # Defaults to false. But when a database hook is used, the # setting here is ignored and one_file_system is considered # true. # one_file_system: true ``` This quirk (using a `*` glob) currently allows me to do what I intend to but I am afraid this behavior might get fixed in the future and that I would silently stop backuping my files. Also, using `*` implies leaving any empty files in that folder aside. #### Environment **borgmatic version:** 1.5.24 **borgmatic installation method:** pip **Borg version:** 1.1.16 **Python version:** 3.9.2 **Database version (if applicable):** 10.4 **operating system and version:** debian:bulleyes-slim docker image

witten commented

2022-04-12 16:53:46 +00:00

Thanks for filing this! I think this is a "works by design" thing that could perhaps be better documented. I don't see the one_file_system flag set to true in your configuration, but even if it was, it probably wouldn't prevent all of your source directories on different filesystems from being backed up. That's because the glob is expanded by borgmatic, and then your three source paths are passed to Borg individually.

To confirm this behavior, you could set one_file_system to true and then observe whether all your paths get backed up.

What one_file_system is designed to do, AFAIK, is to prevent Borg from implicitly including other off-filesystem paths it reaches within those source directories. E.g., if it found /storage/images/external which was a mount point of another filesystem, it'd get excluded.

Thanks for filing this! I think this is a "works by design" thing that could perhaps be better documented. I don't see the `one_file_system` flag set to `true` in your configuration, but even if it was, it probably wouldn't prevent all of your source directories on different filesystems from being backed up. That's because the glob is expanded by borgmatic, and then your three source paths are passed to Borg individually. To confirm this behavior, you could set `one_file_system` to `true` and then observe whether all your paths get backed up. What `one_file_system` is designed to do, AFAIK, is to prevent Borg from implicitly including _other_ off-filesystem paths it reaches within those source directories. E.g., if it found `/storage/images/external` which was a mount point of another filesystem, it'd get excluded.

witten added the

question / support

label 2022-04-12 16:53:54 +00:00

witten referenced this issue from a commit

2022-04-12 18:05:59 +00:00

Clarify one_file_system behavior in schema comment (#520).

witten commented

2022-04-28 17:47:54 +00:00

I've clarified the one_file_system behavior in the schema documentation. I'll close this ticket for now, but please feel free to re-open if you have any other comments or questions related to this behavior.

I've clarified the `one_file_system` behavior in the schema documentation. I'll close this ticket for now, but please feel free to re-open if you have any other comments or questions related to this behavior.

witten closed this issue

2022-04-28 17:47:55 +00:00

rgaudin commented

2022-04-29 09:43:34 +00:00

Thanks ; sorry for not replying earlier. The fact that one_file_system is borg-related and the source_directories is built in borgmatic wasn't clear to me.

👍 1

Sign in to join this conversation.