Excluded patterns are not printed in output of create --dry-run --list #590

Closed
opened 2022-10-02 17:38:19 +00:00 by rcdailey · 4 comments

What I'm trying to do and why

I want to verify that I have set up my patterns file correctly. I want to verify exclusion/inclusion patterns as they relate to my root backup directories.

Steps to reproduce (if a bug)

  1. Create a patterns file and fill it with the contents of patterns.lst below.
  2. Use the specified config.yaml below.
  3. Run borgmatic -c config.yaml create -v -1 --dry-run --list --log-file dryrun.txt --log-file-verbosity 2

patterns.lst:

# Patterns file is included before the Exclusions file
# Inclusions (+) are first so they override exclusions (-)

R /etc

- /etc/*

config.yaml:

location:
  patterns_from:
    - /path/to/patterns.lst
  repositories:
    - ssh://user@myrepo.domain.com/./repo
  remote_path: /usr/local/bin/borg
  exclude_caches: true
  exclude_if_present:
    - .nobackup

retention:
  keep_daily: 3
  keep_weekly: 4
  keep_monthly: 12
  keep_yearly: 2

consistency:
  checks:
    - repository
    - archives

storage:
  encryption_passphrase: "thepassphrase"
  ssh_command: ssh -oStrictHostKeyChecking=no -F /home/robert/.ssh/config
  relocated_repo_access_is_ok: true
  umask: 0002
  compression: zstd

Actual behavior (if a bug)

Observe that the dryrun.txt file does not contain a list of excluded paths, which should appear in the format: x /etc/crontab (with an x in front).

My assertion on the format of the list output is based on this comment.

Expected behavior (if a bug)

The output of borgmatic create --dry-run --list should contain paths starting with x if they were excluded by patterns.

Environment

borgmatic version: 1.7.2

borgmatic installation method: pip install borgmatic

Borg version: 1.2.2

Python version: 3.8.6 (using pyenv)

operating system and version: Ubuntu 20.04

#### What I'm trying to do and why I want to verify that I have set up my patterns file correctly. I want to verify exclusion/inclusion patterns as they relate to my root backup directories. #### Steps to reproduce (if a bug) 1. Create a patterns file and fill it with the contents of `patterns.lst` below. 1. Use the specified `config.yaml` below. 1. Run `borgmatic -c config.yaml create -v -1 --dry-run --list --log-file dryrun.txt --log-file-verbosity 2` `patterns.lst`: ```patch # Patterns file is included before the Exclusions file # Inclusions (+) are first so they override exclusions (-) R /etc - /etc/* ``` `config.yaml`: ```yml location: patterns_from: - /path/to/patterns.lst repositories: - ssh://user@myrepo.domain.com/./repo remote_path: /usr/local/bin/borg exclude_caches: true exclude_if_present: - .nobackup retention: keep_daily: 3 keep_weekly: 4 keep_monthly: 12 keep_yearly: 2 consistency: checks: - repository - archives storage: encryption_passphrase: "thepassphrase" ssh_command: ssh -oStrictHostKeyChecking=no -F /home/robert/.ssh/config relocated_repo_access_is_ok: true umask: 0002 compression: zstd ``` #### Actual behavior (if a bug) Observe that the `dryrun.txt` file does not contain a list of excluded paths, which should appear in the format: `x /etc/crontab` (with an `x` in front). My assertion on the format of the list output is based on [this comment](https://github.com/borgbackup/borg/issues/5834#issuecomment-861660538). #### Expected behavior (if a bug) The output of `borgmatic create --dry-run --list` should contain paths starting with `x` if they were excluded by patterns. #### Environment **borgmatic version:** `1.7.2` **borgmatic installation method:** `pip install borgmatic` **Borg version:** `1.2.2` **Python version:** `3.8.6` (using pyenv) **operating system and version:** Ubuntu 20.04
Owner

Thanks for filing this. I think there are at least two things going on here:

  1. borgmatic explicitly asks Borg to filter down the displayed files list so as to omit excluded files (among other file types). It would be an easy fix to change that so it no longer omits excluded files.
  2. Borg has a bug in which using patterns along with command-line paths causes the patterns to be ignored. It looks like borgmatic is hitting this Borg bug here because, even though you don't have any explicit source_directories, borgmatic passes its own ~/.borgmatic source directory to Borg. A variant of this unfortunate interaction was fixed previously (#574), but it sounds like you've discovered another! So this will need more investigation—and probably another work-around for the Borg bug.
Thanks for filing this. I think there are at least two things going on here: 1. borgmatic explicitly asks Borg to filter down the displayed files list so as to omit excluded files (among other file types). It would be an easy fix to change that so it no longer omits excluded files. 2. Borg has a [bug in which using patterns along with command-line paths causes the patterns to be ignored](https://github.com/borgbackup/borg/issues/6994). It looks like borgmatic is hitting this Borg bug here because, even though you don't have any explicit `source_directories`, borgmatic passes its own `~/.borgmatic` source directory to Borg. A variant of this unfortunate interaction was fixed previously (#574), but it sounds like you've discovered another! So this will need more investigation—and probably another work-around for the Borg bug.
witten added the
bug
data loss
labels 2022-10-04 23:16:45 +00:00
Owner

Both of these issues are now fixed in master and should be part of the next release. For the second issue, I ended up using a fix very similar to the one in #574 (put source directories into patterns to work around the Borg bug). I've confirmed that excluded files now show up correctly in the file list.

Both of these issues are now fixed in master and should be part of the next release. For the second issue, I ended up using a fix very similar to the one in #574 (put source directories into patterns to work around the Borg bug). I've confirmed that excluded files now show up correctly in the file list.
Author

Thank you for the quick turnaround!!

Thank you for the quick turnaround!!
Owner

Just released in borgmatic 1.7.3!

Just released in borgmatic 1.7.3!
Sign in to join this conversation.
No Milestone
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#590
No description provided.