Excluded patterns are not printed in output of create --dry-run --list #590

New issue

Closed

opened 2022-10-02 17:38:19 +00:00 by rcdailey · 4 comments

rcdailey commented 2022-10-02 17:38:19 +00:00

Copy link

What I'm trying to do and why

I want to verify that I have set up my patterns file correctly. I want to verify exclusion/inclusion patterns as they relate to my root backup directories.

Steps to reproduce (if a bug)

1. Create a patterns file and fill it with the contents of patterns.lst below.
2. Use the specified config.yaml below.
3. Run borgmatic -c config.yaml create -v -1 --dry-run --list --log-file dryrun.txt --log-file-verbosity 2

patterns.lst:

# Patterns file is included before the Exclusions file
# Inclusions (+) are first so they override exclusions (-)

R /etc

- /etc/*

config.yaml:

location:
  patterns_from:
    - /path/to/patterns.lst
  repositories:
    - ssh://user@myrepo.domain.com/./repo
  remote_path: /usr/local/bin/borg
  exclude_caches: true
  exclude_if_present:
    - .nobackup

retention:
  keep_daily: 3
  keep_weekly: 4
  keep_monthly: 12
  keep_yearly: 2

consistency:
  checks:
    - repository
    - archives

storage:
  encryption_passphrase: "thepassphrase"
  ssh_command: ssh -oStrictHostKeyChecking=no -F /home/robert/.ssh/config
  relocated_repo_access_is_ok: true
  umask: 0002
  compression: zstd

Actual behavior (if a bug)

Observe that the dryrun.txt file does not contain a list of excluded paths, which should appear in the format: x /etc/crontab (with an x in front).

My assertion on the format of the list output is based on this comment.

Expected behavior (if a bug)

The output of borgmatic create --dry-run --list should contain paths starting with x if they were excluded by patterns.

Environment

borgmatic version: 1.7.2

borgmatic installation method: pip install borgmatic

Borg version: 1.2.2

Python version: 3.8.6 (using pyenv)

operating system and version: Ubuntu 20.04

#### What I'm trying to do and why I want to verify that I have set up my patterns file correctly. I want to verify exclusion/inclusion patterns as they relate to my root backup directories. #### Steps to reproduce (if a bug) 1. Create a patterns file and fill it with the contents of `patterns.lst` below. 1. Use the specified `config.yaml` below. 1. Run `borgmatic -c config.yaml create -v -1 --dry-run --list --log-file dryrun.txt --log-file-verbosity 2` `patterns.lst`: ```patch # Patterns file is included before the Exclusions file # Inclusions (+) are first so they override exclusions (-) R /etc - /etc/* ``` `config.yaml`: ```yml location: patterns_from: - /path/to/patterns.lst repositories: - ssh://user@myrepo.domain.com/./repo remote_path: /usr/local/bin/borg exclude_caches: true exclude_if_present: - .nobackup retention: keep_daily: 3 keep_weekly: 4 keep_monthly: 12 keep_yearly: 2 consistency: checks: - repository - archives storage: encryption_passphrase: "thepassphrase" ssh_command: ssh -oStrictHostKeyChecking=no -F /home/robert/.ssh/config relocated_repo_access_is_ok: true umask: 0002 compression: zstd ``` #### Actual behavior (if a bug) Observe that the `dryrun.txt` file does not contain a list of excluded paths, which should appear in the format: `x /etc/crontab` (with an `x` in front). My assertion on the format of the list output is based on [this comment](https://github.com/borgbackup/borg/issues/5834#issuecomment-861660538). #### Expected behavior (if a bug) The output of `borgmatic create --dry-run --list` should contain paths starting with `x` if they were excluded by patterns. #### Environment **borgmatic version:** `1.7.2` **borgmatic installation method:** `pip install borgmatic` **Borg version:** `1.2.2` **Python version:** `3.8.6` (using pyenv) **operating system and version:** Ubuntu 20.04

witten commented 2022-10-04 23:16:23 +00:00

Owner

Copy link

Thanks for filing this. I think there are at least two things going on here:

1. borgmatic explicitly asks Borg to filter down the displayed files list so as to omit excluded files (among other file types). It would be an easy fix to change that so it no longer omits excluded files.
2. Borg has a bug in which using patterns along with command-line paths causes the patterns to be ignored. It looks like borgmatic is hitting this Borg bug here because, even though you don't have any explicit source_directories, borgmatic passes its own ~/.borgmatic source directory to Borg. A variant of this unfortunate interaction was fixed previously (#574), but it sounds like you've discovered another! So this will need more investigation—and probably another work-around for the Borg bug.

Thanks for filing this. I think there are at least two things going on here: 1. borgmatic explicitly asks Borg to filter down the displayed files list so as to omit excluded files (among other file types). It would be an easy fix to change that so it no longer omits excluded files. 2. Borg has a [bug in which using patterns along with command-line paths causes the patterns to be ignored](https://github.com/borgbackup/borg/issues/6994). It looks like borgmatic is hitting this Borg bug here because, even though you don't have any explicit `source_directories`, borgmatic passes its own `~/.borgmatic` source directory to Borg. A variant of this unfortunate interaction was fixed previously (#574), but it sounds like you've discovered another! So this will need more investigation—and probably another work-around for the Borg bug.

witten added the

bug

data loss

labels 2022-10-04 23:16:42 +00:00

witten referenced this issue from a commit 2022-10-05 05:59:33 +00:00

Fix for potential data loss with "patterns_from". Also, display excluded files (#590).

witten commented 2022-10-05 06:01:17 +00:00

Owner

Copy link

Both of these issues are now fixed in master and should be part of the next release. For the second issue, I ended up using a fix very similar to the one in #574 (put source directories into patterns to work around the Borg bug). I've confirmed that excluded files now show up correctly in the file list.

Both of these issues are now fixed in master and should be part of the next release. For the second issue, I ended up using a fix very similar to the one in #574 (put source directories into patterns to work around the Borg bug). I've confirmed that excluded files now show up correctly in the file list.

witten closed this issue 2022-10-05 06:01:18 +00:00

rcdailey commented 2022-10-05 13:42:32 +00:00

Author

Copy link

Thank you for the quick turnaround!!

Thank you for the quick turnaround!!

👍 1

witten commented 2022-10-12 18:02:27 +00:00

Owner

Copy link

Just released in borgmatic 1.7.3!

Just released in borgmatic 1.7.3!

witten referenced this issue 2022-12-18 20:19:58 +00:00

Disable logging of 'exclude' files #620

lacrimal referenced this issue 2022-12-18 20:41:04 +00:00

Disable logging of 'exclude' files #620

witten referenced this issue 2023-10-24 17:48:09 +00:00

Borgmatic reports success but logs show stacktrace and no archive is created. #776

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

parallel-borgmatic

command-hook-steps

2.1.7

2.1.6

2.1.5

2.1.4

2.1.3

2.1.2

2.1.1

2.1.0

2.0.13

2.0.12

2.0.11

2.0.10

2.0.9

2.0.8

2.0.7

2.0.6

2.0.5

2.0.4

2.0.3

2.0.2

2.0.1

2.0.0

1.9.14

1.9.13

1.9.12

1.9.11

1.9.10

1.9.9

1.9.8

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.14

1.8.13

1.8.12

1.8.11

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.24

1.5.23

1.5.22

1.5.21

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.14

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.7.dev0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.22

1.4.21

1.4.20

1.4.19

1.4.18

1.4.17

1.4.16

1.4.15

1.4.14

1.4.13

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.26

1.3.25

1.3.24

1.3.23

1.3.22

1.3.21

1.3.20

1.3.19

1.3.18

1.3.17

1.3.16

1.3.15

1.3.14

1.3.13

1.3.12

1.3.11

1.3.10

1.3.9

1.3.8

1.3.7

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.18

1.2.17

1.2.16

1.2.15

1.2.14

1.2.13

1.2.12

1.2.11

1.2.10

1.2.9

1.2.8

1.2.7

1.2.6

1.2.5

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.15

1.1.14

1.1.13

1.1.12

1.1.11

1.1.10

1.1.9

1.1.7

1.1.6

1.1.5

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.3

1.0.2

1.0.1

1.0.0

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

0.0.2

Labels

Clear labels   

blocked

  

breaking

  

bug

  

data loss

  

design finalized

  

good first issue

  

new feature area

  

question / support

  

security

  

waiting for response

No labels

blocked

breaking

bug

data loss

design finalized

good first issue

new feature area

question / support

security

waiting for response

Milestone

Clear milestone

No items

No milestone

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

borgmatic-collective/borgmatic#590

No description provided.