Need help with exclude_patterns #453

New Issue

Closed

opened 2021-09-15 09:26:39 +00:00 by auanasgheps · 4 comments

auanasgheps commented 2021-09-15 09:26:39 +00:00

Copy Link

What I'm trying to do and why

I want to set up a filter to exclude some files, like WhatsApp DBs that have dates in the filename, like
msgstore-2021-09-15.1.db.crypt14.

The filter I've come up with is msgstore-20*.* which works if I search for files in WinSCP, but does not work in borgmatic/borg because these files are still included in my backups.

I don't want to exclude all crypt14 files because I want to backup another WhatsApp DB which has the same extension but different file name.

I'm sure I am making a very dumb mistake because a simple filter works, like if I exclude a full path (e.g. a temp folder).

I think this issue also applies to the other files I'm trying to exclude when using asterisks.

Steps to reproduce (if a bug)

Here's the relevant configuration. The first two folder patterns work, the others not really.

exclude_patterns:
    - /srv/dev-disk-by-label-nvme/docker-install
    - /srv/dev-disk-by-label-nvme/docker-apps
    - '*/lost+found'
    - 'thumbs.db'
    - 'Thumbs.db'
    - '$RECYCLE.BIN'
    - 'System Volume Information'
    - 'msgstore-20*.*'
    - 'Folder.jpg'
    - 'AlbumArt*.jpg'

Actual behavior (if a bug)

Expected behavior (if a bug)

Other notes / implementation ideas

I'm not sure what I'm doing wrong, but there are no very good examples to do this kind of exclusions, neither in Borgbackup docs.

Environment

borgmatic version: 1.5.12

borgmatic installation method: Debian Package

Borg version: 1.1.15

Python version: Python 3.7.3

Database version (if applicable): N/A

operating system and version: Debian 10 (OpenMediaVault 5.x)

#### What I'm trying to do and why I want to set up a filter to exclude some files, like WhatsApp DBs that have dates in the filename, like msgstore-2021-09-15.1.db.crypt14. The filter I've come up with is `msgstore-20*.*` which works if I search for files in WinSCP, but does not work in borgmatic/borg because these files are still included in my backups. I don't want to exclude all crypt14 files because I want to backup another WhatsApp DB which has the same extension but different file name. I'm sure I am making a very dumb mistake because a simple filter works, like if I exclude a full path (e.g. a temp folder). I think this issue also applies to the other files I'm trying to exclude when using asterisks.  #### Steps to reproduce (if a bug) Here's the relevant configuration. The first two folder patterns work, the others not really. ``` exclude_patterns: - /srv/dev-disk-by-label-nvme/docker-install - /srv/dev-disk-by-label-nvme/docker-apps - '*/lost+found' - 'thumbs.db' - 'Thumbs.db' - '$RECYCLE.BIN' - 'System Volume Information' - 'msgstore-20*.*' - 'Folder.jpg' - 'AlbumArt*.jpg' ``` #### Actual behavior (if a bug) #### Expected behavior (if a bug) #### Other notes / implementation ideas I'm not sure what I'm doing wrong, but there are no very good examples to do this kind of exclusions, neither in Borgbackup docs. #### Environment **borgmatic version:** 1.5.12 **borgmatic installation method:** Debian Package **Borg version:** 1.1.15 **Python version:** Python 3.7.3 **Database version (if applicable):** N/A **operating system and version:** Debian 10 (OpenMediaVault 5.x)

borgmatic.jpg

135 KiB

witten commented 2021-09-15 16:20:37 +00:00

Owner

Copy Link

Thanks for taking the time to file this! (Sorry nobody got to your question on IRC.) Looking at the Borg docs on exclude patterns, there's one sentence in particular that stands out to me as potentially relevant: "For a path to match a pattern, the full path must match, or it must match from the start of the full path to just before a path separator." I think what this means is that if your pattern is an absolute path or starts with a glob, then it'll match a full path properly. On the other hand, if your pattern does not begin with an absolute path and does not start with a glob, then it can't match a full path. If that's the case, then, msgstore-20*.* wouldn't match anything.

So how to fix this? Couple of options:

• */msgstore-20*.*
• sh:**/msgstore-20*.* (untested)
• re:^.*/msgstore-20.*\..*$ (untested)

Give something along those lines a shot and let me know how they work for you!

Thanks for taking the time to file this! (Sorry nobody got to your question on IRC.) Looking at the [Borg docs on exclude patterns](https://borgbackup.readthedocs.io/en/stable/usage/help.html), there's one sentence in particular that stands out to me as potentially relevant: "For a path to match a pattern, the full path must match, or it must match from the start of the full path to just before a path separator." I think what this means is that if your pattern is an absolute path or starts with a glob, then it'll match a full path properly. On the other hand, if your pattern does not begin with an absolute path and does not start with a glob, then it can't match a full path. If that's the case, then, `msgstore-20*.*` wouldn't match anything. So how to fix this? Couple of options: * `*/msgstore-20*.*` * `sh:**/msgstore-20*.*` (untested) * `re:^.*/msgstore-20.*\..*$` (untested) Give something along those lines a shot and let me know how they work for you!

witten added the

question / support

label 2021-09-15 16:22:06 +00:00

auanasgheps commented 2021-09-16 13:20:55 +00:00

Author

Copy Link

Sorry nobody got to your question on IRC.

Thanks for letting me know but I would have never seen a reply: I signed in as guest and when I refreshed the page my m messages were cleared...

Thanks for the patterns! I will try the first one. On saturday my backup will run and I'll be able to report back. I will also change all the others accordingly.

> Sorry nobody got to your question on IRC. Thanks for letting me know but I would have never seen a reply: I signed in as guest and when I refreshed the page my m messages were cleared... Thanks for the patterns! I will try the first one. On saturday my backup will run and I'll be able to report back. I will also change all the others accordingly.

👍 1

auanasgheps commented 2021-09-18 18:24:49 +00:00

Author

Copy Link

I'm happy to confirm that */msgstore-20*.* made the trick! Thank a lot!

I'm happy to confirm that `*/msgstore-20*.*` made the trick! Thank a lot!

auanasgheps closed this issue 2021-09-18 18:24:49 +00:00

witten commented 2021-09-18 22:43:15 +00:00

Owner

Copy Link

Awesome, so glad to hear it! I've updated the examples in the configuration file to include a pattern like this one.

Awesome, so glad to hear it! I've updated the examples in the configuration file to include a pattern like this one.

👍 1

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

spot-check

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.24

1.5.23

1.5.22

1.5.21

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.14

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.7.dev0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.22

1.4.21

1.4.20

1.4.19

1.4.18

1.4.17

1.4.16

1.4.15

1.4.14

1.4.13

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.26

1.3.25

1.3.24

1.3.23

1.3.22

1.3.21

1.3.20

1.3.19

1.3.18

1.3.17

1.3.16

1.3.15

1.3.14

1.3.13

1.3.12

1.3.11

1.3.10

1.3.9

1.3.8

1.3.7

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.18

1.2.17

1.2.16

1.2.15

1.2.14

1.2.13

1.2.12

1.2.11

1.2.10

1.2.9

1.2.8

1.2.7

1.2.6

1.2.5

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.15

1.1.14

1.1.13

1.1.12

1.1.11

1.1.10

1.1.9

1.1.7

1.1.6

1.1.5

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.3

1.0.2

1.0.1

1.0.0

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

0.0.2

Labels

Clear labels   

bug

  

data loss

  

design finalized

  

good first issue

  

new feature area

  

question / support

  

security

  

waiting for response

No Label

bug

data loss

design finalized

good first issue

new feature area

question / support

security

waiting for response

Milestone

Clear milestone

No items

No Milestone

Assignees

Clear assignees

No Assignees

2 Participants

Notifications

Subscribe

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#453

No description provided.