Bash-style include patterns for source_directories #637

New Issue

Closed

opened 2023-02-09 15:43:30 +00:00 by persal · 2 comments

persal commented 2023-02-09 15:43:30 +00:00

Copy Link

What I'm trying to do and why

We want to make sure no "bak" directories are included in the backup. Having convention for exclude_pattern exist, but if some users do not follow the naming convention, it will of course be backed up. This is what we want to avoid.

So instead, we try to narrow down exactly what to backup. Is it possible to describe folders with bash-expansion {..} or similar?

Lets say we want to backup somedir/year/month/day/* paths:

/data/node_data/*/2023/01/01/*
...
/data/node_data/*/2023/12/31/*

And we want to make sure to exclude paths like:

/data/node_data/*/2023/01.bak/31/*
/data/node_data/*/2023/01.somename/31/*
/data/node_data/*/2023/tmpdir123/31/*

source_directories could look like:

/data/node_data/*/2023/{01,02,03,04,05,06,07,08,09,10,11,12}/{01,02,03,04,05,06,07,08,09,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31}/*

Or even better:

/data/node_data/*/2023/{01..12}/{01..31}/*

Is this possible?

I have tried using the
"patterns" with regex and + and - but find it having quite strang behavior.

Environment

borgmatic version: 1.5.24
Borg version: 1.2.0
Python version: 3.9.2
operating system and version: Debian 11

#### What I'm trying to do and why We want to make sure no "bak" directories are included in the backup. Having convention for exclude_pattern exist, but if some users do not follow the naming convention, it will of course be backed up. This is what we want to avoid. So instead, we try to narrow down exactly what to backup. Is it possible to describe folders with bash-expansion {..} or similar? Lets say we want to backup `somedir/year/month/day/*` paths: ``` /data/node_data/*/2023/01/01/* ... /data/node_data/*/2023/12/31/* ``` And we want to make sure to exclude paths like: ``` /data/node_data/*/2023/01.bak/31/* /data/node_data/*/2023/01.somename/31/* /data/node_data/*/2023/tmpdir123/31/* ``` source_directories could look like: ``` /data/node_data/*/2023/{01,02,03,04,05,06,07,08,09,10,11,12}/{01,02,03,04,05,06,07,08,09,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31}/* ``` Or even better: ``` /data/node_data/*/2023/{01..12}/{01..31}/* ``` Is this possible? I have tried using the "patterns" with regex and + and - but find it having quite strang behavior. #### Environment **borgmatic version:** 1.5.24 **Borg version:** 1.2.0 **Python version:** 3.9.2 **operating system and version:** Debian 11

witten commented 2023-02-09 17:10:09 +00:00

Owner

Copy Link

I don't believe what you're trying to do is currently possible with source_directories, as that only expands simple globs (*) and tildes (~). For anything more complex, you'll need to use patterns, which includes support for regexes, shell-style expressions, etc. Perhaps you can post what you've tried so far with patterns and describe how it's not working? I'd be happy to help debug.

I don't believe what you're trying to do is *currently* possible with `source_directories`, as that only expands simple globs (`*`) and tildes (`~`). For anything more complex, you'll need to use `patterns`, which [includes support for regexes, shell-style expressions, etc.](https://borgbackup.readthedocs.io/en/stable/usage/help.html#borg-patterns) Perhaps you can post what you've tried so far with patterns and describe how it's not working? I'd be happy to help debug.

witten added the

question / support

label 2023-02-09 17:10:20 +00:00

witten commented 2023-03-08 17:43:27 +00:00

Owner

Copy Link

Closing this for now, but please feel free to follow up and/or reopen so we can continue discussing. Thanks!

Closing this for now, but please feel free to follow up and/or reopen so we can continue discussing. Thanks!

witten closed this issue 2023-03-08 17:43:33 +00:00

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

spot-check

1.8.11

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.24

1.5.23

1.5.22

1.5.21

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.14

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.7.dev0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.22

1.4.21

1.4.20

1.4.19

1.4.18

1.4.17

1.4.16

1.4.15

1.4.14

1.4.13

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.26

1.3.25

1.3.24

1.3.23

1.3.22

1.3.21

1.3.20

1.3.19

1.3.18

1.3.17

1.3.16

1.3.15

1.3.14

1.3.13

1.3.12

1.3.11

1.3.10

1.3.9

1.3.8

1.3.7

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.18

1.2.17

1.2.16

1.2.15

1.2.14

1.2.13

1.2.12

1.2.11

1.2.10

1.2.9

1.2.8

1.2.7

1.2.6

1.2.5

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.15

1.1.14

1.1.13

1.1.12

1.1.11

1.1.10

1.1.9

1.1.7

1.1.6

1.1.5

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.3

1.0.2

1.0.1

1.0.0

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

0.0.2

Labels

Clear labels   

bug

  

data loss

  

design finalized

  

good first issue

  

new feature area

  

question / support

  

security

  

waiting for response

No Label

bug

data loss

design finalized

good first issue

new feature area

question / support

security

waiting for response

Milestone

Clear milestone

No items

No Milestone

Assignees

Clear assignees

No Assignees

2 Participants

Notifications

Subscribe

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#637

No description provided.