[Bug] source_directories_must_exist does not expand globs and tildes #682

New Issue

Closed

opened 2023-04-20 00:10:16 +00:00 by holocronweaver · 2 comments

holocronweaver commented 2023-04-20 00:10:16 +00:00

Contributor

Copy Link

What I'm trying to do and why

I am trying to add source_directories_must_exist: true to my existing Borgmatic config so I can make use of this feature. However I any directories with globs or tildes are not being expanded and always fail the existence check.

Steps to reproduce (if a bug)

Add source directories with globs and tildes while enabling source_directories_must_exist.

Example config file:

location:
    source_directories:
        - ~/downloads
        - /home/me/.bash*

    repositories:
        - /home/me/myrepo

    # If true, then source directories must exist, otherwise an
    # error is raised. Defaults to false.
    source_directories_must_exist: true

Removing source_directories_must_exist resolves the issue.

Actual behavior (if a bug)

borgmatic -c backup.yaml
...
summary:
backup.yaml: An error occurred
/home/me/myrepo: Error running actions for repository
Source directories do not exist: ~/downloads, /home/me/.bash*

Expected behavior (if a bug)

A successful backup of source directories containing globs and tildes, as was the case before adding source_directories_must_exist.

Other notes / implementation ideas

Seems no expansion is being done during the existence check.

def check_all_source_directories_exist(source_directories):
    '''
    Given a sequence of source directories, check that they all exist. If any do not, raise an
    exception.
    '''
    missing_directories = [
        source_directory
        for source_directory in source_directories
        if not os.path.exists(source_directory)
    ]
    if missing_directories:
        raise ValueError(f"Source directories do not exist: {', '.join(missing_directories)}")

so should be a simple matter to fix:

if not os.path.exists(expand_directory(source_directory))

Happy to open a pull request to fix and add a unit test.

Environment

borgmatic version: 1.7.12

Use sudo borgmatic --version or sudo pip show borgmatic | grep ^Version

borgmatic installation method: pip

Borg version: 1.2.1

Use sudo borg --version

Python version: 3.10.6

Use python3 --version

Database version (if applicable): n/a

Use psql --version or mysql --version on client and server.

operating system and version: Ubuntu 22.04.2

#### What I'm trying to do and why I am trying to add `source_directories_must_exist: true` to my existing Borgmatic config so I can make use of this feature. However I any directories with globs or tildes are not being expanded and always fail the existence check. #### Steps to reproduce (if a bug) Add source directories with globs and tildes while enabling `source_directories_must_exist`. Example config file: ``` location: source_directories: - ~/downloads - /home/me/.bash* repositories: - /home/me/myrepo # If true, then source directories must exist, otherwise an # error is raised. Defaults to false. source_directories_must_exist: true ``` Removing `source_directories_must_exist` resolves the issue. #### Actual behavior (if a bug) ``` borgmatic -c backup.yaml ... summary: backup.yaml: An error occurred /home/me/myrepo: Error running actions for repository Source directories do not exist: ~/downloads, /home/me/.bash* ``` #### Expected behavior (if a bug) A successful backup of source directories containing globs and tildes, as was the case before adding `source_directories_must_exist`. #### Other notes / implementation ideas Seems [no expansion is being done during the existence check](https://github.com/borgmatic-collective/borgmatic/blob/9f5769f87be3e75a3b86ee1a6f31a6754ec2ed4e/borgmatic/borg/create.py#L317). ``` def check_all_source_directories_exist(source_directories): ''' Given a sequence of source directories, check that they all exist. If any do not, raise an exception. ''' missing_directories = [ source_directory for source_directory in source_directories if not os.path.exists(source_directory) ] if missing_directories: raise ValueError(f"Source directories do not exist: {', '.join(missing_directories)}") ``` so should be a simple matter to fix: ``` if not os.path.exists(expand_directory(source_directory)) ``` Happy to open a pull request to fix and add a unit test. #### Environment **borgmatic version:** 1.7.12 Use `sudo borgmatic --version` or `sudo pip show borgmatic | grep ^Version` **borgmatic installation method:** pip **Borg version:** 1.2.1 Use `sudo borg --version` **Python version:** 3.10.6 Use `python3 --version` **Database version (if applicable):** n/a Use `psql --version` or `mysql --version` on client and server. **operating system and version:** Ubuntu 22.04.2

holocronweaver referenced a pull request that will close this issue 2023-04-20 01:56:58 +00:00

Expand source directories when checking for existence. #683

witten commented 2023-04-20 03:27:49 +00:00

Owner

Copy Link

Good catch! I think your analysis is completely correct here. And yes, I'd welcome a PR if you're up to it.

Good catch! I think your analysis is completely correct here. And yes, I'd welcome a PR if you're up to it.

witten added the

bug

label 2023-04-20 03:27:55 +00:00

witten closed this issue 2023-04-21 06:06:00 +00:00

witten referenced this issue from a commit 2023-04-21 06:06:00 +00:00

Expand source directories when checking for existence (#682).

witten referenced this issue from a commit 2023-04-21 06:06:00 +00:00

Expand source directories when checking for existence (#682).

witten referenced this issue from a commit 2023-04-21 06:15:36 +00:00

Add "source_directories_must_exist" expansion fix to NEWS (#682).

witten commented 2023-05-16 17:04:32 +00:00

Owner

Copy Link

This was just released in borgmatic 1.7.13!

This was just released in borgmatic 1.7.13!

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

spot-check

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.24

1.5.23

1.5.22

1.5.21

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.14

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.7.dev0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.22

1.4.21

1.4.20

1.4.19

1.4.18

1.4.17

1.4.16

1.4.15

1.4.14

1.4.13

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.26

1.3.25

1.3.24

1.3.23

1.3.22

1.3.21

1.3.20

1.3.19

1.3.18

1.3.17

1.3.16

1.3.15

1.3.14

1.3.13

1.3.12

1.3.11

1.3.10

1.3.9

1.3.8

1.3.7

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.18

1.2.17

1.2.16

1.2.15

1.2.14

1.2.13

1.2.12

1.2.11

1.2.10

1.2.9

1.2.8

1.2.7

1.2.6

1.2.5

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.15

1.1.14

1.1.13

1.1.12

1.1.11

1.1.10

1.1.9

1.1.7

1.1.6

1.1.5

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.3

1.0.2

1.0.1

1.0.0

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

0.0.2

Labels

Clear labels   

bug

  

data loss

  

design finalized

  

good first issue

  

new feature area

  

question / support

  

security

  

waiting for response

No Label

bug

data loss

design finalized

good first issue

new feature area

question / support

security

waiting for response

Milestone

Clear milestone

No items

No Milestone

Assignees

Clear assignees

No Assignees

2 Participants

Notifications

Subscribe

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#682

No description provided.