Glob-Expansion (of source directories) does not work on "working directory" #609

New Issue

Closed

opened 2022-10-29 10:18:24 +00:00 by simon-77 · 6 comments

simon-77 commented 2022-10-29 10:18:24 +00:00

Copy Link

Hi, it's me again :)

What I'm trying to do and why

I try to use relative paths during backup via working_directory field.

Issue: using relative-path globs in source_directories field are not resolved relative to working_directory

Steps to reproduce

location:
    working_directory: /path/to/work-dir
    source_directories:
        - "*"

Actual behavior

running $ sudo borgmatic create --verbosity 2 --dry-run lists the create command with * expanded to a list of all files/folders in current directory (where this command is executed)

Partial output:

borg create --dry-run <path-to-repo>::{hostname}-user-{now} /root/.borgmatic <a list of files/folders in current directory> --debug --show-rc

Additionally I am curious why /root/.borgmatic is always included in the backup ?

Expected behavior

1. a list of files in <path/to/work-dir>
2. not having /root/.borgmatic in the backup included - although, if it has a reason that this directory is always included, I am fine with this fact.

Other notes / implementation ideas

Notes about the location.source_directories field:

I expected - * to work, but this throws an YAML-parser error: did not find expected alphabetic or numeric character

Additionally I tried - ./* which gives the same result as - "*" - resolving relative to current directory instead of working directory.

Environment

borgmatic version: 1.4.4

borgmatic installation method: openSUSE package manger

Borg version: 1.2.2

Python version: 3.10.7

operating system and version:
Operating System: openSUSE Tumbleweed 20221026
Kernel Version: 6.0.3-1-default (64-bit)

Hi, it's me again :) #### What I'm trying to do and why I try to use relative paths during backup via `working_directory` field. Issue: using relative-path globs in `source_directories` field are not resolved relative to `working_directory` #### Steps to reproduce ``` YAML location: working_directory: /path/to/work-dir source_directories: - "*" ``` #### Actual behavior running `$ sudo borgmatic create --verbosity 2 --dry-run` lists the create command with `*` expanded to a list of all files/folders in current directory (where this command is executed) Partial output: ``` borg create --dry-run <path-to-repo>::{hostname}-user-{now} /root/.borgmatic <a list of files/folders in current directory> --debug --show-rc ``` Additionally I am curious why `/root/.borgmatic` is always included in the backup ? #### Expected behavior 1) a list of files in <path/to/work-dir> 2) not having /root/.borgmatic in the backup included - although, if it has a reason that this directory is always included, I am fine with this fact. #### Other notes / implementation ideas Notes about the `location.source_directories` field: I expected `- *` to work, but this throws an YAML-parser error: `did not find expected alphabetic or numeric character` Additionally I tried `- ./*` which gives the same result as `- "*"` - resolving relative to current directory instead of working directory. #### Environment borgmatic version: 1.4.4 borgmatic installation method: openSUSE package manger Borg version: 1.2.2 Python version: 3.10.7 operating system and version: Operating System: openSUSE Tumbleweed 20221026 Kernel Version: 6.0.3-1-default (64-bit)

👍 1

simon-77 commented 2022-10-29 10:26:17 +00:00

Author

Copy Link

Additional Note:

The Level-2 output of borgmatic create has no info that the relative working_directory path is used.

I gues borgmatic simply does a cd bevore. Maybe include this in Level-2 output.

Additional Note: The Level-2 output of `borgmatic create` has no info that the relative `working_directory` path is used. I gues borgmatic simply does a `cd` bevore. Maybe include this in Level-2 output.

simon-77 commented 2022-10-29 14:11:49 +00:00

Author

Copy Link

I now have a work-around:

location:
    working_directory: /path/to/work-dir
    source_directories:
        - .

includes all directories and files of /path/to/work-dir in a relative manner.

My initial goal was to only include directories (and files) that do not start with a dot. For this I found the following glob pattern to work: [!.]*
This is how I came up with the simplified example in the original post.

I just noticed: for not having .* directories/files in the repo I could also simply use the exclude patterns.

I now have a work-around: ``` YAML location: working_directory: /path/to/work-dir source_directories: - . ``` includes all directories and files of `/path/to/work-dir` in a relative manner. My initial goal was to only include directories (and files) that do not start with a dot. For this I found the following glob pattern to work: `[!.]*` This is how I came up with the simplified example in the original post. I just noticed: for not having `.*` directories/files in the repo I could also simply use the exclude patterns.

witten commented 2023-03-27 03:18:39 +00:00

Owner

Copy Link

I apologize for the delay in getting to this. Supporting globs relative to the configured working directory seems like a totally reasonable feature!

I expected - * to work, but this throws an YAML-parser error: did not find expected alphabetic or numeric character

Yeah, this is a YAML limitation. You need to quote the asterisk, as you discovered.

The Level-2 output of borgmatic create has no info that the relative working_directory path is used.

Agreed.. This would be a good thing to log!

Additionally I am curious why /root/.borgmatic is always included in the backup ?

The short answer is that ~/.borgmatic is included to support backing up database dumps. More info is here: https://torsion.org/borgmatic/docs/how-to/backup-your-databases/

I apologize for the delay in getting to this. Supporting globs relative to the configured working directory seems like a totally reasonable feature! > I expected - * to work, but this throws an YAML-parser error: did not find expected alphabetic or numeric character Yeah, this is a YAML limitation. You need to quote the asterisk, as you discovered. > The Level-2 output of borgmatic create has no info that the relative working_directory path is used. Agreed.. This would be a good thing to log! > Additionally I am curious why /root/.borgmatic is always included in the backup ? The short answer is that `~/.borgmatic` is included to support backing up database dumps. More info is here: https://torsion.org/borgmatic/docs/how-to/backup-your-databases/

witten added the good first issue label 2023-03-27 03:18:46 +00:00

spali commented 2024-10-09 12:14:04 +00:00

Copy Link

I ran into the same issue with.
I would workaround this limitation with specifying the directory without globs as i only use them to workaround another limitation.
My top-level directory to backup is a symbolic link (unraid share using single disk). So defining top-level directory in source_directories results in only backup the symbolic link.

Situation:
Content to backup: /mnt/user/myshare.
In unraid this may be a symbolic link to the actual disk mount.

source_directories:
  - /mnt/user/myshare

results in having an archive containing only the symbolic link myshare.
I don't want to open a discussion to how symbolic links should be backed up... In my eyes in general they should as they are. Current behavior. But in this special case of a top-level directory it would make sense to make a exception. In most shells you can append a trailing / to symbolic links which most of the time works perfect for this exception.
Unfortunately this workaround doesn't work with borg/borgmatic.

So a workaround would be:

source_directories:
  - /mnt/user/myshare/*
  - /mnt/user/myshare/.*

But this does not work in combination with working_directory if you want to "hide" the common location from the backup.

working_directory: /mnt/user/myshare
source_directories:
  - "*"
  - ".*"

results in all directories and files in the working directory of borg/borgmatic instead the defined working_directory.

I ran into the same issue with. I would workaround this limitation with specifying the directory without globs as i only use them to workaround another limitation. My top-level directory to backup is a symbolic link (unraid share using single disk). So defining top-level directory in `source_directories` results in only backup the symbolic link. Situation: Content to backup: `/mnt/user/myshare`. In unraid this may be a symbolic link to the actual disk mount. ``` source_directories: - /mnt/user/myshare ``` results in having an archive containing only the symbolic link `myshare`. I don't want to open a discussion to how symbolic links should be backed up... In my eyes in general they should as they are. Current behavior. But in this special case of a top-level directory it would make sense to make a exception. In most shells you can append a trailing `/` to symbolic links which most of the time works perfect for this exception. Unfortunately this workaround doesn't work with borg/borgmatic. So a workaround would be: ``` source_directories: - /mnt/user/myshare/* - /mnt/user/myshare/.* ``` But this does not work in combination with `working_directory` if you want to "hide" the common location from the backup. ``` working_directory: /mnt/user/myshare source_directories: - "*" - ".*" ``` results in all directories and files in the working directory of borg/borgmatic instead the defined `working_directory`.

witten referenced this issue from a commit 2024-10-20 23:05:50 +00:00

Apply the "working_directory" option to all actions, not just "create". Also fix the glob expansion of "source_directories" values to respect the "working_directory" option (#609).

witten commented 2024-10-20 23:07:35 +00:00

Owner

Copy Link

So in working on this ticket, I, uh, kind of expanded scope. Now working_directory applies to all borgmatic actions, not just create. And as part of that work, the original ask in this ticket was also implemented.

This will be part of the next release. Thanks to @simon-77 for filing this (and to @spali for weighing in)!

So in working on this ticket, I, uh, kind of expanded scope. Now `working_directory` applies to *all* borgmatic actions, not just `create`. And as part of that work, the original ask in this ticket was also implemented. This will be part of the next release. Thanks to @simon-77 for filing this (and to @spali for weighing in)!

👍 1

witten closed this issue 2024-10-20 23:07:38 +00:00

witten commented 2024-11-03 21:28:51 +00:00

Owner

Copy Link

Released in borgmatic 1.9.0!

Released in borgmatic 1.9.0!

🎉 1

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

command-hook-steps

2.0.11

2.0.10

2.0.9

2.0.8

2.0.7

2.0.6

2.0.5

2.0.4

2.0.3

2.0.2

2.0.1

2.0.0

1.9.14

1.9.13

1.9.12

1.9.11

1.9.10

1.9.9

1.9.8

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.14

1.8.13

1.8.12

1.8.11

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.24

1.5.23

1.5.22

1.5.21

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.14

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.7.dev0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.22

1.4.21

1.4.20

1.4.19

1.4.18

1.4.17

1.4.16

1.4.15

1.4.14

1.4.13

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.26

1.3.25

1.3.24

1.3.23

1.3.22

1.3.21

1.3.20

1.3.19

1.3.18

1.3.17

1.3.16

1.3.15

1.3.14

1.3.13

1.3.12

1.3.11

1.3.10

1.3.9

1.3.8

1.3.7

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.18

1.2.17

1.2.16

1.2.15

1.2.14

1.2.13

1.2.12

1.2.11

1.2.10

1.2.9

1.2.8

1.2.7

1.2.6

1.2.5

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.15

1.1.14

1.1.13

1.1.12

1.1.11

1.1.10

1.1.9

1.1.7

1.1.6

1.1.5

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.3

1.0.2

1.0.1

1.0.0

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

0.0.2

Labels

Clear labels

blocked

breaking

bug

data loss

design finalized

good first issue

new feature area

question / support

security

waiting for response

No Label good first issue

Milestone

No items

No Milestone

Assignees

Clear assignees

witten

No Assignees

3 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#609