Multiple entries of exclude_if_present #280

Closed
opened 2020-01-12 22:52:45 +00:00 by codingforfun · 8 comments

What I'm trying to do and why

I try to add multiple entries to exclude_if_present flag like so:

    exclude_if_present: 
        - .nobackup
        - CMakeCache.txt

I verified that borg (1.1.10) itself can handle specifying --exclude-if-present flag mulitple times with different arguments.

borg create … --exclude-if-present .nobackup --exclude-if-present CMakeCache.txt …

Why??? Because:

I have a code folder where all my coding projects are in.
The most relevant, both in size and number, are C++ projects build with CMake.
CMake creates a separate build directory with all the binary artifacts in it.
This build directory is has the unique property of always having a CMakeCache.txt at it’s top level.
It would be really a mess of trying to exclude all this build artifacts with explicit paths or trying to come up with a complete set of patterns to exclude everything (like '*.pyc' for python). Thus it would be really easy and nice and foolproof to just exclude all the CMake created build trees by allowing to specify
--exclude-if-present .CMakeCache.txt while keeping the more general .nobackup for the general case.
Others might have other use cases for this too.

Actual behavior

borgmatic complains about a wrong type for exclude_if_present, because it expects a string and errors out with a message about broken configuration.

Expected behavior

Would be nice if borgmatic could just translate the list into multiple times specifying the --exclude-if-present flag to borg create with different arguments for each list entry.

Other notes / implementation ideas

Environment

borgmatic version: 1.4.21

borgmatic installation method: pip

Borg version: 1.1.10

Python version: 3.5.2

#### What I'm trying to do and why I try to add multiple entries to `exclude_if_present` flag like so: ```yml exclude_if_present: - .nobackup - CMakeCache.txt ``` I verified that borg (1.1.10) itself can handle specifying `--exclude-if-present` flag mulitple times with different arguments. ``` borg create … --exclude-if-present .nobackup --exclude-if-present CMakeCache.txt … ``` **Why??? Because:** I have a code folder where all my coding projects are in. The most relevant, both in size and number, are C++ projects build with CMake. CMake creates a separate build directory with all the binary artifacts in it. This build directory is has the unique property of always having a CMakeCache.txt at it’s top level. It would be really a mess of trying to exclude all this build artifacts with explicit paths or trying to come up with a complete set of patterns to exclude everything (like '*.pyc' for python). Thus it would be really easy and nice and foolproof to just exclude all the CMake created build trees by allowing to specify `--exclude-if-present .CMakeCache.txt` while keeping the more general `.nobackup` for the general case. Others might have other use cases for this too. #### Actual behavior borgmatic complains about a wrong type for `exclude_if_present`, because it expects a string and errors out with a message about broken configuration. #### Expected behavior Would be nice if borgmatic could just translate the list into multiple times specifying the `--exclude-if-present` flag to `borg create` with different arguments for each list entry. #### Other notes / implementation ideas #### Environment **borgmatic version:** 1.4.21 **borgmatic installation method:** pip **Borg version:** 1.1.10 **Python version:** 3.5.2
Owner

Seems reasonable to me.. Thanks for suggesting this! I think the only tricky part will be dealing with the breaking change, since presumably this involves validating exclude_if_present as a list instead of a string. That means existing configuration files with just a string value may need to be converted manually. Still, probably worth doing.

Seems reasonable to me.. Thanks for suggesting this! I think the only tricky part will be dealing with the breaking change, since presumably this involves validating `exclude_if_present` as a list instead of a string. That means existing configuration files with just a string value may need to be converted manually. Still, probably worth doing.
witten added the
design finalized
label 2020-01-13 18:55:03 +00:00
Author

I had no look into the code yet, but don’t you think this can be solved by introspection. I mean just looking at the given type. If it’s a string, then use the legacy behavior otherwise if it’s a list use the new behavior.

I had no look into the code yet, but don’t you think this can be solved by introspection. I mean just looking at the given type. If it’s a string, then use the legacy behavior otherwise if it’s a list use the new behavior.
Owner

In theory, yes. In practice, the pykwalify YAML validation library that borgmatic uses has a fixed schema where each value has to be of the declared type: string, list, int, bool, etc. As far as I know, it doesn't support a way to say "this value can be a string or a list".

Even so, there may be a way to do some pre-processing to convert any strings found to a list before feeding it to validation, but that's just a random idea. Don't know if it'd actually work.

In theory, yes. In practice, the pykwalify YAML validation library that borgmatic uses has a fixed schema where each value has to be of the declared type: string, list, int, bool, etc. As far as I know, it doesn't support a way to say "this value can be a string *or* a list". Even so, there may be a way to do some pre-processing to convert any strings found to a list before feeding it to validation, but that's just a random idea. Don't know if it'd actually work.
Author

Btw. I currently just use:

    extra_borg_options:
        create: --exclude-if-present CMakeCache.txt

for this and it works like a charm.

Btw. I currently just use: ``` extra_borg_options: create: --exclude-if-present CMakeCache.txt ``` for this and it works like a charm.
Owner

Glad to hear you've got a work-around for the time being.

Glad to hear you've got a work-around for the time being.
Owner

This is implemented now in master. I managed to make it backwards compatible, such that exclude_if_present can either be a string or a list. (Some pre-processing normalization code "upgrades" any strings found to lists before schema validation.)

I'll post here when the next release is out! Thanks again for the idea.

This is implemented now in master. I managed to make it backwards compatible, such that `exclude_if_present` can either be a string or a list. (Some pre-processing normalization code "upgrades" any strings found to lists before schema validation.) I'll post here when the next release is out! Thanks again for the idea.
Author

Awesome! Thanks a lot! Great work.

Awesome! Thanks a lot! Great work.
Owner

Released in borgmatic 1.5.0!

Released in borgmatic 1.5.0!
Sign in to join this conversation.
No Milestone
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#280
No description provided.