Native support for borg recreate (using config yaml, not arbitrary borg command line) #610

New Issue

Open

opened 2022-11-02 16:24:38 +00:00 by y8s · 7 comments

y8s commented 2022-11-02 16:24:38 +00:00

Copy Link

What I'm trying to do and why

Clean out old backups that have obsolete data so I don't have to pay more for borgbase.

Steps to reproduce (if a bug)

borgmatic -c config.yaml borg recreate --exclude 'olddata.file'
or
borgmatic -c config.yaml borg recreate --exclude 'obsoletedir/*'

Actual behavior (if a bug)

it works fine but is cumbersome

Expected behavior (if a bug)

if I had my way:

borgmatic -c config.yaml recreate
(and that's it.. updates to exclude patterns in config.yaml or referenced exclude.txt automatically take care of themselves)

e.g. in the yaml I could:

location: 
...
  exclude_patterns:
        - /obsoletedir/*
        - /*/olddata.file

and then run borgmatic recreate and it'll refresh all those old archives.

Other notes / implementation ideas

See above. I'd like to just edit my config files and excludes and have that just "work" when doing a borgmatic recreate. Essentially it brings the whole repo up to date without much configuration.

It would save a lot of time trying to figure out where I had backed up data I didn't actually need to back up and remove it from all archives.

Environment

borgmatic version: [1.7.4]

borgmatic installation method: [sudo pip3 install --upgrade borgmatic]

Borg version: [1.1.6]

Python version: [3.9.2]

Database version (if applicable): [N/A]

operating system and version: [Debian Bullseye (proxmox 7)]

#### What I'm trying to do and why Clean out old backups that have obsolete data so I don't have to pay more for borgbase. #### Steps to reproduce (if a bug) `borgmatic -c config.yaml borg recreate --exclude 'olddata.file' ` or `borgmatic -c config.yaml borg recreate --exclude 'obsoletedir/*' ` #### Actual behavior (if a bug) it works fine but is cumbersome #### Expected behavior (if a bug) if I had my way: `borgmatic -c config.yaml recreate` (and that's it.. updates to exclude patterns in config.yaml or referenced exclude.txt automatically take care of themselves) e.g. in the yaml I could: ``` location: ... exclude_patterns: - /obsoletedir/* - /*/olddata.file ``` and then run borgmatic recreate and it'll refresh all those old archives. #### Other notes / implementation ideas See above. I'd like to just edit my config files and excludes and have that just "work" when doing a `borgmatic recreate`. Essentially it brings the whole repo up to date without much configuration. It would save a lot of time trying to figure out where I had backed up data I didn't actually need to back up and remove it from all archives. #### Environment **borgmatic version:** [1.7.4] **borgmatic installation method:** [sudo pip3 install --upgrade borgmatic] **Borg version:** [1.1.6] **Python version:** [3.9.2] **Database version (if applicable):** [N/A] **operating system and version:** [Debian Bullseye (proxmox 7)]

👍 1

witten commented 2022-11-02 16:54:36 +00:00

Owner

Copy Link

That makes sense to me as a feature! I hadn't thought about the exclude integration before.. that's a nice idea. Thanks for taking the time to file this.

That makes sense to me as a feature! I hadn't thought about the exclude integration before.. that's a nice idea. Thanks for taking the time to file this.

y8s commented 2022-11-02 17:26:35 +00:00

Author

Copy Link

It just seemed like it was an obvious way to handle retconning my data. There's already all that exclude setup in place, why not extend it to recreate?

Incidentally, you may want to look at the other options recreate can take and how they can be implemented with config yaml changes. E.g. compression, comments, what have you.

Thanks!

It just seemed like it was an obvious way to handle retconning my data. There's already all that exclude setup in place, why not extend it to recreate? Incidentally, you may want to look at the other options recreate can take and how they can be implemented with config yaml changes. E.g. compression, comments, what have you. Thanks!

👍 2

witten added the

good first issue

label 2023-02-04 17:16:30 +00:00

diivi commented 2023-03-22 13:17:00 +00:00

Collaborator

Copy Link

This should be similar to what borgmatic does for borgmatic create right (except the extra arguments that borgmatic create can take)?

It should take the source repos from config and overwrite them (considering the new config).
Or take one repo via --repository and overwrite it.

And it should ignore any changes to the config that borg recreate does not account for?
Quick links for differences between the args borg recreate and borg create take -
https://borgbackup.readthedocs.io/en/stable/usage/recreate.html, https://borgbackup.readthedocs.io/en/stable/usage/create.html

Also,no support for overwriting a particular archive?

Looks like an interesting task, I would like to work on it!
Will look at the other borgmatic commands.

This should be similar to what borgmatic does for `borgmatic create` right (except the extra arguments that borgmatic create can take)? It should take the source repos from config and overwrite them (considering the new config). Or take one repo via `--repository` and overwrite it. And it should ignore any changes to the config that `borg recreate` does not account for? Quick links for differences between the args `borg recreate` and `borg create` take - https://borgbackup.readthedocs.io/en/stable/usage/recreate.html, https://borgbackup.readthedocs.io/en/stable/usage/create.html Also,no support for overwriting a particular archive? Looks like an interesting task, I would like to work on it! Will look at the other borgmatic commands.

witten commented 2023-03-22 17:45:33 +00:00

Owner

Copy Link

That all sounds reasonable.

Also,no support for overwriting a particular archive?

I think it could go either way. There's at least one example on the borg recreate docs that do this. So you could support that with a borgmatic --archive flag if you wanted to.

That all sounds reasonable. > Also,no support for overwriting a particular archive? I think it could go either way. There's at least one example on the `borg recreate` docs that do this. So you could support that with a borgmatic `--archive` flag if you wanted to.

y8s commented 2023-05-22 14:43:10 +00:00

Author

Copy Link

Any news on this? I keep creating and removing VMs and all the old VMs that I never intend to use again are lingering in my archives.

Any news on this? I keep creating and removing VMs and all the old VMs that I never intend to use again are lingering in my archives.

witten commented 2023-05-22 18:37:57 +00:00

Owner

Copy Link

As far as I know, no one is working on this one right now. Pull requests are always welcome. 😄

As far as I know, no one is working on this one right now. Pull requests are always welcome. 😄

y8s commented 2023-05-22 18:40:07 +00:00

Author

Copy Link

As far as I know, no one is working on this one right now. Pull requests are always welcome. 😄

Understood. I best get learning to code I guess.

> As far as I know, no one is working on this one right now. Pull requests are always welcome. 😄 Understood. I best get learning to code I guess.

😆 1

witten added the

new feature area

label 2023-06-28 18:39:59 +00:00

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

No results found.

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.14

1.8.13

1.8.12

1.8.11

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.24

1.5.23

1.5.22

1.5.21

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.14

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.7.dev0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.22

1.4.21

1.4.20

1.4.19

1.4.18

1.4.17

1.4.16

1.4.15

1.4.14

1.4.13

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.26

1.3.25

1.3.24

1.3.23

1.3.22

1.3.21

1.3.20

1.3.19

1.3.18

1.3.17

1.3.16

1.3.15

1.3.14

1.3.13

1.3.12

1.3.11

1.3.10

1.3.9

1.3.8

1.3.7

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.18

1.2.17

1.2.16

1.2.15

1.2.14

1.2.13

1.2.12

1.2.11

1.2.10

1.2.9

1.2.8

1.2.7

1.2.6

1.2.5

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.15

1.1.14

1.1.13

1.1.12

1.1.11

1.1.10

1.1.9

1.1.7

1.1.6

1.1.5

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.3

1.0.2

1.0.1

1.0.0

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

0.0.2

Labels

Clear labels

blocked

bug

data loss

design finalized

good first issue

new feature area

question / support

security

waiting for response

No Label

good first issue

new feature area

Milestone

No items

No Milestone

Assignees

Clear assignees

No Assignees

3 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#610

No description provided.