borgmatic-collective/borgmatic
9
73
Fork 51
Code Issues 52 Pull Requests 3 Actions 1 Packages Releases 158 Activity

Native support for borg recreate (using config yaml, not arbitrary borg command line) #610

New Issue
Closed
opened 2022-11-02 16:24:38 +00:00 by y8s · 16 comments
y8s commented 2022-11-02 16:24:38 +00:00
Copy Link

What I'm trying to do and why

Clean out old backups that have obsolete data so I don't have to pay more for borgbase.

Steps to reproduce (if a bug)

borgmatic -c config.yaml borg recreate --exclude 'olddata.file'
or
borgmatic -c config.yaml borg recreate --exclude 'obsoletedir/*'

Actual behavior (if a bug)

it works fine but is cumbersome

Expected behavior (if a bug)

if I had my way:

borgmatic -c config.yaml recreate
(and that's it.. updates to exclude patterns in config.yaml or referenced exclude.txt automatically take care of themselves)

e.g. in the yaml I could:

location: 
...
  exclude_patterns:
        - /obsoletedir/*
        - /*/olddata.file

and then run borgmatic recreate and it'll refresh all those old archives.

Other notes / implementation ideas

See above. I'd like to just edit my config files and excludes and have that just "work" when doing a borgmatic recreate. Essentially it brings the whole repo up to date without much configuration.

It would save a lot of time trying to figure out where I had backed up data I didn't actually need to back up and remove it from all archives.

Environment

borgmatic version: [1.7.4]

borgmatic installation method: [sudo pip3 install --upgrade borgmatic]

Borg version: [1.1.6]

Python version: [3.9.2]

Database version (if applicable): [N/A]

operating system and version: [Debian Bullseye (proxmox 7)]

#### What I'm trying to do and why Clean out old backups that have obsolete data so I don't have to pay more for borgbase. #### Steps to reproduce (if a bug) `borgmatic -c config.yaml borg recreate --exclude 'olddata.file' ` or `borgmatic -c config.yaml borg recreate --exclude 'obsoletedir/*' ` #### Actual behavior (if a bug) it works fine but is cumbersome #### Expected behavior (if a bug) if I had my way: `borgmatic -c config.yaml recreate` (and that's it.. updates to exclude patterns in config.yaml or referenced exclude.txt automatically take care of themselves) e.g. in the yaml I could: ``` location: ... exclude_patterns: - /obsoletedir/* - /*/olddata.file ``` and then run borgmatic recreate and it'll refresh all those old archives. #### Other notes / implementation ideas See above. I'd like to just edit my config files and excludes and have that just "work" when doing a `borgmatic recreate`. Essentially it brings the whole repo up to date without much configuration. It would save a lot of time trying to figure out where I had backed up data I didn't actually need to back up and remove it from all archives. #### Environment **borgmatic version:** [1.7.4] **borgmatic installation method:** [sudo pip3 install --upgrade borgmatic] **Borg version:** [1.1.6] **Python version:** [3.9.2] **Database version (if applicable):** [N/A] **operating system and version:** [Debian Bullseye (proxmox 7)]
👍 1
witten commented 2022-11-02 16:54:36 +00:00
Owner
Copy Link

That makes sense to me as a feature! I hadn't thought about the exclude integration before.. that's a nice idea. Thanks for taking the time to file this.

That makes sense to me as a feature! I hadn't thought about the exclude integration before.. that's a nice idea. Thanks for taking the time to file this.
y8s commented 2022-11-02 17:26:35 +00:00
Author
Copy Link

It just seemed like it was an obvious way to handle retconning my data. There's already all that exclude setup in place, why not extend it to recreate?

Incidentally, you may want to look at the other options recreate can take and how they can be implemented with config yaml changes. E.g. compression, comments, what have you.

Thanks!

It just seemed like it was an obvious way to handle retconning my data. There's already all that exclude setup in place, why not extend it to recreate? Incidentally, you may want to look at the other options recreate can take and how they can be implemented with config yaml changes. E.g. compression, comments, what have you. Thanks!
👍 2
witten added the good first issue label 2023-02-04 17:16:30 +00:00
diivi commented 2023-03-22 13:17:00 +00:00
Collaborator
Copy Link

This should be similar to what borgmatic does for borgmatic create right (except the extra arguments that borgmatic create can take)?

It should take the source repos from config and overwrite them (considering the new config).
Or take one repo via --repository and overwrite it.

And it should ignore any changes to the config that borg recreate does not account for?
Quick links for differences between the args borg recreate and borg create take -
https://borgbackup.readthedocs.io/en/stable/usage/recreate.html, https://borgbackup.readthedocs.io/en/stable/usage/create.html

Also,no support for overwriting a particular archive?

Looks like an interesting task, I would like to work on it!
Will look at the other borgmatic commands.

This should be similar to what borgmatic does for `borgmatic create` right (except the extra arguments that borgmatic create can take)? It should take the source repos from config and overwrite them (considering the new config). Or take one repo via `--repository` and overwrite it. And it should ignore any changes to the config that `borg recreate` does not account for? Quick links for differences between the args `borg recreate` and `borg create` take - https://borgbackup.readthedocs.io/en/stable/usage/recreate.html, https://borgbackup.readthedocs.io/en/stable/usage/create.html Also,no support for overwriting a particular archive? Looks like an interesting task, I would like to work on it! Will look at the other borgmatic commands.
witten commented 2023-03-22 17:45:33 +00:00
Owner
Copy Link

That all sounds reasonable.

Also,no support for overwriting a particular archive?

I think it could go either way. There's at least one example on the borg recreate docs that do this. So you could support that with a borgmatic --archive flag if you wanted to.

That all sounds reasonable. > Also,no support for overwriting a particular archive? I think it could go either way. There's at least one example on the `borg recreate` docs that do this. So you could support that with a borgmatic `--archive` flag if you wanted to.
y8s commented 2023-05-22 14:43:10 +00:00
Author
Copy Link

Any news on this? I keep creating and removing VMs and all the old VMs that I never intend to use again are lingering in my archives.

Any news on this? I keep creating and removing VMs and all the old VMs that I never intend to use again are lingering in my archives.
witten commented 2023-05-22 18:37:57 +00:00
Owner
Copy Link

As far as I know, no one is working on this one right now. Pull requests are always welcome. 😄

As far as I know, no one is working on this one right now. Pull requests are always welcome. 😄
y8s commented 2023-05-22 18:40:07 +00:00
Author
Copy Link

As far as I know, no one is working on this one right now. Pull requests are always welcome. 😄

Understood. I best get learning to code I guess.

> As far as I know, no one is working on this one right now. Pull requests are always welcome. 😄 Understood. I best get learning to code I guess.
😆 1
witten added the new feature area label 2023-06-28 18:39:59 +00:00
VandalByte commented 2025-03-15 08:36:39 +00:00
Contributor
Copy Link

I would like to try my luck with this.

I would like to try my luck with this.
witten commented 2025-03-15 17:12:41 +00:00
Owner
Copy Link

Awesome, welcome to the borgmatic project! Please have a look at the docs on how to get started with development! It might also save you time if you wanted to outline your approach to the ticket here to get feedback before you start writing code, especially since this is one of the larger "good first issue" tickets.

In terms of where to start in the source code once your design is locked down:

  • borgmatic/commands/arguments.py is where the CLI interface for each action is defined.
  • borgmatic/actions/recreate.py is where any borgmatic logic for the new recreate action should go.
  • borgmatic/borg/recreate.py is where the code to actually call borg recreate should go.
  • borgmatic/unit/tests/... contains the hierarchy of unit tests. Look at some of the existing actions for inspiration.

And let me know if you have any questions!

Awesome, welcome to the borgmatic project! Please have a look at the [docs on how to get started with development](https://torsion.org/borgmatic/docs/how-to/develop-on-borgmatic/)! It might also save you time if you wanted to outline your approach to the ticket here to get feedback before you start writing code, especially since this is one of the larger "good first issue" tickets. In terms of where to start in [the source code](https://torsion.org/borgmatic/docs/reference/source-code/) once your design is locked down: - `borgmatic/commands/arguments.py` is where the CLI interface for each action is defined. - `borgmatic/actions/recreate.py` is where any borgmatic logic for the new `recreate` action should go. - `borgmatic/borg/recreate.py` is where the code to actually call `borg recreate` should go. - `borgmatic/unit/tests/...` contains the hierarchy of unit tests. Look at some of the existing actions for inspiration. And let me know if you have any questions!
VandalByte commented 2025-03-16 06:43:02 +00:00
Contributor
Copy Link

Hey witten
Thanks for the heads up! I've been working with Vorta, so I'm fairly new to the Borgmatic codebase. Once I fully understand the relevant parts needed for this implementation, I’ll go ahead and create a PR and explain the process.

For now, I’m thinking of looking into how create is implemented and using that as inspiration for recreate. I’ll keep you updated on my progress!

I do have a question, it's about testing the changes. From my current understanding, running tox will run the unit tests, right? I’m wondering how to manually test the changes. Does running tox also build the execuatable borgmatic? Or can I just run borgmatic straight after making changes to the code?

Any guidance on how to manually test would be really helpful, Thanks

Hey witten Thanks for the heads up! I've been working with Vorta, so I'm fairly new to the Borgmatic codebase. Once I fully understand the relevant parts needed for this implementation, I’ll go ahead and create a PR and explain the process. For now, I’m thinking of looking into how `create` is implemented and using that as inspiration for `recreate`. I’ll keep you updated on my progress! I do have a question, it's about testing the changes. From my current understanding, running `tox` will run the unit tests, right? I’m wondering how to manually test the changes. Does running `tox` also build the execuatable `borgmatic`? Or can I just run `borgmatic` straight after making changes to the code? Any guidance on how to manually test would be really helpful, Thanks
120EE0980 commented 2025-03-16 08:53:38 +00:00
Contributor
Copy Link

After I make changes, I run(to build editable) pip install -e . and then tox to test across all python versions in tox.ini file.

After I make changes, I run(to build editable)` pip install -e .` and then `tox` to test across all python versions in tox.ini file.
👍 1
witten commented 2025-03-16 17:43:59 +00:00
Owner
Copy Link

Hey witten
Thanks for the heads up! I've been working with Vorta, so I'm fairly new to the Borgmatic codebase. Once I fully understand the relevant parts needed for this implementation, I’ll go ahead and create a PR and explain the process.

For now, I’m thinking of looking into how create is implemented and using that as inspiration for recreate. I’ll keep you updated on my progress!

Sounds good! Just be aware that create is one of the most complex actions in borgmatic. So while it's probably the closest to recreate, it may also help to look at some of the simpler actions as well.

I do have a question, it's about testing the changes. From my current understanding, running tox will run the unit tests, right? I’m wondering how to manually test the changes. Does running tox also build the execuatable borgmatic? Or can I just run borgmatic straight after making changes to the code?

@120EE0980 is correct. Installing editable (via pip install -e or pipx install -e) and then running tox will indeed run tests. But to manually test your changes, yes, you should be able to just run borgmatic right after making modifications to the code. This doesn't have anything to do with tox though.. It's the magic of that editable install that lets you run your changes directly.

> Hey witten > Thanks for the heads up! I've been working with Vorta, so I'm fairly new to the Borgmatic codebase. Once I fully understand the relevant parts needed for this implementation, I’ll go ahead and create a PR and explain the process. > > For now, I’m thinking of looking into how `create` is implemented and using that as inspiration for `recreate`. I’ll keep you updated on my progress! Sounds good! Just be aware that `create` is one of the most complex actions in borgmatic. So while it's probably the closest to `recreate`, it may also help to look at some of the simpler actions as well. > I do have a question, it's about testing the changes. From my current understanding, running `tox` will run the unit tests, right? I’m wondering how to manually test the changes. Does running `tox` also build the execuatable `borgmatic`? Or can I just run `borgmatic` straight after making changes to the code? @120EE0980 is correct. Installing editable (via `pip install -e` or `pipx install -e`) and then running `tox` will indeed run tests. But to manually test your changes, yes, you should be able to just run `borgmatic` right after making modifications to the code. This doesn't have anything to do with `tox` though.. It's the magic of that editable install that lets you run your changes directly.
VandalByte commented 2025-03-16 19:56:30 +00:00
Contributor
Copy Link

Got it! Thanks guys

Also witten, can I consider this as my borgmatic project for the proposal? This is like the subset of what I had in mind, but now that I looked into this a bit more, I think with all the additions and tests to be written, it's a small project on it's own.

Once I finish this I'll have a good understanding of adding borgmatic actions, then I can proceed to add the rest of actions, instead of jumping the gun, I think that will be a smoother transition.

Got it! Thanks guys Also witten, can I consider this as my borgmatic project for the proposal? This is like the subset of what I had in [mind]([url](https://github.com/borgbase/vorta/wiki/Google-Summer-of-Code-2025-Ideas#wrap-all-borg-sub-commands-with-borgmatic-actions)), but now that I looked into this a bit more, I think with all the additions and tests to be written, it's a small project on it's own. Once I finish this I'll have a good understanding of adding borgmatic actions, then I can proceed to add the rest of actions, instead of jumping the gun, I think that will be a smoother transition.
witten commented 2025-03-16 21:26:31 +00:00
Owner
Copy Link

My thinking is that any relevant project can be put into a proposal, not just one of the ideas listed on the GSoC ideas page. So what I'll do is remove the "good first issue" tag because I agree that it's a little bit bigger of a project even if it's (mostly) straightforward. In terms of moving forward on GSoC though, just make sure you're going through the steps listed here. (It sounds like you've already done some of them with Vorta.)

Thanks!

My thinking is that any relevant project can be put into a proposal, not just one of the ideas listed [on the GSoC ideas page](https://github.com/borgbase/vorta/wiki/Google-Summer-of-Code-2025-Ideas). So what I'll do is remove the "good first issue" tag because I agree that it's a little bit bigger of a project even if it's (mostly) straightforward. In terms of moving forward on GSoC though, just make sure you're going through the [steps listed here](https://github.com/borgbase/vorta/wiki/Google-Summer-of-Code-2025-Ideas#getting-started). (It sounds like you've already done some of them with Vorta.) Thanks!
👍 1
witten removed the good first issue label 2025-03-16 21:26:39 +00:00
witten commented 2025-03-29 22:08:22 +00:00
Owner
Copy Link

This is implemented by @VandalByte in main and will be part of the next release!

This is implemented by @VandalByte in main and will be part of the next release!
👍 1
witten commented 2025-04-06 15:45:41 +00:00
Owner
Copy Link

Released in borgmatic 2.0.0!

Released in borgmatic 2.0.0!
🎉 1
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
blocked
breaking
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
No Label new feature area
Milestone
No items
No Milestone
Assignees
Clear assignees
witten
No Assignees
5 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#610
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?