Add Borg 2.0.0b17 features #1098

New issue

Open

opened 2025-05-23 05:20:03 +00:00 by witten · 7 comments

witten commented 2025-05-23 05:20:03 +00:00

Owner

Copy link

What I'd like to do and why

Support relevant Borg 2.0.0b17 features and changes natively in borgmatic:

• transfer: implement --chunker-params to re-chunk while transferring
  • See borgmatic's create and recreate actions (which already have --chunker-params) for inspiration.
• list --depth=N: list files up to N depth in path hierarchy
  • Decide whether this should be an option and a flag or just a flag. Maybe look at the existing list flags/options for inspiration.
• BORG_REPO_PERMISSIONS=all|no-delete|write-only|read-only
  • See the changelog for more info. You may also have to dig into the ticket.
• compact: add --iec option

Other notes / implementation ideas

No response

### What I'd like to do and why Support relevant [Borg 2.0.0b17 features and changes](https://borgbackup.readthedocs.io/en/2.0.0b17/changes.html#version-2-0-0b17-2025-05-23) natively in borgmatic: * [ ] transfer: implement --chunker-params to re-chunk while transferring * See borgmatic's `create` and `recreate` actions (which already have `--chunker-params`) for inspiration. * [ ] list --depth=N: list files up to N depth in path hierarchy * Decide whether this should be an option *and* a flag or just a flag. Maybe look at the existing `list` flags/options for inspiration. * [ ] BORG_REPO_PERMISSIONS=all|no-delete|write-only|read-only * See the changelog for more info. You may also have to dig into [the ticket](https://github.com/borgbackup/borg/issues/8823). * [ ] compact: add --iec option ### Other notes / implementation ideas _No response_

witten added the

good first issue

label 2025-05-23 05:20:20 +00:00

wrexor commented 2025-10-08 18:45:37 +00:00

Copy link

I thought I'd look into getting the no-delete permission to work with my setup but of course nothing is ever simple :)

The borg rename command needs delete permission, consequently borgmatic create doesn't work due to the way streaming process failures are handled (the archive is created with a suffix and renamed upon success).

I'm already using a pull model to prune and compact so having it do the create as well is trivial, but I'd rather let the client initiate its own backups.

Is there any other way to deal with failed streams without using rename?

I thought I'd look into getting the no-delete permission to work with my setup but of course nothing is ever simple :) The borg rename command needs delete permission, consequently borgmatic create doesn't work due to the way streaming process failures are handled (the archive is created with a suffix and renamed upon success). I'm already using a pull model to prune and compact so having it do the create as well is trivial, but I'd rather let the client initiate its own backups. Is there any other way to deal with failed streams without using rename?

witten commented 2025-10-08 20:24:09 +00:00

Author

Owner

Copy link

I thought I'd look into getting the no-delete permission to work with my setup but of course nothing is ever simple :)

The borg rename command needs delete permission, consequently borgmatic create doesn't work due to the way streaming process failures are handled (the archive is created with a suffix and renamed upon success).

I'm already using a pull model to prune and compact so having it do the create as well is trivial, but I'd rather let the client initiate its own backups.

Is there any other way to deal with failed streams without using rename?

Ugh, that's obnoxious! When designing the solution for dealing with failed database dumps, we looked at a bunch of other options, but the rename approach was the only one that ended up panning out in the end. See #1032 if you're interested in the full history.

In terms of how to proceed with this ticket, I think my inclination is for borgmatic to not do the checkpoint + rename whenever no-delete is enabled.

> I thought I'd look into getting the no-delete permission to work with my setup but of course nothing is ever simple :) > > The borg rename command needs delete permission, consequently borgmatic create doesn't work due to the way streaming process failures are handled (the archive is created with a suffix and renamed upon success). > > I'm already using a pull model to prune and compact so having it do the create as well is trivial, but I'd rather let the client initiate its own backups. > > Is there any other way to deal with failed streams without using rename? Ugh, that's obnoxious! When designing the solution for dealing with failed database dumps, we looked at a bunch of other options, but the rename approach was the only one that ended up panning out in the end. See #1032 if you're interested in the full history. In terms of how to proceed with this ticket, I think my inclination is for borgmatic to *not* do the checkpoint + rename whenever `no-delete` is enabled.

witten removed the

good first issue

label 2025-10-08 20:24:18 +00:00

witten referenced this issue 2025-10-08 21:02:28 +00:00

Archive gets created despite failed database hook #1032

wrexor commented 2025-10-09 01:16:07 +00:00

Copy link

Not doing the checkpoint would work but the problem of pruning good archives in favor of bad ones would still exist. I would at least need to know when a stream fails so I could deal with it manually somehow. I also wonder if rename really needs the delete permission but that's way above my pay grade

Seems like the magic pipe solution might still be viable and almost working? I'm already neck deep on a tangent, but once I get some free cycles I might give that a go if no one beats me to it

Not doing the checkpoint would work but the problem of pruning good archives in favor of bad ones would still exist. I would at least need to know when a stream fails so I could deal with it manually somehow. I also wonder if rename really needs the delete permission but that's way above my pay grade Seems like the magic pipe solution might still be viable and almost working? I'm already neck deep on a tangent, but once I get some free cycles I might give that a go if no one beats me to it

witten commented 2025-10-09 01:55:06 +00:00

Author

Owner

Copy link

Not doing the checkpoint would work but the problem of pruning good archives in favor of bad ones would still exist. I would at least need to know when a stream fails so I could deal with it manually somehow.

Yeah, fortunately borgmatic does error when a database stream fails—it just won't cleanup the partial archive if that checkpoint/rename feature is disabled.

I also wonder if rename really needs the delete permission but that's way above my pay grade

Mine too. 😄 You could ask in the #borgbackup IRC channel on libera.chat on you could file a Borg issue/discussion about it.

Seems like the magic pipe solution might still be viable and almost working? I'm already neck deep on a tangent, but once I get some free cycles I might give that a go if no one beats me to it

The sentinel named pipe? IIRC, I wasn't actually able to get that approach working in prototyping. You're welcome to try again, but it might just be easier to deal with the occasional partial archives since you should find out when such errors occur.

> Not doing the checkpoint would work but the problem of pruning good archives in favor of bad ones would still exist. I would at least need to know when a stream fails so I could deal with it manually somehow. Yeah, fortunately borgmatic [does error](https://projects.torsion.org/borgmatic-collective/borgmatic/issues/1032) when a database stream fails—it just won't cleanup the partial archive if that checkpoint/rename feature is disabled. > I also wonder if rename really needs the delete permission but that's way above my pay grade Mine too. 😄 You could ask in the #borgbackup IRC channel on libera.chat on you could file a [Borg issue/discussion](https://github.com/borgbackup/borg/) about it. > Seems like the magic pipe solution might still be viable and almost working? I'm already neck deep on a tangent, but once I get some free cycles I might give that a go if no one beats me to it The sentinel named pipe? IIRC, I wasn't actually able to get that approach working in prototyping. You're welcome to try again, but it might just be easier to deal with the occasional partial archives since you should find out when such errors occur.

wrexor commented 2025-10-09 02:15:00 +00:00

Copy link

The sentinel named pipe? IIRC, I wasn't actually able to get that approach working in prototyping. You're welcome to try again, but it might just be easier to deal with the occasional partial archives since you should find out when such errors occur.

I assumed they were all the same thing, but the OP of #1032 attempted a FIFO pipe solution and I got the impression it might still be feasible even though I'm a pessimist.

But yes, much easier to just disable it and deal with the errors. I guess I just need to figure out how to tell if no_delete is set

> The sentinel named pipe? IIRC, I wasn't actually able to get that approach working in prototyping. You're welcome to try again, but it might just be easier to deal with the occasional partial archives since you should find out when such errors occur. I assumed they were all the same thing, but the OP of #1032 attempted a FIFO pipe solution and I got the impression it might still be feasible even though I'm a pessimist. But yes, much easier to just disable it and deal with the errors. I guess I just need to figure out how to tell if _no_delete_ is set

witten commented 2025-10-09 05:07:00 +00:00

Author

Owner

Copy link

The sentinel named pipe? IIRC, I wasn't actually able to get that approach working in prototyping. You're welcome to try again, but it might just be easier to deal with the occasional partial archives since you should find out when such errors occur.

I assumed they were all the same thing, but the OP of #1032 attempted a FIFO pipe solution and I got the impression it might still be feasible even though I'm a pessimist.

Yup, the sentinel named pipe / FIFO pipe solution is all the same thing. Maybe it's feasible, but yeah it would take some more poking to find out.

But yes, much easier to just disable it and deal with the errors. I guess I just need to figure out how to tell if no_delete is set

So I'm assuming BORG_REPO_PERMISSIONS would be represented in borgmatic with a configuration file option. If that's the case, then it might be as easy as checking that config.get('borg_repo_permissions') == 'no-delete' or whatever, assuming that the configuration has already been loaded into an available config dict (which is the case almost everywhere in borgmatic).

> > The sentinel named pipe? IIRC, I wasn't actually able to get that approach working in prototyping. You're welcome to try again, but it might just be easier to deal with the occasional partial archives since you should find out when such errors occur. > > I assumed they were all the same thing, but the OP of #1032 attempted a FIFO pipe solution and I got the impression it might still be feasible even though I'm a pessimist. Yup, the sentinel named pipe / FIFO pipe solution is all the same thing. Maybe it's feasible, but yeah it would take some more poking to find out. > But yes, much easier to just disable it and deal with the errors. I guess I just need to figure out how to tell if _no_delete_ is set So I'm assuming `BORG_REPO_PERMISSIONS` would be represented in borgmatic with a configuration file option. If that's the case, then it might be as easy as checking that `config.get('borg_repo_permissions') == 'no-delete'` or whatever, assuming that the configuration has already been loaded into an available `config` dict (which is the case almost everywhere in borgmatic).

gzm0 commented 2025-10-11 12:09:26 +00:00

Contributor

Copy link

TY for the mention on #1032 @witten. Just drawing your attention to:

#1032 (comment)

IMHO the sentinel named pipe solution is prone to deadlocking between borg / borgmatic (but that's just a gut feeling).

TY for the mention on #1032 @witten. Just drawing your attention to: https://projects.torsion.org/borgmatic-collective/borgmatic/issues/1032#issuecomment-11308 IMHO the sentinel named pipe solution is prone to deadlocking between borg / borgmatic (but that's just a gut feeling).

👍 1

witten added the

waiting for response

label 2025-10-15 22:21:55 +00:00

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

parallel-borgmatic

command-hook-steps

2.1.7

2.1.6

2.1.5

2.1.4

2.1.3

2.1.2

2.1.1

2.1.0

2.0.13

2.0.12

2.0.11

2.0.10

2.0.9

2.0.8

2.0.7

2.0.6

2.0.5

2.0.4

2.0.3

2.0.2

2.0.1

2.0.0

1.9.14

1.9.13

1.9.12

1.9.11

1.9.10

1.9.9

1.9.8

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.14

1.8.13

1.8.12

1.8.11

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.24

1.5.23

1.5.22

1.5.21

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.14

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.7.dev0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.22

1.4.21

1.4.20

1.4.19

1.4.18

1.4.17

1.4.16

1.4.15

1.4.14

1.4.13

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.26

1.3.25

1.3.24

1.3.23

1.3.22

1.3.21

1.3.20

1.3.19

1.3.18

1.3.17

1.3.16

1.3.15

1.3.14

1.3.13

1.3.12

1.3.11

1.3.10

1.3.9

1.3.8

1.3.7

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.18

1.2.17

1.2.16

1.2.15

1.2.14

1.2.13

1.2.12

1.2.11

1.2.10

1.2.9

1.2.8

1.2.7

1.2.6

1.2.5

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.15

1.1.14

1.1.13

1.1.12

1.1.11

1.1.10

1.1.9

1.1.7

1.1.6

1.1.5

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.3

1.0.2

1.0.1

1.0.0

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

0.0.2

Labels

Clear labels   

blocked

  

breaking

  

bug

  

data loss

  

design finalized

  

good first issue

  

new feature area

  

question / support

  

security

  

waiting for response

No labels

blocked

breaking

bug

data loss

design finalized

good first issue

new feature area

question / support

security

waiting for response

Milestone

Clear milestone

No items

No milestone

Assignees

Clear assignees

No assignees

3 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

borgmatic-collective/borgmatic#1098

No description provided.