Cap backups by size #25

Closed
opened 2018-01-05 05:34:01 +00:00 by import_bot · 5 comments
Collaborator

It would be nice (although I guess it would be some work without direct support from borg) to have support for capping the number of backups by size.

I'll explain myself better and provide an hypothetical algorithm.

The example config file on the borgmatic website has:

keep_daily: 7
keep_weekly: 4
keep_monthly: 6

Which is lovely! Except for one thing: if I've got plenty of space I'd like to keep the highest possible number of monthly snapshots that fit.

So I'd love to say:

keep_daily: 7
keep_weekly: 4
keep_monthly: up to 300GB, but at least 6 anyway

Now, the algorithm would be: when we go above quota we simply drop old versions until we either reach the required minimum or we get below quota again.

Due to deduplication we can't know in advance how much to drop.


Imported from Taiga issue 24 (to do). Created on 2016-07-10T17:01:19+0000 by Leonardo Taglialegne.

It would be nice (although I guess it would be some work without direct support from borg) to have support for capping the number of backups by size. I'll explain myself better and provide an hypothetical algorithm. The example config file on the borgmatic website has: keep_daily: 7 keep_weekly: 4 keep_monthly: 6 Which is lovely! Except for one thing: if I've got plenty of space I'd like to keep the highest possible number of monthly snapshots that fit. So I'd love to say: keep_daily: 7 keep_weekly: 4 keep_monthly: up to 300GB, but at least 6 anyway Now, the algorithm would be: when we go above quota we simply drop old versions until we either reach the required minimum or we get below quota again. Due to deduplication we can't know in advance how much to drop. --- Imported from Taiga issue 24 (to do). Created on 2016-07-10T17:01:19+0000 by Leonardo Taglialegne.
Author
Collaborator

Indeed, that would be a pretty cool feature. A variant of that idea: Just specify the quota separately from all the keep_ options, and somehow have borgmatic adjust the number of archives kept across all the intervals to stay below that quota.


Comment on 2016-07-10T17:44:19+0000 by Dan Helfman.

Indeed, that would be a pretty cool feature. A variant of that idea: Just specify the quota separately from all the `keep_` options, and somehow have borgmatic adjust the number of archives kept across all the intervals to stay below that quota. --- Comment on 2016-07-10T17:44:19+0000 by Dan Helfman.
Author
Collaborator

It seems to be difficult to manage inside borgmatic because it doesn't know which chunks should be removed - except by date, which is a bit rude -, and more importantly can't remove them during borg create.

As a (not so reliable) workaround, borgmatic could ask (store in a cache for the sake of speed?) the size of the last n backups in order to derive a conservative estimation of the size of the current backup, and prune the oldest ones until a sufficient amount of disk space is available. It seems to be a very important feature, as borg doesn't handle full disks situation well.

Btw, I see the source code of borgmatic calls create comes before prune (`command.py:51-54). IMHO they should be swapped, in order to make room with prune before creating the new backup. That would limit the disk space, and potentially avoid conditions that could break Borg.


Comment on 2016-09-13T10:24:29+0000 by Aurel G.

It seems to be difficult to manage inside borgmatic because it doesn't know which chunks should be removed - except by date, which is a bit rude -, and more importantly can't remove them during `borg create`. As a (not so reliable) workaround, borgmatic could ask (store in a cache for the sake of speed?) the size of the last n backups in order to derive a conservative estimation of the size of the current backup, and prune the oldest ones until a sufficient amount of disk space is available. It seems to be a very important feature, as borg doesn't handle full disks situation well. Btw, I see the source code of borgmatic calls `create` comes before `prune` (`command.py:51-54). IMHO they should be swapped, in order to make room with prune before creating the new backup. That would limit the disk space, and potentially avoid conditions that could break Borg. --- Comment on 2016-09-13T10:24:29+0000 by Aurel G.
Author
Collaborator

The next borgmatic release (1.1.0) will prune before create as you suggested.


Comment on 2017-07-23T04:50:49+0000 by Dan Helfman.

The next borgmatic release (1.1.0) will `prune` before `create` as you suggested. --- Comment on 2017-07-23T04:50:49+0000 by Dan Helfman.
Owner

Note that there is a borg init --storage-quota parameter. It's conceivable that using that underlying Borg quota support could drive the feature described in this ticket. For instance, if borgmatic is trying to create an archive and hits an over-quota error, it could try to opportunistically prune archives until getting back under the quota.

Note that there is a `borg init --storage-quota` parameter. It's conceivable that using that underlying Borg quota support could drive the feature described in this ticket. For instance, if borgmatic is trying to create an archive and hits an over-quota error, it could try to opportunistically prune archives until getting back under the quota.
Owner

Closing due to inactivity, but I'd be happy to reopen if there's still user interest.

Closing due to inactivity, but I'd be happy to reopen if there's still user interest.
Sign in to join this conversation.
No Milestone
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#25
No description provided.