Borgmatic does not prune #37
Labels
No Label
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
No Milestone
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: borgmatic-collective/borgmatic#37
Loading…
Reference in New Issue
No description provided.
Delete Branch "%!s(<nil>)"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Borgmatic seemingly refuses to prune. Running sudo borgmatic --prune --verbosity 2 -c /etc/borgmatic.d/eon.yaml does not result in any changes and no archives are pruned during the daily backup run.
Pruning with borg works perfectly, using the following command:
sudo borg prune -s --info --list -d 7 -w 4 -m 1 /mnt/nas/eon
I have attached the config file. I can't find any references to similar problems using a Google search. Any suggestions to solve this problem are welcome.
Imported from Taiga issue 36 (done). Created on 2017-08-15T14:43:38+0000 by ot3.
Hi ot3,
I don't see anything obvious that might cause this based on your config. But I did notice that you are specifying the retention "prefix" in the borgmatic config, but you aren't specifying any prefix when invoking borg prune directly. Have you tried adding that prefix, to see if borg itself still prunes? And are you sure that the "eon" prefix matches the generated archive names that borgmatic produces?
Comment on 2017-08-16T03:37:49+0000 by Dan Helfman.
Also, assuming it doesn't have any secrets in it, attaching the output of your
borgmatic prune -v 2
run may be helpful for debugging.Comment on 2017-08-16T03:40:33+0000 by Dan Helfman.
Hi Dan,
Thanks for getting back to me. I have tried this with and without the prefix; doesn't make any difference. In fact I added the prefix to see if that solved the problem.
Here's the requested output of sudo borgmatic --prune --verbosity 2 -c /etc/borgmatic.d/eon.yaml:
Apparently nothing happens.
Here's borg doing the pruning; done immediately after the previous command:
I also should mention that I'm running Ubuntu 17.04 (GNU/Linux 4.10.0-28-generic x86_64)
Comment on 2017-08-16T09:56:20+0000 by ot3.
Some more ideas: It appears as if the exact retention parameters you're passing to borgmatic and borg, respectively, are a little different. Here's the relevant section of your borgmatic config (comments removed):
However, for your borg command-line, you're providing:
-d 7 -w 4 -m 1
.The delta between the two being:
keep_within
andkeep_yearly
. Also, yourkeep_monthly
value is 6 in borgmatic instead of the 1 passed to borg.So I could imagine that borgmatic is holding onto more archives because it's being told to retain more months. Plus keep_within could be having a similar effect.
So, based on that theory, try passing
-m 6
instead of-m 1
to borg, and also tack on--keep-within 3H
and-y 1
while you're at it. Then, I would expect both borgmatic and borg to have the same pruning behavior. Alternatively, you could change the borgmatic config to match what you're passing to borg: Comment outkeep_within
,keep_yearly
, and setkeep_monthly
to 1.Let me know how that goes.
Comment on 2017-08-17T04:15:20+0000 by Dan Helfman.
Tentative success. Here's what I did:
I followed up with the borg command:
Looking at the list of archives, this is to be expected as apparently the 2017-08-10 archive (Thursday) was somehow pruned. Next I looked at the borgmatic logfile from the nightly backup:
I don't see any pruning going on here, yet Thursday is gone. Somehow the 2017-08-10 archive was pruned without leaving a trace in the logfile or in the output of the commands I later ran manually.EDIT: I expected the logfile to show the actual pruning, just as it did when running the borg command but apparently it only shows the mount of data deleted which I overlooked at first.
As an experiment I changed the retention policy from 7 to 6 days and ran the following commands:
This time it worked.
Tentative conclusion:
Leaving a number of default parameters in the borgmatic retention policy did not result in the expected pruning behavior:Original retention policy:
Current retention policy
I don't know if the behavior as shown is by design, but it might help others in a similar situation to point this out in the docs. I'll report tomorrow if pruning occurs dring standard operation (cron) but I assume it will. Many thanks for your assistance in this matter.
Comment on 2017-08-17T10:56:55+0000 by ot3.
Glad to hear it appears to be working now. I'll update the docs and also look into listing the archives when pruning with
-v 2
, not just providing the summary.Comment on 2017-08-18T04:00:45+0000 by Dan Helfman.
Unfortunately we're not there yet; very puzzling behavior.
Current retention policy [still using yesterday's 'keep_daily: 6' for testing purposes]:
From the mailed borgmatic log as created by cron; backup OK but no pruning:
Running borg to check on the remaining archives; Saturday was not pruned:
Running the same borgmatic pruning command as yesterday; pruning OK
Running borg to check on the remaining archives; Saturday was pruned:
So here we have it: running borgmatic from cron results in a backup without pruning while running the pruning command manually works fine. Normally in case of cron problems, this points to an environment problem. Does this give you a clue? Does the borgmatic code set it's environment to Python3?
Comment on 2017-08-18T10:29:48+0000 by ot3.
Borgmatic runs pruning before creating any archives. (This is a relatively recent change.) The rationale is that pruning may free up space needed to create the archive.
So what could be happening here is that borgmatic, when run by cron without any action flags like
--prune
, will:It's entirely possible that after the archive is created during the cron job, there are now 7 daily archives instead of 6. And so the next time a prune is run (whether manually or on the next cron invocation), the oldest daily archive will get pruned.
Based on your logs, that looks plausible.
Comment on 2017-08-19T03:05:26+0000 by Dan Helfman.
I'll leave the backup routines untouched for a few days and see what happens.
Comment on 2017-08-19T09:45:19+0000 by ot3.
Any luck?
Comment on 2017-08-26T23:12:47+0000 by Dan Helfman.
I was going to report back tomorrow but will do it now [I wanted one more day].
Current retention policy:
Archives:
Conclusion:
Will the same happen to the weekly and monthly archives?
Comment on 2017-08-27T13:05:48+0000 by ot3.
Thanks for reporting back. I've updated the documentation to reflect this perhaps unexpected characteristic: "Note that borgmatic prunes archives before creating an archive, so as to free up space for archiving. This means that when a borgmatic run finishes, there may still be prune-able archives. Not to worry, as they will get cleaned up at the start of the next run."
And yes, weekly and monthly archives work the same way. Although note that not every borgmatic run creates an archive that would be pruned by weekly or monthly retention policies.
So I'll close this for now, but feel free to re-open or re-file if you see other problems with this.
Comment on 2017-08-27T17:19:20+0000 by Dan Helfman.