Borgmatic does not prune #37

Closed
opened 2018-01-05 05:34:27 +00:00 by import_bot · 12 comments
Collaborator

Borgmatic seemingly refuses to prune. Running sudo borgmatic --prune --verbosity 2 -c /etc/borgmatic.d/eon.yaml does not result in any changes and no archives are pruned during the daily backup run.
Pruning with borg works perfectly, using the following command:

sudo borg prune -s --info --list -d 7 -w 4 -m 1 /mnt/nas/eon

I have attached the config file. I can't find any references to similar problems using a Google search. Any suggestions to solve this problem are welcome.


Imported from Taiga issue 36 (done). Created on 2017-08-15T14:43:38+0000 by ot3.

Borgmatic seemingly refuses to prune. Running sudo borgmatic --prune --verbosity 2 -c /etc/borgmatic.d/eon.yaml does not result in any changes and no archives are pruned during the daily backup run. Pruning with borg works perfectly, using the following command: sudo borg prune -s --info --list -d 7 -w 4 -m 1 /mnt/nas/eon I have attached the config file. I can't find any references to similar problems using a Google search. Any suggestions to solve this problem are welcome. --- Imported from Taiga issue 36 (done). Created on 2017-08-15T14:43:38+0000 by ot3.
import_bot added the
question / support
label 2018-01-05 05:34:27 +00:00
Author
Collaborator

Hi ot3,

I don't see anything obvious that might cause this based on your config. But I did notice that you are specifying the retention "prefix" in the borgmatic config, but you aren't specifying any prefix when invoking borg prune directly. Have you tried adding that prefix, to see if borg itself still prunes? And are you sure that the "eon" prefix matches the generated archive names that borgmatic produces?


Comment on 2017-08-16T03:37:49+0000 by Dan Helfman.

Hi ot3, I don't see anything obvious that might cause this based on your config. But I did notice that you are specifying the retention "prefix" in the borgmatic config, but you aren't specifying any prefix when invoking borg prune directly. Have you tried adding that prefix, to see if borg itself still prunes? And are you sure that the "eon" prefix matches the generated archive names that borgmatic produces? --- Comment on 2017-08-16T03:37:49+0000 by Dan Helfman.
Author
Collaborator

Also, assuming it doesn't have any secrets in it, attaching the output of your borgmatic prune -v 2 run may be helpful for debugging.


Comment on 2017-08-16T03:40:33+0000 by Dan Helfman.

Also, assuming it doesn't have any secrets in it, attaching the output of your `borgmatic prune -v 2` run may be helpful for debugging. --- Comment on 2017-08-16T03:40:33+0000 by Dan Helfman.
Author
Collaborator

Hi Dan,

Thanks for getting back to me. I have tried this with and without the prefix; doesn't make any difference. In fact I added the prefix to see if that solved the problem.

Here's the requested output of sudo borgmatic --prune --verbosity 2 -c /etc/borgmatic.d/eon.yaml:

pi@eon:[~]: sudo borgmatic --prune --verbosity 2 -c /etc/borgmatic.d/eon.yaml
using builtin fallback logging configuration
TAM-verified manifest
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
Deleted data:                    0 B                  0 B                  0 B
All archives:               43.23 GB             43.29 GB              5.89 GB

                       Unique chunks         Total chunks
Chunk index:                  103575              1308809
------------------------------------------------------------------------------
pi@eon:[~]:

Apparently nothing happens.

Here's borg doing the pruning; done immediately after the previous command:

pi@eon:[~]: sudo borg prune  -s --info --list -d 7 -w 4 -m 1 /mnt/nas/eon
Keeping archive: eon-2017-08-16T11:41:39.384681       Wed, 2017-08-16 11:41:39
Keeping archive: eon-2017-08-15T00:30:02.279744       Tue, 2017-08-15 00:30:02
Keeping archive: eon-2017-08-14T00:30:02.755813       Mon, 2017-08-14 00:30:03
Keeping archive: eon-2017-08-13T00:30:02.524766       Sun, 2017-08-13 00:30:02
Keeping archive: eon-2017-08-12T00:30:02.015646       Sat, 2017-08-12 00:30:02
Keeping archive: eon-2017-08-11T00:30:02.791884       Fri, 2017-08-11 00:30:03
Keeping archive: eon-2017-08-10T00:30:02.527576       Thu, 2017-08-10 00:30:02
Keeping archive: eon-2017-08-06T00:30:02.240738       Sun, 2017-08-06 00:30:02
Pruning archive: eon-2017-08-09T00:30:02.459540       Wed, 2017-08-09 00:30:02
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
Deleted data:               -4.59 GB             -4.60 GB           -103.82 MB
All archives:               38.64 GB             38.69 GB              5.78 GB

                       Unique chunks         Total chunks
Chunk index:                  103343              1174442
------------------------------------------------------------------------------
pi@eon:[~]:

I also should mention that I'm running Ubuntu 17.04 (GNU/Linux 4.10.0-28-generic x86_64)


Comment on 2017-08-16T09:56:20+0000 by ot3.

Hi Dan, Thanks for getting back to me. I have tried this with and without the prefix; doesn't make any difference. In fact I added the prefix to see if that solved the problem. Here's the requested output of sudo borgmatic --prune --verbosity 2 -c /etc/borgmatic.d/eon.yaml: ``` pi@eon:[~]: sudo borgmatic --prune --verbosity 2 -c /etc/borgmatic.d/eon.yaml using builtin fallback logging configuration TAM-verified manifest ------------------------------------------------------------------------------ Original size Compressed size Deduplicated size Deleted data: 0 B 0 B 0 B All archives: 43.23 GB 43.29 GB 5.89 GB Unique chunks Total chunks Chunk index: 103575 1308809 ------------------------------------------------------------------------------ pi@eon:[~]: ``` Apparently nothing happens. Here's borg doing the pruning; done immediately after the previous command: ``` pi@eon:[~]: sudo borg prune -s --info --list -d 7 -w 4 -m 1 /mnt/nas/eon Keeping archive: eon-2017-08-16T11:41:39.384681 Wed, 2017-08-16 11:41:39 Keeping archive: eon-2017-08-15T00:30:02.279744 Tue, 2017-08-15 00:30:02 Keeping archive: eon-2017-08-14T00:30:02.755813 Mon, 2017-08-14 00:30:03 Keeping archive: eon-2017-08-13T00:30:02.524766 Sun, 2017-08-13 00:30:02 Keeping archive: eon-2017-08-12T00:30:02.015646 Sat, 2017-08-12 00:30:02 Keeping archive: eon-2017-08-11T00:30:02.791884 Fri, 2017-08-11 00:30:03 Keeping archive: eon-2017-08-10T00:30:02.527576 Thu, 2017-08-10 00:30:02 Keeping archive: eon-2017-08-06T00:30:02.240738 Sun, 2017-08-06 00:30:02 Pruning archive: eon-2017-08-09T00:30:02.459540 Wed, 2017-08-09 00:30:02 ------------------------------------------------------------------------------ Original size Compressed size Deduplicated size Deleted data: -4.59 GB -4.60 GB -103.82 MB All archives: 38.64 GB 38.69 GB 5.78 GB Unique chunks Total chunks Chunk index: 103343 1174442 ------------------------------------------------------------------------------ pi@eon:[~]: ``` I also should mention that I'm running Ubuntu 17.04 (GNU/Linux 4.10.0-28-generic x86_64) --- Comment on 2017-08-16T09:56:20+0000 by ot3.
Author
Collaborator

Some more ideas: It appears as if the exact retention parameters you're passing to borgmatic and borg, respectively, are a little different. Here's the relevant section of your borgmatic config (comments removed):

retention:
    keep_within: 3H
    keep_hourly: 24
    keep_daily: 7
    keep_weekly: 4
    keep_monthly: 6
    keep_yearly: 1

However, for your borg command-line, you're providing: -d 7 -w 4 -m 1.

The delta between the two being: keep_within and keep_yearly. Also, your keep_monthly value is 6 in borgmatic instead of the 1 passed to borg.

So I could imagine that borgmatic is holding onto more archives because it's being told to retain more months. Plus keep_within could be having a similar effect.

So, based on that theory, try passing -m 6 instead of -m 1 to borg, and also tack on --keep-within 3H and -y 1 while you're at it. Then, I would expect both borgmatic and borg to have the same pruning behavior. Alternatively, you could change the borgmatic config to match what you're passing to borg: Comment out keep_within, keep_yearly, and set keep_monthly to 1.

Let me know how that goes.


Comment on 2017-08-17T04:15:20+0000 by Dan Helfman.

Some more ideas: It appears as if the exact retention parameters you're passing to borgmatic and borg, respectively, are a little different. Here's the relevant section of your borgmatic config (comments removed): ``` retention: keep_within: 3H keep_hourly: 24 keep_daily: 7 keep_weekly: 4 keep_monthly: 6 keep_yearly: 1 ``` However, for your borg command-line, you're providing: `-d 7 -w 4 -m 1`. The delta between the two being: `keep_within` and `keep_yearly`. Also, your `keep_monthly` value is 6 in borgmatic instead of the 1 passed to borg. So I could imagine that borgmatic is holding onto more archives because it's being told to retain more months. Plus keep_within could be having a similar effect. So, based on that theory, try passing `-m 6` instead of `-m 1` to borg, and also tack on `--keep-within 3H` and `-y 1` while you're at it. Then, I would expect both borgmatic and borg to have the same pruning behavior. Alternatively, you could change the borgmatic config to match what you're passing to borg: Comment out `keep_within`, `keep_yearly`, and set `keep_monthly` to 1. Let me know how that goes. --- Comment on 2017-08-17T04:15:20+0000 by Dan Helfman.
Author
Collaborator

Tentative success. Here's what I did:

  • As suggested, I changed the borgmatic retention policy to be identical to the one I used in the borg command: -d 7 -w 4 -m 1. I commented out all other options. I didn't occur to me that these could cause problems as they were part of the default configuration.
  • I then ran the borgmatic prune command with the modified retention policy; no pruning occurred.
pi@eon:[~]: !1858
sudo borgmatic --prune --verbosity 2 -c /etc/borgmatic.d/eon.yaml
using builtin fallback logging configuration
TAM-verified manifest
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
Deleted data:               -4.58 GB             -4.59 GB           -167.27 MB
All archives:               39.02 GB             39.08 GB              5.81 GB

                       Unique chunks         Total chunks
Chunk index:                  103412              1194531
------------------------------------------------------------------------------
pi@eon:[~]: !1945
sudo borgmatic --prune --verbosity 2 -c /etc/borgmatic.d/eon.yaml
using builtin fallback logging configuration
TAM-verified manifest
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
Deleted data:                    0 B                  0 B                  0 B
All archives:               39.02 GB             39.08 GB              5.81 GB

                       Unique chunks         Total chunks
Chunk index:                  103412              1194531
------------------------------------------------------------------------------

I followed up with the borg command:

pi@eon:[~]: sudo borg prune -s --info --list -d 7 -w 4 -m 1 /mnt/nas/eon
Keeping archive: eon-2017-08-17T00:30:02.646158       Thu, 2017-08-17 00:30:03
Keeping archive: eon-2017-08-16T11:41:39.384681       Wed, 2017-08-16 11:41:39
Keeping archive: eon-2017-08-15T00:30:02.279744       Tue, 2017-08-15 00:30:02
Keeping archive: eon-2017-08-14T00:30:02.755813       Mon, 2017-08-14 00:30:03
Keeping archive: eon-2017-08-13T00:30:02.524766       Sun, 2017-08-13 00:30:02
Keeping archive: eon-2017-08-12T00:30:02.015646       Sat, 2017-08-12 00:30:02
Keeping archive: eon-2017-08-11T00:30:02.791884       Fri, 2017-08-11 00:30:03
Keeping archive: eon-2017-08-06T00:30:02.240738       Sun, 2017-08-06 00:30:02
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
Deleted data:                    0 B                  0 B                  0 B
All archives:               39.02 GB             39.08 GB              5.81 GB

                       Unique chunks         Total chunks
Chunk index:                  103412              1194531
------------------------------------------------------------------------------
pi@eon:[~]:

Looking at the list of archives, this is to be expected as apparently the 2017-08-10 archive (Thursday) was somehow pruned. Next I looked at the borgmatic logfile from the nightly backup:

------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
Deleted data:                    0 B                  0 B                  0 B
All archives:               38.64 GB             38.69 GB              5.78 GB

                       Unique chunks         Total chunks
Chunk index:                  103343              1174442
------------------------------------------------------------------------------
------------------------------------------------------------------------------
Archive name: eon-2017-08-17T00:30:02.646158
Archive fingerprint: 2a3064f237114c956442a07e6d0b6033d4c152ad32a03f8df07e7652d4ed3bb2
Time (start): Thu, 2017-08-17 00:30:03
Time (end):   Thu, 2017-08-17 00:30:50
Duration: 47.44 seconds
Number of files: 175059
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:                4.97 GB              4.97 GB            198.25 MB
All archives:               43.61 GB             43.67 GB              5.98 GB

                       Unique chunks         Total chunks
Chunk index:                  103678              1328907
------------------------------------------------------------------------------
Starting repository check
Checking segments 0.0%
Checking segments 0.1%
Checking segments 0.2%
...
Checking segments 99.7%
Checking segments 99.8%
Checking segments 99.9%

Completed repository check, no problems found.
Starting archive consistency check...
Analyzing archive eon-2017-08-17T00:30:02.646158 (9/9)
Analyzing archive eon-2017-08-16T11:41:39.384681 (8/9)
Analyzing archive eon-2017-08-15T00:30:02.279744 (7/9)
Orphaned objects check skipped (needs all archives checked).
Archive consistency check complete, no problems found.

I don't see any pruning going on here, yet Thursday is gone. Somehow the 2017-08-10 archive was pruned without leaving a trace in the logfile or in the output of the commands I later ran manually.

EDIT: I expected the logfile to show the actual pruning, just as it did when running the borg command but apparently it only shows the mount of data deleted which I overlooked at first.

As an experiment I changed the retention policy from 7 to 6 days and ran the following commands:

pi@eon:[~]: sudo borg list /mnt/nas/eon
eon-2017-08-06T00:30:02.240738       Sun, 2017-08-06 00:30:02
eon-2017-08-11T00:30:02.791884       Fri, 2017-08-11 00:30:03
eon-2017-08-12T00:30:02.015646       Sat, 2017-08-12 00:30:02
eon-2017-08-13T00:30:02.524766       Sun, 2017-08-13 00:30:02
eon-2017-08-14T00:30:02.755813       Mon, 2017-08-14 00:30:03
eon-2017-08-15T00:30:02.279744       Tue, 2017-08-15 00:30:02
eon-2017-08-16T11:41:39.384681       Wed, 2017-08-16 11:41:39
eon-2017-08-17T00:30:02.646158       Thu, 2017-08-17 00:30:03
pi@eon:[~]: sudo borgmatic --prune --verbosity 2 -c /etc/borgmatic.d/eon.yaml
using builtin fallback logging configuration
TAM-verified manifest
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
Deleted data:               -4.60 GB             -4.61 GB           -189.82 MB
All archives:               34.42 GB             34.47 GB              5.62 GB

                       Unique chunks         Total chunks
Chunk index:                  103104              1060132
------------------------------------------------------------------------------
pi@eon:[~]: sudo borg list /mnt/nas/eon
eon-2017-08-06T00:30:02.240738       Sun, 2017-08-06 00:30:02
eon-2017-08-12T00:30:02.015646       Sat, 2017-08-12 00:30:02
eon-2017-08-13T00:30:02.524766       Sun, 2017-08-13 00:30:02
eon-2017-08-14T00:30:02.755813       Mon, 2017-08-14 00:30:03
eon-2017-08-15T00:30:02.279744       Tue, 2017-08-15 00:30:02
eon-2017-08-16T11:41:39.384681       Wed, 2017-08-16 11:41:39
eon-2017-08-17T00:30:02.646158       Thu, 2017-08-17 00:30:03
pi@eon:[~]:

This time it worked. 

Tentative conclusion:

Leaving a number of default parameters in the borgmatic retention policy did not result in the expected pruning behavior:Original retention policy:

# Retention policy for how many backups to keep in each category. See
# https://borgbackup.readthedocs.org/en/stable/usage.html#borg-prune for details.
retention:
    # Keep all archives within this time interval.
    keep_within: 3H

    # Number of hourly archives to keep.
    keep_hourly: 24

    # Number of daily archives to keep.
    keep_daily: 6

    # Number of weekly archives to keep.
    keep_weekly: 4

    # Number of monthly archives to keep.
    keep_monthly: 6

    # Number of yearly archives to keep.
    keep_yearly: 1

    # When pruning, only consider archive names starting with this prefix.
    # prefix: eon

Current retention policy

# Retention policy for how many backups to keep in each category. See
# https://borgbackup.readthedocs.org/en/stable/usage.html#borg-prune for details.
retention:
    # Keep all archives within this time interval.
    #keep_within: 3H

    # Number of hourly archives to keep.
    #keep_hourly: 24

    # Number of daily archives to keep.
    keep_daily: 6

    # Number of weekly archives to keep.
    keep_weekly: 4

    # Number of monthly archives to keep.
    keep_monthly: 6

    # Number of yearly archives to keep.
    #keep_yearly: 1

    # When pruning, only consider archive names starting with this prefix.
    # prefix: eon

I don't know if the behavior as shown is by design, but it might help others in a similar situation to point this out in the docs. I'll report tomorrow if pruning occurs dring standard operation (cron) but I assume it will. Many thanks for your assistance in this matter.


Comment on 2017-08-17T10:56:55+0000 by ot3.

Tentative success. Here's what I did: * As suggested, I changed the borgmatic retention policy to be identical to the one I used in the borg command: -d 7 -w 4 -m 1\. I commented out all other options. I didn't occur to me that these could cause problems as they were part of the default configuration. * I then ran the borgmatic prune command with the modified retention policy; no pruning occurred. ``` pi@eon:[~]: !1858 sudo borgmatic --prune --verbosity 2 -c /etc/borgmatic.d/eon.yaml using builtin fallback logging configuration TAM-verified manifest ------------------------------------------------------------------------------ Original size Compressed size Deduplicated size Deleted data: -4.58 GB -4.59 GB -167.27 MB All archives: 39.02 GB 39.08 GB 5.81 GB Unique chunks Total chunks Chunk index: 103412 1194531 ------------------------------------------------------------------------------ pi@eon:[~]: !1945 sudo borgmatic --prune --verbosity 2 -c /etc/borgmatic.d/eon.yaml using builtin fallback logging configuration TAM-verified manifest ------------------------------------------------------------------------------ Original size Compressed size Deduplicated size Deleted data: 0 B 0 B 0 B All archives: 39.02 GB 39.08 GB 5.81 GB Unique chunks Total chunks Chunk index: 103412 1194531 ------------------------------------------------------------------------------ ``` I followed up with the borg command: ``` pi@eon:[~]: sudo borg prune -s --info --list -d 7 -w 4 -m 1 /mnt/nas/eon Keeping archive: eon-2017-08-17T00:30:02.646158 Thu, 2017-08-17 00:30:03 Keeping archive: eon-2017-08-16T11:41:39.384681 Wed, 2017-08-16 11:41:39 Keeping archive: eon-2017-08-15T00:30:02.279744 Tue, 2017-08-15 00:30:02 Keeping archive: eon-2017-08-14T00:30:02.755813 Mon, 2017-08-14 00:30:03 Keeping archive: eon-2017-08-13T00:30:02.524766 Sun, 2017-08-13 00:30:02 Keeping archive: eon-2017-08-12T00:30:02.015646 Sat, 2017-08-12 00:30:02 Keeping archive: eon-2017-08-11T00:30:02.791884 Fri, 2017-08-11 00:30:03 Keeping archive: eon-2017-08-06T00:30:02.240738 Sun, 2017-08-06 00:30:02 ------------------------------------------------------------------------------ Original size Compressed size Deduplicated size Deleted data: 0 B 0 B 0 B All archives: 39.02 GB 39.08 GB 5.81 GB Unique chunks Total chunks Chunk index: 103412 1194531 ------------------------------------------------------------------------------ pi@eon:[~]: ``` Looking at the list of archives, this is to be expected as apparently the 2017-08-10 archive (Thursday) was somehow pruned. Next I looked at the borgmatic logfile from the nightly backup: ``` ------------------------------------------------------------------------------ Original size Compressed size Deduplicated size Deleted data: 0 B 0 B 0 B All archives: 38.64 GB 38.69 GB 5.78 GB Unique chunks Total chunks Chunk index: 103343 1174442 ------------------------------------------------------------------------------ ------------------------------------------------------------------------------ Archive name: eon-2017-08-17T00:30:02.646158 Archive fingerprint: 2a3064f237114c956442a07e6d0b6033d4c152ad32a03f8df07e7652d4ed3bb2 Time (start): Thu, 2017-08-17 00:30:03 Time (end): Thu, 2017-08-17 00:30:50 Duration: 47.44 seconds Number of files: 175059 ------------------------------------------------------------------------------ Original size Compressed size Deduplicated size This archive: 4.97 GB 4.97 GB 198.25 MB All archives: 43.61 GB 43.67 GB 5.98 GB Unique chunks Total chunks Chunk index: 103678 1328907 ------------------------------------------------------------------------------ Starting repository check Checking segments 0.0% Checking segments 0.1% Checking segments 0.2% ... Checking segments 99.7% Checking segments 99.8% Checking segments 99.9% Completed repository check, no problems found. Starting archive consistency check... Analyzing archive eon-2017-08-17T00:30:02.646158 (9/9) Analyzing archive eon-2017-08-16T11:41:39.384681 (8/9) Analyzing archive eon-2017-08-15T00:30:02.279744 (7/9) Orphaned objects check skipped (needs all archives checked). Archive consistency check complete, no problems found. ``` ~~I don't see any pruning going on here, yet Thursday is gone. Somehow the 2017-08-10 archive was pruned without leaving a trace in the logfile or in the output of the commands I later ran manually.~~ **EDIT**: I expected the logfile to show the actual pruning, just as it did when running the borg command but apparently it only shows the mount of data deleted which I overlooked at first. As an experiment I changed the retention policy from 7 to 6 days and ran the following commands: ``` pi@eon:[~]: sudo borg list /mnt/nas/eon eon-2017-08-06T00:30:02.240738 Sun, 2017-08-06 00:30:02 eon-2017-08-11T00:30:02.791884 Fri, 2017-08-11 00:30:03 eon-2017-08-12T00:30:02.015646 Sat, 2017-08-12 00:30:02 eon-2017-08-13T00:30:02.524766 Sun, 2017-08-13 00:30:02 eon-2017-08-14T00:30:02.755813 Mon, 2017-08-14 00:30:03 eon-2017-08-15T00:30:02.279744 Tue, 2017-08-15 00:30:02 eon-2017-08-16T11:41:39.384681 Wed, 2017-08-16 11:41:39 eon-2017-08-17T00:30:02.646158 Thu, 2017-08-17 00:30:03 pi@eon:[~]: sudo borgmatic --prune --verbosity 2 -c /etc/borgmatic.d/eon.yaml using builtin fallback logging configuration TAM-verified manifest ------------------------------------------------------------------------------ Original size Compressed size Deduplicated size Deleted data: -4.60 GB -4.61 GB -189.82 MB All archives: 34.42 GB 34.47 GB 5.62 GB Unique chunks Total chunks Chunk index: 103104 1060132 ------------------------------------------------------------------------------ pi@eon:[~]: sudo borg list /mnt/nas/eon eon-2017-08-06T00:30:02.240738 Sun, 2017-08-06 00:30:02 eon-2017-08-12T00:30:02.015646 Sat, 2017-08-12 00:30:02 eon-2017-08-13T00:30:02.524766 Sun, 2017-08-13 00:30:02 eon-2017-08-14T00:30:02.755813 Mon, 2017-08-14 00:30:03 eon-2017-08-15T00:30:02.279744 Tue, 2017-08-15 00:30:02 eon-2017-08-16T11:41:39.384681 Wed, 2017-08-16 11:41:39 eon-2017-08-17T00:30:02.646158 Thu, 2017-08-17 00:30:03 pi@eon:[~]: ``` This time it worked.  Tentative conclusion: Leaving a number of default parameters in the borgmatic retention policy did not result in the expected pruning behavior:Original retention policy: ``` # Retention policy for how many backups to keep in each category. See # https://borgbackup.readthedocs.org/en/stable/usage.html#borg-prune for details. retention: # Keep all archives within this time interval. keep_within: 3H # Number of hourly archives to keep. keep_hourly: 24 # Number of daily archives to keep. keep_daily: 6 # Number of weekly archives to keep. keep_weekly: 4 # Number of monthly archives to keep. keep_monthly: 6 # Number of yearly archives to keep. keep_yearly: 1 # When pruning, only consider archive names starting with this prefix. # prefix: eon ``` Current retention policy ``` # Retention policy for how many backups to keep in each category. See # https://borgbackup.readthedocs.org/en/stable/usage.html#borg-prune for details. retention: # Keep all archives within this time interval. #keep_within: 3H # Number of hourly archives to keep. #keep_hourly: 24 # Number of daily archives to keep. keep_daily: 6 # Number of weekly archives to keep. keep_weekly: 4 # Number of monthly archives to keep. keep_monthly: 6 # Number of yearly archives to keep. #keep_yearly: 1 # When pruning, only consider archive names starting with this prefix. # prefix: eon ``` I don't know if the behavior as shown is by design, but it might help others in a similar situation to point this out in the docs. I'll report tomorrow if pruning occurs dring standard operation (cron) but I assume it will. Many thanks for your assistance in this matter. --- Comment on 2017-08-17T10:56:55+0000 by ot3.
Author
Collaborator

Glad to hear it appears to be working now. I'll update the docs and also look into listing the archives when pruning with -v 2, not just providing the summary.


Comment on 2017-08-18T04:00:45+0000 by Dan Helfman.

Glad to hear it appears to be working now. I'll update the docs and also look into listing the archives when pruning with `-v 2`, not just providing the summary. --- Comment on 2017-08-18T04:00:45+0000 by Dan Helfman.
Author
Collaborator

Unfortunately we're not there yet; very puzzling behavior.

Current retention policy [still using yesterday's 'keep_daily: 6' for testing purposes]:

# Number of daily archives to keep.
    keep_daily: 6

    # Number of weekly archives to keep.
    keep_weekly: 4

    # Number of monthly archives to keep.
    keep_monthly: 6

From the mailed borgmatic log as created by cron; backup OK but no pruning:

------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
Deleted data:                    0 B                  0 B                  0 B
All archives:               34.42 GB             34.47 GB              5.62 GB

                       Unique chunks         Total chunks
Chunk index:                  103104              1060132
------------------------------------------------------------------------------
------------------------------------------------------------------------------
Archive name: eon-2017-08-18T00:30:02.250531
Archive fingerprint: 11b1cf2cb401a3f94a52e4c25b1c7f125252dddfcb8611e41a2480305995af49
Time (start): Fri, 2017-08-18 00:30:02
Time (end):   Fri, 2017-08-18 00:30:46
Duration: 43.91 seconds
Number of files: 175058
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:                4.96 GB              4.96 GB            159.16 MB
All archives:               39.38 GB             39.43 GB              5.78 GB

                       Unique chunks         Total chunks
Chunk index:                  103242              1214590
------------------------------------------------------------------------------

Running borg to check on the remaining archives; Saturday was not pruned:

pi@eon:[~]: sudo borg list /mnt/nas/eon
eon-2017-08-06T00:30:02.240738       Sun, 2017-08-06 00:30:02
eon-2017-08-12T00:30:02.015646       Sat, 2017-08-12 00:30:02
eon-2017-08-13T00:30:02.524766       Sun, 2017-08-13 00:30:02
eon-2017-08-14T00:30:02.755813       Mon, 2017-08-14 00:30:03
eon-2017-08-15T00:30:02.279744       Tue, 2017-08-15 00:30:02
eon-2017-08-16T11:41:39.384681       Wed, 2017-08-16 11:41:39
eon-2017-08-17T00:30:02.646158       Thu, 2017-08-17 00:30:03
eon-2017-08-18T00:30:02.250531       Fri, 2017-08-18 00:30:02
pi@eon:[~]:

Running the same borgmatic pruning command as yesterday; pruning OK

pi@eon:[~]: sudo borgmatic --prune --verbosity 2 -c /etc/borgmatic.d/eon.yaml
using builtin fallback logging configuration
TAM-verified manifest
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
Deleted data:               -4.95 GB             -4.96 GB           -182.59 MB
All archives:               34.43 GB             34.48 GB              5.60 GB

                       Unique chunks         Total chunks
Chunk index:                  102944              1060498
------------------------------------------------------------------------------
pi@eon:[~]:

Running borg to check on the remaining archives; Saturday was pruned:

pi@eon:[~]: sudo borg list /mnt/nas/eon
eon-2017-08-06T00:30:02.240738       Sun, 2017-08-06 00:30:02
eon-2017-08-13T00:30:02.524766       Sun, 2017-08-13 00:30:02
eon-2017-08-14T00:30:02.755813       Mon, 2017-08-14 00:30:03
eon-2017-08-15T00:30:02.279744       Tue, 2017-08-15 00:30:02
eon-2017-08-16T11:41:39.384681       Wed, 2017-08-16 11:41:39
eon-2017-08-17T00:30:02.646158       Thu, 2017-08-17 00:30:03
eon-2017-08-18T00:30:02.250531       Fri, 2017-08-18 00:30:02
pi@eon:[~]:

So here we have it: running borgmatic from cron results in a backup without pruning while running the pruning command manually works fine. Normally in case of cron problems, this points to an environment problem. Does this give you a clue? Does the borgmatic code set it's environment to Python3?


Comment on 2017-08-18T10:29:48+0000 by ot3.

Unfortunately we're not there yet; very puzzling behavior. Current retention policy [still using yesterday's 'keep_daily: 6' for testing purposes]: ``` # Number of daily archives to keep. keep_daily: 6 # Number of weekly archives to keep. keep_weekly: 4 # Number of monthly archives to keep. keep_monthly: 6 ``` From the mailed borgmatic log as created by cron; backup OK but no pruning: ``` ------------------------------------------------------------------------------ Original size Compressed size Deduplicated size Deleted data: 0 B 0 B 0 B All archives: 34.42 GB 34.47 GB 5.62 GB Unique chunks Total chunks Chunk index: 103104 1060132 ------------------------------------------------------------------------------ ------------------------------------------------------------------------------ Archive name: eon-2017-08-18T00:30:02.250531 Archive fingerprint: 11b1cf2cb401a3f94a52e4c25b1c7f125252dddfcb8611e41a2480305995af49 Time (start): Fri, 2017-08-18 00:30:02 Time (end): Fri, 2017-08-18 00:30:46 Duration: 43.91 seconds Number of files: 175058 ------------------------------------------------------------------------------ Original size Compressed size Deduplicated size This archive: 4.96 GB 4.96 GB 159.16 MB All archives: 39.38 GB 39.43 GB 5.78 GB Unique chunks Total chunks Chunk index: 103242 1214590 ------------------------------------------------------------------------------ ``` Running borg to check on the remaining archives; Saturday was not pruned: ``` pi@eon:[~]: sudo borg list /mnt/nas/eon eon-2017-08-06T00:30:02.240738 Sun, 2017-08-06 00:30:02 eon-2017-08-12T00:30:02.015646 Sat, 2017-08-12 00:30:02 eon-2017-08-13T00:30:02.524766 Sun, 2017-08-13 00:30:02 eon-2017-08-14T00:30:02.755813 Mon, 2017-08-14 00:30:03 eon-2017-08-15T00:30:02.279744 Tue, 2017-08-15 00:30:02 eon-2017-08-16T11:41:39.384681 Wed, 2017-08-16 11:41:39 eon-2017-08-17T00:30:02.646158 Thu, 2017-08-17 00:30:03 eon-2017-08-18T00:30:02.250531 Fri, 2017-08-18 00:30:02 pi@eon:[~]: ``` Running the same borgmatic pruning command as yesterday; pruning OK ``` pi@eon:[~]: sudo borgmatic --prune --verbosity 2 -c /etc/borgmatic.d/eon.yaml using builtin fallback logging configuration TAM-verified manifest ------------------------------------------------------------------------------ Original size Compressed size Deduplicated size Deleted data: -4.95 GB -4.96 GB -182.59 MB All archives: 34.43 GB 34.48 GB 5.60 GB Unique chunks Total chunks Chunk index: 102944 1060498 ------------------------------------------------------------------------------ pi@eon:[~]: ``` Running borg to check on the remaining archives; Saturday was pruned: ``` pi@eon:[~]: sudo borg list /mnt/nas/eon eon-2017-08-06T00:30:02.240738 Sun, 2017-08-06 00:30:02 eon-2017-08-13T00:30:02.524766 Sun, 2017-08-13 00:30:02 eon-2017-08-14T00:30:02.755813 Mon, 2017-08-14 00:30:03 eon-2017-08-15T00:30:02.279744 Tue, 2017-08-15 00:30:02 eon-2017-08-16T11:41:39.384681 Wed, 2017-08-16 11:41:39 eon-2017-08-17T00:30:02.646158 Thu, 2017-08-17 00:30:03 eon-2017-08-18T00:30:02.250531 Fri, 2017-08-18 00:30:02 pi@eon:[~]: ``` So here we have it: running borgmatic from cron results in a backup without pruning while running the pruning command manually works fine. Normally in case of cron problems, this points to an environment problem. Does this give you a clue? Does the borgmatic code set it's environment to Python3? --- Comment on 2017-08-18T10:29:48+0000 by ot3.
Author
Collaborator

Borgmatic runs pruning before creating any archives. (This is a relatively recent change.) The rationale is that pruning may free up space needed to create the archive.

So what could be happening here is that borgmatic, when run by cron without any action flags like --prune, will:

  1. Prune any archives as per the retention policy.
  2. Create an archive.
  3. Run any configured consistency checks.

It's entirely possible that after the archive is created during the cron job, there are now 7 daily archives instead of 6. And so the next time a prune is run (whether manually or on the next cron invocation), the oldest daily archive will get pruned.

Based on your logs, that looks plausible.


Comment on 2017-08-19T03:05:26+0000 by Dan Helfman.

Borgmatic runs pruning *before* creating any archives. (This is a relatively recent change.) The rationale is that pruning may free up space needed to create the archive. So what could be happening here is that borgmatic, when run by cron without any action flags like `--prune`, will: 1. Prune any archives as per the retention policy. 2. Create an archive. 3. Run any configured consistency checks. It's entirely possible that after the archive is created during the cron job, there are now 7 daily archives instead of 6. And so the next time a prune is run (whether manually or on the next cron invocation), the oldest daily archive will get pruned. Based on your logs, that looks plausible. --- Comment on 2017-08-19T03:05:26+0000 by Dan Helfman.
Author
Collaborator

I'll leave the backup routines untouched for a few days and see what happens.


Comment on 2017-08-19T09:45:19+0000 by ot3.

I'll leave the backup routines untouched for a few days and see what happens. --- Comment on 2017-08-19T09:45:19+0000 by ot3.
Author
Collaborator

Any luck?


Comment on 2017-08-26T23:12:47+0000 by Dan Helfman.

Any luck? --- Comment on 2017-08-26T23:12:47+0000 by Dan Helfman.
Author
Collaborator

I was going to report back tomorrow but will do it now [I wanted one more day].

Current retention policy:

# Number of daily archives to keep.
    keep_daily: 7

    # Number of weekly archives to keep.
    keep_weekly: 4

    # Number of monthly archives to keep.
    keep_monthly: 6

Archives:

pi@eon:[~]: sudo borg list /mnt/nas/eon
eon-2017-08-06T00:30:02.240738       Sun, 2017-08-06 00:30:02
eon-2017-08-13T00:30:02.524766       Sun, 2017-08-13 00:30:02
eon-2017-08-20T04:30:06.596881       Sun, 2017-08-20 04:30:07
eon-2017-08-21T00:30:08.961567       Mon, 2017-08-21 00:30:09
eon-2017-08-22T00:30:02.179638       Tue, 2017-08-22 00:30:02
eon-2017-08-23T00:30:09.075557       Wed, 2017-08-23 00:30:09
eon-2017-08-24T00:30:10.772666       Thu, 2017-08-24 00:30:12
eon-2017-08-25T00:30:11.093285       Fri, 2017-08-25 00:30:12
eon-2017-08-26T00:30:09.582931       Sat, 2017-08-26 00:30:10
eon-2017-08-27T00:30:25.804251       Sun, 2017-08-27 00:30:26
pi@eon:[~]:

Conclusion:

  • pruning is working 
  • borgmatic retains 1 archive more than specified: 8 instead of 7; this wasn't clear from the documentation. 

Will the same happen to the weekly and monthly archives?


Comment on 2017-08-27T13:05:48+0000 by ot3.

I was going to report back tomorrow but will do it now [I wanted one more day]. Current retention policy: ``` # Number of daily archives to keep. keep_daily: 7 # Number of weekly archives to keep. keep_weekly: 4 # Number of monthly archives to keep. keep_monthly: 6 ``` Archives: ``` pi@eon:[~]: sudo borg list /mnt/nas/eon eon-2017-08-06T00:30:02.240738 Sun, 2017-08-06 00:30:02 eon-2017-08-13T00:30:02.524766 Sun, 2017-08-13 00:30:02 eon-2017-08-20T04:30:06.596881 Sun, 2017-08-20 04:30:07 eon-2017-08-21T00:30:08.961567 Mon, 2017-08-21 00:30:09 eon-2017-08-22T00:30:02.179638 Tue, 2017-08-22 00:30:02 eon-2017-08-23T00:30:09.075557 Wed, 2017-08-23 00:30:09 eon-2017-08-24T00:30:10.772666 Thu, 2017-08-24 00:30:12 eon-2017-08-25T00:30:11.093285 Fri, 2017-08-25 00:30:12 eon-2017-08-26T00:30:09.582931 Sat, 2017-08-26 00:30:10 eon-2017-08-27T00:30:25.804251 Sun, 2017-08-27 00:30:26 pi@eon:[~]: ``` Conclusion: * pruning is working  * borgmatic retains 1 archive more than specified: 8 instead of 7; this wasn't clear from the documentation.  Will the same happen to the weekly and monthly archives? --- Comment on 2017-08-27T13:05:48+0000 by ot3.
Author
Collaborator

Thanks for reporting back. I've updated the documentation to reflect this perhaps unexpected characteristic: "Note that borgmatic prunes archives before creating an archive, so as to free up space for archiving. This means that when a borgmatic run finishes, there may still be prune-able archives. Not to worry, as they will get cleaned up at the start of the next run."

And yes, weekly and monthly archives work the same way. Although note that not every borgmatic run creates an archive that would be pruned by weekly or monthly retention policies.

So I'll close this for now, but feel free to re-open or re-file if you see other problems with this.


Comment on 2017-08-27T17:19:20+0000 by Dan Helfman.

Thanks for reporting back. I've updated the documentation to reflect this perhaps unexpected characteristic: "Note that borgmatic prunes archives before creating an archive, so as to free up space for archiving. This means that when a borgmatic run finishes, there may still be prune-able archives. Not to worry, as they will get cleaned up at the start of the next run." And yes, weekly and monthly archives work the same way. Although note that not every borgmatic run creates an archive that would be pruned by weekly or monthly retention policies. So I'll close this for now, but feel free to re-open or re-file if you see other problems with this. --- Comment on 2017-08-27T17:19:20+0000 by Dan Helfman.
Sign in to join this conversation.
No Milestone
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#37
No description provided.