reduce default check frequency #523

New Issue

ramcq · 2022-04-22T09:42:23Z

ramcq commented

2022-04-22 09:42:23 +00:00

What I'm trying to do and why

I've got several container backups which are running every 6 hours (partly to give robustness against failures caused by https://github.com/borgmatic-collective/docker-borgmatic/issues/93) and Borgmatic recently contacted me and said I was causing undue server load due to the checks and needed to reduce the frequency.

I had not modified the default configuration which means all archives are being verified every backup run multiple times daily, so I can understand their concern! Obviously now I will adjust my configuration, but I believe these defaults are un-necessarily paranoid and implicitly encourage this "misbehaviour".

Expected behavior (if a bug)

The ideal to me would be if we could work out a way to achieve the behaviour which was considered in #18, so that checks have a frequency which differs from the backup frequency. I'm aware of the challenge in as much as one borgmatic invocation doesn't know how frequently it's run, and that we don't ideally want to add fragile state here. I've not fully thought it through, but I wonder whether the distribution/age of backups (comparing the rotation settings to the backups available?) could act as a trigger for a periodic check? Ie do a check whenever prune discards a weekly/monthly/...?

Other notes / implementation ideas

Perhaps defaulting check_last to 1 or 2 would be a reasonable middle ground, or even disable the archive check by default, and instead document that a more thorough check should be scheduled at a lower frequency?

I will file a PR on the docker container so that the default crontab.txt runs create/prune/compact daily and check monthly, which feels like a more "exemplary" configuration to me, but it would feel more robust if borgmatic could handle this somehow.

Environment

borgmatic version: 1.5.24

borgmatic installation method: b3vis/borgmatic docker container

Borg version: 1.2.0

Python version: Python 3.9.7

operating system and version: Debian GNU/Linux 11 (bullseye)

#### What I'm trying to do and why I've got several container backups which are running every 6 hours (partly to give robustness against failures caused by https://github.com/borgmatic-collective/docker-borgmatic/issues/93) and Borgmatic recently contacted me and said I was causing undue server load due to the checks and needed to reduce the frequency. I had not modified the default configuration which means all archives are being verified every backup run multiple times daily, so I can understand their concern! Obviously now I will adjust my configuration, but I believe these defaults are un-necessarily paranoid and implicitly encourage this "misbehaviour". #### Expected behavior (if a bug) The ideal to me would be if we could work out a way to achieve the behaviour which was considered in #18, so that checks have a frequency which differs from the backup frequency. I'm aware of the challenge in as much as one borgmatic invocation doesn't know how frequently it's run, and that we don't ideally want to add fragile state here. I've not fully thought it through, but I wonder whether the distribution/age of backups (comparing the rotation settings to the backups available?) could act as a trigger for a periodic check? Ie do a check whenever prune discards a weekly/monthly/...? #### Other notes / implementation ideas Perhaps defaulting check_last to 1 or 2 would be a reasonable middle ground, or even disable the archive check by default, and instead document that a more thorough check should be scheduled at a lower frequency? I will file a PR on the docker container so that the default crontab.txt runs create/prune/compact daily and check monthly, which feels like a more "exemplary" configuration to me, but it would feel more robust if borgmatic could handle this somehow. #### Environment **borgmatic version:** 1.5.24 **borgmatic installation method:** b3vis/borgmatic docker container **Borg version:** 1.2.0 **Python version:** Python 3.9.7 **operating system and version:** Debian GNU/Linux 11 (bullseye)

m3nu commented

2022-04-22 11:39:13 +00:00

Came here to support this change. Except for very small repos, doing checks during every backups is not appropriate and makes the backup much longer.

For us at BorgBase it causes lots of read activity. Usually not an issue, but can exacerbate load at busy times, when too many users do it.

We already updated all the docs and template where we can, but changing the default in Borgmatic would be better.

I don't suggest disabling checks for good as default. Maybe there could be a new default that's less frequent. Like monthly only.

Came here to support this change. Except for very small repos, doing checks during every backups is not appropriate and makes the backup much longer. For us at BorgBase it causes lots of read activity. Usually not an issue, but can exacerbate load at busy times, when too many users do it. We already updated all the docs and template where we can, but changing the default in Borgmatic would be better. I don't suggest disabling checks for good as default. Maybe there could be a new default that's less frequent. Like monthly only.

witten commented

2022-04-23 16:29:08 +00:00

Thanks for suggesting this! I don't mind borgmatic storing a little state.. it could just be a local file with a timestamp of the last consistency check.

As for configuration, I'd like to avoid implementing a whole scheduling system (that's what cron/systemd is for!), but what about something like:

consistency:
    check_frequency: 2 weeks

To support that, borgmatic would have to take a dependency on something like dateparser.

Thoughts? I'm open to alternate ideas as well.

Thanks for suggesting this! I don't mind borgmatic storing a little state.. it could just be a local file with a timestamp of the last consistency check. As for configuration, I'd like to avoid implementing a whole scheduling system (that's what cron/systemd is for!), but what about something like: ``` consistency: check_frequency: 2 weeks ``` To support that, borgmatic would have to take a dependency on something like [dateparser](https://dateparser.readthedocs.io/). Thoughts? I'm open to alternate ideas as well.

witten added the

waiting for response

label 2022-04-28 17:48:19 +00:00

robdavies commented

2022-05-24 13:10:50 +00:00

I'm new to borg/borgmatic and currently trying out configurations to get a good setup. I don't know yet what is the right balance of verification (not too much or too little).

Currently I'm checking last 3 archives-only each backup (not repositories).

I am thinking of running a normal check (repository and archives) each month.

Could it be possible to define two different check lists, each with their own frequency?

Similar to what you have suggested above, if I put archives-only check and a frequency (e.g. each backup), also a reposotiries and archivies check with a frequency (i.e. monthly), is that something you think that borgmatic could manage?

I'm new to borg/borgmatic and currently trying out configurations to get a good setup. I don't know yet what is the right balance of verification (not too much or too little). Currently I'm checking last 3 archives-only each backup (not repositories). I am thinking of running a normal check (repository and archives) each month. Could it be possible to define two different check lists, each with their own frequency? Similar to what you have suggested above, if I put archives-only check and a frequency (e.g. each backup), also a reposotiries and archivies check with a frequency (i.e. monthly), is that something you think that borgmatic could manage?

witten commented

2022-05-24 17:25:22 +00:00

@robdavies: How about something like this?

consistency:
    checks:
        - name: standard
          frequency: all
          types:
              - archives
        - name: monthly
          frequency: 1 month
          types:
              - archives
              - repository

Thoughts?

@robdavies: How about something like this? ``` consistency: checks: - name: standard frequency: all types: - archives - name: monthly frequency: 1 month types: - archives - repository ``` Thoughts?

m3nu commented

2022-05-24 17:54:03 +00:00

If I understood the docs correctly, archives should be fairly fast and repository slow because it reads all the data. Has anyone tried both to see the actual time difference? Also, archives doesn't detect bit rot, so it's not as useful.

There is also a max-duration option, which could be useful to add.

Being backwards-compatible with the config would be good too.

How about:

consistency:
    checks:
      - repository
        frequency: 1 [day,days,week,weeks,month,months]  # 1 month being the default
        max_duration_sec: 1800  # minutes may be more natural than seconds
      - archives
        frequency: always
        check_last: 3
        prefix: '{hostname}-'

This puts groups the option with the corresponding check and adds the frequency option to all of them. disabled still does nothing.

https://borgbackup.readthedocs.io/en/stable/usage/check.html

If I understood the docs correctly, `archives` should be fairly fast and `repository` slow because it reads all the data. Has anyone tried both to see the actual time difference? Also, `archives` doesn't detect bit rot, so it's not as useful. There is also a max-duration option, which could be useful to add. Being backwards-compatible with the config would be good too. How about: ``` consistency: checks: - repository frequency: 1 [day,days,week,weeks,month,months] # 1 month being the default max_duration_sec: 1800 # minutes may be more natural than seconds - archives frequency: always check_last: 3 prefix: '{hostname}-' ``` This puts groups the option with the corresponding check and adds the frequency option to all of them. `disabled` still does nothing. https://borgbackup.readthedocs.io/en/stable/usage/check.html

witten commented

2022-05-24 18:20:08 +00:00

I like that structure! The only downside I can see is it wouldn't allow a given check type to run at multiple frequencies as @robdavies suggested. That may be just fine though, since the archives check running "always" would also run approximately "monthly".

m3nu commented

2022-05-25 02:45:26 +00:00

You don't want to rebuild Cron here. 😄

And personally, I was mainly asking for a different default value and avoid checking the whole repository every time. This is pretty slow and causes lots of read activity on our servers.

I'd still like to see the difference in time between checking the repo and the previoius 3 archives. Will do this in the afternoon.

BTW: How do you plan on keeping the state of the last run? Borg must have a way to keep the state when using max-duration. Maybe random keys/values can be written via borg config? That would be handy.

You don't want to rebuild Cron here. 😄 And personally, I was mainly asking for a different default value and avoid checking the whole repository every time. This is pretty slow and causes lots of read activity on our servers. I'd still like to see the difference in time between checking the repo and the previoius 3 archives. Will do this in the afternoon. BTW: How do you plan on keeping the state of the last run? Borg must have a way to keep the state when using max-duration. Maybe random keys/values can be written via `borg config`? That would be handy.

ramcq commented

2022-05-25 13:04:55 +00:00

You don't want to rebuild Cron here. 😄

I'd agree, this seems like a bit of overkill against my original concern.

You can already use borgmatic to set up crontabs that do the check action, or the non-check actions, on different cadences, or to override the config on the command line so that more or less rigorous checks are done.

The main goal I had is that by /default/ (you take a stock crontab where borgmatic runs daily or whatever) the check has a lesser frequency. So that people who don't even edit the check: section have something reasonable happen, which is the case for the rest of the file.

I think that just needs one config option around frequency, and one stamp file to check the age of the last check.

> You don't want to rebuild Cron here. 😄 I'd agree, this seems like a bit of overkill against my original concern. You can already use borgmatic to set up crontabs that do the check action, or the non-check actions, on different cadences, or to override the config on the command line so that more or less rigorous checks are done. The main goal I had is that by /default/ (you take a stock crontab where borgmatic runs daily or whatever) the check has a lesser frequency. So that people who don't even edit the check: section have something reasonable happen, which is the case for the rest of the file. I think that just needs one config option around frequency, and one stamp file to check the age of the last check.

witten commented

2022-05-26 17:41:30 +00:00

Definitely don't want to rebuild cron!

BTW: How do you plan on keeping the state of the last run? Borg must have a way to keep the state when using max-duration. Maybe random keys/values can be written via borg config? That would be handy.

I was originally just thinking a state file in ~/.borgmatic (per repository?), but borg config is pretty interesting too. Thanks for the suggestion. EDIT: I've determined experimentally that Borg does not allow arbitrary keys/values to be written via borg config.

I think that just needs one config option around frequency, and one stamp file to check the age of the last check.

That would certainly satisfy the original ask. The multiple-frequency thing is compelling though because I think folks often want a one-stop config file for setting up consistency checks.. rather than having to set up multiple cron jobs or systemd services. Of course, this need must be balanced against borgmatic becoming cron. 😄

Definitely don't want to rebuild cron! > BTW: How do you plan on keeping the state of the last run? Borg must have a way to keep the state when using max-duration. Maybe random keys/values can be written via borg config? That would be handy. I was originally just thinking a state file in `~/.borgmatic` (per repository?), but `borg config` is pretty interesting too. Thanks for the suggestion. EDIT: I've determined experimentally that Borg does not allow arbitrary keys/values to be written via `borg config`. > I think that just needs one config option around frequency, and one stamp file to check the age of the last check. That would certainly satisfy the original ask. The multiple-frequency thing is compelling though because I think folks often want a one-stop config file for setting up consistency checks.. rather than having to set up multiple cron jobs or systemd services. Of course, this need must be balanced against borgmatic becoming cron. :smile:

witten removed the

waiting for response

label 2022-05-26 20:05:34 +00:00

witten referenced this issue from a commit

2022-05-29 03:28:11 +00:00

Reduce the default consistency check frequency and support configuring the frequency independently for each check (#523).

witten referenced this issue from a commit

2022-05-29 03:28:11 +00:00

Tweak default check frequency to 1 month (#523).

witten referenced this issue from a commit

2022-05-29 03:28:11 +00:00

Add another note about the consistency checks schema in old versions (#523).

witten referenced this issue from a commit

2022-05-29 03:28:11 +00:00

Add "borgmatic check --force" flag to ignore configured check frequencies (#523).

witten commented

2022-05-29 03:29:42 +00:00

To keep the scope somewhat small, I implemented a subset of this. The following works now:

consistency:
    checks:
        - name: repository
          frequency: 2 weeks
        - name: archives
          frequency: always

If nothing is configured in consistency checks, the default checks are repository and archives with a frequency of 1 month for each. All frequency checking is done on a per-repository basis (and recorded locally on the client in ~/.borgmatic).

I also added an optional --force flag to borgmatic check for ignoring the configured frequencies and running checks unconditionally.

This is all in master and will be part of the next release. I'm of course open to subseqent enhancements on follow-up tickets as needed. Thanks everyone for the discussion and input here!

To keep the scope somewhat small, I implemented a subset of this. The following works now: ```yaml consistency: checks: - name: repository frequency: 2 weeks - name: archives frequency: always ``` If nothing is configured in consistency checks, the default checks are repository and archives with a frequency of 1 month for each. All frequency checking is done on a per-repository basis (and recorded locally on the client in `~/.borgmatic`). I also added an optional `--force` flag to `borgmatic check` for ignoring the configured frequencies and running checks unconditionally. This is all in master and will be part of the next release. I'm of course open to subseqent enhancements on follow-up tickets as needed. Thanks everyone for the discussion and input here!

witten closed this issue

2022-05-29 03:29:42 +00:00

m3nu commented

2022-05-29 03:56:49 +00:00

That's great! Thanks for adding! Definitely goes into this month's BorgBase newsletter.

🎉 1

witten referenced this issue

2022-05-29 04:44:42 +00:00

Is `--max-duration` for checks supported by YAML already? #506

witten commented

2022-05-31 16:44:10 +00:00

This has been released in borgmatic 1.6.2!