Is --max-duration for checks supported by YAML already? #506

New Issue

Open

opened 2022-03-07 10:01:40 +00:00 by ams_tschoening · 7 comments

ams_tschoening commented 2022-03-07 10:01:40 +00:00

Copy Link

What I'm trying to do and why

I'm running repo checks by default using repository and archives, because those checks seemed to have fit pretty well into my backup time window in the past. Though, some days ago this wasn't the case anymore and checking operations which according to former logs took only e.g. 5 minutes, required around 45 minutes now.

Of course this doesn't have anything to do with BORG or BORGMATIC, but instead I'm backing up to some NAS-like device provided by my hoster. The provider doesn't guarantee any performance and most likely changed/did something for some reason, resulting in temporarily bad performance. Things seem to be better now.

The point is, that while I would like to have as much checks as possible, backing up the individual data of all systems is more important of course. Because multiple backups are run one after another only when the former has finished, exceptionally long running checks make things difficult.

That's where --max-duration comes into play, even though it supports slightly different checks only. Though, I can't see --max-duration in the YAML structure for consistency.

So, is it supported already and I need to upgrade BORGMATIC or isn't it supported currently and I need to use it in extra_borg_options/check instead? In the latter case, would you consider supporting it for consistency?

Thanks!

Environment

borgmatic version: 1.5.15
borgmatic installation method: PIP, system wide
Borg version: 1.1.16
Python version: 3.8.10
operating system and version: Ubuntu 20.04

#### What I'm trying to do and why I'm running repo checks by default using `repository` and `archives`, because those checks seemed to have fit pretty well into my backup time window in the past. Though, some days ago this wasn't the case anymore and checking operations which according to former logs took only e.g. 5 minutes, required around 45 minutes now. Of course this doesn't have anything to do with BORG or BORGMATIC, but instead I'm backing up to some NAS-like device provided by my hoster. The provider doesn't guarantee any performance and most likely changed/did something for some reason, resulting in temporarily bad performance. Things seem to be better now. The point is, that while I would like to have as much checks as possible, backing up the individual data of all systems is more important of course. Because multiple backups are run one after another only when the former has finished, exceptionally long running checks make things difficult. That's where [--max-duration](https://borgbackup.readthedocs.io/en/stable/usage/check.html?highlight=check) comes into play, even though it supports slightly different checks only. Though, I can't see `--max-duration` in the YAML structure for `consistency`. So, is it supported already and I need to upgrade BORGMATIC or isn't it supported currently and I need to use it in `extra_borg_options/check` instead? In the latter case, would you consider supporting it for `consistency`? Thanks! #### Environment **borgmatic version:** 1.5.15 **borgmatic installation method:** PIP, system wide **Borg version:** 1.1.16 **Python version:** 3.8.10 **operating system and version:** Ubuntu 20.04

witten commented 2022-03-08 05:59:22 +00:00

Owner

Copy Link

It's supported by borgmatic, but currently it's only at the command-line level rather than the configuration level. E.g., you can do borgmatic check --max-duration. I think your idea of adding it as an option to the consistency section of borgmatic's configuration file makes sense as well! (Implementation note: If max duration is specified in both the configuration file and on the command-line, the command-line value should take precedence.) In the meantime, you can use the command-line work-around. Thanks for filing this!

It's supported by borgmatic, but currently it's only at the command-line level rather than the configuration level. E.g., you can do `borgmatic check --max-duration`. I think your idea of adding it as an option to the `consistency` section of borgmatic's configuration file makes sense as well! (Implementation note: If max duration is specified in both the configuration file and on the command-line, the command-line value should take precedence.) In the meantime, you can use the command-line work-around. Thanks for filing this!

witten added the

design finalized

label 2022-03-08 05:59:46 +00:00

lucaim commented 2022-03-16 13:39:54 +00:00

Copy Link

Hi, thanks for your great work. I just wanted to report that this does not seem to work for me with borg 1.2.0 and borgmatic 1.5.24. Is it on another branch?

/root/.local/bin/borgmatic check --verbosity 2 --syslog-verbosity 2 --max-duration=3600

Also, I think it would be very nice to have this in the config file as you were saying as I have a big repository and checking over SSH in a stable way becomes tricky as connection needs to be maintained for more than 20 hours.

Hi, thanks for your great work. I just wanted to report that this does not seem to work for me with borg 1.2.0 and borgmatic 1.5.24. Is it on another branch? `/root/.local/bin/borgmatic check --verbosity 2 --syslog-verbosity 2 --max-duration=3600` Also, I think it would be very nice to have this in the config file as you were saying as I have a big repository and checking over SSH in a stable way becomes tricky as connection needs to be maintained for more than 20 hours.

witten commented 2022-03-16 17:06:15 +00:00

Owner

Copy Link

My mistake. For some reason I thought it was already implemented as a command-line flag! Looks like it's not. Agreed that it would make sense in the config file instead! As an interim as a work-around, you could add it to check under the extra_borg_options in the configuration file. Example:

storage:
    extra_borg_options:
        check: --max-duration=3600

My mistake. For some reason I thought it was already implemented as a command-line flag! Looks like it's not. Agreed that it would make sense in the config file instead! As an interim as a work-around, you could add it to `check` under the `extra_borg_options` in the configuration file. Example: ```yaml storage: extra_borg_options: check: --max-duration=3600 ```

lucaim commented 2022-03-17 21:51:54 +00:00

Copy Link

Many thanks for the help @witten. It actually works, as the option is correctly present in the command line but (because of borg) it needs to set a lot other very specific options: it needs --repository-only and this in turn is not compatible with --first, --last, --prefix and --verify-data arguments . So in short, I think this would require a dedicated command for the time being (outside of borgmatic). There seems not to be an easy solution to do repository and archives checks together as repository check can have --max-duration but archive check cannot.

Mind that the correct option is --max-duration, just for reference.

Many thanks again for the quick and nice help.

Many thanks for the help @witten. It actually works, as the option is correctly present in the command line but (because of borg) it needs to set a lot other very specific options: it needs `--repository-only` and this in turn is not compatible with `--first`, `--last`, `--prefix` and `--verify-data` arguments . So in short, I think this would require a dedicated command for the time being (outside of borgmatic). There seems not to be an easy solution to do repository and archives checks together as repository check can have `--max-duration` but archive check cannot. Mind that the correct option is `--max-duration`, just for reference. Many thanks again for the quick and nice help.

lucaim commented 2022-03-17 21:53:03 +00:00

Copy Link

PS: Of course you can do it in two steps in borgmatic and include the option only in the repository check.

PS: Of course you can do it in two steps in borgmatic and include the option only in the repository check.

witten commented 2022-03-17 22:03:02 +00:00

Owner

Copy Link

Thanks for pointing this out. Certainly seems like a candidate for a borgmatic option then.. borgmatic could apply the logic to add --max-duration only when --repository-only is implied.. or just to error if that's not the case.

Thanks for pointing this out. Certainly seems like a candidate for a borgmatic option then.. borgmatic could apply the logic to add `--max-duration` only when `--repository-only` is implied.. or just to error if that's not the case.

witten commented 2022-05-29 04:44:42 +00:00

Owner

Copy Link

While there's no --max-duration support yet, you may be interested in the related ticket #523, which allows you to run checks at a configured frequency rather than running them all the time. The documentation for that (not yet released) feature is here: https://torsion.org/borgmatic/docs/how-to/deal-with-very-large-backups/#check-frequency

While there's no `--max-duration` support yet, you may be interested in the related ticket #523, which allows you to run checks at a configured frequency rather than running them all the time. The documentation for that (not yet released) feature is here: https://torsion.org/borgmatic/docs/how-to/deal-with-very-large-backups/#check-frequency

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

spot-check

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.24

1.5.23

1.5.22

1.5.21

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.14

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.7.dev0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.22

1.4.21

1.4.20

1.4.19

1.4.18

1.4.17

1.4.16

1.4.15

1.4.14

1.4.13

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.26

1.3.25

1.3.24

1.3.23

1.3.22

1.3.21

1.3.20

1.3.19

1.3.18

1.3.17

1.3.16

1.3.15

1.3.14

1.3.13

1.3.12

1.3.11

1.3.10

1.3.9

1.3.8

1.3.7

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.18

1.2.17

1.2.16

1.2.15

1.2.14

1.2.13

1.2.12

1.2.11

1.2.10

1.2.9

1.2.8

1.2.7

1.2.6

1.2.5

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.15

1.1.14

1.1.13

1.1.12

1.1.11

1.1.10

1.1.9

1.1.7

1.1.6

1.1.5

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.3

1.0.2

1.0.1

1.0.0

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

0.0.2

Labels

Clear labels   

bug

  

data loss

  

design finalized

  

good first issue

  

new feature area

  

question / support

  

security

  

waiting for response

No Label

bug

data loss

design finalized

good first issue

new feature area

question / support

security

waiting for response

Milestone

Clear milestone

No items

No Milestone

Assignees

Clear assignees

No Assignees

3 Participants

Notifications

Subscribe

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#506

No description provided.