Archive checks are incorrectly skipped if you have multiple config files with different archive name formats #688

New Issue

Closed

opened 2023-05-01 04:27:24 +00:00 by kaysond · 12 comments

kaysond commented 2023-05-01 04:27:24 +00:00

Copy Link

What I'm trying to do and why

Steps to reproduce (if a bug)

Create multiple config files with different archive_name_formats (a la https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/) and the same frequency setting for the archive check.

Run borgmatic check twice.

Actual behavior (if a bug)

The first config file will run the archive check with the correct globbing argument and then touch the ~/.borgmatic/checks/<UUID>/archives file. The remaining config files will look at the archives check timestamp, see that it's recent, and not run at all. This is despite the fact that the archives were not checked, since the archive_name_format is different.

Expected behavior (if a bug)

All archives should be checked per the frequency setting.

Other notes / implementation ideas

It seems that the check timestamps are stored per repository. While that makes sense for the repository check, it doesn't work for the archive checks. There needs to be some way to track the archive check time per archive. A simple approach might be to just use a hash of archive_name_format.

Environment

borgmatic version: 1.7.12

borgmatic installation method: pip

Borg version: 1.2.4

Python version: 3.9.2

Database version (if applicable): N/A

operating system and version: Debian 11

#### What I'm trying to do and why #### Steps to reproduce (if a bug) Create multiple config files with different `archive_name_format`s (a la https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/) and the same `frequency` setting for the `archive` check. Run `borgmatic check` twice. #### Actual behavior (if a bug) The first config file will run the archive check with the correct globbing argument and then touch the `~/.borgmatic/checks/<UUID>/archives` file. The remaining config files will look at the archives check timestamp, see that it's recent, and not run at all. This is despite the fact that the archives were not checked, since the `archive_name_format` is different. #### Expected behavior (if a bug) All archives should be checked per the frequency setting. #### Other notes / implementation ideas It seems that the check timestamps are stored per repository. While that makes sense for the repository check, it doesn't work for the archive checks. There needs to be some way to track the archive check time per archive. A simple approach might be to just use a hash of `archive_name_format`. #### Environment **borgmatic version:** 1.7.12 **borgmatic installation method:** pip **Borg version:** 1.2.4 **Python version:** 3.9.2 **Database version (if applicable):** N/A **operating system and version:** Debian 11

witten commented 2023-05-01 16:50:42 +00:00

Owner

Copy Link

Good catch! Yeah, I think you're right that the path for recording the archives check needs to include a hash of the archive_name_format value or similar for this to work. The trick will be "upgrading" an existing directory of checks in place.

Good catch! Yeah, I think you're right that the path for recording the archives check needs to include a hash of the `archive_name_format` value or similar for this to work. The trick will be "upgrading" an existing directory of checks in place.

witten added the

bug

label 2023-05-01 16:52:23 +00:00

kaysond commented 2023-05-01 18:33:13 +00:00

Author

Copy Link

Good catch! Yeah, I think you're right that the path for recording the archives check needs to include a hash of the archive_name_format value or similar for this to work. The trick will be "upgrading" an existing directory of checks in place.

My thought there was that you could leave the <repo_hash>/archives file as is, and add new files like <repo_hash>/archive_<hash>, since you'd still want them per-repo. And then you can check to see which if any exist and run the checks accordingly.

> Good catch! Yeah, I think you're right that the path for recording the archives check needs to include a hash of the `archive_name_format` value or similar for this to work. The trick will be "upgrading" an existing directory of checks in place. My thought there was that you could leave the `<repo_hash>/archives` file as is, and add new files like `<repo_hash>/archive_<hash>`, since you'd still want them per-repo. And then you can check to see which if any exist and run the checks accordingly.

👍 1

witten referenced this issue from a commit 2023-05-16 06:18:47 +00:00

Fix archive checks being skipped even when particular archives haven't been checked recently (#688).

witten commented 2023-05-16 06:24:17 +00:00

Owner

Copy Link

A fix for this is implemented now in main and will be released soon. I ended up going with a format like <repo_hash>/archives/<archive_filters_hash> (for a subset of archives) and <repo_hash>/archives/all (for all archives). The archive filters hash isn't derived from archive_name_format directly, but rather from the downstream archive filter flags produced from archive_name_format (and a variety of other relevant options/flags that filter down the checked archives to a subset).

The old check paths should get upgraded automatically.

A fix for this is implemented now in main and will be released soon. I ended up going with a format like `<repo_hash>/archives/<archive_filters_hash>` (for a subset of archives) and `<repo_hash>/archives/all` (for all archives). The archive filters hash isn't derived from `archive_name_format` directly, but rather from the downstream archive filter flags produced from `archive_name_format` (and a variety of other relevant options/flags that filter down the checked archives to a subset). The old check paths should get upgraded automatically.

witten closed this issue 2023-05-16 06:24:21 +00:00

kaysond commented 2023-05-16 17:02:55 +00:00

Author

Copy Link

Awesome! I will give this a shot when its released.

The archive filters hash isn't derived from archive_name_format directly, but rather from the downstream archive filter flags produced from archive_name_format (and a variety of other relevant options/flags that filter down the checked archives to a subset).

Makes sense.

and <repo_hash>/archives/all (for all archives)

Why is this needed?

Awesome! I will give this a shot when its released. > The archive filters hash isn't derived from archive_name_format directly, but rather from the downstream archive filter flags produced from archive_name_format (and a variety of other relevant options/flags that filter down the checked archives to a subset). Makes sense. > and <repo_hash>/archives/all (for all archives) Why is this needed?

witten commented 2023-05-16 17:03:28 +00:00

Owner

Copy Link

This was just released in borgmatic 1.7.13! Let me know how it works out for you or if it needs any tweaking.

This was just released in borgmatic 1.7.13! Let me know how it works out for you or if it needs any tweaking.

witten commented 2023-05-16 17:13:34 +00:00

Owner

Copy Link

and <repo_hash>/archives/all (for all archives)

Why is this needed?

It's written to in the case that archives are unfiltered (by archive_name_format or anything else) and therefore all archives are being checked. It's also read in the case that an archive_name_format is used but there's no more specific archives check file on disk. The idea is that if all archives were checked at some point, then that means that any given subset of archives was implicitly checked as well.

> > and <repo_hash>/archives/all (for all archives) > Why is this needed? It's written to in the case that archives are unfiltered (by `archive_name_format` or anything else) and therefore all archives are being checked. It's also *read* in the case that an `archive_name_format` is used but there's no more specific archives check file on disk. The idea is that if all archives were checked at some point, then that means that any given subset of archives was implicitly checked as well.

witten referenced this issue from a commit 2023-05-16 17:21:31 +00:00

Instead of taking the first check time found, take the maximum value (#688)

witten commented 2023-05-16 17:24:43 +00:00

Owner

Copy Link

I did just make one tweak that'll have to wait for the next release: Instead of taking the first check time when multiple are present (e.g., <repo_hash>/archives/<archive_filters_hash> and <repo_hash>/archives/all, take the maximum. That way, if for some reason the all file is newer, that more recent timestamp will get used even if you've since set archive_name_format. Kind of an edge case, but more correct I think.

I did just make one tweak that'll have to wait for the next release: Instead of taking the *first* check time when multiple are present (e.g., `<repo_hash>/archives/<archive_filters_hash>` and `<repo_hash>/archives/all`, take the *maximum*. That way, if for some reason the `all` file is newer, that more recent timestamp will get used even if you've since set `archive_name_format`. Kind of an edge case, but more correct I think.

witten referenced this issue from a commit 2023-05-17 15:49:11 +00:00

Add archive check probing logic tweak to NEWS (#688).

kaysond commented 2023-05-20 20:54:59 +00:00

Author

Copy Link

Based on some quick initial testing this seems to work great! I just deleted the checks dir and started borgmatic check then I'll rerun it to make sure everything looks good.

One thing that still doesn't work great for me, though, is the fact that when the checks do actually run, my health checks go down because it takes so much longer than a normal run.

This why I'd previously disabled checks in my config files, then had a separate cron job with an override to run the checks and call a different healthchecks url. (Which didn't previously work because of this bug).

I'm curious if you have any suggestions on how to best handle this?

Based on some quick initial testing this seems to work great! I just deleted the checks dir and started borgmatic check then I'll rerun it to make sure everything looks good. One thing that still doesn't work great for me, though, is the fact that when the checks do actually run, my health checks go down because it takes so much longer than a normal run. This why I'd previously disabled checks in my config files, then had a separate cron job with an override to run the checks and call a different healthchecks url. (Which didn't previously work because of this bug). I'm curious if you have any suggestions on how to best handle this?

witten commented 2023-05-22 18:02:41 +00:00

Owner

Copy Link

#518 is probably the feature you're looking for to solve this. It would let you define different Healthchecks URLs for different borgmatic actions.

Glad to hear initial testing is working out for you though!

#518 is probably the feature you're looking for to solve this. It would let you define different Healthchecks URLs for different borgmatic actions. Glad to hear initial testing is working out for you though!

kaysond commented 2023-05-22 19:02:56 +00:00

Author

Copy Link

That's exactly it! FYI - the second round of testing worked as well.

That's exactly it! FYI - the second round of testing worked as well.

witten commented 2023-05-22 21:36:24 +00:00

Owner

Copy Link

Awesome!

Awesome!

kaysond referenced this issue 2023-05-28 16:21:28 +00:00

borgmatic check doesn't work if only repo checks are run #704

kaysond referenced this issue 2023-05-28 16:26:37 +00:00

borgmatic doesn't show the borg error output for local repositories #705

witten referenced this issue 2023-05-31 02:48:37 +00:00

borgmatic check doesn't work if only repo checks are run #704

witten referenced this issue 2023-05-31 06:20:34 +00:00

borgmatic check doesn't work if only repo checks are run #704

witten commented 2023-06-04 05:29:27 +00:00

Owner

Copy Link

Just released as part of borgmatic 1.7.14!

Just released as part of borgmatic 1.7.14!

witten referenced this issue 2023-06-18 03:06:59 +00:00

'Not a directory' errors when running consistency check #713

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

No results found.

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.14

1.8.13

1.8.12

1.8.11

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.24

1.5.23

1.5.22

1.5.21

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.14

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.7.dev0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.22

1.4.21

1.4.20

1.4.19

1.4.18

1.4.17

1.4.16

1.4.15

1.4.14

1.4.13

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.26

1.3.25

1.3.24

1.3.23

1.3.22

1.3.21

1.3.20

1.3.19

1.3.18

1.3.17

1.3.16

1.3.15

1.3.14

1.3.13

1.3.12

1.3.11

1.3.10

1.3.9

1.3.8

1.3.7

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.18

1.2.17

1.2.16

1.2.15

1.2.14

1.2.13

1.2.12

1.2.11

1.2.10

1.2.9

1.2.8

1.2.7

1.2.6

1.2.5

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.15

1.1.14

1.1.13

1.1.12

1.1.11

1.1.10

1.1.9

1.1.7

1.1.6

1.1.5

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.3

1.0.2

1.0.1

1.0.0

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

0.0.2

Labels

Clear labels

blocked

bug

data loss

design finalized

good first issue

new feature area

question / support

security

waiting for response

No Label

bug

Milestone

No items

No Milestone

Assignees

Clear assignees

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#688

No description provided.