Recreate not removing excluded files when using lvm snapshots #1053

New issue

Closed

opened 2025-04-03 19:42:14 +00:00 by bb · 7 comments

bb commented 2025-04-03 19:42:14 +00:00

Contributor

Copy link

What I'm trying to do and why

I tried the new recreate command after adding a few additional exclude_patterns to my config. I expected the excluded files to be gone after recreeate, but they stayed available. The main motivation was freeing up space used by big temporary files I didn't exclude during the first few backups.

Steps to reproduce

1. Create one or more backups using a config file:
2. the config file by adding some exclude_patterns, e.g. - /home/foo/bar*/tmp/*.zst
3. Create another backup to confirm that the exclude patterns actually match
4. Observe that the newly excluded files are still existent in the old archives but not existent in the newly created archive, e.g. by running borgmatic -c foo.yaml -v 2 list --find "/home/foo/bar*/tmp/*.zst"
5. Run borgmatic recreate to get rid of all newly excluded files in all archives:borgmatic -c foo -v 2 recreate (observe that borgmatic is actually looking into each old archive)
6. run borgmatic -c foo.yaml -v 2 prune compact (I'm actually not sure if this is needed. Is it? I guess it's not needed for listing, but it might be needed to actually free up the space. Would be great if this was documented.
   • EDIT: it's not needed to get files out of the listing. It's not needed to get the new deduplicated size shown in info. But it is needed to actually free up the space. Running compact logged Remote: compaction freed about 28.37 GB repository space. after applying the workaround mentioned below.)

Actual behavior

list --find still finds the excluded files in the older archives

Expected behavior

list --find finds nothing

Other notes / implementation ideas

low prio issue I run into while testing: existing temporary target

While trying this and that, I interrupted borgmatic using ctrl-c. Later, a recreate was failing due to Archive foo-2025-03-22T03:30:03.675584.recreate already exists. Maybe it could delete that or just chose another name (at least when not using --target)

low prio question

While waiting to recreate a dozen archives, I wondered why I cannot run read-only operations like info, list without acquiring an exclusive lock.

observation and possible reason for this issue

list --find lists the files like this:
tmp/borgmatic-b7547a4w/borgmatic/lvm_snapshots/3751af5bd81011e6d21d/home/foo/bar-baz/tmp/baz_latest.tar.zst

I guess recreate needs to respect the lvm-specific prefix. This may also be relevant for the other snapshotting mechanisms like zfs, btffs.

verification and workaround

To verify this theory, I changed the exclude from - /home/foo/bar*/tmp/*.zst to - '**/home/foo/bar*/tmp/*.zst' and re-ran recreate. This actually removed the files from the old archives. After that, compact freed up the space.

borgmatic version

2.0.0.dev0 (HEAD is d5d04b89dc)

borgmatic installation method

pipx install --editable .

Borg version

borg 1.4.0 (with borg 1.2.8 on the server; the server is a Hetzner storage box)

Python version

Python 3.13.2

Database version (if applicable)

No response

Operating system and version

No response

### What I'm trying to do and why I tried the new recreate command after adding a few additional exclude_patterns to my config. I expected the excluded files to be gone after recreeate, but they stayed available. The main motivation was freeing up space used by big temporary files I didn't exclude during the first few backups. ### Steps to reproduce 1. Create one or more backups using a config file: 2. the config file by adding some `exclude_patterns`, e.g. ` - /home/foo/bar*/tmp/*.zst` 3. Create another backup to confirm that the exclude patterns actually match 4. Observe that the newly excluded files are still existent in the old archives but not existent in the newly created archive, e.g. by running `borgmatic -c foo.yaml -v 2 list --find "/home/foo/bar*/tmp/*.zst"` 5. Run borgmatic recreate to get rid of all newly excluded files in all archives:`borgmatic -c foo -v 2 recreate` (observe that borgmatic is actually looking into each old archive) 6. run `borgmatic -c foo.yaml -v 2 prune compact` (I'm actually not sure if this is needed. Is it? I guess it's not needed for listing, but it might be needed to actually free up the space. Would be great if this was documented. * EDIT: it's not needed to get files out of the listing. It's not needed to get the new deduplicated size shown in `info`. But it *is* needed to actually free up the space. Running compact logged `Remote: compaction freed about 28.37 GB repository space.` after applying the workaround mentioned below.) ### Actual behavior list --find still finds the excluded files in the older archives ### Expected behavior list --find finds nothing ### Other notes / implementation ideas #### low prio issue I run into while testing: existing temporary target While trying this and that, I interrupted borgmatic using ctrl-c. Later, a `recreate` was failing due to `Archive foo-2025-03-22T03:30:03.675584.recreate already exists`. Maybe it could delete that or just chose another name (at least when *not* using `--target`) #### low prio question While waiting to recreate a dozen archives, I wondered why I cannot run read-only operations like info, list without acquiring an exclusive lock. #### observation and possible reason for this issue `list --find` lists the files like this: `tmp/borgmatic-b7547a4w/borgmatic/lvm_snapshots/3751af5bd81011e6d21d/home/foo/bar-baz/tmp/baz_latest.tar.zst` **I guess `recreate` needs to respect the lvm-specific prefix.** This may also be relevant for the other snapshotting mechanisms like zfs, btffs. #### verification and workaround To verify this theory, I changed the exclude from ` - /home/foo/bar*/tmp/*.zst` to ` - '**/home/foo/bar*/tmp/*.zst'` and re-ran `recreate`. This actually removed the files from the old archives. After that, `compact` freed up the space. ### borgmatic version 2.0.0.dev0 (HEAD is d5d04b89dcc8b509d898a6559bab2999972b1e13) ### borgmatic installation method pipx install --editable . ### Borg version borg 1.4.0 (with borg 1.2.8 on the server; the server is a Hetzner storage box) ### Python version Python 3.13.2 ### Database version (if applicable) _No response_ ### Operating system and version _No response_

bb changed title from Recreate not pruning/compacting files when using lvm snapshots to Recreate not removing excluded files when using lvm snapshots 2025-04-03 19:42:58 +00:00

witten referenced this issue from a commit 2025-04-03 19:56:01 +00:00

Add action "--help" note about running compact after recreate (#1053).

witten commented 2025-04-03 20:02:27 +00:00

Owner

Copy link

Wow, thanks for taking this unreleased feature for a spin! That's brave of you. I think your theory is correct that the exclude patterns aren't matching because of how the LVM snapshot paths are stored.

The good news is that the "fix" for LVM paths being stored that way is to upgrade your server to Borg 1.4.x+. With that version of Borg, borgmatic stores LVM snapshot paths without the tmp/borgmatic-b7547a4w/borgmatic/lvm_snapshots/3751af5bd81011e6d21d path prefix, so the archive file path looks just like any other file—which your exclude patterns will match. With older versions of Borg (like 1.2.8), the path stripping feature isn't available and so borgmatic falls back to storing the "full" snapshot path. (This is mentioned briefly in the docs.)

The bad news is that this doesn't help you with any of your already-snapshotted file paths. But the **/ exclude path work-around you discovered should address the problem. And because this is a problem solved "properly" by newer versions of Borg, I'm not really wild about modifying recreate or exclude patterns in general to compensate for it.

Other questions you had:

• Yes, the compact is needed to actually free up space. I just added this to the --help at your suggestion.
• The "archive already exists" issue: When I get a chance, I'll look into this to see if anything can be done!
• Exclusive lock: I think this is just how Borg 1 works. You might consider using --bypass-lock, which sadly is still not implemented in borgmatic (#427), but Borg supports it natively. I will say that I don't think this is an issue with Borg 2.

Wow, thanks for taking this unreleased feature for a spin! That's brave of you. I think your theory is correct that the exclude patterns aren't matching because of how the LVM snapshot paths are stored. The good news is that the "fix" for LVM paths being stored that way is to upgrade your server to Borg 1.4.x+. With that version of Borg, borgmatic stores LVM snapshot paths without the `tmp/borgmatic-b7547a4w/borgmatic/lvm_snapshots/3751af5bd81011e6d21d` path prefix, so the archive file path looks just like any other file—which your exclude patterns will match. With older versions of Borg (like 1.2.8), the path stripping feature isn't available and so borgmatic falls back to storing the "full" snapshot path. (This is mentioned briefly in [the docs](https://torsion.org/borgmatic/docs/how-to/snapshot-your-filesystems/#subvolume-discovery).) The bad news is that this doesn't help you with any of your already-snapshotted file paths. But the `**/` exclude path work-around you discovered should address the problem. And because this is a problem solved "properly" by newer versions of Borg, I'm not really wild about modifying `recreate` or exclude patterns in general to compensate for it. Other questions you had: * Yes, the compact is needed to actually free up space. I just added this to the `--help` at your suggestion. * The "archive already exists" issue: When I get a chance, I'll look into this to see if anything can be done! * Exclusive lock: I think this is just how Borg 1 works. You might consider using `--bypass-lock`, which sadly is still not implemented in borgmatic (#427), but Borg supports it natively. I will say that I don't think this is an issue with Borg 2.

bb commented 2025-04-03 20:51:06 +00:00

Author

Contributor

Copy link

Thank you for this instant and elaborate response including answers to all questions.

It's nice to see your docs fix is not only used when calling with --help on the the command line but also on the website. That's DRY 😎

I like your priority: if it's fixed in borg 1.4, don't waste time on it. I guess I'll stick with the server-side borg version provided by Hetzner for now, but it's nice to know I could configure remote_path if I wanted to.

I'm afraid automatically just prefixing **/* (like with list --find) might even be dangerous if the exclude pattern is not very specific, so it's better up to the user to use this workaround manually, if desired.

Thank you for this instant and elaborate response including answers to all questions. It's nice to see your docs fix is not only used when calling with `--help` on the the command line but also on the website. That's DRY 😎 I like your priority: if it's fixed in borg 1.4, don't waste time on it. I guess I'll stick with the server-side borg version provided by Hetzner for now, but it's nice to know I _could_ configure `remote_path` if I wanted to. I'm afraid automatically just prefixing `**/*` (like with list --find) might even be dangerous if the exclude pattern is not very specific, so it's better up to the user to use this workaround manually, if desired.

bb closed this issue 2025-04-03 20:51:06 +00:00

witten commented 2025-04-03 22:16:11 +00:00

Owner

Copy link

I appreciate your understanding here! I'm actually going to reopen this ticket though so I remember to look into the "archive already exists" issue.

I appreciate your understanding here! I'm actually going to reopen this ticket though so I remember to look into the "archive already exists" issue.

👍 1

witten 2025-04-03 22:16:12 +00:00

• reopened this issue
• added the
  question / support
  label

witten referenced this issue from a commit 2025-04-08 17:13:22 +00:00

Display a nicer error message when running the "recreate" action on a leftover temporary archive from a prior recreate run (#1053).

witten referenced this issue from a commit 2025-04-08 17:59:06 +00:00

Display a nicer error message when the "recreate" action encounters an archive that already exists (#1053).

witten commented 2025-04-08 18:01:48 +00:00

Owner

Copy link

The solution I ended on for this one was to just display a nicer error message:

$ borgmatic -c test.yaml recreate --archive blah -v 2
...
Error running actions for repository
The archive "blah.recreate" is leftover from a prior recreate. Delete it first or select a different archive.

There are also a few other variants of this error message now for related Borg recreate errors.

I went this way because I didn't feel great about auto-deleting an archive that a user wasn't explicitly requesting to be deleted, even if it's "just" a temporary archive. And it's not always a temporary archive; borgmatic recreate --target foo can result in a Borg "archive already exists" error even though foo is not a temporary recreate archive.

Anyway, this is implemented in main and will be part of the next release.

The solution I ended on for this one was to just display a nicer error message: ``` $ borgmatic -c test.yaml recreate --archive blah -v 2 ... Error running actions for repository The archive "blah.recreate" is leftover from a prior recreate. Delete it first or select a different archive. ``` There are also a few other variants of this error message now for related Borg recreate errors. I went this way because I didn't feel great about auto-deleting an archive that a user wasn't explicitly requesting to be deleted, even if it's "just" a temporary archive. And it's not always a temporary archive; `borgmatic recreate --target foo` can result in a Borg "archive already exists" error even though `foo` is not a temporary recreate archive. Anyway, this is implemented in main and will be part of the next release.

🚀 1

witten closed this issue 2025-04-08 18:01:48 +00:00

witten commented 2025-04-08 19:14:32 +00:00

Owner

Copy link

Released in borgmatic 2.0.2!

Released in borgmatic 2.0.2!

👍 1

bb commented 2025-04-16 10:22:30 +00:00

Author

Contributor

Copy link

I want to amend that Hetzner Storage Boxes actually allow choosing between Borg 1.1, 1.2 and 1.4, so we can actually use borg 1.4 by configuring remote_path: borg-1.4. The versions supported by Hetzner, the remote path argument and the current default version are documented here.

I want to amend that Hetzner Storage Boxes actually allow choosing between Borg 1.1, 1.2 and 1.4, so we can actually use borg 1.4 by configuring `remote_path: borg-1.4`. The versions supported by Hetzner, the remote path argument and the current default version are documented [here](https://docs.hetzner.com/storage/storage-box/access/access-ssh-rsync-borg/#borgbackup).

witten commented 2025-04-17 23:55:27 +00:00

Owner

Copy link

Awesome, glad to hear you can change that on the server side!

Awesome, glad to hear you can change that on the server side!

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

parallel-borgmatic

command-hook-steps

2.1.7

2.1.6

2.1.5

2.1.4

2.1.3

2.1.2

2.1.1

2.1.0

2.0.13

2.0.12

2.0.11

2.0.10

2.0.9

2.0.8

2.0.7

2.0.6

2.0.5

2.0.4

2.0.3

2.0.2

2.0.1

2.0.0

1.9.14

1.9.13

1.9.12

1.9.11

1.9.10

1.9.9

1.9.8

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.14

1.8.13

1.8.12

1.8.11

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.24

1.5.23

1.5.22

1.5.21

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.14

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.7.dev0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.22

1.4.21

1.4.20

1.4.19

1.4.18

1.4.17

1.4.16

1.4.15

1.4.14

1.4.13

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.26

1.3.25

1.3.24

1.3.23

1.3.22

1.3.21

1.3.20

1.3.19

1.3.18

1.3.17

1.3.16

1.3.15

1.3.14

1.3.13

1.3.12

1.3.11

1.3.10

1.3.9

1.3.8

1.3.7

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.18

1.2.17

1.2.16

1.2.15

1.2.14

1.2.13

1.2.12

1.2.11

1.2.10

1.2.9

1.2.8

1.2.7

1.2.6

1.2.5

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.15

1.1.14

1.1.13

1.1.12

1.1.11

1.1.10

1.1.9

1.1.7

1.1.6

1.1.5

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.3

1.0.2

1.0.1

1.0.0

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

0.0.2

Labels

Clear labels   

blocked

  

breaking

  

bug

  

data loss

  

design finalized

  

good first issue

  

new feature area

  

question / support

  

security

  

waiting for response

No labels

blocked

breaking

bug

data loss

design finalized

good first issue

new feature area

question / support

security

waiting for response

Milestone

Clear milestone

No items

No milestone

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

borgmatic-collective/borgmatic#1053

No description provided.