"Structure needs cleaning"/"Orphan file not empty on read-only fs" when snapshotting an ext4 fs via LVM #1126

New issue

Closed

opened 2025-08-11 15:28:26 +00:00 by 30p87 · 2 comments

30p87 commented 2025-08-11 15:28:26 +00:00

Copy link

What I'm trying to do and why

Snapshot and mount my in-use ext4-on-LVM data fs; via borgmatic and manually, via LVM.

Steps to reproduce

1. Have original LV in-use, with orphan inodes (so an in-use fs, eg. using a loop dev to test does very likely not work)
2. Snapshot it read-only via LVM
3. If the original fs is already marked as needs-recovery, EXT4-fs fails to recover it due to it being r/o
4. Mount it as r/w
5. Observe EXT4-fs always removing the same amount of orphans, leading to the conclusion that, indeed, the needs-recovery tag is inherited from the original LV

Actual behavior

r/o snapshot:

EXT4-fs (dm-7): write access unavailable, skipping orphan cleanup
EXT4-fs (dm-7): recovery complete
EXT4-fs error (device dm-7): ext4_mark_recovery_complete:6262: comm mount: Orphan file not empty on read-only fs.
EXT4-fs (dm-7): mount failed

r/w snapshot:
EXT4-fs (dm-7): orphan cleanup on readonly fs
EXT4-fs (dm-7): 130 orphan inodes deleted
EXT4-fs (dm-7): recovery complete
EXT4-fs (dm-7): mounted filesystem 32ce7327-b2f0-4375-ae1b-23292898722a ro with ordered data mode. Quota mode: none.

Expected behavior

Snapshot as r/w by default, or check if needed beforehand (eg. sudo debugfs -R features /dev/data/data 2>&1 | grep needs_recovery)

Other notes / implementation ideas

Edit /usr/lib/python3.13/site-packages/borgmatic/hooks/data_source/lvm.py, line 107:
--- 'r', # Read-only.
+++ 'rw', # Read-write.

as quick fix; does not survive a package upgrade ofc

borgmatic version

2.0.7

borgmatic installation method

Official Arch repos

Borg version

borg 1.4.1

Python version

Python 3.13.5

Database version (if applicable)

No response

Operating system and version

NAME="Arch Linux" (latest testing packages)

### What I'm trying to do and why Snapshot and mount my in-use ext4-on-LVM data fs; via borgmatic and manually, via LVM. ### Steps to reproduce 1. Have original LV in-use, with orphan inodes (so an in-use fs, eg. using a loop dev to test does very likely not work) 2. Snapshot it read-only via LVM 3. If the original fs is already marked as needs-recovery, EXT4-fs fails to recover it due to it being r/o 4. Mount it as r/w 5. Observe EXT4-fs always removing the same amount of orphans, leading to the conclusion that, indeed, the needs-recovery tag is inherited from the original LV ### Actual behavior r/o snapshot: EXT4-fs (dm-7): write access unavailable, skipping orphan cleanup EXT4-fs (dm-7): recovery complete EXT4-fs error (device dm-7): ext4_mark_recovery_complete:6262: comm mount: Orphan file not empty on read-only fs. EXT4-fs (dm-7): mount failed r/w snapshot: EXT4-fs (dm-7): orphan cleanup on readonly fs EXT4-fs (dm-7): 130 orphan inodes deleted EXT4-fs (dm-7): recovery complete EXT4-fs (dm-7): mounted filesystem 32ce7327-b2f0-4375-ae1b-23292898722a ro with ordered data mode. Quota mode: none. ### Expected behavior Snapshot as r/w by default, or check if needed beforehand (eg. sudo debugfs -R features /dev/data/data 2>&1 | grep needs_recovery) ### Other notes / implementation ideas Edit /usr/lib/python3.13/site-packages/borgmatic/hooks/data_source/lvm.py, line 107: --- 'r', # Read-only. +++ 'rw', # Read-write. as quick fix; does not survive a package upgrade ofc ### borgmatic version 2.0.7 ### borgmatic installation method Official Arch repos ### Borg version borg 1.4.1 ### Python version Python 3.13.5 ### Database version (if applicable) _No response_ ### Operating system and version NAME="Arch Linux" (latest testing packages)

30p87 changed title from "Orphan file not empty on read-only fs" when snapshotting an ext4 fs via LVM to "Structure needs cleaning"/"Orphan file not empty on read-only fs" when snapshotting an ext4 fs via LVM 2025-08-11 15:40:00 +00:00

witten commented 2025-08-11 17:01:32 +00:00

Owner

Copy link

Thanks for filing this and providing all the detail! I went ahead and went with your r -> rw solution for simplicity. This is changed in main and will be part of the next release.

Thanks for filing this and providing all the detail! I went ahead and went with your `r` -> `rw` solution for simplicity. This is changed in main and will be part of the next release.

witten closed this issue 2025-08-11 17:01:32 +00:00

witten referenced this issue from a commit 2025-08-11 17:01:34 +00:00

Create LVM snapshots as read-write to avoid an error when snapshotting ext4 filesystems with orphaned files that need recovery (#1126).

witten commented 2025-09-20 00:41:14 +00:00

Owner

Copy link

Released in borgmatic 2.0.8!

Released in borgmatic 2.0.8!

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

parallel-borgmatic

command-hook-steps

2.1.7

2.1.6

2.1.5

2.1.4

2.1.3

2.1.2

2.1.1

2.1.0

2.0.13

2.0.12

2.0.11

2.0.10

2.0.9

2.0.8

2.0.7

2.0.6

2.0.5

2.0.4

2.0.3

2.0.2

2.0.1

2.0.0

1.9.14

1.9.13

1.9.12

1.9.11

1.9.10

1.9.9

1.9.8

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.14

1.8.13

1.8.12

1.8.11

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.24

1.5.23

1.5.22

1.5.21

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.14

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.7.dev0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.22

1.4.21

1.4.20

1.4.19

1.4.18

1.4.17

1.4.16

1.4.15

1.4.14

1.4.13

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.26

1.3.25

1.3.24

1.3.23

1.3.22

1.3.21

1.3.20

1.3.19

1.3.18

1.3.17

1.3.16

1.3.15

1.3.14

1.3.13

1.3.12

1.3.11

1.3.10

1.3.9

1.3.8

1.3.7

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.18

1.2.17

1.2.16

1.2.15

1.2.14

1.2.13

1.2.12

1.2.11

1.2.10

1.2.9

1.2.8

1.2.7

1.2.6

1.2.5

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.15

1.1.14

1.1.13

1.1.12

1.1.11

1.1.10

1.1.9

1.1.7

1.1.6

1.1.5

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.3

1.0.2

1.0.1

1.0.0

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

0.0.2

Labels

Clear labels   

blocked

  

breaking

  

bug

  

data loss

  

design finalized

  

good first issue

  

new feature area

  

question / support

  

security

  

waiting for response

No labels

blocked

breaking

bug

data loss

design finalized

good first issue

new feature area

question / support

security

waiting for response

Milestone

Clear milestone

No items

No milestone

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

borgmatic-collective/borgmatic#1126

No description provided.