multiple source_directories broken since version 1.5.5 #333

New issue

Closed

opened 2020-06-19 18:11:53 +00:00 by WladyX · 8 comments

WladyX commented 2020-06-19 18:11:53 +00:00

Copy link

Hi,
From borgmatic version 1.5.5 borgmatic does not back up the secong folder.
/ is backed up, but /media/ssd is not backed up.
Version 1.5.4 worked as expected.

Thank you!

I have the following config file:

egrep -v '#|^$' /etc/borgmatic/config.yaml

location:
    source_directories:
        - /
        - /media/ssd
    repositories:
        - borg@tyrion.redact:pandora
        - borg@pandora.redact:pandora
    one_file_system: true
    numeric_owner: true
    atime: false
    exclude_patterns:
        - /proc
        - /dev
        - /tmp
        - /sys
        - /media/self
        - /var/spool/squid3
storage:
    ssh_command: ssh -i /root/.ssh/borg_rsa
    unknown_unencrypted_repo_access_is_ok: true
retention:
    keep_daily: 7
    keep_monthly: 6
    keep_yearly: 1
consistency:
    checks:
        - repository
        - data
        - extract
    prefix: ''
hooks:
    before_backup:
        - echo "*** Starting pandora backup ***"
        - curl -sX POST redact
    after_backup:
        - echo "******** END OF PANDORA ********"
        - curl -sX POST redact
    on_error:
        - echo "******** !!! ERROR ON PANDORA !!! ********"
        - curl -sX POST redact

same problem with this config:

    source_directories:
        - /media/xps2
        - /media/xps2/Pictures
        - /media/xps2/images

Hi, From borgmatic version 1.5.5 borgmatic does not back up the secong folder. / is backed up, but /media/ssd is not backed up. Version 1.5.4 worked as expected. Thank you! I have the following config file: ``` egrep -v '#|^$' /etc/borgmatic/config.yaml location: source_directories: - / - /media/ssd repositories: - borg@tyrion.redact:pandora - borg@pandora.redact:pandora one_file_system: true numeric_owner: true atime: false exclude_patterns: - /proc - /dev - /tmp - /sys - /media/self - /var/spool/squid3 storage: ssh_command: ssh -i /root/.ssh/borg_rsa unknown_unencrypted_repo_access_is_ok: true retention: keep_daily: 7 keep_monthly: 6 keep_yearly: 1 consistency: checks: - repository - data - extract prefix: '' hooks: before_backup: - echo "*** Starting pandora backup ***" - curl -sX POST redact after_backup: - echo "******** END OF PANDORA ********" - curl -sX POST redact on_error: - echo "******** !!! ERROR ON PANDORA !!! ********" - curl -sX POST redact ``` same problem with this config: ``` source_directories: - /media/xps2 - /media/xps2/Pictures - /media/xps2/images ```

witten commented 2020-06-19 18:20:24 +00:00

Owner

Copy link

Yikes! Thank you for reporting this. I have an idea of what might be going on here: borgmatic 1.5.5 started de-duplicating source directories to prevent Borg from scanning the same source directories twice, which can cause hangs in certain cases. So for instance, source directories of /foo and /foo/bar would get de-duplicated down to just /foo.. because /foo contains /foo/bar, and therefore Borg will implicitly backup both.

Something similiar is going on here: / contains /media/ssd, and so borgmatic is de-duplicating that down to just /. Now it sounds like that might be problematic because you have one_file_system set to true (a totally legitimate setting to have), and therefore Borg doesn't traverse from / to /media/ssd when scanning for files.

I assume that your backup archives contained the contents of /media/ssd before borgmatic 1.5.5, and /media/ssd is missing from archives afterwards? I'm just trying to understand how one_file_system interacts with multiple Borg source directories. Thanks.

Yikes! Thank you for reporting this. I have an idea of what might be going on here: borgmatic 1.5.5 started de-duplicating source directories to prevent Borg from scanning the same source directories twice, which can cause hangs in certain cases. So for instance, source directories of `/foo` and `/foo/bar` would get de-duplicated down to just `/foo`.. because `/foo` contains `/foo/bar`, and therefore Borg will implicitly backup both. Something similiar is going on here: `/` contains `/media/ssd`, and so borgmatic is de-duplicating that down to just `/`. Now it sounds like that might be problematic because you have `one_file_system` set to `true` (a totally legitimate setting to have), and therefore Borg doesn't traverse from `/` to `/media/ssd` when scanning for files. I assume that your backup archives contained the contents of `/media/ssd` before borgmatic 1.5.5, and `/media/ssd` is missing from archives afterwards? I'm just trying to understand how `one_file_system` interacts with multiple Borg source directories. Thanks.

witten added the

bug

label 2020-06-19 18:20:29 +00:00

WladyX commented 2020-06-19 18:23:51 +00:00

Author

Copy link

Wow, fastest issue reply ever :) Thanks for that.

Yes, I confirm having /media/ssd in earlier backups.

Wow, fastest issue reply ever :) Thanks for that. Yes, I confirm having /media/ssd in earlier backups.

witten commented 2020-06-19 18:32:31 +00:00

Owner

Copy link

If only the implementation of the fix could be so speedy! I'll have to think about what I can do here without regressing #316.

If only the implementation of the fix could be so speedy! I'll have to think about what I can do here without regressing #316.

witten added the

data loss

label 2020-06-19 19:25:09 +00:00

witten commented 2020-06-20 03:17:45 +00:00

Owner

Copy link

Okay, I've got an apparent fix in master. borgmatic now checks if two source directories are on the same filesystem before trying to de-duplicate them. If they're on different filesystems, as is the case with your first example, then borgmatic passes both of them to Borg.

Thanks again for bringing this to my attention. I'll add a note here when there's a release with this fix.

same problem with this config:

source_directories:
    - /media/xps2
    - /media/xps2/Pictures
    - /media/xps2/images

A question about this. Are /media/xps2 and its sub-directories on the same filesystem? In any case, I would expect borgmatic 1.5.5/1.5.6 to de-duplicate them down to just /media/xps2 when passing them to Borg. But if they are all on the same filesystem, then Borg should implicitly backup both /media/xps2 and all of its sub-directories.

Is that consistent with what you're seeing in the resulting archive? For instance, is /media/xps2/Pictures present there?

Okay, I've got an apparent fix in master. borgmatic now checks if two source directories are on the same filesystem before trying to de-duplicate them. If they're on different filesystems, as is the case with your first example, then borgmatic passes both of them to Borg. Thanks again for bringing this to my attention. I'll add a note here when there's a release with this fix. > same problem with this config: > > source_directories: > - /media/xps2 > - /media/xps2/Pictures > - /media/xps2/images A question about this. Are `/media/xps2` and its sub-directories on the same filesystem? In any case, I would expect borgmatic 1.5.5/1.5.6 to de-duplicate them down to just `/media/xps2` when passing them to Borg. But if they are all on the same filesystem, then Borg *should* implicitly backup both `/media/xps2` and all of its sub-directories. Is that consistent with what you're seeing in the resulting archive? For instance, is `/media/xps2/Pictures` present there?

witten referenced this issue from a commit 2020-06-20 03:19:18 +00:00

Fix for potential data loss (data not getting backed up) when borgmatic omitted configured source directories in certain situations (#333).

WladyX commented 2020-06-20 07:12:57 +00:00

Author

Copy link

Hey,
Thank you also for trying to fix this so quickly and also for this amazing wrapper :)
/media/xps2 = ext4 fs
/media/xps2/Pictures and /media/xps2/images = encfs mounts. So they should be considered different filesystems.
Once you have a release with this fix i can test both cases.
/media/xps2/Pictures is present but empty in the faulty backups.

Hey, Thank you also for trying to fix this so quickly and also for this amazing wrapper :) /media/xps2 = ext4 fs /media/xps2/Pictures and /media/xps2/images = encfs mounts. So they should be considered different filesystems. Once you have a release with this fix i can test both cases. /media/xps2/Pictures is present but empty in the faulty backups.

witten commented 2020-06-20 21:08:47 +00:00

Owner

Copy link

Great, that sounds like that case should be handled!

Great, that sounds like that case should be handled!

witten commented 2020-06-21 21:45:45 +00:00

Owner

Copy link

I'll add a note here when the fix is released (hopefully soon!).

I'll add a note here when the fix is released (hopefully soon!).

witten closed this issue 2020-06-21 21:45:45 +00:00

witten commented 2020-06-23 18:28:03 +00:00

Owner

Copy link

Released in borgmatic 1.5.7!

Released in borgmatic 1.5.7!

sysedit referenced this issue 2021-01-27 16:23:24 +00:00

Behavior change in 1.5.7 with missing locations #387

witten referenced this issue 2021-01-28 23:51:08 +00:00

Behavior change in 1.5.7 with missing locations #387

witten referenced this issue 2021-10-11 17:39:35 +00:00

Behavior change in 1.5.7 with missing locations #387

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

parallel-borgmatic

command-hook-steps

2.1.7

2.1.6

2.1.5

2.1.4

2.1.3

2.1.2

2.1.1

2.1.0

2.0.13

2.0.12

2.0.11

2.0.10

2.0.9

2.0.8

2.0.7

2.0.6

2.0.5

2.0.4

2.0.3

2.0.2

2.0.1

2.0.0

1.9.14

1.9.13

1.9.12

1.9.11

1.9.10

1.9.9

1.9.8

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.14

1.8.13

1.8.12

1.8.11

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.24

1.5.23

1.5.22

1.5.21

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.14

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.7.dev0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.22

1.4.21

1.4.20

1.4.19

1.4.18

1.4.17

1.4.16

1.4.15

1.4.14

1.4.13

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.26

1.3.25

1.3.24

1.3.23

1.3.22

1.3.21

1.3.20

1.3.19

1.3.18

1.3.17

1.3.16

1.3.15

1.3.14

1.3.13

1.3.12

1.3.11

1.3.10

1.3.9

1.3.8

1.3.7

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.18

1.2.17

1.2.16

1.2.15

1.2.14

1.2.13

1.2.12

1.2.11

1.2.10

1.2.9

1.2.8

1.2.7

1.2.6

1.2.5

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.15

1.1.14

1.1.13

1.1.12

1.1.11

1.1.10

1.1.9

1.1.7

1.1.6

1.1.5

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.3

1.0.2

1.0.1

1.0.0

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

0.0.2

Labels

Clear labels   

blocked

  

breaking

  

bug

  

data loss

  

design finalized

  

good first issue

  

new feature area

  

question / support

  

security

  

waiting for response

No labels

blocked

breaking

bug

data loss

design finalized

good first issue

new feature area

question / support

security

waiting for response

Milestone

Clear milestone

No items

No milestone

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

borgmatic-collective/borgmatic#333

No description provided.