borgmatic-collective/borgmatic
borgmatic-collective
/
borgmatic
15
57
Fork 40
Code Issues 53 Pull Requests 2 Actions Packages Releases 121 Activity

multiple source_directories broken since version 1.5.5 #333

New Issue
Closed
opened 2020-06-19 18:11:53 +00:00 by WladyX · 8 comments
WladyX commented 2020-06-19 18:11:53 +00:00
Copy Link

Hi,
From borgmatic version 1.5.5 borgmatic does not back up the secong folder.
/ is backed up, but /media/ssd is not backed up.
Version 1.5.4 worked as expected.

Thank you!

I have the following config file:

egrep -v '#|^$' /etc/borgmatic/config.yaml

location:
    source_directories:
        - /
        - /media/ssd
    repositories:
        - borg@tyrion.redact:pandora
        - borg@pandora.redact:pandora
    one_file_system: true
    numeric_owner: true
    atime: false
    exclude_patterns:
        - /proc
        - /dev
        - /tmp
        - /sys
        - /media/self
        - /var/spool/squid3
storage:
    ssh_command: ssh -i /root/.ssh/borg_rsa
    unknown_unencrypted_repo_access_is_ok: true
retention:
    keep_daily: 7
    keep_monthly: 6
    keep_yearly: 1
consistency:
    checks:
        - repository
        - data
        - extract
    prefix: ''
hooks:
    before_backup:
        - echo "*** Starting pandora backup ***"
        - curl -sX POST redact
    after_backup:
        - echo "******** END OF PANDORA ********"
        - curl -sX POST redact
    on_error:
        - echo "******** !!! ERROR ON PANDORA !!! ********"
        - curl -sX POST redact

same problem with this config:

    source_directories:
        - /media/xps2
        - /media/xps2/Pictures
        - /media/xps2/images
Hi, From borgmatic version 1.5.5 borgmatic does not back up the secong folder. / is backed up, but /media/ssd is not backed up. Version 1.5.4 worked as expected. Thank you! I have the following config file: ``` egrep -v '#|^$' /etc/borgmatic/config.yaml location: source_directories: - / - /media/ssd repositories: - borg@tyrion.redact:pandora - borg@pandora.redact:pandora one_file_system: true numeric_owner: true atime: false exclude_patterns: - /proc - /dev - /tmp - /sys - /media/self - /var/spool/squid3 storage: ssh_command: ssh -i /root/.ssh/borg_rsa unknown_unencrypted_repo_access_is_ok: true retention: keep_daily: 7 keep_monthly: 6 keep_yearly: 1 consistency: checks: - repository - data - extract prefix: '' hooks: before_backup: - echo "*** Starting pandora backup ***" - curl -sX POST redact after_backup: - echo "******** END OF PANDORA ********" - curl -sX POST redact on_error: - echo "******** !!! ERROR ON PANDORA !!! ********" - curl -sX POST redact ``` same problem with this config: ``` source_directories: - /media/xps2 - /media/xps2/Pictures - /media/xps2/images ```
witten commented 2020-06-19 18:20:24 +00:00
Owner
Copy Link

Yikes! Thank you for reporting this. I have an idea of what might be going on here: borgmatic 1.5.5 started de-duplicating source directories to prevent Borg from scanning the same source directories twice, which can cause hangs in certain cases. So for instance, source directories of /foo and /foo/bar would get de-duplicated down to just /foo.. because /foo contains /foo/bar, and therefore Borg will implicitly backup both.

Something similiar is going on here: / contains /media/ssd, and so borgmatic is de-duplicating that down to just /. Now it sounds like that might be problematic because you have one_file_system set to true (a totally legitimate setting to have), and therefore Borg doesn't traverse from / to /media/ssd when scanning for files.

I assume that your backup archives contained the contents of /media/ssd before borgmatic 1.5.5, and /media/ssd is missing from archives afterwards? I'm just trying to understand how one_file_system interacts with multiple Borg source directories. Thanks.

Yikes! Thank you for reporting this. I have an idea of what might be going on here: borgmatic 1.5.5 started de-duplicating source directories to prevent Borg from scanning the same source directories twice, which can cause hangs in certain cases. So for instance, source directories of `/foo` and `/foo/bar` would get de-duplicated down to just `/foo`.. because `/foo` contains `/foo/bar`, and therefore Borg will implicitly backup both. Something similiar is going on here: `/` contains `/media/ssd`, and so borgmatic is de-duplicating that down to just `/`. Now it sounds like that might be problematic because you have `one_file_system` set to `true` (a totally legitimate setting to have), and therefore Borg doesn't traverse from `/` to `/media/ssd` when scanning for files. I assume that your backup archives contained the contents of `/media/ssd` before borgmatic 1.5.5, and `/media/ssd` is missing from archives afterwards? I'm just trying to understand how `one_file_system` interacts with multiple Borg source directories. Thanks.
witten added the
bug
 label 2020-06-19 18:20:29 +00:00
WladyX commented 2020-06-19 18:23:51 +00:00
Author
Copy Link

Wow, fastest issue reply ever :) Thanks for that.

Yes, I confirm having /media/ssd in earlier backups.

Wow, fastest issue reply ever :) Thanks for that. Yes, I confirm having /media/ssd in earlier backups.
witten commented 2020-06-19 18:32:31 +00:00
Owner
Copy Link

If only the implementation of the fix could be so speedy! I'll have to think about what I can do here without regressing #316.

If only the implementation of the fix could be so speedy! I'll have to think about what I can do here without regressing #316.
witten added the
data loss
 label 2020-06-19 19:25:09 +00:00
witten commented 2020-06-20 03:17:45 +00:00
Owner
Copy Link

Okay, I've got an apparent fix in master. borgmatic now checks if two source directories are on the same filesystem before trying to de-duplicate them. If they're on different filesystems, as is the case with your first example, then borgmatic passes both of them to Borg.

Thanks again for bringing this to my attention. I'll add a note here when there's a release with this fix.

same problem with this config:

source_directories:
    - /media/xps2
    - /media/xps2/Pictures
    - /media/xps2/images

A question about this. Are /media/xps2 and its sub-directories on the same filesystem? In any case, I would expect borgmatic 1.5.5/1.5.6 to de-duplicate them down to just /media/xps2 when passing them to Borg. But if they are all on the same filesystem, then Borg should implicitly backup both /media/xps2 and all of its sub-directories.

Is that consistent with what you're seeing in the resulting archive? For instance, is /media/xps2/Pictures present there?

Okay, I've got an apparent fix in master. borgmatic now checks if two source directories are on the same filesystem before trying to de-duplicate them. If they're on different filesystems, as is the case with your first example, then borgmatic passes both of them to Borg. Thanks again for bringing this to my attention. I'll add a note here when there's a release with this fix. > same problem with this config: > > source_directories: > - /media/xps2 > - /media/xps2/Pictures > - /media/xps2/images A question about this. Are `/media/xps2` and its sub-directories on the same filesystem? In any case, I would expect borgmatic 1.5.5/1.5.6 to de-duplicate them down to just `/media/xps2` when passing them to Borg. But if they are all on the same filesystem, then Borg *should* implicitly backup both `/media/xps2` and all of its sub-directories. Is that consistent with what you're seeing in the resulting archive? For instance, is `/media/xps2/Pictures` present there?
WladyX commented 2020-06-20 07:12:57 +00:00
Author
Copy Link

Hey,
Thank you also for trying to fix this so quickly and also for this amazing wrapper :)
/media/xps2 = ext4 fs
/media/xps2/Pictures and /media/xps2/images = encfs mounts. So they should be considered different filesystems.
Once you have a release with this fix i can test both cases.
/media/xps2/Pictures is present but empty in the faulty backups.

Hey, Thank you also for trying to fix this so quickly and also for this amazing wrapper :) /media/xps2 = ext4 fs /media/xps2/Pictures and /media/xps2/images = encfs mounts. So they should be considered different filesystems. Once you have a release with this fix i can test both cases. /media/xps2/Pictures is present but empty in the faulty backups.
witten commented 2020-06-20 21:08:47 +00:00
Owner
Copy Link

Great, that sounds like that case should be handled!

Great, that sounds like that case should be handled!
witten commented 2020-06-21 21:45:45 +00:00
Owner
Copy Link

I'll add a note here when the fix is released (hopefully soon!).

I'll add a note here when the fix is released (hopefully soon!).
witten commented 2020-06-23 18:28:03 +00:00
Owner
Copy Link

Released in borgmatic 1.5.7!

Released in borgmatic 1.5.7!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
main
spot-check
1.8.10
1.8.9
1.8.8
1.8.7
1.8.6
1.8.5
1.8.4
1.8.3
1.8.2
1.8.1
1.8.0
1.7.15
1.7.14
1.7.13
1.7.12
1.7.11
1.7.10
1.7.9
1.7.8
1.7.7
1.7.6
1.7.5
1.7.4
1.7.3
1.7.2
1.7.1
1.7.0
1.6.6
1.6.5
1.6.4
1.6.3
1.6.2
1.6.1
1.6.0
1.5.24
1.5.23
1.5.22
1.5.21
1.5.20
1.5.19
1.5.18
1.5.17
1.5.16
1.5.15
1.5.14
1.5.13
1.5.12
1.5.11
1.5.10
1.5.9
1.5.8
1.5.7
1.5.7.dev0
1.5.6
1.5.5
1.5.4
1.5.3
1.5.2
1.5.1
1.5.0
1.4.22
1.4.21
1.4.20
1.4.19
1.4.18
1.4.17
1.4.16
1.4.15
1.4.14
1.4.13
1.4.12
1.4.11
1.4.10
1.4.9
1.4.8
1.4.7
1.4.6
1.4.5
1.4.4
1.4.3
1.4.2
1.4.1
1.4.0
1.3.26
1.3.25
1.3.24
1.3.23
1.3.22
1.3.21
1.3.20
1.3.19
1.3.18
1.3.17
1.3.16
1.3.15
1.3.14
1.3.13
1.3.12
1.3.11
1.3.10
1.3.9
1.3.8
1.3.7
1.3.6
1.3.5
1.3.4
1.3.3
1.3.2
1.3.1
1.3.0
1.2.18
1.2.17
1.2.16
1.2.15
1.2.14
1.2.13
1.2.12
1.2.11
1.2.10
1.2.9
1.2.8
1.2.7
1.2.6
1.2.5
1.2.4
1.2.3
1.2.2
1.2.1
1.2.0
1.1.15
1.1.14
1.1.13
1.1.12
1.1.11
1.1.10
1.1.9
1.1.7
1.1.6
1.1.5
1.1.4
1.1.3
1.1.2
1.1.1
1.1.0
1.0.3
1.0.2
1.0.1
1.0.0
0.1.8
0.1.7
0.1.6
0.1.5
0.1.4
0.1.3
0.1.2
0.1.1
0.1.0
0.0.7
0.0.6
0.0.5
0.0.4
0.0.3
0.0.2
Labels
Clear labels   
bug

   
data loss

   
design finalized

   
good first issue

   
new feature area

   
question / support

   
security

   
waiting for response
No Label
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#333
No description provided.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?