location.patterns breaks when /root/.borgmatic exists #574

New Issue

Closed

opened 2022-08-25 09:42:04 +00:00 by Kritzefitz · 5 comments

Kritzefitz commented 2022-08-25 09:42:04 +00:00

Copy Link

What I'm trying to do and why

I'm trying to create a backup while specifying the locations using location.patterns and /root/.borgmatic exists. Reasons to do this include using location.patterns and hooks.*_databases at the same time, because the latter automatically creates /root/.borgmatic.

Steps to reproduce (if a bug)

# borg init -e none /repo
# mkdir /the-files
# echo 'the content' > /the-files/the-file
# cat /etc/borgmatic/config.yaml
location:
    source_directories: []
    repositories:
        - /repo
    patterns:
        - R /the-files

# # This is still working as expected
# borgmatic create
# borgmatic list --archive latest
/repo: Listing archive
/repo: Listing archive 3e1f0673503c-2022-08-25T09:31:38.722964
drwxr-xr-x root   root          0 Thu, 2022-08-25 09:30:04 the-files
-rw-r--r-- root   root         12 Thu, 2022-08-25 09:30:04 the-files/the-file

# # This doesn't back up the files in /the-files
# mkdir /root/.borgmatic
# borgmatic create
# borgmatic list --archive latest
/repo: Listing archive
/repo: Listing archive 3e1f0673503c-2022-08-25T09:32:37.611310
drwxr-xr-x root   root          0 Thu, 2022-08-25 09:32:30 root/.borgmatic

Actual behavior (if a bug)

As soon as /root/.borgmatic exists, it overrides all locations specified in location.patterns and only /root/.borgmatic is included in the backup.

Expected behavior (if a bug)

The roots specified by location.patterns should be included in the backup regardless of the existence of /root/.borgmatic.

Other notes / implementation ideas

This seems to happen, because borgmatic passes /root/.borgmatic as a backup directory to borg create when it exists, which seems to conflict with --patterns-from

# borgmatic create -v2
Ensuring legacy configuration is upgraded
/etc/borgmatic/config.yaml: No commands to run for pre-everything hook
borg --version --debug --show-rc
/etc/borgmatic/config.yaml: No commands to run for pre-actions hook
/etc/borgmatic/config.yaml: No commands to run for pre-backup hook
/repo: Creating archive
borg create --patterns-from /tmp/tmpdc182zsw --debug --show-rc /repo::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /root/.borgmatic
using builtin fallback logging configuration
35 self tests completed in 0.08 seconds
Verified integrity of /repo/index.13
TAM-verified manifest
security: read previous location '/repo'
security: read manifest timestamp '2022-08-25T09:32:37.689635'
security: determined newest manifest timestamp as 2022-08-25T09:32:37.689635
security: repository checks ok, allowing access
Creating archive at "/repo::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}"
Verified integrity of /root/.cache/borg/6b144ea6c59923e26643bd085d87f5227e279f6597e341a0d31755b8df0e7a6c/chunks
Reading files cache ...
Verified integrity of /root/.cache/borg/6b144ea6c59923e26643bd085d87f5227e279f6597e341a0d31755b8df0e7a6c/files
security: read previous location '/repo'
security: read manifest timestamp '2022-08-25T09:32:37.689635'
security: determined newest manifest timestamp as 2022-08-25T09:32:37.689635
security: repository checks ok, allowing access
Processing files ...
Cleaned up 0 uncommitted segment files (== everything after segment 13).
Verified integrity of /repo/hints.13
check_free_space: few segments, not requiring a full free segment
check_free_space: calculated working space for compact as 1913 bytes
check_free_space: required bytes 171081, free bytes 92029419520
security: saving state for 6b144ea6c59923e26643bd085d87f5227e279f6597e341a0d31755b8df0e7a6c to /root/.config/borg/security/6b144ea6c59923e26643bd085d87f5227e279f6597e341a0d31755b8df0e7a6c
security: current location   /repo
security: key type           2
security: manifest timestamp 2022-08-25T09:35:53.980844
terminating with success status, rc 0
/etc/borgmatic/config.yaml: No commands to run for post-backup hook
/etc/borgmatic/config.yaml: No commands to run for post-actions hook
/etc/borgmatic/config.yaml: No commands to run for post-everything hook

summary:
/etc/borgmatic/config.yaml: Successfully ran configuration file

Environment

borgmatic version: 1.7.0, 1.6.3

borgmatic installation method: via pip in docker container python:3-buster (1.7.0), as well as via apt on Debian bookworm (1.6.3)

Borg version: 1.1.9 (in docker container with borgmatic 1.7.0), 1.2.1 (on Debian bookworm with borgmatic 1.6.3)

Python version: 3.10.6 (in both cases)

Database version (if applicable): n/a, the bug was discovered while trying to use database hooks, but also appears when no actual database hook is configured.

operating system and version: docker container python:3-buster, Debian Bookworm

#### What I'm trying to do and why I'm trying to create a backup while specifying the locations using `location.patterns` and `/root/.borgmatic` exists. Reasons to do this include using `location.patterns` and `hooks.*_databases` at the same time, because the latter automatically creates `/root/.borgmatic`. #### Steps to reproduce (if a bug) ``` # borg init -e none /repo # mkdir /the-files # echo 'the content' > /the-files/the-file # cat /etc/borgmatic/config.yaml location: source_directories: [] repositories: - /repo patterns: - R /the-files # # This is still working as expected # borgmatic create # borgmatic list --archive latest /repo: Listing archive /repo: Listing archive 3e1f0673503c-2022-08-25T09:31:38.722964 drwxr-xr-x root root 0 Thu, 2022-08-25 09:30:04 the-files -rw-r--r-- root root 12 Thu, 2022-08-25 09:30:04 the-files/the-file # # This doesn't back up the files in /the-files # mkdir /root/.borgmatic # borgmatic create # borgmatic list --archive latest /repo: Listing archive /repo: Listing archive 3e1f0673503c-2022-08-25T09:32:37.611310 drwxr-xr-x root root 0 Thu, 2022-08-25 09:32:30 root/.borgmatic ``` #### Actual behavior (if a bug) As soon as `/root/.borgmatic` exists, it overrides all locations specified in `location.patterns` and only `/root/.borgmatic` is included in the backup. #### Expected behavior (if a bug) The roots specified by `location.patterns` should be included in the backup regardless of the existence of `/root/.borgmatic`. #### Other notes / implementation ideas This seems to happen, because borgmatic passes `/root/.borgmatic` as a backup directory to `borg create` when it exists, which seems to conflict with `--patterns-from` ``` # borgmatic create -v2 Ensuring legacy configuration is upgraded /etc/borgmatic/config.yaml: No commands to run for pre-everything hook borg --version --debug --show-rc /etc/borgmatic/config.yaml: No commands to run for pre-actions hook /etc/borgmatic/config.yaml: No commands to run for pre-backup hook /repo: Creating archive borg create --patterns-from /tmp/tmpdc182zsw --debug --show-rc /repo::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /root/.borgmatic using builtin fallback logging configuration 35 self tests completed in 0.08 seconds Verified integrity of /repo/index.13 TAM-verified manifest security: read previous location '/repo' security: read manifest timestamp '2022-08-25T09:32:37.689635' security: determined newest manifest timestamp as 2022-08-25T09:32:37.689635 security: repository checks ok, allowing access Creating archive at "/repo::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}" Verified integrity of /root/.cache/borg/6b144ea6c59923e26643bd085d87f5227e279f6597e341a0d31755b8df0e7a6c/chunks Reading files cache ... Verified integrity of /root/.cache/borg/6b144ea6c59923e26643bd085d87f5227e279f6597e341a0d31755b8df0e7a6c/files security: read previous location '/repo' security: read manifest timestamp '2022-08-25T09:32:37.689635' security: determined newest manifest timestamp as 2022-08-25T09:32:37.689635 security: repository checks ok, allowing access Processing files ... Cleaned up 0 uncommitted segment files (== everything after segment 13). Verified integrity of /repo/hints.13 check_free_space: few segments, not requiring a full free segment check_free_space: calculated working space for compact as 1913 bytes check_free_space: required bytes 171081, free bytes 92029419520 security: saving state for 6b144ea6c59923e26643bd085d87f5227e279f6597e341a0d31755b8df0e7a6c to /root/.config/borg/security/6b144ea6c59923e26643bd085d87f5227e279f6597e341a0d31755b8df0e7a6c security: current location /repo security: key type 2 security: manifest timestamp 2022-08-25T09:35:53.980844 terminating with success status, rc 0 /etc/borgmatic/config.yaml: No commands to run for post-backup hook /etc/borgmatic/config.yaml: No commands to run for post-actions hook /etc/borgmatic/config.yaml: No commands to run for post-everything hook summary: /etc/borgmatic/config.yaml: Successfully ran configuration file ``` #### Environment **borgmatic version:** 1.7.0, 1.6.3 **borgmatic installation method:** via pip in docker container `python:3-buster` (1.7.0), as well as via apt on Debian bookworm (1.6.3) **Borg version:** 1.1.9 (in docker container with borgmatic 1.7.0), 1.2.1 (on Debian bookworm with borgmatic 1.6.3) **Python version:** 3.10.6 (in both cases) **Database version (if applicable):** n/a, the bug was discovered while trying to use database hooks, but also appears when no actual database hook is configured. **operating system and version:** docker container `python:3-buster`, Debian Bookworm

witten commented 2022-08-25 16:15:00 +00:00

Owner

Copy Link

Yikes, that's pretty bad! Thanks for the detailed bug report. I will attempt to repro this locally and work on a fix.

Yikes, that's pretty bad! Thanks for the detailed bug report. I will attempt to repro this locally and work on a fix.

witten added the

bug

data loss

labels 2022-08-25 16:15:12 +00:00

witten commented 2022-08-25 16:44:21 +00:00

Owner

Copy Link

I've repro'd this with Borg only (1.2.1), without borgmatic:

borg create --pattern="R /root/tmp" --debug --show-rc test.borg::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /root/.borgmatic

This results in only /root/.borgmatic getting backed up. So this is either a bug in Borg or intended behavior. In either case, borgmatic probably needs to work around it.

I've repro'd this with Borg only (1.2.1), without borgmatic: ```bash borg create --pattern="R /root/tmp" --debug --show-rc test.borg::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /root/.borgmatic ``` This results in only `/root/.borgmatic` getting backed up. So this is either a bug in Borg or intended behavior. In either case, borgmatic probably needs to work around it.

witten referenced this issue 2022-08-25 18:07:12 +00:00

Make `source_directories` optional if roots are specified in patterns file #542

witten commented 2022-08-25 18:07:41 +00:00

Owner

Copy Link

The Borg dev seems to think the Borg patterns behavior described above could be a bug, so I've filed it here: https://github.com/borgbackup/borg/issues/6994. Still gotta figure out what that means for borgmatic.

The Borg dev seems to think the Borg patterns behavior described above could be a bug, so I've filed it here: https://github.com/borgbackup/borg/issues/6994. Still gotta figure out what that means for borgmatic.

witten referenced this issue from a commit 2022-08-25 18:58:17 +00:00

Fix conflict between "patterns" and "source_directories" (#574), make "source_directories" optional (#542).

witten commented 2022-08-25 19:02:22 +00:00

Owner

Copy Link

Okay, I just released a work-around for this Borg issue in borgmatic 1.7.1. Basically, when patterns is used, any source_directories (implicit or otherwise) get converted into patterns instead of being passed to Borg on the command-line. With that change, the overriding behavior you were seeing no longer occurs.

You may also be interested to know that with #542 included in this release, source_directories is now completely optional; you don't need to set it to an empty list any longer.

Thanks again for reporting this!

Okay, I just released a work-around for this Borg issue in borgmatic 1.7.1. Basically, when `patterns` is used, any `source_directories` (implicit or otherwise) get converted into patterns instead of being passed to Borg on the command-line. With that change, the overriding behavior you were seeing no longer occurs. You may also be interested to know that with #542 included in this release, `source_directories` is now completely optional; you don't need to set it to an empty list any longer. Thanks again for reporting this!

witten closed this issue 2022-08-25 19:02:22 +00:00

witten referenced this issue 2022-08-25 19:04:32 +00:00

Make `source_directories` optional if roots are specified in patterns file #542

Kritzefitz commented 2022-08-25 19:49:01 +00:00

Author

Copy Link

Thank you, for the thourough investigation and very timely fix!

Thank you, for the thourough investigation and very timely fix!

👍 1

witten referenced this issue 2022-09-08 15:44:43 +00:00

borgmatic hangs when location.patterns includes /root/.borgmatic and database hooks are active #582

witten referenced this issue 2022-10-04 23:16:23 +00:00

Excluded patterns are not printed in output of create --dry-run --list #590

witten referenced this issue 2022-10-05 06:01:17 +00:00

Excluded patterns are not printed in output of create --dry-run --list #590

witten referenced this issue 2023-10-24 17:48:09 +00:00

Borgmatic reports success but logs show stacktrace and no archive is created. #776

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

spot-check

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.24

1.5.23

1.5.22

1.5.21

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.14

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.7.dev0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.22

1.4.21

1.4.20

1.4.19

1.4.18

1.4.17

1.4.16

1.4.15

1.4.14

1.4.13

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.26

1.3.25

1.3.24

1.3.23

1.3.22

1.3.21

1.3.20

1.3.19

1.3.18

1.3.17

1.3.16

1.3.15

1.3.14

1.3.13

1.3.12

1.3.11

1.3.10

1.3.9

1.3.8

1.3.7

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.18

1.2.17

1.2.16

1.2.15

1.2.14

1.2.13

1.2.12

1.2.11

1.2.10

1.2.9

1.2.8

1.2.7

1.2.6

1.2.5

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.15

1.1.14

1.1.13

1.1.12

1.1.11

1.1.10

1.1.9

1.1.7

1.1.6

1.1.5

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.3

1.0.2

1.0.1

1.0.0

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

0.0.2

Labels

Clear labels   

bug

  

data loss

  

design finalized

  

good first issue

  

new feature area

  

question / support

  

security

  

waiting for response

No Label

bug

data loss

design finalized

good first issue

new feature area

question / support

security

waiting for response

Milestone

Clear milestone

No items

No Milestone

Assignees

Clear assignees

No Assignees

2 Participants

Notifications

Subscribe

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#574

No description provided.