borgmatic-collective/borgmatic
borgmatic-collective/borgmatic
10
66
Fork 27
Code Issues 42 Pull Requests 5 Actions Packages Releases 140 Activity

location.patterns breaks when /root/.borgmatic exists #574

New Issue
Closed
opened 2022-08-25 09:42:04 +00:00 by Kritzefitz · 5 comments
Kritzefitz commented 2022-08-25 09:42:04 +00:00
Copy Link

What I'm trying to do and why

I'm trying to create a backup while specifying the locations using location.patterns and /root/.borgmatic exists. Reasons to do this include using location.patterns and hooks.*_databases at the same time, because the latter automatically creates /root/.borgmatic.

Steps to reproduce (if a bug)

# borg init -e none /repo
# mkdir /the-files
# echo 'the content' > /the-files/the-file
# cat /etc/borgmatic/config.yaml
location:
    source_directories: []
    repositories:
        - /repo
    patterns:
        - R /the-files

# # This is still working as expected
# borgmatic create
# borgmatic list --archive latest
/repo: Listing archive
/repo: Listing archive 3e1f0673503c-2022-08-25T09:31:38.722964
drwxr-xr-x root   root          0 Thu, 2022-08-25 09:30:04 the-files
-rw-r--r-- root   root         12 Thu, 2022-08-25 09:30:04 the-files/the-file

# # This doesn't back up the files in /the-files
# mkdir /root/.borgmatic
# borgmatic create
# borgmatic list --archive latest
/repo: Listing archive
/repo: Listing archive 3e1f0673503c-2022-08-25T09:32:37.611310
drwxr-xr-x root   root          0 Thu, 2022-08-25 09:32:30 root/.borgmatic

Actual behavior (if a bug)

As soon as /root/.borgmatic exists, it overrides all locations specified in location.patterns and only /root/.borgmatic is included in the backup.

Expected behavior (if a bug)

The roots specified by location.patterns should be included in the backup regardless of the existence of /root/.borgmatic.

Other notes / implementation ideas

This seems to happen, because borgmatic passes /root/.borgmatic as a backup directory to borg create when it exists, which seems to conflict with --patterns-from

# borgmatic create -v2
Ensuring legacy configuration is upgraded
/etc/borgmatic/config.yaml: No commands to run for pre-everything hook
borg --version --debug --show-rc
/etc/borgmatic/config.yaml: No commands to run for pre-actions hook
/etc/borgmatic/config.yaml: No commands to run for pre-backup hook
/repo: Creating archive
borg create --patterns-from /tmp/tmpdc182zsw --debug --show-rc /repo::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /root/.borgmatic
using builtin fallback logging configuration
35 self tests completed in 0.08 seconds
Verified integrity of /repo/index.13
TAM-verified manifest
security: read previous location '/repo'
security: read manifest timestamp '2022-08-25T09:32:37.689635'
security: determined newest manifest timestamp as 2022-08-25T09:32:37.689635
security: repository checks ok, allowing access
Creating archive at "/repo::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}"
Verified integrity of /root/.cache/borg/6b144ea6c59923e26643bd085d87f5227e279f6597e341a0d31755b8df0e7a6c/chunks
Reading files cache ...
Verified integrity of /root/.cache/borg/6b144ea6c59923e26643bd085d87f5227e279f6597e341a0d31755b8df0e7a6c/files
security: read previous location '/repo'
security: read manifest timestamp '2022-08-25T09:32:37.689635'
security: determined newest manifest timestamp as 2022-08-25T09:32:37.689635
security: repository checks ok, allowing access
Processing files ...
Cleaned up 0 uncommitted segment files (== everything after segment 13).
Verified integrity of /repo/hints.13
check_free_space: few segments, not requiring a full free segment
check_free_space: calculated working space for compact as 1913 bytes
check_free_space: required bytes 171081, free bytes 92029419520
security: saving state for 6b144ea6c59923e26643bd085d87f5227e279f6597e341a0d31755b8df0e7a6c to /root/.config/borg/security/6b144ea6c59923e26643bd085d87f5227e279f6597e341a0d31755b8df0e7a6c
security: current location   /repo
security: key type           2
security: manifest timestamp 2022-08-25T09:35:53.980844
terminating with success status, rc 0
/etc/borgmatic/config.yaml: No commands to run for post-backup hook
/etc/borgmatic/config.yaml: No commands to run for post-actions hook
/etc/borgmatic/config.yaml: No commands to run for post-everything hook

summary:
/etc/borgmatic/config.yaml: Successfully ran configuration file

Environment

borgmatic version: 1.7.0, 1.6.3

borgmatic installation method: via pip in docker container python:3-buster (1.7.0), as well as via apt on Debian bookworm (1.6.3)

Borg version: 1.1.9 (in docker container with borgmatic 1.7.0), 1.2.1 (on Debian bookworm with borgmatic 1.6.3)

Python version: 3.10.6 (in both cases)

Database version (if applicable): n/a, the bug was discovered while trying to use database hooks, but also appears when no actual database hook is configured.

operating system and version: docker container python:3-buster, Debian Bookworm

#### What I'm trying to do and why I'm trying to create a backup while specifying the locations using `location.patterns` and `/root/.borgmatic` exists. Reasons to do this include using `location.patterns` and `hooks.*_databases` at the same time, because the latter automatically creates `/root/.borgmatic`. #### Steps to reproduce (if a bug) ``` # borg init -e none /repo # mkdir /the-files # echo 'the content' > /the-files/the-file # cat /etc/borgmatic/config.yaml location: source_directories: [] repositories: - /repo patterns: - R /the-files # # This is still working as expected # borgmatic create # borgmatic list --archive latest /repo: Listing archive /repo: Listing archive 3e1f0673503c-2022-08-25T09:31:38.722964 drwxr-xr-x root root 0 Thu, 2022-08-25 09:30:04 the-files -rw-r--r-- root root 12 Thu, 2022-08-25 09:30:04 the-files/the-file # # This doesn't back up the files in /the-files # mkdir /root/.borgmatic # borgmatic create # borgmatic list --archive latest /repo: Listing archive /repo: Listing archive 3e1f0673503c-2022-08-25T09:32:37.611310 drwxr-xr-x root root 0 Thu, 2022-08-25 09:32:30 root/.borgmatic ``` #### Actual behavior (if a bug) As soon as `/root/.borgmatic` exists, it overrides all locations specified in `location.patterns` and only `/root/.borgmatic` is included in the backup. #### Expected behavior (if a bug) The roots specified by `location.patterns` should be included in the backup regardless of the existence of `/root/.borgmatic`. #### Other notes / implementation ideas This seems to happen, because borgmatic passes `/root/.borgmatic` as a backup directory to `borg create` when it exists, which seems to conflict with `--patterns-from` ``` # borgmatic create -v2 Ensuring legacy configuration is upgraded /etc/borgmatic/config.yaml: No commands to run for pre-everything hook borg --version --debug --show-rc /etc/borgmatic/config.yaml: No commands to run for pre-actions hook /etc/borgmatic/config.yaml: No commands to run for pre-backup hook /repo: Creating archive borg create --patterns-from /tmp/tmpdc182zsw --debug --show-rc /repo::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /root/.borgmatic using builtin fallback logging configuration 35 self tests completed in 0.08 seconds Verified integrity of /repo/index.13 TAM-verified manifest security: read previous location '/repo' security: read manifest timestamp '2022-08-25T09:32:37.689635' security: determined newest manifest timestamp as 2022-08-25T09:32:37.689635 security: repository checks ok, allowing access Creating archive at "/repo::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}" Verified integrity of /root/.cache/borg/6b144ea6c59923e26643bd085d87f5227e279f6597e341a0d31755b8df0e7a6c/chunks Reading files cache ... Verified integrity of /root/.cache/borg/6b144ea6c59923e26643bd085d87f5227e279f6597e341a0d31755b8df0e7a6c/files security: read previous location '/repo' security: read manifest timestamp '2022-08-25T09:32:37.689635' security: determined newest manifest timestamp as 2022-08-25T09:32:37.689635 security: repository checks ok, allowing access Processing files ... Cleaned up 0 uncommitted segment files (== everything after segment 13). Verified integrity of /repo/hints.13 check_free_space: few segments, not requiring a full free segment check_free_space: calculated working space for compact as 1913 bytes check_free_space: required bytes 171081, free bytes 92029419520 security: saving state for 6b144ea6c59923e26643bd085d87f5227e279f6597e341a0d31755b8df0e7a6c to /root/.config/borg/security/6b144ea6c59923e26643bd085d87f5227e279f6597e341a0d31755b8df0e7a6c security: current location /repo security: key type 2 security: manifest timestamp 2022-08-25T09:35:53.980844 terminating with success status, rc 0 /etc/borgmatic/config.yaml: No commands to run for post-backup hook /etc/borgmatic/config.yaml: No commands to run for post-actions hook /etc/borgmatic/config.yaml: No commands to run for post-everything hook summary: /etc/borgmatic/config.yaml: Successfully ran configuration file ``` #### Environment **borgmatic version:** 1.7.0, 1.6.3 **borgmatic installation method:** via pip in docker container `python:3-buster` (1.7.0), as well as via apt on Debian bookworm (1.6.3) **Borg version:** 1.1.9 (in docker container with borgmatic 1.7.0), 1.2.1 (on Debian bookworm with borgmatic 1.6.3) **Python version:** 3.10.6 (in both cases) **Database version (if applicable):** n/a, the bug was discovered while trying to use database hooks, but also appears when no actual database hook is configured. **operating system and version:** docker container `python:3-buster`, Debian Bookworm
witten commented 2022-08-25 16:15:00 +00:00
Owner
Copy Link

Yikes, that's pretty bad! Thanks for the detailed bug report. I will attempt to repro this locally and work on a fix.

Yikes, that's pretty bad! Thanks for the detailed bug report. I will attempt to repro this locally and work on a fix.
witten added the
bug
data loss
 labels 2022-08-25 16:15:12 +00:00
witten commented 2022-08-25 16:44:21 +00:00
Owner
Copy Link

I've repro'd this with Borg only (1.2.1), without borgmatic:

borg create --pattern="R /root/tmp" --debug --show-rc test.borg::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /root/.borgmatic

This results in only /root/.borgmatic getting backed up. So this is either a bug in Borg or intended behavior. In either case, borgmatic probably needs to work around it.

I've repro'd this with Borg only (1.2.1), without borgmatic: ```bash borg create --pattern="R /root/tmp" --debug --show-rc test.borg::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /root/.borgmatic ``` This results in only `/root/.borgmatic` getting backed up. So this is either a bug in Borg or intended behavior. In either case, borgmatic probably needs to work around it.
witten commented 2022-08-25 18:07:41 +00:00
Owner
Copy Link

The Borg dev seems to think the Borg patterns behavior described above could be a bug, so I've filed it here: https://github.com/borgbackup/borg/issues/6994. Still gotta figure out what that means for borgmatic.

The Borg dev seems to think the Borg patterns behavior described above could be a bug, so I've filed it here: https://github.com/borgbackup/borg/issues/6994. Still gotta figure out what that means for borgmatic.
witten commented 2022-08-25 19:02:22 +00:00
Owner
Copy Link

Okay, I just released a work-around for this Borg issue in borgmatic 1.7.1. Basically, when patterns is used, any source_directories (implicit or otherwise) get converted into patterns instead of being passed to Borg on the command-line. With that change, the overriding behavior you were seeing no longer occurs.

You may also be interested to know that with #542 included in this release, source_directories is now completely optional; you don't need to set it to an empty list any longer.

Thanks again for reporting this!

Okay, I just released a work-around for this Borg issue in borgmatic 1.7.1. Basically, when `patterns` is used, any `source_directories` (implicit or otherwise) get converted into patterns instead of being passed to Borg on the command-line. With that change, the overriding behavior you were seeing no longer occurs. You may also be interested to know that with #542 included in this release, `source_directories` is now completely optional; you don't need to set it to an empty list any longer. Thanks again for reporting this!
Kritzefitz commented 2022-08-25 19:49:01 +00:00
Author
Copy Link

Thank you, for the thourough investigation and very timely fix!

Thank you, for the thourough investigation and very timely fix!
👍 1
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
blocked
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
No Label
bug
data loss
Milestone
No items
No Milestone
Assignees
Clear assignees
diivi import_bot witten
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#574
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?