borgmatic-collective/borgmatic
borgmatic-collective
/
borgmatic
15
57
Fork 40
Code Issues 54 Pull Requests 1 Actions Packages Releases 121 Activity

Backups include unnecessary config files #725

New Issue
Closed
opened 2023-07-14 15:03:22 +00:00 by hop · 7 comments
hop commented 2023-07-14 15:03:22 +00:00
Copy Link

How to reproduce

Put several config files in a conf.d/ directory and start with borgmatic -c conf.d.

.
├── conf.d
│   ├── a.yaml
│   └── b.yaml
├── fs
│   └── data
└── repo
    ├── README
    ├── config
    ├── data
    ...

a.yaml:

location:
    source_directories:
        - $HOME/borg_test/fs

    repositories:
        - path: $HOME/borg_test/repo
          label: repo_a

storage:
    archive_name_format: 'a-{now}'

retention:
    keep_daily: 7

Same for /etc/borgmatic.d/ and borgmatic without arguments.

Actual behavior

borgmatic includes all config files in all backups.

$ borgmatic -c conf.d --verbosity 2
...
borg create $HOME/borg_test/repo::a-{now} $HOME/.borgmatic $HOME/borg_test/conf.d/a.yaml $HOME/borg_test/conf.d/b.yaml $HOME/borg_test/fs --debug --show-rc

$ borg list repo::a-2023-07-14T16:34:48
drwxr-xr-x user    staff    0 Thu, 2023-07-13 23:32:04 $HOME/.borgmatic
...
-rw------- user    staff  228 Fri, 2023-07-14 16:12:13 $HOME/borg_test/conf.d/a.yaml
-rw------- user    staff  228 Fri, 2023-07-14 16:12:11 $HOME/borg_test/conf.d/b.yaml
drwxr-xr-x user    staff    0 Thu, 2023-07-13 23:40:37 $HOME/borg_test/fs
-rw-r--r-- user    staff  139 Thu, 2023-07-13 23:40:37 $HOME/borg_test/fs/data

Expected behavior

I wouldn't have expected b.yaml to be included in a-backups and vice versa. I also think this could lead to leaked passphrases or other sensitive data.

Environment

borgmatic version: 1.7.15

borgmatic installation method: pip

Borg version: 1.1.15, 1.2.4

Python version: 3.11

Database version (if applicable): n/a

operating system and version: Debian, macOS

#### How to reproduce Put several config files in a `conf.d/` directory and start with `borgmatic -c conf.d`. ``` . ├── conf.d │   ├── a.yaml │   └── b.yaml ├── fs │   └── data └── repo ├── README ├── config ├── data ... ``` **a.yaml**: ```yaml location: source_directories: - $HOME/borg_test/fs repositories: - path: $HOME/borg_test/repo label: repo_a storage: archive_name_format: 'a-{now}' retention: keep_daily: 7 ``` Same for `/etc/borgmatic.d/` and `borgmatic` without arguments. #### Actual behavior borgmatic includes _all_ config files in all backups. ``` $ borgmatic -c conf.d --verbosity 2 ... borg create $HOME/borg_test/repo::a-{now} $HOME/.borgmatic $HOME/borg_test/conf.d/a.yaml $HOME/borg_test/conf.d/b.yaml $HOME/borg_test/fs --debug --show-rc ``` ``` $ borg list repo::a-2023-07-14T16:34:48 drwxr-xr-x user staff 0 Thu, 2023-07-13 23:32:04 $HOME/.borgmatic ... -rw------- user staff 228 Fri, 2023-07-14 16:12:13 $HOME/borg_test/conf.d/a.yaml -rw------- user staff 228 Fri, 2023-07-14 16:12:11 $HOME/borg_test/conf.d/b.yaml drwxr-xr-x user staff 0 Thu, 2023-07-13 23:40:37 $HOME/borg_test/fs -rw-r--r-- user staff 139 Thu, 2023-07-13 23:40:37 $HOME/borg_test/fs/data ``` #### Expected behavior I wouldn't have expected `b.yaml` to be included in a-backups and vice versa. I also think this could lead to leaked passphrases or other sensitive data. #### Environment **borgmatic version:** 1.7.15 **borgmatic installation method:** pip **Borg version:** 1.1.15, 1.2.4 **Python version:** 3.11 **Database version (if applicable):** n/a **operating system and version:** Debian, macOS
witten commented 2023-07-14 15:25:34 +00:00
Owner
Copy Link

Thanks for taking the time to file this. Config files are included in each repository in order to support the (soon-to-be documented) config bootstrap action. The idea is that you should be able to "bootstrap" a fresh system from your backups, and the first step in doing that is to extract all borgmatic configuration files so then borgmatic can work. If each repository only contained a partial set of the configuration files, that wouldn't be possible without multiple config bootstrap invocations. Hope that makes sense.

Thanks for taking the time to file this. Config files are included in each repository in order to support the (soon-to-be documented) [`config bootstrap` action](https://projects.torsion.org/borgmatic-collective/borgmatic/issues/697). The idea is that you should be able to "bootstrap" a fresh system from your backups, and the first step in doing that is to extract all borgmatic configuration files so then borgmatic can work. If each repository only contained a partial set of the configuration files, that wouldn't be possible without multiple `config bootstrap` invocations. Hope that makes sense.
hop commented 2023-07-14 16:11:06 +00:00
Author
Copy Link

It absolutely makes sense, but I don't need b.yaml in my backup of a. The above example should maybe use two different repos to make this a bit clearer.

The use case is to be able to give different clients or users different keys and be able to give them access to only their own repository.

I could iterate over the config files myself and invoke borgmatic for each of them separately, I guess, but some mechanism to do that for a single invocation would be nice.

… if /that/ makes sense.

It absolutely makes sense, but I don't need `b.yaml` in my backup of `a`. The above example should maybe use two different repos to make this a bit clearer. The use case is to be able to give different clients or users different keys and be able to give them access to only their own repository. I could iterate over the config files myself and invoke borgmatic for each of them separately, I guess, but some mechanism to do that for a single invocation would be nice. … if /that/ makes sense.
witten commented 2023-07-14 17:13:55 +00:00
Owner
Copy Link

That makes sense, although that change would make the bootstrap feature potentially less useful for other use cases. (Like restoring all configs with a single borgmatic command.) So here are a few different ideas on how to address this:

  • Add a configuration option for disabling the automatic storing of configuration files altogether. This would prevent the bootstrap action from working, but it'd avoid leaking any information across repositories.
  • Or.. Add a configuration option for restricting automatic storing of configuration files to only the repositories used in those configuration files. This would make the bootstrap action less useful, but it'd still work. And a user would have to intentionally opt in to this.
  • Or.. Optionally encrypt configuration files as stored in each repository with a key specified on the borgmatic config bootstrap command-line. I'm not a huge fan of this idea though, as: 1. It's encrypting something already store in an encrypted repository, and 2. borgmatic would have to get into the business of encryption, which right now it 100% farms out to Borg. The upshot though is you could still bootstrap with a single command (assuming you have the decryption key).

I'm open to other ideas as well.

That makes sense, although that change would make the bootstrap feature potentially less useful for other use cases. (Like restoring all configs with a single borgmatic command.) So here are a few different ideas on how to address this: * Add a configuration option for disabling the automatic storing of configuration files altogether. This would prevent the `bootstrap` action from working, but it'd avoid leaking any information across repositories. * Or.. Add a configuration option for restricting automatic storing of configuration files to only the repositories used in those configuration files. This would make the `bootstrap` action less useful, but it'd still work. And a user would have to intentionally opt in to this. * Or.. Optionally encrypt configuration files as stored in each repository with a key specified on the `borgmatic config bootstrap` command-line. I'm not a huge fan of this idea though, as: 1. It's encrypting something already store in an encrypted repository, and 2. borgmatic would have to get into the business of encryption, which right now it 100% farms out to Borg. The upshot though is you could still bootstrap with a single command (assuming you have the decryption key). I'm open to other ideas as well.
hop commented 2023-07-14 20:01:21 +00:00
Author
Copy Link

I would be happy about the first option over not having any choice at all. In my use case, automatic bootstrapping is not a key issue after system failure. Essentially, the system can be built from scratch and only data needs to be recovered afterwards.

For the second, I'm not familiar enough about the system to tell whether the effort is worth it, but it sounds ideal.

It should be opt-in in any case, I agree.

The third option seems too complicated, brittle and inelegant.

I would be happy about the first option over not having any choice at all. In my use case, automatic bootstrapping is not a key issue after system failure. Essentially, the system can be built from scratch and only data needs to be recovered afterwards. For the second, I'm not familiar enough about the system to tell whether the effort is worth it, but it sounds ideal. It should be opt-in in any case, I agree. The third option seems too complicated, brittle and inelegant.
👍 1
diivi commented 2023-07-29 16:19:26 +00:00
Collaborator
Copy Link

I discussed with witten that I'll implement the first option when I'm free from other GSoC tasks.

I discussed with witten that I'll implement the first option when I'm free from other GSoC tasks.
👍 1
witten added the
design finalized
 label 2023-07-31 20:11:11 +00:00
witten referenced this issue from a commit 2023-07-31 21:04:16 +00:00
Code formatting (#725).
witten commented 2023-07-31 21:05:20 +00:00
Owner
Copy Link

Implemented in main by @diivi via a new store_config_files option that can be set to false. This will be part of the next release! Thanks again for the ticket.

Implemented in main by @diivi via a new `store_config_files` option that can be set to `false`. This will be part of the next release! Thanks again for the ticket.
❤️ 1
witten commented 2023-08-04 04:59:55 +00:00
Owner
Copy Link

Just released in borgmatic 1.8.1!

Just released in borgmatic 1.8.1!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
main
spot-check
1.8.10
1.8.9
1.8.8
1.8.7
1.8.6
1.8.5
1.8.4
1.8.3
1.8.2
1.8.1
1.8.0
1.7.15
1.7.14
1.7.13
1.7.12
1.7.11
1.7.10
1.7.9
1.7.8
1.7.7
1.7.6
1.7.5
1.7.4
1.7.3
1.7.2
1.7.1
1.7.0
1.6.6
1.6.5
1.6.4
1.6.3
1.6.2
1.6.1
1.6.0
1.5.24
1.5.23
1.5.22
1.5.21
1.5.20
1.5.19
1.5.18
1.5.17
1.5.16
1.5.15
1.5.14
1.5.13
1.5.12
1.5.11
1.5.10
1.5.9
1.5.8
1.5.7
1.5.7.dev0
1.5.6
1.5.5
1.5.4
1.5.3
1.5.2
1.5.1
1.5.0
1.4.22
1.4.21
1.4.20
1.4.19
1.4.18
1.4.17
1.4.16
1.4.15
1.4.14
1.4.13
1.4.12
1.4.11
1.4.10
1.4.9
1.4.8
1.4.7
1.4.6
1.4.5
1.4.4
1.4.3
1.4.2
1.4.1
1.4.0
1.3.26
1.3.25
1.3.24
1.3.23
1.3.22
1.3.21
1.3.20
1.3.19
1.3.18
1.3.17
1.3.16
1.3.15
1.3.14
1.3.13
1.3.12
1.3.11
1.3.10
1.3.9
1.3.8
1.3.7
1.3.6
1.3.5
1.3.4
1.3.3
1.3.2
1.3.1
1.3.0
1.2.18
1.2.17
1.2.16
1.2.15
1.2.14
1.2.13
1.2.12
1.2.11
1.2.10
1.2.9
1.2.8
1.2.7
1.2.6
1.2.5
1.2.4
1.2.3
1.2.2
1.2.1
1.2.0
1.1.15
1.1.14
1.1.13
1.1.12
1.1.11
1.1.10
1.1.9
1.1.7
1.1.6
1.1.5
1.1.4
1.1.3
1.1.2
1.1.1
1.1.0
1.0.3
1.0.2
1.0.1
1.0.0
0.1.8
0.1.7
0.1.6
0.1.5
0.1.4
0.1.3
0.1.2
0.1.1
0.1.0
0.0.7
0.0.6
0.0.5
0.0.4
0.0.3
0.0.2
Labels
Clear labels   
bug

   
data loss

   
design finalized

   
good first issue

   
new feature area

   
question / support

   
security

   
waiting for response
No Label
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
3 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#725
No description provided.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?