borgmatic-collective/borgmatic
borgmatic-collective
/
borgmatic
15
57
Fork 40
Code Issues 54 Pull Requests 1 Actions Packages Releases 121 Activity

Feature: Store all configs used to create an archive, inside the created archive. #697

New Issue
Closed
opened 2023-05-14 09:53:08 +00:00 by diivi · 9 comments
diivi commented 2023-05-14 09:53:08 +00:00
Collaborator
Copy Link

What I'm trying to do and why

For Borgmatic to restore data from just a repository and nothing else, the user needs to have the config somewhere. Having it inside the archive will make things easier for the user as borgmatic can later use that config and restore the archive, with no external help.

Other notes / implementation ideas

  • Store configuration within the archive at the same path where it exists on the system being backed up.

For instance, let's say you have configuration at /etc/borgmatic/config.yaml and /etc/borgmatic.d/myapp.yaml. And you pass both of those to borgmatic either implicitly or explicitly. Then upon a "create" action, borgmatic can put those two configuration files into the archive at etc/borgmatic/config.yaml and etc/borgmatic.d/myapp.yaml just as it would with any other file (Borg doesn't store leading slashes in file paths). It already works kind of like this for the ~/.borgmatic/ directory.

  • witten

TODO: This should be as simple as adding the config being used to source_directories somehow?

  • Create a metadata/manifest file that tells borgmatic where these configs are, to list them and so that borgmatic knows what to extract.

The one thing missing though with an approach like this is some sort of metadata/manifest for borgmatic to programmatically retrieve the full list of configuration files within an archive, so that "bootstrap" knows what to extract. This could just be a JSON file shoved into ~/.borgmatic somewhere, for instance.

  • witten

TODO: Check how things get stored inside ~/.borgmatic (afair, every database has a folder in this folder) and then add one file with a pre-determined name (configs-list.json?) that contains info about all config files (just their path for now).

[
	"etc/borgmatic/config.yaml","etc/borgmatic.d/myapp.yaml"
]

This should complete "phase 1" of Borgmatic being able to restore an archive from nothing.

#### What I'm trying to do and why For Borgmatic to restore data from just a repository and nothing else, the user needs to have the config somewhere. Having it inside the archive will make things easier for the user as borgmatic can later use that config and restore the archive, with no external help. #### Other notes / implementation ideas * Store configuration within the archive at the same path where it exists on the system being backed up. > For instance, let's say you have configuration at /etc/borgmatic/config.yaml and /etc/borgmatic.d/myapp.yaml. And you pass both of those to borgmatic either implicitly or explicitly. Then upon a "create" action, borgmatic can put those two configuration files into the archive at etc/borgmatic/config.yaml and etc/borgmatic.d/myapp.yaml just as it would with any other file (Borg doesn't store leading slashes in file paths). It already works kind of like this for the ~/.borgmatic/ directory. - witten TODO: This should be as simple as adding the config being used to source_directories somehow? * Create a metadata/manifest file that tells borgmatic where these configs are, to list them and so that borgmatic knows what to extract. > The one thing missing though with an approach like this is some sort of metadata/manifest for borgmatic to programmatically retrieve the full list of configuration files within an archive, so that "bootstrap" knows what to extract. This could just be a JSON file shoved into ~/.borgmatic somewhere, for instance. - witten TODO: Check how things get stored inside ~/.borgmatic (afair, every database has a folder in this folder) and then add one file with a pre-determined name (configs-list.json?) that contains info about all config files (just their path for now). ```json [ "etc/borgmatic/config.yaml","etc/borgmatic.d/myapp.yaml" ] ``` This should complete "phase 1" of Borgmatic being able to restore an archive from nothing.
witten commented 2023-05-14 21:03:39 +00:00
Owner
Copy Link

TODO: This should be as simple as adding the config being used to source_directories somehow?

I think so. This is where that happens now: https://projects.torsion.org/borgmatic-collective/borgmatic/src/branch/main/borgmatic/borg/create.py#L354

TODO: Check how things get stored inside ~/.borgmatic (afair, every database has a folder in this folder) and then add one file with a pre-determined name (configs-list.json?) that contains info about all config files (just their path for now).

Makes sense! Maybe ~/.borgmatic/bootstrap/configs-list.json or similar?

I like the idea of doing this in discrete, shippable phases so as to keep the work tractable.

> TODO: This should be as simple as adding the config being used to source_directories somehow? I think so. This is where that happens now: https://projects.torsion.org/borgmatic-collective/borgmatic/src/branch/main/borgmatic/borg/create.py#L354 > TODO: Check how things get stored inside ~/.borgmatic (afair, every database has a folder in this folder) and then add one file with a pre-determined name (configs-list.json?) that contains info about all config files (just their path for now). Makes sense! Maybe `~/.borgmatic/bootstrap/configs-list.json` or similar? I like the idea of doing this in discrete, shippable phases so as to keep the work tractable.
witten added this to the bootstrap from nothing milestone 2023-05-14 21:15:07 +00:00
diivi commented 2023-05-15 15:31:04 +00:00
Author
Collaborator
Copy Link

I was trying to back up all configs with:

    location_config['source_directories'] = location_config.get('source_directories', []) + global_arguments.config_paths

Out of the default config_paths (used without the -c arg) that Borgmatic uses, the ones that start with /root raised an error, here's the log for clarity:

Creating archive at "/home/divyansh/Desktop/bkt2::divi-2023-05-15T20:54:29.396008"
/etc/borgmatic.d: [Errno 2] No such file or directory: 'borgmatic.d'
- /etc/borgmatic/config.yaml
- /home/divyansh/Desktop/os/os-ass4/q2.c
- /home/divyansh/Desktop/os/os-ass4/q1.c
- /home/divyansh/Desktop/os/os-ass4/q3.c
- /home/divyansh/Desktop/os/os-ass4/q4.c
- /home/divyansh/Desktop/os/os-ass4/a.out
- /home/divyansh/Desktop/os/2110110680.tar.gz
- /home/divyansh/Desktop/os/a.out
- /home/divyansh/Desktop/os/file.txt
- /home/divyansh/Desktop/os/hello.txt
- /home/divyansh/Desktop/os/wow.txt
- /home/divyansh/Desktop/os/meh.txt
- /home/divyansh/Desktop/os/yo.txt
- /home/divyansh/Desktop/os/.vscode/settings.json
- /home/divyansh/Desktop/os/rr.c
- /home/divyansh/Desktop/os/sjf.c
- /home/divyansh/Desktop/os/os-ass5/a.out
- /home/divyansh/Desktop/os/os-ass5/q1.c
- /home/divyansh/Desktop/os/os-ass5/q2.c
- /home/divyansh/Desktop/os/xyz/x.txt
- /home/divyansh/Desktop/os/ass2/addtrans.c
- /home/divyansh/Desktop/os/ass2/a.out
- /home/divyansh/Desktop/os/helll/hello.sql
/home/divyansh/Desktop/xyz: [Errno 2] No such file or directory: 'xyz'
/root/.config/borgmatic.d: [Errno 2] No such file or directory: 'borgmatic.d'
Local Exception
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/borg/archive.py", line 264, in OsOpen
    fd = os_open(path=path, parent_fd=parent_fd, name=name, flags=flags, noatime=noatime)
  File "/usr/lib/python3/dist-packages/borg/helpers/fs.py", line 290, in os_open
    fd = os.open(fname, _flags_noatime, dir_fd=parent_fd)
FileNotFoundError: [Errno 2] No such file or directory: '/root/.config/borgmatic'

Borg raised this error:

borg.archive.BackupOSError: open_root: [Errno 2] No such file or directory: '/root/.config/borgmatic'

which was not handled by Borgmatic (I even set source_directories_must_exist to false).

Why wasn't this handled by https://github.com/borgmatic-collective/borgmatic/pull/56? Also, how should I go about backing up the configs in the /root folder?

I was trying to back up all configs with: ```python location_config['source_directories'] = location_config.get('source_directories', []) + global_arguments.config_paths ``` Out of the default config_paths (used without the -c arg) that Borgmatic uses, the ones that start with /root raised an error, here's the log for clarity: ```bash Creating archive at "/home/divyansh/Desktop/bkt2::divi-2023-05-15T20:54:29.396008" /etc/borgmatic.d: [Errno 2] No such file or directory: 'borgmatic.d' - /etc/borgmatic/config.yaml - /home/divyansh/Desktop/os/os-ass4/q2.c - /home/divyansh/Desktop/os/os-ass4/q1.c - /home/divyansh/Desktop/os/os-ass4/q3.c - /home/divyansh/Desktop/os/os-ass4/q4.c - /home/divyansh/Desktop/os/os-ass4/a.out - /home/divyansh/Desktop/os/2110110680.tar.gz - /home/divyansh/Desktop/os/a.out - /home/divyansh/Desktop/os/file.txt - /home/divyansh/Desktop/os/hello.txt - /home/divyansh/Desktop/os/wow.txt - /home/divyansh/Desktop/os/meh.txt - /home/divyansh/Desktop/os/yo.txt - /home/divyansh/Desktop/os/.vscode/settings.json - /home/divyansh/Desktop/os/rr.c - /home/divyansh/Desktop/os/sjf.c - /home/divyansh/Desktop/os/os-ass5/a.out - /home/divyansh/Desktop/os/os-ass5/q1.c - /home/divyansh/Desktop/os/os-ass5/q2.c - /home/divyansh/Desktop/os/xyz/x.txt - /home/divyansh/Desktop/os/ass2/addtrans.c - /home/divyansh/Desktop/os/ass2/a.out - /home/divyansh/Desktop/os/helll/hello.sql /home/divyansh/Desktop/xyz: [Errno 2] No such file or directory: 'xyz' /root/.config/borgmatic.d: [Errno 2] No such file or directory: 'borgmatic.d' Local Exception Traceback (most recent call last): File "/usr/lib/python3/dist-packages/borg/archive.py", line 264, in OsOpen fd = os_open(path=path, parent_fd=parent_fd, name=name, flags=flags, noatime=noatime) File "/usr/lib/python3/dist-packages/borg/helpers/fs.py", line 290, in os_open fd = os.open(fname, _flags_noatime, dir_fd=parent_fd) FileNotFoundError: [Errno 2] No such file or directory: '/root/.config/borgmatic' ``` Borg raised this error: ```bash borg.archive.BackupOSError: open_root: [Errno 2] No such file or directory: '/root/.config/borgmatic' ``` which was not handled by Borgmatic (I even set source_directories_must_exist to false). Why wasn't this handled by https://github.com/borgmatic-collective/borgmatic/pull/56? Also, how should I go about backing up the configs in the /root folder?
witten commented 2023-05-15 16:23:57 +00:00
Owner
Copy Link

Where exactly did you put this line?

  location_config['source_directories'] = location_config.get('source_directories', []) + global_arguments.config_paths

If, for instance, it came before the call to check_all_source_directories_exist() in create_archive(), I would expect that feature to raise when it can't find those paths. If it came after, then I would not expect borgmatic to catch that, and it would be consistent with your observed behavior. In any case, I don't think this approach will quite work....

Also, how should I go about backing up the configs in the /root folder?

Here's how I think you should go about backing up configs in any folder: I think you might need to probe for existence before adding them to location_config['source_directories']. And not to raise when they don't exist like the check_all_source_directories_exist() feature, but instead just to omit them if they don't exist. The reason I say that is because unlike standard source directories, it's okay if the default config paths don't exist. They just need to be probed for one at a time and config loaded if it's present there.

The good news is this is already done for you, and I think you might be able to switch out the value you add to location_config['source_directories'] (with some annoying plumbing work). Instead of global_arguments.config_paths, which is the raw, unprocessed set of config paths that may or may not exist, you need the output of borgmatic/config/collect.py:collect_config_filenames() ...

  • Which is currently called in borgmatic/commands/borgmatic.py:main().
  • That calls collect_configuration_run_summary_logs() with configs (a map from processed config filename to loaded config).
  • That calls run_configuration() on a single config file.
  • That calls run_actions().
  • That calls borgmatic/actions/create.py:run_create().
  • That calls borgmatic/borg/create.py:create_archive(), which is where you presumably need the full list of configs so you can add them to source directories.

So maybe the thing to do is to pass the config filename list into run_configuration() and all the way down to create_archive()? If you can think of a better approach, that's fine too.

Where exactly did you put this line? ```python location_config['source_directories'] = location_config.get('source_directories', []) + global_arguments.config_paths ``` If, for instance, it came *before* the call to `check_all_source_directories_exist()` in `create_archive()`, I would expect that feature to raise when it can't find those paths. If it came *after*, then I would not expect borgmatic to catch that, and it would be consistent with your observed behavior. In any case, I don't think this approach will quite work.... > Also, how should I go about backing up the configs in the /root folder? Here's how I think you should go about backing up configs in *any* folder: I think you might need to probe for existence before adding them to `location_config['source_directories']`. And not to raise when they don't exist like the `check_all_source_directories_exist()` feature, but instead just to omit them if they don't exist. The reason I say that is because unlike standard source directories, it's okay if the default config paths don't exist. They just need to be probed for one at a time and config loaded if it's present there. The good news is this is already done for you, and I think you might be able to switch out the value you add to `location_config['source_directories']` (with some annoying plumbing work). Instead of `global_arguments.config_paths`, which is the raw, unprocessed set of config paths that may or may not exist, you need the output of `borgmatic/config/collect.py:collect_config_filenames()` ... * Which is currently called in `borgmatic/commands/borgmatic.py:main()`. * That calls `collect_configuration_run_summary_logs()` with `configs` (a map from processed config filename to loaded config). * That calls `run_configuration()` on a single config file. * That calls `run_actions()`. * That calls `borgmatic/actions/create.py:run_create()`. * That calls `borgmatic/borg/create.py:create_archive()`, which is where you presumably need the full list of configs so you can add them to source directories. So maybe the thing to do is to pass the config filename list into `run_configuration()` and all the way down to `create_archive()`? If you can think of a better approach, that's fine too.
witten commented 2023-05-25 02:23:48 +00:00
Owner
Copy Link

@diivi It occurs to me that borgmatic bootstrap could be renamed to borgmatic config bootstrap, especially if all it's doing is extracting config. Related ticket: #529.

@diivi It occurs to me that `borgmatic bootstrap` _could_ be renamed to `borgmatic config bootstrap`, especially if all it's doing is extracting config. Related ticket: #529.
witten commented 2023-06-10 21:47:43 +00:00
Owner
Copy Link

For the record: The bulk of this was implemented in: https://github.com/borgmatic-collective/borgmatic/pull/71

For the record: The bulk of this was implemented in: https://github.com/borgmatic-collective/borgmatic/pull/71
witten commented 2023-06-10 23:06:59 +00:00
Owner
Copy Link

I tried this feature and it works great! However one thing that occurs to me is that it's not storing absolute paths for config files that are specified as relative paths. Which means that when you go to bootstrap, if you don't happen to be in the same directory where the backup was taken, then the config file won't get extracted into the right place. So do you think it makes sense to abspath() all config files before storing them?

I tried this feature and it works great! However one thing that occurs to me is that it's not storing absolute paths for config files that are specified as relative paths. Which means that when you go to bootstrap, if you don't happen to be in the same directory where the backup was taken, then the config file won't get extracted into the right place. So do you think it makes sense to `abspath()` all config files before storing them?
witten commented 2023-06-15 17:58:25 +00:00
Owner
Copy Link

FYI: bbc7f0596c ... I haven't done this for Fish shell though, because honestly I don't understand how its completions work.

FYI: https://projects.torsion.org/borgmatic-collective/borgmatic/commit/bbc7f0596c13b79e6930ffa1a2b4381e8d2df766 ... I haven't done this for Fish shell though, because honestly I don't understand how its completions work.
witten commented 2023-06-25 05:30:21 +00:00
Owner
Copy Link

Just released in borgmatic 1.7.15. I'm leaving the ticket open for now pending documentation.

Just released in borgmatic 1.7.15. I'm leaving the ticket open for now pending documentation.
witten added the
new feature area
 label 2023-06-28 18:37:04 +00:00
witten commented 2023-07-31 19:17:02 +00:00
Owner
Copy Link

Docs are complete, courtesy of @diivi! https://torsion.org/borgmatic/docs/how-to/extract-a-backup/#extract-the-configuration-files-used-to-create-an-archive

Docs are complete, courtesy of @diivi! https://torsion.org/borgmatic/docs/how-to/extract-a-backup/#extract-the-configuration-files-used-to-create-an-archive
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
main
spot-check
1.8.10
1.8.9
1.8.8
1.8.7
1.8.6
1.8.5
1.8.4
1.8.3
1.8.2
1.8.1
1.8.0
1.7.15
1.7.14
1.7.13
1.7.12
1.7.11
1.7.10
1.7.9
1.7.8
1.7.7
1.7.6
1.7.5
1.7.4
1.7.3
1.7.2
1.7.1
1.7.0
1.6.6
1.6.5
1.6.4
1.6.3
1.6.2
1.6.1
1.6.0
1.5.24
1.5.23
1.5.22
1.5.21
1.5.20
1.5.19
1.5.18
1.5.17
1.5.16
1.5.15
1.5.14
1.5.13
1.5.12
1.5.11
1.5.10
1.5.9
1.5.8
1.5.7
1.5.7.dev0
1.5.6
1.5.5
1.5.4
1.5.3
1.5.2
1.5.1
1.5.0
1.4.22
1.4.21
1.4.20
1.4.19
1.4.18
1.4.17
1.4.16
1.4.15
1.4.14
1.4.13
1.4.12
1.4.11
1.4.10
1.4.9
1.4.8
1.4.7
1.4.6
1.4.5
1.4.4
1.4.3
1.4.2
1.4.1
1.4.0
1.3.26
1.3.25
1.3.24
1.3.23
1.3.22
1.3.21
1.3.20
1.3.19
1.3.18
1.3.17
1.3.16
1.3.15
1.3.14
1.3.13
1.3.12
1.3.11
1.3.10
1.3.9
1.3.8
1.3.7
1.3.6
1.3.5
1.3.4
1.3.3
1.3.2
1.3.1
1.3.0
1.2.18
1.2.17
1.2.16
1.2.15
1.2.14
1.2.13
1.2.12
1.2.11
1.2.10
1.2.9
1.2.8
1.2.7
1.2.6
1.2.5
1.2.4
1.2.3
1.2.2
1.2.1
1.2.0
1.1.15
1.1.14
1.1.13
1.1.12
1.1.11
1.1.10
1.1.9
1.1.7
1.1.6
1.1.5
1.1.4
1.1.3
1.1.2
1.1.1
1.1.0
1.0.3
1.0.2
1.0.1
1.0.0
0.1.8
0.1.7
0.1.6
0.1.5
0.1.4
0.1.3
0.1.2
0.1.1
0.1.0
0.0.7
0.0.6
0.0.5
0.0.4
0.0.3
0.0.2
Labels
Clear labels   
bug

   
data loss

   
design finalized

   
good first issue

   
new feature area

   
question / support

   
security

   
waiting for response
No Label
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
Milestone
Clear milestone
No items
No Milestone
bootstrap from nothing
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#697
No description provided.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?