mount borg repo after backup, in config file #760

New Issue

robdavies · 2023-09-25T13:44:43Z

robdavies commented

2023-09-25 13:44:43 +00:00

What I'm trying to do and why

After a backup, I mount the repo and perform some verification (count files, check some checksums). I do this in after_backup: , the command I use is

borgmatic mount --repository {repository} --archive latest --mount-point /mnt/borg

which works when the config file is config.yaml . I have several config files, when it isn't config.yaml, I have to add -c <config_file> to the mount parameters e.g.

borgmatic -c $HOME/.config/borgmatic/server.yaml mount --repository {repository} --archive latest --mount-point /mnt/borg

Is there a way I can mount the latest repo without having the use the -c <config_file> option? Is there a better way I can do this?

The main reason I'm trying to do this, is when I create a new config file, I can just copy an old one and make one or two changes. I tend to forget this setting and it would be great to have a better way for me to do this

Thanks for your help,
Rob

Steps to reproduce

No response

Actual behavior

No response

Expected behavior

No response

Other notes / implementation ideas

No response

borgmatic version

1.8.2

borgmatic installation method

pip

Borg version

1.2.6

Python version

python 3.8.10

Database version (if applicable)

No response

Operating system and version

linux ubuntu 20.04.6

### What I'm trying to do and why After a backup, I mount the repo and perform some verification (count files, check some checksums). I do this in after_backup: , the command I use is borgmatic mount --repository {repository} --archive latest --mount-point /mnt/borg which works when the config file is config.yaml . I have several config files, when it isn't config.yaml, I have to add -c <config_file> to the mount parameters e.g. borgmatic -c $HOME/.config/borgmatic/server.yaml mount --repository {repository} --archive latest --mount-point /mnt/borg Is there a way I can mount the latest repo without having the use the -c <config_file> option? Is there a better way I can do this? The main reason I'm trying to do this, is when I create a new config file, I can just copy an old one and make one or two changes. I tend to forget this setting and it would be great to have a better way for me to do this Thanks for your help, Rob ### Steps to reproduce _No response_ ### Actual behavior _No response_ ### Expected behavior _No response_ ### Other notes / implementation ideas _No response_ ### borgmatic version 1.8.2 ### borgmatic installation method pip ### Borg version 1.2.6 ### Python version python 3.8.10 ### Database version (if applicable) _No response_ ### Operating system and version linux ubuntu 20.04.6

witten commented

2023-09-25 19:02:48 +00:00

I generally recommend against calling borgmatic from borgmatic, as you can get into an infinite loop or run into contention issues since borgmatic assumes only a single instance is running at once. Although I can see why you might want to do it anyway here.

When you're mounting the repository in the after_backup hook, are you then manually performing counts and doing checksums, or are you using an automated script to do it? I wonder if a proposed feature like this new consistency check would meet your need in a more built-in manner.

In terms of your actual question, there's actually an existing borgmatic variable that should save you from having to manually specify the configuration file name. Try something like this in your after_backup hook:

borgmatic -c {configuration_filename} mount --repository {repository} --archive latest --mount-point /mnt/borg

I generally recommend against calling borgmatic from borgmatic, as you can get into an infinite loop or run into [contention issues](https://projects.torsion.org/borgmatic-collective/borgmatic/issues/755) since borgmatic assumes only a single instance is running at once. Although I can see why you might want to do it anyway here. When you're mounting the repository in the `after_backup` hook, are you then manually performing counts and doing checksums, or are you using an automated script to do it? I wonder if a proposed feature like [this new consistency check](https://projects.torsion.org/borgmatic-collective/borgmatic/issues/656) would meet your need in a more built-in manner. In terms of your actual question, there's actually an [existing borgmatic variable](https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/#variable-interpolation) that should save you from having to manually specify the configuration file name. Try something like this in your `after_backup` hook: ``` borgmatic -c {configuration_filename} mount --repository {repository} --archive latest --mount-point /mnt/borg ```

witten added the

question / support

label 2023-09-25 19:02:56 +00:00

robdavies commented

2023-09-29 09:40:35 +00:00

I tried the "borgmatic -c {configuration_filename}" setting and it worked, I'll use it for now until any new checks are added.

The new consistency check sounds great, it sound similar to why I run the checks I do.

Why :
borgmatic checks (repository, archive) didn't seem to validate the file contents against the original. I didn't understand what either of those checks really checked for, so I implemented my own to validate some/all of the data.

how :
pre_backup :
export list of packages installed (apt,flatpak,pip,etc) > borg_backup directory
calculate checksums of all the files to backup > borg_backup directory

post backup :
mount repo
compare number of files in repo with number of entries of checksum file
validate files in borg_backup directory using checksums stored in extended file attributes

I figured that if I could :

mount the repository
the right amount of files are backed up (catch misconfiguration)
At least some data could be read and validated

I thought that if these all pass then that its a good chance that the rest of the data was valid.

I can run this validation on any repo archive I create, it will validate against the files at the time of backup rather than as they are now.

Validating all files for a few GB wouldn't take too long, though when its 100s GB or multi TB then it can take a long time (longer than the backup itself). I like the idea of checking a random sample, setting a percentage.

Having this replaced with a borgmatic built-in function would be fantastic.

I tried the "borgmatic -c {configuration_filename}" setting and it worked, I'll use it for now until any new checks are added. The new consistency check sounds great, it sound similar to why I run the checks I do. Why : borgmatic checks (repository, archive) didn't seem to validate the file contents against the original. I didn't understand what either of those checks really checked for, so I implemented my own to validate some/all of the data. how : pre_backup : export list of packages installed (apt,flatpak,pip,etc) > borg_backup directory calculate checksums of all the files to backup > borg_backup directory post backup : mount repo compare number of files in repo with number of entries of checksum file validate files in borg_backup directory using checksums stored in extended file attributes *** I figured that if I could : 1) mount the repository 2) the right amount of files are backed up (catch misconfiguration) 3) At least some data could be read and validated I thought that if these all pass then that its a good chance that the rest of the data was valid. I can run this validation on any repo archive I create, it will validate against the files at the time of backup rather than as they are now. Validating all files for a few GB wouldn't take too long, though when its 100s GB or multi TB then it can take a long time (longer than the backup itself). I like the idea of checking a random sample, setting a percentage. Having this replaced with a borgmatic built-in function would be fantastic.

witten commented

2023-09-29 15:46:42 +00:00

Awesome, I'm glad to hear it's working for you now! And thanks for all the context on your existing setup.

borgmatic checks (repository, archive) didn't seem to validate the file contents against the original. I didn't understand what either of those checks really checked for, so I implemented my own to validate some/all of the data.

You are correct that these checks don't validate against the original files. They're just validating that the repository and archives are themselves intact on Borg's side and that they haven't become corrupted. But yeah, they don't do anything to catch misconfiguration.

I'll close this ticket for now, but feel free to follow #656 to be notified when the new built-in spot check is available.

Awesome, I'm glad to hear it's working for you now! And thanks for all the context on your existing setup. > borgmatic checks (repository, archive) didn't seem to validate the file contents against the original. I didn't understand what either of those checks really checked for, so I implemented my own to validate some/all of the data. You are correct that these checks don't validate against the original files. They're just validating that the repository and archives are themselves intact on Borg's side and that they haven't become corrupted. But yeah, they don't do anything to catch misconfiguration. I'll close this ticket for now, but feel free to follow #656 to be notified when the new built-in spot check is available.

witten closed this issue

2023-09-29 15:46:42 +00:00

robdavies referenced this issue

2024-04-17 02:01:46 +00:00

"Spot check" of source directories consistency check #656

Sign in to join this conversation.