borgmatic-collective/borgmatic

Watch

Fork 57

help with config files #1080

New issue

Closed

opened 2025-04-19 16:51:53 +00:00 by robdavies · 8 comments

robdavies commented

2025-04-19 16:51:53 +00:00

I have been converting my configs files to use borgmatic v2 settings, have also split the config files into common.yaml and <repo_name>.yaml as most are the same settings

I used to have the remote repo files as one (borgbase and rsync.net), until I had to have a different remote_path for rsync.net

I now have the following config files, each for a separate repository. Each use common.yaml to share settings :

borgbase.yaml
rsyncnet.yaml
localusb.yaml

Its working well, I now run two config files at the same time (and plan to run three when I use usb), except that before_everything (which is in the common.yaml config file) runs multiple times at the start (seem like one per config file)

Is there anything I can do as I only need to run before_everything once for my usage, even if I have two or three config files in the command line. Its exactly the same as its shared in common.yaml. It calculates checksums of the files to backup, which takes 15-20m each time it runs. Is there another hook I can use instead of it?

Thanks,
Rob

Hi I have been converting my configs files to use borgmatic v2 settings, have also split the config files into common.yaml and <repo_name>.yaml as most are the same settings I used to have the remote repo files as one (borgbase and rsync.net), until I had to have a different remote_path for rsync.net I now have the following config files, each for a separate repository. Each use common.yaml to share settings : - borgbase.yaml - rsyncnet.yaml - localusb.yaml Its working well, I now run two config files at the same time (and plan to run three when I use usb), except that before_everything (which is in the common.yaml config file) runs multiple times at the start (seem like one per config file) Is there anything I can do as I only need to run before_everything once for my usage, even if I have two or three config files in the command line. Its exactly the same as its shared in common.yaml. It calculates checksums of the files to backup, which takes 15-20m each time it runs. Is there another hook I can use instead of it? Thanks, Rob

witten commented

2025-04-19 22:49:19 +00:00

Owner

before_everything commands should only run once regardless of the number of configuration files, so if that's not the case, there might be a bug. My guess is what's happening is that because you've got before_everything in your common configuration file, it's in effect getting included and therefore duplicated multiple times. You can validate this theory by running something like the following:

borgmatic config validate --show --config borgbase.yaml --config rsyncnet.yaml

This will spit out the config file contents after all the includes and merging and normalization have occurred, and it'll show you whether your before_everything gets transformed into commands: and duplicated there.

If it is getting duplicated, then borgmatic might have to somehow get a little smarter about merging duplicate values.

`before_everything` commands *should* only run once regardless of the number of configuration files, so if that's not the case, there might be a bug. My guess is what's happening is that because you've got `before_everything` in your common configuration file, it's in effect getting included and therefore duplicated multiple times. You can validate this theory by running something like the following: ```shell borgmatic config validate --show --config borgbase.yaml --config rsyncnet.yaml ``` This will spit out the config file contents *after* all the includes and merging and normalization have occurred, and it'll show you whether your `before_everything` gets transformed into `commands:` and duplicated there. If it is getting duplicated, then borgmatic might have to somehow get a little smarter about merging duplicate values.

witten added the

bug

label

2025-04-19 22:49:58 +00:00

robdavies commented

2025-04-20 18:02:59 +00:00

Author

My mistake, when I converted the config files to borgmatic v2 settings, I used borgmatic config generate --source input.yaml -d output.yaml , before_everything got converted to

commands:
    - before: everything
      run:
          - $HOME/bin/backup_pre.bsh $HOME/borg_backup
      when:
          - create

Its this that is in the common.yaml in place of before_everything. This looks like its the same hook, just a new way to configure it.

When I ran borgmatic config validate --show --config borgbase.yaml --config rsyncnet.yaml
I found that this new before: everything command got duplicated. There are two commands: in total (I'm assuming one per config file) and its in both of them, with other commands

I tried by removing before: everything from common.yaml and put it into borgbase.yaml and rsyncnet.yaml I got the same result.

My mistake, when I converted the config files to borgmatic v2 settings, I used `borgmatic config generate --source input.yaml -d output.yaml` , before_everything got converted to ``` commands: - before: everything run: - $HOME/bin/backup_pre.bsh $HOME/borg_backup when: - create ``` Its this that is in the common.yaml in place of before_everything. This looks like its the same hook, just a new way to configure it. When I ran `borgmatic config validate --show --config borgbase.yaml --config rsyncnet.yaml` I found that this new `before: everything` command got duplicated. There are two `commands:` in total (I'm assuming one per config file) and its in both of them, with other commands I tried by removing `before: everything` from common.yaml and put it into borgbase.yaml and rsyncnet.yaml I got the same result.

witten commented

2025-04-20 18:21:54 +00:00

Owner

Okay, thanks for verifying. I'll have to think about what if anything I can do about this on the borgmatic side. Maybe transparently deduplicating everything command hooks as they're collected from multiple configuration files.

Okay, thanks for verifying. I'll have to think about what if anything I can do about this on the borgmatic side. Maybe transparently deduplicating `everything` command hooks as they're collected from multiple configuration files.

robdavies commented

2025-04-21 12:50:21 +00:00

Author

That would be great.

Just for a test (to cover bases, so to speak), I did try merging borgbase.yaml and common.yaml, also rsyncnet.yaml and common.yaml into two independent config file. I had the same result.

That would be great. Just for a test (to cover bases, so to speak), I did try merging borgbase.yaml and common.yaml, also rsyncnet.yaml and common.yaml into two independent config file. I had the same result.

👍 1

witten referenced this issue from a commit

2025-04-22 02:19:50 +00:00

If the exact same "everything" command hook is present in multiple configuration files, borgmatic only runs it once (#1080).

witten commented

2025-04-22 02:21:07 +00:00

Owner

Okay, deduplication of everything command hooks across configuration files is now implemented in main and will be part of the next release. Thanks for filing this!

Okay, deduplication of `everything` command hooks across configuration files is now implemented in main and will be part of the next release. Thanks for filing this!

witten

2025-04-22 02:21:07 +00:00