discussion: using different repos, dependent on network conditions #626

New Issue

jzacsh · 2022-12-27T18:09:45Z

jzacsh commented

2022-12-27 18:09:45 +00:00

note: this is more of a discussion-post. I'm not expecting this to be fixed or even truly borgmatic's problem to solve, but hoping this might be useful to others (eg both #205 or my previous #419 are nicely searchable).

What I'm trying to do and why

I want a local repo (on a 2nd SSD in my laptop) to be used as the only repository in "some cases", and otherwise I want the remote repositories to be used and not bother with the local repo. The "some cases" here is: if I'm on my home wifi network (a particular SSID) then I know I have a great internet connection, or if that's not achievable then simply making the switch when there's outright no network connectivity.

Other notes / implementation ideas

I think the answer is that you'd need👍👍 to...

split your config.yaml into two: one for "over-network" backups, one for "local-backups"
have two distinct borgmatic timers and service units going so that the ExecStart CLI line can be distinct on both (for differing -c config args)

I'm just not sure what kind of wonky situation would happen if "local-backups" was running and then you switched networks (or say your internet was out, backup started, but now internet comes back on, and then the "over-network" schedule starts)... would both try to run and make your machine unusable? TODO here is for me to figure out if systemd has a way to make units mutually exclusive.

note: this is more of a discussion-post. I'm not expecting this to be fixed or even truly borgmatic's problem to solve, but hoping this might be useful to others (eg both #205 or my previous #419 are nicely searchable). #### What I'm trying to do and why I want a local repo (on a 2nd SSD in my laptop) to be used as the only repository in "some cases", and otherwise I want the remote repositories to be used and not bother with the local repo. The "some cases" here is: if I'm on my _home_ wifi network (a particular SSID) then I know I have a great internet connection, or if that's not achievable then simply making the switch when there's outright _no_ network connectivity. #### Other notes / implementation ideas I _think_ the answer is that you'd need👍👍 to... 1. [split your `config.yaml` into two](https://torsion.org/borgmatic/docs/how-to/make-per-application-backups/): one for "over-network" backups, one for "local-backups" 2. have two distinct borgmatic timers and service units going so that [the `ExecStart` CLI line](https://gitlab.com/jzacsh/dotfiles/-/blob/d6367f9b892cbd60d8ccba156fdf5f08bb67c025/.host/src/host.oankali/borgmatic.service#L81) can be distinct on both (for differing `-c` config args) I'm just not sure what kind of wonky situation would happen if "local-backups" was running and then you switched networks (or say your internet _was_ out, backup started, but now internet comes back on, and then the "over-network" schedule starts)... would both try to run and make your machine unusable? TODO here is for me to figure out if systemd has a way to make units mutually exclusive.

witten commented

2022-12-27 18:29:37 +00:00

Thanks for filing this.. Discussion tickets are an excellent use of this issue tracker!

Have you seen the documentation on intermittent backups? You could write a before_backup shell script that runs any logic you want (including checking for a particular wifi SSID) and returns an exit code of 75 to indicate that configuration file's backups should be skipped.

With this approach, you'd still need two configuration files as you described. But you would only need a single borgmatic service unit / timer, and the backups would run serially so you wouldn't need to worry about making them mutually exclusive. (Although you wouldn't get any of the benefits of running them in parallel, either).

Thanks for filing this.. Discussion tickets are an excellent use of this issue tracker! Have you seen [the documentation on intermittent backups](https://torsion.org/borgmatic/docs/how-to/backup-to-a-removable-drive-or-an-intermittent-server/)? You could write a `before_backup` shell script that runs any logic you want (including checking for a particular wifi SSID) and returns an exit code of 75 to indicate that configuration file's backups should be skipped. With this approach, you'd still need two configuration files as you described. But you would only need a single borgmatic service unit / timer, and the backups would run serially so you wouldn't need to worry about making them mutually exclusive. (Although you wouldn't get any of the benefits of running them in parallel, either).

jzacsh commented

2022-12-27 23:02:14 +00:00

perfect, you're right, thanks for the reply! For some reason I was imagining the before_backup logic inside of systemd rather than a config value of borgmatic (which makes no sense, because I've used before_backup before for the battery-capacity check in #419).

Anyway, yes this is perfect and I've broken out my configs and tested this works perfectly! Here's a diff if anyone's sufficiently motivated - it's of course specific to my machine/dotfile-setup but I think the two "known fast SSID"-scripts (thin nmcli wrappers) and the example of the YAML split-up, all in one place might be helpful.

There's one thing I have left to figure out: prevent actual serial running (since this is meant as a round-robin, pick one destination, not every-single-one). Example: say one backup starts correctly and then half-way through the state changes (eg: offline backsup starts and then we get onto a good SSID half-way through the backup) then I wouldn't want the online backup to start immediately afterwards. Instead I'd rather wait until the next systemd timer schedule tries to run. Otherwise these backups take forever (slowing down my computer).

I'll close this for now, because this is sufficient, but feel free to reopen/comment for thoughts here! My only thought is to add another hook that somehow figures out if there's a recent successful backup somewhere :| ... feels ugly.

tl;dr for each systemd invokation, have just one borgmatic-config get to a started-state (ie: get beyond the pre/before-hooks successfully), rather than all configs get tried.

perfect, you're right, thanks for the reply! For some reason I was imagining the `before_backup` logic inside of systemd rather than a config value of borgmatic (which makes no sense, because I've used `before_backup` before for the battery-capacity check in #419). Anyway, yes this is perfect and I've broken out my configs and tested this works perfectly! Here's a [diff if anyone's sufficiently motivated](https://gitlab.com/jzacsh/dotfiles/-/compare/2ce308471e36f64b56e9416babe15598dce3f7b4...c5858b3dde0c85f2312d2816a0709d063b8af219?from_project_id=29199579&straight=false) - it's of course specific to my machine/dotfile-setup but I think the two "known fast SSID"-scripts (thin `nmcli` wrappers) and the example of the YAML split-up, all in one place might be helpful. ---- There's one thing I have left to figure out: prevent actual serial running (since this is meant as a round-robin, pick _one_ destination, not every-single-one). Example: say one backup starts correctly and then half-way through the state changes (eg: offline backsup starts and then we get onto a good SSID half-way through the backup) then I wouldn't want the online backup to start immediately afterwards. Instead I'd rather wait until the next systemd timer schedule tries to run. Otherwise these backups take forever (slowing down my computer). I'll close this for now, because this is sufficient, but feel free to reopen/comment for thoughts here! My only thought is to add _another_ hook that somehow figures out if there's a recent successful backup somewhere :| ... feels ugly. **tl;dr** for each systemd invokation, have just **one** borgmatic-config get to a started-state (ie: get beyond the pre/before-hooks successfully), rather than _all_ configs get tried.

jzacsh closed this issue

2022-12-27 23:02:34 +00:00

witten commented

2022-12-27 23:18:34 +00:00

Glad to hear that approach works for you!

As for preventing serial running.. It might be kind of hacky, but you could write another before_backup script for the online backup configuration file that checks the last backup time of the offline repository and, if too recent, exits with code 75.

I don't normally recommend running borgmatic from borgmatic, but maybe something like:

borgmatic -c test.yaml info --archive latest --json --repository /your/offline/repo | jq -r '.[0].archives[0].start'

Something similar could also be done with Borg directly.

Glad to hear that approach works for you! As for preventing serial running.. It might be kind of hacky, but you could write another `before_backup` script for the online backup configuration file that checks the last backup time of the offline repository and, if too recent, exits with code 75. I don't normally recommend running borgmatic from borgmatic, but maybe something like: ```bash borgmatic -c test.yaml info --archive latest --json --repository /your/offline/repo | jq -r '.[0].archives[0].start' ``` Something similar could also be done with Borg directly.

Sign in to join this conversation.