Schedule backups with different triggers (systemd) and parameter? #496

New Issue

borg.haefemeier · 2022-02-08T20:28:17Z

borg.haefemeier commented

2022-02-08 20:28:17 +00:00

What I'm trying to do and why

I setup a borgmatic configuration with per-application configuration and use systemd timer units to trigger the backup. I would like to manage different time slots for specific backup configuration.

First scenario

Create user home backup archive every hour but database backup only every four hours.

Second scenario

Running borgmatic prune only once a day in opposite to borgmatic create every hour.

Thrid scenario

Combination of the scenarios above.

Is there any best practice available to handle these kind of scenarios using systemd.

Environment

borgmatic version: 1.5.21

borgmatic installation method: Fedora dnf package

Borg version: 1.1.17

Python version: 3.10.2

Database version (if applicable): n/a

operating system and version: Linux 5.16.5-200.fc35.x86_64

#### What I'm trying to do and why I setup a borgmatic configuration with per-application configuration and use `systemd` timer units to trigger the backup. I would like to manage different time slots for specific backup configuration. ##### First scenario Create user home backup archive every hour but database backup only every four hours. ##### Second scenario Running `borgmatic prune` only once a day in opposite to `borgmatic create` every hour. ##### Thrid scenario Combination of the scenarios above. Is there any best practice available to handle these kind of scenarios using `systemd`. #### Environment **borgmatic version:** 1.5.21 **borgmatic installation method:** Fedora dnf package **Borg version:** 1.1.17 **Python version:** 3.10.2 **Database version (if applicable):** n/a **operating system and version:** Linux 5.16.5-200.fc35.x86_64

witten commented

2022-02-08 23:24:04 +00:00

Great question! borgmatic has an optional --config flag that allows you to specify a particular per-application configuration file to use instead of picking up all application config files. Something like:

borgmatic --config /etc/borgmatic.d/myapp.yaml

(And if you only ever invoke them separately like that, they wouldn't have to reside in /etc/borgmatic.d.)

So with that approach, you could create separate pairs of systemd services and timers, one per application you want backed up. Each service could invoke borgmatic with a separate config file via --config.

As for your second/third scenarios, a similar setup could work: One systemd service file and timer that invokes borgmatic prune at your desired schedule. And then add create (and optionally check) to the borgmatic command-line for each individual application above so that prune isn't run for them.

If you want to run prune per-application as well (for instance if you're using separate archive prefixes per-application), then you could make separate prune services/timers per application as well, although that might get a bit onerous.

Please let me know if this helps for your use cases!

Great question! `borgmatic` has an optional `--config` flag that allows you to specify a particular per-application configuration file to use instead of picking up all application config files. Something like: `borgmatic --config /etc/borgmatic.d/myapp.yaml` (And if you only ever invoke them separately like that, they wouldn't have to reside in `/etc/borgmatic.d`.) So with that approach, you could create separate pairs of systemd services and timers, one per application you want backed up. Each service could invoke borgmatic with a separate config file via `--config`. As for your second/third scenarios, a similar setup could work: One systemd service file and timer that invokes `borgmatic prune` at your desired schedule. And then [add `create` (and optionally `check`) to the borgmatic command-line](https://torsion.org/borgmatic/docs/how-to/deal-with-very-large-backups/) for each individual application above so that `prune` isn't run for them. If you want to run `prune` per-application as well (for instance if you're using separate archive prefixes per-application), then you *could* make separate `prune` services/timers per application as well, although that might get a bit onerous. Please let me know if this helps for your use cases!

witten added the

question / support

label 2022-02-08 23:24:11 +00:00

borg.haefemeier commented

2022-02-12 22:12:23 +00:00

Thanks for your proposal and ideas. I have started to experiment with systemd template file. I took the borgmatic.service and borgmatic.timer files, rename it into borgmatic@.service and borgmatic@.timer to enable different timer for specific commands. For example borgmatic@create.timer to create a new backup archive using the defined configuration.

This works well and reduce the among of systemd unit files. Taken your thoughts I will try to extend my first steps and will keep you informed.

But it has one big pitfall. The dependency between the scheduled service files and backup repository. In some circumstances (e.g. overlapping timer schedules) and using the same repository it can provoke a locking issue.

Maybe you have a good idea to handle it. I am not a systemd or borg expert, maybe it is possible to define circular dependencies between each unit definitions to prevent parallel started services.

Thanks for your proposal and ideas. I have started to experiment with systemd template file. I took the `borgmatic.service` and `borgmatic.timer` files, rename it into `borgmatic@.service` and `borgmatic@.timer` to enable different timer for specific commands. For example `borgmatic@create.timer` to create a new backup archive using the defined configuration. This works well and reduce the among of systemd unit files. Taken your thoughts I will try to extend my first steps and will keep you informed. But it has one big pitfall. The dependency between the scheduled service files and backup repository. In some circumstances (e.g. overlapping timer schedules) and using the same repository it can provoke a locking issue. Maybe you have a good idea to handle it. I am not a systemd or borg expert, maybe it is possible to define circular dependencies between each unit definitions to prevent parallel started services.

witten commented

2022-03-05 23:00:41 +00:00

I'm not a systemd expert by any means, but my understanding is that you can chain systemd services via dependencies, specifically by the Wants options. This might also mean that you'd only need a single timer per chain of dependent services. Additionally, there are Before and After options that may allow you to sequence service runs to avoid lock contention on a common repository.

Hope this helps. I'll close the ticket for now, but feel free to post additional thoughts here.

I'm not a systemd expert by any means, but my understanding is that you *can* chain systemd services via dependencies, specifically by the [`Wants` options](https://fedoramagazine.org/systemd-unit-dependencies-and-order/). This *might* also mean that you'd only need a single timer per chain of dependent services. Additionally, there are `Before` and `After` options that may allow you to sequence service runs to avoid lock contention on a common repository. Hope this helps. I'll close the ticket for now, but feel free to post additional thoughts here.

witten closed this issue

2022-03-05 23:00:41 +00:00

Sign in to join this conversation.