unified borgmatic command to trigger actions (prune or not) from what is in the config file #701

New Issue

eoli3n · 2023-05-14T20:30:41Z

eoli3n commented

2023-05-14 20:30:41 +00:00

What I'm trying to do and why

Run a repository in append-only mode, with a cron on the repository server to prune.

I want to be able to run "borgmatic" command to performs all tasks without the need to explictily limit to "prune" subcommand.

If I run borgmatic on the repository server, I will create a backup of the repository server, on the client repository.

Repository server side

Current config file to be able to borgmatic prune

Create a borgmatic file on the repository in /etc/borgmatic.d/prune-hostname.yml

location:
  source_directories:
    - / # this is required for borgmatic to run, but useless for pruning
  repositories:
    - /backups/hostname

storage:
  encryption_passphrase: "*********************************"
  compression: zstd
  archive_name_format: 'hostname-{now:%Y-%m-%dT%H:%M:%S}'
  relocated_repo_access_is_ok: true

retention:
  prefix: 'hostname-'
  keep_daily: 7
  keep_weekly: 4
  keep_monthly: 6

Then run

borgmatic prune

if I run borgmatic alone, this will have an non wanted behaviour (backuping the repository server in the client repository)

Expected config file and behaviour

location:
  repositories:
    - /backups/hostname

storage:
  encryption_passphrase: "*********************************"
  compression: zstd
  archive_name_format: 'hostname-{now:%Y-%m-%dT%H:%M:%S}'
  relocated_repo_access_is_ok: true

retention:
  prefix: 'hostname-'
  keep_daily: 7
  keep_weekly: 4
  keep_monthly: 6

Then run

borgmatic

client side

Current config file to be able to create a backup

location:
  exclude_caches: true
  source_directories:
    - /
  repositories:
    - ssh://repository/backups/hostname

storage:
  encryption_passphrase: "***********************"
  compression: zstd
  archive_name_format: '{hostname}-{now:%Y-%m-%dT%H:%M:%S}'
  relocated_repo_access_is_ok: true

consistency:
  check_last: 3

Then, to be able to create a backup without pruning, I need to run

borgmatic create

Expected config file and behaviour

If no retention key is specified in the config file, the following command should work.

borgmatic

The benefit is that this will also borgmatic check without the need to explicit it.

Other notes / implementation ideas

Environment

borgmatic version: 1.7.8

Borg version: borg 1.2.4

#### What I'm trying to do and why Run a repository in append-only mode, with a cron on the repository server to prune. I want to be able to run "borgmatic" command to performs all tasks without the need to explictily limit to "prune" subcommand. If I run borgmatic on the repository server, I will create a backup of the repository server, on the client repository. ### Repository server side #### Current config file to be able to borgmatic prune Create a borgmatic file on the repository in /etc/borgmatic.d/prune-hostname.yml ``` location: source_directories: - / # this is required for borgmatic to run, but useless for pruning repositories: - /backups/hostname storage: encryption_passphrase: "*********************************" compression: zstd archive_name_format: 'hostname-{now:%Y-%m-%dT%H:%M:%S}' relocated_repo_access_is_ok: true retention: prefix: 'hostname-' keep_daily: 7 keep_weekly: 4 keep_monthly: 6 ``` Then run ``` borgmatic prune ``` if I run ``borgmatic`` alone, this will have an non wanted behaviour (backuping the repository server in the client repository) #### Expected config file and behaviour ``` location: repositories: - /backups/hostname storage: encryption_passphrase: "*********************************" compression: zstd archive_name_format: 'hostname-{now:%Y-%m-%dT%H:%M:%S}' relocated_repo_access_is_ok: true retention: prefix: 'hostname-' keep_daily: 7 keep_weekly: 4 keep_monthly: 6 ``` Then run ``` borgmatic ``` ### client side #### Current config file to be able to create a backup ``` location: exclude_caches: true source_directories: - / repositories: - ssh://repository/backups/hostname storage: encryption_passphrase: "***********************" compression: zstd archive_name_format: '{hostname}-{now:%Y-%m-%dT%H:%M:%S}' relocated_repo_access_is_ok: true consistency: check_last: 3 ``` Then, to be able to create a backup without pruning, I need to run ``` borgmatic create ``` #### Expected config file and behaviour If no ``retention`` key is specified in the config file, the following command should work. ``` borgmatic ``` The benefit is that this will also borgmatic check without the need to explicit it. #### Other notes / implementation ideas #### Environment **borgmatic version:** 1.7.8 **Borg version:** borg 1.2.4

eoli3n commented

2023-05-14 20:36:24 +00:00

The other problem is that I cron my backup on the client, and that I backup on two different repositories.
Repository 1 runs in --append-only mode, but not Repository 2.
I monitor return code of borgmatic command.

If I explicitly run borgmatic create && borgmatic check, I will backup and check on repository 1 and repository 2, but will not prune on repository 2.

If I run borgmatic, prune will fail on Repository 1.

I should be able to run borgmatic command to automate what I need on specific repositories.

With the version I run, I can't prune per repository to workaround this.
EDIT : updated to be able to use --repository with prune subcommand
https://github.com/void-linux/void-packages/pull/43872

The other problem is that I cron my backup on the client, and that I backup on two different repositories. Repository 1 runs in ``--append-only`` mode, but not Repository 2. I monitor return code of borgmatic command. If I explicitly run ``borgmatic create && borgmatic check``, I will backup and check on repository 1 and repository 2, but will not prune on repository 2. If I run ``borgmatic``, prune will fail on Repository 1. I should be able to run ``borgmatic`` command to automate what I need on specific repositories. With the version I run, I can't prune per repository to workaround this. EDIT : updated to be able to use ``--repository`` with ``prune`` subcommand https://github.com/void-linux/void-packages/pull/43872

witten commented

2023-05-14 21:33:25 +00:00

Thank you for taking the time to file this! I totally agree that having the ability to run "prune only" or "check only" would be nice without having to specify that on the command-line every time. I'm not sure though about having borgmatic try to figure out what actions to run based on the configured section of the configuration file. I say that because sometimes a configuration section isn't specified but has defaults anyway. For instance, even when consistency isn't specified, borgmatic assumes a default set of checks with a default frequency.

An alternate approach to allow what you're asking for—to be able to just run borgmatic without explicit actions—would be to put those explicit actions in the configuration file (optionally). For instance:

some_section:
     actions:
         - create
         - check

Then, if you didn't override those actions on the command-line, they would be used when running borgmatic and no other actions would run (for that configuration file).

Related ticket: #364.

Let me know your thoughts on whether this would address your use case!

Thank you for taking the time to file this! I totally agree that having the ability to run "prune only" or "check only" would be nice without having to specify that on the command-line every time. I'm not sure though about having borgmatic try to figure out what actions to run based on the configured section of the configuration file. I say that because sometimes a configuration section isn't specified but has defaults anyway. For instance, even when `consistency` isn't specified, borgmatic assumes [a default set of checks with a default frequency](https://torsion.org/borgmatic/docs/how-to/deal-with-very-large-backups/#consistency-check-configuration). An alternate approach to allow what you're asking for—to be able to just run `borgmatic` without explicit actions—would be to put those explicit actions in the configuration file (optionally). For instance: ```yaml some_section: actions: - create - check ``` Then, if you didn't override those actions on the command-line, they would be used when running borgmatic and no other actions would run (for that configuration file). Related ticket: #364. Let me know your thoughts on whether this would address your use case!

eoli3n commented

2023-05-15 18:06:05 +00:00

Yes, the solution about the ability to explicitly limit actions in config file seems perfect to me !

👍 2

witten added the

design finalized

label 2023-05-15 18:07:21 +00:00

witten referenced this issue

2023-10-24 16:07:44 +00:00

How to skip pruning when running from systemd timer #775

kalsan referenced this issue

2023-10-24 18:17:50 +00:00

How to skip pruning when running from systemd timer #775

witten commented

2023-10-31 05:41:56 +00:00

A couple of challenges I encountered with this one so far:

At the point in the code that configuration files are parsed and loaded, e.g. to grab the configured default actions, it's too late to influence the parsing of the command-line based on those actions. That's because the command-line parsing—including applying particular CLI flags to particular action parsers—occurs before the configuration files are loaded. And it goes in that order because you can specify configuration files to load on the command-line. So this is sort of a chicken-and-egg problem, and I'm not sure what, if anything, to do about it.
Second, and maybe more easily surmountable: If you specify multiple configuration files, you could in theory have different configured default actions in each one. To address this, borgmatic could warn or error if it happens.

A couple of challenges I encountered with this one so far: - At the point in the code that configuration files are parsed and loaded, e.g. to grab the configured default actions, it's too late to influence the parsing of the command-line based on those actions. That's because the command-line parsing—including applying particular CLI flags to particular action parsers—occurs before the configuration files are loaded. And it goes in that order because you can specify configuration files to load on the command-line. So this is sort of a chicken-and-egg problem, and I'm not sure what, if anything, to do about it. - Second, and maybe more easily surmountable: If you specify multiple configuration files, you could in theory have different configured default actions in each one. To address this, borgmatic could warn or error if it happens.

witten commented

2023-11-01 03:42:53 +00:00

Okay, here's the solution I came up with to sidestep those challenges: Instead of a configuration option with the default actions to use, I flipped the logic and added a "skip_actions" option with names of the actions that should be skipped for the current configuration file. Example:

skip_actions:
    - prune
    - compact

I feel like this pretty well supports the original append-only use case and also other use cases like a checkless configuration.

Okay, here's the solution I came up with to sidestep those challenges: Instead of a configuration option with the default actions to use, I flipped the logic and added a "skip_actions" option with names of the actions that should be _skipped_ for the current configuration file. Example: ```yaml skip_actions: - prune - compact ``` I feel like this pretty well supports the original append-only use case and also other use cases like a checkless configuration.

witten referenced this issue from a commit

2023-11-01 04:54:52 +00:00

Add a "skip_actions" option to skip running particular actions (#701).

witten referenced this issue

2023-11-01 04:59:12 +00:00

How to do append only? #462

witten commented

2023-11-01 05:00:51 +00:00

This is implemented in main and will be part of the next release!

🚀 1

witten closed this issue

2023-11-01 05:00:51 +00:00