borgmatic-collective/borgmatic

Watch

Fork 57

How to do append only? #462

New issue

Closed

opened 2021-11-07 14:02:05 +00:00 by adrian · 4 comments

adrian commented

2021-11-07 14:02:05 +00:00

What I'm trying to do and why

Use append only backups without prune.

Steps to reproduce (if a bug)

I use the following base configuration as a base:

location:
    source_directories:
        - /home
    repositories:
        - x@x.repo.borgbase.com:repo
    local_path: /usr/local/bin/borg
    exclude_patterns:
        - '/*/.borgkey'
        - '*.pyc'
        - '*.tmp'
    exclude_caches: true
    exclude_if_present:
        - .nobackup
    exclude_nodump: true
storage:
    encryption_passcommand: cat /root/.borgkey
    checkpoint_interval: 43200
consistency:
    checks:
        - repository

The configuration is considered valid:

$ validate-borgmatic-config 
All given configuration files are valid: /etc/borgmatic/config.yaml

Actual behavior (if a bug)

When I trigger the backup with borgmatic --verbosity 2 it kills with

Ensuring legacy configuration is upgraded
/etc/borgmatic/config.yaml: No commands to run for pre-everything hook
/etc/borgmatic/config.yaml: No commands to run for pre-prune hook
/etc/borgmatic/config.yaml: No commands to run for pre-backup hook
/etc/borgmatic/config.yaml: No commands to run for pre-check hook
x@x.repo.borgbase.com:repo: Pruning archives
/usr/local/bin/borg prune --prefix {hostname}- --debug --show-rc x@x.repo.borgbase.com:repo
using builtin fallback logging configuration
35 self tests completed in 0.12 seconds
SSH command line: ['ssh', 'x@x.repo.borgbase.com', 'borg', 'serve', '--umask=077', '--debug']
Remote: using builtin fallback logging configuration
Remote: 35 self tests completed in 0.06 seconds
Remote: using builtin fallback logging configuration
Remote: Initialized logging system for JSON-based protocol
Remote: Resolving repository path b'repo'
Remote: Resolved repository path to '/srv/repos/x/repo'
Remote: Verified integrity of /srv/repos/x/repo/index.43
TAM-verified manifest
security: read previous location 'ssh://x@x.repo.borgbase.com/./repo'
security: read manifest timestamp '2021-11-07T12:51:42.098496'
security: determined newest manifest timestamp as 2021-11-07T12:51:42.098496
security: repository checks ok, allowing access
At least one of the "keep-within", "keep-last", "keep-secondly", "keep-minutely", "keep-hourly", "keep-daily", "keep-weekly", "keep-monthly" or "keep-yearly" settings must be specified.
RemoteRepository: 213 B bytes sent, 3.16 kB bytes received, 5 messages sent
terminating with error status, rc 2
x@x.repo.borgbase.com:repo: Error running actions for repository
Command '/usr/local/bin/borg prune --prefix {hostname}- --debug --show-rc x@x.repo.borgbase.com:repo' returned non-zero exit status 2.
/etc/borgmatic/config.yaml: No commands to run for on-error hook
/etc/borgmatic/config.yaml: Error running configuration file
/etc/borgmatic/config.yaml: No commands to run for post-everything hook

summary:
/etc/borgmatic/config.yaml: Error running configuration file
x@x.repo.borgbase.com:repo: Error running actions for repository
using builtin fallback logging configuration
35 self tests completed in 0.12 seconds
SSH command line: ['ssh', 'x@x.repo.borgbase.com', 'borg', 'serve', '--umask=077', '--debug']
Remote: using builtin fallback logging configuration
Remote: 35 self tests completed in 0.06 seconds
Remote: using builtin fallback logging configuration
Remote: Initialized logging system for JSON-based protocol
Remote: Resolving repository path b'repo'
Remote: Resolved repository path to '/srv/repos/x/repo'
Remote: Verified integrity of /srv/repos/x/repo/index.43
TAM-verified manifest
security: read previous location 'ssh://x@x.repo.borgbase.com/./repo'
security: read manifest timestamp '2021-11-07T12:51:42.098496'
security: determined newest manifest timestamp as 2021-11-07T12:51:42.098496
security: repository checks ok, allowing access
At least one of the "keep-within", "keep-last", "keep-secondly", "keep-minutely", "keep-hourly", "keep-daily", "keep-weekly", "keep-monthly" or "keep-yearly" settings must be specified.
RemoteRepository: 213 B bytes sent, 3.16 kB bytes received, 5 messages sent
terminating with error status, rc 2
Command '/usr/local/bin/borg prune --prefix {hostname}- --debug --show-rc x@x.repo.borgbase.com:repo' returned non-zero exit status 2.

Need some help? https://torsion.org/borgmatic/#issues

With the detail on At least one of the "keep-within", "keep-last", "keep-secondly", "keep-minutely", "keep-hourly", "keep-daily", "keep-weekly", "keep-monthly" or "keep-yearly" settings must be specified..

Looking through the docs leaves me none the wiser:

At least one of the "keep" options is required for pruning to work. To skip pruning entirely, run "borgmatic create" or "check" without the "prune" action. See borgmatic documentation for details.

I did not specify any, I did not specify "prune". I do not even know how to not-specify prune. The program fails because I did not specify at least one keep-* parameter. As soon as I specify one, prune gets executed.

Last try was to just to uncomment the retention key, but that fails as well:

Ensuring legacy configuration is upgraded
/etc/borgmatic/config.yaml /etc/borgmatic.d /root/.config/borgmatic/config.yaml /root/.config/borgmatic.d: No valid configuration files found

summary:
/etc/borgmatic/config.yaml: Error parsing configuration file
An error occurred while parsing a configuration file at /etc/borgmatic/config.yaml:
At 'retention': None is not of type 'object'
/etc/borgmatic/config.yaml /etc/borgmatic.d /root/.config/borgmatic/config.yaml /root/.config/borgmatic.d: No valid configuration files found

Need some help? https://torsion.org/borgmatic/#issues

Expected behavior (if a bug)

Skip prune.

Other notes / implementation ideas

None.

Environment

borgmatic version: 1.5.20

borgmatic installation method:

pip3 on debian

Borg version: borg 1.1.17

Python version: Python 3.8.10

Database version (if applicable): n/a

operating system and version: Debian 8

#### What I'm trying to do and why Use append only backups without prune. #### Steps to reproduce (if a bug) I use the following base configuration as a base: ```yaml location: source_directories: - /home repositories: - x@x.repo.borgbase.com:repo local_path: /usr/local/bin/borg exclude_patterns: - '/*/.borgkey' - '*.pyc' - '*.tmp' exclude_caches: true exclude_if_present: - .nobackup exclude_nodump: true storage: encryption_passcommand: cat /root/.borgkey checkpoint_interval: 43200 consistency: checks: - repository ``` The configuration is considered valid: ```bash $ validate-borgmatic-config All given configuration files are valid: /etc/borgmatic/config.yaml ``` #### Actual behavior (if a bug) When I trigger the backup with `borgmatic --verbosity 2` it kills with ```log Ensuring legacy configuration is upgraded /etc/borgmatic/config.yaml: No commands to run for pre-everything hook /etc/borgmatic/config.yaml: No commands to run for pre-prune hook /etc/borgmatic/config.yaml: No commands to run for pre-backup hook /etc/borgmatic/config.yaml: No commands to run for pre-check hook x@x.repo.borgbase.com:repo: Pruning archives /usr/local/bin/borg prune --prefix {hostname}- --debug --show-rc x@x.repo.borgbase.com:repo using builtin fallback logging configuration 35 self tests completed in 0.12 seconds SSH command line: ['ssh', 'x@x.repo.borgbase.com', 'borg', 'serve', '--umask=077', '--debug'] Remote: using builtin fallback logging configuration Remote: 35 self tests completed in 0.06 seconds Remote: using builtin fallback logging configuration Remote: Initialized logging system for JSON-based protocol Remote: Resolving repository path b'repo' Remote: Resolved repository path to '/srv/repos/x/repo' Remote: Verified integrity of /srv/repos/x/repo/index.43 TAM-verified manifest security: read previous location 'ssh://x@x.repo.borgbase.com/./repo' security: read manifest timestamp '2021-11-07T12:51:42.098496' security: determined newest manifest timestamp as 2021-11-07T12:51:42.098496 security: repository checks ok, allowing access At least one of the "keep-within", "keep-last", "keep-secondly", "keep-minutely", "keep-hourly", "keep-daily", "keep-weekly", "keep-monthly" or "keep-yearly" settings must be specified. RemoteRepository: 213 B bytes sent, 3.16 kB bytes received, 5 messages sent terminating with error status, rc 2 x@x.repo.borgbase.com:repo: Error running actions for repository Command '/usr/local/bin/borg prune --prefix {hostname}- --debug --show-rc x@x.repo.borgbase.com:repo' returned non-zero exit status 2. /etc/borgmatic/config.yaml: No commands to run for on-error hook /etc/borgmatic/config.yaml: Error running configuration file /etc/borgmatic/config.yaml: No commands to run for post-everything hook summary: /etc/borgmatic/config.yaml: Error running configuration file x@x.repo.borgbase.com:repo: Error running actions for repository using builtin fallback logging configuration 35 self tests completed in 0.12 seconds SSH command line: ['ssh', 'x@x.repo.borgbase.com', 'borg', 'serve', '--umask=077', '--debug'] Remote: using builtin fallback logging configuration Remote: 35 self tests completed in 0.06 seconds Remote: using builtin fallback logging configuration Remote: Initialized logging system for JSON-based protocol Remote: Resolving repository path b'repo' Remote: Resolved repository path to '/srv/repos/x/repo' Remote: Verified integrity of /srv/repos/x/repo/index.43 TAM-verified manifest security: read previous location 'ssh://x@x.repo.borgbase.com/./repo' security: read manifest timestamp '2021-11-07T12:51:42.098496' security: determined newest manifest timestamp as 2021-11-07T12:51:42.098496 security: repository checks ok, allowing access At least one of the "keep-within", "keep-last", "keep-secondly", "keep-minutely", "keep-hourly", "keep-daily", "keep-weekly", "keep-monthly" or "keep-yearly" settings must be specified. RemoteRepository: 213 B bytes sent, 3.16 kB bytes received, 5 messages sent terminating with error status, rc 2 Command '/usr/local/bin/borg prune --prefix {hostname}- --debug --show-rc x@x.repo.borgbase.com:repo' returned non-zero exit status 2. Need some help? https://torsion.org/borgmatic/#issues ``` With the detail on `At least one of the "keep-within", "keep-last", "keep-secondly", "keep-minutely", "keep-hourly", "keep-daily", "keep-weekly", "keep-monthly" or "keep-yearly" settings must be specified.`. Looking through the docs leaves me none the wiser: > At least one of the "keep" options is required for pruning to work. To skip pruning entirely, run "borgmatic create" or "check" without the "prune" action. See borgmatic documentation for details. I did not specify any, I did not specify "prune". I do not even know how to not-specify prune. The program fails because I did not specify at least one `keep-*` parameter. As soon as I specify one, `prune` gets executed. Last try was to just to uncomment the retention key, but that fails as well: ``` Ensuring legacy configuration is upgraded /etc/borgmatic/config.yaml /etc/borgmatic.d /root/.config/borgmatic/config.yaml /root/.config/borgmatic.d: No valid configuration files found summary: /etc/borgmatic/config.yaml: Error parsing configuration file An error occurred while parsing a configuration file at /etc/borgmatic/config.yaml: At 'retention': None is not of type 'object' /etc/borgmatic/config.yaml /etc/borgmatic.d /root/.config/borgmatic/config.yaml /root/.config/borgmatic.d: No valid configuration files found Need some help? https://torsion.org/borgmatic/#issues ``` #### Expected behavior (if a bug) Skip prune. #### Other notes / implementation ideas None. #### Environment **borgmatic version:** 1.5.20 **borgmatic installation method:** pip3 on debian **Borg version:** borg 1.1.17 **Python version:** Python 3.8.10 **Database version (if applicable):** n/a **operating system and version:** Debian 8

witten commented

2021-11-07 17:17:44 +00:00

Owner

Thanks for filing this! The way to skip prune is to run the specific borgmatic actions you want, e.g. create and check on the command-line, omitting prune. Example:

borgmatic --verbosity 2 create check

See the documentation for more information.

It sounds like though there may be a documentation gap if this isn't clear, and perhaps this ticket could be used for fixing that.

It also sounds like perhaps you don't want to have specify a keep-* option, given that you'll never be pruning from this machine? One change I could envision is to change the configuration validation to accept a configuration even if zero keep-* options are specified. And then just blow up if the user ever tries to prune in that case.

Thanks for filing this! The way to skip prune is to run the specific borgmatic actions you want, e.g. `create` and `check` on the command-line, omitting `prune`. Example: ```bash borgmatic --verbosity 2 create check ``` See [the documentation](https://torsion.org/borgmatic/docs/how-to/deal-with-very-large-backups/#a-la-carte-actions) for more information. It sounds like though there may be a documentation gap if this isn't clear, and perhaps this ticket could be used for fixing that. It also sounds like perhaps you don't want to have specify a `keep-*` option, given that you'll never be pruning from this machine? One change I could envision is to change the configuration validation to accept a configuration even if zero `keep-*` options are specified. And then just blow up if the user ever tries to `prune` in that case.

adrian commented

2021-11-07 18:09:11 +00:00

Author

Ah, thats how it works. Did not check the command reference on what borgmatic without any parameters actually means, so I assumed it is just a "work with everything configured in the configuration file", not an alias for create, check and prune. I just worked through the setup chapter and did not see any example on a different use case except init. Now the actual error message makes more sense, even though the comment in the yaml itself seems a bit misleading. Also that you should configure prune anyway, even if you intend to never use it.

A hard exit if prune is unconfigured would be the more implicit handling I think, because it comes preconfigured and my first intention was to disable it, so I uncommented everythign related to it.

Ah, thats how it works. Did not check the command reference on what `borgmatic` without any parameters actually means, so I assumed it is just a "work with everything configured in the configuration file", not an alias for create, check and prune. I just worked through the [setup](https://torsion.org/borgmatic/docs/how-to/set-up-backups/) chapter and did not see any example on a different use case except `init`. Now the actual error message makes more sense, even though the comment in the yaml itself seems a bit misleading. Also that you should configure prune anyway, even if you intend to never use it. A hard exit if prune is unconfigured would be the more implicit handling I think, because it comes preconfigured and my first intention was to disable it, so I uncommented everythign related to it.

witten commented

2021-11-15 18:04:19 +00:00

Owner

That makes sense. I'll treat this as a documentation issue and potentially adjust the handling when prune is unconfigured. Thanks!

witten added the

good first issue

label

2023-02-04 17:42:22 +00:00

witten commented

2023-11-01 04:59:12 +00:00

Owner

There's now a new skip_actions configuration option that allows you to disable an action (like prune or compact) completely. See #701 or the documentation for more information.

There's now a new `skip_actions` configuration option that allows you to disable an action (like `prune` or `compact`) completely. See #701 or [the documentation](https://torsion.org/borgmatic/docs/how-to/set-up-backups/#skipping-actions) for more information.

witten closed this issue

2023-11-01 04:59:12 +00:00