Borg server - documentation of checks only #364

New Issue

NoxInmortus · 2020-10-30T10:58:20Z

NoxInmortus commented

2020-10-30 10:58:20 +00:00

What I'm trying to do and why

I'm building a borg/borgmatic little cluster, I have 3 nodes on which I do backups and they are sent to a fourth server only used as a remote repository.

I also wish to use borgmatic on the repository server to manage the whole. I want to use it for manual actions, but I would also like to use it for the consistency checks feature.

I could configure checks on the nodes I want to backup, but then the whole stuff would use more cpu and bandwith than checks ran on the repository server only.

borgmatic configuration on the repository node :

location:
    source_directories: []

    one_file_system: false
    files_cache: ctime,size,inode
    repositories:
        - /var/backups/borgbackup
    exclude_from:
        - /etc/borgmatic/excludes
    exclude_caches: true
    exclude_if_present: .nobackup
    borgmatic_source_directory: /tmp/borgmatic

storage:
    encryption_passphrase: password
    compression: lz4
    remote_rate_limit: 5000
    umask: 0077
    lock_wait: 5
    archive_name_format: '{hostname}-{now}'
    relocated_repo_access_is_ok: true

retention:
    keep_within: 2d
    keep_daily: 7
    keep_weekly: 4
    keep_monthly: 1
    keep_yearly: 1
    prefix: '{hostname}-'

consistency:
    checks:
        - repository
        - extract
        - data

    check_last: 1
    prefix: '{hostname}-'

hooks:
    umask: 0077

borgmatic configuration on the backup nodes :

location:
    source_directories: 
        - /var/www

    one_file_system: false
    files_cache: ctime,size,inode
    repositories:
        - user@remote_server:/var/backups/borgbackup
    exclude_from:
        - /etc/borgmatic/excludes
    exclude_caches: true
    exclude_if_present: .nobackup
    borgmatic_source_directory: /tmp/borgmatic

storage:
    encryption_passphrase: password
    compression: lz4
    remote_rate_limit: 5000
    umask: 0077
    lock_wait: 5
    archive_name_format: '{hostname}-{now}'
    relocated_repo_access_is_ok: true

retention:
    keep_within: 2d
    keep_daily: 6
    keep_weekly: 4
    keep_monthly: 1
    keep_yearly: 2
    prefix: '{hostname}-'

consistency:
    checks:
        - repository
    check_last: 1
    prefix: '{hostname}-'

hooks:
    mysql_databases:
        - name: xxxx
          options: --single-transaction --quick
          password: xxxx
          username: xxxx
        - name: xxxx
          options: --single-transaction --quick
          password: xxxx
          username: xxxx
    umask: 0077

Actual behavior

Trying to launch borgmatic with sources_directories: [] on the repository server show this :

[2020-10-30 11:37:06,322] CRITICAL: /var/backups/borgbackup: Error running actions for repository
[2020-10-30 11:37:06,322] CRITICAL: Command 'borg create --exclude-from /etc/borgmatic/excludes --exclude-caches --exclude-if-present .nobackup --compression lz4 --remote-ratelimit 5000 --files-cache ctime,size,inode --umask 77 --lock-wait 5 --debug --show-rc /var/backups/borgbackup::{hostname}-{now}' returned non-zero exit status 2

Other notes / implementation ideas

I'm not sure if what i'm trying to do is currently possible or not. If it is not, make it a feature request : To be able to run borgmatic for checks only.

Environment

borg and borgmatic latest version, Python 3.5.3, Debian 10 64 bits

#### What I'm trying to do and why I'm building a borg/borgmatic little cluster, I have 3 nodes on which I do backups and they are sent to a fourth server only used as a remote repository. I also wish to use borgmatic on the repository server to manage the whole. I want to use it for manual actions, but I would also like to use it for the consistency checks feature. I could configure checks on the nodes I want to backup, but then the whole stuff would use more cpu and bandwith than checks ran on the repository server only. borgmatic configuration on the repository node : ``` location: source_directories: [] one_file_system: false files_cache: ctime,size,inode repositories: - /var/backups/borgbackup exclude_from: - /etc/borgmatic/excludes exclude_caches: true exclude_if_present: .nobackup borgmatic_source_directory: /tmp/borgmatic storage: encryption_passphrase: password compression: lz4 remote_rate_limit: 5000 umask: 0077 lock_wait: 5 archive_name_format: '{hostname}-{now}' relocated_repo_access_is_ok: true retention: keep_within: 2d keep_daily: 7 keep_weekly: 4 keep_monthly: 1 keep_yearly: 1 prefix: '{hostname}-' consistency: checks: - repository - extract - data check_last: 1 prefix: '{hostname}-' hooks: umask: 0077 ``` borgmatic configuration on the backup nodes : ``` location: source_directories: - /var/www one_file_system: false files_cache: ctime,size,inode repositories: - user@remote_server:/var/backups/borgbackup exclude_from: - /etc/borgmatic/excludes exclude_caches: true exclude_if_present: .nobackup borgmatic_source_directory: /tmp/borgmatic storage: encryption_passphrase: password compression: lz4 remote_rate_limit: 5000 umask: 0077 lock_wait: 5 archive_name_format: '{hostname}-{now}' relocated_repo_access_is_ok: true retention: keep_within: 2d keep_daily: 6 keep_weekly: 4 keep_monthly: 1 keep_yearly: 2 prefix: '{hostname}-' consistency: checks: - repository check_last: 1 prefix: '{hostname}-' hooks: mysql_databases: - name: xxxx options: --single-transaction --quick password: xxxx username: xxxx - name: xxxx options: --single-transaction --quick password: xxxx username: xxxx umask: 0077 ``` #### Actual behavior Trying to launch borgmatic with `sources_directories: []` on the repository server show this : ``` [2020-10-30 11:37:06,322] CRITICAL: /var/backups/borgbackup: Error running actions for repository [2020-10-30 11:37:06,322] CRITICAL: Command 'borg create --exclude-from /etc/borgmatic/excludes --exclude-caches --exclude-if-present .nobackup --compression lz4 --remote-ratelimit 5000 --files-cache ctime,size,inode --umask 77 --lock-wait 5 --debug --show-rc /var/backups/borgbackup::{hostname}-{now}' returned non-zero exit status 2 ``` #### Other notes / implementation ideas I'm not sure if what i'm trying to do is currently possible or not. If it is not, make it a feature request : To be able to run borgmatic for checks only. #### Environment borg and borgmatic latest version, Python 3.5.3, Debian 10 64 bits

👍 1

witten commented

2020-10-30 16:41:49 +00:00

Interesting use case.. Thanks for writing it up! I assume you've seen how to run independent borgmatic actions? E.g.: borgmatic check. Is the issue that you'd like a way to put this check-only behavior in borgmatic's configuration file, rather than specifying it on the command-line? Or would the command-line borgmatic check work for you?

Interesting use case.. Thanks for writing it up! I assume you've seen how to run [independent borgmatic actions](https://torsion.org/borgmatic/docs/how-to/deal-with-very-large-backups/#a-la-carte-actions)? E.g.: `borgmatic check`. Is the issue that you'd like a way to put this check-only behavior in borgmatic's configuration file, rather than specifying it on the command-line? Or would the command-line `borgmatic check` work for you?

NoxInmortus commented

2020-10-30 16:51:26 +00:00

Hello @witten

It really depends, the documentation says

    check (--check, -k)
                        Check archives for consistency

What do that means ? If I compare that to the consistency checks :

    repository: Checks the consistency of the repository itself.
    archives: Checks all of the archives in the repository.
    extract: Performs an extraction dry-run of the most recent archive.
    data: Verifies the data integrity of all archives contents, decrypting and decompressing all data (implies archives as well).

For a "borg server", I would need to be able to start all the availables checks, not only one of them.

Or maybe I did not understood and borgmatic check is fetching the wanted checks in the config.yml ?

Hello @witten It really depends, the documentation says ``` check (--check, -k) Check archives for consistency ``` What do that means ? If I compare that to the consistency checks : ``` repository: Checks the consistency of the repository itself. archives: Checks all of the archives in the repository. extract: Performs an extraction dry-run of the most recent archive. data: Verifies the data integrity of all archives contents, decrypting and decompressing all data (implies archives as well). ``` For a "borg server", I would need to be able to start all the availables checks, not only one of them. Or maybe I did not understood and `borgmatic check` is fetching the wanted checks in the config.yml ?

witten commented

2020-10-30 17:16:05 +00:00

Yes! borgmatic check is running through all the specified checks from the configuration file. I can update the command-help to make this much more clear.

Yes! `borgmatic check` is running through all the specified checks from the configuration file. I can update the command-help to make this much more clear.

NoxInmortus commented

2020-10-30 17:43:13 +00:00

Well that's wonderful. More details of the --check option would be good I concur.
Just need to edit my cron now.

Thanks again for your time @witten

Well that's wonderful. More details of the --check option would be good I concur. Just need to edit my cron now. Thanks again for your time @witten

NoxInmortus closed this issue

2020-10-30 17:43:30 +00:00

witten commented

2020-10-30 19:20:41 +00:00

Sure thing! I'll re-open this ticket to cover improving the documentation of the check action!

Sure thing! I'll re-open this ticket to cover improving the documentation of the `check` action!

witten reopened this issue

2020-10-30 19:20:45 +00:00

witten changed title from ~~Borg server - checks only~~ to Borg server - documentation of checks only

2020-10-30 19:21:01 +00:00

NoxInmortus commented

2020-11-05 08:38:22 +00:00

Hello back @witten !

I'm noticing that if you use borgmatic check the hooks are not used, at least for theses ones (i did not test on_error):

    before_backup
    after_backup
    before_everything
    after_everything

that before and after_backup are not executed, that seems rational, but what about before/after_everything ?

Hello back @witten ! I'm noticing that if you use `borgmatic check` the hooks are not used, at least for theses ones (i did not test on_error): ``` before_backup after_backup before_everything after_everything ``` that before and after_backup are not executed, that seems rational, but what about before/after_everything ?

witten commented

2020-11-05 18:54:31 +00:00

Yeah, before/after_everything currently only trigger if there's a create action. I think it was initially a variant of before_backup that runs before all actions.

Have you seen the before/after_check hooks? If that doesn't work for your use case, could you tell me a little more about the problem there?

Yeah, `before/after_everything` currently only trigger if there's a `create` action. I think it was initially a variant of `before_backup` that runs before all actions. Have you seen the `before/after_check` hooks? If that doesn't work for your use case, could you tell me a little more about the problem there?

NoxInmortus commented

2020-11-05 20:44:32 +00:00

Damn, I was sure I had read correctly the reference documentation... it's what I need, sorry for bothering again.

witten commented

2020-11-05 21:15:47 +00:00

No worries! I'm glad to hear that does it for you. If you can think of any improvements I can make to the docs, do let me know!

witten referenced this issue

2022-03-17 04:35:46 +00:00

Extracting archives to remote pc #494

gerdneuman referenced this issue

2022-03-17 05:02:15 +00:00

Extracting archives to remote pc #494

witten referenced this issue

2023-05-14 21:33:25 +00:00

unified borgmatic command to trigger actions (prune or not) from what is in the config file #701

witten referenced this issue from a commit

2023-06-26 05:50:06 +00:00

Some brief documentation on running only checks (#364).

witten commented

2023-06-26 05:51:09 +00:00

I believe this is documented now! If you find that it's not, please feel free to open another ticket or follow-up here. Thanks for your patience.

https://torsion.org/borgmatic/docs/how-to/deal-with-very-large-backups/

I believe this is documented now! If you find that it's not, please feel free to open another ticket or follow-up here. Thanks for your patience. https://torsion.org/borgmatic/docs/how-to/deal-with-very-large-backups/

witten closed this issue

2023-06-26 05:51:09 +00:00

Sign in to join this conversation.