Borgbase SSH issues #544

New Issue

modem7 · 2022-06-12T18:43:03Z

modem7 commented

2022-06-12 18:43:03 +00:00

What I'm trying to do and why

Backing up to Borgbase

Steps to reproduce (if a bug)

borgmatic --stats -v 1 --files 2>&1

If checks are enabled, it seems that SSH may have too many connections in a short period of time and refuses any further connections. I'm seeing 3-4 SSH connections per backup (without checks), and it could be that SSH sees this as a potential issue.

2022-06-12 19:41:31	logout	my.ip.*		
2022-06-12 19:41:30	login	my.ip.*	MYSERVER	Host: box-eu16.borgbase.com
2022-06-12 19:41:29	logout	my.ip.*		
2022-06-12 19:41:19	login	my.ip.*	MYSERVER	Host: box-eu16.borgbase.com
2022-06-12 19:41:18	logout	my.ip.*		
2022-06-12 19:41:17	login	my.ip.*	MYSERVER	Host: box-eu16.borgbase.com
2022-06-12 19:41:16	logout	my.ip.*		
2022-06-12 19:41:13	login	my.ip.*	MYSERVER	Host: box-eu16.borgbase.com

Actual behavior (if a bug)

summary:
/etc/borgmatic.d/config.yml: Error running configuration file
user@user.repo.borgbase.com:repo: Error running actions for repository
Remote: Connection reset by 95.217.115.47 port 22
Connection closed by remote host. Is borg working on the server?
Command 'borg check --archives-only --prefix {hostname}- --info user@user.repo.borgbase.com:repo' returned non-zero exit status 2.

After a period of time, SSH works again for that particular server with no issues (backup still not working with checks enabled however).

Expected behavior (if a bug)

For checks to work with default SSH behaviour on Borgbase.

I understand it's possibly not a Borg(matic) issue per se, but the behaviour seems to have changed at some point recently, so I'm trying to figure out why/how/when, and get the appropriate parties (Borg/Borgmatic/Borgbase) involved to come to a solution as it probably won't just be me who is having/will have the issue.

This occurs on three different VM's with three different Borgbase servers/repos on multiple VLANs. DNS has been checked, and that's absolutely fine.

Other notes / implementation ideas

location:
    source_directories:
        - /mnt/source/

    repositories:
        - user@user.repo.borgbase.com:repo
    one_file_system: true
    exclude_caches: true

storage:
#   Passphase is set in variable $BORG_PASSPHRASE
    compression: lz4
    archive_name_format: 'backup-{now}'

retention:
    keep_hourly: 0
    keep_daily: 7
    keep_weekly: 4
    keep_monthly: 12
    keep_yearly: 1
    prefix: 'backup-'

consistency:
    checks:
#        - name: disabled
        - name: repository
          frequency: 1 week
        - name: archives
          frequency: always
        - name: extract
          frequency: 2 weeks
        - name: data
          frequency: 1 month
    prefix: 'backup-'

hooks:
    before_everything:
        - echo "Starting a backup job."
    after_everything:
        - echo "Backup created."
    on_error:
        - echo "Error while creating a backup."

    # https://torsion.org/borgmatic/docs/how-to/backup-your-databases/

    # Third-party services to notify you if backups aren't happening.
    healthchecks: https://hc-ping.com/uuid

    # https://torsion.org/borgmatic/docs/reference/configuration/

Environment

borgmatic version: 1.6.3

borgmatic installation method: Docker

Borg version: 1.2.1

Python version: 3.10.4

#### What I'm trying to do and why Backing up to Borgbase #### Steps to reproduce (if a bug) `borgmatic --stats -v 1 --files 2>&1` If checks are enabled, it seems that SSH may have too many connections in a short period of time and refuses any further connections. I'm seeing 3-4 SSH connections per backup (without checks), and it could be that SSH sees this as a potential issue. ``` 2022-06-12 19:41:31 logout my.ip.* 2022-06-12 19:41:30 login my.ip.* MYSERVER Host: box-eu16.borgbase.com 2022-06-12 19:41:29 logout my.ip.* 2022-06-12 19:41:19 login my.ip.* MYSERVER Host: box-eu16.borgbase.com 2022-06-12 19:41:18 logout my.ip.* 2022-06-12 19:41:17 login my.ip.* MYSERVER Host: box-eu16.borgbase.com 2022-06-12 19:41:16 logout my.ip.* 2022-06-12 19:41:13 login my.ip.* MYSERVER Host: box-eu16.borgbase.com ``` #### Actual behavior (if a bug) ``` summary: /etc/borgmatic.d/config.yml: Error running configuration file user@user.repo.borgbase.com:repo: Error running actions for repository Remote: Connection reset by 95.217.115.47 port 22 Connection closed by remote host. Is borg working on the server? Command 'borg check --archives-only --prefix {hostname}- --info user@user.repo.borgbase.com:repo' returned non-zero exit status 2. ``` After a period of time, SSH works again for that particular server with no issues (backup still not working with checks enabled however). #### Expected behavior (if a bug) For checks to work with default SSH behaviour on Borgbase. I understand it's possibly not a Borg(matic) issue per se, but the behaviour seems to have changed at some point recently, so I'm trying to figure out why/how/when, and get the appropriate parties (Borg/Borgmatic/Borgbase) involved to come to a solution as it probably won't just be me who is having/will have the issue. This occurs on three different VM's with three different Borgbase servers/repos on multiple VLANs. DNS has been checked, and that's absolutely fine. #### Other notes / implementation ideas ``` location: source_directories: - /mnt/source/ repositories: - user@user.repo.borgbase.com:repo one_file_system: true exclude_caches: true storage: # Passphase is set in variable $BORG_PASSPHRASE compression: lz4 archive_name_format: 'backup-{now}' retention: keep_hourly: 0 keep_daily: 7 keep_weekly: 4 keep_monthly: 12 keep_yearly: 1 prefix: 'backup-' consistency: checks: # - name: disabled - name: repository frequency: 1 week - name: archives frequency: always - name: extract frequency: 2 weeks - name: data frequency: 1 month prefix: 'backup-' hooks: before_everything: - echo "Starting a backup job." after_everything: - echo "Backup created." on_error: - echo "Error while creating a backup." # https://torsion.org/borgmatic/docs/how-to/backup-your-databases/ # Third-party services to notify you if backups aren't happening. healthchecks: https://hc-ping.com/uuid # https://torsion.org/borgmatic/docs/reference/configuration/ ``` #### Environment **borgmatic version:** 1.6.3 **borgmatic installation method:** Docker **Borg version:** 1.2.1 **Python version:** 3.10.4

modem7 changed title from ~~Borgbase issues~~ to Borgbase SSH issues

2022-06-14 01:54:36 +00:00

witten commented

2022-06-14 17:01:55 +00:00

As far as I know, this is not a borgmatic issue. But I have a few suggestions:

The error message isn't exactly the same, but checkout out this SSH troubleshooting tip that may be relevant.
There's also a long (old) Borg ticket on the issue.
Seems unlikely since you're getting an actual connection reset, but you could try diagnosing your network connectivity between your machine and Borgbase. Tools like ping, tracepath, and mtr may help.
If this is an intermittent thing you simply want to work around, you could try using the borgmatic retries functionality to re-attempt a connection when one fails. See retries and retry_wait in the storage section of borgmatic's configuration.
If none of that does it, maybe contact Borgbase support and see if they've got ideas?

Hope some of that helps!

As far as I know, this is not a borgmatic issue. But I have a few suggestions: * The error message isn't exactly the same, but checkout out [this SSH troubleshooting tip](https://torsion.org/borgmatic/docs/how-to/deal-with-very-large-backups/#broken-pipe-with-remote-repository) that may be relevant. * There's also a [long (old) Borg ticket](https://github.com/borgbackup/borg/issues/636) on the issue. * Seems unlikely since you're getting an actual connection reset, but you could try diagnosing your network connectivity between your machine and Borgbase. Tools like `ping`, `tracepath`, and `mtr` may help. * If this is an intermittent thing you simply want to work around, you could try using the borgmatic retries functionality to re-attempt a connection when one fails. See `retries` and `retry_wait` in the `storage` section of borgmatic's configuration. * If none of that does it, maybe contact Borgbase support and see if they've got ideas? Hope some of that helps!

witten added the

question / support

label 2022-06-14 17:02:04 +00:00

witten added the

waiting for response

label 2022-06-16 23:23:00 +00:00

witten commented

2022-10-05 06:06:24 +00:00

Closing this for now, but feel free to follow up. We can always reopen it. Thanks!

witten closed this issue

2022-10-05 06:06:24 +00:00

Sign in to join this conversation.