Option to auto-adjust Network Bandwith limits #372

New Issue

adrien-github · 2020-11-23T13:24:34Z

adrien-github commented

2020-11-23 13:24:34 +00:00

What I'm trying to do and why

Hi,

We are using Borg to Backup linux laptops on our labs, and it works well !

Now a lot of our users work mainly from home, and some do not have a good connection for backup. In my case for instance I got only 80ko/s in upload (french ADSL). Without limit, the uplink is quickly saturated, and the connexion became totally unresponsive.

So our backup script evaluate the bandwith with iperf, and then set --remote-rate-limit as a third of the max bandwith. Still, if the bandwith is to small (less than 60ko/s in our case) the backup do not start, because it will never finish.

When a laptop is at home, the backup is slow, but well the connexion is still usable for other stuff.
When the laptop it at the workplace, it use the 1GB/s connexion, which is much better.

Could it be possible to implement such an « auto rating limit » in borgmatic ?

It is the only thing left preventing us to use borgmatic :-)

Kind regards,

Adrien

Environment

borgmatic version: 1.2.11 (from Debian Buster)

borgmatic installation method: Debian package

Borg version: 1.1.9

operating system and version: Debian Buster

#### What I'm trying to do and why Hi, We are using Borg to Backup linux laptops on our labs, and it works well ! Now a lot of our users work mainly from home, and some do not have a good connection for backup. In my case for instance I got only 80ko/s in upload (french ADSL). Without limit, the uplink is quickly saturated, and the connexion became totally unresponsive. So our backup script evaluate the bandwith with iperf, and then set --remote-rate-limit as a third of the max bandwith. Still, if the bandwith is to small (less than 60ko/s in our case) the backup do not start, because it will never finish. When a laptop is at home, the backup is slow, but well the connexion is still usable for other stuff. When the laptop it at the workplace, it use the 1GB/s connexion, which is much better. Could it be possible to implement such an « auto rating limit » in borgmatic ? It is the only thing left preventing us to use borgmatic :-) Kind regards, Adrien #### Environment **borgmatic version:** 1.2.11 (from Debian Buster) **borgmatic installation method:** Debian package **Borg version:** 1.1.9 **operating system and version:** Debian Buster

witten commented

2020-11-25 19:18:23 +00:00

Super interesting idea. Thanks for suggesting it. Would you be able to share your existing auto-rate-limiting script? That may help in making a borgmatic-native implementation.

What happens (or should happen) if the available bandwidth changes during the borgmatic backup?

Also note that the Borg FAQ has an entry for (non-auto) bandwidth rate limiting: https://borgbackup.readthedocs.io/en/stable/faq.html#is-there-a-way-to-limit-bandwidth-with-project-name

Super interesting idea. Thanks for suggesting it. Would you be able to share your existing auto-rate-limiting script? That may help in making a borgmatic-native implementation. What happens (or should happen) if the available bandwidth changes during the borgmatic backup? Also note that the Borg FAQ has an entry for (non-auto) bandwidth rate limiting: https://borgbackup.readthedocs.io/en/stable/faq.html#is-there-a-way-to-limit-bandwidth-with-project-name

adrien-github commented

2020-11-30 09:56:17 +00:00

Yes, sure. Be careful, ours scripts are a bit quick and dirty ;) The principle is simple, we just launch the command

iperf -y -C -c BORG_SERVER

then parse it to have the last field (max bandwith) in ko/s. At last we set up borg to use only the third of it.
The rest is small adjustment for our needs: do not limit backup on servers, do not backup if the bandwith is too small (it will take too long).

Here is the part of our bash script:

mk_bandwith() {
    BW_LIMIT="0"
    export BW_LIMIT

    #PCs or servers ?
    hostname | grep "^pc-\|^dmn" > /dev/null 2>&1 || return

    ##Test si Iperf est disponible sur le client
    IPERF="$(command -v iperf)" || return

    #test si netcat est installe
    NC="$(command -v nc)" || return

    #Test si iperf tourne sur le serveur
    "$NC" -w 5 -z "$SRV_BORG" 5001 || exit 1

    #Calcul de la bande passante avec iperf
    BW_MAX=$("$IPERF" -y C -c "$SRV_BORG"  | sed -e "s#.*,\([0-9]*\)#\1#" ) #Retourne la bw en bits
    BW_MAX=$(( "$BW_MAX" / 1024 / 8 )) #Retourne la bw en KiBytes
    BW_LIMIT=$(( "$BW_MAX" / "$RATIO" ))
    export BW_LIMIT

    #Si la bande passante est trop faible on abandonne la sauvegarde
    echo $(("$BW_LIMIT" - "$BW_MIN" )) | grep -- "-\|^0" > /dev/null 2>&1 && exit 1

    return
}

I tried to adapt it in python, in order to use borgmatic. Here is the cron script, which calculate the bandwith, change /etc/borgmatic/config.yaml then launch borgmatic:

#!/usr/bin/env python3

## This scripts need python3-yaml:

import socket
import subprocess
import re
import ruamel.yaml

## Config:
ratio=3 #Limitaion ratio of the bandwith
bw_min=20  #Minimum bandwith in kiBytes (with the ratio)
borgmatic_config = '/etc/borgmatic/config.yaml'
##

# only limit PCs, not server:
if socket.gethostname().startswith(('pc-', 'dmn')):
    # Recuperation du nom du serveur borg, depuis le fichier de conf borgmatic:
    yaml = ruamel.yaml.YAML()
    with open(borgmatic_config) as fp:
        data = yaml.load(fp)
    borg_server = re.split('@|:', data['location']['repositories'][0])[1]

    # Retrieve the bandwith with iperf
    iperf = subprocess.run(['/usr/bin/iperf -y C -c ' + borg_server], shell = True, capture_output=True)
    if iperf.stdout.decode() == '':
        exit(1) # no connexion to the borg server

    # bandwith in ko/s:
    bw_max = int(iperf.stdout.decode().split(',')[-1]) / 1024 / 8
    bw_limit = round(bw_max/ratio)
    if bw_limit < bw_min:
        exit(1) # bande passante too small : no backup

    # setup the bandwith
    data['storage']['remote_rate_limit'] = bw_limit

    with open(borgmatic_config, 'w') as fp:
        yaml.dump(data, fp)

# Now we launch the backup
subprocess.run(['/usr/bin/borgmatic', '--create'])

On the borg server, you should have iperf listening. I could be a security concern if not on a controlled network. Here is our systemd service :

# /etc/systemd/system/iperf.service
[Unit]
Description=iperf server
After=syslog.target network.target auditd.service

[Service]
Restart=on-failure
RestartSec=5s
ExecStart=/usr/bin/iperf -s

[Install]
WantedBy=multi-user.target

Our system is quite naive, it may be improved on several points : use iperf3 instead of iperf, and as you mention it, it does not adapt in realtime during the backup (we assume the backup should be quick, so the bandwith do not change a lot)

Kind regards

Adrien

Yes, sure. Be careful, ours scripts are a bit quick and dirty ;) The principle is simple, we just launch the command ``` iperf -y -C -c BORG_SERVER ``` then parse it to have the last field (max bandwith) in ko/s. At last we set up borg to use only the third of it. The rest is small adjustment for our needs: do not limit backup on servers, do not backup if the bandwith is too small (it will take too long). Here is the part of our bash script: ``` mk_bandwith() { BW_LIMIT="0" export BW_LIMIT #PCs or servers ? hostname | grep "^pc-\|^dmn" > /dev/null 2>&1 || return ##Test si Iperf est disponible sur le client IPERF="$(command -v iperf)" || return #test si netcat est installe NC="$(command -v nc)" || return #Test si iperf tourne sur le serveur "$NC" -w 5 -z "$SRV_BORG" 5001 || exit 1 #Calcul de la bande passante avec iperf BW_MAX=$("$IPERF" -y C -c "$SRV_BORG" | sed -e "s#.*,$[0-9]*$#\1#" ) #Retourne la bw en bits BW_MAX=$(( "$BW_MAX" / 1024 / 8 )) #Retourne la bw en KiBytes BW_LIMIT=$(( "$BW_MAX" / "$RATIO" )) export BW_LIMIT #Si la bande passante est trop faible on abandonne la sauvegarde echo $(("$BW_LIMIT" - "$BW_MIN" )) | grep -- "-\|^0" > /dev/null 2>&1 && exit 1 return } ``` I tried to adapt it in python, in order to use borgmatic. Here is the cron script, which calculate the bandwith, change /etc/borgmatic/config.yaml then launch borgmatic: ``` #!/usr/bin/env python3 ## This scripts need python3-yaml: import socket import subprocess import re import ruamel.yaml ## Config: ratio=3 #Limitaion ratio of the bandwith bw_min=20 #Minimum bandwith in kiBytes (with the ratio) borgmatic_config = '/etc/borgmatic/config.yaml' ## # only limit PCs, not server: if socket.gethostname().startswith(('pc-', 'dmn')): # Recuperation du nom du serveur borg, depuis le fichier de conf borgmatic: yaml = ruamel.yaml.YAML() with open(borgmatic_config) as fp: data = yaml.load(fp) borg_server = re.split('@|:', data['location']['repositories'][0])[1] # Retrieve the bandwith with iperf iperf = subprocess.run(['/usr/bin/iperf -y C -c ' + borg_server], shell = True, capture_output=True) if iperf.stdout.decode() == '': exit(1) # no connexion to the borg server # bandwith in ko/s: bw_max = int(iperf.stdout.decode().split(',')[-1]) / 1024 / 8 bw_limit = round(bw_max/ratio) if bw_limit < bw_min: exit(1) # bande passante too small : no backup # setup the bandwith data['storage']['remote_rate_limit'] = bw_limit with open(borgmatic_config, 'w') as fp: yaml.dump(data, fp) # Now we launch the backup subprocess.run(['/usr/bin/borgmatic', '--create']) ``` On the borg server, you should have iperf listening. I could be a security concern if not on a controlled network. Here is our systemd service : ``` # /etc/systemd/system/iperf.service [Unit] Description=iperf server After=syslog.target network.target auditd.service [Service] Restart=on-failure RestartSec=5s ExecStart=/usr/bin/iperf -s [Install] WantedBy=multi-user.target ``` Our system is quite naive, it may be improved on several points : use iperf3 instead of iperf, and as you mention it, it does not adapt in realtime during the backup (we assume the backup should be quick, so the bandwith do not change a lot) Kind regards Adrien

witten added the

new feature area

label 2023-06-28 18:49:16 +00:00

Sign in to join this conversation.