Feature Request: Store backup results to a JSON file #617

New Issue

crosenbe · 2022-12-01T20:06:05Z

crosenbe commented

2022-12-01 20:06:05 +00:00

What I'm trying to do and why

Monitoring backups is essential and the only way to provide information to a monitoring system is to use hooks. Traps are not reliable, storing to a file is perhaps the better idea. But the status is not enough, to monitor the backups more informations will be useful.

start time of the last backup
finish time of the last backup
status flag of the last backup
duration of the last backup
time of last successful backup
used borgmatic configuration file
valid maximum age of the last backup

Example JSON file

[
    {
        "finished": "24.11.2022 - 01:15:13",
        "started": "24.11.2022 - 01:13:20",
        "finished_unix": 1669248913,
        "status": 0,
        "started_unix": 1669248800,
        "last_success": "24.11.2022 - 01:15:13",
        "config": "/etc/borgmatic/nas/wifictl.yaml",
        "duration": 1,
        "last_success_unix": 1669248913,
        "max_age": 691200
    },
]

This can be done by using hooks and a provided python script, i.e.

hooks:
    before_backup:
        - write-status -c start -C {configuration_filename} -f /var/log/backups/borgmatic.json -m 8

    after_backup:
        - write-status -c success -C {configuration_filename} -f /var/log/backups/borgmatic.json -m 8

    on_error:
        - write-status -c failure -C {configuration_filename} -f /var/log/backups/borgmatic.json -m 8

write-status

import json
import getopt
from datetime import datetime as dt
import sys

def getTime():
    stamp = dt.now()
    return [ stamp.strftime('%d.%m.%Y - %H:%M:%S'), int(stamp.timestamp()) ]

def readJson(file):
    try:
        with open(file, 'r') as json_file:
            backup_results = json.load(json_file)
            return backup_results
    except OSError as e:
        return []

def writeJson(file, backup_results):
    json_string = json.dumps(backup_results, indent=4)
    with open(file, 'w') as json_file:
        json_file.write(json_string)

def run(command, config, file, max_age):
    try:
        max_age = int(max_age) * 86400
    except ValueError:
        usage()
    backup_results = readJson(file)
    time_stamp, time_stamp_unix = getTime()
    if backup_results is not None:
        config_list = [d['config'] for d in backup_results]
        if config not in config_list:
            backup_results.append( {'config': config } )
            config_list.append(config)

    list_entry = config_list.index(config)
    if command == 'start':
        backup_results[list_entry]['started'] = time_stamp
        backup_results[list_entry]['started_unix'] = time_stamp_unix
        backup_results[list_entry]['status'] = 1
    elif command == 'success':
        backup_results[list_entry]['duration'] = int(( time_stamp_unix - backup_results[list_entry]['started_unix'] ) / 60)
        backup_results[list_entry]['finished'] = time_stamp
        backup_results[list_entry]['finished_unix'] = time_stamp_unix
        backup_results[list_entry]['last_success'] = time_stamp
        backup_results[list_entry]['last_success_unix'] = time_stamp_unix
        backup_results[list_entry]['status'] = 0
    elif command == 'failure':
        backup_results[list_entry]['duration'] = int(( time_stamp_unix - backup_results[list_entry]['started_unix'] ) / 60)
        backup_results[list_entry]['finished'] = time_stamp
        backup_results[list_entry]['finished_unix'] = time_stamp_unix
        backup_results[list_entry]['status'] = 2
    else:
        usage()

    if 'last_success_unix' not in backup_results[list_entry]:
        backup_results[list_entry]['last_success_unix'] = 0
    if 'duration' not in backup_results[list_entry]:
        backup_results[list_entry]['duration'] = 0

    backup_results[list_entry]['max_age'] = max_age
    writeJson(file, backup_results)

def usage():
    print('write-status -c <command> -C <configuration> -f <file> -m <max-age>')
    print('     command:       start,success,failure)')
    print('     configuration: path to the borgmatic configuration file (yaml)')
    print('     file:          path to log json file results shoule be written')
    print('     max-age:       maximum age of last successful backup in days')
    exit(0)

def start(argv):
    command = ''
    config = ''
    file = ''
    max_age = ''

    try:
        opts, args = getopt.getopt(argv,"hc:C:f:m:",["help","command=","config=","file=","max-age="])
    except getopt.GetoptError:
        usage()
    for opt,arg in opts:
        if opt in ("-h", "--help"):
            usage()
        elif opt in ("-c", "--command"):
            if arg in ('start', 'success', 'failure'):
                command = arg
            else:
                usage()
        elif opt in ("-C", "--config"):
            config = arg
        elif opt in ("-f", "--file"):
            file = arg
        elif opt in ("-m", "--max-age"):
            max_age = arg

    if '' in (command, config, file, max_age):
        usage()
    else:
        run(command, config, file, max_age)

if __name__ == "__main__":
    start(sys.argv[1:])

Other notes / implementation ideas

It would be more elegant to have this feature integrated and extended providing more information about the backup. The integration could be done by providing relevant information as variable, i.e. defining the max_age inside the borgmatic configuration and to be used like {configuration_filename}. Additional information like transferred data to the backup repository would be also nice to see in the status. But this information is only forwarded from borg through borgmatic. Maybe it's possible to send all forwarded logs through a couple of regex and fetch data and store it also to variables.

Log Example
borgmatic: INFO RemoteRepository: 366.38 MB bytes sent, 32.64 kB bytes received, 215 messages sent

Maybe to fetch the data like that and store this to variables as well.

\s(?P<sent>[^\s]+)\s(?P<sentunit>\w\w)\sbytes\ssent,\s(?P<received>[^\s]+)\s(?P<receivedunit>\w\w)\sbytes\sreceived

#### What I'm trying to do and why Monitoring backups is essential and the only way to provide information to a monitoring system is to use hooks. Traps are not reliable, storing to a file is perhaps the better idea. But the status is not enough, to monitor the backups more informations will be useful. * start time of the last backup * finish time of the last backup * status flag of the last backup * duration of the last backup * time of last successful backup * used borgmatic configuration file * valid maximum age of the last backup *Example JSON file* ``` [ { "finished": "24.11.2022 - 01:15:13", "started": "24.11.2022 - 01:13:20", "finished_unix": 1669248913, "status": 0, "started_unix": 1669248800, "last_success": "24.11.2022 - 01:15:13", "config": "/etc/borgmatic/nas/wifictl.yaml", "duration": 1, "last_success_unix": 1669248913, "max_age": 691200 }, ] ``` This can be done by using hooks and a provided python script, i.e. ``` hooks: before_backup: - write-status -c start -C {configuration_filename} -f /var/log/backups/borgmatic.json -m 8 after_backup: - write-status -c success -C {configuration_filename} -f /var/log/backups/borgmatic.json -m 8 on_error: - write-status -c failure -C {configuration_filename} -f /var/log/backups/borgmatic.json -m 8 ``` *write-status* ``` import json import getopt from datetime import datetime as dt import sys def getTime(): stamp = dt.now() return [ stamp.strftime('%d.%m.%Y - %H:%M:%S'), int(stamp.timestamp()) ] def readJson(file): try: with open(file, 'r') as json_file: backup_results = json.load(json_file) return backup_results except OSError as e: return [] def writeJson(file, backup_results): json_string = json.dumps(backup_results, indent=4) with open(file, 'w') as json_file: json_file.write(json_string) def run(command, config, file, max_age): try: max_age = int(max_age) * 86400 except ValueError: usage() backup_results = readJson(file) time_stamp, time_stamp_unix = getTime() if backup_results is not None: config_list = [d['config'] for d in backup_results] if config not in config_list: backup_results.append( {'config': config } ) config_list.append(config) list_entry = config_list.index(config) if command == 'start': backup_results[list_entry]['started'] = time_stamp backup_results[list_entry]['started_unix'] = time_stamp_unix backup_results[list_entry]['status'] = 1 elif command == 'success': backup_results[list_entry]['duration'] = int(( time_stamp_unix - backup_results[list_entry]['started_unix'] ) / 60) backup_results[list_entry]['finished'] = time_stamp backup_results[list_entry]['finished_unix'] = time_stamp_unix backup_results[list_entry]['last_success'] = time_stamp backup_results[list_entry]['last_success_unix'] = time_stamp_unix backup_results[list_entry]['status'] = 0 elif command == 'failure': backup_results[list_entry]['duration'] = int(( time_stamp_unix - backup_results[list_entry]['started_unix'] ) / 60) backup_results[list_entry]['finished'] = time_stamp backup_results[list_entry]['finished_unix'] = time_stamp_unix backup_results[list_entry]['status'] = 2 else: usage() if 'last_success_unix' not in backup_results[list_entry]: backup_results[list_entry]['last_success_unix'] = 0 if 'duration' not in backup_results[list_entry]: backup_results[list_entry]['duration'] = 0 backup_results[list_entry]['max_age'] = max_age writeJson(file, backup_results) def usage(): print('write-status -c <command> -C <configuration> -f <file> -m <max-age>') print(' command: start,success,failure)') print(' configuration: path to the borgmatic configuration file (yaml)') print(' file: path to log json file results shoule be written') print(' max-age: maximum age of last successful backup in days') exit(0) def start(argv): command = '' config = '' file = '' max_age = '' try: opts, args = getopt.getopt(argv,"hc:C:f:m:",["help","command=","config=","file=","max-age="]) except getopt.GetoptError: usage() for opt,arg in opts: if opt in ("-h", "--help"): usage() elif opt in ("-c", "--command"): if arg in ('start', 'success', 'failure'): command = arg else: usage() elif opt in ("-C", "--config"): config = arg elif opt in ("-f", "--file"): file = arg elif opt in ("-m", "--max-age"): max_age = arg if '' in (command, config, file, max_age): usage() else: run(command, config, file, max_age) if __name__ == "__main__": start(sys.argv[1:]) ``` #### Other notes / implementation ideas It would be more elegant to have this feature integrated and extended providing more information about the backup. The integration could be done by providing relevant information as variable, i.e. defining the max_age inside the borgmatic configuration and to be used like {configuration_filename}. Additional information like transferred data to the backup repository would be also nice to see in the status. But this information is only forwarded from borg through borgmatic. Maybe it's possible to send all forwarded logs through a couple of regex and fetch data and store it also to variables. *Log Example* `borgmatic: INFO RemoteRepository: 366.38 MB bytes sent, 32.64 kB bytes received, 215 messages sent` Maybe to fetch the data like that and store this to variables as well. ``` \s(?P<sent>[^\s]+)\s(?P<sentunit>\w\w)\sbytes\ssent,\s(?P<received>[^\s]+)\s(?P<receivedunit>\w\w)\sbytes\sreceived ```

witten commented

2022-12-02 06:26:04 +00:00

That's an interesting idea about parsing the Borg output for basic backup statistics. The currently supported way to monitor the status and duration of borgmatic backups is to use one of the supported third-party monitoring services, some of which you can host yourself if you prefer. I assume none of those will work for your use case or you're already settled on a different tool? And/or perhaps you need the additional stats like bytes sent/received, etc?

Could you talk about how you intend to consume this JSON file? And what actions (if any) do you intend to take when this downstream tool consumes the JSON? Thanks!

That's an interesting idea about parsing the Borg output for basic backup statistics. The currently supported way to monitor the status and duration of borgmatic backups is to use one of the supported [third-party monitoring services](https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#third-party-monitoring-services), some of which you can host yourself if you prefer. I assume none of those will work for your use case or you're already settled on a different tool? And/or perhaps you need the additional stats like bytes sent/received, etc? Could you talk about how you intend to consume this JSON file? And what actions (if any) do you intend to take when this downstream tool consumes the JSON? Thanks!

crosenbe commented

2022-12-02 08:22:59 +00:00

I consume the file with Zabbix, just monitoring this file and discover it. The big advantage is i don't have to do to much configuration in zabbix in case of iterating this file and creating the monitored items in an automated way. As soon i create a new backup job a new JSON entry will be created and detected by Zabbix. And i don't need additional tools and be able to integrate borgmatic into the central monitoring solution.

Zabbix template

{
    "zabbix_export": {
        "version": "6.0",
        "date": "2022-12-02T08:17:39Z",
        "groups": [
            {
                "uuid": "7df96b18c230490a9a0a9e2307226338",
                "name": "Templates"
            },
        ],
        "templates": [
            {
                "uuid": "60b5c1230248485991079e203c80451e",
                "template": "BorgMatic",
                "name": "BorgMatic",
                "description": "Monitor backup result file.",
                "groups": [
                    {
                        "name": "Templates"
                    },
                ],
                "items": [
                    {
                        "uuid": "8983cc49c662405298b31dbe6164aff0",
                        "name": "Backup Status File",
                        "key": "vfs.file.contents[{$MON_FILE}]",
                        "history": "365d",
                        "trends": "0",
                        "value_type": "TEXT"
                    }
                ],
                "discovery_rules": [
                    {
                        "uuid": "4ea0345efbcb4d4aacc774b104e6031f",
                        "name": "Borgmatic Discovery",
                        "type": "DEPENDENT",
                        "key": "borgmatic.discover",
                        "delay": "0",
                        "item_prototypes": [
                            {
                                "uuid": "3689ee87e9a14e4dba663d2378bcb750",
                                "name": "Borgmatic Last Backup Duration [{#CONFIG}]",
                                "type": "DEPENDENT",
                                "key": "borgmatic.last.duration[{#CONFIG}]",
                                "delay": "0",
                                "units": "m",
                                "preprocessing": [
                                    {
                                        "type": "JSONPATH",
                                        "parameters": [
                                            "$[?(@.config == \"{#CONFIG}\")].duration.first()"
                                        ]
                                    }
                                ],
                                "master_item": {
                                    "key": "vfs.file.contents[{$MON_FILE}]"
                                },
                                "tags": [
                                    {
                                        "tag": "Application",
                                        "value": "Borgmatic"
                                    }
                                ]
                            },
                            {
                                "uuid": "9219741cce5640168dd45614fcce8ca0",
                                "name": "Borgmatic Last Success Backup [{#CONFIG}]",
                                "type": "DEPENDENT",
                                "key": "borgmatic.last_success[{#CONFIG}]",
                                "delay": "0",
                                "units": "unixtime",
                                "preprocessing": [
                                    {
                                        "type": "JSONPATH",
                                        "parameters": [
                                            "$[?(@.config == \"{#CONFIG}\")].last_success_unix.first()"
                                        ]
                                    }
                                ],
                                "master_item": {
                                    "key": "vfs.file.contents[{$MON_FILE}]"
                                },
                                "tags": [
                                    {
                                        "tag": "Application",
                                        "value": "Borgmatic"
                                    }
                                ]
                            },
                            {
                                "uuid": "f5056227e1414e0ea3eff1038c882271",
                                "name": "Borgmatic Valid Max Age [{#CONFIG}]",
                                "type": "DEPENDENT",
                                "key": "borgmatic.max_age[{#CONFIG}]",
                                "delay": "0",
                                "units": "s",
                                "preprocessing": [
                                    {
                                        "type": "JSONPATH",
                                        "parameters": [
                                            "$[?(@.config == \"{#CONFIG}\")].max_age.first()"
                                        ]
                                    }
                                ],
                                "master_item": {
                                    "key": "vfs.file.contents[{$MON_FILE}]"
                                },
                                "tags": [
                                    {
                                        "tag": "Application",
                                        "value": "Borgmatic"
                                    }
                                ]
                            },
                            {
                                "uuid": "247fe516d3374fe48d6b859884217e97",
                                "name": "Borgmatic Status [{#CONFIG}]",
                                "type": "DEPENDENT",
                                "key": "borgmatic.status[{#CONFIG}]",
                                "delay": "0",
                                "valuemap": {
                                    "name": "Backup Status"
                                },
                                "preprocessing": [
                                    {
                                        "type": "JSONPATH",
                                        "parameters": [
                                            "$[?(@.config == \"{#CONFIG}\")].status.first()"
                                        ]
                                    }
                                ],
                                "master_item": {
                                    "key": "vfs.file.contents[{$MON_FILE}]"
                                },
                                "tags": [
                                    {
                                        "tag": "Application",
                                        "value": "Borgmatic"
                                    }
                                ]
                            }
                        ],
                        "trigger_prototypes": [
                            {
                                "uuid": "28e78881fa0e4754bd4d3ac32dc4be20",
                                "expression": "now() - last(/BorgMatic/borgmatic.max_age[{#CONFIG}]) > last(/BorgMatic/borgmatic.last_success[{#CONFIG}])",
                                "recovery_mode": "RECOVERY_EXPRESSION",
                                "recovery_expression": "now() - last(/BorgMatic/borgmatic.max_age[{#CONFIG}]) < last(/BorgMatic/borgmatic.last_success[{#CONFIG}])",
                                "name": "Old Backup [{#CONFIG}]",
                                "priority": "HIGH",
                                "description": "Last Successful Backup is to old",
                                "manual_close": "YES"
                            }
                        ],
                        "master_item": {
                            "key": "vfs.file.contents[{$MON_FILE}]"
                        },
                        "lld_macro_paths": [
                            {
                                "lld_macro": "{#CONFIG}",
                                "path": "$..config.first()"
                            }
                        ]
                    }
                ],
                "macros": [
                    {
                        "macro": "{$MON_FILE}",
                        "value": "/var/log/backups/borgmatic.json"
                    },
                ],
                "valuemaps": [
                    {
                        "uuid": "ac3d95e26ada42c68e7f6c1cc46fb6c8",
                        "name": "Backup Status",
                        "mappings": [
                            {
                                "value": "0",
                                "newvalue": "Success"
                            },
                            {
                                "value": "1",
                                "newvalue": "Running"
                            },
                            {
                                "value": "2",
                                "newvalue": "Failure"
                            }
                        ]
                    }
                ]
            }
        ]
    }
}

I consume the file with Zabbix, just monitoring this file and discover it. The big advantage is i don't have to do to much configuration in zabbix in case of iterating this file and creating the monitored items in an automated way. As soon i create a new backup job a new JSON entry will be created and detected by Zabbix. And i don't need additional tools and be able to integrate borgmatic into the central monitoring solution. *Zabbix template* ``` { "zabbix_export": { "version": "6.0", "date": "2022-12-02T08:17:39Z", "groups": [ { "uuid": "7df96b18c230490a9a0a9e2307226338", "name": "Templates" }, ], "templates": [ { "uuid": "60b5c1230248485991079e203c80451e", "template": "BorgMatic", "name": "BorgMatic", "description": "Monitor backup result file.", "groups": [ { "name": "Templates" }, ], "items": [ { "uuid": "8983cc49c662405298b31dbe6164aff0", "name": "Backup Status File", "key": "vfs.file.contents[{$MON_FILE}]", "history": "365d", "trends": "0", "value_type": "TEXT" } ], "discovery_rules": [ { "uuid": "4ea0345efbcb4d4aacc774b104e6031f", "name": "Borgmatic Discovery", "type": "DEPENDENT", "key": "borgmatic.discover", "delay": "0", "item_prototypes": [ { "uuid": "3689ee87e9a14e4dba663d2378bcb750", "name": "Borgmatic Last Backup Duration [{#CONFIG}]", "type": "DEPENDENT", "key": "borgmatic.last.duration[{#CONFIG}]", "delay": "0", "units": "m", "preprocessing": [ { "type": "JSONPATH", "parameters": [ "$[?(@.config == \"{#CONFIG}\")].duration.first()" ] } ], "master_item": { "key": "vfs.file.contents[{$MON_FILE}]" }, "tags": [ { "tag": "Application", "value": "Borgmatic" } ] }, { "uuid": "9219741cce5640168dd45614fcce8ca0", "name": "Borgmatic Last Success Backup [{#CONFIG}]", "type": "DEPENDENT", "key": "borgmatic.last_success[{#CONFIG}]", "delay": "0", "units": "unixtime", "preprocessing": [ { "type": "JSONPATH", "parameters": [ "$[?(@.config == \"{#CONFIG}\")].last_success_unix.first()" ] } ], "master_item": { "key": "vfs.file.contents[{$MON_FILE}]" }, "tags": [ { "tag": "Application", "value": "Borgmatic" } ] }, { "uuid": "f5056227e1414e0ea3eff1038c882271", "name": "Borgmatic Valid Max Age [{#CONFIG}]", "type": "DEPENDENT", "key": "borgmatic.max_age[{#CONFIG}]", "delay": "0", "units": "s", "preprocessing": [ { "type": "JSONPATH", "parameters": [ "$[?(@.config == \"{#CONFIG}\")].max_age.first()" ] } ], "master_item": { "key": "vfs.file.contents[{$MON_FILE}]" }, "tags": [ { "tag": "Application", "value": "Borgmatic" } ] }, { "uuid": "247fe516d3374fe48d6b859884217e97", "name": "Borgmatic Status [{#CONFIG}]", "type": "DEPENDENT", "key": "borgmatic.status[{#CONFIG}]", "delay": "0", "valuemap": { "name": "Backup Status" }, "preprocessing": [ { "type": "JSONPATH", "parameters": [ "$[?(@.config == \"{#CONFIG}\")].status.first()" ] } ], "master_item": { "key": "vfs.file.contents[{$MON_FILE}]" }, "tags": [ { "tag": "Application", "value": "Borgmatic" } ] } ], "trigger_prototypes": [ { "uuid": "28e78881fa0e4754bd4d3ac32dc4be20", "expression": "now() - last(/BorgMatic/borgmatic.max_age[{#CONFIG}]) > last(/BorgMatic/borgmatic.last_success[{#CONFIG}])", "recovery_mode": "RECOVERY_EXPRESSION", "recovery_expression": "now() - last(/BorgMatic/borgmatic.max_age[{#CONFIG}]) < last(/BorgMatic/borgmatic.last_success[{#CONFIG}])", "name": "Old Backup [{#CONFIG}]", "priority": "HIGH", "description": "Last Successful Backup is to old", "manual_close": "YES" } ], "master_item": { "key": "vfs.file.contents[{$MON_FILE}]" }, "lld_macro_paths": [ { "lld_macro": "{#CONFIG}", "path": "$..config.first()" } ] } ], "macros": [ { "macro": "{$MON_FILE}", "value": "/var/log/backups/borgmatic.json" }, ], "valuemaps": [ { "uuid": "ac3d95e26ada42c68e7f6c1cc46fb6c8", "name": "Backup Status", "mappings": [ { "value": "0", "newvalue": "Success" }, { "value": "1", "newvalue": "Running" }, { "value": "2", "newvalue": "Failure" } ] } ] } ] } } ```

witten commented

2023-03-27 03:48:56 +00:00

Thanks for providing the additional detail, and I apologize for the delay in getting back to this. Here would be my recommendation for anyone looking to integrate a feature like this with borgmatic:

Create a new borgmatic monitoring hook called "json.py" or similar, modeled off one of the existing hooks like Cronitor, Healthchecks, ntfy, etc.
Instead of connecting to an external monitoring service when triggered, write monitoring information to a JSON file at a configured location, similar to the example code above.
borgmatic doesn't currently track any statistics on borgmatic create results other than success/failure, so some data gathering may need to be added in order to support some of the requested JSON fields. It's possible that an initial version could start simple with only a few fields, and then additional fields could be added over time as needed.

Related ticket: #385.

Thanks for providing the additional detail, and I apologize for the delay in getting back to this. Here would be my recommendation for anyone looking to integrate a feature like this with borgmatic: * Create a new borgmatic monitoring hook called "json.py" or similar, modeled off one of the existing hooks like Cronitor, Healthchecks, ntfy, etc. * Instead of connecting to an external monitoring service when triggered, write monitoring information to a JSON file at a configured location, similar to the example code above. * borgmatic doesn't currently track any statistics on `borgmatic create` results other than success/failure, so some data gathering may need to be added in order to support some of the requested JSON fields. It's possible that an initial version could start simple with only a few fields, and then additional fields could be added over time as needed. Related ticket: #385.

witten referenced this issue

2023-03-27 04:08:07 +00:00

Give access to results details in case of succeeded archive creation #385

diivi commented

2023-03-27 05:47:28 +00:00

I can take this, it might make it easier for me to test monitoring hooks too.
Should I also add additional data gathered from the output of the create command to other monitoring hooks?

I can take this, it might make it easier for me to test monitoring hooks too. Should I also add additional data gathered from the output of the `create` command to other monitoring hooks?

witten commented

2023-03-27 05:56:12 +00:00

If it makes sense, sure. Not all of the monitoring hooks have anywhere to slot in that data in a structured way though. I'd also be fine just starting with this proposed hook and worrying about the other monitoring hooks later.

👍 1

witten referenced this issue

2023-03-27 19:27:42 +00:00

Additional environment variables for hooks #436

witten added the

new feature area

label 2023-06-28 18:39:46 +00:00

Sign in to join this conversation.