#100 Ability to call borg with `--stats` independent of `--list --filter AME`

Closed
opened 8 months ago by sdellenb · 10 comments

Before the logging-fix in 1.2.6, using -v 1 (logging.INFO) would only print the borg create stats, which was all the information I personally wanted to get from the nightly backup mails.

Are there any plans to control the borg --stats and --list flags separately in borgmatic?

Before the logging-fix in 1.2.6, using `-v 1` (logging.INFO) would only print the borg create stats, which was all the information I personally wanted to get from the nightly backup mails. Are there any plans to control the borg `--stats` and `--list` flags separately in borgmatic?
witten commented 8 months ago
Owner

I was indeed a little worried about the logging fixes changing the user experience for certain use cases. Looks like that’s happened here!

What exactly were you pulling out of the borg create stats previously? For instance, some of that information is currently available via borgmatic --info, but it’s more summary stats rather than per-archive stats. Put another way, what would you want to control with borg --stats and --list flags separately?

I was indeed a little worried about the logging fixes changing the user experience for certain use cases. Looks like that's happened here! What exactly were you pulling out of the `borg create` stats previously? For instance, some of that information is currently available via `borgmatic --info`, but it's more summary stats rather than per-archive stats. Put another way, what would you want to control with `borg` `--stats` and `--list` flags separately?
sdellenb commented 8 months ago
Poster

I’m running prune, create and check of the last 2 archives in the same borgmatic run. This is the output I’m interested in:

/etc/borgmatic/config.yaml: Parsing configuration file
/etc/borgmatic/config.yaml: Running command for pre-backup hook
2018-10-14 02:00:09 - Starting a backup job.
borgbackup@borgbackup:myserver: Pruning archives
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
Deleted data:               -3.81 GB             -2.39 GB            -54.74 MB
All archives:              195.83 GB            125.08 GB              8.60 GB

                       Unique chunks         Total chunks
Chunk index:                  212296              6807437
------------------------------------------------------------------------------
borgbackup@borgbackup:myserver: Creating archive
------------------------------------------------------------------------------
Archive name: myserver-2018-10-14T02:00:42.925104
Archive fingerprint: fd6c400c80e33f203be0ba50b8a9034d65c94c5d9ade6802c8b43faf9666ce59
Time (start): Sun, 2018-10-14 02:00:45
Time (end):   Sun, 2018-10-14 02:04:35
Duration: 3 minutes 50.18 seconds
Number of files: 149104
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:                3.90 GB              2.47 GB             42.85 MB
All archives:              199.74 GB            127.56 GB              8.64 GB

                       Unique chunks         Total chunks
Chunk index:                  213025              6944836
------------------------------------------------------------------------------
borgbackup@borgbackup:myserver: Running consistency checks
Starting archive consistency check...
Analyzing archive myserver-2018-10-13T02:00:58.561018 (1/2)
Analyzing archive myserver-2018-10-14T02:00:42.925104 (2/2)
Orphaned objects check skipped (needs all archives checked).
Archive consistency check complete, no problems found.
/etc/borgmatic/config.yaml: Running command for post-backup hook
2018-10-14 02:05:27 - Backup created.

When checking the nightly email reports, I mostly just glance at the “Deduplicated size” column of “This archive” to see whether the number is reasonable. I will try the --info flag.

I'm running prune, create and check of the last 2 archives in the same borgmatic run. This is the output I'm interested in: ``` /etc/borgmatic/config.yaml: Parsing configuration file /etc/borgmatic/config.yaml: Running command for pre-backup hook 2018-10-14 02:00:09 - Starting a backup job. borgbackup@borgbackup:myserver: Pruning archives ------------------------------------------------------------------------------ Original size Compressed size Deduplicated size Deleted data: -3.81 GB -2.39 GB -54.74 MB All archives: 195.83 GB 125.08 GB 8.60 GB Unique chunks Total chunks Chunk index: 212296 6807437 ------------------------------------------------------------------------------ borgbackup@borgbackup:myserver: Creating archive ------------------------------------------------------------------------------ Archive name: myserver-2018-10-14T02:00:42.925104 Archive fingerprint: fd6c400c80e33f203be0ba50b8a9034d65c94c5d9ade6802c8b43faf9666ce59 Time (start): Sun, 2018-10-14 02:00:45 Time (end): Sun, 2018-10-14 02:04:35 Duration: 3 minutes 50.18 seconds Number of files: 149104 Utilization of max. archive size: 0% ------------------------------------------------------------------------------ Original size Compressed size Deduplicated size This archive: 3.90 GB 2.47 GB 42.85 MB All archives: 199.74 GB 127.56 GB 8.64 GB Unique chunks Total chunks Chunk index: 213025 6944836 ------------------------------------------------------------------------------ borgbackup@borgbackup:myserver: Running consistency checks Starting archive consistency check... Analyzing archive myserver-2018-10-13T02:00:58.561018 (1/2) Analyzing archive myserver-2018-10-14T02:00:42.925104 (2/2) Orphaned objects check skipped (needs all archives checked). Archive consistency check complete, no problems found. /etc/borgmatic/config.yaml: Running command for post-backup hook 2018-10-14 02:05:27 - Backup created. ``` When checking the nightly email reports, I mostly just glance at the "Deduplicated size" column of "This archive" to see whether the number is reasonable. I will try the `--info` flag.
witten commented 8 months ago
Owner

Got it. Unfortunately --info by itself won’t give you the de-duplicated size of the current / most recent backup, because that output is more oriented towards the “all archives” stats. So, a couple of ideas to get what you’re looking for:

  1. Continue running borgmatic with -v 1, but just grep the output for the for the data you’re interested in. Example: borgmatic -v 1 ... 2>&1 | grep -C 2 size. Not the most elegant solution, but may work.
  2. Or, if you’re interested in consuming this data programmatically, then at least for --create, you could pass the --json argument once #97 is done. That would allow you to get at all the size information directly via JSON values.
  3. Or, now that -v 1 is more verbose than it used to be, maybe there’s a case to be made for also making -v 0 more verbose, and including (only) Borg summary output there.
Got it. Unfortunately `--info` by itself won't give you the de-duplicated size of the current / most recent backup, because that output is more oriented towards the "all archives" stats. So, a couple of ideas to get what you're looking for: 1. Continue running borgmatic with `-v 1`, but just grep the output for the for the data you're interested in. Example: `borgmatic -v 1 ... 2>&1 | grep -C 2 size`. Not the most elegant solution, but may work. 2. Or, if you're interested in consuming this data programmatically, then at least for `--create`, you could pass the `--json` argument once https://projects.torsion.org/witten/borgmatic/pulls/97 is done. That would allow you to get at all the size information directly via JSON values. 3. Or, now that `-v 1` is more verbose than it used to be, maybe there's a case to be made for also making `-v 0` more verbose, and including (only) Borg summary output there.
sdellenb commented 8 months ago
Poster

I currently pipe the output with -v 1 through grep -v -e "^[AME] .*" to filter out the file changes, which pretty much produces the same output as before. It is then mailed to me through postfix.

Personally, I vote for Option 3, but that might be too noisy for someone running silent backups.

I currently pipe the output with `-v 1` through `grep -v -e "^[AME] .*"` to filter out the file changes, which pretty much produces the same output as before. It is then mailed to me through postfix. Personally, I vote for Option 3, but that might be too noisy for someone running silent backups.
witten commented 7 months ago
Owner

Tangentially related discussion: #53.

Tangentially related discussion: #53.

I would also like something like option 3, the possibility to just print “--stats” for create and check.

I was also surprised by the change since I had previously been content with just the stats-output from each run. I am just launching it manually on my personal computers to backup to some external hard drives. The stats-output allowed me a quick overview of whether everything seemed alright or if there was a surprising increase in size.

I would also like something like option 3, the possibility to just print "--stats" for create and check. I was also surprised by the change since I had previously been content with just the stats-output from each run. I am just launching it manually on my personal computers to backup to some external hard drives. The stats-output allowed me a quick overview of whether everything seemed alright or if there was a surprising increase in size.

how about passing the --stats like --progress to borg?

for pruning and creating backups

how about passing the `--stats` like `--progress` to borg? for pruning and creating backups
witten commented 6 months ago
Owner

Continuing the discussion on this PR from none other than Stunkymonkey! #115

Continuing the discussion on this PR from none other than Stunkymonkey! https://projects.torsion.org/witten/borgmatic/pulls/115#issuecomment-1010
witten commented 5 months ago
Owner

Merged Stunkymonkey’s PR. Support for this will be part of the next release!

Merged Stunkymonkey's PR. Support for this will be part of the next release!
witten commented 5 months ago
Owner

Just released in borgmatic 1.2.13!

Just released in borgmatic 1.2.13!
Sign in to join this conversation.
No Milestone
No Assignees
4 Participants
Due Date

No due date set.

Dependencies

This issue currently doesn't have any dependencies.

Loading…
Cancel
Save
There is no content yet.