Ability to call borg with --stats independent of --list --filter AME #100

New Issue

Closed

opened 2018-10-10 14:40:41 +00:00 by sdellenb · 10 comments

sdellenb commented 2018-10-10 14:40:41 +00:00

Copy Link

Before the logging-fix in 1.2.6, using -v 1 (logging.INFO) would only print the borg create stats, which was all the information I personally wanted to get from the nightly backup mails.

Are there any plans to control the borg --stats and --list flags separately in borgmatic?

Before the logging-fix in 1.2.6, using `-v 1` (logging.INFO) would only print the borg create stats, which was all the information I personally wanted to get from the nightly backup mails. Are there any plans to control the borg `--stats` and `--list` flags separately in borgmatic?

witten commented 2018-10-10 15:23:02 +00:00

Owner

Copy Link

I was indeed a little worried about the logging fixes changing the user experience for certain use cases. Looks like that's happened here!

What exactly were you pulling out of the borg create stats previously? For instance, some of that information is currently available via borgmatic --info, but it's more summary stats rather than per-archive stats. Put another way, what would you want to control with borg --stats and --list flags separately?

I was indeed a little worried about the logging fixes changing the user experience for certain use cases. Looks like that's happened here! What exactly were you pulling out of the `borg create` stats previously? For instance, some of that information is currently available via `borgmatic --info`, but it's more summary stats rather than per-archive stats. Put another way, what would you want to control with `borg` `--stats` and `--list` flags separately?

sdellenb commented 2018-10-15 09:06:47 +00:00

Author

Copy Link

I'm running prune, create and check of the last 2 archives in the same borgmatic run.
This is the output I'm interested in:

/etc/borgmatic/config.yaml: Parsing configuration file
/etc/borgmatic/config.yaml: Running command for pre-backup hook
2018-10-14 02:00:09 - Starting a backup job.
borgbackup@borgbackup:myserver: Pruning archives
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
Deleted data:               -3.81 GB             -2.39 GB            -54.74 MB
All archives:              195.83 GB            125.08 GB              8.60 GB

                       Unique chunks         Total chunks
Chunk index:                  212296              6807437
------------------------------------------------------------------------------
borgbackup@borgbackup:myserver: Creating archive
------------------------------------------------------------------------------
Archive name: myserver-2018-10-14T02:00:42.925104
Archive fingerprint: fd6c400c80e33f203be0ba50b8a9034d65c94c5d9ade6802c8b43faf9666ce59
Time (start): Sun, 2018-10-14 02:00:45
Time (end):   Sun, 2018-10-14 02:04:35
Duration: 3 minutes 50.18 seconds
Number of files: 149104
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:                3.90 GB              2.47 GB             42.85 MB
All archives:              199.74 GB            127.56 GB              8.64 GB

                       Unique chunks         Total chunks
Chunk index:                  213025              6944836
------------------------------------------------------------------------------
borgbackup@borgbackup:myserver: Running consistency checks
Starting archive consistency check...
Analyzing archive myserver-2018-10-13T02:00:58.561018 (1/2)
Analyzing archive myserver-2018-10-14T02:00:42.925104 (2/2)
Orphaned objects check skipped (needs all archives checked).
Archive consistency check complete, no problems found.
/etc/borgmatic/config.yaml: Running command for post-backup hook
2018-10-14 02:05:27 - Backup created.

When checking the nightly email reports, I mostly just glance at the "Deduplicated size" column of "This archive" to see whether the number is reasonable.
I will try the --info flag.

I'm running prune, create and check of the last 2 archives in the same borgmatic run. This is the output I'm interested in: ``` /etc/borgmatic/config.yaml: Parsing configuration file /etc/borgmatic/config.yaml: Running command for pre-backup hook 2018-10-14 02:00:09 - Starting a backup job. borgbackup@borgbackup:myserver: Pruning archives ------------------------------------------------------------------------------ Original size Compressed size Deduplicated size Deleted data: -3.81 GB -2.39 GB -54.74 MB All archives: 195.83 GB 125.08 GB 8.60 GB Unique chunks Total chunks Chunk index: 212296 6807437 ------------------------------------------------------------------------------ borgbackup@borgbackup:myserver: Creating archive ------------------------------------------------------------------------------ Archive name: myserver-2018-10-14T02:00:42.925104 Archive fingerprint: fd6c400c80e33f203be0ba50b8a9034d65c94c5d9ade6802c8b43faf9666ce59 Time (start): Sun, 2018-10-14 02:00:45 Time (end): Sun, 2018-10-14 02:04:35 Duration: 3 minutes 50.18 seconds Number of files: 149104 Utilization of max. archive size: 0% ------------------------------------------------------------------------------ Original size Compressed size Deduplicated size This archive: 3.90 GB 2.47 GB 42.85 MB All archives: 199.74 GB 127.56 GB 8.64 GB Unique chunks Total chunks Chunk index: 213025 6944836 ------------------------------------------------------------------------------ borgbackup@borgbackup:myserver: Running consistency checks Starting archive consistency check... Analyzing archive myserver-2018-10-13T02:00:58.561018 (1/2) Analyzing archive myserver-2018-10-14T02:00:42.925104 (2/2) Orphaned objects check skipped (needs all archives checked). Archive consistency check complete, no problems found. /etc/borgmatic/config.yaml: Running command for post-backup hook 2018-10-14 02:05:27 - Backup created. ``` When checking the nightly email reports, I mostly just glance at the "Deduplicated size" column of "This archive" to see whether the number is reasonable. I will try the `--info` flag.

witten commented 2018-10-16 05:49:39 +00:00

Owner

Copy Link

Got it. Unfortunately --info by itself won't give you the de-duplicated size of the current / most recent backup, because that output is more oriented towards the "all archives" stats. So, a couple of ideas to get what you're looking for:

1. Continue running borgmatic with -v 1, but just grep the output for the for the data you're interested in. Example: borgmatic -v 1 ... 2>&1 | grep -C 2 size. Not the most elegant solution, but may work.
2. Or, if you're interested in consuming this data programmatically, then at least for --create, you could pass the --json argument once witten/borgmatic#97 is done. That would allow you to get at all the size information directly via JSON values.
3. Or, now that -v 1 is more verbose than it used to be, maybe there's a case to be made for also making -v 0 more verbose, and including (only) Borg summary output there.

Got it. Unfortunately `--info` by itself won't give you the de-duplicated size of the current / most recent backup, because that output is more oriented towards the "all archives" stats. So, a couple of ideas to get what you're looking for: 1. Continue running borgmatic with `-v 1`, but just grep the output for the for the data you're interested in. Example: `borgmatic -v 1 ... 2>&1 | grep -C 2 size`. Not the most elegant solution, but may work. 2. Or, if you're interested in consuming this data programmatically, then at least for `--create`, you could pass the `--json` argument once https://projects.torsion.org/witten/borgmatic/pulls/97 is done. That would allow you to get at all the size information directly via JSON values. 3. Or, now that `-v 1` is more verbose than it used to be, maybe there's a case to be made for also making `-v 0` more verbose, and including (only) Borg summary output there.

sdellenb commented 2018-10-16 12:16:41 +00:00

Author

Copy Link

I currently pipe the output with -v 1 through grep -v -e "^[AME] .*" to filter out the file changes, which pretty much produces the same output as before. It is then mailed to me through postfix.

Personally, I vote for Option 3, but that might be too noisy for someone running silent backups.

I currently pipe the output with `-v 1` through `grep -v -e "^[AME] .*"` to filter out the file changes, which pretty much produces the same output as before. It is then mailed to me through postfix. Personally, I vote for Option 3, but that might be too noisy for someone running silent backups.

witten commented 2018-10-28 18:38:07 +00:00

Owner

Copy Link

Tangentially related discussion: #53.

Tangentially related discussion: #53.

andersjohansson commented 2018-11-30 09:38:07 +00:00

Copy Link

I would also like something like option 3, the possibility to just print "--stats" for create and check.

I was also surprised by the change since I had previously been content with just the stats-output from each run. I am just launching it manually on my personal computers to backup to some external hard drives. The stats-output allowed me a quick overview of whether everything seemed alright or if there was a surprising increase in size.

I would also like something like option 3, the possibility to just print "--stats" for create and check. I was also surprised by the change since I had previously been content with just the stats-output from each run. I am just launching it manually on my personal computers to backup to some external hard drives. The stats-output allowed me a quick overview of whether everything seemed alright or if there was a surprising increase in size.

Stunkymonkey commented 2018-12-01 11:41:29 +00:00

Contributor

Copy Link

how about passing the --stats like --progress to borg?

for pruning and creating backups

how about passing the `--stats` like `--progress` to borg? for pruning and creating backups

witten commented 2018-12-07 06:00:49 +00:00

Owner

Copy Link

Continuing the discussion on this PR from none other than Stunkymonkey! witten/borgmatic#115 (comment)

Continuing the discussion on this PR from none other than Stunkymonkey! https://projects.torsion.org/witten/borgmatic/pulls/115#issuecomment-1010

witten commented 2018-12-24 00:10:48 +00:00

Owner

Copy Link

Merged Stunkymonkey's PR. Support for this will be part of the next release!

Merged Stunkymonkey's PR. Support for this will be part of the next release!

witten closed this issue 2018-12-24 00:10:48 +00:00

witten commented 2018-12-25 06:46:19 +00:00

Owner

Copy Link

Just released in borgmatic 1.2.13!

Just released in borgmatic 1.2.13!

👍 1

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

command-hook-steps

2.1.2

2.1.1

2.1.0

2.0.13

2.0.12

2.0.11

2.0.10

2.0.9

2.0.8

2.0.7

2.0.6

2.0.5

2.0.4

2.0.3

2.0.2

2.0.1

2.0.0

1.9.14

1.9.13

1.9.12

1.9.11

1.9.10

1.9.9

1.9.8

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.14

1.8.13

1.8.12

1.8.11

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.24

1.5.23

1.5.22

1.5.21

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.14

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.7.dev0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.22

1.4.21

1.4.20

1.4.19

1.4.18

1.4.17

1.4.16

1.4.15

1.4.14

1.4.13

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.26

1.3.25

1.3.24

1.3.23

1.3.22

1.3.21

1.3.20

1.3.19

1.3.18

1.3.17

1.3.16

1.3.15

1.3.14

1.3.13

1.3.12

1.3.11

1.3.10

1.3.9

1.3.8

1.3.7

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.18

1.2.17

1.2.16

1.2.15

1.2.14

1.2.13

1.2.12

1.2.11

1.2.10

1.2.9

1.2.8

1.2.7

1.2.6

1.2.5

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.15

1.1.14

1.1.13

1.1.12

1.1.11

1.1.10

1.1.9

1.1.7

1.1.6

1.1.5

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.3

1.0.2

1.0.1

1.0.0

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

0.0.2

Labels

Clear labels

blocked

breaking

bug

data loss

design finalized

good first issue

new feature area

question / support

security

waiting for response

No Label

Milestone

No items

No Milestone

Assignees

Clear assignees

witten

No Assignees

4 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#100