garbled --progress output #257

Closed
opened 2019-11-27 23:49:18 +00:00 by anarcat · 7 comments

What I'm trying to do and why

I'm trying to do an interactive backup to make sure my backups won't take too much space or pick up ridiculously large files.

Steps to reproduce (if a bug)

Setup this configuration in /etc/borgmatic/config.yaml, fairly
normal:

location:
    source_directories:
      - /
      - /boot
      - /var
      - /home
      - /srv

    repositories:
        - ssh://example.com/

    one_file_system: true

    # numeric_owner: true

    # Any paths matching these patterns are excluded from backups. Globs and tildes
    # are expanded. See the output of "borg help patterns" for more details.
    exclude_patterns:
     - /home/anarcat/.fuse
     - /home/*/.cache/
     - "*/.Trash-*/"
     - "*/.bitcoin/blocks/"
     - "*/build-area/*"
     - "/var/cache/*"
     - "/tmp/*"
     - "/var/tmp/*"
     - /srv/chroot

    # Exclude directories that contain a CACHEDIR.TAG file. See
    # http://www.brynosaurus.com/cachedir/spec.html for details. Defaults to false.
    exclude_caches: true

    # Exclude directories that contain a file with the given filename. Defaults to not
    # set.
    exclude_if_present: .nobackup

    # If true, the exclude_if_present filename is included in backups. Defaults to
    # false, meaning that the exclude_if_present filename is omitted from backups.
    keep_exclude_tags: true

storage:
    # Remote network upload rate limit in kiBytes/second. Defaults to unlimited.
    remote_rate_limit: 1000

retention:
    # Number of daily archives to keep.
    keep_daily: 7

    # Number of weekly archives to keep.
    keep_weekly: 4

    # Number of monthly archives to keep.
    keep_monthly: 6

    # Number of yearly archives to keep.
    keep_yearly: 10

Note that I censored the repository path and removed commented out configuration.

Start borgmatic with --progress.

Actual behavior (if a bug)

This is the output of the command:

root@angela:~# borgmatic  --syslog-verbosity 1 --progress
Creating archive at "ssh://example.com/foo::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}"
A /.autorelabel D 0 N .                                                                                                                                                                        
A /core O 248.31 kB C 248.31 kB D 1 N core                                                                                                                                                     
A /usr/lib/libmapnik.so.3.0.22 2 N usr/lib/libmapnik.so.3.0.22                                                                                                                                 
A /usr/lib/notification-daemon/notification-daemonation-daemon/notification-daemon                                                                                                             
A /usr/lib/libcomedi.so.0.11.0
A /usr/lib/libsolid.so.4.14.38
A /usr/lib/pkg-config.multiarch6 N usr/lib/pkg-config.multiarch                                                                                                                                
A /usr/lib/libGraphicsMagick-Q16.so.3.19.0/libGraphicsMagick-Q16.so.3.19.0                                                                                                                     
A /usr/lib/ledger/libledger.so.3 N usr/lib/ledger/libledger.so.3                                                                                                                               
A /usr/lib/libgjs.so.0.0.0MB D 9 N usr/lib/libgjs.so.0.0.0                                                                                                                                     
A /usr/lib/libkpty.so.4.14.38D 10 N usr/lib/libkpty.so.4.14.38                                                                                                                                 
A /usr/lib/gnupg/gpg-check-pattern
A /usr/lib/gnupg/dirmngr_ldapD 12 N usr/lib/gnupg/dirmngr_ldap                                                                                                                                 
A /usr/lib/gnupg/scdaemon
A /usr/lib/gnupg/gpg-wks-clientD 14 N usr/lib/gnupg/gpg-wks-client                                                                                                                             
A /usr/lib/gnupg/gpg-preset-passphraseusr/lib/gnupg/gpg-preset-passphrase                                                                                                                      
A /usr/lib/gnupg/gpg-protect-tool
26.69 MB O 11.24 MB C 11.24 MB D 17 N usr/lib/libgnustep-base.so.1.26.0

Note the A lines: those mask the "progress" output, visible at the
last line because i hit "enter" while running the backup so that borg
would show it again. I believe this might because borg is called with
--filter AME, a flag that seems impossible to turn off from the
user's perspective.

Expected behavior (if a bug)

I would have expected the following output:

root@angela:~# borgmatic  --syslog-verbosity 1 --progress
Creating archive at "ssh://example.com/foo::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}"
26.69 MB O 11.24 MB C 11.24 MB D 17 N usr/lib/libgnustep-base.so.1.26.0

... ie. with a single line where processed files are shown.

Other notes / implementation ideas

I wonder if this is one of the flags that could be (by default!)
specified as one of the "arbitrary flags passed to borg" (in #235) and
that could then be overriden in the configuration...

In other words, if there are magic flags that are forcibly passed to borg by borgmatic, I would like to have a more generic way to change/disable those...

Environment

borgmatic version: 1.4.8

borgmatic installation method: debian package

Borg version: 1.1.9

Python version: 3.7.3

Database version (if applicable): N/A

operating system and version: Debian 10 "buster", with some
packages from unstable (namely borgmatic).

#### What I'm trying to do and why I'm trying to do an interactive backup to make sure my backups won't take too much space or pick up ridiculously large files. #### Steps to reproduce (if a bug) Setup this configuration in `/etc/borgmatic/config.yaml`, fairly normal: location: source_directories: - / - /boot - /var - /home - /srv repositories: - ssh://example.com/ one_file_system: true # numeric_owner: true # Any paths matching these patterns are excluded from backups. Globs and tildes # are expanded. See the output of "borg help patterns" for more details. exclude_patterns: - /home/anarcat/.fuse - /home/*/.cache/ - "*/.Trash-*/" - "*/.bitcoin/blocks/" - "*/build-area/*" - "/var/cache/*" - "/tmp/*" - "/var/tmp/*" - /srv/chroot # Exclude directories that contain a CACHEDIR.TAG file. See # http://www.brynosaurus.com/cachedir/spec.html for details. Defaults to false. exclude_caches: true # Exclude directories that contain a file with the given filename. Defaults to not # set. exclude_if_present: .nobackup # If true, the exclude_if_present filename is included in backups. Defaults to # false, meaning that the exclude_if_present filename is omitted from backups. keep_exclude_tags: true storage: # Remote network upload rate limit in kiBytes/second. Defaults to unlimited. remote_rate_limit: 1000 retention: # Number of daily archives to keep. keep_daily: 7 # Number of weekly archives to keep. keep_weekly: 4 # Number of monthly archives to keep. keep_monthly: 6 # Number of yearly archives to keep. keep_yearly: 10 Note that I censored the repository path and removed commented out configuration. Start borgmatic with `--progress`. #### Actual behavior (if a bug) This is the output of the command: root@angela:~# borgmatic --syslog-verbosity 1 --progress Creating archive at "ssh://example.com/foo::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}" A /.autorelabel D 0 N . A /core O 248.31 kB C 248.31 kB D 1 N core A /usr/lib/libmapnik.so.3.0.22 2 N usr/lib/libmapnik.so.3.0.22 A /usr/lib/notification-daemon/notification-daemonation-daemon/notification-daemon A /usr/lib/libcomedi.so.0.11.0 A /usr/lib/libsolid.so.4.14.38 A /usr/lib/pkg-config.multiarch6 N usr/lib/pkg-config.multiarch A /usr/lib/libGraphicsMagick-Q16.so.3.19.0/libGraphicsMagick-Q16.so.3.19.0 A /usr/lib/ledger/libledger.so.3 N usr/lib/ledger/libledger.so.3 A /usr/lib/libgjs.so.0.0.0MB D 9 N usr/lib/libgjs.so.0.0.0 A /usr/lib/libkpty.so.4.14.38D 10 N usr/lib/libkpty.so.4.14.38 A /usr/lib/gnupg/gpg-check-pattern A /usr/lib/gnupg/dirmngr_ldapD 12 N usr/lib/gnupg/dirmngr_ldap A /usr/lib/gnupg/scdaemon A /usr/lib/gnupg/gpg-wks-clientD 14 N usr/lib/gnupg/gpg-wks-client A /usr/lib/gnupg/gpg-preset-passphraseusr/lib/gnupg/gpg-preset-passphrase A /usr/lib/gnupg/gpg-protect-tool 26.69 MB O 11.24 MB C 11.24 MB D 17 N usr/lib/libgnustep-base.so.1.26.0 Note the `A` lines: those mask the "progress" output, visible at the last line because i hit "enter" while running the backup so that borg would show it again. I believe this might because `borg` is called with `--filter AME`, a flag that seems impossible to turn off from the user's perspective. #### Expected behavior (if a bug) I would have expected the following output: root@angela:~# borgmatic --syslog-verbosity 1 --progress Creating archive at "ssh://example.com/foo::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}" 26.69 MB O 11.24 MB C 11.24 MB D 17 N usr/lib/libgnustep-base.so.1.26.0 ... ie. with a *single* line where processed files are shown. #### Other notes / implementation ideas I wonder if this is one of the flags that could be (by default!) specified as one of the "arbitrary flags passed to borg" (in #235) and that could then be overriden in the configuration... In other words, if there are magic flags that are forcibly passed to borg by borgmatic, I would like to have a more generic way to change/disable those... #### Environment **borgmatic version:** 1.4.8 **borgmatic installation method:** debian package **Borg version:** 1.1.9 **Python version:** 3.7.3 **Database version (if applicable):** N/A **operating system and version:** Debian 10 "buster", with some packages from unstable (namely borgmatic).
Author

oh and thanks so much for your work on this project, i really appreciate your responsiveness! :) i can start using borgmatic now that 1.4.4 (1.4.8 actually) hit the Debian mirrors...

oh and thanks so much for your work on this project, i really appreciate your responsiveness! :) i can start using borgmatic now that 1.4.4 (1.4.8 actually) hit the Debian mirrors...
Owner

Glad to hear the project is (generally) working out for you, and is now more up-to-date in Debian! Thanks for reporting the issue.. I'll try to see if I can reproduce it. Perhaps --filter AME and --progress are just incompatible from a user experience perspective?

Glad to hear the project is (generally) working out for you, and is now more up-to-date in Debian! Thanks for reporting the issue.. I'll try to see if I can reproduce it. Perhaps `--filter AME` and `--progress` are just incompatible from a user experience perspective?
witten added the
bug
label 2019-11-28 07:03:28 +00:00
Author

Perhaps --filter AME and --progress are just incompatible from a user experience perspective?

I would argue so, yes. Maybe it's just a matter of removing the --filter argument if --progress is provided?

> Perhaps --filter AME and --progress are just incompatible from a user experience perspective? I would argue so, yes. Maybe it's just a matter of removing the `--filter` argument if `--progress` is provided?
Owner

Yup, that would certainly be one option.. assuming that actually solves it! I'll need to play around with some of the Borg flags.

Yup, that would certainly be one option.. assuming that actually solves it! I'll need to play around with some of the Borg flags.
Owner

I was able to reproduce this, even running Borg directly without borgmtaic. Looks like this is an underlying Borg issue. Related: https://github.com/borgbackup/borg/issues/476

Leaving out both --list and --filter does work, but may in theory obscure some Borg file read errors because of the omission of 'E' = regular file, an error happened while accessing/reading this file. That's probably okay, given that it's not the default anyway.

I'm inclined to make the fix here to omit --list and --filter when --progress is given as you suggested.

I was able to reproduce this, even running Borg directly without borgmtaic. Looks like this is an underlying Borg issue. Related: https://github.com/borgbackup/borg/issues/476 Leaving out both `--list` and `--filter` does work, but may in theory obscure some Borg file read errors because of the omission of `'E' = regular file, an error happened while accessing/reading this file`. That's probably okay, given that it's not the default anyway. I'm inclined to make the fix here to omit `--list` and `--filter` when `--progress` is given as you suggested.
Owner

Fixed / worked around in borgmatic 1.4.16, just released. Thanks again for reporting this!

Fixed / worked around in borgmatic 1.4.16, just released. Thanks again for reporting this!
Author

awesome work, thanks!

awesome work, thanks!
Sign in to join this conversation.
No Milestone
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#257
No description provided.