borgmatic-collective/borgmatic
borgmatic-collective/borgmatic
12
62
Fork 49
Code Issues 49 Pull Requests 1 Actions Packages Releases 132 Activity

yes/NO prompt not shown when changing repo location, causing confusion. #914

New Issue
Closed
opened 2024-09-07 22:34:11 +00:00 by H3wastooshort · 6 comments
H3wastooshort commented 2024-09-07 22:34:11 +00:00
Copy Link

What I'm trying to do and why

I am changing the IP Address of a remote repo because the device it is hosted had its IP changed.

Steps to reproduce

  1. make a remote repo
  2. make a backup
  3. change remote repo's address
  4. try making another backup

Actual behavior

A line containing a yes/no prompt is not output until enter is pressed, hanging borgmatic and causing lots of confusion.

root@<source hostname>:/home/pi# borgmatic --verbosity 2 --list --stats
/etc/borgmatic/config.yaml: No commands to run for pre-everything hook
BORG_RSH=*** BORG_EXIT_CODES=*** borg --version --debug --show-rc
/etc/borgmatic/config.yaml: Borg 1.1.16
/etc/borgmatic/config.yaml: Calling ntfy hook function initialize_monitor
/etc/borgmatic/config.yaml: Calling ntfy hook function ping_monitor
<old address>: Running actions for repository
/etc/borgmatic/config.yaml: No commands to run for pre-actions hook
/etc/borgmatic/config.yaml: No commands to run for pre-backup hook
<old address>: Creating archive
ssh://<backup_user>@<new IP>//mnt/hdd/borgbackup/<source hostname>/<source hostname>.borg: Calling mariadb_databases hook function remove_data_source_dumps
ssh://<backup_user>@<new IP>//mnt/hdd/borgbackup/<source hostname>/<source hostname>.borg: Removing MariaDB data source dumps
ssh://<backup_user>@<new IP>//mnt/hdd/borgbackup/<source hostname>/<source hostname>.borg: Calling mysql_databases hook function remove_data_source_dumps
ssh://<backup_user>@<new IP>//mnt/hdd/borgbackup/<source hostname>/<source hostname>.borg: Removing MySQL data source dumps
ssh://<backup_user>@<new IP>//mnt/hdd/borgbackup/<source hostname>/<source hostname>.borg: Calling mongodb_databases hook function remove_data_source_dumps
ssh://<backup_user>@<new IP>//mnt/hdd/borgbackup/<source hostname>/<source hostname>.borg: Removing MongoDB data source dumps
ssh://<backup_user>@<new IP>//mnt/hdd/borgbackup/<source hostname>/<source hostname>.borg: Calling postgresql_databases hook function remove_data_source_dumps
ssh://<backup_user>@<new IP>//mnt/hdd/borgbackup/<source hostname>/<source hostname>.borg: Removing PostgreSQL data source dumps
ssh://<backup_user>@<new IP>//mnt/hdd/borgbackup/<source hostname>/<source hostname>.borg: Calling sqlite_databases hook function remove_data_source_dumps
ssh://<backup_user>@<new IP>//mnt/hdd/borgbackup/<source hostname>/<source hostname>.borg: Removing SQLite data source dumps
BORG_RSH=*** BORG_EXIT_CODES=*** borg create --exclude-from /tmp/tmpffjilqqv --list --filter AMEx- --stats --debug --show-rc ssh://<backup_user>@<new IP>//mnt/hdd/borgbackup/<source hostname>/<source hostname>.borg::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /etc/borgmatic/config.yaml /media/USBdrive/nc-data /media/USBdrive/onedrive /media/USBdrive/synced_files /root/.borgmatic
using builtin fallback logging configuration
35 self tests completed in 0.31 seconds
SSH command line: ['ssh', '-i', '/etc/borgmatic/ssh_id', '-c', 'aes128-ctr', '-m', 'umac-64-etm@openssh.com', '<backup_user>@<new IP>', 'borg', 'serve', '--umask=077', '--debug']
Remote: using builtin fallback logging configuration
Remote: 33 self tests completed in 2.37 seconds
Remote: using builtin fallback logging configuration
Remote: Initialized logging system for JSON-based protocol
Remote: Resolving repository path b'//mnt/hdd/borgbackup/<source hostname>/<source hostname>.borg'
Remote: Resolved repository path to '/mnt/hdd/borgbackup/<source hostname>/<source hostname>.borg'
Remote: Verified integrity of /mnt/hdd/borgbackup/<source hostname>/<source hostname>.borg/index.1523
TAM-verified manifest
security: read previous location 'ssh://<backup_user>@<old address>//mnt/hdd/borgbackup/<source hostname>/<source hostname>.borg'
Warning: The repository at location ssh://<backup_user>@<new IP>//mnt/hdd/borgbackup/<source hostname>/<source hostname>.borg was previously located at ssh://<backup_user>@<old address>//mnt/hdd/borgbackup/<source hostname>/<source hostname>.borg

Only upon pressing enter, this appears:


Do you want to continue? [yN] security: updating location stored in cache and security dir

which will then already have aborted becuse No is the default.

Expected behavior

The last line prompting a choice should have been output as well. Like:

TAM-verified manifest
security: read previous location 'ssh://<backup_user>@<old address>//mnt/hdd/borgbackup/<source hostname>/<source hostname>.borg'
Warning: The repository at location ssh://<backup_user>@<new IP>//mnt/hdd/borgbackup/<source hostname>/<source hostname>.borg was previously located at ssh://<backup_user>@<old address>//mnt/hdd/borgbackup/<source hostname>/<source hostname>.borg
Do you want to continue? [yN] security: updating location stored in cache and security dir

Other notes / implementation ideas

Workaround: press [y] then [Enter] upon seeing Warning: The repository at location <A> was previously located at <B>

borgmatic version

1.8.13

borgmatic installation method

pipx

Borg version

borg 1.1.16 on source, borg 1.2.4 on destination

Python version

Python 3.9.2

Database version (if applicable)

No response

Operating system and version

Debian GNU/Linux 11 (bullseye) on source, Raspbian GNU/Linux 12 (bookworm) on destination

### What I'm trying to do and why I am changing the IP Address of a remote repo because the device it is hosted had its IP changed. ### Steps to reproduce 1. make a remote repo 2. make a backup 3. change remote repo's address 4. try making another backup ### Actual behavior A line containing a yes/no prompt is not output until enter is pressed, hanging borgmatic and causing lots of confusion. ``` root@<source hostname>:/home/pi# borgmatic --verbosity 2 --list --stats /etc/borgmatic/config.yaml: No commands to run for pre-everything hook BORG_RSH=*** BORG_EXIT_CODES=*** borg --version --debug --show-rc /etc/borgmatic/config.yaml: Borg 1.1.16 /etc/borgmatic/config.yaml: Calling ntfy hook function initialize_monitor /etc/borgmatic/config.yaml: Calling ntfy hook function ping_monitor <old address>: Running actions for repository /etc/borgmatic/config.yaml: No commands to run for pre-actions hook /etc/borgmatic/config.yaml: No commands to run for pre-backup hook <old address>: Creating archive ssh://<backup_user>@<new IP>//mnt/hdd/borgbackup/<source hostname>/<source hostname>.borg: Calling mariadb_databases hook function remove_data_source_dumps ssh://<backup_user>@<new IP>//mnt/hdd/borgbackup/<source hostname>/<source hostname>.borg: Removing MariaDB data source dumps ssh://<backup_user>@<new IP>//mnt/hdd/borgbackup/<source hostname>/<source hostname>.borg: Calling mysql_databases hook function remove_data_source_dumps ssh://<backup_user>@<new IP>//mnt/hdd/borgbackup/<source hostname>/<source hostname>.borg: Removing MySQL data source dumps ssh://<backup_user>@<new IP>//mnt/hdd/borgbackup/<source hostname>/<source hostname>.borg: Calling mongodb_databases hook function remove_data_source_dumps ssh://<backup_user>@<new IP>//mnt/hdd/borgbackup/<source hostname>/<source hostname>.borg: Removing MongoDB data source dumps ssh://<backup_user>@<new IP>//mnt/hdd/borgbackup/<source hostname>/<source hostname>.borg: Calling postgresql_databases hook function remove_data_source_dumps ssh://<backup_user>@<new IP>//mnt/hdd/borgbackup/<source hostname>/<source hostname>.borg: Removing PostgreSQL data source dumps ssh://<backup_user>@<new IP>//mnt/hdd/borgbackup/<source hostname>/<source hostname>.borg: Calling sqlite_databases hook function remove_data_source_dumps ssh://<backup_user>@<new IP>//mnt/hdd/borgbackup/<source hostname>/<source hostname>.borg: Removing SQLite data source dumps BORG_RSH=*** BORG_EXIT_CODES=*** borg create --exclude-from /tmp/tmpffjilqqv --list --filter AMEx- --stats --debug --show-rc ssh://<backup_user>@<new IP>//mnt/hdd/borgbackup/<source hostname>/<source hostname>.borg::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /etc/borgmatic/config.yaml /media/USBdrive/nc-data /media/USBdrive/onedrive /media/USBdrive/synced_files /root/.borgmatic using builtin fallback logging configuration 35 self tests completed in 0.31 seconds SSH command line: ['ssh', '-i', '/etc/borgmatic/ssh_id', '-c', 'aes128-ctr', '-m', 'umac-64-etm@openssh.com', '<backup_user>@<new IP>', 'borg', 'serve', '--umask=077', '--debug'] Remote: using builtin fallback logging configuration Remote: 33 self tests completed in 2.37 seconds Remote: using builtin fallback logging configuration Remote: Initialized logging system for JSON-based protocol Remote: Resolving repository path b'//mnt/hdd/borgbackup/<source hostname>/<source hostname>.borg' Remote: Resolved repository path to '/mnt/hdd/borgbackup/<source hostname>/<source hostname>.borg' Remote: Verified integrity of /mnt/hdd/borgbackup/<source hostname>/<source hostname>.borg/index.1523 TAM-verified manifest security: read previous location 'ssh://<backup_user>@<old address>//mnt/hdd/borgbackup/<source hostname>/<source hostname>.borg' Warning: The repository at location ssh://<backup_user>@<new IP>//mnt/hdd/borgbackup/<source hostname>/<source hostname>.borg was previously located at ssh://<backup_user>@<old address>//mnt/hdd/borgbackup/<source hostname>/<source hostname>.borg ``` Only upon pressing enter, this appears: ``` Do you want to continue? [yN] security: updating location stored in cache and security dir ``` which will then already have aborted becuse No is the default. ### Expected behavior The last line prompting a choice should have been output as well. Like: ``` TAM-verified manifest security: read previous location 'ssh://<backup_user>@<old address>//mnt/hdd/borgbackup/<source hostname>/<source hostname>.borg' Warning: The repository at location ssh://<backup_user>@<new IP>//mnt/hdd/borgbackup/<source hostname>/<source hostname>.borg was previously located at ssh://<backup_user>@<old address>//mnt/hdd/borgbackup/<source hostname>/<source hostname>.borg Do you want to continue? [yN] security: updating location stored in cache and security dir ``` ### Other notes / implementation ideas Workaround: press [y] then [Enter] upon seeing `Warning: The repository at location <A> was previously located at <B>` ### borgmatic version 1.8.13 ### borgmatic installation method pipx ### Borg version borg 1.1.16 on source, borg 1.2.4 on destination ### Python version Python 3.9.2 ### Database version (if applicable) _No response_ ### Operating system and version Debian GNU/Linux 11 (bullseye) on source, Raspbian GNU/Linux 12 (bookworm) on destination
H3wastooshort changed title from Promp missing on rep location change, creating confusion. to yes/NO prompt not shown when changing repo location, causing confusion. 2024-09-07 22:37:43 +00:00
witten commented 2024-10-03 15:46:23 +00:00
Owner
Copy Link

Thanks for taking the time to report this. What's going on is that, rather than operating in an interactive mode that allows you to see Borg's prompt, borgmatic is consuming Borg's output on a line-by-line basis and logging it to the console. So any interactive prompts don't display because they don't contain newlines. (And even if they did, you wouldn't be able to see what you're typing because borgmatic is intermediating Borg's input/output rather than giving Borg direct access to the console.) borgmatic does in certain cases switch to interactive mode, e.g. when --progress is specified. But in that scenario, borgmatic can't do fancy logging, which is needed for colored output, multiple logging backends, etc.

Unfortunately that leaves this use case without a good solution unless you or I can think of another approach. In the meantime, what I recommend doing is to set the option relocated_repo_access_is_ok: true in your configuration file (either temporarily or permanently). That will allow Borg to bypass the interactive prompt and just deal with the fact that the repository has been moved.

Thanks for taking the time to report this. What's going on is that, rather than operating in an interactive mode that allows you to see Borg's prompt, borgmatic is consuming Borg's output on a line-by-line basis and logging it to the console. So any interactive prompts don't display because they don't contain newlines. (And even if they did, you wouldn't be able to see what you're typing because borgmatic is intermediating Borg's input/output rather than giving Borg direct access to the console.) borgmatic _does_ in certain cases switch to interactive mode, e.g. when `--progress` is specified. But in that scenario, borgmatic can't do fancy logging, which is needed for colored output, multiple logging backends, etc. Unfortunately that leaves this use case without a good solution unless you or I can think of another approach. In the meantime, what I recommend doing is to set the option `relocated_repo_access_is_ok: true` in your configuration file (either temporarily or permanently). That will allow Borg to bypass the interactive prompt and just deal with the fact that the repository has been moved.
witten added the
bug
 label 2024-10-03 23:46:59 +00:00
witten commented 2024-10-03 23:54:48 +00:00
Owner
Copy Link

Okay, I've actually got a "fix" implemented in main, which will be part of the next release. Now, when the repository location changes, borgmatic will instruct Borg to exit immediately instead of displaying an invisible prompt. Then borgmatic will display a hopefully helpful error message suggesting what to do next. (This message only displays when using Borg 1.4.0+.)

Here's what it looks like:

...
Warning: The repository at location /root/tmp/test2.borg was previously located at /root/tmp/test.borg
Do you want to continue? [yN] no (from BORG_RELOCATED_REPO_ACCESS_IS_OK)
...
terminating with error status, rc 62
Command '/root/borg1.4.0 create --exclude-from /tmp/tmp1rbtzzaq --keep-exclude-tags --exclude-nodump --list --filter AMEx- --debug --show-rc test2.borg::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /root/.borgmatic /root/tmp' returned non-zero exit status 62.

To work around this, set either the "relocated_repo_access_is_ok" or "unknown_unencrypted_repo_access_is_ok" option to "true", as appropriate.

Need some help? https://torsion.org/borgmatic/#issues

The "Do you want to continue?" is not an actual interactive prompt. It's just what Borg displays when defaulting to "no" via the BORG_RELOCATED_REPO_ACCESS_IS_OK environment variable—which borgmatic now sets to "no" when the "relocated_repo_access_is_ok" option is unset.

Anyway, feedback is welcome on this. But I'll close the ticket now as implemented! Thanks again.

Okay, I've actually got a "fix" implemented in main, which will be part of the next release. Now, when the repository location changes, borgmatic will instruct Borg to exit immediately instead of displaying an invisible prompt. Then borgmatic will display a hopefully helpful error message suggesting what to do next. (This message only displays when using Borg 1.4.0+.) Here's what it looks like: ``` ... Warning: The repository at location /root/tmp/test2.borg was previously located at /root/tmp/test.borg Do you want to continue? [yN] no (from BORG_RELOCATED_REPO_ACCESS_IS_OK) ... terminating with error status, rc 62 Command '/root/borg1.4.0 create --exclude-from /tmp/tmp1rbtzzaq --keep-exclude-tags --exclude-nodump --list --filter AMEx- --debug --show-rc test2.borg::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /root/.borgmatic /root/tmp' returned non-zero exit status 62. To work around this, set either the "relocated_repo_access_is_ok" or "unknown_unencrypted_repo_access_is_ok" option to "true", as appropriate. Need some help? https://torsion.org/borgmatic/#issues ``` The "Do you want to continue?" is not an actual interactive prompt. It's just what Borg displays when defaulting to "no" via the `BORG_RELOCATED_REPO_ACCESS_IS_OK` environment variable—which borgmatic now sets to "no" when the "relocated_repo_access_is_ok" option is unset. Anyway, feedback is welcome on this. But I'll close the ticket now as implemented! Thanks again.
👍 1
H3wastooshort commented 2024-10-05 13:46:13 +00:00
Author
Copy Link

this is way better than the invisible propmpt. thank you for taking the time :)

this is way better than the invisible propmpt. thank you for taking the time :)
witten commented 2024-10-05 15:42:10 +00:00
Owner
Copy Link

Sure thing!

Sure thing!
witten commented 2024-11-03 21:34:06 +00:00
Owner
Copy Link

Released in borgmatic 1.9.0!

Released in borgmatic 1.9.0!
👍 1
H3wastooshort commented 2024-11-03 21:37:05 +00:00
Author
Copy Link

thank you ^^

thank you ^^
👍 1
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
blocked
bug
data loss
design finalized
good first issue
new feature area
question / support
security
waiting for response
No Label
bug
Milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#914
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?