running borgmatic borg check --repair doesn't show request to type "YES" to continue #694

New Issue

y8s · 2023-05-10T19:28:03Z

y8s commented

2023-05-10 19:28:03 +00:00

What I'm trying to do and why

Run borgmatic borg check --repair on a busted repo

Steps to reproduce (if a bug)

user@host:~ $ sudo borgmatic -c /etc/borgmatic.d/configuration.yaml borg check --repair

Actual behavior (if a bug)

user@host:~ $ sudo borgmatic -c -v 2 /etc/borgmatic.d/configuration.yaml borg check --repair

output looks like this:

Ensuring legacy configuration is upgraded
borg --version --debug --show-rc
ssh://bbrepoid@bbrepoid.repo.borgbase.com/./repo: Running actions for repository
/etc/borgmatic.d/bb-root.yaml: No commands to run for pre-actions hook
ssh://bbrepoid@bbrepoid.repo.borgbase.com/./repo: Running arbitrary Borg command
borg check ssh://bbrepoid@bbrepoid.repo.borgbase.com/./repo --repair --debug --show-rc
using builtin fallback logging configuration
35 self tests completed in 0.24 seconds
SSH command line: ['ssh', 'bbrepoid@bbrepoid.repo.borgbase.com', 'borg', 'serve', '--umask=077', '--debug']
Remote: using builtin fallback logging configuration
Remote: borg selftest disabled via BORG_SELFTEST env variable
Remote: using builtin fallback logging configuration
Remote: Initialized logging system for JSON-based protocol
Remote: Resolving repository path b'/./repo'
Remote: Resolved repository path to '/srv/repos/bbrepoid/repo'
This is a potentially dangerous function.
check --repair might lead to data loss (for kinds of corruption it is not
capable of dealing with). BE VERY CAREFUL!

command waits indefinitely at this point.

Expected behavior (if a bug)

Ensuring legacy configuration is upgraded
borg --version --debug --show-rc
ssh://bbrepoid@bbrepoid.repo.borgbase.com/./repo: Running actions for repository
/etc/borgmatic.d/bb-root.yaml: No commands to run for pre-actions hook
ssh://bbrepoid@bbrepoid.repo.borgbase.com/./repo: Running arbitrary Borg command
borg check ssh://bbrepoid@bbrepoid.repo.borgbase.com/./repo --repair --debug --show-rc
using builtin fallback logging configuration
35 self tests completed in 0.27 seconds
SSH command line: ['ssh', 'bbrepoid@bbrepoid.repo.borgbase.com', 'borg', 'serve', '--umask=077', '--debug']
Remote: using builtin fallback logging configuration
Remote: borg selftest disabled via BORG_SELFTEST env variable
Remote: using builtin fallback logging configuration
Remote: Initialized logging system for JSON-based protocol
Remote: Resolving repository path b'/./repo'
Remote: Resolved repository path to '/srv/repos/bbrepoid/repo'
This is a potentially dangerous function.
check --repair might lead to data loss (for kinds of corruption it is not
capable of dealing with). BE VERY CAREFUL!
Type 'YES' if you understand this and want to continue: Remote: Starting repository check

command waits for user input

YES 

Remote: Verified integrity of /srv/repos/bbrepoid/repo/index.7948
Remote: Read committed index of transaction 7948
Remote: Cleaned up 0 uncommitted segment files (== everything after segment 7948).
Remote: Segment transaction is    7948
Remote: Determined transaction is 7948
Remote: Found 179 segments
Remote: checking segment file /srv/repos/bbrepoid/repo/data/0/25...
Remote: checking segment file /srv/repos/bbrepoid/repo/data/5/5792...
Remote: checking segment file /srv/repos/bbrepoid/repo/data/5/5794...
(etc)

Other notes / implementation ideas

If I type "YES" after the warning, it works and then I see the line asking me to confirm with YES and the command continues/completes. See the attached image for YES above the request for YES.

Environment

borgmatic version: [1.7.12]

borgmatic installation method: [sudo pip install --upgrade borgmatic]

Borg version: [1.2.4 and tested 1.1.16]

Python version: [3.9.2]

Database version (if applicable): [N/A]

operating system and version: [proxmox 7.4-3 / Debian 11 bullseye]

PRETTY_NAME="LMDE 5 (elsie)"
NAME="LMDE"
VERSION_ID="5"
VERSION="5 (elsie)"
ID=linuxmint
ID_LIKE=debian
HOME_URL="https://www.linuxmint.com/"
SUPPORT_URL="https://forums.linuxmint.com/"
BUG_REPORT_URL="http://linuxmint-troubleshooting-guide.readthedocs.io/en/latest/"
PRIVACY_POLICY_URL="https://www.linuxmint.com/"
#VERSION_CODENAME=elsie
VERSION_CODENAME=bullseye
DEBIAN_CODENAME=bullseye

#### What I'm trying to do and why Run borgmatic borg check --repair on a busted repo #### Steps to reproduce (if a bug) ``` user@host:~ $ sudo borgmatic -c /etc/borgmatic.d/configuration.yaml borg check --repair ``` #### Actual behavior (if a bug) ``` user@host:~ $ sudo borgmatic -c -v 2 /etc/borgmatic.d/configuration.yaml borg check --repair ``` output looks like this: ``` Ensuring legacy configuration is upgraded borg --version --debug --show-rc ssh://bbrepoid@bbrepoid.repo.borgbase.com/./repo: Running actions for repository /etc/borgmatic.d/bb-root.yaml: No commands to run for pre-actions hook ssh://bbrepoid@bbrepoid.repo.borgbase.com/./repo: Running arbitrary Borg command borg check ssh://bbrepoid@bbrepoid.repo.borgbase.com/./repo --repair --debug --show-rc using builtin fallback logging configuration 35 self tests completed in 0.24 seconds SSH command line: ['ssh', 'bbrepoid@bbrepoid.repo.borgbase.com', 'borg', 'serve', '--umask=077', '--debug'] Remote: using builtin fallback logging configuration Remote: borg selftest disabled via BORG_SELFTEST env variable Remote: using builtin fallback logging configuration Remote: Initialized logging system for JSON-based protocol Remote: Resolving repository path b'/./repo' Remote: Resolved repository path to '/srv/repos/bbrepoid/repo' This is a potentially dangerous function. check --repair might lead to data loss (for kinds of corruption it is not capable of dealing with). BE VERY CAREFUL! ``` command waits indefinitely at this point. #### Expected behavior (if a bug) ``` Ensuring legacy configuration is upgraded borg --version --debug --show-rc ssh://bbrepoid@bbrepoid.repo.borgbase.com/./repo: Running actions for repository /etc/borgmatic.d/bb-root.yaml: No commands to run for pre-actions hook ssh://bbrepoid@bbrepoid.repo.borgbase.com/./repo: Running arbitrary Borg command borg check ssh://bbrepoid@bbrepoid.repo.borgbase.com/./repo --repair --debug --show-rc using builtin fallback logging configuration 35 self tests completed in 0.27 seconds SSH command line: ['ssh', 'bbrepoid@bbrepoid.repo.borgbase.com', 'borg', 'serve', '--umask=077', '--debug'] Remote: using builtin fallback logging configuration Remote: borg selftest disabled via BORG_SELFTEST env variable Remote: using builtin fallback logging configuration Remote: Initialized logging system for JSON-based protocol Remote: Resolving repository path b'/./repo' Remote: Resolved repository path to '/srv/repos/bbrepoid/repo' This is a potentially dangerous function. check --repair might lead to data loss (for kinds of corruption it is not capable of dealing with). BE VERY CAREFUL! Type 'YES' if you understand this and want to continue: Remote: Starting repository check ``` command waits for user input ``` YES Remote: Verified integrity of /srv/repos/bbrepoid/repo/index.7948 Remote: Read committed index of transaction 7948 Remote: Cleaned up 0 uncommitted segment files (== everything after segment 7948). Remote: Segment transaction is 7948 Remote: Determined transaction is 7948 Remote: Found 179 segments Remote: checking segment file /srv/repos/bbrepoid/repo/data/0/25... Remote: checking segment file /srv/repos/bbrepoid/repo/data/5/5792... Remote: checking segment file /srv/repos/bbrepoid/repo/data/5/5794... (etc) ``` #### Other notes / implementation ideas If I type "YES" after the warning, it works and then I see the line asking me to confirm with YES and the command continues/completes. See the attached image for YES above the request for YES. #### Environment **borgmatic version:** [1.7.12] **borgmatic installation method:** [sudo pip install --upgrade borgmatic] **Borg version:** [1.2.4 and tested 1.1.16] **Python version:** [3.9.2] **Database version (if applicable):** [N/A] **operating system and version:** [proxmox 7.4-3 / Debian 11 bullseye] ``` PRETTY_NAME="LMDE 5 (elsie)" NAME="LMDE" VERSION_ID="5" VERSION="5 (elsie)" ID=linuxmint ID_LIKE=debian HOME_URL="https://www.linuxmint.com/" SUPPORT_URL="https://forums.linuxmint.com/" BUG_REPORT_URL="http://linuxmint-troubleshooting-guide.readthedocs.io/en/latest/" PRIVACY_POLICY_URL="https://www.linuxmint.com/" #VERSION_CODENAME=elsie VERSION_CODENAME=bullseye DEBIAN_CODENAME=bullseye ```

borgnonsense.png

2.2 MiB

witten commented

2023-05-11 00:00:54 +00:00

I appreciate you taking the time to file this one. So far I haven't been able to reproduce it locally. What's supposed to happen (and what I'm seeing) is that giving --repair or --progress flags should switch borgmatic out of logging mode to interactive mode so that the Borg prompt actually shows up properly. But given that your Borg output text is purple instead of gray/white, that means that switch isn't happening.

So some ideas on things to try:

In the same terminal where you run borgmatic, try running:

python -c "import sys; print(f'terminal is a TTY: {sys.stderr.isatty()}')"

... to see if the interactive terminal detection is broken. The output should show True if it's working.

You can also try running echo $TERM which should show anything other than dumb. That also impacts borgmatic's interactive terminal detection.
What terminal are you running? Maybe try a different terminal to see if that makes a difference?
Does switching to the root user entirely (with either su - or sudo su -) and then running your full borgmatic check command change anything?

Thanks!

I appreciate you taking the time to file this one. So far I haven't been able to reproduce it locally. What's supposed to happen (and what I'm seeing) is that giving `--repair` or `--progress` flags should switch borgmatic out of logging mode to interactive mode so that the Borg prompt actually shows up properly. But given that your Borg output text is purple instead of gray/white, that means that switch isn't happening. So some ideas on things to try: * In the same terminal where you run borgmatic, try running: ```bash python -c "import sys; print(f'terminal is a TTY: {sys.stderr.isatty()}')" ``` ... to see if the interactive terminal detection is broken. The output should show `True` if it's working. * You can also try running `echo $TERM` which should show anything other than `dumb`. That also impacts borgmatic's interactive terminal detection. * What terminal are you running? Maybe try a different terminal to see if that makes a difference? * Does switching to the root user entirely (with either `su -` or `sudo su -`) and then running your full `borgmatic check` command change anything? Thanks!

y8s commented

2023-05-11 13:37:35 +00:00

Here are the results of the tests:

python command:

user@host:~ $ python3 -c "import sys; print(f'terminal is a TTY: {sys.stderr.isatty()}')"
terminal is a TTY: True

echo $TERM

user@host:~ $ echo $TERM
xterm-256color

terminal

GNOME Terminal 3.38.3 (Cinnamon 5.6.8)

switching to super user

root@host:~ # borgmatic --dry-run -v 2 -c /etc/borgmatic.d/bb-root.yaml borg check --repair
Ensuring legacy configuration is upgraded
borg --version --debug --show-rc
ssh://bbrepoid@bbrepoid.repo.borgbase.com/./repo: Running actions for repository
/etc/borgmatic.d/bb-root.yaml: No commands to run for pre-actions hook
ssh://bbrepoid@bbrepoid.repo.borgbase.com/./repo: Running arbitrary Borg command
borg check ssh://bbrepoid@bbrepoid.repo.borgbase.com/./repo --repair --debug --show-rc
using builtin fallback logging configuration
33 self tests completed in 0.26 seconds
SSH command line: ['ssh', 'bbrepoid@bbrepoid.repo.borgbase.com', 'borg', 'serve', '--debug']
Remote: using builtin fallback logging configuration
Remote: borg selftest disabled via BORG_SELFTEST env variable
Remote: using builtin fallback logging configuration
Remote: Initialized logging system for JSON-based protocol
Remote: Resolving repository path b'/./repo'
Remote: Resolved repository path to '/srv/repos/bbrepoid/repo'
This is a potentially dangerous function.
check --repair might lead to data loss (for kinds of corruption it is not
capable of dealing with). BE VERY CAREFUL!
YES
Type 'YES' if you understand this and want to continue: Remote: Starting repository check
Remote: Verified integrity of /srv/repos/bbrepoid/repo/index.7948
Remote: Read committed index of transaction 7948
Remote: Cleaned up 0 uncommitted segment files (== everything after segment 7948).
Remote: Segment transaction is    7948
Remote: Determined transaction is 7948
Remote: Found 179 segments
Remote: checking segment file /srv/repos/bbrepoid/repo/data/0/25...

(same thing)

Happy to help!

Here are the results of the tests: ### python command: ```bash user@host:~ $ python3 -c "import sys; print(f'terminal is a TTY: {sys.stderr.isatty()}')" terminal is a TTY: True ``` ### echo $TERM ```bash user@host:~ $ echo $TERM xterm-256color ``` ### terminal GNOME Terminal 3.38.3 (Cinnamon 5.6.8) ### switching to super user ```bash root@host:~ # borgmatic --dry-run -v 2 -c /etc/borgmatic.d/bb-root.yaml borg check --repair Ensuring legacy configuration is upgraded borg --version --debug --show-rc ssh://bbrepoid@bbrepoid.repo.borgbase.com/./repo: Running actions for repository /etc/borgmatic.d/bb-root.yaml: No commands to run for pre-actions hook ssh://bbrepoid@bbrepoid.repo.borgbase.com/./repo: Running arbitrary Borg command borg check ssh://bbrepoid@bbrepoid.repo.borgbase.com/./repo --repair --debug --show-rc using builtin fallback logging configuration 33 self tests completed in 0.26 seconds SSH command line: ['ssh', 'bbrepoid@bbrepoid.repo.borgbase.com', 'borg', 'serve', '--debug'] Remote: using builtin fallback logging configuration Remote: borg selftest disabled via BORG_SELFTEST env variable Remote: using builtin fallback logging configuration Remote: Initialized logging system for JSON-based protocol Remote: Resolving repository path b'/./repo' Remote: Resolved repository path to '/srv/repos/bbrepoid/repo' This is a potentially dangerous function. check --repair might lead to data loss (for kinds of corruption it is not capable of dealing with). BE VERY CAREFUL! YES Type 'YES' if you understand this and want to continue: Remote: Starting repository check Remote: Verified integrity of /srv/repos/bbrepoid/repo/index.7948 Remote: Read committed index of transaction 7948 Remote: Cleaned up 0 uncommitted segment files (== everything after segment 7948). Remote: Segment transaction is 7948 Remote: Determined transaction is 7948 Remote: Found 179 segments Remote: checking segment file /srv/repos/bbrepoid/repo/data/0/25... ``` (same thing) Happy to help!

witten commented

2023-05-12 23:44:54 +00:00

Thanks for trying these out. It looks like borgmatic is correctly detecting the interactive console, and so that's not the source of the issue. I tried running this locally with GNOME Terminal and a remote repo; still no luck reproducing.

Do you have the interest and ability to try temporarily installing some modified versions of borgmatic source files with additional debug logging on your machine? There is a particular section of logic in the check action I'd be interested in debugging, because I don't see how it can not work—but it's clearly not working on your system.

Either way, please let me know and I can get you some modified source files.

Thanks for trying these out. It looks like borgmatic _is_ correctly detecting the interactive console, and so that's not the source of the issue. I tried running this locally with GNOME Terminal and a remote repo; still no luck reproducing. Do you have the interest and ability to try temporarily installing some modified versions of borgmatic source files with additional debug logging on your machine? There is a particular section of logic in the check action I'd be interested in debugging, because I don't see how it can not work—but it's clearly not working on your system. Either way, please let me know and I can get you some modified source files.

y8s commented

2023-05-13 00:55:23 +00:00

Is there a way to do it without breaking my current install? I use the pre built binary and copy it to /usr/bin which is pretty easy.

I just tested on my other proxmox install using borgmatic 1.6.3 and borg 1.1.14 and it behaves similarly but the text is different.

The latter is pretty vanilla proxmox.

This is a potentially dangerous function.
YES
check --repair might lead to data loss (for kinds of corruption it is not
capable of dealing with). BE VERY CAREFUL!
Type 'YES' if you understand this and want to continue:

edit: this is using juice ssh on my phone. $TERM is linux.

Is there a way to do it without breaking my current install? I use the pre built binary and copy it to `/usr/bin` which is pretty easy. I just tested on my other proxmox install using borgmatic 1.6.3 and borg 1.1.14 and it behaves similarly but the text is different. The latter is pretty vanilla proxmox. ```bash This is a potentially dangerous function. YES check --repair might lead to data loss (for kinds of corruption it is not capable of dealing with). BE VERY CAREFUL! Type 'YES' if you understand this and want to continue: ``` edit: this is using juice ssh on my phone. $TERM is linux.

witten commented

2023-05-13 00:59:50 +00:00

Is there a way to do it without breaking my current install?

If the only changes are the addition of more debugging logs, then it shouldn't break your install. "Shouldn't" being the operative word.

I use the pre built binary and copy it to /usr/bin which is pretty easy.

Are you talking about Borg's pre-built binary? The modifications I'm talking about would only be to borgmatic.. not Borg. So presumably that would only require dropping in some source files to replace the existing ones. And then you could always blow them away via pip install --upgrade borgmatic.

> Is there a way to do it without breaking my current install? If the only changes are the addition of more debugging logs, then it *shouldn't* break your install. "Shouldn't" being the operative word. > I use the pre built binary and copy it to /usr/bin which is pretty easy. Are you talking about Borg's pre-built binary? The modifications I'm talking about would only be to borgmatic.. not Borg. So presumably that would only require dropping in some source files to replace the existing ones. And then you could always blow them away via `pip install --upgrade borgmatic`.

y8s commented

2023-05-13 01:03:14 +00:00

Is there a way to do it without breaking my current install?

If the only changes are the addition of more debugging logs, then it shouldn't break your install. "Shouldn't" being the operative word.

I use the pre built binary and copy it to /usr/bin which is pretty easy.

Are you talking about Borg's pre-built binary? The modifications I'm talking about would only be to borgmatic.. not Borg. So presumably that would only require dropping in some source files to replace the existing ones. And then you could always blow them away via pip install --upgrade borgmatic.

oh right yes nevermind thats borg.

Does global (sudo without --user) install matter?

> > Is there a way to do it without breaking my current install? > > If the only changes are the addition of more debugging logs, then it *shouldn't* break your install. "Shouldn't" being the operative word. > > > I use the pre built binary and copy it to /usr/bin which is pretty easy. > > Are you talking about Borg's pre-built binary? The modifications I'm talking about would only be to borgmatic.. not Borg. So presumably that would only require dropping in some source files to replace the existing ones. And then you could always blow them away via `pip install --upgrade borgmatic`. oh right yes nevermind thats borg. Does global (sudo without --user) install matter?

witten commented

2023-05-13 05:49:16 +00:00

Nope, shouldn't matter. It'll just mean your source file are in a different place. pip show borgmatic should show the path under Location:

Nope, shouldn't matter. It'll just mean your source file are in a different place. `pip show borgmatic` should show the path under `Location:`

y8s commented

2023-05-13 11:25:48 +00:00

sure send over the modified source files. ill give them a shot.

witten commented

2023-05-13 17:31:41 +00:00

Okay, the two modified files are attached. In terms of where to put these on your system: Let's say that the output of pip show borgmatic | grep ^Location is:

Location: /usr/lib/python3.10/site-packages

Then that probably means execute.py should get copied over /usr/lib/python3.10/site-packages/borgmatic/execute.py or similar and check.py should get copied over /usr/lib/python3.10/site-packages/borgmatic/borg/check.py or similar. Make sure you do not copy it over actions/check.py—that's a different file.

After installing these modified source files, running borgmatic again should produce the usual output but with the addition of a bunch of new "#694: ..." log lines. Seeing your entire output including those lines would be most helpful.

Thanks for your patience here!

Okay, the two modified files are attached. In terms of where to put these on your system: Let's say that the output of `pip show borgmatic | grep ^Location` is: ``` Location: /usr/lib/python3.10/site-packages ``` Then that probably means `execute.py` should get copied over `/usr/lib/python3.10/site-packages/borgmatic/execute.py` or similar and `check.py` should get copied over `/usr/lib/python3.10/site-packages/borgmatic/borg/check.py` or similar. Make sure you do *not* copy it over `actions/check.py`—that's a different file. After installing these modified source files, running borgmatic again should produce the usual output but with the addition of a bunch of new "#694: ..." log lines. Seeing your entire output including those lines would be most helpful. Thanks for your patience here!

execute.py

15 KiB

check.py

12 KiB

y8s commented

2023-05-13 19:06:37 +00:00

command executed and waiting for input presumably:

matt@pve:~ $ sudo borgmatic --dry-run -v 2 -c /etc/borgmatic.d/mybk-root.yaml borg check --repair
Ensuring legacy configuration is upgraded
borg --version --debug --show-rc
/usbdrive/backup-ext/borgroot: Running actions for repository
/etc/borgmatic.d/mybk-root.yaml: No commands to run for pre-actions hook
/usbdrive/backup-ext/borgroot: Running arbitrary Borg command
#694: execute_command()
#694: output_log_level: 25
#694: output_file: None
borg check /usbdrive/backup-ext/borgroot --repair --debug --show-rc
#694: do_not_capture: False
#694: stdout: -1
#694: stderr: -2
#694: run_to_completion: True
#694: log_outputs() output_log_level: 25
#694: process_for_output_buffer: {<_io.BufferedReader name=3>: <Popen: returncode: None args: ['borg', 'check', '/usbdrive/backup-ext/borgr...>}
#694: output_buffers: [<_io.BufferedReader name=3>]
#694: append_last_lines() output_log_level: 25
using builtin fallback logging configuration
#694: append_last_lines() output_log_level: 25
33 self tests completed in 0.13 seconds
#694: append_last_lines() output_log_level: 25
This is a potentially dangerous function.
#694: append_last_lines() output_log_level: 25
check --repair might lead to data loss (for kinds of corruption it is not
#694: append_last_lines() output_log_level: 25
capable of dealing with). BE VERY CAREFUL!
#694: still_running: False
#694: exit_code_indicates_error(): False
#694: output_buffers: [<_io.BufferedReader name=3>]

I type "YES" and hit enter:

YES
#694: append_last_lines() output_log_level: 25
Type 'YES' if you understand this and want to continue: Starting repository check
#694: append_last_lines() output_log_level: 25
Verified integrity of /usbdrive/backup-ext/borgroot/index.9435
#694: append_last_lines() output_log_level: 25
Read committed index of transaction 9435
#694: append_last_lines() output_log_level: 25
Cleaned up 0 uncommitted segment files (== everything after segment 9435).
#694: append_last_lines() output_log_level: 25
Segment transaction is    9435
#694: append_last_lines() output_log_level: 25
Determined transaction is 9435
#694: append_last_lines() output_log_level: 25
Found 3467 segments
#694: append_last_lines() output_log_level: 25
checking segment file /usbdrive/backup-ext/borgroot/data/0/1...
#694: append_last_lines() output_log_level: 25
checking segment file /usbdrive/backup-ext/borgroot/data/0/24...

and canceled.

command executed and waiting for input presumably: ```bash matt@pve:~ $ sudo borgmatic --dry-run -v 2 -c /etc/borgmatic.d/mybk-root.yaml borg check --repair Ensuring legacy configuration is upgraded borg --version --debug --show-rc /usbdrive/backup-ext/borgroot: Running actions for repository /etc/borgmatic.d/mybk-root.yaml: No commands to run for pre-actions hook /usbdrive/backup-ext/borgroot: Running arbitrary Borg command #694: execute_command() #694: output_log_level: 25 #694: output_file: None borg check /usbdrive/backup-ext/borgroot --repair --debug --show-rc #694: do_not_capture: False #694: stdout: -1 #694: stderr: -2 #694: run_to_completion: True #694: log_outputs() output_log_level: 25 #694: process_for_output_buffer: {<_io.BufferedReader name=3>: <Popen: returncode: None args: ['borg', 'check', '/usbdrive/backup-ext/borgr...>} #694: output_buffers: [<_io.BufferedReader name=3>] #694: append_last_lines() output_log_level: 25 using builtin fallback logging configuration #694: append_last_lines() output_log_level: 25 33 self tests completed in 0.13 seconds #694: append_last_lines() output_log_level: 25 This is a potentially dangerous function. #694: append_last_lines() output_log_level: 25 check --repair might lead to data loss (for kinds of corruption it is not #694: append_last_lines() output_log_level: 25 capable of dealing with). BE VERY CAREFUL! #694: still_running: False #694: exit_code_indicates_error(): False #694: output_buffers: [<_io.BufferedReader name=3>] ``` I type "YES" and hit enter: ```bash YES #694: append_last_lines() output_log_level: 25 Type 'YES' if you understand this and want to continue: Starting repository check #694: append_last_lines() output_log_level: 25 Verified integrity of /usbdrive/backup-ext/borgroot/index.9435 #694: append_last_lines() output_log_level: 25 Read committed index of transaction 9435 #694: append_last_lines() output_log_level: 25 Cleaned up 0 uncommitted segment files (== everything after segment 9435). #694: append_last_lines() output_log_level: 25 Segment transaction is 9435 #694: append_last_lines() output_log_level: 25 Determined transaction is 9435 #694: append_last_lines() output_log_level: 25 Found 3467 segments #694: append_last_lines() output_log_level: 25 checking segment file /usbdrive/backup-ext/borgroot/data/0/1... #694: append_last_lines() output_log_level: 25 checking segment file /usbdrive/backup-ext/borgroot/data/0/24... ``` and canceled.

witten commented

2023-05-13 20:00:19 +00:00

Thanks! However, it looks like check.py might not be in the right spot, because I'm not seeing any of its debugging output. Are you sure you put it into borgmatic/borg/check.py, overwriting the existing file there?

Also, could I see the consistency section of your borgmatic configuration file?

Thanks! However, it looks like `check.py` might not be in the right spot, because I'm not seeing any of its debugging output. Are you sure you put it into `borgmatic/borg/check.py`, overwriting the existing file there? Also, could I see the `consistency` section of your borgmatic configuration file?

y8s commented

2023-05-14 03:10:48 +00:00

recopied the two files and tried again. note im using python 3.9 not 3.10 in your example above.

and sorry about weird line wraps. i did this on a phone.


matt@pve:~ $ sudo borgmatic --dry-run -v 2 -c /etc/borgmatic.d/mybk-root.yaml borg check --repair
Ensuring legacy configuration is upgraded       borg --version --debug --show-rc
/usbdrive/backup-ext/borgroot: Running actions for repository
/etc/borgmatic.d/mybk-root.yaml: No commands to run for pre-actions hook
/usbdrive/backup-ext/borgroot: Running arbitrary Borg command
#694: execute_command()
#694: output_log_level: 25                      #694: output_file: None
borg check /usbdrive/backup-ext/borgroot --repair --debug --show-rc
#694: do_not_capture: False
#694: stdout: -1
#694: stderr: -2
#694: run_to_completion: True
#694: log_outputs() output_log_level: 25
#694: process_for_output_buffer: {<_io.BufferedReader name=3>: <Popen: returncode: None args: ['borg', 'check', '/usbdrive/backup-ext/borgr...>}#694: output_buffers: [<_io.BufferedReader name=3>]
#694: append_last_lines() output_log_level: 25
using builtin fallback logging configuration
#694: append_last_lines() output_log_level: 25
33 self tests completed in 0.26 seconds
#694: append_last_lines() output_log_level: 25
This is a potentially dangerous function.
#694: append_last_lines() output_log_level: 25
check --repair might lead to data loss (for kinds of corruption it is not
#694: append_last_lines() output_log_level: 25
capable of dealing with). BE VERY CAREFUL!
#694: still_running: False
#694: exit_code_indicates_error(): False
#694: output_buffers: [<_io.BufferedReader name=3>]
YES
#694: append_last_lines() output_log_level: 25
Type 'YES' if you understand this and want to continue: Starting repository check
#694: append_last_lines() output_log_level: 25
Verified integrity of /usbdrive/backup-ext/borgroot/index.9446
#694: append_last_lines() output_log_level: 25
Read committed index of transaction 9446
#694: append_last_lines() output_log_level: 25
Cleaned up 0 uncommitted segment files (== everything after segment 9446).
#694: append_last_lines() output_log_level: 25
Segment transaction is    9446
#694: append_last_lines() output_log_level: 25
Determined transaction is 9446
#694: append_last_lines() output_log_level: 25
Found 3478 segments
#694: append_last_lines() output_log_level: 25
checking segment file /usbdrive/backup-ext/borgroot/data/0/1...
#694: append_last_lines() output_log_level: 25
checking segment file /usbdrive/backup-ext/borgroot/data/0/24...

consistency section

consistency:
    checks:
        - repository
        - archives
    check_last: 3
    prefix: "pveroot-"

recopied the two files and tried again. note im using python 3.9 not 3.10 in your example above. and sorry about weird line wraps. i did this on a phone. ```bash matt@pve:~ $ sudo borgmatic --dry-run -v 2 -c /etc/borgmatic.d/mybk-root.yaml borg check --repair Ensuring legacy configuration is upgraded borg --version --debug --show-rc /usbdrive/backup-ext/borgroot: Running actions for repository /etc/borgmatic.d/mybk-root.yaml: No commands to run for pre-actions hook /usbdrive/backup-ext/borgroot: Running arbitrary Borg command #694: execute_command() #694: output_log_level: 25 #694: output_file: None borg check /usbdrive/backup-ext/borgroot --repair --debug --show-rc #694: do_not_capture: False #694: stdout: -1 #694: stderr: -2 #694: run_to_completion: True #694: log_outputs() output_log_level: 25 #694: process_for_output_buffer: {<_io.BufferedReader name=3>: <Popen: returncode: None args: ['borg', 'check', '/usbdrive/backup-ext/borgr...>}#694: output_buffers: [<_io.BufferedReader name=3>] #694: append_last_lines() output_log_level: 25 using builtin fallback logging configuration #694: append_last_lines() output_log_level: 25 33 self tests completed in 0.26 seconds #694: append_last_lines() output_log_level: 25 This is a potentially dangerous function. #694: append_last_lines() output_log_level: 25 check --repair might lead to data loss (for kinds of corruption it is not #694: append_last_lines() output_log_level: 25 capable of dealing with). BE VERY CAREFUL! #694: still_running: False #694: exit_code_indicates_error(): False #694: output_buffers: [<_io.BufferedReader name=3>] YES #694: append_last_lines() output_log_level: 25 Type 'YES' if you understand this and want to continue: Starting repository check #694: append_last_lines() output_log_level: 25 Verified integrity of /usbdrive/backup-ext/borgroot/index.9446 #694: append_last_lines() output_log_level: 25 Read committed index of transaction 9446 #694: append_last_lines() output_log_level: 25 Cleaned up 0 uncommitted segment files (== everything after segment 9446). #694: append_last_lines() output_log_level: 25 Segment transaction is 9446 #694: append_last_lines() output_log_level: 25 Determined transaction is 9446 #694: append_last_lines() output_log_level: 25 Found 3478 segments #694: append_last_lines() output_log_level: 25 checking segment file /usbdrive/backup-ext/borgroot/data/0/1... #694: append_last_lines() output_log_level: 25 checking segment file /usbdrive/backup-ext/borgroot/data/0/24... ``` consistency section ```yaml consistency: checks: - repository - archives check_last: 3 prefix: "pveroot-" ```

y8s commented

2023-05-14 03:14:15 +00:00

here is where check.py is

matt@pve:/usr/local/lib/python3.9/dist-packages/borgmatic/borg $ ls
borg.py         feature.py   rcreate.py
break_lock.py   flags.py     rinfo.py
check.py        info.py      rlist.py
check.py.bak    __init__.py  state.py
compact.py      init.py      transfer.py
create.py       list.py      umount.py
environment.py  mount.py     version.py
export_tar.py   prune.py
extract.py      __pycache__
matt@pve:/usr/local/lib/python3.9/dist-packages/borgmatic/borg $

here is where check.py is ```bash matt@pve:/usr/local/lib/python3.9/dist-packages/borgmatic/borg $ ls borg.py feature.py rcreate.py break_lock.py flags.py rinfo.py check.py info.py rlist.py check.py.bak __init__.py state.py compact.py init.py transfer.py create.py list.py umount.py environment.py mount.py version.py export_tar.py prune.py extract.py __pycache__ matt@pve:/usr/local/lib/python3.9/dist-packages/borgmatic/borg $ ```

witten commented

2023-05-14 05:39:20 +00:00

OMG, I just figured it out, and the answer has been staring me in the face the whole time! Here's your original command:

sudo borgmatic -c /etc/borgmatic.d/configuration.yaml borg check --repair

Instead of running borgmatic's check action directly, this command runs borgmatic's borg action, designed for running arbitrary Borg commands. And that borg action has a limitation that it captures and logs all output—preventing interactive Borg prompts from working.

The reason I couldn't reproduce this was because I wasn't running the borg action, I was running borgmatic's check directly:

sudo borgmatic -c /etc/borgmatic.d/configuration.yaml check --repair

This action is smarter (because it actually parses the flags instead of blindly passing them to Borg), and realizes if --repair is given the borgmatic needs to run in interactive mode to support Borg's prompts. So if you try that check action instead of borg check, I believe it'll work for you too.

The other bit of serendipity is that there's already a soon-to-be-released fix implemented such that the borg action will no longer capture output for logging—and instead pass through Borg output unmolested. This means that, among other things, the borg check --repair prompt will work as of the next release.

I hope this gets to the bottom of the issue you were having. Thanks for sticking with this, and let me know if you have any questions! You can roll back the modified source files simply by running sudo pip install --upgrade borgmatic.

OMG, I just figured it out, and the answer has been staring me in the face the whole time! Here's your original command: ``` sudo borgmatic -c /etc/borgmatic.d/configuration.yaml borg check --repair ``` Instead of running borgmatic's `check` action directly, this command runs borgmatic's `borg` action, designed for [running arbitrary Borg commands](https://torsion.org/borgmatic/docs/how-to/run-arbitrary-borg-commands/). And that `borg` action has a [limitation](https://torsion.org/borgmatic/docs/how-to/run-arbitrary-borg-commands/#limitations) that it captures and logs all output—preventing interactive Borg prompts from working. The reason I couldn't reproduce this was because I *wasn't* running the `borg` action, I was running borgmatic's `check` directly: ``` sudo borgmatic -c /etc/borgmatic.d/configuration.yaml check --repair ``` This action is smarter (because it actually parses the flags instead of blindly passing them to Borg), and realizes if `--repair` is given the borgmatic needs to run in interactive mode to support Borg's prompts. So if you try that `check` action instead of `borg check`, I believe it'll work for you too. The other bit of serendipity is that there's *already* a soon-to-be-released fix implemented such that the `borg` action will no longer capture output for logging—and instead pass through Borg output unmolested. This means that, among other things, the `borg check --repair` prompt will work as of the next release. I hope this gets to the bottom of the issue you were having. Thanks for sticking with this, and let me know if you have any questions! You can roll back the modified source files simply by running `sudo pip install --upgrade borgmatic`.

y8s commented

2023-05-14 13:11:35 +00:00

either I first tried to run an older version or missed it in the command line options docs.

when was check added to borgmatic?

matt@pve:~ $ sudo borgmatic --dry-run -v 2 -c /etc/borgmatic.d/mybk-root.yaml check --repair    Ensuring legacy configuration is upgraded
borg --version --debug --show-rc
/usbdrive/backup-ext/borgroot: Running actions for repository
/etc/borgmatic.d/mybk-root.yaml: No commands to run for pre-actions hook
/etc/borgmatic.d/mybk-root.yaml: No commands to run for pre-check hook
/usbdrive/backup-ext/borgroot: Running consistency checks
borg info --json /usbdrive/backup-ext/borgroot
#694: check_archives() repair: True
#694: check_archives() progress: False
#694: executing command WITHOUT capture
#694: execute_command()
#694: output_log_level: 20
#694: output_file: <object object at 0x7f5860708910>
borg check --repair --last 3 --glob-archives pveroot-* --debug --show-rc /usbdrive/backup-ext/borgroot >
#694: do_not_capture: True
#694: stdout: None
#694: stderr: None
#694: run_to_completion: True
#694: log_outputs() output_log_level: 20
#694: process_for_output_buffer: {}
#694: output_buffers: []
#694: still_running: False
using builtin fallback logging configuration
33 self tests completed in 0.23 seconds
This is a potentially dangerous function.
check --repair might lead to data loss (for kinds of corruption it is not
capable of dealing with). BE VERY CAREFUL!
Type 'YES' if you understand this and want to continue:

heres an idea!:
when running a borg command that exists in borgmatic, pop up a prompt to confirm the user wants to run the borg command over the native command.

either I first tried to run an older version or missed it in the command line options docs. when was check added to borgmatic? ```bash matt@pve:~ $ sudo borgmatic --dry-run -v 2 -c /etc/borgmatic.d/mybk-root.yaml check --repair Ensuring legacy configuration is upgraded borg --version --debug --show-rc /usbdrive/backup-ext/borgroot: Running actions for repository /etc/borgmatic.d/mybk-root.yaml: No commands to run for pre-actions hook /etc/borgmatic.d/mybk-root.yaml: No commands to run for pre-check hook /usbdrive/backup-ext/borgroot: Running consistency checks borg info --json /usbdrive/backup-ext/borgroot #694: check_archives() repair: True #694: check_archives() progress: False #694: executing command WITHOUT capture #694: execute_command() #694: output_log_level: 20 #694: output_file: <object object at 0x7f5860708910> borg check --repair --last 3 --glob-archives pveroot-* --debug --show-rc /usbdrive/backup-ext/borgroot > #694: do_not_capture: True #694: stdout: None #694: stderr: None #694: run_to_completion: True #694: log_outputs() output_log_level: 20 #694: process_for_output_buffer: {} #694: output_buffers: [] #694: still_running: False using builtin fallback logging configuration 33 self tests completed in 0.23 seconds This is a potentially dangerous function. check --repair might lead to data loss (for kinds of corruption it is not capable of dealing with). BE VERY CAREFUL! Type 'YES' if you understand this and want to continue: ``` heres an idea!: when running a borg command that exists in borgmatic, pop up a prompt to confirm the user wants to run the borg command over the native command.

witten commented

2023-05-14 16:55:30 +00:00

when was check added to borgmatic?

I'm pretty sure it's been there since the beginning! But I'm glad to hear it looks like the prompt is working properly for you now.

when running a borg command that exists in borgmatic, pop up a prompt to confirm the user wants to run the borg command over the native command.

That's a great idea. What I'll do is log a prominent warning when this happens.

> when was check added to borgmatic? I'm pretty sure it's been there since the beginning! But I'm glad to hear it looks like the prompt is working properly for you now. > when running a borg command that exists in borgmatic, pop up a prompt to confirm the user wants to run the borg command over the native command. That's a great idea. What I'll do is log a prominent warning when this happens.

witten closed this issue

2023-05-14 16:55:30 +00:00

witten added the

question / support

label 2023-05-14 16:55:37 +00:00

witten referenced this issue from a commit

2023-05-14 19:51:33 +00:00

Log a warning when "borgmatic borg" is run with an action that borgmatic natively supports (#694).

witten commented

2023-05-16 17:02:30 +00:00

FYI the fix to borgmatic borg has just been released in borgmatic 1.7.13.

FYI the fix to `borgmatic borg` has just been released in borgmatic 1.7.13.