Signal forwarding traceback #368

New Issue

Closed

opened 2020-11-18 05:08:22 +00:00 by witten · 2 comments

witten commented 2020-11-18 05:08:22 +00:00

Owner

Copy Link

Reported by f0x.

What I'm trying to do and why

Send a signal to borgmatic and have it be forwarded on to Borg. (In this case, SIGUSR1 for instrumentation purposes. But applies to other signals too.)

Steps to reproduce (if a bug)

Start a borgmatic create such that it invokes Borg. While Borg is running:

kill -SIGUSR1 [pid of borgmatic]

Actual behavior (if a bug)

Traceback (most recent call last):
  File "/usr/bin/borgmatic", line 33, in <module>
    sys.exit(load_entry_point('borgmatic==1.5.10', 'console_scripts', 'borgmatic')())
  File "/usr/lib/python3.8/site-packages/borgmatic/commands/borgmatic.py", line 785, in main
    summary_logs = parse_logs + list(collect_configuration_run_summary_logs(configs, arguments))
  File "/usr/lib/python3.8/site-packages/borgmatic/commands/borgmatic.py", line 683, in collect_configuration_run_summary_logs
    results = list(run_configuration(config_filename, config, arguments))
  File "/usr/lib/python3.8/site-packages/borgmatic/commands/borgmatic.py", line 124, in run_configuration
    yield from run_actions(
  File "/usr/lib/python3.8/site-packages/borgmatic/commands/borgmatic.py", line 317, in run_actions
    json_output = borg_create.create_archive(
  File "/usr/lib/python3.8/site-packages/borgmatic/borg/create.py", line 281, in create_archive
    return execute_command(full_command, output_log_level, output_file, borg_local_path=local_path)
  File "/usr/lib/python3.8/site-packages/borgmatic/execute.py", line 198, in execute_command
    log_outputs(
  File "/usr/lib/python3.8/site-packages/borgmatic/execute.py", line 71, in log_outputs
    (ready_buffers, _, _) = select.select(output_buffers, [], [])
  File "/usr/lib/python3.8/site-packages/borgmatic/signals.py", line 9, in _handle_signal
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/borgmatic/signals.py", line 9, in _handle_signal
    os.killpg(os.getpgrp(), signal_number)
  File "/usr/lib/python3.8/site-packages/borgmatic/signals.py", line 9, in _handle_signal
    os.killpg(os.getpgrp(), signal_number)
  [Previous line repeated 986 more times]
RecursionError: maximum recursion depth exceeded while calling a Python object 

Expected behavior (if a bug)

No traceback, and borgmatic forwards the signal onto Borg.

Other notes / implementation ideas

The problem here is that borgmatic's existing signal handler forwards the signal to borgmatic's entire process group, which include child processes like Borg. And while that means Borg receives the forwarded signal as designed, it also means borgmatic re-receives its own forwarded signal (being in the same process group).

Then the signal gets handled again, gets forwarded again, repeat ad infinitum.

Boom, stack blown.

Environment

borgmatic version: 1.5.10, master (1.5.11.dev0)

borgmatic installation method: pip install --editable

Borg version: 1.1.14

Python version: 3.8.6

Database version (if applicable): N/A

operating system and version: Manjaro stable

Reported by f0x. #### What I'm trying to do and why Send a signal to borgmatic and have it be forwarded on to Borg. (In this case, SIGUSR1 for instrumentation purposes. But applies to other signals too.) #### Steps to reproduce (if a bug) Start a `borgmatic create` such that it invokes Borg. While Borg is running: ```bash kill -SIGUSR1 [pid of borgmatic] ``` #### Actual behavior (if a bug) ```python Traceback (most recent call last): File "/usr/bin/borgmatic", line 33, in <module> sys.exit(load_entry_point('borgmatic==1.5.10', 'console_scripts', 'borgmatic')()) File "/usr/lib/python3.8/site-packages/borgmatic/commands/borgmatic.py", line 785, in main summary_logs = parse_logs + list(collect_configuration_run_summary_logs(configs, arguments)) File "/usr/lib/python3.8/site-packages/borgmatic/commands/borgmatic.py", line 683, in collect_configuration_run_summary_logs results = list(run_configuration(config_filename, config, arguments)) File "/usr/lib/python3.8/site-packages/borgmatic/commands/borgmatic.py", line 124, in run_configuration yield from run_actions( File "/usr/lib/python3.8/site-packages/borgmatic/commands/borgmatic.py", line 317, in run_actions json_output = borg_create.create_archive( File "/usr/lib/python3.8/site-packages/borgmatic/borg/create.py", line 281, in create_archive return execute_command(full_command, output_log_level, output_file, borg_local_path=local_path) File "/usr/lib/python3.8/site-packages/borgmatic/execute.py", line 198, in execute_command log_outputs( File "/usr/lib/python3.8/site-packages/borgmatic/execute.py", line 71, in log_outputs (ready_buffers, _, _) = select.select(output_buffers, [], []) File "/usr/lib/python3.8/site-packages/borgmatic/signals.py", line 9, in _handle_signal os.killpg(os.getpgrp(), signal_number) File "/usr/lib/python3.8/site-packages/borgmatic/signals.py", line 9, in _handle_signal os.killpg(os.getpgrp(), signal_number) File "/usr/lib/python3.8/site-packages/borgmatic/signals.py", line 9, in _handle_signal os.killpg(os.getpgrp(), signal_number) [Previous line repeated 986 more times] RecursionError: maximum recursion depth exceeded while calling a Python object ``` #### Expected behavior (if a bug) No traceback, and borgmatic forwards the signal onto Borg. #### Other notes / implementation ideas The problem here is that borgmatic's existing signal handler forwards the signal to borgmatic's entire process group, which include child processes like Borg. And while that means Borg receives the forwarded signal as designed, it also means borgmatic re-receives its own forwarded signal (being in the same process group). Then the signal gets handled again, gets forwarded again, repeat ad infinitum. Boom, stack blown. #### Environment **borgmatic version:** 1.5.10, master (1.5.11.dev0) **borgmatic installation method:** `pip install --editable` **Borg version:** 1.1.14 **Python version:** 3.8.6 **Database version (if applicable):** N/A **operating system and version:** Manjaro stable

witten referenced this issue from a commit 2020-11-18 18:29:29 +00:00

Fix signal forwarding from borgmatic to Borg resulting in recursion traceback (#368).

witten commented 2020-11-18 18:32:03 +00:00

Author

Owner

Copy Link

Fixed in master by stopping the infinite recursion when detected. This fix will be part of the next release.

Fixed in master by stopping the infinite recursion when detected. This fix will be part of the next release.

witten closed this issue 2020-11-18 18:32:04 +00:00

witten added the bug label 2020-11-18 18:32:10 +00:00

witten commented 2020-11-19 18:01:49 +00:00

Author

Owner

Copy Link

Just released in borgmatic 1.5.11!

Just released in borgmatic 1.5.11!

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

command-hook-steps

2.0.13

2.0.12

2.0.11

2.0.10

2.0.9

2.0.8

2.0.7

2.0.6

2.0.5

2.0.4

2.0.3

2.0.2

2.0.1

2.0.0

1.9.14

1.9.13

1.9.12

1.9.11

1.9.10

1.9.9

1.9.8

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.14

1.8.13

1.8.12

1.8.11

1.8.10

1.8.9

1.8.8

1.8.7

1.8.6

1.8.5

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.15

1.7.14

1.7.13

1.7.12

1.7.11

1.7.10

1.7.9

1.7.8

1.7.7

1.7.6

1.7.5

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.24

1.5.23

1.5.22

1.5.21

1.5.20

1.5.19

1.5.18

1.5.17

1.5.16

1.5.15

1.5.14

1.5.13

1.5.12

1.5.11

1.5.10

1.5.9

1.5.8

1.5.7

1.5.7.dev0

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.22

1.4.21

1.4.20

1.4.19

1.4.18

1.4.17

1.4.16

1.4.15

1.4.14

1.4.13

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.26

1.3.25

1.3.24

1.3.23

1.3.22

1.3.21

1.3.20

1.3.19

1.3.18

1.3.17

1.3.16

1.3.15

1.3.14

1.3.13

1.3.12

1.3.11

1.3.10

1.3.9

1.3.8

1.3.7

1.3.6

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.18

1.2.17

1.2.16

1.2.15

1.2.14

1.2.13

1.2.12

1.2.11

1.2.10

1.2.9

1.2.8

1.2.7

1.2.6

1.2.5

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.15

1.1.14

1.1.13

1.1.12

1.1.11

1.1.10

1.1.9

1.1.7

1.1.6

1.1.5

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.3

1.0.2

1.0.1

1.0.0

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

0.0.2

Labels

Clear labels

blocked

breaking

bug

data loss

design finalized

good first issue

new feature area

question / support

security

waiting for response

No Label bug

Milestone

No items

No Milestone

Assignees

Clear assignees

witten

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: borgmatic-collective/borgmatic#368