When user terminates the script (also implicitly through for example
`make clang-analyzer`) by using
Ctrl+C (or sending SIGINT more generally) the KeyboardInterrupt
is raised printing stack trace of the execution as shown below:
$ ./scripts/clang-tools/run-clang-tools.py clang-tidy ./compile_commands.json
^CTraceback (most recent call last):
File "./scripts/clang-tools/run-clang-tools.py", line 74, in <module>
main()
File "./scripts/clang-tools/run-clang-tools.py", line 70, in main
pool.map(run_analysis, datastore)
File "/usr/lib64/python3.8/multiprocessing/pool.py", line 364, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/usr/lib64/python3.8/multiprocessing/pool.py", line 765, in get
self.wait(timeout)
File "/usr/lib64/python3.8/multiprocessing/pool.py", line 762, in wait
self._event.wait(timeout)
File "/usr/lib64/python3.8/threading.py", line 558, in wait
Process ForkPoolWorker-6:
Process ForkPoolWorker-1:
Process ForkPoolWorker-5:
Process ForkPoolWorker-7:
Process ForkPoolWorker-2:
Process ForkPoolWorker-3:
Process ForkPoolWorker-4:
Process ForkPoolWorker-8:
signaled = self._cond.wait(timeout)
File "/usr/lib64/python3.8/threading.py", line 302, in wait
waiter.acquire()
KeyboardInterrupt
The patch handles the raise of the KeyboardInterrupt and exits when occurred
with code 130 as documented in: https://tldp.org/LDP/abs/html/exitcodes.html
Signed-off-by: Maciej Falkowski <[email protected]>
---
scripts/clang-tools/run-clang-tools.py | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/scripts/clang-tools/run-clang-tools.py b/scripts/clang-tools/run-clang-tools.py
index 38fc311d2e03..eb0e0ecfce24 100755
--- a/scripts/clang-tools/run-clang-tools.py
+++ b/scripts/clang-tools/run-clang-tools.py
@@ -77,4 +77,7 @@ def main():
if __name__ == "__main__":
- main()
+ try:
+ main()
+ except KeyboardInterrupt:
+ sys.exit(130)
--
2.26.3
On Thu, May 20, 2021 at 4:18 PM Maciej Falkowski
<[email protected]> wrote:
>
> When user terminates the script (also implicitly through for example
> `make clang-analyzer`) by using
> Ctrl+C (or sending SIGINT more generally) the KeyboardInterrupt
> is raised printing stack trace of the execution as shown below:
>
> $ ./scripts/clang-tools/run-clang-tools.py clang-tidy ./compile_commands.json
> ^CTraceback (most recent call last):
> File "./scripts/clang-tools/run-clang-tools.py", line 74, in <module>
> main()
> File "./scripts/clang-tools/run-clang-tools.py", line 70, in main
> pool.map(run_analysis, datastore)
> File "/usr/lib64/python3.8/multiprocessing/pool.py", line 364, in map
> return self._map_async(func, iterable, mapstar, chunksize).get()
> File "/usr/lib64/python3.8/multiprocessing/pool.py", line 765, in get
> self.wait(timeout)
> File "/usr/lib64/python3.8/multiprocessing/pool.py", line 762, in wait
> self._event.wait(timeout)
> File "/usr/lib64/python3.8/threading.py", line 558, in wait
> Process ForkPoolWorker-6:
> Process ForkPoolWorker-1:
> Process ForkPoolWorker-5:
> Process ForkPoolWorker-7:
> Process ForkPoolWorker-2:
> Process ForkPoolWorker-3:
> Process ForkPoolWorker-4:
> Process ForkPoolWorker-8:
> signaled = self._cond.wait(timeout)
> File "/usr/lib64/python3.8/threading.py", line 302, in wait
> waiter.acquire()
> KeyboardInterrupt
With this applied,
$ make LLVM=1 LLVM_IAS=1 -j72 clang-analyzer
^C
Process ForkPoolWorker-5:
make: *** [Makefile:1902: clang-analyzer] Error 130
Reviewed-by: Nick Desaulniers <[email protected]>
Tested-by: Nick Desaulniers <[email protected]>
Thanks for the patch!
> The patch handles the raise of the KeyboardInterrupt and exits when occurred
> with code 130 as documented in: https://tldp.org/LDP/abs/html/exitcodes.html
>
> Signed-off-by: Maciej Falkowski <[email protected]>
> ---
> scripts/clang-tools/run-clang-tools.py | 5 ++++-
> 1 file changed, 4 insertions(+), 1 deletion(-)
>
> diff --git a/scripts/clang-tools/run-clang-tools.py b/scripts/clang-tools/run-clang-tools.py
> index 38fc311d2e03..eb0e0ecfce24 100755
> --- a/scripts/clang-tools/run-clang-tools.py
> +++ b/scripts/clang-tools/run-clang-tools.py
> @@ -77,4 +77,7 @@ def main():
>
>
> if __name__ == "__main__":
> - main()
> + try:
> + main()
> + except KeyboardInterrupt:
> + sys.exit(130)
> --
> 2.26.3
>
--
Thanks,
~Nick Desaulniers
On Sat, May 22, 2021 at 2:18 AM Nick Desaulniers
<[email protected]> wrote:
>
> On Thu, May 20, 2021 at 4:18 PM Maciej Falkowski
> <[email protected]> wrote:
> >
> > When user terminates the script (also implicitly through for example
> > `make clang-analyzer`) by using
> > Ctrl+C (or sending SIGINT more generally) the KeyboardInterrupt
> > is raised printing stack trace of the execution as shown below:
> >
> > $ ./scripts/clang-tools/run-clang-tools.py clang-tidy ./compile_commands.json
> > ^CTraceback (most recent call last):
> > File "./scripts/clang-tools/run-clang-tools.py", line 74, in <module>
> > main()
> > File "./scripts/clang-tools/run-clang-tools.py", line 70, in main
> > pool.map(run_analysis, datastore)
> > File "/usr/lib64/python3.8/multiprocessing/pool.py", line 364, in map
> > return self._map_async(func, iterable, mapstar, chunksize).get()
> > File "/usr/lib64/python3.8/multiprocessing/pool.py", line 765, in get
> > self.wait(timeout)
> > File "/usr/lib64/python3.8/multiprocessing/pool.py", line 762, in wait
> > self._event.wait(timeout)
> > File "/usr/lib64/python3.8/threading.py", line 558, in wait
> > Process ForkPoolWorker-6:
> > Process ForkPoolWorker-1:
> > Process ForkPoolWorker-5:
> > Process ForkPoolWorker-7:
> > Process ForkPoolWorker-2:
> > Process ForkPoolWorker-3:
> > Process ForkPoolWorker-4:
> > Process ForkPoolWorker-8:
> > signaled = self._cond.wait(timeout)
> > File "/usr/lib64/python3.8/threading.py", line 302, in wait
> > waiter.acquire()
> > KeyboardInterrupt
>
> With this applied,
> $ make LLVM=1 LLVM_IAS=1 -j72 clang-analyzer
> ^C
> Process ForkPoolWorker-5:
> make: *** [Makefile:1902: clang-analyzer] Error 130
>
> Reviewed-by: Nick Desaulniers <[email protected]>
> Tested-by: Nick Desaulniers <[email protected]>
>
> Thanks for the patch!
I am not a python expert, but is
"Let's suppress back-trace when a user presses an interrupt"
the common coding style?
If really so, do we need to do something similar in all python scripts?
I do not know what is special about run-clang-tools.py.
For example, if I press Ctrl-C while building Clang
by using tc-build, I see a back-trace.
I have never thought back-tracing was annoying.
The exit code is 130 regardless of this patch.
BTW, I prefer not having "Makefile:" in the patch subject
since this is not touching Makefile at all.
I rather like "clang-tools:" or "scripts/clang-tools:".
> > The patch handles the raise of the KeyboardInterrupt and exits when occurred
> > with code 130 as documented in: https://tldp.org/LDP/abs/html/exitcodes.html
> >
> > Signed-off-by: Maciej Falkowski <[email protected]>
> > ---
> > scripts/clang-tools/run-clang-tools.py | 5 ++++-
> > 1 file changed, 4 insertions(+), 1 deletion(-)
> >
> > diff --git a/scripts/clang-tools/run-clang-tools.py b/scripts/clang-tools/run-clang-tools.py
> > index 38fc311d2e03..eb0e0ecfce24 100755
> > --- a/scripts/clang-tools/run-clang-tools.py
> > +++ b/scripts/clang-tools/run-clang-tools.py
> > @@ -77,4 +77,7 @@ def main():
> >
> >
> > if __name__ == "__main__":
> > - main()
> > + try:
> > + main()
> > + except KeyboardInterrupt:
> > + sys.exit(130)
> > --
> > 2.26.3
> >
>
>
> --
> Thanks,
> ~Nick Desaulniers
--
Best Regards
Masahiro Yamada
On 5/22/21 5:03 AM, Masahiro Yamada wrote:
> On Sat, May 22, 2021 at 2:18 AM Nick Desaulniers
> <[email protected]> wrote:
>> On Thu, May 20, 2021 at 4:18 PM Maciej Falkowski
>> <[email protected]> wrote:
>>> When user terminates the script (also implicitly through for example
>>> `make clang-analyzer`) by using
>>> Ctrl+C (or sending SIGINT more generally) the KeyboardInterrupt
>>> is raised printing stack trace of the execution as shown below:
>>>
>>> $ ./scripts/clang-tools/run-clang-tools.py clang-tidy ./compile_commands.json
>>> ^CTraceback (most recent call last):
>>> File "./scripts/clang-tools/run-clang-tools.py", line 74, in <module>
>>> main()
>>> File "./scripts/clang-tools/run-clang-tools.py", line 70, in main
>>> pool.map(run_analysis, datastore)
>>> File "/usr/lib64/python3.8/multiprocessing/pool.py", line 364, in map
>>> return self._map_async(func, iterable, mapstar, chunksize).get()
>>> File "/usr/lib64/python3.8/multiprocessing/pool.py", line 765, in get
>>> self.wait(timeout)
>>> File "/usr/lib64/python3.8/multiprocessing/pool.py", line 762, in wait
>>> self._event.wait(timeout)
>>> File "/usr/lib64/python3.8/threading.py", line 558, in wait
>>> Process ForkPoolWorker-6:
>>> Process ForkPoolWorker-1:
>>> Process ForkPoolWorker-5:
>>> Process ForkPoolWorker-7:
>>> Process ForkPoolWorker-2:
>>> Process ForkPoolWorker-3:
>>> Process ForkPoolWorker-4:
>>> Process ForkPoolWorker-8:
>>> signaled = self._cond.wait(timeout)
>>> File "/usr/lib64/python3.8/threading.py", line 302, in wait
>>> waiter.acquire()
>>> KeyboardInterrupt
>> With this applied,
>> $ make LLVM=1 LLVM_IAS=1 -j72 clang-analyzer
>> ^C
>> Process ForkPoolWorker-5:
>> make: *** [Makefile:1902: clang-analyzer] Error 130
>>
>> Reviewed-by: Nick Desaulniers <[email protected]>
>> Tested-by: Nick Desaulniers <[email protected]>
>>
>> Thanks for the patch!
>
> I am not a python expert, but is
> "Let's suppress back-trace when a user presses an interrupt"
> the common coding style?
I think that is not a common coding style.
In this case, the back-trace is verbose and it is caused
by a bug in multiprocessing Pool.
> If really so, do we need to do something similar in all python scripts?
> I do not know what is special about run-clang-tools.py.
I think no, I misunderstood the nature of the problem.
The verbose back-trace is a result of the multiprocessing Pool
not handling KeyboardInterrupt properly.
It is the old bug(bugs.python.org/issue8296), yet to be
solved(bugs.python.org/issue22393).
For more info please see:
https://stackoverflow.com/questions/1408356/keyboard-interrupts-with-pythons-multiprocessing-pool
This issue is already once solved in a Linux source tree in a
scripts/checkkconfigsymbols.py:
scripts/checkkconfigsymbols.py:312
try:
return check_symbols_helper(pool, ignore)
except KeyboardInterrupt:
pool.terminate()
pool.join()
sys.exit(1)
Also the comment:
scripts/checkkconfigsymbols.py:321
"""Helper method for check_symbols(). Used to catch keyboard interrupts in
check_symbols() in order to properly terminate running worker processes."""
> For example, if I press Ctrl-C while building Clang
> by using tc-build, I see a back-trace.
> I have never thought back-tracing was annoying.
>
> The exit code is 130 regardless of this patch.
Okay.
>
> BTW, I prefer not having "Makefile:" in the patch subject
> since this is not touching Makefile at all.
>
> I rather like "clang-tools:" or "scripts/clang-tools:".
>
Okay.
The solution should cover the whole bug, I think this patch
may be dismissed then.
Thank you for your feedback!
Best regards,
Maciej Falkowski
>
>
>
>
>
>>> The patch handles the raise of the KeyboardInterrupt and exits when occurred
>>> with code 130 as documented in: https://tldp.org/LDP/abs/html/exitcodes.html
>>>
>>> Signed-off-by: Maciej Falkowski <[email protected]>
>>> ---
>>> scripts/clang-tools/run-clang-tools.py | 5 ++++-
>>> 1 file changed, 4 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/scripts/clang-tools/run-clang-tools.py b/scripts/clang-tools/run-clang-tools.py
>>> index 38fc311d2e03..eb0e0ecfce24 100755
>>> --- a/scripts/clang-tools/run-clang-tools.py
>>> +++ b/scripts/clang-tools/run-clang-tools.py
>>> @@ -77,4 +77,7 @@ def main():
>>>
>>>
>>> if __name__ == "__main__":
>>> - main()
>>> + try:
>>> + main()
>>> + except KeyboardInterrupt:
>>> + sys.exit(130)
>>> --
>>> 2.26.3
>>>
>>
>> --
>> Thanks,
>> ~Nick Desaulniers
>
>
> --
> Best Regards
> Masahiro Yamada