I've noticed that my ssh sessions would hang during test runs, which
is really frustrating when you kick off a 13 hour test before going to
bed, and the second test (1 hour into it) hangs, and you need to kick
it off again in the morning (wasting all that time over night).
I finally figured out the cause. There is a disconnect between
the run_command that executes the test, and the "wait_for_input" that
monitors the test. The wait_for_input has a default of 2 minute timeout
if it doesn't see any output it returns. The run_command takes the
empty string from wait_for_input as the test is finished, and then
stops monitoring it, and calls waitpid() waiting for the test to
exit.
The problem is that if the test has a lot of output, it will continue
writing into the pipe that was suppose to go to the monitor, which has
now exited the loop. When the pipe fills up, it will not finish.
When the test is over, it just hangs waiting for the pipe to flush
(which never happens).
To fix this, change the run_command to by default have an infinite
run (which can be overridden by the new RUN_TIMEOUT option), and
make the wait_for_input also wait indefinitely in this case. It now
Now the tests will have its content continuously read and will exit
normally.
While debugging this, I also found out why you can lose stdout on
the terminal sometimes. Especially if you hit Ctrl^C while the monitor
is running. It was due to missing "end_monitor" which gives back the
tty to the terminal. The first two patches fix that.
git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-ktest.git
devel
Head SHA1: aa9aba9382884554fe6a303744884866d137422d
Steven Rostedt (3):
ktest.pl: Fix missing "end_monitor" when machine check fails
ktest.pl: Give back console on Ctrt^C on monitor
ktest.pl: Add RUN_TIMEOUT option with default unlimited
----
tools/testing/ktest/ktest.pl | 26 +++++++++++++++++++++-----
tools/testing/ktest/sample.conf | 5 +++++
2 files changed, 26 insertions(+), 5 deletions(-)
On Wed, 18 Jan 2023 16:54:35 -0500
Steven Rostedt <[email protected]> wrote:
> I've noticed that my ssh sessions would hang during test runs, which
> is really frustrating when you kick off a 13 hour test before going to
> bed, and the second test (1 hour into it) hangs, and you need to kick
> it off again in the morning (wasting all that time over night).
>
> I finally figured out the cause. There is a disconnect between
> the run_command that executes the test, and the "wait_for_input" that
> monitors the test. The wait_for_input has a default of 2 minute timeout
> if it doesn't see any output it returns. The run_command takes the
> empty string from wait_for_input as the test is finished, and then
> stops monitoring it, and calls waitpid() waiting for the test to
> exit.
>
> The problem is that if the test has a lot of output, it will continue
> writing into the pipe that was suppose to go to the monitor, which has
> now exited the loop. When the pipe fills up, it will not finish.
> When the test is over, it just hangs waiting for the pipe to flush
> (which never happens).
>
> To fix this, change the run_command to by default have an infinite
> run (which can be overridden by the new RUN_TIMEOUT option), and
> make the wait_for_input also wait indefinitely in this case. It now
> Now the tests will have its content continuously read and will exit
> normally.
>
> While debugging this, I also found out why you can lose stdout on
> the terminal sometimes. Especially if you hit Ctrl^C while the monitor
> is running. It was due to missing "end_monitor" which gives back the
> tty to the terminal. The first two patches fix that.
>
>
Thanks for updating. I ran the test and confirmed that the terminal
setting is recovered :)
Tested-by: Masami Hiramatsu (Google) <[email protected]>
BTW, I found another issue that if I didn't set up sendmail,
it doesn't recover stty. Let me send a fix.
>
> git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-ktest.git
> devel
Also, this devel branch seems not pushed yet.
Thank you,
>
> Head SHA1: aa9aba9382884554fe6a303744884866d137422d
>
>
> Steven Rostedt (3):
> ktest.pl: Fix missing "end_monitor" when machine check fails
> ktest.pl: Give back console on Ctrt^C on monitor
> ktest.pl: Add RUN_TIMEOUT option with default unlimited
>
> ----
> tools/testing/ktest/ktest.pl | 26 +++++++++++++++++++++-----
> tools/testing/ktest/sample.conf | 5 +++++
> 2 files changed, 26 insertions(+), 5 deletions(-)
--
Masami Hiramatsu (Google) <[email protected]>