target: fix deadlock on unload
On uniprocessor preemptible kernel, target core deadlocks on unload. The
following events happen:
* iscsit_del_np is called
* it calls send_sig(SIGINT, np->np_thread, 1);
* the scheduler switches to the np_thread
* the np_thread is woken up, it sees that kthread_should_stop() returns
false, so it doesn't terminate
* the np_thread clears signals with flush_signals(current); and goes back
to sleep in iscsit_accept_np
* the scheduler switches back to iscsit_del_np
* iscsit_del_np calls kthread_stop(np->np_thread);
* the np_thread is waiting in iscsit_accept_np and it doesn't respond to
kthread_stop
The deadlock could be resolved if the administrator sends SIGINT signal to
the np_thread with killall -INT iscsi_np
The reproducible deadlock was introduced in commit
db6077fd0b7dd41dc6ff18329cec979379071f87, but the thread-stopping code was
racy even before.
This patch fixes the problem. Using kthread_should_stop to stop the
np_thread is unreliable, so we test np_thread_state instead. If
np_thread_state equals ISCSI_NP_THREAD_SHUTDOWN, the thread exits.
Signed-off-by: Mikulas Patocka <[email protected]>
Cc: [email protected]
---
drivers/target/iscsi/iscsi_target_login.c | 13 +++++++------
1 file changed, 7 insertions(+), 6 deletions(-)
Index: linux-3.14.8/drivers/target/iscsi/iscsi_target_login.c
===================================================================
--- linux-3.14.8.orig/drivers/target/iscsi/iscsi_target_login.c 2014-06-19 00:10:44.634113370 +0200
+++ linux-3.14.8/drivers/target/iscsi/iscsi_target_login.c 2014-06-23 19:05:44.887437402 +0200
@@ -1196,7 +1196,7 @@ old_sess_out:
static int __iscsi_target_login_thread(struct iscsi_np *np)
{
u8 *buffer, zero_tsih = 0;
- int ret = 0, rc, stop;
+ int ret = 0, rc;
struct iscsi_conn *conn = NULL;
struct iscsi_login *login;
struct iscsi_portal_group *tpg = NULL;
@@ -1210,6 +1210,9 @@ static int __iscsi_target_login_thread(s
if (np->np_thread_state == ISCSI_NP_THREAD_RESET) {
np->np_thread_state = ISCSI_NP_THREAD_ACTIVE;
complete(&np->np_restart_comp);
+ } else if (np->np_thread_state == ISCSI_NP_THREAD_SHUTDOWN) {
+ spin_unlock_bh(&np->np_thread_lock);
+ goto exit;
} else {
np->np_thread_state = ISCSI_NP_THREAD_ACTIVE;
}
@@ -1402,10 +1405,8 @@ old_sess_out:
}
out:
- stop = kthread_should_stop();
- /* Wait for another socket.. */
- if (!stop)
- return 1;
+ return 1;
+
exit:
iscsi_stop_login_thread_timer(np);
spin_lock_bh(&np->np_thread_lock);
@@ -1422,7 +1423,7 @@ int iscsi_target_login_thread(void *arg)
allow_signal(SIGINT);
- while (!kthread_should_stop()) {
+ while (1) {
ret = __iscsi_target_login_thread(np);
/*
* We break and exit here unless another sock_accept() call
Hi Mikulas,
On Mon, 2014-06-23 at 13:42 -0400, Mikulas Patocka wrote:
> target: fix deadlock on unload
>
> On uniprocessor preemptible kernel, target core deadlocks on unload. The
> following events happen:
> * iscsit_del_np is called
> * it calls send_sig(SIGINT, np->np_thread, 1);
> * the scheduler switches to the np_thread
> * the np_thread is woken up, it sees that kthread_should_stop() returns
> false, so it doesn't terminate
> * the np_thread clears signals with flush_signals(current); and goes back
> to sleep in iscsit_accept_np
> * the scheduler switches back to iscsit_del_np
> * iscsit_del_np calls kthread_stop(np->np_thread);
> * the np_thread is waiting in iscsit_accept_np and it doesn't respond to
> kthread_stop
>
> The deadlock could be resolved if the administrator sends SIGINT signal to
> the np_thread with killall -INT iscsi_np
>
> The reproducible deadlock was introduced in commit
> db6077fd0b7dd41dc6ff18329cec979379071f87, but the thread-stopping code was
> racy even before.
>
> This patch fixes the problem. Using kthread_should_stop to stop the
> np_thread is unreliable, so we test np_thread_state instead. If
> np_thread_state equals ISCSI_NP_THREAD_SHUTDOWN, the thread exits.
>
> Signed-off-by: Mikulas Patocka <[email protected]>
> Cc: [email protected]
>
Apologies for the delayed response..
Applied to target-pending/master and including in the next -rc3 PULL
request.
Also FYI, I've added '3.12+' to the stable tag to match how far back
commit db6077fd0 has been included in stable.
Thanks,
--nab
On Thu, 26 Jun 2014, Nicholas A. Bellinger wrote:
> Hi Mikulas,
>
> On Mon, 2014-06-23 at 13:42 -0400, Mikulas Patocka wrote:
> > target: fix deadlock on unload
> >
> > On uniprocessor preemptible kernel, target core deadlocks on unload. The
> > following events happen:
> > * iscsit_del_np is called
> > * it calls send_sig(SIGINT, np->np_thread, 1);
> > * the scheduler switches to the np_thread
> > * the np_thread is woken up, it sees that kthread_should_stop() returns
> > false, so it doesn't terminate
> > * the np_thread clears signals with flush_signals(current); and goes back
> > to sleep in iscsit_accept_np
> > * the scheduler switches back to iscsit_del_np
> > * iscsit_del_np calls kthread_stop(np->np_thread);
> > * the np_thread is waiting in iscsit_accept_np and it doesn't respond to
> > kthread_stop
> >
> > The deadlock could be resolved if the administrator sends SIGINT signal to
> > the np_thread with killall -INT iscsi_np
> >
> > The reproducible deadlock was introduced in commit
> > db6077fd0b7dd41dc6ff18329cec979379071f87, but the thread-stopping code was
> > racy even before.
> >
> > This patch fixes the problem. Using kthread_should_stop to stop the
> > np_thread is unreliable, so we test np_thread_state instead. If
> > np_thread_state equals ISCSI_NP_THREAD_SHUTDOWN, the thread exits.
> >
> > Signed-off-by: Mikulas Patocka <[email protected]>
> > Cc: [email protected]
> >
>
> Apologies for the delayed response..
>
> Applied to target-pending/master and including in the next -rc3 PULL
> request.
>
> Also FYI, I've added '3.12+' to the stable tag to match how far back
> commit db6077fd0 has been included in stable.
>
> Thanks,
>
> --nab
Hi
I think db6077fd0 should be backported to stable kernels beginning with
3.10 (because they set np->np_thread = NULL in
__iscsi_target_login_thread). The current 3.10-stable branch misses this
patch.
This patch for unload deadlock should be backported to all stable kernels
(because unload is racy there), but because of different code, we should
make a different patch for old stable branches.
For example in 3.4.95, __iscsi_target_login_thread contains this code:
spin_lock_bh(&np->np_thread_lock);
if (np->np_thread_state == ISCSI_NP_THREAD_RESET) {
np->np_thread_state = ISCSI_NP_THREAD_ACTIVE;
complete(&np->np_restart_comp);
} else {
np->np_thread_state = ISCSI_NP_THREAD_ACTIVE;
}
spin_unlock_bh(&np->np_thread_lock);
If the state is ISCSI_NP_THREAD_SHUTDOWN, the above piece of code will
change it to ISCSI_NP_THREAD_ACTIVE and open the same kthread_should_stop
race.
Mikulas
On Fri, 2014-06-27 at 08:35 -0400, Mikulas Patocka wrote:
>
> On Thu, 26 Jun 2014, Nicholas A. Bellinger wrote:
>
> > Hi Mikulas,
> >
> > On Mon, 2014-06-23 at 13:42 -0400, Mikulas Patocka wrote:
> > > target: fix deadlock on unload
> > >
> > > On uniprocessor preemptible kernel, target core deadlocks on unload. The
> > > following events happen:
> > > * iscsit_del_np is called
> > > * it calls send_sig(SIGINT, np->np_thread, 1);
> > > * the scheduler switches to the np_thread
> > > * the np_thread is woken up, it sees that kthread_should_stop() returns
> > > false, so it doesn't terminate
> > > * the np_thread clears signals with flush_signals(current); and goes back
> > > to sleep in iscsit_accept_np
> > > * the scheduler switches back to iscsit_del_np
> > > * iscsit_del_np calls kthread_stop(np->np_thread);
> > > * the np_thread is waiting in iscsit_accept_np and it doesn't respond to
> > > kthread_stop
> > >
> > > The deadlock could be resolved if the administrator sends SIGINT signal to
> > > the np_thread with killall -INT iscsi_np
> > >
> > > The reproducible deadlock was introduced in commit
> > > db6077fd0b7dd41dc6ff18329cec979379071f87, but the thread-stopping code was
> > > racy even before.
> > >
> > > This patch fixes the problem. Using kthread_should_stop to stop the
> > > np_thread is unreliable, so we test np_thread_state instead. If
> > > np_thread_state equals ISCSI_NP_THREAD_SHUTDOWN, the thread exits.
> > >
> > > Signed-off-by: Mikulas Patocka <[email protected]>
> > > Cc: [email protected]
> > >
> >
> > Apologies for the delayed response..
> >
> > Applied to target-pending/master and including in the next -rc3 PULL
> > request.
> >
> > Also FYI, I've added '3.12+' to the stable tag to match how far back
> > commit db6077fd0 has been included in stable.
> >
> > Thanks,
> >
> > --nab
>
> Hi
>
> I think db6077fd0 should be backported to stable kernels beginning with
> 3.10 (because they set np->np_thread = NULL in
> __iscsi_target_login_thread). The current 3.10-stable branch misses this
> patch.
>
>
> This patch for unload deadlock should be backported to all stable kernels
> (because unload is racy there), but because of different code, we should
> make a different patch for old stable branches.
>
> For example in 3.4.95, __iscsi_target_login_thread contains this code:
> spin_lock_bh(&np->np_thread_lock);
> if (np->np_thread_state == ISCSI_NP_THREAD_RESET) {
> np->np_thread_state = ISCSI_NP_THREAD_ACTIVE;
> complete(&np->np_restart_comp);
> } else {
> np->np_thread_state = ISCSI_NP_THREAD_ACTIVE;
> }
> spin_unlock_bh(&np->np_thread_lock);
> If the state is ISCSI_NP_THREAD_SHUTDOWN, the above piece of code will
> change it to ISCSI_NP_THREAD_ACTIVE and open the same kthread_should_stop
> race.
>
Ok, dropping the 'v3.12+' stable tag from this patch, and will include
it in a PATCH-v3.10.y series together with db6077fd0 once Greg-KH
attempts to queue it up.
As for v3.4.x, care to send along a separate patch with your Tested-by +
Signed-off-by..?
Thanks for the extra stable info!
--nab