From: Nicholas Bellinger <[email protected]>
This patch fixes a regression bug in virtscsi_kick_cmd() that relinquishes
the acquired spinlocks in the incorrect order using the wrong spin_unlock
macros, namely releasing vq->vq_lock before tgt->tgt_lock while invoking
the calls to virtio_ring.c:virtqueue_add_buf() and friends.
This bug was originally introduced in v3.5-rc7 code with:
commit 2bd37f0fde99cbf8b78fb55f1128e8c3a63cf1da
Author: Paolo Bonzini <[email protected]>
Date: Wed Jun 13 16:56:34 2012 +0200
[SCSI] virtio-scsi: split scatterlist per target
Go ahead and make sure that vq->vq_lock is relinquished w/ spin_unlock
first, then release tgt->tgt_lock w/ spin_unlock_irqrestore.
Cc: Paolo Bonzini <[email protected]>
Cc: James Bottomley <[email protected]>
Cc: Christoph Hellwig <[email protected]>
Cc: [email protected]
Signed-off-by: Nicholas Bellinger <[email protected]>
---
drivers/scsi/virtio_scsi.c | 4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/scsi/virtio_scsi.c b/drivers/scsi/virtio_scsi.c
index 595af1a..b2abb8a 100644
--- a/drivers/scsi/virtio_scsi.c
+++ b/drivers/scsi/virtio_scsi.c
@@ -417,11 +417,11 @@ static int virtscsi_kick_cmd(struct virtio_scsi_target_state *tgt,
spin_lock(&vq->vq_lock);
ret = virtqueue_add_buf(vq->vq, tgt->sg, out_num, in_num, cmd, gfp);
- spin_unlock(&tgt->tgt_lock);
+ spin_unlock(&vq->vq_lock);
if (ret >= 0)
ret = virtqueue_kick_prepare(vq->vq);
- spin_unlock_irqrestore(&vq->vq_lock, flags);
+ spin_unlock_irqrestore(&tgt->tgt_lock, flags);
if (ret > 0)
virtqueue_notify(vq->vq);
--
1.7.2.5
On 11/09/2012 02:29 PM, Nicholas A. Bellinger wrote:
> From: Nicholas Bellinger <[email protected]>
>
> This patch fixes a regression bug in virtscsi_kick_cmd() that relinquishes
> the acquired spinlocks in the incorrect order using the wrong spin_unlock
> macros, namely releasing vq->vq_lock before tgt->tgt_lock while invoking
> the calls to virtio_ring.c:virtqueue_add_buf() and friends.
>
> This bug was originally introduced in v3.5-rc7 code with:
>
> commit 2bd37f0fde99cbf8b78fb55f1128e8c3a63cf1da
> Author: Paolo Bonzini <[email protected]>
> Date: Wed Jun 13 16:56:34 2012 +0200
>
> [SCSI] virtio-scsi: split scatterlist per target
>
> Go ahead and make sure that vq->vq_lock is relinquished w/ spin_unlock
> first, then release tgt->tgt_lock w/ spin_unlock_irqrestore.
Did you hit any error? I don't think this order is wrong.
Thanks,
Wanlong Gao
>
> Cc: Paolo Bonzini <[email protected]>
> Cc: James Bottomley <[email protected]>
> Cc: Christoph Hellwig <[email protected]>
> Cc: [email protected]
> Signed-off-by: Nicholas Bellinger <[email protected]>
> ---
> drivers/scsi/virtio_scsi.c | 4 ++--
> 1 files changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/scsi/virtio_scsi.c b/drivers/scsi/virtio_scsi.c
> index 595af1a..b2abb8a 100644
> --- a/drivers/scsi/virtio_scsi.c
> +++ b/drivers/scsi/virtio_scsi.c
> @@ -417,11 +417,11 @@ static int virtscsi_kick_cmd(struct virtio_scsi_target_state *tgt,
>
> spin_lock(&vq->vq_lock);
> ret = virtqueue_add_buf(vq->vq, tgt->sg, out_num, in_num, cmd, gfp);
> - spin_unlock(&tgt->tgt_lock);
> + spin_unlock(&vq->vq_lock);
> if (ret >= 0)
> ret = virtqueue_kick_prepare(vq->vq);
>
> - spin_unlock_irqrestore(&vq->vq_lock, flags);
> + spin_unlock_irqrestore(&tgt->tgt_lock, flags);
>
> if (ret > 0)
> virtqueue_notify(vq->vq);
>
Il 09/11/2012 07:29, Nicholas A. Bellinger ha scritto:
> From: Nicholas Bellinger <[email protected]>
>
> This patch fixes a regression bug in virtscsi_kick_cmd() that relinquishes
> the acquired spinlocks in the incorrect order using the wrong spin_unlock
> macros, namely releasing vq->vq_lock before tgt->tgt_lock while invoking
> the calls to virtio_ring.c:virtqueue_add_buf() and friends.
>
> This bug was originally introduced in v3.5-rc7 code with:
>
> commit 2bd37f0fde99cbf8b78fb55f1128e8c3a63cf1da
> Author: Paolo Bonzini <[email protected]>
> Date: Wed Jun 13 16:56:34 2012 +0200
>
> [SCSI] virtio-scsi: split scatterlist per target
>
> Go ahead and make sure that vq->vq_lock is relinquished w/ spin_unlock
> first, then release tgt->tgt_lock w/ spin_unlock_irqrestore.
That's done on purpose. After you do virtqueue_add_buf, you don't need
the sg list anymore, nor the lock that protects it. The cover letter is
at https://lkml.org/lkml/2012/6/13/295 and had this text:
This series reorganizes the locking in virtio-scsi, introducing
separate scatterlists for each target and "pipelining" the locks so
that one command can be queued while the other is prepared. This
improves performance when there are multiple in-flight operations.
In fact, the patch _introduces_ wrong locking because
virtqueue_kick_prepare needs the vq_lock.
Perhaps what you want is separate local_irq_save/local_irq_restore?
Paolo
> Cc: Paolo Bonzini <[email protected]>
> Cc: James Bottomley <[email protected]>
> Cc: Christoph Hellwig <[email protected]>
> Cc: [email protected]
> Signed-off-by: Nicholas Bellinger <[email protected]>
> ---
> drivers/scsi/virtio_scsi.c | 4 ++--
> 1 files changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/scsi/virtio_scsi.c b/drivers/scsi/virtio_scsi.c
> index 595af1a..b2abb8a 100644
> --- a/drivers/scsi/virtio_scsi.c
> +++ b/drivers/scsi/virtio_scsi.c
> @@ -417,11 +417,11 @@ static int virtscsi_kick_cmd(struct virtio_scsi_target_state *tgt,
>
> spin_lock(&vq->vq_lock);
> ret = virtqueue_add_buf(vq->vq, tgt->sg, out_num, in_num, cmd, gfp);
> - spin_unlock(&tgt->tgt_lock);
> + spin_unlock(&vq->vq_lock);
> if (ret >= 0)
> ret = virtqueue_kick_prepare(vq->vq);
>
> - spin_unlock_irqrestore(&vq->vq_lock, flags);
> + spin_unlock_irqrestore(&tgt->tgt_lock, flags);
>
> if (ret > 0)
> virtqueue_notify(vq->vq);
>
Hi Paolo,
On Fri, 2012-11-09 at 09:42 +0100, Paolo Bonzini wrote:
> Il 09/11/2012 07:29, Nicholas A. Bellinger ha scritto:
> > From: Nicholas Bellinger <[email protected]>
> >
> > This patch fixes a regression bug in virtscsi_kick_cmd() that relinquishes
> > the acquired spinlocks in the incorrect order using the wrong spin_unlock
> > macros, namely releasing vq->vq_lock before tgt->tgt_lock while invoking
> > the calls to virtio_ring.c:virtqueue_add_buf() and friends.
> >
> > This bug was originally introduced in v3.5-rc7 code with:
> >
> > commit 2bd37f0fde99cbf8b78fb55f1128e8c3a63cf1da
> > Author: Paolo Bonzini <[email protected]>
> > Date: Wed Jun 13 16:56:34 2012 +0200
> >
> > [SCSI] virtio-scsi: split scatterlist per target
> >
> > Go ahead and make sure that vq->vq_lock is relinquished w/ spin_unlock
> > first, then release tgt->tgt_lock w/ spin_unlock_irqrestore.
>
> That's done on purpose. After you do virtqueue_add_buf, you don't need
> the sg list anymore, nor the lock that protects it. The cover letter is
> at https://lkml.org/lkml/2012/6/13/295 and had this text:
>
> This series reorganizes the locking in virtio-scsi, introducing
> separate scatterlists for each target and "pipelining" the locks so
> that one command can be queued while the other is prepared. This
> improves performance when there are multiple in-flight operations.
>
> In fact, the patch _introduces_ wrong locking because
> virtqueue_kick_prepare needs the vq_lock.
>
> Perhaps what you want is separate local_irq_save/local_irq_restore?
>
Ahh, that makes more sense now.
Just noticed this while reviewing code that using one spinlock flag's to
release the other looks suspicious, minus the ordering bit..
Using local_irq_* would probably be cleaner than swapping flags between
different locks, and a short comment here would be helpful to explain
the locking order context.
Anyways, no big deal. Thanks for the explanation.
--nab
Il 09/11/2012 20:31, Nicholas A. Bellinger ha scritto:
>> That's done on purpose. After you do virtqueue_add_buf, you don't need
>> the sg list anymore, nor the lock that protects it. The cover letter is
>> at https://lkml.org/lkml/2012/6/13/295 and had this text:
>>
>> This series reorganizes the locking in virtio-scsi, introducing
>> separate scatterlists for each target and "pipelining" the locks so
>> that one command can be queued while the other is prepared. This
>> improves performance when there are multiple in-flight operations.
>>
>> In fact, the patch _introduces_ wrong locking because
>> virtqueue_kick_prepare needs the vq_lock.
>>
>> Perhaps what you want is separate local_irq_save/local_irq_restore?
>
> Ahh, that makes more sense now.
>
> Just noticed this while reviewing code that using one spinlock flag's to
> release the other looks suspicious, minus the ordering bit..
>
> Using local_irq_* would probably be cleaner than swapping flags between
> different locks, and a short comment here would be helpful to explain
> the locking order context.
Well, my plan is to improve the virtio API so I can reuse the higher
layer's scatterlist, and get rid of the lock (not just of the funny
order) altogether. :) Queuing requests is really performance-sensitive,
and it can use any optimization.
But if I can't get to it quick, I'll queue a cleanup using local_irq_*.
Paolo