For multi-queue and large ring-size use case, the following error
occurred when free_unused_bufs:
rcu: INFO: rcu_sched self-detected stall on CPU.
Fixes: 986a4f4d452d ("virtio_net: multiqueue support")
Signed-off-by: Wenliang Wang <[email protected]>
---
v2:
-add need_resched check.
-apply same logic to sq.
v3:
-use cond_resched instead.
v4:
-add fixes tag
---
drivers/net/virtio_net.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 8d8038538fc4..a12ae26db0e2 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -3560,12 +3560,14 @@ static void free_unused_bufs(struct virtnet_info *vi)
struct virtqueue *vq = vi->sq[i].vq;
while ((buf = virtqueue_detach_unused_buf(vq)) != NULL)
virtnet_sq_free_unused_buf(vq, buf);
+ cond_resched();
}
for (i = 0; i < vi->max_queue_pairs; i++) {
struct virtqueue *vq = vi->rq[i].vq;
while ((buf = virtqueue_detach_unused_buf(vq)) != NULL)
virtnet_rq_free_unused_buf(vq, buf);
+ cond_resched();
}
}
--
2.20.1
On Thu, May 04, 2023 at 10:27:06AM +0800, Wenliang Wang wrote:
> For multi-queue and large ring-size use case, the following error
> occurred when free_unused_bufs:
> rcu: INFO: rcu_sched self-detected stall on CPU.
>
> Fixes: 986a4f4d452d ("virtio_net: multiqueue support")
> Signed-off-by: Wenliang Wang <[email protected]>
Acked-by: Michael S. Tsirkin <[email protected]>
Probably a good idea for stable, too.
> ---
> v2:
> -add need_resched check.
> -apply same logic to sq.
> v3:
> -use cond_resched instead.
> v4:
> -add fixes tag
> ---
> drivers/net/virtio_net.c | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> index 8d8038538fc4..a12ae26db0e2 100644
> --- a/drivers/net/virtio_net.c
> +++ b/drivers/net/virtio_net.c
> @@ -3560,12 +3560,14 @@ static void free_unused_bufs(struct virtnet_info *vi)
> struct virtqueue *vq = vi->sq[i].vq;
> while ((buf = virtqueue_detach_unused_buf(vq)) != NULL)
> virtnet_sq_free_unused_buf(vq, buf);
> + cond_resched();
> }
>
> for (i = 0; i < vi->max_queue_pairs; i++) {
> struct virtqueue *vq = vi->rq[i].vq;
> while ((buf = virtqueue_detach_unused_buf(vq)) != NULL)
> virtnet_rq_free_unused_buf(vq, buf);
> + cond_resched();
> }
> }
>
> --
> 2.20.1
On Thu, May 4, 2023 at 10:27 AM Wenliang Wang
<[email protected]> wrote:
>
> For multi-queue and large ring-size use case, the following error
> occurred when free_unused_bufs:
> rcu: INFO: rcu_sched self-detected stall on CPU.
>
> Fixes: 986a4f4d452d ("virtio_net: multiqueue support")
> Signed-off-by: Wenliang Wang <[email protected]>
> ---
> v2:
> -add need_resched check.
> -apply same logic to sq.
> v3:
> -use cond_resched instead.
> v4:
> -add fixes tag
> ---
> drivers/net/virtio_net.c | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> index 8d8038538fc4..a12ae26db0e2 100644
> --- a/drivers/net/virtio_net.c
> +++ b/drivers/net/virtio_net.c
> @@ -3560,12 +3560,14 @@ static void free_unused_bufs(struct virtnet_info *vi)
> struct virtqueue *vq = vi->sq[i].vq;
> while ((buf = virtqueue_detach_unused_buf(vq)) != NULL)
> virtnet_sq_free_unused_buf(vq, buf);
> + cond_resched();
Does this really address the case when the virtqueue is very large?
Thanks
> }
>
> for (i = 0; i < vi->max_queue_pairs; i++) {
> struct virtqueue *vq = vi->rq[i].vq;
> while ((buf = virtqueue_detach_unused_buf(vq)) != NULL)
> virtnet_rq_free_unused_buf(vq, buf);
> + cond_resched();
> }
> }
>
> --
> 2.20.1
>
Hello:
This patch was applied to netdev/net.git (main)
by David S. Miller <[email protected]>:
On Thu, 4 May 2023 10:27:06 +0800 you wrote:
> For multi-queue and large ring-size use case, the following error
> occurred when free_unused_bufs:
> rcu: INFO: rcu_sched self-detected stall on CPU.
>
> Fixes: 986a4f4d452d ("virtio_net: multiqueue support")
> Signed-off-by: Wenliang Wang <[email protected]>
>
> [...]
Here is the summary with links:
- [v4] virtio_net: suppress cpu stall when free_unused_bufs
https://git.kernel.org/netdev/net/c/f8bb51043945
You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html
On Fri, 5 May 2023 11:28:25 +0800, Jason Wang <[email protected]> wrote:
> On Thu, May 4, 2023 at 10:27 AM Wenliang Wang
> <[email protected]> wrote:
> >
> > For multi-queue and large ring-size use case, the following error
> > occurred when free_unused_bufs:
> > rcu: INFO: rcu_sched self-detected stall on CPU.
> >
> > Fixes: 986a4f4d452d ("virtio_net: multiqueue support")
> > Signed-off-by: Wenliang Wang <[email protected]>
> > ---
> > v2:
> > -add need_resched check.
> > -apply same logic to sq.
> > v3:
> > -use cond_resched instead.
> > v4:
> > -add fixes tag
> > ---
> > drivers/net/virtio_net.c | 2 ++
> > 1 file changed, 2 insertions(+)
> >
> > diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> > index 8d8038538fc4..a12ae26db0e2 100644
> > --- a/drivers/net/virtio_net.c
> > +++ b/drivers/net/virtio_net.c
> > @@ -3560,12 +3560,14 @@ static void free_unused_bufs(struct virtnet_info *vi)
> > struct virtqueue *vq = vi->sq[i].vq;
> > while ((buf = virtqueue_detach_unused_buf(vq)) != NULL)
> > virtnet_sq_free_unused_buf(vq, buf);
> > + cond_resched();
>
> Does this really address the case when the virtqueue is very large?
Yes, I also have this question. I think cond_resched() should be called every
time a certain number of buffers are processed.
Thanks.
>
> Thanks
>
> > }
> >
> > for (i = 0; i < vi->max_queue_pairs; i++) {
> > struct virtqueue *vq = vi->rq[i].vq;
> > while ((buf = virtqueue_detach_unused_buf(vq)) != NULL)
> > virtnet_rq_free_unused_buf(vq, buf);
> > + cond_resched();
> > }
> > }
> >
> > --
> > 2.20.1
> >
>
On Fri, May 05, 2023 at 11:28:25AM +0800, Jason Wang wrote:
> On Thu, May 4, 2023 at 10:27 AM Wenliang Wang
> <[email protected]> wrote:
> >
> > For multi-queue and large ring-size use case, the following error
> > occurred when free_unused_bufs:
> > rcu: INFO: rcu_sched self-detected stall on CPU.
> >
> > Fixes: 986a4f4d452d ("virtio_net: multiqueue support")
> > Signed-off-by: Wenliang Wang <[email protected]>
> > ---
> > v2:
> > -add need_resched check.
> > -apply same logic to sq.
> > v3:
> > -use cond_resched instead.
> > v4:
> > -add fixes tag
> > ---
> > drivers/net/virtio_net.c | 2 ++
> > 1 file changed, 2 insertions(+)
> >
> > diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> > index 8d8038538fc4..a12ae26db0e2 100644
> > --- a/drivers/net/virtio_net.c
> > +++ b/drivers/net/virtio_net.c
> > @@ -3560,12 +3560,14 @@ static void free_unused_bufs(struct virtnet_info *vi)
> > struct virtqueue *vq = vi->sq[i].vq;
> > while ((buf = virtqueue_detach_unused_buf(vq)) != NULL)
> > virtnet_sq_free_unused_buf(vq, buf);
> > + cond_resched();
>
> Does this really address the case when the virtqueue is very large?
>
> Thanks
it does in that a very large queue is still just 64k in size.
we might however have 64k of these queues.
> > }
> >
> > for (i = 0; i < vi->max_queue_pairs; i++) {
> > struct virtqueue *vq = vi->rq[i].vq;
> > while ((buf = virtqueue_detach_unused_buf(vq)) != NULL)
> > virtnet_rq_free_unused_buf(vq, buf);
> > + cond_resched();
> > }
> > }
> >
> > --
> > 2.20.1
> >
在 2023/5/7 21:34, Michael S. Tsirkin 写道:
> On Fri, May 05, 2023 at 11:28:25AM +0800, Jason Wang wrote:
>> On Thu, May 4, 2023 at 10:27 AM Wenliang Wang
>> <[email protected]> wrote:
>>> For multi-queue and large ring-size use case, the following error
>>> occurred when free_unused_bufs:
>>> rcu: INFO: rcu_sched self-detected stall on CPU.
>>>
>>> Fixes: 986a4f4d452d ("virtio_net: multiqueue support")
>>> Signed-off-by: Wenliang Wang <[email protected]>
>>> ---
>>> v2:
>>> -add need_resched check.
>>> -apply same logic to sq.
>>> v3:
>>> -use cond_resched instead.
>>> v4:
>>> -add fixes tag
>>> ---
>>> drivers/net/virtio_net.c | 2 ++
>>> 1 file changed, 2 insertions(+)
>>>
>>> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
>>> index 8d8038538fc4..a12ae26db0e2 100644
>>> --- a/drivers/net/virtio_net.c
>>> +++ b/drivers/net/virtio_net.c
>>> @@ -3560,12 +3560,14 @@ static void free_unused_bufs(struct virtnet_info *vi)
>>> struct virtqueue *vq = vi->sq[i].vq;
>>> while ((buf = virtqueue_detach_unused_buf(vq)) != NULL)
>>> virtnet_sq_free_unused_buf(vq, buf);
>>> + cond_resched();
>> Does this really address the case when the virtqueue is very large?
>>
>> Thanks
>
> it does in that a very large queue is still just 64k in size.
> we might however have 64k of these queues.
Ok, but we have other similar loops especially the refill, I think we
may need cond_resched() there as well.
Thanks
>
>>> }
>>>
>>> for (i = 0; i < vi->max_queue_pairs; i++) {
>>> struct virtqueue *vq = vi->rq[i].vq;
>>> while ((buf = virtqueue_detach_unused_buf(vq)) != NULL)
>>> virtnet_rq_free_unused_buf(vq, buf);
>>> + cond_resched();
>>> }
>>> }
>>>
>>> --
>>> 2.20.1
>>>
On Mon, May 08, 2023 at 11:12:03AM +0800, Jason Wang wrote:
>
> 在 2023/5/7 21:34, Michael S. Tsirkin 写道:
> > On Fri, May 05, 2023 at 11:28:25AM +0800, Jason Wang wrote:
> > > On Thu, May 4, 2023 at 10:27 AM Wenliang Wang
> > > <[email protected]> wrote:
> > > > For multi-queue and large ring-size use case, the following error
> > > > occurred when free_unused_bufs:
> > > > rcu: INFO: rcu_sched self-detected stall on CPU.
> > > >
> > > > Fixes: 986a4f4d452d ("virtio_net: multiqueue support")
> > > > Signed-off-by: Wenliang Wang <[email protected]>
> > > > ---
> > > > v2:
> > > > -add need_resched check.
> > > > -apply same logic to sq.
> > > > v3:
> > > > -use cond_resched instead.
> > > > v4:
> > > > -add fixes tag
> > > > ---
> > > > drivers/net/virtio_net.c | 2 ++
> > > > 1 file changed, 2 insertions(+)
> > > >
> > > > diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> > > > index 8d8038538fc4..a12ae26db0e2 100644
> > > > --- a/drivers/net/virtio_net.c
> > > > +++ b/drivers/net/virtio_net.c
> > > > @@ -3560,12 +3560,14 @@ static void free_unused_bufs(struct virtnet_info *vi)
> > > > struct virtqueue *vq = vi->sq[i].vq;
> > > > while ((buf = virtqueue_detach_unused_buf(vq)) != NULL)
> > > > virtnet_sq_free_unused_buf(vq, buf);
> > > > + cond_resched();
> > > Does this really address the case when the virtqueue is very large?
> > >
> > > Thanks
> >
> > it does in that a very large queue is still just 64k in size.
> > we might however have 64k of these queues.
>
>
> Ok, but we have other similar loops especially the refill, I think we may
> need cond_resched() there as well.
>
> Thanks
>
Refill is already per vq isn't it?
> >
> > > > }
> > > >
> > > > for (i = 0; i < vi->max_queue_pairs; i++) {
> > > > struct virtqueue *vq = vi->rq[i].vq;
> > > > while ((buf = virtqueue_detach_unused_buf(vq)) != NULL)
> > > > virtnet_rq_free_unused_buf(vq, buf);
> > > > + cond_resched();
> > > > }
> > > > }
> > > >
> > > > --
> > > > 2.20.1
> > > >
On Mon, May 8, 2023 at 2:08 PM Michael S. Tsirkin <[email protected]> wrote:
>
> On Mon, May 08, 2023 at 11:12:03AM +0800, Jason Wang wrote:
> >
> > 在 2023/5/7 21:34, Michael S. Tsirkin 写道:
> > > On Fri, May 05, 2023 at 11:28:25AM +0800, Jason Wang wrote:
> > > > On Thu, May 4, 2023 at 10:27 AM Wenliang Wang
> > > > <[email protected]> wrote:
> > > > > For multi-queue and large ring-size use case, the following error
> > > > > occurred when free_unused_bufs:
> > > > > rcu: INFO: rcu_sched self-detected stall on CPU.
> > > > >
> > > > > Fixes: 986a4f4d452d ("virtio_net: multiqueue support")
> > > > > Signed-off-by: Wenliang Wang <[email protected]>
> > > > > ---
> > > > > v2:
> > > > > -add need_resched check.
> > > > > -apply same logic to sq.
> > > > > v3:
> > > > > -use cond_resched instead.
> > > > > v4:
> > > > > -add fixes tag
> > > > > ---
> > > > > drivers/net/virtio_net.c | 2 ++
> > > > > 1 file changed, 2 insertions(+)
> > > > >
> > > > > diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> > > > > index 8d8038538fc4..a12ae26db0e2 100644
> > > > > --- a/drivers/net/virtio_net.c
> > > > > +++ b/drivers/net/virtio_net.c
> > > > > @@ -3560,12 +3560,14 @@ static void free_unused_bufs(struct virtnet_info *vi)
> > > > > struct virtqueue *vq = vi->sq[i].vq;
> > > > > while ((buf = virtqueue_detach_unused_buf(vq)) != NULL)
> > > > > virtnet_sq_free_unused_buf(vq, buf);
> > > > > + cond_resched();
> > > > Does this really address the case when the virtqueue is very large?
> > > >
> > > > Thanks
> > >
> > > it does in that a very large queue is still just 64k in size.
> > > we might however have 64k of these queues.
> >
> >
> > Ok, but we have other similar loops especially the refill, I think we may
> > need cond_resched() there as well.
> >
> > Thanks
> >
>
> Refill is already per vq isn't it?
Not for the refill_work().
Thanks
>
>
> > >
> > > > > }
> > > > >
> > > > > for (i = 0; i < vi->max_queue_pairs; i++) {
> > > > > struct virtqueue *vq = vi->rq[i].vq;
> > > > > while ((buf = virtqueue_detach_unused_buf(vq)) != NULL)
> > > > > virtnet_rq_free_unused_buf(vq, buf);
> > > > > + cond_resched();
> > > > > }
> > > > > }
> > > > >
> > > > > --
> > > > > 2.20.1
> > > > >
>
On Mon, May 08, 2023 at 02:13:42PM +0800, Jason Wang wrote:
> On Mon, May 8, 2023 at 2:08 PM Michael S. Tsirkin <[email protected]> wrote:
> >
> > On Mon, May 08, 2023 at 11:12:03AM +0800, Jason Wang wrote:
> > >
> > > 在 2023/5/7 21:34, Michael S. Tsirkin 写道:
> > > > On Fri, May 05, 2023 at 11:28:25AM +0800, Jason Wang wrote:
> > > > > On Thu, May 4, 2023 at 10:27 AM Wenliang Wang
> > > > > <[email protected]> wrote:
> > > > > > For multi-queue and large ring-size use case, the following error
> > > > > > occurred when free_unused_bufs:
> > > > > > rcu: INFO: rcu_sched self-detected stall on CPU.
> > > > > >
> > > > > > Fixes: 986a4f4d452d ("virtio_net: multiqueue support")
> > > > > > Signed-off-by: Wenliang Wang <[email protected]>
> > > > > > ---
> > > > > > v2:
> > > > > > -add need_resched check.
> > > > > > -apply same logic to sq.
> > > > > > v3:
> > > > > > -use cond_resched instead.
> > > > > > v4:
> > > > > > -add fixes tag
> > > > > > ---
> > > > > > drivers/net/virtio_net.c | 2 ++
> > > > > > 1 file changed, 2 insertions(+)
> > > > > >
> > > > > > diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> > > > > > index 8d8038538fc4..a12ae26db0e2 100644
> > > > > > --- a/drivers/net/virtio_net.c
> > > > > > +++ b/drivers/net/virtio_net.c
> > > > > > @@ -3560,12 +3560,14 @@ static void free_unused_bufs(struct virtnet_info *vi)
> > > > > > struct virtqueue *vq = vi->sq[i].vq;
> > > > > > while ((buf = virtqueue_detach_unused_buf(vq)) != NULL)
> > > > > > virtnet_sq_free_unused_buf(vq, buf);
> > > > > > + cond_resched();
> > > > > Does this really address the case when the virtqueue is very large?
> > > > >
> > > > > Thanks
> > > >
> > > > it does in that a very large queue is still just 64k in size.
> > > > we might however have 64k of these queues.
> > >
> > >
> > > Ok, but we have other similar loops especially the refill, I think we may
> > > need cond_resched() there as well.
> > >
> > > Thanks
> > >
> >
> > Refill is already per vq isn't it?
>
> Not for the refill_work().
>
> Thanks
Good point, refill_work probably needs cond_resched, too.
And I guess virtnet_open?
> >
> >
> > > >
> > > > > > }
> > > > > >
> > > > > > for (i = 0; i < vi->max_queue_pairs; i++) {
> > > > > > struct virtqueue *vq = vi->rq[i].vq;
> > > > > > while ((buf = virtqueue_detach_unused_buf(vq)) != NULL)
> > > > > > virtnet_rq_free_unused_buf(vq, buf);
> > > > > > + cond_resched();
> > > > > > }
> > > > > > }
> > > > > >
> > > > > > --
> > > > > > 2.20.1
> > > > > >
> >
On Mon, May 8, 2023 at 2:47 PM Michael S. Tsirkin <[email protected]> wrote:
>
> On Mon, May 08, 2023 at 02:13:42PM +0800, Jason Wang wrote:
> > On Mon, May 8, 2023 at 2:08 PM Michael S. Tsirkin <[email protected]> wrote:
> > >
> > > On Mon, May 08, 2023 at 11:12:03AM +0800, Jason Wang wrote:
> > > >
> > > > 在 2023/5/7 21:34, Michael S. Tsirkin 写道:
> > > > > On Fri, May 05, 2023 at 11:28:25AM +0800, Jason Wang wrote:
> > > > > > On Thu, May 4, 2023 at 10:27 AM Wenliang Wang
> > > > > > <[email protected]> wrote:
> > > > > > > For multi-queue and large ring-size use case, the following error
> > > > > > > occurred when free_unused_bufs:
> > > > > > > rcu: INFO: rcu_sched self-detected stall on CPU.
> > > > > > >
> > > > > > > Fixes: 986a4f4d452d ("virtio_net: multiqueue support")
> > > > > > > Signed-off-by: Wenliang Wang <[email protected]>
> > > > > > > ---
> > > > > > > v2:
> > > > > > > -add need_resched check.
> > > > > > > -apply same logic to sq.
> > > > > > > v3:
> > > > > > > -use cond_resched instead.
> > > > > > > v4:
> > > > > > > -add fixes tag
> > > > > > > ---
> > > > > > > drivers/net/virtio_net.c | 2 ++
> > > > > > > 1 file changed, 2 insertions(+)
> > > > > > >
> > > > > > > diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> > > > > > > index 8d8038538fc4..a12ae26db0e2 100644
> > > > > > > --- a/drivers/net/virtio_net.c
> > > > > > > +++ b/drivers/net/virtio_net.c
> > > > > > > @@ -3560,12 +3560,14 @@ static void free_unused_bufs(struct virtnet_info *vi)
> > > > > > > struct virtqueue *vq = vi->sq[i].vq;
> > > > > > > while ((buf = virtqueue_detach_unused_buf(vq)) != NULL)
> > > > > > > virtnet_sq_free_unused_buf(vq, buf);
> > > > > > > + cond_resched();
> > > > > > Does this really address the case when the virtqueue is very large?
> > > > > >
> > > > > > Thanks
> > > > >
> > > > > it does in that a very large queue is still just 64k in size.
> > > > > we might however have 64k of these queues.
> > > >
> > > >
> > > > Ok, but we have other similar loops especially the refill, I think we may
> > > > need cond_resched() there as well.
> > > >
> > > > Thanks
> > > >
> > >
> > > Refill is already per vq isn't it?
> >
> > Not for the refill_work().
> >
> > Thanks
>
> Good point, refill_work probably needs cond_resched, too.
> And I guess virtnet_open?
Yes, let me draft a patch.
Thanks
>
>
> > >
> > >
> > > > >
> > > > > > > }
> > > > > > >
> > > > > > > for (i = 0; i < vi->max_queue_pairs; i++) {
> > > > > > > struct virtqueue *vq = vi->rq[i].vq;
> > > > > > > while ((buf = virtqueue_detach_unused_buf(vq)) != NULL)
> > > > > > > virtnet_rq_free_unused_buf(vq, buf);
> > > > > > > + cond_resched();
> > > > > > > }
> > > > > > > }
> > > > > > >
> > > > > > > --
> > > > > > > 2.20.1
> > > > > > >
> > >
>