Hi,
This contains a fix for "SRCU: kworker hung in synchronize_srcu":
http://lore.kernel.org/CANZk6aR+CqZaqmMWrC2eRRPY12qAZnDZLwLnHZbNi=xXMB401g@mail.gmail.com
And a few cleanups.
Passed 50 hours of SRCU-P and SRCU-N.
git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
srcu/fixes
HEAD: 7ea5adc5673b42ef06e811dca75e43d558cc87e0
Thanks,
Frederic
---
Frederic Weisbecker (5):
srcu: Fix callbacks acceleration mishandling
srcu: Only accelerate on enqueue time
srcu: Remove superfluous callbacks advancing from srcu_start_gp()
srcu: No need to advance/accelerate if no callback enqueued
srcu: Explain why callbacks invocations can't run concurrently
kernel/rcu/srcutree.c | 55 ++++++++++++++++++++++++++++++++++++---------------
1 file changed, 39 insertions(+), 16 deletions(-)
On Wed, Oct 04, 2023 at 01:28:58AM +0200, Frederic Weisbecker wrote:
> Hi,
>
> This contains a fix for "SRCU: kworker hung in synchronize_srcu":
>
> http://lore.kernel.org/CANZk6aR+CqZaqmMWrC2eRRPY12qAZnDZLwLnHZbNi=xXMB401g@mail.gmail.com
>
> And a few cleanups.
>
> Passed 50 hours of SRCU-P and SRCU-N.
>
> git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
> srcu/fixes
>
> HEAD: 7ea5adc5673b42ef06e811dca75e43d558cc87e0
>
> Thanks,
> Frederic
Very good, and a big "Thank You!!!" to all of you!
I queued this series for testing purposes, and have started a bunch of
SRCU-P and SRCU-N tests on one set of systems, and a single SRCU-P and
SRCU-N on another system, but with both scenarios resized to 40 CPU each.
While that is in flight, a few questions:
o Please check the Co-developed-by rules. Last I knew, it was
necessary to have a Signed-off-by after each Co-developed-by.
o Is it possible to get a Tested-by from the original reporter?
Or is this not reproducible?
o Is it possible to convince rcutorture to find this sort of
bug? Seems like it should be, but easy to say...
o Frederic, would you like to include this in your upcoming
pull request? Or does it need more time?
Thanx, Paul
> ---
>
> Frederic Weisbecker (5):
> srcu: Fix callbacks acceleration mishandling
> srcu: Only accelerate on enqueue time
> srcu: Remove superfluous callbacks advancing from srcu_start_gp()
> srcu: No need to advance/accelerate if no callback enqueued
> srcu: Explain why callbacks invocations can't run concurrently
>
>
> kernel/rcu/srcutree.c | 55 ++++++++++++++++++++++++++++++++++++---------------
> 1 file changed, 39 insertions(+), 16 deletions(-)
On Tue, Oct 03, 2023 at 05:35:31PM -0700, Paul E. McKenney wrote:
> On Wed, Oct 04, 2023 at 01:28:58AM +0200, Frederic Weisbecker wrote:
> > Hi,
> >
> > This contains a fix for "SRCU: kworker hung in synchronize_srcu":
> >
> > http://lore.kernel.org/CANZk6aR+CqZaqmMWrC2eRRPY12qAZnDZLwLnHZbNi=xXMB401g@mail.gmail.com
> >
> > And a few cleanups.
> >
> > Passed 50 hours of SRCU-P and SRCU-N.
> >
> > git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
> > srcu/fixes
> >
> > HEAD: 7ea5adc5673b42ef06e811dca75e43d558cc87e0
> >
> > Thanks,
> > Frederic
>
> Very good, and a big "Thank You!!!" to all of you!
>
> I queued this series for testing purposes, and have started a bunch of
> SRCU-P and SRCU-N tests on one set of systems, and a single SRCU-P and
> SRCU-N on another system, but with both scenarios resized to 40 CPU each.
>
> While that is in flight, a few questions:
>
> o Please check the Co-developed-by rules. Last I knew, it was
> necessary to have a Signed-off-by after each Co-developed-by.
>
> o Is it possible to get a Tested-by from the original reporter?
> Or is this not reproducible?
>
> o Is it possible to convince rcutorture to find this sort of
> bug? Seems like it should be, but easy to say...
And one other thing...
o What other bugs like this one are hiding elsewhere
in RCU?
> o Frederic, would you like to include this in your upcoming
> pull request? Or does it need more time?
Thanx, Paul
> > ---
> >
> > Frederic Weisbecker (5):
> > srcu: Fix callbacks acceleration mishandling
> > srcu: Only accelerate on enqueue time
> > srcu: Remove superfluous callbacks advancing from srcu_start_gp()
> > srcu: No need to advance/accelerate if no callback enqueued
> > srcu: Explain why callbacks invocations can't run concurrently
> >
> >
> > kernel/rcu/srcutree.c | 55 ++++++++++++++++++++++++++++++++++++---------------
> > 1 file changed, 39 insertions(+), 16 deletions(-)
On Tue, Oct 03, 2023 at 08:21:42PM -0700, Paul E. McKenney wrote:
> On Tue, Oct 03, 2023 at 05:35:31PM -0700, Paul E. McKenney wrote:
> > On Wed, Oct 04, 2023 at 01:28:58AM +0200, Frederic Weisbecker wrote:
> > > Hi,
> > >
> > > This contains a fix for "SRCU: kworker hung in synchronize_srcu":
> > >
> > > http://lore.kernel.org/CANZk6aR+CqZaqmMWrC2eRRPY12qAZnDZLwLnHZbNi=xXMB401g@mail.gmail.com
> > >
> > > And a few cleanups.
> > >
> > > Passed 50 hours of SRCU-P and SRCU-N.
> > >
> > > git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
> > > srcu/fixes
> > >
> > > HEAD: 7ea5adc5673b42ef06e811dca75e43d558cc87e0
> > >
> > > Thanks,
> > > Frederic
> >
> > Very good, and a big "Thank You!!!" to all of you!
> >
> > I queued this series for testing purposes, and have started a bunch of
> > SRCU-P and SRCU-N tests on one set of systems, and a single SRCU-P and
> > SRCU-N on another system, but with both scenarios resized to 40 CPU each.
The 200*1h of SRCU-N and the 100*1h of SRCU-p passed other than the usual
tick-stop errors. (Is there a patch for that one?) The 40-CPU SRCU-N
run was fine, but the 40-CPU SRCU-P run failed due to the fanouts setting
a maximum of 16 CPUs. So I started a 10-hour 40-CPU SRCU-P and a pair
of 10-hour 16-CPU SRCU-N runs on one system, and 200*10h of SRCU-N and
100*10h of SRCU-P.
I will let you know how it goes.
Thanx, Paul
> > While that is in flight, a few questions:
> >
> > o Please check the Co-developed-by rules. Last I knew, it was
> > necessary to have a Signed-off-by after each Co-developed-by.
> >
> > o Is it possible to get a Tested-by from the original reporter?
> > Or is this not reproducible?
> >
> > o Is it possible to convince rcutorture to find this sort of
> > bug? Seems like it should be, but easy to say...
>
> And one other thing...
>
> o What other bugs like this one are hiding elsewhere
> in RCU?
>
> > o Frederic, would you like to include this in your upcoming
> > pull request? Or does it need more time?
>
> Thanx, Paul
>
> > > ---
> > >
> > > Frederic Weisbecker (5):
> > > srcu: Fix callbacks acceleration mishandling
> > > srcu: Only accelerate on enqueue time
> > > srcu: Remove superfluous callbacks advancing from srcu_start_gp()
> > > srcu: No need to advance/accelerate if no callback enqueued
> > > srcu: Explain why callbacks invocations can't run concurrently
> > >
> > >
> > > kernel/rcu/srcutree.c | 55 ++++++++++++++++++++++++++++++++++++---------------
> > > 1 file changed, 39 insertions(+), 16 deletions(-)
On Tue, Oct 03, 2023 at 05:35:31PM -0700, Paul E. McKenney wrote:
> On Wed, Oct 04, 2023 at 01:28:58AM +0200, Frederic Weisbecker wrote:
> > Hi,
> >
> > This contains a fix for "SRCU: kworker hung in synchronize_srcu":
> >
> > http://lore.kernel.org/CANZk6aR+CqZaqmMWrC2eRRPY12qAZnDZLwLnHZbNi=xXMB401g@mail.gmail.com
> >
> > And a few cleanups.
> >
> > Passed 50 hours of SRCU-P and SRCU-N.
> >
> > git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
> > srcu/fixes
> >
> > HEAD: 7ea5adc5673b42ef06e811dca75e43d558cc87e0
> >
> > Thanks,
> > Frederic
>
> Very good, and a big "Thank You!!!" to all of you!
>
> I queued this series for testing purposes, and have started a bunch of
> SRCU-P and SRCU-N tests on one set of systems, and a single SRCU-P and
> SRCU-N on another system, but with both scenarios resized to 40 CPU each.
>
> While that is in flight, a few questions:
>
> o Please check the Co-developed-by rules. Last I knew, it was
> necessary to have a Signed-off-by after each Co-developed-by.
Indeed! I'll try to collect the three of them within a few days. If some
are missing, I'll put a Reported-by instead.
>
> o Is it possible to get a Tested-by from the original reporter?
> Or is this not reproducible?
It seems that the issue would trigger rarely. But I hope we can get one.
>
> o Is it possible to convince rcutorture to find this sort of
> bug? Seems like it should be, but easy to say...
So at least the part where advance/accelerate fail is observed from time
to time. But then we must meet two more rare events:
1) The CPU failing to ACC/ADV must also fail to start the grace period because
another CPU was faster.
2) The callbacks invocation must not run until that grace period has ended (even
though we had a previous one completed with callbacks ready).
Or it can run after all but at least the acceleration part of it has to
happen after the end of the new grace period.
Perhaps all these conditions can me met more often if we overcommit the number
of vCPU. For example run 10 SRCU-P instances within 3 real CPUs. This could
introduce random breaks within the torture writers...
Just an idea...
>
> o Frederic, would you like to include this in your upcoming
> pull request? Or does it need more time?
At least the first patch yes. It should be easily backported and
it should be enough to solve the race. I'll just wait a bit to collect
more tags.
Thanks!
>
> Thanx, Paul
>
> > ---
> >
> > Frederic Weisbecker (5):
> > srcu: Fix callbacks acceleration mishandling
> > srcu: Only accelerate on enqueue time
> > srcu: Remove superfluous callbacks advancing from srcu_start_gp()
> > srcu: No need to advance/accelerate if no callback enqueued
> > srcu: Explain why callbacks invocations can't run concurrently
> >
> >
> > kernel/rcu/srcutree.c | 55 ++++++++++++++++++++++++++++++++++++---------------
> > 1 file changed, 39 insertions(+), 16 deletions(-)
On Tue, Oct 03, 2023 at 08:21:42PM -0700, Paul E. McKenney wrote:
> On Tue, Oct 03, 2023 at 05:35:31PM -0700, Paul E. McKenney wrote:
> > On Wed, Oct 04, 2023 at 01:28:58AM +0200, Frederic Weisbecker wrote:
> > > Hi,
> > >
> > > This contains a fix for "SRCU: kworker hung in synchronize_srcu":
> > >
> > > http://lore.kernel.org/CANZk6aR+CqZaqmMWrC2eRRPY12qAZnDZLwLnHZbNi=xXMB401g@mail.gmail.com
> > >
> > > And a few cleanups.
> > >
> > > Passed 50 hours of SRCU-P and SRCU-N.
> > >
> > > git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
> > > srcu/fixes
> > >
> > > HEAD: 7ea5adc5673b42ef06e811dca75e43d558cc87e0
> > >
> > > Thanks,
> > > Frederic
> >
> > Very good, and a big "Thank You!!!" to all of you!
> >
> > I queued this series for testing purposes, and have started a bunch of
> > SRCU-P and SRCU-N tests on one set of systems, and a single SRCU-P and
> > SRCU-N on another system, but with both scenarios resized to 40 CPU each.
> >
> > While that is in flight, a few questions:
> >
> > o Please check the Co-developed-by rules. Last I knew, it was
> > necessary to have a Signed-off-by after each Co-developed-by.
> >
> > o Is it possible to get a Tested-by from the original reporter?
> > Or is this not reproducible?
> >
> > o Is it possible to convince rcutorture to find this sort of
> > bug? Seems like it should be, but easy to say...
>
> And one other thing...
>
> o What other bugs like this one are hiding elsewhere
> in RCU?
Hmm, yesterday I thought RCU would be fine because it has a tick polling
on callbacks anyway. But I'm not so sure, I'll check for real...
Thanks.
>
> > o Frederic, would you like to include this in your upcoming
> > pull request? Or does it need more time?
>
> Thanx, Paul
>
> > > ---
> > >
> > > Frederic Weisbecker (5):
> > > srcu: Fix callbacks acceleration mishandling
> > > srcu: Only accelerate on enqueue time
> > > srcu: Remove superfluous callbacks advancing from srcu_start_gp()
> > > srcu: No need to advance/accelerate if no callback enqueued
> > > srcu: Explain why callbacks invocations can't run concurrently
> > >
> > >
> > > kernel/rcu/srcutree.c | 55 ++++++++++++++++++++++++++++++++++++---------------
> > > 1 file changed, 39 insertions(+), 16 deletions(-)
On Tue, Oct 03, 2023 at 08:30:45PM -0700, Paul E. McKenney wrote:
> On Tue, Oct 03, 2023 at 08:21:42PM -0700, Paul E. McKenney wrote:
> > On Tue, Oct 03, 2023 at 05:35:31PM -0700, Paul E. McKenney wrote:
> > > On Wed, Oct 04, 2023 at 01:28:58AM +0200, Frederic Weisbecker wrote:
> > > > Hi,
> > > >
> > > > This contains a fix for "SRCU: kworker hung in synchronize_srcu":
> > > >
> > > > http://lore.kernel.org/CANZk6aR+CqZaqmMWrC2eRRPY12qAZnDZLwLnHZbNi=xXMB401g@mail.gmail.com
> > > >
> > > > And a few cleanups.
> > > >
> > > > Passed 50 hours of SRCU-P and SRCU-N.
> > > >
> > > > git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
> > > > srcu/fixes
> > > >
> > > > HEAD: 7ea5adc5673b42ef06e811dca75e43d558cc87e0
> > > >
> > > > Thanks,
> > > > Frederic
> > >
> > > Very good, and a big "Thank You!!!" to all of you!
> > >
> > > I queued this series for testing purposes, and have started a bunch of
> > > SRCU-P and SRCU-N tests on one set of systems, and a single SRCU-P and
> > > SRCU-N on another system, but with both scenarios resized to 40 CPU each.
>
> The 200*1h of SRCU-N and the 100*1h of SRCU-p passed other than the usual
> tick-stop errors. (Is there a patch for that one?) The 40-CPU SRCU-N
> run was fine, but the 40-CPU SRCU-P run failed due to the fanouts setting
> a maximum of 16 CPUs. So I started a 10-hour 40-CPU SRCU-P and a pair
> of 10-hour 16-CPU SRCU-N runs on one system, and 200*10h of SRCU-N and
> 100*10h of SRCU-P.
>
> I will let you know how it goes.
Very nice! It might be worth testing the first patch alone as
well if we backport only this one.
Thanks!
> Thanx, Paul
>
> > > While that is in flight, a few questions:
> > >
> > > o Please check the Co-developed-by rules. Last I knew, it was
> > > necessary to have a Signed-off-by after each Co-developed-by.
> > >
> > > o Is it possible to get a Tested-by from the original reporter?
> > > Or is this not reproducible?
> > >
> > > o Is it possible to convince rcutorture to find this sort of
> > > bug? Seems like it should be, but easy to say...
> >
> > And one other thing...
> >
> > o What other bugs like this one are hiding elsewhere
> > in RCU?
> >
> > > o Frederic, would you like to include this in your upcoming
> > > pull request? Or does it need more time?
> >
> > Thanx, Paul
> >
> > > > ---
> > > >
> > > > Frederic Weisbecker (5):
> > > > srcu: Fix callbacks acceleration mishandling
> > > > srcu: Only accelerate on enqueue time
> > > > srcu: Remove superfluous callbacks advancing from srcu_start_gp()
> > > > srcu: No need to advance/accelerate if no callback enqueued
> > > > srcu: Explain why callbacks invocations can't run concurrently
> > > >
> > > >
> > > > kernel/rcu/srcutree.c | 55 ++++++++++++++++++++++++++++++++++++---------------
> > > > 1 file changed, 39 insertions(+), 16 deletions(-)
On Wed, Oct 04, 2023 at 11:36:49AM +0200, Frederic Weisbecker wrote:
> On Tue, Oct 03, 2023 at 08:30:45PM -0700, Paul E. McKenney wrote:
> > On Tue, Oct 03, 2023 at 08:21:42PM -0700, Paul E. McKenney wrote:
> > > On Tue, Oct 03, 2023 at 05:35:31PM -0700, Paul E. McKenney wrote:
> > > > On Wed, Oct 04, 2023 at 01:28:58AM +0200, Frederic Weisbecker wrote:
> > > > > Hi,
> > > > >
> > > > > This contains a fix for "SRCU: kworker hung in synchronize_srcu":
> > > > >
> > > > > http://lore.kernel.org/CANZk6aR+CqZaqmMWrC2eRRPY12qAZnDZLwLnHZbNi=xXMB401g@mail.gmail.com
> > > > >
> > > > > And a few cleanups.
> > > > >
> > > > > Passed 50 hours of SRCU-P and SRCU-N.
> > > > >
> > > > > git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
> > > > > srcu/fixes
> > > > >
> > > > > HEAD: 7ea5adc5673b42ef06e811dca75e43d558cc87e0
> > > > >
> > > > > Thanks,
> > > > > Frederic
> > > >
> > > > Very good, and a big "Thank You!!!" to all of you!
> > > >
> > > > I queued this series for testing purposes, and have started a bunch of
> > > > SRCU-P and SRCU-N tests on one set of systems, and a single SRCU-P and
> > > > SRCU-N on another system, but with both scenarios resized to 40 CPU each.
> >
> > The 200*1h of SRCU-N and the 100*1h of SRCU-p passed other than the usual
> > tick-stop errors. (Is there a patch for that one?) The 40-CPU SRCU-N
> > run was fine, but the 40-CPU SRCU-P run failed due to the fanouts setting
> > a maximum of 16 CPUs. So I started a 10-hour 40-CPU SRCU-P and a pair
> > of 10-hour 16-CPU SRCU-N runs on one system, and 200*10h of SRCU-N and
> > 100*10h of SRCU-P.
> >
> > I will let you know how it goes.
>
> Very nice! It might be worth testing the first patch alone as
> well if we backport only this one.
The 10-hour 40-CPU SRCU-P run and pair of 10-hour 16-CPU SRCU-N runs
completed without failure. The others had some failures, but I need
to look and see if any were unexpected. In the meantime, I started a
two-hour 40-CPU SRCU-P run and a pair of one-hour 16-CPU SRCU-N runs on
just that first commit. Also servicing SIGSHOWER and SIGFOOD. ;-)
Thanx, Paul
> Thanks!
>
>
> > Thanx, Paul
> >
> > > > While that is in flight, a few questions:
> > > >
> > > > o Please check the Co-developed-by rules. Last I knew, it was
> > > > necessary to have a Signed-off-by after each Co-developed-by.
> > > >
> > > > o Is it possible to get a Tested-by from the original reporter?
> > > > Or is this not reproducible?
> > > >
> > > > o Is it possible to convince rcutorture to find this sort of
> > > > bug? Seems like it should be, but easy to say...
> > >
> > > And one other thing...
> > >
> > > o What other bugs like this one are hiding elsewhere
> > > in RCU?
> > >
> > > > o Frederic, would you like to include this in your upcoming
> > > > pull request? Or does it need more time?
> > >
> > > Thanx, Paul
> > >
> > > > > ---
> > > > >
> > > > > Frederic Weisbecker (5):
> > > > > srcu: Fix callbacks acceleration mishandling
> > > > > srcu: Only accelerate on enqueue time
> > > > > srcu: Remove superfluous callbacks advancing from srcu_start_gp()
> > > > > srcu: No need to advance/accelerate if no callback enqueued
> > > > > srcu: Explain why callbacks invocations can't run concurrently
> > > > >
> > > > >
> > > > > kernel/rcu/srcutree.c | 55 ++++++++++++++++++++++++++++++++++++---------------
> > > > > 1 file changed, 39 insertions(+), 16 deletions(-)
On Wed, Oct 04, 2023 at 07:06:58AM -0700, Paul E. McKenney wrote:
> On Wed, Oct 04, 2023 at 11:36:49AM +0200, Frederic Weisbecker wrote:
> > On Tue, Oct 03, 2023 at 08:30:45PM -0700, Paul E. McKenney wrote:
> > > On Tue, Oct 03, 2023 at 08:21:42PM -0700, Paul E. McKenney wrote:
> > > > On Tue, Oct 03, 2023 at 05:35:31PM -0700, Paul E. McKenney wrote:
> > > > > On Wed, Oct 04, 2023 at 01:28:58AM +0200, Frederic Weisbecker wrote:
> > > > > > Hi,
> > > > > >
> > > > > > This contains a fix for "SRCU: kworker hung in synchronize_srcu":
> > > > > >
> > > > > > http://lore.kernel.org/CANZk6aR+CqZaqmMWrC2eRRPY12qAZnDZLwLnHZbNi=xXMB401g@mail.gmail.com
> > > > > >
> > > > > > And a few cleanups.
> > > > > >
> > > > > > Passed 50 hours of SRCU-P and SRCU-N.
> > > > > >
> > > > > > git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
> > > > > > srcu/fixes
> > > > > >
> > > > > > HEAD: 7ea5adc5673b42ef06e811dca75e43d558cc87e0
> > > > > >
> > > > > > Thanks,
> > > > > > Frederic
> > > > >
> > > > > Very good, and a big "Thank You!!!" to all of you!
> > > > >
> > > > > I queued this series for testing purposes, and have started a bunch of
> > > > > SRCU-P and SRCU-N tests on one set of systems, and a single SRCU-P and
> > > > > SRCU-N on another system, but with both scenarios resized to 40 CPU each.
> > >
> > > The 200*1h of SRCU-N and the 100*1h of SRCU-p passed other than the usual
> > > tick-stop errors. (Is there a patch for that one?) The 40-CPU SRCU-N
> > > run was fine, but the 40-CPU SRCU-P run failed due to the fanouts setting
> > > a maximum of 16 CPUs. So I started a 10-hour 40-CPU SRCU-P and a pair
> > > of 10-hour 16-CPU SRCU-N runs on one system, and 200*10h of SRCU-N and
> > > 100*10h of SRCU-P.
> > >
> > > I will let you know how it goes.
> >
> > Very nice! It might be worth testing the first patch alone as
> > well if we backport only this one.
>
> The 10-hour 40-CPU SRCU-P run and pair of 10-hour 16-CPU SRCU-N runs
> completed without failure. The others had some failures, but I need
> to look and see if any were unexpected. In the meantime, I started a
> two-hour 40-CPU SRCU-P run and a pair of one-hour 16-CPU SRCU-N runs on
> just that first commit. Also servicing SIGSHOWER and SIGFOOD. ;-)
And the two-hour 40-CPU SRCU-P run and a pair of two-hour 16-CPU SRCU-N
runs (on only the first commit) completed without incident.
The other set of overnight full-stack runs had only tick-stop errors,
so I started a two-hour set on the first commit.
So far so good!
Thanx, Paul
> > Thanks!
> >
> >
> > > Thanx, Paul
> > >
> > > > > While that is in flight, a few questions:
> > > > >
> > > > > o Please check the Co-developed-by rules. Last I knew, it was
> > > > > necessary to have a Signed-off-by after each Co-developed-by.
> > > > >
> > > > > o Is it possible to get a Tested-by from the original reporter?
> > > > > Or is this not reproducible?
> > > > >
> > > > > o Is it possible to convince rcutorture to find this sort of
> > > > > bug? Seems like it should be, but easy to say...
> > > >
> > > > And one other thing...
> > > >
> > > > o What other bugs like this one are hiding elsewhere
> > > > in RCU?
> > > >
> > > > > o Frederic, would you like to include this in your upcoming
> > > > > pull request? Or does it need more time?
> > > >
> > > > Thanx, Paul
> > > >
> > > > > > ---
> > > > > >
> > > > > > Frederic Weisbecker (5):
> > > > > > srcu: Fix callbacks acceleration mishandling
> > > > > > srcu: Only accelerate on enqueue time
> > > > > > srcu: Remove superfluous callbacks advancing from srcu_start_gp()
> > > > > > srcu: No need to advance/accelerate if no callback enqueued
> > > > > > srcu: Explain why callbacks invocations can't run concurrently
> > > > > >
> > > > > >
> > > > > > kernel/rcu/srcutree.c | 55 ++++++++++++++++++++++++++++++++++++---------------
> > > > > > 1 file changed, 39 insertions(+), 16 deletions(-)
Le Wed, Oct 04, 2023 at 09:47:04AM -0700, Paul E. McKenney a ?crit :
> > The 10-hour 40-CPU SRCU-P run and pair of 10-hour 16-CPU SRCU-N runs
> > completed without failure. The others had some failures, but I need
> > to look and see if any were unexpected. In the meantime, I started a
> > two-hour 40-CPU SRCU-P run and a pair of one-hour 16-CPU SRCU-N runs on
> > just that first commit. Also servicing SIGSHOWER and SIGFOOD. ;-)
>
> And the two-hour 40-CPU SRCU-P run and a pair of two-hour 16-CPU SRCU-N
> runs (on only the first commit) completed without incident.
>
> The other set of overnight full-stack runs had only tick-stop errors,
> so I started a two-hour set on the first commit.
>
> So far so good!
Very nice!
As for the tick-stop error, see the upstream fix:
1a6a46477494 ("timers: Tag (hr)timer softirq as hotplug safe")
Thanks!
On Wed, Oct 04, 2023 at 11:27:29PM +0200, Frederic Weisbecker wrote:
> Le Wed, Oct 04, 2023 at 09:47:04AM -0700, Paul E. McKenney a ?crit :
> > > The 10-hour 40-CPU SRCU-P run and pair of 10-hour 16-CPU SRCU-N runs
> > > completed without failure. The others had some failures, but I need
> > > to look and see if any were unexpected. In the meantime, I started a
> > > two-hour 40-CPU SRCU-P run and a pair of one-hour 16-CPU SRCU-N runs on
> > > just that first commit. Also servicing SIGSHOWER and SIGFOOD. ;-)
> >
> > And the two-hour 40-CPU SRCU-P run and a pair of two-hour 16-CPU SRCU-N
> > runs (on only the first commit) completed without incident.
> >
> > The other set of overnight full-stack runs had only tick-stop errors,
> > so I started a two-hour set on the first commit.
> >
> > So far so good!
>
> Very nice!
>
> As for the tick-stop error, see the upstream fix:
>
> 1a6a46477494 ("timers: Tag (hr)timer softirq as hotplug safe")
Got it, thank you!
And the two-hour set of 200*SRCU-N and 100*SRCU-P had only tick-stop
errors. I am refreshing the test grid, and will run overnight.
Here is hoping!
Thanx, Paul
On Wed, Oct 04, 2023 at 02:54:57PM -0700, Paul E. McKenney wrote:
> On Wed, Oct 04, 2023 at 11:27:29PM +0200, Frederic Weisbecker wrote:
> > Le Wed, Oct 04, 2023 at 09:47:04AM -0700, Paul E. McKenney a ?crit :
> > > > The 10-hour 40-CPU SRCU-P run and pair of 10-hour 16-CPU SRCU-N runs
> > > > completed without failure. The others had some failures, but I need
> > > > to look and see if any were unexpected. In the meantime, I started a
> > > > two-hour 40-CPU SRCU-P run and a pair of one-hour 16-CPU SRCU-N runs on
> > > > just that first commit. Also servicing SIGSHOWER and SIGFOOD. ;-)
> > >
> > > And the two-hour 40-CPU SRCU-P run and a pair of two-hour 16-CPU SRCU-N
> > > runs (on only the first commit) completed without incident.
> > >
> > > The other set of overnight full-stack runs had only tick-stop errors,
> > > so I started a two-hour set on the first commit.
> > >
> > > So far so good!
> >
> > Very nice!
> >
> > As for the tick-stop error, see the upstream fix:
> >
> > 1a6a46477494 ("timers: Tag (hr)timer softirq as hotplug safe")
>
> Got it, thank you!
>
> And the two-hour set of 200*SRCU-N and 100*SRCU-P had only tick-stop
> errors. I am refreshing the test grid, and will run overnight.
And the ten-hour test passed with only the tick-stop errors, representing
2000 hours of SRCU-N and 1000 hours of SRCU-P. (I ran the exact same
stack, without the rebased fix you call out above.)
Looking good!
Thanx, Paul
On Wed, Oct 4, 2023 at 5:25 PM Frederic Weisbecker <[email protected]> wrote:
>
> On Tue, Oct 03, 2023 at 05:35:31PM -0700, Paul E. McKenney wrote:
> > On Wed, Oct 04, 2023 at 01:28:58AM +0200, Frederic Weisbecker wrote:
> > > Hi,
> > >
> > > This contains a fix for "SRCU: kworker hung in synchronize_srcu":
> > >
> > > http://lore.kernel.org/CANZk6aR+CqZaqmMWrC2eRRPY12qAZnDZLwLnHZbNi=xXMB401g@mail.gmail.com
> > >
> > > And a few cleanups.
> > >
> > > Passed 50 hours of SRCU-P and SRCU-N.
> > >
> > > git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
> > > srcu/fixes
> > >
> > > HEAD: 7ea5adc5673b42ef06e811dca75e43d558cc87e0
> > >
> > > Thanks,
> > > Frederic
> >
> > Very good, and a big "Thank You!!!" to all of you!
> >
> > I queued this series for testing purposes, and have started a bunch of
> > SRCU-P and SRCU-N tests on one set of systems, and a single SRCU-P and
> > SRCU-N on another system, but with both scenarios resized to 40 CPU each.
> >
> > While that is in flight, a few questions:
> >
> > o Please check the Co-developed-by rules. Last I knew, it was
> > necessary to have a Signed-off-by after each Co-developed-by.
>
> Indeed! I'll try to collect the three of them within a few days. If some
> are missing, I'll put a Reported-by instead.
>
> >
> > o Is it possible to get a Tested-by from the original reporter?
> > Or is this not reproducible?
>
> It seems that the issue would trigger rarely. But I hope we can get one.
There is currently no way to reproduce this problem in our environment.
The problem has appeared on 2 machines, and each time it occurred, the
test had been running for more than a month.
BTW, I will run tests with these patches in our environment.
>
> >
> > o Is it possible to convince rcutorture to find this sort of
> > bug? Seems like it should be, but easy to say...
>
> So at least the part where advance/accelerate fail is observed from time
> to time. But then we must meet two more rare events:
>
> 1) The CPU failing to ACC/ADV must also fail to start the grace period because
> another CPU was faster.
>
> 2) The callbacks invocation must not run until that grace period has ended (even
> though we had a previous one completed with callbacks ready).
>
> Or it can run after all but at least the acceleration part of it has to
> happen after the end of the new grace period.
>
> Perhaps all these conditions can me met more often if we overcommit the number
> of vCPU. For example run 10 SRCU-P instances within 3 real CPUs. This could
> introduce random breaks within the torture writers...
>
> Just an idea...
>
> >
> > o Frederic, would you like to include this in your upcoming
> > pull request? Or does it need more time?
>
> At least the first patch yes. It should be easily backported and
> it should be enough to solve the race. I'll just wait a bit to collect
> more tags.
>
> Thanks!
>
> >
> > Thanx, Paul
> >
> > > ---
> > >
> > > Frederic Weisbecker (5):
> > > srcu: Fix callbacks acceleration mishandling
> > > srcu: Only accelerate on enqueue time
> > > srcu: Remove superfluous callbacks advancing from srcu_start_gp()
> > > srcu: No need to advance/accelerate if no callback enqueued
> > > srcu: Explain why callbacks invocations can't run concurrently
> > >
> > >
> > > kernel/rcu/srcutree.c | 55 ++++++++++++++++++++++++++++++++++++---------------
> > > 1 file changed, 39 insertions(+), 16 deletions(-)
On Thu, Oct 05, 2023 at 09:54:12AM -0700, Paul E. McKenney wrote:
> On Wed, Oct 04, 2023 at 02:54:57PM -0700, Paul E. McKenney wrote:
> > On Wed, Oct 04, 2023 at 11:27:29PM +0200, Frederic Weisbecker wrote:
> > > Le Wed, Oct 04, 2023 at 09:47:04AM -0700, Paul E. McKenney a ?crit :
> > > > > The 10-hour 40-CPU SRCU-P run and pair of 10-hour 16-CPU SRCU-N runs
> > > > > completed without failure. The others had some failures, but I need
> > > > > to look and see if any were unexpected. In the meantime, I started a
> > > > > two-hour 40-CPU SRCU-P run and a pair of one-hour 16-CPU SRCU-N runs on
> > > > > just that first commit. Also servicing SIGSHOWER and SIGFOOD. ;-)
> > > >
> > > > And the two-hour 40-CPU SRCU-P run and a pair of two-hour 16-CPU SRCU-N
> > > > runs (on only the first commit) completed without incident.
> > > >
> > > > The other set of overnight full-stack runs had only tick-stop errors,
> > > > so I started a two-hour set on the first commit.
> > > >
> > > > So far so good!
> > >
> > > Very nice!
> > >
> > > As for the tick-stop error, see the upstream fix:
> > >
> > > 1a6a46477494 ("timers: Tag (hr)timer softirq as hotplug safe")
> >
> > Got it, thank you!
> >
> > And the two-hour set of 200*SRCU-N and 100*SRCU-P had only tick-stop
> > errors. I am refreshing the test grid, and will run overnight.
>
> And the ten-hour test passed with only the tick-stop errors, representing
> 2000 hours of SRCU-N and 1000 hours of SRCU-P. (I ran the exact same
> stack, without the rebased fix you call out above.)
Thanks a lot!
On Sat, Oct 07, 2023 at 06:24:53PM +0800, zhuangel570 wrote:
> On Wed, Oct 4, 2023 at 5:25 PM Frederic Weisbecker <[email protected]> wrote:
> There is currently no way to reproduce this problem in our environment.
> The problem has appeared on 2 machines, and each time it occurred, the
> test had been running for more than a month.
>
> BTW, I will run tests with these patches in our environment.
Ok, let us know if it ever triggers after this series.
Since I added you in the Co-developed-by: tags, I will need to
also add your Signed-off-by: tag.
Is that ok for you?
Thanks.
On Tue, Oct 10, 2023 at 7:27 PM Frederic Weisbecker <[email protected]> wrote:
>
> On Sat, Oct 07, 2023 at 06:24:53PM +0800, zhuangel570 wrote:
> > On Wed, Oct 4, 2023 at 5:25 PM Frederic Weisbecker <[email protected]> wrote:
> > There is currently no way to reproduce this problem in our environment.
> > The problem has appeared on 2 machines, and each time it occurred, the
> > test had been running for more than a month.
> >
> > BTW, I will run tests with these patches in our environment.
>
> Ok, let us know if it ever triggers after this series.
Sure, I have ported the patch set to our test environment, now 2 machines
already run test for 3 days, everything looks fine.
The patch set is very elegant and completely eliminates the possibility of
unexpected accelerations in our analysis. I am very confident in fixing our
problem.
>
> Since I added you in the Co-developed-by: tags, I will need to
> also add your Signed-off-by: tag.
>
> Is that ok for you?
Sure. Big thanks!
If possible, would you please change my "Signed-off-by" into:
Signed-off-by: Yong He <[email protected]>
>
> Thanks.