The rt boosting in locktorture has a factor variable large enough that
boosting only happens once every minute or so. Add a tunable to educe
the factor so that boosting happens more often, to test paths and arrive
at failure modes earlier. With this change, I can set the factor to
like 50 and have the boosting happens every 10 seconds or so.
Tested with boot parameters:
locktorture.torture_type=mutex_lock
locktorture.onoff_interval=1
locktorture.nwriters_stress=8
locktorture.stutter=0
locktorture.rt_boost=1
locktorture.rt_boost_factor=50
locktorture.nlocks=3
Signed-off-by: Joel Fernandes (Google) <[email protected]>
---
kernel/locking/locktorture.c | 12 +++++++-----
1 file changed, 7 insertions(+), 5 deletions(-)
diff --git a/kernel/locking/locktorture.c b/kernel/locking/locktorture.c
index 5a388ac96a9b..e4529c2166e9 100644
--- a/kernel/locking/locktorture.c
+++ b/kernel/locking/locktorture.c
@@ -47,6 +47,7 @@ torture_param(int, stat_interval, 60,
"Number of seconds between stats printk()s");
torture_param(int, stutter, 5, "Number of jiffies to run/halt test, 0=disable");
torture_param(int, rt_boost, 0, "Perform an rt-boost from the writer, always 1 for rtmutex_lock");
+torture_param(int, rt_boost_factor, 50000, "A factor determining how often rt-boost happens");
torture_param(int, verbose, 1,
"Enable verbose debugging printk()s");
torture_param(int, nlocks, 1,
@@ -132,15 +133,15 @@ static void torture_lock_busted_write_unlock(int tid __maybe_unused)
static void torture_rt_boost(struct torture_random_state *trsp)
{
- const unsigned int factor = 50000; /* yes, quite arbitrary */
+ const unsigned int factor = rt_boost_factor; /* yes, quite arbitrary */
if (!rt_boost)
return;
if (!rt_task(current)) {
/*
- * Boost priority once every ~50k operations. When the
- * task tries to take the lock, the rtmutex it will account
+ * Boost priority once every rt_boost_factor operations. When
+ * the task tries to take the lock, the rtmutex it will account
* for the new priority, and do any corresponding pi-dance.
*/
if (trsp && !(torture_random(trsp) %
@@ -150,8 +151,9 @@ static void torture_rt_boost(struct torture_random_state *trsp)
return;
} else {
/*
- * The task will remain boosted for another ~500k operations,
- * then restored back to its original prio, and so forth.
+ * The task will remain boosted for another 10*rt_boost_factor
+ * operations, then restored back to its original prio, and so
+ * forth.
*
* When @trsp is nil, we want to force-reset the task for
* stopping the kthread.
--
2.38.1.584.g0f3c55d4c2-goog
On Wed, Nov 23, 2022 at 01:21:04AM +0000, Joel Fernandes (Google) wrote:
> The rt boosting in locktorture has a factor variable large enough that
> boosting only happens once every minute or so. Add a tunable to educe
> the factor so that boosting happens more often, to test paths and arrive
> at failure modes earlier. With this change, I can set the factor to
> like 50 and have the boosting happens every 10 seconds or so.
>
> Tested with boot parameters:
> locktorture.torture_type=mutex_lock
> locktorture.onoff_interval=1
> locktorture.nwriters_stress=8
> locktorture.stutter=0
> locktorture.rt_boost=1
> locktorture.rt_boost_factor=50
> locktorture.nlocks=3
>
> Signed-off-by: Joel Fernandes (Google) <[email protected]>
This looks good, and is what I should have done to start with. But
it depends on the first commit, so I will hold off for the moment.
Thanx, Paul
> ---
> kernel/locking/locktorture.c | 12 +++++++-----
> 1 file changed, 7 insertions(+), 5 deletions(-)
>
> diff --git a/kernel/locking/locktorture.c b/kernel/locking/locktorture.c
> index 5a388ac96a9b..e4529c2166e9 100644
> --- a/kernel/locking/locktorture.c
> +++ b/kernel/locking/locktorture.c
> @@ -47,6 +47,7 @@ torture_param(int, stat_interval, 60,
> "Number of seconds between stats printk()s");
> torture_param(int, stutter, 5, "Number of jiffies to run/halt test, 0=disable");
> torture_param(int, rt_boost, 0, "Perform an rt-boost from the writer, always 1 for rtmutex_lock");
> +torture_param(int, rt_boost_factor, 50000, "A factor determining how often rt-boost happens");
> torture_param(int, verbose, 1,
> "Enable verbose debugging printk()s");
> torture_param(int, nlocks, 1,
> @@ -132,15 +133,15 @@ static void torture_lock_busted_write_unlock(int tid __maybe_unused)
>
> static void torture_rt_boost(struct torture_random_state *trsp)
> {
> - const unsigned int factor = 50000; /* yes, quite arbitrary */
> + const unsigned int factor = rt_boost_factor; /* yes, quite arbitrary */
>
> if (!rt_boost)
> return;
>
> if (!rt_task(current)) {
> /*
> - * Boost priority once every ~50k operations. When the
> - * task tries to take the lock, the rtmutex it will account
> + * Boost priority once every rt_boost_factor operations. When
> + * the task tries to take the lock, the rtmutex it will account
> * for the new priority, and do any corresponding pi-dance.
> */
> if (trsp && !(torture_random(trsp) %
> @@ -150,8 +151,9 @@ static void torture_rt_boost(struct torture_random_state *trsp)
> return;
> } else {
> /*
> - * The task will remain boosted for another ~500k operations,
> - * then restored back to its original prio, and so forth.
> + * The task will remain boosted for another 10*rt_boost_factor
> + * operations, then restored back to its original prio, and so
> + * forth.
> *
> * When @trsp is nil, we want to force-reset the task for
> * stopping the kthread.
> --
> 2.38.1.584.g0f3c55d4c2-goog
>
On 2022-11-23 at 01:21:04 +0000, Joel Fernandes (Google) wrote:
> The rt boosting in locktorture has a factor variable large enough that
> boosting only happens once every minute or so. Add a tunable to educe
> the factor so that boosting happens more often, to test paths and arrive
> at failure modes earlier. With this change, I can set the factor to
> like 50 and have the boosting happens every 10 seconds or so.
>
> Tested with boot parameters:
> locktorture.torture_type=mutex_lock
> locktorture.onoff_interval=1
> locktorture.nwriters_stress=8
> locktorture.stutter=0
> locktorture.rt_boost=1
> locktorture.rt_boost_factor=50
> locktorture.nlocks=3
>
> Signed-off-by: Joel Fernandes (Google) <[email protected]>
> ---
> kernel/locking/locktorture.c | 12 +++++++-----
> 1 file changed, 7 insertions(+), 5 deletions(-)
>
> diff --git a/kernel/locking/locktorture.c b/kernel/locking/locktorture.c
> index 5a388ac96a9b..e4529c2166e9 100644
> --- a/kernel/locking/locktorture.c
> +++ b/kernel/locking/locktorture.c
> @@ -47,6 +47,7 @@ torture_param(int, stat_interval, 60,
> "Number of seconds between stats printk()s");
> torture_param(int, stutter, 5, "Number of jiffies to run/halt test, 0=disable");
> torture_param(int, rt_boost, 0, "Perform an rt-boost from the writer, always 1 for rtmutex_lock");
> +torture_param(int, rt_boost_factor, 50000, "A factor determining how often rt-boost happens");
> torture_param(int, verbose, 1,
> "Enable verbose debugging printk()s");
> torture_param(int, nlocks, 1,
> @@ -132,15 +133,15 @@ static void torture_lock_busted_write_unlock(int tid __maybe_unused)
>
> static void torture_rt_boost(struct torture_random_state *trsp)
> {
> - const unsigned int factor = 50000; /* yes, quite arbitrary */
> + const unsigned int factor = rt_boost_factor; /* yes, quite arbitrary */
>
> if (!rt_boost)
> return;
>
> if (!rt_task(current)) {
> /*
> - * Boost priority once every ~50k operations. When the
> - * task tries to take the lock, the rtmutex it will account
> + * Boost priority once every rt_boost_factor operations. When
> + * the task tries to take the lock, the rtmutex it will account
> * for the new priority, and do any corresponding pi-dance.
> */
> if (trsp && !(torture_random(trsp) %
> @@ -150,8 +151,9 @@ static void torture_rt_boost(struct torture_random_state *trsp)
> return;
> } else {
> /*
> - * The task will remain boosted for another ~500k operations,
> - * then restored back to its original prio, and so forth.
> + * The task will remain boosted for another 10*rt_boost_factor
Maybe I understand incorrectly, the code is
cxt.nrealwriters_stress * factor * 2, should it be 2 rather than 10?
May I know where the 10 comes from?
thanks,
Chenyu
On Wed, Dec 21, 2022 at 12:28:39PM +0800, Chen Yu wrote:
> On 2022-11-23 at 01:21:04 +0000, Joel Fernandes (Google) wrote:
> > The rt boosting in locktorture has a factor variable large enough that
> > boosting only happens once every minute or so. Add a tunable to educe
> > the factor so that boosting happens more often, to test paths and arrive
> > at failure modes earlier. With this change, I can set the factor to
> > like 50 and have the boosting happens every 10 seconds or so.
> >
> > Tested with boot parameters:
> > locktorture.torture_type=mutex_lock
> > locktorture.onoff_interval=1
> > locktorture.nwriters_stress=8
> > locktorture.stutter=0
> > locktorture.rt_boost=1
> > locktorture.rt_boost_factor=50
> > locktorture.nlocks=3
> >
> > Signed-off-by: Joel Fernandes (Google) <[email protected]>
> > ---
> > kernel/locking/locktorture.c | 12 +++++++-----
> > 1 file changed, 7 insertions(+), 5 deletions(-)
> >
> > diff --git a/kernel/locking/locktorture.c b/kernel/locking/locktorture.c
> > index 5a388ac96a9b..e4529c2166e9 100644
> > --- a/kernel/locking/locktorture.c
> > +++ b/kernel/locking/locktorture.c
> > @@ -47,6 +47,7 @@ torture_param(int, stat_interval, 60,
> > "Number of seconds between stats printk()s");
> > torture_param(int, stutter, 5, "Number of jiffies to run/halt test, 0=disable");
> > torture_param(int, rt_boost, 0, "Perform an rt-boost from the writer, always 1 for rtmutex_lock");
> > +torture_param(int, rt_boost_factor, 50000, "A factor determining how often rt-boost happens");
> > torture_param(int, verbose, 1,
> > "Enable verbose debugging printk()s");
> > torture_param(int, nlocks, 1,
> > @@ -132,15 +133,15 @@ static void torture_lock_busted_write_unlock(int tid __maybe_unused)
> >
> > static void torture_rt_boost(struct torture_random_state *trsp)
> > {
> > - const unsigned int factor = 50000; /* yes, quite arbitrary */
> > + const unsigned int factor = rt_boost_factor; /* yes, quite arbitrary */
> >
> > if (!rt_boost)
> > return;
> >
> > if (!rt_task(current)) {
> > /*
> > - * Boost priority once every ~50k operations. When the
> > - * task tries to take the lock, the rtmutex it will account
> > + * Boost priority once every rt_boost_factor operations. When
> > + * the task tries to take the lock, the rtmutex it will account
> > * for the new priority, and do any corresponding pi-dance.
> > */
> > if (trsp && !(torture_random(trsp) %
> > @@ -150,8 +151,9 @@ static void torture_rt_boost(struct torture_random_state *trsp)
> > return;
> > } else {
> > /*
> > - * The task will remain boosted for another ~500k operations,
> > - * then restored back to its original prio, and so forth.
> > + * The task will remain boosted for another 10*rt_boost_factor
> Maybe I understand incorrectly, the code is
> cxt.nrealwriters_stress * factor * 2, should it be 2 rather than 10?
It looks that way to me, but I might be missing something. Joel?
Thanx, Paul
> May I know where the 10 comes from?
>
> thanks,
> Chenyu
On Thu, Jan 05, 2023 at 10:27:18AM -0800, Paul E. McKenney wrote:
> On Wed, Dec 21, 2022 at 12:28:39PM +0800, Chen Yu wrote:
> > On 2022-11-23 at 01:21:04 +0000, Joel Fernandes (Google) wrote:
> > > The rt boosting in locktorture has a factor variable large enough that
> > > boosting only happens once every minute or so. Add a tunable to educe
> > > the factor so that boosting happens more often, to test paths and arrive
> > > at failure modes earlier. With this change, I can set the factor to
> > > like 50 and have the boosting happens every 10 seconds or so.
> > >
> > > Tested with boot parameters:
> > > locktorture.torture_type=mutex_lock
> > > locktorture.onoff_interval=1
> > > locktorture.nwriters_stress=8
> > > locktorture.stutter=0
> > > locktorture.rt_boost=1
> > > locktorture.rt_boost_factor=50
> > > locktorture.nlocks=3
> > >
> > > Signed-off-by: Joel Fernandes (Google) <[email protected]>
> > > ---
> > > kernel/locking/locktorture.c | 12 +++++++-----
> > > 1 file changed, 7 insertions(+), 5 deletions(-)
> > >
> > > diff --git a/kernel/locking/locktorture.c b/kernel/locking/locktorture.c
> > > index 5a388ac96a9b..e4529c2166e9 100644
> > > --- a/kernel/locking/locktorture.c
> > > +++ b/kernel/locking/locktorture.c
> > > @@ -47,6 +47,7 @@ torture_param(int, stat_interval, 60,
> > > "Number of seconds between stats printk()s");
> > > torture_param(int, stutter, 5, "Number of jiffies to run/halt test, 0=disable");
> > > torture_param(int, rt_boost, 0, "Perform an rt-boost from the writer, always 1 for rtmutex_lock");
> > > +torture_param(int, rt_boost_factor, 50000, "A factor determining how often rt-boost happens");
> > > torture_param(int, verbose, 1,
> > > "Enable verbose debugging printk()s");
> > > torture_param(int, nlocks, 1,
> > > @@ -132,15 +133,15 @@ static void torture_lock_busted_write_unlock(int tid __maybe_unused)
> > >
> > > static void torture_rt_boost(struct torture_random_state *trsp)
> > > {
> > > - const unsigned int factor = 50000; /* yes, quite arbitrary */
> > > + const unsigned int factor = rt_boost_factor; /* yes, quite arbitrary */
> > >
> > > if (!rt_boost)
> > > return;
> > >
> > > if (!rt_task(current)) {
> > > /*
> > > - * Boost priority once every ~50k operations. When the
> > > - * task tries to take the lock, the rtmutex it will account
> > > + * Boost priority once every rt_boost_factor operations. When
> > > + * the task tries to take the lock, the rtmutex it will account
> > > * for the new priority, and do any corresponding pi-dance.
> > > */
> > > if (trsp && !(torture_random(trsp) %
> > > @@ -150,8 +151,9 @@ static void torture_rt_boost(struct torture_random_state *trsp)
> > > return;
> > > } else {
> > > /*
> > > - * The task will remain boosted for another ~500k operations,
> > > - * then restored back to its original prio, and so forth.
> > > + * The task will remain boosted for another 10*rt_boost_factor
> > Maybe I understand incorrectly, the code is
> > cxt.nrealwriters_stress * factor * 2, should it be 2 rather than 10?
>
> It looks that way to me, but I might be missing something. Joel?
> > May I know where the 10 comes from?
The comment in existing code was 500k ops.
Yes, Chen is right, the comment can be improved to mention the actual
equation. I was just going by the initial comment of ~500K ops. Since factor
now defaults to 50k, this translates to 500k (10 times the factor) ops which
it does for a 4-5 CPU system.
But I am Ok with the comment changing to what Chen suggested though!
thanks,
- Joel