On Fri, Apr 23, 2021 at 04:01:29PM +0530, Srikar Dronamraju wrote:
> > The series also oopses a *lot* and didn't get through a run of basic
> > workloads on x86 on any of three machines. An example oops is
> >
>
> Can you pass me your failing config. I am somehow not been seeing this
> either on x86 or on Powerpc on multiple systems.

The machines have since moved onto testing something else (Rik's patch
for newidle) but the attached config should be close enough.

> Also if possible cat /proc/schedstat and cat
> /proc/sys/kernel/sched_domain/cpu0/domain*/name
>

For the vanilla kernel

SMT
MC
NUMA

> > [ 137.770968] BUG: unable to handle page fault for address: 000000000001a5c8
> > [ 137.777836] #PF: supervisor read access in kernel mode
> > [ 137.782965] #PF: error_code(0x0000) - not-present page
> > [ 137.788097] PGD 8000004098a42067 P4D 8000004098a42067 PUD 4092e36067 PMD 40883ac067 PTE 0
> > [ 137.796261] Oops: 0000 [#1] SMP PTI
> > [ 137.799747] CPU: 0 PID: 14913 Comm: GC Slave Tainted: G E 5.12.0-rc8-llcfallback-v1r1 #1
> > [ 137.809123] Hardware name: SGI.COM C2112-4GP3/X10DRT-P-Series, BIOS 2.0a 05/09/2016
> > [ 137.816765] RIP: 0010:cpus_share_cache+0x22/0x30
> > [ 137.821396] Code: fc ff 0f 0b eb 80 66 90 0f 1f 44 00 00 48 63 ff 48 63 f6 48 c7 c0 c8 a5 01 00 48 8b 0c fd 00 59 9d 9a 48 8b 14 f5 00 59 9d 9a <8b> 14 02 39 14 01 0f 94 c0 c3 0f 1f 40 00 0f 1f 44 00 00 41 57 41
>
> IP says cpus_share_cache, and it takes 2 ints,
> RAX is 000000000001a5c8 but the panic says
> "unable to handle page fault for address: 000000000001a5c8"
> so it must have failed for "per_cpu(sd_llc_id, xx_cpu)"
>

More than likely, I didn't look closely because the intent was to schedule
tests to get some data and do the review later when I had time. tbench
partially completed but oopsed for high thread counts. Another load failed
completely and I didn't test beyond that but tbench for high thread counts
should be reproducible.

--
Mel Gorman
SUSE Labs

Attachments:

(No filename) (2.00 kB)
config.gz (57.94 kB)
Download all attachments

2021-04-26 10:32:39

by Srikar Dronamraju

[permalink] [raw]

Subject: Re: [PATCH 00/10] sched/fair: wake_affine improvements

* Mel Gorman <[email protected]> [2021-04-23 13:38:55]:

Hi Mel,

> On Fri, Apr 23, 2021 at 04:01:29PM +0530, Srikar Dronamraju wrote:
> > > The series also oopses a *lot* and didn't get through a run of basic
> > > workloads on x86 on any of three machines. An example oops is
> > >
> >
> > Can you pass me your failing config. I am somehow not been seeing this
> > either on x86 or on Powerpc on multiple systems.
>
> The machines have since moved onto testing something else (Rik's patch
> for newidle) but the attached config should be close enough.
>
> > Also if possible cat /proc/schedstat and cat
> > /proc/sys/kernel/sched_domain/cpu0/domain*/name
> >
>
> For the vanilla kernel
>
> SMT
> MC
> NUMA

I was able to reproduce the problem and analyze why it would panic in
cpus_share_cache.

In my patch(es), we have code snippets like this.

if (tsds->idle_core != -1) {
if (cpumask_test_cpu(tsds->idle_core, p->cpus_ptr))
return tsds->idle_core;
return this_cpu;
}

Here when we tested the idle_core and cpumask_test_cpu,
tsds->idle_core may not have been -1; However by the time it returns,
tsds->idle_core could be -1;

cpus_share_cpus() then tries to find sd_llc_id for -1 and crashes.

Its more easier to reproduce this on a machine with more cores in a
LLC than say a Power10/Power9. Hence we are hitting this more often
on x86.

One way could be to save the idle_core to a local variable, but that
negates the whole purpose since we may end up choosing a busy CPU. I
will find a way to fix this problem.

--
Thanks and Regards
Srikar Dronamraju

2021-04-26 10:41:48

by Srikar Dronamraju

[permalink] [raw]

* Vincent Guittot <[email protected]> [2021-04-27 16:52:30]:

> Hi Srikar,

Hi Vincent,

> On Thu, 22 Apr 2021 at 12:23, Srikar Dronamraju
> <[email protected]> wrote:
> >
> > Recently we found that some of the benchmark numbers on Power10 were lesser
> > than expected. Some analysis showed that the problem lies in the fact that
> > L2-Cache on Power10 is at core level i.e only 4 threads share the L2-cache.
> >
> >
> > Summary:
> > mc-llc outperforms, this patchset and upstream almost give similar performance.
>
> So mc-llc patch seems to be the best approach IMHO. Although the
> hemisphere don't share cache, they share enough resources so
> cache-snooping is as efficient as sharing cache
>

Yes, mc-llc helps just specific systems like Power10 but its shows better
numbers than my posted patchset.

However in this patchset, we are looking at areas in wakeup (aka idler llcs)
we could optimize which can help other archs too. + the fallback mechanism
is generic enough that we could use it for other Systems too.

I know that there are valid concerns raised by Mel and I working to resolve
them. Some of them are.
- How hot is idle-core
- Crashes when running tbench (I was able to reproduce with kernbench on x86)

Also I am adding some more changes with which we are getting similar
performance as mc-llc.

> > Cc: LKML <[email protected]>
> > Cc: Michael Ellerman <[email protected]>
> > Cc: Gautham R Shenoy <[email protected]>
> > Cc: Parth Shah <[email protected]>
> > Cc: Ingo Molnar <[email protected]>
> > Cc: Peter Zijlstra <[email protected]>
> > Cc: Valentin Schneider <[email protected]>
> > Cc: Dietmar Eggemann <[email protected]>
> > Cc: Mel Gorman <[email protected]>
> > Cc: Vincent Guittot <[email protected]>
> > Cc: Rik van Riel <[email protected]>
> >
> > Srikar Dronamraju (10):
> > sched/fair: Update affine statistics when needed
> > sched/fair: Maintain the identity of idle-core
> > sched/fair: Update idle-core more often
> > sched/fair: Prefer idle CPU to cache affinity
> > sched/fair: Call wake_affine only if necessary
> > sched/idle: Move busy_cpu accounting to idle callback
> > sched/fair: Remove ifdefs in waker_affine_idler_llc
> > sched/fair: Dont iterate if no idle CPUs
> > sched/topology: Introduce fallback LLC
> > powerpc/smp: Add fallback flag to powerpc MC domain
> >
> > arch/powerpc/kernel/smp.c | 7 +-
> > include/linux/sched/sd_flags.h | 7 +
> > include/linux/sched/topology.h | 3 +-
> > kernel/sched/fair.c | 229 +++++++++++++++++++++++++++------
> > kernel/sched/features.h | 1 +
> > kernel/sched/idle.c | 33 ++++-
> > kernel/sched/sched.h | 6 +
> > kernel/sched/topology.c | 54 +++++++-
> > 8 files changed, 296 insertions(+), 44 deletions(-)
> >
> > --
> > 2.18.2
> >

--
Thanks and Regards
Srikar Dronamraju