[ Removed Andi as I believe this is the mysterious thread he was talking
about. Anyone else want to be removed? ]
On Wed, 2011-02-16 at 08:24 -0500, Mathieu Desnoyers wrote:
> * Will Newton ([email protected]) wrote:
> initially:
> foo = 0
> bar = 0
>
> CPU A CPU B
>
> xchg(&foo, 1);
> ll foo
> sc foo
>
> -> interrupt
>
> if (foo == 1)
> xchg(&bar, 1);
> ll bar
> sc bar
> invalidate bar
>
> lbar = bar;
> smp_mb()
Question: Does a mb() flush all cache or does it just make sure that
read/write operations finish before starting new ones?
> lfoo = foo;
IOW, will that smp_mb() really make lfoo read the new foo in memory? If
foo happens to still be in cache and no coherency has been performed to
flush it, would it just simply read foo straight from the cache?
-- Steve
> BUG_ON(lbar == 1 && lfoo == 0);
> invalidate foo
>
> It should be valid to expect that every time "bar" read by CPU B is 1,
> then "foo" is always worth 1. However, in this case, the lack of
> invalidate on foo is keeping the cacheline from reaching CPU B. There
> seems to be a problem with interrupts/NMIs coming right between sc and
> invalidate, as Ingo pointed out.
>
> Thanks,
>
> Mathieu
>
* Steven Rostedt ([email protected]) wrote:
> [ Removed Andi as I believe this is the mysterious thread he was talking
> about. Anyone else want to be removed? ]
>
>
> On Wed, 2011-02-16 at 08:24 -0500, Mathieu Desnoyers wrote:
> > * Will Newton ([email protected]) wrote:
>
> > initially:
> > foo = 0
> > bar = 0
> >
> > CPU A CPU B
> >
> > xchg(&foo, 1);
> > ll foo
> > sc foo
> >
> > -> interrupt
> >
> > if (foo == 1)
> > xchg(&bar, 1);
> > ll bar
> > sc bar
> > invalidate bar
> >
> > lbar = bar;
> > smp_mb()
>
> Question: Does a mb() flush all cache or does it just make sure that
> read/write operations finish before starting new ones?
AFAIK, the Linux kernel memory model semantic only cares about coherent
caches (I'd be interested to learn if I am wrong here). Therefore,
smp_mb() affects ordering of data memory read/writes only, not cache
invalidation -- _however_, it apply only in a memory model where the
underlying accesses are performed on coherent caches.
>
> > lfoo = foo;
>
> IOW, will that smp_mb() really make lfoo read the new foo in memory? If
> foo happens to still be in cache and no coherency has been performed to
> flush it, would it just simply read foo straight from the cache?
If we were to deploy the Linux kernel on an architecture without
coherent caches, I think smp_mb() should imply a cacheline invalidation,
otherwise we completely mess up the order of data writes vs their
observability from each invididual core POV.
This is what I do in liburcu actually. I introduced a "smp_mc() (mc for
memory commit)" macro to specify that cache invalidation is required on
non-cache-coherent archs. smp_mb() imply a smp_mc(). (smp_mc() is
therefore weaker than smp_mb(), because the mb imply ordering of memory
operations performed by a given core, while smp_mc only ensures that
the core caches are synchronized with memory)
Thanks,
Mathieu
>
> -- Steve
>
> > BUG_ON(lbar == 1 && lfoo == 0);
> > invalidate foo
> >
> > It should be valid to expect that every time "bar" read by CPU B is 1,
> > then "foo" is always worth 1. However, in this case, the lack of
> > invalidate on foo is keeping the cacheline from reaching CPU B. There
> > seems to be a problem with interrupts/NMIs coming right between sc and
> > invalidate, as Ingo pointed out.
> >
> > Thanks,
> >
> > Mathieu
> >
>
>
--
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com