Hi, Andi -
I notice that wmb() is a no-op on x86_64 kernels unless
CONFIG_UNORDERED_IO is set. Is there any particular reason for this?
It's not similarly conditional on other platforms, and as a consequence,
in our driver (which requires a write barrier in some situations for
correctness), I have to add the following piece of ugliness:
#if defined(CONFIG_X86_64) && !defined(CONFIG_UNORDERED_IO)
#define ipath_wmb() asm volatile("sfence" ::: "memory")
#else
#define ipath_wmb() wmb()
#endif
<b
On Wednesday 18 January 2006 17:23, Bryan O'Sullivan wrote:
> Hi, Andi -
>
> I notice that wmb() is a no-op
Actually it is a compiler optimizer barrier, not a no-op.
> on x86_64 kernels unless
> CONFIG_UNORDERED_IO is set.
Because x86 is architecturally defined as having ordered writes (unless you use
write combining or non temporal stores which normal kernel code doesn't). So it's
not needed.
> Is there any particular reason for this?
> It's not similarly conditional on other platforms, and as a consequence,
> in our driver (which requires a write barrier in some situations for
> correctness), I have to add the following piece of ugliness:
>
> #if defined(CONFIG_X86_64) && !defined(CONFIG_UNORDERED_IO)
> #define ipath_wmb() asm volatile("sfence" ::: "memory")
> #else
> #define ipath_wmb() wmb()
> #endif
Hmm, I suppose one could add a wc_wmb() or somesuch, but WC
is currently deeply architecture specific so I'm not sure
how you can even use it portably.
Why do you need the barrier?
-Andi
On Wed, 2006-01-18 at 17:29 +0100, Andi Kleen wrote:
> Actually it is a compiler optimizer barrier, not a no-op.
Sorry, braino.
> Hmm, I suppose one could add a wc_wmb() or somesuch, but WC
> is currently deeply architecture specific so I'm not sure
> how you can even use it portably.
>
> Why do you need the barrier?
On x86_64, we fiddle with the MTRRs to enable write combining, which
makes a huge difference to performance. It's not clear to me what we
should even do on other architectures, since the only generic entry
point that even exposes write combining is pci_mmap_page_range, which is
for PCI mmap through userspace, and half the arches I've looked at
ignore its write_combine parameter.
<b
>>>>> "Bryan" == Bryan O'Sullivan <[email protected]> writes:
Bryan> On Wed, 2006-01-18 at 17:29 +0100, Andi Kleen wrote:
>> Why do you need the barrier?
Bryan> On x86_64, we fiddle with the MTRRs to enable write combining,
Bryan> which makes a huge difference to performance. It's not clear
Bryan> to me what we should even do on other architectures, since the
Bryan> only generic entry point that even exposes write combining is
Bryan> pci_mmap_page_range, which is for PCI mmap through userspace,
Bryan> and half the arches I've looked at ignore its write_combine
Bryan> parameter.
A job for mmiowb() perhaps?
Cheers,
Jes
On Wed, 2006-01-18 at 12:06 -0500, Jes Sorensen wrote:
> A job for mmiowb() perhaps?
That might be suitable. It's a no-op on most platforms, but it's only
used by a handful of drivers:
drivers/scsi/qla1280.c
drivers/sn/ioc4.c
drivers/net/bnx2.c
drivers/net/sky2.c
drivers/net/s2io.c
drivers/net/tg3.c
If the semantics were to change so that it really did a write barrier, I
doubt any existing users would notice. In fact, based on the comments
in some drivers, at least some authors think it does already, when it
typically doesn't.
<b
On Wednesday 18 January 2006 18:06, Jes Sorensen wrote:
> >>>>> "Bryan" == Bryan O'Sullivan <[email protected]> writes:
>
> Bryan> On Wed, 2006-01-18 at 17:29 +0100, Andi Kleen wrote:
> >> Why do you need the barrier?
>
> Bryan> On x86_64, we fiddle with the MTRRs to enable write combining,
> Bryan> which makes a huge difference to performance. It's not clear
> Bryan> to me what we should even do on other architectures, since the
> Bryan> only generic entry point that even exposes write combining is
> Bryan> pci_mmap_page_range, which is for PCI mmap through userspace,
> Bryan> and half the arches I've looked at ignore its write_combine
> Bryan> parameter.
>
> A job for mmiowb() perhaps?
No, normal IO mappings are also not write combining on x86, so
it's not needed there.
I guess it's best to just define a wmb_wc() for i386/x86-64 right now
and use that in ipath. I suspect it won't compile anywhere else
anyways.
-Andi
Bryan> On x86_64, we fiddle with the MTRRs to enable write
Bryan> combining, which makes a huge difference to performance.
Bryan> It's not clear to me what we should even do on other
Bryan> architectures, since the only generic entry point that even
Bryan> exposes write combining is pci_mmap_page_range, which is
Bryan> for PCI mmap through userspace, and half the arches I've
Bryan> looked at ignore its write_combine parameter.
Jes> A job for mmiowb() perhaps?
I don't think the semantics of mmiowb() do what is desired here.
mmiowb() is all about ordering writes between separate CPUs, and the
issue at hand here is that write-combining buffers might reorder
writes from a single CPU.
- R.
>>>>> "Andi" == Andi Kleen <[email protected]> writes:
Andi> On Wednesday 18 January 2006 18:06, Jes Sorensen wrote:
>> A job for mmiowb() perhaps?
Andi> No, normal IO mappings are also not write combining on x86, so
Andi> it's not needed there.
True, just seemed nasty to have yet another special purpose wmb()
variation.
Cheers,
Jes