Dear kernel developers,
I'm currently trying to get the Mach64 DRI driver to run on PowerPC. It's
mostly working but there are some strange behaviors (DMA works, MMIO not
really unless you make long waits when submiting, etc.). This is most
likely related with the MMIO programming macros in the kernel module.
My question is: How to code MMIO to be portable across all platforms,
i.e., taking in consideration the endian format and memory caches?
I've search thorougly the answer to this question but found
incomplete/contraditory answers:
- should one use readl/writel or dereference the address directly?
- is the use of readl/writel macros suficient to account for endian
correctness or it's also needed to use the cpu_to_le32/le32_to_cpu macros?
- should one in general (i.e., assuming the worst case) do wmb() on
writes, and mb() on reads?
Although I appreciate answers concerning future developments on this
matter, I need an answer to the current stable kernel release.
Regards,
Jos? Fonseca
PS: Please CC me as I'm not subscribed.
On Tue, 30 Apr 2002, Jos? Fonseca wrote:
> I'm currently trying to get the Mach64 DRI driver to run on PowerPC. It's
> mostly working but there are some strange behaviors (DMA works, MMIO not
> really unless you make long waits when submiting, etc.). This is most
> likely related with the MMIO programming macros in the kernel module.
>
> My question is: How to code MMIO to be portable across all platforms,
> i.e., taking in consideration the endian format and memory caches?
>
> I've search thorougly the answer to this question but found
> incomplete/contraditory answers:
>
> - should one use readl/writel or dereference the address directly?
Use readl/writel, that's why they exist.
> - is the use of readl/writel macros suficient to account for endian
> correctness or it's also needed to use the cpu_to_le32/le32_to_cpu macros?
They'll give you little endian always (i.e. they byteswap on big endian
archs), so normally (register accesses) you shouldn't need additional
swapping.
> - should one in general (i.e., assuming the worst case) do wmb() on
> writes, and mb() on reads?
I don't think mb() will help you. You're probably experiencing PCI posting
problems - when a writel() has executed, that doesn't necessarily mean
that the transaction has actually happened it may (and will) be buffered
for a potentially long time.
However, PCI won't reorder reads vs. writes, so you when you want to be
sure that a write() actually reached the hardware, do a dummy read()
afterwards, that'll flush the write buffer.
--Kai
Kai,
On 2002.04.30 17:19 Kai Germaschewski wrote:
> On Tue, 30 Apr 2002, Jos? Fonseca wrote:
> > - should one in general (i.e., assuming the worst case) do wmb() on
> > writes, and mb() on reads?
>
> I don't think mb() will help you. You're probably experiencing PCI
> posting
> problems - when a writel() has executed, that doesn't necessarily mean
> that the transaction has actually happened it may (and will) be buffered
> for a potentially long time.
>
> However, PCI won't reorder reads vs. writes, so you when you want to be
> sure that a write() actually reached the hardware, do a dummy read()
> afterwards, that'll flush the write buffer.
Unfortunately one of the problems occurs in a idle wait loop, when a
register is being sucessively read.
And if so, how the wmb() example in "Linux Device Drivers"
(http://www.xml.com/ldd/chapter/book/ch08.html#t1) can be explained? The
"Bus-Independent Device Accesses"
(http://www.kernelnewbies.org/documents/kdoc/deviceiobook/x44.html) also
refers what you suggested, but it also mentions the use of memory
barriers. So how and when should they be used?
Regards,
Jos? Fonseca
On Tue, 30 Apr 2002, Jos? Fonseca wrote:
> Unfortunately one of the problems occurs in a idle wait loop, when a
> register is being sucessively read.
Hmmh, suppose you have different problem (most likely elsewhere) then.
> And if so, how the wmb() example in "Linux Device Drivers"
> (http://www.xml.com/ldd/chapter/book/ch08.html#t1) can be explained? The
> "Bus-Independent Device Accesses"
> (http://www.kernelnewbies.org/documents/kdoc/deviceiobook/x44.html) also
> refers what you suggested, but it also mentions the use of memory
> barriers. So how and when should they be used?
Well, my understanding is the following: (if I get something wrong,
hopefully somebody who's reading this will correct me)
the barrier(), {,r,w}mb() stuff is for actually for normal memory
accesses.
About barrier():
If you have
*p1 = 1; *p2 = 2;
the compiler may decide to reorder this to (if it knows that p1 != p2)
*p2 = 2; *p1 = 1;
A barrier() in between will inhibit this reordering.
For some archs, even the barrier() is not sufficient to serialize the
accesses to RAM. The compiler may generate something like
mov [p1], 1
mov [p2], 2
but on e.g. alpha (where the asm would look differently ;-), the processor
may decide to put the second instruction on the memory bus before the
first one. Only an mb in between will guarantee the ordering on the
memory bus.
Now, for IO, basically the same holds, though I wouldn't want to guarantee
that the macros designed for the memory bus would work on the PCI bus as
expected.
However, I do *believe*, that the readl/writel functions implicitly do the
right thing and introduce barriers where needed. On x86 e.g., the macros
do a cast to (volatile *), which will ensure that these functions are
compiled without reordering. As x86 is strongly ordered, no additional
mb() or whatever is necessary (nor does it exist) to make sure that this
ordering will propagate to the PCI bus.
There's definitely people here who know that stuff better than I do,
though.
--Kai
In article <[email protected]> you wrote:
>> - should one in general (i.e., assuming the worst case) do wmb() on
>> writes, and mb() on reads?
readl() and writel() are specified to contain the proper barriers already.
> I don't think mb() will help you. You're probably experiencing PCI posting
> problems - when a writel() has executed, that doesn't necessarily mean
> that the transaction has actually happened it may (and will) be buffered
> for a potentially long time.
>
> However, PCI won't reorder reads vs. writes, so you when you want to be
> sure that a write() actually reached the hardware, do a dummy read()
> afterwards, that'll flush the write buffer.
yup
On Tue, Apr 30, 2002 at 02:41:56PM -0500, Kai Germaschewski wrote:
> Well, my understanding is the following: (if I get something wrong,
> hopefully somebody who's reading this will correct me)
>
> the barrier(), {,r,w}mb() stuff is for actually for normal memory
> accesses.
>
> About barrier():
>
> If you have
>
> *p1 = 1; *p2 = 2;
>
> the compiler may decide to reorder this to (if it knows that p1 != p2)
>
> *p2 = 2; *p1 = 1;
>
> A barrier() in between will inhibit this reordering.
>
> For some archs, even the barrier() is not sufficient to serialize the
> accesses to RAM. The compiler may generate something like
>
> mov [p1], 1
> mov [p2], 2
>
> but on e.g. alpha (where the asm would look differently ;-), the processor
> may decide to put the second instruction on the memory bus before the
> first one. Only an mb in between will guarantee the ordering on the
> memory bus.
That's a good summary of the memory ordering issues one normally runs
into.
> Now, for IO, basically the same holds, though I wouldn't want to guarantee
> that the macros designed for the memory bus would work on the PCI bus as
> expected.
Right. In fact, waiting on I/O busses can take a bit longer than
making sure the processor executes memory transactions in the order
you'd like.
> However, I do *believe*, that the readl/writel functions implicitly do the
> right thing and introduce barriers where needed. On x86 e.g., the macros
> do a cast to (volatile *), which will ensure that these functions are
> compiled without reordering. As x86 is strongly ordered, no additional
> mb() or whatever is necessary (nor does it exist) to make sure that this
> ordering will propagate to the PCI bus.
Right, readl/writel will order things wrt the compiler and
processor, but not necessarily the I/O bus. On IA64, we've introduced
mmiob() to address this. It acts just like mb(), but wrt I/O address
space. The ia64 patch for 2.5 includes some documentation about it,
I'd love to see other arches implement something similar (even as a
simple nop) so that machines with weakly ordered I/O busses will work
as expected.
Jesse