On Altix, DMA may be reordered within the NUMA interconnect.
This can be a problem with Infiniband, where DMA to Completion
Queues can race with data DMA. This patchset allows a driver
to associate a memory region with a "dmaflush" attribute, so
that writes to the memory region flush in-flight DMA, preventing
the CQ/data race.
There are three patches in this set:
[1/3]: add pci_dma_flags_set_dmaflush() to pci interface
[2/3]: redefine pci_dma_flags_set_dmaflush() for sn-ia64
[3/3]: document pci_dma_flags_set_dmaflush()
And there would be additional patches to IB drivers to make use
of the interface, of course.
--
Arthur
On Fri, Aug 24, 2007 at 11:02:32AM -0700, [email protected] wrote:
>
> On Altix, DMA may be reordered within the NUMA interconnect.
> This can be a problem with Infiniband, where DMA to Completion
> Queues can race with data DMA. This patchset allows a driver
> to associate a memory region with a "dmaflush" attribute, so
> that writes to the memory region flush in-flight DMA, preventing
> the CQ/data race.
FYI to linux-pci folks
This patch had previous discussion on LKML:
http://lkml.org/lkml/2007/8/17/336
James Bottomley at one point obliquely referred to
my OLS2003 paper: "DMA Hints on ia64/PARISC"
After reading the thread, my take is we need a more elegant way for a
device driver to handle registration of DMA regions allocated by user
space. The API would "make this page/region act like dma_alloc_coherent()".
That implies strong ordering between CPU and DMA to/from the device.
Maybe the code is the right thing and I want a name that makes
sense in the context of current DMA API.
On IRC, willy suggested an mmap() flag and that sounds reasonable too
though I don't know if it's feasible.
hth,
grant
> There are three patches in this set:
>
> [1/3]: add pci_dma_flags_set_dmaflush() to pci interface
> [2/3]: redefine pci_dma_flags_set_dmaflush() for sn-ia64
> [3/3]: document pci_dma_flags_set_dmaflush()
>
> And there would be additional patches to IB drivers to make use
> of the interface, of course.
>
> --
> Arthur
On Mon, Aug 27, 2007 at 04:05:48PM -0600, Grant Grundler wrote:
> .....
> After reading the thread, my take is we need a more elegant way for a
> device driver to handle registration of DMA regions allocated by user
> space. The API would "make this page/region act like dma_alloc_coherent()".
> That implies strong ordering between CPU and DMA to/from the device.
> Maybe the code is the right thing and I want a name that makes
> sense in the context of current DMA API.
Need to think about this...
>
> On IRC, willy suggested an mmap() flag and that sounds reasonable too
> though I don't know if it's feasible.
>
Yeah, we're doing something like this now as a band-aid solution.
Not a flag to mmap(), but a magic offset value. But it wasn't
acceptable to the maintainer of the mthca IB driver (Roland
Dreier), hence the new proposal....
--
Arthur