To: James Bottomley <James.Bottomley@HansenPartnership.com>
Cc: akepner@sgi.com, Tony Luck <tony.luck@intel.com>,
       Grant Grundler <grundler@parisc-linux.org>,
       Jesse Barnes <jbarnes@virtuousgeek.org>, Jes Sorensen <jes@sgi.com>,
       Randy Dunlap <randy.dunlap@oracle.com>,
       David Miller <davem@davemloft.net>, Muli Ben-Yehuda <muli@il.ibm.com>,
       linux-kernel@vger.kernel.org
Subject: Re: [RFC/PARTIAL PATCH 0/3] dma: passing "attributes" to dma_map_* routines
References: <20080108023222.GP23661@sgi.com>
	<1199809628.3534.34.camel@localhost.localdomain>
	<ada63y4kycs.fsf@cisco.com>
	<1199814853.3534.39.camel@localhost.localdomain>
	<adawsqkjipz.fsf@cisco.com>
	<1199816504.3534.59.camel@localhost.localdomain>
From: Roland Dreier <rdreier@cisco.com>
Date: Wed, 09 Jan 2008 13:00:38 -0800
In-Reply-To: <1199816504.3534.59.camel@localhost.localdomain> (James Bottomley's message of "Tue, 08 Jan 2008 12:21:44 -0600")
Message-ID: <adaprwa7lzd.fsf@cisco.com>
User-Agent: Gnus/5.1008 (Gnus v5.10.8) XEmacs/21.4.21 (linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 2146
Lines: 42

 > What it wants to do is set strict ordering for the bus ... well, that is
 > an attribute in the PCIe standard (it just happens to be the default one
 > for a standard bus, whereas relaxed is the default for altix).  However,
 > set bus attribute strict would be a simple no-op for a standard bus (and
 > set attribute relaxed a no-op for altix).

I don't think that this can work.  As Arthur and I have said several
times, the Altix issue is within the system NUMA interconnect --
nothing to do with the PCI bus.  And as I understand things, to get
good performance, we have to allow reordering within the NUMA
interconnect except that certain transactions need to flush all
earlier writes.  So this attribute needs to be set per-mapping, not
per-bus.

If you have a cleaner way to specify this attribute that Arthur's
idea, then it would be very useful to share that.  If we were starting
from scratch, then probably adding another "flags" parameter to the
DMA mapping functions would be the way to go, but given the number of
calls to dma_map_xxx all around, changing the signature now doesn't
look very appealing.

The reason this hasn't been an issue until now is that almost all
drivers work correctly if the Altix code just sets the "flush" bit for
memory allocated via the consistent/coherent allocators.  However, if
we want the device to write to userspace memory, this doesn't work
(and mapping coherent memory allocated in the kernel into userspace is
a mess on other platforms, because it unnecessarily consumes lowmem
and/or kernel address space).

Also, all of this is quite separate from the PCIe "loose ordering"
stuff.  In fact, it's quite conceivable that SGI could hook up a PCIe
3.0 host bridge to the Altix NUMA interconnect, so that you could have
a PCI bus that supported loose ordering and also a system interconnect
that allowed a different type of reordering too.

 - R.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/