2003-02-28 06:36:11

by Matthew Wilcox

[permalink] [raw]
Subject: Proposal: Eliminate GFP_DMA


I'm thinking about doing something along these lines:

--- linux-2.5.63/include/linux/gfp.h 2003-02-24 13:05:13.000000000 -0600
+++ linux-2.5.63-gfpdma/include/linux/gfp.h 2003-02-27 17:24:39.000000000 -0600
@@ -26,11 +26,8 @@

-/* Flag - indicates that the buffer will be suitable for DMA. Ignored on some
- platforms, used as appropriate on others */
-
-#define GFP_DMA __GFP_DMA
-
+#define GFP_ATOMIC_DMA (GFP_ATOMIC | __GFP_DMA)
+#define GFP_KERNEL_DMA (GFP_KERNEL | __GFP_DMA)

combined with changing some users to use __GFP_DMA if they really do mean
the bitmask. Comments?

--
"It's not Hollywood. War is real, war is primarily not about defeat or
victory, it is about death. I've seen thousands and thousands of dead bodies.
Do you think I want to have an academic debate on this subject?" -- Robert Fisk


2003-02-28 08:46:22

by Andi Kleen

[permalink] [raw]
Subject: Re: Proposal: Eliminate GFP_DMA

Matthew Wilcox <[email protected]> writes:
> -
> +#define GFP_ATOMIC_DMA (GFP_ATOMIC | __GFP_DMA)
> +#define GFP_KERNEL_DMA (GFP_KERNEL | __GFP_DMA)
>
> combined with changing some users to use __GFP_DMA if they really do mean
> the bitmask. Comments?

Sounds like a good 2.7.x early project. Currently we still have too much
driver breakage for the next release to break even more right now.

-Andi

2003-02-28 14:02:14

by Matthew Wilcox

[permalink] [raw]
Subject: Re: Proposal: Eliminate GFP_DMA

On Fri, Feb 28, 2003 at 09:56:41AM +0100, Andi Kleen wrote:
> Matthew Wilcox <[email protected]> writes:
> > -
> > +#define GFP_ATOMIC_DMA (GFP_ATOMIC | __GFP_DMA)
> > +#define GFP_KERNEL_DMA (GFP_KERNEL | __GFP_DMA)
> >
> > combined with changing some users to use __GFP_DMA if they really do mean
> > the bitmask. Comments?
>
> Sounds like a good 2.7.x early project. Currently we still have too much
> driver breakage for the next release to break even more right now.

i'm not the kind of person who just changes the header file and breaks all
the drivers. plan:

- Add the GFP_ATOMIC_DMA & GFP_KERNEL_DMA definitions
- Change the drivers
- Delete the GFP_DMA definition

--
"It's not Hollywood. War is real, war is primarily not about defeat or
victory, it is about death. I've seen thousands and thousands of dead bodies.
Do you think I want to have an academic debate on this subject?" -- Robert Fisk

2003-02-28 14:11:57

by Alan

[permalink] [raw]
Subject: Re: Proposal: Eliminate GFP_DMA

On Fri, 2003-02-28 at 14:12, Matthew Wilcox wrote:
> i'm not the kind of person who just changes the header file and breaks all
> the drivers. plan:
>
> - Add the GFP_ATOMIC_DMA & GFP_KERNEL_DMA definitions
> - Change the drivers
> - Delete the GFP_DMA definition

Needless pain for people maintaining cross release drivers. Save it for
2.7 where we should finally do the honourable deed given x86-64 may well
be mainstream, and simply remove GFP_DMA and expect people to use
pci_*

2003-02-28 14:23:45

by Matthew Wilcox

[permalink] [raw]
Subject: Re: Proposal: Eliminate GFP_DMA

On Fri, Feb 28, 2003 at 03:24:58PM +0000, Alan Cox wrote:
> On Fri, 2003-02-28 at 14:12, Matthew Wilcox wrote:
> > i'm not the kind of person who just changes the header file and breaks all
> > the drivers. plan:
> >
> > - Add the GFP_ATOMIC_DMA & GFP_KERNEL_DMA definitions
> > - Change the drivers
> > - Delete the GFP_DMA definition
>
> Needless pain for people maintaining cross release drivers. Save it for
> 2.7 where we should finally do the honourable deed given x86-64 may well
> be mainstream, and simply remove GFP_DMA and expect people to use
> pci_*

umm. are you volunteering to convert drivers/net/macmace.c to the pci_*
API then? also, GFP_DMA is used on, eg, s390 to get memory below 2GB and
on ia64 to get memory below 4GB.

--
"It's not Hollywood. War is real, war is primarily not about defeat or
victory, it is about death. I've seen thousands and thousands of dead bodies.
Do you think I want to have an academic debate on this subject?" -- Robert Fisk

2003-02-28 14:42:32

by Alan

[permalink] [raw]
Subject: Re: Proposal: Eliminate GFP_DMA

On Fri, 2003-02-28 at 14:34, Matthew Wilcox wrote:
> umm. are you volunteering to convert drivers/net/macmace.c to the pci_*
> API then? also, GFP_DMA is used on, eg, s390 to get memory below 2GB and
> on ia64 to get memory below 4GB.

The ia64 is a fine example of how broken it is. People have to hack around
with GFP_DMA meaning different things on ia64 to everything else. It needs
to die.

Alan

2003-02-28 14:45:56

by Andi Kleen

[permalink] [raw]
Subject: Re: Proposal: Eliminate GFP_DMA

On Fri, Feb 28, 2003 at 03:55:37PM +0000, Alan Cox wrote:
> On Fri, 2003-02-28 at 14:34, Matthew Wilcox wrote:
> > umm. are you volunteering to convert drivers/net/macmace.c to the pci_*
> > API then? also, GFP_DMA is used on, eg, s390 to get memory below 2GB and
> > on ia64 to get memory below 4GB.
>
> The ia64 is a fine example of how broken it is. People have to hack around
> with GFP_DMA meaning different things on ia64 to everything else. It needs
> to die.

At least on x86-64 it is still needed when you need have some hardware
with address limits < 4GB (e.g. an 24bit soundcard)

pci_* on K8 only allows address mask 0xffffffff or unlimited.

-Andi

2003-02-28 15:14:47

by Jeff Garzik

[permalink] [raw]
Subject: Re: Proposal: Eliminate GFP_DMA

On Fri, Feb 28, 2003 at 03:56:15PM +0100, Andi Kleen wrote:
> On Fri, Feb 28, 2003 at 03:55:37PM +0000, Alan Cox wrote:
> > On Fri, 2003-02-28 at 14:34, Matthew Wilcox wrote:
> > > umm. are you volunteering to convert drivers/net/macmace.c to the pci_*
> > > API then? also, GFP_DMA is used on, eg, s390 to get memory below 2GB and
> > > on ia64 to get memory below 4GB.
> >
> > The ia64 is a fine example of how broken it is. People have to hack around
> > with GFP_DMA meaning different things on ia64 to everything else. It needs
> > to die.
>
> At least on x86-64 it is still needed when you need have some hardware
> with address limits < 4GB (e.g. an 24bit soundcard)
>
> pci_* on K8 only allows address mask 0xffffffff or unlimited.

That's a bit broken... I have an ALS4000 PCI soundcard that is a 24-bit
soundcard. pci_set_dma_mask should support 24-bits accordingly,
otherwise it's a bug in your platform implementation... Nobody will be
able to use certain properly-written drivers on your platform otherwise.

Jeff



2003-02-28 15:24:38

by Martin J. Bligh

[permalink] [raw]
Subject: Re: Proposal: Eliminate GFP_DMA

>> > > umm. are you volunteering to convert drivers/net/macmace.c to the pci_*
>> > > API then? also, GFP_DMA is used on, eg, s390 to get memory below 2GB and
>> > > on ia64 to get memory below 4GB.
>> >
>> > The ia64 is a fine example of how broken it is. People have to hack around
>> > with GFP_DMA meaning different things on ia64 to everything else. It needs
>> > to die.
>>
>> At least on x86-64 it is still needed when you need have some hardware
>> with address limits < 4GB (e.g. an 24bit soundcard)
>>
>> pci_* on K8 only allows address mask 0xffffffff or unlimited.
>
> That's a bit broken... I have an ALS4000 PCI soundcard that is a 24-bit
> soundcard. pci_set_dma_mask should support 24-bits accordingly,
> otherwise it's a bug in your platform implementation... Nobody will be
> able to use certain properly-written drivers on your platform otherwise.

If we're going to really sort this out, would be nice to just pass an upper
bound for an address to __alloc_pages, instead of a simple bitmask ;-)

M.

2003-02-28 15:20:12

by Andi Kleen

[permalink] [raw]
Subject: Re: Proposal: Eliminate GFP_DMA

> That's a bit broken... I have an ALS4000 PCI soundcard that is a 24-bit
> soundcard. pci_set_dma_mask should support 24-bits accordingly,
> otherwise it's a bug in your platform implementation... Nobody will be
> able to use certain properly-written drivers on your platform otherwise.

It supports it for pci_alloc_consistent (by mapping it to an GFP_DMA
allocation)

Just for pci_map_single/pci_map_sg() it is not supported and will
error out when the passed address doesn't fit the mask. That behaviour
is 100% compatible with IA32 which does the same.

-Andi

2003-02-28 15:21:55

by Arjan van de Ven

[permalink] [raw]
Subject: Re: Proposal: Eliminate GFP_DMA

On Fri, 2003-02-28 at 15:56, Andi Kleen wrote:
> On Fri, Feb 28, 2003 at 03:55:37PM +0000, Alan Cox wrote:
> > On Fri, 2003-02-28 at 14:34, Matthew Wilcox wrote:
> > > umm. are you volunteering to convert drivers/net/macmace.c to the pci_*
> > > API then? also, GFP_DMA is used on, eg, s390 to get memory below 2GB and
> > > on ia64 to get memory below 4GB.
> >
> > The ia64 is a fine example of how broken it is. People have to hack around
> > with GFP_DMA meaning different things on ia64 to everything else. It needs
> > to die.
>
> At least on x86-64 it is still needed when you need have some hardware
> with address limits < 4GB (e.g. an 24bit soundcard)
>
> pci_* on K8 only allows address mask 0xffffffff or unlimited.

for 2.7 the underlying low level budy allocator could (should?) just
take a DMA bitmask


Attachments:
signature.asc (189.00 B)
This is a digitally signed message part

2003-02-28 15:32:04

by Alan

[permalink] [raw]
Subject: Re: Proposal: Eliminate GFP_DMA

On Fri, 2003-02-28 at 15:34, Martin J. Bligh wrote:
> If we're going to really sort this out, would be nice to just pass an upper
> bound for an address to __alloc_pages, instead of a simple bitmask ;-)

Not all systems work out that way. Some ARM for example have a memory window rather
than a range. It does solve most platforms so its definitely the right basic feature
to have and let awkward boxes overload it.

2003-02-28 15:39:16

by Eli Carter

[permalink] [raw]
Subject: Re: Proposal: Eliminate GFP_DMA

Martin J. Bligh wrote:
>>>>>umm. are you volunteering to convert drivers/net/macmace.c to the pci_*
>>>>>API then? also, GFP_DMA is used on, eg, s390 to get memory below 2GB and
>>>>>on ia64 to get memory below 4GB.
>>>>
>>>>The ia64 is a fine example of how broken it is. People have to hack around
>>>>with GFP_DMA meaning different things on ia64 to everything else. It needs
>>>>to die.
>>>
>>>At least on x86-64 it is still needed when you need have some hardware
>>>with address limits < 4GB (e.g. an 24bit soundcard)
>>>
>>>pci_* on K8 only allows address mask 0xffffffff or unlimited.
>>
>>That's a bit broken... I have an ALS4000 PCI soundcard that is a 24-bit
>>soundcard. pci_set_dma_mask should support 24-bits accordingly,
>>otherwise it's a bug in your platform implementation... Nobody will be
>>able to use certain properly-written drivers on your platform otherwise.
>
>
> If we're going to really sort this out, would be nice to just pass an upper
> bound for an address to __alloc_pages, instead of a simple bitmask ;-)

To do it properly, I think you'd need to give a range, not just an upper
bound. On some ARM / XScale systems, you can specify a window of your
RAM that is visible on the PCI bus. That may be a case too odd to care
about, but I'm going to have to at some point in the future....

Eli
--------------------. "If it ain't broke now,
Eli Carter \ it will be soon." -- crypto-gram
eli.carter(a)inet.com `-------------------------------------------------

2003-02-28 15:38:54

by Dmitry A. Fedorov

[permalink] [raw]
Subject: Re: Proposal: Eliminate GFP_DMA

On 28 Feb 2003, Alan Cox wrote:
> On Fri, 2003-02-28 at 14:12, Matthew Wilcox wrote:
> > i'm not the kind of person who just changes the header file and breaks all
> > the drivers. plan:
> >
> > - Add the GFP_ATOMIC_DMA & GFP_KERNEL_DMA definitions
> > - Change the drivers
> > - Delete the GFP_DMA definition
>
> Needless pain for people maintaining cross release drivers. Save it for
> 2.7 where we should finally do the honourable deed given x86-64 may well
> be mainstream, and simply remove GFP_DMA and expect people to use
> pci_*

But why drivers of ISA bus devices with DMA should use pci_* functions?

I'm personally wouldn't have too much pain with GFP_DMA because I have
compatibility headers and proposed change for them is tiny.

2003-02-28 15:48:26

by Jeff Garzik

[permalink] [raw]
Subject: Re: Proposal: Eliminate GFP_DMA

On Fri, Feb 28, 2003 at 09:44:14PM +0600, Dmitry A. Fedorov wrote:
> On 28 Feb 2003, Alan Cox wrote:
> > On Fri, 2003-02-28 at 14:12, Matthew Wilcox wrote:
> > > i'm not the kind of person who just changes the header file and breaks all
> > > the drivers. plan:
> > >
> > > - Add the GFP_ATOMIC_DMA & GFP_KERNEL_DMA definitions
> > > - Change the drivers
> > > - Delete the GFP_DMA definition
> >
> > Needless pain for people maintaining cross release drivers. Save it for
> > 2.7 where we should finally do the honourable deed given x86-64 may well
> > be mainstream, and simply remove GFP_DMA and expect people to use
> > pci_*
>
> But why drivers of ISA bus devices with DMA should use pci_* functions?
>
> I'm personally wouldn't have too much pain with GFP_DMA because I have
> compatibility headers and proposed change for them is tiny.

Do not let the name "pci_" distract, it works for ISA too :)

We can #define pci_xxx isa_xxx if you like :)

Jeff



2003-02-28 15:53:25

by Russell King

[permalink] [raw]
Subject: Re: Proposal: Eliminate GFP_DMA

On Fri, Feb 28, 2003 at 09:49:18AM -0600, Eli Carter wrote:
> To do it properly, I think you'd need to give a range, not just an upper
> bound. On some ARM / XScale systems, you can specify a window of your
> RAM that is visible on the PCI bus. That may be a case too odd to care
> about, but I'm going to have to at some point in the future....

Which may not start at address zero either.

There are even ARM systems where it'd be useful to be able to say "only
allocate memory in region N where N = first 1MB of every 2MB region.
(Yes, I know, this broken hardware should die, but it just isn't going
away thanks to the marketing powers of large IC manufacturers.)

Maybe the generic solution could be something like the resource allocation
functions which are passed an alignment function? /me hides.

--
Russell King ([email protected]) The developer of ARM Linux
http://www.arm.linux.org.uk/personal/aboutme.html

2003-02-28 15:55:34

by Russell King

[permalink] [raw]
Subject: Re: Proposal: Eliminate GFP_DMA

On Fri, Feb 28, 2003 at 09:44:14PM +0600, Dmitry A. Fedorov wrote:
> On 28 Feb 2003, Alan Cox wrote:
> > On Fri, 2003-02-28 at 14:12, Matthew Wilcox wrote:
> > > i'm not the kind of person who just changes the header file and breaks all
> > > the drivers. plan:
> > >
> > > - Add the GFP_ATOMIC_DMA & GFP_KERNEL_DMA definitions
> > > - Change the drivers
> > > - Delete the GFP_DMA definition
> >
> > Needless pain for people maintaining cross release drivers. Save it for
> > 2.7 where we should finally do the honourable deed given x86-64 may well
> > be mainstream, and simply remove GFP_DMA and expect people to use
> > pci_*
>
> But why drivers of ISA bus devices with DMA should use pci_* functions?
>
> I'm personally wouldn't have too much pain with GFP_DMA because I have
> compatibility headers and proposed change for them is tiny.

Umm, question - I've seen ISA bridges with the ability to perform 32-bit
DMA using the ISA DMA controllers. AFAIK, Linux doesn't make use of this
feature, except on ARM PCI systems with ISA bridges. Is there a reason
why this isn't used on x86 hardware?

--
Russell King ([email protected]) The developer of ARM Linux
http://www.arm.linux.org.uk/personal/aboutme.html

2003-02-28 16:32:33

by Alan

[permalink] [raw]
Subject: Re: Proposal: Eliminate GFP_DMA

On Fri, 2003-02-28 at 15:44, Dmitry A. Fedorov wrote:
> But why drivers of ISA bus devices with DMA should use pci_* functions?
>
> I'm personally wouldn't have too much pain with GFP_DMA because I have
> compatibility headers and proposed change for them is tiny.

ISA devices or ISA like devices on non x86-32 platforms already need to
do this to work on non PC boxes. I'm hoping the generic DMA framework
stuff will make that all much cleaner for 2.7

2003-02-28 17:17:39

by Ivan Kokshaysky

[permalink] [raw]
Subject: Re: Proposal: Eliminate GFP_DMA

On Fri, Feb 28, 2003 at 04:05:50PM +0000, Russell King wrote:
> Umm, question - I've seen ISA bridges with the ability to perform 32-bit
> DMA using the ISA DMA controllers. AFAIK, Linux doesn't make use of this
> feature, except on ARM PCI systems with ISA bridges.

Alpha uses this from day 1, BTW.
Also, in 2.5 we have "isa_bridge" stuff which was intended exactly for
that - it's a pointer to pci device (real ISA bridge with appropriate
dma_mask) that can be used by non-busmastering ISA devices as a pci_dev *
arg to pci_* mapping functions.

> Is there a reason
> why this isn't used on x86 hardware?

Given a huge number of various ISA bridges found in x86 systems,
I don't see a generic way to determine which ones can do 32-bit DMA...
Maybe kind of white list?

Ivan.

2003-02-28 18:07:23

by Deepak Saxena

[permalink] [raw]
Subject: Re: Proposal: Eliminate GFP_DMA

On Feb 28 2003, at 10:58, Jeff Garzik was caught saying:
> On Fri, Feb 28, 2003 at 09:44:14PM +0600, Dmitry A. Fedorov wrote:
> >
> > But why drivers of ISA bus devices with DMA should use pci_* functions?
> >
> > I'm personally wouldn't have too much pain with GFP_DMA because I have
> > compatibility headers and proposed change for them is tiny.
>
> Do not let the name "pci_" distract, it works for ISA too :)
>
> We can #define pci_xxx isa_xxx if you like :)
>
> Jeff

This discussion raises an issue that I've been meaning to bring up for a bit.
Currently, the DMA-API is defined as returning a cpu physical address [1],
but should the API be redefined as returning an address which is valid on
the bus which the device sits on? For example, on most xscale systems I
deal with, there's not a 1:1 mapping from physical to PCI memory space,
so the address which is valid on the bus is not a valid CPU physical
address. Imagine an SOC system with on-chip Ethernets and such that
that have direct access to the physical bus, plus non-1:1 PCI translation.
In such a case, I would need a physical address for the on-chip devices and
a PCI bus address for the PCI devices. We can extend this case to a SOC
which has on-chip peripherals on some custom FooBus with non-1:1 translation
into physical memory. Now, depending on whether my device is on
the PCI bus or on said FooBus, my dma_addr needs to have a different
meaning.

One easy answer is to provide bus-specific bus_map/unmap/etc functions
such as is done with PCI, but this seems rather ugly to me as now every
custom bus needs a new set of functions which IMNHO defeats the purpose
of a Generic DMA API. I think it would be a much cleaner solution to
define the DMA API itself to return an address that is valid on the
the bus pointed to by dev->bus. I don't think this would require any
implementation changes at the moment, but it would be a more flexible
definition of the API allowing for easy addition of new bus types with DMA
capabilities. All the bus-specific magic would be hidden in the
architecture-specific implementations on platforms that require different
mappings for different buses, making the life of driver writers much
simpler. On systems in which everything is 1:1 mapped between physical,
PCI, and FooBuses, no changes would be required.

~Deepak


[1] From Documentation/DMA-API.txt in the 2.5 BK tree:

dma_addr_t
dma_map_single(struct device *dev, void *cpu_addr, size_t size,
enum dma_data_direction direction)
dma_addr_t
pci_map_single(struct device *dev, void *cpu_addr, size_t size,
int direction)

Maps a piece of processor virtual memory so it can be accessed by the
device and returns the _physical_ handle of the memory.


--
Deepak Saxena, Code Monkey - Ph:480.517.0372 Fax:480.517.0262
MontaVista Software - Powering the Embedded Revolution - http://www.mvista.com

2003-02-28 18:17:12

by Deepak Saxena

[permalink] [raw]
Subject: Re: Proposal: Eliminate GFP_DMA


I would like to add/clarify that I'm not suggesting this only for the
mapping functions, but for the whole DMA API. So for example,
dma_alloc_coherent would again return a dma_handle that is valid on
the bus that dev->bus points to.

~Deepak


--
Deepak Saxena, Code Monkey - Ph:480.517.0372 Fax:480.517.0262
MontaVista Software - Powering the Embedded Revolution - http://www.mvista.com

2003-02-28 19:41:52

by Russell King

[permalink] [raw]
Subject: Re: Proposal: Eliminate GFP_DMA

On Fri, Feb 28, 2003 at 11:17:29AM -0700, Deepak Saxena wrote:
> This discussion raises an issue that I've been meaning to bring up for
> a bit. Currently, the DMA-API is defined as returning a cpu physical
> address [1]

The DMA-API should reflect the same as the PCI-DMA API, namely:

| pci_alloc_consistent returns two values: the virtual address which you
| can use to access it from the CPU and dma_handle which you pass to the
| card.

Translated: not a physical address. "Physical" address to me means
the address that appears on the outside of the CPU after it has passed
through the MMU and is heading for the local CPUs system RAM.

So yes, I think this is a bug that exists in the DMA-API document.

It must be an offsetable cookie that the card on its own bus knows how
to deal with to access the memory associated with the mapping. In plain
speak, the bus address for that device.

> One easy answer is to provide bus-specific bus_map/unmap/etc functions
> such as is done with PCI, but this seems rather ugly to me as now every
> custom bus needs a new set of functions which IMNHO defeats the purpose
> of a Generic DMA API.

You can do this today with the API - test dev->bus_type to determine
the type of the bus, convert to the correct device type and grab the
bus information.

The API hasn't really changed - its just that bugs exist in the
documentation that need fixing before 2.6 comes out. Submit a
patch. 8)

--
Russell King ([email protected]) The developer of ARM Linux
http://www.arm.linux.org.uk/personal/aboutme.html

2003-02-28 20:41:05

by Pete Zaitcev

[permalink] [raw]
Subject: Re: Proposal: Eliminate GFP_DMA

> This discussion raises an issue that I've been meaning to bring up for a bit.
> Currently, the DMA-API is defined as returning a cpu physical address [1],
> but should the API be redefined as returning an address which is valid on
> the bus which the device sits on? [...]

> dma_addr_t
> dma_map_single(struct device *dev, void *cpu_addr, size_t size,
> enum dma_data_direction direction)
> dma_addr_t
> pci_map_single(struct device *dev, void *cpu_addr, size_t size,
> int direction)
>
> Maps a piece of processor virtual memory so it can be accessed by the
> device and returns the _physical_ handle of the memory.

That's only a sloppy documentation, obviously it returns a bus
address, not a physical address. Simply the x86 graduate who wrote
it cannot see the difference, that's all. Pay it no mind and
implement it properly, that means copy it from sparc64.
Who reads documentation anyway when we have the source?

-- Pete

2003-02-28 22:00:42

by Russell King

[permalink] [raw]
Subject: Re: Proposal: Eliminate GFP_DMA

On Fri, Feb 28, 2003 at 03:51:23PM -0500, Pete Zaitcev wrote:
> Pay it no mind and
> implement it properly, that means copy it from sparc64.

and fix the documentation so the next unsuspecting individual doesn't
fall into the same trap and ask the same question on the same mailing
list. 8)

--
Russell King ([email protected]) The developer of ARM Linux
http://www.arm.linux.org.uk/personal/aboutme.html

2003-03-04 17:46:01

by Rogier Wolff

[permalink] [raw]
Subject: Re: Proposal: Eliminate GFP_DMA

On Fri, Feb 28, 2003 at 03:55:37PM +0000, Alan Cox wrote:
> On Fri, 2003-02-28 at 14:34, Matthew Wilcox wrote:
> > umm. are you volunteering to convert drivers/net/macmace.c to the pci_*
> > API then? also, GFP_DMA is used on, eg, s390 to get memory below 2GB and
> > on ia64 to get memory below 4GB.

> The ia64 is a fine example of how broken it is. People have to hack
> around with GFP_DMA meaning different things on ia64 to everything
> else. It needs to die.

All the modifier flags on kmalloc and GFP should be "memory allocation
descriptors".

A memory allocation descriptor is a linked list of something like:
<memory pool pointer>
<function to call when out of memory on this pool>
<next memory pool>

An "atomic" modifier will not have a function pointer that waits for
memory when "out of memory".

A DMA pool descriptor will only point to pools that have that
capability.

Drivers will eventually have to start specifying what they really
need: GFP_1M, GFP_16M, GFP_2G, GFP_4G, GFP_36G, GFP_ATOMIC.

The 16M allocation descriptor will also point "back" to the 1M
allocation descriptor. Thus drivers requesting memory "below 16M"
will prefer to use memory between 1M and 16M because that's first
on the list.

I think that this will simplify things in the long run.

Roger.

--
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2600998 **
*-- BitWizard writes Linux device drivers for any device you may have! --*
* The Worlds Ecosystem is a stable system. Stable systems may experience *
* excursions from the stable situation. We are currently in such an *
* excursion: The stable situation does not include humans. ***************

2003-03-04 21:54:29

by Alan

[permalink] [raw]
Subject: Re: Proposal: Eliminate GFP_DMA

On Tue, 2003-03-04 at 17:56, Rogier Wolff wrote:
> All the modifier flags on kmalloc and GFP should be "memory allocation
> descriptors".
> A DMA pool descriptor will only point to pools that have that
> capability.

Much much too simple. DMA to what from where for example ? Post 2.6 its
IMHO a case of making the equivalent of pci_alloc_* pci_map_* work with
generic device objects now we have them.

2003-03-05 07:33:23

by Rogier Wolff

[permalink] [raw]
Subject: Re: Proposal: Eliminate GFP_DMA

On Tue, Mar 04, 2003 at 11:09:30PM +0000, Alan Cox wrote:
> On Tue, 2003-03-04 at 17:56, Rogier Wolff wrote:
> > All the modifier flags on kmalloc and GFP should be "memory allocation
> > descriptors".
> > A DMA pool descriptor will only point to pools that have that
> > capability.
>
> Much much too simple. DMA to what from where for example ? Post 2.6 its
> IMHO a case of making the equivalent of pci_alloc_* pci_map_* work with
> generic device objects now we have them.

I can't think of a case that doesn't fit my model. So, if you're
saying that it's simple, that's good.

dev1 +----------+ dev2
| | CPU / | |
--------+--------+ BRIDGE +---------+--------
| | ----> | |
mem1 +----------+ mem2

If for example, there are two PCI busses, and a memory controller on
both of them, but no DMA cpability in one direction through the PCI
bridge (embedded CPU) then there should be two memory pools.

In the picture above: dev1 can reach mem2, but dev2 can't reach mem1.

PCI devices on the restricted PCI bus (dev2) will have to pass a
memory allocation descriptor that describes just the memory on
that PCI bus, the other one (dev1) can pass a descriptor that
prefers the non-shared memory, (leaving as much as possible for
the devices on the other bus (bus2)), but
reverts to the memory that the other devices can handle as well.

But if we end up "reading" from dev1, and writing the same data
to dev2 we're better off DMAing it to "mem2". So, how to tell
the dev1 driver that allocing off the allocation descriptor
"mem1, then mem2", is not a good a good idea, and that it should
use "mem2, then mem1" is a problem that remains to be solved...


But if I'm missing a case that can't be modelled, please enlighten
me as to what I'm missing....

Roger.


--
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2600998 **
*-- BitWizard writes Linux device drivers for any device you may have! --*
* The Worlds Ecosystem is a stable system. Stable systems may experience *
* excursions from the stable situation. We are currently in such an *
* excursion: The stable situation does not include humans. ***************

2003-03-05 12:37:54

by Alan

[permalink] [raw]
Subject: Re: Proposal: Eliminate GFP_DMA

On Wed, 2003-03-05 at 07:43, Rogier Wolff wrote:
> PCI devices on the restricted PCI bus (dev2) will have to pass a
> memory allocation descriptor that describes just the memory on
> that PCI bus, the other one (dev1) can pass a descriptor that
> prefers the non-shared memory, (leaving as much as possible for
> the devices on the other bus (bus2)), but
> reverts to the memory that the other devices can handle as well.

Which is actually what I said - you need to have the pci_alloc
equivalents for generic devices and pass the device you wish to
do the allocation for. How that allocation occurs doesn't
matter too much and can be arch specific