Hello,
[As I'm not subscribed the list, please include me in Cc: for answers]
I am currently porting 2.6 to a home-made MIPS-based platform using
the MIPS RM9000 processor and the Marvell memory/uart/ethernet
controller.
My physical memory mapping is a bit special : I have 384 MB of
memory. The first 256MB are directly connected to the RM9000, while
the last 128MB are connected to the Marvell controller. _Only_ the
last 128MB are usable for DMA (especially for network traffic). For
the moment, Linux only takes care of the first 256MB, but I can change
it to take care of the complete physical memory space (384 MB).
My problem is the allocation of skbuff. They are allocated using
alloc_skb() in net/core/skbuff.c, and uses the "normal" kmalloc()
allocator. kmalloc() will allocate memory somewhere in the physical
memory space : even if a I allow Linux to allocate memory between
256MB and 384MB, I cannot be sure that it will use memory in this
space to allocate skbuff. If skbuff are not allocated in this space,
then I can't use DMA to transfer the buffers.
As I understand the ZONE_DMA thing, it allows to tell Linux that a
physical memory region located between 0 and some value (16 MB on PCs
for old ISA cards compatibility) is the only area usable for DMA. How
could I declare my 256MB-384MB physical memory reagion to be the only
area usable for DMA ? How can I tell the skbuff functions to allocate
_only_ DMA-able memory ? Moreover, can I make assumptions on the
alignement of final data at the bottom of the network stack (my DMA
controller doesn't like the 2 byte-aligned things).
At the moment, I see only three solutions. The two first aren't not
very satisfying, the third might be a solution, but not perfect
neither (and not sure it would work).
1) Implement a home-made memory allocator dedicated to the allocation
of DMA buffers inside the 256MB-384MB space. Then modify the
net/core/skbuff.c functions to use this allocator to allocate/free
the contents (skb->data) of the skbuffs. I'm not sure that it will
work, but at least, it involves the modification of
architecture-independent code for an architecture-dependent
reason.
2) Modify the Marvell Ethernet driver (drivers/net/mv64340_eth.c) to
change the calls to pci_map_single() and
pci_unmap_single(). The pci_map_single() would allocate (through a
dedicated home-made allocator) a DMA buffer, and copy the contents
of the skbuf to the DMA buffer. The pci_unmap_single() would copy
the contents of the DMA buffer back to the skb->data buffer. I'm
quite sure this would work (this is how the 2.4 port that I have
for this platform work), but it involves the modification of the
Ethernet driver, and above all, a performance hit.
3) Modify net/core/skbuf.c to make sure all kmalloc()'ed areas (for
skbuff contents) are allocated with the GFP_DMA flag. Then, modify
the arch/mips/mm/init.c file to make sure the first 256MB physical
pages don't have the DMA bit, and that the next 128MB will have it
(not sure on how complex it is).
Are there any other solutions available ? If not, which of the
proposed solutions is the best ?
If my problem is unclear, don't hesitate to ask for further details,
Thanks,
Thomas
--
PETAZZONI Thomas - [email protected]
http://thomas.enix.org - Jabber: [email protected]
KOS: http://kos.enix.org/ - Lolut: http://lolut.utbm.info
Fingerprint : 0BE1 4CF3 CEA4 AC9D CC6E 1624 F653 CB30 98D3 F7A7
On Tue, Sep 28, 2004 at 12:08:31PM +0200, Thomas Petazzoni wrote:
> My physical memory mapping is a bit special : I have 384 MB of
> memory. The first 256MB are directly connected to the RM9000, while
> the last 128MB are connected to the Marvell controller. _Only_ the
> last 128MB are usable for DMA (especially for network traffic). For
> the moment, Linux only takes care of the first 256MB, but I can change
> it to take care of the complete physical memory space (384 MB).
>
> My problem is the allocation of skbuff. They are allocated using
> alloc_skb() in net/core/skbuff.c, and uses the "normal" kmalloc()
> allocator. kmalloc() will allocate memory somewhere in the physical
> memory space : even if a I allow Linux to allocate memory between
> 256MB and 384MB, I cannot be sure that it will use memory in this
> space to allocate skbuff. If skbuff are not allocated in this space,
> then I can't use DMA to transfer the buffers.
>
> As I understand the ZONE_DMA thing, it allows to tell Linux that a
> physical memory region located between 0 and some value (16 MB on PCs
> for old ISA cards compatibility) is the only area usable for DMA. How
> could I declare my 256MB-384MB physical memory reagion to be the only
> area usable for DMA ? How can I tell the skbuff functions to allocate
> _only_ DMA-able memory ?
ZONE_DMA has a system specific meaning. On a PCI system ISA could always
be exist through a PCI-to-ISA bridge, so you can't just go and give it
a system specific meaning. It's also needed for PCI devices with a
less than 32-bit DMA limit; those exist in a rich variety.
> Moreover, can I make assumptions on the
> alignement of final data at the bottom of the network stack (my DMA
> controller doesn't like the 2 byte-aligned things).
Well, if you put packets on an aligned address you'll later take a bunch
of missalignment exceptions which are going to severly impact networking
performance ...
> At the moment, I see only three solutions. The two first aren't not
> very satisfying, the third might be a solution, but not perfect
> neither (and not sure it would work).
Change the configuration of the board to put the MV memory at the bottom.
Leave ZONE_DMA what it used to be, < 16MB. Set the ZONE_NORMAL limit to
128MB. Anything above that is non-dmable will go into ZONE_HIGHMEM.
See also CONFIG_LIMITED_DMA in 2.6. It works, it has little compatibility
problems but it's a solution for platform that simply doesn't reflect the
Linux hw architecture very much ...
Ralf