Hi,
I am currently developing a PCIe data capture card hardware and the
device drivers to drive this. I have implemented DMA on the data
capture and the scatter-gather DMA is implemented in the hardware. I
am testing this in an X86_64 architecture machine with 4 GB of RAM. I
am able to successfully dma data into any memory (dma) address >
0x0000_0001_0000_0000. However, my problem is to dma data to any
address less than this. When I try to DMA data to an address less than
0x0000_0001_0000_0000, the hardware device hangs indicating that the
address does not exist.
I have implemented the DMA mask to be full 64 bit and my hardware is
capable of transfering data to any address < 8TB. I am using kernel
version 2.6.23.11.
Could you please let me know what I might be doing wrong?
Thanks and regards,
Sanka
Find a better answer, faster with the new Yahoo!7 Search. http://www.yahoo7.com.au/search
On Wed, 30 Jul 2008 09:00:58 -0700 (PDT)
Sanka Piyaratna <[email protected]> wrote:
> Hi,
>
> I am currently developing a PCIe data capture card hardware and the
> device drivers to drive this. I have implemented DMA on the data
> capture and the scatter-gather DMA is implemented in the hardware. I
> am testing this in an X86_64 architecture machine with 4 GB of RAM. I
> am able to successfully dma data into any memory (dma) address >
> 0x0000_0001_0000_0000. However, my problem is to dma data to any
> address less than this. When I try to DMA data to an address less than
> 0x0000_0001_0000_0000, the hardware device hangs indicating that the
> address does not exist.
Assuming the failures are in the range 640K-1MB then I would imagine your
bridge doesn't permit transfers via DMA to the ISA hole.
Hi Alan,
Actually the failures occur at addresses lot higher than this range, I am noticing the failure at dma address 0x0000000037845000. This would be at around 900MB ?
Thanks
Sanka
----- Original Message ----
From: Alan Cox <[email protected]>
To: Sanka Piyaratna <[email protected]>
Cc: [email protected]
Sent: Thursday, 31 July, 2008 1:17:18 AM
Subject: Re: PCIe device driver question
On Wed, 30 Jul 2008 09:00:58 -0700 (PDT)
Sanka Piyaratna <[email protected]> wrote:
> Hi,
>
> I am currently developing a PCIe data capture card hardware and the
> device drivers to drive this. I have implemented DMA on the data
> capture and the scatter-gather DMA is implemented in the hardware. I
> am testing this in an X86_64 architecture machine with 4 GB of RAM. I
> am able to successfully dma data into any memory (dma) address >
> 0x0000_0001_0000_0000. However, my problem is to dma data to any
> address less than this. When I try to DMA data to an address less than
> 0x0000_0001_0000_0000, the hardware device hangs indicating that the
> address does not exist.
Assuming the failures are in the range 640K-1MB then I would imagine your
bridge doesn't permit transfers via DMA to the ISA hole.
Find a better answer, faster with the new Yahoo!7 Search. http://www.yahoo7.com.au/search
> > am testing this in an X86_64 architecture machine with 4 GB of RAM. I
> > am able to successfully dma data into any memory (dma) address >
> > 0x0000_0001_0000_0000.
How can you DMA "successfully" into this address which is > 4 GB when
you have only 4 GB RAM ? Or am I missing something ?
V. Radhakrishnan
http://www.atr-labs.com
On Wed, 2008-07-30 at 09:09 -0700, Sanka Piyaratna wrote:
> Hi Alan,
>
> Actually the failures occur at addresses lot higher than this range, I am noticing the failure at dma address 0x0000000037845000. This would be at around 900MB ?
>
> Thanks
>
> Sanka
>
>
>
> ----- Original Message ----
> From: Alan Cox <[email protected]>
> To: Sanka Piyaratna <[email protected]>
> Cc: [email protected]
> Sent: Thursday, 31 July, 2008 1:17:18 AM
> Subject: Re: PCIe device driver question
>
> On Wed, 30 Jul 2008 09:00:58 -0700 (PDT)
> Sanka Piyaratna <[email protected]> wrote:
>
> > Hi,
> >
> > I am currently developing a PCIe data capture card hardware and the
> > device drivers to drive this. I have implemented DMA on the data
> > capture and the scatter-gather DMA is implemented in the hardware. I
> > am testing this in an X86_64 architecture machine with 4 GB of RAM. I
> > am able to successfully dma data into any memory (dma) address >
> > 0x0000_0001_0000_0000. However, my problem is to dma data to any
> > address less than this. When I try to DMA data to an address less than
> > 0x0000_0001_0000_0000, the hardware device hangs indicating that the
> > address does not exist.
>
> Assuming the failures are in the range 640K-1MB then I would imagine your
> bridge doesn't permit transfers via DMA to the ISA hole.
>
>
>
> Find a better answer, faster with the new Yahoo!7 Search. http://www.yahoo7.com.au/search
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
On Wed, 30 Jul 2008 09:09:36 -0700 (PDT)
Sanka Piyaratna <[email protected]> wrote:
> Hi Alan,
>
> Actually the failures occur at addresses lot higher than this range, I am noticing the failure at dma address 0x0000000037845000. This would be at around 900MB ?
For x86 we don't do anything special and if your hardware is flagging the
memory as absent perhaps you should consult the hardware/bridge
documentation firt ?
In some cases (over 4GB and 32bit DMA masks set) we will use the GART on
x86-64 but that will still see addresses below 4GB being sent to the
device.
Alan
V.Radhakrishnan wrote:
>>> am testing this in an X86_64 architecture machine with 4 GB of RAM. I
>>> am able to successfully dma data into any memory (dma) address >
>>> 0x0000_0001_0000_0000.
>
> How can you DMA "successfully" into this address which is > 4 GB when
> you have only 4 GB RAM ? Or am I missing something ?
The MMIO and other reserved memory space at the top of the 32-bit memory
space will cause the top part of memory to be relocated above 4GB.
Sanka Piyaratna wrote:
> Hi,
>
> I am currently developing a PCIe data capture card hardware and the
> device drivers to drive this. I have implemented DMA on the data
> capture and the scatter-gather DMA is implemented in the hardware. I
> am testing this in an X86_64 architecture machine with 4 GB of RAM. I
> am able to successfully dma data into any memory (dma) address >
> 0x0000_0001_0000_0000. However, my problem is to dma data to any
> address less than this. When I try to DMA data to an address less than
> 0x0000_0001_0000_0000, the hardware device hangs indicating that the
> address does not exist.
>
> I have implemented the DMA mask to be full 64 bit and my hardware is
> capable of transfering data to any address < 8TB. I am using kernel
> version 2.6.23.11.
>
> Could you please let me know what I might be doing wrong?
The kernel can't do anything to stop you from DMAing anywhere you want
(barring the system having special IOMMU hardware). If you overwrite
something you shouldn't have you'll cause a crash, but the kernel has no
influence on it really.
Unless you're messing up the DMA addresses somehow and writing into a
space that's not actually RAM (like the MMIO memory hole or something),
my guess is it's likely a hardware problem..
I allocate memory in the user land using memalign function (typically I allocate about 500 MB) and pass this to the kernel space. In my device driver, I call get_user_pages() to lock down the memory and extract the relevant pages. A scatter-gather list is generated using these page addresses and hence derive the dma_addresses using page_to_phys() function. These addresses are programmed into a FIFO in the hardware device using a memory mapped register interface (PCI BAR based). Subsequently the hardware start filling up the pages and interrupt when a block of pages are complete. I notice the hardware hang (PCIe packets don't seem to get the acknowledgements from the root complex) when the DMA address is < 0x0000_0001_0000_0000. I have verified in the hardware that the PCIe packet is created with the correct address as programed by the device driver dma_address. If i can guard some how that the memory allocation is with in a certain area, I can stop the
problem from occuring. Are there any bridge functionality in the Intel architecture that may mask a certain region of memory?
Thanks and regards,
Sanka
----- Original Message ----
From: Robert Hancock <[email protected]>
To: Sanka Piyaratna <[email protected]>
Cc: [email protected]
Sent: Thursday, 31 July, 2008 4:54:48 AM
Subject: Re: PCIe device driver question
Sanka Piyaratna wrote:
> Hi,
>
> I am currently developing a PCIe data capture card hardware and the
> device drivers to drive this. I have implemented DMA on the data
> capture and the scatter-gather DMA is implemented in the hardware. I
> am testing this in an X86_64 architecture machine with 4 GB of RAM. I
> am able to successfully dma data into any memory (dma) address >
> 0x0000_0001_0000_0000. However, my problem is to dma data to any
> address less than this. When I try to DMA data to an address less than
> 0x0000_0001_0000_0000, the hardware device hangs indicating that the
> address does not exist.
>
> I have implemented the DMA mask to be full 64 bit and my hardware is
> capable of transfering data to any address < 8TB. I am using kernel
> version 2.6.23.11.
>
> Could you please let me know what I might be doing wrong?
The kernel can't do anything to stop you from DMAing anywhere you want
(barring the system having special IOMMU hardware). If you overwrite
something you shouldn't have you'll cause a crash, but the kernel has no
influence on it really.
Unless you're messing up the DMA addresses somehow and writing into a
space that's not actually RAM (like the MMIO memory hole or something),
my guess is it's likely a hardware problem..
Find a better answer, faster with the new Yahoo!7 Search. http://www.yahoo7.com.au/search
Sanka Piyaratna wrote:
> I allocate memory in the user land using memalign function (typically I allocate about 500 MB) and pass this to the kernel space. In my device driver, I call get_user_pages() to lock down the memory and extract the relevant pages. A scatter-gather list is generated using these page addresses and hence derive the dma_addresses using page_to_phys() function. These addresses are programmed into a FIFO in the hardware device using a memory mapped register interface (PCI BAR based). Subsequently the hardware start filling up the pages and interrupt when a block of pages are complete. I notice the hardware hang (PCIe packets don't seem to get the acknowledgements from the root complex) when the DMA address is < 0x0000_0001_0000_0000. I have verified in the hardware that the PCIe packet is created with the correct address as programed by the device driver dma_address. If i can guard some how that the memory allocation is with in a certain area, I can stop the
> problem from occuring. Are there any bridge functionality in the Intel architecture that may mask a certain region of memory?
How are you formatting the addresses in the TLP? The PCI Express spec
says that for addresses below 4GB the 32-bit addressing format must be
used (you can't use the 64-bit format unconditionally). It could be that
is what is making the chipset unhappy.
Hi Robert,
Thanks for that, I did not notice that in the spec. I am currently using 64 bit format unconditionally and this must be what is going wrong.
Thank you so much for that.
Sanka
----- Original Message ----
From: Robert Hancock <[email protected]>
To: Sanka Piyaratna <[email protected]>
Cc: [email protected]
Sent: Thursday, 31 July, 2008 6:14:54 PM
Subject: Re: PCIe device driver question
Sanka Piyaratna wrote:
> I allocate memory in the user land using memalign function (typically I allocate about 500 MB) and pass this to the kernel space. In my device driver, I call get_user_pages() to lock down the memory and extract the relevant pages. A scatter-gather list is generated using these page addresses and hence derive the dma_addresses using page_to_phys() function. These addresses are programmed into a FIFO in the hardware device using a memory mapped register interface (PCI BAR based). Subsequently the hardware start filling up the pages and interrupt when a block of pages are complete. I notice the hardware hang (PCIe packets don't seem to get the acknowledgements from the root complex) when the DMA address is < 0x0000_0001_0000_0000. I have verified in the hardware that the PCIe packet is created with the correct address as programed by the device driver dma_address. If i can guard some how that the memory allocation is with in a certain area, I can stop the
> problem from occuring. Are there any bridge functionality in the Intel architecture that may mask a certain region of memory?
How are you formatting the addresses in the TLP? The PCI Express spec
says that for addresses below 4GB the 32-bit addressing format must be
used (you can't use the 64-bit format unconditionally). It could be that
is what is making the chipset unhappy.
Find a better answer, faster with the new Yahoo!7 Search. http://www.yahoo7.com.au/search
Hi Robert,
Thanks for the reply. I was thinking that the MMIO and reserve memory
will be below 4 GB was only applicable for 32-bit environments, since I
don't have much experience in 64-bit.
However, I had an IDENTICAL problem over 2 years ago. I had used
posix_memalign() in user space to allocate pages aligned to 4096 byte
pages, allocated several additional memaligned pages in user space, used
mlock() to lock all these pages, gathered the user space addresses into
the original pages as arrays of structures, passed this array into the
kernel using an ioctl() call, used get_user_pages() to extract the
struct page pointers, performed a kmap() to get the kernel virtual
addresses and then extracted the physical addresses and 'sent' this to
the chip to perform DMA.
This situation is almost identical to what has been reported and hence
my interest.
However, I had a PCI access problem. The DMA was just NOT happening on
any machine which had highmem, i.e over 896 MB.
I "solved" the problem since I didn't have much time to do R&D, by
booting with kernel command line option of mem=512M and the DMA went
thru successfully.
This was the linux-2.6.15 kernel then. Since the project was basically
to test the DMA capability of the device, the actual address to where it
was DMA-ed didn't matter, and I got paid for my work. However, this
matter was always at the back of my head.
What could have been the problem with the x86 32-bit PCI ?
Thanks and regards
V. Radhakrishnan
http://www.atr-labs.com
On Wed, 2008-07-30 at 13:21 -0600, Robert Hancock wrote:
> V.Radhakrishnan wrote:
> >>> am testing this in an X86_64 architecture machine with 4 GB of RAM. I
> >>> am able to successfully dma data into any memory (dma) address >
> >>> 0x0000_0001_0000_0000.
> >
> > How can you DMA "successfully" into this address which is > 4 GB when
> > you have only 4 GB RAM ? Or am I missing something ?
>
> The MMIO and other reserved memory space at the top of the 32-bit memory
> space will cause the top part of memory to be relocated above 4GB.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
V.Radhakrishnan wrote:
> Hi Robert,
>
> Thanks for the reply. I was thinking that the MMIO and reserve memory
> will be below 4 GB was only applicable for 32-bit environments, since I
> don't have much experience in 64-bit.
>
> However, I had an IDENTICAL problem over 2 years ago. I had used
> posix_memalign() in user space to allocate pages aligned to 4096 byte
> pages, allocated several additional memaligned pages in user space, used
> mlock() to lock all these pages, gathered the user space addresses into
> the original pages as arrays of structures, passed this array into the
> kernel using an ioctl() call, used get_user_pages() to extract the
> struct page pointers, performed a kmap() to get the kernel virtual
> addresses and then extracted the physical addresses and 'sent' this to
> the chip to perform DMA.
>
> This situation is almost identical to what has been reported and hence
> my interest.
>
> However, I had a PCI access problem. The DMA was just NOT happening on
> any machine which had highmem, i.e over 896 MB.
My guess there was a bug in your DMA mapping code. I don't think kmap is
what is normally used for this. I think with get_user_pages one usually
takes the returned page pointers to create an SG list and uses
dma_map_sg to create a DMA mapping for them.
>
> I "solved" the problem since I didn't have much time to do R&D, by
> booting with kernel command line option of mem=512M and the DMA went
> thru successfully.
>
> This was the linux-2.6.15 kernel then. Since the project was basically
> to test the DMA capability of the device, the actual address to where it
> was DMA-ed didn't matter, and I got paid for my work. However, this
> matter was always at the back of my head.
>
> What could have been the problem with the x86 32-bit PCI ?
>
> Thanks and regards
>
> V. Radhakrishnan
> http://www.atr-labs.com
>
> On Wed, 2008-07-30 at 13:21 -0600, Robert Hancock wrote:
>> V.Radhakrishnan wrote:
>>>>> am testing this in an X86_64 architecture machine with 4 GB of RAM. I
>>>>> am able to successfully dma data into any memory (dma) address >
>>>>> 0x0000_0001_0000_0000.
>>> How can you DMA "successfully" into this address which is > 4 GB when
>>> you have only 4 GB RAM ? Or am I missing something ?
>> The MMIO and other reserved memory space at the top of the 32-bit memory
>> space will cause the top part of memory to be relocated above 4GB.
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>> the body of a message to [email protected]
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>> Please read the FAQ at http://www.tux.org/lkml/
>
>
> My guess there was a bug in your DMA mapping code. I don't think kmap is
> what is normally used for this. I think with get_user_pages one usually
> takes the returned page pointers to create an SG list and uses
> dma_map_sg to create a DMA mapping for them.
Looking at the actual code, I see that I had used kmap() only to obtain
kernel virtual addresses for the array of struct pages obtained from
user space by using get_user_pages.
Subsequently, I had used dma_map_single() and dma_unmap_single() calls
for single buffer calls.
The code didn't have bugs IMHO since it was used for extensive stress
testing the initial FPGA prototype as well as the final ASIC chip ,
sometimes running for over 4 days non-stop without breaking.
However, using Test Access Points on the board and using a Logic
Analyzer showed that DMA was NOT taking place when RAM > 896 MB was
used. The hardware gurus said that PCI bus cycles just didn't seem to be
taking place when RAM > 896 MB was used as the source OR destination
address.
Perhaps this was a problem in the earlier kernel(s) and has since been
rectified ? ( I was using 2.6.15 then ... )
I am just curious since Sanka Piyaratna reported a 'similar' kind of
situation.
Regards
V. Radhakrishnan
On Thu, 2008-07-31 at 11:37 -0600, Robert Hancock wrote:
> V.Radhakrishnan wrote:
> > Hi Robert,
> >
> > Thanks for the reply. I was thinking that the MMIO and reserve memory
> > will be below 4 GB was only applicable for 32-bit environments, since I
> > don't have much experience in 64-bit.
> >
> > However, I had an IDENTICAL problem over 2 years ago. I had used
> > posix_memalign() in user space to allocate pages aligned to 4096 byte
> > pages, allocated several additional memaligned pages in user space, used
> > mlock() to lock all these pages, gathered the user space addresses into
> > the original pages as arrays of structures, passed this array into the
> > kernel using an ioctl() call, used get_user_pages() to extract the
> > struct page pointers, performed a kmap() to get the kernel virtual
> > addresses and then extracted the physical addresses and 'sent' this to
> > the chip to perform DMA.
> >
> > This situation is almost identical to what has been reported and hence
> > my interest.
> >
> > However, I had a PCI access problem. The DMA was just NOT happening on
> > any machine which had highmem, i.e over 896 MB.
>
> My guess there was a bug in your DMA mapping code. I don't think kmap is
> what is normally used for this. I think with get_user_pages one usually
> takes the returned page pointers to create an SG list and uses
> dma_map_sg to create a DMA mapping for them.
>
> >
> > I "solved" the problem since I didn't have much time to do R&D, by
> > booting with kernel command line option of mem=512M and the DMA went
> > thru successfully.
> >
> > This was the linux-2.6.15 kernel then. Since the project was basically
> > to test the DMA capability of the device, the actual address to where it
> > was DMA-ed didn't matter, and I got paid for my work. However, this
> > matter was always at the back of my head.
> >
> > What could have been the problem with the x86 32-bit PCI ?
> >
> > Thanks and regards
> >
> > V. Radhakrishnan
> > http://www.atr-labs.com
> >
> > On Wed, 2008-07-30 at 13:21 -0600, Robert Hancock wrote:
> >> V.Radhakrishnan wrote:
> >>>>> am testing this in an X86_64 architecture machine with 4 GB of RAM. I
> >>>>> am able to successfully dma data into any memory (dma) address >
> >>>>> 0x0000_0001_0000_0000.
> >>> How can you DMA "successfully" into this address which is > 4 GB when
> >>> you have only 4 GB RAM ? Or am I missing something ?
> >> The MMIO and other reserved memory space at the top of the 32-bit memory
> >> space will cause the top part of memory to be relocated above 4GB.
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> >> the body of a message to [email protected]
> >> More majordomo info at http://vger.kernel.org/majordomo-info.html
> >> Please read the FAQ at http://www.tux.org/lkml/
> >
> >
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
V.Radhakrishnan wrote:
>> My guess there was a bug in your DMA mapping code. I don't think kmap is
>> what is normally used for this. I think with get_user_pages one usually
>> takes the returned page pointers to create an SG list and uses
>> dma_map_sg to create a DMA mapping for them.
>
> Looking at the actual code, I see that I had used kmap() only to obtain
> kernel virtual addresses for the array of struct pages obtained from
> user space by using get_user_pages.
>
> Subsequently, I had used dma_map_single() and dma_unmap_single() calls
> for single buffer calls.
I'm suspicious about this usage, I don't know if that will actually
work. There is a dma_map_page call which is meant for doing a DMA
mapping on a struct page which should likely be used instead.
>
> The code didn't have bugs IMHO since it was used for extensive stress
> testing the initial FPGA prototype as well as the final ASIC chip ,
> sometimes running for over 4 days non-stop without breaking.
>
> However, using Test Access Points on the board and using a Logic
> Analyzer showed that DMA was NOT taking place when RAM > 896 MB was
> used. The hardware gurus said that PCI bus cycles just didn't seem to be
> taking place when RAM > 896 MB was used as the source OR destination
> address.
Are you sure the address being passed to the device was correct in this
case? There should be nothing magical about 896MB from a hardware point
of view, and the kernel in general cannot stop you from DMAing anywhere
you like.
>
> Perhaps this was a problem in the earlier kernel(s) and has since been
> rectified ? ( I was using 2.6.15 then ... )
>
> I am just curious since Sanka Piyaratna reported a 'similar' kind of
> situation.
>
> Regards
>
> V. Radhakrishnan
Thanks Robert !
Unfortunately, I cannot test this with recent kernels, since this was a
WiMAX card based on a CardBus inteface and the Toshiba laptop I have
has only 512 MB RAM ( though I have PCs with more RAM, the PCs don't
have a CardBus interface ).
BRgds
RK
On Thu, 2008-07-31 at 12:52 -0600, Robert Hancock wrote:
> V.Radhakrishnan wrote:
> >> My guess there was a bug in your DMA mapping code. I don't think kmap is
> >> what is normally used for this. I think with get_user_pages one usually
> >> takes the returned page pointers to create an SG list and uses
> >> dma_map_sg to create a DMA mapping for them.
> >
> > Looking at the actual code, I see that I had used kmap() only to obtain
> > kernel virtual addresses for the array of struct pages obtained from
> > user space by using get_user_pages.
> >
> > Subsequently, I had used dma_map_single() and dma_unmap_single() calls
> > for single buffer calls.
>
> I'm suspicious about this usage, I don't know if that will actually
> work. There is a dma_map_page call which is meant for doing a DMA
> mapping on a struct page which should likely be used instead.
>
> >
> > The code didn't have bugs IMHO since it was used for extensive stress
> > testing the initial FPGA prototype as well as the final ASIC chip ,
> > sometimes running for over 4 days non-stop without breaking.
> >
> > However, using Test Access Points on the board and using a Logic
> > Analyzer showed that DMA was NOT taking place when RAM > 896 MB was
> > used. The hardware gurus said that PCI bus cycles just didn't seem to be
> > taking place when RAM > 896 MB was used as the source OR destination
> > address.
>
> Are you sure the address being passed to the device was correct in this
> case? There should be nothing magical about 896MB from a hardware point
> of view, and the kernel in general cannot stop you from DMAing anywhere
> you like.
>
> >
> > Perhaps this was a problem in the earlier kernel(s) and has since been
> > rectified ? ( I was using 2.6.15 then ... )
> >
> > I am just curious since Sanka Piyaratna reported a 'similar' kind of
> > situation.
> >
> > Regards
> >
> > V. Radhakrishnan
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/