On Thu, May 28, 2015 at 03:07:05PM +0300, Dmitry Kalinkin wrote:
> This introduces a new dma device that provides a single ioctl call that
> provides DMA read and write functionality to the user space.
>
> Signed-off-by: Dmitry Kalinkin <[email protected]>
> Cc: Igor Alekseev <[email protected]>
> ---
> drivers/staging/vme/devices/vme_user.c | 201 ++++++++++++++++++++++++++++++++-
> drivers/staging/vme/devices/vme_user.h | 11 ++
> 2 files changed, 209 insertions(+), 3 deletions(-)
I want to get Martyn's feedback on this, as it's adding a new feature to
the subsystem that he's going to have to maintain.
thanks,
greg k-h
On 13/06/15 01:28, Greg Kroah-Hartman wrote:
> On Thu, May 28, 2015 at 03:07:05PM +0300, Dmitry Kalinkin wrote:
>> This introduces a new dma device that provides a single ioctl call that
>> provides DMA read and write functionality to the user space.
>>
>> Signed-off-by: Dmitry Kalinkin <[email protected]>
>> Cc: Igor Alekseev <[email protected]>
>> ---
>> drivers/staging/vme/devices/vme_user.c | 201 ++++++++++++++++++++++++++++++++-
>> drivers/staging/vme/devices/vme_user.h | 11 ++
>> 2 files changed, 209 insertions(+), 3 deletions(-)
>
> I want to get Martyn's feedback on this, as it's adding a new feature to
> the subsystem that he's going to have to maintain.
>
Sorry about the *really* late reply, loads of emails some how missed my
periodic search of the mailing list.
I'm happy with the addition of DMA, just not sure whether it's worth
adding an extra device file just to handle DMA. Could the user space
application not just use the control device?
--
Martyn Welch (Lead Software Engineer) | Registered in England and Wales
GE Intelligent Platforms | (3828642) at 100 Barbirolli Square
T +44(0)1327322748 | Manchester, M2 3AB
E [email protected] | VAT:GB 927559189
On Mon, Jul 6, 2015 at 4:22 PM, Martyn Welch <[email protected]> wrote:
>
> Sorry about the *really* late reply, loads of emails some how missed my
> periodic search of the mailing list.
>
> I'm happy with the addition of DMA, just not sure whether it's worth adding
> an extra device file just to handle DMA. Could the user space application
> not just use the control device?
That would require an additional ioctl field for DMA channel id in case we want
to support both DMA channels on tsi148.
It would make sense to save that device minor if Documentation/devices.txt
was good.
But it has only 4 slave and 4 master windows whereas we would want to
make some parameters for vme_user to configure this allocation numbers up
to 8 slaves and 8 masters.
On 06/07/15 14:50, Dmitry Kalinkin wrote:
> On Mon, Jul 6, 2015 at 4:22 PM, Martyn Welch <[email protected]> wrote:
>>
>> Sorry about the *really* late reply, loads of emails some how missed my
>> periodic search of the mailing list.
>>
>> I'm happy with the addition of DMA, just not sure whether it's worth adding
>> an extra device file just to handle DMA. Could the user space application
>> not just use the control device?
> That would require an additional ioctl field for DMA channel id in case we want
> to support both DMA channels on tsi148.
>
Or just dynamically allocate and free a resource for the DMA operation?
> It would make sense to save that device minor if Documentation/devices.txt
> was good.
> But it has only 4 slave and 4 master windows whereas we would want to
> make some parameters for vme_user to configure this allocation numbers up
> to 8 slaves and 8 masters.
>
The vme_user module was originally envisaged as a mechanism to provide
support for applications that had been written to use the original
driver at vmelinux.org. Some functionality was dropped as it was not
good practice (such as receiving VME interrupts in user space, it's not
really doable if the slave card is Release On Register Access rather
than Release on Acknowledge), so the interface became more of a debug
mechanism for me. Others have clearly found it provides enough for them
to allow drivers to be written in user space.
I was thinking that the opposite might be better, no windows were mapped
at module load, windows could be allocated and mapped using the control
device. This would ensure that unused resources were still available for
kernel based drivers and would mean the driver wouldn't be
pre-allocating a bunch of fairly substantially sized slave window
buffers (the buffers could also be allocated to match the size of the
slave window requested). What do you think?
--
Martyn Welch (Lead Software Engineer) | Registered in England and Wales
GE Intelligent Platforms | (3828642) at 100 Barbirolli Square
T +44(0)1327322748 | Manchester, M2 3AB
E [email protected] | VAT:GB 927559189
On Mon, Jul 6, 2015 at 5:48 PM, Martyn Welch <[email protected]> wrote:
>
>
> On 06/07/15 14:50, Dmitry Kalinkin wrote:
>>
>> On Mon, Jul 6, 2015 at 4:22 PM, Martyn Welch <[email protected]> wrote:
>>>
>>>
>>> Sorry about the *really* late reply, loads of emails some how missed my
>>> periodic search of the mailing list.
>>>
>>> I'm happy with the addition of DMA, just not sure whether it's worth
>>> adding
>>> an extra device file just to handle DMA. Could the user space application
>>> not just use the control device?
>>
>> That would require an additional ioctl field for DMA channel id in case we
>> want
>> to support both DMA channels on tsi148.
>>
>
> Or just dynamically allocate and free a resource for the DMA operation?
That seems to be a too high level.
Also notice how vme_user_dma_ioctl is doing without locks now. Acquiring a
resource for operation would introduce at least one.
>
>> It would make sense to save that device minor if Documentation/devices.txt
>> was good.
>> But it has only 4 slave and 4 master windows whereas we would want to
>> make some parameters for vme_user to configure this allocation numbers up
>> to 8 slaves and 8 masters.
>>
>
> The vme_user module was originally envisaged as a mechanism to provide
> support for applications that had been written to use the original driver at
> vmelinux.org.
That part I never understood. vmelinux.org's cvs has a very dated driver
with a very limited capabilities.
This one looks like a grandpa of the one we have (both tsi148 and universe):
ftp://ftp.prosoft.ru/pub/Hardware/Fastwel/CPx/CPC600/Software/Drivers/Linux/tsi148.tar.gz
There is also VME4L driver by MEN (tsi148 only):
https://www.men.de/software/13z014-90/
Some other driver:
http://www.awa.tohoku.ac.jp/~sanshiro/kinoko-e/vmedrv/
Some other driver (universe only):
https://github.com/mgmarino/VMELinux/blob/master/driver/universe.c
Driver by CERN (dynamic window allocation):
https://github.com/cota/ht-drivers/tree/master/vmebridge/driver
The point is: there are many drivers of different quality. All all of them
include some sort of userspace interface and that, as you mention below,
seems to work well for many cases.
All I'm trying to do is to make vme_user to be at least as useful as
drivers above
without looking back at vmelinux.
> Some functionality was dropped as it was not good practice
> (such as receiving VME interrupts in user space, it's not really doable if
> the slave card is Release On Register Access rather than Release on
> Acknowledge),
Didn't know about RORA. I wonder how different this is compared to the
PCI bus case.
> so the interface became more of a debug mechanism for me.
> Others have clearly found it provides enough for them to allow drivers to be
> written in user space.
>
> I was thinking that the opposite might be better, no windows were mapped at
> module load, windows could be allocated and mapped using the control device.
> This would ensure that unused resources were still available for kernel
> based drivers and would mean the driver wouldn't be pre-allocating a bunch
> of fairly substantially sized slave window buffers (the buffers could also
> be allocated to match the size of the slave window requested). What do you
> think?
I'm not a VME expert, but it seems that VME windows are a quiet limited resource
no matter how you allocate your resources. Theoretically we could put up to 32
different boards in a single crate, so there won't be enough windows for each
driver to allocate. That said, there is no way around this when putting together
a really heterogeneous VME system. To overcome such problem, one could
develop a different kernel API that would not provide windows to the
drivers, but
handle reads and writes by reconfiguring windows on the fly, which in turn would
introduce more latency. Those who need such API are welcome to develop it :)
As for dynamic vme_user device allocation, I don't see the point in this.
The only existing kernel VME driver allocates windows in advance, user is just
to make sure to leave one free window if she wants to use that. Module parameter
for window count will be dynamic enough to handle that.
Cheers,
Dmitry
Hi Dmitry,
On 6 July 2015 at 19:24, Dmitry Kalinkin <[email protected]> wrote:
[...]
>
> I'm not a VME expert, but it seems that VME windows are a quiet limited resource
> no matter how you allocate your resources. Theoretically we could put up to 32
> different boards in a single crate, so there won't be enough windows for each
> driver to allocate. That said, there is no way around this when putting together
> a really heterogeneous VME system. To overcome such problem, one could
> develop a different kernel API that would not provide windows to the
> drivers, but
> handle reads and writes by reconfiguring windows on the fly, which in turn would
> introduce more latency.
In my humble opinion using user-space drivers (as workaround for
limited windows/images) introduce more latency than let VME driver
dynamically configure windows/images. After all VME systems usually
aren't so much dynamic by its nature (Who likes continuously put in
and out a board which requires an insertion force between 20 and 50
kg?) and are instead heavily used in critical contexts often in
non-stop way.
In fact this is a big obstacle for adoption of this VME stack (at
least for us). We use VME a lot and we care about latency as well so
we use only kernel-space drivers for ours VME boards but unfortunately
the VME stack let us to link a single board with a single window/image
(so max 8 boards on tsi148) only. That said that stack has proven to
be very rock solid.
Until now we have used a modified version of the old vmelinux.org
stack for sharing windows/images between all (ours) VME kernel drivers
and we would be very happy to see something similar in vanilla (at
least coalescence two adjacent addresses with same modifiers).
> Those who need such API are welcome to develop it :)
I would be glad to try it if the maintainer is willing to receive this
type of changes.
Ciao,
Alessio
On 06/07/15 18:24, Dmitry Kalinkin wrote:
>> Some functionality was dropped as it was not good practice
>> >(such as receiving VME interrupts in user space, it's not really doable if
>> >the slave card is Release On Register Access rather than Release on
>> >Acknowledge),
> Didn't know about RORA. I wonder how different this is compared to the
> PCI bus case.
Little I suspect. What it does mean is that there's no generic mechanism
for clearing down an interrupt, so a device specific interrupt routine
is required, which needs to be in kernel space.
>> >so the interface became more of a debug mechanism for me.
>> >Others have clearly found it provides enough for them to allow drivers to be
>> >written in user space.
>> >
>> >I was thinking that the opposite might be better, no windows were mapped at
>> >module load, windows could be allocated and mapped using the control device.
>> >This would ensure that unused resources were still available for kernel
>> >based drivers and would mean the driver wouldn't be pre-allocating a bunch
>> >of fairly substantially sized slave window buffers (the buffers could also
>> >be allocated to match the size of the slave window requested). What do you
>> >think?
> I'm not a VME expert, but it seems that VME windows are a quiet limited resource
> no matter how you allocate your resources. Theoretically we could put up to 32
> different boards in a single crate, so there won't be enough windows for each
> driver to allocate. That said, there is no way around this when putting together
> a really heterogeneous VME system. To overcome such problem, one could
> develop a different kernel API that would not provide windows to the
> drivers, but
> handle reads and writes by reconfiguring windows on the fly, which in turn would
> introduce more latency. Those who need such API are welcome to develop it:)
>
The aim of the existing APIs is to provide a mechanism for allocating
resources. You're right, the resources are limited when scaling to a 32
slot crate. There's a number of ways to share the resources, though they
tend to all have trade offs. My experience has been that the majority of
VME systems don't tend to stretch up to 32 cards.
> As for dynamic vme_user device allocation, I don't see the point in this.
> The only existing kernel VME driver allocates windows in advance, user is just
> to make sure to leave one free window if she wants to use that. Module parameter
> for window count will be dynamic enough to handle that.
If vme_user grabs all the VME windows, there are no windows available
for any kernel level VME drivers to use. If a kernel level driver loads
before vme_user and is allocated a window, if vme_user demands 8 windows
(and assuming it doesn't deal with some already having been allocated
gracefully, which it doesn't at the moment) then it doesn't load.
Dynamic allocation would leave "unused" resources available rather than
prospectively hogging them.
--
Martyn Welch (Lead Software Engineer) | Registered in England and Wales
GE Intelligent Platforms | (3828642) at 100 Barbirolli Square
T +44(0)1327322748 | Manchester, M2 3AB
E [email protected] | VAT:GB 927559189
> On 08 Jul 2015, at 16:22, Martyn Welch <[email protected]> wrote:
>
> On 06/07/15 18:24, Dmitry Kalinkin wrote:
>>> Some functionality was dropped as it was not good practice
>>> >(such as receiving VME interrupts in user space, it's not really doable if
>>> >the slave card is Release On Register Access rather than Release on
>>> >Acknowledge),
>> Didn't know about RORA. I wonder how different this is compared to the
>> PCI bus case.
>
> Little I suspect. What it does mean is that there's no generic mechanism for clearing down an interrupt, so a device specific interrupt routine is required, which needs to be in kernel space.
Yet PCI somehow managed to settle with UIO.
Now imagine I am working with a board using vme_user API and I need to
implement interrupts. The PCI case teaches me that I need to write a board specific
UIO driver. My board is ROAK and allows to configure it's interrupt to any level and
any status/id. So I only use a standard vme_irq_request API that generates IACK
cycle for me automatically. I also don’t want to limit my end user with a choice of
interrupt level and status/id and so I make it configurable. In the end I’ve got a very
generic ROAK device driver. What did I do wrong?
Cheers,
Dmitry-
On 08/07/15 16:02, Dmitry Kalinkin wrote:
>
>> On 08 Jul 2015, at 16:22, Martyn Welch <[email protected]> wrote:
>>
>> On 06/07/15 18:24, Dmitry Kalinkin wrote:
>>>> Some functionality was dropped as it was not good practice
>>>>> (such as receiving VME interrupts in user space, it's not really doable if
>>>>> the slave card is Release On Register Access rather than Release on
>>>>> Acknowledge),
>>> Didn't know about RORA. I wonder how different this is compared to the
>>> PCI bus case.
>>
>> Little I suspect. What it does mean is that there's no generic mechanism for clearing down an interrupt, so a device specific interrupt routine is required, which needs to be in kernel space.
> Yet PCI somehow managed to settle with UIO.
"If, on the other hand, your hardware needs some action to be performed
after each interrupt, then you must do it in your kernel module."
https://www.kernel.org/doc/htmldocs/uio-howto/adding_irq_handler.html
> Now imagine I am working with a board using vme_user API and I need to
> implement interrupts. The PCI case teaches me that I need to write a board specific
> UIO driver. My board is ROAK and allows to configure it's interrupt to any level and
> any status/id. So I only use a standard vme_irq_request API that generates IACK
> cycle for me automatically. I also don’t want to limit my end user with a choice of
> interrupt level and status/id and so I make it configurable. In the end I’ve got a very
> generic ROAK device driver. What did I do wrong?
>
Nothing by the sounds of it. If you have ROAK hardware, life is a lot
simpler.
Martyn
> Cheers,
> Dmitry
>
--
Martyn Welch (Lead Software Engineer) | Registered in England and Wales
GE Intelligent Platforms | (3828642) at 100 Barbirolli Square
T +44(0)1327322748 | Manchester, M2 3AB
E [email protected] | VAT:GB 927559189