OPENIB.ORG INFINIBAND DRIVERS RELEASE -- 2004-03-05
We are proud to announce the initial availability of the OpenIB.org
InfiniBand stack. This is a fully open source[1] InfiniBand stack
that includes a low-level driver for Mellanox HCA hardware, a
midlayer, and a number of useful upper-layer protocols including
IP-over-InfiniBand, SCSI RDMA Protocol, Sockets Direct Protocol,
uDAPL and MPI. In addition userspace utilities are available,
including the OpenSM subnet manager.
[1] See the section on SDP intellectual property for some important
information, though.
An initial drop of this source is available from
<http://openib.org>. Additional protocols and further enhancements
will be added in upcoming releases. In the next few days, we will
also be bringing up mailing lists, source code control, and so on.
We look forward to working with the entire Linux community to
continue to develop these drivers.
TODO
We are fully aware that our code is far from being acceptable for
inclusion in the Linux kernel, and we are very committed to doing
the hard work required to clean up the source. We believe that in
addition to the benefits of being in the standard kernel, we will
find huge improvements in the quality, performance and robustness of
our drivers.
As a starting point, here is a list of some of the tasks we believe
must be completed before these drivers can be seriously considered
for inclusion in the kernel. We would welcome any suggestions of
tasks that we have forgotten. Of course patches would be even
better!
* Use a more standard naming convention (tTS_IB_TYPE_NAME ->
ib_type_name, tsIbFunctionName -> ib_function_name, etc)
* Get rid of bizarro types such as tUINT32 -- replace with Linux
standard u32 etc.
* Run source through scripts/Lindent (and then fix by hand, since
lots of things will no longer line up)
* Get rid of XXX_exports.c files and move EXPORT_SYMBOL next to
definition in source.
* Change MODULE_PARM -> module_param
* Use DMA mapping API so we work properly with non-coherent
caches, IOMMUs etc. Add a function to get struct device * for an
IB device so ULPs can call sync functions (embed struct device
in struct ib_device?). (How do we handle systems with limited
DMA mapping capacity?)
* Fix up ioctls, userspace interface so that 32-bit userspace
works with a 64-bit kernel.
* Get rid of ib_poll module; move all timers to standard Linux
timers (CM can use a work queue plus schedule_delayed_work).
* Get rid of ib_services module (move TS_TRACE etc. to dev_printk)
* Add missing pieces of API for currently unused IB features: deal
with RD (EECs, etc -- including CM support), verbs extensions.
* Get rid of IPoIB address hash and just use native 20-byte
addresses (keep 6-byte hash as an option for compatibility
with existing applications?)
* Rewrite client query support (SA and DM) to be more extensible,
simpler, deal with component mask.
* Add sysfs support.
* In Mellanox HCA driver, clean up mosal. Some obvious
targets (there's lots more though):
o Grunging through kernel code to find the mlock/munlock
pointers especially has to go (mosal_mlock.c). We need to
get VM experts from LKML to tell us how to handle memory
translation and locking.
o Turn MT_KERNEL back to __KERNEL__
o Use native spinlocks.
o Turn driver into a real PCI driver (rather than
reimplementing a worse version of PCI bus scan).
SDP intellectual property
Microsoft believes that they own certain intellectual property
relating to the Sockets Direct Protocol (SDP)[2]. Therefore, we are
including the following disclaimer required by Microsoft's license
in SDP source that relates to the implementation of the protocol:
"This source code may incorporate intellectual property owned by
Microsoft Corporation. Our provision of this source code does not
include any licenses or any other rights to you under any
Microsoft intellectual property. If you would like a license from
Microsoft (e.g., to rebrand, redistribute), you need to contact
Microsoft directly."
We realize that this is incompatible with open source licensing, and
we are working to find a more satisfactory solution, but for the
time being we are forced to comply with Microsoft's license.
Please make sure you have fully understood the implications of
Microsoft's claims before you redistribute any of the SDP source
that contains the above disclaimer.
[2] <http://www.microsoft.com/mscorp/ip/standards/>
Several people have requested that we split the possibly-encumbered
SDP code into a separate package so that they don't have to download
it to get the free code. To allow that, I have split the kernel code
into two packages:
infiniband-kernel-2004-03-05.tar.bz2
infiniband-kernel-sdp-2004-03-05.tar.bz2
If you wish to build SDP, you will need to download both packages and
move the SDP files into the appropriate place in the tree. For all
other protocols, only the first package (containing only pure dual
GPL/BSD code) is required.
The new snapshot is available now from <http://openib.org/downloads>.
(It also contains a few fixes and code cleanups as compared to the
2004-02-26 drop)
On Sat, Mar 06, 2004 at 01:00:59PM -0800, Roland Dreier wrote:
> Several people have requested that we split the possibly-encumbered
> SDP code into a separate package so that they don't have to download
> it to get the free code. To allow that, I have split the kernel code
> into two packages:
>
> infiniband-kernel-2004-03-05.tar.bz2
> infiniband-kernel-sdp-2004-03-05.tar.bz2
>
Since the openib.org mailing lists don't seem to be alive yet, I'll
bring this up here..
Can we get this split out into the following components?
* kernel level infiniband access layer
* lowlevel hardware driver (aka mellanox driver)
* all other 'upper layer protocols'
All the existing code has big problems with how it interfaces to the
kernel memory management.. it doesn't use the regular interfaces like
get_user_pages, pci_map_page, and pci_map_sq, and goes and looks at the
guts of the pagetables directly.
This is either a problem with the kernel interfaces made available, or a
design issue with the existing codebases. Can someone please tell me
which it is?
--
--------------------------------------------------------------------------
Troy Benjegerdes 'da hozer' [email protected]
Somone asked my why I work on this free (http://www.fsf.org/philosophy/)
software stuff and not get a real job. Charles Shultz had the best answer:
"Why do musicians compose symphonies and poets write poems? They do it
because life wouldn't have any meaning for them if they didn't. That's why
I draw cartoons. It's my life." -- Charles Shultz
Troy> Since the openib.org mailing lists don't seem to be alive
Troy> yet, I'll bring this up here..
Troy> Can we get this split out into the following components?
Troy> * kernel level infiniband access layer
Troy> * lowlevel hardware driver (aka mellanox driver)
Troy> * all other 'upper layer protocols'
Right now, in the infiniband-kernel package, the Mellanox driver is
under drivers/infiniband/hw/mellanox-hca, the 'upper layer protocols'
are under drivers/infiniband/ulp, and the 'access layer' is everything
else in drivers/infiniband. I'm not sure it's worth rolling three
packages for that split right now.
Troy> All the existing code has big problems with how it
Troy> interfaces to the kernel memory management.. it doesn't use
Troy> the regular interfaces like get_user_pages, pci_map_page,
Troy> and pci_map_sq, and goes and looks at the guts of the
Troy> pagetables directly.
Troy> This is either a problem with the kernel interfaces made
Troy> available, or a design issue with the existing
Troy> codebases. Can someone please tell me which it is?
I would say both. Certainly the HCA driver code could have been
written much more cleanly. However, it's not clear how the right way
to handle the memory management requirements of kernel bypass/RDMA
protocols, especially since the driver tries to support kernels all
the way back to the ancient 2.4.9 used by Red Hat AS 2.1 (which
doesn't even have get_user_pages()).
In any case, it's why I put the following items in the TODO list
of my announcement:
* Use DMA mapping API so we work properly with non-coherent
caches, IOMMUs etc. Add a function to get struct device * for an
IB device so ULPs can call sync functions (embed struct device
in struct ib_device?). (How do we handle systems with limited
DMA mapping capacity?)
* In Mellanox HCA driver, clean up mosal. Some obvious
targets (there's lots more though):
o Grunging through kernel code to find the mlock/munlock
pointers especially has to go (mosal_mlock.c). We need to
get VM experts from LKML to tell us how to handle memory
translation and locking.
On Fri, 12 Mar 2004, Troy Benjegerdes wrote:
Troy> Since the openib.org mailing lists don't seem to be alive
Troy> yet, I'll bring this up here..
It would probably be best to wait and discuss openib.org business on
their email lists rather than on the InfiniBand SourceForge email lists
to prevent confusion as to which code we are talking about.
BTW. when are these lists going to be available. I've got a bunch
of comments to provide on the tsTopSpinApi.
We have moved the OpenIB web site to a new server and opened our mailing
list. We invite everyone interested in InfiniBand support for Linux to
join us. To sign up for the mailing list, send email with the subject
"subscribe" to [email protected]. More information and list
archives can be found at <http://www.openib.org/openib-general>.
DNS has just been updated, so it may take some time for changes to
propagate to your site.
- Roland