Ulrich Drepper <[email protected]> wrote in message
news:<[email protected]>...
>
> First, the working group I refer to is this:
>
> http://www.opengroup.org/icsc/
>
> But that's all there is. The socket extension working group
>
> http://www.opengroup.org/icsc/sockets/
>
Most of us IB folks wanted to work with the ICSC stuff waaaaay back but
it never went anywhere and we all had to ship products. So various
stacks developed. All that is being specified by the ICSC is an API.
There are already several APIs into IB that are not IB specific.
These include:
DAPL - http://www.datcollaborative.org/,
http://sourceforge.net/projects/dapl/
MPI - http://www-unix.mcs.anl.gov/mpi/
SDP - (not really an API but is run over IB verbs and RDMA verbs and
meets up with your sockets note) See InfiniBand Specification, Volume 1,
Annex A4, Release 1.1, http://www.rdmaconsortium.org/home
There is no reason we can't implement support for the ICSC APIs if the
Linux community desires this. I can say that these other APIs are
already being used by other programs (or other APIs themselves) and
really can't go away. If folks ask for ICSC support, it will probably
get in there.
> So, these people come up with their own software stacks, unreviewed
> interface extensions, and demand that everybody accepts what they were
> "designing" without the ability to question anything.
> I surely find this completely unacceptable and any consideration of
> accepting anything the Infiniband group comes up with should be
> postponed until every bit of the design can be reviewed. If bits and
> pieces are accepted prematurely it'll just be "now that this is
support
> you have to add this too, otherwise it'll not be useful".
While some of the same people were on the ICSC lists as the IBTA lists
or the DAT lists, it doesn't mean that they are operating as one big
group with some secret agenda. The ICSC group went one way and only
recently produced a spec. The DAT group went another and has had a spec
for some time. The RDMA folks went yet another and produced a spec a
bit ago. While there are some people in both groups, the groups act
independently with different companies and individuals have more
control/influence in one than the other.
All that said, the IB developer community has decided as a whole to open
things up. InfiniCon, TopSpin, Voltaire, DivergeNet, Mellanox, etc.
have all opened up their code. Add to this the already started open
effort on sourceforge (IBAL) and we hope to find a best of breed final
solution. So please do review every single bit posted. If there is
going to be a standard linux infiniband stack it will have to be very
good or else splinter versions and incompatible versions will live on.
-Ack
p.s. While I may work for an IB company, I am expressing my own
opinions, not their official opinions.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Acker, Dave wrote:
> I can say that these other APIs are
> already being used by other programs (or other APIs themselves) and
> really can't go away. If folks ask for ICSC support, it will probably
> get in there.
Spoken in true "proprietary-software, don't get the concept of free
software"-fashion.
Just consider the implications if this would be accepted as an argument.
Everybody who is producing a new API has to create just a couple of
programs using it before publishing the specs to be able to say: well,
here's the API. Use it. Oh, and don't change it, that's not acceptable
because we have code relying on the API. Yep, strong-arming people you
have no control over works out fine, all the time.
The only acceptable order in which things can happen is:
1. develop API
2. propose API to be accepted by "community"/distributions
3. change API if necessary, and go back to 2.
4. write applications using new API
> If folks ask for ICSC support, it will probably
> get in there.
You did not in the least address the main point: IB is just one fiber.
I cannot imagine anybody but the IB people want to have a specific API +
kernel extensions for each separate interconnect fiber.
Get it all over with in one stroke. Come up with an API which covers it
all. The requirements aren't that different. And I singled out ICSC
because it attempts to do just this.
And don't say "we did our part, if the Linux community wants to have
something else it's their to come up with a unified solution". That
would be acceptable only if it wouldn't be the IB people who want their
code to be generally accepted. If you don't care about the later, fine,
leave it all as is. But the code might not be picked up at all.
Furthermore, don't count on much community participation. There aren't
many people out there with the necessary hardware. Or even the ability
to collect experiences. The price for the changes have to be carried by
the parties with the agenda.
> If there is
> going to be a standard linux infiniband stack it will have to be very
> good or else splinter versions and incompatible versions will live on.
And by very good you mean of course your implementation. From the
comment above it is clear that if the standardized stack does not
include your code you intent to keep using your code anyway. Designing
standards is always about compromises.
Nobody ever was/is happy with everything that POSIX requires. But it is
an acceptable compromise. One thing which has been clearly shown by
POSIX is that 99% of all developers prefer portability over getting the
last 5% of performance out of their machines.
The organizations with interconnect interest have to come together to
create just this, a compromise which in the end might not fulfill you
"very good" criteria in all places.
- --
➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
iD8DBQFAXIWg2ijCOnn/RHQRAk9OAKCPAanX45/cK3zdFEN/Y28gk/vHzwCdG57w
XprZ1vZdfkT8VY3CW6T+aR0=
=gYdV
-----END PGP SIGNATURE-----
Ulrich> The only acceptable order in which things can happen is:
Ulrich> 1. develop API
Ulrich> 2. propose API to be accepted by "community"/distributions
Ulrich> 3. change API if necessary, and go back to 2.
Ulrich> 4. write applications using new API
I don't think this is reasonable, since nothing is settled enough for
this to work. On the one hand, InfiniBand and other "fibers" (eg RDMA
over ethernet) are quite experimental. No one is sure of the right
semantics or the best way to use the interconnect. On the other hand,
there are people who want to use this stuff right now (eg
high-performance computing people building clusters, database cluster
people, etc).
There are users who want to use InfiniBand now, and making them wait
through your whole process above is simply untenable. You can't
expect a company selling InfiniBand equipment to say, "Sorry, our
software isn't perfect (although it would work for you now). Come
back in a year or two."
With that in mind, I think the only order things can happen is:
1. develop API
2. implement API
2a.learn from mistakes and go back to 1.
3. write applications using API
4. learn from mistakes and go back to 1.
It's certainly unfortunate that so much InfiniBand software has been
developed behind closed doors, but the industry has finally woken up
and come together around the OpenIB idea to develop Linux support
completely in the open.
When does this software make it into distributions? Obviously that's
up to the distribution. Certainly a commercial distribution has
customers of its own to listen to, and I would assume that the
decision would be made based on the appropriate combination of
technical merit and customer demand.
- Roland