From: Greg Banks <gnb@sgi.com>
Subject: Re: [RFC,PATCH 11/15] knfsd: RDMA transport core
Date: Mon, 21 May 2007 20:02:59 +1000
Message-ID: <20070521100259.GN7482@sgi.com>
References: <1179510352.23385.123.camel@trinity.ogc.int>
	<18001.17958.145762.704892@notabene.brown>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Cc: Tom Talpey <Thomas.Talpey@netapp.com>,
	Linux NFS Mailing List <nfs@lists.sourceforge.net>,
	Peter Leckie <pleckie@melbourne.sgi.com>
To: Neil Brown <neilb@suse.de>
In-Reply-To: <18001.17958.145762.704892@notabene.brown>
Sender: nfs-bounces@lists.sourceforge.net
Errors-To: nfs-bounces@lists.sourceforge.net

On Mon, May 21, 2007 at 05:11:34PM +1000, Neil Brown wrote:
> On Friday May 18, tom@opengridcomputing.com wrote:
> > 
> > This file implements the core transport data management and I/O
> > path. The I/O path for RDMA involves receiving callbacks on interrupt 
> > context. Since all the svc transport locks are _bh locks we enqueue the 
> > transport on a list, schedule a tasklet to dequeue data indications from 
> > the RDMA completion queue. The tasklet in turn takes _bh locks to 
> > enqueue receive data indications on a list for the transport. The 
> > svc_rdma_recvfrom transport function dequeues data from this list in an 
> > NFSD thread context.
> 
> Cannot we simply change the usage of ->sp_lock to always disable
> interrupts?

We could, but the question then is what impact will locking out
interrupts more often have on throughout or latency over Ethernet?

> That would make this much simpler. 

Yes it would.

> How much would it cost?

???

> Alternatively, why can the network layer deliver these notification in
> "bh" context, but the ib layer wants to deliver them in irq context?
> Does doing it in irq context have lower latency or something?

The problem is wider than it appears.  The kernel verbs interface
reports a *lot* of stuff in irq context, not just incoming data.
Many of those events would be best handled by disconnecting the
svc_sock, which implies setting SK_CLOSE and causing svc_sock_enqueue()
to be called.  The former happens; the latter doesn't.  For example,
see qp_event_handler() in net/sunrpc/svc_rdma_transport.c.  This
means that error conditions are not notified until some other event
causes svc_sock_enqueue() to be called.

Greg.
-- 
Greg Banks, R&D Software Engineer, SGI Australian Software Group.
Apparently, I'm Bedevere.  Which MPHG character are you?
I don't speak for SGI.

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs