The following 9 messages implement the NFS/RDMA client,
and are RFC at this time, though many of you have seen
them before. They represent a long cycle of development!
The patches are applied against 2.6.22, and are tested on
both Infiniband (Mellanox) and iWARP (Chelsio) RDMA providers,
using NFSv3 and NFSv4. The changes are almost completely
in the RPC (net/sunrpc) layer, only one line is changed in NFS,
to allow NFSv4 mounts to work on a non-IPv4 server.
The first 5 (patches 01 through 05) are prerequisites for
integration into 2.6.22. 01, 02 and 03 are from Chuck Lever
and implement dynamic RPC transport switch modules,
allowing RPCRDMA to be added. 04 and 05 are small and
allow NFS/RDMA mounts to proceed on non-UDP, non-TCP
protocols.
The 6th (patch 06) is the rpcrdma protocol header file, this
file is also a part of Tom Tucker's knfsd/RDMA submission,
and is included here for completeness.
The remaining three (patches 07, 08 and 09) implement the
RPCRDMA transport. They are organized into three components,
the core configuration and transport implementation, the rpcrdma
protocol handling, and the RDMA verbs interfacing. Patch 07 has
stub versions of two files, which (hopefully) provides clarity but
also allows building the patch cleanly.
Comments are welcome.
Tom.
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
I neglected to include the following patch to xprtrdma/verbs.c
which changes it to use the new ib_dma_*() wrappers.
BTW, it appears that some helpful mailer along the way translated
the tabs in my patches to spaces. If this message receives the
same fate I will attempt a different route next time. If anyone
wants the raw patches in the meantime, let me know.
Tom.
-----
RPCRDMA: use ib_dma interface
The rdma verbs have wrapper hooks for the dma api, use them.
Signed-off-by: Tom Talpey <[email protected]>
---
net/sunrpc/xprtrdma/verbs.c | 24 ++++++++++++------------
1 file changed, 12 insertions(+), 12 deletions(-)
Index: linux-2.6.22/net/sunrpc/xprtrdma/verbs.c
===================================================================
--- linux-2.6.22.orig/net/sunrpc/xprtrdma/verbs.c
+++ linux-2.6.22/net/sunrpc/xprtrdma/verbs.c
@@ -162,8 +162,8 @@ void rpcrdma_event_process(struct ib_wc
switch (wc->opcode) {
case IB_WC_RECV:
rep->rr_len = wc->byte_len;
- dma_sync_single_for_cpu(
- rdmab_to_ia(rep->rr_buffer)->ri_id->device->dma_device,
+ ib_dma_sync_single_for_cpu(
+ rdmab_to_ia(rep->rr_buffer)->ri_id->device,
rep->rr_iov.addr, rep->rr_len, DMA_FROM_DEVICE);
/* Keep (only) the most recent credits, after check validity */
if (rep->rr_len >= 16) {
@@ -1219,7 +1219,7 @@ rpcrdma_register_internal(struct rpcrdma
/*
* All memory passed here was kmalloc'ed, therefore phys-contiguous.
*/
- iov->addr = dma_map_single(ia->ri_id->device->dma_device,
+ iov->addr = ib_dma_map_single(ia->ri_id->device,
va, len, DMA_BIDIRECTIONAL);
iov->length = len;
@@ -1257,7 +1257,7 @@ rpcrdma_deregister_internal(struct rpcrd
{
int rc;
- dma_unmap_single(ia->ri_id->device->dma_device,
+ ib_dma_unmap_single(ia->ri_id->device,
iov->addr, iov->length, DMA_BIDIRECTIONAL);
if (NULL == mr)
@@ -1279,11 +1279,11 @@ rpcrdma_map_one(struct rpcrdma_ia *ia, s
seg->mr_dir = writing ? DMA_FROM_DEVICE : DMA_TO_DEVICE;
seg->mr_dmalen = seg->mr_len;
if (seg->mr_page)
- seg->mr_dma = dma_map_page(ia->ri_id->device->dma_device,
+ seg->mr_dma = ib_dma_map_page(ia->ri_id->device,
seg->mr_page, offset_in_page(seg->mr_offset),
seg->mr_dmalen, seg->mr_dir);
else
- seg->mr_dma = dma_map_single(ia->ri_id->device->dma_device,
+ seg->mr_dma = ib_dma_map_single(ia->ri_id->device,
seg->mr_offset,
seg->mr_dmalen, seg->mr_dir);
}
@@ -1292,10 +1292,10 @@ static void
rpcrdma_unmap_one(struct rpcrdma_ia *ia, struct rpcrdma_mr_seg *seg)
{
if (seg->mr_page)
- dma_unmap_page(ia->ri_id->device->dma_device,
+ ib_dma_unmap_page(ia->ri_id->device,
seg->mr_dma, seg->mr_dmalen, seg->mr_dir);
else
- dma_unmap_single(ia->ri_id->device->dma_device,
+ ib_dma_unmap_single(ia->ri_id->device,
seg->mr_dma, seg->mr_dmalen, seg->mr_dir);
}
@@ -1541,13 +1541,13 @@ rpcrdma_ep_post(struct rpcrdma_ia *ia, s
send_wr.opcode = IB_WR_SEND;
send_wr.imm_data = 0;
if (send_wr.num_sge == 4) /* no need to sync any pad (constant) */
- dma_sync_single_for_device(ia->ri_id->device->dma_device,
+ ib_dma_sync_single_for_device(ia->ri_id->device,
req->rl_send_iov[3].addr, req->rl_send_iov[3].length,
DMA_TO_DEVICE);
- dma_sync_single_for_device(ia->ri_id->device->dma_device,
+ ib_dma_sync_single_for_device(ia->ri_id->device,
req->rl_send_iov[1].addr, req->rl_send_iov[1].length,
DMA_TO_DEVICE);
- dma_sync_single_for_device(ia->ri_id->device->dma_device,
+ ib_dma_sync_single_for_device(ia->ri_id->device,
req->rl_send_iov[0].addr, req->rl_send_iov[0].length,
DMA_TO_DEVICE);
@@ -1580,7 +1580,7 @@ rpcrdma_ep_post_recv(struct rpcrdma_ia *
recv_wr.sg_list = &rep->rr_iov;
recv_wr.num_sge = 1;
- dma_sync_single_for_cpu(ia->ri_id->device->dma_device,
+ ib_dma_sync_single_for_cpu(ia->ri_id->device,
rep->rr_iov.addr, rep->rr_iov.length, DMA_BIDIRECTIONAL);
DECR_CQCOUNT(ep);
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs