Allow INFINIBAND without INFINIBAND_ADDR_TRANS.
Signed-off-by: Greg Thelen <[email protected]>
Cc: Tarick Bedeir <[email protected]>
---
drivers/infiniband/Kconfig | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/infiniband/Kconfig b/drivers/infiniband/Kconfig
index ee270e065ba9..f20a3977087c 100644
--- a/drivers/infiniband/Kconfig
+++ b/drivers/infiniband/Kconfig
@@ -61,7 +61,7 @@ config INFINIBAND_ON_DEMAND_PAGING
pages on demand instead.
config INFINIBAND_ADDR_TRANS
- bool
+ bool "InfiniBand address translation"
depends on INFINIBAND
default y
--
2.17.0.484.g0c8726318c-goog
On Fri, 2018-04-13 at 00:06 -0700, Greg Thelen wrote:
> Allow INFINIBAND without INFINIBAND_ADDR_TRANS.
>
> Signed-off-by: Greg Thelen <[email protected]>
> Cc: Tarick Bedeir <[email protected]>
> ---
> drivers/infiniband/Kconfig | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/infiniband/Kconfig b/drivers/infiniband/Kconfig
> index ee270e065ba9..f20a3977087c 100644
> --- a/drivers/infiniband/Kconfig
> +++ b/drivers/infiniband/Kconfig
> @@ -61,7 +61,7 @@ config INFINIBAND_ON_DEMAND_PAGING
> pages on demand instead.
>
> config INFINIBAND_ADDR_TRANS
> - bool
> + bool "InfiniBand address translation"
> depends on INFINIBAND
> default y
Hello Greg,
Please change the short description from "InfiniBand address translation" (which
is a misleading description) into "RDMA/CM". Please also add a help text that
explains to people who are not RDMA experts what the RDMA/CM is.
Thanks,
Bart.
Allow INFINIBAND without INFINIBAND_ADDR_TRANS.
Signed-off-by: Greg Thelen <[email protected]>
Cc: Tarick Bedeir <[email protected]>
Change-Id: I6fbbf8a432e467710fa65e4904b7d61880b914e5
---
drivers/infiniband/Kconfig | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/drivers/infiniband/Kconfig b/drivers/infiniband/Kconfig
index ee270e065ba9..2a972ed6851b 100644
--- a/drivers/infiniband/Kconfig
+++ b/drivers/infiniband/Kconfig
@@ -61,9 +61,12 @@ config INFINIBAND_ON_DEMAND_PAGING
pages on demand instead.
config INFINIBAND_ADDR_TRANS
- bool
+ bool "RDMA/CM"
depends on INFINIBAND
default y
+ ---help---
+ Support for RDMA communication manager (CM).
+ This allows for a generic connection abstraction over RDMA.
config INFINIBAND_ADDR_TRANS_CONFIGFS
bool
--
2.17.0.484.g0c8726318c-goog
On 4/13/2018 1:27 PM, Greg Thelen wrote:
> Allow INFINIBAND without INFINIBAND_ADDR_TRANS.
>
> Signed-off-by: Greg Thelen <[email protected]>
> Cc: Tarick Bedeir <[email protected]>
> Change-Id: I6fbbf8a432e467710fa65e4904b7d61880b914e5
Forgot to remove the Gerrit thing.
-Denny
On Sat, Apr 14, 2018 at 8:13 AM Dennis Dalessandro <
[email protected]> wrote:
> On 4/13/2018 1:27 PM, Greg Thelen wrote:
> > Allow INFINIBAND without INFINIBAND_ADDR_TRANS.
> >
> > Signed-off-by: Greg Thelen <[email protected]>
> > Cc: Tarick Bedeir <[email protected]>
> > Change-Id: I6fbbf8a432e467710fa65e4904b7d61880b914e5
> Forgot to remove the Gerrit thing.
> -Denny
Ack. My bad. Will repost. Unfortunately checkpatch didn't notice.
Allow INFINIBAND without INFINIBAND_ADDR_TRANS.
Signed-off-by: Greg Thelen <[email protected]>
Cc: Tarick Bedeir <[email protected]>
---
drivers/infiniband/Kconfig | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/drivers/infiniband/Kconfig b/drivers/infiniband/Kconfig
index ee270e065ba9..2a972ed6851b 100644
--- a/drivers/infiniband/Kconfig
+++ b/drivers/infiniband/Kconfig
@@ -61,9 +61,12 @@ config INFINIBAND_ON_DEMAND_PAGING
pages on demand instead.
config INFINIBAND_ADDR_TRANS
- bool
+ bool "RDMA/CM"
depends on INFINIBAND
default y
+ ---help---
+ Support for RDMA communication manager (CM).
+ This allows for a generic connection abstraction over RDMA.
config INFINIBAND_ADDR_TRANS_CONFIGFS
bool
--
2.17.0.484.g0c8726318c-goog
On Sat, 2018-04-14 at 15:34 +0000, Greg Thelen wrote:
> On Sat, Apr 14, 2018 at 8:13 AM Dennis Dalessandro <
> [email protected]> wrote:
>
> > On 4/13/2018 1:27 PM, Greg Thelen wrote:
> > > Allow INFINIBAND without INFINIBAND_ADDR_TRANS.
> > >
> > > Signed-off-by: Greg Thelen <[email protected]>
> > > Cc: Tarick Bedeir <[email protected]>
> > > Change-Id: I6fbbf8a432e467710fa65e4904b7d61880b914e5
> > Forgot to remove the Gerrit thing.
> > -Denny
>
> Ack. My bad. Will repost. Unfortunately checkpatch didn't notice.
Probably because Change-Id: is after a Signed-off-by: line
Hi Greg,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on linus/master]
[also build test ERROR on v4.16 next-20180413]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Greg-Thelen/IB-make-INFINIBAND_ADDR_TRANS-configurable/20180414-234042
config: x86_64-randconfig-x011-201815 (attached as .config)
compiler: gcc-7 (Debian 7.3.0-1) 7.3.0
reproduce:
# save the attached .config to linux build tree
make ARCH=x86_64
All errors (new ones prefixed by >>):
drivers/nvme/host/rdma.o: In function `nvme_rdma_stop_queue':
>> drivers/nvme/host/rdma.c:554: undefined reference to `rdma_disconnect'
drivers/nvme/host/rdma.o: In function `nvme_rdma_create_qp':
>> drivers/nvme/host/rdma.c:258: undefined reference to `rdma_create_qp'
drivers/nvme/host/rdma.o: In function `nvme_rdma_free_queue':
>> drivers/nvme/host/rdma.c:570: undefined reference to `rdma_destroy_id'
drivers/nvme/host/rdma.o: In function `nvme_rdma_alloc_queue':
>> drivers/nvme/host/rdma.c:511: undefined reference to `__rdma_create_id'
>> drivers/nvme/host/rdma.c:523: undefined reference to `rdma_resolve_addr'
drivers/nvme/host/rdma.c:544: undefined reference to `rdma_destroy_id'
drivers/nvme/host/rdma.o: In function `nvme_rdma_addr_resolved':
>> drivers/nvme/host/rdma.c:1461: undefined reference to `rdma_resolve_route'
drivers/nvme/host/rdma.o: In function `nvme_rdma_create_queue_ib':
>> drivers/nvme/host/rdma.c:485: undefined reference to `rdma_destroy_qp'
drivers/nvme/host/rdma.o: In function `nvme_rdma_route_resolved':
>> drivers/nvme/host/rdma.c:1512: undefined reference to `rdma_connect'
drivers/nvme/host/rdma.o: In function `nvme_rdma_conn_rejected':
>> drivers/nvme/host/rdma.c:1436: undefined reference to `rdma_reject_msg'
>> drivers/nvme/host/rdma.c:1437: undefined reference to `rdma_consumer_reject_data'
vim +554 drivers/nvme/host/rdma.c
f41725bb Israel Rukshin 2017-11-26 423
ca6e95bb Sagi Grimberg 2017-05-04 424 static int nvme_rdma_create_queue_ib(struct nvme_rdma_queue *queue)
71102307 Christoph Hellwig 2016-07-06 425 {
ca6e95bb Sagi Grimberg 2017-05-04 426 struct ib_device *ibdev;
71102307 Christoph Hellwig 2016-07-06 427 const int send_wr_factor = 3; /* MR, SEND, INV */
71102307 Christoph Hellwig 2016-07-06 428 const int cq_factor = send_wr_factor + 1; /* + RECV */
71102307 Christoph Hellwig 2016-07-06 429 int comp_vector, idx = nvme_rdma_queue_idx(queue);
71102307 Christoph Hellwig 2016-07-06 430 int ret;
71102307 Christoph Hellwig 2016-07-06 431
ca6e95bb Sagi Grimberg 2017-05-04 432 queue->device = nvme_rdma_find_get_device(queue->cm_id);
ca6e95bb Sagi Grimberg 2017-05-04 433 if (!queue->device) {
ca6e95bb Sagi Grimberg 2017-05-04 434 dev_err(queue->cm_id->device->dev.parent,
ca6e95bb Sagi Grimberg 2017-05-04 435 "no client data found!\n");
ca6e95bb Sagi Grimberg 2017-05-04 436 return -ECONNREFUSED;
ca6e95bb Sagi Grimberg 2017-05-04 437 }
ca6e95bb Sagi Grimberg 2017-05-04 438 ibdev = queue->device->dev;
71102307 Christoph Hellwig 2016-07-06 439
71102307 Christoph Hellwig 2016-07-06 440 /*
0b36658c Sagi Grimberg 2017-07-13 441 * Spread I/O queues completion vectors according their queue index.
0b36658c Sagi Grimberg 2017-07-13 442 * Admin queues can always go on completion vector 0.
71102307 Christoph Hellwig 2016-07-06 443 */
0b36658c Sagi Grimberg 2017-07-13 444 comp_vector = idx == 0 ? idx : idx - 1;
71102307 Christoph Hellwig 2016-07-06 445
71102307 Christoph Hellwig 2016-07-06 446 /* +1 for ib_stop_cq */
ca6e95bb Sagi Grimberg 2017-05-04 447 queue->ib_cq = ib_alloc_cq(ibdev, queue,
ca6e95bb Sagi Grimberg 2017-05-04 448 cq_factor * queue->queue_size + 1,
ca6e95bb Sagi Grimberg 2017-05-04 449 comp_vector, IB_POLL_SOFTIRQ);
71102307 Christoph Hellwig 2016-07-06 450 if (IS_ERR(queue->ib_cq)) {
71102307 Christoph Hellwig 2016-07-06 451 ret = PTR_ERR(queue->ib_cq);
ca6e95bb Sagi Grimberg 2017-05-04 452 goto out_put_dev;
71102307 Christoph Hellwig 2016-07-06 453 }
71102307 Christoph Hellwig 2016-07-06 454
71102307 Christoph Hellwig 2016-07-06 455 ret = nvme_rdma_create_qp(queue, send_wr_factor);
71102307 Christoph Hellwig 2016-07-06 456 if (ret)
71102307 Christoph Hellwig 2016-07-06 457 goto out_destroy_ib_cq;
71102307 Christoph Hellwig 2016-07-06 458
71102307 Christoph Hellwig 2016-07-06 459 queue->rsp_ring = nvme_rdma_alloc_ring(ibdev, queue->queue_size,
71102307 Christoph Hellwig 2016-07-06 460 sizeof(struct nvme_completion), DMA_FROM_DEVICE);
71102307 Christoph Hellwig 2016-07-06 461 if (!queue->rsp_ring) {
71102307 Christoph Hellwig 2016-07-06 462 ret = -ENOMEM;
71102307 Christoph Hellwig 2016-07-06 463 goto out_destroy_qp;
71102307 Christoph Hellwig 2016-07-06 464 }
71102307 Christoph Hellwig 2016-07-06 465
f41725bb Israel Rukshin 2017-11-26 466 ret = ib_mr_pool_init(queue->qp, &queue->qp->rdma_mrs,
f41725bb Israel Rukshin 2017-11-26 467 queue->queue_size,
f41725bb Israel Rukshin 2017-11-26 468 IB_MR_TYPE_MEM_REG,
f41725bb Israel Rukshin 2017-11-26 469 nvme_rdma_get_max_fr_pages(ibdev));
f41725bb Israel Rukshin 2017-11-26 470 if (ret) {
f41725bb Israel Rukshin 2017-11-26 471 dev_err(queue->ctrl->ctrl.device,
f41725bb Israel Rukshin 2017-11-26 472 "failed to initialize MR pool sized %d for QID %d\n",
f41725bb Israel Rukshin 2017-11-26 473 queue->queue_size, idx);
f41725bb Israel Rukshin 2017-11-26 474 goto out_destroy_ring;
f41725bb Israel Rukshin 2017-11-26 475 }
f41725bb Israel Rukshin 2017-11-26 476
eb1bd249 Max Gurtovoy 2017-11-28 477 set_bit(NVME_RDMA_Q_TR_READY, &queue->flags);
eb1bd249 Max Gurtovoy 2017-11-28 478
71102307 Christoph Hellwig 2016-07-06 479 return 0;
71102307 Christoph Hellwig 2016-07-06 480
f41725bb Israel Rukshin 2017-11-26 481 out_destroy_ring:
f41725bb Israel Rukshin 2017-11-26 482 nvme_rdma_free_ring(ibdev, queue->rsp_ring, queue->queue_size,
f41725bb Israel Rukshin 2017-11-26 483 sizeof(struct nvme_completion), DMA_FROM_DEVICE);
71102307 Christoph Hellwig 2016-07-06 484 out_destroy_qp:
1f61def9 Max Gurtovoy 2017-11-06 @485 rdma_destroy_qp(queue->cm_id);
71102307 Christoph Hellwig 2016-07-06 486 out_destroy_ib_cq:
71102307 Christoph Hellwig 2016-07-06 487 ib_free_cq(queue->ib_cq);
ca6e95bb Sagi Grimberg 2017-05-04 488 out_put_dev:
ca6e95bb Sagi Grimberg 2017-05-04 489 nvme_rdma_dev_put(queue->device);
71102307 Christoph Hellwig 2016-07-06 490 return ret;
71102307 Christoph Hellwig 2016-07-06 491 }
71102307 Christoph Hellwig 2016-07-06 492
41e8cfa1 Sagi Grimberg 2017-07-10 493 static int nvme_rdma_alloc_queue(struct nvme_rdma_ctrl *ctrl,
71102307 Christoph Hellwig 2016-07-06 494 int idx, size_t queue_size)
71102307 Christoph Hellwig 2016-07-06 495 {
71102307 Christoph Hellwig 2016-07-06 496 struct nvme_rdma_queue *queue;
8f4e8dac Max Gurtovoy 2017-02-19 497 struct sockaddr *src_addr = NULL;
71102307 Christoph Hellwig 2016-07-06 498 int ret;
71102307 Christoph Hellwig 2016-07-06 499
71102307 Christoph Hellwig 2016-07-06 500 queue = &ctrl->queues[idx];
71102307 Christoph Hellwig 2016-07-06 501 queue->ctrl = ctrl;
71102307 Christoph Hellwig 2016-07-06 502 init_completion(&queue->cm_done);
71102307 Christoph Hellwig 2016-07-06 503
71102307 Christoph Hellwig 2016-07-06 504 if (idx > 0)
71102307 Christoph Hellwig 2016-07-06 505 queue->cmnd_capsule_len = ctrl->ctrl.ioccsz * 16;
71102307 Christoph Hellwig 2016-07-06 506 else
71102307 Christoph Hellwig 2016-07-06 507 queue->cmnd_capsule_len = sizeof(struct nvme_command);
71102307 Christoph Hellwig 2016-07-06 508
71102307 Christoph Hellwig 2016-07-06 509 queue->queue_size = queue_size;
71102307 Christoph Hellwig 2016-07-06 510
71102307 Christoph Hellwig 2016-07-06 @511 queue->cm_id = rdma_create_id(&init_net, nvme_rdma_cm_handler, queue,
71102307 Christoph Hellwig 2016-07-06 512 RDMA_PS_TCP, IB_QPT_RC);
71102307 Christoph Hellwig 2016-07-06 513 if (IS_ERR(queue->cm_id)) {
71102307 Christoph Hellwig 2016-07-06 514 dev_info(ctrl->ctrl.device,
71102307 Christoph Hellwig 2016-07-06 515 "failed to create CM ID: %ld\n", PTR_ERR(queue->cm_id));
71102307 Christoph Hellwig 2016-07-06 516 return PTR_ERR(queue->cm_id);
71102307 Christoph Hellwig 2016-07-06 517 }
71102307 Christoph Hellwig 2016-07-06 518
8f4e8dac Max Gurtovoy 2017-02-19 519 if (ctrl->ctrl.opts->mask & NVMF_OPT_HOST_TRADDR)
0928f9b4 Sagi Grimberg 2017-02-05 520 src_addr = (struct sockaddr *)&ctrl->src_addr;
8f4e8dac Max Gurtovoy 2017-02-19 521
0928f9b4 Sagi Grimberg 2017-02-05 522 queue->cm_error = -ETIMEDOUT;
0928f9b4 Sagi Grimberg 2017-02-05 @523 ret = rdma_resolve_addr(queue->cm_id, src_addr,
0928f9b4 Sagi Grimberg 2017-02-05 524 (struct sockaddr *)&ctrl->addr,
71102307 Christoph Hellwig 2016-07-06 525 NVME_RDMA_CONNECT_TIMEOUT_MS);
71102307 Christoph Hellwig 2016-07-06 526 if (ret) {
71102307 Christoph Hellwig 2016-07-06 527 dev_info(ctrl->ctrl.device,
71102307 Christoph Hellwig 2016-07-06 528 "rdma_resolve_addr failed (%d).\n", ret);
71102307 Christoph Hellwig 2016-07-06 529 goto out_destroy_cm_id;
71102307 Christoph Hellwig 2016-07-06 530 }
71102307 Christoph Hellwig 2016-07-06 531
71102307 Christoph Hellwig 2016-07-06 532 ret = nvme_rdma_wait_for_cm(queue);
71102307 Christoph Hellwig 2016-07-06 533 if (ret) {
71102307 Christoph Hellwig 2016-07-06 534 dev_info(ctrl->ctrl.device,
d8bfceeb Sagi Grimberg 2017-10-11 535 "rdma connection establishment failed (%d)\n", ret);
71102307 Christoph Hellwig 2016-07-06 536 goto out_destroy_cm_id;
71102307 Christoph Hellwig 2016-07-06 537 }
71102307 Christoph Hellwig 2016-07-06 538
5013e98b Sagi Grimberg 2017-10-11 539 set_bit(NVME_RDMA_Q_ALLOCATED, &queue->flags);
71102307 Christoph Hellwig 2016-07-06 540
71102307 Christoph Hellwig 2016-07-06 541 return 0;
71102307 Christoph Hellwig 2016-07-06 542
71102307 Christoph Hellwig 2016-07-06 543 out_destroy_cm_id:
71102307 Christoph Hellwig 2016-07-06 @544 rdma_destroy_id(queue->cm_id);
eb1bd249 Max Gurtovoy 2017-11-28 545 nvme_rdma_destroy_queue_ib(queue);
71102307 Christoph Hellwig 2016-07-06 546 return ret;
71102307 Christoph Hellwig 2016-07-06 547 }
71102307 Christoph Hellwig 2016-07-06 548
71102307 Christoph Hellwig 2016-07-06 549 static void nvme_rdma_stop_queue(struct nvme_rdma_queue *queue)
71102307 Christoph Hellwig 2016-07-06 550 {
a57bd541 Sagi Grimberg 2017-08-28 551 if (!test_and_clear_bit(NVME_RDMA_Q_LIVE, &queue->flags))
a57bd541 Sagi Grimberg 2017-08-28 552 return;
a57bd541 Sagi Grimberg 2017-08-28 553
71102307 Christoph Hellwig 2016-07-06 @554 rdma_disconnect(queue->cm_id);
71102307 Christoph Hellwig 2016-07-06 555 ib_drain_qp(queue->qp);
71102307 Christoph Hellwig 2016-07-06 556 }
71102307 Christoph Hellwig 2016-07-06 557
71102307 Christoph Hellwig 2016-07-06 558 static void nvme_rdma_free_queue(struct nvme_rdma_queue *queue)
71102307 Christoph Hellwig 2016-07-06 559 {
5013e98b Sagi Grimberg 2017-10-11 560 if (!test_and_clear_bit(NVME_RDMA_Q_ALLOCATED, &queue->flags))
a57bd541 Sagi Grimberg 2017-08-28 561 return;
a57bd541 Sagi Grimberg 2017-08-28 562
bd9f0759 Sagi Grimberg 2017-10-19 563 if (nvme_rdma_queue_idx(queue) == 0) {
bd9f0759 Sagi Grimberg 2017-10-19 564 nvme_rdma_free_qe(queue->device->dev,
bd9f0759 Sagi Grimberg 2017-10-19 565 &queue->ctrl->async_event_sqe,
bd9f0759 Sagi Grimberg 2017-10-19 566 sizeof(struct nvme_command), DMA_TO_DEVICE);
bd9f0759 Sagi Grimberg 2017-10-19 567 }
bd9f0759 Sagi Grimberg 2017-10-19 568
71102307 Christoph Hellwig 2016-07-06 569 nvme_rdma_destroy_queue_ib(queue);
71102307 Christoph Hellwig 2016-07-06 @570 rdma_destroy_id(queue->cm_id);
71102307 Christoph Hellwig 2016-07-06 571 }
71102307 Christoph Hellwig 2016-07-06 572
:::::: The code at line 554 was first introduced by commit
:::::: 7110230719602852481c2793d054f866b2bf4a2b nvme-rdma: add a NVMe over Fabrics RDMA host driver
:::::: TO: Christoph Hellwig <[email protected]>
:::::: CC: Jens Axboe <[email protected]>
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
Hi Greg,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on linus/master]
[also build test ERROR on v4.16 next-20180413]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Greg-Thelen/IB-make-INFINIBAND_ADDR_TRANS-configurable/20180414-234042
config: i386-randconfig-x005-201815 (attached as .config)
compiler: gcc-7 (Debian 7.3.0-1) 7.3.0
reproduce:
# save the attached .config to linux build tree
make ARCH=i386
All errors (new ones prefixed by >>):
drivers/nvme/host/rdma.o: In function `nvme_rdma_stop_queue':
drivers/nvme/host/rdma.c:554: undefined reference to `rdma_disconnect'
drivers/nvme/host/rdma.o: In function `nvme_rdma_free_queue':
drivers/nvme/host/rdma.c:570: undefined reference to `rdma_destroy_id'
drivers/nvme/host/rdma.o: In function `nvme_rdma_alloc_queue':
drivers/nvme/host/rdma.c:511: undefined reference to `__rdma_create_id'
drivers/nvme/host/rdma.c:523: undefined reference to `rdma_resolve_addr'
drivers/nvme/host/rdma.c:544: undefined reference to `rdma_destroy_id'
drivers/nvme/host/rdma.o: In function `nvme_rdma_create_qp':
drivers/nvme/host/rdma.c:258: undefined reference to `rdma_create_qp'
drivers/nvme/host/rdma.o: In function `nvme_rdma_create_queue_ib':
drivers/nvme/host/rdma.c:485: undefined reference to `rdma_destroy_qp'
drivers/nvme/host/rdma.o: In function `nvme_rdma_addr_resolved':
drivers/nvme/host/rdma.c:1461: undefined reference to `rdma_resolve_route'
drivers/nvme/host/rdma.o: In function `nvme_rdma_route_resolved':
drivers/nvme/host/rdma.c:1512: undefined reference to `rdma_connect'
drivers/nvme/host/rdma.o: In function `nvme_rdma_conn_rejected':
drivers/nvme/host/rdma.c:1436: undefined reference to `rdma_reject_msg'
drivers/nvme/host/rdma.c:1437: undefined reference to `rdma_consumer_reject_data'
drivers/infiniband/ulp/srp/ib_srp.o: In function `srp_create_ch_ib':
>> drivers/infiniband/ulp/srp/ib_srp.c:585: undefined reference to `rdma_create_qp'
>> drivers/infiniband/ulp/srp/ib_srp.c:647: undefined reference to `rdma_destroy_qp'
drivers/infiniband/ulp/srp/ib_srp.o: In function `srp_disconnect_target':
>> drivers/infiniband/ulp/srp/ib_srp.c:977: undefined reference to `rdma_disconnect'
drivers/infiniband/ulp/srp/ib_srp.o: In function `srp_new_rdma_cm_id':
>> drivers/infiniband/ulp/srp/ib_srp.c:336: undefined reference to `__rdma_create_id'
>> drivers/infiniband/ulp/srp/ib_srp.c:345: undefined reference to `rdma_resolve_addr'
>> drivers/infiniband/ulp/srp/ib_srp.c:369: undefined reference to `rdma_destroy_id'
drivers/infiniband/ulp/srp/ib_srp.o: In function `srp_rdma_lookup_path':
>> drivers/infiniband/ulp/srp/ib_srp.c:790: undefined reference to `rdma_resolve_route'
drivers/infiniband/ulp/srp/ib_srp.o: In function `srp_send_req':
>> drivers/infiniband/ulp/srp/ib_srp.c:938: undefined reference to `rdma_connect'
drivers/infiniband/ulp/srp/ib_srp.o: In function `srp_free_ch_ib':
drivers/infiniband/ulp/srp/ib_srp.c:677: undefined reference to `rdma_destroy_id'
drivers/infiniband/ulp/srp/ib_srp.o: In function `srp_rdma_cm_handler':
drivers/infiniband/ulp/srp/ib_srp.c:2808: undefined reference to `rdma_disconnect'
vim +585 drivers/infiniband/ulp/srp/ib_srp.c
7dad6b2e Bart Van Assche 2014-10-21 542
509c07bc Bart Van Assche 2014-10-30 543 static int srp_create_ch_ib(struct srp_rdma_ch *ch)
aef9ec39 Roland Dreier 2005-11-02 544 {
509c07bc Bart Van Assche 2014-10-30 545 struct srp_target_port *target = ch->target;
62154b2e Bart Van Assche 2014-05-20 546 struct srp_device *dev = target->srp_host->srp_dev;
aef9ec39 Roland Dreier 2005-11-02 547 struct ib_qp_init_attr *init_attr;
73aa89ed Ishai Rabinovitz 2012-11-26 548 struct ib_cq *recv_cq, *send_cq;
73aa89ed Ishai Rabinovitz 2012-11-26 549 struct ib_qp *qp;
d1b4289e Bart Van Assche 2014-05-20 550 struct ib_fmr_pool *fmr_pool = NULL;
5cfb1782 Bart Van Assche 2014-05-20 551 struct srp_fr_pool *fr_pool = NULL;
509c5f33 Bart Van Assche 2016-05-12 552 const int m = 1 + dev->use_fast_reg * target->mr_per_cmd * 2;
aef9ec39 Roland Dreier 2005-11-02 553 int ret;
aef9ec39 Roland Dreier 2005-11-02 554
aef9ec39 Roland Dreier 2005-11-02 555 init_attr = kzalloc(sizeof *init_attr, GFP_KERNEL);
aef9ec39 Roland Dreier 2005-11-02 556 if (!init_attr)
aef9ec39 Roland Dreier 2005-11-02 557 return -ENOMEM;
aef9ec39 Roland Dreier 2005-11-02 558
561392d4 Steve Wise 2016-02-17 559 /* queue_size + 1 for ib_drain_rq() */
1dc7b1f1 Christoph Hellwig 2015-11-13 560 recv_cq = ib_alloc_cq(dev->dev, ch, target->queue_size + 1,
1dc7b1f1 Christoph Hellwig 2015-11-13 561 ch->comp_vector, IB_POLL_SOFTIRQ);
73aa89ed Ishai Rabinovitz 2012-11-26 562 if (IS_ERR(recv_cq)) {
73aa89ed Ishai Rabinovitz 2012-11-26 563 ret = PTR_ERR(recv_cq);
da9d2f07 Roland Dreier 2010-02-24 564 goto err;
aef9ec39 Roland Dreier 2005-11-02 565 }
aef9ec39 Roland Dreier 2005-11-02 566
1dc7b1f1 Christoph Hellwig 2015-11-13 567 send_cq = ib_alloc_cq(dev->dev, ch, m * target->queue_size,
1dc7b1f1 Christoph Hellwig 2015-11-13 568 ch->comp_vector, IB_POLL_DIRECT);
73aa89ed Ishai Rabinovitz 2012-11-26 569 if (IS_ERR(send_cq)) {
73aa89ed Ishai Rabinovitz 2012-11-26 570 ret = PTR_ERR(send_cq);
da9d2f07 Roland Dreier 2010-02-24 571 goto err_recv_cq;
9c03dc9f Bart Van Assche 2010-02-02 572 }
9c03dc9f Bart Van Assche 2010-02-02 573
aef9ec39 Roland Dreier 2005-11-02 574 init_attr->event_handler = srp_qp_event;
5cfb1782 Bart Van Assche 2014-05-20 575 init_attr->cap.max_send_wr = m * target->queue_size;
7dad6b2e Bart Van Assche 2014-10-21 576 init_attr->cap.max_recv_wr = target->queue_size + 1;
aef9ec39 Roland Dreier 2005-11-02 577 init_attr->cap.max_recv_sge = 1;
aef9ec39 Roland Dreier 2005-11-02 578 init_attr->cap.max_send_sge = 1;
5cfb1782 Bart Van Assche 2014-05-20 579 init_attr->sq_sig_type = IB_SIGNAL_REQ_WR;
aef9ec39 Roland Dreier 2005-11-02 580 init_attr->qp_type = IB_QPT_RC;
73aa89ed Ishai Rabinovitz 2012-11-26 581 init_attr->send_cq = send_cq;
73aa89ed Ishai Rabinovitz 2012-11-26 582 init_attr->recv_cq = recv_cq;
aef9ec39 Roland Dreier 2005-11-02 583
19f31343 Bart Van Assche 2018-01-22 584 if (target->using_rdma_cm) {
19f31343 Bart Van Assche 2018-01-22 @585 ret = rdma_create_qp(ch->rdma_cm.cm_id, dev->pd, init_attr);
19f31343 Bart Van Assche 2018-01-22 586 qp = ch->rdma_cm.cm_id->qp;
19f31343 Bart Van Assche 2018-01-22 587 } else {
62154b2e Bart Van Assche 2014-05-20 588 qp = ib_create_qp(dev->pd, init_attr);
19f31343 Bart Van Assche 2018-01-22 589 if (!IS_ERR(qp)) {
19f31343 Bart Van Assche 2018-01-22 590 ret = srp_init_ib_qp(target, qp);
19f31343 Bart Van Assche 2018-01-22 591 if (ret)
19f31343 Bart Van Assche 2018-01-22 592 ib_destroy_qp(qp);
19f31343 Bart Van Assche 2018-01-22 593 } else {
73aa89ed Ishai Rabinovitz 2012-11-26 594 ret = PTR_ERR(qp);
19f31343 Bart Van Assche 2018-01-22 595 }
19f31343 Bart Van Assche 2018-01-22 596 }
19f31343 Bart Van Assche 2018-01-22 597 if (ret) {
19f31343 Bart Van Assche 2018-01-22 598 pr_err("QP creation failed for dev %s: %d\n",
19f31343 Bart Van Assche 2018-01-22 599 dev_name(&dev->dev->dev), ret);
da9d2f07 Roland Dreier 2010-02-24 600 goto err_send_cq;
aef9ec39 Roland Dreier 2005-11-02 601 }
aef9ec39 Roland Dreier 2005-11-02 602
002f1567 Bart Van Assche 2015-08-10 603 if (dev->use_fast_reg) {
5cfb1782 Bart Van Assche 2014-05-20 604 fr_pool = srp_alloc_fr_pool(target);
5cfb1782 Bart Van Assche 2014-05-20 605 if (IS_ERR(fr_pool)) {
5cfb1782 Bart Van Assche 2014-05-20 606 ret = PTR_ERR(fr_pool);
5cfb1782 Bart Van Assche 2014-05-20 607 shost_printk(KERN_WARNING, target->scsi_host, PFX
5cfb1782 Bart Van Assche 2014-05-20 608 "FR pool allocation failed (%d)\n", ret);
5cfb1782 Bart Van Assche 2014-05-20 609 goto err_qp;
5cfb1782 Bart Van Assche 2014-05-20 610 }
002f1567 Bart Van Assche 2015-08-10 611 } else if (dev->use_fmr) {
d1b4289e Bart Van Assche 2014-05-20 612 fmr_pool = srp_alloc_fmr_pool(target);
d1b4289e Bart Van Assche 2014-05-20 613 if (IS_ERR(fmr_pool)) {
d1b4289e Bart Van Assche 2014-05-20 614 ret = PTR_ERR(fmr_pool);
d1b4289e Bart Van Assche 2014-05-20 615 shost_printk(KERN_WARNING, target->scsi_host, PFX
d1b4289e Bart Van Assche 2014-05-20 616 "FMR pool allocation failed (%d)\n", ret);
d1b4289e Bart Van Assche 2014-05-20 617 goto err_qp;
d1b4289e Bart Van Assche 2014-05-20 618 }
d1b4289e Bart Van Assche 2014-05-20 619 }
d1b4289e Bart Van Assche 2014-05-20 620
509c07bc Bart Van Assche 2014-10-30 621 if (ch->qp)
9566b054 Bart Van Assche 2017-10-11 622 srp_destroy_qp(ch);
509c07bc Bart Van Assche 2014-10-30 623 if (ch->recv_cq)
1dc7b1f1 Christoph Hellwig 2015-11-13 624 ib_free_cq(ch->recv_cq);
509c07bc Bart Van Assche 2014-10-30 625 if (ch->send_cq)
1dc7b1f1 Christoph Hellwig 2015-11-13 626 ib_free_cq(ch->send_cq);
73aa89ed Ishai Rabinovitz 2012-11-26 627
509c07bc Bart Van Assche 2014-10-30 628 ch->qp = qp;
509c07bc Bart Van Assche 2014-10-30 629 ch->recv_cq = recv_cq;
509c07bc Bart Van Assche 2014-10-30 630 ch->send_cq = send_cq;
73aa89ed Ishai Rabinovitz 2012-11-26 631
7fbc67df Sagi Grimberg 2015-08-24 632 if (dev->use_fast_reg) {
7fbc67df Sagi Grimberg 2015-08-24 633 if (ch->fr_pool)
7fbc67df Sagi Grimberg 2015-08-24 634 srp_destroy_fr_pool(ch->fr_pool);
7fbc67df Sagi Grimberg 2015-08-24 635 ch->fr_pool = fr_pool;
7fbc67df Sagi Grimberg 2015-08-24 636 } else if (dev->use_fmr) {
7fbc67df Sagi Grimberg 2015-08-24 637 if (ch->fmr_pool)
7fbc67df Sagi Grimberg 2015-08-24 638 ib_destroy_fmr_pool(ch->fmr_pool);
7fbc67df Sagi Grimberg 2015-08-24 639 ch->fmr_pool = fmr_pool;
7fbc67df Sagi Grimberg 2015-08-24 640 }
7fbc67df Sagi Grimberg 2015-08-24 641
da9d2f07 Roland Dreier 2010-02-24 642 kfree(init_attr);
da9d2f07 Roland Dreier 2010-02-24 643 return 0;
da9d2f07 Roland Dreier 2010-02-24 644
da9d2f07 Roland Dreier 2010-02-24 645 err_qp:
19f31343 Bart Van Assche 2018-01-22 646 if (target->using_rdma_cm)
19f31343 Bart Van Assche 2018-01-22 @647 rdma_destroy_qp(ch->rdma_cm.cm_id);
19f31343 Bart Van Assche 2018-01-22 648 else
95c2ef50 Israel Rukshin 2017-05-11 649 ib_destroy_qp(qp);
da9d2f07 Roland Dreier 2010-02-24 650
da9d2f07 Roland Dreier 2010-02-24 651 err_send_cq:
1dc7b1f1 Christoph Hellwig 2015-11-13 652 ib_free_cq(send_cq);
da9d2f07 Roland Dreier 2010-02-24 653
da9d2f07 Roland Dreier 2010-02-24 654 err_recv_cq:
1dc7b1f1 Christoph Hellwig 2015-11-13 655 ib_free_cq(recv_cq);
aef9ec39 Roland Dreier 2005-11-02 656
da9d2f07 Roland Dreier 2010-02-24 657 err:
aef9ec39 Roland Dreier 2005-11-02 658 kfree(init_attr);
aef9ec39 Roland Dreier 2005-11-02 659 return ret;
aef9ec39 Roland Dreier 2005-11-02 660 }
aef9ec39 Roland Dreier 2005-11-02 661
:::::: The code at line 585 was first introduced by commit
:::::: 19f313438c7754e6cc2bceddeebeaa5132e2e0a1 IB/srp: Add RDMA/CM support
:::::: TO: Bart Van Assche <[email protected]>
:::::: CC: Doug Ledford <[email protected]>
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
On Fri, Apr 13, 2018 at 12:06:44AM -0700, Greg Thelen wrote:
> Allow INFINIBAND without INFINIBAND_ADDR_TRANS.
Why? We are pushing everyone heavily to use RDMA/CM, so making it
optional seems rather counter-intuitive.
You'll also have to fix tons of ULPs to explicitly depend on
INFINIBAND_ADDR_TRANS, or make code conditional on it.
On Sun, Apr 15, 2018 at 5:06 AM Christoph Hellwig <[email protected]> wrote:
> On Fri, Apr 13, 2018 at 12:06:44AM -0700, Greg Thelen wrote:
> > Allow INFINIBAND without INFINIBAND_ADDR_TRANS.
> Why? We are pushing everyone heavily to use RDMA/CM, so making it
> optional seems rather counter-intuitive.
Bugs. We don't currently use CM. But fuzzy testing has been finding a
fair number CM bugs, so we were thinking of disabling it until we need it.
> You'll also have to fix tons of ULPs to explicitly depend on
> INFINIBAND_ADDR_TRANS, or make code conditional on it.
I think I've identified the set of options which use
INFINIBAND_ADDR_TRANS without a kconfig depends:
* CIFS_SMB_DIRECT
* INFINIBAND_SRPT
* NVME_RDMA
* NVME_TARGET_RDMA
I have patches for the above, but need to finish the commit logs. Let me
know if they'll be nacked and I'll just patch my kernel and forget
upstreaming.
On Mon, Apr 16, 2018 at 04:02:12AM +0000, Greg Thelen wrote:
> I think I've identified the set of options which use
> INFINIBAND_ADDR_TRANS without a kconfig depends:
> * CIFS_SMB_DIRECT
> * INFINIBAND_SRPT
> * NVME_RDMA
> * NVME_TARGET_RDMA
>
> I have patches for the above, but need to finish the commit logs. Let me
> know if they'll be nacked and I'll just patch my kernel and forget
> upstreaming.
I think those are what is needed. But better do a full cycle through
the build bot first..
Btw, should we rename the config symbol to something containing
RDMACM? The current name seems rather confusing?
在 2018/4/13 20:47, Bart Van Assche 写道:
> On Fri, 2018-04-13 at 00:06 -0700, Greg Thelen wrote:
>> Allow INFINIBAND without INFINIBAND_ADDR_TRANS.
>>
>> Signed-off-by: Greg Thelen <[email protected]>
>> Cc: Tarick Bedeir <[email protected]>
>> ---
>> drivers/infiniband/Kconfig | 2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/drivers/infiniband/Kconfig b/drivers/infiniband/Kconfig
>> index ee270e065ba9..f20a3977087c 100644
>> --- a/drivers/infiniband/Kconfig
>> +++ b/drivers/infiniband/Kconfig
>> @@ -61,7 +61,7 @@ config INFINIBAND_ON_DEMAND_PAGING
>> pages on demand instead.
>>
>> config INFINIBAND_ADDR_TRANS
>> - bool
>> + bool "InfiniBand address translation"
>> depends on INFINIBAND
>> default y
>
> Hello Greg,
>
> Please change the short description from "InfiniBand address translation" (which
> is a misleading description) into "RDMA/CM". Please also add a help text that
> explains to people who are not RDMA experts what the RDMA/CM is.
>
> Thanks,
>
> Bart.
>
Hi, Bart
We need to select the INFINIBAND without INFINIBAND_ADDR_TRANS if use cm for others?
Thanks
Lijun Ou
>
>
> �{.n�+�������+%��lzwm��b�맲��r��zX��ݙ�����ܨ}���Ơz�&j:+v�������zZ+��+zf���h���~����i���z� �w���?����&�)ߢf
>
On Mon, Apr 16, 2018 at 01:56:20AM -0700, Christoph Hellwig wrote:
> On Mon, Apr 16, 2018 at 04:02:12AM +0000, Greg Thelen wrote:
> > I think I've identified the set of options which use
> > INFINIBAND_ADDR_TRANS without a kconfig depends:
> > * CIFS_SMB_DIRECT
> > * INFINIBAND_SRPT
> > * NVME_RDMA
> > * NVME_TARGET_RDMA
> >
> > I have patches for the above, but need to finish the commit logs. Let me
> > know if they'll be nacked and I'll just patch my kernel and forget
> > upstreaming.
>
> I think those are what is needed. But better do a full cycle through
> the build bot first..
Yes, you have to post the full series and have the build bots say it
is clean, collect acks from the various maintainers and then we can
apply the whole thing through RDMA..
I n't want to apply kconfig changes piecemeal, we just get bug reports
from build bots that way.
Jason