Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1764061Ab3DDHfk (ORCPT ); Thu, 4 Apr 2013 03:35:40 -0400 Received: from mail.linux-iscsi.org ([67.23.28.174]:35548 "EHLO linux-iscsi.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1764040Ab3DDHfh (ORCPT ); Thu, 4 Apr 2013 03:35:37 -0400 From: "Nicholas A. Bellinger" To: target-devel Cc: linux-rdma , linux-scsi , linux-kernel , Roland Dreier , Or Gerlitz , Alexander Nezhinsky , Nicholas Bellinger Subject: [RFC-v3 0/9] Add support for iSCSI Extensions for RDMA (ISER) target mode Date: Thu, 4 Apr 2013 07:24:07 +0000 Message-Id: <1365060256-21506-1-git-send-email-nab@linux-iscsi.org> X-Mailer: git-send-email 1.7.2.5 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 6685 Lines: 138 From: Nicholas Bellinger This series is the third RFC for iSCSI Extensions for RDMA (ISER) target mode support planned for an future v3.10 merge. This series refactors existing traditional iscsi-target mode logic in order for external ib_isert.ko module code to function with areas common to traditional TCP socket based iSCSI and RDMA verbs based ISER operation. This includes a basic iscsit_transport API that allows different transports to reside under the existing iscsi-target configfs control plane, using an pre-defined network portal attribute to enable a rdma_cm listener on top of existing ipoib portals. At this point the code is functional and pushing sustained RDMA_WRITE + RDMA_READ traffic using open-iscsi on top of multiple iser network portals + multiple IB HCA ports + multiple LUNs. Thus far we're using Mellanox IB HCAs for initial development, and will be verfiying with RCoE capable NICs as well in the near future. This RFC-v3 code is available in git against v3.9-rc3 here: git://git.kernel.org/pub/scm/linux/kernel/git/nab/target-pending.git iser_target-rfc-v3 The changes included since RFC-v2 include: - Convert to use per isert_cq_desc->cq_[rx,tx]_work + drop tasklets (Or + nab) - Move IB_EVENT_QP_LAST_WQE_REACHED warn into correct isert_qp_event_callback (Or) - Drop unnecessary IB_ACCESS_REMOTE_* access flag usage in isert_create_device_ib_res (Or) - Add common isert_init_send_wr(), and convert isert_put_* calls (Or) - Move to verbs+core logic to single ib_isert.[c,h] (Or + nab) - Add kmem_cache isert_cmd_cache usage for descriptor allocation (nab) - Move common ib_post_send() logic used by isert_put_*() to isert_post_response() (nab) - Add isert_put_reject call in isert_response_queue() for posting ISCSI_REJECT response. (nab) - Add ISTATE_SEND_REJECT checking in isert_do_control_comp. (nab) - Add extra target_put_sess_cmd call in iscsit_add_reject_from_cmd after completion - Add iscsit_build_reject for REJECT PDU init + convert iscsit_send_reject() - Add iscsi_post_login_start_timers FIXME for ISER Many thanks again to Or Gerlitz from Mellanox for v2 review feedback! With the per isert_cq_desc->cq_[rx,tx]_work cmwq dispatch now in place, process context switch overhead is reduced to each cq callback, instead of per each ib_poll_cq ib_wc descriptor as with tasklets in RFC-v2 code. This allows RFC-v3 to reach within ~%5 of single-lun NULLIO small block IOPs performance parity vs. existing STGT iser code, at almost 1/10 the number of total context switches. As before, review patches are broken down into: Patch #1 adds the export of target_get_sess_cmd to be used by iscsi-target Patch #2 -> #4 include iscsi-target API template, conversion of iscsi/tcp login path to use API template, plus add iser RFC parameter keys. Patch #5 -> #6 allow external iscsi_cmd descriptor allocation / free, and refactoring of RX side PDU request handling to allow incoming PDU logic to be called by external ib_isert workqueue process context. Patch #7 allows iscsi-target to use per transport API template immediate / response callbacks in the per-connection TX thread completion path, and refactoring of response PDU creation for export to external ib_isert code. Patch #8 adds the pre-defined iser network portal attribute under the existing iscsi-target configfs tree. Patch #9 -> #12 is the external ib_isert.ko module code seperated into individual commits for review. Note that I'll be dropping #1 -> #8 changes into target-pending/for-next shortly, and planning to do the same with ib_isert code soon unless there is an objection from Roland. Please review.. Thank you! --nab Nicholas Bellinger (9): target: Add export of target_get_sess_cmd symbol iscsi-target: Add iscsit_transport API template iscsi-target: Initial traditional TCP conversion to iscsit_transport iscsi-target: Add iser-target parameter keys + setup during login iscsi-target: Add per transport iscsi_cmd alloc/free iscsi-target: Refactor RX PDU logic + export request PDU handling iscsi-target: Refactor TX queue logic + export response PDU creation iscsi-target: Add iser network portal attribute iser-target: Add iSCSI Extensions for RDMA (iSER) target driver drivers/infiniband/Kconfig | 1 + drivers/infiniband/Makefile | 1 + drivers/infiniband/ulp/isert/Kconfig | 6 + drivers/infiniband/ulp/isert/Makefile | 2 + drivers/infiniband/ulp/isert/ib_isert.c | 2270 ++++++++++++++++++++++++ drivers/infiniband/ulp/isert/ib_isert.h | 142 ++ drivers/infiniband/ulp/isert/isert_proto.h | 47 + drivers/target/iscsi/Makefile | 3 +- drivers/target/iscsi/iscsi_target.c | 1169 ++++++++----- drivers/target/iscsi/iscsi_target.h | 3 +- drivers/target/iscsi/iscsi_target_configfs.c | 94 +- drivers/target/iscsi/iscsi_target_core.h | 26 +- drivers/target/iscsi/iscsi_target_device.c | 7 +- drivers/target/iscsi/iscsi_target_erl1.c | 13 +- drivers/target/iscsi/iscsi_target_login.c | 470 ++++-- drivers/target/iscsi/iscsi_target_login.h | 6 + drivers/target/iscsi/iscsi_target_nego.c | 182 +-- drivers/target/iscsi/iscsi_target_nego.h | 11 +- drivers/target/iscsi/iscsi_target_parameters.c | 87 +- drivers/target/iscsi/iscsi_target_parameters.h | 16 +- drivers/target/iscsi/iscsi_target_tmr.c | 4 +- drivers/target/iscsi/iscsi_target_tpg.c | 6 +- drivers/target/iscsi/iscsi_target_transport.c | 55 + drivers/target/iscsi/iscsi_target_util.c | 53 +- drivers/target/iscsi/iscsi_target_util.h | 1 + drivers/target/target_core_transport.c | 4 +- include/target/iscsi/iscsi_transport.h | 83 + include/target/target_core_fabric.h | 2 +- 28 files changed, 3938 insertions(+), 826 deletions(-) create mode 100644 drivers/infiniband/ulp/isert/Kconfig create mode 100644 drivers/infiniband/ulp/isert/Makefile create mode 100644 drivers/infiniband/ulp/isert/ib_isert.c create mode 100644 drivers/infiniband/ulp/isert/ib_isert.h create mode 100644 drivers/infiniband/ulp/isert/isert_proto.h create mode 100644 drivers/target/iscsi/iscsi_target_transport.c create mode 100644 include/target/iscsi/iscsi_transport.h -- 1.7.2.5 -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/