Subject: Re: [PATCH 3/9] IB: add a helper to safely drain a QP
To: Steve Wise <swise@opengridcomputing.com>,
        "'Christoph Hellwig'" <hch@lst.de>, linux-rdma@vger.kernel.org
References: <1447422410-20891-1-git-send-email-hch@lst.de>
 <1447422410-20891-4-git-send-email-hch@lst.de>
 <564851BB.1020004@dev.mellanox.co.il>
 <564A067B.8030504@opengridcomputing.com>
 <003001d1209c$ecb70760$c6251620$@opengridcomputing.com>
 <564A2270.1040004@dev.mellanox.co.il>
 <003801d120a1$799338d0$6cb9aa70$@opengridcomputing.com>
Cc: bart.vanassche@sandisk.com, axboe@fb.com, linux-scsi@vger.kernel.org,
        linux-kernel@vger.kernel.org
From: Sagi Grimberg <sagig@dev.mellanox.co.il>
Message-ID: <5652EA67.1000707@dev.mellanox.co.il>
Date: Mon, 23 Nov 2015 12:28:55 +0200
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101
 Thunderbird/38.3.0
MIME-Version: 1.0
In-Reply-To: <003801d120a1$799338d0$6cb9aa70$@opengridcomputing.com>
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 7bit
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 2429
Lines: 87


> That won't work for iWARP.  Is this code new?  I didn't see any errors that would result from this code when I tested iSER over
> cxgb4 with the old iwarp support patches.

Steve,

I think I figured out why this works with iWARP.

For iWARP, rdma_disconnect() calls iw_cm_disconnect() with abrupt=0
which would make iw_cm_disconnect() move the QP into SQ_DRAIN state"

int iw_cm_disconnect(struct iw_cm_id *cm_id, int abrupt)
{
	...

         if (qp) {
                 if (abrupt)
                         ret = iwcm_modify_qp_err(qp);
                 else
                         ret = iwcm_modify_qp_sqd(qp);

                 /*
                  * If both sides are disconnecting the QP could
                  * already be in ERR or SQD states
                  */
                 ret = 0;
	}
}

IFAIK, SQD state allows the ULP to post work requests on the send
queue and expect these work requests to FLUSH.

So Maybe we should have:
void ib_drain_qp(struct ib_qp *qp)
{
     struct ib_qp_attr attr = { };
     struct ib_stop_cqe rstop, sstop;
     struct ib_recv_wr rwr = {}, *bad_rwr;
     struct ib_send_wr swr = {}, *bad_swr;
     enum ib_qp_state state;
     int ret;

     if rdma_cap_ib_cm(id->device, id->port_num) {
	state = IB_QPS_ERR;
     else if rdma_cap_iw_cm(id->device, id->port_num)
         state = IB_QPS_SQD;
     else
        return;

     rwr.wr_cqe = &rstop.cqe;
     rstop.cqe.done = ib_stop_done;
     init_completion(&rstop.done);

     swr.wr_cqe = &sstop.cqe;
     sstop.cqe.done = ib_stop_done;
     swr.send_flags = IB_SEND_SIGNALED;
     init_completion(&sstop.done);

     attr.qp_state = state;
     ret = ib_modify_qp(qp, &attr, IB_QP_STATE);
     if (ret) {
         WARN_ONCE(ret, "failed to drain QP: %d\n", ret);
         return;
     }

     ret = ib_post_recv(qp, &rwr, &bad_rwr);
     if (ret) {
         WARN_ONCE(ret, "failed to drain recv queue: %d\n", ret);
         return;
     }

     ret = ib_post_send(qp, &swr, &bad_swr);
     if (ret) {
         WARN_ONCE(ret, "failed to drain send queue: %d\n", ret);
         return;
     }

     wait_for_completion(&rstop.done);
     wait_for_completion(&sstop.done);
}

Thoughts?
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/