Subject: Re: [PATCH 4/9] srpt: chain RDMA READ/WRITE requests
To: Sagi Grimberg <sagig@dev.mellanox.co.il>, Christoph Hellwig <hch@lst.de>,
        <linux-rdma@vger.kernel.org>
References: <1447422410-20891-1-git-send-email-hch@lst.de>
 <1447422410-20891-5-git-send-email-hch@lst.de> <564BD1AF.60200@sandisk.com>
 <564C41B8.2080904@dev.mellanox.co.il>
CC: <axboe@fb.com>, <linux-scsi@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>
From: Bart Van Assche <bart.vanassche@sandisk.com>
Message-ID: <564CA83B.4060403@sandisk.com>
Date: Wed, 18 Nov 2015 08:32:59 -0800
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
 Thunderbird/38.3.0
MIME-Version: 1.0
In-Reply-To: <564C41B8.2080904@dev.mellanox.co.il>
Content-Type: text/plain; charset="windows-1252"; format=flowed
Content-Transfer-Encoding: 7bit
SpamDiagnosticOutput: 1:23
SpamDiagnosticMetadata: NSPM
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Nov 2015 16:32:59.6259
 (UTC)
X-MS-Exchange-CrossTenant-Id: fcd9ea9c-ae8c-460c-ab3c-3db42d7ac64d
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=fcd9ea9c-ae8c-460c-ab3c-3db42d7ac64d;Ip=[63.163.107.173];Helo=[milsmgep12.sandisk.com]
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY1PR02MB1384
X-Microsoft-Exchange-Diagnostics: 1;CY1PR02MB1102;2:PHrx76QBLEYoc81jJKa7rDq9Fv7N0yqwfpQzHAY3urFZe9GHG4xKnqIg10DnbEO72egfyMg0xgq6nNGqWkK8QGIy/ZSX6skvCmyzEVSuLx+t0/JhtXfWkMuYDg+rYDvl4hrX9wPd5jK+Y7jTN0TSC6eQuN0eUIN4bNa61hYYRp8=;23:88iaSts64DYcD5GRzgZz4q/wtgeF7QIlc62xbufz25KmmmhPn+/djSJLta5wfMR+VJqLyUSZw/xYXiOL6AJsnsjY/E3g8HvPmXgppcL3QQsJIgVYdpQsMO5nqkyywZgYvc7Y+OULJrx0gx2a07z3MJwZ491gYGkm4tY7SMxHGnckhcR3rCFXRx4io21YjqDd
X-OriginatorOrg: sandisk.com
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
Content-Length: 2716
Lines: 64

On 11/18/2015 01:15 AM, Sagi Grimberg wrote:
> On 18/11/2015 03:17, Bart Van Assche wrote:
>> On 11/13/2015 05:46 AM, Christoph Hellwig wrote:
>>> -        ret = ib_post_send(ch->qp, &wr.wr, &bad_wr);
>>> -        if (ret)
>>> -            break;
>>> +        if (i == n_rdma - 1) {
>>> +            /* only get completion event for the last rdma read */
>>> +            if (dir == DMA_TO_DEVICE)
>>> +                wr->wr.send_flags = IB_SEND_SIGNALED;
>>> +            wr->wr.next = NULL;
>>> +        } else {
>>> +            wr->wr.next = &ioctx->rdma_ius[i + 1].wr;
>>> +        }
>>>       }
>>>
>>> +    ret = ib_post_send(ch->qp, &ioctx->rdma_ius->wr, &bad_wr);
>>>       if (ret)
>>>           pr_err("%s[%d]: ib_post_send() returned %d for %d/%d\n",
>>>                    __func__, __LINE__, ret, i, n_rdma);
>>
>> Hello Christoph,
>
> Hi Bart,
>
>>
>> Chaining RDMA requests is a great idea. But it seems to me that this
>> patch is based on the assumption that posting multiple RDMA requests
>> either succeeds as a whole or fails as a whole. Sorry but I'm not sure
>> that the verbs API guarantees this. In the ib_srpt driver a QP can be
>> changed at any time into the error state and there might be drivers that
>> report an immediate failure in that case.
>
> I'm not so sure it actually matters if some WRs succeeded. In the normal
> flow when srpt has enough available work requests (sq_wr_avail) they
> should all succeed otherwise we're in trouble. If the QP transitioned
> to ERROR state, then some failed, but those that succeeded will
> generate flush completions, and srpt should handle it correctly
> shouldn't it?
>
>> I think even when chaining
>> RDMA requests that we still need a mechanism to wait until ongoing RDMA
>> transfers have finished if some but not all RDMA requests have been
>> posted.
>
> I'm not an expert on srpt, can you explain how this mechanism will help?

Hello Sagi,

As you know events like a cable pull can cause some of the RDMA work 
requests to succeed and others to fail. It is essential that all RDMA 
work requests related to the same SCSI command have finished before the 
buffers these requests operate upon are reused. The purpose of the 
SRPT_RDMA_ABORT request is to wait for the RDMA requests that were 
posted without IB_SEND_SIGNALED and for which no error completion will 
be received. BTW, I think this consideration applies to all SCSI target 
drivers and not only to SRP target drivers.

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/