Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932476AbbKYGV3 (ORCPT ); Wed, 25 Nov 2015 01:21:29 -0500 Received: from quartz.orcorp.ca ([184.70.90.242]:46716 "EHLO quartz.orcorp.ca" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932343AbbKYGV1 (ORCPT ); Wed, 25 Nov 2015 01:21:27 -0500 Date: Tue, 24 Nov 2015 23:21:13 -0700 From: Jason Gunthorpe To: Caitlin Bestler Cc: Bart Van Assche , Christoph Hellwig , "linux-rdma@vger.kernel.org" , "sagig@dev.mellanox.co.il" , "axboe@fb.com" , "linux-scsi@vger.kernel.org" , "linux-kernel@vger.kernel.org" Subject: Re: [PATCH 2/9] IB: add a proper completion queue abstraction Message-ID: <20151125062113.GA29187@obsidianresearch.com> References: <20151124100839.48b52fb35c6f209c51bccbb9807b6df0.f113bf890f.wbe@email24.secureserver.net> <20151124175226.GA10391@obsidianresearch.com> <56552132.7090701@asomi.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <56552132.7090701@asomi.com> User-Agent: Mutt/1.5.23 (2014-03-12) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 1969 Lines: 52 On Tue, Nov 24, 2015 at 06:47:14PM -0800, Caitlin Bestler wrote: > Acknowledge packet for the last packet of the request has been > committed to the wire (including the appropriate fields for RDMA > READ response). > The target side cannot generate a response before it receives the request. > The target side verbs cannot generate the completion before the acknowledge > packet has been > committed to the wire. Sure, but, I keep saying this, the responder behavior is largely irrelevant to what the target is able/required to do. > Therefore the initiating side cannot receive a response before the write > operation has completed. Wrong. The ladder diagram would be Requestor Responder Responder Verbs SEND1 -----------> Process X--------- ACK (lost) --------> recv1 CQ <------- send2 WQE recv2 CQE <-------- SEND2 Packet [..] send1 CQE <-------- ACK (resent) The Ack may be lost, causing the send CQE to arrive after the recv CQE, even though the responder did everything in a specific order. The fundamental issue is that the responder cannot detect the lost ACK. The PSN of the ACK packet is part of the Requestor's PSN space, not part of the Responders: 9.7.5.1.1 GENERATING PSNs FOR ACKNOWLEDGE MESSAGES C9-95: For responses to SEND requests or RDMA WRITE requests the responder shall insert in the PSN field of the response the PSN of the most recent request packet being acknowledged. Or stated another way, the value of the AckReq bit in SEND1 has no impact on the contents of the SEND2 packet - thus there is no way for the requestor to detect the loss of the ACK and hold off delivering recv2 CQE. Jason -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/