Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752419AbbKXHDT (ORCPT ); Tue, 24 Nov 2015 02:03:19 -0500 Received: from quartz.orcorp.ca ([184.70.90.242]:56935 "EHLO quartz.orcorp.ca" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751485AbbKXHDR (ORCPT ); Tue, 24 Nov 2015 02:03:17 -0500 Date: Tue, 24 Nov 2015 00:03:01 -0700 From: Jason Gunthorpe To: Caitlin Bestler Cc: Bart Van Assche , Christoph Hellwig , "linux-rdma@vger.kernel.org" , "sagig@dev.mellanox.co.il" , "axboe@fb.com" , "linux-scsi@vger.kernel.org" , "linux-kernel@vger.kernel.org" Subject: Re: [PATCH 2/9] IB: add a proper completion queue abstraction Message-ID: <20151124070301.GA23597@obsidianresearch.com> References: <20151123203712.GB5640@obsidianresearch.com> <56537F59.4080708@sandisk.com> <20151123212822.GE6062@obsidianresearch.com> <56538AFD.9080103@sandisk.com> <20151123221806.GA7152@obsidianresearch.com> <56539421.9050705@sandisk.com> <20151123230659.GA8287@obsidianresearch.com> <20151124000011.GA9301@obsidianresearch.com> <5653CCF0.7050501@asomi.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5653CCF0.7050501@asomi.com> User-Agent: Mutt/1.5.23 (2014-03-12) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 2178 Lines: 51 On Mon, Nov 23, 2015 at 06:35:28PM -0800, Caitlin Bestler wrote: > Is it possible for an IB HCA to transmit a response on a QP and not > in that packet or a previous packet acknowledge something that it > has delivered to the user? AFAIK, the rules of ack coalescing do not interact with the send side. Even if they did, that is the wrong place to look at. We must look at the receiver. Ordered ack,data on the wire may suffer a packet loss and the ack may not reach the reciever. In this case can the reciever detect the lost ack and not progress the data? For IB, it cannot. The ack sequencing is part of the transmitters recv FSM, and does not interact with the send FSM. I feel this a deliberate IB design choice to be simple and efficient in hardware. > My recollection of the IB verbs is that they were unlikely to have > overlooked something like that. If it did slip through then there > should be an errata. verbs reflects the wire protocol and the wire protocol has no way to create a linkage between the send and recv sides of a RC QP. It is not a spec bug, there is no errata. > But regardless of specification lawyering, is this an implementation > issue. All IB implementations have no choice but to act this way - the wire protocol provides no way to guarentee ack vs data sequencing at the receiver, so there is simply no way to guarentee the sendq advances strictly causally with the recvq. > Are there actual HCAs that make this mistake? All IB HCAs have this behavior and require apps to see a send CQ completion before making any statements about the state of the send Q or buffers handed over to the HCA. Tom and I have seen this in real systems under proper stress conditions. [Which is why I am so certain about this, because when I first hit it years ago I dug into the spec and figured out it was not a HW bug I was looking at] This is a direct consequence of how IB runs the ACK protocol. Jason -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/