Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752834AbbKXCmh (ORCPT ); Mon, 23 Nov 2015 21:42:37 -0500 Received: from p3plsmtpa08-08.prod.phx3.secureserver.net ([173.201.193.109]:37331 "EHLO p3plsmtpa08-08.prod.phx3.secureserver.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751568AbbKXCmf (ORCPT ); Mon, 23 Nov 2015 21:42:35 -0500 X-Greylist: delayed 430 seconds by postgrey-1.27 at vger.kernel.org; Mon, 23 Nov 2015 21:42:35 EST Subject: Re: [PATCH 2/9] IB: add a proper completion queue abstraction To: Jason Gunthorpe References: <20151113220636.GA32133@obsidianresearch.com> <20151114071344.GE27738@lst.de> <20151123203712.GB5640@obsidianresearch.com> <56537F59.4080708@sandisk.com> <20151123212822.GE6062@obsidianresearch.com> <56538AFD.9080103@sandisk.com> <20151123221806.GA7152@obsidianresearch.com> <56539421.9050705@sandisk.com> <20151123230659.GA8287@obsidianresearch.com> <20151124000011.GA9301@obsidianresearch.com> Cc: Bart Van Assche , Christoph Hellwig , "linux-rdma@vger.kernel.org" , "sagig@dev.mellanox.co.il" , "axboe@fb.com" , "linux-scsi@vger.kernel.org" , "linux-kernel@vger.kernel.org" From: Caitlin Bestler Message-ID: <5653CCF0.7050501@asomi.com> Date: Mon, 23 Nov 2015 18:35:28 -0800 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <20151124000011.GA9301@obsidianresearch.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 1734 Lines: 43 On 11/23/2015 4:00 PM, Jason Gunthorpe wrote: > On Mon, Nov 23, 2015 at 03:30:42PM -0800, Caitlin Bestler wrote: >> The receive completion can be safely assumed to indicate transmit >> completion over a reliable connection unless your peer has gone >> completely bonkers and is replying to a command that it did not >> receive. > Perhaps iWarp is different and does specify this ordering but IB does > not. > > The issue with IB is how the ACK protocol is designed. There is not > strong ordering between ACKs and data transfers. A HCA can send > ACK,DATA and the network could drop the ACK. The recevier side does > not know the ACK was lost and goes ahead to process DATA. > > Since only ACK advances the sendq and DATA advances the recvq it is > trivial to get a case where the recvq is advanced with a reply while > the sendq continues to wait for the ACK to be resent. > > Further IB allows ACK coalescing and has no rules for how an ACK is > placed. It is entirely valid for a HCA to RECV,REPLY,ACK - for > instance. > > Is it possible for an IB HCA to transmit a response on a QP and not in that packet or a previous packet acknowledge something that it has delivered to the user? My recollection of the IB verbs is that they were unlikely to have overlooked something like that. If it did slip through then there should be an errata. But regardless of specification lawyering, is this an implementation issue. Are there actual HCAs that make this mistake? -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/