Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753430AbbKXM7z (ORCPT ); Tue, 24 Nov 2015 07:59:55 -0500 Received: from p3plsmtpa06-07.prod.phx3.secureserver.net ([173.201.192.108]:48243 "EHLO p3plsmtpa06-07.prod.phx3.secureserver.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752826AbbKXM7y (ORCPT ); Tue, 24 Nov 2015 07:59:54 -0500 X-Greylist: delayed 430 seconds by postgrey-1.27 at vger.kernel.org; Tue, 24 Nov 2015 07:59:54 EST Subject: Re: [PATCH 2/9] IB: add a proper completion queue abstraction To: Jason Gunthorpe , Caitlin Bestler References: <20151123203712.GB5640@obsidianresearch.com> <56537F59.4080708@sandisk.com> <20151123212822.GE6062@obsidianresearch.com> <56538AFD.9080103@sandisk.com> <20151123221806.GA7152@obsidianresearch.com> <56539421.9050705@sandisk.com> <20151123230659.GA8287@obsidianresearch.com> <20151124000011.GA9301@obsidianresearch.com> <5653CCF0.7050501@asomi.com> <20151124070301.GA23597@obsidianresearch.com> Cc: Bart Van Assche , Christoph Hellwig , "linux-rdma@vger.kernel.org" , "sagig@dev.mellanox.co.il" , "axboe@fb.com" , "linux-scsi@vger.kernel.org" , "linux-kernel@vger.kernel.org" From: Tom Talpey Message-ID: <56545D98.1020904@talpey.com> Date: Tue, 24 Nov 2015 07:52:40 -0500 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <20151124070301.GA23597@obsidianresearch.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 1408 Lines: 28 On 11/24/2015 2:03 AM, Jason Gunthorpe wrote: > On Mon, Nov 23, 2015 at 06:35:28PM -0800, Caitlin Bestler wrote: >> Are there actual HCAs that make this mistake? > > All IB HCAs have this behavior and require apps to see a send CQ > completion before making any statements about the state of the send Q > or buffers handed over to the HCA. Tom and I have seen this in real > systems under proper stress conditions. [Which is why I am so certain > about this, because when I first hit it years ago I dug into the spec > and figured out it was not a HW bug I was looking at] To be clear, I saw the reply-completion-before-request-completion on Windows, not Linux, but the principle is identical. It's simply a fact of life on a multiprocessor, unless you want to throw in locks and synchronization rules that drivers have to follow to enforce ordered completions across queues. Which trust me, you don't. In Windows SMB Direct, we added reference counts around pretty much every verb interaction associated with each upper layer operation, and did not retire them until all refcounts went to zero. It is excruciatingly correct yet performs incredibly well. Tom. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/