Received: by 10.223.185.116 with SMTP id b49csp179687wrg; Fri, 2 Mar 2018 16:22:55 -0800 (PST) X-Google-Smtp-Source: AG47ELuDiuJ6lrOlgQEU9J962uPpreM/vkzY1aYTP4b4AnKCzVkrEEE508de4uAY5DmKgFMhfF19 X-Received: by 2002:a17:902:7e4a:: with SMTP id a10-v6mr6773652pln.207.1520036575795; Fri, 02 Mar 2018 16:22:55 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1520036575; cv=none; d=google.com; s=arc-20160816; b=E3Ltq6Y3/K4B9jk+LfTOlSLqUB+l3p27z8zfWUboUiKHBLwCFa0pdvXrO5ZpZErFVK 1ZBJMidglCA3P0lK6fSXjpA1a5ap5+YXGUBDYdjWnjt4yihfli1+uGzYHRuhVkGR1Ps/ 9OT27jUNJD4jNovSXzBcshbbP5ENi8Lz/u/eQI5Q5/SAqtLeYVIFUmluxKsQucZP4Atv 4jcIDz6O/gZk15liIVEgqZJK6Sps9gGfbpxbhAyaCjOXEziw1r7x2bL1ipNG0+FTYH4o lX3pYWl8SReYWFZ3Si0CjJKyx3m0KjO/vlJGM7hgExO/9oEuQg3cfhyE+qnbsW2kyxnF YgVw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:subject:content-transfer-encoding :content-language:in-reply-to:mime-version:user-agent:date :message-id:from:references:cc:to:arc-authentication-results; bh=2SYWuax+TVMXVKIVfVGTRmm74PvZvxWGuWKFb0z02+0=; b=cJWX2O5wF/Vw3EuZ0OxbtkubXwg1qBEFQSI231/iL55aXhSwni0SyJ5cbd8CbMjTcS K7mBe4JfaAUeouFBafsDq8o/N0EBuKJg3IZ9NPrTqTzh3qaJREnh3cYeK3gIXAMUbufq BKXpkbIAWDOH+WDFEngJnLKGq9q5p0tLAKIx/nCcsTGwWCmWb0eykrv6SueydFwnQQ3F xFgNGNpT5qCjYNWxpX7rlfbd2EiqL1RRn+5bb/YiwO0E2+7BKILHMWnwZzE/4yHC/eCz A5sgOQ4PafMKtDs2pgaBthomYNXBKuiZ39YkadjBqyYKJAmbqagT2L/83llzyWFr8DhU 6eFQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id z5si839477pfe.255.2018.03.02.16.22.40; Fri, 02 Mar 2018 16:22:55 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1427452AbeCBRMb (ORCPT + 99 others); Fri, 2 Mar 2018 12:12:31 -0500 Received: from ale.deltatee.com ([207.54.116.67]:44528 "EHLO ale.deltatee.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1424699AbeCBRM2 (ORCPT ); Fri, 2 Mar 2018 12:12:28 -0500 Received: from guinness.priv.deltatee.com ([172.16.1.162]) by ale.deltatee.com with esmtp (Exim 4.89) (envelope-from ) id 1eroCe-00089N-Ep; Fri, 02 Mar 2018 10:10:41 -0700 To: Jason Gunthorpe , Stephen Bates Cc: Keith Busch , Sagi Grimberg , "linux-kernel@vger.kernel.org" , "linux-pci@vger.kernel.org" , "linux-nvme@lists.infradead.org" , "linux-rdma@vger.kernel.org" , "linux-nvdimm@lists.01.org" , "linux-block@vger.kernel.org" , Christoph Hellwig , Jens Axboe , Bjorn Helgaas , Max Gurtovoy , Dan Williams , =?UTF-8?B?SsOpcsO0bWUgR2xpc3Nl?= , Benjamin Herrenschmidt , Alex Williamson , Steve Wise References: <20180228234006.21093-11-logang@deltatee.com> <749e3752-4349-0bdf-5243-3d510c2b26db@grimberg.me> <40d69074-31a8-d06a-ade9-90de7712c553@deltatee.com> <5649098f-b775-815b-8b9a-f34628873ff4@grimberg.me> <20180301184249.GI19007@ziepe.ca> <20180301224540.GL19007@ziepe.ca> <77591162-4CCD-446E-A27C-1CDB4996ACB7@raithlin.com> <20180301234930.GG14799@localhost.localdomain> <20180302161842.GB14861@ziepe.ca> From: Logan Gunthorpe Message-ID: Date: Fri, 2 Mar 2018 10:10:31 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.6.0 MIME-Version: 1.0 In-Reply-To: <20180302161842.GB14861@ziepe.ca> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-SA-Exim-Connect-IP: 172.16.1.162 X-SA-Exim-Rcpt-To: swise@opengridcomputing.com, alex.williamson@redhat.com, benh@kernel.crashing.org, jglisse@redhat.com, dan.j.williams@intel.com, maxg@mellanox.com, bhelgaas@google.com, axboe@kernel.dk, hch@lst.de, linux-block@vger.kernel.org, linux-nvdimm@lists.01.org, linux-rdma@vger.kernel.org, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, sagi@grimberg.me, keith.busch@intel.com, sbates@raithlin.com, jgg@ziepe.ca X-SA-Exim-Mail-From: logang@deltatee.com X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on ale.deltatee.com X-Spam-Level: X-Spam-Status: No, score=-8.9 required=5.0 tests=ALL_TRUSTED,BAYES_00, GREYLIST_ISWHITE,T_RP_MATCHES_RCVD autolearn=ham autolearn_force=no version=3.4.1 Subject: Re: [PATCH v2 10/10] nvmet: Optionally use PCI P2P memory X-SA-Exim-Version: 4.2.1 (built Tue, 02 Aug 2016 21:08:31 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 02/03/18 09:18 AM, Jason Gunthorpe wrote: > This allocator is already seems not useful for the P2P target memory > on a Mellanox NIC due to the way it has a special allocation flow > (windowing) and special usage requirements.. > > Nor can it be usefull for the doorbell memory in the NIC. No one says every P2P use has to use P2P memory. Doorbells are obviously not P2P memory. But it's the p2mem interface that's important and the interface absolutely does not belong in the NVMe driver. Once you have the P2P memory interface you need an allocator behind it and the obvious place is in the P2P code to handle the common case where you're just mapping a BAR. We don't need to implement a genalloc in every driver that has P2P memory attached with it. If some hardware wants to expose memory that requires complicated allocation it's up to them to solve that problem but that isn't enough justification, to me, to push common code into every driver. > Both of these are existing use cases for P2P with out of tree patches.. And we have out of tree code that uses the generic allocator part of p2pmem. > The allocator seems to only be able to solve the CMB problem, and I > think due to that it is better to put this allocator in the NVMe > driver and not the core code.. At least until we find a 2nd user that > needs the same allocation scheme... See the first P2PMEM RFC. We used Chelsio's NIC instead of the CMB with a very similar allocation scheme. We'd still be enabling that NIC in the same way if we didn't run into hardware issues with it. A simple BAR with memory behind it is always going to be the most common case. Logan