Received: by 10.223.185.116 with SMTP id b49csp7809194wrg; Thu, 1 Mar 2018 11:28:24 -0800 (PST) X-Google-Smtp-Source: AG47ELvHk9HBDLC7HpPTl6VaTENXMBudFoRHAFPvwu1x6qx2Vf0yYFPiGWtklF2SxRBzRNFIA3jo X-Received: by 2002:a17:902:7182:: with SMTP id b2-v6mr2830251pll.331.1519932504576; Thu, 01 Mar 2018 11:28:24 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1519932504; cv=none; d=google.com; s=arc-20160816; b=e5RvKgJzoHPk5lVfqBDeh/Kw3Y2sXlpfly7N/J64Tcn+A/S89WlHjWou97pNm2Ci0S Jho5stArXG8PImHfIxo8uzWbSVuoV+nxIEOKRxdrXXXbdiFOcwVTe28ombomTNStWxbL jzFiw4mQmU6gx9z7T03HEiQE3v4EqbUgkPG9C2HD7ys8ywGu1VBSojAvPLTK+q57zaXI 8bAkaUMpkRPhbGQbXWe9MZa5mxiHpbHys0lwPUpaprJzAtLXtJInPeeHyQgOJWAGi9YS IAiGZfQG1TnCGiUXrPSYOwUJxHxRCYcEMoTR8ouZLgZacnUstI/STKaFQ/elZlQgMquc Xgyg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:subject:content-transfer-encoding :content-language:in-reply-to:mime-version:user-agent:date :message-id:from:references:cc:to:arc-authentication-results; bh=tT2DBiQm/m8q3EHv69g9bbNzU2KZGx9wPlwNz5DK20A=; b=CnfF0mI7W0KyRfwaMPgq6Yd5S2AIkgTWVwKvdeNu893K7X1RU4cRahoSnayvL2Os2B K8eXQmMR/vxAPi3EYYuB0xGDh4Q7PG0dGYiqXBIIjXG+gPc8/QJDgR+iQ4lscmo3L1E5 EIXmqsSu5uN/EZf38u2wmSHVdtmNPELlgdsomc8dO62Ydk/oQXp5CCXTCer30ZbT785R 7afJTuDgDW4gHqgdZTlyWPvJooEStk6lJcXtBO1TDPE6qGKuA/Rx4aXwnvmmSJB960un IyaMe+LBQ8gALlelb85hd5AuxLjPgZUAgnQT37eYR+svmfDj5+JWYsCqFMJZPb5OPBXI tYpQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id s26si3455223pfg.264.2018.03.01.11.28.09; Thu, 01 Mar 2018 11:28:24 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1161294AbeCAT1V (ORCPT + 99 others); Thu, 1 Mar 2018 14:27:21 -0500 Received: from ale.deltatee.com ([207.54.116.67]:37664 "EHLO ale.deltatee.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1033435AbeCAT1T (ORCPT ); Thu, 1 Mar 2018 14:27:19 -0500 Received: from guinness.priv.deltatee.com ([172.16.1.162]) by ale.deltatee.com with esmtp (Exim 4.89) (envelope-from ) id 1erTr7-0000Fz-Ry; Thu, 01 Mar 2018 12:27:06 -0700 To: Jason Gunthorpe , Sagi Grimberg Cc: linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org, linux-nvdimm@lists.01.org, linux-block@vger.kernel.org, Stephen Bates , Christoph Hellwig , Jens Axboe , Keith Busch , Bjorn Helgaas , Max Gurtovoy , Dan Williams , =?UTF-8?B?SsOpcsO0bWUgR2xpc3Nl?= , Benjamin Herrenschmidt , Alex Williamson , Steve Wise References: <20180228234006.21093-1-logang@deltatee.com> <20180228234006.21093-11-logang@deltatee.com> <749e3752-4349-0bdf-5243-3d510c2b26db@grimberg.me> <40d69074-31a8-d06a-ade9-90de7712c553@deltatee.com> <5649098f-b775-815b-8b9a-f34628873ff4@grimberg.me> <20180301184249.GI19007@ziepe.ca> From: Logan Gunthorpe Message-ID: Date: Thu, 1 Mar 2018 12:27:03 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.6.0 MIME-Version: 1.0 In-Reply-To: <20180301184249.GI19007@ziepe.ca> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-SA-Exim-Connect-IP: 172.16.1.162 X-SA-Exim-Rcpt-To: swise@opengridcomputing.com, alex.williamson@redhat.com, benh@kernel.crashing.org, jglisse@redhat.com, dan.j.williams@intel.com, maxg@mellanox.com, bhelgaas@google.com, keith.busch@intel.com, axboe@kernel.dk, hch@lst.de, sbates@raithlin.com, linux-block@vger.kernel.org, linux-nvdimm@lists.01.org, linux-rdma@vger.kernel.org, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, sagi@grimberg.me, jgg@ziepe.ca X-SA-Exim-Mail-From: logang@deltatee.com X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on ale.deltatee.com X-Spam-Level: X-Spam-Status: No, score=-8.9 required=5.0 tests=ALL_TRUSTED,BAYES_00, GREYLIST_ISWHITE,T_RP_MATCHES_RCVD autolearn=ham autolearn_force=no version=3.4.1 Subject: Re: [PATCH v2 10/10] nvmet: Optionally use PCI P2P memory X-SA-Exim-Version: 4.2.1 (built Tue, 02 Aug 2016 21:08:31 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 01/03/18 11:42 AM, Jason Gunthorpe wrote: > On Thu, Mar 01, 2018 at 08:35:55PM +0200, Sagi Grimberg wrote: > This is also why I don't entirely understand why this series has a > generic allocator for p2p mem, it makes little sense to me. > Why wouldn't the nmve driver just claim the entire CMB of its local > device for its own use? Why involve p2p core code in this? We'd prefer to have a generic way to get p2pmem instead of restricting ourselves to only using CMBs. We did work in the past where the P2P memory was part of an IB adapter and not the NVMe card. So this won't work if it's an NVMe only interface. As Stephen mentioned, we also use a couple devices that only exposes P2P memory and this isn't related to NVMe at all. So there's value in having a generic interface and allocator to enable all devices to provide this memory. If there were a hypothetical situation where a driver wants to use some of the memory for P2P and some of it for other purposes then they'd just divide it themselves and only pass a subset to pci_p2pdma_add_resource(). However, as per our changes to the nvme-pci code, it's really just easier to use the allocator for everything inside the driver. Logan