Received: by 10.213.65.68 with SMTP id h4csp633567imn; Tue, 13 Mar 2018 15:50:46 -0700 (PDT) X-Google-Smtp-Source: AG47ELtaChbLIwVnLdEXz/AyBlqgZG4n5PsW8VtY1EmMK5JeZ8jdf3FmdAZ9RtVFN7suBA2tk104 X-Received: by 2002:a17:902:8c83:: with SMTP id t3-v6mr2018962plo.310.1520981446878; Tue, 13 Mar 2018 15:50:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1520981446; cv=none; d=google.com; s=arc-20160816; b=nv3Wa02+T1prnvJyT3d337XzCZlvKcTsypKcxkd9iNMk2m6T6at1VpAgTSsgtT2t85 qh2v4Gh+Ocsgaar1RzYL9TxRwkJJWdOsUANWTWqRKHOJF3DVVWxJ0FYLmDZHnj+q4UD3 2wFmjy7Bmm/J4PoUWpwU5pzzVIMN3knoPW4RaLNhz894O2TLj+Cm0aiso1l29Sf0UbOK VgjfSgwez4iHJt9aZAwgzw+25qLO2npmf0lYWyu1bSHCj8et9aEMP5TmPYfdWTPC3S+9 G6y76axzh8UVnrL4EtqWJkvtiOdwIPUveuXHwcoMKlXPk62Pev38SVCpMxenFWDadjzc CivA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:subject:content-transfer-encoding :content-language:in-reply-to:mime-version:user-agent:date :message-id:from:references:cc:to:arc-authentication-results; bh=uqECLRixyoM4TZPT97ie9LMBEHqm3g3WbY/S7EKXLd0=; b=SMfXZuy70hSKdrFMCpP2mTYfaZh4lEinXiRmZdZU64mIBG934Mu0l6VUDZ/KHHzHwX B3cZhlAEZ3ryTifDI45gYtWyA11nTat5kYV8CvKUZgrao/GjOsD8FqNoArz5rK+TNaVZ +bGy2/jV96cf0P4VxjcQ/enKEdzINWYQX47Y1KgnQW00W3Qy99FELBaQap2xzsoljXbt Hp9eIdB+Oi+U9gkqFuopLziIq06lSVBTGazoJv8Wk9tRqPU8FloQUW5e3ed/tcZwYNtq PBZkgdegGB3BLUroEy8VSb7ZChGx+MEHUo3V//k1dgow4HvEOv5i9NH4vAVxe0Wb9LBF Pr9w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id q10-v6si854229pll.237.2018.03.13.15.50.32; Tue, 13 Mar 2018 15:50:46 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753063AbeCMWtW (ORCPT + 99 others); Tue, 13 Mar 2018 18:49:22 -0400 Received: from ale.deltatee.com ([207.54.116.67]:36060 "EHLO ale.deltatee.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751300AbeCMWtT (ORCPT ); Tue, 13 Mar 2018 18:49:19 -0400 Received: from guinness.priv.deltatee.com ([172.16.1.162]) by ale.deltatee.com with esmtp (Exim 4.89) (envelope-from ) id 1evsj9-0008Jt-6c; Tue, 13 Mar 2018 16:49:04 -0600 To: Sinan Kaya , linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org, linux-nvdimm@lists.01.org, linux-block@vger.kernel.org Cc: Stephen Bates , Christoph Hellwig , Jens Axboe , Keith Busch , Sagi Grimberg , Bjorn Helgaas , Jason Gunthorpe , Max Gurtovoy , Dan Williams , =?UTF-8?B?SsOpcsO0bWUgR2xpc3Nl?= , Benjamin Herrenschmidt , Alex Williamson References: <20180312193525.2855-1-logang@deltatee.com> <20180312193525.2855-2-logang@deltatee.com> <59fd2f5d-177f-334a-a9c4-0f8a6ec7c303@codeaurora.org> <24d8e5c2-065d-8bde-3f5d-7f158be9c578@deltatee.com> <52cbbbc4-c488-f83f-8d02-14d455b4efd7@codeaurora.org> <3e738f95-d73c-4182-2fa1-8664aafb1ab7@deltatee.com> <703aa92c-0c1c-4852-5887-6f6e6ccde0fb@codeaurora.org> <3ea80992-a0fc-08f2-d93d-ae0ec4e3f4ce@codeaurora.org> <4eb6850c-df1b-fd44-3ee0-d43a50270b53@deltatee.com> <757fca36-dee4-e070-669e-f2788bd78e41@codeaurora.org> <4f761f55-4e9a-dccb-d12f-c59d2cd689db@deltatee.com> <016dc910-f96a-8a60-4bda-fa24eea98ea5@codeaurora.org> From: Logan Gunthorpe Message-ID: <2b152932-2f44-408b-e3ed-b4608d95f82e@deltatee.com> Date: Tue, 13 Mar 2018 16:48:55 -0600 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.6.0 MIME-Version: 1.0 In-Reply-To: <016dc910-f96a-8a60-4bda-fa24eea98ea5@codeaurora.org> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-SA-Exim-Connect-IP: 172.16.1.162 X-SA-Exim-Rcpt-To: alex.williamson@redhat.com, benh@kernel.crashing.org, jglisse@redhat.com, dan.j.williams@intel.com, maxg@mellanox.com, jgg@mellanox.com, bhelgaas@google.com, sagi@grimberg.me, keith.busch@intel.com, axboe@kernel.dk, hch@lst.de, sbates@raithlin.com, linux-block@vger.kernel.org, linux-nvdimm@lists.01.org, linux-rdma@vger.kernel.org, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, okaya@codeaurora.org X-SA-Exim-Mail-From: logang@deltatee.com X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on ale.deltatee.com X-Spam-Level: X-Spam-Status: No, score=-8.9 required=5.0 tests=ALL_TRUSTED,BAYES_00, GREYLIST_ISWHITE,T_RP_MATCHES_RCVD autolearn=ham autolearn_force=no version=3.4.1 Subject: Re: [PATCH v3 01/11] PCI/P2PDMA: Support peer-to-peer memory X-SA-Exim-Version: 4.2.1 (built Tue, 02 Aug 2016 21:08:31 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 13/03/18 04:29 PM, Sinan Kaya wrote: > If hardware doesn't support it, blacklisting should have been the right > path and I still think that you should remove all switch business from the code. > I did not hear enough justification for having a switch requirement > for P2P. I disagree. > You are also saying that root ports have issues not because of functionality but > because of performance. No... performance can also be an issue but the main justification for this is that many root ports do not support P2P at all and can fail in different ways (usually just dropping the TLPs). > What if I come up with a very cheap/crappy switch (something like used in data > mining)? Good luck. That's not how hardware is designed. PCIe switches that have any hope to compete with the existing market will need features like NTB, non-transparent ports, etc... and that implies a certain symmetry (ie there isn't a special host port because there may be more than one and it may move around) which implies that packets will always be able to forward between each ports which implies P2P will work. > I have been doing my best to provide feedback. It feels like you are throwing > them over the wall to be honest. > > You keep implying "not my problem". Well, the fact of the matter is that extending this in all the ways people like you want face huge problems on all sides. These are not trivial issues and holding back work that works for our problem because it doesn't solve your problem is IMO just going to grind development in this area to a halt. We have to have something we can agree on which is safe to start building on. The building can't just emerge fully formed in one go. P2P proposal go back a long time and have never gotten anywhere because there are limitations and people want it to do things that are hard but don't want to contribute the work to solving those problems. >> Well, if it's a problem for someone they'll have to solve it. We're >> targeting JBOFs that have no use for ACS / IOMMU groups at all. > > IMO, you (not somebody) should address this one way or the other before this > series land in upstream. The real way to address this (as I've mentioned before) is with some way of doing ACS and iomem groups dynamically. But this is a huge project in itself and is never going to be part of the P2P patchset. > Another assumption: There are other architectures like ARM64 where IOMMU > is enabled by default even if you don't use VMs for security reasons. > IOMMU blocks stray transactions. True, but it doesn't change my point: ACS is not a requirement for Linux many many systems do not have it on at all or by default. > Didn't the ACS behavior change suddenly for no good reason when we enabled > your code even though I might not be using the P2P but I happen to have > a kernel with P2P config option? Well no, presumably you made a conscious choice to turn the config option on and build a custom kernel for your box. That doesn't seem very sudden and the reason is that the two concepts are very much at odds with each other: you can't have isolation and still have transactions between devices. Logan