Received: by 10.192.165.148 with SMTP id m20csp4860045imm; Tue, 8 May 2018 16:06:44 -0700 (PDT) X-Google-Smtp-Source: AB8JxZpFIaUtMEEuLftM87aITcAC8LKyufgUh60RmjX1r2mSB+SSZmyPQAzhZ9r678RwEz3sWHOQ X-Received: by 2002:a17:902:b94c:: with SMTP id h12-v6mr11372326pls.322.1525820803942; Tue, 08 May 2018 16:06:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1525820803; cv=none; d=google.com; s=arc-20160816; b=qwI4KiQD5S2Ml+gaJrSGuu5B0bUqvVhkmZUJ/GuS6g9Zs+qHYaXod5g671jFM5wwFY g5C9zMSUhoELahiquq8VMBTprSCrev66JgH+1kOXkG3pLG0sfzlT09xdBMFwzUkhxTUm 8hT/MftBy2+5vKVuYYFhDByLeFskXh5DpgElL6oQRKWg7ftyv4Q32UMzjbPqqlmSRm0V 2vYERyf3dRuJEkgOUm/+EpIrAAx0zQdPY/gE8QK3YzHySOtNJ2AGkAl/z5JCzZSYLz7S d+0/dP9KZl41qzVww5ILGM/DEB9w1IcVz0pm2CpuYv/9zBH/0rgfugDfeJje0e61NAup XCoQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding :content-language:in-reply-to:mime-version:user-agent:date :message-id:from:references:cc:to:subject:arc-authentication-results; bh=MYVxwzri3SMTO9n0nQNThkyHVO7sDsf8zONYJBijb+g=; b=Ooh3O4MdL4feXQI99G3JXMQpkOyAxvCyzeikK0YsXY6qqE5ucBWNctHstBKtAVs9Sh kaSm1ixIIXDQXGHSXl6kCXBNUzDYjsgETjS9YzUYsPBptQsr9b11xpsPcfjVJXaLszIc n8XdxWw94eruCN93QUP8jtFVEdeo7Hky6+OJ7DPa0dugQQ9lH6craDj0Qw6zxXm25JNG e15zqz9ga6MPtksaHT/ISZ8I5A2ru+rpH8sfXcyvwxwLYFZJD2ne0W0c9qY09URiGsxh /4aR8kZFlAvLPI8JSKNLD4IAgOHGN+dyoZISZ+yDBsGkj0oIIMWMcw/pd0A4TuIFVrwU p2RA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=redhat.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id f34-v6si25474860plf.362.2018.05.08.16.06.29; Tue, 08 May 2018 16:06:43 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933246AbeEHXGX (ORCPT + 99 others); Tue, 8 May 2018 19:06:23 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:59850 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S932961AbeEHXGU (ORCPT ); Tue, 8 May 2018 19:06:20 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id B223211F2E9; Tue, 8 May 2018 23:06:19 +0000 (UTC) Received: from [10.18.17.89] (dhcp-17-89.bos.redhat.com [10.18.17.89]) by smtp.corp.redhat.com (Postfix) with ESMTP id 848A7215CDA7; Tue, 8 May 2018 23:06:17 +0000 (UTC) Subject: Re: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches To: Stephen Bates , Dan Williams , Logan Gunthorpe Cc: Linux Kernel Mailing List , "linux-pci@vger.kernel.org" , "linux-nvme@lists.infradead.org" , linux-rdma , linux-nvdimm , "linux-block@vger.kernel.org" , Christoph Hellwig , Jens Axboe , Keith Busch , Sagi Grimberg , Bjorn Helgaas , Jason Gunthorpe , Max Gurtovoy , =?UTF-8?B?SsOpcsO0bWUgR2xpc3Nl?= , Benjamin Herrenschmidt , Alex Williamson , =?UTF-8?Q?Christian_K=c3=b6nig?= References: <20180423233046.21476-1-logang@deltatee.com> <20180423233046.21476-5-logang@deltatee.com> <64C231F5-DE36-415F-B308-3A423B0BBACB@raithlin.com> <15433946-f7f5-f610-4e80-380fb59920e5@redhat.com> <3C9FB262-A93C-4C8F-B1E0-85C6D6F78BC2@raithlin.com> From: Don Dutile Message-ID: Date: Tue, 8 May 2018 19:06:17 -0400 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.6.0 MIME-Version: 1.0 In-Reply-To: <3C9FB262-A93C-4C8F-B1E0-85C6D6F78BC2@raithlin.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Tue, 08 May 2018 23:06:19 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Tue, 08 May 2018 23:06:19 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'ddutile@redhat.com' RCPT:'' Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 05/08/2018 05:27 PM, Stephen Bates wrote: > Hi Don > >> Well, p2p DMA is a function of a cooperating 'agent' somewhere above the two devices. >> That agent should 'request' to the kernel that ACS be removed/circumvented (p2p enabled) btwn two endpoints. >> I recommend doing so via a sysfs method. > > Yes we looked at something like this in the past but it does hit the IOMMU grouping issue I discussed earlier today which is not acceptable right now. In the long term, once we get IOMMU grouping callbacks to VMs we can look at extending p2pdma in this way. But I don't think this is viable for the initial series. > > >> So I don't understand the comments why VMs should need to know. > > As I understand it VMs need to know because VFIO passes IOMMU grouping up into the VMs. So if a IOMMU grouping changes the VM's view of its PCIe topology changes. I think we even have to be cognizant of the fact the OS running on the VM may not even support hot-plug of PCI devices. Alex: Really? IOMMU groups are created by the kernel, so don't know how they would be passed into the VMs, unless indirectly via PCI(e) layout. At best, twiddling w/ACS enablement (emulation) would cause VMs to see different IOMMU groups, but again, VMs are not the security point/level, the host/HV's are. > >> Is there a thread I need to read up to explain /clear-up the thoughts above? > > If you search for p2pdma you should find the previous discussions. Thanks for the input! > > Stephen > > >