Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Mon, 25 Oct 2021 13:51:19 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Jean-Philippe Brucker <jean-philippe@linaro.org>
Cc:     "Tian, Kevin" <kevin.tian@intel.com>,
        Alex Williamson <alex.williamson@redhat.com>,
        "Liu, Yi L" <yi.l.liu@intel.com>, "hch@lst.de" <hch@lst.de>,
        "jasowang@redhat.com" <jasowang@redhat.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "parav@mellanox.com" <parav@mellanox.com>,
        "lkml@metux.net" <lkml@metux.net>,
        "pbonzini@redhat.com" <pbonzini@redhat.com>,
        "lushenming@huawei.com" <lushenming@huawei.com>,
        "eric.auger@redhat.com" <eric.auger@redhat.com>,
        "corbet@lwn.net" <corbet@lwn.net>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "yi.l.liu@linux.intel.com" <yi.l.liu@linux.intel.com>,
        "Tian, Jun J" <jun.j.tian@intel.com>, "Wu, Hao" <hao.wu@intel.com>,
        "Jiang, Dave" <dave.jiang@intel.com>,
        "jacob.jun.pan@linux.intel.com" <jacob.jun.pan@linux.intel.com>,
        "kwankhede@nvidia.com" <kwankhede@nvidia.com>,
        "robin.murphy@arm.com" <robin.murphy@arm.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "iommu@lists.linux-foundation.org" <iommu@lists.linux-foundation.org>,
        "dwmw2@infradead.org" <dwmw2@infradead.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
        "david@gibson.dropbear.id.au" <david@gibson.dropbear.id.au>,
        "nicolinc@nvidia.com" <nicolinc@nvidia.com>
Subject: Re: [RFC 10/20] iommu/iommufd: Add IOMMU_DEVICE_GET_INFO
Message-ID: <20211025165119.GA2744544@nvidia.com>
References: <20210923114219.GG964074@nvidia.com>
 <BN9PR11MB5433519229319BA951CA97638CAA9@BN9PR11MB5433.namprd11.prod.outlook.com>
 <20210930222355.GH964074@nvidia.com>
 <BN9PR11MB5433530032DC8400B71FCB788CB89@BN9PR11MB5433.namprd11.prod.outlook.com>
 <20211014154259.GT2744544@nvidia.com>
 <BN9PR11MB543327BB6D58AEF91AD2C9D18CB99@BN9PR11MB5433.namprd11.prod.outlook.com>
 <BL1PR11MB5429973588E4FBCEC8F519A88CBF9@BL1PR11MB5429.namprd11.prod.outlook.com>
 <YXF/+jxRtjnlXU7w@myrica>
 <20211021232223.GM2744544@nvidia.com>
 <YXJs7+nQJ++EKIyT@myrica>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <YXJs7+nQJ++EKIyT@myrica>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?2AWuWXX5+/nE17iKFxsFA/RXI7P5TDNV1xwyJomgIG1NLuXUQ5kb4R3ifjri?=
 =?us-ascii?Q?ST2v/nPtRwweCHFpTuzOK2n6Sc8lGBQ84brdgLBMIpylHks/B6uqE49Xcprf?=
 =?us-ascii?Q?ww07pewm5e7YElg0teTKxJ6j+/CtrS8yCCRnGv47JWo0bP1LJVvFS/HCv8P4?=
 =?us-ascii?Q?tiUsVZfws90+3SpkISj6/ZcnAmlq/63C421Xmd8nwiKUzCf3iCRiM9TzAThU?=
 =?us-ascii?Q?PKvm18keP2rMYsujnBOjxRTxiUblJc5NIf//h8BHuDfK2UT3MsTammZQ/Wte?=
 =?us-ascii?Q?oyKPQ5+ohGBIHOhnDsAYDfy+/FD4DrXicuwpwnXSGT3zMy3Qad2LwPeqL5j/?=
 =?us-ascii?Q?up3LR5df7/uMBos6K2qcZU6K/IohHiD7dfrZf6bBIzL2vVd1zqNpEp+6ToOD?=
 =?us-ascii?Q?28cS/3s64lJI+km5wXavkb30+4/rrUOAoI4fUxY2nZau0TqCsL6lt877m0L4?=
 =?us-ascii?Q?Oo46csK3FlnKBbC+ED5Olh5jdzK0bU8fcSsGcxPm+rLM4oQuHoDGu3+1lxP+?=
 =?us-ascii?Q?ZLuBaK6Tj46kl9pC9AJAkzi6n3O2P8eC4DSxjUVop1egKvpwm4m64LsSjfEU?=
 =?us-ascii?Q?RMr5BSXxwln9a7pm4ieYk83dnVSDJJQcdDa4hLAS356MGkljFxSDt4GGx6j7?=
 =?us-ascii?Q?U9MpK10UL/8qdY36PJSU3uNivqBERoaYo4LeCsPfbdlxhjgRQRlDezqLuNca?=
 =?us-ascii?Q?FBr25hqbdHs8jRr5qxPDZykX+06mWXBvOlFWSkJoysVNk3OFSZdM5EFf0/hk?=
 =?us-ascii?Q?rupQkfXv+h/D9ETnhgVwEeA7c/ixwZIRpxahRf44B2GaJ52j5lvJsLoUnssQ?=
 =?us-ascii?Q?VLAVwRlM2BSSAyKYWIYcKJopt9HiSQoNvJjzGohZAtaM5P9pXRvWYg2GqJfF?=
 =?us-ascii?Q?08smwzh5pVtwC1v+MCghV1+EwJam+v74nG7RINaMrP5atRGNUHXGUQ/LZHZ5?=
 =?us-ascii?Q?5qnUbgBjOsVg63hh0YKUO4J2LTJADw58q5tcmr50zEBe1Rf7XQ8NFd/SsZeN?=
 =?us-ascii?Q?489tCucgejuMehsBj9dK5hhgSM8qQgInHzzt5AO375Gu8Dl5aNEz8RUnw6wt?=
 =?us-ascii?Q?vunSMu1520rHR9J9mnNw7OXo+W4tAhRkpk9bn5obitD01lUSo9sx4/wJYRp+?=
 =?us-ascii?Q?qD490BwloorZzAzjyrMzDVJ/79nCPGWozmT9bg4adGBn6P58USWdHPXQzvJB?=
 =?us-ascii?Q?yWeUGSGrsWKCa/IEV0lFXGjxv5QCjF3Jy+HoNN4DkRL3ZzOQNZpqjehO0UCc?=
 =?us-ascii?Q?Fhys1ox7FmDRLo2x85hK7VAaCkoCST4b8fOymq9qyEf7i9qbR6m0YOnfAfzs?=
 =?us-ascii?Q?OX1C/sWF7yEMtjzy3KOVsopfPNj/G5hsPc/CRlNqys19knQldZ0mWCO8VVuC?=
 =?us-ascii?Q?lO9WC05z49ESBnKw2udQqtfGLx/b2CMQkZV5+rpfdZIjsYKeXFMn29p7KZVo?=
 =?us-ascii?Q?u4VTWVlbfCZoWAeYhFq1rdE615iV2exOfyjeY2mTYfQ8ygUmxTYVMX/Cu1Ey?=
 =?us-ascii?Q?PnRD5LVuDvZlSsykS1P+36NCkbBv+IiYvQWc6ZIyP0sbDFCv0AvT7fB0uJpj?=
 =?us-ascii?Q?RdQFZvxYgnhNoabfOMI=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 78019065-b8b4-45ea-346b-08d997d7ac8b
X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB5506.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Oct 2021 16:51:22.1229
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Q8t9I6+gAvQsU9b9BnjibjC8+x0hWtILjJ0AF+QNWtUitfX8JnhmTfCkmS/vrsIW
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5380
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Fri, Oct 22, 2021 at 08:49:03AM +0100, Jean-Philippe Brucker wrote:
> On Thu, Oct 21, 2021 at 08:22:23PM -0300, Jason Gunthorpe wrote:
> > On Thu, Oct 21, 2021 at 03:58:02PM +0100, Jean-Philippe Brucker wrote:
> > > On Thu, Oct 21, 2021 at 02:26:00AM +0000, Tian, Kevin wrote:
> > > > > I'll leave it to Jean to confirm. If only coherent DMA can be used in
> > > > > the guest on other platforms, suppose VFIO should not blindly set
> > > > > IOMMU_CACHE and in concept it should deny assigning a non-coherent
> > > > > device since no co-ordination with guest exists today.
> > > > 
> > > > Jean, what's your opinion?
> > > 
> > > Yes a sanity check to prevent assigning non-coherent devices would be
> > > good, though I'm not particularly worried about non-coherent devices. PCIe
> > > on Arm should be coherent (according to the Base System Architecture). So
> > > vfio-pci devices should be coherent, but vfio-platform and mdev are
> > > case-by-case (hopefully all coherent since it concerns newer platforms).
> > > 
> > > More worrying, I thought we disabled No-Snoop for VFIO but I was wrong,
> > > it's left enabled. On Arm I don't think userspace can perform the right
> > > cache maintenance operations to maintain coherency with a device that
> > > issues No-Snoop writes. Userspace can issue clean+invalidate but not
> > > invalidate alone, so there is no equivalent to
> > > arch_sync_dma_for_cpu().
> > 
> > So what happens in a VM? Does a VM know that arch_sync_dma_for_cpu()
> > is not available?
> 
> This would only affect userspace drivers, it's only host or guest
> userspace that cannot issue the maintenance operations. The VM can do
> arch_sync_dma_for_cpu()

This seems out of sync with what the KVM people asked for - any
operation a VM can do should be doable inside a normal process as
well?

Jason