Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Thu, 14 Oct 2021 12:42:59 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     "Tian, Kevin" <kevin.tian@intel.com>
Cc:     Alex Williamson <alex.williamson@redhat.com>,
        "Liu, Yi L" <yi.l.liu@intel.com>, "hch@lst.de" <hch@lst.de>,
        "jasowang@redhat.com" <jasowang@redhat.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "jean-philippe@linaro.org" <jean-philippe@linaro.org>,
        "parav@mellanox.com" <parav@mellanox.com>,
        "lkml@metux.net" <lkml@metux.net>,
        "pbonzini@redhat.com" <pbonzini@redhat.com>,
        "lushenming@huawei.com" <lushenming@huawei.com>,
        "eric.auger@redhat.com" <eric.auger@redhat.com>,
        "corbet@lwn.net" <corbet@lwn.net>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "yi.l.liu@linux.intel.com" <yi.l.liu@linux.intel.com>,
        "Tian, Jun J" <jun.j.tian@intel.com>, "Wu, Hao" <hao.wu@intel.com>,
        "Jiang, Dave" <dave.jiang@intel.com>,
        "jacob.jun.pan@linux.intel.com" <jacob.jun.pan@linux.intel.com>,
        "kwankhede@nvidia.com" <kwankhede@nvidia.com>,
        "robin.murphy@arm.com" <robin.murphy@arm.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "iommu@lists.linux-foundation.org" <iommu@lists.linux-foundation.org>,
        "dwmw2@infradead.org" <dwmw2@infradead.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
        "david@gibson.dropbear.id.au" <david@gibson.dropbear.id.au>,
        "nicolinc@nvidia.com" <nicolinc@nvidia.com>
Subject: Re: [RFC 10/20] iommu/iommufd: Add IOMMU_DEVICE_GET_INFO
Message-ID: <20211014154259.GT2744544@nvidia.com>
References: <20210919063848.1476776-1-yi.l.liu@intel.com>
 <20210919063848.1476776-11-yi.l.liu@intel.com>
 <20210922152407.1bfa6ff7.alex.williamson@redhat.com>
 <20210922234954.GB964074@nvidia.com>
 <BN9PR11MB5433409DF766AAEF1BB2CF258CA39@BN9PR11MB5433.namprd11.prod.outlook.com>
 <BN9PR11MB54333BDB1E58387FD9999DF18CA39@BN9PR11MB5433.namprd11.prod.outlook.com>
 <20210923114219.GG964074@nvidia.com>
 <BN9PR11MB5433519229319BA951CA97638CAA9@BN9PR11MB5433.namprd11.prod.outlook.com>
 <20210930222355.GH964074@nvidia.com>
 <BN9PR11MB5433530032DC8400B71FCB788CB89@BN9PR11MB5433.namprd11.prod.outlook.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <BN9PR11MB5433530032DC8400B71FCB788CB89@BN9PR11MB5433.namprd11.prod.outlook.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?4jijM/pUAM3BOr10awphRrYFl//SZ1cvMRlMy1IQjRnd1Tez3p5Jg2eemWnc?=
 =?us-ascii?Q?fyZwI6l7tIPIoTDUUF4lEk7MqkK9y1DSBkVs0k2OgIZg//TPAcPuNoXGRYkx?=
 =?us-ascii?Q?p3OALXZk7r4ukDwOSIK4oxdtLM/OB459WinsGCpCmBRVxaGi7zSJvq+v6exm?=
 =?us-ascii?Q?dIsHt9PbPFHgMhsqjJlkdwZehttFXnKi+UaISvu59ApxJ9jAaYJX5mPex94l?=
 =?us-ascii?Q?zvtg0OxTuf0jmxCPSvZYCIhCAtF53yZ96BR2QGK4gp2BCVFcSkUK5Jc6wHDn?=
 =?us-ascii?Q?l9gcqmLYEtzMmobhDWiad8e19FdhSSHn1UkIXMDdAaC6MCZGQKvhRmKIv8ZJ?=
 =?us-ascii?Q?uyKlOGc3kTR5TsglTNWq7tIX4Q+G6KhQQd7s6I+QiesyQiMLs0L8TFWaURUG?=
 =?us-ascii?Q?4vJ6nsGcF3jThQQlP2tPRKNDsL0MBlbphqSVmUJ9LKGVMMxZHdcf/T6yS7zh?=
 =?us-ascii?Q?+mDCHUXnk0+i4ubLpPCmEvIMzc8bDMqyd4JkhKL2nAeAp9cLLUv43hbLGz7l?=
 =?us-ascii?Q?d0FIm9wZ6MMiBovUi+9cFbxlnjZeYaD6ujCgwkklWhpqqxDTOLYUZTHGFaaV?=
 =?us-ascii?Q?WMUHIGjQbEMrl8b9QvSSQKei4jA6YoHKaPpAeL+x+zdufC3dp2H/thPRS1Rn?=
 =?us-ascii?Q?L0F3zEgk4/evcgNrjI/tt6Nf8z09USKJorjwwzOtqRznPLRxnB5O0+fQfOdd?=
 =?us-ascii?Q?MrQkVQ/a9RMC3S5ru4jxq7xzPtT9IP30mCdde05NgBmSdjbSykZjlwQ2ZiTH?=
 =?us-ascii?Q?i68Jrr8roIc63EPR9SeKTvjfhzkLirR28D02R4UcaubAu9NfMV65CPcFSmqz?=
 =?us-ascii?Q?2giyKdDX/uPkyJGf4+yBHxcI/ZOBviYQA2U6erpeG+cgzvGCnp6QsZqt8kBt?=
 =?us-ascii?Q?RuIP3fP7Zc36SYo6dhfHzuyCib6L89Q5XEHwGJvvK7R53eU+3LBZ4O0xyLpS?=
 =?us-ascii?Q?jpzZbJBffE+xzIVdbFx7Eqcm8C4+8gzwQW7DGdthvdUJz+sajI9KRs7ahL2l?=
 =?us-ascii?Q?Zdz9NLnX7IcgKP2hW4s8+VQhcb4GSKqN7gtaeAjBuNLUUa6BtG1w5WiBmVuS?=
 =?us-ascii?Q?nIkKN9aS86kZovCLfeERPecDrrR9lMavfCK6+cn9G8nPZPEr1ZCb2MuRYQtH?=
 =?us-ascii?Q?gBsvy6z0snTNElraUg+5g2QiszJgS9nxFVnrbJma5SVn5nGDwd3syfh4NFC9?=
 =?us-ascii?Q?5vnVGL1hJsLY96agJyJZsq9ox5fwYOQ46oA2OG0mJ4XYud5N+ujC+Is30UsK?=
 =?us-ascii?Q?WO/nqylLUKz6/3+HCI5TwoCVedv9G8UgoGeNSa7fJ9cxGL9BWiYt8kNHsKik?=
 =?us-ascii?Q?3SvS7f0whYqmd/wEpKIj7r+zsOlibz7q15Q86X7pRH4bLA=3D=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 07271f29-8153-46c9-0d97-08d98f294da0
X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB5506.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Oct 2021 15:43:01.1030
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: i+vz/8pxeib3hRjb48UJHQM8Jie9CXu6kunKCCnXaeEhU5VoDWFt+aFhUADYdGYS
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5127
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Thu, Oct 14, 2021 at 09:11:58AM +0000, Tian, Kevin wrote:

> But in both cases cache maintenance instructions are available from 
> guest p.o.v and no coherency semantics would be violated.

You've described how Intel's solution papers over the problem.

In part wbinvd is defined to restore CPU cache coherence after a
no-snoop DMA. Having wbinvd NOP breaks this contract.

To counter-act the broken wbinvd the IOMMU completely prevents the use
of no-snoop DMA. It converts them to snoop instead.

The driver thinks it has no-snoop. The platform appears to support
no-snoop. The driver issues wbinvd - but all of it does nothing.

Don't think any of this is even remotely related to what ARM is doing
here. ARM has neither the broken VM cache ops, nor the IOMMU ability
to suppress no-snoop.

> > > I think the key is whether other archs allow driver to decide DMA
> > > coherency and indirectly the underlying I/O page table format.
> > > If yes, then I don't see a reason why such decision should not be
> > > given to userspace for passthrough case.
> > 
> > The choice all comes down to if the other arches have cache
> > maintenance instructions in the VM that *don't work*
>
> Looks vfio always sets IOMMU_CACHE on all platforms as long as
> iommu supports it (true on all platforms except intel iommu which
> is dedicated for GPU):
> 
> vfio_iommu_type1_attach_group()
> {
> 	...
> 	if (iommu_capable(bus, IOMMU_CAP_CACHE_COHERENCY))
> 		domain->prot |= IOMMU_CACHE;
> 	...
> }
> 
> Should above be set according to whether a device is coherent?

For IOMMU_CACHE there are two questions related to the overloaded
meaning:

 - Should VFIO ask the IOMMU to use non-coherent DMA (ARM meaning)
   This depends on how the VFIO user expects to operate the DMA.
   If the VFIO user can issue cache maintenance ops then IOMMU_CACHE
   should be controlled by the user. I have no idea what platforms
   support user space cache maintenance ops.

 - Should VFIO ask the IOMMU to suppress no-snoop (Intel meaning)
   This depends if the VFIO user has access to wbinvd or not.

   wbinvd is a privileged instruction so normally userspace will not
   be able to access it.

   Per Paolo recommendation there should be a uAPI someplace that
   allows userspace to issue wbinvd - basically the suppress no-snoop
   is also user controllable.

The two things are very similar and ultimately are a choice userspace
should be making.

From something like a qemu perspective things are more murkey - eg on
ARM qemu needs to co-ordinate with the guest. Whatever IOMMU_CACHE
mode VFIO is using must match the device coherent flag in the Linux
guest. I'm guessing all Linux guest VMs only use coherent DMA for all
devices today. I don't know if the cache maintaince ops are even
permitted in an ARM VM.

Jason