Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   "Tian, Kevin" <kevin.tian@intel.com>
To:     Jason Gunthorpe <jgg@nvidia.com>,
        Alex Williamson <alex.williamson@redhat.com>
CC:     "Liu, Yi L" <yi.l.liu@intel.com>, "hch@lst.de" <hch@lst.de>,
        "jasowang@redhat.com" <jasowang@redhat.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "jean-philippe@linaro.org" <jean-philippe@linaro.org>,
        "parav@mellanox.com" <parav@mellanox.com>,
        "lkml@metux.net" <lkml@metux.net>,
        "pbonzini@redhat.com" <pbonzini@redhat.com>,
        "lushenming@huawei.com" <lushenming@huawei.com>,
        "eric.auger@redhat.com" <eric.auger@redhat.com>,
        "corbet@lwn.net" <corbet@lwn.net>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "yi.l.liu@linux.intel.com" <yi.l.liu@linux.intel.com>,
        "Tian, Jun J" <jun.j.tian@intel.com>, "Wu, Hao" <hao.wu@intel.com>,
        "Jiang, Dave" <dave.jiang@intel.com>,
        "jacob.jun.pan@linux.intel.com" <jacob.jun.pan@linux.intel.com>,
        "kwankhede@nvidia.com" <kwankhede@nvidia.com>,
        "robin.murphy@arm.com" <robin.murphy@arm.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "iommu@lists.linux-foundation.org" <iommu@lists.linux-foundation.org>,
        "dwmw2@infradead.org" <dwmw2@infradead.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
        "david@gibson.dropbear.id.au" <david@gibson.dropbear.id.au>,
        "nicolinc@nvidia.com" <nicolinc@nvidia.com>
Subject: RE: [RFC 10/20] iommu/iommufd: Add IOMMU_DEVICE_GET_INFO
Thread-Topic: [RFC 10/20] iommu/iommufd: Add IOMMU_DEVICE_GET_INFO
Thread-Index: AQHXrSGNbNtRgavabUSKJjvt8l12BauwlhaAgAAouwCAACufAIAAEL4Q
Date:   Thu, 23 Sep 2021 03:38:10 +0000
Message-ID: <BN9PR11MB54333BDB1E58387FD9999DF18CA39@BN9PR11MB5433.namprd11.prod.outlook.com>
References: <20210919063848.1476776-1-yi.l.liu@intel.com>
 <20210919063848.1476776-11-yi.l.liu@intel.com>
 <20210922152407.1bfa6ff7.alex.williamson@redhat.com>
 <20210922234954.GB964074@nvidia.com>
 <BN9PR11MB5433409DF766AAEF1BB2CF258CA39@BN9PR11MB5433.namprd11.prod.outlook.com>
In-Reply-To: <BN9PR11MB5433409DF766AAEF1BB2CF258CA39@BN9PR11MB5433.namprd11.prod.outlook.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?DAInMCGBNFJtdHeEg1w4K7dR3GhXCls5Z7CoOL24h9LaPMXzTklCoKEFmJIM?=
 =?us-ascii?Q?12myTKdSs5K54U3xbX5iMITSPKOmvANfT21/AU9q7znI2TclEVPERMkAp1jo?=
 =?us-ascii?Q?XcIxiVJQrQcmJjaxMg19IxM7h5VxA49B2Ppdshsx3oPc92bU6g9A/WdPOG7e?=
 =?us-ascii?Q?FAMb+Pc0lOhAjZjbheONwRFa0XnLWutoO8ITodEjbJ9xZ5qNvLc42/sKzYuv?=
 =?us-ascii?Q?lPDHX8e5kT3CwfBLJO/ADmV4LD1zTwffY2d8NnpRd8iO+wxLARtMdw6vDpok?=
 =?us-ascii?Q?L8RxEGJdX+2wm6UYw2faiiHVtZ3hLxJNzxl9VsQOqiBwsb6pzQqfS/Urd5oE?=
 =?us-ascii?Q?SiVHLxY19my89z9+d511H+U7FHZJlH3phu3VLSatyLXMQX3zVlVd5vCtLZh2?=
 =?us-ascii?Q?xSB+oEbCLYZCho9hRueuTf85QXTRDKFnHnBFuxCvQYRilJlVBBOSKMegOAg5?=
 =?us-ascii?Q?Dh4DY5W6kiU8ov/9JzUllUdmyCVwvz9i2bLXfaP+zWuGKGbcUQwFOrgD7COE?=
 =?us-ascii?Q?7U0ujvbkwGShrWcdzPtkG/oKVyvw7DokTYn76HS6DbAnFzUs1C0jdMu+hDyg?=
 =?us-ascii?Q?FpeBDBTn0xSiYNOM29H5ec5mhcWrMQXWptD2gAs6I6dO4NMS20POxKp4LDBh?=
 =?us-ascii?Q?ne9kVIzllPkAS4aLk147T+iPz9MsM3ebfBM88yhooGMwkJUQ9EyoQdWK7mEJ?=
 =?us-ascii?Q?/oJExPpyFqp+YN3cCW2M9zu4kc1aL5nKLJUt4MqHmgSR+MyQlJUs2vplOTSM?=
 =?us-ascii?Q?Pus6foBAV8vCNhAYMPnGt4OofBbtZofFvNIJjB0b4BzB7tFQdI2labX65ym5?=
 =?us-ascii?Q?5xbzdiFXnVm+SxRrusIily4o0xAvjR4yS9noSX7/MAaNtdRgOcxSSQGEzSZG?=
 =?us-ascii?Q?W5g5u8n35CoWl6NXP1uikt5PpP7+44M8hBRfOO/9Ch+C2shsv0ZBh3ra5Ksr?=
 =?us-ascii?Q?vS0pjDuKsngPm7KrdL24D2hU2AGVIkohB0IjkJV9GWgIIzcPX/4aIHcU3KRA?=
 =?us-ascii?Q?+dtIHO2BGENS9tDqF/hgPOF03SN5izDVJmp60SUYZpZsZkXrDFtZt1d0wMJB?=
 =?us-ascii?Q?cpGwZOquEM7PAa+TaSoSIX7alctBewpg4+4OSzVJ8SGGnZNtY920n3QUwAyY?=
 =?us-ascii?Q?bx/Sk31+qVf7VJBLt3hU1+PmfZPddPPvZNNprdDd/9b4GjDKHdwf27PzJwVT?=
 =?us-ascii?Q?HPoLVk9gru5nrL+83CbDicotJcZPPrRyN5p8hZSrXcqlZ+zvne2SDWNVgzft?=
 =?us-ascii?Q?8MtQ0CnKz+7iZJ6pQoOxzPXhUdHjIv9S1VKNi7/Uroz3XkiV/9G/cTgFO61L?=
 =?us-ascii?Q?CqBx3L2oTEt4RC91hV2zGScd?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5433.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: e6df1ccb-c876-4197-5f71-08d97e4390a8
X-MS-Exchange-CrossTenant-originalarrivaltime: 23 Sep 2021 03:38:10.2815
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: eJLntXlFQT8QqcEaZrVMzaqohz6fMxvDFokejiBFcoy/iZjz984Tmbt9QTKnY2u1/16Bk+8wmE7ZWsxVAw0b8g==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN9PR11MB5466
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> From: Tian, Kevin
> Sent: Thursday, September 23, 2021 11:11 AM
>=20
> >
> > The required behavior for iommufd is to have the IOMMU ignore the
> > no-snoop bit so that Intel HW can disable wbinvd. This bit should be
> > clearly documented for its exact purpose and if other arches also have
> > instructions that need to be disabled if snoop TLPs are allowed then
> > they can re-use this bit. It appears ARM does not have this issue and
> > does not need the bit.
>=20
> Disabling wbinvd is one purpose. imo the more important intention
> is that iommu vendor uses different PTE formats between snoop and
> !snoop. As long as we want allow userspace to opt in case of isoch
> performance requirement (unlike current vfio which always choose
> snoop format if available), such mechanism is required for all vendors.
>=20

btw I'm not sure whether the wbinvd trick is Intel specific. All other
platforms (amd, arm, s390, etc.) currently always claim OMMU_CAP_
CACHE_COHERENCY (the source of IOMMU_CACHE). They didn't hit
this problem because vfio always sets IOMMU_CACHE to force every
DMA to snoop. Will they need to handle similar wbinvd-like trick (plus
necessary memory type virtualization) when non-snoop format is enabled?=20
Or are their architectures highly optimized to afford isoch traffic even=20
with snoop (then fine to not support user opt-in)?

Thanks
Kevin