From:   "Tian, Kevin" <kevin.tian@intel.com>
To:     Jason Gunthorpe <jgg@nvidia.com>,
        Nicolin Chen <nicolinc@nvidia.com>
CC:     Robin Murphy <robin.murphy@arm.com>,
        "will@kernel.org" <will@kernel.org>,
        "eric.auger@redhat.com" <eric.auger@redhat.com>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "shameerali.kolothum.thodi@huawei.com" 
        <shameerali.kolothum.thodi@huawei.com>,
        "jean-philippe@linaro.org" <jean-philippe@linaro.org>,
        "linux-arm-kernel@lists.infradead.org" 
        <linux-arm-kernel@lists.infradead.org>,
        "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: RE: [PATCH v1 14/14] iommu/arm-smmu-v3: Add
 arm_smmu_cache_invalidate_user
Thread-Topic: [PATCH v1 14/14] iommu/arm-smmu-v3: Add
 arm_smmu_cache_invalidate_user
Thread-Index: AQHZUnWe1bQRwtfhAUSI/T7BqHBa167yh7YAgAALsACAANbagIAAyRsAgAqMnjA=
Date:   Fri, 17 Mar 2023 09:41:34 +0000
Message-ID: <BN9PR11MB52768F4D3E21C5231C1A04D38CBD9@BN9PR11MB5276.namprd11.prod.outlook.com>
References: <cover.1678348754.git.nicolinc@nvidia.com>
 <aa327f9ea61e5a4771c13e53639e33955b9acde3.1678348754.git.nicolinc@nvidia.com>
 <1467e666-1b6c-c285-3f79-f8e8b088718b@arm.com> <ZAn7uC9UweiNdGkJ@nvidia.com>
 <ZAqv87fjbdynVaHA@Asurada-Nvidia> <ZAtYphmOuEqQ1BiC@nvidia.com>
In-Reply-To: <ZAtYphmOuEqQ1BiC@nvidia.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?h/rNhaRr/8bXCuivEIH6xO12rqRIU97Wr7wczel0jaK6cKN0G7mf3UlwKw6A?=
 =?us-ascii?Q?ZLnIsMZipnvefhvFunrGSy+7V+kBfa5BbYIadGwHWycj3iWr+068+p8DETsu?=
 =?us-ascii?Q?Q/um+TjrBLLUQ2xocCUzRbHDsh1cM2mP8RZaUJTv2xmdIIugn9unw5PpqHxd?=
 =?us-ascii?Q?vUyOJpCIngPOZOy65rT6hgRg/E2n9q+WfNriWIMxmzCRTaylAg0LE3ZnQh/X?=
 =?us-ascii?Q?KXjD3O/E5JeH/ZVE9cp9gdlQpRWOlAQiMrE+pQzjLrWuKlSh3Q14SS3/s9A3?=
 =?us-ascii?Q?/GUapRE3G7u/8zrgfzFj6Nco2TX3lgMv2FjUNKZz47kY2YirJuOKOdSCIMfj?=
 =?us-ascii?Q?M5H5R8IOdLpM9hGIlSGJBYwD7mAWOLZucR+O2tk827tRE6vzbmUwYkuiI6Ci?=
 =?us-ascii?Q?z7th2fj4ASBpIedplenymg0J62UYQCmT6l0L6QHJQFUGD5sxl8EjuSYOx79F?=
 =?us-ascii?Q?+kq0d8DSfzdHODhjm9o4kvEB0AaZgImTQg9Cji8u0DSl0y9IF6ZLJicPABSn?=
 =?us-ascii?Q?afMmjeqhp6BmoRr5++MGZ2Hnj7dOriqNW3s20B0Fo+b3wmGUIPAJ1iZ0GD9f?=
 =?us-ascii?Q?eJIhkMJ4RBinCOIKwmziVdgh1t1ahnDZu0TQA2RhslLkLKJudzqwAAq6ziax?=
 =?us-ascii?Q?Ul7SbZtf7N/o1DDezX/70Fp3mvol7q+zIdlOPWIkKwOEsRmCd1wDgy5mE/r9?=
 =?us-ascii?Q?4nTa/Wqs78t+VECXEVNbP7GoyMdNmi0533dBQgEq4/VpXqvZAkTQVv3sXmAR?=
 =?us-ascii?Q?vnW10VGIvRs4sQ5PwJL5sVwbpf1CpP4D051vhGTLgA93333rjmOGiipUcJSw?=
 =?us-ascii?Q?hxBWbhx3NVAuMGz1Ze+FkwB1dUprbR5H9fm5V1mNGzuWww1juW6lKIw+fAMq?=
 =?us-ascii?Q?lUrJF+3+5SfMxjJXF21D7RFjdQz69orAIbb8XlZlZmVLAeiyUj8uwfkqf5Pd?=
 =?us-ascii?Q?UhBpDF65VFd1HG2sUY/7jBDziNRnx9JmxZlLbkiXjrpmhLT+UKBlm+z1cuvH?=
 =?us-ascii?Q?oILdt+ahtOb9xxrnFIMnVwG6UcWPFQ/VCK7VKZg1iVgCUbsJO8ZRusLF+rHh?=
 =?us-ascii?Q?sYxGF2cPLHv9TCzT+cjBJaO/Y3N/MeYy4zctzJaLvDOcS10dNS1IjHfzl+2C?=
 =?us-ascii?Q?MbMqjZbpFrCIN1Iehdcak3eAMeamDX1FeI5XrxH7OagczETKQT/+n7CAhGds?=
 =?us-ascii?Q?bJZxenR78G0bxn9bTAJJWV0oZxp8tCs4SonrKWQn+EZp+bWJBtVqLculBfcI?=
 =?us-ascii?Q?7UQxzr2sHHQVpG1jiDvtnO/gCzXim5FIEVYe0+cGdLKQcwlu1Y60fSXsrc3x?=
 =?us-ascii?Q?3Ds4shw+hcTTveFeuEoQpJ3E3jCXuoooShC1yy6pUL+RMV5f+0/P49Z9B+mt?=
 =?us-ascii?Q?NNFsozeQgA6FzMUIMQ1BBb/Obgx/N0dxrRsGqFuN7BT9UPUN6P3AonX6Ec6D?=
 =?us-ascii?Q?oBKSjLt7GQdVY9NzTlUSPhQQzM+Y5diMeJlZlrqdw0UIXtd3UJ6SIQ3g1Wh+?=
 =?us-ascii?Q?KelnMHWC2s5GRUd9tL8bMc1ujw/K5+3OzfzvyMyuaD5h+pngpVD2vwptZ+Eo?=
 =?us-ascii?Q?PSgGJd5MQpJ3JRMd5txvsvQldsnFef5h1SQbhpEm?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5276.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: f9ae8212-dce4-458e-3064-08db26cbcbe0
X-MS-Exchange-CrossTenant-originalarrivaltime: 17 Mar 2023 09:41:34.3116
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: qVSUxRBwRhnu0xE2qH2NiipF+uV2jdsMmrUqkmlTAA/eH/aJ8TkZOdyvTQYL4RkGkSDg7WkZHeldiQkPhrfdNg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA3PR11MB7583
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Saturday, March 11, 2023 12:20 AM
>=20
> What I'm broadly thinking is if we have to make the infrastructure for
> VCMDQ HW accelerated invalidation then it is not a big step to also
> have the kernel SW path use the same infrastructure just with a CPU
> wake up instead of a MMIO poke.
>=20
> Ie we have a SW version of VCMDQ to speed up SMMUv3 cases without HW
> support.
>=20

I thought about this in VT-d context. Looks there are some difficulties.

The most prominent one is that head/tail of the VT-d invalidation queue
are in MMIO registers. Handling it in kernel iommu driver suggests
reading virtual tail register and updating virtual head register. Kind of=20
moving some vIOMMU awareness into the kernel which, iirc, is not
a welcomed model.

vhost doesn't have this problem as its vring structure fully resides in
memory including ring tail/head. As long as kernel vhost driver understands
the structure and can send/receive notification to/from kvm then the
in-kernel acceleration works seamlessly.

Not sure whether SMMU has similar obstacle as VT-d. But this is my
impression why vhost-iommu is preferred when talking about such
optimization before.