From:   "Tian, Kevin" <kevin.tian@intel.com>
To:     Jason Gunthorpe <jgg@nvidia.com>,
        Nicolin Chen <nicolinc@nvidia.com>
CC:     Robin Murphy <robin.murphy@arm.com>,
        "will@kernel.org" <will@kernel.org>,
        "eric.auger@redhat.com" <eric.auger@redhat.com>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "shameerali.kolothum.thodi@huawei.com" 
        <shameerali.kolothum.thodi@huawei.com>,
        "jean-philippe@linaro.org" <jean-philippe@linaro.org>,
        "linux-arm-kernel@lists.infradead.org" 
        <linux-arm-kernel@lists.infradead.org>,
        "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: RE: [PATCH v1 14/14] iommu/arm-smmu-v3: Add
 arm_smmu_cache_invalidate_user
Thread-Topic: [PATCH v1 14/14] iommu/arm-smmu-v3: Add
 arm_smmu_cache_invalidate_user
Thread-Index: AQHZUnWe1bQRwtfhAUSI/T7BqHBa167yh7YAgAALsACAANbagIAAyRsAgAqMnjCABPKxgIAANdgAgAAeawCAAPG3IA==
Date:   Tue, 21 Mar 2023 08:34:00 +0000
Message-ID: <BN9PR11MB5276E6E3FCA90582AA61BDDE8C819@BN9PR11MB5276.namprd11.prod.outlook.com>
References: <cover.1678348754.git.nicolinc@nvidia.com>
 <aa327f9ea61e5a4771c13e53639e33955b9acde3.1678348754.git.nicolinc@nvidia.com>
 <1467e666-1b6c-c285-3f79-f8e8b088718b@arm.com> <ZAn7uC9UweiNdGkJ@nvidia.com>
 <ZAqv87fjbdynVaHA@Asurada-Nvidia> <ZAtYphmOuEqQ1BiC@nvidia.com>
 <BN9PR11MB52768F4D3E21C5231C1A04D38CBD9@BN9PR11MB5276.namprd11.prod.outlook.com>
 <ZBhYq8o4gqFZVb6k@nvidia.com> <ZBiF1hZVrp/aJRM6@Asurada-Nvidia>
 <ZBifWkY2Lt/U1Z7R@nvidia.com>
In-Reply-To: <ZBifWkY2Lt/U1Z7R@nvidia.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?U+bR+4Z8jCyEahTHG9WrG5aE9MZzh+T/zEVK1c60lil1NMG9Tf9ySp4VohoN?=
 =?us-ascii?Q?+IapmzGGbKdj9FJ6Hq/Q+J68NJObhCNIxOze0b8gb7+3wR7q0IpbKRlels06?=
 =?us-ascii?Q?JMlbgXJFV/sfg/XlNmXO6SyLMhdmBV8Pdn5zRy8yJX3f/wP3+kHhpjs2jvbP?=
 =?us-ascii?Q?oyc8NzwZAmAyEo1YSm4/6r9X2+fdYXuzka+3Y4BgNcc/uvvkMB8D3PbN02q3?=
 =?us-ascii?Q?ZTwD/XpMSSwiwNgidueNLfdGqXpaA3vSmFgKELUPrgll10YnbUqaeq8KFPZo?=
 =?us-ascii?Q?k2HuPm2ntA5/Zux8+kVj6Q5cf3G2oaSfndSkDdiq+x8fXSkI3+GKUJi2j48j?=
 =?us-ascii?Q?0VlB04Zbr2ITIt7TLKNN8HTxaun/s+V21fGPnBhgPnc5butCT0xWOBTeR9/G?=
 =?us-ascii?Q?cH85dVH5W+xDwVbkDX+IKm+Wsk2/sYsdFmNZnD4hX7C1GZzn0438CeRnPP+P?=
 =?us-ascii?Q?xin9U+yVCTKjzTJqbmNe8Eb0S9jAlJkG/BoC18bkjpLZzXwHBO2/X0m14GRY?=
 =?us-ascii?Q?7LN/6pqo6saM58JGXmMH0jo45K41y6E6CvfFMgRgMyFQ4KWDn8mZmflRFI0r?=
 =?us-ascii?Q?aSata/L5Uku6kVpnJ0wvqaG0fU/Tv/s52h5lgYdF4CQyC6ylv37PpNEB/8ny?=
 =?us-ascii?Q?y4fRRrV67qVJUn7IASqwuVN4Vc1lUU4VFM+W5eWUEhyC1MRuO8khwOejlAiY?=
 =?us-ascii?Q?4p3A5LV3r3X/KFTTQ1jl0BA3pXFNbrd/dG2HszCSxcckL7mMX9D00EH4ASJn?=
 =?us-ascii?Q?R+E6NJARggAGa+zPYaTaF2RwuLNR77/6uIDjNw6NRkKx5ocyOAJ0xo5+aYUj?=
 =?us-ascii?Q?tb3GgFci/cVHPPyNmNSQVIGgUYpJnQ2VYGROlq1GusvxQk749ocgUMGf1LyY?=
 =?us-ascii?Q?MDPkxI+kTFhIvK0xqSJgnsh62B2UPXAqJDSErQlikZwqtTCrtY7c7N9MX23Q?=
 =?us-ascii?Q?W4bn9f8HDdUY6bI/0rZPogZDfrdsHPhXq7339zMKFwOcKKOOc867G/UxSsv9?=
 =?us-ascii?Q?Joe1Nle6TX4kGwqFFNwtoabPwIloVzjP2Hz1GzEfwzp4Mza8xaWniEbCcyZ7?=
 =?us-ascii?Q?PKftNGsbYddLE32OZcPtHA1YidOWXlUdYeQblZEo9spwy1ScrTKWhpTfgOow?=
 =?us-ascii?Q?d1vJIsZLj6Bcwgf7SJ+jaMheRyHiOgySHF/t6qdsYuwLc8GR2mpvluE224aM?=
 =?us-ascii?Q?WssuaogUfjL9tBRyqRLuVpChjmQ28gKHS7qdRjARR0BBEaym7J5CR0sIk8Cg?=
 =?us-ascii?Q?t6EjBDyIt4QdV9h7jOr02YkBBuBGuny+ZZfFtEpqw+anLY3sPg0gBZw8tVXy?=
 =?us-ascii?Q?ETQ20icEBivgZgH/1XdZ3a26Njq2+upj9JWF7+NmwCvBFYaUrOdvYlmS+Dwq?=
 =?us-ascii?Q?OkwZtLwUNz2D3rQlS3ORQyF2liHJskhQwKZaQASi2iRy2iIyOkuEGt87Vsta?=
 =?us-ascii?Q?/bByozclDe6sZHTnoAlo/F6xRDlJKt2n7DUeUSQb88/HBJ46e9OMiQVReUgC?=
 =?us-ascii?Q?RdKMcqihwW+pH7xxy5nBu89+XFRi9ritRI5QHXjnwJ8CQKkWc0Vch143Bdeh?=
 =?us-ascii?Q?jVrkK9UcWPY7Wfri5zssljgC0UTQrOddzjWGoFsQ?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5276.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 383ceaa4-5346-4b6e-c19e-08db29e70546
X-MS-Exchange-CrossTenant-originalarrivaltime: 21 Mar 2023 08:34:00.2327
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: usfMItCRS+M7NItF/J3GQl3UbMdTWryY3/pTGo7E7oi+SAQq5kYvc8ZZPLQC3TUz3XCPvQN0apWP1249IXbPEA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR11MB7318
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Tuesday, March 21, 2023 2:01 AM
>=20
> On Mon, Mar 20, 2023 at 09:12:06AM -0700, Nicolin Chen wrote:
> > On Mon, Mar 20, 2023 at 09:59:23AM -0300, Jason Gunthorpe wrote:
> > > On Fri, Mar 17, 2023 at 09:41:34AM +0000, Tian, Kevin wrote:
> > > > > From: Jason Gunthorpe <jgg@nvidia.com>
> > > > > Sent: Saturday, March 11, 2023 12:20 AM
> > > > >
> > > > > What I'm broadly thinking is if we have to make the infrastructur=
e for
> > > > > VCMDQ HW accelerated invalidation then it is not a big step to al=
so
> > > > > have the kernel SW path use the same infrastructure just with a C=
PU
> > > > > wake up instead of a MMIO poke.
> > > > >
> > > > > Ie we have a SW version of VCMDQ to speed up SMMUv3 cases
> without HW
> > > > > support.
> > > > >
> > > >
> > > > I thought about this in VT-d context. Looks there are some difficul=
ties.
> > > >
> > > > The most prominent one is that head/tail of the VT-d invalidation
> queue
> > > > are in MMIO registers. Handling it in kernel iommu driver suggests
> > > > reading virtual tail register and updating virtual head register. K=
ind of
> > > > moving some vIOMMU awareness into the kernel which, iirc, is not
> > > > a welcomed model.
> > >
> > > qemu would trap the MMIO and generate an IOCTL with the written head
> > > pointer. It isn't as efficient as having the kernel do the trap, but
> > > does give batching.
> >
> > Rephrasing that to put into a design: the IOCTL would pass a
> > user pointer to the queue, the size of the queue, then a head
> > pointer and a tail pointer? Then the kernel reads out all the
> > commands between the head and the tail and handles all those
> > invalidation commands only?
>=20
> Yes, that is one possible design
>=20

If we cannot have the short path in the kernel then I'm not sure the
value of using native format and queue in the uAPI. Batching can
be enabled over any format.

Btw probably a dumb question. The current invalidation IOCTL is
per hwpt. If picking a native format does it suggest making the IOCTL
per iommufd given native format is per IOMMU and could carry
scope bigger than a hwpt.