Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35;
From:   "Zhang, Tina" <tina.zhang@intel.com>
To:     Jason Gunthorpe <jgg@ziepe.ca>
CC:     "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "David Woodhouse" <dwmw2@infradead.org>,
        Lu Baolu <baolu.lu@linux.intel.com>,
        "Joerg Roedel" <joro@8bytes.org>, Will Deacon <will@kernel.org>,
        Robin Murphy <robin.murphy@arm.com>,
        "Tian, Kevin" <kevin.tian@intel.com>,
        Nicolin Chen <nicolinc@nvidia.com>,
        Michael Shavit <mshavit@google.com>,
        Vasant Hegde <vasant.hegde@amd.com>
Subject: RE: [PATCH v8 0/5] Share sva domains with all devices bound to a mm
Thread-Topic: [PATCH v8 0/5] Share sva domains with all devices bound to a mm
Thread-Index: AQHaAJO2VqePiK4FsEOuJbSheMRPMLBOMH8AgADH82A=
Date:   Wed, 18 Oct 2023 04:43:41 +0000
Message-ID: <MW5PR11MB5881EF500A27E87D58E5111E89D5A@MW5PR11MB5881.namprd11.prod.outlook.com>
References: <20231017004802.109618-1-tina.zhang@intel.com>
 <20231017164150.GF282036@ziepe.ca>
In-Reply-To: <20231017164150.GF282036@ziepe.ca>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?LL6+itLhAZorMYxpns4D0BiKgIszhtmGtdYkrwAGLRcz6kT5iJRicZ9kPcL8?=
 =?us-ascii?Q?zN73GnZn86oy5a7Jq+ULMZ6AEraUCX475sJJytWZDbWDxNmtdfebAth8pKCK?=
 =?us-ascii?Q?c2ZwyzFKQSlB6g2IzXaY+YwNpX1uDRhKLec4vBsH8mEJVSu94ABpk0pdKM4v?=
 =?us-ascii?Q?X+n5vU3wYHnps79iDaQylWFKg8ZuSiY5dwZtfqCwaApFF65Z+tVBxG/m1gzO?=
 =?us-ascii?Q?OSozoW9OnDEEldVYKW/ydkIG7fKgwd3CDhkff45gGto1YsuzxRUudAPNOEo6?=
 =?us-ascii?Q?QMccLe8W6ccMkpVPjF4MuInfmK+KNLhh0X0tb6+QXLTQARl6Pso9pJWfzJSR?=
 =?us-ascii?Q?3DBrwruwb0/cu0J9HNz6ne4J6RS7LpYEwi8DFVjFUaU/s+N6TrH89dHm5/Vr?=
 =?us-ascii?Q?81ryS+MQ3GeC4LJ42WSAYaalGZCaHjycaT6AjiuiTS8+BhJu/CXTm1ZQxLic?=
 =?us-ascii?Q?zMxgtRfqpeMyVaQUtdpmi2sUe/8y19MPPIivPGWlqF6uk6K6R6jaTYbbkeUF?=
 =?us-ascii?Q?Ngg6/TynvjTw57M6ZEovJzuQVkYnH7hWc5mXlFCiSPwiIuVoEH1lFjRUVU0l?=
 =?us-ascii?Q?WDqekozPopr0MCi5KIJrhiTpF61gy0VE22wnvZiAsBAlMii9uAFt16gqMg7L?=
 =?us-ascii?Q?7qccp5p0I72gbnGAB/DeIIvHUp6J6n4T7niVn5iOTXvHWpwXuY0aERUiI7Z2?=
 =?us-ascii?Q?cf+KStUMy8jqc0Ce89a/gcBlpNcblCC0SqiM4KqsGbSXfyBKeTHqGu2J0Cxr?=
 =?us-ascii?Q?86fPmYMAZ9FoyAxv60Tlg5mFiOKS1xS/qZlj+6j+qq0xaSnxTOBoYf1ny9GL?=
 =?us-ascii?Q?Im6SxT+1r0zXYPoAHrI0Oz8tP41d/FitHHJLspUAQdA9YPbTT+vEARqWpGUm?=
 =?us-ascii?Q?2X5LaP+yYEnP2cZjQNFfsNF04SWnm2ma1bEhp1dPXzQfxBMk2VJuUiZNrENR?=
 =?us-ascii?Q?uRVpgmXbSJyhRfm2p3Anwcw58zyYp+Vyaj1bbK69CFrqZvNLYif862bKmv4P?=
 =?us-ascii?Q?Bs/rmU4d+e4TEum5TEQs16X3R9mbYV+5OBMTqurJTLDGMPNnM1bn/Vwb4klK?=
 =?us-ascii?Q?Y3MitONxD9cJne3jg2BH+8J/kU/empL661yED3occwiPZkzcOgUgNj4t/usx?=
 =?us-ascii?Q?aZzFutBPaqa3q58Eh7Td925BPdZnTlFpd0qNoTLPz/VfV+9gWtaf2OH/ZZuN?=
 =?us-ascii?Q?Cs9c7ChGB4ZLPKCLoQ09X5eIcuf23t1lF0AtIL78RgEmssZWYeQXLzYVPDNB?=
 =?us-ascii?Q?zXfe/eno4gPmJwKJmsRHIf3HFJE4yUBayTSecjradtu3RlwfPsME4vLEtPUI?=
 =?us-ascii?Q?JosxIZ7B5mA4gCLnz4Y4EHzLwq6fROQC+0dkDNBzEypZx5fy9MvGBw4yRTUJ?=
 =?us-ascii?Q?9G44wga0mfpn/LtbkgyBB4qHt4S62UhdTIqn6M7vNdKxClepwv1euX5o5wiq?=
 =?us-ascii?Q?kuPV8RlBISydQD0lHvx04Aj8Tljxez6deKgipiDg3V9U0dN+Vk+EZ25tYdUJ?=
 =?us-ascii?Q?BxS79l7R1lb4ewvGrPDCOnJPYGKyVRKZ4PgR+7j/6iU0rhWyCNnBZ0Kq8Ozh?=
 =?us-ascii?Q?P2nyi1Xc8ToY1qaAhjzpNueeBgdViORTezCR/x4a?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: MW5PR11MB5881.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 78e82aff-d5db-4561-d064-08dbcf94cd7b
X-MS-Exchange-CrossTenant-originalarrivaltime: 18 Oct 2023 04:43:41.2294
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: 5PXWmbqElYBJc6GAUZoP6riZmWfeorTyEY85+m406NMFHgh3tYYqeKOBP1rnLg4c0/lQYthPLdSIDQXxzzVxtQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR11MB5492
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Tue, 17 Oct 2023 21:52:04 -0700 (PDT)

Hi,

> -----Original Message-----
> From: Jason Gunthorpe <jgg@ziepe.ca>
> Sent: Wednesday, October 18, 2023 12:42 AM
> To: Zhang, Tina <tina.zhang@intel.com>
> Cc: iommu@lists.linux.dev; linux-kernel@vger.kernel.org; David Woodhouse
> <dwmw2@infradead.org>; Lu Baolu <baolu.lu@linux.intel.com>; Joerg
> Roedel <joro@8bytes.org>; Will Deacon <will@kernel.org>; Robin Murphy
> <robin.murphy@arm.com>; Tian, Kevin <kevin.tian@intel.com>; Nicolin Chen
> <nicolinc@nvidia.com>; Michael Shavit <mshavit@google.com>; Vasant
> Hegde <vasant.hegde@amd.com>
> Subject: Re: [PATCH v8 0/5] Share sva domains with all devices bound to a
> mm
>=20
> On Tue, Oct 17, 2023 at 08:47:57AM +0800, Tina Zhang wrote:
> > This series is to share sva(shared virtual addressing) domains with
> > all devices bound to one mm.
> >
> > Problem
> > -------
> > In the current iommu core code, sva domain is allocated per IOMMU
> > group, when device driver is binding a process address space to a
> > device (which is handled in iommu_sva_bind_device()). If one than more
> > device is bound to the same process address space, there must be more
> > than one sva domain instance, with each device having one. In other
> > words, the sva domain doesn't share between those devices bound to the
> > same process address space, and that leads to two problems:
> > 1) device driver has to duplicate sva domains with enqcmd, as those
> > sva domains have the same PASID and are relevant to one virtual address
> space.
> > This makes the sva domain handling complex in device drivers.
> > 2) IOMMU driver cannot get sufficient info of the IOMMUs that have
> > devices behind them bound to the same virtual address space, when
> > handling mmu_notifier_ops callbacks. As a result, IOMMU IOTLB
> > invalidation is performed per device instead of per IOMMU, and that
> > may lead to superfluous IOTLB invalidation issue, especially in a
> > virtualization environment where all devices may be behind one virtual
> IOMMU.
> >
> > Solution
> > --------
> > This patch-set tries to fix those two problems by allowing sharing sva
> > domains with all devices bound to a mm. To achieve this, a new
> > structure pointer is introduced to mm to replace the old PASID field,
> > which can keep the info of PASID as well as the corresponding shared sv=
a
> domains.
> > Besides, function iommu_sva_bind_device() is updated to ensure a new
> > sva domain can only be allocated when the old ones cannot work for the
> IOMMU.
> > With these changes, a device driver can expect one sva domain could
> > work for per PASID instance(e.g., enqcmd PASID instance), and
> > therefore may get rid of handling sva domain duplication. Besides,
> > IOMMU driver (e.g., intel vt-d driver) can get sufficient info (e.g.,
> > the info of the IOMMUs having their devices bound to one virtual
> > address space) when handling mmu_notifier_ops callbacks, to remove the
> redundant IOTLB invalidations.
> >
> > Arguably there shouldn't be more than one sva_domain with the same
> > PASID, and in any sane configuration there should be only 1 type of
> > IOMMU driver that needs only 1 SVA domain. However, in reality, IOMMUs
> > on one platform may not be identical to each other. Thus, attaching a
> > sva domain that has been successfully bound to device A behind a IOMMU
> > A, to device B behind IOMMU B may get failed due to the difference
> > between IOMMU A and IOMMU B. In this case, a new sva domain with the
> > same PASID needs to be allocated to work with IOMMU B. That's why we
> > need a list to keep sva domains of one PASID. For the platform where
> > IOMMUs are compatible to each other, there should be one sva domain in
> the list.
> >
> > v8:
> >  - CC more people
> >  - CC iommu@lists.linux.dev mailing list.
> >    When sending version 7, some issue happened in my CC list and that
> caused
> >    version 7 wasn't sent to iommu@lists.linux.dev.
> >  - Rebase to v6.6-rc6 and make a few format changes.
>=20
> You should based it on Joerg's tree so he can take it without conflcits.
>=20
> The conflicts are trivial though (Take Michael's version and switch
> mm->pasid with mm_get_enqcmd_pasid(mm))
>=20
> It looks fine, please lets get it in this cycle, the ARM and AMD SVA seri=
es
> depend on it.
The V9 will be based on the next branch of Joerg's tree.

Like Baolu mentioned, besides ARM and AMD SVA series, we also have a VT-d s=
eries waiting for it.

Regards,
-Tina
>=20
> Jason