Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   "Zhang, Tina" <tina.zhang@intel.com>
To:     Jason Gunthorpe <jgg@nvidia.com>
CC:     Michael Shavit <mshavit@google.com>, Will Deacon <will@kernel.org>,
        "Robin Murphy" <robin.murphy@arm.com>,
        Joerg Roedel <joro@8bytes.org>,
        "jean-philippe@linaro.org" <jean-philippe@linaro.org>,
        "nicolinc@nvidia.com" <nicolinc@nvidia.com>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
        "linux-arm-kernel@lists.infradead.org" 
        <linux-arm-kernel@lists.infradead.org>,
        "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: RE: [PATCH v2 14/18] iommu/arm-smmu-v3: Support domains with shared
 CDs
Thread-Topic: [PATCH v2 14/18] iommu/arm-smmu-v3: Support domains with shared
 CDs
Thread-Index: AQHZmHAAx5RbbLZUakKJUTzXs8dX169+AmQAgAAYP4CAASOlgIAr2NewgAhWl4CAAG5csA==
Date:   Tue, 11 Jul 2023 00:26:56 +0000
Message-ID: <MW5PR11MB58818828160F5AA41756D45A8931A@MW5PR11MB5881.namprd11.prod.outlook.com>
References: <20230606120854.4170244-1-mshavit@google.com>
 <20230606120854.4170244-15-mshavit@google.com> <ZH9oQMmI8CezMCnx@nvidia.com>
 <CAKHBV25eDNXEu+6GWDJqpw1h5N-Jq-GPzf81kNH1dyw3Z=KsZA@mail.gmail.com>
 <ZIBxPd1/JCAle6yP@nvidia.com>
 <MW5PR11MB588179952AEDB50D9259549C892FA@MW5PR11MB5881.namprd11.prod.outlook.com>
 <ZKw38RrJ0FG8zbqt@nvidia.com>
In-Reply-To: <ZKw38RrJ0FG8zbqt@nvidia.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?PAw88NvYfxJaJvVfO49McvF3VhBv8z1df4AHrIpvOnuJZ2Ht33JFDKWmcwRh?=
 =?us-ascii?Q?gg9Vph3HVeIJFR4Ulqz3v3uRyfNU/bGo6m7Rq1P76DVrDE+b72PgYdgYauKJ?=
 =?us-ascii?Q?cPxzWP0TInM4/CQ/fCvU4jabv2tZsCKqwC5OQNaTnC+S3us2OtSqMjIvxr4Q?=
 =?us-ascii?Q?2T8h7zd2PDcyWl/rqj6y5CzQ/mQc3stIZr6jJsf1Lkx7ho+C2zHhfYtLf3W3?=
 =?us-ascii?Q?ONvjj1Cn5siHrijRB46Pn4gvxczFSK21CTNBhHAZjCNjjfVEt/lvzuRh9vdE?=
 =?us-ascii?Q?Wi/CEpSWWqGltzKZik5/6z/+2j+Ua+PROpxxuKPs6yKXAJ7U44jh1T0g0C4h?=
 =?us-ascii?Q?hrhZKd2kaG3iPhwKSawMfQQosvG/SOAHKUCY+pRrX7eGSsSzmItr/S6apY+p?=
 =?us-ascii?Q?RxfjhabixHywfIkb8Aa70bE4z5uMw4AO9y1g0myD/ZXO4BOLptgcGiiUtSm6?=
 =?us-ascii?Q?3OMI+QiA83R9wR3bY7xWcRKhG3EJJ6esoQ4kfiqZbntoujH4xwzWL6/qqvCK?=
 =?us-ascii?Q?EZOVoDdxfnUXE0i1yNKDU/LniI8CCOmEiGmAMOXasAxTcbsYp9w3JvA4e4tg?=
 =?us-ascii?Q?vZN/4r2VU5ynqe4AsXtI7ro8YBwbABlwmT3IJVhWcU9crAXsFON/5paQAr64?=
 =?us-ascii?Q?H903RLmHlowZlCgzedFowjkTDXnJpJvk8/PHVu6ow2llHVSB65MDrUGJoWOr?=
 =?us-ascii?Q?GtUC8TkfDSY+PBsDsAmMFPWrxuskdPeP9sXUXS6taoQKKB3f5jjkY6niGiLN?=
 =?us-ascii?Q?NAP8iZVwu/+8vQHpxALYNCH8ApRii2pl/6NA/2UF1VOsauFEFivOrgTfXAWY?=
 =?us-ascii?Q?1cIm5sQUGz7cuaqFIVXdOFxUKTL1dyxLICGgNpSO5hClrVhyhuxNlh1ORCga?=
 =?us-ascii?Q?s9CAy6zUklTNpg3tNr4hXm7cibwjo3uyhf4TXbCOivF9VsrEQJncPbNXJ9RQ?=
 =?us-ascii?Q?ldosPtdmzzO7+/02udtKZ8M/Rr4Xvvl7UmnOncgjyIfIdpbDL/am3SmsRkyv?=
 =?us-ascii?Q?egUYFjW+VxCcPWaVygO8iW1JDky7msjOXwBpLQxYsx7FVJOIVvUpHMLV/2k6?=
 =?us-ascii?Q?EgmWcXnm+vKPQSO9qpor1cxPTLMf1i4+p3N7FO8yLWk+kbad3tdFM1hhuCsN?=
 =?us-ascii?Q?EPcXQ2XxcZhaUqtrr5hX4J0jUnfyLqwfYUXiYz5jz9zvgfEp1YocTQB+6vDK?=
 =?us-ascii?Q?WxXyiVvlFiCsW5GaOJcLcWWMG3UzTsljVhnSpWiIm0lglixbn99yFjO2iLhs?=
 =?us-ascii?Q?c3LlcbovmlTaWYtaBIWIEuZKY/dJ7bqVn1lrhKdRWaeLifrwlK5CypiD0jf6?=
 =?us-ascii?Q?wIaPjPRctKAfyqswJxQ54QF/0JXrbLU3FHIIlP7SbPX2p+6Yma8rCLo/MG2a?=
 =?us-ascii?Q?2Yxeu58FeAJ0dp5PCldF+kmSCBdp5Ijf/ygQw7Oyz72sPMLcoQ9uSRtImE/x?=
 =?us-ascii?Q?KZK4qbFKcdlDMzGi0hfqrknqwziNxd+baW+Ln4B+wyYQ2C6oP8PPN/PHwr0O?=
 =?us-ascii?Q?2H8JJZKKqkWT1uh5h6dlHO9eg92zzvXNq1hCJjC0+i+VpHy3S9/S9KIlsEX5?=
 =?us-ascii?Q?+fRvtcj/wLRdd2VlwjCQwDOIw0j5R0dOXFcXlzuY?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: MW5PR11MB5881.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 4df0d9bb-7525-452b-2e05-08db81a588ac
X-MS-Exchange-CrossTenant-originalarrivaltime: 11 Jul 2023 00:26:56.5194
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: XtP23EmXb8KGARzKo6uCsFthG054bLnnrOhPAzB0DjbEQKG+EpZMKkkE4I7s5Y5NSti+y8tx/NNYie1FuuAOuA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA2PR11MB5116
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,
        RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE,
        URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


> -----Original Message-----
> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Tuesday, July 11, 2023 12:55 AM
> To: Zhang, Tina <tina.zhang@intel.com>
> Cc: Michael Shavit <mshavit@google.com>; Will Deacon <will@kernel.org>;
> Robin Murphy <robin.murphy@arm.com>; Joerg Roedel <joro@8bytes.org>;
> jean-philippe@linaro.org; nicolinc@nvidia.com; baolu.lu@linux.intel.com;
> linux-arm-kernel@lists.infradead.org; iommu@lists.linux.dev; linux-
> kernel@vger.kernel.org
> Subject: Re: [PATCH v2 14/18] iommu/arm-smmu-v3: Support domains with
> shared CDs
>=20
> On Wed, Jul 05, 2023 at 09:56:50AM +0000, Zhang, Tina wrote:
>=20
> > > Possibly what we should do is conver the u32 pasid in the mm_struct
> > > to a struct iommu_mm_data * and put alot more stuff in there. eg a
> > > linked list of all SVA domains.
>=20
> > If we are going to have 1:1 between SVA domain and pasid, why we need
> > a linked list of all SVA domains? Would a SVA domain pointer be
> > enough?
>=20
> Kevin asked this, we can't assume that a single SVA domain is going to wo=
rk in
> a multi-iommu-driver system, which we are trying to enable at the core le=
vel..
>=20
> > I've got a patch-set which takes this suggestion to add an
> > iommu_mm_data struct field to mm_struct. I'll send it out for review
> > soon. The motivation of that patch-set is to let the
> > invalidate_range() callback use the SVA domain referenced by
> > mm->iommu_mm_data->sva_domain to do per-iommu IOTLB invalidation.
>=20
> Huh?
>=20
> You are supposed to put the struct mmu_notifier inside the sva_domain
> struct and use container_of().
No. Just want to use iommu_mm to replace the old pasid field.
https://lore.kernel.org/linux-iommu/20230707013441.365583-5-tina.zhang@inte=
l.com/

The motivation is mainly to for performance optimization for vIOMMU, though=
 it could also benefit native world.

Currently, in invalidate_range() callback implemented by VT-d driver, due t=
o lacking sva_domain knowledge (i.e., intel_invalidate_range() doesn't know=
 which IOMMUs' IOTLB should be invalidated), intel_invalidate_range() just =
performs IOMMU IOTLB per device and that leads to superfluous IOTLB invalid=
ations. For example, if there are two devices physically behind an IOMMU an=
d those two devices are used by one user space application, it means in eac=
h invalidate_range() callback, two IOTLB and two device-IOTLB invalidations=
 would be performed, instead of one IOTLB plus two device-TLB invalidations=
 (i.e., one IOTLB invalidation is redundant). Things will become worse in a=
 virtualized environment, especially with one vIOMMU exposed to guest, as i=
nvoking IOTLB is much more expensive than doing it in native and conceptual=
ly there could be more devices behind a vIOMMU that means more superfluous =
IOTLB invalidations.

So, we want to add sva_domain related info to mm to remove the superfluous =
IOTLB invalidations.

Thanks,
-Tina

>=20
> This is another reason why I'd prefer we de-duplicate SVA domains at the
> core code level as duplicitive notifiers are expensive..
>=20
> Please don't add stuff to the mm just for this reason.
>=20
> Jason