From:   "Tian, Kevin" <kevin.tian@intel.com>
To:     Jacob Pan <jacob.jun.pan@linux.intel.com>,
        Jason Gunthorpe <jgg@nvidia.com>
CC:     "Luck, Tony" <tony.luck@intel.com>,
        LKML <linux-kernel@vger.kernel.org>,
        "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
        Lu Baolu <baolu.lu@linux.intel.com>,
        Joerg Roedel <joro@8bytes.org>,
        "dmaengine@vger.kernel.org" <dmaengine@vger.kernel.org>,
        "vkoul@kernel.org" <vkoul@kernel.org>,
        Robin Murphy <robin.murphy@arm.com>,
        Will Deacon <will@kernel.org>,
        David Woodhouse <dwmw2@infradead.org>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "Liu, Yi L" <yi.l.liu@intel.com>,
        "Yu, Fenghua" <fenghua.yu@intel.com>,
        "Jiang, Dave" <dave.jiang@intel.com>,
        "Zanussi, Tom" <tom.zanussi@intel.com>
Subject: RE: [PATCH 3/4] iommu/sva: Support reservation of global PASIDs
Thread-Topic: [PATCH 3/4] iommu/sva: Support reservation of global PASIDs
Thread-Index: AQHZTKHLpFv8Ca2hQ0+TT/QlO+gFRa7nPClwgAJeMICABCPvAIAATvUA////3YCAAAQCgIAABegAgAAIRICAAAStgIAElbqAgApc9+A=
Date:   Thu, 16 Mar 2023 07:25:17 +0000
Message-ID: <BN9PR11MB5276DE801B64423E7BF3B68E8CBC9@BN9PR11MB5276.namprd11.prod.outlook.com>
References: <20230302005959.2695267-1-jacob.jun.pan@linux.intel.com>
        <20230302005959.2695267-4-jacob.jun.pan@linux.intel.com>
        <BN9PR11MB52765C5E0DC0759880C08E258CB29@BN9PR11MB5276.namprd11.prod.outlook.com>
        <20230303134753.660d0755@jacob-builder> <ZAXkLN39VUSl+t65@nvidia.com>
        <20230306094408.2d675d5b@jacob-builder> <ZAYmS4Sx6bm+ziDY@nvidia.com>
        <20230306095759.1dd65cca@jacob-builder> <ZAYunPcgSOGFK8Qi@nvidia.com>
        <SJ1PR11MB6083DFA2C1D00B00C3918982FCB69@SJ1PR11MB6083.namprd11.prod.outlook.com>
        <ZAY5d2MSXjWRGF0n@nvidia.com> <20230309090623.7ea2c3fe@jacob-builder>
In-Reply-To: <20230309090623.7ea2c3fe@jacob-builder>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?/WhKrfW9ll/oFRsn8Mi1gvPgKpoEmdCjw7JBwL6KOmtCdU1rm7AdAhc6xcjC?=
 =?us-ascii?Q?r6bXMBPyyZzz1KD/SXLjm6KODo6nHegDKYK2NfW8qprNTEUVrSL/R30+CiCh?=
 =?us-ascii?Q?cqK2BUZoFri16TtDL6hRmx43J68lOEAiRpNxkAVAnokPFqeAZDBhN+D/f8Vg?=
 =?us-ascii?Q?lob27t9rr3lSb/oPf6xKTo17irJQ++J60VO7nKCQQlBGtFCAXRKQhyGZaoCc?=
 =?us-ascii?Q?ltNoGK80pOLHt5tfLlluVWaigUJFzLdmb4+ZyIuJtlhLx9u8SJZNdei2gLy3?=
 =?us-ascii?Q?I6+BQqs3E1Ea6NhqojoFkCaZaa3GA9PIJTiOwtWalIbhD/Wl1WpHZZ4WR/tY?=
 =?us-ascii?Q?2BNsYZBXlvLSu34o5e2YPBy5vPw6jLbwNXQJ16KUZMhv9An8Wjk8VWN9gspM?=
 =?us-ascii?Q?abX2jjTuFQ62Igdsgc4iG3rDbJI+ZWtTyGgY+RKkWh2/1ecwDSFyu4P5wMCU?=
 =?us-ascii?Q?u3DKjaZuwDO4sHO69wBuvD2fyATcEfQ3KepAaiPEB7iIZ//oKvYYzNDaCsAb?=
 =?us-ascii?Q?ZwuVoZt70ATBrxZyd85XsJKUXu+2QbBWvKDJTRKd6R3TlS6FEZsm2GIOViLa?=
 =?us-ascii?Q?sV0e3gAMcNwIGztbMrXGtkZrwT/7rE9s3L6K1PYnemI+I1MgRYyg1sjwuwem?=
 =?us-ascii?Q?uFvYgPxtr9Jl+wCHfNK3oL6GKJO6+U2//N4udVUiOtKiUWjC7EPbsUGkznue?=
 =?us-ascii?Q?wB8pG/IfyHLNiczJUy0i9jix/w7YXZy55mGVZtU3BSdGRp7fbBNEfXHDEioe?=
 =?us-ascii?Q?YhUfUyzPNLSApGVfdCkd/azHRIKyfnmK+wWsizkf5TY/R/5Urxa50DDRIk0X?=
 =?us-ascii?Q?uYR0G8rNalUSgihPzt6KwEOqYI8uxgm4OnYpB9r8r2MmOnBD078ipihtSEUf?=
 =?us-ascii?Q?rEf7SbIpPG1FuLWGYrduNSL6LL4xwFdT118NmKhkSYPTOaJB+TFyln1uBXYu?=
 =?us-ascii?Q?KC5f+cniXxQHQ3ZmuY60O24B771b2ElkReqZhreb/QT8IWY6rbR4s2Yi0rfH?=
 =?us-ascii?Q?bl3nzwOOWsb5QjnPEynVxL4r2pttFTd3GNaaQJ4vUXOpZdRqFp1ftqtvZt7X?=
 =?us-ascii?Q?CVLWRLaavXfRq7sy4IzsIsPUm409DJMAvtkMTmlzRHK/jTlYCxzLfZ3Ox1zO?=
 =?us-ascii?Q?TQZydJZiPsRpcz9cFzfp1lqhlDfoOhnJkrCszLzG43chdsExL0zL36/gI7uJ?=
 =?us-ascii?Q?7bfxAzTGbqV+w8zrd0JoTyw1E+HD0puynebrfu07rUU804IxJXFnCxzfiG+K?=
 =?us-ascii?Q?K+ijBmdlzCsVRCEsn+oYtdexx6vS5RvHazFtsXqaO656tFSM22rxmwDzkau/?=
 =?us-ascii?Q?Rh8SvLc3TxnyPeqvRogMlfMCMjd5GI81dgCHBrL0E0HDVIy36BYfw+LlKC7G?=
 =?us-ascii?Q?pjOcwv+3XPEzNVPmcuT1UEm3fapKMmvUWNo+bkT57oX59DCbp4LeAHbTnh53?=
 =?us-ascii?Q?xCx2U0Mbp8RPHVholq76qGPrFJQNk+zJkqXSCR4DBuFSt1rqSIrTY0LCL8iB?=
 =?us-ascii?Q?ACFYkPUhgCwaTDA/KFFbZHJQwrjXQFdqLN1MXBXcheIv7z576qyHiIXdRxvL?=
 =?us-ascii?Q?9LY35jtGi6ZDKkoIckJeC3YNpwVl6iiPT7RJwjLz?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5276.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 9b4d4402-37f1-4cdf-0477-08db25ef97c3
X-MS-Exchange-CrossTenant-originalarrivaltime: 16 Mar 2023 07:25:17.6140
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: t9scVoDf0tVM5xMbhiHN/+ys+zuL41EqYZ2Wij68mHAtyYlycYLtkoOtMpIXvM192Cw6Y88BdZbNtq4JhhjOuQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR11MB6349
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> From: Jacob Pan <jacob.jun.pan@linux.intel.com>
> Sent: Friday, March 10, 2023 1:06 AM
>=20
> Hi Jason,
>=20
> On Mon, 6 Mar 2023 15:05:27 -0400, Jason Gunthorpe <jgg@nvidia.com>
> wrote:
>=20
> > On Mon, Mar 06, 2023 at 06:48:43PM +0000, Luck, Tony wrote:
> > > >> ENQCMDS does not have the restriction of using a single CPU MSR to
> > > >> store PASIDs, PASID is supplied to the instruction operand.
> > > >
> > > > Huh? That isn't what it says in the programming manual. It says the
> > > > PASID only comes from the IA32_PASID msr and the only two operands
> are
> > > > the destination MMIO and the memory source for the rest of the
> > > > payload.
> > >
> > > Jason,
> > >
> > > Two different instructions with only one letter different in the name=
.
> > >
> > > ENQCMD - ring 3 instruction. The PASID is inserted into the descripto=
r
> > > pushed to the device from the IA32_PASID MSR.
> > >
> > > ENQCMDS - ring 0 instruction (see that trailing "S" for Supervisor
> > > mode). In this case the submitter can include any PASID value they wa=
nt
> > > in the in-memory copy of the descriptor and ENQCMDS will pass that to
> > > the device.
> >
> > Ah, well, my comment wasn't talking about ENQCMDS :)
> >
> > If ENQCMDS can take in an arbitary PASID then there is no
> > justification here to use the global allocator.
> >
> > The rational is more like:
> >
> >  IDXD uses PASIDs that come from the SVA allocator. It needs to create
> >  an internal kernel-only PASID that is non-overlapping so allow the SVA
> >  allocator to reserve PASIDs for driver use.
> >
> >  IDXD has to use the global SVA PASID allocator beacuse its userspace
> >  will use ENQCMD which requires global PASIDs.
> >
> yes, great summary. I think that is the same as what I was trying to say
> earlier :)
> "due the unforgiving nature of ENQCMD that requires global PASIDs,
> ENQCMDS
> has no choice but to allocate from the same numberspace to avoid conflict=
."
>=20
> In that sense, I feel the global allocator should be staying with SVA
> instead of moving to iommu core (as Kevin suggested). Because we are tryi=
ng
> to have non-overlapping pasid with SVA.
>=20

I still doubt 'reserve' is the right interface to define.

for DMA domain probably yes as it's static and one-off.

but thinking louder when the same driver starts to support SIOV we
need allocating additional PASIDs on demand which is hardly to be
fit in a reservation interface.