Date:   Mon, 13 Feb 2023 19:45:46 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Jacob Pan <jacob.jun.pan@linux.intel.com>
Cc:     Jean-Philippe Brucker <jean-philippe@linaro.org>,
        LKML <linux-kernel@vger.kernel.org>, iommu@lists.linux.dev,
        Lu Baolu <baolu.lu@linux.intel.com>,
        Joerg Roedel <joro@8bytes.org>,
        Jean-Philippe Brucker <jean-philippe@linaro.com>,
        Robin Murphy <robin.murphy@arm.com>,
        David Woodhouse <dwmw2@infradead.org>,
        Raj Ashok <ashok.raj@intel.com>,
        "Tian, Kevin" <kevin.tian@intel.com>, Yi Liu <yi.l.liu@intel.com>
Subject: Re: [PATCH 2/2] iommu/ioasid: Remove custom IOASID allocator
Message-ID: <Y+rLqg66BBPsfsRK@nvidia.com>
References: <20230210230206.3160144-1-jacob.jun.pan@linux.intel.com>
 <20230210230206.3160144-3-jacob.jun.pan@linux.intel.com>
 <Y+pjTR80bnl9rMK1@myrica>
 <20230213103455.02f3ac37@jacob-builder>
 <Y+qR5zC8ZOkVphgv@nvidia.com>
 <20230213134402.194ed7fd@jacob-builder>
 <Y+rFW80DUHvf3hQP@nvidia.com>
 <20230213154345.0f76eea1@jacob-builder>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20230213154345.0f76eea1@jacob-builder>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?CywUlh8q9Gnt0gt6PsVU7CTMw/PEw9XjyZLy4y16iMhzuOdms3jBom3d7n2K?=
 =?us-ascii?Q?Ivt9Ad/ipxfi86j29yYWFotL6/bjBDnhh4x9Bmic1mA/qd1bjVbP1uLjlXOo?=
 =?us-ascii?Q?h5l3VzfUwcfTOitC6LlFACFPZD/rPvFSq5bPjkB+Da8ziwsYNv0xxYXhYHtZ?=
 =?us-ascii?Q?0TRgQR5Snmm9Flvi9ZHkuUUWiHsgWBgXpQyUNypI3lFz2wT9wWLr2VyRnpTi?=
 =?us-ascii?Q?dq456UbcL3bYoTQHlKFJpKbj25/GTM4l3TmlMhwb/5YUBc5v63Cc6TqpiQaJ?=
 =?us-ascii?Q?Ki1q4yLvjmEGSAN9Qo0cMBLwiiMN2/1iA3HY+kO0hUgae1U446xNNZv2Hdzl?=
 =?us-ascii?Q?nYiBubRu07mtFLTRLRFIoa/vpex5AIpBIE1O5oxsz71vEp81urd1Gk6UUyUI?=
 =?us-ascii?Q?XPeTMdIa9n68qxEQloTDsT11tE7KokwCdmBC80JIP7uHuJZMGgJRZCG0JI5Q?=
 =?us-ascii?Q?jiQWB81yYpJb7h2KYFPImipNZtV7aPHGl7/oS8mSq9Jvw2Ud7a+D4pvWcZ0a?=
 =?us-ascii?Q?JjMLEGPuHKoW6ww+jX8LnaN2SWpys9lKYrs43ZIU/wDNN93tm5zYH89+QvSI?=
 =?us-ascii?Q?FEcEjDQfLYcTLXP+gtLZ9dtRAvlIuLtQVSmBuu359LjEqJfgTHa2D+tRZft1?=
 =?us-ascii?Q?766SO7IygMVM600gl90TdoMmZLPF9RzvFXfpYhAWju48v6z8Udfr1ycXqFI4?=
 =?us-ascii?Q?RHzLzHnd/rTOJeQG2zTjCKYqivo4Kz+Yn5Fdhd5hQs93H/qNsnv+nXTp3/O5?=
 =?us-ascii?Q?6X3lvdii9NxzPpHkgGEJvMPmp5/JWS9w3bkjvqVLe6GivXsWBrfx3ZJveYed?=
 =?us-ascii?Q?O8go7nMRGJmlnWZuUtc2ayvGGbSMu+Bx1+eZtIc2c9X+GX/sITNL9tqWeNVU?=
 =?us-ascii?Q?3vX02KDmMBPKEKCj3IeFqO3sNTGYas+ZPclAoQ6fW7NrtXWAO0DwO/QiMdOJ?=
 =?us-ascii?Q?yWk5Q0v7TJ1MqPj865s0/s4586f4CcapsPaGmUQkMBQXVDuz9mj0g1mTle5T?=
 =?us-ascii?Q?ljO9Kd4w6mkG1Gz7HXLAeTAKTBrjZub9bBBAo/atxiUzWZj39E2cYiEgwHyX?=
 =?us-ascii?Q?qDNdX0yDeIn87xGHgVyVZEX1wr/G9YFA9Fju65bRKsk5eiOZs3m36ItMmi18?=
 =?us-ascii?Q?nOC6YhaAPAJt9FGi7U1Ie5gXf4gPQftT5IgGC7uUuGxS4y49oNV1lyRByqhb?=
 =?us-ascii?Q?LUEHaNPyTx/cZoHj9IcaeH02C4p6BLJg2DOa01HK6gp8PzDRAL5gzOUPRE0O?=
 =?us-ascii?Q?y9w7zI/KUF9/xeWtjEH/VTYKP9G0eQ+DuFYg92eQsOQZw68wevkYe3kcJmWw?=
 =?us-ascii?Q?Y+XAOc2fH0DbEZBR/enIJ7h0xSks0WWtCab1zivv5n9bbCAKHx596wL0eCH1?=
 =?us-ascii?Q?42gaFOUMX8GB3OaRX5CDbMV3KKT31oRe+nB1OR9qjVjEsqsIdrnv8oqWr/2i?=
 =?us-ascii?Q?pHiRcqq5LTE1/BsCAAiQK8CZIl2969b6Vuk1O+Cxkg1JAZulHn+/CvhDLv6k?=
 =?us-ascii?Q?bq2Cvkc2reYVVPCe2I0kFzQZIvNulUXNf4fp+zCz/5SLFYPcxSkT1wWAlqtM?=
 =?us-ascii?Q?cuK6IlU/0dp+Oa7882BSUCBpCPFN1hpo+RxRAF4x?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 0ab9237b-19a6-4fe9-64bb-08db0e1c6ec7
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Feb 2023 23:45:48.4583
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: SCZ6WdJKAW5N5n+BsmeoC85LCF8QJIAVC2WnmVsi8Tt/X64oP7xeJKND8a6zcpHW
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB8308
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Mon, Feb 13, 2023 at 03:43:45PM -0800, Jacob Pan wrote:
> Hi Jason,
> 
> On Mon, 13 Feb 2023 19:18:51 -0400, Jason Gunthorpe <jgg@nvidia.com> wrote:
> 
> > On Mon, Feb 13, 2023 at 01:44:02PM -0800, Jacob Pan wrote:
> > > > Your DMA API PASID thing will simply need one new API to alloc/free a
> > > > PASID from the iommu_global_pasid_ida  
> > > It should satisfy what we need right now.
> > > Just wondering if we were to do resource management of global PASIDs,
> > > say with the new misc cgroup controller, do we plan to expand in iommu
> > > sva code? If yes, do we keep DMA API PASID in a separate range/set?  
> > 
> > I would say all shared PASIDs held by userspace should be captured by
> > by a resource limit, it doesn't matter if they are global PASIDs or
> > device local shared PASIDs.
> agreed, I was just thinking in-kernel DMA PASID is not held by userspace,
> might be good to keep them in separate pool, thus keeping ioasid_set.

Then you just don't charge a cgroup when you get these, you won't have
a cgroup anyhow in that context.

The "ioasid_set" is really the xarray, and you don't have an option to
use two xarrays with the same RID, so we definately wouldn't have two
pools.

> > So if a cgroup comes it is just a matter of putting charges in the
> > right place which is auditable by looking at calls to attach pasid
> > functions.
> shouldn't we charge cg during allocation? Or it might be too early for
> iommufd so we have to wait  until attach?

We need to sort this all out. I would expect that iommufd will have an
allocation API.

Jason