Date:   Mon, 13 Feb 2023 15:48:25 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Jean-Philippe Brucker <jean-philippe@linaro.org>
Cc:     Jacob Pan <jacob.jun.pan@linux.intel.com>,
        LKML <linux-kernel@vger.kernel.org>, iommu@lists.linux.dev,
        Lu Baolu <baolu.lu@linux.intel.com>,
        Joerg Roedel <joro@8bytes.org>,
        Jean-Philippe Brucker <jean-philippe@linaro.com>,
        Robin Murphy <robin.murphy@arm.com>,
        David Woodhouse <dwmw2@infradead.org>,
        Raj Ashok <ashok.raj@intel.com>,
        "Tian, Kevin" <kevin.tian@intel.com>, Yi Liu <yi.l.liu@intel.com>
Subject: Re: [PATCH 2/2] iommu/ioasid: Remove custom IOASID allocator
Message-ID: <Y+qUCTGEOOeV4rua@nvidia.com>
References: <20230210230206.3160144-1-jacob.jun.pan@linux.intel.com>
 <20230210230206.3160144-3-jacob.jun.pan@linux.intel.com>
 <Y+pjTR80bnl9rMK1@myrica>
 <20230213103455.02f3ac37@jacob-builder>
 <Y+qP0q9U0J2UfIxX@myrica>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <Y+qP0q9U0J2UfIxX@myrica>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?WM8edvTJZYw9Mqkcck2HneJ1XIuy/s/m1ppg5TbEo3L4YTW13i1qIzS21Dqg?=
 =?us-ascii?Q?3RaZmpgyko0nDf7xX2FctJ1efpSQHo12R5IstWDqs6U5zU8qPpA1idLeNnSf?=
 =?us-ascii?Q?JHj3cYui3yTBuG3+eztbM8SKc7/pSC5iVhLfjF8409eljKLceTfP3Q7+/kh0?=
 =?us-ascii?Q?L0lW7FCmX+4XN2wvdvVqkEzHvJgRUtOaDYRFa2fcnAdMLFeE4lmnvz0SXM++?=
 =?us-ascii?Q?CcuNShWhPbQZqxzMC9C7tCJ6KGJtcNw5burqd7rhdc7wqw4GMT198yH0lG9b?=
 =?us-ascii?Q?Lw4qj+bVUjbrA11GENRB4pohhuhXhVTBAHlaVMlmNHg58YFV39Qbws5HcdTx?=
 =?us-ascii?Q?MY9dI+YIhPj/XLtbP893LeF1VBUmIUX0anj3WV2kWuvg88NlKYNwq9tecdEh?=
 =?us-ascii?Q?3TevgHUZjizb/0pMMYzNY5vMtN/UzeVVzS6ME/W6AROOveO0A2O3pyJu1kZx?=
 =?us-ascii?Q?FH9zYxNxTK1YqBXCKR5/mmZhyV1NFVrJJeh0pN4jBoy1YEHdal2m5Z8QxZ2L?=
 =?us-ascii?Q?QmsVh+n+m+aGp8x7kfDQk6/+Q0/LH8J+FAb6GeHbhnbqyY+Bg/8WnmvfTw8W?=
 =?us-ascii?Q?zxY8lY8N9IZNiTCNKA47uPTHgfat8vo5bh67RRClHXQALJnRTHb21x0GTYcL?=
 =?us-ascii?Q?60ARXuXxzTKm+JUqpD6wIBWJjPhgYFrWXRvZkVu9YU/SG2XRI0lw92h32KnT?=
 =?us-ascii?Q?ksDvwD5/bU5kfhryQ00cKYctPcz13bPEyLBENo+iWwIOtKZNUw54xUwo15CT?=
 =?us-ascii?Q?GEfYl1OOAzpx5zErZnlFRxOaU8ZtqxQ9syEf/Ct4Zd/IYuw9bD0/EHEgNXq3?=
 =?us-ascii?Q?iuhxgcpcqNGicFUqRveAG/7z9eX2BqCqbzCrukh4a3A5XrSXLYo9++c2LXNx?=
 =?us-ascii?Q?06/XIV5Y7s+cKhr9pzqDuGMHyw3CUlyK8WUmNRYtVRLqYmMz9ZNwjoBdX/qR?=
 =?us-ascii?Q?v0acSiiVaNOgoc92yGuOpjy6mVmwcl5MhB9giD0xRygJMu2xZfGKoSxIbGHE?=
 =?us-ascii?Q?t6psggpytIqUuGv5WzJpESvhEoSikwkMCyvyJmjM9TqnJsF6K7YTohl8ijH1?=
 =?us-ascii?Q?C6vEU7Cz1aIGFxBOo4m78YRPKnTO8rERuZoHZR6eOpdZtpPossR9rUdxP3he?=
 =?us-ascii?Q?yMbZwXiBwZ5Fpc2jG2afNYD604J5bgYnbwJLztBZnlSgmdVlmUqj8b4RD8TE?=
 =?us-ascii?Q?FikoC3j+pQDrTjg9JdKPb5OWhn4iDVeNgwKkAlCt1VJWDg8tgoE8K9vIHBBw?=
 =?us-ascii?Q?c57WqVJDJOLwssbFEr66vhcStJuNDQvnH7xH2Dr4T1uNsHdEqzq2kbKv3Vrd?=
 =?us-ascii?Q?jAH5+DR+3Xs5Ujr+fCp3UYWynxY8vCHcPFFFBHaFL8NM1AlhbkcEDKct6xVY?=
 =?us-ascii?Q?GIz+t3E+3LbFtnnD7RJqJw/wnJxEcX88MAfKkYnT2YCInpi00N/KfI4yfvC9?=
 =?us-ascii?Q?oTSDVYDPsDw3ereNbYu0Mxzsm6yk01f2DBj6XBS/iLN0msVy2pOFwF/G23rP?=
 =?us-ascii?Q?sYQcV29SCGtE3CrZEXW5cY2uLd9nzNGZIwpB7XaldX0N+h6KgnrQenr0xLux?=
 =?us-ascii?Q?+Nc6i/QWBCmaYdwGeiIXu6asK1XKuNQe0KG/RPZS?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: cacd84d9-3695-4135-050e-08db0dfb4619
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Feb 2023 19:48:26.8334
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Nsj2pZRBJulfeCf2TCkZVKct6AfTsPfAnwdZJaMVNW3yL+Ig4fJCgeAn4dU4VJvQ
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5705
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Mon, Feb 13, 2023 at 07:30:26PM +0000, Jean-Philippe Brucker wrote:
> On Mon, Feb 13, 2023 at 10:34:55AM -0800, Jacob Pan wrote:
> > > On a related note, it looks like 100b8a14a370 ("iommu/vt-d: Add pasid
> > > private data helpers") removed the last user of ioasid_set_data(). I guess
> > > that could be dropped too, unless you plan to still use it?
> > > 
> > You are right, will remove.
> > I was planning on the other way around which will convert VT-d's private
> > pasid data helpers to common ioasid code, but when I look closer the
> > private pasid xa is just holding a list of pasid/mm which could be per iommu
> > not global. Another cleanup I suppose.
> 
> Yes that should probably be common to SVA. I'm planning to take another
> look at SVA on the SMMU side following the recent API changes, and from a
> quick glance the problem that VT-d's private helpers solves is
> common.

I think there is something really wrong that the drivers are somehow
tied to the mm_struct in some deep way.

The flow should have the SVA iommu_domain be created and associated
with a mm_struct in the driver.

At this instant the iommu_domain should extract the top of page table
from the mm_struct and from then on the driver never touches the
mm_struct again. The top of page table is just like any other
iommu_domain except it isn't mappable and it isn't allocated/freed.

The driver will establish a mmu_notifier in the driver's private
iommu_domain space to capture invalidations.

Page fault is handled via common code

Any invalidations act exactly like any other invalidation - the driver
goes through all the cache tags associated with the iommu_domain and
cleans them.

What have I missed that requires more common code or tracking a list
of pasid/mm?

PASID attachment and invalidation is general, and has nothing to do
with SVA.

If the core code provies help for anything it should be to help the
driver keep track of what cache tags are linked to which iommu_domain.

The comingling of SVA and PASID has made a big mess in the drivers we
need to unwind.

Jason