Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Tue, 28 Mar 2023 13:39:26 -0700
From:   Jacob Pan <jacob.jun.pan@linux.intel.com>
To:     "Tian, Kevin" <kevin.tian@intel.com>
Cc:     Baolu Lu <baolu.lu@linux.intel.com>,
        LKML <linux-kernel@vger.kernel.org>,
        "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
        Jason Gunthorpe <jgg@nvidia.com>,
        Joerg Roedel <joro@8bytes.org>,
        "dmaengine@vger.kernel.org" <dmaengine@vger.kernel.org>,
        "vkoul@kernel.org" <vkoul@kernel.org>,
        Robin Murphy <robin.murphy@arm.com>,
        Will Deacon <will@kernel.org>,
        "David Woodhouse" <dwmw2@infradead.org>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "Liu, Yi L" <yi.l.liu@intel.com>,
        "Yu, Fenghua" <fenghua.yu@intel.com>,
        "Jiang, Dave" <dave.jiang@intel.com>,
        "Luck, Tony" <tony.luck@intel.com>,
        "Zanussi, Tom" <tom.zanussi@intel.com>,
        jacob.jun.pan@linux.intel.com
Subject: Re: [PATCH v2 5/8] iommu/vt-d: Make device pasid attachment
 explicit
Message-ID: <20230328133926.6f6c2ed2@jacob-builder>
In-Reply-To: <BN9PR11MB5276BCF726D0B813046479A18C889@BN9PR11MB5276.namprd11.prod.outlook.com>
References: <20230327232138.1490712-1-jacob.jun.pan@linux.intel.com>
        <20230327232138.1490712-6-jacob.jun.pan@linux.intel.com>
        <71b60cc2-4ab0-3777-6bc9-3a6a4174f743@linux.intel.com>
        <BN9PR11MB5276BCF726D0B813046479A18C889@BN9PR11MB5276.namprd11.prod.outlook.com>
Organization: OTC
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

Hi Kevin,

On Tue, 28 Mar 2023 07:44:52 +0000, "Tian, Kevin" <kevin.tian@intel.com>
wrote:

> > From: Baolu Lu <baolu.lu@linux.intel.com>
> > Sent: Tuesday, March 28, 2023 1:49 PM
> >=20
> > On 3/28/23 7:21 AM, Jacob Pan wrote: =20
> > > diff --git a/drivers/iommu/intel/iommu.h b/drivers/iommu/intel/iommu.h
> > > index 65b15be72878..b6c26f25d1ba 100644
> > > --- a/drivers/iommu/intel/iommu.h
> > > +++ b/drivers/iommu/intel/iommu.h
> > > @@ -595,6 +595,7 @@ struct dmar_domain {
> > >
> > >   	spinlock_t lock;		/* Protect device tracking
> > > lists */ struct list_head devices;	/* all devices' list */
> > > +	struct list_head dev_pasids;	/* all attached pasids */
> > >
> > >   	struct dma_pte	*pgd;		/* virtual
> > > address */ int		gaw;		/* max guest
> > > address width */ @@ -708,6 +709,7 @@ struct device_domain_info {
> > >   	u8 ats_supported:1;
> > >   	u8 ats_enabled:1;
> > >   	u8 dtlb_extra_inval:1;	/* Quirk for devices need
> > > extra flush */
> > > +	u8 dev_attached:1;	/* Device context activated */
> > >   	u8 ats_qdep;
> > >   	struct device *dev; /* it's NULL for PCIe-to-PCI bridge */
> > >   	struct intel_iommu *iommu; /* IOMMU used by this device */
> > > @@ -715,6 +717,12 @@ struct device_domain_info {
> > >   	struct pasid_table *pasid_table; /* pasid table */
> > >   };
> > >
> > > +struct device_pasid_info {
> > > +	struct list_head link_domain;	/* link to domain
> > > siblings */
> > > +	struct device *dev;		/* physical device
> > > derived from */
> > > +	ioasid_t pasid;			/* PASID on physical
> > > device */ +}; =20
> >=20
> > The dev_pasids list seems to be duplicate with iommu_group::pasid_array.
> >=20
> > The pasid_array is de facto per-device as the PCI subsystem requires ACS
> > to be enabled on the upstream path to the root port.
> >=20
> > pci_enable_pasid():
> > 385         if (!pci_acs_path_enabled(pdev, NULL, PCI_ACS_RR |
> > PCI_ACS_UF)) 386                 return -EINVAL;
> >=20
> > For such PCI topology, pci_device_group() always assigns an exclusive
> > iommu group (a.k.a. singleton group).
> >=20
> > So, how about moving the pasid_array from struct iommu_group to struct
> > dev_iommu? With this refactoring, the individual iommu driver has no
> > need to create their own pasid array or list.
> >=20
> > Instead of using iommu_group::mutex, perhaps the pasid_array needs its
> > own lock in struct dev_iommu after moving.
> >  =20
>=20
> What you suggested is a right thing and more friendly to pasid attach
> in iommufd [1].
>=20
> but dev_pasids list here is a different thing. It tracks which [device,
> pasid] is attached to the domain. w/o this information you'll have to
> walk the pasid_array of every attached device under the domain and search
> for every pasid entry pointing to the said domain. It's very inefficient.=
=20
>=20
> of course if this can be done more generally it'd be nice.=F0=9F=98=8A
>=20
> [1] https://lore.kernel.org/linux-iommu/ZAjbDxSzxYPqSCjo@nvidia.com/
Yes, it would be nice as the next step. But so far only ENQCMDS usages may
not justify.

Thanks,

Jacob