Date:   Fri, 10 Dec 2021 10:18:20 -0800
From:   Jacob Pan <jacob.jun.pan@linux.intel.com>
To:     Jason Gunthorpe <jgg@nvidia.com>
Cc:     Lu Baolu <baolu.lu@linux.intel.com>,
        iommu@lists.linux-foundation.org,
        LKML <linux-kernel@vger.kernel.org>,
        Joerg Roedel <joro@8bytes.org>,
        Christoph Hellwig <hch@infradead.org>,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        Jean-Philippe Brucker <jean-philippe@linaro.com>,
        Jacob Pan <jacob.jun.pan@intel.com>,
        Raj Ashok <ashok.raj@intel.com>,
        "Kumar, Sanjay K" <sanjay.k.kumar@intel.com>,
        Dave Jiang <dave.jiang@intel.com>,
        Tony Luck <tony.luck@intel.com>, Yi Liu <yi.l.liu@intel.com>,
        "Tian, Kevin" <kevin.tian@intel.com>,
        Barry Song <21cnbao@gmail.com>,
        "Zanussi, Tom" <tom.zanussi@intel.com>,
        Dan Williams <dan.j.williams@intel.com>,
        jacob.jun.pan@linux.intel.com
Subject: Re: [PATCH 3/4] iommu/vt-d: Support PASID DMA for in-kernel usage
Message-ID: <20211210101820.58b24b77@jacob-builder>
In-Reply-To: <20211210174848.GH6385@nvidia.com>
References: <1638884834-83028-1-git-send-email-jacob.jun.pan@linux.intel.com>
        <1638884834-83028-4-git-send-email-jacob.jun.pan@linux.intel.com>
        <20211208132255.GS6385@nvidia.com>
        <20211208111659.6de22e52@jacob-builder>
        <9f724b3a-6028-43d7-b4fc-d8a939e7b2cf@linux.intel.com>
        <20211209152113.64b817b9@jacob-builder>
        <921a766f-d826-2ca4-f739-4d196b32a681@linux.intel.com>
        <20211210095025.38af67ce@jacob-builder>
        <20211210174848.GH6385@nvidia.com>
Organization: OTC
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Precedence: bulk

Hi Jason,

On Fri, 10 Dec 2021 13:48:48 -0400, Jason Gunthorpe <jgg@nvidia.com> wrote:

> On Fri, Dec 10, 2021 at 09:50:25AM -0800, Jacob Pan wrote:
> 
> > > Tying pasid to an iommu_domain is not a good idea. An iommu_domain
> > > represents an I/O address translation table. It could be attached to a
> > > device or a PASID on the device.  
> > 
> > I don;t think we can avoid storing PASID at domain level or the group's
> > default domain. IOTLB flush is per domain. Default domain of DMA type
> > is already tying to PASID0, right?  
> 
> No, it is just wrong.
> 
> If the HW requires a list of everything that is connected to the
> iommu_domain then it's private iommu_domain should have that list.
> 
What I have in this patchset is in the private dmar_domain
struct dmar_domain {
	...
	u32		kernel_pasid;	/* for in-kernel DMA w/
PASID */ atomic_t	kernel_pasid_user; /* count of kernel_pasid users
*/ struct iommu_domain domain;	/* generic domain data structure for
					   iommu core */
};

Perhaps I am missing the point. "private domain" is still "domain level" as
what I stated. Confused :(

> But it is a *list* not a single PASID.
> 
We could have a list when real use case comes.

> If one device has 10 PASID's pointing to this domain you must flush
> them all if that is what the HW requires.
> 
Yes. My point is that other than PASID 0 is a given, we must track the 10
PASIDs to avoid wasted flush. It also depend on how TLBs are tagged and
flush granularity available. But at the API level, should we support all the
cases?

> Jason


Thanks,

Jacob