Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <7312331a-71a1-deea-6d17-bd6d968e87d4@linux.intel.com>
Date:   Thu, 6 Apr 2023 10:49:50 +0800
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.9.0
Cc:     baolu.lu@linux.intel.com, LKML <linux-kernel@vger.kernel.org>,
        iommu@lists.linux.dev, Robin Murphy <robin.murphy@arm.com>,
        Joerg Roedel <joro@8bytes.org>, dmaengine@vger.kernel.org,
        vkoul@kernel.org, Will Deacon <will@kernel.org>,
        David Woodhouse <dwmw2@infradead.org>,
        Raj Ashok <ashok.raj@intel.com>,
        "Tian, Kevin" <kevin.tian@intel.com>, Yi Liu <yi.l.liu@intel.com>,
        "Yu, Fenghua" <fenghua.yu@intel.com>,
        Dave Jiang <dave.jiang@intel.com>,
        Tony Luck <tony.luck@intel.com>,
        "Zanussi, Tom" <tom.zanussi@intel.com>
Subject: Re: [PATCH v3 7/7] dmaengine/idxd: Re-enable kernel workqueue under
 DMA API
Content-Language: en-US
To:     Jason Gunthorpe <jgg@nvidia.com>,
        Jacob Pan <jacob.jun.pan@linux.intel.com>
References: <20230331231137.1947675-1-jacob.jun.pan@linux.intel.com>
 <20230331231137.1947675-8-jacob.jun.pan@linux.intel.com>
 <ZC1mUKH/03QhkcR9@nvidia.com>
From:   Baolu Lu <baolu.lu@linux.intel.com>
In-Reply-To: <ZC1mUKH/03QhkcR9@nvidia.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 4/5/23 8:15 PM, Jason Gunthorpe wrote:
> On Fri, Mar 31, 2023 at 04:11:37PM -0700, Jacob Pan wrote:
>>   static void idxd_disable_system_pasid(struct idxd_device *idxd)
>>   {
>> +	struct pci_dev *pdev = idxd->pdev;
>> +	struct device *dev = &pdev->dev;
>> +	struct iommu_domain *domain;
>> +	union gencfg_reg gencfg;
>> +
>> +	domain = iommu_get_domain_for_dev(dev);
>> +	if (!domain || domain->type == IOMMU_DOMAIN_BLOCKED)
>> +		return;
>> +
>> +	iommu_detach_device_pasid(domain, dev, idxd->pasid);
> This sequence is kinda weird, we shouldn't pass in domain to
> detach_device_pasid, IMHO. We already know the domain because we store
> it in an xarray, it just creates weirdness if the user passes in the
> wrong domain.

The initial idea was that the driver has a domain and it wants to attach
the domain to a pasid of the device. During attaching, iommu core will
save the domain in its xarray.

After use, driver want to detach the domain from the pasid by calling
iommu_detach_device_pasid(). The iommu core will compare the input
domain and the one it saved. A warning will be triggered if they are
different.

	WARN_ON(xa_erase(&group->pasid_array, pasid) != domain);

Logically speaking, @domain for detach_device_pasid is unnecessary,
because the pasid array is essentially per-device (as we discussed
before. the pci_enable_pasid() ensures this), although it is currently
placed in the group structure. In that case, the driver can and should
own everything about the pasid and domain. The roles of the iommu core
and the individual driver are only to handle requests of installing or
withdrawing a domain on/from a device's pasid.

Best regards,
baolu