Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 8 Sep 2022 17:25:32 +0100
From:   Jean-Philippe Brucker <jean-philippe@linaro.org>
To:     Jason Gunthorpe <jgg@nvidia.com>
Cc:     Baolu Lu <baolu.lu@linux.intel.com>,
        Joerg Roedel <joro@8bytes.org>,
        Christoph Hellwig <hch@infradead.org>,
        Bjorn Helgaas <bhelgaas@google.com>,
        Kevin Tian <kevin.tian@intel.com>,
        Ashok Raj <ashok.raj@intel.com>, Will Deacon <will@kernel.org>,
        Robin Murphy <robin.murphy@arm.com>,
        Jean-Philippe Brucker <jean-philippe@linaro.com>,
        Dave Jiang <dave.jiang@intel.com>,
        Fenghua Yu <fenghua.yu@intel.com>,
        Vinod Koul <vkoul@kernel.org>,
        Eric Auger <eric.auger@redhat.com>,
        Liu Yi L <yi.l.liu@intel.com>,
        Jacob jun Pan <jacob.jun.pan@intel.com>,
        Zhangfei Gao <zhangfei.gao@linaro.org>,
        Zhu Tony <tony.zhu@intel.com>, iommu@lists.linux.dev,
        linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH v13 09/13] iommu/sva: Refactoring
 iommu_sva_bind/unbind_device()
Message-ID: <YxoXfCQcD3yC5ppn@myrica>
References: <20220906124458.46461-1-baolu.lu@linux.intel.com>
 <20220906124458.46461-10-baolu.lu@linux.intel.com>
 <Yxd2+d/VOjdOgrR2@myrica>
 <682d8922-200d-8c89-7142-83e7b3754b8d@linux.intel.com>
 <YxhqbhMmWLeFS512@myrica>
 <YxjV1y/FF0nCI/WO@nvidia.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <YxjV1y/FF0nCI/WO@nvidia.com>
Precedence: bulk

On Wed, Sep 07, 2022 at 02:33:11PM -0300, Jason Gunthorpe wrote:
> On Wed, Sep 07, 2022 at 10:54:54AM +0100, Jean-Philippe Brucker wrote:
> 
> > Is iommu_domain still going to represent both a device context (whole
> > PASID table) and individual address spaces, or are you planning to move
> > away from that?  What happens when a driver does:
> > 
> >   d1 = iommu_domain_alloc()
> >   iommu_attach_device(d1)
> >   d2 = iommu_sva_bind_device()
> >   iommu_detach_device(d1)
> > 
> > Does detach
> > (a) only disable the non-PASID address space?
> > (b) disable everything?
> > (c) fail because the driver didn't unbind first?
> 
> I think it must be (a), considering how everything is defined and the
> needs for vIOMMU emulation.

Yes (a) is probably better. The SMMU driver currently implements (c) to
ensure that you can't switch device driver without unbinding everything
first, and we should keep that check somewhere

> 
> If it is any other option then we have a problem of what to do if the
> guest VM asks to change the page table associated with the RID while a
> PASID is attached.
>  
> > I'm asking because the SMMU driver is still using smmu_domain to represent
> > all address spaces + the non-PASID one, and using the same type
> > "iommu_domain" for the new object makes things unreadable. I think
> > internally we'll want to use distinct variable names, something like
> > "domain" and "address_space". If (a) is not the direction you're going,
> > then it may be worth renaming the API as well.
> > 
> > I'm also not sure why set_dev_pasid() is a domain_ops of the SVA domain,
> > but acts on the parent domain which contains the PASID table. Shouldn't it
> > be an IOMMU op like remove_dev_pasid(), or on the parent domain?
> 
> There is no "parent domain"
> 
> PASID or RID+PASID are completely equal concepts for binding.
> 
> If you are thinking "parent domain" because SMMU is storing the PASID
> table in the RID's iommu_domain, then I think that is a misplacement
> in the SMMU driver...
> 
> The PASID table belongs in the iommu driver's per-group data
> structure. The iommu domain should only have the actual IOPTEs.
> 
> Otherwise everything blows up if you attach an iommu_domain to two
> RIDs - the API demands that every RID gets its own PASID mapping, even
> if the RID shares iommu_domains. We do not have an API to share PASID
> tables.

Well, we still do since SMMU implements it. Changing the API is fine, but
someone will need to rework the SMMU driver to align with the new meaning
of iommu_domain. I can take a stab if no one volunteers but probably not
before next year.

Thanks,
Jean

> 
> Thus the PASID table is NOT part of the iommu_domain.
> 
> The exception will be for nested translation where we will have a
> special ARM iommu_domain that contains the PASID table in userspace
> memory. When this domain is attached it will logically claim the RID
> and every PASID and thus disable the PASID API for that RID.
> 
> Remember also that an UNMANAGED iommu_domain should be attachable to
> many PASID's and RID's concurrently.
> 
> Jason