Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates
 216.228.117.160 as permitted sender) receiver=protection.outlook.com;
 client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C
Date:   Thu, 9 Feb 2023 14:18:07 -0800
From:   Nicolin Chen <nicolinc@nvidia.com>
To:     Jason Gunthorpe <jgg@nvidia.com>
CC:     "Tian, Kevin" <kevin.tian@intel.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "will@kernel.org" <will@kernel.org>,
        "robin.murphy@arm.com" <robin.murphy@arm.com>,
        "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
        "shuah@kernel.org" <shuah@kernel.org>,
        "Liu, Yi L" <yi.l.liu@intel.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "linux-kselftest@vger.kernel.org" <linux-kselftest@vger.kernel.org>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>
Subject: Re: [PATCH v2 05/10] iommufd: Add replace support in
 iommufd_access_set_ioas()
Message-ID: <Y+VxH1upHzyR8gV/@Asurada-Nvidia>
References: <cover.1675802050.git.nicolinc@nvidia.com>
 <931be169ff4a1f4d4f1ed060d722c2dc17ce6667.1675802050.git.nicolinc@nvidia.com>
 <BN9PR11MB5276AE27E37866B82A6E03608CD99@BN9PR11MB5276.namprd11.prod.outlook.com>
 <Y+VXfQbXakNSHSLw@Asurada-Nvidia>
 <Y+Vcc6x2JVPwkl5+@nvidia.com>
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <Y+Vcc6x2JVPwkl5+@nvidia.com>
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Feb 2023 22:18:24.8670
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 2fc6ad97-a294-47a4-d25f-08db0aeb8fd2
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com]
X-MS-Exchange-CrossTenant-AuthSource: CO1PEPF00001A5F.namprd05.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR12MB8417
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Thu, Feb 09, 2023 at 04:49:55PM -0400, Jason Gunthorpe wrote:
> On Thu, Feb 09, 2023 at 12:28:45PM -0800, Nicolin Chen wrote:
> > On Thu, Feb 09, 2023 at 03:13:08AM +0000, Tian, Kevin wrote:
> >  
> > > > --- a/drivers/iommu/iommufd/device.c
> > > > +++ b/drivers/iommu/iommufd/device.c
> > > > @@ -509,11 +509,23 @@ int iommufd_access_set_ioas(struct
> > > > iommufd_access *access, u32 ioas_id)
> > > >               iommufd_ref_to_users(obj);
> > > >       }
> > > >
> > > > +     /*
> > > > +      * Set ioas to NULL to block any further iommufd_access_pin_pages().
> > > > +      * iommufd_access_unpin_pages() can continue using access-
> > > > >ioas_unpin.
> > > > +      */
> > > > +     access->ioas = NULL;
> > > > +
> > > >       if (cur_ioas) {
> > > > +             if (new_ioas) {
> > > > +                     mutex_unlock(&access->ioas_lock);
> > > > +                     access->ops->unmap(access->data, 0, ULONG_MAX);
> > > > +                     mutex_lock(&access->ioas_lock);
> > > > +             }
> > > 
> > > why does above only apply to a valid new_ioas? this is the cleanup on
> > > cur_ioas then required even when new_ioas=NULL.
> >   
> > Though it'd make sense to put it in the common path, our current
> > detach routine doesn't call this unmap. If we do so, it'd become
> > something new to the normal detach routine. Or does this mean the
> > detach routine has been missing an unmap call so far?
> 
> By the time vfio_iommufd_emulated_unbind() is called the driver's
> close_device() has already returned
> 
> At this point the driver should have removed all active pins.
> 
> We should not call back into the driver with unmap after its
> close_device() returns.

I see. Just found that vfio_device_last_close().

> However, this function is not on the close_device path so it should
> always flush all existing mappings before attempting to change the
> ioas to anything.

OK. That means I also need to fix my change here:

diff --git a/drivers/iommu/iommufd/device.c b/drivers/iommu/iommufd/device.c
index 8a9834fc129a..ba3fd35b7540 100644
--- a/drivers/iommu/iommufd/device.c
+++ b/drivers/iommu/iommufd/device.c
@@ -465,7 +465,10 @@ void iommufd_access_destroy_object(struct iommufd_object *obj)
        struct iommufd_access *access =
                container_of(obj, struct iommufd_access, obj);

-       iommufd_access_set_ioas(access, 0);
+       if (access->ioas) {
+               iopt_remove_access(&access->ioas->iopt, access);
+               refcount_dec(&access->ioas->obj.users);
+       }
        iommufd_ctx_put(access->ictx);
        mutex_destroy(&access->ioas_lock);
 }

Otherwise, the close_device path would invoke this function via
the unbind() call.

Thanks
Nic