Date:   Thu, 9 Feb 2023 16:49:55 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Nicolin Chen <nicolinc@nvidia.com>
Cc:     "Tian, Kevin" <kevin.tian@intel.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "will@kernel.org" <will@kernel.org>,
        "robin.murphy@arm.com" <robin.murphy@arm.com>,
        "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
        "shuah@kernel.org" <shuah@kernel.org>,
        "Liu, Yi L" <yi.l.liu@intel.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "linux-kselftest@vger.kernel.org" <linux-kselftest@vger.kernel.org>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>
Subject: Re: [PATCH v2 05/10] iommufd: Add replace support in
 iommufd_access_set_ioas()
Message-ID: <Y+Vcc6x2JVPwkl5+@nvidia.com>
References: <cover.1675802050.git.nicolinc@nvidia.com>
 <931be169ff4a1f4d4f1ed060d722c2dc17ce6667.1675802050.git.nicolinc@nvidia.com>
 <BN9PR11MB5276AE27E37866B82A6E03608CD99@BN9PR11MB5276.namprd11.prod.outlook.com>
 <Y+VXfQbXakNSHSLw@Asurada-Nvidia>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <Y+VXfQbXakNSHSLw@Asurada-Nvidia>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?r+fTq31BmkvM3pY1fV9K4G9ff8ZOgp6IiGcIOEUiI7NwHjFFQs2lQFbXHtYe?=
 =?us-ascii?Q?pfAGw8XDTKyCwmkKTvpv4XjGDTyC0ARIzdva5kD2IIDlFAIDDQIGuMdlRhmB?=
 =?us-ascii?Q?upXA/58RX5DgGoCg8Y33DNtv9k+dVxIbWyLwrmYvZe8znFBAgahSIsCTxVdj?=
 =?us-ascii?Q?pBBWmVoWTr8mVrbNnlgKV1LKSGlw4vFAekkKgSpofCIkwED2UcN0Lcu40Ugn?=
 =?us-ascii?Q?uTJAL+FKvtGC0Xr4RIoRWKUjOixL7o0p6tkH6KGpUT+81rpPGHm0PPT7xtGo?=
 =?us-ascii?Q?T60BpdWf6p7jkMgWU3vCEn1SLp4djjz9AmDC3fa+Okyycp8Ejk+lDyz6NlN4?=
 =?us-ascii?Q?gEauRA6ANyj77PFQ2Ldyh0h4STvBLT1YqccGZ0yVtnv2f5EBF15wNS6B2efm?=
 =?us-ascii?Q?RDGhrs3u4UooLs1dVpLEXOuZL4BCPjplawo8x9ERa1vptEN5x5otyo5+8zuu?=
 =?us-ascii?Q?fWzSuFw36rZUPmwLPI8M11ZUKtsZ3WY0XJQu094nwJGz9c1aAAYseqIbg2Dp?=
 =?us-ascii?Q?eNkyZnjQOtS8fZQKDIyMoQopWYdoImfXQlf38sUQxnp66wgElpTdo+RN80nL?=
 =?us-ascii?Q?pRybbU7xi+EqzWvuQHtBDYarqQVveHaO5BdizrkWR4YrqnQgXxa2RgrmTOX6?=
 =?us-ascii?Q?7hEL9vRDeyqHD9ejozWLmeNSelw99sMOdIIKEflBmrR0K9c25peRvBxBo0/v?=
 =?us-ascii?Q?DPzvw4rC/y2HKvd1wGYsJer66kz00v+M7wZoDUD5MJwT+sAkgqn1lsygCkuq?=
 =?us-ascii?Q?Jv9ez3xHf9pZLxKBvHMG3o/QjPauPssQ4CcADi/+IhISEWkPJEeugsHswwtx?=
 =?us-ascii?Q?jyn1HO22b3zYDs0hDzz7dbSiTCu/tmj4W1F0rlWXkOoOMiQnfWgdxJIcgXQa?=
 =?us-ascii?Q?rKW7BTY2dlkmqQbYiEZSZXKRqkjoEKFkKjJUqeqXXWZXMhWxp7r6mXdWj6fC?=
 =?us-ascii?Q?xDhVjLHG6vmk6tK9q2gpDzBUvALRSz2+Mx8TolS0ne4d0J7J0X8+BNYxPffZ?=
 =?us-ascii?Q?JaMrrFBk176AOB4iGlgkg/0lqaj4XcpVWITrVPll6c3tI+Q9TuPjqsyL0/Pp?=
 =?us-ascii?Q?GatuspV0hWfAgqtzlEpC0bPKMkmGlz7i25Jv//JSjOS5gFI4FD7xbgnII6b7?=
 =?us-ascii?Q?XpVn5JKP2HvWAIBbbnbvqhomuwzHOI8QZ9ldAaEcdKMiYVRRklszWaKWtOH2?=
 =?us-ascii?Q?EqFNX9pyewiJOapIzAIdoLRXvTqV3grzQAgt+dV6tXUh08oGRNUx5TyW2GBq?=
 =?us-ascii?Q?hpz0+HHd8SEynZ/DRNtXo7UyLg5C+N/D9QTRybIdo8UYXVGtKX1FfQdPsIEY?=
 =?us-ascii?Q?ekB3lzR0F1LWvMv0w+5yX6iXT5AuR+GuEDI78iw2dOOFQmjjuKysOxzJyaRs?=
 =?us-ascii?Q?TunjOoyGfI/FLSJiFluZeeM+Pxf7A3MsegbfHg+4BHq+zHv4KLDTC6/kcPyc?=
 =?us-ascii?Q?PR4+v192Wsm6dX9fvvV5Ta4q76/NTbaV7MwAadVFINk0Ry9fEoHupMpUBMw/?=
 =?us-ascii?Q?ua3Ha7ay0K1pyUKCdXUb2UtYDZx8JSG/e6vcgfnQr5qdmrlwL8SvxBoKeliz?=
 =?us-ascii?Q?xTtlbQ23ZZjmAeqp+yB7OS/NplSykwMnYqgDTTSC?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 0d95b749-2982-4397-7cc1-08db0adf344e
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Feb 2023 20:49:57.6788
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: aG1f12+iPtbD2BKA++XTVGY8eaYSMyTRKRidNxyZGPjeN+3GqvtXg2PiFcvgQRg/
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB7038
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Thu, Feb 09, 2023 at 12:28:45PM -0800, Nicolin Chen wrote:
> On Thu, Feb 09, 2023 at 03:13:08AM +0000, Tian, Kevin wrote:
>  
> > > --- a/drivers/iommu/iommufd/device.c
> > > +++ b/drivers/iommu/iommufd/device.c
> > > @@ -509,11 +509,23 @@ int iommufd_access_set_ioas(struct
> > > iommufd_access *access, u32 ioas_id)
> > >               iommufd_ref_to_users(obj);
> > >       }
> > >
> > > +     /*
> > > +      * Set ioas to NULL to block any further iommufd_access_pin_pages().
> > > +      * iommufd_access_unpin_pages() can continue using access-
> > > >ioas_unpin.
> > > +      */
> > > +     access->ioas = NULL;
> > > +
> > >       if (cur_ioas) {
> > > +             if (new_ioas) {
> > > +                     mutex_unlock(&access->ioas_lock);
> > > +                     access->ops->unmap(access->data, 0, ULONG_MAX);
> > > +                     mutex_lock(&access->ioas_lock);
> > > +             }
> > 
> > why does above only apply to a valid new_ioas? this is the cleanup on
> > cur_ioas then required even when new_ioas=NULL.
>   
> Though it'd make sense to put it in the common path, our current
> detach routine doesn't call this unmap. If we do so, it'd become
> something new to the normal detach routine. Or does this mean the
> detach routine has been missing an unmap call so far?

By the time vfio_iommufd_emulated_unbind() is called the driver's
close_device() has already returned

At this point the driver should have removed all active pins.

We should not call back into the driver with unmap after its
close_device() returns.

However, this function is not on the close_device path so it should
always flush all existing mappings before attempting to change the
ioas to anything.

Jason