Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Mon, 27 Sep 2021 10:09:35 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     "Tian, Kevin" <kevin.tian@intel.com>
Cc:     "Liu, Yi L" <yi.l.liu@intel.com>,
        "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
        "hch@lst.de" <hch@lst.de>,
        "jasowang@redhat.com" <jasowang@redhat.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "jean-philippe@linaro.org" <jean-philippe@linaro.org>,
        "parav@mellanox.com" <parav@mellanox.com>,
        "lkml@metux.net" <lkml@metux.net>,
        "pbonzini@redhat.com" <pbonzini@redhat.com>,
        "lushenming@huawei.com" <lushenming@huawei.com>,
        "eric.auger@redhat.com" <eric.auger@redhat.com>,
        "corbet@lwn.net" <corbet@lwn.net>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "yi.l.liu@linux.intel.com" <yi.l.liu@linux.intel.com>,
        "Tian, Jun J" <jun.j.tian@intel.com>, "Wu, Hao" <hao.wu@intel.com>,
        "Jiang, Dave" <dave.jiang@intel.com>,
        "jacob.jun.pan@linux.intel.com" <jacob.jun.pan@linux.intel.com>,
        "kwankhede@nvidia.com" <kwankhede@nvidia.com>,
        "robin.murphy@arm.com" <robin.murphy@arm.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "iommu@lists.linux-foundation.org" <iommu@lists.linux-foundation.org>,
        "dwmw2@infradead.org" <dwmw2@infradead.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
        "david@gibson.dropbear.id.au" <david@gibson.dropbear.id.au>,
        "nicolinc@nvidia.com" <nicolinc@nvidia.com>
Subject: Re: [RFC 06/20] iommu: Add iommu_device_init[exit]_user_dma
 interfaces
Message-ID: <20210927130935.GZ964074@nvidia.com>
References: <20210919063848.1476776-1-yi.l.liu@intel.com>
 <20210919063848.1476776-7-yi.l.liu@intel.com>
 <20210921170943.GS327412@nvidia.com>
 <BN9PR11MB5433DA330D4583387B59AA7F8CA29@BN9PR11MB5433.namprd11.prod.outlook.com>
 <20210922123931.GI327412@nvidia.com>
 <BN9PR11MB5433CE19425E85E7F52093278CA79@BN9PR11MB5433.namprd11.prod.outlook.com>
 <20210927115342.GW964074@nvidia.com>
 <BN9PR11MB5433502FEF11940984774F278CA79@BN9PR11MB5433.namprd11.prod.outlook.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <BN9PR11MB5433502FEF11940984774F278CA79@BN9PR11MB5433.namprd11.prod.outlook.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?UjDfeb01+UqvmaxxeRPCtEv07zQOBThVAwYy2zvpn2kMCPqBVt6XOktEmZy0?=
 =?us-ascii?Q?RxSXTHAfObOWz10WR5lA7Nb6+0MmqdQhGYbgWp684z2BvJepQ5EulWiRh66U?=
 =?us-ascii?Q?RItboFwJf2CxzVlHbsE8bwBjv9HQOG+Q9GkuZVdPuwK5om5AA12T/AkkBkAz?=
 =?us-ascii?Q?AzV6ydJuD/hauHI/91UwJjrJlbw+vni3x3xzu2AUd2SZzxbUIdNLA+LeOF1L?=
 =?us-ascii?Q?U8iBsWjWkShyVMZPlhZnflJSkajIwNJd11WWa5Lb9Jk3GTxUmwwVN4UTFdsZ?=
 =?us-ascii?Q?Qy+boR5as9p1JJOrGrnl2V/UkOTCJ0f3cFdr51QBYo4Voue10KtWouztYUSG?=
 =?us-ascii?Q?Mn5w57fUSf3/uSSVoCDaPiRD872ElJoogxr+1636exMJEtcgdcQkeLEAxJuP?=
 =?us-ascii?Q?Q26GB+MM/vpb1qaiSt6rb3InoOO47ae2citV4fA0BNiDZHrsozl8FBAu+syj?=
 =?us-ascii?Q?6qmyn8EY6/g+lZpDhBqDlS70ND9YspaLCYLvJJEJHzbyggmD30mfvqPfjQQ+?=
 =?us-ascii?Q?NzLGYdYNoUqvkv3MLx5iP3GAPx34bjdqEreAtz/W5j7hSG3HygDFLmcEVpnz?=
 =?us-ascii?Q?gf4ZkYYlKqlSyhUfzkeMA64E/kZ/0TraBvZmJsW9wztyN4ilQ5185RJs1QjV?=
 =?us-ascii?Q?n9CnMKiQDZpNzKRAP8UhvLAOdSEwT9RcZvySrWJtpUcMvySw2gW2GaIBZf54?=
 =?us-ascii?Q?0Z5TruMTcsJOwaxoAsfYUbX6lBTHK4ApJjELQuh2cTsl210kB8eNLZpLWtDD?=
 =?us-ascii?Q?f+C8+rZT+aLKXTxWQFdMYyhdTRH07aKv3jLfTuRw2hjL/VDIHGj1qxEcTyo1?=
 =?us-ascii?Q?y9pAgSNGWPomdrJDouDR5WdIaoCynKGeFeAkbrDBhC30/blU5shEgRogubx7?=
 =?us-ascii?Q?1HOZm/T24pVWrga0eraWCUBP1S3ACiTHvtoluShKXFMBmXqAehE5KdkWNsrU?=
 =?us-ascii?Q?Xz9GeUzpy/LdyjFvEmFMpql3ZU0+BoHfQJwFbKZin3GU2BIVjG6t8S03U7Ns?=
 =?us-ascii?Q?FPlNdJCSqNwByWPWPe64jTbTNlUVDapuEJiQwpZ5iBtfveCGLwGlvFpoMkQa?=
 =?us-ascii?Q?4DEbK4Dmbkr3ALK8Vo4llaQDT7DQlKAAv4TVYq8dsrYjOZNMBHderICBMbqT?=
 =?us-ascii?Q?U5zZhpQiLW80GqAq5x1WPA9JxpwW6zxcebpQGE4RGCn4gOH2I75bB206lFB6?=
 =?us-ascii?Q?gB22//83kU3Fy7kMWDJKkLHNE5OkHTODv7rpBITYLXADdOw49LAqoUvTHE2l?=
 =?us-ascii?Q?yae8Tv7VkXR1cwpBfZsAq7bw51gxB+Lctlm7eUvJxs4Z+sajtp4oi2HJuo6+?=
 =?us-ascii?Q?PBcSpPQ57opRcSmxMO/3ihQO?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 202543a6-b1f4-4557-9bf7-08d981b80e47
X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB5506.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Sep 2021 13:09:36.5833
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: eDmdSqHZv/2vDB69GOIa4WXkznhzLsFk5yQG6fCnJwZA1rSFT8g4nFKp7BkAE52s
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR12MB5539
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Mon, Sep 27, 2021 at 01:00:08PM +0000, Tian, Kevin wrote:

> > I think for such a narrow usage you should not change the struct
> > device_driver. Just have pci_stub call a function to flip back to user
> > mode.
> 
> Here we want to ensure that kernel dma should be blocked
> if the group is already marked for user-dma. If we just blindly
> do it for any driver at this point (as you commented earlier):
> 
> +       ret = iommu_set_kernel_ownership(dev);
> +       if (ret)
> +               return ret;
> 
> how would pci-stub reach its function to indicate that it doesn't 
> do dma and flip back?

> Do you envision a simpler policy that no driver can be bound
> to the group if it's already set for user-dma? what about vfio-pci
> itself?

Yes.. I'm not sure there is a good use case to allow the stub drivers
to load/unload while a VFIO is running. At least, not a strong enough
one to justify a global change to the driver core..

> > > +static int iommu_dev_viable(struct device *dev, void *data)
> > > +{
> > > +	enum dma_hint hint = *data;
> > > +	struct device_driver *drv = READ_ONCE(dev->driver);
> > 
> > Especially since this isn't locked properly or safe.
> 
> I have the same worry when copying from vfio. Not sure how
> vfio gets safe with this approach...

Fixing the locking in vfio_dev_viable is part of deleting the unbound
list. Once it properly uses the device_lock and doesn't race with the
driver core like this things are much better. Don't copy this stuff
into the iommu core without fixing it.

https://github.com/jgunthorpe/linux/commit/fa6abb318ccca114da12c0b5b123c99131ace926
https://github.com/jgunthorpe/linux/commit/45980bd90b023d1eea56df70d1c395bdf4cc7cf1

I can't remember if the above is contingent on some of the mdev
cleanups or not.. Have to get back to it.

Jason