Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Wed, 22 Sep 2021 09:39:31 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     "Tian, Kevin" <kevin.tian@intel.com>
Cc:     "Liu, Yi L" <yi.l.liu@intel.com>,
        "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
        "hch@lst.de" <hch@lst.de>,
        "jasowang@redhat.com" <jasowang@redhat.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "jean-philippe@linaro.org" <jean-philippe@linaro.org>,
        "parav@mellanox.com" <parav@mellanox.com>,
        "lkml@metux.net" <lkml@metux.net>,
        "pbonzini@redhat.com" <pbonzini@redhat.com>,
        "lushenming@huawei.com" <lushenming@huawei.com>,
        "eric.auger@redhat.com" <eric.auger@redhat.com>,
        "corbet@lwn.net" <corbet@lwn.net>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "yi.l.liu@linux.intel.com" <yi.l.liu@linux.intel.com>,
        "Tian, Jun J" <jun.j.tian@intel.com>, "Wu, Hao" <hao.wu@intel.com>,
        "Jiang, Dave" <dave.jiang@intel.com>,
        "jacob.jun.pan@linux.intel.com" <jacob.jun.pan@linux.intel.com>,
        "kwankhede@nvidia.com" <kwankhede@nvidia.com>,
        "robin.murphy@arm.com" <robin.murphy@arm.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "iommu@lists.linux-foundation.org" <iommu@lists.linux-foundation.org>,
        "dwmw2@infradead.org" <dwmw2@infradead.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
        "david@gibson.dropbear.id.au" <david@gibson.dropbear.id.au>,
        "nicolinc@nvidia.com" <nicolinc@nvidia.com>
Subject: Re: [RFC 06/20] iommu: Add iommu_device_init[exit]_user_dma
 interfaces
Message-ID: <20210922123931.GI327412@nvidia.com>
References: <20210919063848.1476776-1-yi.l.liu@intel.com>
 <20210919063848.1476776-7-yi.l.liu@intel.com>
 <20210921170943.GS327412@nvidia.com>
 <BN9PR11MB5433DA330D4583387B59AA7F8CA29@BN9PR11MB5433.namprd11.prod.outlook.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <BN9PR11MB5433DA330D4583387B59AA7F8CA29@BN9PR11MB5433.namprd11.prod.outlook.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?flNm2Bh7f2SQ83HwLHAo/PVQrmgziJzYZlYm6KOr1wDooJ5zH2P6s5k2LpEW?=
 =?us-ascii?Q?K1fZWrppQCDsbYmjc6DK2zMfpbXnjLWh51IKmU88tUljwv5+wdkaxlILRgcg?=
 =?us-ascii?Q?EyDX7imtcCzWatS/scYwrO8sO9HTRg1bpfy0OhSgZvRR0UXcRCKQW/VU1RbS?=
 =?us-ascii?Q?KPnHmNbVRCGcRRGeslEAzJ4kKRBMVWxbc3+BiMD5BF0TVjkC1TLMD844Sazn?=
 =?us-ascii?Q?d/VjSw4ZQS6ndcZQHkQPeNm6RVEs3yuOZ+fICrUQQZsdKDOZNzRUo9ub9Cvw?=
 =?us-ascii?Q?rXhJ9haw3aQpbFNvFtyoinL3kBmJloRRvk2Ql6eFUDQ2P71sabG5/MGn5NPO?=
 =?us-ascii?Q?70CnXkdJxdnlCJDILq3EKsS3cIQxYpbG0WtGw7Q1tYUq2bLov47rUESv/4Eu?=
 =?us-ascii?Q?uI7QETCc8N52abfgQWXhFTgPPduimHsXT+XZobA6JQe+giyWbjWk8vr/U7u/?=
 =?us-ascii?Q?ygs/FiOwKR7g9FJywCKVzJThncCkbCZUpw1dyC4nlwSbwRNaorJAmAVEhShX?=
 =?us-ascii?Q?abbGB5YfA9vNVFwq0yX6oE023VA0OkwflYPHBxyAHCyFpWNC3sipGb4Q/0xL?=
 =?us-ascii?Q?T0wpce3Vp7TzmyYdZAwSxOmEnNf7z1z4aY6CHmq0hwK5x6GWkF/Dro+/Odns?=
 =?us-ascii?Q?Ig4TTsDHzLR0E/xFnBItSPLRsHFsR71jVrLdu/LiqK1+Xj+TS1SRX0pg1bLW?=
 =?us-ascii?Q?bla0O3EZYHZVfOHUDi+TS/OcUPh33sZC8DC+YrF0RZhKpzXh1FYHsgxXyYZ9?=
 =?us-ascii?Q?DfciO6LVRm81Oz1zSBL0/cbfn4W5sF7AR4b4EdI6kgXCi1QLQ6BMQygJP2i9?=
 =?us-ascii?Q?V5yFVsGVYBv5gFwJD67PhFGWqEr6h666H/ciEIaUMcT6uzc+WPHWu1ni8+3K?=
 =?us-ascii?Q?x2S/vpSXEg2BLKKsB8BNOWl4ax5L57Knh/nDkY1bigaNPYJOmj4kb7cZCViL?=
 =?us-ascii?Q?JXOgcS1bjQie8m+l3ZBYa6OkonPbw3hT2LtcKGYiAnJkT7hz3s745fS13utQ?=
 =?us-ascii?Q?6zfD4qu3PdZuDoVsf5ibaKqttSia+gJ6H+1YsFj0iQNEaSHYdbxYpd7IpCnk?=
 =?us-ascii?Q?Va+sOJLqRIw+pi6gtZilJxLTrHJ0YsFX/uYdkWLsIFfGyBgtmjbnlOpnzKu0?=
 =?us-ascii?Q?Dlwxsqu7CQ5K7JBgjq8WirfIM6SkahdPmszztGI/mey7NgQrDijfovGt2S4x?=
 =?us-ascii?Q?xi0hoZJ8ahWd/8syBPLcNw7RADMnEgMzmT5RcEHbadS6VMxbQ3Sog0OoPaXN?=
 =?us-ascii?Q?KjbT0vAxbdrIoSNnZkTLRuO9ny9OwvJ0uYsNq/3EOTLdXJskxZv3BWQkvyso?=
 =?us-ascii?Q?tZtpNV7tMPauNVEUogeOW/ur?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 4c03ed3b-8908-4e17-0ddf-08d97dc606ea
X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB5506.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Sep 2021 12:39:32.6487
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: W7J7jW0oMgg9Q2P1rcJ8DOtZYYJtNWOVtZ0t4OLCyQCp9epp0K4OuRP/TuzriF9p
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5063
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Wed, Sep 22, 2021 at 01:47:05AM +0000, Tian, Kevin wrote:

> > IIRC in VFIO the container is the IOAS and when the group goes to
> > create the device fd it should simply do the
> > iommu_device_init_user_dma() followed immediately by a call to bind
> > the container IOAS as your #3.
> 
> a slight correction.
> 
> to meet vfio semantics we could do init_user_dma() at group attach
> time and then call binding to container IOAS when the device fd
> is created. This is because vfio requires the group in a security context
> before the device is opened. 

Is it? Until a device FD is opened the group fd is kind of idle, right?

> > Ie the basic flow would see the driver core doing some:
> 
> Just double confirm. Is there concern on having the driver core to
> call iommu functions? 

It is always an interesting question, but I'd say iommu is
foundantional to Linux and if it needs driver core help it shouldn't
be any different from PM, pinctl, or other subsystems that have
inserted themselves into the driver core.

Something kind of like the below.

If I recall, once it is done like this then the entire iommu notifier
infrastructure can be ripped out which is a lot of code.


diff --git a/drivers/base/dd.c b/drivers/base/dd.c
index 68ea1f949daa90..e39612c99c6123 100644
--- a/drivers/base/dd.c
+++ b/drivers/base/dd.c
@@ -566,6 +566,10 @@ static int really_probe(struct device *dev, struct device_driver *drv)
                goto done;
        }
 
+       ret = iommu_set_kernel_ownership(dev);
+       if (ret)
+               return ret;
+
 re_probe:
        dev->driver = drv;
 
@@ -673,6 +677,7 @@ static int really_probe(struct device *dev, struct device_driver *drv)
                dev->pm_domain->dismiss(dev);
        pm_runtime_reinit(dev);
        dev_pm_set_driver_flags(dev, 0);
+       iommu_release_kernel_ownership(dev);
 done:
        return ret;
 }
@@ -1214,6 +1219,7 @@ static void __device_release_driver(struct device *dev, struct device *parent)
                        dev->pm_domain->dismiss(dev);
                pm_runtime_reinit(dev);
                dev_pm_set_driver_flags(dev, 0);
+               iommu_release_kernel_ownership(dev);
 
                klist_remove(&dev->p->knode_driver);
                device_pm_check_callbacks(dev);