Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Mon, 16 May 2022 20:11:06 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Alex Williamson <alex.williamson@redhat.com>
Cc:     Matthew Rosato <mjrosato@linux.ibm.com>,
        linux-s390@vger.kernel.org, cohuck@redhat.com,
        schnelle@linux.ibm.com, farman@linux.ibm.com, pmorel@linux.ibm.com,
        borntraeger@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com,
        gerald.schaefer@linux.ibm.com, agordeev@linux.ibm.com,
        svens@linux.ibm.com, frankja@linux.ibm.com, david@redhat.com,
        imbrenda@linux.ibm.com, vneethv@linux.ibm.com,
        oberpar@linux.ibm.com, freude@linux.ibm.com, thuth@redhat.com,
        pasic@linux.ibm.com, pbonzini@redhat.com, corbet@lwn.net,
        kvm@vger.kernel.org, linux-kernel@vger.kernel.org,
        linux-doc@vger.kernel.org
Subject: Re: [PATCH v7 17/22] vfio-pci/zdev: add open/close device hooks
Message-ID: <20220516231106.GQ1343366@nvidia.com>
References: <20220513191509.272897-1-mjrosato@linux.ibm.com>
 <20220513191509.272897-18-mjrosato@linux.ibm.com>
 <20220516172734.GE1343366@nvidia.com>
 <7a31ec36-ceaf-dcef-8bd0-2b4732050aed@linux.ibm.com>
 <20220516183558.GN1343366@nvidia.com>
 <20220516133839.7e116489.alex.williamson@redhat.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20220516133839.7e116489.alex.williamson@redhat.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?BKPAzHcLCP0NSizlKu6eRB/a9D1wCDxeVgchrHQud+/RcoLG0kYhhVZCJhiV?=
 =?us-ascii?Q?LjSIozloLEIhNiTN/xSI/2wF/n2jrRR3Z0JZbVMiUYtmMi5TzzMCBcYJnjvm?=
 =?us-ascii?Q?tQc0E6LlRwWq72Uq33V51YOID6JxVm/jkfem7ls2ze9W10Mi4xIfnBQ1W0Q+?=
 =?us-ascii?Q?otv5uobI1PI+0VjDIMa98bctqz7rVn9OYG97FRNIRotz9JHIsyE6RkD7j+in?=
 =?us-ascii?Q?rZ+di915Mb9k0tQnSXFzKlR234K45AWVuWy9NQa1zfeRsdfiraaDO4VFbNS/?=
 =?us-ascii?Q?7lyyWONR+bQAfRA2QQi55qaHFiIruwu6ZZGKac0ZhtARSajQdYbrkGNhKKQo?=
 =?us-ascii?Q?F99OrUQqHhHU/03R+ZSye7P5cBAtm/fNBG1HMhYd2lN3ahcqziGdgRS+Zrxj?=
 =?us-ascii?Q?96Z4Z4GV1mIEsZjFrlheCkuSO9W0F/PL7nS63OFKUFn+0zVa4TofRDeXiL/x?=
 =?us-ascii?Q?T6NtbWYmz0hYzhQRuat4Qp52WPbzzA645PhidZ8tcParn3mJR4T03FnSEe9v?=
 =?us-ascii?Q?bW0nHheJW+qS82vhkRQjwofKeGMJ/4sxcqOILx/tVEfUsGUOfR8L9ciXZylS?=
 =?us-ascii?Q?IDdR6SyzwvWQosb1z5ZpY94Q5zHfTTDwlVm+b4MSNiBfu4Q1K/DsYokZZMZR?=
 =?us-ascii?Q?kRqtFa86NFDeGKIy3YbPVEmzM4W+KjDGkz9CsYI5aF7gumHCxfkOLPBz84he?=
 =?us-ascii?Q?yVjTc4329EYtrPKsQR4Xv4WKeQgLxE3J0BE5RChCvuXiEf0T2Ke9RQBu4bOa?=
 =?us-ascii?Q?FPFTmLQp7Rukhb/5TXxwfAds66KLElsN5lkFfZSYyIlPe/rQhBxSzl2i1o1U?=
 =?us-ascii?Q?ZryYROk0uRmllSpbBbEyo8z2KMvIbnR7QXg5ChJSyq767nIqmaSUmceos6cj?=
 =?us-ascii?Q?tnxTkgFPChPfEkpGcLWbAILqpvLgjHd3kY4k0yLurfJjgnJ8yEdEbtXbq0VD?=
 =?us-ascii?Q?tb74Ey0CEc82YkDyw1ZxcZsGlHk267ecFj0i49By9OfEMDpwrRqiNon+DDfe?=
 =?us-ascii?Q?S2kIsrAFkXl7U7vB26FhJq1W1ws5pTGhnVhM8d/iigdVy3tymEqoUrrKh3yo?=
 =?us-ascii?Q?rgCRYBDKpA4DwIA0AEcJWF9wZ2v4mzA1IkcXD2eOu4/3OhLatf0t97ffezhv?=
 =?us-ascii?Q?6XmpPmMSBnc2WwMZzMEH14tDx3S9ptiKZzrNfpOLtTMlrENenLtKOEaD9x9D?=
 =?us-ascii?Q?67NAa1PZLR96SS8Y59RJUd63W1Kt5asBfHDruF7oTqKecF/HVovTYqV67ASj?=
 =?us-ascii?Q?CUGFGXZ2aj8qVVAkhpxc9tirtcY5n7hiByp5aSN0CT14Ntkap3HdmVD9/GVQ?=
 =?us-ascii?Q?vBbyroIoUXP3hsx2C7L1bkZWZjIZyZgDrK7an+nRHkx03ITUL+kWxFtJhkN9?=
 =?us-ascii?Q?D0vApRL79p1mU/dr9gg5uNxS3vDc8fQSjlgqz8ElLcas45ZvwtjGtKCJdIXW?=
 =?us-ascii?Q?pTle4ubBp3UNtIHwxOpGWahxZcppPRzVNiekKtO9riyBRMuh/oEZ5Lgme+r7?=
 =?us-ascii?Q?+h7wxXLRGcoBphh+vZPTMU6Z8KkQWi/VGVAJRYNi+u2LprRwcfaQFRIIG4iD?=
 =?us-ascii?Q?uPmBdLUuiXP72fmVymMwJ8mef9mJcIn8hlTxQgntrD65lOeo7aCwBmQpeLIO?=
 =?us-ascii?Q?VVmpDK79TLPvpIbKvGJnZ/Q75VEMqYCrref+hBp7mLxsDswZgvXwwdbP63+T?=
 =?us-ascii?Q?vqvIzF35oKcXAKFw2+hhPV3Hv8b/ieU2cnTtTKpUraakyMWrzTuOnd1hYeP2?=
 =?us-ascii?Q?3amA4t/eIg=3D=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: ed4726cd-f740-4899-f5ee-08da37915bc4
X-MS-Exchange-CrossTenant-AuthSource: MN2PR12MB4192.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 May 2022 23:11:07.7016
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: OR7DCa3+3G/v6/ZquAMcSosEgnl/1v2ucpZu1cwRy2ml+B9f09sriGoNOZoTqNGF
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5948
X-Spam-Status: No, score=-1.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Mon, May 16, 2022 at 01:38:39PM -0600, Alex Williamson wrote:
> On Mon, 16 May 2022 15:35:58 -0300
> Jason Gunthorpe <jgg@nvidia.com> wrote:
> 
> > On Mon, May 16, 2022 at 02:30:46PM -0400, Matthew Rosato wrote:
> > 
> > > Conceptually I think this would work for QEMU anyway (it always sets the kvm
> > > before we open the device).  I tried to test the idea quickly but couldn't
> > > get the following to apply on vfio-next or your vfio_group_locking -- but I
> > > understand what you're trying to do so I'll re-work and try it out.  
> > 
> > I created it on 8c9350e9bf43de1ebab3cc8a80703671e6495ab4 which is the
> > vfio_group_locking.. I can send you a github if it helps
> > https://github.com/jgunthorpe/linux/commits/vfio_group_lockin
> > 
> > > @Alex can you think of any usecase/reason why we would need to be able to
> > > set the KVM sometime after the device was opened?  Doing something like
> > > below would break that, as this introduces the assumption that the group is
> > > associated with the KVM before the device is opened (and if it's not, the
> > > device open fails).  
> > 
> > Keep in mind that GVT already hard requires this ordering to even
> > allow open_device to work - so it already sets a floor for what
> > userspace can do..
> 
> How is this going to work when vfio devices are exposed directly?  We
> currently have a strict ordering through the group to get to the
> device, and it's therefore a reasonable requirement for userspace to
> register the group with kvm before opening the device.  Is the notifier
> and async KVM registration necessary to support this dependency with
> direct device access?  I don't have as clear a picture of the ordering
> with with direct access.  Thanks,

With the device FD there is already a zombie state after open("/dev/...") but
before the internal open_device op is called. This state ends after
the iommufd is assigned, then we can go to open_device.

It is reasonable that the KVM would have to be setup before assigning
the iommfd to the device fd.

Jason