Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
Date:   Tue, 8 Feb 2022 14:51:41 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Alex Williamson <alex.williamson@redhat.com>
Cc:     Matthew Rosato <mjrosato@linux.ibm.com>,
        linux-s390@vger.kernel.org, cohuck@redhat.com,
        schnelle@linux.ibm.com, farman@linux.ibm.com, pmorel@linux.ibm.com,
        borntraeger@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com,
        gerald.schaefer@linux.ibm.com, agordeev@linux.ibm.com,
        frankja@linux.ibm.com, david@redhat.com, imbrenda@linux.ibm.com,
        vneethv@linux.ibm.com, oberpar@linux.ibm.com, freude@linux.ibm.com,
        thuth@redhat.com, pasic@linux.ibm.com, kvm@vger.kernel.org,
        linux-kernel@vger.kernel.org
Subject: Re: [PATCH v3 24/30] vfio-pci/zdev: wire up group notifier
Message-ID: <20220208185141.GH4160@nvidia.com>
References: <20220204211536.321475-1-mjrosato@linux.ibm.com>
 <20220204211536.321475-25-mjrosato@linux.ibm.com>
 <20220208104319.4861fb22.alex.williamson@redhat.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20220208104319.4861fb22.alex.williamson@redhat.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?y/XwfOuGXHPhbHmsEiRvwaazJ6UbxMYsqRMusDgJC6kJ0yfR7cnau5AUYF5X?=
 =?us-ascii?Q?SdgmY0f5CslqRhrem8sEmbieuMRxNAi1UT4CVe5dKkCbKYf04npnJ5Eeyl/A?=
 =?us-ascii?Q?nq3tv7h8vNpmuVgqqGq8Zq1oSsjL4YhDj6E615mpCDZRwxpRCAFTHPoEU0Fu?=
 =?us-ascii?Q?Dfet3w1i5L+qgmgMvmueMXTQu5s16xKIazP2J8RyYF5b7E94bykdxCt2m1z2?=
 =?us-ascii?Q?aRNWSkOSs43Yq6M+3cxsO+Tl5W8JTx9/OYUMIYc45Vm+9VHGiPiaj8LNBDlI?=
 =?us-ascii?Q?8qqAoqZFxt+XevbMV/dD16ljcUB23AHs7ZWnhmLo/UBjrKzKV/9luFuTpcY+?=
 =?us-ascii?Q?rvIo+Tu+ivvlYNsppxtvGJfFcrFsTy2O3gzTEAlhuQZ+wgF1UDIzQc+hXeJ1?=
 =?us-ascii?Q?QBB0WqAFHakUlunpycfBXhKL7J3RU36m1JpOEr4ENYSl1UvK4pQCi1K7oBIB?=
 =?us-ascii?Q?FLu21OOOSvRRHTEr10UZLLBPBKmrWI3zOrExWUQXuju4HqLrIU3OjYKOAilO?=
 =?us-ascii?Q?greoI9gAzKpIq4Qijz/5je8pr7EJ0qeX21/z7gV5U+8LSgDUEw98u6LtCZV1?=
 =?us-ascii?Q?bpvgylLu07TnzgW5MZlnTAir2qiyfGqyV43eQMXNC1RsPUeP8W5Hm4JaPKUZ?=
 =?us-ascii?Q?z4zio0HfOdTDaZwXFr1lD/07cBL+GsB5L5kfs0tjvjIbYeBoY3cyM6/cbUaN?=
 =?us-ascii?Q?qZ4xMsXGShtfxEAzqm3hVEAYd6ROTjZXaSJq5SmCmg/r0WX8ojYrOvMIcM+p?=
 =?us-ascii?Q?j5LTFQANOyAsf38luzowAjFAPgzc2cucbKKqej4c0iKQd9tEaTjhiZjtQTPu?=
 =?us-ascii?Q?X8Mk7w876AuZ7rh7JRNOuRw+EI+W/BtGqg2MiFN2Tmp4WloG9pha9KhxZBbk?=
 =?us-ascii?Q?Y0f6eI8KRNlVjLT1Sy2jquzmSCJfSO65GQZPEwF/a3XefevUZ+qHLEq1vONZ?=
 =?us-ascii?Q?HbyEdXT9Lg/CP90qb3RdPetIPKF0KJmUrbm5jeX8XUKkzOXm3ufA7qKk8EDo?=
 =?us-ascii?Q?QV+KnQQK8rDq/qRUPDmOyQTrTPhjCJBgC1q70JgviAbn43vh8YeCDByp1+mp?=
 =?us-ascii?Q?SoGKITcOBg4eklLdjzb3Qu8kYPMxd/fkLDL9qyrE+vkdg9ecvc8JLnP/h+t/?=
 =?us-ascii?Q?RheXKwipLBi4daoNJYL/DRegx1kDA3+D34207Vza5sqW2/rvUK4HdhdOqIOc?=
 =?us-ascii?Q?7MOsFRJCdyXolyDGrQtyxLcHV440jK9XK1Tuz92Q8pKgq/dQz45kbGJ9HVDN?=
 =?us-ascii?Q?fI0WCwJH5dY5EHHQFptFFc4Nd7NDhshVfsyeUlNmgBwlj7f9wXXBOedehubf?=
 =?us-ascii?Q?5kWw5ohYXHxbhSTsQ2igkl3VzPphxHhAv9RhTHVzpyCgf2CfPoTYPDTFOF3r?=
 =?us-ascii?Q?TEP0Lj/RB4WXeVklpEwk2HIVF1OGJ9cdGn65fnSxR4Wjmz4juryydCiOl6ks?=
 =?us-ascii?Q?y9F/tFBg9effcHaoZxTSCq+gppXfYCgT32tE0DFrPPaCIOrn0esuwIYRHcEF?=
 =?us-ascii?Q?ofeB8SaVN+fjvI/YKdMch7RZ7zyf5hbNY2CIdlUBsuqS47m8Ptcr27XklEGV?=
 =?us-ascii?Q?92dOrr5VCDmKzmWYUaM=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 8d814342-e9b8-4685-c692-08d9eb340c4a
X-MS-Exchange-CrossTenant-AuthSource: MN2PR12MB4192.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Feb 2022 18:51:42.8876
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: RcYLXCpTU3DqEpdPpzS9j8kcyJ+3tT6JzuzEbDwWxcNH+iJcEj9cxZtAshCmQNFF
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR12MB4495
X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,RDNS_NONE,SPF_HELO_NONE,T_SCC_BODY_TEXT_LINE
	autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Tue, Feb 08, 2022 at 10:43:19AM -0700, Alex Williamson wrote:
> On Fri,  4 Feb 2022 16:15:30 -0500
> Matthew Rosato <mjrosato@linux.ibm.com> wrote:
> 
> > KVM zPCI passthrough device logic will need a reference to the associated
> > kvm guest that has access to the device.  Let's register a group notifier
> > for VFIO_GROUP_NOTIFY_SET_KVM to catch this information in order to create
> > an association between a kvm guest and the host zdev.
> > 
> > Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
> >  arch/s390/include/asm/kvm_pci.h  |  2 ++
> >  drivers/vfio/pci/vfio_pci_core.c |  2 ++
> >  drivers/vfio/pci/vfio_pci_zdev.c | 46 ++++++++++++++++++++++++++++++++
> >  include/linux/vfio_pci_core.h    | 10 +++++++
> >  4 files changed, 60 insertions(+)
> > 
> > diff --git a/arch/s390/include/asm/kvm_pci.h b/arch/s390/include/asm/kvm_pci.h
> > index e4696f5592e1..16290b4cf2a6 100644
> > +++ b/arch/s390/include/asm/kvm_pci.h
> > @@ -16,6 +16,7 @@
> >  #include <linux/kvm.h>
> >  #include <linux/pci.h>
> >  #include <linux/mutex.h>
> > +#include <linux/notifier.h>
> >  #include <asm/pci_insn.h>
> >  #include <asm/pci_dma.h>
> >  
> > @@ -32,6 +33,7 @@ struct kvm_zdev {
> >  	u64 rpcit_count;
> >  	struct kvm_zdev_ioat ioat;
> >  	struct zpci_fib fib;
> > +	struct notifier_block nb;
> >  };
> >  
> >  int kvm_s390_pci_dev_open(struct zpci_dev *zdev);
> > diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
> > index f948e6cd2993..fc57d4d0abbe 100644
> > +++ b/drivers/vfio/pci/vfio_pci_core.c
> > @@ -452,6 +452,7 @@ void vfio_pci_core_close_device(struct vfio_device *core_vdev)
> >  
> >  	vfio_pci_vf_token_user_add(vdev, -1);
> >  	vfio_spapr_pci_eeh_release(vdev->pdev);
> > +	vfio_pci_zdev_release(vdev);
> >  	vfio_pci_core_disable(vdev);
> >  
> >  	mutex_lock(&vdev->igate);
> > @@ -470,6 +471,7 @@ EXPORT_SYMBOL_GPL(vfio_pci_core_close_device);
> >  void vfio_pci_core_finish_enable(struct vfio_pci_core_device *vdev)
> >  {
> >  	vfio_pci_probe_mmaps(vdev);
> > +	vfio_pci_zdev_open(vdev);
> >  	vfio_spapr_pci_eeh_open(vdev->pdev);
> >  	vfio_pci_vf_token_user_add(vdev, 1);
> >  }
> 
> If this handling were for a specific device, I think we'd be suggesting
> this is the point at which we cross over to a vendor variant making use
> of vfio-pci-core rather than hooking directly into the core code. 

Personally, I think it is wrong layering for VFIO to be aware of KVM
like this. This marks the first time that VFIO core code itself is
being made aware of the KVM linkage.

It copies the same kind of design the s390 specific mdev use of
putting VFIO in charge of KVM functionality. If we are doing this we
should just give up and admit that KVM is a first-class part of struct
vfio_device and get rid of the notifier stuff too, at least for s390.

Reading the patches and descriptions pretty much everything is boiling
down to 'use vfio to tell the kvm architecture code to do something' -
which I think needs to be handled through a KVM side ioctl.

Or, at the very least, everything needs to be described in some way
that makes it clear what is happening to userspace, without kvm,
through these ioctls.

This seems especially true now that it seems s390 PCI support is
almost truely functional, with actual new userspace instructions to
issue MMIO operations that work outside of KVM. 

I'm not sure how this all fits together, but I would expect an outcome
where DPDK could run on these new systems and not have to know
anything more about s390 beyond using the proper MMIO instructions via
some compilation time enablement.

(I've been reviewing s390 patches updating rdma for a parallel set of
stuff)

> this is meant to extend vfio-pci proper for the whole arch.  Is there a
> compromise in using #ifdefs in vfio_pci_ops to call into zpci specific
> code that implements these arch specific hooks and the core for
> everything else?  SPAPR code could probably converted similarly, it
> exists here for legacy reasons. [Cc Jason]

I'm not sure I get what you are suggesting? Where would these ifdefs
be?

> Also, please note the DEVICE_FEATURE generalizations in the latest
> series from NVIDIA for mlx5 migration support:
 
> https://lore.kernel.org/all/20220207172216.206415-8-yishaih@nvidia.com/

Yes, please don't implement a bunch of new FEATURE code without taking
the cleanup patches for feature support from that series too.

I can put them on a branch for you if you needed.

Jason