Received-SPF: softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) client-ip=23.128.96.19;
Date:   Wed, 27 Apr 2022 12:14:00 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Matthew Rosato <mjrosato@linux.ibm.com>
Cc:     linux-s390@vger.kernel.org, alex.williamson@redhat.com,
        cohuck@redhat.com, schnelle@linux.ibm.com, farman@linux.ibm.com,
        pmorel@linux.ibm.com, borntraeger@linux.ibm.com, hca@linux.ibm.com,
        gor@linux.ibm.com, gerald.schaefer@linux.ibm.com,
        agordeev@linux.ibm.com, svens@linux.ibm.com, frankja@linux.ibm.com,
        david@redhat.com, imbrenda@linux.ibm.com, vneethv@linux.ibm.com,
        oberpar@linux.ibm.com, freude@linux.ibm.com, thuth@redhat.com,
        pasic@linux.ibm.com, pbonzini@redhat.com, corbet@lwn.net,
        kvm@vger.kernel.org, linux-kernel@vger.kernel.org,
        linux-doc@vger.kernel.org
Subject: Re: [PATCH v6 15/21] KVM: s390: pci: add routines to start/stop
 interpretive execution
Message-ID: <20220427151400.GY2125828@nvidia.com>
References: <20220426200842.98655-1-mjrosato@linux.ibm.com>
 <20220426200842.98655-16-mjrosato@linux.ibm.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20220426200842.98655-16-mjrosato@linux.ibm.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?HY1yu7bEu5xdA6izINQN5+R0jgiJXFDb0l4iS2/LB4QHAvyGtp8GC8pm4XXI?=
 =?us-ascii?Q?MLEGgetHk7QKVTOj+zSSl1fIowiQuDMIWNfj0bvwYdae97ItRAOsdpVyNjXA?=
 =?us-ascii?Q?Mhxfmctcb5iBqEBTxhnp72F1Ntd04G7umGbqBiXmpx0AR0gMA6KDLZqwRhBw?=
 =?us-ascii?Q?L8ipCUHmcfFuD5gf5HDeGXwkiQy+21qqvE9XqyIDFJ7O0ziySUeV/tzOspGG?=
 =?us-ascii?Q?v/W2ak1EgLAZQr6zkODnqXqh9vSiLENrELR/N23GfAT86wRNKOI040IaDkZ9?=
 =?us-ascii?Q?8cvgA6OLQNSs8lbplo7NwWADxdUTGg5WDjNvPUzXpvMd9ZgxNl9jyo4frSWz?=
 =?us-ascii?Q?e6BKFtHw7HfDq73b9Bl7QM3pAX96RkqhRh+wOsPV21BqDHZPMIEjZOfzhcJ9?=
 =?us-ascii?Q?unJqGXswlMFWqKzlTf1Ix1V2I5gS4Te7RtGkGJMKyH2qIZMWTDvqBoumVcFf?=
 =?us-ascii?Q?ure3WsGr8JwuTCnAP0DRbJ7vZjVt6+tZXQOOhgPcBGswQGypFFDSxL9FhAVg?=
 =?us-ascii?Q?5EbmSZ/2EY1nZo7gFCZqrhDsRiqCg74w3NtHTjnQvWqKGAYICRpwAmh3Oodv?=
 =?us-ascii?Q?TDcRf36R7m/nsE+DhwlocSF/f3LhFbp4w1iw955wOUqYSM6PVKXXzoVJ+UOW?=
 =?us-ascii?Q?RSpvcGyd+fLHBNe9nK7Gb+DfmVyl7cNSAUT4gK+7YIGti5CjaPMEKksFD7O4?=
 =?us-ascii?Q?6bY48E5o4uIduZR19hl6n1uIALf1YRfM51tA1+RAWuqGVKI2ItY5+cAcKALs?=
 =?us-ascii?Q?0dQ/lhomShoIMElKDA4KfAK3vq+Jn0CQvEOz1pUqSchU2zdhNPhGHH9Ekiwr?=
 =?us-ascii?Q?LEE4dvh0yD1O+vvANXQLfYLS80Cv2Oz2dxakwOLen9WqrO9sanCvWhRqPpnz?=
 =?us-ascii?Q?msUC7PR1IJXpNMLYoIjuoun3ZbltiTnIWd7Jrm7C9LkfOP6whqHk27xhAMey?=
 =?us-ascii?Q?bG6azAcMRtJe1T4T4U/5ZlKiPCVsQUkXCrP8BkVAl8ZEWj4RjubwQsKBgE9p?=
 =?us-ascii?Q?2qGXN2DeMKx5lSoRJ/Rqc6suTs27v1MC8HWzTs5tSLhZNsoPybCUtLS8r6gx?=
 =?us-ascii?Q?IBZWjxXIEB2/mNL2N36EGWKGNv327a+2UG0QtlpjKxkPOpfvdAVz2HDqrkBA?=
 =?us-ascii?Q?WQsWWfgRyZIp26QDAQeKC8yf4omZ86HhuYagwt4SxaemjyzS3ikWuilB2ZJF?=
 =?us-ascii?Q?5Fn5V10Woz//RkxxnTE+jHMOXF32serh665SVAaXl5jV9kfDimC9if2+D0wm?=
 =?us-ascii?Q?WWJtFeD+WnlYrLPxWTF/S8vEQ64TTpvhHMqvxciACTAnRAxSFgctvYXwSpPT?=
 =?us-ascii?Q?hq8HBaPn+NVKJw3d9B1X8aLgZUVfuffCugr8pIrg2rmiOTT4PxX9RtEsaYxO?=
 =?us-ascii?Q?AD5Md2gkr0dKGDfwgyJciPfyjUGbvdNZ6a/QxGfPWvw1rUiQkOiUSjcNVP17?=
 =?us-ascii?Q?KzLAxZyJXRCXdE4V6cU5cld/eyRhFfTUzyfMLl6h3x4cLcsixMHw/1pYItzG?=
 =?us-ascii?Q?ZE0FXSL5jd374saFv94tTM+h6qhoAdtUGV6m/D8SI39ceEblV1zL3lmTgDQO?=
 =?us-ascii?Q?UysXNAzG+yaR77SjNDRG2LjWP6ygT8+edaXAwW6Gwelzy6lxK5kv2ULrGoPh?=
 =?us-ascii?Q?cMjKfhVU5jBibpY/Fqe2MmYfmuXjAO5qw4u5BOYECG2vHPU1JVyesCHWm65Y?=
 =?us-ascii?Q?5lKKH1dpNrCE0+TA4uab75JSN8HxLyRa3AXtJlvPcWaytyyvK003M9ET+9Hj?=
 =?us-ascii?Q?4FL7NJTlcw=3D=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: a3afe478-331a-47fd-d379-08da28608f5e
X-MS-Exchange-CrossTenant-AuthSource: MN2PR12MB4192.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Apr 2022 15:14:01.6017
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: WkObWjOcs8sXpi6DnpO8wz5RAs2cQWT+f9do+/boBCDMKi3Cxz7qmjMjBv6MtCS/
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5077
X-Spam-Status: No, score=-2.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,RDNS_NONE,SPF_HELO_NONE autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Tue, Apr 26, 2022 at 04:08:36PM -0400, Matthew Rosato wrote:

> +int kvm_s390_pci_register_kvm(struct zpci_dev *zdev, struct kvm *kvm)
> +{
> +	if (!zdev)
> +		return 0;
> +
> +	/*
> +	 * Register device with this KVM (or remove the KVM association if 0).
> +	 * If interpetation facilities are available, enable them and let
> +	 * userspace indicate whether or not they will be used (specify SHM bit
> +	 * to disable).
> +	 */
> +	if (kvm)
> +		return register_kvm(zdev, kvm);
> +	else
> +		return unregister_kvm(zdev);
> +}
> +EXPORT_SYMBOL_GPL(kvm_s390_pci_register_kvm);

I think it is cleaner to expose both the register/unregister APIs and
not multiplex them like this

> +void kvm_s390_pci_clear_list(struct kvm *kvm)
> +{
> +	struct kvm_zdev *tmp, *kzdev;
> +	LIST_HEAD(remove);
> +
> +	spin_lock(&kvm->arch.kzdev_list_lock);
> +	list_for_each_entry_safe(kzdev, tmp, &kvm->arch.kzdev_list, entry)
> +		list_move_tail(&kzdev->entry, &remove);
> +	spin_unlock(&kvm->arch.kzdev_list_lock);
> +
> +	list_for_each_entry_safe(kzdev, tmp, &remove, entry)
> +		unregister_kvm(kzdev->zdev);

Hum, I wonder if this is a mistake in kvm:

static void kvm_destroy_vm(struct kvm *kvm)
{
[..]
	kvm_arch_destroy_vm(kvm);
	kvm_destroy_devices(kvm);

kvm_destroy_devices() triggers the VFIO notifier with NULL. Indeed for
correctness I would expect the VFIO users to have been notified to
release the kvm before the kvm object becomes partially destroyed?

Maybe you should investigate re-ordering this at the KVM side and just
WARN_ON(!list_empty) in the arch code?

(vfio has this odd usage model where it should use the kvm pointer
without taking a ref on it so long as the unregister hasn't been
called)

If you keep it like this then the locking in register/unregister looks
not broad enough and has to cover the zdev->kzdev also.

Overall I think it is OK designed like this, aside from the ugly
symbol_get in vfio which I hope you can resolve.

Jason