Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <b9575614-a234-0c36-7601-9c09d159c3b0@linux.ibm.com>
Date:   Wed, 27 Apr 2022 16:20:10 -0400
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.8.0
Subject: Re: [PATCH v6 15/21] KVM: s390: pci: add routines to start/stop
 interpretive execution
Content-Language: en-US
To:     Jason Gunthorpe <jgg@nvidia.com>
Cc:     linux-s390@vger.kernel.org, alex.williamson@redhat.com,
        cohuck@redhat.com, schnelle@linux.ibm.com, farman@linux.ibm.com,
        pmorel@linux.ibm.com, borntraeger@linux.ibm.com, hca@linux.ibm.com,
        gor@linux.ibm.com, gerald.schaefer@linux.ibm.com,
        agordeev@linux.ibm.com, svens@linux.ibm.com, frankja@linux.ibm.com,
        david@redhat.com, imbrenda@linux.ibm.com, vneethv@linux.ibm.com,
        oberpar@linux.ibm.com, freude@linux.ibm.com, thuth@redhat.com,
        pasic@linux.ibm.com, pbonzini@redhat.com, corbet@lwn.net,
        kvm@vger.kernel.org, linux-kernel@vger.kernel.org,
        linux-doc@vger.kernel.org
References: <20220426200842.98655-1-mjrosato@linux.ibm.com>
 <20220426200842.98655-16-mjrosato@linux.ibm.com>
 <20220427151400.GY2125828@nvidia.com>
From:   Matthew Rosato <mjrosato@linux.ibm.com>
In-Reply-To: <20220427151400.GY2125828@nvidia.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 4/27/22 11:14 AM, Jason Gunthorpe wrote:
> On Tue, Apr 26, 2022 at 04:08:36PM -0400, Matthew Rosato wrote:
> 
>> +int kvm_s390_pci_register_kvm(struct zpci_dev *zdev, struct kvm *kvm)
>> +{
>> +	if (!zdev)
>> +		return 0;
>> +
>> +	/*
>> +	 * Register device with this KVM (or remove the KVM association if 0).
>> +	 * If interpetation facilities are available, enable them and let
>> +	 * userspace indicate whether or not they will be used (specify SHM bit
>> +	 * to disable).
>> +	 */
>> +	if (kvm)
>> +		return register_kvm(zdev, kvm);
>> +	else
>> +		return unregister_kvm(zdev);
>> +}
>> +EXPORT_SYMBOL_GPL(kvm_s390_pci_register_kvm);
> 
> I think it is cleaner to expose both the register/unregister APIs and
> not multiplex them like this
> 

OK

>> +void kvm_s390_pci_clear_list(struct kvm *kvm)
>> +{
>> +	struct kvm_zdev *tmp, *kzdev;
>> +	LIST_HEAD(remove);
>> +
>> +	spin_lock(&kvm->arch.kzdev_list_lock);
>> +	list_for_each_entry_safe(kzdev, tmp, &kvm->arch.kzdev_list, entry)
>> +		list_move_tail(&kzdev->entry, &remove);
>> +	spin_unlock(&kvm->arch.kzdev_list_lock);
>> +
>> +	list_for_each_entry_safe(kzdev, tmp, &remove, entry)
>> +		unregister_kvm(kzdev->zdev);
> 
> Hum, I wonder if this is a mistake in kvm:
> 
> static void kvm_destroy_vm(struct kvm *kvm)
> {
> [..]
> 	kvm_arch_destroy_vm(kvm);
> 	kvm_destroy_devices(kvm);
> 
> kvm_destroy_devices() triggers the VFIO notifier with NULL. Indeed for
> correctness I would expect the VFIO users to have been notified to
> release the kvm before the kvm object becomes partially destroyed?
> 
> Maybe you should investigate re-ordering this at the KVM side and just
> WARN_ON(!list_empty) in the arch code?
> 
> (vfio has this odd usage model where it should use the kvm pointer
> without taking a ref on it so long as the unregister hasn't been
> called)
>

The issue there is that I am unregistering the notifier during 
close_device for each zPCI dev, which will have already happened -- so 
by the time we get to kvm_destroy_devices(), whether it's before or 
after kvm_arch_destroy_vm, there are no longer any zPCI notifiers 
registered that will trigger.

One way to solve this is to have the zpci close_device hook also trigger 
the work that a KVM_DEV_VFIO_GROUP_DEL would (if the device is being 
closed, the KVM association for that device isn't applicable anymore so 
go ahead and clean up).

Then, since we know each device will get closed (either by userspace or 
by kvm), I don't need something like kvm_s390_pci_clear_list at all.

> If you keep it like this then the locking in register/unregister looks
> not broad enough and has to cover the zdev->kzdev also.

But I would still need to revisit the locking with the above idea.

> 
> Overall I think it is OK designed like this, aside from the ugly
> symbol_get in vfio which I hope you can resolve.
> 
> Jason