Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Subject: Re: [PATCH] s390/vfio-ap: fix memory leak in mdev remove callback
To:     Cornelia Huck <cohuck@redhat.com>
Cc:     linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org,
        borntraeger@de.ibm.com, pasic@linux.vnet.ibm.com,
        jjherne@linux.ibm.com, jgg@nvidia.com, alex.williamson@redhat.com,
        kwankhede@nvidia.com, stable@vger.kernel.org,
        Tony Krowiak <akrowiak@stny.rr.com>
References: <20210505172826.105304-1-akrowiak@linux.ibm.com>
 <20210506122245.20f4ba21.cohuck@redhat.com>
From:   Tony Krowiak <akrowiak@linux.ibm.com>
Message-ID: <51acc512-58a9-2e69-d759-4efbbea941a8@linux.ibm.com>
Date:   Mon, 10 May 2021 13:44:16 -0400
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.8.1
MIME-Version: 1.0
In-Reply-To: <20210506122245.20f4ba21.cohuck@redhat.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
Content-Language: en-US
Precedence: bulk


On 5/6/21 6:22 AM, Cornelia Huck wrote:
> On Wed,  5 May 2021 13:28:26 -0400
> Tony Krowiak <akrowiak@linux.ibm.com> wrote:
>
>> The mdev remove callback for the vfio_ap device driver bails out with
>> -EBUSY if the mdev is in use by a KVM guest. The intended purpose was
>> to prevent the mdev from being removed while in use; however, returning a
>> non-zero rc does not prevent removal. This could result in a memory leak
>> of the resources allocated when the mdev was created. In addition, the
>> KVM guest will still have access to the AP devices assigned to the mdev
>> even though the mdev no longer exists.
>>
>> To prevent this scenario, cleanup will be done - including unplugging the
>> AP adapters, domains and control domains - regardless of whether the mdev
>> is in use by a KVM guest or not.
>>
>> Fixes: 258287c994de ("s390: vfio-ap: implement mediated device open callback")
>> Cc: stable@vger.kernel.org
>> Signed-off-by: Tony Krowiak <akrowiak@stny.rr.com>
>> Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com>
>> ---
>>   drivers/s390/crypto/vfio_ap_ops.c | 39 +++++++++++++++++++++++--------
>>   1 file changed, 29 insertions(+), 10 deletions(-)
>>
>> diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
>> index b2c7e10dfdcd..757166da947e 100644
>> --- a/drivers/s390/crypto/vfio_ap_ops.c
>> +++ b/drivers/s390/crypto/vfio_ap_ops.c
>> @@ -335,6 +335,32 @@ static void vfio_ap_matrix_init(struct ap_config_info *info,
>>   	matrix->adm_max = info->apxa ? info->Nd : 15;
>>   }
>>   
>> +static bool vfio_ap_mdev_has_crycb(struct ap_matrix_mdev *matrix_mdev)
>> +{
>> +	return (matrix_mdev->kvm && matrix_mdev->kvm->arch.crypto.crycbd);
>> +}
>> +
>> +static void vfio_ap_mdev_clear_apcb(struct ap_matrix_mdev *matrix_mdev)
>> +{
>> +	/*
>> +	 * If the KVM pointer is in the process of being set, wait until the
>> +	 * process has completed.
>> +	 */
>> +	wait_event_cmd(matrix_mdev->wait_for_kvm,
>> +		       !matrix_mdev->kvm_busy,
>> +		       mutex_unlock(&matrix_dev->lock),
>> +		       mutex_lock(&matrix_dev->lock));
>> +
>> +	if (vfio_ap_mdev_has_crycb(matrix_mdev)) {
>> +		matrix_mdev->kvm_busy = true;
>> +		mutex_unlock(&matrix_dev->lock);
>> +		kvm_arch_crypto_clear_masks(matrix_mdev->kvm);
>> +		mutex_lock(&matrix_dev->lock);
>> +		matrix_mdev->kvm_busy = false;
>> +		wake_up_all(&matrix_mdev->wait_for_kvm);
>> +	}
>> +}
> Looking at vfio_ap_mdev_unset_kvm(), do you need to unhook the kvm here
> as well?
>
> (Or can you maybe even combine the two functions into one?)

I contemplated just calling the vfio_ap_unset_kvm() function from
the vfio_ap_mdev_remove() function, but my thinking at the time
was that some of the other things done in the unset function, such
as kvm_put_kvm() etc., might cause problems.
After thinking about it some more, the vfio_ap_mdev_remove()
function will not get called until the vfio_ap_mdev_release()
callback is invoked unless something weird happens. Since the
remove callback gets rid of the mdev and the release callback
calls the vfio_ap_unset_kvm() function anyway, I now see no harm
in just calling the unset function from the remove callback and
eliminating the function above.

>
>> +
>>   static int vfio_ap_mdev_create(struct mdev_device *mdev)
>>   {
>>   	struct ap_matrix_mdev *matrix_mdev;
>> @@ -366,16 +392,9 @@ static int vfio_ap_mdev_remove(struct mdev_device *mdev)
>>   	struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
>>   
>>   	mutex_lock(&matrix_dev->lock);
>> -
>> -	/*
>> -	 * If the KVM pointer is in flux or the guest is running, disallow
>> -	 * un-assignment of control domain.
>> -	 */
>> -	if (matrix_mdev->kvm_busy || matrix_mdev->kvm) {
>> -		mutex_unlock(&matrix_dev->lock);
>> -		return -EBUSY;
>> -	}
>> -
>> +	WARN(vfio_ap_mdev_has_crycb(matrix_mdev),
>> +	     "Removing mdev leaves KVM guest without any crypto devices");
>> +	vfio_ap_mdev_clear_apcb(matrix_mdev);
>>   	vfio_ap_mdev_reset_queues(mdev);
>>   	list_del(&matrix_mdev->node);
>>   	kfree(matrix_mdev);