Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Subject: Re: [PATCH 1/1] s390/vfio-ap: fix circular lockdep when
 setting/clearing crypto masks
To:     Halil Pasic <pasic@linux.ibm.com>
Cc:     Cornelia Huck <cohuck@redhat.com>, linux-s390@vger.kernel.org,
        linux-kernel@vger.kernel.org, kvm@vger.kernel.org,
        stable@vger.kernel.org, borntraeger@de.ibm.com,
        kwankhede@nvidia.com, pbonzini@redhat.com,
        alex.williamson@redhat.com, pasic@linux.vnet.ibm.com
References: <20210209194830.20271-1-akrowiak@linux.ibm.com>
 <20210209194830.20271-2-akrowiak@linux.ibm.com>
 <20210210115334.46635966.cohuck@redhat.com>
 <20210210162429.261fc17c.pasic@linux.ibm.com>
 <20210210163237.315d9a68.pasic@linux.ibm.com>
 <59e8f084-c9ec-ce25-2326-b206e30d04d0@linux.ibm.com>
 <20210210234606.1d0dbdec.pasic@linux.ibm.com>
From:   Tony Krowiak <akrowiak@linux.ibm.com>
Message-ID: <8c461602-8c2c-4dd9-1d2b-5e424fc701f8@linux.ibm.com>
Date:   Thu, 11 Feb 2021 09:21:26 -0500
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101
 Thunderbird/68.5.0
MIME-Version: 1.0
In-Reply-To: <20210210234606.1d0dbdec.pasic@linux.ibm.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Content-Language: en-US
Precedence: bulk


On 2/10/21 5:46 PM, Halil Pasic wrote:
> On Wed, 10 Feb 2021 17:05:48 -0500
> Tony Krowiak <akrowiak@linux.ibm.com> wrote:
>
>> On 2/10/21 10:32 AM, Halil Pasic wrote:
>>> On Wed, 10 Feb 2021 16:24:29 +0100
>>> Halil Pasic <pasic@linux.ibm.com> wrote:
>>>   
>>>>> Maybe you could
>>>>> - grab a reference to kvm while holding the lock
>>>>> - call the mask handling functions with that kvm reference
>>>>> - lock again, drop the reference, and do the rest of the processing?
>>>> I agree, matrix_mdev->kvm can go NULL any time and we are risking
>>>> a null pointer dereference here.
>>>>
>>>> Another idea would be to do
>>>>
>>>>
>>>> static void vfio_ap_mdev_unset_kvm(struct ap_matrix_mdev *matrix_mdev)
>>>> {
>>>>           struct kvm *kvm;
>>>>                                                           
>>>>           mutex_lock(&matrix_dev->lock);
>>>>           if (matrix_mdev->kvm) {
>>>>                   kvm = matrix_mdev->kvm;
>>>>                   matrix_mdev->kvm = NULL;
>>>>                   mutex_unlock(&matrix_dev->lock);
>>>>                   kvm_arch_crypto_clear_masks(kvm);
>>>>                   mutex_lock(&matrix_dev->lock);
>>>>                   matrix_mdev->kvm->arch.crypto.pqap_hook = NULL;
>>> s/matrix_mdev->kvm/kvm
>>>>                   vfio_ap_mdev_reset_queues(matrix_mdev->mdev);
>>>>                   kvm_put_kvm(kvm);
>>>>           }
>>>>           mutex_unlock(&matrix_dev->lock);
>>>> }
>>>>
>>>> That way only one unset would actually do the unset and cleanup
>>>> and every other invocation would bail out with only checking
>>>> matrix_mdev->kvm.
>>> But the problem with that is that we enable the the assign/unassign
>>> prematurely, which could interfere wit reset_queues(). Forget about
>>> it.
>> Not sure what you mean by this.
>>
>>
> I mean because above I first do
> (1) matrix_mdev->kvm = NULL;
> and then do
> (2) vfio_ap_mdev_reset_queues(matrix_mdev->mdev);
> another thread could do
> static ssize_t unassign_adapter_store(struct device *dev,
>                                        struct device_attribute *attr,
>                                        const char *buf, size_t count)
> {
>          int ret;
>          unsigned long apid;
>          struct mdev_device *mdev = mdev_from_dev(dev);
>          struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
>                                                                                  
>          /* If the guest is running, disallow un-assignment of adapter */
>          if (matrix_mdev->kvm)
>                  return -EBUSY;
> ...
> }
> between (1) and (2), and we would not bail out with -EBUSY because !!kvm
> because of (1). That means we would change matrix_mdev->matrix and we
> would not reset the queues that correspond to the apid that was just
> removed, because by the time we do the reset_queues, the queues are
> not in the matrix_mdev->matrix any more.
>
> Does that make sense?

Yes, it makes sense. I guess I didn't look closely at your
suggestion when I said it was exactly what I implemented
after agreeing with Connie. I had a slight difference in
my implementation:

static void vfio_ap_mdev_unset_kvm(struct ap_matrix_mdev *matrix_mdev)
{
     struct kvm *kvm;

     mutex_lock(&matrix_dev->lock);

     if (matrix_mdev->kvm) {
         kvm = matrix_mdev->kvm;
         mutex_unlock(&matrix_dev->lock);
         kvm_arch_crypto_clear_masks(kvm);
         mutex_lock(&matrix_dev->lock);
         kvm->arch.crypto.pqap_hook = NULL;
         vfio_ap_mdev_reset_queues(matrix_mdev->mdev);
         matrix_mdev->kvm = NULL;
         kvm_put_kvm(kvm);
     }

     mutex_unlock(&matrix_dev->lock);
}

In your scenario, the unassignment would fail with -EBUSY because
the matrix_mdev->kvm pointer would not have yet been
cleared. The other problem with your implementation is that
IRQ resources would not get cleared after the reset because
the matrix_mdev->kvm pointer would be NULL at that time.