Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Subject: Re: [PATCH] s390/vfio-ap: do not open code locks for
 VFIO_GROUP_NOTIFY_SET_KVM notification
To:     Jason Gunthorpe <jgg@nvidia.com>, Halil Pasic <pasic@linux.ibm.com>
Cc:     linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org,
        borntraeger@de.ibm.com, cohuck@redhat.com,
        pasic@linux.vnet.ibm.com, jjherne@linux.ibm.com,
        alex.williamson@redhat.com, kwankhede@nvidia.com,
        frankja@linux.ibm.com, david@redhat.com, imbrenda@linux.ibm.com,
        hca@linux.ibm.com
References: <20210707154156.297139-1-akrowiak@linux.ibm.com>
 <20210713013815.57e8a8cb.pasic@linux.ibm.com>
 <5dd3cc05-f789-21a3-50c7-ee80d850a105@linux.ibm.com>
 <20210713184517.48eacee6.pasic@linux.ibm.com>
 <20210713170533.GF136586@nvidia.com>
From:   Tony Krowiak <akrowiak@linux.ibm.com>
Message-ID: <9512a7fb-cc55-cd9b-cdf9-7c19d0567311@linux.ibm.com>
Date:   Tue, 13 Jul 2021 15:04:28 -0400
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.8.1
MIME-Version: 1.0
In-Reply-To: <20210713170533.GF136586@nvidia.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
Content-Language: en-US
Precedence: bulk


On 7/13/21 1:05 PM, Jason Gunthorpe wrote:
> On Tue, Jul 13, 2021 at 06:45:17PM +0200, Halil Pasic wrote:
>
>> Jason may give it another try to convince us that 0cc00c8d4050 only
>> silenced lockdep, but vfio_ap remained prone to deadlocks. To my best
>> knowledge using condition variable and a mutex is one of the well known
>> ways to implement an rwlock.
> The well known pattern is to use a rwsem.
>
> This:
>          wait_event_cmd(matrix_mdev->wait_for_kvm,
>                         !matrix_mdev->kvm_busy,
>                         mutex_unlock(&matrix_dev->lock),
>                         mutex_lock(&matrix_dev->lock));
>
>
> Is not really a rwsem, and is invsible to lockdep.

The lockdep splat was due to holding the matrix_dev->lock
mutex while the kvm->lock was taken to plug the AP devices
into the guest. The same problem would occur whether an
rwsem or the mutex was used.

The lockdep splat was resolved by setting the
matrix_mdev->kvm_busy flag and unlocking the matrix_dev->lock
mutex while the AP devices were being plugged into the guest.
All other functions needing the matrix_dev->lock mutex would wait
on a queue until the matrix_mdev->kvm_busy flag is cleared before
locking the matrix_dev->lock mutex.

Now, I understand that this technique is invisible to lockdep,
but I don't see how we can ever end up in a deadlock with
this design since the matrix_dev->lock mutex will never get
locked as long as the matrix_mdev->kvm_busy flag is set.

>
> Jason