Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Wed, 19 May 2021 14:59:41 +0200
From:   Halil Pasic <pasic@linux.ibm.com>
To:     Christian Borntraeger <borntraeger@de.ibm.com>
Cc:     Tony Krowiak <akrowiak@linux.ibm.com>, linux-s390@vger.kernel.org,
        linux-kernel@vger.kernel.org, cohuck@redhat.com,
        pasic@linux.vnet.ibm.com, jjherne@linux.ibm.com, jgg@nvidia.com,
        alex.williamson@redhat.com, kwankhede@nvidia.com,
        stable@vger.kernel.org, Tony Krowiak <akrowiak@stny.rr.com>
Subject: Re: [PATCH v2] s390/vfio-ap: fix memory leak in mdev remove
 callback
Message-ID: <20210519145941.216cae45.pasic@linux.ibm.com>
In-Reply-To: <9c2b4711-5a26-15b0-8651-67a88bf12270@de.ibm.com>
References: <20210510214837.359717-1-akrowiak@linux.ibm.com>
        <20210512203536.4209c29c.pasic@linux.ibm.com>
        <4c156ab8-da49-4867-f29c-9712c2628d44@linux.ibm.com>
        <20210513194541.58d1628a.pasic@linux.ibm.com>
        <243086e2-08a0-71ed-eb7e-618a62b007e4@linux.ibm.com>
        <20210514021500.60ad2a22.pasic@linux.ibm.com>
        <594374f6-8cf6-4c22-0bac-3b224c55bbb6@linux.ibm.com>
        <20210517211030.368ca64b.pasic@linux.ibm.com>
        <966a60ad-bdde-68d0-ae2f-06121c6ad970@de.ibm.com>
        <9ebd5fd8-b093-e5bc-e680-88fa7a9b085c@linux.ibm.com>
        <494af62b-dc9a-ef2c-1869-d8f5ed239504@de.ibm.com>
        <20210518173351.39646b45.pasic@linux.ibm.com>
        <ca5f1c72-09a3-d270-44a0-bda54c554f67@de.ibm.com>
        <20210519012709.3bcc30e7.pasic@linux.ibm.com>
        <250189ed-bded-5261-d8f3-f75787be7aeb@de.ibm.com>
        <9c2b4711-5a26-15b0-8651-67a88bf12270@de.ibm.com>
Organization: IBM
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Precedence: bulk

On Wed, 19 May 2021 13:22:56 +0200
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> On 19.05.21 10:17, Christian Borntraeger wrote:
> > 
> > 
> > On 19.05.21 01:27, Halil Pasic wrote:  
> >> On Tue, 18 May 2021 19:01:42 +0200
> >> Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> >>  
> >>> On 18.05.21 17:33, Halil Pasic wrote:  
> >>>> On Tue, 18 May 2021 15:59:36 +0200
> >>>> Christian Borntraeger <borntraeger@de.ibm.com> wrote:  
> >> [..]  
> >>>>>>>
> >>>>>>> Would it help, if the code in priv.c would read the hook once
> >>>>>>> and then only work on the copy? We could protect that with rcu
> >>>>>>> and do a synchronize rcu in vfio_ap_mdev_unset_kvm after
> >>>>>>> unsetting the pointer?  
> >>>>
> >>>> Unfortunately just "the hook" is ambiguous in this context. We
> >>>> have kvm->arch.crypto.pqap_hook that is supposed to point to
> >>>> a struct kvm_s390_module_hook member of struct ap_matrix_mdev
> >>>> which is also called pqap_hook. And struct kvm_s390_module_hook
> >>>> has function pointer member named "hook".  
> >>>
> >>> I was referring to the full struct.  
> >>>>>>
> >>>>>> I'll look into this.  
> >>>>>
> >>>>> I think it could work. in priv.c use rcu_readlock, save the
> >>>>> pointer, do the check and call, call rcu_read_unlock.
> >>>>> In vfio_ap use rcu_assign_pointer to set the pointer and
> >>>>> after setting it to zero call sychronize_rcu.  
> >>>>
> >>>> In my opinion, we should make the accesses to the
> >>>> kvm->arch.crypto.pqap_hook pointer properly synchronized. I'm
> >>>> not sure if that is what you are proposing. How do we usually
> >>>> do synchronisation on the stuff that lives in kvm->arch?  
> >>>
> >>> RCU is a method of synchronization. We  make sure that structure
> >>> pqap_hook is still valid as long as we are inside the rcu read
> >>> lock. So the idea is: clear pointer, wait until all old readers
> >>> have finished and the proceed with getting rid of the structure.  
> >>
> >> Yes I know that RCU is a method of synchronization, but I'm not
> >> very familiar with it. I'm a little confused by "read the hook
> >> once and then work on a copy". I guess, I would have to read up
> >> on the RCU again to get clarity. I intend to brush up my RCU knowledge
> >> once the patch comes along. I would be glad to have your help when
> >> reviewing an RCU based solution for this.  
> > 
> > Just had a quick look. Its not trivial, as the hook function itself
> > takes a mutex and an rcu section must not sleep. Will have a deeper
> > look.  
> 
> 
> As a quick hack something like this could work. The whole locking is pretty
> complicated and this makes it even more complex so we might want to do
> a cleanup/locking rework later on.
> 

Hm, seems our emails crossed mid air...

> 
> index 9928f785c677..fde6e02aab54 100644
> --- a/arch/s390/kvm/priv.c
> +++ b/arch/s390/kvm/priv.c
> @@ -609,6 +609,7 @@ static int handle_io_inst(struct kvm_vcpu *vcpu)
>    */
>   static int handle_pqap(struct kvm_vcpu *vcpu)
>   {
> +       struct kvm_s390_module_hook *pqap_hook;
>          struct ap_queue_status status = {};
>          unsigned long reg0;
>          int ret;
> @@ -657,14 +658,21 @@ static int handle_pqap(struct kvm_vcpu *vcpu)
>           * Verify that the hook callback is registered, lock the owner
>           * and call the hook.
>           */
> -       if (vcpu->kvm->arch.crypto.pqap_hook) {
> -               if (!try_module_get(vcpu->kvm->arch.crypto.pqap_hook->owner))
> +       rcu_read_lock();
> +       pqap_hook = rcu_dereference(vcpu->kvm->arch.crypto.pqap_hook);
> +       if (pqap_hook) {
> +               if (!try_module_get(pqap_hook->owner)) {
> +                       rcu_read_unlock();
>                          return -EOPNOTSUPP;
> -               ret = vcpu->kvm->arch.crypto.pqap_hook->hook(vcpu);
> -               module_put(vcpu->kvm->arch.crypto.pqap_hook->owner);
> +               }

Up to this point the local pqap_hook is guaranteed to point to a valid
object if not NULL, ...
> +               rcu_read_unlock();

... and after this point IMHO it is not.

> +               ret = pqap_hook->hook(vcpu);

So IMHO the pointer deference here is still problematic, but that can
be fixed easily as I described in that email I've sent 3 minutes after
yours. IMHO we need a local copy of cpu->kvm->arch.crypto.pqap_hook->hook
taken within the rcu read critical section. Do you agree?

Regards,
Halil

> +               module_put(pqap_hook->owner);
>                  if (!ret && vcpu->run->s.regs.gprs[1] & 0x00ff0000)
>                          kvm_s390_set_psw_cc(vcpu, 3);
>                  return ret;
> +       } else {
> +               rcu_read_unlock();
>          }
>          /*
>           * A vfio_driver must register a hook.
> diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
> index f90c9103dac2..a7124abd6aed 100644
> --- a/drivers/s390/crypto/vfio_ap_ops.c
> +++ b/drivers/s390/crypto/vfio_ap_ops.c
> @@ -1194,6 +1194,7 @@ static void vfio_ap_mdev_unset_kvm(struct ap_matrix_mdev *matrix_mdev)
>                  mutex_lock(&matrix_dev->lock);
>                  vfio_ap_mdev_reset_queues(matrix_mdev->mdev);
>                  matrix_mdev->kvm->arch.crypto.pqap_hook = NULL;
> +               synchronize_rcu();
>                  kvm_put_kvm(matrix_mdev->kvm);
>                  matrix_mdev->kvm = NULL;
>                  matrix_mdev->kvm_busy = false;