Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <20220407195908.633003-1-pgonda@google.com> <CAFNjLiXC0AdOw5f8Ovu47D==ex7F0=WN_Ocirymz4xL=mWvC5A@mail.gmail.com>
 <CAMkAt6r-Mc_YN-gVHuCpTj4E1EmcvyYpP9jhtHo5HRHnoNJAdA@mail.gmail.com>
 <CAMkAt6r+OMPWCbV_svUyGWa0qMzjj2UEG29G6P7jb6uH6yko2w@mail.gmail.com>
 <62e9ece1-5d71-f803-3f65-2755160cf1d1@redhat.com> <CAMkAt6q6YLBfo2RceduSXTafckEehawhD4K4hUEuB4ZNqe2kKg@mail.gmail.com>
 <4c0edc90-36a1-4f4c-1923-4b20e7bdbb4c@redhat.com> <CAMkAt6oL5qi7z-eh4z7z8WBhpc=Ow6WtcJA5bDi6-aGMnz135A@mail.gmail.com>
 <CAMkAt6rmDrZfN5DbNOTsKFV57PwEnK2zxgBTCbEPeE206+5v5w@mail.gmail.com> <0d282be4-d612-374d-84ba-067994321bab@redhat.com>
In-Reply-To: <0d282be4-d612-374d-84ba-067994321bab@redhat.com>
From:   Peter Gonda <pgonda@google.com>
Date:   Fri, 29 Apr 2022 09:35:45 -0600
Message-ID: <CAMkAt6ragq4OmnX+n628Yd5pn51qFv4qV20upGR6tTvyYw3U5A@mail.gmail.com>
Subject: Re: [PATCH v3] KVM: SEV: Mark nested locking of vcpu->lock
To:     Paolo Bonzini <pbonzini@redhat.com>
Cc:     John Sperbeck <jsperbeck@google.com>,
        kvm list <kvm@vger.kernel.org>,
        David Rientjes <rientjes@google.com>,
        Sean Christopherson <seanjc@google.com>,
        LKML <linux-kernel@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Thu, Apr 28, 2022 at 5:59 PM Paolo Bonzini <pbonzini@redhat.com> wrote:
>
> On 4/28/22 23:28, Peter Gonda wrote:
> >
> > So when actually trying this out I noticed that we are releasing the
> > current vcpu iterator but really we haven't actually taken that lock
> > yet. So we'd need to maintain a prev_* pointer and release that one.
>
> Not entirely true because all vcpu->mutex.dep_maps will be for the same
> lock.  The dep_map is essentially a fancy string, in this case
> "&vcpu->mutex".
>
> See the definition of mutex_init:
>
> #define mutex_init(mutex)                                              \
> do {                                                                   \
>          static struct lock_class_key __key;                            \
>                                                                         \
>          __mutex_init((mutex), #mutex, &__key);                         \
> } while (0)
>
> and the dep_map field is initialized with
>
>          lockdep_init_map_wait(&lock->dep_map, name, key, 0, LD_WAIT_SLEEP);
>
> (i.e. all vcpu->mutexes share the same name and key because they have a
> single mutex_init-ialization site).  Lockdep is as crude in theory as it
> is effective in practice!
>
> >
> >           bool acquired = false;
> >           kvm_for_each_vcpu(...) {
> >                   if (!acquired) {
> >                      if (mutex_lock_killable_nested(&vcpu->mutex, role)
> >                          goto out_unlock;
> >                      acquired = true;
> >                   } else {
> >                      if (mutex_lock_killable(&vcpu->mutex, role)
> >                          goto out_unlock;
>
> This will cause a lockdep splat because it uses subclass 0.  All the
> *_nested functions is allow you to specify a subclass other than zero.

OK got it. I now have this to lock:

         kvm_for_each_vcpu (i, vcpu, kvm) {
                  if (prev_vcpu != NULL) {
                          mutex_release(&prev_vcpu->mutex.dep_map, _THIS_IP_);
                          prev_vcpu = NULL;
                  }

                  if (mutex_lock_killable_nested(&vcpu->mutex, role))
                          goto out_unlock;
                  prev_vcpu = vcpu;
          }

But I've noticed the unlocking is in the wrong order since we are
using kvm_for_each_vcpu() I think we need a kvm_for_each_vcpu_rev() or
something. Which maybe a bit for work:
https://elixir.bootlin.com/linux/latest/source/lib/xarray.c#L1119.
Then I think we could something like this to unlock:

         bool acquired = true;
          kvm_for_each_vcpu_rev(i, vcpu, kvm) {
                  if (!acquired) {
                          mutex_acquire(&vcpu->mutex.dep_map, 0, role,
_THIS_IP_);
                  }
                  mutex_unlock(&vcpu->mutex);
                  acquired = false;
          }
>
> Paolo
>
> >                   }
> >           }
> >
> > To unlock:
> >
> >           kvm_for_each_vcpu(...) {
> >              mutex_unlock(&vcpu->mutex);
> >           }
> >
> > This way instead of mocking and releasing the lock_dep we just lock
> > the fist vcpu with mutex_lock_killable_nested(). I think this
> > maintains the property you suggested of "coalesces all the mutexes for
> > a vm in a single subclass".  Thoughts?
>