Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <895ef450-4fb3-5d29-a6ad-790657106a5a@intel.com>
In-Reply-To: <895ef450-4fb3-5d29-a6ad-790657106a5a@intel.com>
From:   John Stultz <jstultz@google.com>
Date:   Fri, 7 Jul 2023 13:23:07 -0700
Message-ID: <CANDhNCpiLFP-YKhNFWeZ8TJbygFfGczVfRFJo6xMzVT2NrKKPA@mail.gmail.com>
Subject: Re: ww_mutex.sh hangs since v5.16-rc1
To:     Li Zhijian <zhijianx.li@intel.com>
Cc:     peterz@infradead.org, mingo@redhat.com, will@kernel.org,
        longman@redhat.com, boqun.feng@gmail.com,
        open list <linux-kernel@vger.kernel.org>,
        "linux-kselftest@vger.kernel.org" <linux-kselftest@vger.kernel.org>,
        "lkp@lists.01.org" <lkp@lists.01.org>,
        Chris Wilson <chris@chris-wilson.co.uk>,
        Dietmar Eggemann <dietmar.eggemann@arm.com>,
        Joel Fernandes <joelaf@google.com>,
        Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Tue, Nov 30, 2021 at 5:26=E2=80=AFPM Li Zhijian <zhijianx.li@intel.com> =
wrote:
>
> LKP/0Day found that ww_mutex.sh cannot complete since v5.16-rc1, but
> I'm pretty sorry that we failed to bisect the FBC, instead, the bisection=
 pointed
> to a/below merge commit(91e1c99e17) finally.
>
> Due to this hang, other tests in the same group are also blocked in 0Day,=
 we
> hope we can fix this hang ASAP.
>
> So if you have any idea about this, or need more debug information, feel =
free to let me know :)
>
> BTW, ww_mutex.sh was failed in v5.15 without hang, and looks it cannot re=
produce on a vm.
>

So, as part of the proxy-execution work, I've been recently trying to
understand why the patch series was causing apparent hangs in the
ww_mutex test with large(64) cpu counts.
I was assuming my changes were causing a lost wakeup somehow, but as I
dug in I found it looked like the stress_inorder_work() function was
live-locking.

I noticed that adding printks to the logic would change the behavior,
and finally realized I could reproduce a livelock against mainline by
adding a printk before the "return -EDEADLK;" in __ww_mutex_kill(),
making it clear the logic was timing sensitive.  Then searching around
I found this old and unresolved thread.

Part of the issue is that we may not hit the timeout check at the end
of the loop, as the EDEADLK case short-cuts back to retry, allowing
the test to effectively get stuck.

But I know with ww_mutexes there's supposed to be a guarantee of
forward progress as the older context wins, but it's not clear to me
that works here. The EDEADLK case results in a releasing and
reacquiring of the locks (only with the contended lock being taken
first), and if a second EDEADLK occurs, it starts over again from
scratch (though with the new contended lock being chosen first instead
- which seems to lose any progress).

So maybe the test has broken that guarentee in how it restarts, or
with 128 threads trying to acquire a random order of 16 locks without
contention (and the order shifting slightly each time it does see
contention) it might just be a very big space to resolve if we don't
luck into good timing.

Anyway, I wanted to get some feedback from folks who have a better
theoretical understanding of the ww_mutexes. With large cpu counts are
we just asking for trouble here? Is the test doing something wrong? Or
is there possibly a ww_mutex bug under this?

thanks
-john