Received-SPF: pass (google.com: domain of linux-kernel+bounces-31231-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Content-Type: multipart/mixed; boundary="------------efg6hB0FNWC65xkaiJmoB8uN"
Message-ID: <fc3fd07a-218d-406c-918b-e7f701968eb0@linux.ibm.com>
Date: Fri, 19 Jan 2024 14:56:10 +0100
User-Agent: Mozilla Thunderbird
Subject: Re: Several tst-robust* tests time out with recent Linux kernel
Content-Language: en-US
To: "Edgecombe, Rick P" <rick.p.edgecombe@intel.com>,
        "peterz@infradead.org" <peterz@infradead.org>
Cc: "xry111@xry111.site" <xry111@xry111.site>,
        "andrealmeid@igalia.com" <andrealmeid@igalia.com>,
        "fweimer@redhat.com" <fweimer@redhat.com>,
        "linux-mm@kvack.org" <linux-mm@kvack.org>,
        "libc-alpha@sourceware.org" <libc-alpha@sourceware.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "tglx@linutronix.de" <tglx@linutronix.de>,
        "linux-api@vger.kernel.org" <linux-api@vger.kernel.org>,
        "linux-arch@vger.kernel.org" <linux-arch@vger.kernel.org>,
        Heiko Carstens <hca@linux.ibm.com>,
        Sven Schnelle <svens@linux.ibm.com>
References: <4bda9f2e06512e375e045f9e72edb205104af19c.camel@xry111.site>
 <d69d50445284a5e0d98a64862877c1e6ec22a9a8.camel@xry111.site>
 <20231114153100.GY8262@noisy.programming.kicks-ass.net>
 <20231114154017.GI4779@noisy.programming.kicks-ass.net>
 <87ttpowajb.fsf@oldenburg.str.redhat.com>
 <20231114201402.GA25315@noisy.programming.kicks-ass.net>
 <822f3a867e5661ce61cea075a00ce04a4e4733f3.camel@intel.com>
 <20231115085102.GY3818@noisy.programming.kicks-ass.net>
 <564119521b61b5a38f9bdfe6c7a41fcbb07049c9.camel@intel.com>
 <158f6a47727a40c163e3fa6041a24388549c68f2.camel@intel.com>
From: Stefan Liebler <stli@linux.ibm.com>
In-Reply-To: <158f6a47727a40c163e3fa6041a24388549c68f2.camel@intel.com>
Precedence: bulk
MIME-Version: 1.0

This is a multi-part message in MIME format.
--------------efg6hB0FNWC65xkaiJmoB8uN
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit

On 17.11.23 02:22, Edgecombe, Rick P wrote:
> A bit more info...
> 
> The error returned to userspace is originating from:
> https://github.com/torvalds/linux/blob/master/kernel/futex/pi.c#L295
> 
> 'uval' is often zero in that error case, but sometimes just a
> mismatching value like: uval=0x567, task_pid_vnr()=0x564
> 
> 
> Depending on the number of CPUs the VM is running on it reproduces or
> not. When it does reproduce, the newly added path here is taken:
> https://github.com/torvalds/linux/blob/master/kernel/futex/pi.c#L1185
> The path is taken a lot during the test, sometimes >400 times before
> the above linked error is generated during the syscall. When it doesn't
> reproduce, I never saw that new path taken.
> 
> More print statements make the reproduction less reliable, so it does
> seem to have a race in the mix at least somewhat. Otherwise, I haven't
> tried to understand what is going on here with all this highwire
> locking.
> 
> Hope it helps.
Hi,

I've also observed fails in glibc testcase nptl/tst-robust8pi with:
mutex_timedlock of 66 in thread 7 failed with 22
=> pthread_mutex_timedlock returns 22=EINVAL

I've saw it on s390x. There I've used kernel with
commit 120d99901eb288f1d21db3976df4ba347b28f9c7
s390/vfio-ap: do not reset queue removed from host config

But I also saw it on a x86_64 kvm-guest with Fedora 39 and
copr-repository with vanilla kernel:
Linux fedora 6.7.0-0.rc8.20240107gt52b1853b.366.vanilla.fc39.x86_64 #1
SMP PREEMPT_DYNAMIC Sun Jan  7 06:17:30 UTC 2024 x86_64 GNU/Linux

And reported it to libc-alpha ("FAILING nptl/tst-robust8pi"
https://sourceware.org/pipermail/libc-alpha/2024-January/154150.html)
where Florian Weimer pointed me to this thread.

I've reduced the test (see attachement) and now have only one process
with three threads. I only use one mutex with attributes like the
original testcase: PTHREAD_MUTEX_ROBUST_NP, PTHREAD_PROCESS_SHARED,
PTHREAD_PRIO_INHERIT.
Every thread is doing a loop with pthread_mutex_timedlock(abstime={0,0})
and if locked, pthread_mutex_unlock.

I've added some uprobes before and after the futex-syscall in
__futex_lock_pi64(in pthread_mutex_timedlock) and futex_unlock_pi(in
pthread_mutex_unlock). For me __ASSUME_FUTEX_LOCK_PI2 is not available,
but __ASSUME_TIME64_SYSCALLS is defined.

For me it looks like this (simplified ubprobes-trace):
<thread> <timestamp>: <probe>
t1 4309589.419744: before syscall in __futex_lock_pi64

t3 4309589.419745: before syscall in futex_unlock_pi
t2 4309589.419745: before syscall in __futex_lock_pi64

t3 4309589.419747: after syscall in futex_unlock_pi
t2 4309589.419747: after syscall in __futex_lock_pi64 ret=-22=EINVAL

t1 4309589.419748: after syscall in __futex_lock_pi64 ret=-110=ETIMEDOUT

Can you please have a look again?

Bye,
Stefan Liebler
--------------efg6hB0FNWC65xkaiJmoB8uN
Content-Type: text/x-csrc; charset=UTF-8; name="tst-robust8pi-20240118.c"
Content-Disposition: attachment; filename="tst-robust8pi-20240118.c"
Content-Transfer-Encoding: base64

Ly9DRkxBR1M9LXB0aHJlYWQKLy9MREZMQUdTPS1scHRocmVhZAojaW5jbHVkZSA8c3RkaW8u
aD4KI2luY2x1ZGUgPHN0ZGxpYi5oPgojaW5jbHVkZSA8cHRocmVhZC5oPgojaW5jbHVkZSA8
YXNzZXJ0Lmg+CiNpbmNsdWRlIDxlcnJuby5oPgojaW5jbHVkZSA8dW5pc3RkLmg+CgojZGVm
aW5lIE5VTV9USFJFQURTIDMKI2RlZmluZSBUSFJFQURfRlVOQyB0aHJfZnVuYwojZGVmaW5l
IFVTRV9CQVJSSUVSIDEKI2lmbmRlZiBST1VORFMKIyBkZWZpbmUgUk9VTkRTIDEwMDAwMDAw
MAojZW5kaWYKCnR5cGVkZWYgc3RydWN0IHRocl9pbmZvCnsKICBpbnQgbnI7CiAgcHRocmVh
ZF90IHRocmVhZDsKfSBfX2F0dHJpYnV0ZV9fICgoYWxpZ25lZCAoMjU2KSkpIHRocl9pbmZv
X3Q7CgojZGVmaW5lIFRIUl9JTklUKCkJCQkJXAogIHRocl9pbmZvX3QgKnRociA9ICh0aHJf
aW5mb190ICopIGFyZzsKCiNkZWZpbmUgVEhSX1BSSU5URihmbXQsIC4uLikJCQlcCiAgcHJp
bnRmICgiIyVkOiAiIGZtdCwgdGhyLT5uciwgX19WQV9BUkdTX18pCgojZGVmaW5lIFRIUl9Q
VVRTKG1zZykJCQkJXAogIHByaW50ZiAoIiMlZDogIiBtc2cgIlxuIiwgdGhyLT5ucikKCiNp
ZiBVU0VfQkFSUklFUiAhPSAwCnN0YXRpYyBwdGhyZWFkX2JhcnJpZXJfdCB0aHJzX2JhcnJp
ZXI7CiNlbmRpZgoKc3RhdGljIHB0aHJlYWRfbXV0ZXhfdCBtdHg7CnN0YXRpYyBjb25zdCBz
dHJ1Y3QgdGltZXNwZWMgYmVmb3JlID0geyAwLCAwIH07CgovKiAjIyMjIyMjIyMjIyMjIyMj
IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjCiAg
IHRocmVhZCBmdW5jCiAgICMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj
IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyAgKi8KCnN0YXRpYyB2b2lkICoKdGhyX2Z1bmMg
KHZvaWQgKmFyZykKewogIFRIUl9JTklUICgpOwogIGludCBzdGF0ZSA9IDA7CiAgaW50IGZj
dDsKCiNpZiAwCiAgLyogMyB0aHJlYWRzLCAxeGZjdD0wPXB0aHJlYWRfbXV0ZXhfbG9jaywg
MnhmY3Q9MT1wdGhyZWFkX211dGV4X3RpbWVkbG9jazogRUlOVkFMLiAgKi8KICBmY3QgPSAo
dGhyLT5uciArIDEpICUgMjsKI2VsaWYgMAogIC8qIDMgdGhyZWFkcywgMnhmY3Q9MD1wdGhy
ZWFkX211dGV4X2xvY2ssIDF4ZmN0PTE9cHRocmVhZF9tdXRleF90aW1lZGxvY2s6IG5vIGZh
aWxzLiAgKi8KICBmY3QgPSAodGhyLT5ucikgJSAyOwojZWxpZiAxCiAgLyogPjMgdGhyZWFk
cywgZmN0PTE9b25seSBwdGhyZWFkX211dGV4X3RpbWVkbG9jazogRUlOVkFMLiAgKi8KICBm
Y3QgPSAxOwojZW5kaWYKCiAgaW50IHJvdW5kID0gMDsKICBUSFJfUFJJTlRGICgic3RhcnRl
ZDogZmN0PSVkXG4iLCBmY3QpOwojaWYgVVNFX0JBUlJJRVIgIT0gMAogIHB0aHJlYWRfYmFy
cmllcl93YWl0ICgmdGhyc19iYXJyaWVyKTsKI2VuZGlmCiAgd2hpbGUgKDEpCiAgICB7Cgog
ICAgICBpZiAoc3RhdGUgPT0gMCkKCXsKCSAgcm91bmQgKys7CgkgIGludCBlOwoKCSAgc3dp
dGNoIChmY3QpCgkgICAgewoJICAgIGNhc2UgMDoKCSAgICAgIGUgPSBwdGhyZWFkX211dGV4
X2xvY2sgKCZtdHgpOwoJICAgICAgaWYgKGUgIT0gMCkKCQl7CgkJICBUSFJfUFJJTlRGICgi
bXV0ZXhfbG9jayBmYWlsZWQgd2l0aCAlZCAocm91bmQ9JWQpXG4iLCBlLCByb3VuZCk7CgkJ
ICBleGl0ICgxKTsKCQl9CgkgICAgICBzdGF0ZSA9IDE7CgkgICAgICBicmVhazsKCSAgICBj
YXNlIDE6CgkgICAgICBlID0gcHRocmVhZF9tdXRleF90aW1lZGxvY2sgKCZtdHgsICZiZWZv
cmUpOwoJICAgICAgaWYgKGUgIT0gMCAmJiBlICE9IEVUSU1FRE9VVCkKCQl7CgkJICBUSFJf
UFJJTlRGICgibXV0ZXhfdGltZWRsb2NrIGZhaWxlZCB3aXRoICVkIChyb3VuZD0lZClcbiIs
IGUsIHJvdW5kKTsKCQkgIGV4aXQgKDEpOwoJCX0KCSAgICAgIGJyZWFrOwoJICAgIGRlZmF1
bHQ6CgkgICAgICBlID0gcHRocmVhZF9tdXRleF90cnlsb2NrICgmbXR4KTsKCSAgICAgIGlm
IChlICE9IDAgJiYgZSAhPSBFQlVTWSkKCQl7CgkJICBUSFJfUFJJTlRGICgibXV0ZXhfdHJ5
bG9jayBmYWlsZWQgd2l0aCAlZCAocm91bmQ9JWQpXG4iLCBlLCByb3VuZCk7CgkJICBleGl0
ICgxKTsKCQl9CgkgICAgICBicmVhazsKCSAgICB9CgoJICBpZiAoZSA9PSBFT1dORVJERUFE
KQoJICAgIHB0aHJlYWRfbXV0ZXhfY29uc2lzdGVudCAoJm10eCk7CgoJICBpZiAoZSA9PSAw
IHx8IGUgPT0gRU9XTkVSREVBRCkKCSAgICBzdGF0ZSA9IDE7Cgl9CiAgICAgIGVsc2UKCXsK
CSAgaW50IGUgPSBwdGhyZWFkX211dGV4X3VubG9jayAoJm10eCk7CgkgIGlmIChlICE9IDAp
CgkgICAgewoJICAgICAgVEhSX1BSSU5URiAoIm11dGV4X3VubG9jayBvZiBmYWlsZWQgd2l0
aCAlZCAocm91bmQ9JWQpXG4iLCBlLCByb3VuZCk7CgkgICAgICBleGl0ICgxKTsKCSAgICB9
CgkgIHN0YXRlID0gMDsKCX0KCiAgICAgIGlmIChyb3VuZCA+PSBST1VORFMpCgl7CgkgIFRI
Ul9QUklOVEYgKCJSRUFDSEVEIHJvdW5kICVkLiA9PiBleGl0XG4iLCBST1VORFMpOwoJICBp
ZiAoc3RhdGUgIT0gMCkKCSAgICB7CgkgICAgICBpbnQgZSA9IHB0aHJlYWRfbXV0ZXhfdW5s
b2NrICgmbXR4KTsKCSAgICAgIGlmIChlICE9IDApCgkJewoJCSAgVEhSX1BSSU5URiAoIm11
dGV4X3VubG9ja0BleGl0IG9mIGZhaWxlZCB3aXRoICVkIChyb3VuZD0lZClcbiIsIGUsIHJv
dW5kKTsKCQkgIGV4aXQgKDEpOwoJCX0KCSAgICB9CgkgIGJyZWFrOwoJfQogICAgfQoKICBy
ZXR1cm4gTlVMTDsKfQoKaW50Cm1haW4gKHZvaWQpCnsKICBpbnQgaTsKICBwcmludGYgKCJt
YWluOiBzdGFydCAlZCB0aHJlYWRzLlxuIiwgTlVNX1RIUkVBRFMpOwoKI2lmIFVTRV9CQVJS
SUVSICE9IDAKICBwdGhyZWFkX2JhcnJpZXJfaW5pdCAoJnRocnNfYmFycmllciwgTlVMTCwg
TlVNX1RIUkVBRFMgKyAxKTsKI2VuZGlmCgogIHB0aHJlYWRfbXV0ZXhhdHRyX3QgbWE7CiAg
aWYgKHB0aHJlYWRfbXV0ZXhhdHRyX2luaXQgKCZtYSkgIT0gMCkKICAgIHsKICAgICAgcHV0
cyAoIm11dGV4YXR0cl9pbml0IGZhaWxlZCIpOwogICAgICByZXR1cm4gMDsKICAgIH0KICBp
ZiAocHRocmVhZF9tdXRleGF0dHJfc2V0cm9idXN0ICgmbWEsIFBUSFJFQURfTVVURVhfUk9C
VVNUX05QKSAhPSAwKQogICAgewogICAgICBwdXRzICgibXV0ZXhhdHRyX3NldHJvYnVzdCBm
YWlsZWQiKTsKICAgICAgcmV0dXJuIDE7CiAgICB9CiAgaWYgKHB0aHJlYWRfbXV0ZXhhdHRy
X3NldHBzaGFyZWQgKCZtYSwgUFRIUkVBRF9QUk9DRVNTX1NIQVJFRCkgIT0gMCkKICAgIHsK
ICAgICAgcHV0cyAoIm11dGV4YXR0cl9zZXRwc2hhcmVkIGZhaWxlZCIpOwogICAgICByZXR1
cm4gMTsKICAgIH0KICBpZiAocHRocmVhZF9tdXRleGF0dHJfc2V0cHJvdG9jb2wgKCZtYSwg
UFRIUkVBRF9QUklPX0lOSEVSSVQpICE9IDApCiAgICB7CiAgICAgIHB1dHMgKCJwdGhyZWFk
X211dGV4YXR0cl9zZXRwcm90b2NvbCBmYWlsZWQiKTsKICAgICAgcmV0dXJuIDE7CiAgICB9
CgogIGlmIChwdGhyZWFkX211dGV4X2luaXQgKCZtdHgsICZtYSkgIT0gMCkKICAgIHsKICAg
ICAgcHV0cyAoInB0aHJlYWRfbXV0ZXhfaW5pdCBmYWlsZWQiKTsKICAgICAgcmV0dXJuIDE7
CiAgICB9CgogIHRocl9pbmZvX3QgdGhyc1tOVU1fVEhSRUFEU107CiAgZm9yIChpID0gMDsg
aSA8IE5VTV9USFJFQURTOyBpKyspCiAgICB7CiAgICAgIHRocnNbaV0ubnIgPSBpOwogICAg
ICBhc3NlcnQgKHB0aHJlYWRfY3JlYXRlICgmKHRocnNbaV0udGhyZWFkKSwgTlVMTCwgVEhS
RUFEX0ZVTkMsICYodGhyc1tpXSkpCgkgICAgICA9PSAwKTs7CiAgICB9CgojaWYgVVNFX0JB
UlJJRVIgIT0gMAogIC8qIEFsbCB0aHJlYWRzIHN0YXJ0IHdvcmsgYWZ0ZXIgdGhpcyBiYXJy
aWVyLiAgKi8KICBwdGhyZWFkX2JhcnJpZXJfd2FpdCAoJnRocnNfYmFycmllcik7CiNlbmRp
ZgoKICBmb3IgKGkgPSAwOyBpIDwgTlVNX1RIUkVBRFM7IGkrKykKICAgIHsKICAgICAgcHRo
cmVhZF9qb2luICh0aHJzW2ldLnRocmVhZCwgTlVMTCk7CiAgICB9CgojaWYgVVNFX0JBUlJJ
RVIgIT0gMAogIHB0aHJlYWRfYmFycmllcl9kZXN0cm95ICgmdGhyc19iYXJyaWVyKTsKI2Vu
ZGlmCgogIHByaW50ZiAoIm1haW46IGVuZC5cbiIpOwogIHJldHVybiBFWElUX1NVQ0NFU1M7
Cn0K

--------------efg6hB0FNWC65xkaiJmoB8uN--