Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Content-Type: multipart/mixed; boundary="------------atppxto8rPUknDM1CKkUqQJb"
Message-ID: <bd5feeb3-bc44-d4d2-7708-eea9243b49a4@gmail.com>
Date:   Fri, 4 Aug 2023 13:07:41 +0200
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: ARM board lockups/hangs triggered by locks and mutexes
Content-Language: en-US
From:   =?UTF-8?B?UmFmYcWCIE1pxYJlY2tp?= <zajec5@gmail.com>
To:     Peter Zijlstra <peterz@infradead.org>,
        Ingo Molnar <mingo@redhat.com>, Will Deacon <will@kernel.org>,
        Waiman Long <longman@redhat.com>,
        Boqun Feng <boqun.feng@gmail.com>,
        Russell King <linux@armlinux.org.uk>,
        Daniel Lezcano <daniel.lezcano@linaro.org>,
        Thomas Gleixner <tglx@linutronix.de>,
        Florian Fainelli <f.fainelli@gmail.com>,
        linux-clk@vger.kernel.org,
        "linux-arm-kernel@lists.infradead.org" 
        <linux-arm-kernel@lists.infradead.org>,
        Network Development <netdev@vger.kernel.org>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>
Cc:     OpenWrt Development List <openwrt-devel@lists.openwrt.org>,
        bcm-kernel-feedback-list <bcm-kernel-feedback-list@broadcom.com>
References: <CACna6rxpzDWE5-gnmpgMgfzPmmHvEGTZk4GJvJ8jLSMazh2bVA@mail.gmail.com>
In-Reply-To: <CACna6rxpzDWE5-gnmpgMgfzPmmHvEGTZk4GJvJ8jLSMazh2bVA@mail.gmail.com>
Precedence: bulk

This is a multi-part message in MIME format.
--------------atppxto8rPUknDM1CKkUqQJb
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

On 2.08.2023 00:10, Rafał Miłecki wrote:
> Unfortunately enabling *any* of following options:
> CONFIG_DEBUG_RT_MUTEXES=y
> CONFIG_DEBUG_SPINLOCK=y
> CONFIG_DEBUG_MUTEXES=y
> seems to make locksup/hangs go away. I tried for few hours.

I decided to find out why enabling CONFIG_DEBUG_MUTEXES "fixes" kernel /
device stability for me. I tried enabling manually code that normally
hides behind the #ifdev CONFIG_DEBUG_MUTEXES.

Attached to this e-mail is a small patch that is enough to make my
kernel stable (mutex-fix-bcm53573.diff).

#####

It's not what's the most interesting thought. What really doesn't make
sense anymore is that below diff (on top of attached one) brings back
hangs/lockups.

I triple checked that. Dropping a single unused function breaks kernel /
device stability on BCM53573!

AFAIK the only thing below diff actually affects is location of symbols
(I actually verified that by comparing System.map before and after -
over 22'000 of relocated symbols).

Can some unfortunate location of symbols cause those hangs/lockups?


diff --git a/kernel/locking/mutex-debug.c b/kernel/locking/mutex-debug.c
index 4fe40910f..c440222a4 100644
--- a/kernel/locking/mutex-debug.c
+++ b/kernel/locking/mutex-debug.c
@@ -34,6 +34,8 @@ void debug_mutex_lock_common(struct mutex *lock, struct mutex_waiter *waiter)
  	INIT_LIST_HEAD(&waiter->list);
  }

+/* Dropping below function brings back hangs/lockups & reboots */
+#if 0
  void debug_mutex_wake_waiter(struct mutex *lock, struct mutex_waiter *waiter)
  {
  	lockdep_assert_held(&lock->wait_lock);
@@ -41,6 +43,7 @@ void debug_mutex_wake_waiter(struct mutex *lock, struct mutex_waiter *waiter)
  	DEBUG_LOCKS_WARN_ON(waiter->magic != waiter);
  	DEBUG_LOCKS_WARN_ON(list_empty(&waiter->list));
  }
+#endif

  void debug_mutex_free_waiter(struct mutex_waiter *waiter)
  {

--------------atppxto8rPUknDM1CKkUqQJb
Content-Type: text/x-patch; charset=UTF-8; name="mutex-fix-bcm53573.diff"
Content-Disposition: attachment; filename="mutex-fix-bcm53573.diff"
Content-Transfer-Encoding: base64

ZGlmZiAtLWdpdCBhL2luY2x1ZGUvbGludXgvbXV0ZXguaCBiL2luY2x1ZGUvbGludXgvbXV0
ZXguaAppbmRleCA0NzliYzk2YzMuLjE1YmQ0NjkxYiAxMDA2NDQKLS0tIGEvaW5jbHVkZS9s
aW51eC9tdXRleC5oCisrKyBiL2luY2x1ZGUvbGludXgvbXV0ZXguaApAQCAtNTcsOSArNTcs
NyBAQCBzdHJ1Y3QgbXV0ZXggewogCXN0cnVjdCBvcHRpbWlzdGljX3NwaW5fcXVldWUgb3Nx
OyAvKiBTcGlubmVyIE1DUyBsb2NrICovCiAjZW5kaWYKIAlzdHJ1Y3QgbGlzdF9oZWFkCXdh
aXRfbGlzdDsKLSNpZmRlZiBDT05GSUdfREVCVUdfTVVURVhFUwogCXZvaWQJCQkqbWFnaWM7
Ci0jZW5kaWYKICNpZmRlZiBDT05GSUdfREVCVUdfTE9DS19BTExPQwogCXN0cnVjdCBsb2Nr
ZGVwX21hcAlkZXBfbWFwOwogI2VuZGlmCkBAIC03MywxMiArNzEsMTAgQEAgc3RydWN0IG11
dGV4X3dhaXRlciB7CiAJc3RydWN0IGxpc3RfaGVhZAlsaXN0OwogCXN0cnVjdCB0YXNrX3N0
cnVjdAkqdGFzazsKIAlzdHJ1Y3Qgd3dfYWNxdWlyZV9jdHgJKnd3X2N0eDsKLSNpZmRlZiBD
T05GSUdfREVCVUdfTVVURVhFUwogCXZvaWQJCQkqbWFnaWM7Ci0jZW5kaWYKIH07CiAKLSNp
ZmRlZiBDT05GSUdfREVCVUdfTVVURVhFUworI2lmIDEgLy9kZWYgQ09ORklHX0RFQlVHX01V
VEVYRVMKIAogI2RlZmluZSBfX0RFQlVHX01VVEVYX0lOSVRJQUxJWkVSKGxvY2tuYW1lKQkJ
CQlcCiAJLCAubWFnaWMgPSAmbG9ja25hbWUKZGlmZiAtLWdpdCBhL2luY2x1ZGUvbGludXgv
c2NoZWQuaCBiL2luY2x1ZGUvbGludXgvc2NoZWQuaAppbmRleCBkMGU2Mzk0OTcuLjhmZWY0
NDg1ZSAxMDA2NDQKLS0tIGEvaW5jbHVkZS9saW51eC9zY2hlZC5oCisrKyBiL2luY2x1ZGUv
bGludXgvc2NoZWQuaApAQCAtOTU4LDEwICs5NTgsOCBAQCBzdHJ1Y3QgdGFza19zdHJ1Y3Qg
ewogCXN0cnVjdCBydF9tdXRleF93YWl0ZXIJCSpwaV9ibG9ja2VkX29uOwogI2VuZGlmCiAK
LSNpZmRlZiBDT05GSUdfREVCVUdfTVVURVhFUwogCS8qIE11dGV4IGRlYWRsb2NrIGRldGVj
dGlvbjogKi8KIAlzdHJ1Y3QgbXV0ZXhfd2FpdGVyCQkqYmxvY2tlZF9vbjsKLSNlbmRpZgog
CiAjaWZkZWYgQ09ORklHX0RFQlVHX0FUT01JQ19TTEVFUAogCWludAkJCQlub25fYmxvY2tf
Y291bnQ7CmRpZmYgLS1naXQgYS9rZXJuZWwvbG9ja2luZy9NYWtlZmlsZSBiL2tlcm5lbC9s
b2NraW5nL01ha2VmaWxlCmluZGV4IDQ1NDUyZmFjZi4uYjIyZTZlY2Q4IDEwMDY0NAotLS0g
YS9rZXJuZWwvbG9ja2luZy9NYWtlZmlsZQorKysgYi9rZXJuZWwvbG9ja2luZy9NYWtlZmls
ZQpAQCAtMTIsNyArMTIsNyBAQCBDRkxBR1NfUkVNT1ZFX211dGV4LWRlYnVnLm8gPSAkKEND
X0ZMQUdTX0ZUUkFDRSkKIENGTEFHU19SRU1PVkVfcnRtdXRleC1kZWJ1Zy5vID0gJChDQ19G
TEFHU19GVFJBQ0UpCiBlbmRpZgogCi1vYmotJChDT05GSUdfREVCVUdfTVVURVhFUykgKz0g
bXV0ZXgtZGVidWcubworb2JqLXkgKz0gbXV0ZXgtZGVidWcubwogb2JqLSQoQ09ORklHX0xP
Q0tERVApICs9IGxvY2tkZXAubwogaWZlcSAoJChDT05GSUdfUFJPQ19GUykseSkKIG9iai0k
KENPTkZJR19MT0NLREVQKSArPSBsb2NrZGVwX3Byb2MubwpkaWZmIC0tZ2l0IGEva2VybmVs
L2xvY2tpbmcvbXV0ZXguYyBiL2tlcm5lbC9sb2NraW5nL211dGV4LmMKaW5kZXggYjAyZmZm
MjgyLi42ZGMzZjgwYTMgMTAwNjQ0Ci0tLSBhL2tlcm5lbC9sb2NraW5nL211dGV4LmMKKysr
IGIva2VybmVsL2xvY2tpbmcvbXV0ZXguYwpAQCAtOTQ2LDkgKzk0Niw3IEBAIF9fbXV0ZXhf
bG9ja19jb21tb24oc3RydWN0IG11dGV4ICpsb2NrLCBsb25nIHN0YXRlLCB1bnNpZ25lZCBp
bnQgc3ViY2xhc3MsCiAKIAltaWdodF9zbGVlcCgpOwogCi0jaWZkZWYgQ09ORklHX0RFQlVH
X01VVEVYRVMKIAlERUJVR19MT0NLU19XQVJOX09OKGxvY2stPm1hZ2ljICE9IGxvY2spOwot
I2VuZGlmCiAKIAl3dyA9IGNvbnRhaW5lcl9vZihsb2NrLCBzdHJ1Y3Qgd3dfbXV0ZXgsIGJh
c2UpOwogCWlmICh3d19jdHgpIHsKQEAgLTE0MTcsOSArMTQxNSw3IEBAIGludCBfX3NjaGVk
IG11dGV4X3RyeWxvY2soc3RydWN0IG11dGV4ICpsb2NrKQogewogCWJvb2wgbG9ja2VkOwog
Ci0jaWZkZWYgQ09ORklHX0RFQlVHX01VVEVYRVMKIAlERUJVR19MT0NLU19XQVJOX09OKGxv
Y2stPm1hZ2ljICE9IGxvY2spOwotI2VuZGlmCiAKIAlsb2NrZWQgPSBfX211dGV4X3RyeWxv
Y2sobG9jayk7CiAJaWYgKGxvY2tlZCkK

--------------atppxto8rPUknDM1CKkUqQJb--