Received: by 2002:a05:6358:a55:b0:ec:fcf4:3ecf with SMTP id 21csp857327rwb; Thu, 12 Jan 2023 13:19:45 -0800 (PST) X-Google-Smtp-Source: AMrXdXtCiMUt02zsMAo9Tt/GjEGOHM88QEe9yUBhSvbKYEojIaohMux/HmirPC45mbIoa60xctXL X-Received: by 2002:a17:906:850c:b0:7c0:f4f8:582a with SMTP id i12-20020a170906850c00b007c0f4f8582amr71534636ejx.52.1673558385419; Thu, 12 Jan 2023 13:19:45 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673558385; cv=none; d=google.com; s=arc-20160816; b=V1YKk0vOZt5UtX7HxUr9H3k08lfbFNkHgxBInwSvwvQkhkEZML0d7vJx8/I+/srH/u F7MpAKT+y8EWkY04DmB0Bz4M9z4lBMe2jYr0k2HOR8hqSGgQTwZ/sM5PeyzM3F0HR7X6 tGZ6dVlXJj2JpCaqI1HX7NqHGUbZ4uRbGJpplpQTxR9FDiHtVpVCVX7Mas0gsTOiDbhL jssnarUgPP2bmVxgDGGwDXnHDeHehp/OpqSP7HS13ZsAAaWiMZ+zJxRvd/LytxLp9HsR LhWNXmgNdj1rY9jEEAxnO6PAQlOKF1sQ/TethKY+V1GJ12HXsb2mGrg7kMlPzqvZG46y r5qA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :content-language:accept-language:in-reply-to:references:message-id :date:thread-index:thread-topic:subject:cc:to:from; bh=aBKrP90HLRvSJabOYeQ6Tl3ZhpMkmaOsIcOjXkZ9r3g=; b=oGZ85IHtVyS85KpdDWonO4sH8HyTy9TdTnVjVrJ84mLmm4CilwrjQNX9jpYj16NZs7 721cQ9k1BMLz0l9N5bc3W0Tao7ZQXyl+mDSC2UXdMbOT386WeG3fUH8WVYxhb17RgwPA S4sdZlCIn97jc5ltQlJxC7r5SNBAtPBwAui3qzoBStinfbybBThGklmSG0p5gAmidtyX IO4cY0kxrpaZEq4cKEoV59T1TyaSSNi1Isgy/NBL1n9gW+inkYgpE6SmR5/UjHDHNewL wBN75YT6vX5vXVg27R9qdVMdGImFmXYvYG9t4D4SHl7M+m0BS6fRj0mhrGyUEmwxo5Th /88g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Return-Path: Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id xa5-20020a170906fd8500b0084d43aadb70si10283859ejb.127.2023.01.12.13.19.33; Thu, 12 Jan 2023 13:19:45 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240544AbjALVHM convert rfc822-to-8bit (ORCPT + 50 others); Thu, 12 Jan 2023 16:07:12 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47444 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240875AbjALVGm (ORCPT ); Thu, 12 Jan 2023 16:06:42 -0500 Received: from frasgout.his.huawei.com (frasgout.his.huawei.com [185.176.79.56]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 03F6F59F95 for ; Thu, 12 Jan 2023 12:51:07 -0800 (PST) Received: from lhrpeml100001.china.huawei.com (unknown [172.18.147.200]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4NtGss5dG9z67HtH; Fri, 13 Jan 2023 04:50:57 +0800 (CST) Received: from lhrpeml500002.china.huawei.com (7.191.160.78) by lhrpeml100001.china.huawei.com (7.191.160.183) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.34; Thu, 12 Jan 2023 20:51:05 +0000 Received: from lhrpeml500002.china.huawei.com ([7.191.160.78]) by lhrpeml500002.china.huawei.com ([7.191.160.78]) with mapi id 15.01.2375.034; Thu, 12 Jan 2023 20:51:05 +0000 From: Jonas Oberhauser To: "paulmck@kernel.org" , "riel@surriel.com" , "davej@codemonkey.org.uk" CC: "linux-kernel@vger.kernel.org" , "kernel-team@meta.com" Subject: RE: [PATCH diagnostic qspinlock] Diagnostics for excessive lock-drop wait loop time Thread-Topic: [PATCH diagnostic qspinlock] Diagnostics for excessive lock-drop wait loop time Thread-Index: AQHZJh4apQ0ukQ/LOEaXjXF7PGrkyq6a/rvg Date: Thu, 12 Jan 2023 20:51:04 +0000 Message-ID: <896a2d84918e4adc8a4d00d72510eb3d@huawei.com> References: <20230112003627.GA3133092@paulmck-ThinkPad-P17-Gen-1> In-Reply-To: <20230112003627.GA3133092@paulmck-ThinkPad-P17-Gen-1> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.81.220.202] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 8BIT MIME-Version: 1.0 X-CFilter-Loop: Reflected X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Paul, -----Original Message----- From: Paul E. McKenney [mailto:paulmck@kernel.org] > We see systems stuck in the queued_spin_lock_slowpath() loop that waits for the lock to become unlocked in the case where the current CPU has set pending state. Interesting! Do you know if the hangs started with a recent patch? What codepaths are active (virtualization/arch/...)? Does it happen extremely rarely? Do you have any additional information? I saw a similar situation a few years ago in a proprietary kernel, but it only happened once ever and I gave up on looking for the reason after a few days (including some time combing through the compiler generated assembler). Have fun, jonas