Received: by 2002:a25:e74b:0:0:0:0:0 with SMTP id e72csp721221ybh; Tue, 21 Jul 2020 06:26:29 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzI5y21OXlqaD5la2R4suVz6L9VxpIrRKriuqxpo8O6qzNJPav4epq8h87EvNqZbujHyWiP X-Received: by 2002:a50:ee87:: with SMTP id f7mr26520646edr.355.1595337989678; Tue, 21 Jul 2020 06:26:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1595337989; cv=none; d=google.com; s=arc-20160816; b=vM2WxKrr4t0ali75vK8t1E3C7AntmUzL3GY9/H4+wJroExzoZ3BucL9n3tPkFWuZ6Z BpJLELd0oky4bpSh4SKMH8WmNV96tRTIuBmn3TlMKXobm1a1mVDUFeLTftb3iSq5dKxv qimWfcog81NUKnzip4yDm4XkAYdIxtFM/Quau0Q+94HK9NA0vjYPXuEkdftviaaPOGgj gYHEBJZAnTXrQCJhtbq4uJyccGUFdbLgQGAP4pXBUFZk8DcLy7bcWA4KRq1fsvQPjNjO RC9h9V3ZX8kFQ/7tLdTZQO4cMvL9/QNzXWD994gQMiKueApu9nJpWsM5qU+8ulAvWFDy 6ZVA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:user-agent:in-reply-to :content-transfer-encoding:content-disposition:mime-version :references:message-id:subject:cc:to:from:date:dkim-signature; bh=Vl6nY10QRE8L9pFubciH3YlE4tKocCVwUrJH3mwnBNc=; b=VOE5mRmOU1MdOIOkaPP0hsVeGeCe8xVFPdF2rWjtV3Y/2fXk24JbuPE11NHpDt8wkh zVTIvV6sVif1Ji34OvZBExHTEJzKyEUUTCUFW9kQSN7rY/8RTAZBF/clR6YJ0jhOdAvC v1HtEWrStWXQ5mqk8eWPKoqqpBaiDvGMYb8rS6MQUc6nslgxBXp1mDqZOAIaMJUEFneV UUlc4ZJZb1oIv6lyiYwd6erPBr1gYCB42eAOLzHEbtbhZt3Z/B+6yfAJPO2AbdtbFvxu h3SkrJs0X2IIyoCGXV5qrYuoIRvdsWWLh2udvZOtHIAKxNDYDTEkzyK123XudvBFgnVc rY3g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@lca.pw header.s=google header.b=K3+TBw5i; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id h20si12551682eje.434.2020.07.21.06.26.07; Tue, 21 Jul 2020 06:26:29 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@lca.pw header.s=google header.b=K3+TBw5i; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726506AbgGUNX4 (ORCPT + 99 others); Tue, 21 Jul 2020 09:23:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44136 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726192AbgGUNXz (ORCPT ); Tue, 21 Jul 2020 09:23:55 -0400 Received: from mail-qk1-x744.google.com (mail-qk1-x744.google.com [IPv6:2607:f8b0:4864:20::744]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7DB45C061794 for ; Tue, 21 Jul 2020 06:23:55 -0700 (PDT) Received: by mail-qk1-x744.google.com with SMTP id u64so5954129qka.12 for ; Tue, 21 Jul 2020 06:23:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lca.pw; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to :user-agent; bh=Vl6nY10QRE8L9pFubciH3YlE4tKocCVwUrJH3mwnBNc=; b=K3+TBw5iH3oRutMINHodqnnpw+ILqFjg71lC1X6m2kb6Md5rz6bvLqUnKDsBG79NFA bD9XgacSi+rNHjzxTKWyhnKE0EwnRmMA0uqL/Gw28wU/WeC/IDenPPdM3k735pwZwOIK zEOGEKF1kGb+e2BHjiP29JtPH14KxtHu0sXS1Og2BxaOFOwU7Y6OySeHBTTgbBXydcai QC/y9HFU3zXTKyAhVHT1FE8b1Ou8A66+ObgK/9fvElBuVI+NYG4hcAQ5/ZgSlWJa0hwe rG9dJpVxutJwZiocT75qbFHgj+QR9L45EGHmHgB1hUQ7BemEam6RjuGqsoFsFHjrw/2l dNJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to:user-agent; bh=Vl6nY10QRE8L9pFubciH3YlE4tKocCVwUrJH3mwnBNc=; b=JgG2LD8Htb+9ZwcM5RcX/foJQ106br7r6v6c1WfLHvnp/Up5LI1G4jT7+YvutJvUcp ZMvK5AD4I9QWFkf4PbpJ9y+zvpK8aWjwIfqtbcpX7Kh1Dzj/3jPmj0XuM1SKfQKTd2GQ tA14q06VSSGF55KylzMSAuQpK0wTVG2P6yh8R71R8Ozep63wVrjadVc3WzfqOD1pna+v tL6a8F5MPllGxipYVY/gRPxb1IkNJ6flengf1MDK6ZSSWS6IOFLcgp5pPZto3Ewe+cvK 7cMh2uARlc8TcD+mglxeiKpHXJweeJiZ109cqJo/cjnZ/QBsdoOWnLyfTrj2Gzp3jy9m cSJA== X-Gm-Message-State: AOAM532aZ93WPnQKJyn+spPEGu4fJaGUCkeWgF71YnrokXx65sCT3Bj6 RVzTSvXmydcGpEZpcvfe/BA2TQ== X-Received: by 2002:ae9:dfc2:: with SMTP id t185mr8700616qkf.49.1595337834564; Tue, 21 Jul 2020 06:23:54 -0700 (PDT) Received: from lca.pw (pool-71-184-117-43.bstnma.fios.verizon.net. [71.184.117.43]) by smtp.gmail.com with ESMTPSA id a5sm22832242qtd.84.2020.07.21.06.23.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Jul 2020 06:23:54 -0700 (PDT) Date: Tue, 21 Jul 2020 09:23:44 -0400 From: Qian Cai To: Michal Hocko Cc: linux-mm@kvack.org, LKML , Andrew Morton , Linus Torvalds , Tim Chen Subject: Re: [RFC PATCH] mm: silence soft lockups from unlock_page Message-ID: <20200721132343.GA4261@lca.pw> References: <20200721112529.GJ4061@dhcp22.suse.cz> <664A07B6-DBCD-4520-84F1-241A4E7A339F@lca.pw> <20200721121752.GK4061@dhcp22.suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20200721121752.GK4061@dhcp22.suse.cz> User-Agent: Mutt/1.10.1 (2018-07-13) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Jul 21, 2020 at 02:17:52PM +0200, Michal Hocko wrote: > On Tue 21-07-20 07:44:07, Qian Cai wrote: > > > > > > > On Jul 21, 2020, at 7:25 AM, Michal Hocko wrote: > > > > > > Are these really important? I believe I can dig that out from the bug > > > report but I didn't really consider that important enough. > > > > Please dig them out. We have also been running those things on > > “large” powerpc as well and never saw such soft-lockups. Those > > details may give us some clues about the actual problem. > > I strongly suspect this is not really relevant but just FYI this is > 16Node, 11.9TB with 1536CPUs system. Okay, we are now talking about the HPC special case. Just brain-storming some ideas here. 1) What about increase the soft-lockup threshold early at boot and restore afterwards? As far as I can tell, those soft-lockups are just a few bursts of things and then cure itself after the booting. 2) Reading through the comments above page_waitqueue(), it said rare hash collisions could happen, so sounds like in this HPC case, it is rather easy to hit those hash collisons. Thus, need to deal with that instead? 3) The commit 62906027091f ("mm: add PageWaiters indicating tasks are waiting for a page bit") mentioned that, "Putting two bits in the same word opens the opportunity to remove the memory barrier between clearing the lock bit and testing the waiters bit, after some work on the arch primitives (e.g., ensuring memory operand widths match and cover both bits)." Do you happen to know if this only happen on powerpc? Also, probably need to dig out if those memory barrier is still there that could be removed to speed up things. > > > Once we > > understand the problem better, we may judge if this “hack” is > > really worth it. > > I do not have access to the machine so I can only judge from the boot > log I have in hands. And from that it is pretty clear that > $ grep BUG tmp/attachment.txt | wc -l > 896 > > $ grep BUG tmp/attachment.txt | grep "\[systemd" | wc -l > 860 > > $ grep do_fault+0x448 tmp/attachment.txt | wc -l > 860 > > that the boot struggles, lockups happen from udev workers and most of > them are stuck at the very same place which is unlock_page. The rest is > a part of the changelog. > -- > Michal Hocko > SUSE Labs