Received: by 2002:ad5:474a:0:0:0:0:0 with SMTP id i10csp3462367imu; Mon, 17 Dec 2018 21:36:56 -0800 (PST) X-Google-Smtp-Source: AFSGD/UqM/7uQSBoJ0jaO4MMb8ueioGiJLMyJTTJt3rFWTxdaoqiqvRGAJwiqQzVeGLxHlvmL+jY X-Received: by 2002:a63:b105:: with SMTP id r5mr14704733pgf.442.1545111416616; Mon, 17 Dec 2018 21:36:56 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1545111416; cv=none; d=google.com; s=arc-20160816; b=eg+OMqiIYwMRr9VN2L9mU40mymngcZAxXBWERMdvMZyb6a6JPXUwOzviWMR1vh8lB+ qtE5Kv9uA/iMt6gBAAvfuvNkyxzsPPVCxlnj2/KsiCHM5jQe1wj4ei6vujzr165JR23a DcGY3ATA4J8PSMHx2L5xN6HjAuhL1HDfXjZyrwe3RB+fRxwZdJ4pBFymYPeRd3ezyjEK maNuhkit07i4VroLQhxKmgpL2kQGVx2SwtTBMqDwmLuJD3EkWXZT0+4I9FmAMtp4IRqV ClRHb0Ewe8D+AP24mnqap2Lp++eeJDCdju9WxhBZHN3DuJ9vsL0eVVBOAAe9M61kWDty 5X/w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:message-id:user-agent:in-reply-to :content-disposition:mime-version:references:reply-to:subject:cc:to :from:date; bh=m3riRWSKeSg22sRv+1k6GTwEBLW5m0uxicvCvNCjAfU=; b=z0w1A7lu03TCSTX+9TJ6gbLXsg3QnXQ+gFAz1YcRQm145fg+Kqm3cfoUGydjATeFUq wSxGiZI7xHiLiHqv3UaSYk13kgD6Y5NjwFzlSX904cbCZiWKmMgMYmoH2BlgnhfYbyR1 RpGwoxn6XH3EAvyiH3XNZfbuXkxl0TveZDa9h1mVDaWxbMJ6cheVZ90KGqZfTQr9j4fw PXDdDbJ0ypdAagy8mUSBeJMU2xZc4/RDksrp+7aVL0x6HUpCrpq9QiLKpNlDBcwsvh6o CaXILmjWnVQq5Zs1g1DVPckcD6PBIAs26LiXy69DHaDcGvk7lKdxUxHrGY48X3Djpdk+ ZBZA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=ibm.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id q5si12218388pgb.245.2018.12.17.21.36.41; Mon, 17 Dec 2018 21:36:56 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726395AbeLRFea (ORCPT + 99 others); Tue, 18 Dec 2018 00:34:30 -0500 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:43152 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726377AbeLRFea (ORCPT ); Tue, 18 Dec 2018 00:34:30 -0500 Received: from pps.filterd (m0098413.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id wBI5XSTB058716 for ; Tue, 18 Dec 2018 00:34:28 -0500 Received: from e13.ny.us.ibm.com (e13.ny.us.ibm.com [129.33.205.203]) by mx0b-001b2d01.pphosted.com with ESMTP id 2pepctser8-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 18 Dec 2018 00:34:28 -0500 Received: from localhost by e13.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 18 Dec 2018 05:34:27 -0000 Received: from b01cxnp22035.gho.pok.ibm.com (9.57.198.25) by e13.ny.us.ibm.com (146.89.104.200) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Tue, 18 Dec 2018 05:34:23 -0000 Received: from b01ledav003.gho.pok.ibm.com (b01ledav003.gho.pok.ibm.com [9.57.199.108]) by b01cxnp22035.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id wBI5YMR614352534 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Tue, 18 Dec 2018 05:34:22 GMT Received: from b01ledav003.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id CBB18B2067; Tue, 18 Dec 2018 05:34:22 +0000 (GMT) Received: from b01ledav003.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 963BDB205F; Tue, 18 Dec 2018 05:34:22 +0000 (GMT) Received: from paulmck-ThinkPad-W541 (unknown [9.85.153.1]) by b01ledav003.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 18 Dec 2018 05:34:22 +0000 (GMT) Received: by paulmck-ThinkPad-W541 (Postfix, from userid 1000) id 1541916C11E1; Mon, 17 Dec 2018 21:34:26 -0800 (PST) Date: Mon, 17 Dec 2018 21:34:26 -0800 From: "Paul E. McKenney" To: "Zhang, Jun" Cc: "He, Bo" , Steven Rostedt , "linux-kernel@vger.kernel.org" , "josh@joshtriplett.org" , "mathieu.desnoyers@efficios.com" , "jiangshanlai@gmail.com" , "Xiao, Jin" , "Zhang, Yanmin" , "Bai, Jie A" , "Sun, Yi J" , "Chang, Junxiao" , "Mei, Paul" Subject: Re: rcu_preempt caused oom Reply-To: paulmck@linux.ibm.com References: <20181213181136.GL4170@linux.ibm.com> <20181214021527.GR4170@linux.ibm.com> <20181214051011.GS4170@linux.ibm.com> <20181214053841.GA16100@linux.ibm.com> <20181217042623.GF4170@linux.ibm.com> <88DC34334CA3444C85D647DBFA962C2735AD64A0@SHSMSX104.ccr.corp.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <88DC34334CA3444C85D647DBFA962C2735AD64A0@SHSMSX104.ccr.corp.intel.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-TM-AS-GCONF: 00 x-cbid: 18121805-0064-0000-0000-0000038808DE X-IBM-SpamModules-Scores: X-IBM-SpamModules-Versions: BY=3.00010241; HX=3.00000242; KW=3.00000007; PH=3.00000004; SC=3.00000271; SDB=6.01133371; UDB=6.00589186; IPR=6.00913539; MB=3.00024728; MTD=3.00000008; XFM=3.00000015; UTC=2018-12-18 05:34:27 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18121805-0065-0000-0000-00003BB834C1 Message-Id: <20181218053426.GP4170@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2018-12-18_02:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1812180049 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Dec 18, 2018 at 02:46:43AM +0000, Zhang, Jun wrote: > Hello, paul > > In softirq context, and current is rcu_preempt-10, rcu_gp_kthread_wake don't wakeup rcu_preempt. > Maybe next patch could fix it. Please help review. > > diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c > index 0b760c1..98f5b40 100644 > --- a/kernel/rcu/tree.c > +++ b/kernel/rcu/tree.c > @@ -1697,7 +1697,7 @@ static bool rcu_future_gp_cleanup(struct rcu_state *rsp, struct rcu_node *rnp) > */ > static void rcu_gp_kthread_wake(struct rcu_state *rsp) > { > - if (current == rsp->gp_kthread || > + if (((current == rsp->gp_kthread) && !in_softirq()) || Close, but not quite. Please see below. > !READ_ONCE(rsp->gp_flags) || > !rsp->gp_kthread) > return; > > [44932.311439, 0][ rcu_preempt] rcu_preempt-10 [001] .n.. 44929.401037: rcu_grace_period: rcu_preempt 19063548 reqwait > ...... > [44932.311517, 0][ rcu_preempt] rcu_preempt-10 [001] d.s2 44929.402234: rcu_future_grace_period: rcu_preempt 19063548 19063552 0 0 3 Startleaf > [44932.311536, 0][ rcu_preempt] rcu_preempt-10 [001] d.s2 44929.402237: rcu_future_grace_period: rcu_preempt 19063548 19063552 0 0 3 Startedroot Good catch! If the rcu_preempt kthread had just entered the function swait_event_idle_exclusive(), which had just called __swait_event_idle() which had just called ___swait_event(), which had just gotten done checking the "condition", then yes, the rcu_preempt kthread could sleep forever. This is a very narrow race window, but that matches your experience with its not happening often -- and my experience with it not happening at all. However, for this to happen, the wakeup must happen within a softirq handler that executes upon return from an interrupt that interrupted ___swait_event() just after the "if (condition)". For this, we don't want in_softirq() but rather in_serving_softirq(), as shown in the patch below. The patch you have above could result in spurious wakeups, as it is checking for bottom halves being disabled, not just executing within a softirq handler. Which might be better than not having enough wakeups, but let's please try for just the right number. ;-) So could you please instead test the patch below? And if it works, could I please have your Signed-off-by so that I can queue it? My patch is quite clearly derived from yours, after all! And you should get credit for finding the problem and arriving at an approximate fix, after all. Thanx, Paul ------------------------------------------------------------------------ diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index e9392a9d6291..b9205b40b621 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -1722,7 +1722,7 @@ static bool rcu_future_gp_cleanup(struct rcu_state *rsp, struct rcu_node *rnp) */ static void rcu_gp_kthread_wake(struct rcu_state *rsp) { - if (current == rsp->gp_kthread || + if ((current == rsp->gp_kthread && !in_serving_softirq()) || !READ_ONCE(rsp->gp_flags) || !rsp->gp_kthread) return;