Received: by 2002:ac0:a581:0:0:0:0:0 with SMTP id m1-v6csp1161418imm; Fri, 22 Jun 2018 11:21:09 -0700 (PDT) X-Google-Smtp-Source: ADUXVKJ1nXcOaZ0opXawPc+XwAnSmdTrAH1ZWjRp8++9cdY/CW9TeHrpyjPrEyjeH6UORMe08Qb5 X-Received: by 2002:a62:3b5d:: with SMTP id i90-v6mr2941367pfa.24.1529691669885; Fri, 22 Jun 2018 11:21:09 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1529691669; cv=none; d=google.com; s=arc-20160816; b=QCc5eyjLJN/TByfeegIRtthjq6IRuZ2ykjJJKMfh7m+IrshTXBJWJEx9LeOouw657u MXEZDpUJCWFQWw24lt0hTKWaeIP2lovhHO6P6cD/6vUoXWmFhhRf199Ktp4010ezHRiE xnvu6fB7WmSMuqqqyeFIF+n4DaVsvP48+yaPqTpa2/XgfWaX1D2gD2L53XZ3jCanPt9W dlii4k3T35BpwChYcckUie6yoojSxCMUJFcR8aFubLkSHJoDwePKfktk3VDJwtajDSIr uqMSReCerhRU+mUkniVBYVZE+OJcuioJMU8C6t8WFPGAaMT9+CMSwMT+MVmR+m+/Z3Rr +Zig== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:user-agent:in-reply-to :content-disposition:mime-version:references:message-id:subject:cc :to:from:date:dkim-signature:arc-authentication-results; bh=N6w/C3eeYoLAa2v/BhFfHURLHlMISlkJH+8ytQR2BXY=; b=AjvU8QB2umM11vUaj06trs0futJTVOJteHFSeMxjzGPRljwUo2LZ2IquRobsdC1nhS 1deLYncew/7wBXNAVFsgopOARaojp07hb/clJJ2QcxViZrEEU6ECvTAJHOmI6medZWjN FWQlqfgze6kOegNiJoIVdEgpOERiOTAsTHDuDgF7Rmw497aM1vVORjV/BHjr1LiKAHYA Ro6uNVVdHeIFsREBLSabXJWfh7d6WjFHlpYL+GmUFTGJ/Gq1eLk7ZcUs4BGQ7d31F2J5 0/t2h7PyJ7PmW7bfkBD6SeYgFM/yaZKGAnV2D5u42oflozHQBxwXzJ2n2Lgol43oIjLG 7Xbg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=vCiNYMgS; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id b94-v6si8774870plb.135.2018.06.22.11.20.54; Fri, 22 Jun 2018 11:21:09 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=vCiNYMgS; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S934027AbeFVSTU (ORCPT + 99 others); Fri, 22 Jun 2018 14:19:20 -0400 Received: from mail-pl0-f67.google.com ([209.85.160.67]:44203 "EHLO mail-pl0-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S933691AbeFVSTT (ORCPT ); Fri, 22 Jun 2018 14:19:19 -0400 Received: by mail-pl0-f67.google.com with SMTP id z9-v6so3855418plk.11 for ; Fri, 22 Jun 2018 11:19:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=N6w/C3eeYoLAa2v/BhFfHURLHlMISlkJH+8ytQR2BXY=; b=vCiNYMgSLjLA2jkDHIS1pPVxdcIDpsKpV5eg3DnaFBTqbWG8sXxm9EJjl3EsVYkRkt DSTpGyjnZ3M9gYb2IHj48d8ocd6rKj1ENHCZK2fcJ3/2fjq8oLhABCA8wTEeFY4bf1cd IrKDoCv8lz9zLmZ6N7XgtPpJiWvDnfpYJGTDQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=N6w/C3eeYoLAa2v/BhFfHURLHlMISlkJH+8ytQR2BXY=; b=ga2oNbeoTbf61syVhXkmnw/f5HPapLUkYl3oKSEeNrdrJ0Upn5NnQnx/vjTn362OsF 6RJAVl4NJfJRUjVFqLgxsKKdaY8qm8TelHORZomdykrFtSR8o/CX0v/zAl01NQuw6XpT Gc5d9Xn0eINkhzsTxyl127/A8tfS1qd2BaUlqkcsALt77iYbPian2gnLlK7dkHZcxUnF xulhUyuSu8o2+5qYKMEoHYjUT79pb8UAmTAwUpe8H8w/70/a4UBZYu6z3m8DF3wqSKM+ l404/0TroEfFko3tlBHOPAft6jmy9UXifKNz1dChe+qQn+gm/UAov4zqdITHn3OD4/ce sz7Q== X-Gm-Message-State: APt69E2G3YydEEkaBoR2JXyijZ7S/ykmG6fS2oUToY9BzmIrxPFsNezL z/F5Kx0iaIeiefqxRIWFLh9peQ== X-Received: by 2002:a17:902:7d89:: with SMTP id a9-v6mr2754107plm.238.1529691558494; Fri, 22 Jun 2018 11:19:18 -0700 (PDT) Received: from localhost ([2620:0:1000:1600:3122:ea9c:d178:eb]) by smtp.gmail.com with ESMTPSA id y14-v6sm9781941pgv.37.2018.06.22.11.19.17 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 22 Jun 2018 11:19:17 -0700 (PDT) Date: Fri, 22 Jun 2018 11:19:16 -0700 From: Joel Fernandes To: "Paul E. McKenney" Cc: Byungchul Park , Byungchul Park , jiangshanlai@gmail.com, josh@joshtriplett.org, Steven Rostedt , Mathieu Desnoyers , linux-kernel@vger.kernel.org, kernel-team@lge.com, luto@kernel.org Subject: Re: [RFC 2/2] rcu: Remove ->dynticks_nmi_nesting from struct rcu_dynticks Message-ID: <20180622181916.GA13628@joelaf.mtv.corp.google.com> References: <1529484440-20634-1-git-send-email-byungchul.park@lge.com> <1529484440-20634-2-git-send-email-byungchul.park@lge.com> <20180620145814.GQ3593@linux.vnet.ibm.com> <20180620164902.GW3593@linux.vnet.ibm.com> <20180622055659.GA255098@joelaf.mtv.corp.google.com> <20180622132843.GN3593@linux.vnet.ibm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20180622132843.GN3593@linux.vnet.ibm.com> User-Agent: Mutt/1.9.2 (2017-12-15) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, Jun 22, 2018 at 06:28:43AM -0700, Paul E. McKenney wrote: > On Thu, Jun 21, 2018 at 10:56:59PM -0700, Joel Fernandes wrote: > > Hi Paul, > > > > On Wed, Jun 20, 2018 at 09:49:02AM -0700, Paul E. McKenney wrote: > > > On Thu, Jun 21, 2018 at 01:05:22AM +0900, Byungchul Park wrote: > > > > On Wed, Jun 20, 2018 at 11:58 PM, Paul E. McKenney > > > > wrote: > > > > > On Wed, Jun 20, 2018 at 05:47:20PM +0900, Byungchul Park wrote: > > > > >> Hello folks, > > > > >> > > > > >> I'm careful in saying that ->dynticks_nmi_nesting can be removed but I > > > > >> think it's possible since the only thing we are interested in with > > > > >> regard to ->dynticks_nesting or ->dynticks_nmi_nesting is whether rcu is > > > > >> idle or not. > > > > > > > > > > Please keep in mind that NMIs cannot be masked, which means that the > > > > > rcu_nmi_enter() and rcu_nmi_exit() pair can be invoked at any point in > > > > > the process, between any consecutive pair of instructions. The saving > > > > > > And yes, I should have looked at this patch more closely before replying. > > > But please see below. > > > > > > > I believe I understand what NMI is and why you introduced > > > > ->dynticks_nmi_nesting. Or am I missing something? > > > > > > Perhaps the fact that there are architectures that can enter interrupt > > > handlers and never leave them when the CPU is non-idle. One example of > > > this is the usermode upcalls in the comment that you removed. > > > > I spent some time tonight and last night trying to understand this concept of > > never leaving an interrupt, I hope you don't mind me asking this dumb > > question... perhaps I will learn something : Could you let me know how is it > > possible that an interrupt never exits? > > > > Typically an interrupt never exiting sounds like a hard-lockup. This is how > > hardlock detector works: Since regular interrupts in linux can't nest, the > > hardlockup detector checks if hrtimer interrupts are being handled and if > > not, then it throws a splat, panics the kernel etc. So I am a bit troubled by > > this interrupt never exiting concept.. > > > > Further since an interrupt is an atomic context, it cannot sleep or schedule > > into usermode so how are these upcalls handled from the interrupt? > > It has been some years since I traced the code flow, but what happened No problem, thanks for the discussion. :) > back then is that it switches itself from an interrupt handler to not > without actually returning from the interrupt. This can only happen when > interrupting a non-idle process, thankfully, and RCU's dyntick-idle code > relies on this restriction. If I remember correctly, the code ends up > executing in the context of the interrupted process, but it has been some > years, so please apply appropriate skepticism. Sure. So in a later thread you mentioned "usermode helpers". I took a closer look at that subsystem, and it seems you can execute usermode helpers from atomic sections with help of UMH_NO_WAIT flag. Then I checked where this flag is used and it turns out its from the mce_work_trigger function in x86/kernel/cpu/mcheck/dev-mcelog.c which can be called infact from an interrupt context (mce_notify_irq). Is this the usecase you remember causing this weird transitions to userspace? > Please take a look at the "Interrupts and NMIs" section of the file > Documentation/RCU/Design/Requirements/Requirements.html for a bit > more information. Sure, thanks for the pointer. > > Lastly, can you point me to an example how the rcu_nmi_enter/exit() pair can go > > out sync? That is they aren't paired and nested properly? In my mind they > > always should be but I may be missing the usecase. I'm happy to try and > > reproduce and trace this if you can let me know how to so that I can study > > it better. > > I have never seen NMIs be unpaired or improperly nested. However, > given that rcu_irq_enter() invokes rcu_nmi_enter() and rcu_irq_exit() > invokes rcu_nmi_exit(), it is definitely the case that rcu_nmi_enter() > and rcu_nmi_exit() need to deal with unpaired and improperly nested > invocations. Just wondering how would the fact that rcu_irq_enter calls into rcu_nmi_enter cause an improper nesting? Just to define what "improper nesting" means, if we can go through an example. Do you mean a scenario like? rcu_nmi_enter (called because of NMI) rcu_nmi_enter (called because of IRQ) rcu_nmi_exit (called because of NMI) rcu_nmi_exit (called because of IRQ) Such scenario seems impossible to me because the IRQ can't be entered after the NMI entered. On the other hand, if you meant that when IRQ is being handled, an NMI fires just before the rcu_irq_enter calls rcu_nmi_enter, then the worst that could happen seems to be that the rcu_nmi_enter/exit pairs will not be nested within the outer rcu_nmi_enter/exit pair even though the NMI interrupted the IRQ. So it'll be something like: rcu_nmi_enter (called because of NMI) rcu_nmi_exit (called because of NMI) rcu_nmi_enter (called because of IRQ) rcu_nmi_exit (called because of IRQ) Even though what really happened in the real world is: IRQ entered NMI entered NMI exited IRQ exited This also seems reasonable to me, but is this what you meant by improper nesting of the rcu_nmi_enter/exit? If yes, what makes it unreasonable? > So why this function-call structure? Well, you see, NMI handlers can > take what appear to RCU to be normal interrupts... > > (And I just added that fun fact to Requirements.html.) Yes, I'll definitely go through all the interrupt requirements in the doc and thanks for referring me to it. thanks, - Joel