Return-Path: <linux-kernel-owner@vger.kernel.org>
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
	id S1756908AbbBEK7Y (ORCPT <rfc822;w@1wt.eu>);
	Thu, 5 Feb 2015 05:59:24 -0500
Received: from mailout3.w1.samsung.com ([210.118.77.13]:61270 "EHLO
	mailout3.w1.samsung.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
	with ESMTP id S1756062AbbBEK7X (ORCPT
	<rfc822;linux-kernel@vger.kernel.org>);
	Thu, 5 Feb 2015 05:59:23 -0500
X-AuditID: cbfec7f4-b7f126d000001e9a-83-54d34c77ecd7
Message-id: <1423133958.25197.7.camel@AMDC1943>
Subject: Re: [PATCH v2] ARM: Don't use complete() during __cpu_die
From: Krzysztof Kozlowski <k.kozlowski@samsung.com>
To: Mark Rutland <mark.rutland@arm.com>
Cc: Russell King <linux@arm.linux.org.uk>,
        "linux-arm-kernel@lists.infradead.org" 
	<linux-arm-kernel@lists.infradead.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "paulmck@linux.vnet.ibm.com" <paulmck@linux.vnet.ibm.com>,
        Arnd Bergmann <arnd@arndb.de>,
        Bartlomiej Zolnierkiewicz <b.zolnierkie@samsung.com>,
        Marek Szyprowski <m.szyprowski@samsung.com>,
        Stephen Boyd <sboyd@codeaurora.org>,
        Catalin Marinas <Catalin.Marinas@arm.com>,
        Will Deacon <Will.Deacon@arm.com>
Date: Thu, 05 Feb 2015 11:59:18 +0100
In-reply-to: <20150205105327.GC11344@leverpostej>
References: <1423131270-24047-1-git-send-email-k.kozlowski@samsung.com>
 <20150205105327.GC11344@leverpostej>
Content-type: text/plain; charset=UTF-8
X-Mailer: Evolution 3.10.4-0ubuntu2
MIME-version: 1.0
Content-transfer-encoding: 7bit
X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrHLMWRmVeSWpSXmKPExsVy+t/xa7rlPpdDDL4vFLD4O+kYu8XGGetZ
	Ld4v62G02PT4GqvF5V1z2CxuX+a1WHvkLrvF0usXmSzebv7OavHjTDeLxcuPJ1gcuD3WzFvD
	6NHS3MPm8fvXJEaPy329TB4PDm1m8di8pN6jb8sqRo/Pm+QCOKK4bFJSczLLUov07RK4Mja+
	/8Je8FGpYs2Z98wNjA+kuhg5OSQETCTuXr3GBGGLSVy4t56ti5GLQ0hgKaPEwr0PmCCcz4wS
	b+48AXI4OHgF9CVO/QwEaRAWcJKY9OAWO4jNJmAssXn5EjYQW0RAXaJn1xcWkF5mge/MEsfv
	rwHbwCKgKnGpq4MVxOYUMJSY8uMEWLOQQJ7E4lvfwWxmoOZJ8xYxg+ySEFCWaOx3AwnzCghK
	/Jh8jwWiRF5i85q3zBMYBWYh6ZiFpGwWkrIFjMyrGEVTS5MLipPScw31ihNzi0vz0vWS83M3
	MULi5MsOxsXHrA4xCnAwKvHwNrRdDBFiTSwrrsw9xCjBwawkwrvF63KIEG9KYmVValF+fFFp
	TmrxIUYmDk6pBkaORYdal/q5qu19+rJUiLNqhtoiPr06/26dVNlnh1oyZxg4npgxecFEe6kt
	X4+Zhj5O0syc9TMtO2iVq+eNXBml7qZVv+1agzsML7Yt3XC0P/pYxqrtFfLuX9fvrLq4ZlvD
	9s+H5zQ9mHjzgqXw1fUXpW11dWXfy5mo3LjCvD/q8r7EbYektS4psRRnJBpqMRcVJwIArpgW
	+3ECAAA=
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
Content-Length: 4616
Lines: 99

On czw, 2015-02-05 at 10:53 +0000, Mark Rutland wrote:
> On Thu, Feb 05, 2015 at 10:14:30AM +0000, Krzysztof Kozlowski wrote:
> > The complete() should not be used on offlined CPU. Rewrite the
> > wait-complete mechanism with wait_on_bit_timeout().
> > 
> > The CPU triggering hot unplug (e.g. CPU0) will loop until some bit is
> > cleared. In each iteration schedule_timeout() is used with initial sleep
> > time of 1 ms. Later it is increased to 10 ms.
> > 
> > The dying CPU will clear the bit which is safe in that context.
> > 
> > This fixes following RCU warning on ARMv8 (Exynos 4412, Trats2) during
> > suspend to RAM:
> 
> Nit: isn't Exynos4412 a quad-A9 (ARMv7 rather than ARMv8)?

Yes, it should be ARMv7. However still this should be fixed for both
architectures.

> 
> > [   31.113925] ===============================
> > [   31.113928] [ INFO: suspicious RCU usage. ]
> > [   31.113935] 3.19.0-rc7-next-20150203 #1914 Not tainted
> > [   31.113938] -------------------------------
> > [   31.113943] kernel/sched/fair.c:4740 suspicious rcu_dereference_check() usage!
> > [   31.113946]
> > [   31.113946] other info that might help us debug this:
> > [   31.113946]
> > [   31.113952]
> > [   31.113952] RCU used illegally from offline CPU!
> > [   31.113952] rcu_scheduler_active = 1, debug_locks = 0
> > [   31.113957] 3 locks held by swapper/1/0:
> > [   31.113988]  #0:  ((cpu_died).wait.lock){......}, at: [<c005a114>] complete+0x14/0x44
> > [   31.114012]  #1:  (&p->pi_lock){-.-.-.}, at: [<c004a790>] try_to_wake_up+0x28/0x300
> > [   31.114035]  #2:  (rcu_read_lock){......}, at: [<c004f1b8>] select_task_rq_fair+0x5c/0xa04
> > [   31.114038]
> > [   31.114038] stack backtrace:
> > [   31.114046] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 3.19.0-rc7-next-20150203 #1914
> > [   31.114050] Hardware name: SAMSUNG EXYNOS (Flattened Device Tree)
> > [   31.114076] [<c0014ce4>] (unwind_backtrace) from [<c0011c30>] (show_stack+0x10/0x14)
> > [   31.114091] [<c0011c30>] (show_stack) from [<c04dc048>] (dump_stack+0x70/0xbc)
> > [   31.114105] [<c04dc048>] (dump_stack) from [<c004f83c>] (select_task_rq_fair+0x6e0/0xa04)
> > [   31.114118] [<c004f83c>] (select_task_rq_fair) from [<c004a83c>] (try_to_wake_up+0xd4/0x300)
> > [   31.114129] [<c004a83c>] (try_to_wake_up) from [<c00598a0>] (__wake_up_common+0x4c/0x80)
> > [   31.114140] [<c00598a0>] (__wake_up_common) from [<c00598e8>] (__wake_up_locked+0x14/0x1c)
> > [   31.114150] [<c00598e8>] (__wake_up_locked) from [<c005a134>] (complete+0x34/0x44)
> > [   31.114167] [<c005a134>] (complete) from [<c04d6ca4>] (cpu_die+0x24/0x84)
> > [   31.114179] [<c04d6ca4>] (cpu_die) from [<c005a508>] (cpu_startup_entry+0x328/0x358)
> > [   31.114189] [<c005a508>] (cpu_startup_entry) from [<40008784>] (0x40008784)
> > [   31.114226] CPU1: shutdown
> > 
> > Signed-off-by: Krzysztof Kozlowski <k.kozlowski@samsung.com>
> > 
> > ---
> > Changes since v1:
> > 1. Use adaptive sleep time when waiting for CPU die (idea and code
> >    from Paul E. McKenney). Paul also acked the patch but I made evem more
> >    changes.
> > 
> > 2. Add another bit (CPU_DIE_TIMEOUT_BIT) for synchronizing power down
> >    failure in case:
> >     CPU0 (killing)                           CPUx (killed)
> >      wait_for_cpu_die
> >      timeout
> >                                               cpu_die()
> >                                                clear_bit()
> >                                                self power down
> > 
> >    In this case the bit would be cleared and CPU would be powered down
> >    introducing wrong behavior in next power down sequence (CPU0 would
> >    see the bit cleared).
> >    I think that such race is still possible but was narrowed to very
> >    short time frame. Any CPU up will reset the bit to proper values.
> 
> In the case of shutting down 2 CPUs in quick succession (without an
> intervening boot of a CPU), surely this does not solve the potential
> race on the wait_cpu_die variable?

Right, the race is not fully fixed.

> 
> I think we instead need a percpu synchronisation variable, which would
> prevent racing on the value between CPUs, and a CPU would have to be
> brought up before we could decide to kill it again. With that I think we
> only need a single bit, too.

You mean a single bit-value per cpu?

Best regards,
Krzysztof
> 
> Thanks,
> Mark.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/