Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752566AbdHVUyD (ORCPT ); Tue, 22 Aug 2017 16:54:03 -0400 Received: from smtp.codeaurora.org ([198.145.29.96]:56352 "EHLO smtp.codeaurora.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751673AbdHVUx6 (ORCPT ); Tue, 22 Aug 2017 16:53:58 -0400 DMARC-Filter: OpenDMARC Filter v1.3.2 smtp.codeaurora.org B3F6B60274 Authentication-Results: pdx-caf-mail.web.codeaurora.org; dmarc=none (p=none dis=none) header.from=codeaurora.org Authentication-Results: pdx-caf-mail.web.codeaurora.org; spf=none smtp.mailfrom=jhugo@codeaurora.org Subject: Re: [BUG] Deadlock due due to interactions of block, RCU, and cpu offline To: Paolo Bonzini , paulmck@linux.vnet.ibm.com Cc: linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, pprakash@codeaurora.org, Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Jens Axboe , Sebastian Andrzej Siewior , Thomas Gleixner , Richard Cochran , Boris Ostrovsky , Richard Weinberger References: <20170327181711.GF3637@linux.vnet.ibm.com> <20170620234623.GA16200@linux.vnet.ibm.com> <20170621161853.GB3721@linux.vnet.ibm.com> <20170623033456.GA15959@linux.vnet.ibm.com> <20170628001130.GB3721@linux.vnet.ibm.com> <20170630001855.GL2393@linux.vnet.ibm.com> <20170820205658.GS11320@linux.vnet.ibm.com> From: Jeffrey Hugo Message-ID: Date: Tue, 22 Aug 2017 14:53:55 -0600 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.2.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 4197 Lines: 100 On 8/22/2017 10:12 AM, Paolo Bonzini wrote: > On 20/08/2017 22:56, Paul E. McKenney wrote: >>> KVM: async_pf: avoid async pf injection when in guest mode >>> KVM: cpuid: Fix read/write out-of-bounds vulnerability in cpuid emulation >>> arm: KVM: Allow unaligned accesses at HYP >>> arm64: KVM: Allow unaligned accesses at EL2 >>> arm64: KVM: Preserve RES1 bits in SCTLR_EL2 >>> KVM: arm/arm64: Handle possible NULL stage2 pud when ageing pages >>> KVM: nVMX: Fix exception injection >>> kvm: async_pf: fix rcu_irq_enter() with irqs enabled >>> KVM: arm/arm64: vgic-v3: Fix nr_pre_bits bitfield extraction >>> KVM: s390: fix ais handling vs cpu model >>> KVM: arm/arm64: Fix isues with GICv2 on GICv3 migration >>> >>> Nothing really stands out to me which would "fix" the issue. >> >> My guess would be an undo of the change that provoked the problem >> in the first place. Did you try bisecting within the above group >> of commits? >> >> Either way, CCing Paolo for his thoughts? > > There is "kvm: async_pf: fix rcu_irq_enter() with irqs enabled", but it > would have caused splats, not deadlocks. > > If you are using nested virtualization, "KVM: async_pf: avoid async pf > injection when in guest mode" can be a wildcard, but only if you have > memory pressure. > > My bet is still on the former changing the timing just a little bit. > > Paolo > I'm sorry, I must have done the bisect incorrectly. I attempted to bisect the KVM changes from the merge, but was seeing that the issue didn't repro with any of them. I double checked the merge commit, and found it did not introduce a "fix". I redid the bisect, and it identified the following change this time. I double checked that reverting the change reintroduces the deadlock, and cherry-picking the change onto 4.12-rc4 (known to exhibit the issue) causes the issue to disappear. I'm pretty sure (knock on wood) that the bisect result is actually correct this time. commit 6460495709aeb651896bc8e5c134b2e4ca7d34a8 Author: James Wang Date: Thu Jun 8 14:52:51 2017 +0800 Fix loop device flush before configure v3 While installing SLES-12 (based on v4.4), I found that the installer will stall for 60+ seconds during LVM disk scan. The root cause was determined to be the removal of a bound device check in loop_flush() by commit b5dd2f6047ca ("block: loop: improve performance via blk-mq"). Restoring this check, examining ->lo_state as set by loop_set_fd() eliminates the bad behavior. Test method: modprobe loop max_loop=64 dd if=/dev/zero of=disk bs=512 count=200K for((i=0;i<4;i++))do losetup -f disk; done mkfs.ext4 -F /dev/loop0 for((i=0;i<4;i++))do mkdir t$i; mount /dev/loop$i t$i;done for f in `ls /dev/loop[0-9]*|sort`; do \ echo $f; dd if=$f of=/dev/null bs=512 count=1; \ done Test output: stock patched /dev/loop0 18.1217e-05 8.3842e-05 /dev/loop1 6.1114e-05 0.000147979 /dev/loop10 0.414701 0.000116564 /dev/loop11 0.7474 6.7942e-05 /dev/loop12 0.747986 8.9082e-05 /dev/loop13 0.746532 7.4799e-05 /dev/loop14 0.480041 9.3926e-05 /dev/loop15 1.26453 7.2522e-05 Note that from loop10 onward, the device is not mounted, yet the stock kernel consumes several orders of magnitude more wall time than it does for a mounted device. (Thanks for Mike Galbraith , give a changelog review.) Reviewed-by: Hannes Reinecke Reviewed-by: Ming Lei Signed-off-by: James Wang Fixes: b5dd2f6047ca ("block: loop: improve performance via blk-mq") Signed-off-by: Jens Axboe Considering the original analysis of the issue, it seems plausible that this change could be fixing it. -- Jeffrey Hugo Qualcomm Datacenter Technologies as an affiliate of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.